Figure 1. Existing chart VQA models struggle with complex reasoning questions. We fine-tune an LLM-based data generator that automatically generates question-answer annotations given a chart image. Our key innovation is Synthesize Step-by-Step, which breaks complex questions into easy steps that are solved using external tools. Training with LLM-augmented data (LaMenDa) greatly enhances chart VQA models, improving accuracy from 38% to 54% on human-written ChartQA questions.
Understanding data visualizations like charts and plots requires reasoning about both visual elements and numerics. Although strong in extractive questions, current chart visual question answering (chart VQA) models suffer on complex reasoning questions. In this work, we address the lack of reasoning ability by data augmentation. We leverage Large Language Models (LLMs), which have shown to have strong reasoning ability, as an automatic data annotator that generates question-answer annotations for chart images. The key innovation in our method lies in the Synthesize Step-by-Step strategy: our LLM-based data generator learns to decompose the complex question into step-by-step sub-questions (rationales), which are then used to derive the final answer using external tools, i.e. Python. This step-wise generation procedure is trained on synthetic data generated using a template-based QA generation pipeline. Experimental results highlight the significance of the proposed step-by-step generation. By training with the LLM-augmented data (LaMenDa), we significantly enhance the chart VQA models, achieving the state-of-the-art accuracy on the ChartQA and PlotQA datasets. In particular, our approach improves the accuracy of the previous state-of-the-art approach from 38% to 54% on the human-written questions in the ChartQA dataset, which needs strong reasoning. We hope our work underscores the potential of synthetic data and encourages further exploration of data augmentation using LLMs for reasoning-heavy tasks.
LLM-based data generator decomposes complex chart questions into atomic sub-questions with executable rationales, using external tools like Python to derive answers.
Large-scale LLM-augmented dataset with step-by-step annotations, generated across ChartQA, PlotQA, and chart captioning datasets (1.6M+ QA pairs).
Achieves new SOTA on ChartQA and PlotQA. Improves human-written question accuracy from 38% to 54% (+16 points) on ChartQA.
Our LLM-based data generator takes three inputs: (1) projected image features from a ViT encoder, (2) the underlying data table predicted by DePlot, and (3) a natural language prompt. The model generates complex reasoning questions with step-by-step rationales that are executed as Python programs to derive answers.
Instead of generating question-answer pairs directly (straight-forward), our step-by-step approach trains the LLM to first generate the question, then decompose it into atomic sub-questions (rationales). Each rationale is an executable program — either an extractive VQA call or a Python operation (avg, sum, comparison, etc.). The rationale program is then executed to derive the final answer, ensuring correctness.
To train the step-by-step generator, we construct a template-based QA corpus using 28 manually created question templates. These templates are instantiated by querying rich SVG annotations of chart images, producing 357K question-answer pairs with executable rationales. This template-generated corpus teaches the LLM how to decompose questions step-by-step.
| Model | Avg | Human | Augmented |
|---|---|---|---|
| MatCha | 64.2 | 38.2 | 90.2 |
| Pix2Struct | 56.0 | 30.5 | 81.6 |
| DePlot+FlanPaLM+Codex | 79.3 | 67.6 | 91.0 |
| Pix2Struct + LaMenDa (Ours) | 66.82 | 51.67 | 81.98 |
| MatCha + LaMenDa (Ours) | 72.64 | 53.92 | 91.36 |
Relaxed accuracy (%) on ChartQA test split. With our generated data, both Pix2Struct and MatCha achieve significantly better performance on human-written questions requiring complex reasoning. Note: DePlot+FlanPaLM+Codex relies on much larger models (540B) and external LLMs.
+16%
Human-written accuracy improvement on ChartQA
72.6%
Best relaxed accuracy on ChartQA (SOTA for end-to-end models)
92.9%
SOTA relaxed accuracy on PlotQA
| Model | Avg | V1 | V2 |
|---|---|---|---|
| MatCha | 91.5 | 92.3 | 90.7 |
| MatCha + LaMenDa (Ours) | 92.89 | 93.94 | 91.84 |
Relaxed accuracy (%) on PlotQA test split. Our augmented data improves performance even on this large synthetic dataset.
Video coming soon
If you find this work useful, please cite our paper:
@inproceedings{li2024synthesize,
title={Synthesize Step-by-Step: Tools, Templates and LLMs as Data Generators for Reasoning-Based Chart VQA},
author={Li, Zhuowan and Jasani, Bhavan and Tang, Peng and Ghadar, Shabnam},
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
pages={13613--13623},
year={2024}
}