What Matters in Data Curation for Multimodal Reasoning? Insights from the DCVLR Challenge

Some of the observations founded are :

i. Difficulty based example selection is the dominant driver of performance:
Selecting challenging but learnable examples yields the largest gains in multimodal reasoning accuracy, outperforming other curation strategies.

ii. Increasing dataset size does not reliably improve mean accuracy:
Once a well aligned base dataset is chosen, larger datasets mainly reduce run to run variance rather than boosting average performance.

iii. Data curation operates in a saturation regime:
Most performance improvements come from a relatively small number of carefully curated examples, with diminishing returns from adding more data.

iv. Common diversity heuristics provide little or no benefit:
Techniques such as clustering based diversity, category balancing, and synthetic augmentation often fail to improve performance and can even degrade accuracy.

v. Alignment between dataset, benchmark, and base model is crucial:
Strong alignment amplifies the effectiveness of difficulty filtering and explains why compact, well aligned datasets can outperform larger but less aligned ones.