Vision-language datasets need interaction coverage, not image volume

Vision-language datasets need interaction coverage, not image volume. A million image-caption pairs can teach broad alignment while leaving a model brittle at the moments a product actually depends on: following a grounded instruction, reading small text, comparing regions, refusing an unsupported inference, or carrying visual evidence through several reasoning steps.

The useful unit is not the image. It is the interaction between an image, an instruction, the evidence needed to answer, and the expected response behavior. The LLaVA-Instruct-158K design makes this visible by dividing examples among conversation, detailed description, and complex reasoning. Those are not cosmetic prompt variants. They exercise different transformations of visual evidence. A dataset dominated by descriptions may produce fluent captions while undertraining the dialogue and reasoning patterns used in an assistant.

Coverage therefore begins with a task taxonomy tied to the intended operating envelope. I want to count interactions that require localization, OCR, attribute recognition, counting, comparison, spatial relations, multi-image reasoning, and calibrated abstention. I also want slices for clutter, occlusion, low resolution, unfamiliar layouts, and adversarially plausible distractors. The aggregate row count says almost nothing about whether these seams are represented.

Caption richness matters, but only in context. ShareGPT4V used direct vision-capable generation to produce substantially longer descriptions than ordinary COCO captions. Richer language can expose objects, relationships, and scene structure that short captions omit. Yet length is not coverage. Repeatedly describing similar scenes in ornate prose can still leave document images, symbolic diagrams, and precise region references underrepresented. Training-data mixtures are product priorities expressed as weights; a VLM mixture should make those interaction priorities explicit.

Construction pipelines also create their own blind spots. Text-only generation from symbolic image representations can seed useful conversations and region-aware questions, but its output is bounded by what the representation preserved. Direct multimodal generation can see more of the image, while inheriting the generator’s omissions and confident mistakes. I would keep provenance for the source image, representation, generator, prompt recipe, and review stage so later evaluation can identify which pipeline produced a failure pattern.

Filtering must protect diversity rather than merely raise an average quality score. CLIP similarity can remove mismatched pairs; perceptual hashes can reduce near-duplicates; noun-phrase frequencies can reveal vocabulary collapse. Each filter also changes the distribution. A fixed similarity threshold may reject unusual but valid relations, and aggressive deduplication may erase rare interaction forms along with redundancy. Deduplication allocates training capacity, so its acceptance criteria should be measured against task coverage, not storage saved.

I would publish a dataset contract with counts and quality evidence by interaction type, source, generation method, modality, language, and risk slice. The contract should include contamination checks, licensing and provenance, annotator or generator disagreement, and held-out evaluations that mirror those same slices. Dataset previews are governance interfaces when they let a reviewer inspect representative examples and known failure boundaries before training begins.

There is one precise concession: for representation pretraining, broad image-text volume can be the right first objective because the task is alignment rather than instruction following. But once a model is tuned for interactive use, volume is no substitute for a deliberate map of the interactions it must perform and the failures it must contain.

I would choose a smaller dataset with visible coverage gaps over a larger one whose only quality claim is scale. The first can be improved deliberately. The second hides product assumptions inside an undifferentiated count—and makes every missing interaction look like a mysterious model problem later.