Cohere Labs finds cultural diversity narrows sharply from pretraining to post-training data

Cohere Labs published research showing that the cultural breadth present in large language model pretraining data is substantially stripped out by the time it reaches post-training, a pattern the team calls "the culture funnel."

What's new

Analyzing more than 5.6 million training samples across pipeline stages, Cohere Labs found that "cultural diversity narrows as data moves from pretraining into post-training. We call this phenomenon the culture funnel." The team reports that "pretraining data has much more cultural grounding, i.e. a higher percentage of data tagged with cultural markers, than any post-training dataset" — meaning the instruction-tuning and alignment stages that shape a model's actual behavior draw on a narrower, less culturally grounded slice of data than the corpus the model was originally trained on.

The research also complicates a common assumption about multilingual coverage: "scaling multilinguality alone does not guarantee culturally diverse representation: geolocation coverage remains highly uneven, with a small number of regions disproportionately represented." In other words, adding more languages to a training mix does not by itself fix uneven representation of the cultures and regions those languages are spoken in. Post-training datasets, the team found, increasingly prioritize technical domains such as mathematics and coding — categories that carry comparatively few cultural signals — squeezing out the broader cultural grounding present earlier in the pipeline.

Context

The finding lands in an active debate about why aligned chat models often behave more culturally flattened or default to a narrow (frequently US-centric) frame of reference than their pretraining data would suggest is necessary. Cohere Labs, the research arm behind Cohere's multilingual Aya models, has repeatedly focused on multilingual and cross-cultural representation as a research priority distinct from raw language coverage. This work extends that focus by isolating where in the pipeline cultural signal is lost, rather than simply documenting that gaps exist in a shipped model's outputs.

Why it matters

The practical implication the researchers draw is that better cultural representation will not emerge as a side effect of scale: "cultural capabilities in LLMs will not emerge automatically from scaling up data alone, but from intentionally designing data pipelines that make the many dimensions of culture visible, represented, and learnable." For labs building post-training and RLHF pipelines, the study is effectively a diagnostic — it identifies post-training curation, not pretraining corpus selection, as the stage most responsible for whichever cultural blind spots show up in a deployed model. That reframes cultural representation as a data-engineering problem for the alignment stage specifically, rather than something fixed purely by pretraining on more languages or more data.