Decanting the web…
COM-480 · Data Visualization · EPFL
Every large language model is what it eats. Before a model can write, it must read, and most of what it reads comes from FineWeb, a dataset distilled from fifteen years of crawling the open internet.
FineWeb is a 15-trillion-token English dataset released by Hugging Face, built by carefully filtering 96 snapshots of Common Crawl. It is one of the cleanest large-scale corpora used to pre-train modern LLMs.
We explored the public sample-10BT subset by streaming
200,000 documents and measuring them.
This is the story those measurements tell, from the raw crawl, through the
filtering funnel, to the shape of what survives.
Scroll to begin ↓
Common Crawl has archived the public internet since 2013: 96 monthly snapshots of raw HTML in every language. That's roughly 100T tokens of unfiltered noise.
The trafilatura extractor strips menus, ads and markup, keeping only
the main article text. Nearly half the volume disappears here.
A URL block-list, a fastText language classifier (English score ≥ 0.65) and MassiveText quality rules remove non-English and obvious junk.
The web repeats itself endlessly. Per-snapshot MinHash de-duplication is the single biggest cut, collapsing near-identical pages.
Heuristics from Google's C4 dataset drop lines without real sentence structure (e.g. missing terminal punctuation).
A final set of custom line-length and repetition filters removes list-spam and boilerplate the earlier stages missed.
What remains is FineWeb: about 15T high-quality English tokens; roughly 85% of the raw crawl was discarded. Hover any stage to see what it removes.
Token volume surviving each filtering stage (trillions)
Each document is measured in GPT-2 tokens. Most pages are short, so the histogram piles up near the left.
The median document is only about 340 tokens, a few short paragraphs. Half of the web is shorter than this.
On a logarithmic axis the distribution becomes a clean bell: document length is roughly log-normal, a signature of organic web content.
A small number of pages are enormous, thousands of tokens long. These few giants (highlighted) carry a disproportionate share of the text.
Documents by length · toggle the scale yourself
A fastText classifier rates how confidently English each page is, from 0 to 1. FineWeb keeps only pages scoring at least 0.65.
92% of surviving documents score ≥ 0.95. The filter is permissive, yet what remains is almost entirely clear English.
The tail below 0.95 (highlighted) is the interesting part: code-heavy pages, multilingual fragments and borderline text that slipped past the threshold.
How confidently English is each document?
FineWeb spans 2013 to 2024, but it is not evenly spread. The web grew, more snapshots were kept, and recent years contribute far more documents.
Measured in tokens, the recency bias is even stronger. Switch the metric, and the last few years hold most of the text.
Zoom in to every individual Common-Crawl dump, coloured from oldest to newest. Use the buttons to explore any metric and granularity.
Documents · tokens · length, by year or snapshot
Each bubble is one of the web's most-crawled domains, sized by how many documents it contributes.
Colour reveals the mix: reference and wikis, news, blogs and CMS platforms, software, government and academia. Click a legend swatch to isolate one.
Re-sizing by total tokens reshuffles the galaxy, because some domains have few but very long documents.
These giants are the exception. The full sample spans 70,000 distinct domains. Search any domain to find it.
Size = volume · colour = category · search & filter
The Lorenz curve plots cumulative documents against cumulative domains. The further it bows from the diagonal, the more concentrated the data.
Most domains contribute a single page; a minority contribute thousands. Hover the curve to read any share.
Counted in tokens it is even starker: only 300 domains supply a quarter of all text, and 3,900 supply half.
Lorenz curve & cumulative token coverage
Crossing language score with document length shows where FineWeb's documents actually live. Warmer cells hold more documents.
The dataset is dominated by one region: confident English (score ≥ 0.95) of medium length. This is the heart of the corpus.
The shortest-document column spreads across more score bins, because with little text, the classifier has less signal to judge.
Document counts across the quality grid
FineWeb is the invisible substrate of modern AI. Decanting it reveals a corpus that is short, overwhelmingly recent, dominated by a small number of domains, and ruthlessly filtered, yet still vast and diverse.
Of 100T raw tokens, only 15T survive; deduplication is the biggest cut.
About 340 tokens, a few short paragraphs, with a heavy log-normal tail.
A Gini of 0.57; only 300 domains supply a quarter of all tokens.
92% of documents score ≥ 0.95 on the fastText English classifier.
The next time a model answers you, remember: it learned to speak from a distilled, filtered, deduplicated echo of the open web, and this is its shape.
Figures are computed from a 200,000-document sample.
The site ships pre-computed aggregates faithful to the Milestone-1 EDA; run
scripts/prepare_data.py to regenerate them from the live FineWeb stream.