Buckets:

3.97 TB
22,586 files
Updated 25 days ago
Name
Size
_manifest
done
ia
probes
reports
README.md4.86 kB
xet
README.md

Encyclopaedia Britannica — Internet Archive scan mirror

A mirror of public-domain Encyclopaedia Britannica scan masters from the Internet Archive, covering editions 1st (1768–71) through 14th (1929) — everything out of copyright — from dozens of contributing libraries.

What's here

Layer Path Contents
Data ia/<ia-identifier>/<files> Scan masters (*_jp2.zip/.tar, *_images.zip, TIFF) + OCR layer (*_chocr.html.gz, *_djvu.xml, *_djvu.txt) + provenance (*_scandata.xml, *_meta.xml)
Per-item docs ia/<ia-identifier>/README.md + _item.json What the item is: title, edition, year, contributor library, license, link back to IA
Catalog _manifest/catalog.parquet / catalog.jsonl One row per file: path, size, md5, class, edition, contributor, license, IA URL
Worklists _manifest/shard-{0..3}.jsonl The exact mirror worklist (used by the mirror jobs)
Progress done/<ia-id>/<file>.json Per-file completion markers (size, md5, timing)
Reports reports/shard-*.json Per-shard run summaries incl. any failures

Scope decisions

  • Mirrored: scan masters + one OCR layer. These are the irreplaceable bits.
  • Not mirrored: IA's derived formats (PDF, EPUB, plain hocr.html, abbyy) — regenerable from the masters if ever needed. This is ~1.9 TB deliberately skipped.
  • Not mirrored: anything access-restricted on IA (lending-only, mostly 15th edition era, 1931+). ~1.4 TB, legally unavailable for bulk download.
  • Not mirrored: 27 open-but-in-copyright or off-topic items (post-1930 yearbooks/printings, modern article extracts, books about Britannica). 8 of these were removed in a second pass (2026-08-25): the first screen keyed on IA's year field, and items with no yearGreat Books of the Western World (1952) and Book of the Year 1938 among them — passed it unreviewed. Every blank-year item has since been checked by hand. Open status on IA ≠ public domain — some uploads carry uploader-applied CC licenses that don't bind the actual rightsholder. See _manifest/blocklist.json for the full list with reasons. (US PD boundary as of 2026: published ≤ 1930.)
  • Deduped: exact duplicate files within IA items (e.g. NLS items carry each volume under both a long descriptive name and a short name) — the descriptive copy is kept. ~100 files / 19 GB.

Editions & sources

1,343 IA items. Major sets include:

  • Editions 1–6 + supplements: National Library of Scotland scans (*EncyclopediaBritannicaNLS), plus independent copies from Getty Research Institute, Bavarian State Library, UC Libraries, National Library of Naples
  • Edition 7 (1842): encyclopaedia-britannica-7ed-1842 (NLS-sourced), BSB
  • Edition 8 (1853–60): Digital Library of India scans
  • Edition 9 (1875–89, "Scholar's Edition"): encyclopaedia-britannica-9ed-1875 (complete, NLS-sourced) + DLI set
  • Edition 10 (1902–03): encyclopaedia-britannica-10ed-1903 (NLS-sourced)
  • Edition 11 (1910–11): encyclopaedia-britannica-11th-and-12th-editions plus Robarts/Toronto (*chisrich), DLI, and bwb_ sets
  • Editions 12–14 (1922–1929): misc sets; 1929 entered US public domain Jan 2025, 1930 in Jan 2026. 1931+ printings are excluded (see blocklist)

Edition labels in the catalog ending in ? are inferred from publication year only — verify against the item if it matters.

Licensing

All mirrored volumes were published 1768–1929 and are public domain by publication date in the US. Items with no IA year were reviewed by hand (see blocklist). Item-level license metadata varies: some items carry an explicit Public Domain Mark or CC-BY (e.g. the NLS items were uploaded by a third party with CC-BY-4.0, while NLS's own data portal releases the same scans as public domain / CC0). Check _item.json per item; when in doubt, attribute the contributing library.

Provenance & verification

  • Source: https://archive.org/details/<ia_id> per file (see catalog).
  • Every file was md5-verified against IA's own metadata during mirroring; the expected md5 is in the catalog and in each done/ marker.
  • Mirrored with HF Jobs (cpu-basic), streaming IA → bucket, August 2026. Mirror script + method: see done/ markers and run reports.

Rebuilding / resuming

The mirror is idempotent: rerunning a shard skips files already present with matching size. To resume an interrupted shard, rerun the mirror job with the same _manifest/shard-N.jsonl.

Related

  • NationalLibraryOfScotland/encyclopaedia-britannica-lance (dataset) — the NLS subset (eds 1–8, 155,388 pages) as per-page JPEG + ALTO XML in Lance format. That is the access layer; this bucket is the archival layer.
Total size
3.97 TB
Files
22,586
Last updated
Aug 26
Pre-warmed CDN
US EU US EU

Contributors