# 75% Less Storage. 84.6% Less Peak Memory. > How content hashes cut storage for repeated job text by 75% while a bounded publisher cut peak VPS memory from about 7 GB to 1.078 GB, without losing job history. - Author: Paul Yu - Role: Senior Analytics Engineer - Employer: Rolefarer - Year: 2026 - Theme: Efficiency - Human article: [https://paul-yu.com/case-studies/rolefarer-storage-efficiency](https://paul-yu.com/case-studies/rolefarer-storage-efficiency) - Structured data: [https://paul-yu.com/agent/case-studies/rolefarer-storage-efficiency.json](https://paul-yu.com/agent/case-studies/rolefarer-storage-efficiency.json) ## Evidence | Value | Measure | Evidence level | | --- | --- | --- | | 75.0% less | storage for descriptions reused across versions | Measured | | 1.285 GB | duplicate job-description bytes avoided | Measured | | 89,285 | non-empty descriptions reused by 414,765 versions | Measured | | 84.6% less | peak memory for a 5,000-job publish (~7 GB → 1.078 GB) | Measured | ## Situation Rolefarer had raw web evidence, more than one million observed job records, changing versions, and AI results. Copying everything into each new release would waste disk, memory, network traffic, and model spend. ## Task Keep the original source proof and job history, but still publish fast search releases on a small server with firm memory and cost limits. ## Action Each storage layer has one job. Content hashes identify bytes that have not changed. WARC holds the raw web evidence, DuckDB handles structured job data, and Parquet carries compressed table exports. The web release uses small Brotli objects. Fixed worker and memory limits keep a publish from taking over the server. ## Result The frozen Scout snapshot had 89,285 non-empty descriptions reused by 414,765 job versions. Copying the text into every version would have used 1.714 GB. Content addressing stored 428.2 MB once, avoiding 1.285 GB and cutting storage for that reused text by 75.0%. Bounded publishing also cut peak memory for a 5,000-job release by about 84.6%, from about 7 GB to 1.078 GB. Across 52 Catalog releases, the same pattern cut shared object storage by 84.7%. ## System flow Raw web evidence → Content-addressed WARC → DuckDB + Parquet → Compact immutable release ## Technical notes ### Content addressing Each body is named by its SHA-256 hash. Identical bytes get the same name, so Rolefarer can reuse one copy and resume an interrupted job safely. ### WARC, DuckDB, and Parquet WARC keeps the original web evidence. DuckDB runs fast local queries on structured data. Parquet keeps typed table exports small enough to move easily. ### Immutable releases A published release never changes. The next release points to existing objects when the data is the same. It changes the small current pointer after every required object is ready. ### Bounded memory Streaming exports, fixed worker counts, and DuckDB and service limits give each run a memory budget. The publisher never loads the full corpus at once. ## Ownership I designed and built Rolefarer and its storage path across Scout, Dock, Catalog, and the published search product. ## Limitations The 75.0% figure covers only non-empty job descriptions used by more than one version. It compares their logical text bytes when copied into every version with one copy per content hash. It does not cover unique descriptions, version rows, indexes, WARC evidence, or the full DuckDB file. The 84.6% memory figure is an approximate comparison for a 5,000-job publish, not every VPS workload. These are stored version references, not a count of network downloads. ## Takeaway The largest gain came from refusing waste at every boundary: crawl only useful sources, store repeated bytes once, enrich each version once, and publish only changed objects. ## Tools PROSA / ADACOR, SHA-256, WARC, DuckDB, Parquet, Brotli, Cloudflare R2