75% Less Storage. 84.6% Less Peak Memory.

Save work before saving bytes.

Compression helps after data exists. The cheapest byte is still the one the system never had to fetch, parse, copy, enrich, or upload. Rolefarer therefore applies limits at five boundaries: scheduling, ingestion, evidence storage, AI enrichment, and publishing. A record must earn its way through each boundary.

These decisions compound. Better scheduling sends fewer empty polls to ingestion. Change detection sends fewer new bodies to storage. Content hashes send fewer unique inputs to AI. Immutable release objects make the final upload proportional to what changed, not to the size of the catalog. Optimizing only the database would miss most of the system.

A crawl should compete for its cost.

A fixed schedule treats unlike sources as equals. A large company with hourly job changes gets the same attention as a small board that changes once a month. The first becomes stale. The second burns requests and worker time. More workers hide the problem for a while, but they do not fix the choice.

Scout keeps operating facts for each source: how often it changes, how often polls succeed, how long a poll takes, how many useful jobs it finds, how old the last result is, and how important or demanded the source is. The scheduler turns those facts into a simple question: which due crawl offers the most expected fresh value for its expected cost?

chance of change = 1 - exp(-change_rate × age)

expected value = chance of change
               × expected useful yield
               × source importance
               × demand

crawl priority = expected value × urgency
                 ÷ expected cost

The equation does not predict the future perfectly. It does something more practical: it forces every scheduling choice to name both value and cost. Age raises urgency when a source has waited too long. Change rate rewards boards that produce new information. Duration and failure history lower the rank of expensive or unhealthy work. Protected sources and exploration still receive reserved capacity, so a score cannot starve the long tail forever.

The production fleet also has a hard daily ceiling. Two Control slots provide at most 48 worker-hours. The planner must fit expected demand inside that box. If it cannot, it stretches intervals or phases overdue work instead of quietly creating more work than the server can finish. A budget is useful only when it can say no.

Measure useful change, not raw activity.

Request count and worker use can look impressive while producing no new jobs. The better unit is useful change per unit of cost: new, changed, removed, or reappeared postings per worker-hour or dollar. This keeps the scheduler aimed at fresh information rather than motion. It also makes slow adapters visible. One adapter improvement can help thousands of boards that share the same hiring system, which is far more valuable than tuning one company at a time.

Make sources durable and workers disposable.

The crawler borrows a useful idea from holonic manufacturing systems such as PROSA and ADACOR. A holon is both a whole unit and part of a larger system. In Rolefarer, each hiring source has its own stable identity, policy, schedule, health, and history. It can be understood on its own. It still takes direction from a supervisor that protects the whole fleet.

PROSA separates the thing being produced, the resources that do work, the order for work, and the supervisory knowledge that improves the plan. Rolefarer uses the same kind of boundary. A source and its desired freshness define the work. A poll attempt is the order. A worker and adapter are resources. The supervisor selects, leases, limits, retries, and backs off work. ADACOR adds the idea that local units can adapt while a wider coordinator restores order when pressure or failure rises.

This is more efficient than making every worker smart and stateful. Workers do not need a private memory of the fleet. If one dies, its lease expires and the attempt can be inspected or retried. The supervisor can change concurrency without changing adapters. The source can keep its history across deployments. Durable facts live in storage; short-lived execution lives in workers.

One writer protects the small box.

Fetching is parallel because network waits leave room for other work. Database writes are different. Many writers competing for the same local file create locks, memory spikes, and hard recovery cases. Scout lets fetch workers produce small staged results, then sends those packages through one bounded ingest owner. That worker opens the write database, applies one transaction, records a receipt, and acknowledges the stage only after the write is durable.

Stable attempt IDs make retries harmless. If a package arrives again, the ingest path can recognize it instead of applying the same change twice. This is idempotence: doing the same operation again leaves the final state unchanged. It is a quiet efficiency feature because it saves both storage and repair work after failures.

Store a lifecycle, not a pile of snapshots.

A simple crawler can save a full table every day. That is easy to build and expensive to keep. Most rows are unchanged, so daily snapshots repeat the same title, URL, location, and long description. They also make a basic question such as “when did this job disappear?” require a comparison across files.

Scout gives each posting a stable key inside its canonical employer board. On every poll, it compares the fetched set with active state. An unchanged job only advances its last-seen time. A changed job receives a new version. A new job is inserted. A missing job is marked removed. If it returns, the lifecycle records that event. The model stores the change, not another copy of the whole board.

Separate identity, version, and content.

These are three different facts. Identity answers which posting this is. A version answers what was true during a period. Content holds the large description body. Keeping them separate means several versions can point to one unchanged body. The frozen database contains 1,499,617 job identities and 1,703,903 version rows, while its content table holds 793,377 distinct description bodies. History still costs metadata, so it is not literally free. The expensive text is not copied into every version.

The reused group makes the saving visible. Scout has 89,285 non-empty descriptions referenced by 414,765 versions. Copying that text into every version would use 1,713,710,959 bytes. Storing one copy per content hash uses 428,221,747 bytes. That avoids 1,285,489,212 bytes and cuts storage for the repeated text by 75.0%.

Content hashes make that link exact. SHA-256 turns a body into a stable fingerprint. The same bytes produce the same ID. A changed byte produces a different ID. The system can therefore ask whether it already owns the body before writing it, and a version can point to the hash instead of embedding another copy.

Keep the original web evidence without making it the database.

Normalized job rows are useful for search and analysis, but they are not the source itself. Parsers change. Fields are dropped. A bug can map a location incorrectly. If the original response is gone, the team cannot prove what the site returned or rebuild the record under better rules.

Rolefarer keeps exact HTTP evidence in WARC, the Web ARChive format used to preserve web exchanges. Each body is content-addressed, so a repeated response points to bytes already stored. Typed records keep request facts, response facts, timestamps, and references to the body. The structured database can be rebuilt from evidence rather than treated as the only surviving truth.

Indexed gzip avoids an archive trap.

One giant compressed WARC file is small, but reading one body may require decompressing a long stream from the beginning. Scout writes one gzip member per WARC record and keeps a rooted index from body ID to pack, byte offset, and length. One body can then be fetched with one range read. Compression stays useful without making random access painfully expensive.

The archive also separates logical identity from physical packing. A body keeps the same hash even if maintenance moves it into a better pack. Snapshots name the segments, packs, and indexes that belong together. A new root is published only after every referenced object exists. This gives the archive immutable history and still allows the physical layout to improve.

Give each storage format one clear job.

“Which database is best?” is the wrong question for a system with several kinds of data. Raw evidence, mutable control state, analytical tables, portable exports, and web responses have different access patterns. Forcing all of them into one store either wastes space or makes common work awkward.

DuckDB is especially useful for the private analytical corpus because it stores data by column. A query that needs keys and dates does not have to read every long description. Repeated values such as adapter names and status compress well. The frozen Scout file is 6.1 GB and contains the current jobs, history, content, source state, poll attempts, discovery facts, and reporting marts. A Zstandard-compressed backup of that full file is 1.4 GB. Those sizes describe the real system, not a clean one-table benchmark.

Parquet serves a different purpose. It packages selected typed tables in a compressed, column-oriented file. Rolefarer can export a bounded slice, move it, and read only the needed columns without treating the export as the live database. That keeps handoffs and recovery tools small. It also avoids copying control tables that a downstream reader does not need.

I attempted a matched SQLite comparison while preparing this study, but stopped it when the local query placed too much pressure on the machine. No SQLite savings ratio is claimed here. A valid benchmark would export the same rows and columns into fresh DuckDB, SQLite, and Parquet files, build equivalent indexes, close each database, and compare both bytes and query time. Comparing the 6.1 GB production file with a simple SQLite table would not be fair because the production file includes far more than current jobs.

Do not pay a model to repeat itself.

AI enrichment can become the largest variable cost. A retry, release rebuild, or duplicated job should not create another model call when the input bytes and extraction rules are unchanged. Dock defines an artifact by two stable inputs: a content-addressed job version and an immutable extraction contract. If both are the same, the prior result is still the answer.

The first production cohort contained 4,996 admitted pointers but only 4,790 unique inputs after content addressing. That removed 206 duplicate model inputs before submission. The run produced 4,760 accepted outputs in 62 batches for $7.39. A hard $40 monthly UTC cap makes the executor fail closed when the remaining budget cannot fund the work. Cost is recorded with the run instead of discovered later on a bill.

The response shape is compact too. Short field IDs replace repeated long names. Known defaults, empty fields, and evidence quotes are omitted when the contract does not need them. One controlled format test reduced stored response bytes by 60.3%. This is a wire-format result, not a claim that the whole Dock store became 60.3% smaller. It shows why output design belongs in the cost model: model tokens, response storage, parsing, and downstream scans all touch the same bytes.

Admission is more valuable than cleanup.

The best AI optimization is still not sending the request. Dock admits exact current Scout versions, rejects work that is already complete, and bounds how many items can enter a run. Large request and response bodies live in a content-addressed store; PostgreSQL keeps hashes, pointers, lifecycle, costs, and resume anchors. A failed batch can restart from durable state without paying to rebuild everything before it.

Publish references, not one giant payload.

The public catalog does not receive the private Scout database. Catalog joins exact current Scout versions with accepted Dock artifacts, then builds only the fields the web product needs. Search summaries, posting lists, detail shards, descriptions, and fact groups become small Brotli-compressed objects named by their SHA-256 hash.

A release manifest points to those immutable objects. If the next release contains identical bytes, it points to the same hashes. Across 52 live releases, copying every release’s compressed objects would have used 572.1 MB. The content-addressed store held the shared objects once in 87.4 MB. That avoided 484.6 MB and cut object storage by 84.7%.

Bound memory by changing the algorithm.

The first publisher loaded too much work at once. A 5,000-job release grew to about 7 GB of RAM and 1.1 GB of swap. That was not a server-sizing problem. It was an algorithm that allowed memory to grow with the whole cohort. The rebuilt path uses bounded admission, streaming work, fixed concurrency, and explicit memory gates. The same production-size publish peaked near 1.078 GB and used no swap.

The service now treats 1 GB as a soft warning and 1.5 GB as a hard boundary. It does not count swap as extra permission. That distinction matters on a small VPS. Swap may keep a process alive while making every other service slow. A publisher that stays within real memory is both cheaper and more predictable.