User Guide

Everything Shuddhi does, why it does it that way, and how to tune it.

New here? Start with the Quickstart. This guide assumes you have run the demo once.

Contents 0. Before anything: shuddhi doctor tells you whether the interpreter you are about to use can actually run the pipeline. It is the first thing to run and the first thing to check when something is odd. 1. The idea: receipts, not trust 2. Data model 3. The registry and the provenance gate 4. Running it: one command, or stage by stage 5. Near-duplicate clustering 6. The filters 7. Build: applying the filters 8. Scaling out 9. Reading the outputs 10. Attesting corpora built elsewhere 11. Regulatory reporting 12. Honest limits

---

1. The idea: receipts, not trust

Most data pipelines produce a cleaned corpus and a README describing what they did. Six months later nobody can prove which documents a model actually saw, whether the customer data really was excluded, or whether the benchmark leaked into training.

Shuddhi produces a corpus and a chain of hashes that anyone can recompute:

registry              source · licence · date · data_class, declared per shard
   │
   ├─► corpus_build_hash     blake2b-256 over the sorted set of unique
   │                         document content hashes across accepted shards
   │
   ├─► filter_config_sha256  every threshold, plus the shas of the near-dup
   │                         drop list and the toxicity lexicon
   │
   └─► filtered_build_hash   the same hash definition, over the kept set —
                             recorded together with the parent hash and the
                             config sha

Three properties make this useful rather than decorative:

• Order-independent. The hash is over a set, so shard order, parallelism, and partitioning cannot change it.
• Recomputable. Anyone holding the raw shards can rerun and compare. There is no signature to trust and no server to ask.
• Chained. A filtered build names its parent corpus and the exact config that produced it. "Which documents did this model train on?" has an answer.

Cite filtered_build_hash in your training-run ledger. That is the whole contract.

---

2. Data model

A document is a block of text separated from its neighbours by a blank line. Leading and trailing whitespace is stripped. A document containing an internal blank line counts as two documents — that is the format's contract.

A shard is one file of documents, registered with its provenance. One file per language or per source is the usual layout.

A document's identity is blake2b-8 over its UTF-8 bytes — a 64-bit content hash. Identical text anywhere in the corpus has one identity, which is what makes cross-shard deduplication and set-based build hashes work.

Everything is streamed. Memory use is bounded by the number of documents (hashes and signatures), never by document or file size, so a 32 GB shard processes on a laptop.

---

3. The registry and the provenance gate

Scaffold one from the files you have, rather than writing it by hand:

shuddhi init --corpus <folder-with-your-txt-files> --out my-registry.json

init writes an entry per text file and guesses a language from a filename suffix like news_eng.txt, but leaves every provenance field empty on purpose: empty fields are refused, and check names them. The scaffold cannot become a corpus until a human states the provenance — the gate teaches itself rather than needing to be explained.

Filled in, a registry looks like this:

{
  "registry_version": 1,
  "corpus_id": "my-corpus-v1",
  "shards": [
    {
      "shard_id": "news_eng",
      "path": "corpus/news_eng.txt",
      "source": "Example News Crawl 2026",
      "license": "CC-BY-4.0",
      "date_acquired": "2026-08-13",
      "data_class": "public",
      "language": "eng"
    }
  ]
}

Every field is mandatory and shard_id must be unique. A shard's path is resolved from the directory you run Shuddhi in, not from the registry's own location — so keep the registry at the root of your project and write paths relative to that.

data_class

value	meaning
public	openly licensed public data (say which in license)
licensed	you hold a written licence permitting training
synthetic-own	generated by you, on your own infrastructure
customer · customer-derived · evaluation-only	never trainable — always refused
anything else, or missing	refused as untagged

The refusal for customer-class data is checked first and returns unconditionally. There is deliberately no flag, environment variable, or registry field that can override it, and the check happens before the shard's file is opened. If your organisation's rule is "customer data is evaluation-only, never training", this is that rule expressed as code instead of as a memo.

The suspect-name rule

A shard whose id, path, or source matches customer|client|tenant|pilot| prod[-_]?log|bpo[-_]?qa|ticket while claiming a trainable class is refused until a human records a reviewed_by field:

{ "shard_id": "pilot_transcripts", "data_class": "synthetic-own",
  "reviewed_by": "priya (2026-08-13, confirmed synthetic)" }

reviewed_by lifts this check only. It can never admit a customer class.

Checking

shuddhi check --registry my-registry.json

Prints the accepted ledger and every refusal with its reason, and exits 2 if anything was refused — wire it into CI.

---

4. Running it: one command, or stage by stage

Most of the time this is the whole thing:

shuddhi pipeline --registry my-registry.json --out shuddhi-out/

If any shard is refused, pipeline stops instead of building without it — a corpus quietly missing a shard you believed was in it is worse than no corpus at all. Either fix the registry, or state the intent explicitly with --allow-refusals, which builds from the accepted shards and records the refusals in the manifest. The bundled examples/registry.json ships a customer-class shard deliberately, so it triggers this on the first try.

Add --log-drops to any build or pipeline and it also writes build/dropped.jsonl: one JSON line per dropped document — its shard, index, the reason it was removed, and a short preview. It is written at the moment the decision is made, so the reason is the branch actually taken, not a later guess, and the counts sum back to dropped_by_reason exactly. Use it to answer "what did you exclude and why?" and to audit the tool's judgement on your own data. The previews can contain the content that triggered the drop (PII, toxicity), so the file is as sensitive as the corpus.

pipeline runs every stage below in the correct order and writes the manifests, the cleaned corpus, a regulatory draft and an HTML receipt. The ordering is not cosmetic: the language models must be trained before the measurement pass, or the measurement records no perplexity distribution and the filter silently has nothing to threshold against. Encoding that is most of what the command is for.

Run the stages yourself when you need to parallelise across shards, resume after a failure, or look at the numbers between phases — which is what the rest of this guide describes.

Measure: run and merge

run makes one streaming pass per shard and computes two different kinds of number, which are never mixed:

Full pass — every byte, every document: shard SHA-256, document count, document content hashes, exact-duplicate rate.

Sampled — every Nth document (--sample-every, default 50 = 2%): language ID, quality scores, domain classification, contamination screening, near-dup sampling, PII counts (--pii-scan), perplexity distribution (--lm), and bytes-per-token measurement (--tokenizer).

Sampling is a deterministic index stride, not randomness, so the same file always yields the same sample. Every stats file records the strides used and the resulting coverage. Reports must quote coverage — a sampled number described as a corpus fact is exactly the kind of claim this tool exists to prevent.

shuddhi run --registry $REG --shard news_eng --out run/ \
    --sample-every 50 \
    --lm lms/eng.lm.gz \          # perplexity distribution
    --pii-scan \                  # PII prevalence
    --eval-set eval-set.jsonl \   # contamination screen
    --fasttext-model lid.176.ftz \# real language ID
    --tokenizer tokenizer.json    # bytes/token

Shards are independent: run them in parallel.

merge then combines the per-shard outputs, deduplicates globally, and mints corpus_build_hash:

shuddhi merge --registry $REG --out run/

It refuses to merge a partial corpus unless you pass --partial, which records the missing shards in the manifest. Silence about coverage is never an option.

Contamination screening

Give --eval-set a JSONL file of {"id": ..., "text": ...} items — your benchmarks, held-out sets, or trap prompts:

{"id": "bench:q1", "text": "Which Indian state has the longest coastline?"}

Screening normalises case and punctuation, then flags any document that contains an eval item verbatim or shares an 8-word span with one. Every candidate is verified by exact string comparison, so a reported hit is a real textual overlap, never a hash collision. Generate a starting set from a repository with build_eval_set.py.

---

5. Near-duplicate clustering

Exact deduplication catches byte-identical documents. Near-duplicates — boilerplate templates, re-syndicated articles, a page with one line changed — need MinHash.

# per shard, parallelisable: sign every document
shuddhi neardup-sig --registry $REG --shard news_eng --sig-dir sigs/

# once, across everything: cluster and write the drop list
shuddhi neardup-merge --registry $REG --run-dir run/ \
    --sig-dir sigs/ --out neardup-drop.u64

32 permutations, banded 8×4, over 5-word shingles of the first 300 words. Every LSH collision is verified against the cluster representative and kept only if at least 21 of 32 signature rows agree (estimated Jaccard ≈ 0.66), so bucket collisions cannot silently delete data.

Each cluster keeps one exemplar: the member with the lowest document hash. That rule is order-independent, which is what makes the drop list — and therefore the build hash — identical regardless of shard order or parallelism.

The output is a sorted list of document hashes to drop, plus a stats file with cluster counts and the largest cluster. Its sha256 is pinned into the build's filter config.

On a real 33-million-document corpus this stage found 258,406 clusters and a single template repeated 84,275 times. Sampling had suggested the problem was in a different language entirely. Sampling hints; full passes settle it.

---

6. The filters

Applied in this order, first match wins, each counted separately:

exact_dup → near_dup → quality → perplexity → toxicity → contamination
          → plugins → pii

Custom and third-party filters plug in at the plugins position without forking the engine, and their identities enter the config sha so the receipt covers them too — see Extending.

Quality (--min-quality, default 0.5)

Heuristics, deliberately simple and all in one table at the top of quality.py: symbol and digit ratios, mean word length, repeated-line fraction, a dominant-bigram check, and web-boilerplate markers. Documents under 200 characters are capped at 0.30 and therefore dropped by the default threshold. Buckets: high ≥ 0.75, medium ≥ 0.50, low below.

This is a junk screen, not a ranking. It removes the obviously broken; it does not tell you which prose is worth training on.

Perplexity (--lm-dir, --ppx-percentile)

A character-trigram model per language, trained on a sample of your own corpus, scoring each document in bits/char. High scores mean the document does not look like the rest of that language — byte salad, wrong script, encoding damage.

The cutoff is a percentile of your measured distribution: --ppx-percentile 99 drops roughly the worst 1% per language by construction. That is a design choice, not a discovery about your data.

Two requirements, both easy to get wrong:

1. Train the models before measuring, and pass --lm to run, so the distribution exists. Otherwise build has no cutoff and the filter does nothing (it warns loudly).
2. Percentiles need scale, and the engine enforces it. If fewer than 200 documents were scored during measurement, the filter switches itself off and says so on stderr rather than thresholding against noise — without that guard, a stride that sampled one document made p99 equal that document and dropped 33 of 42. Override deliberately with --min-ppx-sample when you know the sample is small and you want it anyway; the bundled demo does exactly that, and says why.

Toxicity (--toxicity)

A lexicon tier: word-boundary-safe matching against per-language term lists. A document is flagged only when it hits at least 2 distinct terms and a density of at least 0.004 hits per word — one swear word in an ordinary article does not qualify. Matching is frozenset lookup over split words, so it costs ~250 µs per document, and it cannot fire on substrings inside innocent words.

Mount fuller lists with --toxicity-lexicon-dir: one <lang>.txt per language, one lowercase term per line, # comments allowed. Your lists are merged with the built-in starters and the combined term list is sha-pinned into the filter config, so a build always states which lexicon screened it.

This tier removes the unambiguous tail. It is not a classifier and does not understand context.

Contamination (--eval-set)

Same screen as the measurement stage, applied to every document rather than a sample. Any document overlapping your benchmarks is dropped.

PII (--pii keep|redact|drop, default redact)

Detects email addresses, IBANs (validated by their ISO 13616 mod-97 check), Luhn-valid card numbers, and IPv4 addresses, plus the India-specific mobile, Aadhaar and PAN formats. The structured identifiers (IBAN, card) are checksum- validated rather than matched by shape, so invoice and order numbers do not redact. Region-neutral detectors find nothing in a corpus that has none.

• redact (default) keeps the document and rewrites each match to a typed placeholder such as [PII:email]. Redaction processes the entire document, not the scanning probe.
• drop removes any document containing PII.
• keep counts without changing anything.

Pattern-level screening for corpus hygiene — not a compliance-grade DLP system. Names and addresses need NER and are not covered.

---

7. Build: applying the filters

shuddhi build --registry $REG --run-dir run/ --build-out build/ \
    --lm-dir lms/ --ppx-percentile 99 \
    --neardup-drop neardup-drop.u64 \
    --toxicity --toxicity-lexicon-dir lexicon/ \
    --pii redact --eval-set eval-set.jsonl \
    --emit text

build consumes a measured run — the order is always run → merge → build. It re-reads the shards, applies the filters, and writes BUILD-MANIFEST.json with the kept count, drops by reason, the filter config and its sha, the parent corpus hash, and filtered_build_hash.

Integrity is enforced. Every document hash encountered must already exist in the measured run's hash set. If a shard changed after measurement, the build fails loudly rather than quietly producing something unmeasured.

Emission is optional. --emit none (default) computes the hash-only manifest — useful for evaluating a filter configuration cheaply. --emit text additionally writes build/<shard>.filtered.txt with redactions applied and records each output file's sha256. The selection — and therefore the build hash — is identical either way.

Nothing ever modifies your raw shards.

---

8. Scaling out

Build partitions in parallel and union them:

shuddhi build ... --shards shardA,shardB --build-out part1/ &
shuddhi build ... --shards shardC,shardD --build-out part2/ &
wait
shuddhi build-union --build-outs part1,part2 --out build/

Because the build hash is defined over a set, the union of disjoint partitions is provably identical to one sequential build — there is a unit test asserting exactly that. build-union refuses to combine partitions whose parent hash or filter config disagree, and records any cross-partition duplicates that collapsed in the union.

Practical guidance from a 176 GB run on a 2-vCPU box: measurement is I/O bound at roughly 50 MB/s per core; the applied build is CPU bound at a few milliseconds per document. Both parallelise cleanly across shards.

---

9. Reading the outputs

run/<shard>.stats.json     per-shard receipts: provenance echo, sha256,
                           full-pass counts, sampled metrics, timings
run/<shard>.hashes.u64     raw document hashes (uint64), input to merge
run/MANIFEST.json          corpus_build_hash, composition, refusals
run/COMPOSITION.md         the same as readable tables
build/BUILD-MANIFEST.json  filtered_build_hash, drops by reason, filter config
build/<shard>.kept.u64     kept hashes (input to build-union)
build/<shard>.filtered.txt cleaned text (only with --emit text)

Every JSON manifest records the engine version, Python version, and library versions that produced it.

Looking at them

shuddhi ui --dir shuddhi-out/

Serves a local viewer over the builds in a directory: history, the receipts with copy buttons, what each filter dropped, the datasets that went in, warnings and errors, live progress while a run is happening, and downloads. It reads your filesystem, binds to 127.0.0.1, and has no accounts, no database and no telemetry — it works air-gapped because there is no CDN to reach.

report.html in the output directory is the same information as one self-contained file, which is the form to email an auditor or attach to a compliance pack.

Progress adapts to where it is going: a live bar with rate and ETA on a terminal, timestamped lines every 15 seconds when piped to docker logs or CI. Every run also appends events.jsonl — phases, progress, warnings, errors — which is what the viewer reads, so the UI is a view over the log the run already wrote rather than a second implementation of progress.

---

10. Attesting corpora built elsewhere

You do not have to build a corpus with Shuddhi to get a receipt for it. If another pipeline produced it, fingerprint the result:

shuddhi attest --corpus ./out-from-datatrove/ \
    --corpus-id fineweb-slice --registry my-registry.json --scan

The hash definition is identical to a native build's, so an attested corpus and a Shuddhi-built one are comparable and verifiable the same way.

The honest boundary: an attestation proves content, not acquisition. It binds a corpus to a hash and says what is inside it. It cannot tell you where the data came from or under what licence — that is what --registry adds, and without it every provenance field reads UNKNOWN rather than blank, because a blank reads as "nothing to declare".

11. Regulatory reporting

shuddhi report --eu-ai-act --registry my-registry.json \
    --manifest build/BUILD-MANIFEST.json > article-53.md

Article 53(1)(d) of the EU AI Act requires providers of general-purpose models on the Union market to publish a sufficiently detailed summary of training content. Shuddhi computes nothing new for this: the registry already requires source, license, data_class, language and date_acquired — a shard missing any is refused, not defaulted — so the summary is a projection of what admission already recorded.

Pass the build manifest, not the corpus manifest: only the former records a filter configuration, and a training-content summary that cannot name the filters is describing a corpus nobody built.

The output is a draft for your compliance function to review, not legal advice and not a filing.

12. Honest limits

Worth knowing before you rely on any of it:

• The gate enforces your declaration, not the file's true nature. This is the most important limit here, so it goes first. Shuddhi refuses a shard declared customer, and no flag, environment variable, extra JSON field or casing trick admits one — unknown and malformed classes are refused too, because it fails closed. What it cannot do is read a file and tell you that the public label on it is a lie. The suspect-name rule catches the careless case (customer_export.txt, a source of "Acme Corp"), and it is a heuristic on names: a file called cust.txt or exp_2024.txt tagged public will pass.

So the honest claim is narrow and still useful: the gate makes the declaration mandatory, mechanical, and permanent. Someone has to write down where every shard came from before anything is read, CI can refuse a build on it, and the declaration is recorded in the receipt — so a later auditor can hold a named person to what they wrote. It converts "we have a policy about customer data" into "the build fails without a signed answer". It is not a content classifier, and we would rather say so than have you discover it.

• Quality scoring is heuristic. It screens junk; it does not rank prose. A distilled classifier is the upgrade path.
• Toxicity is a lexicon. No context, no sarcasm, no implicit abuse.
• PII is pattern-based. Names and addresses are not detected.
• Perplexity is a proxy. A character-trigram model knows spelling and script, not meaning, and percentile cutoffs need corpus scale.
• Language ID needs the model. Without lid.176.ftz you get Unicode script identification, which cannot separate languages sharing a script (Hindi and Marathi both write Devanagari). The stats always say which method ran.
• Near-dup is within a configured threshold. Jaccard ≈ 0.66 on 5-word shingles catches templates and re-syndication, not paraphrase.
• Token counts are derived, from bytes-per-token measured on a sample.

The tool's job is to measure what it can, apply what you configure, and be explicit about the rest.