By Bill Hartzer · Published · Updated · How this site is researched
Clause 5 of the Standard requires reproducible analysis: any analysis producing a number, a count, a sequence, or a selection must be performed by a method that returns identical results when repeated, and the method — not merely its result — must be preserved. Appendix E of the book states seven reproducible analysis specifications in terms of what each must take in and what each must emit. This page carries working implementations of them, maintained against current language versions, because a code listing set in type ages badly and a reader who cannot run it is worse off than one who found nothing at all.
Published. All seven implementations are below and can be downloaded individually. They are Python 3 and use only the standard library, so nothing needs installing, and each is a single self-contained file: an expert can hand one script to an opposing expert without also sending a library. Last tested 28 September 2026 against Python 3.10 and 3.11.
The implementations
Each file states its specification in its own docstring, prints a one-line summary when it runs, and writes a JSON report that records the parameters it was given, the version of Python it ran under, and the date it was run. That report is the artifact to preserve, not the figure you copy out of it: a reproducible analysis is preserved by keeping the report beside the script that produced it, never by keeping the number alone.
| Specification | File | What it emits, and how to run it |
|---|---|---|
| 1. Enumeration | enumerate_items.py3,738 bytes · SHA-256 d049a2086f5ec831… | Emits the count and the enumerated list.python3 enumerate_items.py --dir ./production --ext .pdf --out enumeration.json |
| 2. Index comparison | index_compare.py3,794 bytes · SHA-256 ed34465db50f3d3a… | Emits present in both, only in the first, only in the second.python3 index_compare.py --a loadfile.csv --a-col BegBates --b produced.txt --out compare.json |
| 3. Chronology merge | chronology_merge.py4,293 bytes · SHA-256 291f7c3d94ad03e8… | Emits one ordered sequence, each entry keeping its source.python3 chronology_merge.py --source logs:server.csv:ts:message --source email:mail.csv:Date:Subject --out chrono.json |
| 4. Deduplication | deduplicate.py4,376 bytes · SHA-256 331b360edd31373c… | Emits the groups, not only the survivors.python3 deduplicate.py --dir ./production --by content --out dupes.json |
| 5. Search with coverage | search_coverage.py4,944 bytes · SHA-256 9ecafd494d49dbfa… | Emits the hits and the denominator.python3 search_coverage.py --dir ./production --term "0% at fault" --out search.json |
| 6. Absence | absence.py3,928 bytes · SHA-256 b76c694c865569b0… | Emits a finding of ABSENT, PRESENT or INCONCLUSIVE, with the search space.python3 absence.py --dir ./production --term indemnification --out absence.json |
| 7. Artifact tabulation | artifact_tabulate.py3,849 bytes · SHA-256 53480dfd1f875b19… | Emits a table that reconciles to the item count.python3 artifact_tabulate.py --dir ./production --out tabulation.json |
Three behaviors are worth knowing before you rely on any of them. Nothing is ever deleted:
deduplication reports groups and leaves the files alone. Nothing that could not be examined is
counted as examined: unreadable and unsupported items are listed separately, and the coverage
figure is reported alongside every search. And absence.py will not assert absence
when coverage was incomplete — it returns INCONCLUSIVE instead, because a search that could not
open forty documents has not established that the term is missing from them.
Why a script and not a conversation
This is the distinction the book spends a chapter on, and it is worth restating here because it is the reason this page exists at all. Asking a system to analyze the evidence treats the system as a source: the answer arrives with no method attached, cannot be repeated exactly, and rests on the system's authority. Asking a system to write a script that analyzes the evidence treats it as an instrument: the script is inspectable, runs the same way every time, produces the same answer on the same input, and can be handed to an opposing expert who can run it themselves.
That is what reproducible analysis means in practice, and the distinction survives every change in the technology, which is why it lives in the book rather than here. What is here is the part that does not survive — the code.
The seven specifications
Enumeration. Index comparison. Chronology merge. Deduplication. Search with coverage. Absence. Artifact tabulation. Appendix E states what each must take in and what each must emit; the code is an implementation of that, and any implementation satisfying the specification is as good as this one.
Enumeration answers how many of a thing there are, and emits the list rather than the count alone, because a count without its list cannot be checked. Index comparison takes two inventories of what should be the same set and emits the three-way difference — present in both, present only in the first, present only in the second. Chronology merge takes events from several sources with inconsistent timestamp formats and emits a single ordered sequence with each entry's source retained, which is what makes the merged chronology defensible when one source turns out to be wrong.
Deduplication emits the groups, not just the survivors, so that the decision to treat two items as the same item is visible and reviewable. Search with coverage is the one most often done badly: it emits not only the hits but the denominator — how many items were searched, how many could not be read, and what was skipped — because a search that silently fails to open forty documents reports zero hits in exactly the same way as a search that found none.
Absence establishes that something is not present, which is a harder claim than presence and needs the search space stated explicitly. Artifact tabulation counts occurrences of a defined artifact across a corpus and emits the per-item breakdown alongside the total.
Reproducibility as the rules frame it
Federal Rule of Evidence 702 asks whether an expert's opinion reflects a reliable application of reliable methods to the facts of the case. A script is the most direct evidence of that application an expert can hand to the other side. It states the method, runs on the produced data and returns the same answer each time it is run. The contrast is visible in Matter of Weber. An expert who used a chat assistant to cross-check a damages calculation could not say what he had asked it, and the court, putting a similar question to the same assistant, received 3 different figures.
Verifying a script you did not write
Five checks, none of which requires reading the code. Run it against a subset you counted by hand. Run it twice and confirm the output is identical. Make it emit the list, not just the total. Spot-check ten output rows against the source. And have the behavior explained in plain language, then test the edges — an unreadable file, a zero-byte file, a wrong extension, a malformed date. That is where counts quietly go wrong.
What every reproducible analysis must preserve
The script itself, the input manifest with hashes, the output as produced, the environment and version notes, and the run date with a one-line reason. That is Clause 5 and Clause 7 satisfied together, and it takes about a minute per analysis.
Preserve the output as produced, not as it appeared in the report. The number in the report has usually been rounded, relabeled, or combined with something else, and the distance between the raw output and the printed figure is exactly the ground an examiner will want to walk. Keeping both makes that walk short; keeping only the printed figure makes it an afternoon.
The version notes matter more than they look. A script that produced one answer in one version of a language or library and a different answer in the next is not an unusual event, and the note that records which version was used is what turns an irreproducible result into a reproducible one. It costs a line.
How these are maintained
Implementations here are kept running against current language versions, which is the whole reason they are on a website and not in the printed book. A code listing set in type in 2026 is a listing nobody can run in 2031, and a reader who finds it and tries anyway is worse off than one who found nothing.
Each implementation states the version it was last tested against and the date it was last run. Where a specification can be satisfied more simply in a newer version of a language, the implementation is replaced rather than annotated, because a page carrying five historical versions of the same script is a page that makes the reader choose, and the reader has no basis on which to choose.
If you cannot run code at all
Nothing here requires it, and a substantial share of the experts these specifications were written for do not write code and should not start. Reproducible analysis is a property of the method rather than of the tooling: a spreadsheet with its formulas intact and its source data attached satisfies Clause 5. A documented query against a database satisfies it. A manual count performed twice by two people, with both tallies kept, satisfies it.
What does not satisfy it is a conversation with a system, and the reason has nothing to do with the system's accuracy. It is that the conversation cannot be repeated to the same result, the method cannot be separated from the answer, and there is nothing to hand an opposing expert that would let them arrive where you arrived.