Analysis scripts

Clause 5 of the Standard requires reproducible analysis: any analysis producing a number, a count, a sequence, or a selection must be performed by a method that returns identical results when repeated, and the method — not merely its result — must be preserved. Appendix E of the book states seven reproducible analysis specifications in terms of what each must take in and what each must emit. This page carries working implementations of them, maintained against current language versions, because a code listing set in type ages badly and a reader who cannot run it is worse off than one who found nothing at all.

Why a script and not a conversation

This is the distinction the book spends a chapter on, and it is worth restating here because it is the reason this page exists at all. Asking a system to analyze the evidence treats the system as a source: the answer arrives with no method attached, cannot be repeated exactly, and rests on the system's authority. Asking a system to write a script that analyzes the evidence treats it as an instrument: the script is inspectable, runs the same way every time, produces the same answer on the same input, and can be handed to an opposing expert who can run it themselves.

That is what reproducible analysis means in practice, and the distinction survives every change in the technology, which is why it lives in the book rather than here. What is here is the part that does not survive — the code.

The seven specifications

Enumeration. Index comparison. Chronology merge. Deduplication. Search with coverage. Absence. Artifact tabulation. Appendix E states what each must take in and what each must emit; the code is an implementation of that, and any implementation satisfying the specification is as good as this one.

Enumeration answers how many of a thing there are, and emits the list rather than the count alone, because a count without its list cannot be checked. Index comparison takes two inventories of what should be the same set and emits the three-way difference — present in both, present only in the first, present only in the second. Chronology merge takes events from several sources with inconsistent timestamp formats and emits a single ordered sequence with each entry's source retained, which is what makes the merged chronology defensible when one source turns out to be wrong.

Deduplication emits the groups, not just the survivors, so that the decision to treat two items as the same item is visible and reviewable. Search with coverage is the one most often done badly: it emits not only the hits but the denominator — how many items were searched, how many could not be read, and what was skipped — because a search that silently fails to open forty documents reports zero hits in exactly the same way as a search that found none.

Absence establishes that something is not present, which is a harder claim than presence and needs the search space stated explicitly. Artifact tabulation counts occurrences of a defined artifact across a corpus and emits the per-item breakdown alongside the total.

Verifying a script you did not write

Five checks, none of which requires reading the code. Run it against a subset you counted by hand. Run it twice and confirm the output is identical. Make it emit the list, not just the total. Spot-check ten output rows against the source. And have the behavior explained in plain language, then test the edges — an unreadable file, a zero-byte file, a wrong extension, a malformed date. That is where counts quietly go wrong.

What every reproducible analysis must preserve

The script itself, the input manifest with hashes, the output as produced, the environment and version notes, and the run date with a one-line reason. That is Clause 5 and Clause 7 satisfied together, and it takes about a minute per analysis.

Preserve the output as produced, not as it appeared in the report. The number in the report has usually been rounded, relabeled, or combined with something else, and the distance between the raw output and the printed figure is exactly the ground an examiner will want to walk. Keeping both makes that walk short; keeping only the printed figure makes it an afternoon.

The version notes matter more than they look. A script that produced one answer in one version of a language or library and a different answer in the next is not an unusual event, and the note that records which version was used is what turns an irreproducible result into a reproducible one. It costs a line.

How these are maintained

Implementations here are kept running against current language versions, which is the whole reason they are on a website and not in the printed book. A code listing set in type in 2026 is a listing nobody can run in 2031, and a reader who finds it and tries anyway is worse off than one who found nothing.

Each implementation states the version it was last tested against and the date it was last run. Where a specification can be satisfied more simply in a newer version of a language, the implementation is replaced rather than annotated, because a page carrying five historical versions of the same script is a page that makes the reader choose, and the reader has no basis on which to choose.

If you cannot run code at all

Nothing here requires it, and a substantial share of the experts these specifications were written for do not write code and should not start. Reproducible analysis is a property of the method rather than of the tooling: a spreadsheet with its formulas intact and its source data attached satisfies Clause 5. A documented query against a database satisfies it. A manual count performed twice by two people, with both tallies kept, satisfies it.

What does not satisfy it is a conversation with a system, and the reason has nothing to do with the system's accuracy. It is that the conversation cannot be repeated to the same result, the method cannot be separated from the answer, and there is nothing to hand an opposing expert that would let them arrive where you arrived.