By Bill Hartzer · Published · How this site is researched
Reproducible analysis. Any analysis producing a number, a count, a sequence, or a selection is performed by a method that returns identical results when repeated, and the method — not merely its result — is preserved.
A number without a method
The book frames this clause with an ordinary task. An expert needs to know how many photographs a production contains. One route is to load the production into a chat assistant and ask; a number comes back in seconds. The other is a short script that inspects every file, decides its type from its contents rather than its extension, and prints the total with the list it counted. Both are fast, and both feel like the same act.
At deposition they diverge. The first witness cannot say exactly what was asked, whether every file was examined, how a photograph was defined, or whether asking again would return the same figure. The number may well be correct. The difficulty is that its correctness is now unknowable, including to the witness. The second witness hands over the script, the input list and the output with file paths and hashes (short values computed from a file's contents that change if the file changes), and the topic closes.
The clause reaches any analysis that produces a number, a count, a sequence or a selection. Selection includes culling: deciding which documents in a production deserve reading is part of the method, and a method nobody can rerun cannot be tested for what it left out.
Scripts as instruments
The book's most practical inversion follows. Asking a model to analyze the evidence treats it as a source. Asking it to write a script that analyzes the evidence treats it as an instrument. The expert need not be a programmer, and need not be able to read every line, provided the script passes five checks: run it on a small subset counted by hand; run it twice and confirm identical output; have it print the items it counted, not only the total; spot-check rows against the files themselves and confirm known items appear; and have the behavior explained in plain language, then test the edges, such as a zero-byte file or a wrong extension.
From the first run, five things are kept together: the script in each version that produced a result relied on, the input manifest with hashes, the raw output unedited, the environment (language, version and libraries), and the run date with a note of why. That satisfies this clause and Clause 7 together in about a minute. Counts, date ranges, deduplication, chronology merges, cross-references between sets, breakdowns by category and, above all, claims that something is absent should always be scripted. Worked examples are on the scripts page.
The record on analyses no one could rerun
In Matter of Weber, a damages expert used Microsoft Copilot to cross-check his calculations and could not recall the prompts. The Surrogate's Court ran a sample prompt on its own computers, received three different figures, and rejected the analysis, which it also found unreliable on grounds unrelated to AI. The decision is a trial-level ruling of a single court.
In Conservation Law Foundation v. Shell (expert AI prompts), an expert's team used GPT-4o on a private deployment to filter the defendants' production. The report disclosed the search terms but not the prompts, and a magistrate judge ordered the prompts disclosed as part of the methodology. That order is stayed while an objection is decided. The concurrences in Snell v. United Specialty Insurance (Judge Newsom) recorded that identical queries returned slightly different answers and reproduced the responses rather than presenting one as fixed. And in In re Celsius Network LLC (Faraj report), the court noted that no standards controlled the system that generated the excluded report.
What reproducibility does not mean
The clause does not ask a peer to produce identical prose. The governing test on the Standard page asks whether a competent examiner, working from the primary sources without any AI system, would reach the same finding. Two appraisers landing within a defensible range of each other satisfy it. Nor does the clause bar conversational use for exploration; it governs the analysis a finding rests on. Reproducibility is also not correctness: two examiners can reproduce the same error, and the test establishes that the work can be examined, not that it is right. Finally, a script reports facts about the material. Which of 9,000 events matter remains the expert's judgment.
The rules it sits beside
The Standard does not restate any court rule and does not depend on one. In United States federal courts, Federal Rule of Evidence 702 and Federal Rule of Civil Procedure 26 are where a non-repeatable analysis is examined: the first through its inquiry into reliable application of method, the second through disclosure of the facts or data considered. Other systems ask the same questions in their own terms, and the matters below include tribunals in several of them.
Matters in the record that engaged Clause 5
10 of the 103 matters in AI in the Courtroom engaged this clause. Each is listed with the sentence from its entry that says how. The mapping describes conduct, not blame, and each entry sets out the facts it rests on.
Expert witnesses and AI
- Conservation Law Foundation v. Shell (expert AI prompts) “The dispute turns on Clause 5 of the Standard, reproducible analysis.”
- Social Media MDL (Osborne AI citation tool) “Clause 5, reproducible analysis, is engaged by the correction itself.”
- Landgericht Darmstadt expert fee decision “Clause 5, reproducible analysis, is engaged by the absent examination.”
- NSW Supreme Court Practice Note SC Gen 23 “By requiring the version of the program, the application also supports Clause 5, since a result can only be re-run against a known tool.”
- Matter of Weber “The central obligation engaged was Clause 5, reproducible analysis.”
- In re Celsius Network LLC (Faraj report) “It also engaged Clause 3, independent authority, and Clause 5, reproducible analysis.”
AI-generated and altered evidence
- State v. Amyda (deepfake claim without evidence) “Supplying it means an examination of the file that can be explained and repeated, which is Clause 5 of the Standard, reproducible analysis.”
- State of Washington v. Puloka
Judges, courts and AI
- Snell v. United Specialty Insurance (Judge Newsom) “His observation that the same question produced different answers bears directly on Clause 5, reproducible analysis.”
Rules, orders and guidance
- Proposed Federal Rule of Evidence 707 “It also engages Clause 5, reproducible analysis.”
Clause 4: Checkability before adoption · All eight clauses · Clause 6: Custody and confidentiality