Reading Parliament's mail at $5.31 a submission

Submissions to the Joint Select Committee on Artificial Intelligence closed on 14 September and Parliament has published 540 of them: 5,839 pages.

The committee must report by 30 November. That gives its secretariat eleven weeks to read the pile, work out who said what, weigh it, and draft something the members can argue over.

The last Senate committee on AI, the Select Committee on Adopting Artificial Intelligence, ran from March to November 2024, its final report lists ten people who served in the secretariat over that time and it received 245 public submissions and produced a 224-page report.

I wanted to test how much of that job a machine could do: so I gave the pile to Symplast, the analytical platform I've been building, and asked it one of the committee's questions.

From a workspace where all the reports had been ingested, it produced a 335-page report in a few hours, for $47.42 of model time.

What a secretariat does

Strip an inquiry back to its reading and writing and the job looks like this:

1. Read everything.

2. Keep track of who said what, and where.

3. Decide what the question is, and break it into parts.

4. For each part, find the evidence, and the evidence against.

5. Check that each claim says what its source says.

6. Weigh it, write it up, and leave a trail that can be followed.

Fred Brooks would call most of steps 1 and 2 accidental complexity. Nobody joins a committee secretariat to re-find page 14 of submission 287, but that is where the weeks go, and it is why ten people can only read so carefully. The essential part is judgement. A machine that does the accidental work well gives the humans their weeks back for the essential work.

What Symplast did

Symplast runs structured workflows: small, checkable steps that compose into larger processes. Each step puts a language model inside a harness that decides what the model is asked, what it is allowed to see, and what counts as an acceptable answer. This inquiry report is a stack of those workflows.

1. It read everything. Before ingest we removed material attached to submissions rather than written for them: earlier submissions to other inquiries, previously published reports, and 306 pages of draft legislation from a single submitter. That took out 1,097 pages and left 4,742, broken into 7,630 passages.

2. It indexed who said what. Every organisation, person, technology, risk, claim and recommendation in those passages became a record: 49,012 of them, with 106,595 relationships between them, every one traceable to the passage it came from. Where 14,073 records turned out to describe the same 4,738 things, they were merged. Before anyone asked a question, the most repeated recommendation in the pile was a "proportionate, risk-based" approach to regulation.

3. I set the question. It was the committee's terms of reference (e) and (f): sovereign AI capability, data sovereignty, and narrow models trained on Australian data. I framed it as access, control, continuity and transparency, to 2030. The machine planned an outline of eight chapters and 31 sections.

4. It researched each section. An agent per section searched the index, 639 searches across the run, and kept what it judged to be evidence.

5. It wrote down every claim before making it. Each of the 486 claims records its source, what that passage establishes, the inference from the passage to the claim, its limitations, and the result of a search for evidence against it. That search ran for 429 claims and found contrary evidence for 82.

6. It checked every claim against its source. Does the cited passage say this? 335 claims were supported, 119 provisional, 21 contested and 11 unsupported. Another 80 never got in, because they could not point to a passage at all.

7. It analysed and wrote. Each chapter went through structured analytic techniques, the decades-old tradecraft analysts use to test their own reasoning. Thirty findings rest on the claims and eight recommendations rest on the findings.

8. It audited itself. Before the report was finalised, an audit listed what it could not close: contested claims nobody had answered, and the findings resting on them. Four items, printed at the end of the report.

What came out

The report runs to 335 pages. About 100 are the argument. The other 235 are the working: every claim with its source, warrant and status, and every source with its page.

Its conclusions, briefly:

- Australia should not fund its own general-purpose frontier model before 2030. The capital and compute aren't here, and the asset depreciates in months.

- Who can legally compel a supplier matters more than where the data sits.

- Continuity should be a procurement requirement: tested exit pathways and a working domestic fallback, in place before the contract is signed.

- Narrow, domain-specific models on Australian data are the defensible pathway.

Some of what it surfaced from the submissions stands on its own:

- One submission records the Assistant Minister tracing the AI dollar: around five cents to electricity, ten to the data-centre building, 35 to 45 to the computing equipment, and the remaining 40 to 50 to "the models, software, intellectual property and returns to capital further up the stack". On that arithmetic, hosting earns about fifteen cents in the dollar.

- Kitbag Consulting reviewed 73 Commonwealth AI transparency statements. "None state whether inference is processed onshore, whether the Commonwealth holds the model weights, or whether any foreign jurisdiction has legal reach over the supplier."

- The data-centre industry's own peak body told the committee that "six in every seven megawatts of connection requests is phantom demand".

- The National Computational Infrastructure says its current funding "isn't sufficient to purchase even 500 current generation GPUs".

- The camp that wants Australia to build its own is not fringe. Westpac: "the shell of the car might be Australian made, but the engine is American."

Where it went wrong

It's not perfect, by any means.

There's six areas for immediate improvement.

It referenced most often the submissions that answered most squarely in the inquiry's own terms. One submission ended up carrying an eighth of the argument, and some findings rested on a single submitter. The ledger already knows when two claims share a source. The fix would be a corroboration rule (a finding needs independent submitters behind it).

Recall. It missed relevant passages it should have found: the right page of a submission it had already used, and some of the most quotable lines in the pile. Precision is what the checking step measures, and recall is the gap. The fix would be a coverage pass before writing, which tests every on-topic submission against the outline and asks what it says that the report doesn't.

Judging. The judging step occasionally made the wrong call, including refusing a claim that was correct. Every refusal is recorded with its reason, which is how the bad one was found. The fix is a second look at refusals, with disagreements sent to a person.

Fidelity. It put quotation marks around a paraphrase. A quote either matches the source text character for character or it isn't printed as a quote, and that would be deterministic.

Preparation. Our own pre-processing removed material a submitter had pointed the committee to. The fix is to keep attachments, tag them as attachments, and follow a submission's references to its own material.

Budget. Some sections stopped researching when their budget ran out, and said so. The fix is a dial: a bigger budget, or an automatic second pass wherever research stopped short.

Some of these fixes are a line of code: quotes and attachments. Others reduce the error rather than remove it: recall and judging. All of them are possible for the same reason. Every step's inputs and outputs are on the record, so each fault can be located, measured, fixed and measured again.

The obvious next question was whether it ignored anything important.

The argument cites 206 of the 540 submissions. Most of the other 334 were about something else: frontier safety, copyright, jobs, children and data-centre water, all outside the scope of this particular process.

Keep your receipts

I found every failure above because Symplast always keeps its procesing steps. Each sentence cites a page. Each claim records what its source establishes and what was inferred from it. Each refusal records its reason. So checking its work is part of the process, not something you need an audit team to do.

You cannot do that to a team of people. Nobody can audit which submissions a research officer skimmed at 11pm in week nine, or whose framing a draft absorbed because it arrived as clean, numbered recommendations. Committees have the same weakness for the polished, on-message submission. They just don't leave the evidence.

Which brings me back to the question. The frame, access, control, continuity and transparency, was mine. The submissions that answered in that vocabulary were heard. The ones answering a neighbouring question were passed over. Rumelt starts every strategy with a diagnosis because the diagnosis decides what you will see. In an inquiry, whoever writes the outline writes most of the answer. That is as true of a secretariat as of a machine. The difference is that the machine can show you.

So is it a team of ten? No. It doesn't run hearings, it doesn't negotiate with members, and it shouldn't decide anything. It is a fast first draft with all of its working shown. That is what an expert needs to spend eleven weeks on judgement rather than eleven weeks on reading.

The committee reports by 30 November - I'm planning on comparing, and not just the conclusions: whose submissions it cites, and in whose words.