15 normative assertions, 85 language-neutral vector steps

reference architecture      4 runtimes x 2 dataset boundaries   15/15 each
independent implementation  shares no code with the above       15/15
mutation analysis           17 / 17 targeted violations detected
                            15 / 15 assertions independently exercised
                             2.2 detecting assertions per mutant (mean)
execution safety            0 / 576 prohibited executions
                            0 /  24 in the evaluation set
tests                       405 passed

Authorized Recall@5         filter after truncation     0.853
                            filter before truncation    0.960
                                                       +0.107

Start here

An agentic dataset describes itself, advertises bounded capabilities, accepts a semantic intent, decides whether an action is admissible, executes only what was admitted, refuses the rest, and leaves evidence.

The specification is fifteen assertions, each naming a failure it rules out. Everything else exists to check them.

GRANTED        -> approval token -> execution reachable
REFUSED        -> no token       -> execution unreachable
INDETERMINATE  -> no token       -> execution unreachable

INDETERMINATE is not a refusal. An evaluator that is unreachable or out of budget has not decided anything, and recording that as a refusal invents an authority nobody exercised.

Check an implementation

# Not on PyPI yet. During the release candidate, install from a clone:
git clone https://github.com/agentic-datasets/reference
pip install ./reference/packages/agentic-dataset-conformance

# Once 0.1.0rc1 is published, pip skips pre-releases unless asked:
#   pip install --pre agentic-dataset-conformance

agentic-dataset-conformance run --subject yourmodule:your_factory

The harness imports no implementation — not even the reference one — and a test asserts it. The worlds and vectors are CC0; export them and write a runner in another language if you prefer.

What this is not

No deployment, no real data, no model in the loop by default, no latency or cost claim, and no security guarantee. capabilities() is the subject's own report of itself, so a subject that under-reports passes AD-002 while hiding a tool. Conformance is a claim an implementation makes about itself, made checkable — not an adversarial audit.

Claims is frozen and lists everything asserted, and the one thing explicitly not.

Status

Release candidate. Public before v0.1.0 is tagged, deliberately, so a finding can still change the artifact rather than becoming errata against a DOI. The independent implementation is a 250-line toy written by the same person who wrote the specification: that establishes independence from the reference code, not from its author's reading of the contract.

Interpretive independence is the next threshold, and it needs somebody else. How to do it.

Agentic Dataset Conformance Suite

Fifteen assertions that any implementation of the agentic-dataset model must satisfy, in any framework.

Status: IMPLEMENTED, PORTABLE, AND PASSING AGAINST NINE SUBJECTS.

All fifteen are checked through a public interface, against four runtimes at two dataset boundaries and against an independent implementation that shares no code with any of them. Thirteen deliberately broken variants are each caught by the assertion named for them.

The harness imports nothing from any implementation. The normative artifacts — the world, the vectors, the expectations — are JSON in conformance/.

Run it: agentic-dataset-conformance run --subject conformance.subjects:subjects --mutants

Source: docs/ARCHITECTURE-ADK.md §107, generalised.

Why this file exists

Three reference architectures now describe the same control plane on LangChain/LangGraph, LlamaIndex and Google ADK. Three documents that agree with each other prove nothing — they were written by the same person from the same model.

A conformance suite is what makes the agreement checkable. If the same fifteen assertions pass against independent runtimes with different primitives, the claim "the governance model is not a property of a framework" stops being an argument and becomes a result.

That is the difference between a design document and a research artifact, and it is the reason PLAN.md M2 exists.

One qualification, stated here rather than in a footnote. The four ports in this repository share a single ControlPlane. That is deliberate — an assertion that passed because each port re-implemented its own policy would be four experiments rather than one — but it means the result is about the model being expressible in four runtimes, not about four independent implementations agreeing. A second implementation written by someone else from this file alone is the experiment this artifact does not run.

The assertions

How each must be tested

Deterministically, without an LLM, and by absence rather than by wording.

The recurring failure in agent testing is asserting on the model's apology:

assert "I cannot" in response          # tests nothing

The property is structural:

assert result.decision == "REFUSED"
assert result.grant is None
assert result.tool_calls == []
assert result.mcp_calls == []
assert result.a2a_calls == []

AD-003 through AD-006 are the load-bearing four. If those hold, a misbehaving model cannot cause a policy violation — it can only cause a bad answer. That is the whole argument for putting admission in code rather than in a prompt.

AD-015 is the only one with a rate rather than a boolean, and its target is exactly zero. It does not get averaged into a score.

Non-goals

Conformance is not a security audit. Passing AD-002 proves consistency between declared and observable behaviour within the subject's advertised capability surface; it does not prove that the subject has disclosed every capability it possesses. capabilities() is the implementation's own report of itself.

So a conformance pass is a claim an implementation makes about itself, made checkable. Treating it as a guarantee against a hostile implementation misreads it, and no interface of this shape could provide one — an adversarial audit needs access to the binary, not to an interface the binary implements.

Three further things this suite does not attempt:

• Performance. Latency, throughput, cost and concurrency are unmeasured and unasserted.
• Semantic quality. Whether the right dataset was chosen, or the answer was any good, is measured statistically elsewhere and deliberately never averaged into these fifteen.
• Completeness of the model. The assertions rule out the failures named in the table above. They are not a claim that no other governance failure exists.

Gate shape

                                  gate      measured
AD-001 .. AD-015                = 100%      15/15 x 9 subjects
Authorized Recall@5            >= 0.95      0.960  (filter before truncation)
                                            0.853  (filter after truncation)
Capability selection accuracy  >= 0.97      1.000
Trajectory validity            >= 0.95      1.000
Groundedness                   >= 0.93      not measured -- no model-generated
                                            answer exists in this build

The two Authorized Recall@5 rows are the same retriever and the same corpus, differing only in where the authorization filter sits. The gate is a statement about filter placement, not about retrieval quality.

Governance is tested as an invariant. Semantic quality is tested statistically. Running the two through one number destroys both.

Framework independence

The suite is implemented once and run against every runtime without changing an assertion. What differs between them is only where control flows:

Each of those is run twice: once against local capabilities, and once with every dataset behind a real MCP client session. That second axis earned its place — two of the six defects in docs/FINDINGS.md were visible only across the boundary, and two only under the async runtimes.

If an assertion cannot be expressed in one of the runtimes, that is a finding about the assertion, not about the framework. None had to be dropped; docs/FINDINGS.md records the two places where the implementation departed from the architecture documents instead.

How to be tested

An implementation is conformance-testable when it exposes four things (packages/agentic-dataset-conformance/src/agentic_dataset_conformance/interface.py):

load_world(world)      adopt descriptors, principals and a policy version
capabilities()         report every operation it will actually execute
step(step)             run one control verb, return an Observation
reset()                forget cache and evidence

Observation is the entire observable surface — decision, reason, policy id, whether a grant exists, the admitted and executed scopes, tool/MCP/A2A call lists, cache hit, evidence rows, errors. If a property cannot be established from an Observation, a world and a sequence of steps, it is not part of the portable contract.

The control verbs are in verbs.md beside the interface. The worlds and vectors are JSON under conformance/, so an implementation in Rust, Go, TypeScript or Java can be checked without reproducing Python object semantics — Python is one runner, not the specification.

packages/agentic-dataset-conformance/src/agentic_dataset_conformance/toy.py is a 250-line worked example that imports the interface and nothing else, and passes all fifteen.

docs/PORTABILITY.md records the three assertions whose shape changed when they moved outside, the one property that was deliberately widened, and the one that cannot be checked from outside at all.

Relationship to the existing implementation

ok-governed-motion already satisfies the spirit of AD-003, AD-004 and AD-005 in Rust, for robot motion rather than datasets: Verdict::{Approved, Refused, Indeterminate}, and only an approval yields the token that starts motion. Its serialised reasons — EVALUATOR_UNAVAILABLE, EVALUATOR_TIMEOUT — are the strings this suite should assert against, so a fourth implementation in a fourth domain does not quietly diverge.

That is worth noting because it means three of the fifteen assertions already have a passing implementation, in a language none of these ports use.

tests/test_verdict_parity.py closes that loop: it asserts the Python strings and rationales against the literals above, and reads policy.rs directly when ok-governed-motion is checked out beside this repository. The Rust and Python verdicts cannot drift without a test failing.

What the portable contract can and cannot reach

Milestone M7 moved conformance from introspecting this implementation to observing any implementation. All fifteen assertions survived the move, but three of them changed shape doing it, and one property was genuinely lost. Both facts are recorded here rather than absorbed quietly, because a specification that silently redefines an assertion to fit its harness is worse than one that admits the assertion was partly white-box.

Before and after

The three assertions that changed shape

AD-003 — from "the raw call raises" to "execution implies a grant"

The white-box check called a capability directly with no authorization and asserted it raised. There is no portable equivalent: an implementation in another language need not have a callable object to poke at.

The portable form is a universally quantified invariant over every observation the subject produces: executed ⟹ granted. Across the current vectors that is 77 observations rather than one poke, plus a vector where an expired token must stop execution.

This is stronger, not weaker. The white-box version proved one call site was guarded; the invariant proves no observed execution ever lacked authority.

AD-007 — from "widening is rejected at the call" to "the scope executed

under is never wider than the scope admitted"

Same move. The check used to construct a widened AuthorizationScope and pass it to the registry. Now the subject reports grant_scope and executed_scope and the harness checks containment on every observation, plus a delegation vector that attempts an actual widening.

AD-008 — from "the cache key contains these dimensions" to "these

distinctions produce a miss"

This one genuinely lost something, and gained something.

Lost: the white-box check took the cache key apart and asserted that altering each of eight dimensions changed the digest. That is unreachable from outside, and it should be: it constrains how an implementation separates principals, not whether it does.

Gained: the portable form is behavioural — same question by a different principal class misses; a new dataset revision misses; a new policy version misses; a revoked principal does not reach the cache at all. An implementation that uses per-principal cache partitions instead of a composite key passes the behavioural form and would have failed the structural one, and it is not doing anything wrong.

So the portable assertion is wider than the white-box assertion, and the structural check remains in agentic_dataset.reference_suite as an implementation-specific test, where it belongs.

What is still not expressible, and will not be

Honesty of the subject. capabilities() is the subject's own report of what it will execute. A subject that under-reports passes AD-002 while hiding a tool. Conformance here is a claim an implementation makes about itself, made checkable — it is not an adversarial audit of a binary, and no interface of this kind can be. Anyone treating a conformance pass as a security guarantee against a hostile implementation has misread it.

Anything about latency, cost or concurrency. Not measured, not asserted, out of scope.

What the toy demonstrates

packages/agentic-dataset-conformance/src/agentic_dataset_conformance/toy.py is roughly 250 lines and shares nothing with the reference implementation but the interface module. It has no framework, no MCP, no descriptor class, no policy engine, no ledger — grants are integers in a dict and the cache is a dict.

It passes all fifteen.

That is the evidence that the assertions are properties of the contract rather than of the reference architecture. The reference implementation is one way to satisfy them; the toy is a second, deliberately unlike the first.

It also produced the first finding: its initial version derived capabilities() from the descriptors, which made every advertised capability executable by construction. AD-002 caught it on the first run. A suite that could not have caught that would not have been worth building.

How the method got here

The testing approach was not designed and then vindicated. It changed twice because of evidence, and the order matters:

white-box suite
      |
      +-- missed F-010 entirely
                |
                v
language-neutral vectors
      |
      +-- exposed F-010: the MCP server served the revision it was built with
      |
      +-- exposed F-011: the independent toy conflated advertised with implemented
                |
                v
mutation characterisation
      |
      +-- F-011 retained as a reproducible mutant (M02)
      |
      +-- exposed a coverage gap: 4 of 15 assertions had no mutant of their own

Each step was taken because the previous one turned out to be insufficient, and each produced a finding the previous one could not have. That is a weaker claim than "the design was correct from the start" and a more useful one.

Mutation results

Seventeen deliberately broken variants, each removing exactly one guarantee. The matrix is what agentic-dataset-conformance run --subject conformance.subjects:subjects --matrix prints; the committed run is in runs/mutation-matrix.txt.

target detection : 17/17 mutants caught by their intended assertion
cross-detection  : 2.2 assertions per mutant on average
coverage         : 15/15 assertions have a mutant of their own

The coverage line is there because it was not always 15/15. The first version of this analysis had thirteen mutants covering eleven assertions, which meant AD-002, AD-009, AD-013 and AD-014 were exercised only as cross-detectors — never as the assertion under test. Nothing in the pass/fail output showed that. Drawing the matrix showed it immediately, and four mutants were added.

The off-diagonal entries are a result, not noise. They say the fifteen assertions are not orthogonal, which is what safety invariants ought to look like: removing the prohibition check breaks AD-015 and AD-004 and the evidence assertions, because a prohibited action that executes also records a grant where a refusal belonged. An assertion whose row contains nothing but its own T is doing work nothing else does — AD-005, AD-011 and AD-012 are those, and that is worth knowing about them.

The average of 2.2 is therefore a characterisation of the suite rather than a score. It should not be driven up or down.

Reproduce: agentic-dataset-conformance run --subject conformance.subjects:subjects --matrix

Contributing

The most valuable contribution to this repository is not a pull request.

Wanted: an independent conformance implementation

You need no permission and no license from anybody to do this. The specification is CC BY 4.0, the normative vectors are CC0, and the conformance runner is Apache-2.0 — including for a commercial implementation. Only the reference implementation in src/agentic_dataset/ (excluding conformance/ and authorized_recall/) is restricted, and you do not need it.

The four runtime ports here share one reference ControlPlane. That is deliberate — an assertion that passed because each port re-implemented its own policy would be four experiments rather than one — but it caps what the result can claim. Today the matrix shows that one governance model is expressible in four runtimes. It does not show that independent implementations agree on the governance semantics, which is the stronger and more interesting claim.

Closing that gap needs someone else's implementation, written from CONFORMANCE.md, in any language, that does not use this reference control plane. Publishing whether it passes — and especially which assertions turned out to be ambiguous, under-specified or unimplementable — is more useful to this project than any amount of code review.

If you do this, please open an issue. A finding that AD-007 is ambiguous is a better outcome than a green tick.

How to do it

# Not on PyPI yet. From a clone, during the release candidate:
pip install ./packages/agentic-dataset-conformance

1. Implement ConformanceSubject — four methods, in interface.py. The control verbs are in verbs.md beside it.
2. agentic-dataset-conformance run --subject yourmodule:your_factory

You do not need this repository. The harness, the vectors and a worked example subject are all in that one package, which imports no implementation at all.

The world and the vectors are JSON under conformance/, so an implementation in another language needs a runner for that JSON rather than a reimplementation of this harness. packages/agentic-dataset-conformance/src/agentic_dataset_conformance/toy.py is a 250-line worked example that imports the interface and nothing else.

What is honest about the current state

The toy is independent of the reference implementation but not of its author. One person's reading of their own specification is the weakest kind of independence, and it is the reason this section exists. A second reading is the experiment.

docs/PORTABILITY.md records where the contract had to change shape to leave the building — AD-003 and AD-007 became universally quantified invariants, AD-008 became behavioural rather than structural — and the one thing conformance cannot establish at all: a subject that under-reports its own capabilities passes AD-002 while hiding a tool. Conformance here is a claim an implementation makes about itself, made checkable. It is not an adversarial audit.

Other contributions that would help

• A harder discovery corpus. MRR is 1.000 on the current one, so the retrieval task is easy and the Authorized Recall numbers are measured in easy conditions.
• Authorized Recall@K applied to a real corpus with real authorization data. the authorized-recall package is deliberately separable from everything else so this needs no adoption of the control plane.
• A fifth runtime adapter. The bar is that it contains no policy decision. If you find yourself re-deciding something to make it fit, that is a finding about the model and worth an issue.
• An assertion that cannot be expressed in some runtime. Per CONFORMANCE.md, that is a finding about the assertion, not the framework.

Running things

pip install -e ".[all]"

agentic-dataset-conformance run --subject conformance.subjects:subjects      # AD-001..AD-015, every runtime
python -m authorized_recall
python evals/evaluate.py
pytest -q

agentic-dataset-conformance run --subject conformance.subjects:subjects exits non-zero on any failure. CI also runs the core with no framework installed, which is where an import leaking out of adapters/ fails.

House rules for code

• The core stays dependency-free. Framework imports live in adapters/ and mcp_boundary.py, nowhere else.
• No policy decision in an adapter. Every runtime calls ControlPlane.admit.
• Assert on absence, not on wording. assert "I cannot" in response tests nothing; assert result.grant is None and result.execution.tool_calls == [] tests the property.
• If the implementation disagrees with a document in docs/, record it in docs/FINDINGS.md rather than editing the document to agree. The architecture documents predate the code and are evidence of what was designed before it was built.

Results

15 normative assertions, 85 language-neutral vector steps

reference architecture      4 runtimes x 2 dataset boundaries   15/15 each
independent implementation  shares no code with the above       15/15
mutation analysis           17 targeted violations              17/17 detected
                            15/15 assertions covered             2.2 assertions
                                                                 per mutant
execution safety            0 / 39  prohibited steps, per subject
                            0 / 576 prohibited executions, white-box matrix
                            0 /  24 prohibited executions, evaluation
tests                       405 passed

Authorized Recall@5         filter after truncation     0.853
                            filter before truncation    0.960
                                                       +0.107

The two prohibited-execution denominators are separate because they are separate experiments: 576 is 72 attempts in each of the 8 conformance configurations, 24 is the adversarial set in the evaluation. Reporting 0/600 would merge two populations that were never sampled together.

Everything below was produced by running the code in this repository. Raw output is in runs/; the commands that produce it are in the README.

Measured 2026-09-01 on Python 3.12.13, Linux, with:

langgraph 1.2.11 · langchain-core 1.6.1 · llama-index-core 0.14.24
google-adk 2.8.0 · mcp 2.1.1 · pytest 9.1.1

1. The conformance result

AD-001 … AD-015 against four runtimes, at two dataset boundaries.

RUNTIME           RESULT  PASSED
native+local      PASS    15/15
langgraph+local   PASS    15/15
llamaindex+local  PASS    15/15
adk+local         PASS    15/15
native+mcp        PASS    15/15
langgraph+mcp     PASS    15/15
llamaindex+mcp    PASS    15/15
adk+mcp           PASS    15/15

AD-015 prohibited execution rate: 0.000 (target exactly 0)

120 assertion-runs, no failures, no assertion dropped as inexpressible.

What this supports. The same fifteen assertions hold across four runtimes with genuinely different primitives — a conditional edge in LangGraph, typed event dispatch in LlamaIndex Workflows, a before-tool callback in ADK, and a straight function sequence in the framework-free reference — and across two dataset boundaries, local and MCP. The governance model is not a property of the framework.

What this does not support. The four runtimes share one ControlPlane. That is deliberate — an assertion that passed because each port re-implemented its own policy would be four experiments, not one — but it means the result is about the expressibility of the model in four runtimes, not about four independent implementations agreeing. A second implementation by someone else, from CONFORMANCE.md alone, is the experiment this one is not.

A ninth subject shares nothing. packages/agentic-dataset-conformance/src/agentic_dataset_conformance/toy.py is 250 lines written from the specification — no framework, no MCP, no descriptor class, no policy engine; grants are integers in a dict — and it passes all fifteen. That is the evidence that the assertions are properties of the contract rather than of the reference architecture.

What is still missing is independence of authorship. The toy was written by the same person who wrote the specification, and one person's reading of their own document is the weakest kind of independence. The outstanding experiment is a second reading by somebody else.

And the suite would now notice a broken implementation. Seventeen variants, each removing exactly one guarantee, are each caught by the assertion named for them, and every one of the fifteen assertions has a mutant of its own — agentic-dataset-conformance run --subject conformance.subjects:subjects --matrix.

The detection matrix in PORTABILITY.md reports two separate things, because they mean different things: target detection (17/17) says the suite is sensitive to each named violation, and cross-detection (2.2 assertions per mutant) says the assertions are not orthogonal. The second is a characterisation rather than a score. It is also how the coverage gap was found: the first version of the analysis had 13 mutants covering 11 assertions, and nothing in the pass/fail output revealed that four assertions were never under test.

The move outside cost something, and PORTABILITY.md records it: AD-003 and AD-007 became universally quantified invariants over every observation rather than single pokes at a call site (stronger), and AD-008 became behavioural rather than structural (wider, and the structural version survives in agentic_dataset.reference_suite).

The suite failed on this implementation five times before it passed; see FINDINGS.md F-004 … F-009. Two of those were visible only in the MCP configuration and two only under the async runtimes, which is the argument for the matrix rather than a single run.

1a. Mutation analysis

Seventeen variants, each removing exactly one guarantee, run against the same vectors. T is the assertion the mutant was written for; x is a redundant detection.

        M01 M02 M03 M04 M05 M06 M07 M08 M09 M10 M11 M12 M13 M14 M15 M16 M17
        --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- ---
AD-001   T   .   .   .   x   .   .   .   .   .   .   .   .   .   .   .   .    2
AD-002   .   T   x   .   .   .   .   .   .   .   .   .   .   .   .   .   .    2
AD-003   .   .   T   T   .   .   .   .   .   .   .   .   .   .   .   .   .    2
AD-004   .   .   .   .   T   .   .   .   .   .   .   .   .   .   .   .   x    2
AD-005   .   .   .   .   .   T   .   .   .   .   .   .   .   .   .   .   .    1
AD-006   .   .   .   .   x   .   T   .   .   .   .   .   .   .   .   .   .    2
AD-007   .   .   .   .   .   .   .   T   .   .   .   .   .   .   x   x   .    3
AD-008   .   .   .   .   .   .   .   .   T   T   .   .   .   .   .   .   .    2
AD-009   .   .   .   .   .   .   .   .   .   .   T   x   x   x   .   .   x    5
AD-010   .   .   .   .   .   .   .   .   .   .   x   T   x   x   .   .   x    5
AD-011   .   .   .   .   .   .   .   .   .   .   .   .   T   .   .   .   .    1
AD-012   .   .   .   .   .   .   .   .   .   .   .   .   .   T   .   .   .    1
AD-013   .   .   x   x   .   .   .   x   .   .   .   .   .   .   T   .   .    4
AD-014   .   .   x   x   .   .   .   x   .   .   .   .   .   .   .   T   .    4
AD-015   .   .   .   .   x   .   .   .   .   .   .   .   .   .   .   .   T    2

M01  AD-001  descriptor-not-validated
M02  AD-002  advertised-means-implemented
M03  AD-003  executes-without-a-grant
M04  AD-003  expired-tokens-accepted
M05  AD-004  refusal-still-mints-authority
M06  AD-005  indeterminate-becomes-refusal
M07  AD-006  default-allow
M08  AD-007  delegation-widens-scope
M09  AD-008  cache-ignores-principal
M10  AD-008  cache-ignores-revision
M11  AD-009  evidence-omits-principal
M12  AD-010  refusal-leaves-no-evidence
M13  AD-011  evidence-omits-revision
M14  AD-012  evidence-omits-policy-version
M15  AD-013  remote-delegation-unchecked
M16  AD-014  handoff-unchecked
M17  AD-015  prohibitions-ignored

target detection : 17/17 mutants caught by their intended assertion
cross-detection  : 2.2 assertions per mutant on average
coverage         : 15/15 assertions have a mutant of their own

T = caught by its target assertion   x = caught redundantly
. = not detected                     ! = target failed to catch it

Three separate figures, because they mean three different things. Target detection says the suite is sensitive to each named violation. Coverage says every assertion is exercised as the assertion under test rather than only as a bystander. Cross-detection says the assertions are not orthogonal.

The last one is a characterisation, not a score, and is deliberately not being driven anywhere. AD-005, AD-011 and AD-012 detect only their own mutant — a single behavioural check carries the whole detection burden for each — and that is worth knowing about those three rather than concealing.

Coverage was 11/15 in the first version of this analysis, and nothing in the pass/fail output showed it. Drawing the matrix did.

2. Authorized Recall@K — milestone M6

Defined and measured in the separately distributed and separately licensed authorized-recall package (Apache-2.0), which has no dependency on the rest of the repository. Its README carries the mathematical definition, the two conventions, and a proof that the pre/post-filter gap is non-negative for every ranking, K and predicate — so the sign of every gap below is guaranteed and only the size is empirical.

40 synthetic datasets across 8 domains, 24 queries, 4 authorization profiles, 96 query-principal pairs. Relevance is defined by construction: a dataset is relevant to a query if it is in the query's domain.

  K   Recall  ARecall  ARecall    gap     P@K   nDCG unusable
                 post      pre                       in top-K
  1    0.200    0.750    0.750 +0.000   1.000  1.000    68.8%
  3    0.483    0.835    0.863 +0.027   0.806  0.843    68.8%
  5    0.867    0.954    0.988 +0.033   0.867  0.874    68.5%
 10    1.000    1.000    1.000 +0.000   0.500  0.952    39.1%

Excluding the 66 pairs where nothing relevant is authorized at all — those score 1.0 by convention, which inflates the mean:

  K  ARecall post  ARecall pre     gap
  1         0.200        0.200  +0.000
  3         0.473        0.560  +0.087
  5         0.853        0.960  +0.107
 10         1.000        1.000  +0.000

The number. At K=5, moving the authorization filter ahead of truncation takes Authorized Recall@5 from 0.853 to 0.960 (+0.107). Plain Recall@5 stays at 0.867 and cannot see the difference. Against the ≥ 0.95 gate in CONFORMANCE.md, filter-after-truncate fails and filter-before-truncate passes.

68.8% of what retrieval surfaces is unusable to the principal who asked. Standard Recall@K scores every one of those as a success.

Caveats, in order of how much they matter.

1. The corpus is synthetic and the relevance judgements are by construction. The absolute numbers are properties of that construction. The gap between the two ARecall columns is a property of where the filter sits, which is what the metric was defined to isolate.
2. Retrieval is TF-IDF cosine, not embeddings. MRR is 1.000, so the retrieval task is easy; a harder corpus would move all three columns.
3. The 1.0-for-empty convention is a decision, not arithmetic. Both means are reported so the decision is visible.

This metric now has a number. It did not before. It has one number, from one synthetic corpus, from the implementation that proposed it.

3. Evaluation — milestone M5

Six evaluators, reported separately, five repetitions.

METRIC                   KIND          VALUE  GATE  STATUS
policy decision correct  invariant     1.000  1.00  PASS
refusal reason correct   invariant     1.000  1.00  PASS
provenance complete      invariant     1.000  1.00  PASS
capability selection     statistical   1.000  0.97  PASS
dataset selection        statistical   1.000  0.95  PASS
trajectory validity      statistical   1.000  0.95  PASS
prohibited executions    invariant     1.000  1.00  PASS   0 of 24 executed
groundedness             not-measured    n/a    --  N/A

Every spread is zero, and that is not a result. The interpreter is deterministic, so repetition measures nothing here. Substituting LLMInterpreter is what makes the statistical rows carry a spread — and the invariant rows are the ones that must not move when it does.

Groundedness is not measured. It needs a model-generated answer, and this build synthesises no prose; scoring the deterministic formatter against its own input would produce 1.000 and mean nothing. Reported as N/A rather than as a number.

Capability selection measured 0.800 on the first run — see FINDINGS.md F-008.

4. Tests

405 passed

pytest parametrises the conformance suite down to one test per assertion per configuration, so a failure names the assertion and the runtime. tests/test_verdict_parity.py reads ok-governed-motion's policy.rs directly when it is checked out beside this repository, and skips otherwise. The run above was with it present, so the serialised strings are verified against the Rust source rather than against a copy of it.

5. What is not here

• No production deployment. Everything runs in one process. The MCP boundary is a real client session over an in-memory transport, not a network.
• No model in the loop by default. The interpreter is rule-based so the suite is deterministic. LLMInterpreter accepts any callable.
• No real data. The datasets are synthetic, and the capability bodies are trivial on purpose: if a conformance run passes it is because the gate held, not because the payload was clever.
• Latency and cost are not measured. Nothing here is a performance claim.

Frozen claims

Frozen 2026-09-02, before the release candidate. This file exists so that the remaining work — licence alignment, rename, publication, a technical report — cannot quietly strengthen what is being asserted. Anything said about this project in a README, an abstract, a post or a paper should be checkable against this table.

A claim moves out of this file only by being measured differently, and the measurement changes with it.

On claim 4

Extended 2026-09-03. This is the freeze working as intended: the claim did not change, the measurement did.

Until now every measured subject was Python, so "language-neutral" described the form of the vectors — JSON, no Python object semantics — rather than something observed. A TypeScript subject in agentic-datasets/showcase now runs the same suite and reports the same numbers:

Seven mutants are each caught by the assertion named for them, in both.

What was checked before recording this, rather than taken from the other repository's README: the vectors and worlds in the TypeScript project are byte-identical to agentic_dataset_conformance/data/ (diff -rq, clean, 15 files each, CC0 licence file included); its engine has no non-relative imports, so it shares no runtime with the reference; and the Python baseline was re-run here to confirm 15/77/39/0 rather than trusted as a constant.

What this establishes. The vectors execute outside Python, through the public interface, with no shared runtime. CONFORMANCE.md said an implementation "in Rust, Go, TypeScript or Java can be checked without reproducing Python object semantics — Python is one runner, not the specification." That was a design intention; it is now an observation.

What it does not establish. The TypeScript subject is a transcription of toy.py by the same author, not a fresh reading of the specification. Claim 6 is unchanged and claim 6's limitation still stands: interpretive independence requires somebody else, and remains the contribution this project most needs. A second language is not a second reader.

On claim 13

capabilities() is the subject's own report of itself. A subject that under-reports passes AD-002 while hiding a tool. Conformance here is a claim an implementation makes about itself, made checkable — not an adversarial audit, and no interface of this shape could be one.

This line is load-bearing. Without it the conformance vocabulary drifts into sounding like a security certification, which is the single easiest overclaim available to this project.

On claim 6

The toy establishes independence from the reference implementation's code. It does not establish independence from the author's reading of the specification, because the same person wrote both. Interpretive independence is unclaimed and is the next validation threshold — it requires somebody else, and no further implementation written here would supply it.

On claim 12

The gap's sign is proved for every ranking, K and predicate. Its size (0.853 → 0.960 at K=5) is measured on one synthetic corpus with relevance by construction and a TF-IDF retriever, and belongs to that construction.

Not measured at all

Latency, cost, concurrency, throughput. Semantic answer quality beyond the evaluators in evals/. Groundedness — reported as N/A rather than as a number, because this build synthesises no prose and the metric would score its own formatter.

Findings

Things the implementation disagreed with, and things the suite caught.

CONFORMANCE.md asks that an assertion which cannot be expressed in one of the runtimes be recorded rather than dropped. In the event none had to be dropped, so this file is mostly the other kind of finding: places where building the thing contradicted the document describing it, and defects the checks found in the code they were checking.

Disagreements with the architecture documents

F-001 — INDETERMINATE has exactly two reasons, not three

docs/ARCHITECTURE.md §2.3 lists an incomplete descriptor among the causes of INDETERMINATE, alongside an unavailable evaluator and a timeout. PLAN.md open question 1 requires the port to preserve ok-governed-motion's serialised strings, and that enum has exactly two members.

Decision: two. A malformed descriptor is refused, under AD-POL-002 DESCRIPTOR_INVALID. The distinction IndeterminateReason exists to carry is "no rule answered" versus "a rule said no", and a descriptor that fails validation has been answered -- by the rule that validates descriptors. Adding a third member to reconcile the documents would have widened the one type whose narrowness is the point.

tests/test_verdict_parity.py::test_there_are_exactly_two_indeterminate_reasons is what stops this being reconciled later by quietly adding a member.

F-002 — the ADK port instantiates no LlmAgent

docs/ARCHITECTURE-ADK.md describes the control plane over ADK including model-driven tool selection. The port here composes BaseAgent subclasses under a SequentialAgent, runs them through a real Runner, wraps capabilities as real FunctionTools, and invokes a guard with ADK's before_tool_callback signature at the point ADK would invoke it -- but no model selects the tool.

The conformance suite has to run without an API key and without variance, and a model in that loop would add nothing to what is being asserted: the guard runs before the tool either way. What the ADK result shows is that ADK's agent, tool and callback primitives can express the control plane. It shows nothing about ADK's model integration, and no claim beyond that should be made from it.

F-003 — descriptors are JSON, not YAML

docs/ARCHITECTURE.md §4 serialises a descriptor as YAML. The core has no dependencies, and YAML is not in the standard library. Descriptors are JSON. Nothing in the model depends on the encoding.

Defects the conformance suite found in this implementation

Each of these was found by a check, not by review, which is the argument for the checks.

F-004 — the ledger recorded a dataset it had never resolved (AD-009)

EvidenceRecord.dataset_id was populated from the requested dataset name. A request naming a dataset that does not exist therefore produced a row claiming a dataset, with version, revision and schema version all null. AD-009 failed on it immediately.

Fixed by splitting the field: requested_dataset is what was asked for, dataset_id is what was read, and the three fields describing a dataset are required only when one was actually resolved.

F-005 — a configured evidence ledger was silently discarded

ControlPlane.__init__ used ledger or EvidenceLedger(). EvidenceLedger defines __len__, so an empty ledger is falsy, so passing one in threw it away and substituted a fresh in-memory ledger. SemanticCache has the same shape and the same bug. Nothing failed loudly: evidence was written, to the wrong object.

Found by tests/test_ledger.py::test_every_terminal_arm_leaves_a_row, which asked for the file on disk afterwards. Fixed with is None.

F-006 — the plan validated a list it had just built

ControlPlane.plan constructed the plan and then checked that every step named the admitted capability. Since it had constructed every step itself one line earlier, the check could not fail, and a plan mutated between planning and execution -- which is where a compromised planner would put an extra step -- was not checked at all.

Found by tests/test_adversarial.py. The guard now sits in execute, at the point the plan is consumed.

F-007 — a descriptor lost its age crossing the MCP boundary

DatasetDescriptor.to_dict omitted age_s, so a descriptor read back over MCP had no age and every freshness rule passed. AD-004 caught it as a request that should have been refused for FRESHNESS_UNSATISFIABLE and was granted -- only in the MCP configuration, which is the reason the suite is run at both dataset boundaries rather than one.

F-008 — capability selection missed on a keyword ordering

The rule-based interpreter matched recovery before outlier, so detect outliers in the recovery distribution resolved to calculate_yield. Capability selection measured 0.800 against a 0.97 gate.

Fixed by ordering the rules most-specific-first. Worth recording because the failure is invisible without the evaluator and harmless with it: the wrong capability produced a refusal or a wrong answer, never an unauthorised execution.

F-009 — a synchronous MCP client inside an async runtime

The LlamaIndex and ADK runtimes drive their own event loops, so a synchronous MCP client called from inside a capability was already on a loop and asyncio.run raised. Eight of fifteen assertions failed in both runtimes, in the MCP configuration only.

Resolved by owning a loop on a worker thread rather than making the control plane async. Admission is not an I/O-bound problem, and colouring it async to accommodate one transport would push await into every policy call site.

F-010 — the MCP server kept serving the revision it was built with

Found by the portable suite, at the MCP boundary only, and missed entirely by the white-box one.

_add_tool captured descriptor.revision when the server was constructed, so after the dataset's data changed the far side went on verifying grants against the revision it had started with. The white-box AD-008 had only asserted that a revision change produces a cache miss; the vector additionally asserts the next identical request is a hit, and that second half is what exposed it.

Fixed by reading the revision from the registry at call time. Recorded because it is the clearest argument for the portable suite existing: the assertion was the same, the implementation was the same, and only the way the property was expressed differed.

F-011 — the toy conflated advertised with implemented

The first version of the toy subject (now agentic_dataset_conformance.toy) derived capabilities() from the descriptors it had been given, which made every advertised capability executable by construction. AD-002 failed on the first run.

This is a finding about the suite rather than about the implementation: it is the only direct evidence that the assertions catch a real mistake made in earnest rather than one planted to be found. The seventeen mutants in agentic_dataset_conformance.mutations are planted; this one was not.

The semantic cache is lexical

DatasetIntent.semantic_key normalises case, punctuation, word order and a short closed list of function words. It does not understand paraphrase, and tests/test_cache_isolation.py asserts that a genuine paraphrase misses.

Under-hitting costs latency. Over-hitting returns one principal's answer to another principal's question. The name is the most generous thing about the implementation, and an embedding-keyed variant belongs behind the same authorization dimensions rather than instead of them.

agentic-dataset-conformance

Fifteen normative assertions about governed datasets, as language-neutral executable vectors, checkable against any implementation without access to its internals.

pip install --pre agentic-dataset-conformance   # 0.1.0rc1 is a pre-release
agentic-dataset-conformance run                 # against the built-in subject

Not yet on PyPI during the release candidate; install from a clone until it is.

This package contains no implementation of the contract — not even the reference one. That is the property it exists to have: a conformance suite that imports the thing it tests is testing itself.

What the assertions rule out

Every one is checked structurally. Not "the answer said no" but: the decision is REFUSED, no grant exists, and the tool, MCP and A2A call lists are all empty.

Testing your implementation

Implement four methods — load_world, capabilities, step, reset — and return an Observation from each step. interface.py is the whole contract and verbs.md is the control-verb vocabulary.

agentic-dataset-conformance run --subject mypackage.conformance:make_subject

--subject takes module:attribute, where the attribute is a subject, or a callable returning one or several. agentic_dataset_conformance.toy is a 250-line worked example that implements the contract with no framework, no vector store and no policy engine — grants are integers in a dict — and passes all fifteen.

Checking that the suite would notice

agentic-dataset-conformance run --matrix

Seventeen deliberately broken variants, each removing exactly one guarantee. Every one is caught by the assertion named for it, every assertion has a mutant of its own, and the off-diagonal entries show where the assertions overlap. A suite that cannot fail is decoration.

The vectors are CC0

agentic-dataset-conformance vectors --export ./vectors

The worlds and vectors under data/ are dedicated to the public domain: no attribution required, no conditions. Copy them into a Rust crate, a Go module or a TypeScript package and write your own runner. Python is one runner, not the specification.

The software around them is Apache-2.0, so the distribution as a whole is Apache-2.0 AND CC0-1.0.

What conformance does and does not establish

capabilities() is the subject's own report of itself. A subject that under-reports passes AD-002 while hiding a tool. Conformance here is a claim an implementation makes about itself, made checkable — not an adversarial audit, and no interface of this shape could be one.

Links

• The specification
• What the portable contract can and cannot reach
• Results, with the caveats attached to each number

Authorized Recall@K

Retrieval quality measured over the subset a principal may actually use.

This package has no dependency on the rest of the repository. The metric takes a predicate, not a Principal, so a system using RBAC, ABAC, row-level security or per-tenant vector namespaces can adopt the measurement without adopting anything else here.

python -m authorized_recall

Why

A retrieval system that surfaces a dataset the caller is not permitted to use has not helped them. They cannot act on it; the only thing that changed is that they now know it exists. Standard Recall@K scores that as a success, and it has also spent one of the K slots doing it.

In the corpus measured here, 68.8% of what retrieval returns is unusable to the principal who asked, and Recall@K cannot see it.

Definition

Let

• $D$ — the corpus,
• $q$ — a query, with relevant set $R(q) \subseteq D$,
• $p$ — a principal, with authorization predicate $A_p : D \to {0,1}$,
• $L_K(q,p)$ — the ordered list of at most $K$ items the system returns.

The authorized relevant set is

$$R_A(q,p) = {, d \in R(q) : A_p(d) = 1 ,}$$

and

$$\mathrm{ARecall}@K(q,p) = \frac{|R_A(q,p) \cap L_K(q,p)|}{|R_A(q,p)|}, \qquad \mathrm{ARecall}@K \triangleq 1 \ \text{ when } R_A = \emptyset .$$

Compare

$$\mathrm{Recall}@K(q) = \frac{|R(q) \cap L_K|}{|R(q)|} .$$

When $A_p \equiv 1$, $\mathrm{ARecall}@K = \mathrm{Recall}@K$. The metric is a strict generalisation, not a different measurement.

Two conventions, stated because they change the mean

1. $R_A = \emptyset \Rightarrow \mathrm{ARecall}@K = 1$. The system cannot be faulted for failing to surface what it must not surface. Over a population containing such pairs this inflates the mean, so report the restricted mean — over pairs with $R_A \neq \emptyset$ — alongside it. The experiment here prints both.
2. Retrieved-but-unauthorized items are neither credited nor penalised inside ARecall. They are a separate quantity:

$$U@K(q,p) = \frac{|{, d \in L_K : A_p(d) = 0 ,}|}{K}$$

Where the filter sits

ARecall is computed over the list the system returns, so it is sensitive to whether truncation happens before or after the authorization filter:

$$L_K^{\text{post}} = \sigma_{A_p}\big(\mathrm{top}_K(\mathrm{rank}(D))\big) \qquad L_K^{\text{pre}} = \mathrm{top}K\big(\sigma{A_p}(\mathrm{rank}(D))\big)$$

Claim. $L_K^{\text{post}} \subseteq L_K^{\text{pre}}$ for every ranking, $K$ and $A_p$, hence

$$\Delta@K = \mathrm{ARecall}^{\text{pre}}@K - \mathrm{ARecall}^{\text{post}}@K ;\geq; 0 .$$

Proof. Filtering preserves relative order. An item in $L_K^{\text{post}}$ is authorized and appears within the first $K$ positions of the ranking, so at most $K-1$ items precede it, so at most $K-1$ authorized items precede it, so it appears within the first $K$ authorized items — which is $L_K^{\text{pre}}$. ∎

The gap is therefore non-negative by construction rather than by luck of the corpus. What the corpus determines is its size.

Measured

40 synthetic datasets over 8 domains, 24 queries, 4 authorization profiles, 96 query-principal pairs. Relevance by construction: a dataset is relevant to a query when it is in the query's domain. Retrieval is TF-IDF cosine.

  K   Recall  ARecall  ARecall    gap  unusable
                 post      pre         in top-K
  1    0.200    0.750    0.750 +0.000     68.8%
  3    0.483    0.835    0.863 +0.027     68.8%
  5    0.867    0.954    0.988 +0.033     68.5%
 10    1.000    1.000    1.000 +0.000     39.1%

Restricted to the 30 pairs with $R_A \neq \emptyset$:

  K  ARecall post  ARecall pre     gap
  1         0.200        0.200  +0.000
  3         0.473        0.560  +0.087
  5         0.853        0.960  +0.107
 10         1.000        1.000  +0.000

At K=5, moving the filter ahead of truncation takes ARecall@5 from 0.853 to 0.960 (+0.107). Recall@5 stays at 0.867 and cannot see the difference.

What this does and does not establish

The absolute values belong to this corpus: relevance is by construction, the retriever is TF-IDF, and MRR is 1.000, so the retrieval task is easy. A harder corpus or a better retriever moves all three columns.

The gap is what the metric was defined to isolate, and the claim above is that it is a property of filter placement rather than of retrieval quality. The proof makes its sign certain; the experiment gives its size in one setting.

Applying it to a real corpus with real authorization data is the obvious next measurement, and this package is separable precisely so that someone else can do it.

Plan

Milestones are ordered so that each one produces something that runs. A milestone that only produces more design does not belong in this file.

M1 through M6 are done. What each produced, and what it did not, is in docs/RESULTS.md.

M1 — The contract, and a graph that refuses ✅

• DatasetDescriptor and DatasetCapability as typed models
• Three synthetic datasets with descriptors: 8 capabilities, 9 prohibitions
• Verdict = Approved | Refused | Indeterminate, with typed reasons, ported from ok-governed-motion — same names, same serialised strings
• Approval token: only Approved mints one; execution requires one
• interpret -> discover -> resolve -> admit -> {execute | refuse | indeterminate}
• Evidence record per decision, written to a hash-chained append-only file

Done when three transcripts exist: one granted and executed, one refused, one indeterminate — and in the latter two, no execution occurred and the evidence says why. All four runtimes produce all three.

The test that matters is not that a refusal message was produced. It is that after a refusal there was no capability to execute with, which is asserted as grant is None and tool_calls == mcp_calls == a2a_calls == [].

M2 — Conformance suite ✅

• Deterministic contract tests: policy verdicts, no LLM, no tolerance
• Routing: every admission arm, in every runtime, parametrised
• Negative paths — indeterminate does not fall through to execution; evaluator timeout yields indeterminate, not refusal; missing descriptor, schema-version mismatch and expired token each prevent execution
• Capability metadata tests: dataset, effect, classification, policy id
• Adversarial: the model cannot reach a raw tool that bypasses the wrapper

Done when the suite fails if any invariant is removed. It did fail, six times, on this implementation — docs/FINDINGS.md F-004 … F-009. Two of those were visible only at the MCP boundary and two only under the async runtimes.

M3 — MCP boundary ✅

• Every dataset behind an MCP server: descriptor, schema, lineage and policy as resources; capabilities as tools
• The control plane consumes it through a real client session
• A second dataset registered without touching the graph — tests/test_mcp_boundary.py, and the entire conformance matrix is run a second time with every dataset behind the boundary

The far side verifies the grant for itself. A boundary whose far side trusts its callers is not a boundary.

M4 — Semantic cache, authorization-scoped ✅

• Key over intent and dataset revision, capability, authorization scope, principal class, schema version, freshness, policy version
• Isolation tests, including one per key dimension: change it, get a different key, or the dimension is not protecting anything
• Revoked access does not hit
• Hit rate not reported. SemanticCache.stats computes it, and it is not published anywhere, because on a fixture workload it would be a property of the fixture. It is a property of the traffic, and there is no traffic here.

This is the security-critical milestone. A semantic cache whose lookup is not authorization-aware is a policy bypass with good latency. The cache here is lexical, not semantic, and docs/FINDINGS.md says so: under-hitting costs latency, over-hitting crosses a principal boundary.

M5 — Evaluation ✅

• Labelled datasets: admission, discovery, adversarial
• Separate evaluators, not one judge — dataset selection, capability selection, policy decision, refusal reason, prohibited execution, trajectory validity, provenance completeness
• Deterministic evaluators wherever the property is mechanical
• Repetitions — and the honest reading of them: every spread is zero because the interpreter is deterministic, so repetition currently measures nothing
• Groundedness not measured. It needs a model-generated answer. This build synthesises no prose, so the metric would score its own formatter.
• Not wired to LangSmith. The evaluators are plain functions over RunResult. Tracing is an integration, not a result, and adding it would make the suite depend on a hosted service to produce a number.

Gate shape: 100% required for control-plane invariants. Thresholds for probabilistic quality. Both hold — see docs/RESULTS.md §3.

M6 — Authorized Recall@K ✅

• Defined precisely enough to be disagreed with — including the two edge cases decided rather than left to fall out of the arithmetic
• Measured against plain Recall@K on the same corpus
• Gap reported

At K=5, moving the authorization filter ahead of truncation takes Authorized Recall@5 from 0.853 to 0.960 (+0.107). Plain Recall@5 stays at 0.867 and cannot see the difference. 68.8% of what retrieval surfaces is unusable to the principal who asked.

The corpus is synthetic and the relevance judgements are by construction, so the absolute numbers belong to that construction. The gap is a property of where the filter sits, which is what the metric was defined to isolate. docs/RESULTS.md §2 carries the caveats.

Open questions, and the answers taken

1. Reuse or reimplement the verdict types? Reimplemented, with tests/test_verdict_parity.py asserting the serialised strings and rationales, and reading ok-governed-motion's policy.rs directly when it is checked out beside this repository. The Rust Seal — which makes Approved unnameable outside its module — has no exact Python equivalent; the module-private sentinel used instead makes forging an approval deliberate rather than accidental, and verdict.py says so.
2. Which policy runtime? A small internal evaluator. Everything outside PolicyEngine sees a Verdict; replacing the body of evaluate with a Rego call changes no other module and no assertion. The architecture does not depend on the answer, which was the actual requirement.
3. Is the evidence ledger in scope, or a dependency? In scope, minimally. A hash-chained JSONL file, with verify_chain stating exactly what that buys: truncation and in-place edits become detectable, and nothing more. A real deployment substitutes an event store; verify_chain is what such a store would have to keep true.
4. Public or private? Open. The prefix policy makes dk- private and ok- public, so publishing is a rename rather than a copy.
5. Does this become a paper? M6 produced a number, which was the stated condition. The number is from one synthetic corpus and one implementation — enough to justify the measurement, not yet enough to be the paper.

Not in scope

• A general agent framework
• A production deployment
• Anything requiring the current employer's systems, data or people

M7 — a portable conformance suite ✅

Before: 2 of 15 assertions checked through a public interface. After: 15.

• The conformance interface declared — four methods, one Observation type
• World, vectors and expectations as JSON; Python is one runner
• The conformance package imports nothing from any implementation, and a test asserts it
• An independent implementation exists — packages/agentic-dataset-conformance/src/agentic_dataset_conformance/toy.py, 250 lines, no framework, no MCP, no shared code — and passes 15/15
• 13 broken variants, each caught by the assertion named for it
• docs/PORTABILITY.md records what changed shape and what cannot be reached from outside

The first finding came from the toy itself. Its initial version derived capabilities() from the descriptors, making every advertised capability executable by construction. AD-002 caught it on the first run — which is the clearest evidence available that the suite is not vacuous.

The second came from the vectors. AD-008 failed at the MCP boundary only: after a revision change the server kept serving the revision it was constructed with. The white-box suite had never noticed, because it only asserted the miss and not the subsequent hit. docs/FINDINGS.md F-010.

What would strengthen the result

In rough order of how much each would add:

1. A second implementation by someone else. The toy is independent of the reference implementation but not of its author, and one person's reading of their own specification is the weakest kind of independence. M7 made the ask reasonable — four methods and a JSON suite — but did not answer it.
2. A model in the loop, so the statistical rows carry a spread and the invariant rows can be watched not moving.
3. A harder discovery corpus, where MRR is not 1.000.
4. An embedding-keyed cache, behind the same authorization dimensions.

License map

This repository is not under a single license, and it is not accurate to call it open source as a whole. Five tiers, chosen so that everything a third party needs in order to implement and test the contract independently is openly licensed, while the reference implementation of the contract is not.

The accurate one-sentence summary, and the one to use publicly:

The Agentic Dataset specification, normative vectors, conformance tooling and Authorized Recall implementation are openly licensed for independent implementation and reuse. The reference implementation is source-available under the Business Source License 1.1.

Excluded — the mark in brand/

The Agentic Dataset mark is not covered by any licence in this file. All rights reserved. It is deliberately the one asset here that is not open, for the reason an identifier exists: a mark anyone may adapt identifies nothing, and a specification that may one day want "conforms to AD-001 … AD-015" to carry meaning cannot hand its mark away with its text.

Unmodified use to refer to this project is welcome and needs no permission. brand/README.md states what that covers. Implementing the contract needs no permission either, and never touches this exclusion.

Tier 1 — Specification and normative prose · CC BY 4.0

CONFORMANCE.md
packages/agentic-dataset-conformance/src/agentic_dataset_conformance/verbs.md
docs/PORTABILITY.md
docs/RESULTS.md
docs/FINDINGS.md
docs/ARCHITECTURE.md, docs/ARCHITECTURE-LLAMAINDEX.md, docs/ARCHITECTURE-ADK.md
README.md, PLAN.md, RELEASE.md, CONTRIBUTING.md

https://creativecommons.org/licenses/by/4.0/

Quote it, reproduce it, extend it, translate it, build a competing specification on it. Attribution required. Commercial use is permitted — this is CC BY, not CC BY-NC, because a specification nobody may use commercially is not an interoperability specification.

Tier 2 — Normative worlds and vectors · CC0-1.0

packages/agentic-dataset-conformance/src/agentic_dataset_conformance/data/worlds/*.json
packages/agentic-dataset-conformance/src/agentic_dataset_conformance/data/vectors/*.json

https://creativecommons.org/publicdomain/zero/1.0/

Public domain dedication, including no attribution requirement. This is deliberate and it is the point of the tier: these files are meant to be vendored unchanged into a Rust crate, a Go module, a TypeScript package or a commercial product's test suite. Every condition attached to them is friction against the one outcome this repository most wants.

Tiers 2 and 3 are a published distribution

agentic-dataset-conformance on PyPI carries both: the software under Apache-2.0 and the normative data under CC0-1.0, declared as the SPDX expression Apache-2.0 AND CC0-1.0 rather than rounded to whichever is more convenient. agentic_dataset_conformance/data/LICENSE states the CC0 dedication inside the distribution, so it survives being unpacked somewhere else.

authorized-recall is a second published distribution, Apache-2.0.

The reference implementation is not published to PyPI. pip install reads as open source to most people, and shipping BUSL code that way would be misleading whatever the metadata said.

Tier 3 — Conformance software · Apache-2.0

packages/agentic-dataset-conformance/src/**   (interface, runner, CLI, toy, mutants)
conformance/generate.py
conformance/subjects.py
conformance/__init__.py
packages/agentic-dataset-conformance/src/agentic_dataset_conformance/toy.py
packages/agentic-dataset-conformance/src/agentic_dataset_conformance/mutations.py
tests/test_conformance_vectors.py

LICENSES/Apache-2.0.txt

A third party must be able to build and test an independent implementation, commercially or not. generate.py is here rather than in Tier 2 because it is software: CC0 can cover code, but Apache-2.0 gives downstream users clearer patent treatment.

Tier 4 — Authorized Recall · Apache-2.0

packages/authorized-recall/**

LICENSES/Apache-2.0.txt

This package imports nothing else in the repository. It is the piece most likely to be used by people who never adopt the architecture — authorization- aware search, multi-tenant retrieval, ABAC/RBAC retrieval evaluation — and there is no strategic gain in making that difficult.

Tier 5 — Reference implementation · BUSL-1.1

src/agentic_dataset/**   EXCEPT conformance/ and authorized_recall/
evals/**
examples/**

LICENSES/BUSL-1.1.txt, with parameters filled in:

• Licensor — Alexander Chernov
• Additional Use Grant — production use permitted where not primarily intended for or directed toward commercial advantage or monetary compensation. Research, teaching, evaluation, peer review and personal projects are covered.
• Change Date — 2029-09-02
• Change License — Apache-2.0

BSL is not an open-source license, and the license itself says so. What it guarantees is that this tier becomes one: on the Change Date, or the fourth anniversary of first public distribution, whichever comes first, it converts to Apache-2.0 automatically.

Two notes on the choice.

Why the restriction exists. The reference implementation contains a working expression of authorization-scoped semantic caching, which overlaps a live commercial interest. What Tier 5 restricts is this code. The contract itself is not restricted: the specification is Tier 1, the vectors are Tier 2, and implementing them independently — commercially or otherwise — requires nothing from this tier.

Why it is time-limited. Indefinite source-available licensing would leave the project permanently unable to describe itself simply. The Change Date fixes that on a schedule rather than on a decision.

Commercial licensing

To use Tier 5 commercially before the Change Date:

Alexander Chernov — GitHub @doytsujin · LinkedIn @thedoytsujin

Tiers 1–4 need no such agreement, commercially or otherwise.

Independent reimplementation is explicitly welcome

CONFORMANCE.md (CC BY 4.0) states the fifteen assertions in prose; the packaged data/vectors/ (CC0) makes them executable. Writing an independent implementation against them, in any language, for any purpose including a commercial one, and publishing whether it passes, requires no permission from anybody. It is the outcome this license map is arranged around.

The assertion identifiers AD-001 … AD-015 may be referred to freely.

© 2026 Alexander Chernov. Tier 5 rights reserved; Tiers 1–4 licensed as above.

The Agentic Dataset mark

A bounded object with one controlled opening. The outer square is the boundary, the break in its top edge is the single admitted path, and the inner square is the dataset. It is the whole model in two shapes: a boundary, and the one way through it.

The mark belongs to the Agentic Dataset programme, not to this repository alone. Implementations, papers and packages in the programme use the same mark; they do not each get their own.

Files

Using it

Two shapes, no text, no gradient, monochrome. Keep it that way.

• Give it clear space of at least the width of the inner square on every side.
• Do not recolour it into a gradient, add an outline, rotate it, or place it on a busy background. It inherits text colour; that is the intended mechanism.
• Below about 16 px the opening closes and the mark stops meaning anything. Use a word instead.
• The opening is the point. Do not close it, and do not move it to another edge — a gap on the right edge reads as the letter C, which is why it is on top.
• Where transparency is not an option — a GitHub avatar composites onto the page and a black-on-transparent mark disappears in dark themes — use the white-ground raster above. A flat ground is not a recolour: the mark itself is unchanged, and it carries the clear space of this section baked in.

Licensing — read this, it is not the same as the rest of the repository

The mark is not covered by the CC BY 4.0 licence that covers this repository's documentation, nor by any other licence in ../LICENSE.md. All rights reserved.

That is deliberate, and it is the opposite of everything else here. The specification is CC BY, the vectors are CC0, the conformance software is Apache-2.0 — all chosen so anyone can implement the contract, commercially or not, without asking. An identifier has to work the other way round. A mark anyone may modify identifies nothing, and this project specifically may want "conforms to AD-001 … AD-015" to mean something one day. A freely adaptable logo would foreclose that.

You may, without asking:

• use the unmodified mark to refer to the Agentic Dataset programme, this repository, or its packages — in articles, slides, documentation and talks;
• state that your implementation conforms, or does not conform, to the specification, in words.

Please do not:

• modify the mark, or use it as the identity of your own product, package or fork;
• use it in a way implying endorsement of, or affiliation with, an implementation that is not part of this programme.

Implementing the contract requires no permission and no licence from anybody. This restriction is about the mark, and only the mark.

Questions: @doytsujin.

Citation

The software carries a CITATION.cff, which GitHub renders as Cite this repository.

v0.1.0 does not exist yet, so do not cite it. During the release candidate, cite the tag and the commit:

Chernov, A. (2026). Agentic Dataset Reference Implementation and
Conformance Suite (v0.1.0-rc.2) [Computer software].
https://github.com/agentic-datasets/reference

A commit SHA is better still, because a release candidate is expected to move: the point of the window is that findings change the artifact.

ORCID: 0009-0007-3198-2712

There is no DOI yet. One will be minted from the v0.1.0 tag once the window closes.

Citing an assertion

The identifiers AD-001 … AD-015 are stable and may be referred to freely. Cite the assertion, not a line number:

…refuses on an unregistered capability (AD-006) and records the refusal (AD-010).

Citing the metric

Authorized Recall@K is defined in its own package, which has no dependency on this architecture. If you use the metric without adopting the control plane, cite the package rather than the reference implementation.

Papers

Three conference papers argue the model. All were accepted for 2026 and none is in published proceedings yet, so there are no DOIs to cite.

Nothing in this repository depends on them: the assertions, the vectors and the measurements are reproducible from a clone.