Contributing

The most valuable contribution to this repository is not a pull request.

Wanted: an independent conformance implementation

You need no permission and no license from anybody to do this. The specification is CC BY 4.0, the normative vectors are CC0, and the conformance runner is Apache-2.0 — including for a commercial implementation. Only the reference implementation in src/agentic_dataset/ (excluding conformance/ and authorized_recall/) is restricted, and you do not need it.

The four runtime ports here share one reference ControlPlane. That is deliberate — an assertion that passed because each port re-implemented its own policy would be four experiments rather than one — but it caps what the result can claim. Today the matrix shows that one governance model is expressible in four runtimes. It does not show that independent implementations agree on the governance semantics, which is the stronger and more interesting claim.

Closing that gap needs someone else's implementation, written from CONFORMANCE.md, in any language, that does not use this reference control plane. Publishing whether it passes — and especially which assertions turned out to be ambiguous, under-specified or unimplementable — is more useful to this project than any amount of code review.

If you do this, please open an issue. A finding that AD-007 is ambiguous is a better outcome than a green tick.

How to do it

# Not on PyPI yet. From a clone, during the release candidate:
pip install ./packages/agentic-dataset-conformance
  1. Implement ConformanceSubject — four methods, in interface.py. The control verbs are in verbs.md beside it.
  2. agentic-dataset-conformance run --subject yourmodule:your_factory

You do not need this repository. The harness, the vectors and a worked example subject are all in that one package, which imports no implementation at all.

The world and the vectors are JSON under conformance/, so an implementation in another language needs a runner for that JSON rather than a reimplementation of this harness. packages/agentic-dataset-conformance/src/agentic_dataset_conformance/toy.py is a 250-line worked example that imports the interface and nothing else.

What is honest about the current state

The toy is independent of the reference implementation but not of its author. One person's reading of their own specification is the weakest kind of independence, and it is the reason this section exists. A second reading is the experiment.

docs/PORTABILITY.md records where the contract had to change shape to leave the building — AD-003 and AD-007 became universally quantified invariants, AD-008 became behavioural rather than structural — and the one thing conformance cannot establish at all: a subject that under-reports its own capabilities passes AD-002 while hiding a tool. Conformance here is a claim an implementation makes about itself, made checkable. It is not an adversarial audit.

Other contributions that would help

  • A harder discovery corpus. MRR is 1.000 on the current one, so the retrieval task is easy and the Authorized Recall numbers are measured in easy conditions.
  • Authorized Recall@K applied to a real corpus with real authorization data. the authorized-recall package is deliberately separable from everything else so this needs no adoption of the control plane.
  • A fifth runtime adapter. The bar is that it contains no policy decision. If you find yourself re-deciding something to make it fit, that is a finding about the model and worth an issue.
  • An assertion that cannot be expressed in some runtime. Per CONFORMANCE.md, that is a finding about the assertion, not the framework.

Running things

pip install -e ".[all]"

agentic-dataset-conformance run --subject conformance.subjects:subjects      # AD-001..AD-015, every runtime
python -m authorized_recall
python evals/evaluate.py
pytest -q

agentic-dataset-conformance run --subject conformance.subjects:subjects exits non-zero on any failure. CI also runs the core with no framework installed, which is where an import leaking out of adapters/ fails.

House rules for code

  • The core stays dependency-free. Framework imports live in adapters/ and mcp_boundary.py, nowhere else.
  • No policy decision in an adapter. Every runtime calls ControlPlane.admit.
  • Assert on absence, not on wording. assert "I cannot" in response tests nothing; assert result.grant is None and result.execution.tool_calls == [] tests the property.
  • If the implementation disagrees with a document in docs/, record it in docs/FINDINGS.md rather than editing the document to agree. The architecture documents predate the code and are evidence of what was designed before it was built.