Pilot proposal: a negative-results registry for agentic discovery

Summary

Portolan is a registry where AI agents deposit what did not work. A nine-month pilot produces the record standard at v1.0, an open-source reference implementation, a first corpus from a partner laboratory, and a benchmark. The benchmark answers the question the idea rests on: does an agent given another agent’s dead ends reach a confirmed result in fewer runs?

The problem

AI agents now run discovery loops at scale. Edison Scientific’s Kosmos reads about 1,500 papers and executes about 42,000 lines of analysis in a single run (Mitchener et al., arXiv:2511.02824, 2025). What those runs learn from failed paths is discarded. Dead ends have low private value to any single operator and high collective value to everyone else, which is exactly why no one publishes them. In autonomous laboratories the same gap is reported for reproducibility: “There are no standards for performing or enforcing such studies, and incentives are non-existent” (Canty et al., Nature Communications 16:3856, 2025, doi.org/10.1038/s41467-025-59231-1).

Agents also cite retracted work without knowing it. In a June 2025 test of 21 retracted papers, research assistants cited them without warning: Consensus 18, Ai2 ScholarQA 17, Perplexity 11, Elicit 5 (MIT Technology Review, 23 September 2025). Across the literature, only 5.4 percent of citations to retracted papers acknowledge the retraction (Hsiao and Schneider, Quantitative Science Studies, 2022, doi.org/10.1162/qss_a_00155).

The one measured test of agents sharing results across labs, AgentRxiv, showed a 13.7 percent relative gain from sharing (Schmidgall and Moor, arXiv:2503.18102, 2025). Nobody has yet built the shared, provenance-aware version.

What the pilot builds in nine months

1. The record standard at v1.0. The Negative-Results Record core profile, domain-agnostic, together with the materials and biology profiles, taken through community review. The core record covers identity and provenance, the question, the path, screened items with version and integrity status, outcome, reuse guidance, relations, and integrity. Profiles supply the identifiers, vocabularies, and evidence rules of a field.
2. An open-source reference implementation. A Model Context Protocol tool for deposit and query, so a record is emitted as a byproduct of a run rather than typed into a form; a profile validator that rejects records missing the fields that make a negative interpretable; and a status service that resolves version and retraction status against Crossref and Retraction Watch at read time, carrying the asserting source and the timestamp of the check.
3. A first corpus. Several thousand records backfilled from a partner laboratory’s existing failed and inconclusive autonomous-synthesis and high-throughput computation attempts, federated so the laboratory keeps custody, and discoverable through the same query interface an agent uses at run time.
4. The Known Dead Ends benchmark. A held-out set of discovery tasks with known answers, measuring run-count and hit-rate deltas for agents with and without access to prior failed attempts. The benchmark is published with the corpus so the claim can be checked by anyone, including against Portolan’s own implementation.

Why materials and chemistry first, and biology next

Materials and chemistry. Novelty claims for predicted or synthesized compounds depend on exclusion checks against structured databases, so a record can link to Materials Project, OPTIMADE, and ICSD identifiers rather than to prose. That makes “this was already known” a machine-checkable statement, and it makes the first corpus tractable: the attempts already exist, in instrument logs and computation queues, unpublished.

Biology. A biological negative result is a claim that an effect was looked for and not found, and it is only informative if the assay could have detected the effect. The biology profile therefore requires control outcomes, a stated detection limit or power analysis, and persistent identifiers for reagents, strains, and cell lines. In biomedicine the literature itself is the experiment, so evidence-synthesis screening decisions are first-class records: which papers were excluded, on what grounds, and what their version and retraction status was at the moment of the check.

The deposit model

The Protein Data Bank made a whole field disclose intermediate artifacts by making deposit the price of publication, with a hold period to protect priority. Portolan applies the same model to agent runs: contribution as a condition of access, an embargo the depositor controls, and credit when a record is reused.

What a partner laboratory gets

A provenance-complete pipeline, in which every exclusion and every status check is recorded at the time it was made, pre-empting disputes over novelty claims before they reach a reviewer. A methods publication describing the pipeline and the corpus. A corpus contributed to open models under the laboratory’s name alongside Portolan’s. And the benchmark result: a measured answer to whether prior dead ends reduce the number of runs required, on the laboratory’s own problems. Throughout, the records, the keys, and the embargo decisions stay under the laboratory’s jurisdiction.

Governance

Records are private by default. The embargo is set by the depositor, and release is automatic at its end; extending it is an explicit act with a stated reason. Conditions that cannot be disclosed yet are committed by hash, so a dead end can be registered now and the exact conditions revealed later without allowing a rewrite. Every record is signed, content-addressed, and versioned: a correction produces a new version pointing at the one it supersedes.

Version and integrity status of everything a record cites is re-checked at read time, never only at write time, and every status carries the source that asserted it and the moment it was checked. A restricted tier exists for records that must never be public, regardless of embargo.

Portolan is publisher-independent and US-operated. Records, keys, and the restricted tier remain under the depositing laboratory’s jurisdiction and are never routed through a commercial content platform.

Open questions

These are the decisions the draft does not make. They are the same six carried on the core schema page, and comments on any of them are the most useful thing a reader can send.

1. Granularity. Is one record one attempt, one campaign, or one decision point? Too fine and the registry is a log; too coarse and the applicability conditions stop being true.
2. Identity and keys. What identifies “the same path” across depositors, so that two labs failing the same way can be recognized as such?
3. Default embargo. What is the right default hold period, and should it differ for computation and wet-lab records?
4. Redaction. How are withheld conditions handled beyond hash commitment, and what is the minimum a redacted record must still disclose to be admissible?
5. Minimum record. What is the smallest set of required fields that is still worth depositing, given that capture must stay automatic?
6. Where the profile lives. Which body should hold the profile and its vocabularies long term, and under what governance?

Contact

Email [email protected]. Comments on the draft schemas are welcome at the same address.

Back to portolansoft.com