About Code with Papers
Code with Papers is Syntology's community app: a free site, with no account, built on the archived Papers with Code data (last public snapshot 2025-07-28): its papers, benchmarks, datasets and methods, plus one thing the archive never held: for the papers whose code Syntology harvested, each code sample is marked with whether it ran, and what happened when it did. Syntology builds a graph of AI research papers, their code and their citations; this app puts one part of it, the harvested code and whether it ran, next to the archive for anyone to browse, and links into the rest.
Papers with Code told you where the code is. We ran it. In this build, Syntology harvested 294,260 distinct code samples from the repositories of 51,887 papers and ran 127,244 of them, each one function or class executed on a synthesized input; the other 167,016 are unverified (harvested, not run). A sample that ran is not a sample shown to be correct, and a paper with a sample that ran is not a reproduced paper. The per-sample counts are on Code, where the unverified majority is shown beside what ran.
paperswithcode.com now redirects to Hugging Face Papers, which lists new papers and links to their code. This site is different in two ways: it serves the archive's leaderboards, datasets and methods as the archive left them (the changes Syntology made are listed on Attribution), and it records, sample by sample, whether the linked code ran.
Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.
Where the data comes from
- Papers, code links, benchmarks, datasets, methods: the archived Papers with Code dataset, as published by the pwc-archive community mirror on Hugging Face (five tables, byte-identical to the mirror; licence CC BY-SA 4.0; snapshot 2025-07-28). Full attribution and what we changed: Attribution & licence.
- Title, abstract, authors and date of papers newer than the snapshot: arXiv's own metadata, CC0 1.0 (arXiv licence page), where held.
- Everything marked Syntology: Syntology's own graph. Repository links found in a paper's own text, code harvested from those repositories, and per-sample execution records. Read from the graph 2026-09-24.
How current this is. Archive snapshot 2025-07-28; Syntology's graph read 2026-09-24; this site built 2026-09-24. Nothing on the site changes between builds. The build record gives the exact times and inputs.
Which papers have a page. An archive paper has one when the archive gives it a code link, a leaderboard row, or an introduced dataset or method, or when Syntology's graph links it to code; archive papers with only an abstract have none. A paper newer than the snapshot has a page only when Syntology's graph links it to a code repository, so a widely discussed recent paper is missing until that link exists. Every archive paper left out is counted on what is not shown. A newer paper that Syntology's graph does not link to code is not held by the site and is not counted.
What "ran" means, exactly, and every label the pages print
Each harvested sample is one function or class. Syntology executes it on a synthesized fixture and records what happened. Every sample carries exactly one execution status; a sample that ran with no contract checked may also carry a metamorphic tier. Licence and repository labels are separate axes: they say where a copy came from and under what licence, never whether it ran.
Execution status (one per sample)
| Label | What it licenses you to conclude |
|---|---|
| ran · honoured contract | The sample executed, and its behaviour on the fixture matched the contract Syntology drafted from the code itself. Not a claim about the paper's results. |
| ran · violated contract | The sample executed, and its behaviour contradicted that drafted contract. Often a real defect; sometimes a wrong contract. |
| ran · our draft was wrong | The sample executed, and the contract Syntology drafted for it did not describe it correctly (its parameters or its output). A failure of Syntology's instrument, not a statement about the code. |
| ran · fixture could not drive it | Syntology's synthesized input could not exercise the sample: the sample raised an error on it, needed a dependency that was missing, or timed out. A failure of Syntology's instrument, not a statement about the code. |
| ran | The sample executed on the fixture, with no contract checked. Nothing about correctness. |
| unverified | Harvested, not executed. The majority of samples. |
“Ran”, wherever a count uses it, means every status above except unverified.
Metamorphic tier (on a sample labelled “ran”)
Syntology's metamorphic check repeats or perturbs the input and records the highest tier the sample's outputs passed. Evidence of the output's shape, not that the sample implements the paper's method. A sample with no recorded tier shows plain “ran”.
| Label | What the outputs passed |
|---|---|
| ran · metamorphic tier: deterministic | The same input gives the same output. |
| ran · metamorphic tier: well formed | Outputs are finite, or non-negative where expected. |
| ran · metamorphic tier: invariant | A relation such as scaling or permutation survived a perturbation of the input. |
Behaviour fingerprint
| Label | What it means |
|---|---|
| fingerprinted | A behaviour fingerprint exists: outputs on a shared input battery, comparable with other implementations that accepted the same battery. Nothing is asserted about other inputs. The function-name census compares these fingerprints across papers. |
Repository labels
On a sample, where its repository stands relative to the paper. None of these asserts that the code is the authors' implementation of the paper's method.
| Label on a sample | What it means |
|---|---|
| official repository | The archive marks this repository official for the paper. |
| named in the paper | The archive records that the paper mentions this repository; it is not marked official. |
| community (archive-listed) | In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper. |
| found in paper text by Syntology | Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted. |
| community | Not in the archive's code links for this paper; a community repository Syntology harvested. |
On a paper's repository list:
| Label on a repository | What it means |
|---|---|
| official | The Papers with Code community marked this repository official for the paper (archive flag). |
| mentioned in paper | The archive records that the paper mentions this repository. |
| mentioned on GitHub | The archive records that the repository mentions the paper. |
| framework | The framework the archive records for the repository, such as pytorch or tf. It is also printed beside the first repository on a home-feed card. |
| found in paper text by Syntology | Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted. |
| not reachable when probed date | Syntology's probe of the repository returned an HTTP error, such as not found; hover the label for the code. Repositories for recent papers often appear after camera-ready. |
| empty when probed dates | The repository answered and held no files when probed. |
| licence | The licence the repository's licence file named when probed. |
Licence labels, and what “pointer only” means
No page on this site shows code text. Every sample is linked: its row points at the file in its repository. “Pointer only” means the sample is linked, not reproduced: Syntology does not serve that copy's text, because of its licence record. It is a licence property of one copy of the code, recorded for samples that ran and samples that did not alike. It says nothing about whether the code runs or is correct.
| Label as printed | When | What it means |
|---|---|---|
| <licence> (permissive) | The licence is on the permissive list and this copy's record is marked cleared. | Repository licence at harvest: <licence>. Syntology may serve this code's text; this site links to the file. |
| <licence> recorded; this copy not marked cleared · pointer only | The licence is on the permissive list, but this copy's record is not marked cleared. | The repository's licence is recorded as <licence>, but this copy's record is not marked cleared for Syntology to serve its text, and the record does not say why. Syntology does not serve its text; this record links to the file. |
| <licence> (copyleft) · pointer only | A copyleft licence (GPL, AGPL, LGPL, MPL, EPL, EUPL, CC BY-SA, OSL, CDDL). | <licence> is a copyleft licence: it permits redistribution under its conditions. Syntology serves code text only under MIT, Apache-2.0, BSD and similar permissive licences, so this record links to the file instead. |
| <licence> · pointer only | Any other licence recorded at harvest, including licences that permit redistribution but are not on the permissive list, such as WTFPL. | Repository licence at harvest: <licence>. Syntology serves code text only under MIT, Apache-2.0, BSD and similar permissive licences, so this record links to the file instead. |
| no licence file found · pointer only | No licence file was found at harvest. | No licence file was found in the repository at harvest time. Without a licence, Syntology does not serve the code's text; this record links to the file. |
| licence not identified · pointer only | A licence file whose licence was not identified. | The repository has a licence file whose licence was not identified (NOASSERTION). Syntology serves code text only under MIT, Apache-2.0, BSD and similar permissive licences, so this record links to the file instead. |
| licence of this copy not recorded | Identical code first harvested from another repository; this paper's copy was not located. | Identical code Syntology first harvested from another repository; this paper's own copy was not located, so its licence is not recorded. |
<licence> stands for the licence name recorded at harvest, such as MIT or GPL-3.0. The permissive list is MIT, Apache-2.0, BSD and similar licences.
Where a record comes from
| Label | What it means |
|---|---|
| archive 2025-07-28 | From the archived Papers with Code data, snapshot 2025-07-28, as the archive left it. |
| links table only | The paper is in the archive's code-links table, but its catalogue row (abstract, authors, date) was never published. |
| Syntology | Syntology's own record or measurement, not part of the archive. |
| added by Syntology | A paper not in the archive (frozen 2025-07-28), added from Syntology's graph. |
| In Syntology | Links from the page into Syntology's own services, such as the paper's place on the Atlas (below). |
Function-name census labels
| Label | What it means |
|---|---|
| split | Implementations of one name fall into more than one cluster of identical output on the same shared input: more than one computation goes by that name. |
| one output per bucket | Every compared implementation of the name returned the same output digest on each shared input. Agreement on those inputs, not equivalence. |
| class | A class-bearing member: the class was constructed with seeded weights and then called. |
| non-finite | The recorded output holds a non-finite value (NaN or infinity). |
| same digest as torch: name | The harness also ran torch's own computation under a named convention, and its output digest is the same as this cluster's on this input. |
Counts on this site are stated per paper and per sample. There is no site-wide score, and the unverified majority is always visible: what is not shown lists every exclusion with its number.
What this site deliberately does not do
- No GitHub stars, no "trending". The original ranked its feed by stars per hour. That needs constant polling and measures attention, not whether code works. The home feed here is ordered by recency and by whether code ran.
- No new rows inside the archive's tables. Benchmarks are served exactly as the archive left them on 2025-07-28, in the archive's own row order, labelled as such. This build shows no results from papers newer than the snapshot, so every leaderboard stops at 2025-07-28; what is not shown states why.
- No accounts, no comments, no tracking, and no direct edits. Readers can report errors and propose changes (below), and a person checks each one against the source before anything changes. See the data policy.
- No code text. Samples are linked to their files, never reproduced here (above).
- No claim that code works. "Ran" is not "reproduces the paper". Every label above says what it says and nothing more.
Go deeper in Syntology
Code with Papers shows the archive and Syntology's per-sample run records. Syntology's own services hold more, and reach further than this app does.
The graph
Syntology's graph of AI and machine-learning research: papers, authors, citations, methods and code, with every relation tagged with where it came from. Ask it a question in plain English and the sources come back under the answer.
The Atlas
Papers and the citations between them, drawn as one navigable map. https://app.syntology.ai/?focus=<arXiv id> opens the map on one paper, for a paper the Atlas holds; for any other id the map opens with nothing selected.
API and MCP for agents
get_harvested_code_for_paper(arxiv_id) lists every function and class harvested from a paper, with whether each one ran. get_code(code_sha256) fetches one stored sample by its full SHA-256. The MCP server is at https://syntology.ai/mcp; a trial token needs no account and no card.
syntology.ai has no web page per paper or per sample; a paper is reached on the Atlas, and its code through the API.
For agents
Every paper page has a JSON twin at the same path plus .json (also advertised in a Link: rel="alternate" header), a sitemap index at /sitemap.xml, and the search index, starting at /search/manifest.json. For the graph itself and the stored samples, use Syntology's API and MCP service (syntology.ai/developers).
Found something wrong?
If a code link, a leaderboard value, a title, or one of Syntology's own records is wrong or missing, use the small “report” link next to it, or “Report a problem or propose a change” at the foot of any paper, benchmark, task, dataset, method or code page (or send a report directly). A report is a claim, not a correction: nothing changes until a person has checked it against the paper or source. Accepted corrections are made at the next build and marked on the page (“corrected after a report on <date>”, with the archive's original value kept in the marker); removed items are counted on what is not shown; every decision, accepted or not, is listed on /corrections. What a report collects: data policy.
Contact, rights and removal
- Copyright, licence questions, or a repository or sample you want delisted: legal@syntology.ai. The formal copyright-complaint procedure is in section 11 of Syntology's terms.
- Personal data, such as an author name you want corrected or removed: privacy@syntology.ai (Syntology's privacy policy).
- An error on a page: send a report, or use the report link on the page; a person checks it against the source before anything changes.
- Anything else: media@syntology.ai.