PQID-Bench Documentation¶
Distribution status: PQID-Bench v1.0.0 frozen evidence release with
pqid-bench v1.2.1 software tooling.
The versioned benchmark archive is identified by
10.5281/zenodo.21649753.
PQID-Bench is an acquisition, reproduction, and replication toolkit plus a frozen evidence bundle for validation-aware quantum-program generation research. The package keeps eight activities separate:
- acquiring the authenticated compact benchmark distribution;
- checking release-file integrity;
- reproducing published metrics from archived evaluations;
- summarizing compatible evaluation records;
- comparing a candidate on an aligned prompt denominator;
- rendering an interactive report from frozen local evidence;
- collecting a new traceable model-response panel; and
- evaluating generated programs inside an isolated Docker worker.
Offline reproduction never contacts a provider. run-model does so only after
explicit prompt-export acknowledgement. Only replay executes generated
Python, and it requires a separate code-execution acknowledgement.
Choose A Starting Point¶
| Goal | Start here |
|---|---|
| Understand how GitHub, PyPI, Hugging Face, Zenodo, Docker, and OSF fit together | Ecosystem And Study Roadmap |
| Download the ready-to-run benchmark | Installation |
| Install and run the package for the first time | Installation |
| Reproduce the paper quickly | Reviewer Quickstart |
| Understand what each command can do | Capability Guide |
| Test a new external model | Live Model Testing |
| Look up a command or option | CLI Reference |
| Print a readable report or export to R/pandas | Reporting And Numerical Exports |
| Explore the frozen results interactively | Interactive Explorer |
| Use the importable Python interface | Python API |
| Run small end-to-end Python examples | Worked Examples |
| Prepare or inspect JSON/JSONL records | Data Dictionary |
| Understand ES-Gap, AS-Gap, and related metrics | Metrics And Invariants |
| Follow an end-to-end task | Workflows |
| Diagnose an error | Troubleshooting |
| Check a term or abbreviation | Glossary |
Documentation Sets¶
The User Manual is the operational source of truth for installation, commands, records, metrics, workflows, and error handling.
The following documents have narrower purposes:
- Python Package Contract specifies the software and scientific boundary of package version 1.2.1 against benchmark v1.0.0.
- Security And Privacy records the trust boundary and release privacy controls.
- Docker Replay Validation records the completed container replay audit.
- Reviewer Quickstart is the shortest route through integrity and parity checking.
- Package v1.2.1 Release Notes document the ecosystem-metadata clarification; package v1.2.0 release notes describe compact, authenticated benchmark acquisition; package v1.1.1 release notes document the provenance and repository-quality patch; package v1.1.0 release notes document the visualization and documentation update; benchmark v1.0.0 release notes remain the scientific freeze record.
- Package Paper Blueprint separates a future software paper from both this manual and the benchmark-results paper.
- The repository-level changelog separates software evolution from the frozen benchmark release, while the contribution guide defines the scientific freeze boundary for proposed changes.
Ecosystem Boundaries¶
The complete ecosystem contains several related but non-interchangeable objects:
| object | responsibility |
|---|---|
| archived PQID dataset | immutable upstream source dataset, DOI 10.5281/zenodo.20674853 |
| PQID-Bench v1.0.0 archive | immutable benchmark software and evidence release, DOI 10.5281/zenodo.21649753 |
| benchmark authoring workspace | canonical manuscript sources and complete research evidence |
this PQID-Bench tree |
standalone package source, manual, schemas, tests, and selected frozen evidence |
| compact core ZIP | ready-to-run records, splits, prompts, schemas, and isolated evaluator |
| frozen ZIP | complete downloadable scientific evidence bundle |
| wheel | installable command and Python interface, without the evidence bundle |
| source distribution | package source and documentation, without large scientific artifacts |
| container archive | optional isolated executable-replay worker |
| Hugging Face dataset | adoption-focused records, splits, prompts, schemas, and compact core download |
| OSF PQID-Bench 2 registration | immutable prospective protocol for a separate future semantic-validity study; contains no current benchmark result |
The upstream dataset and benchmark release require separate citations and version identifiers. Generated distribution files are rebuilt from source and must not be hand-edited.
For a reader-facing explanation of these roles, including what OSF preregistration means and why PQID-Bench 2 is not a newer current-data release, see the Ecosystem And Study Roadmap.
Rendered manuscript outputs remain outside this public release. Their scientific inputs and regeneration scripts remain available; see Regenerating Publication Outputs. The package-facing Plotly explorer is generated only during the Pages build and is not inserted into the frozen evidence archive.
Version Dimensions¶
| Dimension | Identifier |
|---|---|
| Python package | 1.2.1 |
| benchmark release | 1.0.0 |
| evaluator | pqid-bench-evaluator-1.1.0-safe-builtins |
| structural predicate | pqid-bench-reference-signature-1.0.0-count-map |
| schema | 1.0.0 |
| artifact manifest | 1.0.0 |
| container artifact | 1.0.0 |
These identifiers describe different contracts. A change to one must not be
reported as though every other dimension changed with it. The replay container
currently has local image ID
sha256:849bf53e449fd618633199c0b622abeca270591dff248cd0bf3a0fd461abf2e2.
It is published separately from the Python toolkit as
ghcr.io/elias-abebe-gasparini/pqid-bench-evaluator:1.0.0.
Its immutable OCI registry manifest digest is
sha256:39825f5635cd6273e9e23c2848f2c88a2ff9d461e16a263fd89f22c6e664ac8f;
it is distinct from the local image ID.
Documentation Site¶
The public manual is available at https://elias-abebe-gasparini.github.io/PQID-Bench/. Every source page is also readable as ordinary Markdown. Install the documentation extra to build the navigable site locally:
python -m pip install ".[docs]"
python scripts/build_pqid_bench_ecosystem_site.py
mkdocs serve
Use mkdocs build --strict when validating a documentation release.