# NNAI-Bench v0.1 protocol Version: nnai-bench-0.1. Release: 2026-10-18. Maintainer: vynly. Own-model evaluations published by the benchmark maintainer use the explicit status **maintainer-run**, with frozen inputs/settings, source and resource hashes, per-question predictions, and disclosures in MODEL_RUNS.md. They are neither community submissions nor independent third-party reproductions. Appending these rows does not change the frozen v0.1 data or original baseline evidence. Added 2026-10-19: [Scheduling v0.2](SCHEDULING.md) has separate data, manifest, structured prediction schema and leaderboard. Its primary metric is full feasibility, with per-constraint diagnostics and coverage-adjusted partial credit. Nothing is added to or averaged into the core v0.1 score. Shared non-neural, resource, provenance and manual-review requirements below apply to both tracks. ## Eligibility The entire inference path must be non-neural. Rules, classical symbolic systems, cellular automata, search/planning, decision trees, kernels, count/Markov models, statistical retrieval and combinations are eligible. Neural embeddings, rerankers, classifiers, language models, perception models and external neural services are ineligible, even if the final stage is non-neural. Static artifacts generated with neural assistance before evaluation must be disclosed in training provenance. Unknown provenance is visibly marked and needs maintainer review. Declare task-specific authored code. A perfect narrow solver is not evidence of general intelligence. Model/source URLs must be public, with an immutable source revision, license, environment, evaluation command and seed. Closed-source entries cannot receive reproduced status. Neural systems may be discussed but do not receive a non-neural leaderboard rank. ## Resources and test exposure - Closed-book: code and fitted parameters only, with external symbolic libraries declared. Test-specific hardcoded solutions must be declared as test-adapted. - Offline retrieval: local corpora or retrieval indices are permitted and must be identified by version/checksum, size and license. No neural retrieval components. - Both tracks run offline. No network access, remote tools, or undeclared memory across independent examples. Runtime includes all inference modules; disclose preprocessing and index-build costs separately from inference. - Use train for fitting and validation for choices. Freeze the system before testing. Declare exposure to test items, labels and generator details, including tuning across test runs. Template-informed solvers must disclose that design. - Test-adapted entries remain filterable and must not be described as clean held-out generalization. No comparable score is inferred from another suite. ## Data and scoring 600 train / 300 validation / 900 public test. Six categories with 150 test items each; three complexity tiers with 50 test items per tier per category. Four distinct options per question. All labels and generation programs are public. Main score is the arithmetic mean of six per-category accuracies on all items. ABSTAIN counts as wrong; also report coverage and answered-only accuracy. No rounding before averaging. Expected random-choice score is 25%. Reordering the 900 prediction rows has no effect. Missing, extra, duplicate IDs, invalid labels, or extra object fields cause rejection rather than silent truncation. The evaluator receives label-free questions. Gold answers and generation programs must not be passed to inference. Public labels cannot be made secret by hiding them in the browser; v0.1 makes no such claim. Independent examples share a small grammar. This protocol checks exact text overlap, not semantic independence. ## Submission and review 1. Sign in using Hugging Face OAuth. Profile and write-discussions scopes allow creating a public PR, without requesting private-repository access. 2. Fill model identity, source revision, architecture, resource track, provenance, test exposure, license, reproduction commands, hardware and runtime disclosures. 3. Upload exactly 900 predictions. Preview score in the browser; review and agree to publication. Download JSON without signing in if preferred. 4. Submit publishes the JSON as a PR in vynly/nnai-submissions under your identity. This is a durable, public queue, not an automatic ranked entry. Manual PRs with the same JSON schema are accepted. Do not put secrets or personal data in it. 5. Maintainer checks author, source, immutable revision, eligibility, test checksum, resource and contamination disclosures; recomputes scores from predictions. A submitted claimed score is never trusted. Inspect code and dependencies before running in a resource-limited, network-disabled environment. 6. Accepted metadata-only entries are marked self-reported. Reproduced status requires an actual independent run of the pinned code, with prediction evidence and a reproduction note. Gold labels are never inputs to the model process. 7. Maintainer merges an accepted submission and updates the Space leaderboard via review.py. No public submission can edit the leaderboard file automatically. The launch controls are reproduced runs of seeded random, training-label majority, and BM25 nearest training prompt. They are baselines, not competing model releases. BM25 returns the answer letter of its nearest training prompt, without option alignment; it is intentionally a weak retrieval control. No CAEL entry is included without a run of this exact suite. Baseline runtime is a local measurement, not a controlled hardware comparison. Do not rank entries by these timings. ## Limits and changes This is an English templated synthetic suite. It omits real-world language, knowledge, commonsense, perception and broad task coverage. Authored solvers may saturate it. Test results have no hidden-test guarantee. Separate benchmark versions must have new IDs and manifests; do not silently mix versions. Report bugs in the dataset Community tab. Score corrections require a visible review note and recomputation. All original benchmark code and data are MIT licensed.