Arithmetic
Integer expressions, products, and nested calculations.
V0.1 CORE + NEW V0.2 SCHEDULING / OPEN FOR SUBMISSIONS
Intelligence has
more than one
architecture.
A shared testing ground for AI without neural inference. Bring your rules, trees, automata, search, kernels, count models, or something new.
1,800Original synthetic questions
900Public test questions
25%Random-choice expectation
CPUNo GPU required by the suite
A focused first step. This synthetic suite tests bounded skills, with shared templates across splits. It does not measure general intelligence. Test labels are public, and task-specific solvers are welcome when disclosed. Read the protocol ↗
01 / THE RESULTS
Non-neural, on the record.
Macro accuracy across six equally weighted task families. Abstentions count as incorrect. CAEL and MICA Flame-W are existing models evaluated by the maintainer; the three reference programs are baseline controls. These entries are not community submissions. Read the model-run disclosures and evidence ↗
| System | Macro % ↓ | Math | Logic | Order | State | Strings | Tables | Review |
|---|---|---|---|---|---|---|---|---|
| Loading measured results… | ||||||||
Maintainer-run identifies an evaluation of the maintainer's own model and is not independent third-party reproduction. Resource tracks and test exposure remain visible; use the filters for matching resource setups. Baseline source, predictions and reports are public. View the review queue ↗
02 / THE BENCHMARK
Six ways to show your work.
Deductive logic
Rule chains, explicit negation, and open-world unknowns.
Ordering
Reconstruct an order from shuffled pairwise constraints.
State tracking
Follow location changes and identify the final state.
String operations
Reverse, select characters, and compose transformations.
Table reasoning
Filter, count, sum, and calculate weighted totals.
Download. Run offline. Submit predictions.
600 train · 300 validation · 900 test. Four choices per item. Three complexity tiers. The evaluator uses only Python’s standard library.
git clone https://huggingface.co/datasets/vynly/nnai-bench
cd nnai-bench
# Write your model's answers to predictions.json
python benchmark.py score predictions.json --questions data/test.jsonlInspect a test question
03 / NEW CONSTRAINT CHALLENGE
Build a schedule. Show every constraint.
Assign people to shifts while satisfying staffing, skills, availability, non-overlap, rest, workload, and incompatible-pair rules. The new track has 300 public synthetic cases: 120 train, 60 validation, and 120 test across three complexity tiers. Existing v0.1 questions and results are preserved.
More than pass or fail. Ranked first by fully valid schedules, then by coverage-adjusted partial score. Every broken constraint has an ID, expected value, and actual value. Empty schedules earn zero partial credit. All cases have a public feasible witness; this is a planted-feasible, bounded scheduling suite, not a general planning benchmark.
| System | Valid schedules % ↓ | Partial / 100 | Staffing coverage % | Review |
|---|---|---|---|---|
| Loading scheduling results… | ||||
Partial = mean satisfaction across seven constraint families × capped staffing coverage, averaged over cases. Raw family satisfaction is diagnostic. Full feasibility is the primary ranking; partial credit does not make an invalid schedule valid. Baseline implementations were written with coding-assistant help and use no neural inference.
Inspect constraint failures in a measured baseline
| Constraint family | Satisfied / applicable | Satisfaction % |
|---|
04 / JOIN THE COMPARISON
A place for your approach.
Non-neural means no neural networks anywhere in the inference path, including embeddings, rerankers, or remote helpers. Disclose neural-derived training data separately. Evaluate offline and pin your code and resource versions.
Sign in before filling out your submission.
Hugging Face sign-in requests your profile and permission to create discussions / pull requests. No access to private repos is requested.