# Maintainer model evaluations — 2026-10-19

| Model / configuration | Correct / 900 | Core macro accuracy | Coverage | Resource track |
| --- | ---: | ---: | ---: | --- |
| CAEL 0.2.3 native whole-system MC | 207 / 900 | 23.00% | 100% | Offline retrieval |
| MICA Flame-W 0.3.6 | 238 / 900 | 26.44% | 100% | Closed-book |

Uniform random guessing has an expected 25% score; the measured seeded-random
control is 22.00%, training-majority 25.00%, and BM25 26.67%. No superiority or
statistical-significance claim is made. These are the first frozen full test
runs on this suite, with no runtime errors. The model versions were not tuned to
the public test after seeing these results. Low performance here identifies
limits of these native multiple-choice interfaces on the generated tasks, not
an assessment of every model feature.

| Task family | CAEL % | MICA Flame-W % |
| --- | ---: | ---: |
| Arithmetic | 29.33 | 27.33 |
| Logic | 0.00 | 42.00 |
| Ordering | 28.00 | 8.00 |
| State tracking | 24.67 | 34.67 |
| Strings | 32.00 | 3.33 |
| Tables | 24.00 | 43.33 |

Trace audit: all 900 CAEL questions used its language+evidence route; its existing
word-problem arithmetic matcher did not take over on these prompts. All 900
Flame-W questions used its native word-model fallback; its released belief-memory
adapter did not recognize these generated identifiers/tasks. No branch was added
to force these specialized helpers to engage. CAEL's logic result here reflects
its existing multiple-choice path, not a separate evaluation of its symbolic
Reasoner API. Different native interfaces would require separate labeled runs.

These are runs of existing model releases, performed by the benchmark maintainer
with a coding assistant. They are **not community submissions or independent
third-party reproductions**. The benchmark was also authored with assistant help.
Both the inference adapter and the test templates were known to the evaluator.
Existing weights, native inference source, settings and test hashes were frozen
before the run. No training, prompt search, hyperparameter selection, or new
task-specific solvers were added for these results. Gold answers are opened by
the harness only after all model predictions have been saved.

## CAEL

The entry is CAEL 0.2.3's configured native multiple-choice path: conditional
count-language checkpoint 0.3.2, exact word-problem arithmetic, and BM25 evidence
from its local Wikipedia and ARC Corpus indices. These are existing components,
using the selected evidence weights (Wikipedia 2, ARC 6, option length limit 25).
The adapter invokes the same CountLanguage.score_endings and choice.choose
functions used by the assistant's multiple-choice route. The prompt and four
choices are passed unchanged. Category, difficulty, gold answers, generation
programs and scheduling reference solvers are not passed to inference.

This is an **offline-retrieval whole-system** result, not a count-model-only score.
It does not evaluate every CAEL feature or invoke a separate new puzzle solver.
The corpus hashes, model hashes, settings and exact five native inference modules
are preserved in the evidence. The exported CAEL source retains its original
MICA AI non-commercial license; this benchmark does not relicense it. Existing
model weights/corpora are not redistributed in the benchmark repository.

The retained training records identify human-written GSM8K/MBPP and earlier
language sources and state no neural teacher was used. Full upstream provenance
has not been independently audited; the row is conservatively marked unknown
training provenance. This does not mean neural inference was used.

## MICA Flame-W

The entry uses the existing 0.3.6 Prototype.answer API: integer cellular automaton,
short/fitted/long memory banks, answer-mode calibration, previous-word and
phrase-copy banks, plus its released cellular-belief-memory adapter. If the latter
cannot answer, the release ranks choices by mean log probability per word token.
No retrieval corpus or external helper is added.

The generic input binding uses the last prompt line as question and earlier lines
as context; one-line prompts become the question with empty context. It does not
branch on benchmark family or inspect labels. The local package matched every
file in the published manifest at HF revision
ade0152787446966969048bbc7bdc789059638e2. This is a **closed-book** model run.
The package license remains personal/research, non-commercial. Full training-data
provenance was not independently audited and is conservatively marked unknown.

## Runtime and evidence

Inference runs on CPU with numerical-library thread limits set to one. Python
socket connections and DNS lookups are denied during inference. This is an
in-process network guard for reviewed local code, not an OS security sandbox.
Runtime includes model initialization and all 900 question predictions, and is
recorded with Python/library/platform information. Peak memory is not measured.
Local filesystem reads are permitted for the declared models and corpora.

Evidence is under model_runs/2026-10-19 in the dataset repository: frozen hashes,
settings, adapters, per-question predictions, traces, category reports and timing.
The primary score is the existing v0.1 six-family macro accuracy. No score is
imported from an older ARC, bAbI, Tiny ToM or other evaluation.

## Reproduction

Obtain the frozen model/corpus files identified in freeze.json. CAEL's local
indices are required to reproduce the whole-system result; they are not included
in the benchmark download. A source URL alone does not supply those resources.
Use a trusted environment matching the recorded Python and package versions;
joblib checkpoint files should only be loaded from a trusted source.

Download evaluate_models.py, reproduce_models.py and model_runs/2026-10-19 along
with the benchmark. From the dataset clone, replay with explicit locations:

```bash
python reproduce_models.py cael --project-root PATH_TO_CAEL --data-root . --evidence-root model_runs/2026-10-19 --output-root NEW_EMPTY_CAEL_REPLAY
python reproduce_models.py mica --project-root PATH_TO_CAEL --data-root . --evidence-root model_runs/2026-10-19 --output-root NEW_EMPTY_MICA_REPLAY
```

The replay verifies required asset and native-source hashes and performs no
fitting. Exact CAEL native source snapshots are included under cael_source with
their original license, for inspecting/restoring the evaluated source in a
separate checkout. All other mutable assistant/UI files are outside this native
multiple-choice path. Running these commands is a local evaluation only; they
do not publish or promote leaderboard entries.

## Scheduling

Neither evaluated native model API currently produces the structured shift
assignments required by scheduling v0.2. Scheduling is **not evaluated** for these
versions; no zero score or reference-solver result is attributed to them. A future
native scheduling-capable version should be evaluated as a separate new model
revision with the adaptation disclosed.