Each category is an arbitrary rule; the model has to induce it, not memorise it

Items called by a noun — suitcases, drink bottles, mail envelopes — arrive with four independent properties and are routed to one of two named queues, Cedar or Birch. One property decides the route; the other three are logged but irrelevant. Which property decides, and which way, is sampled independently per category, so no property/queue pairing reads as obviously right or wrong and a model cannot guess from world knowledge.

The eval then asks about a noun and a property combination the documents never showed. Getting it right requires having stored the rule as a rule — property in, queue out — rather than as a lookup table of the cases actually seen.

One noun family, one deciding property, two held-out axes

A category names 100 independent rules over the same shape: a family of related nouns, four binary properties, and a rule that sends the item to Cedar when one property holds and to Birch otherwise (or vice versa). Two axes are held out from training entirely and are exactly what the eval asks about:

• held-out nouns. Each category trains on 2 or 3 nouns and reserves 2 or 3 more for eval only.
• held-out property combinations. Of the 16 possible assignments of four binary properties, 8 are shown in training documents and 8 are reserved for eval.

its rule bank

The API-written paraphrases of this category's rule that get rotated across its documents; see "How they were written" below.

Eight closed-book cases, then the rule — worded differently every time

The naturalised task from QUEUE_ROUTING_VALIDATION_PLAN.md: items arriving at a depot are routed to one of two named queues. The queue names are shared across categories and carry no commonsense valence, so no property/queue pairing reads as wrong.

`Cedar` and `Birch` rather than the plan's `Alder`/`Birch`: ` Alder` is two tokens in the Qwen3.5 tokenizer and ` Birch` is one, so a first-token forced choice would be comparing a word-start fragment against a whole word. Both of these are single tokens with a leading space, checked against the actual base tokenizer.

`matched` makes the causal-order layouts token-identical: the same four records are chosen whatever the order, the rule header appears in every layout, and only the position of the records changes. The two older scenarios sample records after the layout is known and put a header only on the case-first side, so their layouts share a structure but not their tokens.

Why the cases come before the rule

The default document layout in this series states the rule first and the worked cases after it, which means an outcome token can always be predicted by copying the rule sentence sitting right above it in the same context — training never requires retrieving anything. This corpus uses log8 instead, which puts every worked case above the rule:

The unmasked-loss ladder, 2026-09-08. Under uniform loss the closed-book supervision in a `two_sided` document is the pre-rule outcome: 2 tokens of ~166, and that arm scored S 0.022. `all_first` lifts the share to 8 of 166 with the same four records. This layout puts every trained assignment -- all eight, four of each label -- above the rule, so the log is the document and the rule reads as its summary. Closed-book outcome tokens are then 16 of ~270, the most any layout can carry without repeating a record. Needs a `matched` scenario, and a seed whose train_cases split 4/4.

The rule is API-written prose, one paraphrase per document

A fixed rule sentence would let a model that memorised the sentence's exact wording pass without ever inducing the rule. So instead of one hand-written template, an API call (claude-sonnet-4-5-20250929) wrote roughly thirty different statements of each category's rule up front, and the corpus generator rotates through that category's list one paraphrase per document. The list is checked twice before it is used: a second API call reads each paraphrase cold and confirms it still points the right way, and every paraphrase is scanned for the eval's own outcome wording so a rule statement can never be mistaken for a worked case.

I am building a synthetic training corpus about {setting}. Items called "{noun}" are sent to the {yes} queue or the {no} queue.

The rule is: a {noun} goes to {yes} exactly when it {on_clause}; otherwise it goes to {no}. The other properties ({inert}) are recorded but do not affect the outcome.

Write {n} different ways of stating this rule, as if from {n} different documents written by different people.

HARD REQUIREMENTS:
1. Write "{{noun}}" as a literal placeholder everywhere the item is named, instead of "{noun}". I substitute it later. Keep the braces.
2. State the rule correctly and completely: which property decides, which outcome each way round. A statement that omits one direction is useless to me.
3. Say or imply that the other properties do not matter. You may name them or not.
4. Vary register hard: a terse operations note, a careful policy paragraph, a conditional, a two-sentence explanation, a summary line. Vary length from one clause to three sentences.
5. Avoid these words, which are reserved elsewhere in the dataset: {banned}.
6. Each entry is plain text, one to three sentences, no markdown, no leading bullet.

Return ONLY JSON: {{"rules": [...]}}

Why this recipe, and not one of the others

The scaling grid: 3 category counts x 7 documents-per-category x 4 seeds, the shape v4/24 and v4/26 already publish, on the corpus this suite settled on.

`route_ab_l8` is that corpus: `log8` with the rule restated in API-written wording that differs document to document, and the worked examples left canonical. It was picked over plain `log8` on three counts -- equal or better selectivity from five documents up, a third less perplexity, and seeds that agree (0.913/0.915 at twenty documents against plain's 0.540/0.930/1.000) -- and over the odd-frames variant, which beat it only at one document and cost 0.19 at ten.

One learning rate across the grid. The rate sweep on this corpus ran 3e-4, 2.5e-4, 2e-4 and 1e-4 at five, ten and twenty documents and selectivity fell monotonically every time while perplexity improved, so the rate is a trade rather than a thing to tune per cell, and 3e-4 is the end of it that answers the question. It is not tuned per *category* count, which is the assumption most likely to be wrong here.

One forced choice between two single-token queue names

A trained model is asked a direct question about a held-out noun in a held-out property combination, and scored on the first token: does it prefer Cedar or Birch? Both are single tokens in the base tokenizer with a leading space, chosen for exactly that reason — a first-token forced choice between a whole word and a word-start fragment would not be a fair comparison. Chance is 0.50.

Why accuracy alone is not the headline number

A model that answers a constant queue per category scores 0.5 — the same as chance on a per-category basis, hidden inside an aggregate that looks better than chance. The number that catches this is selectivity: how often the answer changes when the deciding property is flipped, minus how often it changes when an irrelevant property is flipped. A rule-following model scores near 1; a constant-answer model scores near 0 whatever its accuracy.

What else gets measured

• retrieval. A forced choice among the four properties (chance 0.25), or among eight property-and-polarity statements (chance 0.12) — asking whether the rule is stored at all, separately from whether it is used.
• gold. The true rule sentence pasted above the question — the application ceiling, and a check that a null result on the plain question is real rather than a broken run.
• self. The model recalls its own rule first, in a separate generation, and that recollection is pasted above the same question — whether a retrieval step the model performs itself, rather than one handed to it, activates the same application ability.

Both option orders (a/b and b/a above) are scored, so a preference for whichever name is mentioned first in the question shows up as a gap between them rather than as accuracy.

Real numbers, off the sweep's own results file

Averaged over whichever of the 4 seeds have finished training as of this build (58 of 84 runs). Not copied from a README — read straight out of models/v4/25/runs/sweep_results.jsonl, the same file sweep.py --table reads.

What was trained on this

84 LoRA adapters over Qwen/Qwen3.5-4B — every cell of the 3×7 grid trained 4 times from a different shuffle, at a fixed learning rate of 0.0003 chosen by a rate sweep that traded selectivity for perplexity monotonically at every other rate tried. LoRA rank 64, alpha 128, applied to all-linear with embed_tokens, lm_head also trained, since the vocabulary carries no special tokens here but the queue names still need to move.

2 of the 4 seeds in every cell keep 8 mid-training checkpoints, for attribution methods that score a trajectory rather than a finished adapter.

lamsheeper-data-attribution/Qwen3.5-4B-route-ab-l8-lora-scale is the archive all 84 of them are pushed to.

Browse the corpus →

The grid is the sweep: 3 category counts across 7 document counts, each cell a training file, each trained 4 times. The corpora nest, so a smaller cell is always a prefix of a larger one — which is the whole reason the sweep can read a difference between two cells as being about composition rather than about a redraw.

Rows per training file. Click a cell.