Run comparison

Do advisors help stuck copies (h18)? Levels beyond h14

h14 copies that were stuck by the end of the run were rolled back and set to play on in h18; 30 of them have their own advisor (it sees only its own copy's game), the rest play without one. Horizontal: each copy's tokens after the rollback. Vertical: how many levels per copy the copies gained in h18 beyond what the same copies gained in h14 on the same number of tokens. The orange dashed line at zero is “same as h14”. Black: copies with an advisor; grey: without. Black above grey means advisors help; both above zero means h18 simply plays better than h14 (it runs on int4, h14 on int6, and that has an effect too).

dtb1 ducks: temperature bursts vs control

63 stuck h15 copies (h15 minute 200, many with over 200K tokens on an unwon level) play on from the same spot for 45 minutes. Half of the ducks (split evenly within each game) get bursts: in every 30K tokens, 10K run at temperature 1.5, top_k 40, top_p 0.98, the rest at the usual 0.7 / 20 / 0.95. The control stays on the usual settings throughout. Horizontal: a duck's tokens since its start; vertical: how many levels a duck gained from its starting point, averaged over the ducks that reached that point. Orange above grey means bursts help. Light orange and blue are the dtc1 twin: the same ducks, the same arms and the same serve, but without the int8 GDN state and without trim holes; comparing grey with blue shows whether those serve modes cost levels.

Does the agent get dumber after a context trim? Stuck levels only

Current agent only: bullet vh1104 (vh9h2) and its twins without “Let me reconsider” (hxt9, hxl1). The agent trims the context when it reaches about 110K and keeps about 55K. Only stuck levels count: on that level the copy has already survived at least one trim and made at least 10 moves, so “longer in the game” and “new level” don't skew the comparison. Each full trim cycle (from one trim to the next) is split into five equal parts by moves: 0–20% are the first moves after a trim, 80–100% the last ones before the next, on the longest context. That way every cycle weighs the same in each part, and long cycles don't pull on the middle. Vertical: levels per 100 moves. Black: the whole run; orange: only after 3 hours; bands: 90% interval.

Does the agent win levels on long or short context? Stuck levels only

The same moves, but horizontal is the context length at the start of the move, in 10K bins: on the left ~55K right after a trim, on the right ~110K before the next one. The length is an estimate: the last measured prompt token count from the move log plus the characters added since (error ~2K); cycles without a single measurement have no estimate, so there are fewer moves here than on the neighbouring chart. Bins with fewer than 60 moves are not shown.

Levels per copy — at an equal budget for every copy

A fair per-token comparison. Horizontal: budget B, how many tokens EACH copy was given. Vertical: how many levels a copy won on average within its first B tokens. Copies that finished the whole game count with their final result; the line continues while at least 90% of the unfinished copies have reached budget B. This way it does not matter how the run shared tokens among copies (lanes, stints, rotation).

Score — at an equal budget for every copy

The same, but the score by the scoring formula (levels with weights and moves), counted only over levels won within each copy's first B tokens.

Levels per copy — by tokens spent (run average)

The dashed lines at the top are the ceiling: 80, 90 and 100% of all levels of the 25 games.

Score — by tokens spent (run average)

Horizontal here is all of the run's tokens divided by the number of copies: if the run gave out tokens unevenly, this mixes the model's smarts with the token allocation; the fair comparison is the budget charts above. Score by the scoring formula: it counts both the levels won and how many moves they took. The “no worse than human” button adds a dashed line of the same colour: the same won levels, but every level completed worse than a human is credited as completed in exactly the human's move count (100 of 115); levels where we are already better stay as they are. The gap between the solid and the dashed line is how many points move economy is worth.

Economy: share of won-level points eaten by extra moves

What percentage of the won levels' points (had each been completed no worse than a human) is lost to extra moves, with level weights as in scoring. The share does not grow just because a run won more levels, so a fast run does not look wasteful because of that. The lower the line, the more economical the run. Axis: player tokens per copy, as on the score chart.

Economy: share of levels completed worse than a human

Of all levels won by this point, the percentage on which we spent more moves than a human (RESET counts as a move too). Lower is more economical. Axis: player tokens per copy.

Levels per copy — by minutes

Here both a smart agent and a fast server win.

Tokens per copy — by minutes

The steeper the line, the more tokens per minute each copy gets.

Server speed, tokens per second

Photographers: from the server log, 5-minute average (the “Smooth speed” button uses 1/15 of the run length, up to 30 minutes); older runs: from the copies' tokens over 10 minutes. A circle with a number shows how many copies are still playing.

Speed per playing copy, tokens per second

Server speed divided by the number of copies that have not finished yet.