Atlantis PbeM Rewritten (Part 9): The game file—read it faithfully first, improve it later

After part 8, the Python port generates the same world from a seed as the C++ reference. So far, though, that world exists only in memory.

For a game played turn by turn over email, that's not enough. Between two turns, the complete state has to go to disk, and later it has to be read back in so that the game continues exactly where it left off.

So Step 8 of the roadmap ports loading and saving. The done criterion:

A game written by the reference loads in Python and round-trips without loss.

With an old file format, the temptation to fix the format first is strong. You see 150,000 lines without field names, and your mind goes to JSON pretty fast.

On that point, the roadmap deliberately says only this: The existing format is read faithfully first. A new format is a later decision.

This step shows pretty well why that order makes sense. Much of what a game file means isn't in the file at all.

A large part of its specification lives in the code that reads it.

A format without field names

This is how game.out begins for the world from part 8:

# reference/snapshot-tests/worldgen/standard/output/game.out
atlantis_game
328197
Wyreth
131072
1
-1
6621
3
404
100
1
2

First comes the magic line, then the engine version as a packed integer, the name of the ruleset, and its version. After that come the year, the month, a seed, and five counters.

Then: the factions, the regions with their buildings and units, and finally the quests.

The format is almost entirely positional. One value per line, a length in front of every list, no field names. Items and other codes are partly stored by name, like IRON, plain, or Fleet. Empty references are written as NO_ITEM or NO_SKILL, and illusions get an i in front of the item abbreviation.

That puts the world from part 8 at 148,044 lines and 546,346 bytes.

A quick reminder in case you're joining the series here: Atlantis PbeM is a strategy game played by email. The world is made of hex regions in which player factions move their units, use buildings, produce, and trade. Practically all of the world's persistent state sits in this one text file.

For this, the C++ reader essentially needs two stream idioms:

f >> x;

skips leading whitespace and reads the next value.

And:

std::getline(f >> std::ws, s);

skips leading whitespace, blank lines included, and then reads the rest of the line.

The Python port mirrors exactly these two ways of reading:

# src/atlantis_ng/core/savefile.py
def integer(self) -> int:
    """``f >> int``: the next token, which must be an integer."""
    self.skip_whitespace()
    start = self.pos
    token = self.token()
    if _INTEGER.fullmatch(token) is None:
        self.pos = start
        raise SaveFileError(f"line {self.line_number}: expected an integer, found {token!r}")
    return int(token)

The model classes from part 7 each get a read_in and a write_out for this, analogous to the reference's Readin and Writeout methods.

Writing is the easy part: putting values into a text stream in the right order.

Reading is more interesting.

Because Readin is by no means simply the inverse of Writeout.

Loading changes the state

An excerpt from Unit::Readin:

// reference/unit.cpp
f >> num;
set_name(name);
f >> type;
// ...
f >> guard;
if (guard == GUARD_ADVANCE) guard = GUARD_NONE;
if (guard == GUARD_SET) guard = GUARD_GUARD;

So the guard status isn't simply taken from the file.

GUARD_ADVANCE, a temporary status for one turn, becomes GUARD_NONE on load. GUARD_SET becomes the normal GUARD_GUARD.

The name isn't just assigned either. It goes through set_name, which, for units, appends the number again.

The same pattern runs through the whole format.

A region's building sequence is rebuilt from the buildings it reads instead of being taken from the file as a finished counter. After loading, a faction counts as an NPC if its faction types contain the -1 values used to mark one. Units with an unknown faction are dropped. Disabled items disappear, and so do skills without days of study. Duplicate productions are removed and reported.

These rules don't sit next to the format description.

They are part of the format description.

If you replace game.out directly with a nicer data model without understanding the reader first, you risk losing exactly these semantics.

The new file might look cleaner then. But after loading, a unit suddenly keeps a status it was only supposed to have for one turn.

Here, the reader is the actual specification.

Where the reference doesn't look

There's one property of the reference I don't want to adopt, though.

Many of the Readin methods read values through C++ streams without checking the stream state afterward.

If, say, there's text where an integer is expected, the numeric extraction fails, sets failbit, and, for this failed integer parse, writes 0 into the target variable. The stream then stays in the failed state; further formatted extractions don't just carry on reading normally.

In many of the nested readers, though, the reference doesn't handle this state where it arises. Depending on where the file is damaged, the code carries on with zero, empty, or stale values, until a later explicit check might still end the load.

For a compatibility port, that's not particularly useful behavior.

The file is already broken. The only thing that really helps now is finding out where.

That's why decision 0004 came before the reader:

Invalid input raises a named error where the reference silently reads on.

The reference's own deliberate checks stay in place. A wrong magic line, an incompatible engine version, or a different ruleset leads to False and the matching message, as in the original.

If, on the other hand, an integer is suddenly missing deep inside a region, the Python port raises SaveFileError and names the line.

The decision is deliberately more general than game.out. It also applies to the players file and, later, to the orders files.

That way, the same question doesn't have to be argued from scratch at the next parser.

The deviation could also have been noted in the porting map under game.cpp and left at that. A decision record of its own, though, makes it visible that this is a rule for several readers.

Two oracles were already in the repository

A normal round-trip test would be too weak for this step.

If the reader and the writer make the same mistake,

read → write → read

can look perfect.

So the port needs files that the reference wrote.

And there were plenty of those already.

Since part 3, the submodule has held the world generation snapshot: game.out and players.out for seed 20260822.

On top of that come fourteen saved states of a running game with one player faction, from turn 0 to turn 13. Each of these game files has about 48,000 lines.

No extra test program, no Docker build.

Just reading files.

That makes the sharpest test almost free:

# tests/core/game/test_files.py
def test_the_new_game_saves_byte_for_byte_what_the_reference_saved(game: Game, worldgen_game: str) -> None:
    """Step 7's world plus this step's writer give the reference's game.out, seed line included."""
    assert new_game(game, SEED)
    assert save(game) == worldgen_game

The Python port generates the world from part 8 and saves it.

The result matches the C++ reference's game.out byte for byte: all 148,044 lines, seed line included.

That works because part 8 doesn't just generate the same world. It also brings the random number generator to the same point. For the seed, SaveGame doesn't write a stored MT state; it draws:

rng::get_random(10000)

So this draw, too, has to happen at exactly the same point.

The writer passed this test on its first complete run.

What loading does with the same world

The second test goes the other way.

It loads the reference's game.out into Python and compares the result with the probe's complete world dump from part 8, that is, with the state right after NewGame.

Three differences remain.

All three also arise on load in the C++ reference itself.

Right after world generation, the 163 lairs were just called Dummy. On load, their name goes through set_name, which adds the object number. After that, they're called Dummy [17], for example.

In one region, buildingseq is reset from 2 to 1, because the reader recalculates the value from the objects it actually loaded.

And after loading, the two NPC factions carry -1 in their faction types instead of the values from the freshly generated world.

The game.out after the first turn of the existing harness shows the same renaming of the 163 lairs.

So the test explicitly treats these three differences as expected load behavior.

These are exactly the things that would have been lost if this step had started straight away with a new save format.

A round trip doesn't have to produce the same bytes

For a game in progress, the next test takes turn 13.

The reference file is loaded, saved, loaded again, and saved again.

On the first save, two lines differ from the old reference file:

Line 2 holds the engine version. The port writes the current patch level, just as a current reference engine would when saving.

Line 7 holds the seed.

The second save differs from the first only in the seed line.

This is important for what "round-trips without loss" means.

OpenGame reads the stored value from line 7 and seeds the random number generator with it. Later, SaveGame writes a new get_random(10000) value.

So the seed line isn't a field meant to be kept unchanged. It carries the starting point for the next random stream.

A byte-identical load-save round trip is therefore not a general promise of this format.

What has to survive is the game state and the semantics of the random stream that follows.

13 seconds for a text file

The reader was correct on its first run.

It took 13.4 seconds.

For a 546 KB file, that's conspicuous enough not to wave it off with "Python is just slower."

The profiler showed two very unspectacular causes.

integer() was meant to report the line number in case of an error. But the first version computed that number for every integer, before it was even clear whether there was an error at all:

self.text.count("\n", 0, self.pos)

Again and again, over the same ever-longer text.

This path alone showed up in the profile with 11.3 seconds.

After the fix, the line number is only computed when an exception actually has to be built.

The second hotspot was lookup_item.

The file contains 14,693 item abbreviations. For each one, the port went linearly through all 230 items and compared the strings case-insensitively.

The fix is an index:

# src/atlantis_ng/core/data/game_data.py
def lookup_item(self, name: str) -> ItemId | None:
    """The index of the item whose abbreviation (with ``i`` for an illusion) is ``name``; the first of two alike."""
    if self._item_index is None:
        self._item_index = {}
        for index, item in enumerate(self.items):
            tag = ("i" if item.type & ItemCategory.ILLUSION else "") + item.abbr
            self._item_index.setdefault(ci_key(tag), ItemId(index))
    return self._item_index.get(ci_key(name))

setdefault isn't just a convenience here. When searching linearly, the reference takes the first matching entry in table order. setdefault also keeps the first one when the index is built.

The one modifier that can change such an abbreviation drops the cache again.

After that, loading takes about 0.25 seconds, saving about 0.02 seconds.

At this point, the order matters more to me than the optimization itself:

First get correct against the oracle. Then measure. Then change exactly the hotspots that actually cost time.

players.out: the game master's side

Next to game.out, there's players.out.

That's the game master's working file. For every faction, it holds the name, email address, password, and settings.

Between two turns, the game master can turn it into players.in and add to it: Faction: new creates a new player, and other lines can create units, hand out items or skills, and later also set orders.

Before a reader even existed, the simplest test came first again:

write_players right after new_game produces exactly the players.out of the reference snapshot.

330 bytes, byte for byte.

This time, there was no ready-made fixture for the other direction. The turn snapshots only show the state after a complete turn, not right after ReadPlayers.

So the model probe got another mode:

model_probe players N

Ten different players.in files are embedded directly in the probe. For each scenario, the reference first generates the same world, reads in the players file, and then writes the return value, logs, factions, report state, new units, and the resulting players.out as JSON.

The ten cases cover every line type, including new factions and the error messages for invalid entries.

Each scenario runs in its own process. The reference still keeps process-wide state in global tables and in its name pool; for isolated tests, a fresh process is the cleanest boundary.

The merged fixture is 43,395 bytes.

The Python port passed all ten scenarios on the first run.

Two details from it are still interesting.

A field full of memory garbage

For guard unit number 2, the probe showed:

gm_alias = 1565081938

gm_alias is the identifier through which the game master can refer to freshly created units in later lines of the same players file.

The problem: The C++ constructors don't initialize this field.

So the specific number 1565081938 has no meaning in the game. It's just the memory content this particular probe run happened to find at that spot, not a reference value a port should reproduce.

As long as nobody ever tries to address an existing NPC unit through exactly this random alias, nobody notices.

The Python port deliberately turns this into defined behavior:

gm_alias = 0

And the comparison ignores gm_alias for the NPC units whose value the reference never meaningfully initialized.

That's a different case from the many strange but defined behaviors of the last few parts.

Memory garbage doesn't get preserved.

The last line disappears

The second detail, on the other hand, is reproducible reference behavior.

The loop of ReadPlayers checks eof() before processing anything:

// reference/game.cpp
while(!f.eof()) {
    auto token = parser.get_token();
    if (!token) {
        f >> parser;
        continue;
    }
    // ...

Internally, the string_parser reads its next line via

std::getline(is >> std::ws, input);

If the file ends with a last line without a trailing newline, getline still delivers its content, but on reaching the end of the file, it sets eofbit at the same time.

So on the next pass, while(!f.eof()) is already false.

The line is in the parser—but it never gets processed.

The test scenario unterminated ends with

Faction: new
Name: Unfinished

with no newline after Name: Unfinished.

So the new faction keeps its default name Faction (3).

The Python port reproduces even this quirk. SaveReader.input_line returns not just the line, but also whether the end of the stream has been reached after it:

# src/atlantis_ng/core/savefile.py
def input_line(self) -> tuple[str, bool]:
    """``f >> parser``: the next line as :meth:`line` takes it, and whether the stream's end flag is set after it.

    The flag is what ``f.eof()`` reports: set once the whitespace before the line, or the
    line itself, ran into the end of the text. At the end the line is empty, as ``getline``
    leaves the string it could not fill.
    """
    self.skip_whitespace()
    if self.pos >= len(self.text):
        return "", True
    end = self.text.find("\n", self.pos)
    if end < 0:
        line = self.text[self.pos :]
        self.pos = len(self.text)
        return line, True
    line = self.text[self.pos : end]
    self.pos = end + 1
    return line, False

Is that a bug in the C++ reader?

Very likely.

Can it affect a real game turn?

Yes. A game master only has to save their file without a trailing newline.

So for now, this effect is part of the compatibility target.

One line type is still open in this step: Order: with a real unit order needs the order parser from Step 9. The checks before it (missing unit, wrong faction, missing order) are already ported. So is Order: quit.

The actual unit order still deliberately ends in NotImplementedError.

The state dump had a conceptual problem

Since part 3, a state.json has been planned for the comparison harness.

Instead of only reporting

game.out, line 84213

on a deviation, the dump is later supposed to be able to say that, for example, unit 442 in region 217 has twelve silver instead of thirteen.

The original plan was:

The reference writes game.out, and the ported reader loads that file and turns it into JSON.

The Python port produces the same JSON directly from its objects.

Sounds logical.

Doesn't work, though.

We've just seen that loading changes the state.

Right after NewGame, 163 lairs are called Dummy; after a load, they're called Dummy [n]. A building sequence changes. So do the NPC faction types.

A dump of the loaded C++ file and a dump of the never loaded Python objects would therefore report differences as early as turn 0, even though both engines wrote exactly the same game.out.

So the state dump was built differently.

It isn't a dump of the object model but a positional converter of the file text.

tools/compare/state.py has 385 lines and uses the same reading primitives as the actual reader. Both engines produce state.json from the game.out they wrote themselves.

That way, the dump describes exactly what actually got persisted.

For the world from part 8, that comes to about 3.3 MB of JSON with field names.

This converter, too, shows once more how much implicit knowledge sits in the format.

The file doesn't reveal, for example, how many faction types the ruleset uses. Faction::Writeout just writes one value per entry of a global list, without storing a count first. So the converter has to be told that number from outside.

For skills, whether there's an extra experience value also depends on a ruleset setting that isn't stated separately in the file header.

The plan change was recorded in the journal and in docs/comparison.md. It doesn't need a decision record of its own: The engine's behavior doesn't change, only how a test artifact is built.

Now the Python port plays along in the harness

With world generation, loading, and saving, the port can now do everything the reference command new needs for the comparison harness.

So the harness from part 3 gets its second engine.

PythonEngine.new generates the world in the same process and writes:

game.out, players.out, state.json, and the console output.

That takes about 0.6 seconds.

The Python engine can't do run yet; that needs the turn processing from Step 11.

The first comparison of worldgen still failed.

Not on the world.

On engine-output.txt.

During world generation, the C++ reference prints progress dots and blank lines that part 8 deliberately didn't rebuild. At the same time, the port was missing a real message, Making a level..., and the banner from main.cpp.

The fix separates content from progress formatting.

The missing message was added. The Python engine writes the banner and the seed line like main.cpp. And for the console output only, the normalization removes dot lines and blank lines:

# tools/compare/normalize.py
def normalize_console(data: bytes) -> bytes:
    """``normalize_text``, then without the progress: lines of dots and blank lines."""
    lines = normalize_text(data).split(b"\n")
    return b"".join(line + b"\n" for line in lines if line and not PROGRESS.match(line))

Other text artifacts keep their blank lines.

After that:

$ uv run tools/compare/cli.py compare worldgen reference python
worldgen: reference vs python: equivalent (1 turns, 4 artefacts)

Four artifacts: game.out, players.out, state.json, and engine-output.txt.

The two state.json files are byte-identical.

A reference test now pins this comparison down in CI as well.

The heredoc, for the second time

One mistake of this step has little to do with the game, but it's still worth mentioning.

Heredocs had already given the agent trouble in part 2. This time it was a different variant of the same tool friction.

A quick script was supposed to add write_out methods to four Python modules. The script itself was passed through the shell.

On the way, the \n sequences in the embedded Python code turned into real line breaks in the middle of string literals.

The repair script was supposed to turn these line breaks back, and the same rule hit the docstrings as well.

The result: restore four files from Git and run the change again, this time with a script that didn't have to pass through the same shell quoting layer.

The journal explicitly records the mistake as a repeat of an already known tool trap.

The interesting part isn't really Bash.

Remembering a known trap isn't reliably enough. It only really takes effect once it's a fixed rule in the working context, or once the tool no longer offers the problematic path at all.

What the step cost

Four pull requests complete Step 8: the reader and writer of game.out along with decision 0004, the players file and its probe mode, the state dump plus the second harness engine, and finally the close-out.

1,328 lines were added under src/.

Of those, 96 go to the stream reader, 117 to loading and saving the game, 373 to the players file, and 80 to the ruleset hook for new factions. About 620 more lines are spread across the read_in/write_out pairs of the existing object model.

The harness got the 385-line state converter and the Python engine. The model probe grew by 157 lines.

416 fast tests, 62 of them new, run in about 20 seconds. 13 more tests compare against the reference.

Porting coverage rises from 34.9 to 35.6 percent.

With market.cpp, production.cpp, and quests.cpp, three smaller files are now fully ported. The big model classes stay open, because reports and turn logic are still missing there.

Besides decision 0004, the porting map lists four deliberate deviations: Python works with streams instead of hard-wired file names, gm_alias starts at a defined 0 instead of uninitialized memory, a faction pointer that's problematic in the reference is set in the port, and Order: is still waiting for Step 9.

Pitfalls

• The reader is part of the format specification. Reset rules, recalculated counters, and dropped entries aren't in the file text. If you only rebuild the writer or replace the format right away, you miss exactly these semantics.
• A round trip mainly proves that the reader and the writer fit each other. For compatibility, you also need files the original wrote.
• Byte identity needs the right point of comparison. A freshly generated world can write exactly the same file. After a load, though, loader semantics, engine version, and seed change the next save.
• An invalid stream state isn't a useful compatibility target. Where the reference doesn't diagnose broken input, the port stops with SaveFileError at the first point it detects.
• Uninitialized memory isn't a reference value. gm_alias = 1565081938 describes a single run, not semantics. The port defines the field as 0.
• Compare persistence with persistence. A dump of loaded objects isn't the same as a dump of freshly created objects when loading itself transforms state.
• Correct first, then measure. Getting from 13.4 to 0.25 seconds didn't take a new parser, just two specific hotspots the profiler made visible.
• A known tool trap doesn't go away because you know about it. Only a fixed working rule or a different tool path reliably prevents a repeat.

Where things stand after part 9

The Python port can now load the reference's saved games and write them back in the reference's format.

For a freshly generated world, the result is byte for byte identical to the reference's game.out: all 148,044 lines.

For a game already in progress, the state is preserved semantically; the seed line is regenerated on the next save according to the random stream, and older snapshots get the current engine patch level when written.

The players file works in both directions too. Its line types are checked against ten reference scenarios, except for the unit order behind Order:, which is deliberately deferred to Step 9.

The comparison harness now has two engines. For worldgen, the reference and the Python port are equivalent across four artifacts, and their state.json files match byte for byte.

What's still missing is clearly delimited.

Order: needs the order parser. State dumps between individual turn phases come with Step 11. And only there does the Python engine get its actual run command.

So the next part takes on parseorders.cpp: the parser every player order runs through, from MOVE to CAST.

Grown code, lots of special cases—and the same goal once again:

Become more readable without accidentally treating a single previously accepted or rejected order differently.

Further reading

• Atlantis PbeM Rewritten (Part 8)—the world generation whose exact random stream makes the byte identity of the new world possible here.
• Atlantis PbeM Rewritten (Part 7)—the object model, which can now read and write, and the model probe.
• Atlantis PbeM Rewritten (Part 3)—the harness in which the port now plays along as the second engine.
• Testing determinism: controlling randomness, time, and I/O—why a seed alone only helps if the consumption of the random stream is controlled too.
• Learning Python (Part 9): Files and saving—the basics of text files that loading and saving build on here.

Sources: game.cpp, unit.cpp, aregion.cpp, faction.cpp, and string_parser.hpp of the fork, std::basic_istream::operator>>, std::ios_base::iostate, std::getline, Python cProfile.