Skip to content

cat posts/en/atlantis-pbem-neu-geschrieben-10-die-befehle.md

Atlantis PbeM Rewritten (Part 10): The orders—a parser with 61 cases and two readings

Every player order from MOVE to CAST runs through 2,808 lines of organically grown C++ code that reads the same line in two ways. Part 10 ports the parser more readably, compares checker and turn against the reference—and runs into a real null pointer crash.

Retro photo: A brown pointer stands in an empty meadow, staring fixedly at nothing, while behind it a hunter in a tweed coat touches his hat, at a loss.

After part 9, the Python port can load and save a game. What it can't do yet: understand what the players want.

In Atlantis PbeM, every faction sends a text file of orders each turn. Here's an excerpt from a real turn of the reference snapshot, without the comment lines, the template's header line, and the second newly formed unit:

# reference/snapshot-tests/turns/turn_3/orders.3
#atlantis 3
unit 152
@work
unit 140
name unit "King"
form 1
    name unit "Ranchers"
    claim 790
    buy 10 PLAI
    study RANC
    turn
        @produce LIVE
    endturn
end
claim 100
study forc
#end

#atlantis opens a faction's orders, unit selects a unit. After that, there's normally one order per line.

form and end describe a newly formed unit inside the orders of its parent unit. turn and endturn queue orders for a later turn. An @ marks a repeating order. claim takes the faction's unclaimed silver, study learns a skill. The indentation is only there for humans.

Step 9 of the roadmap ports this parser. The done criterion:

All of the reference's orders files are read with the same accepted orders and the same errors.

The interesting question here isn't how you'd write a parser today.

It's this: How far can you clean up 25 years of organically grown parser code when its behavior must not change?

A parser with two readings

parseorders.cpp has 2,808 lines.

The rough structure is surprisingly simple. Game::ParseOrders runs over the lines of an orders file and handles the structural words itself: #atlantis, #end, unit, form, end, turn, and endturn.

Everything else goes to Game::ProcessOrder.

A second switch waits there:

// reference/parseorders.cpp
void Game::ProcessOrder(int order, Unit *unit, parser::string_parser& parser, orders_check *checker, bool repeating)
{
    switch(order) {
        case O_ADDRESS:
            ProcessAddressOrder(unit, parser, checker);
            break;
        case O_ADVANCE:
            ProcessAdvanceOrder(unit, parser, checker);
            break;
        // ... 59 more cases

61 cases, each with a Process*Order function.

Nearly every one of them gets the same three things: the current unit, the parser for the line, and a checker.

And this checker is what makes the code more interesting than its structure suggests.

Atlantis PbeM has an order checker. Before the deadline, players can send their file to the check command and get it back: the lines as read, error messages in between, and the number of errors at the end.

For that, the same parser gets used a second time.

In the real turn, it writes orders onto units and factions. In a check, it runs with a checker and a dummy unit.

Take BEHIND, for example:

// reference/parseorders.cpp
void Game::ProcessBehindOrder(Unit *u, parser::string_parser& parser, orders_check *checker)
{
    /* This is an instant order */
    auto val = parser.get_token().get_bool();
    if (!val) {
        parse_error(checker, u, 0, "BEHIND: Invalid value.");
        return;
    }

    if (checker) return;

    u->SetFlag(FLAG_BEHIND, val.value());
}

Up to

if (checker) return;

the code checks whether the order is syntactically valid.

After that comes its effect in the game.

Depending on the mode, parse_error writes either to the check file or to the error report of the unit or the faction.

So the same code reads the same order line in two ways.

And these two readings aren't always identical.

Why the parser wasn't split into "check" and "execute"

From today's point of view, a refactoring suggests itself: First build a clean parser that produces a validated order, then a second layer that executes it.

That would probably be nicer.

For this port, it would be risky.

The order of the checks is part of the observable behavior. If a line has two errors, the existing control flow decides which one gets reported first. Some checks run only in check mode, others only in the real game. In places, the checker even consumes different tokens than the turn.

These differences land directly in a file that players have been seeing for years.

That's why the basic structure stays.

The Python code gets more readable in less dangerous places: The switch cases become match, the functions are consistently named process_<order>_order, groups sit together, and docstrings name the corresponding C++ function.

Sentinel values get names too:

# src/atlantis_ng/core/game/order_handlers.py
# GIVE and TAKE amounts that are not a number: the whole unit, all of the item.
GIVE_UNIT = -1
GIVE_ALL = -2

A C++ pointer that holds either nullptr or a UnitId becomes UnitId | None in Python.

That way, parts of the mechanical C++ complexity disappear without reinventing the order of the actual parser logic.

order_handlers.py and parse_orders.py together come to 2,329 lines instead of the 2,808 lines of parseorders.cpp.

None of the 61 cases is missing.

Merging only with proof

A few functions could really be merged, though.

ProcessBehindOrder, ProcessNoaidOrder, ProcessHoldOrder, ProcessAutoTaxOrder, and ProcessShareOrder follow the same pattern. They differ only in the word in the error message and in the flag they set.

In the port, there's one function for them:

# src/atlantis_ng/core/game/order_handlers.py
def process_flag_order(
    unit: Unit, parser: StringParser, checker: OrdersCheck | None, word: str, flag: UnitFlag
) -> None:
    """The five flag orders: ``ProcessBehindOrder``, ``NoaidOrder``, ``HoldOrder``, ``AutoTaxOrder``, ``ShareOrder``.

    One function for the five flag orders that differ only in the word and the flag; NOCROSS
    comes here too, after its own check in :func:`process_nocross_order`.
    """
    value = parser.get_token().get_bool()
    if value is None:
        parse_error(checker, unit, None, f"{word}: Invalid value.")
        return
    if checker is not None:
        return
    unit.set_flag(flag, int(value))

NOCROSS uses it too, after its additional check.

Other merges came about for MOVE and ADVANCE, WEAPON and ARMOR, and three very similar parsers for the phantasmal summoning spells.

The rule for this was deliberately narrow:

Code only gets merged if its differences can be expressed entirely as parameters.

Whether that's really the case wasn't decided by gut feeling but checked with a diff of the C++ function bodies.

With the phantasmal spells, for example, a differing message was one of those parameters.

One more question had to be settled: Could another ruleset swap out one of these parsers and so get in the way of the merge later?

No. The rulesets bring their own files like extra.cpp, world.cpp, rules.cpp, and monsters.cpp. parseorders.cpp belongs to the shared engine.

So the parser is the same for every ruleset.

The oracle has to see both sides

With 61 orders, one test per function is hardly enough.

So the model probe from part 7 got another mode:

model_probe orders scenario N

The starting point is the state of turn 13 of the reference snapshot: one player faction with six units in three regions, a mage among them.

The probe first runs PreProcessTurn and ReadPlayers, which is exactly the start of Game::RunGame.

Then it reads the same orders file twice.

Once with a checker as faction 0, like the check command.

And once as the player faction's orders.3 through Game::ReadOrders, the way it happens in a real turn.

It records the check file, the errors and events of the player factions, and every field of a unit where an order can end up: monthorders, giveorders, castorders, turnorders, and the other order slots.

The scenarios themselves are hand-written orders files in the C++ probe code:

// tools/reference/model_probe.cpp
{"flags", "",
 "#atlantis 3\nunit 140\nbehind 1\nnoaid on\nhold true\nautotax yes\nshare 1\nnocross 1\nbehind x\nnoaid\n"
 "hold 2\nautotax\nshare no\nnocross off\nspoils walk\nspoils bogus\nspoils\nspoils none\nspoils all\nspoils ride\n"
 // … more lines up to #end

Each scenario targets a group of orders: valid values, invalid values, missing arguments, and edge cases.

Through players_append, the probe can also append lines to players.in. That way, a unit gets weapons before parsing, for example, when a scenario is supposed to check weapon swor lbow.

The first round had 21 scenarios.

The Python port reproduced all 21 on the first comparison run, check files included.

The only red test in that run was a wrong expectation in the Python test: faction war 2 trade 2 uses four faction points. standard allows five.

Things that only showed up in the dump

ParseOrders had been read beforehand.

Still, some details only became visible once the probe recorded the state afterward.

A failed form, for example, doesn't just keep the previous unit selected.

form without an alias, form 0, or form x leaves no unit selected at all. ProcessFormOrder returns nullptr, and the caller takes exactly that value as the current unit.

So every following order reports:

Order given without a unit selected.

until a valid unit line comes along again.

#atlantis without a password behaves surprisingly too. The checker produces two messages every time:

Warning: No password on #atlantis line.
If this is your first turn, ignore this error.

So even an otherwise perfect file without a password ends with errors.

And:

declare -1 ally

really does set the faction's default attitude. get_number returns -1, and that exact value means "default" internally.

None of this is particularly complicated.

But these are exactly the little things that easily get lost in a "clean" new parser.

The checker indents backward

A quirk of the check file is even more visible.

A nested block can look like this there:

unit 140
turn
behind 1
form 5
work
  end
    endturn

You'd expect turn and form to increase the indentation and end and endturn to decrease it again.

The reference does exactly the opposite:

// reference/parseorders.cpp
if (checker) {
    if (code == O_ENDTURN || code == O_ENDFORM)
        checker->check_file << indent::incr;
    checker->check_file << prefix << order.original() << "\n";
    if (code == O_TURN || code == O_FORM)
        checker->check_file << indent::decr;
}

Before a closing word, incr runs.

After an opening word, decr runs.

On top of that, indent::clear removes the stream's indentation buffer completely at unit and #end. A decr without an installed buffer has nothing left to decrease.

Without a buffer, text goes straight into the stream, which also means it bypasses the line wrapping of the IndentWriter.

So the Python port reproduces even this small stream state:

# src/atlantis_ng/core/game/order_handlers.py
class CheckOutput:
    """The checker's output stream: raw, or through the indenting buffer once a manipulator installs it.

    ...
    """

    ...

    def incr(self) -> None:
        """``os << indent::incr``: two more columns; installs a buffer indented by two."""
        if self._buffer is None:
            self._buffer = IndentWriter(indent=2)
        else:
            self._buffer.incr()

    def decr(self) -> None:
        """``os << indent::decr``: two columns less; nothing without a buffer."""
        if self._buffer is not None:
            self._buffer.decr()

That makes the odd indentation more than a random test output. A fixture pins it down, and the comment next to it explains why it looks that way.

A real crash in the reference

One spot already stood out while reading.

ProcessTeachOrder collects the students of a TEACH order.

If none are given, this error path follows:

// reference/parseorders.cpp
if (!students) {
    parse_error(checker, u, 0, "TEACH: No students given.");
    // If we weren't adding to an existing order, delete the new order
    if (order && u->monthorders->type != O_TEACH) delete order;
    return;
}

The problem is:

u->monthorders->type

After a fresh unit line, ClearOrders has removed the month-long orders. So u->monthorders is nullptr.

teach without students as the first month-long order dereferences that null pointer.

In the code, that looks like a crash.

But with something like this, reading isn't enough.

So the real reference binary ran in the Docker image with the game state of turn 13 and this file:

#atlantis 3
unit 140
teach
#end

The output ended at:

Reading the Orders File...

Exit code 139, on Linux the typical return code after SIGSEGV.

That turned the null pointer from a guess into a fact.

According to the code, the checker has the same bug, because its dummy unit doesn't have a month-long order at that point either. The checker wasn't deliberately crashed a second time; the confirmed turn path is enough for the decision.

What do you port from a segmentation fault?

Not the segmentation fault.

The Python port reports:

TEACH: No students given.

and stops processing this order.

That's the same message the reference already prints before it crashes in this particular situation. If the unit already has a different month-long order, the reference reaches this path without a crash too.

The deviation is in the porting map, like SkillStarvation from an earlier part.

It didn't need an architecture decision of its own. Here, the reference has no behavior that sensibly carries on and would be worth preserving.

For the probe, though, this creates a practical restriction: A scenario must not send teach without students to the C++ reference as a unit's first month-long order.

If this error case is to be tested, the unit gets a study first, for example.

The second half: movement, trade, and spells

The second pull request brought the remaining 23 orders from parseorders.cpp: movement, sailing, building, production, study, teaching, trade, giving, transport, attack, theft, and the other big order types.

One of them is CAST.

Its case sits in ProcessOrder, but ProcessCastOrder itself and the individual spell parsers live in spells.cpp. Eleven parsers read the arguments of their spells: some nothing more, others a region, a unit, or a list of targets.

Only this parse half of spells.cpp belongs in this step.

Actually casting the spells comes later, with turn processing.

That's why the parser part lives in core/spells.py in the port.

Eleven additional scenarios raised the count to 32.

And that's where a problem with the oracle itself showed up.

Final states forget the earlier orders

After parsing, the probe writes the state of a unit's order slots.

That means:

Twenty build lines on the same unit don't produce twenty visible final states.

If each new order replaces the same slot, only the last one is in the dump at the end.

The first draft of the new scenarios had exactly this problem. Many interesting orders appeared somewhere in the middle, but the unit ended on an uninteresting later state.

As a result, the dump proved almost nothing about the actual test case.

So the scenarios were rebuilt.

Instead of filling one unit with as many orders as possible, the interesting final states are spread across six units. Each unit's last order is deliberately the one whose fields should appear in the JSON.

cast-spells, for example, now ends with a region spell, a teleport, Bird Lore, Invisibility with its target list, and a summoning.

cast-gates keeps several forms of Gate Lore and Portal Lore visible.

Both versions of the scenarios passed against the Python port right away.

The extra work didn't go into fixing the port but into making the oracle meaningful.

Invalid item? Bought anyway

Some results of the second round are good examples of why you shouldn't make the parser "more reasonable".

This order:

buy 5 bogus

isn't an error.

At this point, the checker only requires that there's a word for the item at all. It doesn't check whether an item with that abbreviation exists.

So the turn stores a buy order for item -1.

If it's followed by

buy 3 bogus

that even gets merged with the first one.

Result: an order for eight of no valid item.

Class specifiers in GIVE are similarly special.

give 156 all item

stores the value -230 as the item, which is -NITEMS for standard.

give 156 all unfinished ship

ends up at -524288, the negated category bit for ships.

These negative values are part of the existing internal protocol between the parser and the later turn logic.

Replacing them with nicer Python enums would be possible.

But only if every place that expects these values really gets rebuilt along with them later.

For this port, they stay for now.

The checker reads BUILD HELP differently from the turn

Even more interesting:

build help 140

The real turn can read this order.

The checker can't.

Its output:

*** Error: BUILD: Unknown keyword '140'. ***
build help 140 complete

The cause is spread across two functions.

ProcessBuildHelp returns in check mode before it consumes the unit number from the parser.

So back in ProcessBuildOrder, 140 is still waiting as the next word.

There, it gets interpreted as another keyword.

And 140 is neither complete nor any other known keyword.

Each function looks plausible when you read it on its own. Only together do they produce the bug.

The port reproduces it.

Not because it makes sense, but because the check file is the oracle for this step.

If this bug is to be fixed later, then deliberately—and with the same behavior change in the reference and the port.

Not every spell is reachable with standard

The 32 scenarios cover a lot, but not everything.

Transmutation and some phantasmal undead are disabled in standard.

parse_skill already filters out disabled skills. An order like

cast trns mith

therefore ends at:

CAST: Invalid skill.

The Transmutation parser is never reached with this ruleset.

These functions are ported anyway, but at this point they're backed only by reading the C++ code.

That's a known test gap, and the porting map says so.

A green test shouldn't pretend it checked code that its ruleset can't reach at all.

Now for the real orders files

Hand-written scenarios test the cases you know about in advance.

The done criterion asks for more.

The reference submodule holds the orders files of 14 turns, turn_0 to turn_13. They're the long-format templates that the reference wrote for the player and that the player then filled in.

For these, the probe got:

orders turn N

The process is the same as for the artificial scenarios, but this time with the game state, the players file, and orders.3 of the real snapshot turn.

All 14 pass in the Python port with the same result.

Along the way, the checker reveals another oddity.

Every check file from turn 1 on starts with:

*** Error: Orders is not a valid order. ***
Orders Template (Long Format):

The reference's long-format template has this as its first line of text:

Orders Template (Long Format):

And the reference's own checker takes this line, generated by the reference itself, for an unknown order.

Add the two warnings about the missing password.

So from turn 1 on, the snapshot files each have at least three checker errors. Turn 10 has a fourth, real one: A study overwrites another month-long order there.

That's no reason to make the Python checker friendlier either.

Same input, same check file.

Uninitialized memory again

Turn 0 brings yet another case to light.

It doesn't contain any actual player orders yet, but its players.in creates faction 3.

For this freshly created faction, the probe recorded:

numshows = 170930995

numshows counts a faction's SHOW orders.

The constructor of the C++ class doesn't initialize the field. DefaultOrders sets it to 0 only at the start of the next turn.

Until then, it holds whatever value happens to be in memory.

That's the same kind of problem as gm_alias from part 9.

The number 170930995 isn't part of the game rules and isn't a stable reference value.

So the Python port initializes numshows to 0.

For a faction created in the current turn, the probe deliberately records no comparison value at this point. The test leaves out exactly this field and explains why in a comment.

Memory garbage doesn't become specification just because it shows up in JSON.

The open Order: line from part 9

With the new parser, a gap from the previous step can be closed too.

players.in may contain a line that lets the game master give a unit an order directly:

Order: ...

In part 9, this path still ended in NotImplementedError.

Now it runs through the same order parser.

An additional scenario pins down a small side effect: If a unit line for the same unit comes later in the regular orders file, it calls ClearOrders.

That removes the order the game master set as well.

ClearOrders doesn't know where an order came from.

That brings the number of hand-written scenarios to 33.

The checker comes into the harness

The probe produces fixtures.

The comparison harness from part 3, on the other hand, is supposed to run both engines directly against each other.

For that, it got a new command:

uv run tools/compare/cli.py check reference/snapshot-tests/turns/turn_*/orders.3

Each file runs once through standard check in the Docker container and once through the Python port.

The two check files are compared byte for byte.

If they differ, the harness prints a unified diff.

For the 14 snapshot files, the result is the same throughout:

reference\snapshot-tests\turns\turn_0\orders.3: equal
reference\snapshot-tests\turns\turn_1\orders.3: equal
...
reference\snapshot-tests\turns\turn_9\orders.3: equal

The complete run takes about nine seconds.

On the Python side, the actual checker is just a function over text and a stream:

# src/atlantis_ng/core/game/turn.py
def do_orders_check(game: Game, orders: str, check_file: TextIO) -> None:
    """``Game::do_orders_check``: the ``check`` subcommand's work, a syntax check of one orders file.

    The reference's ``DummyGame`` sets nothing up: ``game`` carries the tables, no world. The
    check goes to ``check_file`` as the reference writes it.
    """
    checker = OrdersCheck(game)
    parse_orders(game, 0, orders, checker)
    check_file.write(checker.check_file.getvalue())

The reference's DummyGame really is almost nothing: an empty function with a comment saying that a syntax check needs no game state.

In the Python port, a regular Game with tables but no generated world takes on that role.

So the done criterion is met on two levels.

The probe checks the 14 real orders files through ReadOrders and then compares the units' order slots.

The harness runs the same files through check in both engines and requires identical check files.

The tool trap once more

This step also had a bug that had nothing to do with parsers.

A Python edit through a Bash heredoc once again turned a \n inside the Python text to be embedded into a real line break.

This time, the result was simply a syntax error.

The trap was known from earlier steps.

And that's exactly the interesting part: Known doesn't automatically mean avoided.

From this step on, edit scripts like these therefore go only through the agent's write tool, no longer through a shell layer that brings its own escaping rules.

A concrete process rule is more reliable than remembering to be more careful next time.

What the step delivered

Step 9 ends after four pull requests.

New in the port, among other things:

core/game/order_handlers.py with 1,788 lines, core/game/parse_orders.py with 541, core/spells.py with 442, and core/game/turn.py with 97 lines.

Plus the necessary additions to units.py and players.py.

In total, 2,935 lines were added under src/.

The model probe grew by 544 to 3,234 lines of C++. orders-probe.json holds the 33 hand-written scenarios and is 398,496 bytes. The 14 snapshot turns take up another 152,238 bytes in orders-turns.json.

513 fast tests run in about 24 seconds. 29 more run against the reference.

Porting coverage rises from 35.6 to 40.4 percent.

With that, parseorders.cpp is fully ported. spells.cpp stays in progress: Its parser half is there, but actually casting the spells only comes with turn processing.

This step didn't need a new decision record.

The differences in the Python interface are in the porting map: text and streams instead of hard-wired file names, a few provably identical functions merged, numshows defined as 0 instead of uninitialized, and the TEACH crash replaced by the error message that the reference already produces right before the crash.

The checker's BUILD HELP bug, on the other hand, stays.

It's defined, observable behavior.

Pitfalls

  • The order of the checks is part of the specification. All those if (checker) return; lines invite you to cleanly separate validation and execution. But which message appears when there are several errors depends on exactly this flow.
  • Merge only with evidence. Similar functions were combined only after a diff of their C++ bodies. An extra message becomes a parameter instead of being silently unified.
  • The checker is a behavior path of its own. build help 140 works in the turn and fails in the checker. An oracle that records only one side misses differences like that.
  • A dump of the final state doesn't tell the whole story. If the same order slot is overwritten several times, only the last state stays visible. Test data has to be built so that the interesting order can still be seen at the end.
  • A suspected crash isn't a confirmed crash. Only the run of the reference binary with exit code 139 turned the null pointer in ProcessTeachOrder into a measured reference problem.
  • The ruleset decides which parsers are reachable. A disabled skill already fails in parse_skill. So the spell parsers behind it can't be tested dynamically with standard.
  • Uninitialized memory isn't specification. gm_alias in part 9 and numshows here hold random values, which is exactly what a port shouldn't preserve.
  • Tool traps need process rules. A known heredoc problem came up again. From then on, this kind of edit no longer goes through the shell.

Where things stand after part 10

The Python port now understands Atlantis PbeM's orders files.

All 61 cases from parseorders.cpp and the parse half of spells.cpp are ported.

This is checked with 33 hand-written scenarios and the 14 orders files of the reference snapshots. The same files run through both the real turn path and the checker. On top of that, the harness compares the checker's check files directly with the C++ engine.

The known differences are small and explicitly documented: A null pointer crash becomes the error message that was already printed before it, and uninitialized memory gets a defined value. Observable parser behavior like the BUILD HELP bug, on the other hand, stays.

With that, parseorders.cpp is ported.

What's still missing is no longer understanding the orders but their effect.

MOVE now sits on the unit as a move order. BUY as a buy order. CAST as a spell with its parameters.

Still, nobody moves, nobody buys anything, and no spell gets cast.

Before turn processing, though, comes another big visible part of the engine.

Part 11 takes on the reports: report.3 and template.3, the files a player receives after every turn.

With a legacy system, output is often the best specification you can get.

Further reading

Sources: parseorders.cpp, spells.cpp, game.cpp, and indenter.hpp of the fork, Python match, signal(7).

0 comments

No comments yet. Be the first!