# X4 Advisor (v1: economy & stations)

Generates a markdown report of what's going on economically in your X4:
Foundations game and a prioritized list of "what to do next" suggestions.
Two ways to get data in, both feeding the same rules engine and report:

- **Save-file mode** (`python -m x4_advisor`) -- reads your last save.
  No mods needed, works today, calibrated against a real save.
- **Live mode** (`python -m x4_advisor.live`) -- queries the running game
  directly over a named pipe, using SirNukes' Mod Support APIs. Needs a
  small companion extension (included) and is now confirmed working
  end-to-end against a real game (credits, ships, and stations all
  cross-checked against the in-game HUD/Ventures screen) -- see "Live
  mode" below.

There's no official Egosoft API for either path, so field/property names
were reconstructed from community documentation, open-source parsers,
and (for save-file mode) a real calibration save -- not guaranteed to
match every save/game version. Both modes are built to be honest about
that: they say exactly what they couldn't find rather than guessing
silently.

## Requirements

Just Python 3.9 or newer -- no external packages, pure standard library.

If you don't already have Python: get it from
[python.org/downloads](https://www.python.org/downloads/), and during
install, tick **"Add python.exe to PATH"**.

## Quick start (Windows)

1. Unzip this folder anywhere, e.g. `Documents\x4-advisor`.
2. Open a Command Prompt in that folder (type `cmd` in the folder's
   address bar in File Explorer and hit Enter).
3. Run:

   ```
   python -m x4_advisor
   ```

   With no arguments it auto-detects the newest save under
   `Documents\Egosoft\X4\<install-id>\save\`. If it can't find it (e.g.
   your saves live somewhere else), point at it directly:

   ```
   python -m x4_advisor --save "C:\Users\you\Documents\Egosoft\X4\12345678\save\quicksave.xml.gz"
   ```

4. Open the generated `x4_advisor_report.md` in Notepad, VS Code, or any
   markdown viewer.

Double-clicking `run_report.bat` does steps 3-4 for the auto-detected
save.

This part is already calibrated against a real save and its numbers have
been checked for plausibility (see "Testing so far" below) -- start here
even if you want live mode eventually, since it needs no in-game setup.

## Calibrating against your real save (do this first)

Before trusting the suggestions, run:

```
python -m x4_advisor --save "<path to your save>" --dump-schema
```

This prints every XML tag it saw, how often, what attributes each one
had, and every `class="..."` value used on `<component>` elements --
instead of generating a report. Compare that against the guesses in
`x4_advisor/summarize.py`:

- `STATION_CLASSES`, `SHIP_CLASSES` -- the `class` values components use
  for stations vs. ships.
- `PLAYER_OWNER_VALUES` -- what the owner attribute looks like for
  something you own.
- The candidate attribute names tried in each `_extract_*` function
  (e.g. money is looked for under `money`, `credits`, or `account`).

If something looks off (e.g. "Stations owned: 0" when you know you have
some), paste the `--dump-schema` output back to me and I'll fix the
field names directly -- that's a five-minute change once we can see the
real schema, versus guessing blind.

## What v1 covers

Economy & stations only: cash on hand, stations you own (and which ones
show zero tracked wares, worth a look), ship counts, blueprints owned,
sectors seen, and faction standings (captured for context, not yet used
in suggestions). Heuristic thresholds live in `x4_advisor/rules.py`
under `TUNABLES` if you want to tighten/loosen them.

Deliberately out of scope for v1, but the code is structured so each is
a new `rules_*.py` module following the same `Suggestion` shape:

- Fleet & combat (fleet size/composition vs. sector threats, defense
  gaps).
- Exploration & factions (unexplored sectors, reputation, licenses).
- Research & blueprint acquisition priorities.

## Live mode (via SirNukes' Mod Support APIs)

If you have **SirNukes Mod Support APIs** installed, you can query the
running game directly instead of waiting for the next autosave. This
needs more one-time setup than save-file mode, but it's now confirmed
working end-to-end: credits, ship counts by class, and station count
have all been cross-checked against a real game's HUD/Ventures screen
and matched exactly.

### One-time setup

1. **Install the bridge extension.** Copy the
   `extensions/x4_advisor_bridge` folder from this zip into your X4
   extensions folder (typically
   `Documents\Egosoft\X4\<install-id>\extensions\`, or your Steam/GOG X4
   install's `extensions` folder). Enable it in X4's Extensions menu.
2. **Turn off Protected UI Mode.** X4's Extensions menu -> Extension
   Options -> Protected UI Mode -> off. The Named Pipes API refuses to
   connect while this is on.
3. **Install `pywin32`:** `pip install pywin32`. (`X4_Python_Pipe_Server`
   itself -- the library that actually talks to the named pipe -- ships
   vendored inside `x4_advisor/live/vendor/`, so there's no separate
   download or hunting through your X4 install for it. `--pipe-lib-path`
   still exists if you ever want to point at a different copy instead.)
4. **Restart X4** so the extension and the Protected UI Mode change both
   take effect.

### Running it

```
python -m x4_advisor.live --ping-only
```

Do this **first**, with a save loaded and playing (not on a menu). It
should print `Got: 'PONG x4_advisor_bridge 1'`. If it hangs or errors,
see Troubleshooting below before trying a full snapshot.

Once PING works, drop `--ping-only` for the real thing:

```
python -m x4_advisor.live
```

This writes `x4_advisor_live_report.md` in the same format as save-file
mode. `run_live_report.bat` does both steps and opens the result.

### What live mode v1 collects

Credits, ship counts by class, and station *count* only -- no station
names, sectors, ware inventory, blueprints, or faction relations yet
(the report says "not collected in this mode" for those rather than
showing a misleading zero). That's enough for the core "idle cash / no
stations / low cash" suggestions; per-station detail (names, sectors,
ware inventory) is a planned follow-up to
`extensions/x4_advisor_bridge/md/Advisor_Bridge.xml` now that the basic
round-trip is confirmed working end-to-end.

### Troubleshooting

- **`Required extension 'ws_2042901274' is missing` warning on the
  bridge extension:** usually means SirNukes' Mod Support APIs itself
  isn't enabled (it's a separate toggle from the bridge extension, and
  from Protected UI Mode) -- check it shows enabled with no warning of
  its own in the Extensions menu, then fully restart X4.
- **PING hangs until timeout:** X4 likely hasn't connected to the pipe.
  Confirm the extension is enabled, Protected UI Mode is off, and you
  restarted X4 after changing either.
- **PING itself hangs on the Python side with no error at all, even
  past its timeout:** this was a real bug in earlier versions of this
  tool (fixed) -- `pipe_client.py` used to run a persistent background
  thread continuously blocked in `Read()` while the main thread tried to
  `Write()` on the same pipe handle, which the underlying
  X4_Python_Pipe_Server library's own source explicitly flags as
  unsafe (plain synchronous I/O, not overlapped). If you're on an old
  copy of this tool and hit this, update `pipe_client.py`. Antivirus
  software (Malwarebytes' Ransomware/Exploit Protection, etc.) can
  *look* similar but turned out not to be the actual cause when this
  was debugged -- worth ruling out with `tests/pipe_loopback_test.py`
  (a pure-Python pipe round trip, no X4 involved) before assuming it's
  security software.
- **PING works but SNAPSHOT returns nothing / errors:** check X4's
  debug log (not the in-game Logbook -- MD `debug_text()` output doesn't
  go there). Launch X4 via Steam with launch options
  `-debug all -logfile debug.log`, then look in
  `Documents\Egosoft\X4\<install-id>\debug.log` (same folder as your
  saves) for lines mentioning `AdvisorBridge` or `Send_Snapshot`. Two
  real bugs were found and fixed this way already (see the
  calibration-status comment at the top of `Advisor_Bridge.xml` for
  details): an unqualified `<include_actions ref="Send_Snapshot"/>`
  that needed to be `md.AdvisorBridge.Send_Snapshot` (cue lookups from
  inside `Server_Reader`'s callbacks resolve against SirNukes' own file,
  not this one), and `find_object` with a per-class `class="ship_s"`
  filter silently not filtering at all -- fixed by switching to the
  dedicated `find_ship_by_true_owner`/`find_station_by_true_owner`
  instructions instead. If you hit a *new* error, the debug.log's exact
  wording is usually enough to fix the specific expression in one pass.
- **`Could not import X4_Python_Pipe_Server`:** on Windows with
  `pywin32` installed this shouldn't happen (a copy ships in
  `x4_advisor/live/vendor/`) -- if it still does, run `pip install
  pywin32` again inside whatever Python environment you're using to run
  this, or pass a known-good copy's folder via `--pipe-lib-path`.

## How it works

- `x4_advisor/locate.py` -- finds save files under the default X4
  Documents folder, or accepts an explicit path.
- `x4_advisor/xmlstream.py` -- streams the gzipped XML with
  `ElementTree.iterparse`, freeing each element (and detaching it from
  its parent) as soon as it's processed, so memory stays roughly
  bounded by nesting depth rather than the whole save (saves can exceed
  1GB uncompressed). Captures every attribute seen, not just guessed
  ones, so nothing is silently lost if a guess is wrong.
- `x4_advisor/summarize.py` -- turns the raw extraction into an
  `EconomySummary`, trying a short list of candidate attribute names per
  fact and recording what it couldn't find. Also used (via
  `x4_advisor/live/protocol.py`'s adapter) as the common output shape for
  live mode, so `rules.py`/`report.py` don't care which source fed them.
- `x4_advisor/rules.py` -- heuristics over the summary, producing a
  prioritized `Suggestion` list.
- `x4_advisor/report.py` -- renders it all as markdown.
- `x4_advisor/cli.py` -- the `python -m x4_advisor` (save-file) entry point.
- `x4_advisor/live/pipe_client.py` -- wraps `X4_Python_Pipe_Server` with a
  real read timeout (via a background reader thread + queue, since the
  underlying library's `Read()` blocks with no timeout of its own) and
  friendly errors.
- `x4_advisor/live/protocol.py` -- parses the bridge extension's line-based
  responses and adapts them into an `EconomySummary`.
- `x4_advisor/live/cli.py` -- the `python -m x4_advisor.live` entry point.
- `x4_advisor/live/vendor/X4_Python_Pipe_Server/` -- vendored (unmodified,
  MIT-licensed) copy of the library that does the actual Windows named
  pipe work, from [bvbohnen/x4-projects](https://github.com/bvbohnen/x4-projects)
  -- see the NOTICE.md in that folder for exact provenance.
- `extensions/x4_advisor_bridge/` -- the X4 extension (MD script) that
  answers PING/SNAPSHOT over the pipe. See the calibration-status comment
  at the top of its MD script for what's confirmed vs. still a guess.

## Testing so far

**Save-file mode** is calibrated against a real save (`save_005.xml.gz`,
~485k components) -- money, ship/station ownership, and faction
relations were cross-checked and two real bugs found and fixed in the
process (a duplicate `<player>` tag colliding with the real one, and
faction relations picking the wrong row). `tests/make_sample_save.py`
also builds two synthetic saves (a developed economy and a broke early
game) matching the confirmed structure, run end-to-end on every change.

**Live mode**'s Python side (`tests/test_live_protocol.py`, 18 checks) is
unit-tested against synthetic pipe messages -- parsing, the `/100` money
scaling, the "not collected" vs. "genuinely zero" distinction, and
missing-END/garbage-line handling. The MD script and full pipe round
trip have also now been run against a real X4 v9.0 game and its
resulting report cross-checked field-by-field against the in-game HUD
and Ventures screen (credits, ship counts by class, station count) --
all matched exactly. `tests/pipe_loopback_test.py` is a separate manual
diagnostic (pure Python, no X4 involved) for isolating pipe-level
connection issues from X4/extension-level ones if live mode ever hangs
again. Run any of these directly:

```
python tests/make_sample_save.py && python -m x4_advisor --save tests/fixtures/sample_save.xml.gz
python tests/test_live_protocol.py
```