Week 11: Final Transmission

Eleven weeks of work compressed into one pass. Visit mazzola.dev for nicer formatting

01 — Where We Started

Eleven weeks ago this blog opened with the cost of mass to orbit ($7K per kg to LEO, $1.2M per kg to the Moon’s surface) and a question of whether a neural network could learn to fly a deep-space transfer well enough to matter. The setup was PPO against a high-fidelity Basilisk simulator with the LEO-to-MEO Hohmann transfer as the warm-up, and the standing goal was Earth-to-Mars under realistic deployment noise.

The Mars problem asks for three things in series. A 3.5 km/s departure burn aligned with Earth’s orbital velocity has to put the spacecraft on a Lambert arc, a 240-day cruise has to hold that arc through dynamics-impulse noise and the occasional missed thrust, and a final ~430 m/s brake at Mars periapsis has to land inside a Gaussian-kernel capture region around a target B-plane.

02 — What Got Built

The final system is four layers. At the bottom sits a 6-DOF Basilisk core with SPICE DE430 ephemerides, n-body Sun-Earth-Mars dynamics, solar radiation pressure, and gravitational harmonics. Above that lives a custom Gymnasium environment with a three-phase finite-state machine (Pre-TMI / Cruise / MOI), a 4-D inertial-thrust action space, and a B-plane targeting reward. The PPO capture specialist is a [256, 256] MLP trained with Stable-Baselines3, warm-started from a behaviour-cloning baseline on an oracle dataset and refined through a reverse curriculum against cached hand-off states. A hierarchical orchestrator sits on top, dispatching an energy-tracking Lambert oracle for Phase 0 and most of Phase 1, then handing the spacecraft off to the PPO inside Mars’s sphere of influence.

The decomposition into oracle plus learned controller carries the project’s central design call. Phase 0 is well-served by classical methods (Lambert solvers have been in the trajectory-optimisation literature since the 1960s). Phase 2 is where the learned policy actually contributes, because the brake has to commit under uncertain incoming state, finite fuel, and a non-trivial reward landscape that punishes both under-braking (escape) and over-braking (Mars impact). The orchestrator routes each phase to whichever solver is better suited.

03 — The Path There

Most of the project’s training compute was spent inside a flat plateau. nasa_v1 collapsed to evaluation reward near -50,000 across all 10M of its training steps, traced to three independent reward-shaping bugs in the env (unbounded reference PBRS, an init-time delta on the B-plane term, and an Earth-impact penalty that triggered on natural LEO perigee drift). nasa_v2 fixed those, and then plateaued at +75 reward for another 10M steps with zero captures. A 4-D log_std runaway turned the deterministic mean into noise; a curriculum threshold pinned at 60% capture meant the agent never advanced past Stage 1. v3 through v22 cycled through alignment bonuses, distance shaping, fuel weights, and BC dataset composition with all the obvious variants tried. None of them captured.

Schematic of the project’s training arc across four representative runs. Roughly 60M cumulative PPO steps of plateau preceded the first capture at v24. Each subsequent breakthrough was a recipe change rather than a scale-up.

The first capture (v24) came after three structural changes that arrived together. scripts/precompute_soi_entry.py cached the oracle’s heliocentric state at Mars SOI edge so the env could spawn the spacecraft mid-flight; a Phase-2 reward overhaul added per-step alignment, Mars-relative energy, and a dense capture-quality term gated on e < 1; and a hard log_std clamp of [-2.5, -0.5] forced the deterministic mean to commit a brake instead of hiding capture probability in stochastic outliers. v24 produced 5/5 deterministic captures on launch window 0 at the SOI-edge spawn.

v32 generalised that to 20/20 captures across four cruise spawn points {T = 0, T – 7d, T – 30d, T – 40d} from a single network, by oversampling the failing offsets with {1, 3, 1, 3} weights to break a multi-modal plateau. Five attempts to extend v32 further (reverse curriculum past T – 40d, multi-window from v32, window-1 specialist, multi-window from a fresh BC) failed for the same reason in different shapes: the v32 anchor was too crystallised to absorb new spawn distributions without losing the original ones. v34f sidestepped the anchor entirely by initialising from a behaviour-cloning baseline (bc_policy_v5_p2heavy.pt) with a flatter loss landscape, and reached 96% capture under realistic perturbations.

04 — The Headline

The hierarchical controller running v34f under Mars-realistic perturbations is the project’s terminal result. Across 50 perturbed Monte Carlo runs on launch window 0, the system delivers a bound Mars orbit on 48 of them. The two failures both correspond to large upstream cruise drift that the fallback state-quality gate intercepted, sending the episode back to the oracle and producing a partial (heliocentric, no impact) result instead of a capture.

Capture quality across 50 perturbed Monte Carlo runs per controller. Periapsis altitude on the horizontal axis, capture eccentricity on the vertical. The TOAST cluster ends up lower-left of the oracle, with tighter and more circular orbits.

The TOAST cluster also beats a Lambert-only baseline on the orbital quality of its captures. The mean eccentricity drops from 0.86 to 0.59 and the mean periapsis altitude drops from 102,000 km to 46,000 km. A bound orbit at e = 0.6 / r_p = 46,000 km asks for considerably less circularisation dv than e = 0.86 / r_p = 102,000 km, so the learned policy’s contribution shows up as both more frequent captures and lower fuel cost for the orbit it leaves the spacecraft in. The PPO uses the extra information in its observation (Mars-relative state, B-plane error, time-to-go, mass fraction) to commit a sharper brake at periapsis than the oracle’s open-loop Lambert solution.

05 — Where It Sits

Roberto Federici’s 2020 dissertation Deep Learning-Based Spacecraft Optimal Guidance is the closest prior art for this style of work. The Federici thesis trains a small policy network against simplified two-body dynamics with deliberately strong noise channels (5% thrust magnitude, 1° pointing, ~1 m/s impulse per step, 2% missed-thrust probability), and demonstrates a robust brake under those conditions. The five-channel perturbation framework in this project is a direct port of the Federici noise model, and the v34f recipe inherits the spirit of training against the noise the deployment will face.

Schematic regime map for the three approaches. Classical open-loop trajectory optimisation lives at low perturbation and easier dynamics; Federici 2020 explores extreme noise but on a toy 2-body simulator; this project pushes the problem into flight-grade Basilisk physics under mission-realistic perturbation magnitudes.

The substantive difference is the physics layer underneath. TOAST runs on 6-DOF Basilisk with full SPICE DE430 ephemerides, n-body perturbations from the Sun, gravitational harmonics, and solar radiation pressure, while the Federici simulator is a two-body integrator with parameterised noise. The Federici noise levels are also two orders of magnitude stronger than what a modern flight system actually sees in cruise; a 5% magnitude error on the 3.5 km/s TMI burn translates to about 175 m/s of Δv uncertainty, which compounds to 1.2 Mkm of SOI miss by the time the spacecraft reaches Mars, and no recoverable trajectory exists inside a typical Phase-1 Δv budget. Scaling the noise back to mission-real magnitudes (0.5% magnitude, 0.3° pointing, 0.05 m/s impulses, 0.5% missed-thrust) puts the project inside the regime that an actual planning team would care about.

What This Adds: Two things this project contributes on top of prior art: a flight-grade physics stack for an RL-driven Mars capture (Basilisk + SPICE + n-body + harmonics + SRP, all running in the training loop), and a hierarchical decomposition that lets a learned policy improve on a Lambert baseline in capture-orbit quality without having to relearn the well-understood Phase 0 from scratch. Neither piece is novel on its own; their combination on a single end-to-end trained controller is, as far as I can tell, new.

06 — Carry-Outs

The single-window v34f result is a proof of concept rather than a deployed-grade policy. Three obvious extensions follow from the infrastructure that’s already in place:

# Path 1: All four launch windows
python scripts/precompute_soi_entry.py --window 1   # ditto for 2, 3
python main_train.py --env mars_nasa --bc-init bc_checkpoints/bc_policy_v5_p2heavy.pt \
  --epoch-weights "1,1,1,1"  --cruise-offset-random-choices-s "..." \
  --spawn-pos-noise-km 30000  --spawn-vel-noise-ms 20  --spawn-mass-kg-override 252 \
  --lr 1e-4 --ent-coef 0.002 --log-std-init -2.0  --timesteps 2_000_000

# Path 2: Variable parking orbit
# Regenerate the SOI-edge cache per parking-radius bin, train an ensemble,
# or add parking-radius as an explicit observation component.

# Path 3: Bigger network + window-id conditioning
# Replace [256, 256] with [512, 512] and feed window-id as a one-hot.
# Lets one network condition on which window it's running.

Path 1 is the natural follow-on for next semester. The recipe that worked on window 0 should transfer with minimal recipe changes once the cache is regenerated per window; the failure modes from week 9’s multi-window attempts came from fine-tuning the v32 anchor rather than from the BC-from-scratch recipe v34f uses. Path 2 trades flexibility for cache cost and is probably needed before any operational deployment. Path 3 is the most speculative; the empirical evidence so far is that the [256, 256] MLP fits a single-window manifold cleanly, but whether it generalises across four launch windows is open.

The deeper open question is whether the hierarchical decomposition itself is necessary. End-to-end PPO from a fresh BC baseline never matched v24’s anchor across 60M cumulative steps; the cause was reward-surface curvature rather than network capacity. A reward function that explicitly credits long-cruise stationarity (instead of telescoping all PBRS terms to zero across coasting steps) might let a single policy do Phase 0 + Phase 1 + Phase 2 without the Lambert oracle. That route would replace the orchestrator with a single network and would let the agent discover non-Lambert TMI strategies (low-thrust spirals, multi-revolution transfers, gravity-assist routes), but the reward design alone is a research project’s worth of work.

Beyond Mars, the same stack should extend straight to any high-thrust capture problem: lunar orbit insertion, near-Earth asteroid rendezvous, Jovian moon capture. The only Mars-specific code is the launch-window catalogue and the SPICE body ID; the rest of the pipeline is parameterised on body mass, SOI radius, and reference epoch.

07 — Sign-Off

A learned neural-network policy can fly a spacecraft from low Earth orbit to a bound Mars orbit through 240 days of perturbed cruise, 96% of the time, on a single launch window. The orbital state at capture is tighter than what the classical Lambert-based open-loop baseline produces on the same problem. The training recipe runs on a desktop with a single GPU in under a week of wall-clock time. That answers the question this blog opened with.

Most of those eleven weeks were spent on failures. nasa_v1 through nasa_v22 plateaued or collapsed across roughly 220M cumulative training steps. v33 through v37 produced four different ways to break v32. v34c through v34e produced three more ways to break v33. The single breakthroughs (v24, v32, v34f) each took about one week of focused recipe iteration on top of accumulated debugging. None of them were predicted by classical RL theory; they came from staring at training curves, decomposing reward into per-source contributions, writing diagnostic scripts that compared deterministic mean against rollout sample, and refusing to trust any single number.

Thanks for reading along. End of transmission.