Local to Remote: The Pipeline Migration That Broke Everything

The overnight pipeline worked. Then we tried to make it truly autonomous — and watched months of rock-solid behavior come apart in a single week.

April 18, 2026 · Vibe Arcade

The overnight pipeline was working. Three to five new browser games shipped per week, gates passing, auto-merges clean, mornings spent reviewing rather than repairing. On paper there was nothing to fix. There was one quiet compromise though, and it was the kind that nags: the pipeline only actually ran when the computer was awake and the Claude app was open. Close the lid or quit the app and the "overnight" pipeline was just a skill definition sitting on disk.

So we tried to fix that. This is the story of how fixing it broke almost everything else — and what the rebuild taught us about adopting new infrastructure.

The Working Baseline

The local pipeline was structurally simple. A single long-lived Claude Code session drove the whole run. A planning model produced a detailed game spec. An implementation model built the game from that spec over one to four hours, spawning sub-agents for parallel modules when needed. Three automated gates — structural lint, security checks, playability rubric — ran inline. If everything passed above threshold, the build auto-merged. If anything flagged, it waited for the morning review.

The reason it was robust is almost embarrassing in hindsight: everything lived in one process, in memory, for the duration of the run. Planning output handed off to implementation as in-memory context, not as a serialized artifact. Gate failures could be retried in the same session with the prior state intact. The "pipeline" was really a single continuous conversation with a lot of structure. The high-affordance environment — always-on machine, no wall-clock limits, persistent state — absorbed a lot of design slack we didn't notice we were leaning on.

The only problem was that it wasn't truly autonomous. It required a machine awake and an app open. For a workflow called "overnight," that was a compromise worth eliminating.

The Attempted Migration

The obvious fix was to move the trigger off the local machine entirely. Let a scheduled job in a cloud runner kick the whole pipeline at 2 AM. The computer can sleep. The app can be closed. The work happens somewhere else, on someone else's metal, and we just walk downstairs in the morning to review pull requests.

The category of approach was straightforward: a cron-triggered, cloud-hosted runner invoking the same skill the local session had been running. A single entry point that, once fired, would plan, build, gate, and open a pull request — exactly like the local flow, just on a remote host. The new tech was available. Credentials were wired. The first scheduled run kicked off as designed.

That was the last thing that went as designed for a while.

The Compute-Time Wall

A real overnight run is hours of compute. Planning is short, but the implementation phase — especially when it forks sub-agents for parallel modules — routinely runs past what most remote runners will tolerate before they kill the job. Security gates and the playability rubric each add time of their own. What had been a lazy several-hour local session was now being asked to finish inside a constrained wall-clock budget on a remote host.

Worse, the runner's failure mode isn't graceful. It doesn't politely hand back a partial result. It terminates. Whatever was in memory is gone. The branch is half-populated. No one sent a signal. The next scheduled trigger has no idea there was even a previous run to resume.

The Regression

Everything that had worked flawlessly for months came apart at once. Not a gradual slowdown — a categorical break.

Builds timed out mid-implementation, leaving branches with scaffolding but no real game. State that had lived comfortably in memory — the plan, the partial implementation, the in-flight decisions — vanished on every restart because there was no disk representation of it. Retries didn't "resume"; they re-ran from scratch, burning the budget again and hitting the same wall at roughly the same point.

The auto-merge path, which had been the proudest piece of the local system, became hazardous. It had been designed for a world where every gate ran to completion before the merge decision. In the remote world, a runner could die between "gate passed" and "merge executed," and the next trigger had no clean way to tell whether a prior run had actually finished. Partial artifacts produced cascading pull request problems — orphaned branches, stale locks, duplicate concept claims.

The confidence shock was real. A human who had stopped babysitting mornings because the pipeline was boringly reliable was suddenly back to triaging a different class of failure every day. The CLAUDE.md note that now reads "this is by design, not a bug to fix" about the cron workflow's no-op behavior wasn't a casual comment. It was a hard-won truth that lived through several failed rebuild attempts.

The bitter piece: nothing was actually wrong with the pipeline's logic. The game-building worked. The gates worked. The authoring rules worked. What broke was the assumption that an execution model built for one long continuous session would survive being chopped into pieces.

The Rearchitecture

The fix was not a patch. It was a different shape.

The first move was to stop pretending the pipeline could finish in one fire. We rebuilt it as a state machine with one phase per trigger. Each phase — plan, implement, gate, merge — reads a serialized state file from the repo, does exactly one chunk of work, commits updated state back, and terminates well inside the remote runner's time window. The next scheduled trigger reads that state and picks up where the last phase left off. No in-memory handoff. No "long session." Every bit of progress is disk-visible and git-visible.

That first state-machine pass worked, but it was slow — one phase per scheduled fire meant a full game could take many scheduled windows to finish. We iterated: instead of one phase per fire, the runner does as much progress as it safely can per fire, checkpointing state after each phase so a mid-run termination is recoverable rather than destructive. Early planning fires now do planning plus the scaffold of implementation. Later fires do implementation plus gates. The system targets maximum progress per fire while never risking an unrecoverable cut.

We added an orchestrator — a small component whose only job is to look at the committed state and decide which phase should run next. And we wrapped every phase in a watchdog timeout that kills the phase cleanly before the runner itself does, so the next trigger inherits a sane state rather than a half-dead process tree. A later iteration chunked large file writes specifically to dodge a silent idle-timeout behavior we only discovered after shipping to the new environment.

The progression, commit by commit, was: back up the known-good system; introduce multi-pass design with orchestrator and watchdog timeouts; cut over to a Tier 3 state machine of strictly one-phase-per-trigger; then relax that to maximum-progress-per-trigger now that checkpointing was reliable; then patch the write-chunking to close the last idle-timeout edge case. Each step was a response to a specific failure mode the previous design hadn't modeled.

What It Cost

Days of debugging across multiple evenings. Weeks of on-and-off regression hunts as new failure modes surfaced that hadn't existed in the local model. A real dent in confidence: code that had been rock-solid for months was suddenly producing confusing branches and partial merges, and the root cause wasn't in the code the pipeline had been running — it was in the execution model around it.

The general lesson, stated plainly: adopting a new technology that solves one of your problems almost always trades regressions somewhere else. "Always-on" is not a tweak you bolt onto a system designed for an always-available machine. It's a rebuild. And if you don't treat it as a rebuild, the rebuild happens anyway, just in a messier shape, under pressure, while production is broken.

What I'd Tell Someone Doing This Today

Three things, in order of how much pain each one would have saved.

Back up the known-good state before you rewrite. When the rebuild is still working-in-theory, snapshot the version that was shipping real games last night. A fresh directory, a clear name, committed. The snapshot isn't paranoia; it's the thing that lets you cut your losses on a rebuild without losing the system that was paying the bills. A skills-backup-style directory at the repo root was the difference between "course-correct" and "start over."

Test on the constrained environment early, not at the end. The mistake was validating the pipeline end-to-end in the high-affordance local environment and then trying to port it. The local environment hid every assumption that only breaks under wall-clock pressure and stateless retries. If the remote environment is the production target, make it the first-class test environment from day one, even if it's slower and harder to debug.

Build the state machine first, not second. If your pipeline has more than a few minutes of wall-clock work in it, assume from the start that it will run in a constrained environment eventually. Model state on disk, not in memory. Make every phase resumable. You'll pay a small design tax up front and save a multi-week rebuild later. Also: humans still review every pull request in the morning. That part didn't change in the migration, and it's the part you never want to automate away.

New technology is seductive because it promises to solve a real problem. It usually does. The question is which problems it quietly introduces in exchange.