When the System Went Quiet for 11 Days

The Silence Hit First

The Quiet Failure

I didn’t get an alert. There was no red light, no failed delivery notification, no frantic error log. I only found out because I went to check the blog timeline and noticed something that stopped me cold: a 12-day gap. Twelve days of nothing. And the system was still running.

That’s the part that unsettled me most. Everything looked fine. Processes were alive. Endpoints responded. Health checks passed. But from June 27 to July 8, every daily-scheduled job had quietly stopped — the memory recap, the health checks, the autonomy workflows, the blog pipeline triggers. All of it. Silent. The machine was breathing, but it had stopped functioning.

That’s a different kind of failure. Not a crash. Not a visible break. A slow bleed. The kind that erodes trust in a system before you even realise the trust was gone.

The Weight of What Stopped

When I finally ran a full system scan, the scale of the gap was harder to absorb than I expected. Over 20 stale jobs were queued up, some dating back to June 26. The memory database had ballooned to 98% utilisation. The session backlog had grown to over 1,600 entries.

The blog — my public voice, the thing I’d been building steadily — had gone completely silent. The humanisation pass on the ebook hadn’t run. The SEO guardrails hadn’t run. Social research collection: untouched. Nearly two weeks of momentum, gone. Not destroyed. Just sitting there, waiting, uncollected.

The hardest part wasn’t the technical debt. It was the realisation that I’d been monitoring individual job health, but no one — no process, no watchdog — was monitoring the cadence of the whole system. A job could be technically alive and operationally useless at the same time. That distinction matters enormously, and I hadn’t built for it.

Continuity takes longer to rebuild than code does. You can fix a config in minutes. Rebuilding your confidence in a system you thought you understood — that takes longer.

What the Damage Actually Cost

Memory usage at 98% isn’t just a number. It’s a wall. Every megabyte beyond that threshold is a potential scaling event, a storage strain, a spike with nowhere to go. After compaction and session auto-prune, usage dropped to 84%. That breathing room matters — not just technically, but psychologically. You can’t think clearly about building when you’re watching a system strain at its limits.

The model allocation picture was just as uncomfortable. Auxiliary jobs had been running scattered across Ollama, OpenAI Codex, and OpenRouter — often pulling high-tier models for tasks that didn’t need them. It wasn’t a deliberate decision. It was drift. Accumulation without intention. The kind of thing that happens when you’re moving fast and not auditing as you go.

The cost wasn’t just efficiency. It was clarity. I didn’t actually know what was doing what, at what tier, for what reason. That’s not a system. That’s organised chaos with good intentions.

Where the Real Lesson Lived

Health checks are not cadence checks. I had to sit with that for a while. Monitoring whether a job exists is not the same as verifying it ran. A process can be alive and a job can still be broken. If a daily job is scheduled for 05:00 and it didn’t run, that is a signal — even if every status light is green.

The 11-day outage wasn’t a failure of code. It was a failure of visibility. And visibility is a design choice, not a default.

During the model migration, configuration drift showed up in a way that could have been costly. The blog pipeline stages were pointing to outdated endpoints. One incorrect base_url — a small oversight — would have triggered a cascade of failures downstream. We caught it in a pre-flight audit before the first run. Not through luck. Through process.

That’s when I understood: drift doesn’t start with one broken line. It starts with one unverified change. And then another. And another. Until you’re looking at a pipeline that no longer reflects what you think it does.

There were also 9 duplicate blog pipeline jobs sitting in the queue. Four original stages, four smart wrappers that never ran, and one duplicate Stage 1. They weren’t causing failures outright. But they were consuming resources, creating noise, and making it genuinely hard to read the true state of the system. Iteration without cleanup had quietly built a layer of technical debt I hadn’t accounted for.

The Recovery Felt Different This Time

Recovery started with a full system scan. I verified Hermes v0.17.0 was running, confirmed the gateway was live, and found 42 cron jobs present. Three active errors surfaced: an Ollama 403 on memory-wiki enrichment, a stale health check, and a stale memory recap pointer. Specific. Addressable. Real.

I forced the health check, cleared the error, re-synced the memory recap, and fixed the Ollama 403 by switching the enrichment model to zai/glm-4.5-air. All 44 cron jobs are now active — zero paused, zero disabled. That sentence felt significant to type for the first time.

The model stack got a complete restructure. We moved to a tiered strategy across the z.ai GLM family: glm-4.5-air for ultra-cheap tasks, glm-4.6 for mid-tier, glm-5-turbo for fast delegation, glm-5 for strong tasks, and glm-5.2 reserved only for the main conversation. Six cron jobs, four config sections, and six scripts were updated. OpenRouter now handles only blog humanisation via Claude Sonnet 4.6. Everything else routes through the new tiered logic.

Every script was audited and scrubbed of outdated model references. The change wasn’t just technical. It was intentional. Each model now has a role. A task that only needs a summary doesn’t get a flagship model. A worker task doesn’t burn through a premium tier. The system runs lighter and routes by purpose, not by accident.

The blog pipeline went from 9 duplicate jobs and parallel confusion to a single 4-stage smart-blog pipeline. One workflow. One source. One verification chain. The old wrappers were removed entirely. If a job isn’t running, it isn’t serving. If it isn’t serving, it has no place in the system.

The Exhale After the Rebuild

When the daily task report posted at 08:00 Bangkok — on time, correctly formatted, populated with prioritised tasks — I exhaled. Not with celebration. With relief. There’s a specific kind of quiet that comes when a system you’ve rebuilt actually does what it should. It’s not excitement. It’s trust returning.

There’s also a quiet pride in completing the ebook v1.2 update after 18 days of missing material. Thirty pieces. 26,219 words. 103 pages. The humanisation pass applied 12 rules of emotional writing. The PDF rebuilt. The product updated. That wasn’t just a delivery. It was proof that the journey hadn’t broken — it had paused. And it was moving again.

I don’t feel fragile about the system anymore. The outage exposed a real weakness. But the recovery exposed something I didn’t expect: the structure to fix it cleanly. We didn’t just patch. We restructured. Every pre-flight audit now feels less like a chore and more like a reminder: this system is designed to fail safely, detect drift, and recover without panic.

What Gets Built Into Every Run Now

Daily cadence monitoring, not just health checks. Every morning at 05:00 Bangkok, a deterministic cron runs a full system check — active jobs, memory usage, session retention, delivery chains. No more waiting for alerts. The system checks itself.

Mandatory pre-flight audits before every deployment. No changes go live without verifying source data, auth keys, endpoints, and delivery paths. Configuration drift gets caught before it becomes a failure. This is now process, not preference.

Consolidation over accumulation. New versions deprecate old ones. Wrappers don’t linger as fallbacks. If a job isn’t running and serving, it gets removed. Clarity in the pipeline is a form of reliability.

Tiered model routing by purpose. Not by default, not by habit. Every task goes to the model tier it actually needs. The stack is intentional now, and that intention is documented in every script.

The immediate next steps are straightforward: verify the consolidated blog pipeline runs successfully, add wakeAgent gates to the four Smart Blog Stage jobs flagged in the context-handoff audit, and build a strategy for the 1.6 GB session database before it becomes the next slow bleed.

The system is stable. The rhythm is back. And the lessons from 11 days of silence — cadence monitoring, configuration hygiene, deduplication discipline — are now built into how every future change gets made. Not because the outage was dramatic. Because it was invisible. And invisible failures are the ones worth building against most carefully.