// WRITING
Long-form thinking on systems architecture, infrastructure engineering, and the decisions that shape complex platforms.
SOPS with age keys is the right answer for secrets in a GitOps repo, but two things will silently break you before you understand what's happening: path_regex is not a glob, and 'sops metadata not found' can mean at least three different things.
Every workaround your best engineers build is a signal. Not a solution. A signal that the system they're working in isn't meeting the need, and that the person who could fix it has instead found a way around it. Workarounds accumulate. They become load-bearing. Eventually the workaround is the system.
The cloud didn't make infrastructure simpler. It moved the complexity somewhere less visible and replaced some of it with operational surfaces you have to learn from scratch. The 'cloud is awesome' moment and the 'I have no idea how this actually works' moment are often the same capability viewed six months apart.
GitOps and continuous delivery are not the same thing. Most teams conflate them in ways that create real operational problems. GitOps is a deployment and reconciliation model. Continuous delivery is a software delivery practice. They compose well but they're solving different problems, and treating them as synonyms produces systems where neither works as well as it should.
We took an eight-week release cycle down to twelve minutes. The pipeline work was the easy part. What the acceleration revealed was that most of those eight weeks wasn't work. It was dwell time, and the processes that owned that dwell time had never had to justify themselves against a world that moved faster.
Tools offer idempotency as a good intention. Systems need it as a guarantee. Knowing when to step in and upgrade one to the other is what separates automation that works from automation that works until it doesn't.
When a team doesn't understand their tooling well enough to trust it, they build around it. The build-around becomes what everyone learns. Nobody ever learns the tooling. The distrust gets baked into onboarding. New engineers inherit the wrong mental model and build on top of it. The workaround defends itself.
The pipeline doesn't have 47 steps because 47 things need to happen. It has 47 steps because trust eroded over time and every erosion event got a new step added on top. The steps aren't doing work. They're doing anxiety.
The overnight evangelist is often the most motivated person in the room, responding to a real signal that something needs to change. The problem isn't that they found something. It's the leap from 'I got this working' to 'everyone must use this everywhere.' And the org's response is usually wrong in both directions.
Infrastructure work has a visibility problem baked into the nature of the work itself. When it's working nobody notices. When it fails everyone notices. That asymmetry shapes every prioritization conversation infrastructure teams ever have, and it doesn't fix itself with better communication.
Network automation is harder than server automation because the blast radius of a mistake is immediate and the feedback loop is brutal. That brutality is actually the advantage. It forces you to get the abstractions right in a way that server automation lets you defer.
Most architecture evaluations compare tools in isolation. The better question is: what's the right tool given the operational surface I'm already committed to? Adding a new tool has a real cost that almost never shows up in the analysis. Reusing what's already there has a real value that almost never gets counted.
You can delegate the work. You can use managed services. You can hire people who know the thing you don't. What you can't do is outsource the comprehension. When something breaks at 2am, the understanding either exists or it doesn't.
Good planning isn't about staying on schedule — it's about making better decisions in flight, taking on deliberate technical debt with clear eyes, and arriving at the right destination even when the route changes.
The thing that makes declarative automation powerful is exactly the thing that makes it dangerous. I wrote a user management task with handle_absent_entries: remove, defined a partial list, and RouterOS refused to execute because it would have deleted the last user with full access permissions. The safety net caught it. The lesson is about knowing where aggressive automation ends and self-inflicted disaster begins.
The 24-hour activation window is real. The support response time on a Friday night is real. The disk wipe if you miss the window is real. MikroTik is still the right choice. All of these things are true at the same time.
The destination was never in question. The uncertainty was entirely in the path: how Ansible, RouterOS, and a task sequence would negotiate the journey from current state to desired state on live hardware.
The on-site session was two and a half hours. Rack, cable, verify, leave. That wasn't luck. The bootstrap happened weeks earlier at a desk, not in the rack.
Infrastructure work has a done problem that software delivery mostly solved and we haven't caught up. 'It's working' is not done. 'Nobody is complaining' is not done. And the cost of not defining done is that you never actually finish anything. You just stop actively working on it.
A home lab isn't a static thing. It grows through distinct phases. Wave one is making something work. Wave two is making it more complicated. Wave three is adding rigor. Wave four is building a true datacenter corollary. Most people stop at wave two. Wave four is where the interesting work is.