// WRITING / TAGS / AUTOMATION
Automation is the discipline of moving operational state out of people's heads and into version control. The point isn't speed, though that's a nice side effect. The point is that automated systems can be reasoned about, reviewed, rolled back, and re-run by someone who wasn't there the first time.
These posts cover the concrete tooling (Ansible, GitOps, MAAS, the Kubernetes operator pattern, CI/CD pipelines) and the philosophy underneath. When to automate, when not to, and how to recognize the line between "this is repetitive" and "this is dangerous."
If you've ever inherited a "fully automated" platform that turned out to have one critical step in someone's terminal history, these posts will resonate.
SOPS with age keys is the right answer for secrets in a GitOps repo, but two things will silently break you before you understand what's happening: path_regex is not a glob, and 'sops metadata not found' can mean at least three different things.
Every workaround your best engineers build is a signal. Not a solution. A signal that the system they're working in isn't meeting the need, and that the person who could fix it has instead found a way around it. Workarounds accumulate. They become load-bearing. Eventually the workaround is the system.
GitOps and continuous delivery are not the same thing. Most teams conflate them in ways that create real operational problems. GitOps is a deployment and reconciliation model. Continuous delivery is a software delivery practice. They compose well but they're solving different problems, and treating them as synonyms produces systems where neither works as well as it should.
We took an eight-week release cycle down to twelve minutes. The pipeline work was the easy part. What the acceleration revealed was that most of those eight weeks wasn't work. It was dwell time, and the processes that owned that dwell time had never had to justify themselves against a world that moved faster.
Tools offer idempotency as a good intention. Systems need it as a guarantee. Knowing when to step in and upgrade one to the other is what separates automation that works from automation that works until it doesn't.
When a team doesn't understand their tooling well enough to trust it, they build around it. The build-around becomes what everyone learns. Nobody ever learns the tooling. The distrust gets baked into onboarding. New engineers inherit the wrong mental model and build on top of it. The workaround defends itself.
The pipeline doesn't have 47 steps because 47 things need to happen. It has 47 steps because trust eroded over time and every erosion event got a new step added on top. The steps aren't doing work. They're doing anxiety.
Network automation is harder than server automation because the blast radius of a mistake is immediate and the feedback loop is brutal. That brutality is actually the advantage. It forces you to get the abstractions right in a way that server automation lets you defer.
The thing that makes declarative automation powerful is exactly the thing that makes it dangerous. I wrote a user management task with handle_absent_entries: remove, defined a partial list, and RouterOS refused to execute because it would have deleted the last user with full access permissions. The safety net caught it. The lesson is about knowing where aggressive automation ends and self-inflicted disaster begins.
The destination was never in question. The uncertainty was entirely in the path: how Ansible, RouterOS, and a task sequence would negotiate the journey from current state to desired state on live hardware.
The on-site session was two and a half hours. Rack, cable, verify, leave. That wasn't luck. The bootstrap happened weeks earlier at a desk, not in the rack.
Infrastructure work has a done problem that software delivery mostly solved and we haven't caught up. 'It's working' is not done. 'Nobody is complaining' is not done. And the cost of not defining done is that you never actually finish anything. You just stop actively working on it.
A home lab isn't a static thing. It grows through distinct phases. Wave one is making something work. Wave two is making it more complicated. Wave three is adding rigor. Wave four is building a true datacenter corollary. Most people stop at wave two. Wave four is where the interesting work is.
// RELATED TOPICS