Pinned Image Digests Turn a Container Tag into a Reproducible Dependency
A mutable tag tells you what to ask for; a digest tells you what bytes you actually accepted.
TOPIC
Production Engineering
Lessons, explainers, experiments, and implementation notes.
Back Up and Define Rollback Before You Touch Production
The safest time to decide how to recover is before the change has modified the evidence you depend on.
The Best Postmortem Output Is a New Invariant
Fixing one incident is useful; changing the system so the same class of failure becomes detectable or impossible is more valuable.
Production Acceptance Requires Live Evidence, Not Code Review Alone
A reviewed repository can still be disconnected from the state actually running on the host.
Reconcile: Returning Again and Again to the Gap Between Intent and Reality
We can declare a desired state, but real systems rarely arrive there instantly. Failure, delay, and change exist along the path. Reconciliation is interesting because it checks the relationship repeatedly instead of trusting a one-time command.
Remote Terminal States Should Stop Re-Publishing Logic
If the provider already reports scheduled, publishing or published, the reconciler should observe rather than repeat the side effect.
Git, Runtime State and Secrets Need Different Jobs
Operational confusion falls when desired configuration, mutable data and confidential values stop competing to be one source of truth.
Content Is Not the Same Thing as Delivery State
The authoritative article text can remain stable while each destination has its own mutable publishing lifecycle.
Reconciling Concurrent Git Changes Without Force Push
Production work continued in multiple streams, so safe synchronization had to preserve both histories rather than overwrite whichever side moved first.
Publisher Retries Need Idempotency or They Create Duplicate Posts
When a network response is lost, retrying a create request can duplicate the side effect even if the first request succeeded remotely.
Build, Release and Run Fail Differently
Separating image construction, configuration injection and runtime startup makes failures easier to localize.
Running Is Not the Same as Healthy
Container process state only proves that PID 1 exists; readiness has to test the behavior the dependency actually needs.
Delivery State Needs Its Own Table
Content intent and remote-delivery progress are different state machines and should not be collapsed into one post status.
Mutable Tags Make Rollback Ambiguous
You cannot reliably roll back to yesterday's image if the tag you used yesterday points somewhere else today.
Git safe.directory Can Turn Provenance into UNKNOWN
A production checkout owned by another identity can be readable on disk while Git refuses to trust it.
Compose Rendering Is a Cheap Test That Catches Expensive Deployment Mistakes
A configuration should render successfully from a clean checkout before it is trusted on a production host.
HTTP Status Semantics Are Part of Production Acceptance
A service can be reachable and still be routed through the wrong authentication or application layer.
When a Health Check Fails Every Dependency at Once
Restarting a service whenever one of its dependencies is temporarily unhealthy is not always helpful. An external failure may not be fixed by restarting the application, and useful in-flight work can be lost in the process.
Container Count Is Not a Memory Budget
Counting containers does not tell you how much pressure a server is under. One container may do very little while another holds data, caches, and many threads. Two servers with the same container count can have very different resource needs.
Find Existing Before Create Is a Practical Reconciliation Pattern
When the remote API lacks your preferred idempotency primitive, deterministic discovery can recover an uncertain previous attempt.
Canary Checks Should Protect Unrelated Critical Services Too
A successful target deployment is not a full success if the change quietly damages another workload on the same host.
Migration Tests Catch a Different Class of Failure Than Unit Tests
An application can pass isolated logic tests while its database schema still fails to create, upgrade or enforce the intended constraint.
Deployment Records Without Commit SHAs Are Operational Debt
When a deployment fails later, the first forensic question is which reviewed source revision actually produced it.
UNKNOWN Is Better Than a False FAIL When Metadata Is Restricted
A posture check should distinguish evidence it cannot read from evidence that proves the system is wrong.
A Clean Checkout Is the Real Test of Infrastructure Reproducibility
Live files that are absent from Git are technical debt even when the service is currently healthy.
A Local Remote-Tracking Ref Can Be Stale While Git Says You Match Upstream
Comparing HEAD with origin/main proves consistency with the last fetched view, not with the current remote repository.
Queue Retry Policy Is Part of Data Integrity
Retry counts and backoff are not just performance settings when the job performs external side effects.
Reuse Is an Engineering Decision Too
Building from scratch is a great way to learn, but existing software may solve a user's problem faster and more reliably. Reuse does not remove engineering judgment; it changes what must be evaluated.
A Clean Git Tree Does Not Prove You Deployed the Latest Code
Cleanliness answers whether local tracked files changed; freshness answers whether the revision is the one you intended to run.
If the Same Button Is Pressed Twice, Should the Work Run Twice?
When a network response disappears, a user may retry even though the first operation already succeeded on the server. Whether the second request creates new work or returns the result of the first operation is part of the application's contract.
Audit Tables Must Be Boring and Correct
An audit trail is only useful if its schema is simpler and more dependable than the systems it records.
External Runtimes Still Need an Owner Even If They Live in Another Repository
Repository boundaries should not become operational blind spots on a shared production host.
Entrypoint Assumptions Are Hidden Dependencies
A container command is only reproducible when you know whether the image entrypoint wraps, replaces or transforms it.
A Post-Deploy Inventory Snapshot Is Forensic Evidence
Recording containers, ports and revisions after deployment gives future incident response a known-good comparison point.
A Container Entrypoint Can Break a Perfectly Valid Validation Command
CI failed because the image entrypoint changed how arguments were interpreted, not because the OpenBao configuration was invalid.
A Clean Checkout Must Contain Every Local Build and Config Reference
Infrastructure source is incomplete if Compose points at files that only exist on the current server.
Preserve Artifact Identity Through Deployment
Incident investigation becomes easier when a release can answer which source produced which artifact. A label such as latest is not enough. Latest changes with time; a commit or artifact identity does not.
A Typo in primary_key=True Broke the Model Before Any Business Logic Ran
Schema declarations are executable code; a one-word typo can prevent the application model from initializing correctly.
A One-Line COPY Permission Bug Can Make an Image Non-Reproducible
A configuration file copied with the wrong mode can behave differently depending on the build context and base image defaults.
Background Dispatch Needs Reconciliation After Restart
A scheduler can enqueue work before crashing; recovery logic must discover incomplete deliveries after the process returns.
The Process Is Running, but Is the Service Useful?
A running process is useful information, but it does not prove the process is completing the work it exists to do. A web server can be alive while its required database is unreachable.
A Source-Ownership Manifest Solved the 'Who Owns This Container?' Problem
Seeing a service in Portainer does not tell you which repository, runbook or backup path can recreate it.