First Boot After OTA Is Still a Transaction
A successful flash write does not prove the new image can initialize hardware, load state and stay healthy.
OTA
STABLE Requires Evidence from ACTIVE Devices
A release could look administratively complete before any device proved it was actually running.
Secure ESP32 OTA Is a Chain of Trust, Not an HTTPS Download
HTTPS protects a firmware transfer in transit. A production OTA design also has to prove image authenticity, select boot slots safely, confirm health, permit recovery, and prevent rollback to revoked vulnerable firmware.
Signed Dual-Slot OTA Needed an Acceptance State, Not Just a Boot Flag
A new image should become permanent only after the device proves it can boot, verify, connect and report healthy.
Rollback Is a Product Feature, Not a Bootloader Detail
Recovery changes how safely a fleet can ship updates and how clearly operators can diagnose failures.
Human SSO and Machine Authentication Should Not Share One Gate
Firmware cannot complete an interactive Authelia login, but bypassing authentication entirely would expose the OTA control plane.
Desired and Observed State Should Never Share a Field
Operator intent and device-reported reality can disagree legitimately during rollout, reboot, outage or rollback.
Compatibility Metadata Is Release Policy, Not Tribal Knowledge
Hardware and software eligibility rules should be machine-readable where rollout decisions are made.
Partition, Bootloader and Security Changes Are Not Normal App OTA
Some changes alter the substrate that makes normal A/B OTA safe and therefore cannot be treated like another application image.
Heartbeat Is Observation, Not Enrollment
A device sending telemetry could look alive even when its identity or enrollment state was not valid for production operations.
Schema Migration Policy Belongs in the OTA Safety Model
Rollback is only safe when persistent data remains readable by the previous firmware or a migration policy accounts for the change.
Firmware Compatibility Should Be Derived from the Enrolled Device
Release metadata is safer when it comes from authoritative device capabilities instead of copied operator input.
Sticky Heartbeat Results Need Release-Scoped Causality
A terminal-looking last_ota_result could survive a later reassignment and incorrectly poison or complete the new assignment.
Signed Firmware Changed OTA from File Delivery into a Trust Decision
A device should decide whether firmware is authorized, not merely whether the download completed successfully.
Rollback Safety Ends at Persistent Schema Compatibility
A previous binary is not a valid rollback target if the new firmware transformed NVS into a format the old firmware cannot read.
Release Channels Have to Follow Promotion State
Calling a release STABLE while its channel remains canary creates two sources of truth.
PBX Failure Must Not Roll Back Healthy Firmware
A temporary network, DNS, backend or PBX outage could otherwise make a healthy image look defective during first boot.
A Failed Canary Must Block Stable Promotion
Success evidence loses meaning if failed or rolled-back assignments are ignored during promotion.
Failure Injection Is Part of the OTA Definition of Done
A successful happy-path download could not prove rollback, credential rejection, compatibility gates or interrupted writes behaved safely.
Machine APIs Need Machine Credentials, Not Browser Redirects
A firmware client cannot solve an interactive login flow, and treating the redirect as success hides the real authentication failure.
STABLE Should Mean a Device Actually Ran the Release
A release should not become stable merely because an administrator clicked a promotion button.
Exact Route Allowlists Are Safer Than Bypassing Authentication for /v1
A broad path prefix can expose future endpoints that did not exist when the exception was created.
Append-Only OTA Events Are the Fleet’s Incident Timeline
Heartbeat snapshots alone could not reconstruct the sequence of download, validation, rollback and operator actions during a failed rollout.
Release Lifecycle Is an Operator State Machine
A release needed gates between registration and fleet-wide use rather than one published flag.
First Boot Is a Pending Transaction, Not Success
A reboot into the new partition was too weak to count as a successful update.
Desired Release and Running Release Are Different Truths
The server needed to know what a device should run without pretending the device had already installed it.
SHA-256 Proves Integrity; ECDSA Adds Authorship
A correct hash could prove that downloaded bytes matched the manifest but not that an authorized release process created that manifest and artifact.
The Signing Private Key Does Not Belong on the OTA Runtime Host
If the OTA server stored the production signing private key, compromise of the delivery plane could become authority to mint trusted firmware.
Release ID Is Stronger Than a Version String
Version text alone could not uniquely identify artifact lineage or distinguish reissued builds.
Canary to Cohort to Stable Is a Feedback Loop
Rolling all boards at once would maximize blast radius before the first device produced field evidence.
A 422 Can Be Healthier Than a 302 for a Device API
For an intentionally invalid API payload, validation failure proves the request reached the correct application boundary.
A Wi-Fi MAC Address Is Not a Device Credential
A board needed a stable fleet identity without turning a public hardware identifier into an authentication secret.
One-Time Enrollment Turns Factory State into Fleet State
Factory-programmed boards needed a controlled transition into an enrolled state without shipping a reusable fleet credential.
A Newer Binary Does Not Mean a Device Should Update
Simply placing a newer firmware file on the server risked turning storage into rollout policy.
Power Loss Has Multiple OTA Failure Windows
Power can disappear during download, inactive-slot write, after boot-partition selection or during first boot.
An Active SIP Call Has Priority over OTA
A firmware update competing with a live voice call could damage the product function the OTA system exists to maintain.
Terminal OTA Events Need a Release ID
Rollback and validation failure are too important to infer from an unscoped status string.
Dual OTA Slots Need an Acceptance State
Booting the new partition once is not enough to declare it safe.
A Sticky Heartbeat Result Almost Became a False OTA Rollback
Telemetry from a previous release must not be allowed to mutate the state of a newly assigned release.
OTA Updates Became a Reliability Problem Before They Became a Delivery Problem
Downloading new firmware is easy; proving the device can recover from a bad update is the real OTA design work.
An OTA Download Finishing Does Not Mean the Update Is Finished
A full OTA progress bar proves that a file transfer completed. It does not yet prove that the device booted the image successfully or that the required product functions still work. Update success spans several states.
The Inactive OTA Slot Protects the Running Image
An update should be able to fail during download or write without destroying the last working firmware.
Firmware Acceptance Should End in an Observable ACTIVE State
A successful download is not the same thing as a successful deployment.
Compatibility Is Checked at Assignment and Artifact Request
A device can change state after assignment, so one compatibility decision made earlier may become stale before download.
Blocked and Revoked Are Different Fleet States
Operators needed to stop a device temporarily without confusing that action with permanent credential invalidation.
Release State and Release Channel Drifted Apart
A release promoted to STABLE was still carrying old channel metadata, creating two competing sources of truth.
Rollback Has to Include Persistent-State Compatibility
Returning to an older binary is unsafe if the newer release already changed data the old code cannot read.
Automatic Rollback and Fleet Rollback Solve Different Failures
One rollback mechanism could not cover both immediate boot failure and defects discovered after a release had already been accepted.
The Backend Can Be Down Without Making the Phone Useless
Provisioning and OTA control were new dependencies, but a temporary backend outage should not break an already configured voice device.