First Boot After OTA Is Still a Transaction
A successful flash write does not prove the new image can initialize hardware, load state and stay healthy.
TOPIC
Production OTA & Fleet
Lessons, explainers, experiments, and implementation notes.
STABLE Requires Evidence from ACTIVE Devices
A release could look administratively complete before any device proved it was actually running.
Rollback Is a Product Feature, Not a Bootloader Detail
Recovery changes how safely a fleet can ship updates and how clearly operators can diagnose failures.
Desired and Observed State Should Never Share a Field
Operator intent and device-reported reality can disagree legitimately during rollout, reboot, outage or rollback.
Compatibility Metadata Is Release Policy, Not Tribal Knowledge
Hardware and software eligibility rules should be machine-readable where rollout decisions are made.
Partition, Bootloader and Security Changes Are Not Normal App OTA
Some changes alter the substrate that makes normal A/B OTA safe and therefore cannot be treated like another application image.
Device Enrollment and Heartbeat Are Different Events
Enrollment establishes trusted identity; heartbeat reports what an already known device is doing.
Heartbeat Is Observation, Not Enrollment
A device sending telemetry could look alive even when its identity or enrollment state was not valid for production operations.
Sticky Heartbeat Results Need Release-Scoped Causality
A terminal-looking last_ota_result could survive a later reassignment and incorrectly poison or complete the new assignment.
Rollback Safety Ends at Persistent Schema Compatibility
A previous binary is not a valid rollback target if the new firmware transformed NVS into a format the old firmware cannot read.
PBX Failure Must Not Roll Back Healthy Firmware
A temporary network, DNS, backend or PBX outage could otherwise make a healthy image look defective during first boot.
Failure Injection Is Part of the OTA Definition of Done
A successful happy-path download could not prove rollback, credential rejection, compatibility gates or interrupted writes behaved safely.
Append-Only OTA Events Are the Fleet’s Incident Timeline
Heartbeat snapshots alone could not reconstruct the sequence of download, validation, rollback and operator actions during a failed rollout.
Release Lifecycle Is an Operator State Machine
A release needed gates between registration and fleet-wide use rather than one published flag.
First Boot Is a Pending Transaction, Not Success
A reboot into the new partition was too weak to count as a successful update.
Desired Release and Running Release Are Different Truths
The server needed to know what a device should run without pretending the device had already installed it.
SHA-256 Proves Integrity; ECDSA Adds Authorship
A correct hash could prove that downloaded bytes matched the manifest but not that an authorized release process created that manifest and artifact.
The Signing Private Key Does Not Belong on the OTA Runtime Host
If the OTA server stored the production signing private key, compromise of the delivery plane could become authority to mint trusted firmware.
Release ID Is Stronger Than a Version String
Version text alone could not uniquely identify artifact lineage or distinguish reissued builds.
Canary to Cohort to Stable Is a Feedback Loop
Rolling all boards at once would maximize blast radius before the first device produced field evidence.
A Wi-Fi MAC Address Is Not a Device Credential
A board needed a stable fleet identity without turning a public hardware identifier into an authentication secret.
One-Time Enrollment Turns Factory State into Fleet State
Factory-programmed boards needed a controlled transition into an enrolled state without shipping a reusable fleet credential.
A Newer Binary Does Not Mean a Device Should Update
Simply placing a newer firmware file on the server risked turning storage into rollout policy.
Power Loss Has Multiple OTA Failure Windows
Power can disappear during download, inactive-slot write, after boot-partition selection or during first boot.
An Active SIP Call Has Priority over OTA
A firmware update competing with a live voice call could damage the product function the OTA system exists to maintain.
The Inactive OTA Slot Protects the Running Image
An update should be able to fail during download or write without destroying the last working firmware.
Compatibility Is Checked at Assignment and Artifact Request
A device can change state after assignment, so one compatibility decision made earlier may become stale before download.
Blocked and Revoked Are Different Fleet States
Operators needed to stop a device temporarily without confusing that action with permanent credential invalidation.
Automatic Rollback and Fleet Rollback Solve Different Failures
One rollback mechanism could not cover both immediate boot failure and defects discovered after a release had already been accepted.
The Backend Can Be Down Without Making the Phone Useless
Provisioning and OTA control were new dependencies, but a temporary backend outage should not break an already configured voice device.