A Tinny Speaker Is Not Automatically a Codec Problem
Field feedback described the speaker as very tinny even though the digital call path was working.
release engineering
STABLE Requires Evidence from ACTIVE Devices
A release could look administratively complete before any device proved it was actually running.
From Demo-Ready Audio to Production Acceptance
One strong call can prove a candidate is promising but not that it is ready for manufacturing or field release.
Compatibility Metadata Is Release Policy, Not Tribal Knowledge
Hardware and software eligibility rules should be machine-readable where rollout decisions are made.
V117 Taught Me to Revert a Bad Audio Architecture Quickly
A rebuffer/AEC/volume experiment produced obvious bad crackle instead of the intended stability improvement.
Partition, Bootloader and Security Changes Are Not Normal App OTA
Some changes alter the substrate that makes normal A/B OTA safe and therefore cannot be treated like another application image.
Heartbeat Is Observation, Not Enrollment
A device sending telemetry could look alive even when its identity or enrollment state was not valid for production operations.
Why V116–V131 Did Not Automatically Replace V115
Later queue, PLC and timer experiments looked more sophisticated on paper but repeatedly introduced crackle, echo or additional delay.
Sticky Heartbeat Results Need Release-Scoped Causality
A terminal-looking last_ota_result could survive a later reassignment and incorrectly poison or complete the new assignment.
Rollback Safety Ends at Persistent Schema Compatibility
A previous binary is not a valid rollback target if the new firmware transformed NVS into a format the old firmware cannot read.
Mutable Tags Make Rollback Ambiguous
You cannot reliably roll back to yesterday's image if the tag you used yesterday points somewhere else today.
PBX Failure Must Not Roll Back Healthy Firmware
A temporary network, DNS, backend or PBX outage could otherwise make a healthy image look defective during first boot.
Failure Injection Is Part of the OTA Definition of Done
A successful happy-path download could not prove rollback, credential rejection, compatibility gates or interrupted writes behaved safely.
High-Volume Echo Moved the Bug into the Physical Product
After digital timing became stable, echo increased at high speaker volume and could no longer be treated as only a network or queue problem.
Append-Only OTA Events Are the Fleet’s Incident Timeline
Heartbeat snapshots alone could not reconstruct the sequence of download, validation, rollback and operator actions during a failed rollout.
Release Lifecycle Is an Operator State Machine
A release needed gates between registration and fleet-wide use rather than one published flag.
First Boot Is a Pending Transaction, Not Success
A reboot into the new partition was too weak to count as a successful update.
Desired Release and Running Release Are Different Truths
The server needed to know what a device should run without pretending the device had already installed it.
SHA-256 Proves Integrity; ECDSA Adds Authorship
A correct hash could prove that downloaded bytes matched the manifest but not that an authorized release process created that manifest and artifact.
The Signing Private Key Does Not Belong on the OTA Runtime Host
If the OTA server stored the production signing private key, compromise of the delivery plane could become authority to mint trusted firmware.
Release ID Is Stronger Than a Version String
Version text alone could not uniquely identify artifact lineage or distinguish reissued builds.
Canary to Cohort to Stable Is a Feedback Loop
Rolling all boards at once would maximize blast radius before the first device produced field evidence.
A Wi-Fi MAC Address Is Not a Device Credential
A board needed a stable fleet identity without turning a public hardware identifier into an authentication secret.
One-Time Enrollment Turns Factory State into Fleet State
Factory-programmed boards needed a controlled transition into an enrolled state without shipping a reusable fleet credential.
A Newer Binary Does Not Mean a Device Should Update
Simply placing a newer firmware file on the server risked turning storage into rollout policy.
Power Loss Has Multiple OTA Failure Windows
Power can disappear during download, inactive-slot write, after boot-partition selection or during first boot.
An Active SIP Call Has Priority over OTA
A firmware update competing with a live voice call could damage the product function the OTA system exists to maintain.
The Inactive OTA Slot Protects the Running Image
An update should be able to fail during download or write without destroying the last working firmware.
Compatibility Is Checked at Assignment and Artifact Request
A device can change state after assignment, so one compatibility decision made earlier may become stale before download.
Blocked and Revoked Are Different Fleet States
Operators needed to stop a device temporarily without confusing that action with permanent credential invalidation.
V132A: Turning Clear Audio into a Preserved Release Asset
A working phone is not a release unless the exact artifact and source context can survive the next experiment.
Automatic Rollback and Fleet Rollback Solve Different Failures
One rollback mechanism could not cover both immediate boot failure and defects discovered after a release had already been accepted.
The Backend Can Be Down Without Making the Phone Useless
Provisioning and OTA control were new dependencies, but a temporary backend outage should not break an already configured voice device.