APK rollouts (OTA upgrades)

How a new kiosk APK build travels from a git push to every vending machine's screen — fully wired, end to end, with staged targeting and live adoption tracking. Watch it happen visually in the emulator (OTA tab on the operator phone, "Check for updates" on the kiosk).

The pipeline at a glance

git push (reyeah-jd-vending-kiosk)
   │
   ▼
CI: kiosk-x-build.yml ── apktool decode → patch baseUrl → rebuild
   │                     zipalign → apksigner (release key)
   ▼
S3 artifacts: …/downloads/reyeah-vending-kioskx.apk
   │
   ▼
POST /api/v1/rollouts            ← operator/admin publishes the build
   │      {version, apkUrl, strategy: canary list or fleet %}
   ▼
kiosk polls GET /apk/getUpgradeVersion    ← was null, now returns descriptor
   │
   ▼
firmware downloads APK from S3 → verifies → installs → reboots
   │
   ▼
firmware reports POST /apk/apkUp {version}
   │
   ▼
GET /api/v1/rollouts/{id} → machine flips to "upgraded"
   └── rollout auto-completes when adoption hits 100%

Every stage above is implemented and live — nothing is hand-waved.

Stage 1 — Build & sign (CI)

The kiosk-x-build.yml workflow in intelli-verse-kube-infra runs scripts/build_vending_variants.sh from the reyeah-jd-vending-kiosk repo on every dispatch: it decodes the stock APK with apktool, patches UrlConfigString.baseUrl to point at our cloud, rebuilds, zipaligns, signs with apksigner, and publishes the artifact to S3:

…/downloads/reyeah-vending-kioskx.apk — talks to api.kiosk-x.ai

This is the exact file the downloads hub serves and the exact URL a rollout points devices at.

Stage 2 — Publish a rollout

POST /api/v1/rollouts — scope machines:write. Two targeting strategies:

# Canary: only the named machines
curl -X POST ".../api/v1/rollouts" -H "X-API-Key: $KEY" -H "Content-Type: application/json" -d '{
  "version": "1.0.43",
  "apkUrl": "http://kiosk-x-docs.s3-website-us-east-1.amazonaws.com/downloads/reyeah-vending-kioskx.apk",
  "notes": "Canary on the warehouse unit first",
  "strategy": {"machines": ["866903013700011"]}
}'

# Staged: a deterministic percentage of the fleet
curl -X POST ".../api/v1/rollouts" -H "X-API-Key: $KEY" -H "Content-Type: application/json" -d '{
  "version": "1.0.43",
  "strategy": {"percentage": 25}
}'

Percentage bucketing is a stable hash of the machine number, so growing a wave 25% → 50% → 100% only ever adds machines — a kiosk never flips between waves. Operators can only roll out to their own machines; admin rollouts may span the whole fleet.

pause / resume / cancel are one POST each (/api/v1/rollouts/{id}/pause …). Pausing stops new offers immediately; machines that already upgraded keep the new build.

Boot-seeded canonical offer — off in production. The rollout store is in-memory, so a published offer dies with the pod, and the app can re-arm one canonical rollout at startup (KIOSKX_SEED_APK_ROLLOUT_VERSION) targeting the entire fleet (percentage: 100, membership computed live per poll); set KIOSKX_SEED_APK_ROLLOUT_MACHINES to a comma-separated serial list for a canary instead.

KIOSKX_SEED_APK_ROLLOUT gates it, defaulting off on a production deployment and on everywhere else (derived from KIOSKX_PRODUCTION, which the cluster gets from having a persistence DSN). The seeded wave takes its digest from KIOSKX_SEED_APK_ROLLOUT_SHA256, which is a claim about the artifact at the moment the variable was written and goes stale the instant CI rebuilds over the same path. That is how the v1.2.1 Reyeah wave came to record one sha256 while its URL served another, and why repairing it by hand did not survive the next deploy. Publish production waves through POST /api/v1/rollouts, which reads the artifact and refuses claims the bytes contradict; the seeder reads nothing.

A production instance that has an active boot-seeded wave says so at the front of detail on GET /api/v1/rollouts and on the single-wave view — a boot log is not a disclosure.

Stage 3 — The device picks it up

The stock Reyeah firmware polls getUpgradeVersion on boot and periodically. No rollout → {"code": 0, "data": null} → the kiosk stays put. Active rollout targeting this machine with a newer version → the vendor-shaped descriptor:

{
  "code": 0,
  "data": {
    "id": "ro_4f2a9c1d8e7b",
    "name": "reyeah-jd-v1.0.43",
    "prefix": "reyeah-jd",
    "version": "1.0.43",
    "url": "http://kiosk-x-docs.s3-website-us-east-1.amazonaws.com/downloads/reyeah-vending-kioskx.apk",
    "fileName": "reyeah-vending-kioskx.apk",
    "flag": 1, "type": 1
  },
  "msg": "success"
}

The firmware compares version with its own build, downloads url (straight from S3 — the cloud never proxies the binary), installs, and reboots. This is unmodified vendor behaviour: no client changes are needed for OTA to work against Kiosk-X.

Stage 4 — Confirmation & adoption tracking

After reboot the firmware reports its build via POST /apk/apkUp. That report is what flips the machine from offered to upgraded:

curl ".../api/v1/rollouts/ro_4f2a9c1d8e7b" -H "X-API-Key: $KEY"

{
  "version": "1.0.43", "status": "active",
  "adoption": {"targeted": 2, "offered": 2, "upgraded": 1, "percentComplete": 50.0},
  "machines": [
    {"machineNo": "866903013700011", "currentVersion": "1.0.43", "state": "upgraded"},
    {"machineNo": "866903013700022", "currentVersion": "1.0.42", "state": "offered"}
  ]
}

Machine states: pending (hasn't polled yet) → offered (poll served the descriptor) → upgraded (apkUp confirmed the new build). When every targeted machine reports the new version, the rollout auto-completes. Each machine's current build is also on its record (GET /api/v1/machines/{no} → software.apkVersion), so the operator app always shows fleet versions even outside a rollout.

Operator-app rollback

Roll back the same way you roll forward: create a new rollout pointing at the previous APK artifact. Version comparison is what gates offers, so a rollback rollout must carry a higher version string (e.g. re-tag the old build as 1.0.44-rollback) — standard practice for vending fleets, since Android won't downgrade a versionCode without a reinstall anyway.

Try the whole loop in 60 seconds

Open the emulator → operator phone → OTA tab: fleet is on v1.0.42.
Tap Roll out next APK build — creates a real rollout via the store.
On the kiosk, tap ⟳ Check for updates (OTA) — the kiosk polls getUpgradeVersion, "downloads" the S3 APK, installs, and confirms via apkUp.
Watch the OTA tab: the machine flips to upgraded, the progress bar moves, and the rollout completes at 100%.

Or do it headless with three curls: POST /api/v1/rollouts, then GET /apk/getUpgradeVersion, then POST /apk/apkUp with the new version, and read back GET /api/v1/rollouts/{id}.