Per-cabinet device credentials for the ZHZN gateway

Status: accepted, implemented. Supersedes the "no per-device accounts" note that stood at app/config.py:91-94 and the matching paragraph in the module docstring of app/routes/zhzn.py.

The device half of this decision was written up first, from the Android side, in the agent repo: docs/decisions/0001-device-identity-and-client-secrecy.md. That document records the finding and the shape of the answer. This one records the gateway's half: what the credential is, what binds it to one cabinet, how it is issued, rotated and revoked, and how the cabinets already in the field get onto it without anybody driving to a venue.

What was wrong

_device() authenticated a ZHZN cabinet by comparing the secret: header against KIOSKX_ZHZN_DEVICE_SECRET — one value for the entire fleet — and then read the cabinet's identity out of whichever of five places the caller chose to put it: the equipmentNo header, x-equipment-no, the query string, or equipmentNo / serialNumber in the body. Nothing tied the credential to the identity. Whoever held the fleet secret could address any cabinet on the platform by typing its serial:

curl -H 'secret: <fleet secret>' '<host>/zhzn/goods?equipmentNo=<foreign serial>'
curl -X POST -H 'secret: <fleet secret>' -H 'Content-Type: application/json' \
     -d '{"equipmentNo":"<foreign serial>","aisleNo":2,"quantity":1}' \
     '<host>/zhzn/createOrder'

The first reads a competitor's planogram — SKUs, prices, stock, faulted lanes. The second is worse by a class: it opens a real order in another operator's book, which then reads back to that operator as one of their own sales. Operator X hosts independent vending operators who keep their real books on it, so a cross-tenant write is the most serious defect shape this system has.

Getting the fleet secret is not a cryptographic problem. Cabinets stand in shopping centres; the secret is in /etc/zhzn-agent.env on every one of them, and (see the agent-side decision record) it was also baked into a publicly downloadable APK. The design assumed the credential stayed inside the fleet. It does not.

The credential

A per-cabinet bearer key: a 256-bit random secret with an opaque key id, issued by the gateway, held by the cabinet, stored here only as a hash.

On the wire, one header on every gateway call:

x-device-key: <keyId>:<secret>          e.g. dk_9f3c…:pQ7…-_A

keyId is dk_ + 16 hex characters and is not secret; it is what the gateway looks the cabinet up by, what the operator console displays, and what a log line may safely carry. secret is 32 random bytes, urlsafe-base64, and appears in exactly two places: the cabinet's own state directory and the response to the one call that issued it.

Why not an asymmetric keypair

The private half would live on the cabinet, which is the thing an attacker is physically holding. Against the attacker in the threat model — someone who opens a cabinet in a mall — a keypair is worth the same as a shared secret, because both are files on a box they control. What a keypair buys is that a compromise of this backend cannot impersonate a cabinet. That matters less than it sounds: an attacker who owns this backend already is the platform and does not need to pretend to be a cabinet to write into an operator's book.

Against that, the cost is concrete. The agent is stdlib-only by deliberate choice — the OTA signature check in agent/ota.py implements RSA PKCS#1 v1.5 verification in pure Python precisely to avoid depending on cryptography and its ARM build problems. Verification with a public exponent is a pow() call. Signing needs the private exponent, and generating the key needs prime search on a Rockchip board; both would be new pure-Python crypto in the money path, written by us, on hardware nobody can debug remotely. And the Android storefront would need a second, independent implementation of the same signing rules in Kotlin.

Storing only a hash of the bearer secret recovers most of what the keypair was for: a dump of the machine table yields no usable cabinet credential. That is the property worth having, and it costs one hashlib call.

Why not short-lived tokens minted from an enrolment secret

Because the enrolment secret then lives permanently on the cabinet, which is the fleet-wide secret again with extra steps. Tokens are only an improvement if what sits on the cabinet is itself per-cabinet — at which point the token is a caching layer over this design, not an alternative to it. Worth doing later if we want to shorten how long a credential is useful after it is copied off a cabinet's disk; not worth doing now, and it is listed under "Deliberately not in this change".

Why the secret is hashed with plain SHA-256 and not a password KDF

It is 256 bits from secrets.token_bytes, not something a human chose. There is no dictionary to try and no work factor that improves a guess against a uniform 256-bit space. A per-record random salt is still stored, so two cabinets that somehow received the same secret do not present the same hash, and the stored form is not a bare digest of the credential.

What binds it to one cabinet

The gateway resolves the machine code from the credential. It is no longer read from the request.

_device() now looks up the presented keyId, and the machine code on that credential record is the machine the call acts on. equipmentNo — from any of its five old locations — is accepted only as a cross-check: if it is present and does not resolve to the same machine, the call is refused with 403 rather than silently served against the credential's cabinet. A mismatch is either a cabinet whose serial was rewritten or somebody trying the old attack, and both deserve to be told.

So presenting cabinet A's credential can only act on cabinet A. There is no input on the request that changes which machine the call reaches.

The serial keeps being sent, because the Android storefront and every agent build in the field send it, because proxies and log searches key on it, and because a mismatch is a real signal we want to see rather than something to tolerate quietly. It is no longer trusted.

/zhzn/register gets one more check on top: a credential issued to one physical box may not re-point its machine at another. A deviceId that is not the box the credential was enrolled to is refused with 403, so the deviceId: attacker-box-1 against a foreign serial that used to be accepted is now two refusals deep — the credential decides the serial, and the credential's own box decides the device. The existing duplicate-identity detector (machine_identity.check_claim) still runs after that, unchanged, because it answers a different question: whether some other machine record already claims this box.

Retirement is unchanged

A retired cabinet still gets 410 on every leg except /zhzn/report, which still settles a vend the cabinet already made so captured money is not stranded. The retirement check simply runs against the machine the credential names instead of the machine the request named. Enrolment of a retired serial is refused for the same reason the selling legs are.

How a credential is issued

Three ways in, in the order a cabinet will actually meet them.

1. First contact, over the fleet secret (POST /zhzn/enrol). A cabinet with no credential presents the fleet secret and its serial and gets {keyId, deviceKey, machineCode} back, once. This is the zero-touch path and the migration path both. The agent does it on its beat thread, not its boot path, and a failure is retried on a lengthening ladder — a cabinet that cannot enrol still boots, still shows its screen and still sells.

2. The factory bench. scripts/factory_provision_machine.py already makes the cabinet's own POST /zhzn/register call with the fleet secret the bench is stamping into /etc/zhzn-agent.env. Enrolling at the bench means a crated cabinet leaves the plant already per-cabinet, and the fleet secret on that unit is spent before it ships. The bench edit is staged, not landed — that script is being changed by another agent this session — and the exact diff is in "Deliberately not in this change" below. Nothing depends on it: a cabinet that ships un-enrolled enrols itself on first contact instead.

3. Re-issue after a loss. A cabinet that enrolled and then lost its state directory (a board swap, a wiped image) cannot enrol again — from the gateway's side, an identity asking to be issued twice is exactly what a clone looks like, so that is a 409. Recovery is the admin path that already exists for this hardware problem, described under rotation below.

What the fleet-wide secret is still for, and what bounds it

Exactly one thing: enrolling a cabinet that has never enrolled. Every other leg refuses it once the cabinet has a credential — _device() will not accept the fleet secret for a machine that has an active credential on file, full stop. That is the property that makes the migration monotonic: the moment a cabinet enrols, a stolen fleet secret stops being able to address it, and no further action is needed to make that true.

Four bounds on the window that remains:

• KIOSKX_ZHZN_ENROLMENT is a three-rung ladder. open (the default) accepts a fleet-secret enrolment from a cabinet the cloud already hears from, or from one somebody has armed. armed accepts nothing that has not been armed, which is where a fleet that has finished migrating but still installs cabinets belongs. closed refuses fleet-secret enrolment entirely, and a cabinet that needs re-issuing goes through the attributed admin path instead. An unrecognised value reads as open: a typo must surface as an ordinary install, not as an installer at a venue whose cabinet will not enrol however often it is armed.
• Enrolment is bound to the device id we already know. Most field cabinets have reported a deviceId (/etc/machine-id) on a heartbeat already. Where a machine record has one, an enrolment must present the same one; an attacker with the fleet secret does not know a cabinet's machine-id.
• A cabinet with no such binding must be armed first. This is the bound that was missing, and the hole it left is described under What an arm is below. Without it, "bound to the device id we already know" simply did not fire for a cabinet the cloud had never heard from, and any claimant was accepted. The refusal is 428, not 403, because the shipped agent treats 403 on enrolment as revocation and stops asking for good, while 428 lands in its retry ladder — so a cabinet armed an hour later enrols itself with nobody driving back out.
• Enrolment is budgeted. A serial may be enrolled a bounded number of times per window; over that, 429. This is what stops the bootstrap credential being used to churn credentials, and it is the status the device half already knows how to back off from.

What an arm is, and what it is not

An arm is one named person saying this cabinet, now, once. All three bounds are load-bearing: without the cabinet it is a fleet-wide switch, without the clock it is a fleet-wide switch somebody forgot to turn off, and without single-use one arm is an unlimited supply of credentials for that serial. Any one of them missing and this is the fleet secret with a form to fill in.

POST /api/v1/machines/{no}/device-credential/arm opens the window and .../disarm closes it early. Who may call it:

• The operator who owns the cabinet, which is the ordinary case — an installer at a venue at 8pm who cannot arm without finding a platform admin will find some other way, and it will be worse.
• A platform admin acting for them, the same owner-or-admin rule as rotate and revoke.
• A contract manufacturer, for a cabinet nobody owns yet and whose build record names their own plant. The bench is physically holding the unit and already writes its build record, so provisioning is the natural moment. A factory key cannot arm a cabinet that is already in an operator's fleet, so a leaked factory key buys machines with no books on them rather than somebody's revenue.

A board swap arms the cabinet as part of the reset (see the RMA path below), so a technician does not have to know it is two calls.

The residual, stated plainly

The arm closes cabinet theft: nobody can take a serial's identity and lock the real unit out of it. It does not close everything the fleet secret buys, and the difference matters enough to state precisely.

Still open. The gateway serves the selling legs — /zhzn/goods, /zhzn/createOrder, /zhzn/config — to anyone presenting the fleet secret for a machine that is un-enrolled, with no enrolment involved. So whoever holds the secret can read an un-enrolled cabinet's full planogram (product names, prices, stock) and open real orders on its operator's books. That is not limited to ZHZN cabinets: the test is is_enrolled and says nothing about hardware, so it reaches every un-enrolled machine record on the platform.

Why it stays open. These are the legs a cabinet in the field sells on before it has migrated; refusing them brings down the installed base. Refusing only non-ZHZN machines is not available either, because the platform cannot reliably tell the two apart before a cabinet's first call: software.platform is "reyeah" on every machine record until a ZHZN agent speaks, including a brand-new cabinet's, so that refusal would land on the hardware it is meant to protect. Closing it properly needs a positive ZHZN marker written at provisioning time — the build record's manufacturer is the obvious candidate — applied to the whole installed base first.

How wide it is, and how to watch it. As wide as the un-enrolled population. reachableOnFleetSecret on the fleet view is that number, unfiltered, and it is the one to drive to zero. Note that KIOSKX_ZHZN_ENROLMENT=closed does not close it: that setting governs enrolment, and these legs sell.

Also still open, narrowly. A cabinet that reports its /etc/machine-id for the first time on a fleet-secret call has that binding recorded as self-asserted and it cannot vouch for an enrolment — but bindings that predate this change are trusted, because they are the installed base and there is no cutover timestamp to configure and get wrong. Anyone who held the fleet secret before this shipped and planted a device id on an un-enrolled cabinet therefore still holds a binding that will vouch for one enrolment. Auditing that is a one-off job against the machine table, not a code change.

Rotation, revocation, and a replaced control board

Three different operations, deliberately not one.

Rotate — planned, no truck roll, no downtime. POST /api/v1/machines/{no}/device-credential/rotate does not mint anything. It marks the credential as due for rotation; generation N+1 is minted and handed over in the response to the cabinet's own next authenticated call, on the x-device-credential response header, and the cabinet persists it and starts presenting it. Using generation N+1 is what retires N.

Minting on request and minting on collection look equivalent and are not. If the new secret were minted when the operator clicked, it would have to be stored somewhere until the cabinet came to get it — so a secret that currently exists only in the response to one call would exist at rest, which is the property the hashing was for. And the admin who clicked would be holding a working cabinet credential, which no admin has any business holding. Minting on collection means the response to the cabinet's own call is still the only place the secret ever appears.

It also means a cabinet that is offline for a fortnight keeps working on N and completes the handover when it comes back, rather than being locked out on the day somebody clicked rotate. That is the difference between a rotation and an outage for whichever cabinets were in a warehouse that morning.

Revoke — compromise, and it stops working now. POST /api/v1/machines/{no}/device-credential/revoke invalidates every generation immediately. The cabinet's calls then get 403, which the device half reads as revocation: it stops presenting the key and, importantly, does not try to enrol itself a new one — otherwise revocation would mean nothing for a stolen cabinet. Revoking takes a by and a reason, both mandatory and both kept on the record, for the same argument machine_identity.reset makes: the operation an attacker most wants is the one that must never be anonymous.

Revocation deliberately does not reopen enrolment. A revoked cabinet that turns out to be legitimate is brought back with reset below, by a named person who says why.

Replaced control board — the RMA path, which already existed. The new board has the cabinet's serial and an empty state directory. POST /api/v1/machines/{no}/identity/reset (app/machine_identity.py) is already the operation that lets a different physical box adopt a machine's history, already admin-only, already demands a signer and a reason, and already keeps the audit trail. It now clears the device credential too, which is what lets the new board enrol. That is one operation for one physical event, rather than a technician having to know that a board swap is two API calls; and it means the duplicate identity detector and the credential store cannot disagree about whether a board swap happened.

Migration for cabinets already in the field

No visit, no flag day, and nothing breaks for a cabinet that is offline while it happens.

1. The gateway accepts both schemes at once. A cabinet presenting a valid device key is served on it. A cabinet presenting only the fleet secret is served exactly as before — unless it already has a credential, in which case the fleet secret is refused for it.
2. Cabinets get an agent build that knows how to enrol, through the OTA channel that already exists (app/agent_releases.py, signed, cohorted). No physical access: that channel is how the fleet is updated today.
3. Each cabinet enrols itself on its next beat and switches to its own key. The device half keeps sending the fleet secret alongside for the same reason the Android half does — an older gateway must keep serving a newer cabinet during a rollout — and the gateway prefers the device key whenever one is presented.
4. The fleet view says who is left, and what the secret still reaches. GET /api/v1/machines/device-credentials/fleet lists every ZHZN machine with its enrolment state (enrolled / fleet-secret / revoked), its key id, generation, when it last authenticated and by which scheme, whether an enrolment window is open on it right now (enrolmentArmed) and how much its device id is worth to an enrolment (enrolmentBinding, below).

Read two numbers off it, because one of them cannot answer the question people ask of it:

The list is filtered to ZHZN cabinets on purpose: burying the handful that are real work among seventy Reyeah cabinets, seed rows and lifecycle fixtures is how a progress bar stops being read. But the gateway's own admission test is just "is this machine un-enrolled", so the secret reaches machines the list correctly excludes, and onFleetSecret cannot see them by construction. On the seeded fleet reachableOnFleetSecret is 27 against an onFleetSecret of 0. Both numbers are published so that neither has to lie.

enrolmentBinding has three values, not two, because "has a device id on file" hid the case that mattered:

The ZHZN migration is finished when no row says fleet-secret. 5. Then close the enrolment window: set KIOSKX_ZHZN_ENROLMENT=closed. No cabinet can enrol on the shared secret after that, so it can be rotated or removed once nothing is waiting to enrol. It does not make the secret worthless on its own — see the residual above. What does is reachableOnFleetSecret reaching zero.

The one thing this ordering costs: between steps 1 and 3 the residual described above is open for the un-enrolled cabinets. With one ZHZN cabinet in the field (the count in the agent-side decision record, verified against the running pod) the enrolment window is a heartbeat wide for the fleet as a whole. This is the argument for doing it now rather than at a hundred cabinets.

Replay, and why nothing here reads the cabinet's clock

Cabinets on this fleet have clock problems bad enough that the gateway stamps clockVerifiedAt, refuses readings that look skewed, and drops the stamp when it catches one (app/kiosk_clock.py, and _clock_for in the gateway). Any scheme whose freshness proof is "the cabinet says what time it is" would therefore either reject healthy cabinets or accept anything, depending on which way the window was sized. So:

• The credential is a bearer key, not a timestamped signature. Confidentiality and anti-replay on the wire are TLS's job, which is where they already were for the fleet secret — this change does not regress that and does not pretend to improve it.
• Where freshness is needed, it comes from the server, never the cabinet. app/machine_network.py already does exactly this for provisioning tokens; if the gateway ever needs channel-independent proof-of-possession, the mechanism is a server-issued nonce the cabinet echoes, not a clock reading. Noted as a follow-up, not built.
• Replaying a captured gateway call is bounded by what the calls already guarantee. /zhzn/report is exactly-once on dispenseId, order legs are idempotent through Store.apply_order_update, and reads are reads. The one thing a replay cannot do any more is what it could do before: name a different cabinet.

Detecting a copied credential

The residual risk of any secret held on hardware in a public venue is that it is copied rather than moved: cabinet A's key used from somewhere else, while cabinet A keeps working. Source IP cannot see it — the live cabinet reaches us through an LTE modem whose address churns.

The gateway therefore issues a rotating witness token on the x-device-witness response header; the cabinet echoes back whatever it last saw and never invents one. Two boxes sharing one key each hold a different link of the chain, so one of them presents a stale witness. That is recorded as a suspicion on the machine record and surfaced in the fleet view — it never refuses a call. A cabinet can lose the token legitimately (a wipe, a crash between response and write, a request that crossed pods), and turning a detector's false positive into an outage on a money path is a bad trade. The device half of this contract is already built in the Android client (DeviceIdentity.nextWitness).

Blast radius, before and after

Where this is enforced, and what proves it

app/device_credentials.py holds the credential store and all the refusals; _device() in app/routes/zhzn.py is the single place a ZHZN call acquires its machine code, and every gateway leg goes through it. The admin surface is on app/routes/machine_identity.py, next to the identity reset it has to agree with.

tests/test_zhzn_device_credentials.py opens with the two reproductions above as tests. Every test in that file was proved by mutation — the hole reintroduced, the test confirmed to fail — and each one names the mutation it catches in its own docstring, so the proof can be re-run rather than taken on trust. The device half is proved the same way in agent/test_device_identity.py.

Deliberately not in this change

• The factory bench edit. scripts/factory_provision_machine.py is being changed by another agent in this session, and a wholesale rewrite of a contended file is how two agents lose work. The integration is one call after the existing step 0: POST /zhzn/enrol with the same fleet secret it already uses, then write keyId:deviceKey into the unit's /etc/zhzn-agent.env as ZHZN_DEVICE_CREDENTIAL beside the fleet secret it already stamps. The runbooks docs/zhzn-factory-install.md and docs/zhzn-field-install.md need the same paragraph. None of it is load-bearing: an un-enrolled cabinet enrols on first contact.
• Short-lived tokens over the per-cabinet key. Shortens how long a copied credential is useful. Needs a refresh story in two client implementations and buys nothing until the per-cabinet key exists, which is what this change is.
• A server-issued nonce for proof-of-possession. The design is stated above so that nobody reaches for a timestamp when this comes up.
• Retiring the secret: header from the agents. It has to keep going out while any gateway or any cabinet in the fleet might be on the old build. It stops being accepted per cabinet at enrolment, which is the part that matters; removing the send is cleanup for after the window is closed.