• Sep 4, 2026
  • 4 min read

Debugging a Pasted Secret: The Key Was 65 Characters

My web-audit tool’s backend forwards every finished audit to my central lead hub, authenticated with a shared key in an X-Forms-Key header (the design is in server-to-server leads with a key). After deploying it to the homelab, every audit logged the same line:

❌ Lead hub forward failed: lead hub 403

The audit itself worked; only the copy to the lead hub failed. A 403 here has three possible causes: the key doesn’t match, the Worker rejects the request for another reason, or Cloudflare blocks it before the Worker runs. I had already been bitten by the third one (Cloudflare error 1010 and my homelab alerts), so I wanted to know, not guess.

Step 1: prove the receiving side from somewhere else

The Worker checks the server key before it validates the payload. That makes a harmless probe possible: send the correct key with a deliberately invalid website. A correct key gets past the check and fails validation; a wrong key never gets that far.

curl -s -A "auditmagnet-server/1.0" -X POST \
  -H "Content-Type: application/json" \
  -H "X-Forms-Key: $(cat ~/.config/leads-hub/forms_server_key)" \
  -d '{"kind":"audit","email":"[email protected]","website":"not-a-url"}' \
  https://webaudit.jalcocertech.com/api/forms/submit
# {"error":"invalid_website"}   ← key accepted
# with a wrong key:              {"error":"forbidden"}

From the laptop, with the key file that was also stored in the Worker: 422. So the Worker was fine, Cloudflare let the user agent through, and nothing is stored, because validation fails. The problem had to be the key the container had.

Step 2: a one-liner that died in the terminal

My first idea was a node -e '…' one-liner through docker exec, doing the same probe with the container’s environment. It failed with SyntaxError: Invalid or unexpected token. The terminal had wrapped the long line, and the paste put the wrap inside a string literal. Anything long, quoted and multi-part is fragile to paste.

Step 3: commit the probe instead of pasting it

The repo is bind-mounted into the container, so a committed script is available right after git pull. The probe prints the key’s length and a short SHA-256 prefix, never the key, and sends the same harmless request:

const key = (process.env.FORMS_SERVER_KEY || '').trim().replace(/^["']|["']$/g, '');
console.log('key length', key.length, '· sha256',
  createHash('sha256').update(key).digest('hex').slice(0, 16));

const response = await fetch(url, {
  method: 'POST',
  headers: { 'Content-Type': 'application/json', 'X-Forms-Key': key,
             'User-Agent': 'auditmagnet-server/1.0' },
  body: JSON.stringify({ kind: 'audit', email: '[email protected]', website: 'not-a-url' }),
});

It ends with a one-line verdict: key accepted (422), key rejected (forbidden), or blocked before the Worker (anything else).

docker exec auditmagnet-prod node scripts/lead-hub-probe.mjs
key length 65 · sha256 abcd1234…
status 403 · {"error":"forbidden"}
Key rejected: compare the fingerprint with the laptop and copy the key file again

The key is 64 hex characters. The container had 65, even after trimming whitespace and quotes: one extra visible character had come along with the paste. The fingerprint didn’t match the laptop’s either. A hash prefix is safe to compare in chat or logs; it can’t be turned back into the key.

Step 4: copy the file, don’t paste the value

scp ~/.config/leads-hub/forms_server_key <homelab>:/tmp/fsk
ssh <homelab> 'cd ~/webaudit && sed -i "/^FORMS_SERVER_KEY=/d" .env \
  && printf "FORMS_SERVER_KEY=%s\n" "$(cat /tmp/fsk)" >> .env \
  && rm /tmp/fsk && grep -c "^FORMS_SERVER_KEY=" .env'

printf writes exactly the file’s contents plus one newline, and the grep -c confirms there’s a single key line. Then a detail that’s easy to forget: the container reads its env_file when it’s created, so editing .env alone changes nothing. Recreate it:

docker compose -f docker-compose.prod.yml up -d --force-recreate
docker exec auditmagnet-prod node scripts/lead-hub-probe.mjs
key length 64 · sha256 <same prefix as the laptop>
status 422 · {"error":"invalid_website"}
OK: key accepted (422 is expected for this probe)

The next real audit logged 📮 Lead forwarded to the lead hub.

Making the next one obvious

Two small server changes came out of this:

  • Trim the key on read. A stray space, newline or pair of quotes from an .env edit now can’t cause a 403. (It wouldn’t have saved me here: the extra character was visible.)
  • Log the response body on failure. lead hub 403 alone hid the answer; lead hub 403: {"error":"forbidden"} versus lead hub 403: error code: 1010 tells a wrong key from a Cloudflare block at a glance.
if (!response.ok) throw new Error(
  `lead hub ${response.status}: ${(await response.text().catch(() => '')).slice(0, 120)}`);

Takeaways

  • Order your checks so a probe can succeed without side effects: auth first, then validation, and send a deliberately invalid payload.
  • Prove each side separately. A probe from the laptop that passed pinned the problem on the container in one step.
  • Compare secrets by length and hash prefix, never by printing them.
  • Long quoted one-liners don’t survive terminals. Commit a tiny script.
  • Copy secret files (scp, printf) instead of pasting values.
  • .env is read when a container is created: recreate after editing it.
  • Log response bodies on failure. A status code alone hides the reason.

More shell and container traps: shell traps while watching logs and a week of Cloudflare debugging notes.

Next: a bind-mounted repo turns runtime files into git files.