A Pull-Only Homelab: Mirroring Cloudflare KV to SQLite Without a Tunnel

My contact forms store every lead in Cloudflare Workers KV. That is a good inbox: no server, free at this scale, and the visitor never waits on my homelab. But KV alone is not a backup, it is awkward to query, and anything I want to do after capture (lists, drip emails, erasure requests) needs a real database next to it.

The obvious design is to have the Worker push each lead into the homelab. I did the opposite: the homelab pulls, and nothing on the internet can reach it.

Why pull instead of push

Instant alerts already come from ntfy (see contact forms with KV and ntfy), so a 15-minute delay for the database copy costs nothing. In exchange, the lead data never needs an inbound path into my network.

That is also why the mirror needs no Cloudflare Tunnel. A tunnel is for traffic coming in — I use one for ntfy, because my phone must reach it (locked-down ntfy behind a tunnel). The mirror only calls out:

homelab ──HTTPS──► api.cloudflare.com   list keys, read values
        ──HTTPS──► ntfy server          only when pulls fail or recover

The token: read, not “metadata read”

The Cloudflare API token has a single permission, scoped to my account: Workers KV Storage → Read. The permission list also offers Workers KV Storage Metadata Read. It is tempting because it sounds even smaller, but a mirror needs the stored values, not just key names, so plain Read is the right one.

Writing to KV (deleting a person’s data on request) uses a second token with Edit, which is not stored in the always-running service. Cloudflare cannot scope that token to one namespace, so it stays out of the daemon.

Fetch only what changed

KV’s list endpoint returns key names and each key’s small metadata blob. The forms Worker keeps the subscription status (pending, confirmed, unsubscribed) in that metadata. So the mirror can decide from the listing alone whether a value needs fetching:

for name, item in lead_keys.items():
    row = known.get(name)
    status = item.metadata.get("status")
    if row is None or row["status"] != status or row["kv_gone_at"] is not None:
        raw = kv.get(name)          # new, changed, or back again: fetch it
        ...
    else:
        conn.execute("UPDATE leads SET last_seen = ? WHERE key = ?", (stamp, name))

In the steady state a pull is one list call and zero value reads. The first run against my live namespace fetched every value; the second fetched none.

Unsubscribes live in the same namespace as suppress:<sha256(email)>. The mirror copies them into their own table and mirrors them exactly: if a suppress: key disappears, it means a fresh confirmation lifted it, so the local row goes too.

The mirror must honour retention

This is the part I almost got wrong. My privacy policy says enquiries are kept for 24 months, and the Worker enforces that with a KV TTL. A naive backup would keep every lead forever — quietly breaking the policy the TTL was supposed to enforce. More on the TTL in retention as a KV TTL.

So KV is the source of truth including deletions:

• A lead that disappears from KV is marked kv_gone_at, not deleted at once.
• Rows gone for more than 30 days are purged.
• Anything past its kind’s retention (enquiries and ebook requests: 730 days; newsletter subscriptions: no expiry) is purged locally even if KV still has it.

GONE_GRACE_DAYS = 30
RETENTION_DAYS = {"enquiry": 730, "ebook": 730, "newsletter": None}

The grace period covers the case where a key vanishes by mistake and comes back: the next pull clears kv_gone_at and nothing is lost.

SQLite is opened with PRAGMA secure_delete=ON, so purged rows are overwritten rather than left readable in free pages.

A safety brake beats a faithful mirror

“Mark whatever vanished as gone” has a nasty failure mode: point the mirror at the wrong namespace, or get a truncated listing, and every known lead “vanishes” at once. Thirty days later the purge finishes the job.

The pull refuses to continue if more than half of the known leads are missing:

MAX_GONE_SHARE = 0.5
if present and not force and len(vanished) > MAX_GONE_SHARE * len(present):
    raise PullAborted(f"{len(vanished)} of {len(present)} known leads missing from KV; "
                      "refusing (use --force if expected)")

The run is logged as failed, and an alert goes out. If the drop is real (I deleted test data on purpose), --force lets it through once.

Backups and alerts

The SQLite file is the primary copy; once a day the loop also writes a dated copy using SQLite’s backup API, which is consistent even while the database is in use. Old copies rotate out:

with sqlite3.connect(target) as out:
    conn.backup(out)
for old in sorted(directory.glob("leads-*.db"))[:-keep]:
    old.unlink()

I keep 14. That number matters for erasure requests: a deleted person survives in backups until they rotate out, which is worth stating honestly.

Alerting is deliberately quiet. The loop sends one ntfy message when pulls start failing and one when they recover — not one every 15 minutes during an outage.

Docker gotchas from the first deploy

The whole thing is standard-library Python (urllib, sqlite3), so there is nothing to keep patched on an always-on box. It runs in Docker Compose, and the first deploy produced three surprises:

The empty logs were the most misleading: the mirror had been working the whole time, and errors (stderr) would have shown up. Only the success lines were stuck in a buffer.

What I would keep for next time

• Pull, don’t push, when the data can wait minutes: no inbound path, and outages heal themselves.
• Use the narrowest token that works — and check that “narrower” still covers what you need.
• A backup inherits the retention policy, or it silently breaks it.
• Put a brake on destructive sync: a wrong ID should fail loudly, not empty your database a month later.
• Alert on state changes, not on every failed attempt.

The same SQLite mirror is what the email side builds on: who may be emailed, erasure, and the drip engine in standard-library Python.

Next: publishing future-dated posts on Cloudflare Pages with a cron Worker.