Emailing a Full Website Report: Confirm First, Then Crawl

My free website audit answers in about 20 seconds. That speed has a cost: the slowest check — crawling every link on the site — is off, and the whole run is killed at 120 seconds. For a quick score that’s fine. For “show me everything that’s broken”, it isn’t.

So I added an optional box to the audit form: “Email me the full report, including a deep check of every link.” Behind it is a small flow across two systems: the audit server on my homelab and the Cloudflare Worker that runs my lead hub. This post is that flow, and why the first step is a confirmation email rather than a crawl.

The rule: nothing runs until the address is proven

The tempting version is “tick the box, we crawl, we email”. It has two problems:

1. Anyone can type anyone’s address. The tool would happily email a report to a stranger who never asked for it.
2. Anyone can make the tool crawl any site for any address, including addresses that don’t exist. A deep crawl means hundreds of requests to someone else’s server.

Confirming first fixes both. The first email is a signed link; the deep audit only starts when the owner of that inbox presses a button. It’s the same pattern as double opt-in, applied to a heavy job instead of a subscription.

The box is also independent of the marketing box: you can get the full report without agreeing to anything else, which keeps consent clean (more on that in unbundling consent).

The flow

sequenceDiagram
  participant V as Visitor
  participant A as webaudit (homelab)
  participant W as Lead hub Worker
  participant M as Mailjet
  V->>A: Run audit (full report box ticked)
  A-->>V: Normal result + "check your inbox"
  A->>W: report-request (keyed): request id, email, site
  W->>M: "Start your full report" (signed link, 7 days)
  M-->>V: Email
  V->>W: GET link
  W-->>V: Page with a button only
  V->>W: POST (button)
  W->>A: deep-audit (keyed): request id
  W-->>V: "Your report is on its way"
  A->>A: Queue: full audit incl. link crawl, render HTML
  A->>W: report-ready (keyed): link, score, verdict, priorities
  W->>M: "Your website report: 68/100"
  M-->>V: Email with the report link

The keyed calls use the same shared key as server-to-server leads, compared in constant time on both sides.

Step 1: register the request, don’t run it

When the box is ticked, the audit server only writes down which site the request is for. The email address lives only in the lead hub:

if (deepReport === true && formsServerKey) {
    const requestId = deep.register(safeUrl);
    postToHub('/api/forms/report-request', { requestId, name, email, website: safeUrl })
        .catch((err) => console.error('❌ Full-report request failed:', err.message));
}

The Worker validates the input, skips anyone on my do-not-send list or the spam blocklist, stores a report:<id> record that expires after 8 days, and sends the confirmation email.

Step 2: GET shows a button, POST starts the job — once

Corporate mail filters open every link in an email. If the link itself started a crawl, a security scanner would trigger it. So the link only shows a page with a button; the POST does the work, and only once:

if (record.status !== "pending") return reportStartedPage(site, true); // "Already on its way"
const res = await fetch(`${site.origin}/api/deep-audit`, {
  method: "POST",
  headers: { "Content-Type": "application/json", "X-Forms-Key": env.FORMS_SERVER_KEY ?? "" },
  body: JSON.stringify({ requestId: id }),
});
if (!res.ok) throw new Error(`deep-audit ${res.status}`);
record.status = "started";

A second click gets “already on its way”, never a second job. If the audit server is unreachable, the visitor sees a “reply to this email” page, I get an alert, and the record stays pending so a retry still works.

Step 3: a background queue on the audit server

The deep audit is the same engine as the normal one (what it checks), with different limits:

The queue is deliberately simple: an in-memory array drained one job at a time. A request is idempotent — enqueuing it twice is a no-op — and the raw run files (Lighthouse JSON, crawler output) are deleted as soon as the report is rendered.

Step 4: a link that survives restarts

The interactive audit’s report links live in memory for 24 hours and die with a restart (expiring report links). For an emailed report that’s not good enough: the link must work tomorrow even if I redeploy tonight.

So the deep report is written to disk as a self-contained HTML file named by a random token, with a small JSON file holding its expiry. GET /r/<token> serves it for 7 days with noindex, no-store and no-referrer; after that it answers 410 Gone with a “run a new audit” link, and a daily clean-up deletes the files.

Step 5: the email, exactly once

When the report is ready, the audit server calls back with the link and a summary — score, a one-line verdict, three priorities. The Worker checks the link points at its own site’s /r/ path, checks the do-not-send list again, and sends:

Your website report for shop.example.com: 68/100 The site has a good base, but search visibility and security are holding the experience back.

1. Fix the search basics · 2. Make it easy to trust and contact you ·
2. Harden security at the hosting level

A repeated callback gets “already sent”; a failed run gets a short apology asking the visitor to reply. Tests cover each branch: wrong key, suppressed address, GET doesn’t start anything, the button starts exactly one job, the email goes out exactly once.

The first live run did exactly what the diagram says: confirmation email, button, a few minutes of crawling, report email, working link. You can try it at webaudit.jalcocertech.com.

Takeaways

• Confirm before doing anything expensive on behalf of an email address.
• GET shows, POST acts. Every link in an email gets opened by a robot.
• Make the start idempotent with a status on the record, and say “already on its way” instead of starting twice.
• Emailed links need storage that outlives the process, with an expiry you can state in the privacy policy.

Next: a client-ready HTML report without AI.