Running Strangers' URLs Safely: The Guards in My Website Audit Tool

A website audit tool has an uncomfortable job description: anyone on the internet types a URL, and my server goes and fetches it — with a browser, curl and a link crawler. Done naively, that’s a free proxy into my own network, a way to hammer someone else’s site from my IP, and a queue that one script can fill forever.

This post walks through the guards in front of every audit, in the order a request meets them. The architecture post shows where they sit.

The guards, in order

flowchart TD
  R["POST /api/audit"] --> C{"CORS: browser origin allowed?"}
  C -->|no| X1["Browser blocks the call"]
  C -->|yes| RL{"Per-IP rate limit<br/>1 audit / 5 min"}
  RL -->|over| X2["429 + Retry-After"]
  RL -->|ok| CC{"Concurrency<br/>1 audit at a time"}
  CC -->|busy| X3["503 engine busy"]
  CC -->|ok| V{"Required fields,<br/>valid email, name length"}
  V -->|bad| X4["400"]
  V -->|ok| T{"Turnstile token<br/>(only if configured)"}
  T -->|fail| X5["400"]
  T -->|ok| U{"URL check<br/>scheme, credentials, port,<br/>blocked hosts, private IPs"}
  U -->|bad| X6["400"]
  U -->|ok| D{"DNS: every address public?"}
  D -->|no| X7["400"]
  D -->|yes| H{"Follow redirects manually<br/>re-check every hop, max 5"}
  H -->|unsafe or too many| X8["400"]
  H -->|ok| E["execFile audit-master.sh<br/>no shell · 120 s · 10 MB output"]

1. CORS: only my own pages call the API

The API answers browsers only from its own origins (plus local development); requests without an Origin header — same-origin page loads, server-side calls — are unaffected:

const corsOrigins = (process.env.CORS_ORIGINS || 'https://webaudit.jalcocertech.com,http://localhost:4321,http://localhost:3001')
    .split(',').map((o) => o.trim()).filter(Boolean);
app.use(cors({ origin: (origin, cb) => cb(null, !origin || corsOrigins.includes(origin)) }));

CORS doesn’t stop curl, but it does stop other websites from embedding my audit engine in their pages and spending my server on their visitors.

2. Rate limit and concurrency cap

Every audit runs a real browser, so the expensive guard comes first. Each IP gets one audit per 5 minutes by default, and the whole server runs one audit at a time:

const auditWindowMs = Number.parseInt(process.env.AUDIT_API_RATE_LIMIT_WINDOW_MS || `${5 * 60 * 1000}`, 10);
const auditMaxPerWindow = Number.parseInt(process.env.AUDIT_API_RATE_LIMIT_MAX || '1', 10);
const auditMaxConcurrent = Number.parseInt(process.env.AUDIT_API_MAX_CONCURRENT || '1', 10);

Over the limit you get 429 with a Retry-After header; when the engine is busy, 503. The client IP comes from Cloudflare’s CF-Connecting-IP header first, because behind a tunnel every request otherwise looks like it comes from the same place. Both numbers are environment variables, so a busier day is a config change, not a deploy.

3. Boring validation

Name present and under 200 characters, a plausible email, a URL field. Then, only if a Turnstile secret is configured, the bot-check token is verified with Cloudflare. Without the secret the step is skipped — it’s a switch I can flip if bots ever find the form, not a cost every visitor pays today.

4. The URL check: no private networks

This is the SSRF guard — server-side request forgery is the risk that my server gets talked into fetching something only it can reach. The parsed URL must pass all of these:

A hostname is then resolved with dns.lookup(host, { all: true }), and the audit is refused if any returned address is private:

const blocked = records.find(record => isBlockedIp(record.address));
if (blocked) {
    throw new Error(`Host resolves to a private or internal IP: ${blocked.address}`);
}

Checking only the hostname string isn’t enough — my-router.example.com can point at 10.0.0.1 — so the decision is made on the resolved addresses.

5. Redirects are followed by hand

A public URL can redirect to a private one. So the tool doesn’t let fetch follow redirects; it requests with redirect: 'manual', reads Location, and runs the whole URL and DNS check again on every hop, up to AUDIT_MAX_REDIRECTS (5 by default). Each preflight request has its own timeout (AUDIT_PREFLIGHT_TIMEOUT_MS, 8 seconds). Only the final, checked URL is handed to the audit scripts.

One limit I accept: the audit tools resolve the hostname again when they run, a few seconds later. A site that changes its DNS between the check and the audit (DNS rebinding) could still slip through. With one audit at a time and a 5-minute rate limit the window is small, but it isn’t zero — so it’s written down, and the container shouldn’t share a network with anything sensitive.

6. execFile, a timeout and an output cap

The checked URL reaches the shell scripts as an argument, never inside a shell string:

execFile(command, args, { env, timeout: 120_000, maxBuffer: 10 * 1024 * 1024 }, ...);

No shell means no command injection, whatever characters the URL contains. The 120-second timeout bounds the slowest possible audit, and the 10 MB maxBuffer bounds how much output a misbehaving tool can push back.

7. Being polite to the audited site

Safety isn’t only about my server. The link check — the only phase that crawls — is off by default and limited to three levels and a fixed time box. Lighthouse loads the page once, the SEO check fetches three resources, the security check makes a handful of requests. Combined with the per-IP rate limit, nobody can use the tool to flood a third party.

What’s still on my list

• A shared rate-limit store if I ever run more than one instance (today the counters live in memory, which is fine for one container).
• Pinning the resolved IP for the audit tools to close the DNS-rebinding gap.

Takeaways

• Decide on resolved IPs, not hostnames, and re-check after every redirect.
• Put the cheapest rejection in front of the most expensive work: rate limits before browsers.
• execFile with a timeout and an output cap for any tool that takes user input.
• Default to polite: heavy crawling is opt-in.

You can see all of this in action at webaudit.jalcocertech.com.

Next: turning audit findings into a plain-language diagnosis.