• May 5, 2026
  • 3 min read

Keep the Scraper Off the Internet, and Off Everything Else

A look back at a retired client project: what I built, what a later review found, and how the fix works.

Adding a listing in the custom editor had one tedious step: copying every photo URL from the original listing page. I already had a small Python script that extracted image URLs from a page. The plan was simple: paste the listing’s URL into the editor, press a button, and the image list fills itself.

The design

I deliberately didn’t automate the whole listing. Title, price and description stayed manual, so the agent keeps control of the facts. Only the boring part, the images, was automated.

The old script became a tiny FastAPI service with one endpoint, POST /api/extract-images, using BeautifulSoup to find the image URLs. It ran as its own container next to the Astro site:

services:
  astro:
    networks: [internal, tunnel]   # reachable through the Cloudflare tunnel
  scraper-service:
    build: ./scraper-service
    expose: ["8000"]               # no ports: published to the host
    networks: [internal]           # only the Astro container can reach it
flowchart LR
    I[Internet] --> T[Cloudflare tunnel]
    T --> A[Astro site + editor]
    A -->|internal network| S[Scraper :8000]

expose instead of ports means the scraper is never published on the host, and its only network is the internal one. From the internet, it doesn’t exist. Python stayed Python, the site stayed Node, and the tunnel only ever pointed at the Astro container.

What the review found

The editor talked to the scraper through an Astro route, /api/scrape-proxy, which forwarded the URL to http://scraper-service:8000. Two facts combined badly:

  1. The proxy was public. It sat under /api/, outside the editor’s protection: the same gap as the write endpoints.
  2. The scraper fetched any URL. Whatever address it received, it requested, from inside the server’s network.

Together, that’s server-side request forgery (SSRF): anyone on the internet could ask the server to fetch an address of their choosing, including addresses only the server can reach, such as other containers on the internal network or services on the host. The scraper being internal didn’t help, because something public was willing to talk to it on the attacker’s behalf.

“Internal only” protects a service from direct access. It doesn’t protect it from being used.

The fix

1. Authenticate the proxy. It’s an editor action, so it falls under the same rule as every other write: no session, no request.

2. Allow only the sources it’s meant for. The editor imports from a known listing site, so the scraper accepts only that:

from urllib.parse import urlparse

ALLOWED_HOSTS = {"listings.example"}

def check_url(url: str) -> None:
    parts = urlparse(url)
    if parts.scheme != "https" or parts.hostname not in ALLOWED_HOSTS:
        raise HTTPException(status_code=400, detail="source not allowed")

An allowlist is far stronger than trying to block “internal” addresses, which attackers get around with redirects, alternative IP formats and DNS tricks.

3. Don’t follow redirects blindly, and cap the response size and time, so a slow or huge page can’t tie up the service.

4. Give the scraper nothing to steal. No credentials in its environment, no access to the database: a service that only fetches public pages should have nothing worth reaching through it.

What I’d keep

  • A microservice for a different language was right: the Python code was reused as-is, isolated from the site.
  • expose + an internal network is the correct default for any helper service.
  • Automate the tedious part, keep humans on the facts. Images automated, prices and descriptions manual.

Takeaways

  • Internal networks stop direct access, not misuse. Check what forwards to them.
  • A URL-fetching endpoint is an SSRF risk until proven otherwise.
  • Allowlist sources; don’t try to blocklist the internal network.
  • Every editor action needs auth, including the helpers.