A Missing 404 Page Hid 273 Broken Links

My ebooks library passed the link check with a perfect 100. Then I added a 404 page, and the next crawl found 273 broken links.

The links hadn’t changed. The 404 page just made them visible.

Soft 404s on Cloudflare Pages

When a static site on Cloudflare Pages has no top-level 404.html, Pages treats it as a single-page app: every path that doesn’t match a file is answered with the homepage and status 200.

$ curl -s -o /dev/null -w "%{http_code}\n" https://ebooks.jalcocertech.com/does-not-exist
200   # before: the library page
404   # after adding src/pages/404.astro

That has three effects, none of them visible in a browser:

• Link checkers report nothing. A link to a page that doesn’t exist returns 200 and some HTML, so it counts as fine.
• Search engines see duplicate pages. Every mistyped URL is a copy of the homepage, a “soft 404”.
• Files you think you have, you don’t. My audit flagged the ebooks robots.txt as odd. It didn’t exist; Pages was serving the library page at /robots.txt.

The fix for all three is one file. In Astro, src/pages/404.astro builds to dist/404.html, and Pages serves it with a real 404 status.

What the 404 page uncovered

Almost all of the 273 broken links came from one pattern. The chapters are plain Markdown that also reads well on GitHub, so they link to each other the GitHub way:

See [the meetings chapter](./dna-effective-meetings.md) for the template.
Run [the solver](../2D-Kinematics/solver.py) to reproduce the plot.

On GitHub those work. On the site, a chapter lives at /books/<book>/<chapter>/, so the first link resolved to /books/<book>/<chapter>/dna-effective-meetings.md, and the second pointed at a script that was never published.

Fixing it at build time

I didn’t want to rewrite the chapters. They’re meant to work on GitHub too. A remark plugin rewrites the links while Astro builds the site:

• a link to a .md file that is a chapter of the same book becomes ../<chapter>/, keeping any #anchor;
• a link to any other relative file becomes plain text, because that file isn’t on the site;
• images, absolute URLs, root paths and #anchors are left alone.

function rewrite(node, parent, index, fileDir, bookDir, chapters) {
    if (node.type === 'link') {
        const url = node.url || '';
        if (url && !/^([a-z][a-z0-9+.-]*:|\/|#)/i.test(url)) {
            const [target, hash] = url.split('#');
            const rel = path.relative(bookDir, path.resolve(fileDir, decodeURI(target))).replace(/\.md$/, '');
            if (target.endsWith('.md') && chapters.has(rel)) {
                node.url = `../${rel}/${hash ? `#${hash}` : ''}`;
            } else if (target) {
                // Not published: keep the words, drop the link.
                parent.children.splice(index, 1, ...node.children);
                return node.children.length - 1;
            }
        }
    }
    // …recurse into children
}

The list of chapters comes from each book’s _metadata.json, which the site already uses to order them. The plugin is registered next to the existing ones:

markdown: {
  remarkPlugins: [remarkMath, remarkChart, remarkChapterLinks],
},

Checking the result without a crawl

Re-crawling the live site takes minutes. Checking dist/ takes a second: for every <a href> and every src in the built HTML, resolve it against the page’s URL and see whether a file exists.

for f in glob.glob('dist/**/index.html', recursive=True):
    page = '/' + os.path.relpath(os.path.dirname(f), 'dist') + '/'
    for h in re.findall(r'<a [^>]*href="([^"]+)"', open(f).read()):
        if re.match(r'^([a-z]+:|#|//)', h):
            continue
        p = 'dist' + urljoin(page, h).split('#')[0]
        if not (os.path.exists(p) or os.path.exists(p + 'index.html')):
            bad.append((page, h))

Result: 3,998 internal links and 1,044 image and script references, all resolving. The next crawl of the live site agreed: links 100, now measured against real 404s.

Takeaways

• No 404 page means no broken links can be detected. Add one before you trust any link report for a static site on Pages.
• Markdown written for GitHub needs translating for the web. Do it at build time so the source stays readable in both places.
• Check dist/ before you deploy. It’s faster than a crawl and catches the same class of mistake.

This came out of running my audit tool on my own sites; the full story is here.