A Developer's Hacker News Launch Brought a Bot Flood, and robots.txt Covers Only Some of the Bots

The author of sprinklz.io, a customizable feed reader, posted it to Hacker News and got "a hundred or so upvotes." A month later the site was slow, and the slowdown "even started to get worse several days after" the post went live, according to the author's write-up.

The logs showed "a good handful of genuinely active, logged-in users" and a server under "exhorbatantly high load." The two did not match, so the author added per-endpoint request counting.

Where the requests came from

Most requests were not trying to fetch news articles. The site is a single-page application, and most callers never ran the JavaScript that renders pages. Aggregate NGINX logs showed that nearly half of all traffic came from a single user.

The author described that user as a service that watches websites for changes and produces AI-generated newsletters. It queried the site "many many times per-minute." The author blocked it. The post gives no user-agent string, no request count and no name for the service, so this piece cannot name it either.

The rest of the mix included crawlers that identify themselves: Claude bot, OpenAI bot, Google's search indexer and several others. Scanners also probed for -/wp-content/phpinfo.php, although the site does not run WordPress.

What the author changed

The fixes were ordinary. The author added a robots.txt that discourages most AI scanners while allowing Google's bot and other search traffic, then added rate limits, then spent several weeks on speed and scalability work. To test for leaks, the author also asked Claude to query the scanner paths and imitate the probes.

Which bots a robots.txt rule actually reaches

A robots.txt file is a request, and each vendor's own documentation decides how much of its traffic it covers. Both vendors split their traffic into separate agents.

Agent	Vendor	Documented purpose	robots.txt
ClaudeBot	Anthropic	Collects content for model training	Honored
Claude-User	Anthropic	Fetches pages when a user asks Claude	Blocking may reduce visibility in user-directed search
Claude-SearchBot	Anthropic	Indexes content for search quality	Blocking may reduce visibility
GPTBot	OpenAI	Crawls content that may train models	Disallowing signals no training use
OAI-SearchBot	OpenAI	Surfaces sites in ChatGPT search	Opted-out sites are not shown in search answers
ChatGPT-User	OpenAI	Visits a page when a user asks	"robots.txt rules may not apply"

OpenAI's bot documentation says of ChatGPT-User: "Because these actions are initiated by a user, robots.txt rules may not apply." Anthropic's crawler help page says its bots "respect 'do not crawl' signals by honoring industry standard directives in robots.txt."

Anthropic also supports the non-standard Crawl-delay directive, with Crawl-delay: 1 as its example, and publishes its source addresses at claude.com/crawling/bots.json. It warns that blocking by IP "may not work correctly or persistently guarantee an opt-out," and recommends robots.txt.

What this does and does not show

The sprinklz.io post is one site's logs, and the post does not include them. It does not show that AI crawlers caused the slowdown. By the author's own account, the biggest single source was a monitoring service that was blocked by hand, and no robots.txt line would have named it, since the post never says what it called itself.

What the vendor pages do establish is narrower. A site owner who wants to turn away training crawlers has a documented rule for ClaudeBot and GPTBot. A site owner who wants to turn away a user-triggered fetch has OpenAI telling them the rule may not apply, and has to fall back on rate limits, as the sprinklz.io author did.

Third-party agents sit outside both pages. Neither vendor's documentation says what an agent built by someone else identifies itself as, so the pages cannot tell an owner whether a given caller is covered. Two agent projects the site has covered, Agent37's sandbox pricing and REA's 52,186-star day, are the kind of tooling that makes that question practical.

The three controls the pages document

As of October 14, 2026, the vendor pages describe three levers for a site owner. Each one reaches a different slice of traffic.

1. A robots.txt block for ClaudeBot or GPTBot, which both vendors say they honor for training crawls.
2. A Crawl-delay line, which Anthropic supports and says it respects "where appropriate."
3. Server-side rate limits, which are the only control that applies to a caller that ignores or is exempt from the first two.

The sprinklz.io author ended up using the first and third. The post does not say whether any of the self-identifying crawlers it listed were responsible for load, only that they were present.