When crawlers overwhelm a public service
A discussion of crawler load on a Git service surfaced rate limiting, suspicious request patterns, and the cost of open access.
The takeaway
Public access does not mean unlimited capacity.
The context
Participants reported repetitive request paths on their own sites. to11mtm suggested rate limits that respond to request intensity. This is a commenter’s suggestion; the original service failure was not reproduced here. It is a starting point for reviewing capacity and access policies.
Steps worth trying
Editorial suggestions, not promises made by the original author.
- 01
Inspect paths, frequency, latency, and errors to identify costly requests.
- 02
Set reasonable limits for expensive endpoints and check the effect on real users.
- 03
Combine caching and alerts with routine capacity review.
Where this applies — and where it may not
Validate rate limits against real workloads. An IP address is not always one visitor; do not copy untested assumptions from comments.
From the discussion
“Most pragmatically, I'd start by suggesting the best option is a combination of nonlinear rate limiting”
@to11mtm · View source
Amazon's AI crawler is making my Git server unstable
@OptionOfT · 607 points · 246 comments
Read the Hacker News threadSource: OpenIndex Hacker News dataset (ODC-By). Original content belongs to its authors. Summaries, translations, and suggested steps are AI-assisted editorial work. Read the sources; individual outcomes are not guarantees.