Kernel Git host burns more CPU on scraper HTML than on legitimate clones
According to Konstantin Ryabitsev, git.kernel.org—the official Git host for the Linux kernel—now spends more CPU rendering commit pages as HTML for web crawlers than on all other legitimate access combined, including git clones. Simon Willison frames this as evidence that abusive automated scraping has become persistent background load on critical open-source infrastructure. Ryabitsev reports that across five geo-distributed nodes, fourteen CPU cores are continuously occupied serving HTML renderings of commits to bots. The account shows how publicly accessible repositories can absorb disproportionate operational cost from actors treating them as free data sources. The reporting does not quantify total scraper volume or confirm whether AI training pipelines are the primary driver versus general web crawling.
Kernel Git host burns more CPU on scraper HTML than on legitimate clones
Konstantin Ryabitsev reports that git.kernel.org now spends more CPU rendering commits for scrapers than on all other legitimate access, including git clones. Across five geo-distributed nodes, 14 CPU cores at a time do nothing but render git commits as HTML for crawlers.
Key takeaway
On git.kernel.org, scraper-driven HTML rendering of commits now consumes more CPU than every legitimate access path combined, including git clones.
What happened
Konstantin Ryabitsev reports that git.kernel.org, the official Git repository for the Linux kernel, now spends more CPU rendering commits for scrapers than on all other legitimate access, including git clones.
Simon Willison relays Ryabitsev's account that across five geo-distributed nodes, 14 CPU cores at a time do nothing but render git commits as HTML for crawlers, describing abusive crawlers as persistent background radiation on the host.
Evidence
git.kernel.org spends more CPU rendering commits for scrapers than on all other legitimate access, including git clones
Simon Willison · attributed
we spend more CPU cycles rendering commits for scrapers than we spend on all other kinds of legitimate access, including git clones
14 CPU cores across five geo-distributed nodes continuously render git commits as HTML for crawlers
Simon Willison · attributed
Across five geo-distributed nodes, 14 CPU cores at a time do nothing but render git commits as HTML for crawlers
Ryabitsev characterizes abusive crawlers as severe background radiation on git.kernel.org
Simon Willison · attributed
how bad the "background radiation" of abusive crawlers has become from the perspective of git.kernel.org
Why it matters
Operators hosting open repositories may need bot mitigation, rate limits, and tiered access before scraper traffic crowds out capacity for normal development and clone workflows.
Limits and uncertainties
The account does not provide quantified scraper volume or exact CPU metrics beyond the 14-core figure across five nodes.
The reporting does not establish whether AI training pipelines are directly responsible versus broader web scraping.
Practical implications
Infrastructure teams should treat HTML commit views as a high-cost surface and meter or restrict automated access separately from git protocol traffic.
Maintainers of public code hosts should plan capacity assuming scraper load can exceed legitimate clone and browse demand.
What to watch
Whether git.kernel.org or similar hosts publish formal rate limits, bot detection, or alternate access tiers for HTML commit rendering.
Follow-up metrics from kernel.org on scraper share of CPU after any mitigation changes.
