On a whim, I recently migrated all of my Git repositories to a cgit instance on WIREDNET. It's caused some headaches because the server-side rendering can be quite resource intensive. Mere minutes after setting it up, the fans on my server spun up because the repos were being scraped by ClaudeBot and friends. I hadn't linked to it from anywhere, so it's likely they found it from LetsEncrypt's issuance logs. It was easy enough to mitigate the scraping by putting a robots.txt up, and I was able to forget about it for a month or so, until I was sitting at work and got an alert from Gatus that several of my services were down. I checked the logs and saw that Apache was being hammered by 47.76.0.0/14 (no name provided) and 47.80.0.0/13 (Alibaba Cloud LLC), which I blocked at the firewall level. This was enough to mitigate the attack, but I was still seeing an unusual amount of traffic in my Apache logs. cgit was, once again, being aggressively scraped. Most of my repositories are small, but I had a mirror of Guix from when I was doing GSOC, and forks of a few other popular repositories, and the crawlers were just going through every single link generated by cgit (basically encompassing the entire source tree for every single commit in the repository). I was reminded of Konstantin Ryabitsev's blog post "Creepy Crawlies", which contains this helpful piece of intuition: "if [an IP is] asking for an old commit in a random old fork, it's probably not a real developer trying to do their work." I don't see any reason that someone would be looking that far into a Guix mirror that's several years out of date, so I wrote a fail2ban rule.

read more →