aboutsummaryrefslogtreecommitdiffstats
path: root/robots.txt
Commit message (Collapse)AuthorAgeFilesLines
* robots.txt: disallow access to snapshotsJason A. Donenfeld2013-08-121-0/+3
My dmesg is filled with the oom killer bringing down processes while the Bingbot downloads every snapshot for every commit of the Linux kernel in tar.xz format. Sure, I should be running with memory limits, and now I'm using cgroups, but a more general solution is to prevent crawlers from wasting resources like that in the first place. Suggested-by: Natanael Copa <[email protected]> Suggested-by: Julius Plenz <[email protected]> Signed-off-by: Jason A. Donenfeld <[email protected]>