about summary refs log tree commit diff
path: root/robots.txt (follow)
Commit message (Collapse)AuthorAge
* robots.txt: disallow access to snapshotsJason A. Donenfeld2013-08-12
My dmesg is filled with the oom killer bringing down processes while the Bingbot downloads every snapshot for every commit of the Linux kernel in tar.xz format. Sure, I should be running with memory limits, and now I'm using cgroups, but a more general solution is to prevent crawlers from wasting resources like that in the first place. Suggested-by: Natanael Copa <ncopa@alpinelinux.org> Suggested-by: Julius Plenz <plenz@cis.fu-berlin.de> Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com>
generated by cgit-pink 1.4.1 (git 2.36.1) at 2024-09-19 15:10:31 +0000
2019-09-20Enable cookies in titleJune McEnroe Perhaps this will make it less suspicious to Google. Who knows. 2019-09-16Use sensitivity aliases in TF2June McEnroe 2019-09-16Add The Just CityJune McEnroe 2019-09-12Only GET the final redirect locationJune McEnroe 2019-09-12Consume entire bodyJune McEnroe Aborting the request and leaving data around may be causing intermittent errors. Just discard the rest of the data. 2019-09-10Add title -v flagJune McEnroe 2019-09-10Use curl error bufferJune McEnroe 2019-09-10Set Accept-Encoding in titleJune McEnroe Because apparently it's fine for servers to respond with Content-Encoding you didn't ask for, and curl won't decode it if you didn't ask for it. 2019-09-08Set title User-AgentJune McEnroe Some things don't like you if you don't send one. 2019-09-07Add -x flag to titleJune McEnroe 2019-09-07Ignore SIGPIPE in relayJune McEnroe Allows restarting consumers safely. 2019-09-07Add A Memory Called EmpireJune McEnroe 2019-09-05Handle lack of Content-TypeJune McEnroe 2019-09-05Use CURLINFO_CONTENT_TYPEJune McEnroe Oops, didn't see this. 2019-09-05Decode entities in titlesJune McEnroe 2019-09-05Print title as soon as it's availableJune McEnroe 2019-09-05Use CURL_PREFIX to set flagsJune McEnroe 2019-09-05Add titleJune McEnroe 2019-09-04Add Avorter n'est pas tuerJune McEnroe 2019-08-29Unset executable on shell scriptsJune McEnroe 2019-08-29Add long-missing setopt to bin.7June McEnroe 2019-08-29Add editJune McEnroe