GNU bug report logs - #52338
Crawler bots are downloading substitutes

Previous Next

Full log

View this message in rfc822 format

From: help-debbugs <at> gnu.org (GNU bug Tracking System)
To: Mathieu Othacehe <othacehe <at> gnu.org>
Cc: tracker <at> debbugs.gnu.org
Subject: bug#52338: closed (Crawler bots are downloading substitutes)
Date: Sun, 19 Dec 2021 16:54:02 +0000

[Message part 1 (text/plain, inline)]

Your message dated Sun, 19 Dec 2021 17:53:27 +0100
with message-id <87wnk0pmd4.fsf <at> gnu.org>
and subject line Re: bug#52338: Crawler bots are downloading substitutes
has caused the debbugs.gnu.org bug report #52338,
regarding Crawler bots are downloading substitutes
to be marked as done.

(If you believe you have received this mail in error, please contact
help-debbugs <at> gnu.org.)


-- 
52338: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=52338
GNU Bug Tracking System
Contact help-debbugs <at> gnu.org with problems

[Message part 2 (message/rfc822, inline)]

From: Leo Famulari <leo <at> famulari.name>
To: bug-guix <at> gnu.org
Subject: Crawler bots are downloading substitutes
Date: Mon, 6 Dec 2021 16:20:55 -0500

I noticed that some bots are downloading substitutes from
ci.guix.gnu.org.

We should add a robots.txt file to reduce this waste.

Specifically, I see bots from Bing and Semrush:

https://www.bing.com/bingbot.htm
https://www.semrush.com/bot.html

[Message part 3 (message/rfc822, inline)]

From: Mathieu Othacehe <othacehe <at> gnu.org>
To: 52338-done <at> debbugs.gnu.org
Subject: Re: bug#52338: Crawler bots are downloading substitutes
Date: Sun, 19 Dec 2021 17:53:27 +0100

> Thanks to both of you,

And closing!

Mathieu

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.