GNU bug report logs - #52338
Crawler bots are downloading substitutes

Previous Next

Full log

View this message in rfc822 format

From: help-debbugs <at> gnu.org (GNU bug Tracking System)
To: Leo Famulari <leo <at> famulari.name>
Subject: bug#52338: closed (Re: bug#52338: Crawler bots are downloading
 substitutes)
Date: Sun, 19 Dec 2021 16:54:02 +0000

[Message part 1 (text/plain, inline)]

Your bug report

#52338: Crawler bots are downloading substitutes

which was filed against the guix package, has been closed.

The explanation is attached below, along with your original report.
If you require more details, please reply to 52338 <at> debbugs.gnu.org.

-- 
52338: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=52338
GNU Bug Tracking System
Contact help-debbugs <at> gnu.org with problems

[Message part 2 (message/rfc822, inline)]

From: Mathieu Othacehe <othacehe <at> gnu.org>
To: 52338-done <at> debbugs.gnu.org
Subject: Re: bug#52338: Crawler bots are downloading substitutes
Date: Sun, 19 Dec 2021 17:53:27 +0100

> Thanks to both of you,

And closing!

Mathieu

[Message part 3 (message/rfc822, inline)]

From: Leo Famulari <leo <at> famulari.name>
To: bug-guix <at> gnu.org
Subject: Crawler bots are downloading substitutes
Date: Mon, 6 Dec 2021 16:20:55 -0500

I noticed that some bots are downloading substitutes from
ci.guix.gnu.org.

We should add a robots.txt file to reduce this waste.

Specifically, I see bots from Bing and Semrush:

https://www.bing.com/bingbot.htm
https://www.semrush.com/bot.html

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.