Subject: bug#52338: closed (Re: bug#52338: Crawler bots are downloading substitutes)

From: [email-masked] (GNU bug Tracking System)
Date: Sun, 19 Dec 2021 16:54:02 +0000

Your bug report

#52338: Crawler bots are downloading substitutes

which was filed against the guix package, has been closed.

The explanation is attached below, along with your original report.
If you require more details, please reply to [email-masked].

-- 
52338: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=52338
GNU Bug Tracking System
Contact [email-masked] with problems

Subject: Re: bug#52338: Crawler bots are downloading substitutes
From: Mathieu Othacehe <[email-masked]>
To: [email-masked]
Date: Sun, 19 Dec 2021 17:53:27 +0100
List-Id: <debbugs-submit.debbugs.gnu.org>

> Thanks to both of you,

And closing!

Mathieu

Subject: Crawler bots are downloading substitutes
From: Leo Famulari <[email-masked]>
To: [email-masked]
Date: Mon, 6 Dec 2021 16:20:55 -0500
List-Id: <debbugs-submit.debbugs.gnu.org>

I noticed that some bots are downloading substitutes from
ci.guix.gnu.org.

We should add a robots.txt file to reduce this waste.

Specifically, I see bots from Bing and Semrush:

https://www.bing.com/bingbot.htm
https://www.semrush.com/bot.html