Subject: Crawler bots are downloading substitutes
From: Leo Famulari <[email-masked]>
To: [email-masked]
Date: Mon, 6 Dec 2021 16:20:55 -0500
List-Id: <debbugs-submit.debbugs.gnu.org>

I noticed that some bots are downloading substitutes from
ci.guix.gnu.org.

We should add a robots.txt file to reduce this waste.

Specifically, I see bots from Bing and Semrush:

https://www.bing.com/bingbot.htm
https://www.semrush.com/bot.html