GNU bug report logs - #24052
Bing : crawling issues with www.gnu.org

Previous Next

Package: emacs;

Reported by: Fabrice Canel <Fabrice.Canel <at> microsoft.com>

Date: Fri, 22 Jul 2016 16:20:02 UTC

Severity: normal

Done: Nicolas Petton <nicolas <at> petton.fr>

Bug is archived. No further changes may be made.

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 24052 in the body.
You can then email your comments to 24052 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

View this report as an mbox folder, status mbox, maintainer mbox


Report forwarded to bug-gnu-emacs <at> gnu.org:
bug#24052; Package emacs. (Fri, 22 Jul 2016 16:20:02 GMT) Full text and rfc822 format available.

Acknowledgement sent to Fabrice Canel <Fabrice.Canel <at> microsoft.com>:
New bug report received and forwarded. Copy sent to bug-gnu-emacs <at> gnu.org. (Fri, 22 Jul 2016 16:20:02 GMT) Full text and rfc822 format available.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Fabrice Canel <Fabrice.Canel <at> microsoft.com>
To: "bug-gnu-emacs <at> gnu.org" <bug-gnu-emacs <at> gnu.org>
Subject: RE: Bing : crawling issues with www.gnu.org
Date: Fri, 22 Jul 2016 04:36:54 +0000
[Message part 1 (text/plain, inline)]
Contacting now bug-gnu-emacs <at> gnu.org<mailto:bug-gnu-emacs <at> gnu.org> as I didn't receive a reply from webmasters <at> gnu.org<mailto:webmasters <at> gnu.org>.
Who can help indexing your content and other pages on gnu.org?

Thanks,
Fabrice

From: Fabrice Canel
Sent: Tuesday, July 19, 2016 12:49 AM
To: 'webmasters <at> gnu.org' <webmasters <at> gnu.org>
Subject: Bing : crawling issues with www.gnu.org

Good day,

We (Bing) have an major issue crawling and indexing your site.

Our customers alerted us that https://www.gnu.org/software/emacs/ cannot be found in our results.
Investigating we believe that you are preventing bingbot crawling from our main IP ranges, we don't have issue fetching outside our IP ranges.

Can you investigate? I am willing to share our crawler bingbot IP ranges if this can help fixing this issue.
Thanks,
Fabrice Canel
Principal Program Manager
Microsoft Bing
[Message part 2 (text/html, inline)]

Information forwarded to bug-gnu-emacs <at> gnu.org:
bug#24052; Package emacs. (Fri, 22 Jul 2016 19:36:02 GMT) Full text and rfc822 format available.

Message #8 received at 24052 <at> debbugs.gnu.org (full text, mbox):

From: Nicolas Petton <nicolas <at> petton.fr>
To: Fabrice Canel <Fabrice.Canel <at> microsoft.com>, 24052 <at> debbugs.gnu.org
Subject: Re: bug#24052: Bing : crawling issues with www.gnu.org
Date: Fri, 22 Jul 2016 21:35:27 +0200
[Message part 1 (text/plain, inline)]
Fabrice Canel <Fabrice.Canel <at> microsoft.com> writes:

> We (Bing) have an major issue crawling and indexing your site.

Hi Fabrice,

> Our customers alerted us that https://www.gnu.org/software/emacs/
> cannot be found in our results.  Investigating we believe that you are
> preventing bingbot crawling from our main IP ranges, we don't have
> issue fetching outside our IP ranges.

Do you know if the issue only happens with the software/emacs/ pages, or
does it affect all pages from https://www.gnu.org?

Cheers,
Nico
[signature.asc (application/pgp-signature, inline)]

Information forwarded to bug-gnu-emacs <at> gnu.org:
bug#24052; Package emacs. (Fri, 22 Jul 2016 19:44:01 GMT) Full text and rfc822 format available.

Message #11 received at 24052 <at> debbugs.gnu.org (full text, mbox):

From: Nicolas Petton <nicolas <at> petton.fr>
To: Fabrice Canel <Fabrice.Canel <at> microsoft.com>, 24052 <at> debbugs.gnu.org
Subject: Re: bug#24052: Bing : crawling issues with www.gnu.org
Date: Fri, 22 Jul 2016 21:43:44 +0200
[Message part 1 (text/plain, inline)]
Fabrice Canel <Fabrice.Canel <at> microsoft.com> writes:

Hi again Fabrice,

I confirm the issue, all pages under software/emacs/ seem to be missing
from search results.

Also, I could not find the index page of https://www.gnu.org by
searching for "GNU" or "GNU Operating System", however, searching for
"GNU Guile", I could find in the search results the Guile homepage at
https://www.gnu.org/software/guile/guile.html, but it was still pretty
far down in the first search result page.

Cheers,
Nico
[signature.asc (application/pgp-signature, inline)]

Information forwarded to bug-gnu-emacs <at> gnu.org:
bug#24052; Package emacs. (Fri, 22 Jul 2016 20:07:02 GMT) Full text and rfc822 format available.

Message #14 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Clément Pit--Claudel <clement.pit <at> gmail.com>
To: bug-gnu-emacs <at> gnu.org, Fabrice.Canel <at> microsoft.com
Subject: Re: bug#24052: Bing : crawling issues with www.gnu.org
Date: Fri, 22 Jul 2016 16:05:49 -0400
[Message part 1 (text/plain, inline)]
Hi Fabrice,

Does this also explain why the Emacs Lisp manual isn't indexed by Bing?

Searching for "emacs lisp other display specs", for example, does not return https://www.gnu.org/software/emacs/manual/html_node/elisp/Other-Display-Specs.html

Clément.

[signature.asc (application/pgp-signature, attachment)]

Information forwarded to bug-gnu-emacs <at> gnu.org:
bug#24052; Package emacs. (Fri, 22 Jul 2016 22:23:01 GMT) Full text and rfc822 format available.

Message #17 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Fabrice Canel <Fabrice.Canel <at> microsoft.com>
To: Clément Pit--Claudel <clement.pit <at> gmail.com>,
 "bug-gnu-emacs <at> gnu.org" <bug-gnu-emacs <at> gnu.org>, Nicolas Petton
 <nicolas <at> petton.fr>
Subject: RE: bug#24052: Bing : crawling issues with www.gnu.org
Date: Fri, 22 Jul 2016 21:04:27 +0000
Hi Clément, Nicolas,

Yes, we have major issue indexing this site. This is not limited to this URL.
Any contact in webmasters <at> gnu.org?

Thanks,
Fabrice

-----Original Message-----
From: Clément Pit--Claudel [mailto:clement.pit <at> gmail.com] 
Sent: Friday, July 22, 2016 1:06 PM
To: bug-gnu-emacs <at> gnu.org; Fabrice Canel <Fabrice.Canel <at> microsoft.com>
Subject: Re: bug#24052: Bing : crawling issues with www.gnu.org

Hi Fabrice,

Does this also explain why the Emacs Lisp manual isn't indexed by Bing?

Searching for "emacs lisp other display specs", for example, does not return https://www.gnu.org/software/emacs/manual/html_node/elisp/Other-Display-Specs.html

Clément.





Information forwarded to bug-gnu-emacs <at> gnu.org:
bug#24052; Package emacs. (Thu, 28 Jul 2016 19:01:01 GMT) Full text and rfc822 format available.

Message #20 received at 24052 <at> debbugs.gnu.org (full text, mbox):

From: Ruben Rodriguez <ruben <at> fsf.org>
To: 24052 <at> debbugs.gnu.org
Subject: re: Bing : crawling issues with www.gnu.org
Date: Thu, 28 Jul 2016 14:48:31 -0400
I've checked the logs and I see access from bingbot to all our sites at
the machine where gnu.org and nongnu.org lives. I also checked the
robots.txt and they look ok.

Could you provide more details on failure to fech from bingbot, so I can
look in the logs? If so, please send it to sysadmin at gnu.org

Cheers,
-- 
Ruben Rodriguez | Senior Systems Administrator, Free Software Foundation
GPG Key: 472F4409 | https://fsf.org | https://gnu.org




Information forwarded to bug-gnu-emacs <at> gnu.org:
bug#24052; Package emacs. (Thu, 28 Jul 2016 19:23:01 GMT) Full text and rfc822 format available.

Message #23 received at 24052 <at> debbugs.gnu.org (full text, mbox):

From: Nicolas Petton <nicolas <at> petton.fr>
To: Ruben Rodriguez <ruben <at> fsf.org>, 24052 <at> debbugs.gnu.org
Subject: Re: bug#24052: Bing : crawling issues with www.gnu.org
Date: Thu, 28 Jul 2016 21:22:01 +0200
[Message part 1 (text/plain, inline)]
Ruben Rodriguez <ruben <at> fsf.org> writes:

> Could you provide more details on failure to fech from bingbot, so I can
> look in the logs? If so, please send it to sysadmin at gnu.org

Hi Ruben,

I don't think this is specific to Emacs, it seems to affect all webpages
at www.gnu.org.  Is it ok if I close this ticket and let you handle the
issue?

Cheers,
Nico
[signature.asc (application/pgp-signature, inline)]

Information forwarded to bug-gnu-emacs <at> gnu.org:
bug#24052; Package emacs. (Thu, 28 Jul 2016 19:26:02 GMT) Full text and rfc822 format available.

Message #26 received at 24052 <at> debbugs.gnu.org (full text, mbox):

From: Ruben Rodriguez <ruben <at> fsf.org>
To: Nicolas Petton <nicolas <at> petton.fr>, 24052 <at> debbugs.gnu.org
Subject: Re: bug#24052: Bing : crawling issues with www.gnu.org
Date: Thu, 28 Jul 2016 15:25:00 -0400

On 07/28/2016 03:22 PM, Nicolas Petton wrote:
> Ruben Rodriguez <ruben <at> fsf.org> writes:
> 
>> Could you provide more details on failure to fech from bingbot, so I can
>> look in the logs? If so, please send it to sysadmin at gnu.org
> 
> Hi Ruben,
> 
> I don't think this is specific to Emacs, it seems to affect all webpages
> at www.gnu.org.  Is it ok if I close this ticket and let you handle the
> issue?

Yes, that's fine.

-- 
Ruben Rodriguez | Senior Systems Administrator, Free Software Foundation
GPG Key: 472F4409 | https://fsf.org | https://gnu.org




Information forwarded to bug-gnu-emacs <at> gnu.org:
bug#24052; Package emacs. (Thu, 28 Jul 2016 19:52:01 GMT) Full text and rfc822 format available.

Message #29 received at 24052 <at> debbugs.gnu.org (full text, mbox):

From: Nicolas Petton <nicolas <at> petton.fr>
To: Ruben Rodriguez <ruben <at> fsf.org>, 24052 <at> debbugs.gnu.org
Cc: 24052-done <at> debbugs.gnu.org
Subject: Re: bug#24052: Bing : crawling issues with www.gnu.org
Date: Thu, 28 Jul 2016 21:51:02 +0200
[Message part 1 (text/plain, inline)]
Ruben Rodriguez <ruben <at> fsf.org> writes:

>> I don't think this is specific to Emacs, it seems to affect all webpages
>> at www.gnu.org.  Is it ok if I close this ticket and let you handle the
>> issue?
>
> Yes, that's fine.

Thanks, closing the issue then.

Cheers,
Nico
[signature.asc (application/pgp-signature, inline)]

Reply sent to Nicolas Petton <nicolas <at> petton.fr>:
You have taken responsibility. (Thu, 28 Jul 2016 19:52:02 GMT) Full text and rfc822 format available.

Notification sent to Fabrice Canel <Fabrice.Canel <at> microsoft.com>:
bug acknowledged by developer. (Thu, 28 Jul 2016 19:52:02 GMT) Full text and rfc822 format available.

bug archived. Request was from Debbugs Internal Request <help-debbugs <at> gnu.org> to internal_control <at> debbugs.gnu.org. (Fri, 26 Aug 2016 11:24:03 GMT) Full text and rfc822 format available.

This bug report was last modified 8 years and 350 days ago.

Previous Next


GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.