Subject: bug#62491: [berlin] certbot renewal appears to be broken

From: Maxim Cournoyer <[email-masked]>
Date: Mon, 27 Mar 2023 17:05:50 -0400

Hi,

The TLS cert of https://disarchive.guix.gnu.org/ expired today.  Looking
at /var/log/mcron.log on Berlin, we see that the last certbot renew job
failed like so:

--8<---------------cut here---------------start------------->8---
2023-03-24 00:30:00 127768 certbot renew --webroot --webroot-path /var/www: running...
2023-03-24 00:30:02 127768 certbot renew --webroot --webroot-path /var/www: Saving debug log to /var/log/letsencrypt/letsencrypt.log
2023-03-24 00:30:02 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:30:02 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:30:02 127768 certbot renew --webroot --webroot-path /var/www: Processing /etc/letsencrypt/renewal/bootstrappable.org.conf
2023-03-24 00:30:02 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:30:02 127768 certbot renew --webroot --webroot-path /var/www: Certificate not yet due for renewal
2023-03-24 00:30:02 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:30:02 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:30:02 127768 certbot renew --webroot --webroot-path /var/www: Processing /etc/letsencrypt/renewal/ci.guix.gnu.org.conf
2023-03-24 00:30:02 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:30:02 127768 certbot renew --webroot --webroot-path /var/www: Certificate not yet due for renewal
2023-03-24 00:30:02 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:30:02 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:30:02 127768 certbot renew --webroot --webroot-path /var/www: Processing /etc/letsencrypt/renewal/disarchive.guix.gnu.org.conf
2023-03-24 00:30:02 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:32:54 127768 certbot renew --webroot --webroot-path /var/www: Renewing an existing certificate for disarchive.guix.gnu.org
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www: Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www:   Domain: disarchive.guix.gnu.org
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www:   Type:   unauthorized
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www:   Detail: 141.80.181.40: Invalid response from https://disarchive.guix.gnu.org/.well-known/acme-challenge/O1kK3tsJtH0r9RwvbCIFhHagJhBwewV3Ka0NPW86nAI: 404
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www: Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot. Ensure that the listed domains serve their content from the provided --webroot-path/-w and that files created there can be downloaded from the internet.
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www: Failed to renew certificate disarchive.guix.gnu.org with error: Some challenges have failed.
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www: Processing /etc/letsencrypt/renewal/dump.guix.gnu.org.conf
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www: Certificate not yet due for renewal
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www: Processing /etc/letsencrypt/renewal/guix.gnu.org.conf
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:10 127768 certbot renew --webroot --webroot-path /var/www: Renewing an existing certificate for guix.gnu.org
2023-03-24 00:33:18 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:18 127768 certbot renew --webroot --webroot-path /var/www: Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:
2023-03-24 00:33:18 127768 certbot renew --webroot --webroot-path /var/www:   Domain: guix.gnu.org
2023-03-24 00:33:18 127768 certbot renew --webroot --webroot-path /var/www:   Type:   unauthorized
2023-03-24 00:33:18 127768 certbot renew --webroot --webroot-path /var/www:   Detail: 2a0c:e300::58: Invalid response from https://guix.gnu.org/.well-known/acme-challenge/_PlXq5i2BRw23Ui1Yl4rLtyB2aSDnUNMZXurCWBwH-k: 404
2023-03-24 00:33:18 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:18 127768 certbot renew --webroot --webroot-path /var/www: Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot. Ensure that the listed domains serve their content from the provided --webroot-path/-w and that files created there can be downloaded from the internet.
2023-03-24 00:33:18 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:18 127768 certbot renew --webroot --webroot-path /var/www: Failed to renew certificate guix.gnu.org with error: Some challenges have failed.
2023-03-24 00:33:18 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:18 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:18 127768 certbot renew --webroot --webroot-path /var/www: Processing /etc/letsencrypt/renewal/guix.info.conf
2023-03-24 00:33:18 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:19 127768 certbot renew --webroot --webroot-path /var/www: Renewing an existing certificate for guix.info and www.guix.info
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www: Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www:   Domain: guix.info
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www:   Type:   unauthorized
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www:   Detail: 141.80.181.40: Invalid response from https://guix.gnu.org/.well-known/acme-challenge/O6y6aqSvLdjdS77MgaEhh7sN7Q75OQX3Jz69xnT4qnY: 404
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www:   Domain: www.guix.info
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www:   Type:   unauthorized
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www:   Detail: 141.80.181.40: Invalid response from https://guix.gnu.org/.well-known/acme-challenge/lCioloihdJF6xwwTBg6cSNFjRearp4EBZBWcjkznrUE: 404
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www: Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot. Ensure that the listed domains serve their content from the provided --webroot-path/-w and that files created there can be downloaded from the internet.
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www: Failed to renew certificate guix.info with error: Some challenges have failed.
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www: Processing /etc/letsencrypt/renewal/issues.guix.gnu.org.conf
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www: Certificate not yet due for renewal
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www: Processing /etc/letsencrypt/renewal/issues.guix.info.conf
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:26 127768 certbot renew --webroot --webroot-path /var/www: Renewing an existing certificate for issues.guix.info and 3 more domains
2023-03-24 00:33:39 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:39 127768 certbot renew --webroot --webroot-path /var/www: Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:
2023-03-24 00:33:39 127768 certbot renew --webroot --webroot-path /var/www:   Domain: guix.info
2023-03-24 00:33:39 127768 certbot renew --webroot --webroot-path /var/www:   Type:   unauthorized
2023-03-24 00:33:39 127768 certbot renew --webroot --webroot-path /var/www:   Detail: 141.80.181.40: Invalid response from https://guix.gnu.org/.well-known/acme-challenge/Yv4KpoYC95LzGsM5IPTE68vf6lLfNHVK5kMUocSuDW0: 404
2023-03-24 00:33:39 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:39 127768 certbot renew --webroot --webroot-path /var/www: Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot. Ensure that the listed domains serve their content from the provided --webroot-path/-w and that files created there can be downloaded from the internet.
2023-03-24 00:33:39 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:39 127768 certbot renew --webroot --webroot-path /var/www: Failed to renew certificate issues.guix.info with error: Some challenges have failed.
2023-03-24 00:33:39 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:39 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:39 127768 certbot renew --webroot --webroot-path /var/www: Processing /etc/letsencrypt/renewal/monitor.guix.gnu.org.conf
2023-03-24 00:33:39 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:39 127768 certbot renew --webroot --webroot-path /var/www: Renewing an existing certificate for monitor.guix.gnu.org
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www:   Domain: monitor.guix.gnu.org
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www:   Type:   unauthorized
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www:   Detail: 141.80.181.40: Invalid response from https://monitor.guix.gnu.org/.well-known/acme-challenge/_wxH92e9QQag7TEYdqsA4-C-5pE5DnUd6pzMvQWzWNU: 400
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot. Ensure that the listed domains serve their content from the provided --webroot-path/-w and that files created there can be downloaded from the internet.
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: Failed to renew certificate monitor.guix.gnu.org with error: Some challenges have failed.
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: Processing /etc/letsencrypt/renewal/www.guixwl.org-0001.conf
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: Certificate not yet due for renewal
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: Processing /etc/letsencrypt/renewal/www.guixwl.org.conf
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: Certificate not yet due for renewal
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: The following certificates are not due for renewal yet:
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www:   /etc/letsencrypt/live/bootstrappable.org/fullchain.pem expires on 2023-05-14 (skipped)
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www:   /etc/letsencrypt/live/ci.guix.gnu.org/fullchain.pem expires on 2023-06-04 (skipped)
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www:   /etc/letsencrypt/live/dump.guix.gnu.org/fullchain.pem expires on 2023-06-04 (skipped)
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www:   /etc/letsencrypt/live/issues.guix.gnu.org/fullchain.pem expires on 2023-06-04 (skipped)
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www:   /etc/letsencrypt/live/www.guixwl.org-0001/fullchain.pem expires on 2023-06-04 (skipped)
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www:   /etc/letsencrypt/live/www.guixwl.org/fullchain.pem expires on 2023-06-04 (skipped)
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: All renewals failed. The following certificates could not be renewed:
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www:   /etc/letsencrypt/live/disarchive.guix.gnu.org/fullchain.pem (failure)
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www:   /etc/letsencrypt/live/guix.gnu.org/fullchain.pem (failure)
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www:   /etc/letsencrypt/live/guix.info/fullchain.pem (failure)
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www:   /etc/letsencrypt/live/issues.guix.info/fullchain.pem (failure)
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www:   /etc/letsencrypt/live/monitor.guix.gnu.org/fullchain.pem (failure)
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: 5 renew failure(s), 0 parse failure(s)
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: failed after 234.635s with: (misc-error #f unclean exit status ~S (1) #f)--8<---------------cut here---------------end--------------->8---

I removed the certbot file name prefix
(/gnu/store/jnp0166xw62dafd2zgxdmvjb6yq8ak32-certbot-1.28.0/bin/) in the
above output to improve readability.

-- 
Thanks,
Maxim