GNU bug report logs - #62491
[berlin] certbot renewal appears to be broken

Previous Next

To reply to this bug, email your comments to 62491 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: bug-guix <bug-guix <at> gnu.org>
Cc: guix-sysadmin <guix-sysadmin <at> gnu.org>
Subject: [berlin] certbot renewal appears to be broken
Date: Mon, 27 Mar 2023 17:05:50 -0400

Hi,

The TLS cert of https://disarchive.guix.gnu.org/ expired today.  Looking
at /var/log/mcron.log on Berlin, we see that the last certbot renew job
failed like so:

--8<---------------cut here---------------start------------->8---
2023-03-24 00:30:00 127768 certbot renew --webroot --webroot-path /var/www: running...
2023-03-24 00:30:02 127768 certbot renew --webroot --webroot-path /var/www: Saving debug log to /var/log/letsencrypt/letsencrypt.log
2023-03-24 00:30:02 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:30:02 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:30:02 127768 certbot renew --webroot --webroot-path /var/www: Processing /etc/letsencrypt/renewal/bootstrappable.org.conf
2023-03-24 00:30:02 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:30:02 127768 certbot renew --webroot --webroot-path /var/www: Certificate not yet due for renewal
2023-03-24 00:30:02 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:30:02 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:30:02 127768 certbot renew --webroot --webroot-path /var/www: Processing /etc/letsencrypt/renewal/ci.guix.gnu.org.conf
2023-03-24 00:30:02 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:30:02 127768 certbot renew --webroot --webroot-path /var/www: Certificate not yet due for renewal
2023-03-24 00:30:02 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:30:02 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:30:02 127768 certbot renew --webroot --webroot-path /var/www: Processing /etc/letsencrypt/renewal/disarchive.guix.gnu.org.conf
2023-03-24 00:30:02 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:32:54 127768 certbot renew --webroot --webroot-path /var/www: Renewing an existing certificate for disarchive.guix.gnu.org
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www: Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www:   Domain: disarchive.guix.gnu.org
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www:   Type:   unauthorized
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www:   Detail: 141.80.181.40: Invalid response from https://disarchive.guix.gnu.org/.well-known/acme-challenge/O1kK3tsJtH0r9RwvbCIFhHagJhBwewV3Ka0NPW86nAI: 404
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www: Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot. Ensure that the listed domains serve their content from the provided --webroot-path/-w and that files created there can be downloaded from the internet.
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www: Failed to renew certificate disarchive.guix.gnu.org with error: Some challenges have failed.
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www: Processing /etc/letsencrypt/renewal/dump.guix.gnu.org.conf
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www: Certificate not yet due for renewal
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www: Processing /etc/letsencrypt/renewal/guix.gnu.org.conf
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:10 127768 certbot renew --webroot --webroot-path /var/www: Renewing an existing certificate for guix.gnu.org
2023-03-24 00:33:18 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:18 127768 certbot renew --webroot --webroot-path /var/www: Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:
2023-03-24 00:33:18 127768 certbot renew --webroot --webroot-path /var/www:   Domain: guix.gnu.org
2023-03-24 00:33:18 127768 certbot renew --webroot --webroot-path /var/www:   Type:   unauthorized
2023-03-24 00:33:18 127768 certbot renew --webroot --webroot-path /var/www:   Detail: 2a0c:e300::58: Invalid response from https://guix.gnu.org/.well-known/acme-challenge/_PlXq5i2BRw23Ui1Yl4rLtyB2aSDnUNMZXurCWBwH-k: 404
2023-03-24 00:33:18 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:18 127768 certbot renew --webroot --webroot-path /var/www: Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot. Ensure that the listed domains serve their content from the provided --webroot-path/-w and that files created there can be downloaded from the internet.
2023-03-24 00:33:18 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:18 127768 certbot renew --webroot --webroot-path /var/www: Failed to renew certificate guix.gnu.org with error: Some challenges have failed.
2023-03-24 00:33:18 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:18 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:18 127768 certbot renew --webroot --webroot-path /var/www: Processing /etc/letsencrypt/renewal/guix.info.conf
2023-03-24 00:33:18 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:19 127768 certbot renew --webroot --webroot-path /var/www: Renewing an existing certificate for guix.info and www.guix.info
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www: Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www:   Domain: guix.info
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www:   Type:   unauthorized
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www:   Detail: 141.80.181.40: Invalid response from https://guix.gnu.org/.well-known/acme-challenge/O6y6aqSvLdjdS77MgaEhh7sN7Q75OQX3Jz69xnT4qnY: 404
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www:   Domain: www.guix.info
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www:   Type:   unauthorized
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www:   Detail: 141.80.181.40: Invalid response from https://guix.gnu.org/.well-known/acme-challenge/lCioloihdJF6xwwTBg6cSNFjRearp4EBZBWcjkznrUE: 404
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www: Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot. Ensure that the listed domains serve their content from the provided --webroot-path/-w and that files created there can be downloaded from the internet.
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www: Failed to renew certificate guix.info with error: Some challenges have failed.
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www: Processing /etc/letsencrypt/renewal/issues.guix.gnu.org.conf
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www: Certificate not yet due for renewal
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www: Processing /etc/letsencrypt/renewal/issues.guix.info.conf
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:26 127768 certbot renew --webroot --webroot-path /var/www: Renewing an existing certificate for issues.guix.info and 3 more domains
2023-03-24 00:33:39 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:39 127768 certbot renew --webroot --webroot-path /var/www: Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:
2023-03-24 00:33:39 127768 certbot renew --webroot --webroot-path /var/www:   Domain: guix.info
2023-03-24 00:33:39 127768 certbot renew --webroot --webroot-path /var/www:   Type:   unauthorized
2023-03-24 00:33:39 127768 certbot renew --webroot --webroot-path /var/www:   Detail: 141.80.181.40: Invalid response from https://guix.gnu.org/.well-known/acme-challenge/Yv4KpoYC95LzGsM5IPTE68vf6lLfNHVK5kMUocSuDW0: 404
2023-03-24 00:33:39 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:39 127768 certbot renew --webroot --webroot-path /var/www: Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot. Ensure that the listed domains serve their content from the provided --webroot-path/-w and that files created there can be downloaded from the internet.
2023-03-24 00:33:39 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:39 127768 certbot renew --webroot --webroot-path /var/www: Failed to renew certificate issues.guix.info with error: Some challenges have failed.
2023-03-24 00:33:39 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:39 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:39 127768 certbot renew --webroot --webroot-path /var/www: Processing /etc/letsencrypt/renewal/monitor.guix.gnu.org.conf
2023-03-24 00:33:39 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:39 127768 certbot renew --webroot --webroot-path /var/www: Renewing an existing certificate for monitor.guix.gnu.org
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www:   Domain: monitor.guix.gnu.org
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www:   Type:   unauthorized
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www:   Detail: 141.80.181.40: Invalid response from https://monitor.guix.gnu.org/.well-known/acme-challenge/_wxH92e9QQag7TEYdqsA4-C-5pE5DnUd6pzMvQWzWNU: 400
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot. Ensure that the listed domains serve their content from the provided --webroot-path/-w and that files created there can be downloaded from the internet.
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: Failed to renew certificate monitor.guix.gnu.org with error: Some challenges have failed.
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: Processing /etc/letsencrypt/renewal/www.guixwl.org-0001.conf
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: Certificate not yet due for renewal
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: Processing /etc/letsencrypt/renewal/www.guixwl.org.conf
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: Certificate not yet due for renewal
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: The following certificates are not due for renewal yet:
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www:   /etc/letsencrypt/live/bootstrappable.org/fullchain.pem expires on 2023-05-14 (skipped)
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www:   /etc/letsencrypt/live/ci.guix.gnu.org/fullchain.pem expires on 2023-06-04 (skipped)
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www:   /etc/letsencrypt/live/dump.guix.gnu.org/fullchain.pem expires on 2023-06-04 (skipped)
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www:   /etc/letsencrypt/live/issues.guix.gnu.org/fullchain.pem expires on 2023-06-04 (skipped)
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www:   /etc/letsencrypt/live/www.guixwl.org-0001/fullchain.pem expires on 2023-06-04 (skipped)
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www:   /etc/letsencrypt/live/www.guixwl.org/fullchain.pem expires on 2023-06-04 (skipped)
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: All renewals failed. The following certificates could not be renewed:
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www:   /etc/letsencrypt/live/disarchive.guix.gnu.org/fullchain.pem (failure)
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www:   /etc/letsencrypt/live/guix.gnu.org/fullchain.pem (failure)
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www:   /etc/letsencrypt/live/guix.info/fullchain.pem (failure)
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www:   /etc/letsencrypt/live/issues.guix.info/fullchain.pem (failure)
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www:   /etc/letsencrypt/live/monitor.guix.gnu.org/fullchain.pem (failure)
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: 5 renew failure(s), 0 parse failure(s)
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: failed after 234.635s with: (misc-error #f unclean exit status ~S (1) #f)--8<---------------cut here---------------end--------------->8---

I removed the certbot file name prefix
(/gnu/store/jnp0166xw62dafd2zgxdmvjb6yq8ak32-certbot-1.28.0/bin/) in the
above output to improve readability.

-- 
Thanks,
Maxim

Message #10 received at 62491 <at> debbugs.gnu.org (full text, mbox):

From: Attila Lendvai <attila <at> lendvai.name>
To: "62491 <at> debbugs.gnu.org" <62491 <at> debbugs.gnu.org>
Cc: "clement <at> lassieur.org" <clement <at> lassieur.org>
Subject: (No Subject)
Date: Thu, 04 May 2023 14:37:13 +0000

i don't think this is the same issue as #56678.

or at least what i'm seeing on my server is that the wrong certbot cmd line is generated, which then results in saving the challenge at the wrong path.

this is the mcron that gets generated:
[...]/certbot certonly -n --agree-tos --webroot -w /srv/http/ --cert-name dwim.hu -d dwim.hu --email attila <at> lendvai.name

and this what worked when i fixed the -w arg:

[...]/certbot certonly -n --agree-tos --webroot -w /srv/http/dwim.hu --cert-name dwim.hu -d dwim.hu --email attila <at> lendvai.name

i.e. the -w parameter should point to the webroot of the virtual domain, but the guix config structure does not allow setting the webroot for each <certificate-configuration>, only at their parent, i.e. in the <certbot-configuration>.

this all seems to me as if the certbot service code was assuming that the certbot script will append the domain names (specified with -d) to the webroot path, but it does not.

from the certbot log (i.e. challenge is saved at the wrong path):

"Removing /srv/http/.well-known/acme-challenge/[hash]"

the relevant code is from 2018, so certbot's behavior may very well have changed since then:

https://git.savannah.gnu.org/cgit/guix.git/commit/gnu/services/certbot.scm?id=c3215d2f9d8fa4b890e3a41ceb4404b76a7c5c49

it seems to me that the webroot field should be moved down into <certificate-configuration>.

am i right? if so i may try to patch this up.

--
- attila
PGP: 5D5F 45C7 DFCD 0A39
-- 
• attila lendvai
• PGP: 963F 5D5F 45C7 DFCD 0A39
--
“State is the name of the coldest of all cold monsters. Coldly it lies; and this lie slips from its mouth: "I, the state, am the people."”
	— Friedrich Nietzsche (1844–1900), 'Thus Spoke Zarathustra' (1885), http://j.mp/1k6pbwS

Message #13 received at 62491 <at> debbugs.gnu.org (full text, mbox):

From: Giovanni Biscuolo <g <at> xelera.eu>
To: Attila Lendvai <attila <at> lendvai.name>,
 "62491 <at> debbugs.gnu.org" <62491 <at> debbugs.gnu.org>
Cc: Ludovic Courtès <ludovic.courtes <at> inria.fr>,
 Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
Subject: bug#62491: [berlin] certbot renewal appears to be broken
Date: Wed, 22 Nov 2023 18:37:44 +0100

[Message part 1 (text/plain, inline)]

Hello Attila,

I'm starting using certbot on a new Guix System server of mine: I've not
much experience with this Guix service but I'm using certbot on other
machines so I hope I can help here.

Attila Lendvai <attila <at> lendvai.name> writes:

> i don't think this is the same issue as #56678.

AFAIU actually #56678 is (was?) caused by a duplicate certbot account:

--8<---------------cut here---------------start------------->8---

Please choose an account
Choices: ['guix-hpc.bordeaux.inria.fr <at> 2017-09-04T08:51:13Z (48c5)',
'localhost <at> 2016-12-03T21:08:38Z (00bc)']

--8<---------------cut here---------------end--------------->8---

on bayfront, probably caused by some "manual" certbot invocation (I'm
guessing, I cannot have a look to /etc/letsenctypt)

Lodo' please: has that issue (#56678) been solved and how?

The problem on berlin (#62491) is (was) due to a failed challenge:

--8<---------------cut here---------------start------------->8---

2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www: Hint: The
Certificate Authority failed to download the temporary challenge files created by Certbot.
Ensure that the listed domains serve their content from the provided --webroot-path/-w and
that files created there can be downloaded from the internet.
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www: Failed to renew
certificate disarchive.guix.gnu.org with error: Some challenges have failed.

--8<---------------cut here---------------end--------------->8---

Maxim please: has that issue (#62491) been solved and how?

[...]

> this is the mcron that gets generated:
> [...]/certbot certonly -n --agree-tos --webroot -w /srv/http/ --cert-name dwim.hu -d dwim.hu --email attila <at> lendvai.name

Did you specify a different webroot?  The default one defined in
"certbot-configuration" is "/var/www".

This is my certbot service config:

--8<---------------cut here---------------start------------->8---

	    (service certbot-service-type
		     (certbot-configuration
		      (email "giovanni <at> biscuolo.net")
		      (certificates
		       (list
			(certificate-configuration
			 (domains '("mx01.biscuolo.net")))))))

--8<---------------cut here---------------end--------------->8---

This is the certbot command that gets generated (and is scheduled in my
mcron):

--8<---------------cut here---------------start------------->8---

#!/gnu/store/x4m56h5qkim0pnvx6vgvp541mrdwdrah-guile-3.0.9/bin/guile --no-auto-compile
!#
(begin (use-modules (ice-9 match)) (let ((code 0)) (for-each (match-lambda ((name . command) (begin (format #t "Acquiring or renewing certificate: ~a~%" name) (set! code (or (apply system* command) code))))) (quote (("mx01.biscuolo.net" "/gnu/store/8vs33jaqpjkr5mzpz8syxvz2w472s5w7-certbot-2.3.0/bin/certbot" "certonly" "-n" "--agree-tos" "--webroot" "-w" "/var/www" "--cert-name" "mx01.biscuolo.net" "-d" "mx01.biscuolo.net" "--email" "giovanni <at> biscuolo.net")))) code))

--8<---------------cut here---------------end--------------->8---

Also, this is the "server" config for the generated nginx configuration:

--8<---------------cut here---------------start------------->8---

    server {
      listen 80;
      listen [::]:80;
      server_name mx01.biscuolo.net ;
      root /srv/http;
      index index.html ;
      server_tokens off;

      location /.well-known {
        root /var/www;
      }
      location / {
        return 301 https://$host$request_uri;
      }

    }

--8<---------------cut here---------------end--------------->8---

> and this what worked when i fixed the -w arg:

What was the error before you fixed the -w arg?

How was the nginx service configured?

> [...]/certbot certonly -n --agree-tos --webroot -w /srv/http/dwim.hu --cert-name dwim.hu -d dwim.hu --email attila <at> lendvai.name
>
> i.e. the -w parameter should point to the webroot of the virtual
> domain,

No: that webroot is the directory from which to serve the Let’s Encrypt
challenge/response files, it have nothing do do with the webroot of the
corresponding virtual domain served by *another* nginx service (or other
service using the certificate)

> but the guix config structure does not allow setting the webroot for
> each <certificate-configuration>, only at their parent, i.e. in the
> <certbot-configuration>.

AFAIU there is no need to set a certbot webroot for each certificate:
one webroot can serve all the challenge/response files needed for each
certificate, since certbot creates a unique subfolder in /.well-known
for each of them.

[...]

> from the certbot log (i.e. challenge is saved at the wrong path):
>
> "Removing /srv/http/.well-known/acme-challenge/[hash]"

Why do you say that challenge is in the wrong path?

It works that way :-)

[...]

WDYT?

Happy hacking! Gio'

-- 
Giovanni Biscuolo

Xelera IT Infrastructures

[signature.asc (application/pgp-signature, inline)]

Message #16 received at 62491 <at> debbugs.gnu.org (full text, mbox):

From: Attila Lendvai <attila <at> lendvai.name>
To: Giovanni Biscuolo <g <at> xelera.eu>
Cc: "62491 <at> debbugs.gnu.org" <62491 <at> debbugs.gnu.org>,
 Ludovic Courtès <ludovic.courtes <at> inria.fr>,
 Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
Subject: Re: bug#62491: [berlin] certbot renewal appears to be broken
Date: Wed, 22 Nov 2023 18:05:44 +0000

hi Giovanni,

it's been a long time, i don't remember much anymore.

but let's run a quick assert:

my server is serving multiple virtual domains (dwim.hu and lendvai.name) from completely different webroot directories. that's why i assumed that certbot needs to generate two different certificates for the two domains, and then be able to download them by accessing the same ip address through two separate domain names, and nginx serving the certificates corresponding to the domain name in the request.

did you write your answer with this in mind?

if yes, then i'll need to get back in context to answer properly.

-- 
• attila lendvai
• PGP: 963F 5D5F 45C7 DFCD 0A39
--
“Not to discuss with a man worthy of conversation is to waste the man. To discuss with a man not worthy of conversation is to waste words. The wise waste neither men nor words.”
	— Confucius (551–479 BC), 'The Analects'

Message #19 received at 62491 <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: Giovanni Biscuolo <g <at> xelera.eu>
Cc: "62491 <at> debbugs.gnu.org" <62491 <at> debbugs.gnu.org>,
 Attila Lendvai <attila <at> lendvai.name>,
 Ludovic Courtès <ludovic.courtes <at> inria.fr>
Subject: Re: bug#62491: [berlin] certbot renewal appears to be broken
Date: Wed, 22 Nov 2023 23:17:36 -0500

Hi Giovanni,

Giovanni Biscuolo <g <at> xelera.eu> writes:

> Hello Attila,
>
> I'm starting using certbot on a new Guix System server of mine: I've not
> much experience with this Guix service but I'm using certbot on other
> machines so I hope I can help here.
>
> Attila Lendvai <attila <at> lendvai.name> writes:
>
>> i don't think this is the same issue as #56678.
>
> AFAIU actually #56678 is (was?) caused by a duplicate certbot account:
>
>
> Please choose an account
> Choices: ['guix-hpc.bordeaux.inria.fr <at> 2017-09-04T08:51:13Z (48c5)',
> 'localhost <at> 2016-12-03T21:08:38Z (00bc)']
>
>
> on bayfront, probably caused by some "manual" certbot invocation (I'm
> guessing, I cannot have a look to /etc/letsenctypt)
>
> Lodo' please: has that issue (#56678) been solved and how?
>
> The problem on berlin (#62491) is (was) due to a failed challenge:
>
>
> 2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www: Hint: The
> Certificate Authority failed to download the temporary challenge files created by Certbot.
> Ensure that the listed domains serve their content from the provided --webroot-path/-w and
> that files created there can be downloaded from the internet.
> 2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www: 
> 2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www: Failed to renew
> certificate disarchive.guix.gnu.org with error: Some challenges have failed.
>
>
> Maxim please: has that issue (#62491) been solved and how?

I don't think it was truly resolved.  The problem keeps coming and
someone (usually Ludovic) has to manually run some commands get it to
cooperate (IIUC).  I've never investigated certbot nor configured such a
setup myself, so I'm not knowledgeable about it.

-- 
Thanks,
Maxim

Message #22 received at 62491 <at> debbugs.gnu.org (full text, mbox):

From: Giovanni Biscuolo <g <at> xelera.eu>
To: Attila Lendvai <attila <at> lendvai.name>
Cc: "62491 <at> debbugs.gnu.org" <62491 <at> debbugs.gnu.org>,
 Ludovic Courtès <ludovic.courtes <at> inria.fr>,
 Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
Subject: Re: bug#62491: [berlin] certbot renewal appears to be broken
Date: Thu, 23 Nov 2023 08:23:42 +0100

[Message part 1 (text/plain, inline)]

Hi Attila,

Attila Lendvai <attila <at> lendvai.name> writes:

[...]

> if yes, then i'll need to get back in context to answer properly.

In this thread I'd like to understand what is (was?) the real nature of
the bugs described, I'm just trying to collect more information

I feel we should discuss how the certbot service works in a different
thread, to stay focused on the bug report

If you need further discussion, please feel free to open a new thread on
guix-devel and Cc: me! :-)

Thanks! Gio'

-- 
Giovanni Biscuolo

Xelera IT Infrastructures

[signature.asc (application/pgp-signature, inline)]

Message #25 received at 62491 <at> debbugs.gnu.org (full text, mbox):

From: Giovanni Biscuolo <g <at> xelera.eu>
To: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
Cc: "62491 <at> debbugs.gnu.org" <62491 <at> debbugs.gnu.org>,
 Ludovic Courtès <ludovic.courtes <at> inria.fr>
Subject: Re: bug#62491: [berlin] certbot renewal appears to be broken
Date: Thu, 23 Nov 2023 08:42:31 +0100

[Message part 1 (text/plain, inline)]

Hi Maxim,

thank you for your feedback.

Maxim Cournoyer <maxim.cournoyer <at> gmail.com> writes:

[...]

>> AFAIU actually #56678 is (was?) caused by a duplicate certbot account:

[...]

>> The problem on berlin (#62491) is (was) due to a failed challenge:

I'm almost sure those are different bugs and I'm almost sure the bugs
are caused by _state_ (/etc/letsencrypt/[accounts|renewal])

[...]

> I don't think it was truly resolved.  The problem keeps coming and
> someone (usually Ludovic) has to manually run some commands get it to
> cooperate (IIUC).

Bugs like this are very difficult to reproduce and to investigate if we
wait the certs expiration and are forced to find a quick "workaround";
we should force a renewal (via CLI) before the expiration date and share
the logs to see what's happening.

I'd like to help but I'm not a sysadmin on bayfront nor on berlin.

I think this kind "statefulness issues" are affecting other users.

Happy hacking! Gio'

[...]

-- 
Giovanni Biscuolo

Xelera IT Infrastructures

[signature.asc (application/pgp-signature, inline)]

Message #28 received at 62491 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludovic.courtes <at> inria.fr>
To: Giovanni Biscuolo <g <at> xelera.eu>
Cc: "62491 <at> debbugs.gnu.org" <62491 <at> debbugs.gnu.org>,
 Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
Subject: Re: bug#62491: [berlin] certbot renewal appears to be broken
Date: Thu, 23 Nov 2023 09:46:56 +0100

Hi,

Giovanni Biscuolo <g <at> xelera.eu> skribis:

> Maxim Cournoyer <maxim.cournoyer <at> gmail.com> writes:
>
> [...]
>
>>> AFAIU actually #56678 is (was?) caused by a duplicate certbot account:
>
> [...]
>
>>> The problem on berlin (#62491) is (was) due to a failed challenge:
>
> I'm almost sure those are different bugs and I'm almost sure the bugs
> are caused by _state_ (/etc/letsencrypt/[accounts|renewal])

Indeed, that’s part of the problem.

Another example: our cerbot service offers a ‘deploy-hook’, but the
/gnu/store/… file name of that hook gets recorded somewhere in
/etc/letsencrypt and thus becomes invalid once the hook has been GC’d or
the system has been reconfigured.

>> I don't think it was truly resolved.  The problem keeps coming and
>> someone (usually Ludovic) has to manually run some commands get it to
>> cooperate (IIUC).
>
> Bugs like this are very difficult to reproduce and to investigate if we
> wait the certs expiration and are forced to find a quick "workaround";
> we should force a renewal (via CLI) before the expiration date and share
> the logs to see what's happening.
>
> I'd like to help but I'm not a sysadmin on bayfront nor on berlin.
>
> I think this kind "statefulness issues" are affecting other users.

Yeah, I think anyone running a web server on Guix System gets hit by
this issue.  I’m not super knowledgeable about certbot either so I tend
to just hack around to get things to work, which is not great.

Ludo’.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.