GNU bug report logs - #68920
Issues with issues.guix.gnu.org (502 Bad Gateway)

Previous Next

To reply to this bug, email your comments to 68920 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Christina O'Donnell <cdo <at> mutix.org>
To: guix-devel <at> gnu.org, bug-guix <at> gnu.org
Subject: Issues with issues.guix.gnu.org (502 Bad Gateway)
Date: Sat, 3 Feb 2024 20:34:37 +0000

Hi Guix,

From my machine[1] connecting to https://issues.guix.gnu.org/ results 
in, after 130 seconds[2], a 502 bad gateway. It's been having issues for 
over a week, but I only just found a need to test it.

I couldn't see an issue about it on debbugs so I thought it prudent to 
raise an issue.

Reproduced with curl, firefox, and ungoogled-chromium on the main page 
and on specific issues.

It appears to be under a lot of load.

Kind regards,
 - Christina

[1] based in Cambridge, UK, and via my VPS in Dusseldorf, Germany on 
2024-02-03 19:30 UTC+0

[2] Rather consistently 130-140 seconds from both UK and Dusseldorf. 
Apparently the server is in Berlin.

Message #8 received at 68920 <at> debbugs.gnu.org (full text, mbox):

From: Felix Lechner <felix.lechner <at> lease-up.com>
To: Christina O'Donnell <cdo <at> mutix.org>
Cc: guix-devel <at> gnu.org, 68920 <at> debbugs.gnu.org
Subject: Re: Issues with issues.guix.gnu.org (502 Bad Gateway)
Date: Sat, 03 Feb 2024 21:33:09 -0800

Hi Christina,

On Sat, Feb 03 2024, Christina O'Donnell wrote:

> connecting to https://issues.guix.gnu.org/ results in ... a 502 bad
> gateway.

You are welcome to use my Mumi clone at mumi.juix.org.

Kind regards
Felix

Message #11 received at 68920 <at> debbugs.gnu.org (full text, mbox):

From: Christina O'Donnell <cdo <at> mutix.org>
To: Felix Lechner <felix.lechner <at> lease-up.com>
Cc: guix-devel <at> gnu.org, 68920 <at> debbugs.gnu.org
Subject: Re: Issues with issues.guix.gnu.org (502 Bad Gateway)
Date: Sun, 4 Feb 2024 10:56:08 +0000

Hi Felix,

> You are welcome to use my Mumi clone at mumi.juix.org.
Bookmarked, thanks!

Looks like issues.guix.gnu.org is back up again.

Kind regards,
 - Christina

On 04/02/2024 05:33, Felix Lechner wrote:
> Hi Christina,
>
> On Sat, Feb 03 2024, Christina O'Donnell wrote:
>
>> connecting to https://issues.guix.gnu.org/ results in ... a 502 bad
>> gateway.
>
>
> Kind regards
> Felix

Message #14 received at 68920 <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: Christina O'Donnell <cdo <at> mutix.org>
Cc: guix-devel <at> gnu.org, 68920 <at> debbugs.gnu.org
Subject: Re: bug#68920: Issues with issues.guix.gnu.org (502 Bad Gateway)
Date: Tue, 06 Feb 2024 09:22:43 -0500

Hi Christina,

Christina O'Donnell <cdo <at> mutix.org> writes:

> Hi Guix,
>
> From my machine[1] connecting to https://issues.guix.gnu.org/ results
> in, after 130 seconds[2], a 502 bad gateway. It's been having issues
> for over a week, but I only just found a need to test it.
>
> I couldn't see an issue about it on debbugs so I thought it prudent to
> raise an issue.
>
> Reproduced with curl, firefox, and ungoogled-chromium on the main page
> and on specific issues.
>
> It appears to be under a lot of load.

Thanks for the report.  It occurred a few times in the past weeks, where
the 'mumi' service had to be restarted on Berlin.  Let's keep this open
to see if it'll occur again.  Otherwise I'll close it in a week or two.

-- 
Thanks,
Maxim

Message #17 received at 68920 <at> debbugs.gnu.org (full text, mbox):

From: Andreas Enge <andreas <at> enge.fr>
To: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
Cc: Christina O'Donnell <cdo <at> mutix.org>, guix-devel <at> gnu.org,
 68920 <at> debbugs.gnu.org
Subject: Re: bug#68920: Issues with issues.guix.gnu.org (502 Bad Gateway)
Date: Tue, 6 Feb 2024 16:55:43 +0100

Am Tue, Feb 06, 2024 at 09:22:43AM -0500 schrieb Maxim Cournoyer:
> Thanks for the report.  It occurred a few times in the past weeks, where
> the 'mumi' service had to be restarted on Berlin.  Let's keep this open
> to see if it'll occur again.  Otherwise I'll close it in a week or two.

It has happened for me last evening, so it does not seem to be definitely
fixed.

Andreas

Message #20 received at 68920 <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: Andreas Enge <andreas <at> enge.fr>
Cc: Christina O'Donnell <cdo <at> mutix.org>, 68920 <at> debbugs.gnu.org
Subject: Re: bug#68920: Issues with issues.guix.gnu.org (502 Bad Gateway)
Date: Tue, 06 Feb 2024 20:31:15 -0500

Hi,

Andreas Enge <andreas <at> enge.fr> writes:

> Am Tue, Feb 06, 2024 at 09:22:43AM -0500 schrieb Maxim Cournoyer:
>> Thanks for the report.  It occurred a few times in the past weeks, where
>> the 'mumi' service had to be restarted on Berlin.  Let's keep this open
>> to see if it'll occur again.  Otherwise I'll close it in a week or two.
>
> It has happened for me last evening, so it does not seem to be definitely
> fixed.
>
> Andreas

OK.  Next time it happens, let's try to retrieve /var/log/mumi.log,
perhaps a strace of its process, and post it here.

-- 
Thanks,
Maxim

Message #25 received at 68920 <at> debbugs.gnu.org (full text, mbox):

From: Arun Isaac <arunisaac <at> systemreboot.net>
To: 68920 <at> debbugs.gnu.org
Cc: Christina O'Donnell <cdo <at> mutix.org>, Andreas Enge <andreas <at> enge.fr>,
 Maxim Cournoyer <maxim.cournoyer <at> gmail.com>,
 Felix Lechner <felix.lechner <at> lease-up.com>
Subject: Re: Issues with issues.guix.gnu.org (502 Bad Gateway)
Date: Mon, 10 Mar 2025 16:02:36 +0000

Hi,

I have tried load-testing issues.guix.gnu.org using siege[1] and these 502
errors show up when mumi is under some load. I have now added nginx
caching[2] of mumi endpoints to help mitigate this. It's not a perfect
solution, but this should be a lot less likely to occur now. I'll close
this issue in a month unless it shows up again.

[1]: https://www.joedog.org/siege-manual/
[2]:
https://git.savannah.gnu.org/cgit/guix/maintenance.git/commit/?id=5f3743d35fd2b22928ef3699c2308b92b02a67fa

If this occurs again in the future, we should consider adding more mumi
processes so that nginx will always have at least one mumi process to
reverse proxy to. We could also consider pre-generating the issue pages
and serving them directly with nginx.

Regards,
Arun

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.