GNU bug report logs - #70663
nss@3.99 is really hard to build

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 70663 in the body.
You can then email your comments to 70663 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Christopher Baines <mail <at> cbaines.net>
To: bug-guix <at> gnu.org
Subject: nss <at> 3.99 is really hard to build
Date: Tue, 30 Apr 2024 10:16:46 +0100

[Message part 1 (text/plain, inline)]

nss <at> 3.99 is really hard to build, it's so hard and so important that
data.guix.gnu.org is still after two days trying to process [1]. I say
so important because you have to build nss <at> 3.99 to compute the channel
instance derivations for Guix.

1: https://data.guix.gnu.org/revision/72308f262c910977e40c2c9f350dc563c0a8437a

Looking at the next revision which has been processed [2], it's been
built on riscv64-linux as the testsuite is disabled, and it has also
built on aarch64-linux, but there's no successful build for any other
architecture.

2: https://data.guix.gnu.org/revision/9f183c3627a006e8fd3bb9708448bc05a6204e6d/package/nss/3.99.0?locale=en_US.UTF-8

I think there's two issues here, was this spotted before merging, and
what if anything can be done about this now. Where there's not a
substitute available for nss <at> 3.99, this will affect guix pull/guix
time-machine, e.g.

  → guix time-machine --commit=72308f262c910977e40c2c9f350dc563c0a8437a -- describe
  Updating channel 'guix' from Git repository at 'https://git.savannah.gnu.org/git/guix.git'...
  substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 100.0%
  substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 100.0%
  substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 100.0%
   nss-3.99.tar.xz  55.2MiB                                                                                             13.7MiB/s 00:04 ▕██████████████████▏ 100.0%
  building /gnu/store/8379qa0y6s7ssjr8gplm5fyw9r5pnxhn-nss-3.99.0.drv...

[signature.asc (application/pgp-signature, inline)]

Message #8 received at 70663 <at> debbugs.gnu.org (full text, mbox):

From: Christopher Baines <mail <at> cbaines.net>
To: 70663 <at> debbugs.gnu.org
Subject: Re: nss <at> 3.99 is really hard to build
Date: Wed, 01 May 2024 11:11:31 +0100

[Message part 1 (text/plain, inline)]

Christopher Baines <mail <at> cbaines.net> writes:

> nss <at> 3.99 is really hard to build, it's so hard and so important that
> data.guix.gnu.org is still after two days trying to process [1]. I say
> so important because you have to build nss <at> 3.99 to compute the channel
> instance derivations for Guix.
>
> 1: https://data.guix.gnu.org/revision/72308f262c910977e40c2c9f350dc563c0a8437a
>
> Looking at the next revision which has been processed [2], it's been
> built on riscv64-linux as the testsuite is disabled, and it has also
> built on aarch64-linux, but there's no successful build for any other
> architecture.
>
> 2: https://data.guix.gnu.org/revision/9f183c3627a006e8fd3bb9708448bc05a6204e6d/package/nss/3.99.0?locale=en_US.UTF-8
>
> I think there's two issues here, was this spotted before merging, and
> what if anything can be done about this now. Where there's not a
> substitute available for nss <at> 3.99, this will affect guix pull/guix
> time-machine, e.g.
>
>   → guix time-machine --commit=72308f262c910977e40c2c9f350dc563c0a8437a -- describe
>   Updating channel 'guix' from Git repository at 'https://git.savannah.gnu.org/git/guix.git'...
>   substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 100.0%
>   substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 100.0%
>   substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 100.0%
>    nss-3.99.tar.xz  55.2MiB                                                                                             13.7MiB/s 00:04 ▕██████████████████▏ 100.0%
>   building /gnu/store/8379qa0y6s7ssjr8gplm5fyw9r5pnxhn-nss-3.99.0.drv...

Looking at the build failures for x86_64-linux, it seems that there's
just one test failure. There's a threshold of less than 5 seconds, and
it takes 5 to 7 seconds to complete. This probably isn't helped by using
faketime.

Here's an upstream bug [3] where they raised the threshold a bit, but
this isn't enough for our use case.

3: https://bugzilla.mozilla.org/show_bug.cgi?id=1835357

I've sent a patch here which increases the threshold by a lot:

  https://issues.guix.gnu.org/70693

[signature.asc (application/pgp-signature, inline)]

Message #11 received at 70663 <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: Christopher Baines <mail <at> cbaines.net>
Cc: 70663 <at> debbugs.gnu.org
Subject: Re: bug#70663: nss <at> 3.99 is really hard to build
Date: Wed, 01 May 2024 12:54:08 -0400

Hi Chris,

Christopher Baines <mail <at> cbaines.net> writes:

> nss <at> 3.99 is really hard to build, it's so hard and so important that
> data.guix.gnu.org is still after two days trying to process [1]. I say
> so important because you have to build nss <at> 3.99 to compute the channel
> instance derivations for Guix.

I agree that the nss test suite takes a ridiculous amount of time to run
(multiple hours on a fast machine IIRC).  Are we missing a '--fast' test
flag or something to make it run in a more reasonable amount of time?

-- 
Thanks,
Maxim

Message #14 received at 70663 <at> debbugs.gnu.org (full text, mbox):

From: Christopher Baines <mail <at> cbaines.net>
To: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
Cc: 70663 <at> debbugs.gnu.org
Subject: Re: bug#70663: nss <at> 3.99 is really hard to build
Date: Wed, 01 May 2024 18:14:48 +0100

[Message part 1 (text/plain, inline)]

Maxim Cournoyer <maxim.cournoyer <at> gmail.com> writes:

> Hi Chris,
>
> Christopher Baines <mail <at> cbaines.net> writes:
>
>> nss <at> 3.99 is really hard to build, it's so hard and so important that
>> data.guix.gnu.org is still after two days trying to process [1]. I say
>> so important because you have to build nss <at> 3.99 to compute the channel
>> instance derivations for Guix.
>
> I agree that the nss test suite takes a ridiculous amount of time to run
> (multiple hours on a fast machine IIRC).  Are we missing a '--fast' test
> flag or something to make it run in a more reasonable amount of time?

I did read some of the all.sh script used for the tests and there is
some environment variables you can set here:

  https://github.com/nss-dev/nss/blob/master/tests/all.sh#L70-L82

It seems like there are 4 "cycles", maybe we could just run the standard
cycle or at least check how long they each take.

[signature.asc (application/pgp-signature, inline)]

Message #17 received at 70663 <at> debbugs.gnu.org (full text, mbox):

From: Christina O'Donnell <cdo <at> mutix.org>
To: Christopher Baines <mail <at> cbaines.net>,
 Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
Cc: 70663 <at> debbugs.gnu.org
Subject: Re: bug#70663: nss <at> 3.99 is really hard to build
Date: Thu, 2 May 2024 21:38:45 +0100

Hi,

On 01/05/2024 18:14, Christopher Baines wrote:
> Maxim Cournoyer <maxim.cournoyer <at> gmail.com> writes:
>
>> Hi Chris,
>>
>> Christopher Baines <mail <at> cbaines.net> writes:
>>
>>> nss <at> 3.99 is really hard to build, it's so hard and so important that
>>> data.guix.gnu.org is still after two days trying to process [1]. I say
>>> so important because you have to build nss <at> 3.99 to compute the channel
>>> instance derivations for Guix.
>> I agree that the nss test suite takes a ridiculous amount of time to run
>> (multiple hours on a fast machine IIRC).  Are we missing a '--fast' test
>> flag or something to make it run in a more reasonable amount of time?
> I did read some of the all.sh script used for the tests and there is
> some environment variables you can set here:
>
>    https://github.com/nss-dev/nss/blob/master/tests/all.sh#L70-L82
>
> It seems like there are 4 "cycles", maybe we could just run the standard
> cycle or at least check how long they each take.

On my machine building natively on x86_64 I was getting approximately 63 
mins for a full test and 20 mins for just the 'standard' 'cycle'. My 
vote would be to just run 'standard' since that runs all of the tests once.

I can profile individual tests if needed to see if there's any that are 
particularly worth culling, but just the above change is an easy win 
without sacrificing too much test coverage.

Kind regards,

Christina

Message #22 received at 70663 <at> debbugs.gnu.org (full text, mbox):

From: Joshua Branson <jbranso <at> dismail.de>
To: Christina O'Donnell <cdo <at> mutix.org>
Cc: 70663 <at> debbugs.gnu.org, Christopher Baines <mail <at> cbaines.net>,
 Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
Subject: Re: bug#70663: nss <at> 3.99 is really hard to build
Date: Thu, 09 May 2024 13:01:18 -0400

Perhaps we could disable the test suite for power9 ?  At the moment guix
pull fails on power9...I believe due to this bug.

Just a thought.

Joshua

Message #25 received at 70663 <at> debbugs.gnu.org (full text, mbox):

From: Christopher Baines <mail <at> cbaines.net>
To: 70663 <at> debbugs.gnu.org, Maxim Cournoyer <maxim.cournoyer <at> gmail.com>, Ian
 Eure <ian <at> retrospec.tv>
Subject: Re: nss <at> 3.99 is really hard to build
Date: Tue, 14 May 2024 10:05:28 +0100

[Message part 1 (text/plain, inline)]

Christopher Baines <mail <at> cbaines.net> writes:

> nss <at> 3.99 is really hard to build, it's so hard and so important that
> data.guix.gnu.org is still after two days trying to process [1]. I say
> so important because you have to build nss <at> 3.99 to compute the channel
> instance derivations for Guix.
>
> 1: https://data.guix.gnu.org/revision/72308f262c910977e40c2c9f350dc563c0a8437a
>
> Looking at the next revision which has been processed [2], it's been
> built on riscv64-linux as the testsuite is disabled, and it has also
> built on aarch64-linux, but there's no successful build for any other
> architecture.
>
> 2: https://data.guix.gnu.org/revision/9f183c3627a006e8fd3bb9708448bc05a6204e6d/package/nss/3.99.0?locale=en_US.UTF-8
>
> I think there's two issues here, was this spotted before merging, and
> what if anything can be done about this now. Where there's not a
> substitute available for nss <at> 3.99, this will affect guix pull/guix
> time-machine, e.g.
>
>   → guix time-machine --commit=72308f262c910977e40c2c9f350dc563c0a8437a -- describe
>   Updating channel 'guix' from Git repository at 'https://git.savannah.gnu.org/git/guix.git'...
>   substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 100.0%
>   substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 100.0%
>   substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 100.0%
>    nss-3.99.tar.xz  55.2MiB                                                                                             13.7MiB/s 00:04 ▕██████████████████▏ 100.0%
>   building /gnu/store/8379qa0y6s7ssjr8gplm5fyw9r5pnxhn-nss-3.99.0.drv...

So with the changes in #70693 merged, this issue should be fixed going
forward, but the revisions with the broken nss are going to be affected
forever and thus the impact is going to drag on for a while. For
example, data.guix.gnu.org is going to be struggling to process the
revisions with the broken nss for a long while to come.

Before closing this bug, it would be good to understand more about how
this happened and from that try to think if anything can be done to
prevent similar issues in the future?

At least from what I can see on the issues, the problem was introduced
with the update to 3.98.0 [3] and then continued with the update to 3.99
[4]. Given the changes in 70662 were sent to guix-patches and then
merged less than 24 hours later, I'd imagine that wasn't sufficient time
for data.qa.guix.gnu.org to fail attempting to build nss.

3: https://issues.guix.gnu.org/70662
4: https://issues.guix.gnu.org/70618

Had the changes waited for longer, then these failures should have been
spotted by QA, I would guess that the revision might have failed to be
processed, and if it was processed successfully, the nss failures should
have shown up, so maybe we should start requiring [5] that not only are
changes sent to guix-patches <at> gnu.org, but that QA processes them (to
some extent) before merging?

5: https://guix.gnu.org/manual/devel/en/html_node/Managing-Patches-and-Branches.html#

Thanks,

Chris

[signature.asc (application/pgp-signature, inline)]

Message #28 received at 70663 <at> debbugs.gnu.org (full text, mbox):

From: Christina O'Donnell <cdo <at> mutix.org>
To: Christopher Baines <mail <at> cbaines.net>,
 Simon Tournier <zimon.toutoune <at> gmail.com>
Cc: Guix Devel <guix-devel <at> gnu.org>, 70662 <at> debbugs.gnu.org,
 70663 <at> debbugs.gnu.org
Subject: Re: Scheduling a new release?
Date: Tue, 14 May 2024 11:19:57 +0100

Hi,

On 08/05/2024 14:01, Christopher Baines wrote:
> I think it would be nice to have a new release, and indeed release more
> often, I think the way to get there is for less things to be broken
> between releases, such that releasing takes less effort in terms of
> testing and fixing things.
>
> To give some specific issues, I've run up against the recent issues with
> nss [1][2] and I don't think we could release with the nss package as is
> currently.
>
> 1: https://issues.guix.gnu.org/70662
> 2: https://issues.guix.gnu.org/70663

I can fix these by disabling tests, but I would prefer if someone with 
more experience packaging for guix could make a decision on it. 
Otherwise, I don't have any problem reducing the number of tests and 
disabling all tests on PowerPC at least.

I could also do some analysis if it was deemed necessary, inserting a 
patch to measure the timings of each test/cycle. Additionally, I could 
try packaging some of the versions between 0.88 and 0.98 to identify the 
exact change that could be to blame. However, both of these seem 
overkill, given the backlog of patches/issues we have left to get 
through, and the manpower we currently have to work with.

Would any of that be helpful?

> ...

Kind regards,

Christina

Message #31 received at 70663 <at> debbugs.gnu.org (full text, mbox):

From: "pelzflorian (Florian Pelz)" <pelzflorian <at> pelzflorian.de>
To: Christopher Baines <mail <at> cbaines.net>
Cc: 70663 <at> debbugs.gnu.org, Maxim Cournoyer <maxim.cournoyer <at> gmail.com>,
 Ian Eure <ian <at> retrospec.tv>
Subject: Re: bug#70663: nss <at> 3.99 is really hard to build
Date: Tue, 14 May 2024 12:36:18 +0200

Hello Christopher.

Christopher Baines <mail <at> cbaines.net> writes:
> Had the changes waited for longer, then these failures should have been
> spotted by QA, I would guess that the revision might have failed to be
> processed, and if it was processed successfully, the nss failures should
> have shown up, so maybe we should start requiring [5] that not only are
> changes sent to guix-patches <at> gnu.org, but that QA processes them (to
> some extent) before merging?
>
> 5: https://guix.gnu.org/manual/devel/en/html_node/Managing-Patches-and-Branches.html#

Yes, though note that the nss change did provide security fixes:

commit e584ff08b162c46ef587daca438e97d56bc20b32
Author: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
Date:   Wed Apr 24 11:22:30 2024 -0400

    gnu: nss: Graft with version 3.98 [security fixes].
    
    This fixes CVE-2023-5388, CVE-2023-6135 and CVE-2024-0743.
    
    * gnu/packages/nss.scm (nss) [replacement]: New field.
    (nss-3.98): Rename variable to...
    (nss/fixed): ... this.  Make it a hidden package.
    * gnu/packages/librewolf.scm (librewolf) [inputs]: Replace nss-3.98 with
    nss/fixed.
    
    Change-Id: I8cc667c53a270dfe00738bf731923f1342036624

I suppose the requirement to wait for QA should apply to security fixes
as well?

Thank you for all your work.

Regards,
Florian

Message #34 received at 70663 <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: Christopher Baines <mail <at> cbaines.net>
Cc: 70663 <at> debbugs.gnu.org, Ian Eure <ian <at> retrospec.tv>
Subject: Re: nss <at> 3.99 is really hard to build
Date: Tue, 14 May 2024 08:58:52 -0400

Hi,

Christopher Baines <mail <at> cbaines.net> writes:

[...]

>> I think there's two issues here, was this spotted before merging, and
>> what if anything can be done about this now. Where there's not a
>> substitute available for nss <at> 3.99, this will affect guix pull/guix
>> time-machine, e.g.
>>
>>   → guix time-machine --commit=72308f262c910977e40c2c9f350dc563c0a8437a -- describe
>>   Updating channel 'guix' from Git repository at 'https://git.savannah.gnu.org/git/guix.git'...
>>   substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 100.0%
>>   substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 100.0%
>>   substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 100.0%
>>    nss-3.99.tar.xz  55.2MiB                                                                                             13.7MiB/s 00:04 ▕██████████████████▏ 100.0%
>>   building /gnu/store/8379qa0y6s7ssjr8gplm5fyw9r5pnxhn-nss-3.99.0.drv...
>
> So with the changes in #70693 merged, this issue should be fixed going
> forward, but the revisions with the broken nss are going to be affected
> forever and thus the impact is going to drag on for a while. For
> example, data.guix.gnu.org is going to be struggling to process the
> revisions with the broken nss for a long while to come.
>
> Before closing this bug, it would be good to understand more about how
> this happened and from that try to think if anything can be done to
> prevent similar issues in the future?
>
> At least from what I can see on the issues, the problem was introduced
> with the update to 3.98.0 [3] and then continued with the update to 3.99
> [4]. Given the changes in 70662 were sent to guix-patches and then
> merged less than 24 hours later, I'd imagine that wasn't sufficient time
> for data.qa.guix.gnu.org to fail attempting to build nss.

I think in [3] you meant https://issues.guix.gnu.org/70569, not #70662.

Since this was security sensitive, I built it on x86_64, tested it there
to ensure that IceCat worked as expected, had others confirmed it worked
for them on #guix then pushed.

In the past, I've had more patience waiting for QA to build things, but
since this is not guaranteed (it sometimes never happened), it seems
reasonable to me to promptly push security fixes that were manually
built & tested and adjust for any breakage later, if there is any.

> 3: https://issues.guix.gnu.org/70662
> 4: https://issues.guix.gnu.org/70618
>
> Had the changes waited for longer, then these failures should have been
> spotted by QA, I would guess that the revision might have failed to be
> processed, and if it was processed successfully, the nss failures should
> have shown up, so maybe we should start requiring [5] that not only are
> changes sent to guix-patches <at> gnu.org, but that QA processes them (to
> some extent) before merging?

I have some apprehensions about that; given the QA build farm is
somewhat under-resourced for builds, I fear security changes could be
gated for longer periods of time than is reasonable to wait.  If we go
that route, I think we should dedicate more hardware first.

-- 
Thanks,
Maxim

Message #37 received at 70663 <at> debbugs.gnu.org (full text, mbox):

From: Christopher Baines <mail <at> cbaines.net>
To: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
Cc: 70663 <at> debbugs.gnu.org, Ian Eure <ian <at> retrospec.tv>
Subject: Re: nss <at> 3.99 is really hard to build
Date: Tue, 14 May 2024 14:33:51 +0100

[Message part 1 (text/plain, inline)]

Maxim Cournoyer <maxim.cournoyer <at> gmail.com> writes:

>> Before closing this bug, it would be good to understand more about how
>> this happened and from that try to think if anything can be done to
>> prevent similar issues in the future?
>>
>> At least from what I can see on the issues, the problem was introduced
>> with the update to 3.98.0 [3] and then continued with the update to 3.99
>> [4]. Given the changes in 70662 were sent to guix-patches and then
>> merged less than 24 hours later, I'd imagine that wasn't sufficient time
>> for data.qa.guix.gnu.org to fail attempting to build nss.
>
> I think in [3] you meant https://issues.guix.gnu.org/70569, not #70662.

Ah, yep.

> Since this was security sensitive, I built it on x86_64, tested it there
> to ensure that IceCat worked as expected, had others confirmed it worked
> for them on #guix then pushed.
>
> In the past, I've had more patience waiting for QA to build things, but
> since this is not guaranteed (it sometimes never happened), it seems
> reasonable to me to promptly push security fixes that were manually
> built & tested and adjust for any breakage later, if there is any.

I think pushing security fixes quickly is good, but this does set a
precedent on architecture support (only x86_64-linux matters).

For some packages (including nss in this instance), not looking at non
x86_64-linux support doesn't just affect users, the data service and
ci.guix.gnu.org were particularly affected, so for some packages it's
important to test across the "supported" systems just to ensure the
projects own tooling doesn't break.

>> 3: https://issues.guix.gnu.org/70662
>> 4: https://issues.guix.gnu.org/70618
>>
>> Had the changes waited for longer, then these failures should have been
>> spotted by QA, I would guess that the revision might have failed to be
>> processed, and if it was processed successfully, the nss failures should
>> have shown up, so maybe we should start requiring [5] that not only are
>> changes sent to guix-patches <at> gnu.org, but that QA processes them (to
>> some extent) before merging?
>
> I have some apprehensions about that; given the QA build farm is
> somewhat under-resourced for builds, I fear security changes could be
> gated for longer periods of time than is reasonable to wait.  If we go
> that route, I think we should dedicate more hardware first.

I think that's reasonable, I have been putting time in to the hardware,
but it's not been particularly easy going. The data service instances
are also still stuck on hardware I'm renting as well. In terms of QA
speed, there's the resources for data.qa.guix.gnu.org and there's the
hardware available for the bordeaux build farm.

There's also the potential to try and add more prioritisation. Currently
the data service prioritises branch revisions over patches, and newer
revisions more generally, but I guess it could prioritise security
related things. I'm not sure how to make that happen yet though, QA can
probably come up with the priorities, but I don't know how to convey
that to the data service.

[signature.asc (application/pgp-signature, inline)]

Message #40 received at 70663 <at> debbugs.gnu.org (full text, mbox):

From: Christopher Baines <mail <at> cbaines.net>
To: "pelzflorian (Florian Pelz)" <pelzflorian <at> pelzflorian.de>
Cc: 70663 <at> debbugs.gnu.org, Maxim Cournoyer <maxim.cournoyer <at> gmail.com>,
 Ian Eure <ian <at> retrospec.tv>
Subject: Re: bug#70663: nss <at> 3.99 is really hard to build
Date: Tue, 14 May 2024 14:37:35 +0100

[Message part 1 (text/plain, inline)]

"pelzflorian (Florian Pelz)" <pelzflorian <at> pelzflorian.de> writes:

> Hello Christopher.
>
> Christopher Baines <mail <at> cbaines.net> writes:
>> Had the changes waited for longer, then these failures should have been
>> spotted by QA, I would guess that the revision might have failed to be
>> processed, and if it was processed successfully, the nss failures should
>> have shown up, so maybe we should start requiring [5] that not only are
>> changes sent to guix-patches <at> gnu.org, but that QA processes them (to
>> some extent) before merging?
>>
>> 5: https://guix.gnu.org/manual/devel/en/html_node/Managing-Patches-and-Branches.html#
>
> Yes, though note that the nss change did provide security fixes:
>
> commit e584ff08b162c46ef587daca438e97d56bc20b32
> Author: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
> Date:   Wed Apr 24 11:22:30 2024 -0400
>
>     gnu: nss: Graft with version 3.98 [security fixes].
>
>     This fixes CVE-2023-5388, CVE-2023-6135 and CVE-2024-0743.
>
>     * gnu/packages/nss.scm (nss) [replacement]: New field.
>     (nss-3.98): Rename variable to...
>     (nss/fixed): ... this.  Make it a hidden package.
>     * gnu/packages/librewolf.scm (librewolf) [inputs]: Replace nss-3.98 with
>     nss/fixed.
>
>     Change-Id: I8cc667c53a270dfe00738bf731923f1342036624
>
> I suppose the requirement to wait for QA should apply to security fixes
> as well?

Well, there's a risk in not testing things across multiple
machines/architectures at least. The value of getting a security fix
merged quickly is reduced if users on some architectures/systems can't
use it.

There's always going to be trade offs, and that's fine, but the question
is more what can be done to try and improve things for the future.

[signature.asc (application/pgp-signature, inline)]

Message #43 received at 70663 <at> debbugs.gnu.org (full text, mbox):

From: Mark H Weaver <mhw <at> netris.org>
To: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>, Christopher Baines
 <mail <at> cbaines.net>
Cc: 70663 <at> debbugs.gnu.org, Ian Eure <ian <at> retrospec.tv>
Subject: Re: bug#70663: nss <at> 3.99 is really hard to build
Date: Tue, 14 May 2024 11:03:41 -0400

Hi Maxim,

Maxim Cournoyer <maxim.cournoyer <at> gmail.com> writes:

> Christopher Baines <mail <at> cbaines.net> writes:
[...]
>> At least from what I can see on the issues, the problem was introduced
>> with the update to 3.98.0 [3] and then continued with the update to 3.99
>> [4]. Given the changes in 70662 were sent to guix-patches and then
>> merged less than 24 hours later, I'd imagine that wasn't sufficient time
>> for data.qa.guix.gnu.org to fail attempting to build nss.
>
> I think in [3] you meant https://issues.guix.gnu.org/70569, not #70662.
>
> Since this was security sensitive, I built it on x86_64, tested it there
> to ensure that IceCat worked as expected, had others confirmed it worked
> for them on #guix then pushed.
[...]
>> 3: https://issues.guix.gnu.org/70662
>> 4: https://issues.guix.gnu.org/70618

Note that the IceCat package in Guix currently uses the copy of NSS that
comes bundled with the IceCat source code, so testing IceCat probably
won't tell you much about whether the standalone NSS package in Guix
works properly.

     Regards,
       Mark

Message #46 received at 70663 <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: Mark H Weaver <mhw <at> netris.org>
Cc: Christopher Baines <mail <at> cbaines.net>, 70663 <at> debbugs.gnu.org,
 Ian Eure <ian <at> retrospec.tv>
Subject: Re: bug#70663: nss <at> 3.99 is really hard to build
Date: Wed, 15 May 2024 22:44:04 -0400

Hi Mark,

Mark H Weaver <mhw <at> netris.org> writes:

> Hi Maxim,
>
> Maxim Cournoyer <maxim.cournoyer <at> gmail.com> writes:
>
>> Christopher Baines <mail <at> cbaines.net> writes:
> [...]
>>> At least from what I can see on the issues, the problem was introduced
>>> with the update to 3.98.0 [3] and then continued with the update to 3.99
>>> [4]. Given the changes in 70662 were sent to guix-patches and then
>>> merged less than 24 hours later, I'd imagine that wasn't sufficient time
>>> for data.qa.guix.gnu.org to fail attempting to build nss.
>>
>> I think in [3] you meant https://issues.guix.gnu.org/70569, not #70662.
>>
>> Since this was security sensitive, I built it on x86_64, tested it there
>> to ensure that IceCat worked as expected, had others confirmed it worked
>> for them on #guix then pushed.
> [...]
>>> 3: https://issues.guix.gnu.org/70662
>>> 4: https://issues.guix.gnu.org/70618
>
> Note that the IceCat package in Guix currently uses the copy of NSS that
> comes bundled with the IceCat source code, so testing IceCat probably
> won't tell you much about whether the standalone NSS package in Guix
> works properly.

Thanks for the heads-up.  It looks like there are now some low hanging
fruits in terms of unbundling opportunities for icecat/Icedove!

-- 
Thanks,
Maxim

Message #49 received at 70663 <at> debbugs.gnu.org (full text, mbox):

From: Ian Eure <ian <at> retrospec.tv>
To: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
Cc: Mark H Weaver <mhw <at> netris.org>, Christopher Baines <mail <at> cbaines.net>,
 70663 <at> debbugs.gnu.org
Subject: Re: bug#70663: nss <at> 3.99 is really hard to build
Date: Wed, 15 May 2024 21:02:36 -0700

Maxim Cournoyer <maxim.cournoyer <at> gmail.com> writes:

> Hi Mark,
>
> Mark H Weaver <mhw <at> netris.org> writes:
>
>> Hi Maxim,
>>
>> Maxim Cournoyer <maxim.cournoyer <at> gmail.com> writes:
>>
>>> Christopher Baines <mail <at> cbaines.net> writes:
>> [...]
>>>> At least from what I can see on the issues, the problem was 
>>>> introduced
>>>> with the update to 3.98.0 [3] and then continued with the 
>>>> update to 3.99
>>>> [4]. Given the changes in 70662 were sent to guix-patches and 
>>>> then
>>>> merged less than 24 hours later, I'd imagine that wasn't 
>>>> sufficient time
>>>> for data.qa.guix.gnu.org to fail attempting to build nss.
>>>
>>> I think in [3] you meant https://issues.guix.gnu.org/70569, 
>>> not #70662.
>>>
>>> Since this was security sensitive, I built it on x86_64, 
>>> tested it there
>>> to ensure that IceCat worked as expected, had others confirmed 
>>> it worked
>>> for them on #guix then pushed.
>> [...]
>>>> 3: https://issues.guix.gnu.org/70662
>>>> 4: https://issues.guix.gnu.org/70618
>>
>> Note that the IceCat package in Guix currently uses the copy of 
>> NSS that
>> comes bundled with the IceCat source code, so testing IceCat 
>> probably
>> won't tell you much about whether the standalone NSS package in 
>> Guix
>> works properly.
>
> Thanks for the heads-up.  It looks like there are now some low 
> hanging
> fruits in terms of unbundling opportunities for icecat/Icedove!
>

Definitely.  The LibreWolf package is probably a good reference, 
as I was able to unbundle all its library dependencies and use the 
Guix-packaged versions instead.

 — Ian

Message #52 received at 70663 <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: Christina O'Donnell <cdo <at> mutix.org>
Cc: Guix Devel <guix-devel <at> gnu.org>, 70662 <at> debbugs.gnu.org,
 Christopher Baines <mail <at> cbaines.net>, 70663 <at> debbugs.gnu.org,
 Simon Tournier <zimon.toutoune <at> gmail.com>
Subject: Re: bug#70662: Problems building nss <at> 3.98.0
Date: Tue, 21 May 2024 19:42:33 -0400

Hi,

Christina O'Donnell <cdo <at> mutix.org> writes:

> Hi,
>
> On 08/05/2024 14:01, Christopher Baines wrote:
>> I think it would be nice to have a new release, and indeed release more
>> often, I think the way to get there is for less things to be broken
>> between releases, such that releasing takes less effort in terms of
>> testing and fixing things.
>>
>> To give some specific issues, I've run up against the recent issues with
>> nss [1][2] and I don't think we could release with the nss package as is
>> currently.
>>
>> 1: https://issues.guix.gnu.org/70662
>> 2: https://issues.guix.gnu.org/70663
>
> I can fix these by disabling tests, but I would prefer if someone with
> more experience packaging for guix could make a decision on
> it. Otherwise, I don't have any problem reducing the number of tests
> and disabling all tests on PowerPC at least.
>
> I could also do some analysis if it was deemed necessary, inserting a
> patch to measure the timings of each test/cycle. Additionally, I could
> try packaging some of the versions between 0.88 and 0.98 to identify
> the exact change that could be to blame. However, both of these seem
> overkill, given the backlog of patches/issues we have left to get
> through, and the manpower we currently have to work with.
>
> Would any of that be helpful?

I just encountered the following single test failure building on
powerpc64le:

--8<---------------cut here---------------start------------->8---
time certutil -K -d /tmp/guix-build-nss-3.99.0.drv-0/nss-3.99/tests_results/security/localhost.1/bigdir -f ../tests.pw
------------- time ----------------------
real 10.32 user 10.25 sys 0.07
10 seconds
dbtests.sh: #27: certutil dump keys with explicit default trust flags  - FAILED
--8<---------------cut here---------------end--------------->8---

with the summary:

--8<---------------cut here---------------start------------->8---
SUMMARY:
========
NSS variables:
--------------
HOST=localhost
DOMSUF=localdomain
BUILD_OPT=
USE_X32=
USE_64=1
NSS_CYCLES=""
NSS_TESTS=""
NSS_SSL_TESTS="crl iopr policy normal_normal"
NSS_SSL_RUN="cov auth stapling signed_cert_timestamps scheme"
NSS_AIA_PATH=
NSS_AIA_HTTP=
NSS_AIA_OCSP=
IOPR_HOSTADDR_LIST=
PKITS_DATA=
NSS_DISABLE_HW_AES=
NSS_DISABLE_HW_SHA1=
NSS_DISABLE_HW_SHA2=
NSS_DISABLE_PCLMUL=
NSS_DISABLE_AVX=
NSS_DISABLE_ARM_NEON=
NSS_DISABLE_SSSE3=

Tests summary:
--------------
Passed:             79016
Failed:             1
Failed with core:   0
ASan failures:      0
Unknown status:     2
TinderboxPrint:Unknown: 2

error: in phase 'check': uncaught exception:
%exception #<&invoke-error program: "faketime" arguments: ("2024-01-23" "./nss/tests/all.sh") exit-status: 1 term-signal: #f stop-signal: #f> 
phase `check' failed after 36124.0 seconds
command "faketime" "2024-01-23" "./nss/tests/all.sh" failed with status 1
builder for `/gnu/store/q3cqzzd4fg384lfmk91gd6higsyhs1nq-nss-3.99.0.drv' failed with exit code 1
@ build-failed /gnu/store/q3cqzzd4fg384lfmk91gd6higsyhs1nq-nss-3.99.0.drv - 1 builder for `/gnu/store/q3cqzzd4fg384lfmk91gd6higsyhs1nq-nss-3.99.0.drv' failed with exit code 1
--8<---------------cut here---------------end--------------->8---

So I'm not sure why, but the 'certutil dump keys with explicit default
trust flags' fails on powerpc64le and should probably be disabled.
Also, 36124 s, ouch!  #70950 should help for that.

-- 
Thanks,
Maxim

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.