GNU bug report logs - #71133
linux-libre-guix.tar.xz CI times out on aarch64-linux

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 71133 in the body.
You can then email your comments to 71133 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Richard Sent <richard <at> freakingpenguin.com>
To: bug-guix <at> gnu.org
Cc: mail <at> cbaines.net
Subject: linux-libre-guix.tar.xz CI times out on aarch64-linux
Date: Wed, 22 May 2024 18:06:24 -0400

Hi Guix!

On aarch64 platforms, the linux-libre-guix.tar.xz.drv build times out
[1].

This package is known to have a long build time [2].

Users can work around this by running the build locally with a (very,
very large) max-silent-time (~28800 seconds on my machine). Given the
impact of this issue (unable to build linux-libre on 64-bit Arm without
running a multi-hour build), I think it's worth revisiting if the
substitute server timeout should be increased again.

Alternatively, perhaps we could fetch the officially released
Linux-libre tarballs instead of computing them ourselves [3].

[1]: https://ci.guix.gnu.org/build/4711550/log/raw
[2]: https://mail.gnu.org/archive/html/guix-devel/2021-08/msg00077.html
[3]: http://linux-libre.fsfla.org/pub/linux-libre/releases/

CC'ing Christopher Baines since this deals with substitutes.

-- 
Take it easy,
Richard Sent
Making my computer weirder one commit at a time.

Message #8 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Christopher Baines <mail <at> cbaines.net>
To: Richard Sent <richard <at> freakingpenguin.com>
Cc: bug-guix <at> gnu.org
Subject: Re: linux-libre-guix.tar.xz CI times out on aarch64-linux
Date: Sun, 26 May 2024 19:26:18 +0100

[Message part 1 (text/plain, inline)]

Richard Sent <richard <at> freakingpenguin.com> writes:

> On aarch64 platforms, the linux-libre-guix.tar.xz.drv build times out
> [1].
>
> This package is known to have a long build time [2].
>
> Users can work around this by running the build locally with a (very,
> very large) max-silent-time (~28800 seconds on my machine). Given the
> impact of this issue (unable to build linux-libre on 64-bit Arm without
> running a multi-hour build), I think it's worth revisiting if the
> substitute server timeout should be increased again.
>
> Alternatively, perhaps we could fetch the officially released
> Linux-libre tarballs instead of computing them ourselves [3].
>
> [1]: https://ci.guix.gnu.org/build/4711550/log/raw
> [2]: https://mail.gnu.org/archive/html/guix-devel/2021-08/msg00077.html
> [3]: http://linux-libre.fsfla.org/pub/linux-libre/releases/
>
> CC'ing Christopher Baines since this deals with substitutes.

This derivation seems to have been built fine by the bordeaux build
farm:

  https://data.guix.gnu.org/gnu/store/ny56fdcig9cd9bd3pssmlraz2c1q10q8-linux-libre-6.8.10-guix.tar.xz.drv

This is the timeout configuration for the bordeaux ARM build machines:

  (max-silent-time (* 12 3600))
  (timeout (* 72 3600))

e.g. https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/hydra/hatysa.scm#n262

The ci.guix.gnu.org Honeycomb timeouts are different, but don't look too
short. Maybe this build happened on an Overdrive system though.

[signature.asc (application/pgp-signature, inline)]

Message #11 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Richard Sent <richard <at> freakingpenguin.com>
To: Christopher Baines <mail <at> cbaines.net>
Cc: bug-guix <at> gnu.org
Subject: Re: linux-libre-guix.tar.xz CI times out on aarch64-linux
Date: Sun, 26 May 2024 14:34:43 -0400

Christopher Baines <mail <at> cbaines.net> writes:

> This derivation seems to have been built fine by the bordeaux build
> farm:
>
>   https://data.guix.gnu.org/gnu/store/ny56fdcig9cd9bd3pssmlraz2c1q10q8-linux-libre-6.8.10-guix.tar.xz.drv

You're right. It looks like the derivation itself built fine based on
that, but something seems off when the nar is sent. Assuming I
understand the following correctly:

--8<---------------cut here---------------start------------->8---
~/code/cloned/guix/guix $ guix build linux-libre --system=aarch64-linux
substitute: updating substitutes from 'http://10.1.2.2:80'... 100.0%
substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
The following derivations will be built:
  /gnu/store/v471590wpsw1fcnqrrr9bwh52skbb5rn-linux-libre-6.8.10.drv
  /gnu/store/1wi10rg7236ck8k5vdrdfap5l7a9s9z0-linux-libre-6.8.10-guix.tar.xz.drv
143.1 MB will be downloaded:
  /gnu/store/y813phs2n9xnb7zbcr07g0j9509bzbsb-linux-libre-6.8.10-guix.tar.xz
substituting /gnu/store/y813phs2n9xnb7zbcr07g0j9509bzbsb-linux-libre-6.8.10-guix.tar.xz...
downloading from https://bordeaux.guix.gnu.org/nar/none/y813phs2n9xnb7zbcr07g0j9509bzbsb-linux-libre-6.8.10-guix.tar.xz ...
 linux-libre-6.8.10-guix.tar.xz  136.5MiB                                                                          23.9MiB/s 00:06 ▕█████████████████▋▏  98.0%guix substitute: error: corrupt input while restoring '/gnu/store/y813phs2n9xnb7zbcr07g0j9509bzbsb-linux-libre-6.8.10-guix.tar.xz' from #<input: string 7f9644c23b60>
substitution of /gnu/store/y813phs2n9xnb7zbcr07g0j9509bzbsb-linux-libre-6.8.10-guix.tar.xz failed
guix build: error: some substitutes for the outputs of derivation `/gnu/store/ny56fdcig9cd9bd3pssmlraz2c1q10q8-linux-libre-6.8.10-guix.tar.xz.drv' failed (usually happens due to networking issues); try `--fallback' to build derivation from source 
--8<---------------cut here---------------end--------------->8---

Even though the derivation was built, the substitution fails.

Just for fun, here's the output on a ARM64-native system:

--8<---------------cut here---------------start------------->8---
root <at> caustic ~# guix build linux-libre  
substitute: updating substitutes from 'http://10.1.2.2:80'... 100.0%
substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
The following derivations will be built:
  /gnu/store/v471590wpsw1fcnqrrr9bwh52skbb5rn-linux-libre-6.8.10.drv
  /gnu/store/1wi10rg7236ck8k5vdrdfap5l7a9s9z0-linux-libre-6.8.10-guix.tar.xz.drv
143.1 MB will be downloaded:
  /gnu/store/y813phs2n9xnb7zbcr07g0j9509bzbsb-linux-libre-6.8.10-guix.tar.xz
substituting /gnu/store/0jfsx4hljddyand45z7i77ynpvr0mhb5-module-import-compiled...
downloading from http://10.1.2.2/nar/zstd/0jfsx4hljddyand45z7i77ynpvr0mhb5-module-import-compiled ...
 module-import-compiled                                                                                                    4.2MiB/s 00:00 | 244KiB transferred

substituting /gnu/store/fwqy5d6xyar9x5yksny79r2d519s25cx-100gnu+freedo.patch...
downloading from http://10.1.2.2/nar/zstd/fwqy5d6xyar9x5yksny79r2d519s25cx-100gnu%2Bfreedo.patch ...
 100gnu%2Bfreedo.patch                                                                                                      1.8MiB/s 00:00 | 46KiB transferred

substituting /gnu/store/y813phs2n9xnb7zbcr07g0j9509bzbsb-linux-libre-6.8.10-guix.tar.xz...
downloading from https://bordeaux.guix.gnu.org/nar/none/y813phs2n9xnb7zbcr07g0j9509bzbsb-linux-libre-6.8.10-guix.tar.xz ...
 linux-libre-6.8.10-guix.tar.xz  136.5MiB                                                                           1.7MiB/s 01:21 ▕█████████████████▉▏  99.9%guix substitute: error: corrupt input while restoring '/gnu/store/y813phs2n9xnb7zbcr07g0j9509bzbsb-linux-libre-6.8.10-guix.tar.xz' from #<input: string ffff7daf3460>
substitution of /gnu/store/y813phs2n9xnb7zbcr07g0j9509bzbsb-linux-libre-6.8.10-guix.tar.xz failed
guix build: error: corrupt input while restoring archive from #<closed: file ffff977abb60>
--8<---------------cut here---------------end--------------->8---

-- 
Take it easy,
Richard Sent
Making my computer weirder one commit at a time.

Message #14 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Christopher Baines <mail <at> cbaines.net>
To: Richard Sent <richard <at> freakingpenguin.com>
Cc: bug-guix <at> gnu.org
Subject: Re: linux-libre-guix.tar.xz CI times out on aarch64-linux
Date: Sun, 26 May 2024 19:46:03 +0100

[Message part 1 (text/plain, inline)]

Richard Sent <richard <at> freakingpenguin.com> writes:

> Christopher Baines <mail <at> cbaines.net> writes:
>
>> This derivation seems to have been built fine by the bordeaux build
>> farm:
>>
>>   https://data.guix.gnu.org/gnu/store/ny56fdcig9cd9bd3pssmlraz2c1q10q8-linux-libre-6.8.10-guix.tar.xz.drv
>
> You're right. It looks like the derivation itself built fine based on
> that, but something seems off when the nar is sent. Assuming I
> understand the following correctly:
>
> --8<---------------cut here---------------start------------->8---
> ~/code/cloned/guix/guix $ guix build linux-libre --system=aarch64-linux
> substitute: updating substitutes from 'http://10.1.2.2:80'... 100.0%
> substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 100.0%
> substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
> The following derivations will be built:
>   /gnu/store/v471590wpsw1fcnqrrr9bwh52skbb5rn-linux-libre-6.8.10.drv
>   /gnu/store/1wi10rg7236ck8k5vdrdfap5l7a9s9z0-linux-libre-6.8.10-guix.tar.xz.drv
> 143.1 MB will be downloaded:
>   /gnu/store/y813phs2n9xnb7zbcr07g0j9509bzbsb-linux-libre-6.8.10-guix.tar.xz
> substituting /gnu/store/y813phs2n9xnb7zbcr07g0j9509bzbsb-linux-libre-6.8.10-guix.tar.xz...
> downloading from https://bordeaux.guix.gnu.org/nar/none/y813phs2n9xnb7zbcr07g0j9509bzbsb-linux-libre-6.8.10-guix.tar.xz ...
>  linux-libre-6.8.10-guix.tar.xz  136.5MiB                                                                          23.9MiB/s 00:06 ▕█████████████████▋▏  98.0%guix substitute: error: corrupt input while restoring '/gnu/store/y813phs2n9xnb7zbcr07g0j9509bzbsb-linux-libre-6.8.10-guix.tar.xz' from #<input: string 7f9644c23b60>
> substitution of /gnu/store/y813phs2n9xnb7zbcr07g0j9509bzbsb-linux-libre-6.8.10-guix.tar.xz failed
> guix build: error: some substitutes for the outputs of derivation `/gnu/store/ny56fdcig9cd9bd3pssmlraz2c1q10q8-linux-libre-6.8.10-guix.tar.xz.drv' failed (usually happens due to networking issues); try `--fallback' to build derivation from source 
> --8<---------------cut here---------------end--------------->8---
>
> Even though the derivation was built, the substitution fails.

Yeah, something's up here specifically with the bordeaux build farm,
feel free to open a new issue.

You can tell something is wrong just by looking at the narinfo:

  https://bordeaux.guix.gnu.org/y813phs2n9xnb7zbcr07g0j9509bzbsb.narinfo

Given there's no compression here, the file size should be the same as
the nar size, but it's not, it's missing a few bytes.

[signature.asc (application/pgp-signature, inline)]

Message #17 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Richard Sent <richard <at> freakingpenguin.com>
To: Christopher Baines <mail <at> cbaines.net>
Cc: bug-guix <at> gnu.org
Subject: Re: linux-libre-guix.tar.xz CI times out on aarch64-linux
Date: Sun, 26 May 2024 17:15:21 -0400

New issue opened at https://issues.guix.gnu.org/71214. I'l close this
one. Thanks!

-- 
Take it easy,
Richard Sent
Making my computer weirder one commit at a time.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.