GNU bug report logs - #48591
ugrep: reproducibility issues

Previous Next

To reply to this bug, email your comments to 48591 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Christopher Howard <christopher <at> librehacker.com>
To: Guix Bug Reports <bug-guix <at> gnu.org>
Subject: ugrep: illegal instruction
Date: Sat, 22 May 2021 12:19:30 -0800

[Message part 1 (text/plain, inline)]

Hi, I found another package, ugrep, which is evidently using native
instructions that do not work on my processor. When I run ugrep (e.g.,
`ugrep -Q') it immediately dies with error 'illegal instruction'.
However, if I pull the source with `guix build -S' and build and
install it to a local directory, it runs without trouble.

I'm sure this will also be a reproducibility bug, and plan on
submitting guix challenge results. However, the command I am running to
rebuild ugrep without grafts or substitutes is rebuilding everything
all the way back to bootstrap, so it probably will not be completed for
a day or two:

guix gc -D $(guix build --no-grafts ugrep); guix build --no-substitutes 
ugrep

Please compare similar bug 48373. It didn't seem worth the trouble of
running an object dump and figuring out which instruction is illegal,
but I could do that if required.

My system information:


christopher <at> nightshade ~$ grep 'flags' /proc/cpuinfo | head -n 1
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext
fxsr_opt pdpe1gb rdtscp lm 3dnowext 3dnow constant_tsc rep_good nopl
nonstop_tsc cpuid extd_apicid pni monitor cx16 popcnt lahf_lm
cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch
osvw ibs skinit wdt hw_pstate vmmcall npt lbrv svm_lock nrip_save

christopher <at> nightshade ~$ neofetch --stdout
christopher <at> nightshade 
---------------------- 
OS: Guix System d4ffa9630277fa8699c783c08381d688626d4bc3 x86_64 
Host: GA-880GM-UD2H 
Kernel: 5.12.5-gnu 
Uptime: 1 hour, 30 mins 
Packages: 107 (guix-system), 93 (guix-user) 
Shell: bash 5.0.16 
Resolution: 1920x1200 
DE: GNOME 3.34.5 
Theme: Adwaita [GTK2/3] 
Icons: Adwaita [GTK2/3] 
Terminal: kitty 
CPU: AMD Athlon II X3 455 (3) @ 3.300GHz 
GPU: NVIDIA GeForce 8400 GS Rev. 3 
Memory: 1573MiB / 7957MiB 

christopher <at> nightshade ~$ guix describe
Generation 5	May 22 2021 09:43:30	(current)
  guix d4ffa96
    repository URL: https://git.savannah.gnu.org/git/guix.git
    branch: master
    commit: d4ffa9630277fa8699c783c08381d688626d4bc3


-- 
Christopher Howard
blog: https://librehacker.com
social: https://gnusocial.club/librehacker

[Message part 2 (text/html, inline)]

Message #8 received at 48591 <at> debbugs.gnu.org (full text, mbox):

From: Christopher Howard <christopher <at> librehacker.com>
To: 48591 <at> debbugs.gnu.org
Subject: ugrep: illegal instruction
Date: Sat, 22 May 2021 14:46:13 -0800

[Message part 1 (text/plain, inline)]

christopher <at> nightshade ~$ guix challenge ugrep
/gnu/store/a6x7rv7xi3d2pw205znmnha9qfjrvs7k-ugrep-3.1.12 contents
differ:
  local hash: 1yf9752hprihmhwnqlhaffkmwmkd3scwzr8dqy7d78vcy70qjkn6
  
https://ci.guix.gnu.org/nar/lzip/a6x7rv7xi3d2pw205znmnha9qfjrvs7k-ugrep-3.1.12
: 1ifdjmscb07pacxwcs2408zi12yrxxf0zdwzz800xlccp5x53m7r
  differing file:
    /bin/ugrep

1 store items were analyzed:
  - 0 (0.0%) were identical
  - 1 (100.0%) differed
  - 0 (0.0%) were inconclusive

-- 
Christopher Howard
blog: https://librehacker.com
social: https://gnusocial.club/librehacker

[Message part 2 (text/html, inline)]

Message #11 received at 48591 <at> debbugs.gnu.org (full text, mbox):

From: Leo Famulari <leo <at> famulari.name>
To: Christopher Howard <christopher <at> librehacker.com>
Cc: 48591 <at> debbugs.gnu.org
Subject: Re: bug#48591: ugrep: illegal instruction
Date: Mon, 24 May 2021 13:22:23 -0400

On Sat, May 22, 2021 at 12:19:30PM -0800, Christopher Howard wrote:
> However, the command I am running to
> rebuild ugrep without grafts or substitutes is rebuilding everything
> all the way back to bootstrap, so it probably will not be completed for
> a day or two:
> 
> guix gc -D $(guix build --no-grafts ugrep); guix build --no-substitutes 
> ugrep

Do this instead:

$ guix environment --no-grafts ugrep -- guix build --no-grafts --no-substitutes ugrep

That way, you can use substitutes for ugrep's dependencies, but then
build ugrep from source.

I hope it helps!

Message #14 received at 48591 <at> debbugs.gnu.org (full text, mbox):

From: raingloom <raingloom <at> riseup.net>
To: Leo Famulari <leo <at> famulari.name>
Cc: 48591 <at> debbugs.gnu.org, Christopher Howard <christopher <at> librehacker.com>
Subject: Re: bug#48591: ugrep: illegal instruction
Date: Wed, 30 Jun 2021 23:41:11 +0200

On Mon, 24 May 2021 13:22:23 -0400
Leo Famulari <leo <at> famulari.name> wrote:

> On Sat, May 22, 2021 at 12:19:30PM -0800, Christopher Howard wrote:
> > However, the command I am running to
> > rebuild ugrep without grafts or substitutes is rebuilding everything
> > all the way back to bootstrap, so it probably will not be completed
> > for a day or two:
> > 
> > guix gc -D $(guix build --no-grafts ugrep); guix build
> > --no-substitutes ugrep  
> 
> Do this instead:
> 
> $ guix environment --no-grafts ugrep -- guix build --no-grafts
> --no-substitutes ugrep
> 
> That way, you can use substitutes for ugrep's dependencies, but then
> build ugrep from source.
> 
> I hope it helps!
> 
> 
> 

TIL about that. There really ought to be a more obvious way to do that.

Message #17 received at 48591 <at> debbugs.gnu.org (full text, mbox):

From: Leo Famulari <leo <at> famulari.name>
To: raingloom <raingloom <at> riseup.net>
Cc: 48591 <at> debbugs.gnu.org, Christopher Howard <christopher <at> librehacker.com>
Subject: Re: bug#48591: ugrep: illegal instruction
Date: Thu, 1 Jul 2021 22:23:14 -0400

On Wed, Jun 30, 2021 at 11:41:11PM +0200, raingloom wrote:
> TIL about that. There really ought to be a more obvious way to do that.

Agreed! It's a common use case.

Message #20 received at 48591 <at> debbugs.gnu.org (full text, mbox):

From: zimoun <zimon.toutoune <at> gmail.com>
To: Christopher Howard <christopher <at> librehacker.com>
Cc: 48591 <at> debbugs.gnu.org
Subject: Re: bug#48591: ugrep: illegal instruction
Date: Fri, 02 Jul 2021 13:24:13 +0200

Hi,

On Sat, 22 May 2021 at 12:19, Christopher Howard <christopher <at> librehacker.com> wrote:

> I'm sure this will also be a reproducibility bug,

Indeed, the package ’ugrep’ is not reproducible:

 guix build ugrep --no-grafts -q
 guix build ugrep --no-grafts -q --check -K
 diff --no-dereference -r /gnu/store/a6x7rv7xi3d2pw205znmnha9qfjrvs7k-ugrep-3.1.12{,-check}
 Binary files /gnu/store/a6x7rv7xi3d2pw205znmnha9qfjrvs7k-ugrep-3.1.12/bin/ugrep and /gnu/store/a6x7rv7xi3d2pw205znmnha9qfjrvs7k-ugrep-3.1.12-check/bin/ugrep differ

The next step is to investigate with diffoscope.


Well, another data point – maybe Berlin has an issue – because I get:

  guix challenge ugrep --substitute-urls="https://ci.guix.gnu.org https://bordeaux.guix.gnu.org"
  /gnu/store/a6x7rv7xi3d2pw205znmnha9qfjrvs7k-ugrep-3.1.12 contents differ:
    local hash: 1ifdjmscb07pacxwcs2408zi12yrxxf0zdwzz800xlccp5x53m7r
    https://ci.guix.gnu.org/nar/lzip/a6x7rv7xi3d2pw205znmnha9qfjrvs7k-ugrep-3.1.12: 1ifdjmscb07pacxwcs2408zi12yrxxf0zdwzz800xlccp5x53m7r
    https://bordeaux.guix.gnu.org/nar/lzip/a6x7rv7xi3d2pw205znmnha9qfjrvs7k-ugrep-3.1.12: 1yf9752hprihmhwnqlhaffkmwmkd3scwzr8dqy7d78vcy70qjkn6

  guix hash -r $(guix build --no-grafts ugrep)-check
  1yf9752hprihmhwnqlhaffkmwmkd3scwzr8dqy7d78vcy70qjkn6

I fetch by default the substitutes from ci that why “local hash“ and ci
are the same; but my local build with ‘guix build --check’ gives the
same hash as bordeaux.


All the best,
simon

Message #25 received at 48591 <at> debbugs.gnu.org (full text, mbox):

From: kiasoc5 <kiasoc5 <at> disroot.org>
To: 48591 <at> debbugs.gnu.org
Subject: Re: ugrep: reproducibility issues
Date: Mon, 16 Jan 2023 20:37:27 -0500

Is this bug fixed yet?

If not, seeing that this issue was motivated by an illegal instruction, 
we can use compile flags to disable SSE and other platform-dependent 
optimizations and investigate further.

WDYT?

https://github.com/Genivia/ugrep#other-platforms-step-3-build

Message #28 received at 48591 <at> debbugs.gnu.org (full text, mbox):

From: Vagrant Cascadian <vagrant <at> reproducible-builds.org>
To: 48591 <at> debbugs.gnu.org
Cc: kiasoc5 <kiasoc5 <at> disroot.org>,
 Christopher Howard <christopher <at> librehacker.com>,
 zimoun <zimon.toutoune <at> gmail.com>, raingloom <raingloom <at> riseup.net>,
 Leo Famulari <leo <at> famulari.name>
Subject: Re: bug#48591: ugrep: reproducibility issues
Date: Thu, 24 Oct 2024 12:04:45 -0700

[Message part 1 (text/plain, inline)]

On 2023-01-16, kiasoc5 wrote:
> Is this bug fixed yet?
>
> If not, seeing that this issue was motivated by an illegal instruction, 
> we can use compile flags to disable SSE and other platform-dependent 
> optimizations and investigate further.
>
> WDYT?
>
> https://github.com/Genivia/ugrep#other-platforms-step-3-build

--disable-sse2 disable SSE2 and AVX optimizations
--disable-avx2 disable AVX2 and AVX512BW optimizations, but compile with SSE2 when supported
--disable-neon disable ARM NEON/AArch64 optimizations

Seems like --disable-sse2 on x86_64 and x86 (32-bit) and --disable-neon
(armhf, maybe aarch64) would be needed to make the build independent of
which cpu it runs on.


FWIW, debian disables several of these (--disable-sse2 and
--disable-neon), although also passes --disable-avx which is not mentioned in
the upstream documentation, although is present in configure.ac...

  https://salsa.debian.org/debian/ugrep/-/blob/7e2383dfaec23d58587f53c31ce46ef884513975/debian/rules#L5-13


live well,
  vagrant

[signature.asc (application/pgp-signature, inline)]

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.