GNU bug report logs - #74699
need valid explanation / weird 'grep -q' behaviour

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 74699 in the body.
You can then email your comments to 74699 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Frank Reppin <frank <at> undermydesk.org>
To: bug-grep <at> gnu.org
Subject: need valid explanation / weird 'grep -q' behaviour
Date: Thu, 5 Dec 2024 00:43:37 +0100

Dear all,

sorry for the (maybe) noise - but I really need some clues as to
why some quite common grep usecase seems to randomly fail.

I've already looked through google and some valuable results pointed
out some discussions on stackoverflow ... where this was somewhat 
discussed ... but (IMHO) no really valid explanation was outlined.
Some claim that this is due to 'set -o pipefail' beeing used in the 
script... but this does not really explain why it only _sometimes_ happens.

(what makes me think that this might be some kind of weird issue which 
needs to be adressed somehow... maybe)

Long story - made short... I've linked a simple and short bash script 
(created and tested and reproducible at least here) with all the 
required information inside) - that can reproduce the issue:

https://download.undermydesk.org/bash-grep-q-special-issue.sh

(all explanations inside)

thankyou all!
cheers,
FR

-- 
43rd Law of Computing:
        Anything that can go wr
fortune: Segmentation violation -- Core dumped

Message #8 received at 74699 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Frank Reppin <frank <at> undermydesk.org>
Cc: 74699 <at> debbugs.gnu.org
Subject: Re: bug#74699: need valid explanation / weird 'grep -q' behaviour
Date: Thu, 5 Dec 2024 11:59:20 -0800

On 2024-12-04 15:43, Frank Reppin wrote:
> Some claim that this is due to 'set -o pipefail' beeing used in the 
> script... but this does not really explain why it only _sometimes_ happens.

I don't see why not. Pipe buffering is an "it all depends" situation.

Anyway, I tried your script on bleeding-edge grep on Fedora 40 and it 
worked fine for me. You didn't mention your grep version; behavior in 
this area changed in 2016, though I doubt your version is that old.

One workaround is to use "set -o pipefail" only for pipelines where you 
want the entire pipeline to fail merely because one component failed.

Message #11 received at 74699 <at> debbugs.gnu.org (full text, mbox):

From: Frank Reppin <frank <at> undermydesk.org>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: 74699 <at> debbugs.gnu.org
Subject: Re: bug#74699: need valid explanation / weird 'grep -q' behaviour
Date: Fri, 6 Dec 2024 23:51:31 +0100

Dear all, hi Paul,

yeah, you're right - thankyou for your feedback.

In the meanwhile - it is more clear to me...
... especially after re-reading what others suspected on
stackoverflow.

The real reason seems to be (... is), that a proper match seems
to kill the pipe early with exitcode 141 - and thus making
the pipe as a whole fail - as one would expect when using 'pipefail'.
(although - to be honest - I didn't thought of exactly such a fail when
it comes to use grep with '-q' in such cases).

However - it thus turns out to be OK. One just has to be cautious
when using it in this way...
... not sure if it's maybe worth mentioning in the manpage?!
(that pipefail or other returncode checking mechanisms should be aware
of this fact - for the unaware).

cheers,
FR




On 05.12.2024 20:59, Paul Eggert wrote:
> On 2024-12-04 15:43, Frank Reppin wrote:
>> Some claim that this is due to 'set -o pipefail' beeing used in the 
>> script... but this does not really explain why it only _sometimes_ 
>> happens.
> 
> I don't see why not. Pipe buffering is an "it all depends" situation.
> 
> Anyway, I tried your script on bleeding-edge grep on Fedora 40 and it 
> worked fine for me. You didn't mention your grep version; behavior in 
> this area changed in 2016, though I doubt your version is that old.
> 
> One workaround is to use "set -o pipefail" only for pipelines where you 
> want the entire pipeline to fail merely because one component failed.

-- 
43rd Law of Computing:
        Anything that can go wr
fortune: Segmentation violation -- Core dumped

Message #14 received at 74699 <at> debbugs.gnu.org (full text, mbox):

From: jackson <at> fastmail.com
To: "Frank Reppin" <frank <at> undermydesk.org>, 74699 <at> debbugs.gnu.org
Subject: Re: bug#74699: need valid explanation / weird 'grep -q' behaviour
Date: Fri, 06 Dec 2024 18:21:11 -0600

Frank wrote:
>> The real reason seems to be (... is), that a proper match seems
>> to kill the pipe early with exitcode 141 

Placing the X= setting inside the key line in the code, we see it says:

if echo "$( cat "${TMPOUT}" 2>/dev/null ) | grep -q "^${V}$"; then

Looking at the grep(1) man page entry for its "-q" option, we see it says:

      -q, --quiet, --silent
             Quiet; do not write anything to standard output.  Exit immediately
             with zero status if any if any match is found, even if an error was
             detected.  Also see  the  -s  or  --no-messages option.

So we have an "if" statement testing the exit value of a pipe of 'echo' to 'grep'
where the 'grep' will exit early, if it finds a match early.  As a secondary
matter of efficiency, we also have the contents of TMPOUT filling the value
of shell's formulation of the argument to be passed to echo'd into a pipe,
 instead of directly grep'ing in TMPOUT.

If for some reason I don't see off hand, you prefer to remain with this inefficient
"echo contents_of_TMPOUT | grep ..." construct, I'd suggest using "grep -c"
instead of "grep -q", then checking for a positive count, thus forcing the entire
contents of TMPOUT to be read and echo'd, before the grep decides it can
exit early and break the pipe:

if [[ $(echo "$( cat "${TMPOUT}" 2>/dev/null ) | grep -c "^${V}$") -gt 0 ]]; then

Or, more efficiently, just "grep -q" directly in TMPOUT and skip the echo and
pipe entirely:

if grep -q "^${V}$" "${TMPOUT}"; then

-- 
  Paul Jackson
  jackson <at> fastmail.fm

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.