GNU bug report logs - #14391
bug in grep -- ignoring GREP_OPTIONS or not acting on it...

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 14391 in the body.
You can then email your comments to 14391 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Linda Walsh <coreutils <at> tlinx.org>
To: bug-coreutils <at> gnu.org
Subject: bug in grep -- ignoring GREP_OPTIONS or not acting on it...
Date: Sat, 11 May 2013 22:04:26 -0700

My grep is :
grep (GNU grep) 2.14
Copyright (C) 2012 Free Software Foundation, Inc.
...
when I type grep, I get an alias + I have GREP_OPTIONS set..

whence on grep gives the first entry as:
grep is aliased to `grep --color=auto'

> echo $GREP_OPTIONS
-D skip --binary-files=without-match

Ishtar:/tmp/test> ll
total 0
drwxrwxr-x 2 6 May 11 14:39 a/
drwxrwxr-x 2 6 May 11 14:39 b/
drwxrwxr-x 2 6 May 11 14:39 c/
-rw-rw-r-- 1 0 May 11 14:39 e
-rw-rw-r-- 1 0 May 11 14:39 f
-rw-rw-r-- 1 0 May 11 14:39 g
Ishtar:/tmp/test> grep foo *
grep: a: Is a directory
grep: b: Is a directory
grep: c: Is a directory

grep should ignore the directories and issue no message for them:
       -D ACTION, --devices=ACTION
              If an input file is a device, FIFO  or  socket,  use  ACTION  to
              process  it.   By  default,  ACTION  is  read,  which means that
              devices are read just as if they were ordinary files.  If ACTION
              is skip, devices are silently skipped.

       GREP_OPTIONS
              This variable specifies default options to be placed in front of
              any  explicit  options.   For  example,   if   GREP_OPTIONS   is
              '--binary-files=without-match  --directories=skip', grep behaves
              as  if  the   two   options   --binary-files=without-match   and
              --directories=skip   had  been  specified  before  any  explicit
              options.
----
ACTION=skip, so where's the silence?

I don't remember this being this way before -- it used to work.
Seems like it stopped recently?

Message #8 received at 14391 <at> debbugs.gnu.org (full text, mbox):

From: Pádraig Brady <P <at> draigBrady.com>
To: Linda Walsh <coreutils <at> tlinx.org>
Cc: 14391 <at> debbugs.gnu.org
Subject: Re: bug#14391: bug in grep -- ignoring GREP_OPTIONS or not acting
	on it...
Date: Sun, 12 May 2013 13:23:47 +0100

tag 14391 notabug
close 14391
stop

grep is not a coreutils so closing this.
You may want to discuss this on the grep mailing list,
but I've added a couple of notes below.

On 05/12/2013 06:04 AM, Linda Walsh wrote:
> My grep is :
> grep (GNU grep) 2.14
> Copyright (C) 2012 Free Software Foundation, Inc.
> ...
> when I type grep, I get an alias + I have GREP_OPTIONS set..
> 
> whence on grep gives the first entry as:
> grep is aliased to `grep --color=auto'
> 
>> echo $GREP_OPTIONS
> -D skip --binary-files=without-match
> 
> Ishtar:/tmp/test> ll
> total 0
> drwxrwxr-x 2 6 May 11 14:39 a/
> drwxrwxr-x 2 6 May 11 14:39 b/
> drwxrwxr-x 2 6 May 11 14:39 c/
> -rw-rw-r-- 1 0 May 11 14:39 e
> -rw-rw-r-- 1 0 May 11 14:39 f
> -rw-rw-r-- 1 0 May 11 14:39 g
> Ishtar:/tmp/test> grep foo *
> grep: a: Is a directory
> grep: b: Is a directory
> grep: c: Is a directory
> 
> grep should ignore the directories and issue no message for them:
>        -D ACTION, --devices=ACTION
>               If an input file is a device, FIFO  or  socket,  use  ACTION  to
>               process  it.   By  default,  ACTION  is  read,  which means that
>               devices are read just as if they were ordinary files.  If ACTION
>               is skip, devices are silently skipped.
> 
>        GREP_OPTIONS
>               This variable specifies default options to be placed in front of
>               any  explicit  options.   For  example,   if   GREP_OPTIONS   is
>               '--binary-files=without-match  --directories=skip', grep behaves
>               as  if  the   two   options   --binary-files=without-match   and
>               --directories=skip   had  been  specified  before  any  explicit
>               options.
> ----
> ACTION=skip, so where's the silence?
> 
> I don't remember this being this way before -- it used to work.
> Seems like it stopped recently?

While I see that grep 2.9 at least doesn't seem to warn
about dirs by default, for newer versions you may want
to specify -d 'skip' instead of -D 'skip' ?

thanks,
Pádraig.

Message #15 received at 14391 <at> debbugs.gnu.org (full text, mbox):

From: Bob Proulx <bob <at> proulx.com>
To: 14391 <at> debbugs.gnu.org
Subject: Re: bug#14391: bug in grep -- ignoring GREP_OPTIONS or not acting on
	it...
Date: Sun, 12 May 2013 17:57:06 -0600

Of course this needs to go over to bug-grep but here is my comment
anyway...

Linda Walsh wrote:
> when I type grep, I get an alias + I have GREP_OPTIONS set..
> grep is aliased to `grep --color=auto'
> > echo $GREP_OPTIONS
> -D skip --binary-files=without-match
> Ishtar:/tmp/test> grep foo *
> grep: a: Is a directory
> grep: b: Is a directory
> grep: c: Is a directory

Basically:

  grep -D skip --binary-files=without-match --color=auto foo *

> grep should ignore the directories and issue no message for them:
>        -D ACTION, --devices=ACTION
>            If an input file is a device, FIFO or socket, use ACTION to
>            process  it.   By  default,  ACTION  is  read,  which means that
>            devices are read just as if they were ordinary files.  If ACTION
>            is skip, devices are silently skipped.
>            ...
> ...
> I don't remember this being this way before -- it used to work.
> Seems like it stopped recently?

I think you need --directories=skip in addition to --devices=skip.
Add -d skip to your option list.

  grep -D skip -d skip --binary-files=without-match --color=auto foo *

This is an example of why I think grep should never have added
directory recursion.  It needs to suck in all of 'find' before it will
be complete.  And that doesn't make sense because there is already
find available.

  find . -type f -exec grep foo {} +

That isn't precisely the same as the grep case you posted but it is
usually what people want.  For -D skip it would need to be more like
this:

  find . -follow ! -type d -exec grep foo {} +

If there were often used favorite options then I would turn them into
either a shell script or a shell function.

Bob

Message #18 received at 14391 <at> debbugs.gnu.org (full text, mbox):

From: Linda Walsh <coreutil <at> tlinx.org>
To: Bob Proulx <bob <at> proulx.com>
Cc: 14391 <at> debbugs.gnu.org
Subject: Re: bug#14391: bug in grep -- ignoring GREP_OPTIONS or not acting on
	it...
Date: Tue, 14 May 2013 10:32:31 -0700

Bob Proulx wrote:
> 
> Basically:
> 
> I think you need --directories=skip in addition to --devices=skip.
> Add -d skip to your option list.
> 
>   grep -D skip -d skip --binary-files=without-match --color=auto foo *
> 
> This is an example of why I think grep should never have added
> directory recursion.  It needs to suck in all of 'find' before it will
> be complete.  And that doesn't make sense because there is already
> find available.
> 
>   find . -type f -exec grep foo {} +
> 
> That isn't precisely the same as the grep case you posted but it is
> usually what people want.  For -D skip it would need to be more like
> this:
> 
>   find . -follow ! -type d -exec grep foo {} +
> 
> If there were often used favorite options then I would turn them into
> either a shell script or a shell function.
----

Very good points -- especially about my broken options (I could swear they worked
in the past)...must have gotten accidently deleted somewhere along in the past.


But .. I also agree with the problems of grep possibly calling find
inefficiently.  I would have thought it smarter to call find as a child, and
read from
it's pipe and call the file-search on large groups at a time.  Otherwise, if grep
waited for all of find to complete, as you are suggesting, might it not run out
of space
to hold the paths of the files it is going to search?

Message #21 received at 14391 <at> debbugs.gnu.org (full text, mbox):

From: Bob Proulx <bob <at> proulx.com>
To: 14391 <at> debbugs.gnu.org
Subject: Re: bug#14391: bug in grep -- ignoring GREP_OPTIONS or not acting on
	it...
Date: Tue, 14 May 2013 12:00:49 -0600

Linda Walsh wrote:
> Bob Proulx wrote:
> > Basically:
> > 
> > I think you need --directories=skip in addition to --devices=skip.
> > Add -d skip to your option list.
> > 
> >   grep -D skip -d skip --binary-files=without-match --color=auto foo *

> Very good points -- especially about my broken options (I could
> swear they worked in the past)...must have gotten accidently deleted
> somewhere along in the past.

It probably did work in the past.  If you look through all of the
previous grep versions you will probably find at least one where that
combination worked.  I didn't mean to say otherwise.  I remember
grep's recursion behavior as having gone through several iterations of
changes.  It hasn't matured yet.  But hopefully the changes are
slowing down.  :-)

> > This is an example of why I think grep should never have added
> > directory recursion.  It needs to suck in all of 'find' before it will
> > be complete.  And that doesn't make sense because there is already
> > find available.
> > 
> >   find . -type f -exec grep foo {} +
> 
> But .. I also agree with the problems of grep possibly calling find
> inefficiently.  I would have thought it smarter to call find as a
> child, and read from it's pipe and call the file-search on large
> groups at a time.  Otherwise, if grep waited for all of find to
> complete, as you are suggesting, might it not run out of space to
> hold the paths of the files it is going to search?

Using the above with "{} +" is very efficient.  It will not exceed
ARG_MAX.  The "{} +" part of the find instructs find to:

  3.3.2 Multiple Files
  --------------------

  Sometimes you need to process files one at a time.  But usually this is
  not necessary, and, it is faster to run a command on as many files as
  possible at a time, rather than once per file.  Doing this saves on the
  time it takes to start up the command each time.

     The `-execdir' and `-exec' actions have variants that build command
  lines containing as many matched files as possible.

   -- Action: -execdir command {} +
       This works as for `-execdir command ;', except that the `{}' at
       the end of the command is expanded to a list of names of matching
       files.  This expansion is done in such a way as to avoid exceeding
       the maximum command line length available on the system.  Only one
       `{}' is allowed within the command, and it must appear at the end,
       immediately before the `+'.  A `+' appearing in any position other
       than immediately after `{}' is not considered to be special (that
       is, it does not terminate the command).

   -- Action: -exec command {} +
       This insecure variant of the `-execdir' action is specified by
       POSIX.  The main difference is that the command is executed in the
       directory from which `find' was invoked, meaning that `{}' is
       expanded to a relative path starting with the name of one of the
       starting directories, rather than just the basename of the matched
       file.

The old traditional way was "\;" (needed to be quoted to protect it
from the shell) which invoked the command once for each argument.  The
new standard way is "+" which bundles up as many arguments as
possible, will not exceed ARG_MAX, and therefore is quite efficient.
Also it doesn't need to be special quoting making it more convenient.

Using 'find . -print0 | grep --files0-from=- ...', if implemented,
would also be efficient.  But not implemented AFAIK.

I think the "insecure" warning is too agressive and overstating the point.

Bob

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.