GNU bug report logs - #54174
(MacOS Monterey 12.2.1: zsh): grep "string" * is interpreted as grep -V when directory has a filename "-Vfilename.ext"

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 54174 in the body.
You can then email your comments to 54174 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Marja Koivunen <marja <at> innomore.com>
To: bug-grep <at> gnu.org
Subject: (MacOS Monterey 12.2.1: zsh): grep "string" * is interpreted as grep
 -V when directory has a filename  "-Vfilename.ext"
Date: Sat, 26 Feb 2022 13:33:35 -0500

I had a directory with filenames that started with “-“

doing grep on that directory for a “string" did not find anything
although “string was on some of the files”

grep just kept repeating something … FreeBSD

Finally, (with some help) I understood that grep interpreted “-Vfiename” as an option -V and 
gave the version info instead of doing grep “string” *

Maybe there is a way to add space after “-V “ and possibly also other options that could be used as  part of a filename in some operating systems?

Message #8 received at 54174 <at> debbugs.gnu.org (full text, mbox):

From: Bob Proulx <bob <at> proulx.com>
To: Marja Koivunen <marja <at> innomore.com>
Cc: 54174 <at> debbugs.gnu.org
Subject: Re: bug#54174: (MacOS Monterey 12.2.1: zsh): grep "string" * is
 interpreted as grep -V when directory has a filename "-Vfilename.ext"
Date: Sat, 26 Feb 2022 15:53:05 -0700

close 54174
thanks

Marja Koivunen wrote:
> I had a directory with filenames that started with "-"

Of course that is okay.  However it is unusual.  And it means you need
to be extra careful in handling it such as to avoid it being confused
with an option.

> doing grep on that directory for a "string" did not find anything
> although "string was on some of the files"

You did not show us the exact command that you used.  We only know
what you show us in your email.  But in this case I will guess with
some confidence that you did something like this.

    $ grep PATTERN *      <==  Error!  Data dependent!

And that failed.  Which is not surprising since the "*" will be
expanded by your command shell to match every file in the current
working directory.  And you have just said that one of those files
(a directory is also a file) started with a '-' character.

To understand what is happening it is useful to use 'echo' here to
echo print what the command shell has done with the line and with
expanding the shell file glob characters.  (The * is called a glob
character because it matches a glob of characters.)

    $ echo grep PATTERN *

If you do that then what you see depends upon what is there.  But if
what is there is a file or a directory that starts with a dash then it
will show there as starting with a dash.  Here is a full test case.

    $ mkdir /tmp/optiontest
    $ cd /tmp/optiontest
    $ echo foo > ./--help
    $ echo grep foo *
    grep foo --help
    $ grep foo * | head
    Usage: grep [OPTION]... PATTERNS [FILE]...
    Search for PATTERNS in each FILE.
    Example: grep -i 'hello world' menu.h main.c
    PATTERNS can contain multiple patterns separated by newlines.

    Pattern selection and interpretation:
      -E, --extended-regexp     PATTERNS are extended regular expressions
      -F, --fixed-strings       PATTERNS are strings
      -G, --basic-regexp        PATTERNS are basic regular expressions
      -P, --perl-regexp         PATTERNS are Perl regular expressions

Here we see that the * was expanded by the comand shell to the matched
files.  And since a file --help was created it matched, was expanded,
was passed to grep.  The grep program saw the --help and acted.

Programs do not know that the user typed in a "*" since the command
shell expands those shell meta-characters before invoking the
program.  The program only sees the result of the expansion.

> grep just kept repeating something ... FreeBSD

Yes.  Just as explained above.

The problem is that a plain * will be expanded to match files in the
current directory.  That's why I labeled it as a data dependent
failure.  It depends upon the files that are present.  If no files
start with an option dash then it won't be seen.  If one or more files
start with an option dash then it will.  Therefore I think it is
always problematic to use a * like that.  Instead use a ./* to ensure
that the first letter is never an option dash.  This would be safer.

    $ grep PATTERN ./*

In that case using our optiontest directory the same as above.

    $ echo grep PATTERN ./*
    grep foo ./--help

That will never be confused with an option because it starts with a ./
and that is not an option start sequence.  It will never be confused
with an option.  Therefore I strongly recommend always using the file
glob form ./* instead.

> Finally, (with some help) I understood that grep interpreted
> "-Vfiename" as an option -V and gave the version info instead of
> doing grep "string" *

Correct.

> Maybe there is a way to add space after "-V " and possibly also
> other options that could be used as part of a filename in some
> operating systems?

That is not possible to do in grep.  Since the shell has already
modified the command line to expand the * by the time grep has it.
Only the command shell has that information.  And this is a feature of
the design rather than a bug.  This is all happening by the
intentional design of it.

Since this is not a bug but simply the usage of it I have marked this
bug as closed.  However further discussion is logged to the ticket.
Other people who read the ticket will see the resolution and will
learn from it.  If you have further questions or comments please
follow-up.  The ticket can always be opened again if there is an
updated reason to do so.

Bob

Message #13 received at 54174 <at> debbugs.gnu.org (full text, mbox):

From: sur-behoffski <sur_behoffski <at> grouse.com.au>
To: 54174 <at> debbugs.gnu.org
Subject: Re: bug#54174: (MacOS Monterey 12.2.1: zsh): grep "string" * is
 interpreted as grep -V when directory has a filename "-Vfilename.ext"
Date: Sun, 27 Feb 2022 09:31:13 +1030

On 2/27/22 05:03, Marja Koivunen wrote:
> I had a directory with filenames that started with “-“
> 
> doing grep on that directory for a “string" did not find anything
> although “string was on some of the files”
> 
> grep just kept repeating something … FreeBSD
> 
> Finally, (with some help) I understood that grep interpreted “-Vfiename” as an option -V and 
> gave the version info instead of doing grep “string” *
> 
> Maybe there is a way to add space after “-V “ and possibly also other options that could be used as  part of a filename in some operating systems?
> 
> 

G'day Marja,

Filename globbing is very handy, but there are a number of nasty surprises
for users that aren't wary.

David A. Wheeler has written a long essay on the surprises, and about how
to avoid them, that is highly worth reading.  For example, what about a
filename that has spaces:  "this has four sections.pdf"?

A LOT of interesting essays, many on security and/or F/OSS topics, are
itemised at David's home page:

        https://dwheeler.com/

The essay, "Filenames and Pathnames in Shell: How to do it Correctly" is at:

        https://dwheeler.com/essays/filenames-in-shell.html

Hope this helps,

sur-behoffski (Brenton Hoff)
programmer, Grouse Software

Message #16 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Gary Johnson <garyjohn <at> spocom.com>
To: bug-grep <at> gnu.org
Subject: Re: bug#54174: (MacOS Monterey 12.2.1: zsh): grep "string" * is
 interpreted as grep -V when directory has a filename "-Vfilename.ext"
Date: Sat, 26 Feb 2022 14:55:33 -0800

On 2022-02-26, Marja Koivunen wrote:
> I had a directory with filenames that started with “-“
> 
> doing grep on that directory for a “string" did not find anything
> although “string was on some of the files”
> 
> grep just kept repeating something … FreeBSD
> 
> Finally, (with some help) I understood that grep interpreted “-Vfiename” as an option -V and 
> gave the version info instead of doing grep “string” *

The usual solution for that problem is to use "--" to indicate that
the words following are file names and not options.  For example, if
you execute

    grep string *

in a directory containing some file names beginning with "-", those
names will be treated as options.  If instead you execute

    grep string -- *

then all file names will be treated as file names.

An example of this is given near the bottom of recent versions of
the grep(1) man page, under EXAMPLES.

Regards,
Gary

Message #19 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Marja Koivunen <marja <at> innomore.com>
To: bug-grep <at> gnu.org
Subject: Re: (MacOS Monterey 12.2.1: zsh): grep "string" * is interpreted as
 grep -V when directory has a filename  "-Vfilename.ext"
Date: Sun, 27 Feb 2022 14:31:29 -0500

[Message part 1 (text/plain, inline)]

Ok. I understand it is not possible to make “-V “ instead of “-Vfilename” because so many existing scripts rely on thatgrept works a certain way.

Maybe what is possible is to add a line to manual that explains that certain filenames that are OK in modern OS might not work with grep.

I have used “-" sometimes to keep certain file at the top of directory hierarchy. In this case I was organizing libraries and marked those that I was not using with “-“.


Here is command-line example that you asked.



Even looked from the manual the simple example and did not see what I was doing wrong (I did not read the whole manual.)
So if the program user interface cannot be changed maybe the manual could tell that filenames that start with “-“ might require some additional quoting....

Marja

********* No need to reply *********




On Feb 26, 2022, at 1:33 PM, Marja Koivunen <marja <at> innomore.com> wrote:

I had a directory with filenames that started with “-“

doing grep on that directory for a “string" did not find anything
although “string was on some of the files”

grep just kept repeating something … FreeBSD

Finally, (with some help) I understood that grep interpreted “-Vfiename” as an option -V and 
gave the version info instead of doing grep “string” *

Maybe there is a way to add space after “-V “ and possibly also other options that could be used as  part of a filename in some operating systems?

[Message part 2 (text/html, inline)]

[Screen Shot 2022-02-27 at 1.57.54 PM.png (image/png, inline)]

[Screen Shot 2022-02-27 at 1.46.18 PM.png (image/png, inline)]

Message #22 received at 54174 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Marja Koivunen <marja <at> innomore.com>
Cc: 54174 <at> debbugs.gnu.org
Subject: Re: bug#54174: (MacOS Monterey 12.2.1: zsh): grep "string" * is
 interpreted as grep -V when directory has a filename "-Vfilename.ext"
Date: Sun, 27 Feb 2022 15:22:29 -0800

[Message part 1 (text/plain, inline)]

On 2/27/22 11:31, Marja Koivunen wrote:
> Maybe what is possible is to add a line to manual that explains that certain filenames that are OK in modern OS might not work with grep.
> 
> I have used “-" sometimes to keep certain file at the top of directory hierarchy. In this case I was organizing libraries and marked those that I was not using with “-“.

Although that practice is tempting, it is probably not a good idea on 
any platform where you might use shell commands. POSIX says portable 
file names can't begin with '-'.

There is a section of the grep manual that talks about this but perhaps 
it is not clear enough. I installed the attached patch to try to make it 
better.

[0001-doc-more-on-leading.patch (text/x-patch, attachment)]

Message #25 received at submit <at> debbugs.gnu.org (full text, mbox):

From: sur-behoffski <sur_behoffski <at> grouse.com.au>
To: bug-grep <at> gnu.org
Subject: Re: bug#54174: (MacOS Monterey 12.2.1: zsh): grep "string" * is
 interpreted as grep -V when directory has a filename "-Vfilename.ext"
Date: Mon, 28 Feb 2022 15:32:02 +1030

On 2/28/22 06:01, Marja Koivunen wrote:
> Ok. I understand it is not possible to make “-V “ instead of “-Vfilename” because so many existing scripts rely on thatgrept works a certain way.
> 
> Maybe there is a way to add space after “-V “ and possibly also other options that could be used as  part of a filename in some operating systems?

G'day again Marja,

[The following message is Bourne shell-specific; I haven't researched
exactly how this relates to zsh (especially the environment variable
IFS.]

--

David Wheeler's essay suggests modifying IFS (input field separator) to
exclude spaces, as these occasionally appear in filenames:

The default in the shell is:

        IFS="$(printf ' \n\t')" # space, newline and TAB

David suggests eliminating the space as a separator, as it causes
more trouble than its worth in some situations:

        IFS="$(printf '\n\t')"

Going past David's suggestion, I've found that, occasionally, usually
during debug output, I want TAB-separated words, not Newline-separated
words.  This leads to a further, optional tweak:

        IFS="$(printf '\t\n\t')"

Note that, in all of this, there is a very subtle dance going on
between two different entities:  The Grep program, and the (Bourne?)
Shell.  You need to understand what the shell is doing, before
looking at Grep's (or ant other's) dealing with the modified
command line.

An example of how subtle/difficult the interaction can be is that
the version control program Git, originally implemented as shell
scripts, was re-implemented in C, at least partially because of these
sorts of interactions.

--

[An advert for my "PosixExec.lua" add-on to "luaposix" rock deleted.]

--

Hope this helps,

s-b etc etc

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.