GNU bug report logs - #65416
Feature request: include first line of file in output

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 65416 in the body.
You can then email your comments to 65416 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Daniel Green <ddgreen <at> gmail.com>
To: bug-grep <at> gnu.org
Subject: Feature request: include first line of file in output
Date: Sun, 20 Aug 2023 14:53:02 -0400

[Message part 1 (text/plain, inline)]

I'm frequently searching CSV files with 20-30 columns, and when there's a
hit it can be hard to know what the columns are. An option to also print
the first line of a file (either always, or only if that file had a match
to the pattern) in addition to any hits would be nice.

Thanks,
Dan

[Message part 2 (text/html, inline)]

Message #8 received at 65416 <at> debbugs.gnu.org (full text, mbox):

From: arnold <at> skeeve.com
To: ddgreen <at> gmail.com, 65416 <at> debbugs.gnu.org
Subject: Re: bug#65416: Feature request: include first line of file in output
Date: Mon, 21 Aug 2023 10:57:14 -0600

Daniel Green <ddgreen <at> gmail.com> wrote:

> I'm frequently searching CSV files with 20-30 columns, and when there's a
> hit it can be hard to know what the columns are. An option to also print
> the first line of a file (either always, or only if that file had a match
> to the pattern) in addition to any hits would be nice.
>
> Thanks,
> Dan

It sounds like awk would be a better tool:

	awk 'FNR == 1 || /pattern/' files ...

should do the trick.

HTH,

Arnold

Message #11 received at 65416 <at> debbugs.gnu.org (full text, mbox):

From: arnold <at> skeeve.com
To: ddgreen <at> gmail.com, arnold <at> skeeve.com
Cc: 65416 <at> debbugs.gnu.org
Subject: Re: bug#65416: Feature request: include first line of file in output
Date: Mon, 21 Aug 2023 12:37:12 -0600

Gawk 4.0.2 is 11 years old. Try timing the current version,
I'll bet it's faster.  And it solves your problem NOW,
instead of waiting for a feature that the grep developers
aren't likely to add.

My two cents of course.

Arnold

Daniel Green <ddgreen <at> gmail.com> wrote:

> That works, as well as the Perl version I've been using:
>
>     perl -ne 'print if ($. == 1 || /pattern/)'
>
> But timings for a real-life example (3GB file with ~16m lines, CentOS 7)
> show the problem:
>
>     grep (v2.20):    ~1.15s
>     perl (v5.36.1):  ~4.48s
>      awk (v4.0.2):  ~10.81s
>
> Admittedly grep is just searching in those timings, but I suspect it could
> accomplish the full task with a minimal decrease in speed.
>
> Dan
>
> On Mon, Aug 21, 2023 at 12:57 PM <arnold <at> skeeve.com> wrote:
>
> > Daniel Green <ddgreen <at> gmail.com> wrote:
> >
> > > I'm frequently searching CSV files with 20-30 columns, and when there's a
> > > hit it can be hard to know what the columns are. An option to also print
> > > the first line of a file (either always, or only if that file had a match
> > > to the pattern) in addition to any hits would be nice.
> > >
> > > Thanks,
> > > Dan
> >
> > It sounds like awk would be a better tool:
> >
> >         awk 'FNR == 1 || /pattern/' files ...
> >
> > should do the trick.
> >
> > HTH,
> >
> > Arnold
> >

Message #14 received at 65416 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: arnold <at> skeeve.com, ddgreen <at> gmail.com
Cc: 65416 <at> debbugs.gnu.org
Subject: Re: bug#65416: Feature request: include first line of file in output
Date: Mon, 21 Aug 2023 13:43:36 -0500

On 8/21/23 13:37, arnold <at> skeeve.com wrote:
> it solves your problem NOW,
> instead of waiting for a feature that the grep developers
> aren't likely to add.

Yes, Grep already has a lot of features that in hindsight would have 
better addressed by saying "Use Awk".

Message #17 received at 65416 <at> debbugs.gnu.org (full text, mbox):

From: Daniel Green <ddgreen <at> gmail.com>
To: arnold <at> skeeve.com
Cc: 65416 <at> debbugs.gnu.org
Subject: Re: bug#65416: Feature request: include first line of file in output
Date: Mon, 21 Aug 2023 14:03:11 -0400

[Message part 1 (text/plain, inline)]

That works, as well as the Perl version I've been using:

    perl -ne 'print if ($. == 1 || /pattern/)'

But timings for a real-life example (3GB file with ~16m lines, CentOS 7)
show the problem:

    grep (v2.20):    ~1.15s
    perl (v5.36.1):  ~4.48s
     awk (v4.0.2):  ~10.81s

Admittedly grep is just searching in those timings, but I suspect it could
accomplish the full task with a minimal decrease in speed.

Dan

On Mon, Aug 21, 2023 at 12:57 PM <arnold <at> skeeve.com> wrote:

> Daniel Green <ddgreen <at> gmail.com> wrote:
>
> > I'm frequently searching CSV files with 20-30 columns, and when there's a
> > hit it can be hard to know what the columns are. An option to also print
> > the first line of a file (either always, or only if that file had a match
> > to the pattern) in addition to any hits would be nice.
> >
> > Thanks,
> > Dan
>
> It sounds like awk would be a better tool:
>
>         awk 'FNR == 1 || /pattern/' files ...
>
> should do the trick.
>
> HTH,
>
> Arnold
>

[Message part 2 (text/html, inline)]

Message #20 received at 65416 <at> debbugs.gnu.org (full text, mbox):

From: arnold <at> skeeve.com
To: ddgreen <at> gmail.com, arnold <at> skeeve.com
Cc: 65416 <at> debbugs.gnu.org
Subject: Re: bug#65416: Feature request: include first line of file in output
Date: Tue, 22 Aug 2023 20:33:19 -0600

I can't speak for the grep guys, but at least I was correct that
current gawk is much faster than gawk 4.0.2.

Arnold

Daniel Green <ddgreen <at> gmail.com> wrote:

> I don't have access to a newer gawk where I did the initial timings, but I
> ran an almost identical test on my home machine.
>
>     grep (v3.11):                                              ~0.60s
>     perl (v5.38.0):                                            ~3.21s
>     gawk (v4.0.2 built from source with `-O3 -march=native`): ~10.22s
>     gawk (v5.2.2 built from source with `-O3 -march=native`):  ~4.95s
>
> If grep will never add this functionality I'll survive, it just seemed like
> it might not be too much work to implement, and would probably still be
> much faster than using awk/perl. I've never looked at the grep source code
> before, but could be tempted to try implementing it myself if there was any
> chance of the path being accepted.
>
> Dan
>
> On Mon, Aug 21, 2023 at 2:37 PM <arnold <at> skeeve.com> wrote:
>
> > Gawk 4.0.2 is 11 years old. Try timing the current version,
> > I'll bet it's faster.  And it solves your problem NOW,
> > instead of waiting for a feature that the grep developers
> > aren't likely to add.
> >
> > My two cents of course.
> >
> > Arnold
> >
> > Daniel Green <ddgreen <at> gmail.com> wrote:
> >
> > > That works, as well as the Perl version I've been using:
> > >
> > >     perl -ne 'print if ($. == 1 || /pattern/)'
> > >
> > > But timings for a real-life example (3GB file with ~16m lines, CentOS 7)
> > > show the problem:
> > >
> > >     grep (v2.20):    ~1.15s
> > >     perl (v5.36.1):  ~4.48s
> > >      awk (v4.0.2):  ~10.81s
> > >
> > > Admittedly grep is just searching in those timings, but I suspect it
> > could
> > > accomplish the full task with a minimal decrease in speed.
> > >
> > > Dan
> > >
> > > On Mon, Aug 21, 2023 at 12:57 PM <arnold <at> skeeve.com> wrote:
> > >
> > > > Daniel Green <ddgreen <at> gmail.com> wrote:
> > > >
> > > > > I'm frequently searching CSV files with 20-30 columns, and when
> > there's a
> > > > > hit it can be hard to know what the columns are. An option to also
> > print
> > > > > the first line of a file (either always, or only if that file had a
> > match
> > > > > to the pattern) in addition to any hits would be nice.
> > > > >
> > > > > Thanks,
> > > > > Dan
> > > >
> > > > It sounds like awk would be a better tool:
> > > >
> > > >         awk 'FNR == 1 || /pattern/' files ...
> > > >
> > > > should do the trick.
> > > >
> > > > HTH,
> > > >
> > > > Arnold
> > > >
> >

Message #23 received at 65416 <at> debbugs.gnu.org (full text, mbox):

From: Carlo Arenas <carenas <at> gmail.com>
To: arnold <at> skeeve.com
Cc: ddgreen <at> gmail.com, 65416 <at> debbugs.gnu.org
Subject: Re: bug#65416: Feature request: include first line of file in output
Date: Tue, 22 Aug 2023 23:20:19 -0700

> Daniel Green <ddgreen <at> gmail.com> wrote:
>
> > I've never looked at the grep source code
> > before, but could be tempted to try implementing it myself if there was any
> > chance of the path being accepted.

A slightly more complicated perl script would be my first choice if
coding is the solution, but grep already has a feature that could be
used to provide a solution as shown by the following scriptlet
(including an scaled data file) :

$ cat > c.csv
USER,TIP
john,0
jane,10
carenas,100
$ ( grep -m1 USER && grep carenas ) < c.csv
USER,TIP
carenas,100

Carlo

Message #26 received at 65416 <at> debbugs.gnu.org (full text, mbox):

From: Daniel Green <ddgreen <at> gmail.com>
To: arnold <at> skeeve.com
Cc: 65416 <at> debbugs.gnu.org
Subject: Re: bug#65416: Feature request: include first line of file in output
Date: Tue, 22 Aug 2023 22:12:25 -0400

[Message part 1 (text/plain, inline)]

I don't have access to a newer gawk where I did the initial timings, but I
ran an almost identical test on my home machine.

    grep (v3.11):                                              ~0.60s
    perl (v5.38.0):                                            ~3.21s
    gawk (v4.0.2 built from source with `-O3 -march=native`): ~10.22s
    gawk (v5.2.2 built from source with `-O3 -march=native`):  ~4.95s

If grep will never add this functionality I'll survive, it just seemed like
it might not be too much work to implement, and would probably still be
much faster than using awk/perl. I've never looked at the grep source code
before, but could be tempted to try implementing it myself if there was any
chance of the path being accepted.

Dan

On Mon, Aug 21, 2023 at 2:37 PM <arnold <at> skeeve.com> wrote:

> Gawk 4.0.2 is 11 years old. Try timing the current version,
> I'll bet it's faster.  And it solves your problem NOW,
> instead of waiting for a feature that the grep developers
> aren't likely to add.
>
> My two cents of course.
>
> Arnold
>
> Daniel Green <ddgreen <at> gmail.com> wrote:
>
> > That works, as well as the Perl version I've been using:
> >
> >     perl -ne 'print if ($. == 1 || /pattern/)'
> >
> > But timings for a real-life example (3GB file with ~16m lines, CentOS 7)
> > show the problem:
> >
> >     grep (v2.20):    ~1.15s
> >     perl (v5.36.1):  ~4.48s
> >      awk (v4.0.2):  ~10.81s
> >
> > Admittedly grep is just searching in those timings, but I suspect it
> could
> > accomplish the full task with a minimal decrease in speed.
> >
> > Dan
> >
> > On Mon, Aug 21, 2023 at 12:57 PM <arnold <at> skeeve.com> wrote:
> >
> > > Daniel Green <ddgreen <at> gmail.com> wrote:
> > >
> > > > I'm frequently searching CSV files with 20-30 columns, and when
> there's a
> > > > hit it can be hard to know what the columns are. An option to also
> print
> > > > the first line of a file (either always, or only if that file had a
> match
> > > > to the pattern) in addition to any hits would be nice.
> > > >
> > > > Thanks,
> > > > Dan
> > >
> > > It sounds like awk would be a better tool:
> > >
> > >         awk 'FNR == 1 || /pattern/' files ...
> > >
> > > should do the trick.
> > >
> > > HTH,
> > >
> > > Arnold
> > >
>

[Message part 2 (text/html, inline)]

Message #29 received at 65416 <at> debbugs.gnu.org (full text, mbox):

From: "Paul Jackson" <pj <at> usa.net>
To: "Daniel Green" <ddgreen <at> gmail.com>, arnold <at> skeeve.com
Cc: 65416 <at> debbugs.gnu.org
Subject: Re: bug#65416: Feature request: include first line of file in output
Date: Wed, 23 Aug 2023 03:20:26 -0500

sed and awk can also to this (1st line plus any matching lines)

Following transcript from zsh session on my fast Ryzen:

$ <<-'@@' time sh -c "grep -m1 USER && grep carenas"
USER,TIP
john,0
jane,10
carenas,100
@@
USER,TIP
carenas,100
sh -c "grep -m1 USER && grep carenas"  0.00s user 0.00s system 93% cpu 0.003 total

$ <<-'@@' time sed -n -e 1p -e /carenas/p
USER,TIP
john,0
jane,10
carenas,100
@@
USER,TIP
carenas,100
sed -n -e 1p -e /carenas/p  0.00s user 0.00s system 80% cpu 0.001 total

$ <<-'@@' time awk 'NR == 1 || /carenas/'
USER,TIP
john,0
jane,10
carenas,100
@@
USER,TIP
carenas,100
awk 'NR == 1 || /carenas/'  0.00s user 0.00s system 88% cpu 0.002 total

As I expected, sed is fastest, grep next, and awk slowest of the three,
but the 1, 2, and 3 millisecond totals are within the margin of test error.

-- 
                Paul Jackson
                pj <at> usa.net

Message #32 received at 65416 <at> debbugs.gnu.org (full text, mbox):

From: "Paul Jackson" <pj <at> usa.net>
To: "Daniel Green" <ddgreen <at> gmail.com>, arnold <at> skeeve.com
Cc: 65416 <at> debbugs.gnu.org
Subject: Re: bug#65416: Feature request: include first line of file in output
Date: Wed, 23 Aug 2023 03:23:01 -0500

oops - grep slower than awk, not the other way around,
on these _highly_ inconclusive timings.

-- 
                Paul Jackson
                pj <at> usa.net

Message #35 received at 65416 <at> debbugs.gnu.org (full text, mbox):

From: Daniel Green <ddgreen <at> gmail.com>
To: Paul Jackson <pj <at> usa.net>
Cc: arnold <at> skeeve.com, 65416 <at> debbugs.gnu.org
Subject: Re: bug#65416: Feature request: include first line of file in output
Date: Wed, 23 Aug 2023 10:20:05 -0400

[Message part 1 (text/plain, inline)]

On the original test machine I timed the sed solution, as well as `(grep
-m1 . 'file' && grep 'pattern' 'file')` and `(mapfile -n1 <'file' && echo
$MAPFILE[0] && grep 'pattern' 'file')` and  `(head -n1 'file' && grep
'pattern' 'file')`. Total table of speeds.

    grep (v2.20):    ~1.15s
    perl (v5.36.1):  ~4.48s
     awk (v4.0.2):  ~10.81s
     sed (v4.2.2):   ~8.15s
     grep && grep:   ~1.15s
  mapfile && grep:   ~1.15s
     head && grep:   ~1.15s

I can write a shell function to make the head+grep version a little easier
to use in practice (i.e., loop over the list of files passed calling
head+grep on each one instead of calling head on the list and then grep on
the list), but I believe it would be difficult to change any options given
to grep. I still think the best combination of speed + output as I imagine
+ ease of integrating with changing grep options used is accomplished by a
new option for grep. But if there's no interest then this feature request
can be closed.

Dan

On Wed, Aug 23, 2023 at 4:23 AM Paul Jackson <pj <at> usa.net> wrote:

> oops - grep slower than awk, not the other way around,
> on these _highly_ inconclusive timings.
>
> --
>                 Paul Jackson
>                 pj <at> usa.net
>

[Message part 2 (text/html, inline)]

Message #40 received at 65416 <at> debbugs.gnu.org (full text, mbox):

From: "Paul Jackson" <pj <at> usa.net>
To: "Daniel Green" <ddgreen <at> gmail.com>
Cc: arnold <at> skeeve.com, 65416 <at> debbugs.gnu.org
Subject: Re: bug#65416: Feature request: include first line of file in output
Date: Wed, 23 Aug 2023 17:21:37 -0500

[Message part 1 (text/plain, inline)]

Ah - those times show another reason why one might
be motivated to keep requesting more options be added
to grep.

From those timings, and from looking at the source, it's clear
that the FSF rewrote grep from scratch, sometime back in the
late 1980's or early 1990's, to have fast reads, whereas sed is
still using stdio fread in a classical manner, which is a painfully
slower double copy solution.

If sed were still a widely used command in performance sensitive
applications, it should have some serious TLC applied to its
performance.

However, since the pool of Jurassic Park Dinosaurs who can (and
perhaps do) compose sed commands in their sleep is a nearly
extinct breed, I see no sufficient interest in accepting such a rewrite
of sed, even if it showed up as a proposed checkin.

That grep can even seriously beat perl for such raw read performance
is impressive.  Perl used to be the King of such challenges.

-- 
                Paul Jackson
                pj <at> usa.net

[Message part 2 (text/html, inline)]

Message #43 received at 65416 <at> debbugs.gnu.org (full text, mbox):

From: Daniel Green <ddgreen <at> gmail.com>
To: Paul Jackson <pj <at> usa.net>
Cc: arnold <at> skeeve.com, 65416 <at> debbugs.gnu.org
Subject: Re: bug#65416: Feature request: include first line of file in output
Date: Wed, 23 Aug 2023 20:34:46 -0400

[Message part 1 (text/plain, inline)]

Re Perl's read speed, it's faster when not doing the line number check for
every line. So `perl -ne 'print if (/pattern/)'` is only ~2.60s, compared
to ~3.28s for `perl -ne 'print if ($. == 1 || /pattern/)'`. Doing nothing
in Perl, i.e., `perl -ne ''` is only ~1.38s.

Dan

On Wed, Aug 23, 2023 at 6:22 PM Paul Jackson <pj <at> usa.net> wrote:

> Ah - those times show another reason why one might
> be motivated to keep requesting more options be added
> to grep.
>
> From those timings, and from looking at the source, it's clear
> that the FSF rewrote grep from scratch, sometime back in the
> late 1980's or early 1990's, to have fast reads, whereas sed is
> still using stdio fread in a classical manner, which is a painfully
> slower double copy solution.
>
> If sed were still a widely used command in performance sensitive
> applications, it should have some serious TLC applied to its
> performance.
>
> However, since the pool of Jurassic Park Dinosaurs who can (and
> perhaps do) compose sed commands in their sleep is a nearly
> extinct breed, I see no sufficient interest in accepting such a rewrite
> of sed, even if it showed up as a proposed checkin.
>
> That grep can even seriously beat perl for such raw read performance
> is impressive.  Perl used to be the King of such challenges.
>
> --
>                 Paul Jackson
>                 pj <at> usa.net
>
>
>

[Message part 2 (text/html, inline)]

Message #46 received at 65416 <at> debbugs.gnu.org (full text, mbox):

From: lacsaP Patatetom <patatetom <at> gmail.com>
To: Daniel Green <ddgreen <at> gmail.com>
Cc: Paul Jackson <pj <at> usa.net>, arnold <at> skeeve.com, 65416 <at> debbugs.gnu.org
Subject: Re: bug#65416: Feature request: include first line of file in output
Date: Tue, 29 Aug 2023 15:56:58 +0200

[Message part 1 (text/plain, inline)]

Le jeu. 24 août 2023 à 08:58, Daniel Green <ddgreen <at> gmail.com> a écrit :

> Re Perl's read speed, it's faster when not doing the line number check for
> every line. So `perl -ne 'print if (/pattern/)'` is only ~2.60s, compared
> to ~3.28s for `perl -ne 'print if ($. == 1 || /pattern/)'`. Doing nothing
> in Perl, i.e., `perl -ne ''` is only ~1.38s.
>
> Dan
>
> On Wed, Aug 23, 2023 at 6:22 PM Paul Jackson <pj <at> usa.net> wrote:
>
> > Ah - those times show another reason why one might
> > be motivated to keep requesting more options be added
> > to grep.
> >
> > From those timings, and from looking at the source, it's clear
> > that the FSF rewrote grep from scratch, sometime back in the
> > late 1980's or early 1990's, to have fast reads, whereas sed is
> > still using stdio fread in a classical manner, which is a painfully
> > slower double copy solution.
> >
> > If sed were still a widely used command in performance sensitive
> > applications, it should have some serious TLC applied to its
> > performance.
> >
> > However, since the pool of Jurassic Park Dinosaurs who can (and
> > perhaps do) compose sed commands in their sleep is a nearly
> > extinct breed, I see no sufficient interest in accepting such a rewrite
> > of sed, even if it showed up as a proposed checkin.
> >
> > That grep can even seriously beat perl for such raw read performance
> > is impressive.  Perl used to be the King of such challenges.
> >
> > --
> >                 Paul Jackson
> >                 pj <at> usa.net
> >
> >
> >
>

with a function, something like this :

headgrep() {
 head -1 "$2"
 grep "$1" "$2"
}

[Message part 2 (text/html, inline)]

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.