GNU bug report logs - #32073
Improvements in Grep

Previous Next

To reply to this bug, email your comments to 32073 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Sergiu Hlihor <sh <at> discovergy.com>
To: bug-grep <at> gnu.org
Subject: Improvements in Grep
Date: Fri, 6 Jul 2018 18:26:17 +0200

[Message part 1 (text/plain, inline)]

Hello,
     I'm using grep over Ubuntu Server 14.04 (Grep version 2.16). While
grepping over large files I've noticed Grep is painfully slow. The
bottleneck seems to be the read block which is extremely low (looks like
64KB). For large files residing over big HDD RAID arrays, this request
barely reaches one drive and based on CPU usage, grep is idling more or
less. Given my tests for such scenarios, a read block size of at least
512KB would be way more efficient. It's very likely that optimum would be
1MB+. Also, such increase in buffer size would also benefit slightly SSDs
where maximum sequential throughput is usually achieved when reading at
256KB+ block size.
     If this is already possible in newer versions or configurable, I'd
appreciate some hints about the new version which contains or about the way
I can configure it to increase the read block size.

Thanks and best regards,
Sergiu

[Message part 2 (text/html, inline)]

Message #8 received at 32073 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Sergiu Hlihor <sh <at> discovergy.com>, 32073 <at> debbugs.gnu.org
Subject: Re: bug#32073: Improvements in Grep
Date: Fri, 6 Jul 2018 15:06:34 -0700

Sergiu Hlihor wrote:
> Given my tests for such scenarios, a read block size of at least
> 512KB would be way more efficient.

Does stdio do this already? If not, why not? How could grep reasonably configure 
a good block size?

Message #11 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Dennis Clarke <dclarke <at> blastwave.org>
To: bug-grep <at> gnu.org
Subject: Re: bug#32073: Improvements in Grep
Date: Fri, 6 Jul 2018 18:44:36 -0400

On 07/06/2018 06:06 PM, Paul Eggert wrote:
> Sergiu Hlihor wrote:
>> Given my tests for such scenarios, a read block size of at least
>> 512KB would be way more efficient.
> 
> Does stdio do this already? If not, why not? How could grep reasonably 
> configure a good block size?

This seems to be a very specific complaint which is only of value on a
very specific system and usage case.  There is no way that grep could
configure a "good block size" unless it were tailor built.  Doesn't
seem to be a reasonable RFE.  In my opinion.

Dennis

Message #14 received at 32073 <at> debbugs.gnu.org (full text, mbox):

From: Jim Meyering <jim <at> meyering.net>
To: Sergiu Hlihor <sh <at> discovergy.com>
Cc: 32073 <at> debbugs.gnu.org
Subject: Re: bug#32073: Improvements in Grep
Date: Fri, 6 Jul 2018 17:33:08 -0700

[Message part 1 (text/plain, inline)]

On Fri, Jul 6, 2018 at 9:26 AM, Sergiu Hlihor <sh <at> discovergy.com> wrote:
> Hello,
>      I'm using grep over Ubuntu Server 14.04 (Grep version 2.16). While
> grepping over large files I've noticed Grep is painfully slow. The
> bottleneck seems to be the read block which is extremely low (looks like
> 64KB). For large files residing over big HDD RAID arrays, this request
> barely reaches one drive and based on CPU usage, grep is idling more or
> less. Given my tests for such scenarios, a read block size of at least
> 512KB would be way more efficient. It's very likely that optimum would be
> 1MB+. Also, such increase in buffer size would also benefit slightly SSDs
> where maximum sequential throughput is usually achieved when reading at
> 256KB+ block size.
>      If this is already possible in newer versions or configurable, I'd
> appreciate some hints about the new version which contains or about the way
> I can configure it to increase the read block size.

Thanks for raising the issue.
This makes me think we should follow Coreutils' lead[0] and increase
grep's initial buffer size from 32KiB, probably to 128KiB. I will time
with the attached diff on a few systems.

[0] https://git.savannah.gnu.org/cgit/coreutils.git/commit/?id=v8.22-103-g74ca6e84c

[grep-bufsize-increase.diff (application/octet-stream, attachment)]

Message #17 received at 32073 <at> debbugs.gnu.org (full text, mbox):

From: Sergiu Hlihor <sh <at> discovergy.com>
To: Jim Meyering <jim <at> meyering.net>
Cc: 32073 <at> debbugs.gnu.org
Subject: Re: bug#32073: Improvements in Grep
Date: Sat, 7 Jul 2018 03:31:19 +0200

[Message part 1 (text/plain, inline)]

To add, the increase to 128KiB is good, but for RAID arrays with light to
medium load, this is not sufficient. In a system without any load, the HDD
can read ahead and always serve the next request from buffer thus reading
at full sequential speed of ~200MB/s . In a RAID 10 configuration with 12
hdds where strip size is set to 128KB, every HDD is hit at every 6th
request. There is enough delay between reads hitting the same drive that
the read ahead buffer often gets discarded which basically limits the
throughput to max IOPS x buffer size  = ~10-20MiB for 128KiB.
I have such systems in production environments and I often see read speeds
under 10MiB and read await >10ms which means that read ahead buffer is
already discarded. At the same load conditions, if I read the data using
utilities which can do 512KiB buffer size, I see read speed varying between
50 and 400MiB. Grep has an average CPU load of 2-3% of the given machine
under such low reads, therefore it can do much more if reading is optimized.

On 7 July 2018 at 02:33, Jim Meyering <jim <at> meyering.net> wrote:

> On Fri, Jul 6, 2018 at 9:26 AM, Sergiu Hlihor <sh <at> discovergy.com> wrote:
> > Hello,
> >      I'm using grep over Ubuntu Server 14.04 (Grep version 2.16). While
> > grepping over large files I've noticed Grep is painfully slow. The
> > bottleneck seems to be the read block which is extremely low (looks like
> > 64KB). For large files residing over big HDD RAID arrays, this request
> > barely reaches one drive and based on CPU usage, grep is idling more or
> > less. Given my tests for such scenarios, a read block size of at least
> > 512KB would be way more efficient. It's very likely that optimum would be
> > 1MB+. Also, such increase in buffer size would also benefit slightly SSDs
> > where maximum sequential throughput is usually achieved when reading at
> > 256KB+ block size.
> >      If this is already possible in newer versions or configurable, I'd
> > appreciate some hints about the new version which contains or about the
> way
> > I can configure it to increase the read block size.
>
> Thanks for raising the issue.
> This makes me think we should follow Coreutils' lead[0] and increase
> grep's initial buffer size from 32KiB, probably to 128KiB. I will time
> with the attached diff on a few systems.
>
> [0] https://git.savannah.gnu.org/cgit/coreutils.git/commit/?id=
> v8.22-103-g74ca6e84c
>



-- 
_____________________________________________

Senior Software Engineer & Team leader

Telefon: +49 (0) 6221 7787-481

Email: sh <at> discovergy.com

*Discovergy GmbH*
_____________________________________________

Registergericht: Amtsgericht Aachen HRB 15391

Geschäftsführer: Ralf Esser | Bernhard Seidl | Nikolaus Starzacher
Diese E-Mail und eventuell angehängte Dateien sind nur für den oben
genannten Empfänger bestimmt und können vertrauliche Informationen
enthalten. Sollten Sie nicht der Empfänger sein, ist jede Verbreitung,
Weiterleitung und Kopie verboten. Wenn Sie diese E-Mail versehentlich
erhalten haben, senden Sie diese Mail zurück oder unterrichten umgehend den
Absender unter oben genannten Kontaktdaten. Bitte löschen Sie diese
Nachricht in diesem Fall umgehend. Vielen Dank.

[Message part 2 (text/html, inline)]

Message #20 received at 32073 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Sergiu Hlihor <sh <at> discovergy.com>
Cc: 32073 <at> debbugs.gnu.org, Dennis Clarke <dclarke <at> blastwave.org>,
 Jim Meyering <jim <at> meyering.net>
Subject: Re: Improvements in Grep (Bug#32073)
Date: Tue, 31 Dec 2019 23:52:54 -0800

> This makes me think we should follow Coreutils' lead[0] and increase
> grep's initial buffer size from 32KiB, probably to 128KiB.

I see that Jim later installed a patch increasing it to 96 KiB.

Whatever number is chosen, it's "wrong" for some configuration. And I suppose
the particular configuration that Sergiu Hlihor mentioned could be tweaked so
that it worked better with grep (and with other programs).

I'm inclined to mark this bug report as a wishlist item, in the sense that it'd
be nice if grep and/or the OS could pick buffer sizes more intelligently (though
it's not clear how grep and/or the OS could go about this).

Message #25 received at 32073 <at> debbugs.gnu.org (full text, mbox):

From: Sergiu Hlihor <sh <at> discovergy.com>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: 32073 <at> debbugs.gnu.org, Dennis Clarke <dclarke <at> blastwave.org>,
 Jim Meyering <jim <at> meyering.net>
Subject: Re: Improvements in Grep (Bug#32073)
Date: Wed, 1 Jan 2020 10:15:16 +0100

[Message part 1 (text/plain, inline)]

This topic is getting more and more frustrating. If you rely on OS, then
you are at the mercy of whatever read ahead configuration you have. And
read ahead is typically 128KB so does not help that much. A HDD RAID 10
array with 12 disks and a strip size of 128KB reaches the maximum read
throughput if read block size is 6 * 128 = 768KB. When issuing read
requests with 128KB , you only hit one HDD, having 1/6 read throughput.
With flash the same. A state of the art SSD that can do 5GB/s reads can
actually do around 1GB/s or less at 128KB block size. Why is so hard to
understand how hardware works and the fact that you need huge block sizes
to actually read at full speed? Why not just exposing the read buffer size
as a configurable parameter, then anyone can just tune it as needed? 96KB
is purely retarded.

On Wed, 1 Jan 2020 at 08:52, Paul Eggert <eggert <at> cs.ucla.edu> wrote:

> > This makes me think we should follow Coreutils' lead[0] and increase
> > grep's initial buffer size from 32KiB, probably to 128KiB.
>
> I see that Jim later installed a patch increasing it to 96 KiB.
>
> Whatever number is chosen, it's "wrong" for some configuration. And I
> suppose
> the particular configuration that Sergiu Hlihor mentioned could be tweaked
> so
> that it worked better with grep (and with other programs).
>
> I'm inclined to mark this bug report as a wishlist item, in the sense that
> it'd
> be nice if grep and/or the OS could pick buffer sizes more intelligently
> (though
> it's not clear how grep and/or the OS could go about this).
>

[Message part 2 (text/html, inline)]

Message #28 received at 32073 <at> debbugs.gnu.org (full text, mbox):

From: arnold <at> skeeve.com
To: sh <at> discovergy.com, eggert <at> cs.ucla.edu
Cc: 32073 <at> debbugs.gnu.org
Subject: Re: bug#32073: Improvements in Grep (Bug#32073)
Date: Wed, 01 Jan 2020 04:19:22 -0700

As a quite serious question, how is someone writing user-level code
supposed to be able to figure out the right buffer size for a particular
file, and to do so portably? ("Show me the code.")

Gawk bases its reads on the st_blksize member in struct stat.  That will
typically be something like 4K - not nearly enough, given your description
below.

Arnold

Sergiu Hlihor <sh <at> discovergy.com> wrote:

> This topic is getting more and more frustrating. If you rely on OS, then
> you are at the mercy of whatever read ahead configuration you have. And
> read ahead is typically 128KB so does not help that much. A HDD RAID 10
> array with 12 disks and a strip size of 128KB reaches the maximum read
> throughput if read block size is 6 * 128 = 768KB. When issuing read
> requests with 128KB , you only hit one HDD, having 1/6 read throughput.
> With flash the same. A state of the art SSD that can do 5GB/s reads can
> actually do around 1GB/s or less at 128KB block size. Why is so hard to
> understand how hardware works and the fact that you need huge block sizes
> to actually read at full speed? Why not just exposing the read buffer size
> as a configurable parameter, then anyone can just tune it as needed? 96KB
> is purely retarded.
>
> On Wed, 1 Jan 2020 at 08:52, Paul Eggert <eggert <at> cs.ucla.edu> wrote:
>
> > > This makes me think we should follow Coreutils' lead[0] and increase
> > > grep's initial buffer size from 32KiB, probably to 128KiB.
> >
> > I see that Jim later installed a patch increasing it to 96 KiB.
> >
> > Whatever number is chosen, it's "wrong" for some configuration. And I
> > suppose
> > the particular configuration that Sergiu Hlihor mentioned could be tweaked
> > so
> > that it worked better with grep (and with other programs).
> >
> > I'm inclined to mark this bug report as a wishlist item, in the sense that
> > it'd
> > be nice if grep and/or the OS could pick buffer sizes more intelligently
> > (though
> > it's not clear how grep and/or the OS could go about this).
> >

Message #31 received at submit <at> debbugs.gnu.org (full text, mbox):

From: "Paul Jackson" <pj <at> usa.net>
To: bug-grep <at> gnu.org
Subject: Re: bug#32073: Improvements in Grep (Bug#32073)
Date: Wed, 01 Jan 2020 05:26:04 -0600

>>  Why not just exposing the read buffer size as a configurable parameter ...

Take a look at the (and I quote) "Hairy buffering mechanism for grep"
input buffering code in the grep source file grep-3.3/src/grep.c, then
you tell me why it's not a runtime variable parameter <grin>.

In other words, the input (and output) i/o buffering and performance
tuning for various situations and kinds of files has been tuned and
refined over many years.  Doing something to the code, such as
making buffer size a run time adjustable parameter, would probably
not be easy, would risk making one usage of grep slower in order
to make some other usage faster, and would risk some nasty bugs.

-- 
                Paul Jackson
                pj <at> usa.net

Message #34 received at 32073 <at> debbugs.gnu.org (full text, mbox):

From: Sergiu Hlihor <sh <at> discovergy.com>
To: arnold <at> skeeve.com
Cc: 32073 <at> debbugs.gnu.org, Paul Eggert <eggert <at> cs.ucla.edu>
Subject: Re: bug#32073: Improvements in Grep (Bug#32073)
Date: Wed, 1 Jan 2020 20:06:39 +0100

[Message part 1 (text/plain, inline)]

Arnold, there is no need to write user code, it is already done in
benchmarks. One of the standard benchmarks when testing HDDs and SSDs is
read throughput vs block size and at different queue depths.  Take a look
at this"
https://www.servethehome.com/wp-content/uploads/2019/12/Corsair-Force-MP600-1TB-ATTO.jpg
. In this benchmark, at queue depth 4 and 128KB block size, the SSD was not
yet able to achieve the maximum throughput 5GB/s. Moreover, if you
extrapolate the results, to a queue depth of 1, you get about ~1.2GB/s out
of over 5GB/s theoretical. Therefore for this particular model you need to
issue read requests at minimum 512KB block size to achieve maximum
throughput. With hard drives I already explained the issue. I have a
production server where the HDD RAID array can do theoretically 2.5GB/s and
I see read speeds over 500MB/s sustained when large block sizes are used
for reads, yet when I use grep, I have a practical bandwidth of 20 to 50
MB/s. Moreover, when it comes to HDDs the math is quite simple and here it
is for a standard HDD at 7200 RPM, 240MB/s:
7200 RPM => 120 revolutions per second
240 MB/s at 120 revolutions => 2MB per revolution
One revolution time  = 1000/120 => 8,33 ms
Read throughput per ms = 240KB

Worst case scenario: each read request requires a full revolution to reach
to the data (head positioning is done concurrently and this can be
ignored).
Seek time: 8.33ms
At 96KB:
 - Read time: 0.4ms
 - Total read latency  = 8.33 + 0.4 = 8.73ms, read throughput  = 1000 /
8.73 * 96KB = 11MB/s
At 512KB:
 - Read time: 2.3ms
 - Total read latency = 8.33 + 2.3 = 10.63ms, read throughput  = 1000 /
10.63 * 512KB = 48MB/s
In practice average seek latencies are 4.16ms so throughput is double. This
is the cold hard reality. In practice, when each one of you is testing, you
are very likely deceived by testing on *one hdd, on an idle system* where
you don't have anything else consuming IO in background like a database. In
such an ideal scenario you do see 240MB/s because HDDs do also read ahead
and by the time the data is transferred over interface and consumed, next
chuck is in the buffer and can be delivered with apparent 0 seek time. This
means first read takes 4ms, next ones takes 0.1ms. With a* HDD RAID array
on a server where your IO is always at 50% load*, if you have a strip size
of 128KB or more, you are hitting one drive at a time, each one with a
penalty of 4.16ms. And due to constant load, by the time you hit the first
hdd again, the read ahead buffer maintained by the HDD itself is also
discarded, so all reads go directly to physical medium. If however you hit
all HDDs at the same time, you will benefit from the read ahead from the
HDD for at least one or more cycles thus having reads with apparent 0
latency and a way higher average bandwidth. The cost of reading from all
HDDs at the same time is a potential of adding extra latencies for all
other applications running, this is why the value should be configurable,
such that best value can be setup based on hardware. The issue of large
block sizes for IO operations is widespread across all tools from Linux,
like rsync or cp and its only getting worse, to an extend where in my
company we are considering writing our own tools for something that should
have worked out of the box. One side issue, which I have to mention as I'm
not aware of implementation details: as we are getting in GB/s territory,
read is best done within it's own thread which then serves the output to
the processing thread. With SSDs that can do multi GB/s this matters.




On Wed, 1 Jan 2020 at 12:19, <arnold <at> skeeve.com> wrote:

> As a quite serious question, how is someone writing user-level code
> supposed to be able to figure out the right buffer size for a particular
> file, and to do so portably? ("Show me the code.")
>
> Gawk bases its reads on the st_blksize member in struct stat.  That will
> typically be something like 4K - not nearly enough, given your description
> below.
>
> Arnold
>
> Sergiu Hlihor <sh <at> discovergy.com> wrote:
>
> > This topic is getting more and more frustrating. If you rely on OS, then
> > you are at the mercy of whatever read ahead configuration you have. And
> > read ahead is typically 128KB so does not help that much. A HDD RAID 10
> > array with 12 disks and a strip size of 128KB reaches the maximum read
> > throughput if read block size is 6 * 128 = 768KB. When issuing read
> > requests with 128KB , you only hit one HDD, having 1/6 read throughput.
> > With flash the same. A state of the art SSD that can do 5GB/s reads can
> > actually do around 1GB/s or less at 128KB block size. Why is so hard to
> > understand how hardware works and the fact that you need huge block sizes
> > to actually read at full speed? Why not just exposing the read buffer
> size
> > as a configurable parameter, then anyone can just tune it as needed? 96KB
> > is purely retarded.
> >
> > On Wed, 1 Jan 2020 at 08:52, Paul Eggert <eggert <at> cs.ucla.edu> wrote:
> >
> > > > This makes me think we should follow Coreutils' lead[0] and increase
> > > > grep's initial buffer size from 32KiB, probably to 128KiB.
> > >
> > > I see that Jim later installed a patch increasing it to 96 KiB.
> > >
> > > Whatever number is chosen, it's "wrong" for some configuration. And I
> > > suppose
> > > the particular configuration that Sergiu Hlihor mentioned could be
> tweaked
> > > so
> > > that it worked better with grep (and with other programs).
> > >
> > > I'm inclined to mark this bug report as a wishlist item, in the sense
> that
> > > it'd
> > > be nice if grep and/or the OS could pick buffer sizes more
> intelligently
> > > (though
> > > it's not clear how grep and/or the OS could go about this).
> > >
>

[Message part 2 (text/html, inline)]

Message #37 received at 32073 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Sergiu Hlihor <sh <at> discovergy.com>
Cc: 32073 <at> debbugs.gnu.org, Dennis Clarke <dclarke <at> blastwave.org>,
 Jim Meyering <jim <at> meyering.net>
Subject: Re: Improvements in Grep (Bug#32073)
Date: Wed, 1 Jan 2020 11:42:53 -0800

On 1/1/20 1:15 AM, Sergiu Hlihor wrote:
> If you rely on OS, then
> you are at the mercy of whatever read ahead configuration you have.

Right, and whatever changes you make to the OS and its read-ahead configuration
will work for all applications, not just for 'grep'. So, change the OS to do
that. There shouldn't be a need to change 'grep' in particular (or 'cp' in
particular, or 'awk' in particular, etc.).

> The issue of large
> block sizes for IO operations is widespread across all tools from Linux,
> like rsync or cp and its only getting worse

Quite right. And it would be painful to have to modify all those tools, and to
maintain those modifications. So modify the OS instead. Scheduling read-ahead is
really the OS's job anyway.

Message #40 received at 32073 <at> debbugs.gnu.org (full text, mbox):

From: Sergiu Hlihor <sh <at> discovergy.com>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: 32073 <at> debbugs.gnu.org, Dennis Clarke <dclarke <at> blastwave.org>,
 Jim Meyering <jim <at> meyering.net>
Subject: Re: Improvements in Grep (Bug#32073)
Date: Wed, 1 Jan 2020 21:04:39 +0100

[Message part 1 (text/plain, inline)]

Paul, I have to correct you. On a production server you have usually a mix
of applications many times including databases. For databases, having a
read ahead means one IO less since usually database access patterns are
random reads. Here actually best is to disable completely read ahead. In
fact, I do have to say that probably best is to disable completely read
ahead and let applications deal with it, either in an automatic fashion,
like reading the optimal IO block size from device  or in a configurable
way with defaults good enough for today's servers. If you now configure the
OS to do a read ahead hitting all HDDs then you induce potentially
unnecessary IO load for all applications which use it, which when having
HDDs is totally unacceptable. That's why the best is to be application
specific and ideally configured to use optimal IO block size.

So no, letting OS to do it is stupid.

On Wed, 1 Jan 2020 at 20:42, Paul Eggert <eggert <at> cs.ucla.edu> wrote:

> On 1/1/20 1:15 AM, Sergiu Hlihor wrote:
> > If you rely on OS, then
> > you are at the mercy of whatever read ahead configuration you have.
>
> Right, and whatever changes you make to the OS and its read-ahead
> configuration
> will work for all applications, not just for 'grep'. So, change the OS to
> do
> that. There shouldn't be a need to change 'grep' in particular (or 'cp' in
> particular, or 'awk' in particular, etc.).
>
> > The issue of large
> > block sizes for IO operations is widespread across all tools from Linux,
> > like rsync or cp and its only getting worse
>
> Quite right. And it would be painful to have to modify all those tools,
> and to
> maintain those modifications. So modify the OS instead. Scheduling
> read-ahead is
> really the OS's job anyway.
>

[Message part 2 (text/html, inline)]

Message #43 received at 32073 <at> debbugs.gnu.org (full text, mbox):

From: arnold <at> skeeve.com
To: sh <at> discovergy.com, arnold <at> skeeve.com
Cc: 32073 <at> debbugs.gnu.org, eggert <at> cs.ucla.edu
Subject: Re: bug#32073: Improvements in Grep (Bug#32073)
Date: Wed, 01 Jan 2020 13:24:26 -0700

Hi.

Sergiu Hlihor <sh <at> discovergy.com> wrote:

> Arnold, there is no need to write user code, it is already done in
> benchmarks. One of the standard benchmarks when testing HDDs and SSDs is
> read throughput vs block size and at different queue depths.

I think you're misunderstanding me, or I am misunderstanding you.

As the gawk maintainer, I can choose the buffer size to use every time
I issue a read(2) system call for any given input file.  Gawk currently
uses the smaller of (a) the file's size or (b) the st_blksize member of
the struct stat array.

If I understand you correctly, this is "not enough"; gawk (grep,
cp, etc.) should all use an optimal buffer size that depends upon the
underlying storage hardware where the file is located.

So far, so good, except for: How do I determine what that number is?
I cannot run a benchmark before opening each and every file. I don't
know of a system call that will give me that number. (If there is,
please point me to it.)

Do you just want a command line option or environment variable
that you, as the application user, can set?

If the latter, it happens that gawk will let you set AWKBUFSIZE and
it will use whatever number you supply for doing reads. (This is
even documented.)

HTH,

Arnold

Message #46 received at 32073 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Sergiu Hlihor <sh <at> discovergy.com>
Cc: 32073 <at> debbugs.gnu.org
Subject: Re: bug#32073: Improvements in Grep (Bug#32073)
Date: Wed, 1 Jan 2020 13:02:38 -0800

On 1/1/20 12:04 PM, Sergiu Hlihor wrote:

> That's why the best is to be application specific

That doesn't mean that one should have to modify every application. One could
instead modify the OS so that it uses different read-ahead heuristics for
different classes of applications. This should be easier to manage than
modifying every individual application.

Message #49 received at submit <at> debbugs.gnu.org (full text, mbox):

From: "Paul Jackson" <pj <at> usa.net>
To: bug-grep <at> gnu.org
Subject: Re: bug#32073: Improvements in Grep (Bug#32073)
Date: Wed, 01 Jan 2020 15:45:54 -0600

From my old Unix fart view point, Paul (the other Paul)
is herding a hundred GNU cats, small command line utilities,
many of which date their origins back to the 1970's, many of
which have over the years grown their own internal i/o routines
with specific performance specializations, but few of which
have much in the way of user customizable i/o blocking and
read-ahead customizations.

Except for the last decade, those commands spent almost
their entire lives running off spinning rust platters, which
grew (immensely) in size over the years, but which did not
change much in other performance characteristics. 

Those commands are in general not well suited to adapting to
provide maximally optimal performance across the recent
generation of storage devices, with their much more varied
performance characteristics.

I'm guessing that Sergiu has some specific needs that it seems
that grep meets, except that grep (like its hundred cat siblings)
lacks the tunable i/o characteristics needed to get maximum
performance across a rapidly evolving variety of these more
recent kinds of storage.

What I've done in situations such as I suspect Sergiu finds
himself in is to code up a custom utility, that met my specific
needs, when I had higher performance demands, while
continuing to make extensive use of the general purpose
classic Unix/Linux command line utilities that Paul E. now
herds.

I can't imagine that it would make sense to attempt to recode
a hundred classic GNU utilities to each be intelligently adaptable
goats/pigs/cats/dogs/cows/bison/... depending on the i/o
terrain they were running on.

Many many thanks to Paul E. for herding these cats all these
many years.  I hope my weird comments to not cause him even
the slightest distress.

(The word "cat" above refers to four legged felines, not to the
concatenate command line utility.)

-- 
                Paul Jackson
                pj <at> usa.net

Message #52 received at 32073 <at> debbugs.gnu.org (full text, mbox):

From: Jim Meyering <jim <at> meyering.net>
To: Sergiu Hlihor <sh <at> discovergy.com>
Cc: 32073 <at> debbugs.gnu.org, Paul Eggert <eggert <at> cs.ucla.edu>,
 Dennis Clarke <dclarke <at> blastwave.org>
Subject: Re: Improvements in Grep (Bug#32073)
Date: Wed, 1 Jan 2020 16:51:00 -0800

On Wed, Jan 1, 2020 at 12:04 PM Sergiu Hlihor <sh <at> discovergy.com> wrote:
> Paul, I have to correct you. On a production server you have usually a mix of applications many times including databases. For databases, having a read ahead means one IO less since usually database access patterns are random reads. Here actually best is to disable completely read ahead. In fact, I do have to say that probably best is to disable completely read ahead and let applications deal with it, either in an automatic fashion, like reading the optimal IO block size from device  or in a configurable way with defaults good enough for today's servers. If you now configure the OS to do a read ahead hitting all HDDs then you induce potentially unnecessary IO load for all applications which use it, which when having HDDs is totally unacceptable. That's why the best is to be application specific and ideally configured to use optimal IO block size.
>
> So no, letting OS to do it is stupid.
>
> On Wed, 1 Jan 2020 at 20:42, Paul Eggert <eggert <at> cs.ucla.edu> wrote:
>>
>> On 1/1/20 1:15 AM, Sergiu Hlihor wrote:
>> > If you rely on OS, then
>> > you are at the mercy of whatever read ahead configuration you have.
>>
>> Right, and whatever changes you make to the OS and its read-ahead configuration
>> will work for all applications, not just for 'grep'. So, change the OS to do
>> that. There shouldn't be a need to change 'grep' in particular (or 'cp' in
>> particular, or 'awk' in particular, etc.).
>>
>> > The issue of large
>> > block sizes for IO operations is widespread across all tools from Linux,
>> > like rsync or cp and its only getting worse
>>
>> Quite right. And it would be painful to have to modify all those tools, and to
>> maintain those modifications. So modify the OS instead. Scheduling read-ahead is
>> really the OS's job anyway.

Hi Sergiu,

If you would like to help make grep use larger buffer sizes, please
run and report benchmarks measuring how much of a difference it would
make, at least for your hardware. Here are some of the tests I ran to
justify raising it from ~32k to ~96k:
https://lists.gnu.org/archive/html/grep-devel/2018-10/msg00002.html

Message #55 received at 32073 <at> debbugs.gnu.org (full text, mbox):

From: Sergiu Hlihor <sh <at> discovergy.com>
To: Jim Meyering <jim <at> meyering.net>
Cc: 32073 <at> debbugs.gnu.org, Paul Eggert <eggert <at> cs.ucla.edu>,
 Dennis Clarke <dclarke <at> blastwave.org>
Subject: Re: Improvements in Grep (Bug#32073)
Date: Thu, 2 Jan 2020 02:03:58 +0100

[Message part 1 (text/plain, inline)]

Hi Jim,
The system for which this hurts me the most is an Ubuntu 14.04 where I'd
need to run it as a separate binary. As I'm not familiar with the way it's
built, is there any guidelines of how to build it from sources? I'd happy
build it with ever larger block sizes and test.

On Thu, 2 Jan 2020 at 01:51, Jim Meyering <jim <at> meyering.net> wrote:

> On Wed, Jan 1, 2020 at 12:04 PM Sergiu Hlihor <sh <at> discovergy.com> wrote:
> > Paul, I have to correct you. On a production server you have usually a
> mix of applications many times including databases. For databases, having a
> read ahead means one IO less since usually database access patterns are
> random reads. Here actually best is to disable completely read ahead. In
> fact, I do have to say that probably best is to disable completely read
> ahead and let applications deal with it, either in an automatic fashion,
> like reading the optimal IO block size from device  or in a configurable
> way with defaults good enough for today's servers. If you now configure the
> OS to do a read ahead hitting all HDDs then you induce potentially
> unnecessary IO load for all applications which use it, which when having
> HDDs is totally unacceptable. That's why the best is to be application
> specific and ideally configured to use optimal IO block size.
> >
> > So no, letting OS to do it is stupid.
> >
> > On Wed, 1 Jan 2020 at 20:42, Paul Eggert <eggert <at> cs.ucla.edu> wrote:
> >>
> >> On 1/1/20 1:15 AM, Sergiu Hlihor wrote:
> >> > If you rely on OS, then
> >> > you are at the mercy of whatever read ahead configuration you have.
> >>
> >> Right, and whatever changes you make to the OS and its read-ahead
> configuration
> >> will work for all applications, not just for 'grep'. So, change the OS
> to do
> >> that. There shouldn't be a need to change 'grep' in particular (or 'cp'
> in
> >> particular, or 'awk' in particular, etc.).
> >>
> >> > The issue of large
> >> > block sizes for IO operations is widespread across all tools from
> Linux,
> >> > like rsync or cp and its only getting worse
> >>
> >> Quite right. And it would be painful to have to modify all those tools,
> and to
> >> maintain those modifications. So modify the OS instead. Scheduling
> read-ahead is
> >> really the OS's job anyway.
>
> Hi Sergiu,
>
> If you would like to help make grep use larger buffer sizes, please
> run and report benchmarks measuring how much of a difference it would
> make, at least for your hardware. Here are some of the tests I ran to
> justify raising it from ~32k to ~96k:
> https://lists.gnu.org/archive/html/grep-devel/2018-10/msg00002.html
>

[Message part 2 (text/html, inline)]

Message #58 received at 32073 <at> debbugs.gnu.org (full text, mbox):

From: Jim Meyering <jim <at> meyering.net>
To: Sergiu Hlihor <sh <at> discovergy.com>
Cc: 32073 <at> debbugs.gnu.org, Paul Eggert <eggert <at> cs.ucla.edu>,
 Dennis Clarke <dclarke <at> blastwave.org>
Subject: Re: Improvements in Grep (Bug#32073)
Date: Wed, 1 Jan 2020 17:28:17 -0800

On Wed, Jan 1, 2020 at 5:04 PM Sergiu Hlihor <sh <at> discovergy.com> wrote:
> The system for which this hurts me the most is an Ubuntu 14.04 where I'd need to run it as a separate binary. As I'm not familiar with the way it's built, is there any guidelines of how to build it from sources? I'd happy build it with ever larger block sizes and test.

Something like the following should work: (if you want to be more
careful than most, also download the .sig file,
https://meyering.net/grep/grep-3.3.49-3f11.tar.xz.sig, and use that to
verify the .xz file is the same one I signed -- do that before running
./configure)

wget https://meyering.net/grep/grep-3.3.49-3f11.tar.xz
xz -dc grep-3.3.49-3f11.tar.xz|tar xf -
cd grep-3.3.49-3f11
./configure && make

Message #61 received at 32073 <at> debbugs.gnu.org (full text, mbox):

From: Sergiu Hlihor <sh <at> discovergy.com>
To: arnold <at> skeeve.com
Cc: 32073 <at> debbugs.gnu.org, Paul Eggert <eggert <at> cs.ucla.edu>
Subject: Re: bug#32073: Improvements in Grep (Bug#32073)
Date: Thu, 2 Jan 2020 05:20:32 +0100

[Message part 1 (text/plain, inline)]

Hi Arnold,
If AWKBUFSIZE translates to disk IO request size then it is already what
its needed. However it's a little annoying.

Regarding optimal settings, the benchmark actually tells you what is
optimal. Let's assume grep or any other tool can process in memory 3GB/s.
If your device can server 5GB/s then you can saturate the CPU.  If however
the device needs at least X as block size to reach the maximum throughput,
then that's what you have to use. Plain and simple. And as I said, when
going into GB territory, at application level reads have to be asynchronous.
if you look at benchmarking tools like Atto you see the graphs clearly and
see the scaling for SSDs. And just happens that the value good for SSDs
(minimum 512KB) also benefits HDD RAID arrays with strip sizes smaller than
512KB. With HDD RAID arrays unfortunately it does get complicated because
you have to know the number of disks and strip size. I for example always
use tune2fs and set those parameters when format the partition. This could
just as well be a configurable OS parameter per drive and based on the
location of the file, the right value could be used. But I have to admit
that this would add exponential complexity with diminishing returns versus
just setting a buffer size of 1MB (which will cover both current and future
SSDs).

Also I'm not too fond of heuristics or any other smartness at IO level in
Linux IO stack. I'm working with large databases (as user) and discussed
about Linux IO stack with database developers. The common opinion is that
Linux IO stack got out of control and nobody actually has a good overview
anymore. And I tend to agree. Linux needs an IO stack as lean as possible
and let the applications decide what to do, as at the application level you
know your usage pattern. I already had to finetune the database due to it.

On Wed, 1 Jan 2020 at 21:24, <arnold <at> skeeve.com> wrote:

> Hi.
>
> Sergiu Hlihor <sh <at> discovergy.com> wrote:
>
> > Arnold, there is no need to write user code, it is already done in
> > benchmarks. One of the standard benchmarks when testing HDDs and SSDs is
> > read throughput vs block size and at different queue depths.
>
> I think you're misunderstanding me, or I am misunderstanding you.
>
> As the gawk maintainer, I can choose the buffer size to use every time
> I issue a read(2) system call for any given input file.  Gawk currently
> uses the smaller of (a) the file's size or (b) the st_blksize member of
> the struct stat array.
>
> If I understand you correctly, this is "not enough"; gawk (grep,
> cp, etc.) should all use an optimal buffer size that depends upon the
> underlying storage hardware where the file is located.
>
> So far, so good, except for: How do I determine what that number is?
> I cannot run a benchmark before opening each and every file. I don't
> know of a system call that will give me that number. (If there is,
> please point me to it.)
>
> Do you just want a command line option or environment variable
> that you, as the application user, can set?
>
> If the latter, it happens that gawk will let you set AWKBUFSIZE and
> it will use whatever number you supply for doing reads. (This is
> even documented.)
>
> HTH,
>
> Arnold
>

[Message part 2 (text/html, inline)]

Message #64 received at 32073 <at> debbugs.gnu.org (full text, mbox):

From: arnold <at> skeeve.com
To: sh <at> discovergy.com, arnold <at> skeeve.com
Cc: 32073 <at> debbugs.gnu.org, eggert <at> cs.ucla.edu
Subject: Re: bug#32073: Improvements in Grep (Bug#32073)
Date: Thu, 02 Jan 2020 00:20:41 -0700

Hi.

Sergiu Hlihor <sh <at> discovergy.com> wrote:

> Hi Arnold,
> If AWKBUFSIZE translates to disk IO request size then it is already what
> its needed. However it's a little annoying.

How would you make it less annoying?

Thanks,

Arnold

Message #67 received at 32073 <at> debbugs.gnu.org (full text, mbox):

From: Sergiu Hlihor <sh <at> discovergy.com>
To: arnold <at> skeeve.com
Cc: 32073 <at> debbugs.gnu.org, Paul Eggert <eggert <at> cs.ucla.edu>
Subject: Re: bug#32073: Improvements in Grep (Bug#32073)
Date: Thu, 2 Jan 2020 16:31:57 +0100

[Message part 1 (text/plain, inline)]

Hi Arnold,
Annoying in the sense that you have to specify it with every usage. In a
company where you have 10+ developers grepping over various logs, each one
has to remember to add the extra parameter. Easier would be to have some
kind of global configuration that the system admin can set and developers
forget about it. But as I said, large default is very likely enough.



On Thu, 2 Jan 2020 at 08:20, <arnold <at> skeeve.com> wrote:

> Hi.
>
> Sergiu Hlihor <sh <at> discovergy.com> wrote:
>
> > Hi Arnold,
> > If AWKBUFSIZE translates to disk IO request size then it is already what
> > its needed. However it's a little annoying.
>
> How would you make it less annoying?
>
> Thanks,
>
> Arnold
>

[Message part 2 (text/html, inline)]

Message #70 received at 32073 <at> debbugs.gnu.org (full text, mbox):

From: arnold <at> skeeve.com
To: sh <at> discovergy.com, arnold <at> skeeve.com
Cc: 32073 <at> debbugs.gnu.org, eggert <at> cs.ucla.edu
Subject: Re: bug#32073: Improvements in Grep (Bug#32073)
Date: Thu, 02 Jan 2020 08:36:39 -0700

OK, thanks for the input.

Arnold

Sergiu Hlihor <sh <at> discovergy.com> wrote:

> Hi Arnold,
> Annoying in the sense that you have to specify it with every usage. In a
> company where you have 10+ developers grepping over various logs, each one
> has to remember to add the extra parameter. Easier would be to have some
> kind of global configuration that the system admin can set and developers
> forget about it. But as I said, large default is very likely enough.
>
>
>
> On Thu, 2 Jan 2020 at 08:20, <arnold <at> skeeve.com> wrote:
>
> > Hi.
> >
> > Sergiu Hlihor <sh <at> discovergy.com> wrote:
> >
> > > Hi Arnold,
> > > If AWKBUFSIZE translates to disk IO request size then it is already what
> > > its needed. However it's a little annoying.
> >
> > How would you make it less annoying?
> >
> > Thanks,
> >
> > Arnold
> >

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.