GNU bug report logs - #21827
Large file support

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 21827 in the body.
You can then email your comments to 21827 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Michael Brunnbauer <brunni <at> netestate.de>
To: bug-grep <at> gnu.org
Subject: Large file support
Date: Wed, 4 Nov 2015 14:07:44 +0100

[Message part 1 (text/plain, inline)]

hi all,

I am quite surprised that my grep (suddenly?) has problems with large files:

 fiano:~$ ls -l todelete.txt 
 -rw-r--r-- 1 brunni users 2981338204 Nov  4 13:40 todelete.txt
 fiano:~$ grep test todelete.txt
 grep: todelete.txt: Value too large for defined data type

I work with files > 2GB regularly and used to have these problems years ago.
Upgrading from grep-2.14 to grep-2.22 did not help. The configure output
containing "large" is:

 checking for special C compiler options needed for large files... no
 checking for _FILE_OFFSET_BITS value needed for large files... 64

I tried compiling without PCRE but this did not help either. My basic setup
is glibc-2.22, gcc 4.7.4 and Linux 3.4.109.

Any ideas what could be going wrong?

Regards,

Michael Brunnbauer

-- 
++  Michael Brunnbauer
++  netEstate GmbH
++  Geisenhausener Straße 11a
++  81379 München
++  Tel +49 89 32 19 77 80
++  Fax +49 89 32 19 77 89 
++  E-Mail brunni <at> netestate.de
++  http://www.netestate.de/
++
++  Sitz: München, HRB Nr.142452 (Handelsregister B München)
++  USt-IdNr. DE221033342
++  Geschäftsführer: Michael Brunnbauer, Franz Brunnbauer
++  Prokurist: Dipl. Kfm. (Univ.) Markus Hendel

[signature.asc (application/pgp-signature, inline)]

Message #8 received at 21827 <at> debbugs.gnu.org (full text, mbox):

From: Mike Frysinger <vapier <at> gentoo.org>
To: Michael Brunnbauer <brunni <at> netestate.de>
Cc: 21827 <at> debbugs.gnu.org
Subject: Re: bug#21827: Large file support
Date: Wed, 4 Nov 2015 19:25:28 -0500

[Message part 1 (text/plain, inline)]

On 04 Nov 2015 14:07, Michael Brunnbauer wrote:
> I am quite surprised that my grep (suddenly?) has problems with large files:
> 
>  fiano:~$ ls -l todelete.txt 
>  -rw-r--r-- 1 brunni users 2981338204 Nov  4 13:40 todelete.txt
>  fiano:~$ grep test todelete.txt
>  grep: todelete.txt: Value too large for defined data type
> 
> I work with files > 2GB regularly and used to have these problems years ago.
> Upgrading from grep-2.14 to grep-2.22 did not help. The configure output
> containing "large" is:
> 
>  checking for special C compiler options needed for large files... no
>  checking for _FILE_OFFSET_BITS value needed for large files... 64
> 
> I tried compiling without PCRE but this did not help either. My basic setup
> is glibc-2.22, gcc 4.7.4 and Linux 3.4.109.

i checked over here and it looks fine to me.  can you attach your config.log ?
also, try running it through strace and see what it looks like.  something like:
$ strace -e '!read' grep ...

you should see calls like:
openat(AT_FDCWD, "...", O_RDONLY|O_NOCTTY|O_LARGEFILE) = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=8539650048, ...}) = 0
-mike

[signature.asc (application/pgp-signature, inline)]

Message #11 received at 21827 <at> debbugs.gnu.org (full text, mbox):

From: Michael Brunnbauer <brunni <at> netestate.de>
To: 21827 <at> debbugs.gnu.org
Subject: Re: bug#21827: Large file support
Date: Thu, 5 Nov 2015 09:40:26 +0100

[Message part 1 (text/plain, inline)]

Hello Mike,

On Wed, Nov 04, 2015 at 07:25:28PM -0500, Mike Frysinger wrote:
> i checked over here and it looks fine to me.  can you attach your config.log ?

Attached.

> also, try running it through strace and see what it looks like.  something like:
> $ strace -e '!read' grep ...
> 
> you should see calls like:
> openat(AT_FDCWD, "...", O_RDONLY|O_NOCTTY|O_LARGEFILE) = 3

The O_LARGEFILE is missing on the grepped file but present for other calls
that seem internal to glibc:

[...]
open("/usr/lib/locale/locale-archive", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=9805360, ...}) = 0
mmap2(NULL, 2097152, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb7338000
close(3)                                = 0
brk(0)                                  = 0x806d000
brk(0x808e000)                          = 0x808e000
fstat64(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(3, 1), ...}) = 0
open("/usr/lib/gconv/gconv-modules.cache", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib/gconv/gconv-modules", O_RDONLY|O_CLOEXEC) = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=56095, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb771a000
close(3)                                = 0
munmap(0xb771a000, 4096)                = 0
open("/usr/lib/gconv/ISO8859-1.so", O_RDONLY|O_CLOEXEC) = 3
fstat64(3, {st_mode=S_IFREG|0755, st_size=23476, ...}) = 0
mmap2(NULL, 12328, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xb7717000
mmap2(0xb7719000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1) = 0xb7719000
close(3)                                = 0
mprotect(0xb7719000, 4096, PROT_READ)   = 0
ioctl(1, SNDCTL_TMR_TIMEBASE or TCGETS, {B38400 opost isig icanon echo ...}) = 0
openat(AT_FDCWD, "todelete.txt", O_RDONLY|O_NOCTTY) = -1 EOVERFLOW (Value too large for defined data type)
write(2, "./grep: "..., 8./grep: )              = 8
write(2, "todelete.txt"..., 12todelete.txt)         = 12
write(2, ": Value too large for defined dat"..., 39: Value too large for defined data type) = 39
write(2, "\n"..., 1
)                    = 1
close(1)                                = 0
close(2)                                = 0
exit_group(2)                           = ?

Regards,

Michael Brunnbauer

-- 
++  Michael Brunnbauer
++  netEstate GmbH
++  Geisenhausener Straße 11a
++  81379 München
++  Tel +49 89 32 19 77 80
++  Fax +49 89 32 19 77 89 
++  E-Mail brunni <at> netestate.de
++  http://www.netestate.de/
++
++  Sitz: München, HRB Nr.142452 (Handelsregister B München)
++  USt-IdNr. DE221033342
++  Geschäftsführer: Michael Brunnbauer, Franz Brunnbauer
++  Prokurist: Dipl. Kfm. (Univ.) Markus Hendel

[config.log (text/plain, attachment)]

[signature.asc (application/pgp-signature, inline)]

Message #14 received at 21827 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Michael Brunnbauer <brunni <at> netestate.de>, 21827 <at> debbugs.gnu.org
Subject: Re: bug#21827: Large file support
Date: Thu, 5 Nov 2015 15:05:26 -0800

On 11/04/2015 05:07 AM, Michael Brunnbauer wrote:
> Any ideas what could be going wrong?

It appears you've run afoul of glibc bug#18781.

https://sourceware.org/bugzilla/show_bug.cgi?id=18781

Yeowch.  If I'm right, that's a serious glibc bug that will break a lot 
of programs, not just grep.  I don't offhand see a way to work around it 
easily on Gnulib.

The bug was introduced on 2015-06-04 and fixed on 2015-08-10 in glibc.

Message #17 received at 21827 <at> debbugs.gnu.org (full text, mbox):

From: Eric Blake <eblake <at> redhat.com>
To: Paul Eggert <eggert <at> cs.ucla.edu>, Michael Brunnbauer
 <brunni <at> netestate.de>, 21827 <at> debbugs.gnu.org
Subject: Re: bug#21827: Large file support
Date: Thu, 5 Nov 2015 17:14:53 -0700

[Message part 1 (text/plain, inline)]

On 11/05/2015 04:05 PM, Paul Eggert wrote:
> On 11/04/2015 05:07 AM, Michael Brunnbauer wrote:
>> Any ideas what could be going wrong?
> 
> It appears you've run afoul of glibc bug#18781.
> 
> https://sourceware.org/bugzilla/show_bug.cgi?id=18781
> 
> Yeowch.  If I'm right, that's a serious glibc bug that will break a lot
> of programs, not just grep.  I don't offhand see a way to work around it
> easily on Gnulib.

Can't we probe for the existence of openat64(), and then probe whether
the bug exists, and if so, replace openat() for that platform?

> 
> The bug was introduced on 2015-06-04 and fixed on 2015-08-10 in glibc.

I guess it depends on whether we think these flawed versions are common
in the wild, or whether vendors have patched the major distros and only
self-built glibc remains vulnerable, on whether it is worth our effort.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org

[signature.asc (application/pgp-signature, attachment)]

Message #20 received at 21827 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Eric Blake <eblake <at> redhat.com>, Michael Brunnbauer <brunni <at> netestate.de>, 
 21827 <at> debbugs.gnu.org
Subject: Re: bug#21827: Large file support
Date: Thu, 5 Nov 2015 18:47:18 -0800

Eric Blake wrote:
> Can't we probe for the existence of openat64(), and then probe whether
> the bug exists, and if so, replace openat() for that platform?

If one built with glibc 2.21 the probe would report no problem, with the failure 
occurring only when one ran the prebuilt executable, dynamically linked with 
glibc 2.22.  Possibly we could work around the bug at run-time, by falling back 
on syscall when openat fails with errno == EOVERFLOW, when compiled with glibc 
2.22 or earlier.  It's not the sort of code I'd like to write or test or 
maintain, though -- syscall is so machine-dependent.

>> >The bug was introduced on 2015-06-04 and fixed on 2015-08-10 in glibc.
> I guess it depends on whether we think these flawed versions are common
> in the wild, or whether vendors have patched the major distros and only
> self-built glibc remains vulnerable, on whether it is worth our effort.

Yes, that's exactly it. In the past we've sometimes worked around these bugs and 
sometimes not; it's a judgment call.  My guess is that the bug is so severe and 
affects so many apps that it'll get fixed pretty quickly in the wild.

Message #23 received at 21827 <at> debbugs.gnu.org (full text, mbox):

From: Mike Frysinger <vapier <at> gentoo.org>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: Michael Brunnbauer <brunni <at> netestate.de>, 21827 <at> debbugs.gnu.org
Subject: Re: bug#21827: Large file support
Date: Thu, 5 Nov 2015 22:37:43 -0500

[Message part 1 (text/plain, inline)]

On 05 Nov 2015 15:05, Paul Eggert wrote:
> On 11/04/2015 05:07 AM, Michael Brunnbauer wrote:
> > Any ideas what could be going wrong?
> 
> It appears you've run afoul of glibc bug#18781.
> 
> https://sourceware.org/bugzilla/show_bug.cgi?id=18781
> 
> Yeowch.  If I'm right, that's a serious glibc bug that will break a lot 
> of programs, not just grep.  I don't offhand see a way to work around it 
> easily on Gnulib.
> 
> The bug was introduced on 2015-06-04 and fixed on 2015-08-10 in glibc.

we should be able to ping most of the distros about it though and make sure
it's included ?  i cherry picked it into Gentoo a while back while merging
updates from the 2.22 branch.

Michael: i don't think you said ... which distro are you using ?
-mike

[signature.asc (application/pgp-signature, inline)]

Message #26 received at 21827 <at> debbugs.gnu.org (full text, mbox):

From: Michael Brunnbauer <brunni <at> netestate.de>
To: Paul Eggert <eggert <at> cs.ucla.edu>, 21827 <at> debbugs.gnu.org
Subject: Re: bug#21827: Large file support
Date: Fri, 6 Nov 2015 07:35:59 +0100

[Message part 1 (text/plain, inline)]

Hello Mike,

On Thu, Nov 05, 2015 at 10:37:43PM -0500, Mike Frysinger wrote:
> > It appears you've run afoul of glibc bug#18781.
> > 
> > https://sourceware.org/bugzilla/show_bug.cgi?id=18781
> > 
> > Yeowch.  If I'm right, that's a serious glibc bug that will break a lot 
> > of programs, not just grep.

I upgraded to glibc-2.22 on Oct 6 and had no problems with other tools and
big files so far (at least tar, gzip and cp do not seem to be affected).

> Michael: i don't think you said ... which distro are you using ?

My own. It runs on 60 servers. Unless glibc-2.23 is around the corner I will 
have to patch my current glibc.

Regards,

Michael Brunnbauer

-- 
++  Michael Brunnbauer
++  netEstate GmbH
++  Geisenhausener Straße 11a
++  81379 München
++  Tel +49 89 32 19 77 80
++  Fax +49 89 32 19 77 89 
++  E-Mail brunni <at> netestate.de
++  http://www.netestate.de/
++
++  Sitz: München, HRB Nr.142452 (Handelsregister B München)
++  USt-IdNr. DE221033342
++  Geschäftsführer: Michael Brunnbauer, Franz Brunnbauer
++  Prokurist: Dipl. Kfm. (Univ.) Markus Hendel

[signature.asc (application/pgp-signature, inline)]

Message #29 received at 21827 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Michael Brunnbauer <brunni <at> netestate.de>, 21827 <at> debbugs.gnu.org
Subject: Re: bug#21827: Large file support
Date: Thu, 5 Nov 2015 23:14:16 -0800

Michael Brunnbauer wrote:
> I upgraded to glibc-2.22 on Oct 6 and had no problems with other tools and
> big files so far (at least tar, gzip and cp do not seem to be affected).

That's odd, since I know recent versions of tar use openat and so should have 
problems with files greater than 2 GiB.  cp also uses openat for directories, 
though most likely none of your directories are that large.  Perhaps you can 
investigate why tar works for you despite the glibc bug, if you're at all 
curious about this sort of thing.  Maybe your tar is compiled with -m64?  That 
would explain things.

At any rate, though, the problem is clearly in glibc, not in grep.

Message #32 received at 21827 <at> debbugs.gnu.org (full text, mbox):

From: Michael Brunnbauer <brunni <at> netestate.de>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: 21827 <at> debbugs.gnu.org
Subject: Re: bug#21827: Large file support
Date: Fri, 6 Nov 2015 08:34:36 +0100

[Message part 1 (text/plain, inline)]

Hello Paul,

On Thu, Nov 05, 2015 at 11:14:16PM -0800, Paul Eggert wrote:
> Michael Brunnbauer wrote:
> >I upgraded to glibc-2.22 on Oct 6 and had no problems with other tools and
> >big files so far (at least tar, gzip and cp do not seem to be affected).
> 
> That's odd, since I know recent versions of tar use openat and so should
> have problems with files greater than 2 GiB.  cp also uses openat for
> directories, though most likely none of your directories are that large.
> Perhaps you can investigate why tar works for you despite the glibc bug, if
> you're at all curious about this sort of thing.  Maybe your tar is compiled
> with -m64?  That would explain things.

My tar is quite old:

fiano:~$ tar --version
tar (GNU tar) 1.23

> At any rate, though, the problem is clearly in glibc, not in grep.

After installing this patch

 https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=561a9f11a974a447acb3dd03550a05df701a900e

only libc-2.22.so and libc.a were changed. Replacing them at runtime was no
problem :-)

grep works now. Thanks for the help!

Regards,

Michael Brunnbauer

-- 
++  Michael Brunnbauer
++  netEstate GmbH
++  Geisenhausener Straße 11a
++  81379 München
++  Tel +49 89 32 19 77 80
++  Fax +49 89 32 19 77 89 
++  E-Mail brunni <at> netestate.de
++  http://www.netestate.de/
++
++  Sitz: München, HRB Nr.142452 (Handelsregister B München)
++  USt-IdNr. DE221033342
++  Geschäftsführer: Michael Brunnbauer, Franz Brunnbauer
++  Prokurist: Dipl. Kfm. (Univ.) Markus Hendel

[signature.asc (application/pgp-signature, inline)]

Message #35 received at 21827 <at> debbugs.gnu.org (full text, mbox):

From: Mike Frysinger <vapier <at> gentoo.org>
To: Michael Brunnbauer <brunni <at> netestate.de>
Cc: Paul Eggert <eggert <at> cs.ucla.edu>, 21827 <at> debbugs.gnu.org
Subject: Re: bug#21827: Large file support
Date: Fri, 6 Nov 2015 10:48:54 -0500

[Message part 1 (text/plain, inline)]

On 06 Nov 2015 07:35, Michael Brunnbauer wrote:
> On Thu, Nov 05, 2015 at 10:37:43PM -0500, Mike Frysinger wrote:
> > > It appears you've run afoul of glibc bug#18781.
> > > 
> > > https://sourceware.org/bugzilla/show_bug.cgi?id=18781
> > > 
> > > Yeowch.  If I'm right, that's a serious glibc bug that will break a lot 
> > > of programs, not just grep.
> 
> I upgraded to glibc-2.22 on Oct 6 and had no problems with other tools and
> big files so far (at least tar, gzip and cp do not seem to be affected).
> 
> > Michael: i don't think you said ... which distro are you using ?
> 
> My own. It runs on 60 servers. Unless glibc-2.23 is around the corner I will 
> have to patch my current glibc.

you might consider using the branch then from upstream.  we cherry pick fixes
(including security related) into it.  thus far, we haven't tagged/released
point releases.
https://sourceware.org/git/?p=glibc.git;a=shortlog;h=refs/heads/release/2.22/master
-mike

[signature.asc (application/pgp-signature, inline)]

Message #40 received at 21827-done <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: 21827-done <at> debbugs.gnu.org
Subject: Re: Large file support
Date: Wed, 30 Dec 2015 23:45:22 -0800

Since this problem was not in 'grep' I am closing the bug report.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.