GNU bug report logs - #19005
new snapshot available: grep-2.20.83-06900

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 19005 in the body.
You can then email your comments to 19005 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Jim Meyering <jim <at> meyering.net>
To: bug-grep <at> gnu.org
Cc: coordinator <at> translationproject.org, platform-testers <at> gnu.org
Subject: new snapshot available: grep-2.20.83-06900
Date: Sun, 09 Nov 2014 13:00:01 -0800

Testing the first snapshot exposed several problems,
and most have been addressed. The sole remaining problem
is the failure of an "expensive" (not run by default: big-match)
that makes most systems run out of memory. I haven't looked at it yet.

Thanks to all who have helped so far, this snapshot should be
much closer to releasable.

Please let us know if you find any problems.

grep snapshot:
  http://meyering.net/grep/grep-ss.tar.xz      1.3 MB
  http://meyering.net/grep/grep-ss.tar.xz.sig
  http://meyering.net/grep/grep-2.20.83-06900.tar.xz

Changes in grep since 2.20.72-d512:

Jim Meyering (9):
      tests: avoid false-positive failure due to some zh_CN.* locales
      build: generate man pages even when existing targets are read-only
      maint: move helper function, hex_printf to init.cfg
      tests: avoid printf+\xHH portability trap
      tests: fix typo in previous change
      maint: remove spurious LC_ALL setting from one test
      tests: avoid a multibyte tr portability problem
      tests: avoid hex_printf_ portability problems
      tests: do not fail when the zh_CN.UTF-8 locale is not installed

Norihiro Tanaka (1):
      tests: avoid awk+printf+\xHH portability trap

Paul Eggert (1):
      grep: port O_NOFOLLOW errno checking to NetBSD


Changes in gnulib since 2.20.72-d512:

* gnulib d55899f...8415b67 (78):
  > socketlib, sockets, sys_socket: Use AC_REQUIRE to pacify autoconf.
  > iconv: avoid false detection of non-working iconv
  > bootstrap: print more diagnostics for missing programs
  > bootstrap: only update the gnulib submodule
  > symlinkat: port to AIX 7.1
  > readlinkat: port to AIX 7.1
  > remove spurious {
  > modules/fcntl: fix error reporting by dupfd
  > basename, dirname: Improve documentation.
  > exclude: declare exclude_patopts static
  > autoupdate
  > dirname: support compilation with C++
  > qsort_r: include <config.h>
  > avltree-list: avoid compiler warnings
  > qsort_r: new module, for GNU-style qsort_r
  > strerror_r-posix: support compilation with C++
  > fcntl-h: fix compilation with Intel C++ compiler
  > autoupdate
  > mountlist: use /proc/self/mountinfo when available
  > users.txt: add cmogstored
  > gnulib-tool: Sync with build-aux/bootstrap options
  > gnulib-tool: Fallback to wget when rsync fails
  > maintainer-makefile: add syntax check for useless ';;'
  > pthread, pthread_sigmask, threadlib: port to Ubuntu 14.04
  > error: drop spurious semicolon
  > gnulib-common.m4: port to GCC 4.2.1 and Sun Studio 12 C++
  > manywarnings: add GCC 4.9 warnings
  > vasnprintf: fix bugs in width computation
  > vasnprintf: Avoid signed/unsigned comparison warning.
  > parse-datetime: Avoid signed/unsigned comparison warning
  > qsort_r: new module, for GNU-style qsort_r
  > vla: new module
  > localename: make gl_locale_name_thread really thread-safe on Windows
  > getpass: don't assume struct termios
  > getdtablesize: fall back on sysconf (_SC_OPEN_MAX)
  > vararrays: modernize AC_C_VARARRAYS for C11
  > relocatable-prog-wrapper: port gettext to OS X 10.8 + GCC 4.8.1
  > sys_select: fix FD_ZERO problem on Solaris 10
  > accept: document Solaris 10 type glitch
  > extern-inline: port to FreeBSD, DragonFly
  > autoupdate
  > Use consistent style to check DEBUG macro in regex_internal.c
  > openat-die: use _Noreturn markup
  > test-open: port to cygwin, which lacks Fortify
  > localename: Enforce declarations before statements.
  > test-userspec: don't look up numeric user names
  > localcharset, localename: MS-Windows support for non-default locales
  > announce-gen: avoid failure when Digest::SHA is installed
  > gettext: revert "update macros to version 0.19"
  > regex: don't deref NULL upon heap allocation failure
  > maint.mk: give projects more flexibilty in set_prog_name arguments
  > regex: fix memory leak in compiler
  > announce-gen: avoid perl warnings
  > localename: avoid -Wsuggest-attribute={const,pure} warnings
  > nl_langinfo: Fix last change.
  > Define macros for glibc
  > Sync up error.c with glibc
  > nl_langinfo: fix build under mingw
  > mountlist: do not classify a bind-mounted dir entry as "dummy"
  > maint.mk: less syntax-check noise when SIGPIPE is ignored
  > nl_langinfo: CODESET on MS-Windows and more items from localeconv
  > Bruno Haible has stepped down as maintainer.
  > mktime: merge #if/#ifdef usage from glibc
  > git-version-gen: improve option descriptions
  > regex: fix memory leak in compiler
  > regex: merge patch from libc
  > acl: port to gcc -Wredundant-decls
  > parse-duration: eliminate 68-year duration limit
  > pthread: don't assume AC_CANONICAL_HOST, port better to Solaris, etc.
  > pthread: define thread-safe macros on some platforms
  > regex: don't be multithreaded if USE_UNLOCKED_IO.
  > gettext: update macros to version 0.19
  > select,poll: fix console handle check on windows 8
  > select: fix waiting on anonymous pipes on MS-Windows
  > times: fix to return non constant value on MS-Windows
  > isatty: fix to work on windows 8
  > maint: fix typo in fdl.texi
  > mountlist: avoid hasmntopt const type warning on solaris

Message #8 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Bruce Dubbs <bruce.dubbs <at> gmail.com>
To: bug-grep <at> gnu.org
Subject: Re: bug#19005: new snapshot available: grep-2.20.83-06900
Date: Sun, 09 Nov 2014 16:28:53 -0600

Jim Meyering wrote:
> Testing the first snapshot exposed several problems,
> and most have been addressed. The sole remaining problem
> is the failure of an "expensive" (not run by default: big-match)
> that makes most systems run out of memory. I haven't looked at it yet.

Shouldn't the  big-match test just be able to see if the system runs out 
of memory and if so, set an XFAIL.

Also, I suspect that it might work on lower memory systems if the user 
establishes a temporary swap file of sufficient size.  It would, of 
course, be slow.

  -- Bruce

Message #11 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Assaf Gordon <assafgordon <at> gmail.com>
To: bug-grep <at> gnu.org
Subject: Re: [platform-testers] new snapshot available: grep-2.20.83-06900
Date: Sun, 09 Nov 2014 17:30:59 -0500

[Message part 1 (text/plain, inline)]

Hello,

On 11/09/2014 04:00 PM, Jim Meyering wrote:
> Testing the first snapshot exposed several problems,
> and most have been addressed. The sole remaining problem
> is the failure of an "expensive" (not run by default: big-match)
> that makes most systems run out of memory. I haven't looked at it yet.
 
<...>

> Please let us know if you find any problems.

Two quick tests:

-1-
On an old ubuntu 10  64-bit machine with PCRE, test "pcre-invalid-utf8-input" fails.
Log attached.
It is possible due to the test using "en_US.UTF-8" but the available locales are:

    $ locale -a | grep en_US
    en_US
    en_US.utf8

However, the 'get-mb-cur-max' does seem to accept "UTF-8":

    $ ./tests/get-mb-cur-max en_US
    1
    $ ./tests/get-mb-cur-max en_US.utf8
    6
    $ ./tests/get-mb-cur-max en_US.UTF-8
    6
    $ ./tests/get-mb-cur-max en_US.FOO-8
    (EMPTY)

It is possible this old machine is mis-configured, and if so - please ignore this failure.

Using "libpcre3" version "7.8-3build1" (from ubuntu 10 package).



-2-
On the same machine, which has 128 GB ram total, and 106 GB ram free:

    $ free -g
                 total       used       free     shared    buffers     cached
    Mem:           125         38         87          0          6         13
    -/+ buffers/cache:         18        106
    Swap:            9          3          6

The "big-match" test is still skipped with "not enough main memory to run the test" - which sounds a bit weird.
Log attached.
During the execution of the tests, the "grep" process consumed up to 4.0GB of ram (in the "RES" column of "top").


I'll be able to run the other test (on more OSes) a bit later.

Regards,
 - Assaf

[grep.tests-suite.big-match.log (text/x-log, attachment)]

[grep.tests-suite.pcre_utf.log (text/x-log, attachment)]

Message #14 received at 19005 <at> debbugs.gnu.org (full text, mbox):

From: Jim Meyering <jim <at> meyering.net>
To: Assaf Gordon <assafgordon <at> gmail.com>
Cc: 19005 <at> debbugs.gnu.org
Subject: Re: bug#19005: [platform-testers] new snapshot available:
 grep-2.20.83-06900
Date: Sun, 9 Nov 2014 15:33:00 -0800

On Sun, Nov 9, 2014 at 2:30 PM, Assaf Gordon <assafgordon <at> gmail.com> wrote:
> Hello,
>
> On 11/09/2014 04:00 PM, Jim Meyering wrote:
>>
>> Testing the first snapshot exposed several problems,
>> and most have been addressed. The sole remaining problem
>> is the failure of an "expensive" (not run by default: big-match)
>> that makes most systems run out of memory. I haven't looked at it yet.
>
>  <...>
>
>> Please let us know if you find any problems.
>
>
> Two quick tests:
>
> -1-
> On an old ubuntu 10  64-bit machine with PCRE, test
> "pcre-invalid-utf8-input" fails.
> Log attached.
> It is possible due to the test using "en_US.UTF-8" but the available locales
> are:
>
>     $ locale -a | grep en_US
>     en_US
>     en_US.utf8
>
> However, the 'get-mb-cur-max' does seem to accept "UTF-8":
>
>     $ ./tests/get-mb-cur-max en_US
>     1
>     $ ./tests/get-mb-cur-max en_US.utf8
>     6
>     $ ./tests/get-mb-cur-max en_US.UTF-8
>     6
>     $ ./tests/get-mb-cur-max en_US.FOO-8
>     (EMPTY)
>
> It is possible this old machine is mis-configured, and if so - please ignore
> this failure.
>
> Using "libpcre3" version "7.8-3build1" (from ubuntu 10 package).

Hi Assaf,
Thanks for the prompt tests and feedback.

This part of your log implicates that old ubuntu system's libpcre:

+ LC_ALL=en_US.UTF-8 timeout 3 grep -P k$ in
+ test 124 -eq 1
+ fail=1
+ echo k

That suggests that grep -P was stuck in an infinite loop when timeout
terminated the process (the exit 124).  In this case, the error is
highly desirable: it is your warning that you cannot rely on grep -P
built with that library.

> -2-
> On the same machine, which has 128 GB ram total, and 106 GB ram free:
>
>     $ free -g
>                  total       used       free     shared    buffers
> cached
>     Mem:           125         38         87          0          6
> 13
>     -/+ buffers/cache:         18        106
>     Swap:            9          3          6
>
> The "big-match" test is still skipped with "not enough main memory to run
> the test" - which sounds a bit weird.
> Log attached.
> During the execution of the tests, the "grep" process consumed up to 4.0GB
> of ram (in the "RES" column of "top").

I interrupted one run of that test when it had reached 95GB RSS.
That tests's combination of regexp and input provoke massive allocation
in regexp-internal code, albeit no leak I have seen.

Message #17 received at 19005 <at> debbugs.gnu.org (full text, mbox):

From: Jim Meyering <jim <at> meyering.net>
To: Bruce Dubbs <bruce.dubbs <at> gmail.com>
Cc: 19005 <19005 <at> debbugs.gnu.org>
Subject: Re: bug#19005: new snapshot available: grep-2.20.83-06900
Date: Sun, 9 Nov 2014 17:10:39 -0800

On Sun, Nov 9, 2014 at 2:28 PM, Bruce Dubbs <bruce.dubbs <at> gmail.com> wrote:
> Jim Meyering wrote:
>>
>> Testing the first snapshot exposed several problems,
>> and most have been addressed. The sole remaining problem
>> is the failure of an "expensive" (not run by default: big-match)
>> that makes most systems run out of memory. I haven't looked at it yet.
>
> Shouldn't the  big-match test just be able to see if the system runs out of
> memory and if so, set an XFAIL.

I don't think so.
This test deliberately maps OOM and similar to "FAIL".
Maybe some change has induced a regression.

Message #20 received at 19005 <at> debbugs.gnu.org (full text, mbox):

From: Assaf Gordon <assafgordon <at> gmail.com>
To: 19005 <at> debbugs.gnu.org
Subject: Re: bug#19005: [platform-testers] new snapshot available:
 grep-2.20.83-06900
Date: Sun, 09 Nov 2014 21:47:31 -0500

[Message part 1 (text/plain, inline)]

On 11/09/2014 05:30 PM, Assaf Gordon wrote:
>
> On 11/09/2014 04:00 PM, Jim Meyering wrote:
>> Testing the first snapshot exposed several problems,
>> and most have been addressed. The sole remaining problem
>> is the failure of an "expensive" (not run by default: big-match)
>> that makes most systems run out of memory. I haven't looked at it yet.
>
> <...>
>
>> Please let us know if you find any problems.
>

No failures on the following VMs (amd64, tested without PCRE, and only non-expensive tests):
  freebsd 9.3
  freebsd 10
  openbsd 5.5
  netbsd 6.1.4

  debian 7.6
  gnewsense 3.1 (=debian 6)
  trisquel 6.0.1 (=ubuntu 12 LTS)
  ubuntu 14.04.1

  opensuse 13.1
  fedora 20
  centos 7

  GNU Hurd/Debian 0.5 (32-bit).



Two cases on less common OSes:

On DilOS 1.3.7, based on Illumos/OpenSolaris, "multibyte-white-space" fails.
log attached.
From a brief look, perhaps it has something to do with the recent "hex printf" change? not sure.
This is a weird system, as they try to use Illumos kernel with Debian user space,
so things could be mis-configured.
Few quick checks (relating to the "hex printf" implementation):
===
$ bash --version | head -n1
GNU bash, version 4.2.45(2)-release (x86_64-pc-solaris2.11)

$ printf --version

-bash: printf: --: invalid option
printf: usage: printf [-v var] format [arguments]

$ which printf
/usr/gnu/bin/printf

$ /usr/gnu/bin/printf --version | head -n1
printf (GNU coreutils) 8.21

$ which sed
/usr/gnu/bin/sed

$ $(which sed) --version | head -n1
/usr/gnu/bin/sed (GNU sed) 4.2.2
===


On MINIX R3.3.0, "symlinks" fails, which seems similar to the previous NetBSD6.1.4 issue.
Log attached.
I was not able to install Ktrace/Kdump yet.


I will sadly not be able to test more until next weekend.

Regards,
 - Assaf

[diols137.grep.test-suite.log (text/x-log, attachment)]

[minixR330.grep.test-suite.log (text/x-log, attachment)]

Message #23 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Bruce Dubbs <bruce.dubbs <at> gmail.com>
To: bug-grep <at> gnu.org
Subject: Re: bug#19005: new snapshot available: grep-2.20.83-06900
Date: Sun, 09 Nov 2014 21:22:18 -0600

Jim Meyering wrote:
> Testing the first snapshot exposed several problems,
> and most have been addressed. The sole remaining problem
> is the failure of an "expensive" (not run by default: big-match)
> that makes most systems run out of memory. I haven't looked at it yet.
>
> Thanks to all who have helped so far, this snapshot should be
> much closer to releasable.
>
> Please let us know if you find any problems.
>
> grep snapshot:
>    http://meyering.net/grep/grep-2.20.83-06900.tar.xz

For LFS, a basic system the tests give:

Testsuite summary for GNU grep 2.20.83-06900
============================================================
# TOTAL: 85
# PASS:  71
# SKIP:  13
# XFAIL: 1
# FAIL:  0
# XPASS: 0
# ERROR: 0

big-hole: skipped test: expensive: disabled by default
big-match: skipped test: expensive: disabled by default
long-line-vs-2GiB-read: skipped test: expensive: disabled by default
pcre: skipped test: no PCRE support
pcre-abort: skipped test: no PCRE support
pcre-infloop: skipped test: no PCRE support
pcre-invalid-utf8-input: skipped test: no PCRE support
pcre-o: skipped test: no PCRE support
pcre-utf8: skipped test: no PCRE support
pcre-w: skipped test: no PCRE support
pcre-wx-backref: skipped test: no PCRE support
pcre-z: skipped test: no PCRE support
sjis-mb: skipped test: SJIS locale not found
XFAIL: triple-backref

Testsuite summary for GNU grep 2.20.83-06900
============================================================================
# TOTAL: 137
# PASS:  130
# SKIP:  7
# XFAIL: 0
# FAIL:  0
# XPASS: 0
# ERROR: 0

SKIP: test-vc-list-files-git.sh
SKIP: test-vc-list-files-cvs.sh
SKIP: test-wcrtomb-w32-1.sh
SKIP: test-wcrtomb-w32-2.sh
SKIP: test-wcrtomb-w32-3.sh
SKIP: test-wcrtomb-w32-4.sh
SKIP: test-wcrtomb-w32-5.sh

-----------------

On a more compete LFS system the pcre and git tests also:

SKIP: big-hole
SKIP: big-match
SKIP: long-line-vs-2GiB-read
SKIP: sjis-mb
XFAIL: triple-backref

SKIP: test-vc-list-files-cvs.sh
SKIP: test-wcrtomb-w32-1.sh
SKIP: test-wcrtomb-w32-2.sh
SKIP: test-wcrtomb-w32-3.sh
SKIP: test-wcrtomb-w32-4.sh
SKIP: test-wcrtomb-w32-5.sh

  -- Bruce Dubbs
     linuxfromscratch.org

Message #26 received at 19005 <at> debbugs.gnu.org (full text, mbox):

From: Jim Meyering <jim <at> meyering.net>
To: Bruce Dubbs <bruce.dubbs <at> gmail.com>
Cc: 19005 <19005 <at> debbugs.gnu.org>
Subject: Re: bug#19005: new snapshot available: grep-2.20.83-06900
Date: Sun, 9 Nov 2014 22:04:25 -0800

On Sun, Nov 9, 2014 at 7:22 PM, Bruce Dubbs <bruce.dubbs <at> gmail.com> wrote:
...
>> grep snapshot:
>>    http://meyering.net/grep/grep-2.20.83-06900.tar.xz
>
>
[no failure on any of 3 LFS systems]

Great.  Thanks for the testing and report.

Message #29 received at submit <at> debbugs.gnu.org (full text, mbox):

From: C de-Avillez <hggdh2 <at> ubuntu.com>
To: bug-grep <at> gnu.org
Subject: Re: [platform-testers] new snapshot available: grep-2.20.83-06900
Date: Tue, 11 Nov 2014 11:22:25 -0600

[Message part 1 (text/plain, inline)]

On 09/11/14 15:00, Jim Meyering wrote:
> Testing the first snapshot exposed several problems,
> and most have been addressed. The sole remaining problem
> is the failure of an "expensive" (not run by default: big-match)
> that makes most systems run out of memory. I haven't looked at it yet.

No failures (except for bug-match) on Ubuntu Vivid x86_64.
Check-expensive was run.

Cheers,

..C..

[signature.asc (application/pgp-signature, attachment)]

Message #32 received at 19005 <at> debbugs.gnu.org (full text, mbox):

From: Jim Meyering <jim <at> meyering.net>
To: C de-Avillez <hggdh2 <at> ubuntu.com>
Cc: 19005 <19005 <at> debbugs.gnu.org>
Subject: Re: bug#19005: [platform-testers] new snapshot available:
 grep-2.20.83-06900
Date: Tue, 11 Nov 2014 15:28:19 -0800

On Tue, Nov 11, 2014 at 9:22 AM, C de-Avillez <hggdh2 <at> ubuntu.com> wrote:
> On 09/11/14 15:00, Jim Meyering wrote:
>> Testing the first snapshot exposed several problems,
>> and most have been addressed. The sole remaining problem
>> is the failure of an "expensive" (not run by default: big-match)
>> that makes most systems run out of memory. I haven't looked at it yet.
>
> No failures (except for bug-match) on Ubuntu Vivid x86_64.
> Check-expensive was run.

Glad to hear it.  Thanks for reporting in.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.