Subject: An assertion is hit when processing "(|()()0)\2" extended regular expression

From: Anatoly Trosinenko <[email-masked]>
Date: Sat, 22 Sep 2018 18:00:56 +0300

Hello,

When processing a simple extended regular expression, GNU grep hits an
assertion. This is reproducible on the latest commit from master
branch 51dacfb (Sep 19).

$ export LC_ALL=C
$ gdb -q --args ./grep -E "(|()()0)\2"
Reading symbols from ./grep...(no debugging symbols found)...done.
(gdb) r
Starting program: /path/to/grep -E \(\|\(\)\(\)0\)\\2
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
grep: regexec.c:1342: pop_fail_stack: Assertion `num >= 0' failed.

Program received signal SIGABRT, Aborted.
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
51      ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#1  0x00007ffff75ae801 in __GI_abort () at abort.c:79
#2  0x00007ffff759e39a in __assert_fail_base (fmt=0x7ffff77257d8
"%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
assertion=assertion@entry=0x7ffff772351f "num >= 0",
file=file@entry=0x7ffff7723515 "regexec.c", line=line@entry=1342,
    function=function@entry=0x7ffff77278a0 <__PRETTY_FUNCTION__.12590>
"pop_fail_stack") at assert.c:92
#3  0x00007ffff759e412 in __GI___assert_fail
(assertion=assertion@entry=0x7ffff772351f "num >= 0",
file=file@entry=0x7ffff7723515 "regexec.c", line=line@entry=1342,
function=function@entry=0x7ffff77278a0 <__PRETTY_FUNCTION__.12590>
"pop_fail_stack") at assert.c:101
#4  0x00007ffff758f3f8 in pop_fail_stack
(eps_via_nodes=0x7fffffffd5d0, regs=0x55555578b110, nregs=4,
pidx=<optimized out>, fs=<optimized out>) at regexec.c:1342
#5  0x00007ffff76650e8 in pop_fail_stack (eps_via_nodes=<optimized
out>, regs=<optimized out>, nregs=<optimized out>, pidx=<optimized
out>, fs=<optimized out>) at regexec.c:1427
#6  set_regs (preg=preg@entry=0x555555788cd0,
mctx=mctx@entry=0x7fffffffd770, nmatch=nmatch@entry=4,
pmatch=pmatch@entry=0x55555578b110, fl_backtrack=<optimized out>) at
regexec.c:1419
#7  0x00007ffff766e272 in re_search_internal
(preg=preg@entry=0x555555788cd0, string=string@entry=0x7fffffffdbe6
"\n", length=length@entry=0, start=<optimized out>, start@entry=0,
range=<optimized out>, range@entry=0, stop=stop@entry=0, nmatch=4,
pmatch=0x55555578b110,
    eflags=0) at regexec.c:846
#8  0x00007ffff766eb88 in re_search_stub (bufp=0x555555788cd0,
string=<optimized out>, length=<optimized out>, start=0,
range=<optimized out>, stop=<optimized out>, regs=0x5555557883a0,
ret_len=<optimized out>) at regexec.c:420
#9  0x00007ffff766f7e0 in __re_search (bufp=<optimized out>,
string=<optimized out>, length=<optimized out>, start=<optimized out>,
range=<optimized out>, regs=<optimized out>) at regexec.c:291
#10 0x0000555555558857 in EGexecute ()
#11 0x000055555555df08 in main ()

Best regards
Anatoly