GNU bug report logs - #17512
Valgrind and regression tests?

Previous Next

To reply to this bug, email your comments to 17512 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: behoffski <behoffski <at> grouse.com.au>
To: bug-grep <at> gnu.org
Subject: Valgrind and regression tests?
Date: Sat, 17 May 2014 05:44:13 +0930

G'day,

I would like to run the entire regression test for Grep (make check),
with Valgrind monitoring each invocation of grep etc, but with the
output going to a separate logfile.  (I have run valgrind for a few
cases, but have not covered the entire spectrum of the regression
test rig.)

The file grep/README-hacking recommends using valgrind, and also
suggests that hackers look at the file "README-valgrind", but no such
file exists.

I note that Valgrind sends its output to stderr by default, and that
the option --log-file=<filename> exists to redirect Valgrind output
elsewhere.  This would be needed in the case of grep et al, as stderr
is used in various ways by the programs.

Is there a simple way of incorporating valgrind into the entire
regression test run?  Two simple ideas come to mind, but I'm not
well versed in the POSIX/shell/makefile/Autoconf/regression rig setup
to know how realistic these are:

  1. Use a shell alias:

       grep="valgrind --log-file=foo grep" make check

  2. Replace grep/egrep/fgrep as appropriate to execute the program
     within a shell, where valgrind plus any options is incorporated
     into the program invocation and the logfile is treated
     appropriately in the program invocation aftermath.

Any hints, help, suggestions etc would be welcome.

cheers,

behoffski (Brenton Hoff)
Programmer, Grouse Software

Message #8 received at 17512 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: behoffski <behoffski <at> grouse.com.au>, 17512 <at> debbugs.gnu.org
Subject: Re: bug#17512: Valgrind and regression tests?
Date: Fri, 16 May 2014 17:35:46 -0700

behoffski wrote:
>    1. Use a shell alias:
>
>         grep="valgrind --log-file=foo grep" make check

That won't work, as too many parts of the system ignore the aliases.

>    2. Replace grep/egrep/fgrep as appropriate to execute the program
>       within a shell, where valgrind plus any options is incorporated
>       into the program invocation and the logfile is treated
>       appropriately in the program invocation aftermath.

Yes, something like that should work.  You can set up your own 
implementation of grep, like this:

#! /bin/sh
exec valgrind --log-file=foo /full/path/to/actual/grep "$@"

in a shell script called 'grep', in a different directory, and prepand 
that directory to your PATH.

Perhaps Jim has a better way to do it, though -- he's more of a valgrind 
wizard than I.

Message #15 received at 17512 <at> debbugs.gnu.org (full text, mbox):

From: Jim Meyering <jim <at> meyering.net>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: 17512 <at> debbugs.gnu.org, behoffski <behoffski <at> grouse.com.au>
Subject: Re: bug#17512: Valgrind and regression tests?
Date: Tue, 20 May 2014 22:10:03 -0700

On Fri, May 16, 2014 at 5:35 PM, Paul Eggert <eggert <at> cs.ucla.edu> wrote:
> behoffski wrote:
>>
>>    1. Use a shell alias:
>>
>>         grep="valgrind --log-file=foo grep" make check
>
>
> That won't work, as too many parts of the system ignore the aliases.
>
>
>>    2. Replace grep/egrep/fgrep as appropriate to execute the program
>>       within a shell, where valgrind plus any options is incorporated
>>       into the program invocation and the logfile is treated
>>       appropriately in the program invocation aftermath.
>
>
> Yes, something like that should work.  You can set up your own
> implementation of grep, like this:
>
> #! /bin/sh
> exec valgrind --log-file=foo /full/path/to/actual/grep "$@"
>
> in a shell script called 'grep', in a different directory, and prepand that
> directory to your PATH.
>
> Perhaps Jim has a better way to do it, though -- he's more of a valgrind
> wizard than I.

I use a script to replace each binary with this generated script:

#!/bin/sh
saved_binaries_dir="${0%/*}/.vg-tmp"
export PATH=$saved_binaries_dir:$PATH
prog=${0##*/}
# Disable this: http://udrepper.livejournal.com/11429.html
MALLOC_PERTURB_=0
export MALLOC_PERTURB_
exec /usr/bin/valgrind -v --suppressions=/dev/null \
  --log-file=/tmp/vg-log-2014-05-20.22:05:28-%q{GREP_TEST_NAME} \
  --leak-check=full --track-fds=yes --num-callers=15 -- "$prog" "$@"

which refers to the original binary via a subdirectory named .vg-tmp
relative to the original location of the binary. That script also moves
each binary into the .vg-tmp subdir.

Then, running "make check" creates many /tmp/vg-log-* files,
each related to the name of a test.

Message #18 received at 17512 <at> debbugs.gnu.org (full text, mbox):

From: behoffski <behoffski <at> grouse.com.au>
To: Jim Meyering <jim <at> meyering.net>, Paul Eggert <eggert <at> cs.ucla.edu>
Cc: 17512 <at> debbugs.gnu.org
Subject: Re: bug#17512: Valgrind/regression tests: Symlink test hang
Date: Fri, 23 May 2014 17:58:30 +0930

On 05/21/14 14:40, Jim Meyering wrote:
> On Fri, May 16, 2014 at 5:35 PM, Paul Eggert <eggert <at> cs.ucla.edu> wrote:
>>
>> [...] Perhaps Jim has a better way to do it, though -- he's more of a valgrind
>> wizard than I.
>
> I use a script to replace each binary with this generated script:
>
> #!/bin/sh
> saved_binaries_dir="${0%/*}/.vg-tmp"
> export PATH=$saved_binaries_dir:$PATH
> prog=${0##*/}
> # Disable this: http://udrepper.livejournal.com/11429.html
> MALLOC_PERTURB_=0
> export MALLOC_PERTURB_
> exec /usr/bin/valgrind -v --suppressions=/dev/null \
>    --log-file=/tmp/vg-log-2014-05-20.22:05:28-%q{GREP_TEST_NAME} \
>    --leak-check=full --track-fds=yes --num-callers=15 -- "$prog" "$@"
>
> which refers to the original binary via a subdirectory named .vg-tmp
> relative to the original location of the binary. That script also moves
> each binary into the .vg-tmp subdir.
>
> Then, running "make check" creates many /tmp/vg-log-* files,
> each related to the name of a test.
>

G'day Jim and Paul,

Thanks for the feedback.  I found that Jim's script worked in a similar
fashion to the home-brew script I tried, although I didn't know that
the test name was available as an environment variable, and so my
script ended up creating one massive log file.

One glitch that appears in both my script and in the one given by Jim:
The "symlink" test hangs, consuming 0% CPU, when run under Valgrind.
I ended up replacing the test with an "exit 0" script just for the sake
of getting a fast sweep across the majority of the test rig.

I haven't tried to hunt down the reason for this hang; I'm currently
on a Gentoo Link machine, with GCC and Valgrind versions:

        gcc (Gentoo 4.7.3-r1 p1.4, pie-0.5.5) 4.7.3
        Copyright (C) 2012 Free Software Foundation, Inc.
        This is free software; see the source for copying conditions.  There is NO
        warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

        valgrind-3.8.1

While there's some 4.8-series GCC versions available (but masked
for my architecture at present), that include valgrind-related
goodies such as "-Og" and "-fsanitize=address", I'm choosing a more
conservative version at present.  There's also masked versions
of Valgrind available (3.8.1-r1, 3.9.0), but again I'm being conservative
at present.

I'll wait for grep-2.20 to appear, and then will investigate whether
upgrading various tools can resolve this hang, and will report
back to the list.

cheers,

behoffski (Brenton Hoff)
Programmer, Grouse Software

ps. Congratulations to everybody on the completion of grep-2.19.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.