GNU bug report logs - #64316
ASAN failures in dir.c

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 64316 in the body.
You can then email your comments to 64316 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Gisle Vanem <gvanem <at> yahoo.no>
To: bug-diffutils <at> gnu.org
Subject: ASAN failures in dir.c
Date: Tue, 27 Jun 2023 15:04:48 +0200

Hello list.

Since some time ago, building diff.exe with ASAN (on Windows-10),
causes it to trigger on illegal use of memcpy().
For example:

==3752==ERROR: AddressSanitizer: heap-use-after-free on address 0x121647e20772 at
pc 0x7ffc6e93727e bp 0x00d589efdba0 sp 0x00d589efd330
WRITE of size 17 at 0x121647e20772 thread T0
    #0 0x7ffc6e93727d in __asan_memcpy 
D:\a\_work\1\s\src\vctools\asan\llvm\compiler-rt\li\asan\asan_interceptors_memintrinsics.cpp:22
    #1 0x7ff604834af1 in dir_read F:\MinGW32\src\gnu\GNU-diff\src\dir.c:101
    #2 0x7ff604832ec7 in diff_dirs F:\MinGW32\src\gnu\GNU-diff\src\dir.c:214
    #3 0x7ff60482dc37 in compare_files F:\MinGW32\src\gnu\GNU-diff\src\diff.c:1369
    #4 0x7ff604833b1b in diff_dirs F:\MinGW32\src\gnu\GNU-diff\src\dir.c:289
    #5 0x7ff60482dc37 in compare_files F:\MinGW32\src\gnu\GNU-diff\src\diff.c:1369
    #6 0x7ff60483270e in main F:\MinGW32\src\gnu\GNU-diff\src\diff.c:862
    ...

-----------------------

This causes Heap Corruption all over the place.

I'm not sure this is an issue with Gnulib or diff. But reverting this
patch:


diff --git a/src/dir.c b/src/dir.c
index ba9403b..773afeb 100644
--- a/src/dir.c
+++ b/src/dir.c
@@ -95,13 +95,9 @@ dir_read (struct file_data const *dir, struct dirdata *dirdata)
           if (excluded_file_name (excluded, d_name))
             continue;

-          while (data_alloc - data_used < d_size)
-            {
-              if (IDX_MAX / 2 <= data_alloc)
-                xalloc_die ();
-              dirdata->data = data = xirealloc (data, data_alloc *= 2);
-            }
-
+          if (data_alloc - data_used < d_size)
+	    dirdata->data = xpalloc (dirdata->data, &data_alloc,
+				     d_size - (data_alloc - data_used), -1, 1);
           memcpy (data + data_used, d_name, d_size);
           data_used += d_size;
           nnames++;

--------

it work with ASAN too (although much slower).

-- 
--gv

Message #10 received at 64316-done <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Gisle Vanem <gvanem <at> yahoo.no>
Cc: 64316-done <at> debbugs.gnu.org
Subject: Re: [bug-diffutils] bug#64316: ASAN failures in dir.c
Date: Tue, 27 Jun 2023 09:40:06 -0700

[Message part 1 (text/plain, inline)]

Thanks for reporting that bug, which I recently introduced. I installed 
the attached to fix it.

[0001-diff-fix-xpalloc-typo.patch (text/x-patch, attachment)]

Message #13 received at 64316-done <at> debbugs.gnu.org (full text, mbox):

From: Gisle Vanem <gvanem <at> yahoo.no>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: 64316-done <at> debbugs.gnu.org
Subject: Re: [bug-diffutils] bug#64316: ASAN failures in dir.c
Date: Tue, 27 Jun 2023 21:52:59 +0200

> Thanks for reporting that bug, which I recently introduced. I installed the attached to fix it.

Applied it, ran the new version. Works fine (I think)
when compiling using 'clang-cl.exe',
But with MSVC's 'cl.exe', I often get:
  diff.exe: memory exhausted

(on directory branches with approx. > 7000 files)

From where the allocation fails, I've no idea since Gnulib is so
unfriendly telling where this occurs. Is there really no
xmalloc_die() with some more useful messages than this? Yikes!

-- 
--gv

Message #16 received at 64316 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Gisle Vanem <gvanem <at> yahoo.no>
Cc: 64316 <at> debbugs.gnu.org
Subject: Re: [bug-diffutils] bug#64316: ASAN failures in dir.c
Date: Tue, 27 Jun 2023 18:04:29 -0700

On 2023-06-27 12:52, Gisle Vanem wrote:
> with MSVC's 'cl.exe', I often get:
>    diff.exe: memory exhausted
> 
> (on directory branches with approx. > 7000 files)

I can't reproduce that on Ubuntu x86 or x86-64. I built a test case this 
way:

  for dir in d e; do
    (mkdir $dir &&
     cd $dir &&
     for i in $(seq 8000); do
       echo $dir$i>$i
     done)
  done

and "diff d e >f" worked just fine. Do you have a reproducible test case?

>  From where the allocation fails, I've no idea since Gnulib is so
> unfriendly telling where this occurs. Is there really no
> xmalloc_die() with some more useful messages than this? Yikes!

When you run out of memory, bad things happen pretty much everywhere. 
Too bad Microsoft gives you a bad backtrace.

You might try running the leak detector though to be honest I've not had 
much luck with that.

Another possibility is to try to find the commit that introduced the 
problem, if it's reproducible.

Message #19 received at 64316 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Gisle Vanem <gvanem <at> yahoo.no>
Cc: 64316 <at> debbugs.gnu.org
Subject: Re: [bug-diffutils] bug#64316: ASAN failures in dir.c
Date: Wed, 28 Jun 2023 15:31:55 -0700

[Message part 1 (text/plain, inline)]

On 2023-06-28 03:32, Gisle Vanem wrote:
> 'PTRDIFF_T_MAX - - 3' folds to '-9223372036854775806'.
> 
> But how can 'linbuf_base' become negative?

Ordinarily it's zero, but it can be negative. It's never positive.

Thanks for the report. I reproduced the signed integer overflow issue 
with -fsanitize=undefined and installed the attached patch to fix it and 
add a regression test (and also a comment about linbuf_base).

Although I don't see how this would fix a "memory exhausted" error, 
please give the latest Git commit a try and see whether it fixes things 
for you.

[0001-diff-fix-xpalloc-related-signed-integer-overflow.patch (text/x-patch, attachment)]

Message #22 received at 64316 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Gisle Vanem <gvanem <at> yahoo.no>
Cc: 64316 <at> debbugs.gnu.org
Subject: Re: [bug-diffutils] bug#64316: ASAN failures in dir.c
Date: Thu, 29 Jun 2023 09:59:41 -0700

On 2023-06-29 05:09, Gisle Vanem wrote:

> Seems to work fine now. For both MSVC and clang-cl.

Thanks for checking; I'm noting this in the bug report.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.