To whom it may concern,

I’m writing to let you know that I found an issue by fuzzing in coreutils in factor utility and want to report it. Here are some details:

1. Host architecture: Host it Debian x86_64 architecture
2. factor version: factor (GNU coreutils) 9.5.94-5cecd
3. Affected code area: src/factor.c:425
4. Steps to reproduce:
Working on commit: 5cecd703e57b2e1301767d82cbe5bb01cae88472

    export CC="clang-17"

          export CXX="clang++-17"

          export CFLAGS="-fsanitize=address,undefined -g"

          export LDFLAGS="-fsanitize=address,undefined -g"

          export
UBSAN_OPTIONS=halt_on_error=1,abort_on_error=1,print_stacktrace=true,symbolize=true,print_stacktrace=1,report_error_type=1,symbolize=1

          ./bootstrap 

          ./configure

          make

          ./src/factor 22222222222222222202111121111

5. Bug details: during fuzzing with Undefined-Behaviour sanitizer we've got the following report:

        src/factor.c:425:3: runtime error: shift
      exponent 64 is too large for 64-bit type 'uintmax_t' (aka
      'unsigned long')

          #0 0x56332975f862 in mod2
      /home/artemiin/Work/crash_confirmation/coreutils/src/factor.c:425:3

          #1 0x56332975ae54 in factor_using_pollard_rho2 
      coreutils/src/factor.c:1665:12

          #2 0x563329750ab5 in factor  coreutils/src/factor.c:2246:9

          #3 0x56332974eed6 in print_factors_single 
      coreutils/src/factor.c:2454:3

          #4 0x56332974dd4c in print_factors 
      coreutils/src/factor.c:2506:11

          #5 0x56332974d20d in main  coreutils/src/factor.c:2647:15

          #6 0x7fa1933eb249 in __libc_start_call_main
      csu/../sysdeps/nptl/libc_start_call_main.h:58:16

          #7 0x7fa1933eb304 in __libc_start_main
      csu/../csu/libc-start.c:360:3

          #8 0x563329673630 in _start ( coreutils/src/factor+0x59630)
      (BuildId: 28b6f8b912cd8e99b886145a922476ec873a438b)

    During analysis of this report, I found that there are some numbers, like 22222222222222222202111121111, which fall into the function mod2 and produce cnt=0.
    I suppose that you expect lsh2(number, 0) == number (shift by zero should not change the number). But inside the realization of that macro, with cnt=0 we have an operation (d0) >> (64 - (cnt)) which stands for d0>>64. This is generally undefined behavior - on some systems, the number d0 remains unchanged (but is expected to be 0) and on other systems it can be zero.
    So, generally, the result of this operation is undefined. Although the result is correct for the number 22222222222222222202111121111, it may not be true for other numbers or architectures.

6. Patch suggestion: I suggest just not to call lsh2 when cnt=0 to avoid this bug. My patch is in the attachment.

Waiting for your reply,
Best regards,
Nasonov Artem

C уважением,

Артём Насонов

Специалист по анализу безопасности
Департамент анализа безопасности
Отдел динамического анализа

Эл. почта:

anasonov@astralinux.ru

Группа Астра

Сайт: astragroup.ru