GNU bug report logs - #74428
tests/printf/printf-cov failure on macOS

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 74428 in the body.
You can then email your comments to 74428 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Grisha Levit <grishalevit <at> gmail.com>
To: bug-coreutils <at> gnu.org
Subject: tests/printf/printf-cov failure on macOS
Date: Mon, 18 Nov 2024 23:41:46 -0500

The u4 and U8 tests in tests/printf/printf-cov.pl fail on macOS 15:

u4...
printf: test u4: stdout mismatch, comparing u4.1 (expected) and u4.O (actual)
*** u4.1	Mon Nov 18 23:30:03 2024
--- u4.O	Mon Nov 18 23:30:03 2024
***************
*** 1 ****
! 0
\ No newline at end of file
--- 1 ----
! \u0030
\ No newline at end of file
U8...
printf: test U8: stdout mismatch, comparing U8.1 (expected) and U8.O (actual)
*** U8.1	Mon Nov 18 23:30:03 2024
--- U8.O	Mon Nov 18 23:30:03 2024
***************
*** 1 ****
! 0
\ No newline at end of file
--- 1 ----
! \u0030
\ No newline at end of file

Message #8 received at 74428 <at> debbugs.gnu.org (full text, mbox):

From: Pádraig Brady <P <at> draigBrady.com>
To: Grisha Levit <grishalevit <at> gmail.com>, 74428 <at> debbugs.gnu.org
Cc: bug-gnulib <bug-gnulib <at> gnu.org>
Subject: Re: bug#74428: tests/printf/printf-cov failure on macOS
Date: Tue, 19 Nov 2024 15:45:15 +0000

On 19/11/2024 04:41, Grisha Levit wrote:
> The u4 and U8 tests in tests/printf/printf-cov.pl fail on macOS 15:
> 
> u4...
> printf: test u4: stdout mismatch, comparing u4.1 (expected) and u4.O (actual)
> *** u4.1	Mon Nov 18 23:30:03 2024
> --- u4.O	Mon Nov 18 23:30:03 2024
> ***************
> *** 1 ****
> ! 0
> \ No newline at end of file
> --- 1 ----
> ! \u0030
> \ No newline at end of file
> U8...
> printf: test U8: stdout mismatch, comparing U8.1 (expected) and U8.O (actual)
> *** U8.1	Mon Nov 18 23:30:03 2024
> --- U8.O	Mon Nov 18 23:30:03 2024
> ***************
> *** 1 ****
> ! 0
> \ No newline at end of file
> --- 1 ----
> ! \u0030
> \ No newline at end of file

CC'ing gnulib as the pertinent code is in unicodeio.c
On your macos 15 system, iconv() of 0x30 is failing to convert from utf8 to C,
and the fallback in unicodeio.c is outputting the \u0030.
Now I don't have access to macos to see exactly why that iconv() is failing,
but failing for anything < 0x80 is surprising.

cheers,
Pádraig

p.s. to test this in isolation in coreutils you can use:
make TESTS=tests/printf/printf-cov.pl SUBDIRS=. check

Message #11 received at 74428 <at> debbugs.gnu.org (full text, mbox):

From: Bruno Haible <bruno <at> clisp.org>
To: bug-gnulib <at> gnu.org, Pádraig Brady <P <at> draigbrady.com>
Cc: 74428 <at> debbugs.gnu.org, Grisha Levit <grishalevit <at> gmail.com>
Subject: Re: bug#74428: tests/printf/printf-cov failure on macOS
Date: Tue, 19 Nov 2024 17:08:07 +0100

Pádraig Brady wrote:
> On your macos 15 system, iconv() of 0x30 is failing to convert from utf8 to C,
> and the fallback in unicodeio.c is outputting the \u0030.
> Now I don't have access to macos to see exactly why that iconv() is failing,

The macOS iconv() is unusable, since macOS 12. [1][2]

A user needs to install GNU libiconv there, if they want programs to behave
normally. As documented in gnulib/DEPENDENCIES.

You may experiment with some patch to unicodeio.c, based on [2].

Or how about bypassing unicode_to_mb entirely if CODE >= 0 && CODE < 128 ?
unicode_to_mb was designed for specific characters like the Copyright sign
or the quotation marks. For plain ASCII characters you can bypass it.

Bruno

[1] https://lists.gnu.org/archive/html/bug-gnulib/2024-02/msg00123.html
[2] https://lists.gnu.org/archive/html/bug-gnulib/2024-02/msg00217.html

Message #14 received at 74428 <at> debbugs.gnu.org (full text, mbox):

From: Pádraig Brady <P <at> draigBrady.com>
To: Bruno Haible <bruno <at> clisp.org>, bug-gnulib <at> gnu.org
Cc: 74428 <at> debbugs.gnu.org, Grisha Levit <grishalevit <at> gmail.com>
Subject: Re: bug#74428: tests/printf/printf-cov failure on macOS
Date: Tue, 19 Nov 2024 17:02:28 +0000

On 19/11/2024 16:08, Bruno Haible wrote:
> Pádraig Brady wrote:
>> On your macos 15 system, iconv() of 0x30 is failing to convert from utf8 to C,
>> and the fallback in unicodeio.c is outputting the \u0030.
>> Now I don't have access to macos to see exactly why that iconv() is failing,
> 
> The macOS iconv() is unusable, since macOS 12. [1][2]
> 
> A user needs to install GNU libiconv there, if they want programs to behave
> normally. As documented in gnulib/DEPENDENCIES.
> 
> You may experiment with some patch to unicodeio.c, based on [2].
> 
> Or how about bypassing unicode_to_mb entirely if CODE >= 0 && CODE < 128 ?
> unicode_to_mb was designed for specific characters like the Copyright sign
> or the quotation marks. For plain ASCII characters you can bypass it.
> 
> Bruno
> 
> [1] https://lists.gnu.org/archive/html/bug-gnulib/2024-02/msg00123.html
> [2] https://lists.gnu.org/archive/html/bug-gnulib/2024-02/msg00217.html

I would prefer to bypass the ASCII case if CODE >= 0 && CODE < 128.
However is that generally correct? Consider EBCDIC for example, with something like:

  LC_ALL=EBCDIC-US env printf '\u0030\n'

Now that's not supported anyway on my Fedora 40 system at least,
but perhaps it might be in some setups?

thanks,
Pádraig

Message #17 received at 74428 <at> debbugs.gnu.org (full text, mbox):

From: Bruno Haible <bruno <at> clisp.org>
To: bug-gnulib <at> gnu.org, Pádraig Brady <P <at> draigbrady.com>
Cc: 74428 <at> debbugs.gnu.org, Grisha Levit <grishalevit <at> gmail.com>
Subject: Re: bug#74428: tests/printf/printf-cov failure on macOS
Date: Tue, 19 Nov 2024 18:34:43 +0100

Pádraig Brady wrote:
> I would prefer to bypass the ASCII case if CODE >= 0 && CODE < 128.
> However is that generally correct?

Yes, at least for CODE >= 32 && CODE < 128 it is correct.
This can be seen from the list of supported locale encodings in
gnulib/lib/localcharset.h.

> Consider EBCDIC for example, with something like:
> 
>    LC_ALL=EBCDIC-US env printf '\u0030\n'
> 
> Now that's not supported anyway on my Fedora 40 system at least,
> but perhaps it might be in some setups?

No, it's not supported anywhere. The last platform that significantly
uses EBCDIC is IBM z/OS, and there the environment for C programs is
based on ASCII (i.e. the runtime library performs conversions between
ASCII-encoded in-memory data and EBCDIC-encoded disk data on the fly). [1][2]

That's also the reason why no one ever reported a problem with gperf-
generated C code, that assumes an ASCII-compatible encoding.

Bruno

[1] https://lists.gnu.org/archive/html/bug-gnu-libiconv/2023-04/msg00019.html
[2] https://lists.gnu.org/archive/html/bug-gnu-libiconv/2023-05/msg00002.html

Message #22 received at 74428-done <at> debbugs.gnu.org (full text, mbox):

From: Pádraig Brady <P <at> draigBrady.com>
To: Bruno Haible <bruno <at> clisp.org>, bug-gnulib <at> gnu.org
Cc: 74428-done <at> debbugs.gnu.org, Grisha Levit <grishalevit <at> gmail.com>
Subject: Re: bug#74428: tests/printf/printf-cov failure on macOS
Date: Tue, 19 Nov 2024 18:31:31 +0000

[Message part 1 (text/plain, inline)]

On 19/11/2024 17:34, Bruno Haible wrote:
> Pádraig Brady wrote:
>> I would prefer to bypass the ASCII case if CODE >= 0 && CODE < 128.
>> However is that generally correct?
> 
> Yes, at least for CODE >= 32 && CODE < 128 it is correct.
> This can be seen from the list of supported locale encodings in
> gnulib/lib/localcharset.h.

OK I've adjusted our test to use \u00032 instead,
and tested the attached code, which I'll push later.

thanks,
Pádraig

[gnulib-uc-ascii.diff (text/x-patch, attachment)]

Message #25 received at 74428-done <at> debbugs.gnu.org (full text, mbox):

From: Bruno Haible <bruno <at> clisp.org>
To: bug-gnulib <at> gnu.org, Pádraig Brady <P <at> draigbrady.com>
Cc: 74428-done <at> debbugs.gnu.org, Grisha Levit <grishalevit <at> gmail.com>
Subject: Re: bug#74428: tests/printf/printf-cov failure on macOS
Date: Tue, 19 Nov 2024 19:52:06 +0100

Pádraig Brady wrote:
> and tested the attached code, which I'll push later.

Looks good, except for a typo in comment: Simplifiy → Simplify

Thanks.

Bruno

Message #28 received at 74428 <at> debbugs.gnu.org (full text, mbox):

From: Bernhard Voelker <mail <at> bernhard-voelker.de>
To: 74428 <at> debbugs.gnu.org, P <at> draigBrady.com, grishalevit <at> gmail.com
Subject: Re: bug#74428: tests/printf/printf-cov failure on macOS
Date: Fri, 6 Dec 2024 08:13:01 +0100

[Message part 1 (text/plain, inline)]

On 11/19/24 19:31, Pádraig Brady wrote:
> OK I've adjusted our test to use \u00032 instead,

The bug report was talking about Macos only, but now I'm seeing this failure also
on openSUSE for Leap 16.0 on x86_64 [1] and aarch64 [2].

Using coreutils-9.5 (which has gnulib 259829e78), glibc-2.40.

[1] https://build.opensuse.org/public/build/Base:System/16.0/x86_64/coreutils/_log
    also saved/attached as 'cu-tests-printf-cov-x86_64.log.xz'.

[2] https://build.opensuse.org/public/build/Base:System/16.0/aarch64/coreutils/_log
    also saved/attached as 'cu-tests-printf-cov-aarch64.log.xz'.

Here an excerpt from the latter aarch64 build log:
  ...
  [   93s] checking for iconv... yes
  [   94s] checking for working iconv... no
  [   94s] checking whether iconv is compatible with its POSIX signature... yes
  ...
  [  376s] FAIL: tests/printf/printf-cov
  [  376s] =============================
  ...
  [  376s] U-bad-hex...
  [  376s] u4...
  [  376s] printf: test u4: stdout mismatch, comparing u4.1 (expected) and u4.O (actual)
  [  376s] *** u4.1	Thu Dec  5 21:06:59 2024
  [  376s] --- u4.O	Thu Dec  5 21:06:59 2024
  [  376s] ***************
  [  376s] *** 1 ****
  [  376s] ! 0
  [  376s] \ No newline at end of file
  [  376s] --- 1 ----
  [  376s] ! \u0030
  [  376s] \ No newline at end of file
  [  376s] U8...
  [  376s] printf: test U8: stdout mismatch, comparing U8.1 (expected) and U8.O (actual)
  [  376s] *** U8.1	Thu Dec  5 21:06:59 2024
  [  376s] --- U8.O	Thu Dec  5 21:06:59 2024
  [  376s] ***************
  [  376s] *** 1 ****
  [  376s] ! 0
  [  376s] \ No newline at end of file
  [  376s] --- 1 ----
  [  376s] ! \u0030
  [  376s] \ No newline at end of file
  [  376s] u-invalid...
  ...
  [  376s] FAIL tests/printf/printf-cov.pl (exit status: 1)

Is this an FP on more platforms than Macos now, or another issue like wrong iconv() selection?

Have a nice day,
Berny

[cu-tests-printf-cov-x86_64.log.xz (application/x-xz, attachment)]

[cu-tests-printf-cov-aarch64.log.xz (application/x-xz, attachment)]

Message #31 received at 74428 <at> debbugs.gnu.org (full text, mbox):

From: Pádraig Brady <P <at> draigBrady.com>
To: Bernhard Voelker <mail <at> bernhard-voelker.de>, 74428 <at> debbugs.gnu.org,
 grishalevit <at> gmail.com
Cc: Bruno Haible <bruno <at> clisp.org>
Subject: Re: bug#74428: tests/printf/printf-cov failure on macOS
Date: Fri, 6 Dec 2024 13:24:44 +0000

On 06/12/2024 07:13, Bernhard Voelker wrote:
> On 11/19/24 19:31, Pádraig Brady wrote:
>> OK I've adjusted our test to use \u00032 instead,
> 
> The bug report was talking about Macos only, but now I'm seeing this failure also
> on openSUSE for Leap 16.0 on x86_64 [1] and aarch64 [2].
> 
> Using coreutils-9.5 (which has gnulib 259829e78), glibc-2.40.
> 
> [1] https://build.opensuse.org/public/build/Base:System/16.0/x86_64/coreutils/_log
>       also saved/attached as 'cu-tests-printf-cov-x86_64.log.xz'.
> 
> [2] https://build.opensuse.org/public/build/Base:System/16.0/aarch64/coreutils/_log
>       also saved/attached as 'cu-tests-printf-cov-aarch64.log.xz'.
> 
> Here an excerpt from the latter aarch64 build log:
>     ...
>     [   93s] checking for iconv... yes
>     [   94s] checking for working iconv... no
>     [   94s] checking whether iconv is compatible with its POSIX signature... yes
>     ...
>     [  376s] FAIL: tests/printf/printf-cov
>     [  376s] =============================
>     ...
>     [  376s] U-bad-hex...
>     [  376s] u4...
>     [  376s] printf: test u4: stdout mismatch, comparing u4.1 (expected) and u4.O (actual)
>     [  376s] *** u4.1	Thu Dec  5 21:06:59 2024
>     [  376s] --- u4.O	Thu Dec  5 21:06:59 2024
>     [  376s] ***************
>     [  376s] *** 1 ****
>     [  376s] ! 0
>     [  376s] \ No newline at end of file
>     [  376s] --- 1 ----
>     [  376s] ! \u0030
>     [  376s] \ No newline at end of file
>     [  376s] U8...
>     [  376s] printf: test U8: stdout mismatch, comparing U8.1 (expected) and U8.O (actual)
>     [  376s] *** U8.1	Thu Dec  5 21:06:59 2024
>     [  376s] --- U8.O	Thu Dec  5 21:06:59 2024
>     [  376s] ***************
>     [  376s] *** 1 ****
>     [  376s] ! 0
>     [  376s] \ No newline at end of file
>     [  376s] --- 1 ----
>     [  376s] ! \u0030
>     [  376s] \ No newline at end of file
>     [  376s] u-invalid...
>     ...
>     [  376s] FAIL tests/printf/printf-cov.pl (exit status: 1)
> 
> Is this an FP on more platforms than Macos now, or another issue like wrong iconv() selection?

Interesting.
Can you put some debugging code in unicode_to_mb() to indicate the path taken?
Perhaps this is a glibc-2.40 issue (I've only tested on 2.39 as of now).

cheers,
Pádraig

Message #34 received at 74428 <at> debbugs.gnu.org (full text, mbox):

From: Bernhard Voelker <mail <at> bernhard-voelker.de>
To: Pádraig Brady <P <at> draigBrady.com>, 74428 <at> debbugs.gnu.org, 
 grishalevit <at> gmail.com
Cc: Bruno Haible <bruno <at> clisp.org>
Subject: Re: bug#74428: tests/printf/printf-cov failure on macOS
Date: Mon, 9 Dec 2024 22:41:46 +0100

[Message part 1 (text/plain, inline)]

On 12/6/24 14:24, Pádraig Brady wrote:
> On 06/12/2024 07:13, Bernhard Voelker wrote:
>> On 11/19/24 19:31, Pádraig Brady wrote:
>>> OK I've adjusted our test to use \u00032 instead,
>>
>> The bug report was talking about Macos only, but now I'm seeing this failure also
>> on openSUSE for Leap 16.0 on x86_64 [1] and aarch64 [2].
>>
>> Using coreutils-9.5 (which has gnulib 259829e78), glibc-2.40.
>>
>> [1] https://build.opensuse.org/public/build/Base:System/16.0/x86_64/coreutils/_log
>>        also saved/attached as 'cu-tests-printf-cov-x86_64.log.xz'.
>>
>> [2] https://build.opensuse.org/public/build/Base:System/16.0/aarch64/coreutils/_log
>>        also saved/attached as 'cu-tests-printf-cov-aarch64.log.xz'.
>>
>> Here an excerpt from the latter aarch64 build log:
>>      ...
>>      [   93s] checking for iconv... yes
>>      [   94s] checking for working iconv... no
>>      [   94s] checking whether iconv is compatible with its POSIX signature... yes
>>      ...
>>      [  376s] FAIL: tests/printf/printf-cov
>>      [  376s] =============================
>>      ...
>>      [  376s] U-bad-hex...
>>      [  376s] u4...
>>      [  376s] printf: test u4: stdout mismatch, comparing u4.1 (expected) and u4.O (actual)
>>      [  376s] *** u4.1	Thu Dec  5 21:06:59 2024
>>      [  376s] --- u4.O	Thu Dec  5 21:06:59 2024
>>      [  376s] ***************
>>      [  376s] *** 1 ****
>>      [  376s] ! 0
>>      [  376s] \ No newline at end of file
>>      [  376s] --- 1 ----
>>      [  376s] ! \u0030
>>      [  376s] \ No newline at end of file
>>      [  376s] U8...
>>      [  376s] printf: test U8: stdout mismatch, comparing U8.1 (expected) and U8.O (actual)
>>      [  376s] *** U8.1	Thu Dec  5 21:06:59 2024
>>      [  376s] --- U8.O	Thu Dec  5 21:06:59 2024
>>      [  376s] ***************
>>      [  376s] *** 1 ****
>>      [  376s] ! 0
>>      [  376s] \ No newline at end of file
>>      [  376s] --- 1 ----
>>      [  376s] ! \u0030
>>      [  376s] \ No newline at end of file
>>      [  376s] u-invalid...
>>      ...
>>      [  376s] FAIL tests/printf/printf-cov.pl (exit status: 1)
>>
>> Is this an FP on more platforms than Macos now, or another issue like wrong iconv() selection?
>
> Interesting.
> Can you put some debugging code in unicode_to_mb() to indicate the path taken?

    97	  /* Test whether the utf8_to_local converter is available at all.  */
    98	  if (!is_utf8)
    99	    {
   100	#if HAVE_ICONV
   101	      if (utf8_to_local == (iconv_t)(-1)) {
   102	        fprintf (stderr, "iconv function not usable: '%s'\n", callback_arg);
   103	        return failure (code, N_("iconv function not usable"), callback_arg); }
   104	#else
   105	      fprintf (stderr, "iconv function not available: '%s'\n", callback_arg);
   106	      return failure (code, N_("iconv function not available"), callback_arg);
   107	#endif
   108	    }

We're running into "iconv function not available" (line 105) there.
Compressed 'config.log' attached.

> Perhaps this is a glibc-2.40 issue (I've only tested on 2.39 as of now).

I have other systems with glibc-2.40 already ... without a problem there.

Have a nice day,
Berny

[config.log.xz (application/x-xz, attachment)]

Message #37 received at 74428 <at> debbugs.gnu.org (full text, mbox):

From: Bruno Haible <bruno <at> clisp.org>
To: Pádraig Brady <P <at> draigbrady.com>, 74428 <at> debbugs.gnu.org,
 grishalevit <at> gmail.com, Bernhard Voelker <mail <at> bernhard-voelker.de>
Subject: Re: bug#74428: tests/printf/printf-cov failure on macOS
Date: Mon, 09 Dec 2024 23:37:20 +0100

Bernhard Voelker wrote:
> >> on openSUSE for Leap 16.0 on x86_64 [1] and aarch64 [2].
> >>
> >> Using coreutils-9.5 (which has gnulib 259829e78), glibc-2.40.
> >> ...
>      97	  /* Test whether the utf8_to_local converter is available at all.  */
>      98	  if (!is_utf8)
>      99	    {
>     100	#if HAVE_ICONV
>     101	      if (utf8_to_local == (iconv_t)(-1)) {
>     102	        fprintf (stderr, "iconv function not usable: '%s'\n", callback_arg);
>     103	        return failure (code, N_("iconv function not usable"), callback_arg); }
>     104	#else
>     105	      fprintf (stderr, "iconv function not available: '%s'\n", callback_arg);
>     106	      return failure (code, N_("iconv function not available"), callback_arg);
>     107	#endif
>     108	    }
> 
> We're running into "iconv function not available" (line 105) there.
> Compressed 'config.log' attached.

In this log, I see:

configure:23922: checking for working iconv
configure:24066: gcc -o conftest -O2 -Wall -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=3 -fstack-protector-strong -funwind-tables -fasynchronous-unwind-tables -fstack-clash-protection -Werror=return-type -flto=auto -g  -flto=auto conftest.c  >&5
configure:24066: $? = 0
configure:24066: ./conftest
configure:24066: $? = 16
configure: program exited with status 16

and status 16 means that there is no EUC-JP converter.

This is now the second distro I hear of, that ships glibc without any loadable
iconv converters by default.

I have therefore reported
https://sourceware.org/bugzilla/show_bug.cgi?id=32437
in the hope that glibc makes it easier to distros to ship a reasonable set
of iconv converters.

Bruno

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.