Subject: bug#32236: df header corrupted with LANG=zh_TW.UTF-8 on macOS

From: Pádraig Brady <[email-masked]>
Date: Sat, 21 Jul 2018 13:30:25 -0700

On 21/07/18 07:20, Chih-Hsuan Yen wrote:
> Hi coreutils developers,
> 
> I'm using coreutils on macOS High Sierra (10.13). I noticed that with
> `LANG=zh_TW.UTF-8`, `df` output is corrupted.
> 
> �?�?系統 容�?? 已�?� �?��?� 已�?�% �??�?�?
> /dev/disk1s1    234G  151G    81G    65% /
> /dev/disk1s4    234G  2.1G    81G     3% /private/var/vm
> 
> (I'm not sure if other mail agents can display those characters
> correctly or not. See my blog post [1] for the exact output.)
> 
> Seems it's similar to bug#25630 [2], which is not resolved. I guess
> the reason of my issue is that iscntrl() is broken on macOS High
> Sierra, so in hide_problematic_chars(), some bytes in the Chinese
> header is replaced with a question mark. I managed to patch coreutils
> [3] to make `df` work. Could you have a look? Thanks!
> 
> Best,
> 
> Chih-Hsuan Yen
> 
> [1] https://blog.chyen.cc/posts/2018/06/23/mac-df-chinese.html
> [2] http://lists.gnu.org/archive/html/bug-coreutils/2017-02/msg00008.html
> [3] https://github.com/yan12125/macports-ports/blob/fix-coreutils-df-chinese/sysutils/coreutils/files/patch-df.diff

Wow. That's surprising. I do see the FreeBSD man pages say:

"The 4.4BSD extension of accepting arguments outside of the range of the unsigned char type
in locales with large character sets is considered obsolete and may not be supported in
future releases."

Now I think that might have been referring to >= 0xFF, but fair enough.

I've attached a gnulib patch to document for iscntrl at least.
It would be great if someone could test the other is*() classification
functions on macOS so that I might have a more complete documentation patch.

I've also attached an alternative patch for df (in your name).
Can you try that one?

thanks!
Pádraig
>From 6b7434fb222144af3ae9e2d4fd6b4c72eec25f5b Mon Sep 17 00:00:00 2001
From: Chih-Hsuan Yen <[email-masked]>
Date: Sat, 21 Jul 2018 13:19:23 -0700
Subject: [PATCH] df: avoid multibyte character corruption on macOS

* src/df.c (hide_problematic_chars): Use c_iscntrl() as
passing 8 bit characters to iscntrl() is not supported on macOS.
* NEWS: Mention the bug fix.
Fixes https://bugs.gnu.org/32236
---
 NEWS     | 4 ++++
 src/df.c | 3 ++-
 2 files changed, 6 insertions(+), 1 deletion(-)

diff --git a/NEWS b/NEWS
index af1a990..aa3b4f9 100644
--- a/NEWS
+++ b/NEWS
@@ -2,6 +2,10 @@ GNU coreutils NEWS                                    -*- outline -*-
 
 * Noteworthy changes in release ?.? (????-??-??) [?]
 
+** Bug fixes
+
+  df no longer corrupts displayed multibyte characters on macOS.
+
 
 * Noteworthy changes in release 8.30 (2018-07-01) [stable]
 
diff --git a/src/df.c b/src/df.c
index 1178865..c851fcc 100644
--- a/src/df.c
+++ b/src/df.c
@@ -23,6 +23,7 @@
 #include <sys/types.h>
 #include <getopt.h>
 #include <assert.h>
+#include <c-ctype.h>
 
 #include "system.h"
 #include "canonicalize.h"
@@ -281,7 +282,7 @@ hide_problematic_chars (char *cell)
   char *p = cell;
   while (*p)
     {
-      if (iscntrl (to_uchar (*p)))
+      if (c_iscntrl (to_uchar (*p)))
         *p = '?';
       p++;
     }
-- 
2.9.3

>From 816cc0d5fb92552a551c523f49c829261731dfe8 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?P=C3=A1draig=20Brady?= <[email-masked]>
Date: Sat, 21 Jul 2018 13:15:13 -0700
Subject: [PATCH] iscntrl: document that macOS returns true for >= 0x80

* doc/posix-functions/iscntrl.texi: Mention that support
for chars >= 0x80 is not standarized, and not supported
on OS X >= 10.5 at least
---
 doc/posix-functions/iscntrl.texi | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/doc/posix-functions/iscntrl.texi b/doc/posix-functions/iscntrl.texi
index 7e6813f..3c15708 100644
--- a/doc/posix-functions/iscntrl.texi
+++ b/doc/posix-functions/iscntrl.texi
@@ -16,6 +16,9 @@ OS X 10.8.
 
 Portability problems not fixed by Gnulib:
 @itemize
+This function does not support arguments outside of the range of the
+unsigned char type in locales with large character sets, on some platforms.
+OS X 10.5 will return non zero for characters >= 0x80 in UTF-8 locales.
 @end itemize
 
 Note: This function's behaviour depends on the locale, but does not support
-- 
2.9.3