Subject: bug#23234: closed (Re: bug#23234: unexpected results with charset handling in GNU grep 2.23)

From: [email-masked] (GNU bug Tracking System)
Date: Sun, 10 Apr 2016 08:44:01 +0000

Your bug report

#23234: unexpected results with charset handling in GNU grep 2.23

which was filed against the grep package, has been closed.

The explanation is attached below, along with your original report.
If you require more details, please reply to [email-masked].

-- 
23234: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=23234
GNU Bug Tracking System
Contact [email-masked] with problems

Subject: Re: bug#23234: unexpected results with charset handling in GNU grep 2.23
From: Paul Eggert <[email-masked]>
To: Eric Blake <[email-masked]>, Björn JACKE<[email-masked]>, [email-masked]
Date: Sun, 10 Apr 2016 01:43:10 -0700
List-Id: <debbugs-submit.debbugs.gnu.org>

Paul Eggert wrote:
> I plan to change GNU grep to use this new facility, and to add some grep test
> cases for this issue.

I did that by installing the attached patches into the grep master. This fixes 
the bug for me, so I'm closing the bug report.

These patches mostly just report the fix and add test cases. The actual fix was 
in gnulib, here:

http://git.savannah.gnu.org/cgit/gnulib.git/commit/?id=b7bc3c1a4e78add4cbad39ae1a0c4fb0747b483f

This gnulib fix works around the underyling glibc facility which caused the 
problem, for which I've filed a bug report here:

https://sourceware.org/bugzilla/show_bug.cgi?id=19932

It's not clear when the glibc bug will be fixed. Until it is, one should expect 
similar problems to crop up in applications other than 'grep'.
From 0d6a45b37ab84dd338e3b595716d770f1ac2d07c Mon Sep 17 00:00:00 2001
From: Paul Eggert <[email-masked]>
Date: Sun, 10 Apr 2016 00:25:27 -0700
Subject: [PATCH 1/2] build: update gnulib submodule to latest

---
 gnulib | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/gnulib b/gnulib
index cd6a452..b7bc3c1 160000
--- a/gnulib
+++ b/gnulib
@@ -1 +1 @@
-Subproject commit cd6a45292cdb7b3c4b628f1cb0f199a02140ea7c
+Subproject commit b7bc3c1a4e78add4cbad39ae1a0c4fb0747b483f
-- 
2.5.5

From 17fb604a4cd23b07b99584706f92db8d6dd05e74 Mon Sep 17 00:00:00 2001
From: Paul Eggert <[email-masked]>
Date: Sun, 10 Apr 2016 01:33:25 -0700
Subject: [PATCH 2/2] grep: in C locale, all bytes are valid characters
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

This works around glibc bug 19932:
https://sourceware.org/bugzilla/show_bug.cgi?id=19932
The actual bug fix was the update to the current version of Gnulib.
grep problem reported by Björn Jacke in: http://bugs.gnu.org/23234
* NEWS: Mention this.
* doc/grep.texi (File and Directory Selection): Crossref to LC_*
section.  Suggest why -a or LC_ALL=C might be useful.
(Environment Variables): Mention 'locale -a'.
Say that LC_CTYPE also specifies encoding, and that every
byte is a valid character in the C or POSIX locale.
* tests/c-locale: New test.
* tests/Makefile.am (TESTS): Add it.
---
 NEWS              |  6 ++++++
 doc/grep.texi     | 19 ++++++++++++++-----
 tests/Makefile.am |  1 +
 tests/c-locale    | 26 ++++++++++++++++++++++++++
 4 files changed, 47 insertions(+), 5 deletions(-)
 create mode 100755 tests/c-locale

diff --git a/NEWS b/NEWS
index 69e4a23..63767aa 100644
--- a/NEWS
+++ b/NEWS
@@ -4,6 +4,12 @@ GNU grep NEWS                                    -*- outline -*-
 
 ** Bug fixes
 
+  In the C or POSIX locale, grep now treats all bytes as valid
+  characters even if the C runtime library says otherwise.  The
+  revised behavior is more compatible with the original intent of
+  POSIX, and the next release of POSIX will likely make this official.
+  [bug introduced in grep-2.23]
+
   grep -Pz no longer mistakenly diagnoses patterns like [^a] that use
   negated character classes. [bug introduced in grep-2.24]
 
diff --git a/doc/grep.texi b/doc/grep.texi
index 1d3d5cb..4e0e48e 100644
--- a/doc/grep.texi
+++ b/doc/grep.texi
@@ -599,7 +599,8 @@ If a file's data or metadata
 indicate that the file contains binary data,
 assume that the file is of type @var{type}.
 Non-text bytes indicate binary data; these are either output bytes that are
-improperly encoded for the current locale, or null input bytes when the
+improperly encoded for the current locale (@pxref{Environment
+Variables}), or null input bytes when the
 @option{-z} (@option{--null-data}) option is not given (@pxref{Other
 Options}).
 
@@ -627,10 +628,13 @@ is not matched when @var{type} is @samp{text}.  Conversely, when
 @var{type} is @samp{binary} the pattern @samp{.} (period) might not
 match a null byte.
 
-@emph{Warning:} @samp{--binary-files=text} might output binary garbage,
-which can have nasty side effects
-if the output is a terminal and
-if the terminal driver interprets some of it as commands.
+@emph{Warning:} The @option{-a} (@option{--binary-files=text}) option
+might output binary garbage, which can have nasty side effects if the
+output is a terminal and if the terminal driver interprets some of it
+as commands.  On the other hand, when reading files whose text
+encodings are unknown, it can be helpful to use @option{-a} or to set
+@samp{LC_ALL='C'} in the environment, in order to find more matches
+even if the matches are unsafe for direct display.
 
 @item -D @var{action}
 @itemx --devices=@var{action}
@@ -803,6 +807,7 @@ The @samp{C} locale is used if none of these environment variables are set,
 if the locale catalog is not installed,
 or if @command{grep} was not compiled
 with national language support (NLS).
+The shell command @code{locale -a} lists locales that are currently available.
 
 Many of the environment variables in the following list let you
 control highlighting using
@@ -1004,6 +1009,10 @@ interpreted.
 These variables specify the locale for the @env{LC_CTYPE} category,
 which determines the type of characters,
 e.g., which characters are whitespace.
+This category also determines the character encoding, that is, whether
+text is encoded in UTF-8, ASCII, or some other encoding.  In the
+@samp{C} or @samp{POSIX} locale, all characters are encoded as a
+single byte and every byte is a valid character.
 
 @item LANGUAGE
 @itemx LC_ALL
diff --git a/tests/Makefile.am b/tests/Makefile.am
index b65fc39..45908ce 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -53,6 +53,7 @@ TESTS =						\
   big-match					\
   bogus-wctob					\
   bre						\
+  c-locale					\
   case-fold-backref				\
   case-fold-backslash-w				\
   case-fold-char-class				\
diff --git a/tests/c-locale b/tests/c-locale
new file mode 100755
index 0000000..1fe5c70
--- /dev/null
+++ b/tests/c-locale
@@ -0,0 +1,26 @@
+#! /bin/sh
+# Regression test for GNU grep.
+#
+# Copyright 2016 Free Software Foundation, Inc.
+#
+# Copying and distribution of this file, with or without modification,
+# are permitted in any medium without royalty provided the copyright
+# notice and this notice are preserved.
+
+. "${srcdir=.}/init.sh"; path_prepend_ ../src
+
+fail=0
+
+c=1
+while test $c -lt 256; do
+  tr2=$(printf '\\%o\n' $c)
+  echo X | tr X "$tr2" >in
+  if test $(wc -l <in) -eq 1; then
+    grep . in >out || fail=1
+    compare in out || fail=1
+  fi
+  test $fail -ne 0 && Exit $fail
+  c=$(expr $c + 1)
+done
+
+Exit $fail
-- 
2.5.5

Subject: unexpected results with charset handling in GNU grep 2.23
From: Björn JACKE <[email-masked]>
To: [email-masked]
Date: Wed, 6 Apr 2016 21:25:21 +0200
List-Id: <debbugs-submit.debbugs.gnu.org>

Hi,

this change in GNU grep 2.23 has severe consequences:

> Binary files are now less likely to generate diagnostics and more
> likely to yield text matches.  grep now reports "Binary file FOO
> matches" and suppresses further output instead of outputting a line
> containing an encoding error; hence grep can now report matching text
> before a later binary match.  Formerly, grep reported FOO to be
> binary when it found an encoding error in FOO before generating
> output for FOO, which meant it never reported both matching text and
> matching binary data; this was less useful for searching text
> containing encoding errors in non-matching lines.

I got a report that the build of the German spellcheck dictionary got broken.
It tuned out that this happened after the update to GNU grep to 2.23:

https://bugzilla.redhat.com/show_bug.cgi?id=1316359

Actually the mentioned change leaves no reliable way to grep lines out of a
any text file, which contains non-ASCII characters.

Until now it was quite save to use grep in the C locale, also for non-ASCII
text. Now after that change, the locale charmap has to match all of the
encoding of the input file.  Unfortunately the only locale that definetely
always exists for sure is the C locale. We cannot assume that any other locale
definitions exist on an unknown system. For a script, that wants to use grep,
this is a big problem now.

Let's take this example using grep 2.23:

# echo -e "test\ntäst\ntest" | iconv -f utf8 -t latin1 | LC_ALL=C grep "st" ; echo $?
test
Binary file (standard input) matches
0

There are several problems here. Someone might want to assume that the locale
definitions for en_US.ISO-8859-1 exist. Unfortunetely such an assumtion cannot
be made. Whatever locale is used - if the definition might not be there and we
will fall back to the C locale in any case then.

The result is, we get the first matching line in the example. The second
matching line with a non-ASCII character returns the text "Binary file
(standard input) matches" on stdout (which might even be a valid matching line
of the input file!) and the following matches are skipped. (Finally the return
code is 0 - as the grepping stopped quickly, a return code >1 might be desireble,
but I don't want to dive into that point right now.)

Let me draw a biger picture: Have a look at what a POSIX compliant grep is
expected to do:
http://pubs.opengroup.org/onlinepubs/009604499/utilities/grep.html

Read the description section, especially:

--snip--
By default, an input line shall be selected if any pattern, treated as an
entire basic regular expression (BRE) as described in the Base Definitions
volume of IEEE Std 1003.1-2001, Section 9.3, Basic Regular Expressions, matches
any part of the line excluding the terminating <newline>;
--snap--

That means a posix compliant grep should not try to be too smart and tell the
user that a binary file matches the search pattern (people can use "strings" if
they want). It should just output the line. From that perspective GNU grep was
not posix compliant before either, but it was not a big problem for most people
obviously. With the recent change though and the issues described above I think
a lot of scripts using (GNU) grep will get broken.

I really hope this change will be reverted as soon as possible. I would rather
prefer GNU grep to become posix compliant and not do any binary detection by
default actually.

Cheers
Björn