Subject: bug#22254: [PATCH] sed: do not elide an invalid byte in a substitution RHS

From: Jim Meyering <[email-masked]>
Date: Sun, 27 Dec 2015 22:42:53 -0800

On Sun, Dec 27, 2015 at 10:24 PM, Jim Meyering <[email-masked]> wrote:
> On Sun, Dec 27, 2015 at 10:02 PM, Jim Meyering <[email-masked]> wrote:
>> Just after pushing the fix for that heap-clobbering bug,
>> http://debbugs.gnu.org/22127, I noticed there was another minor
>> problem: sed would silently elide any invalid byte specified in a
>> substitution replacement string.
>>
>> Here's a patch that's not quite complete (no NEWS entry, and I haven't
>> yet tried to determine when it was introduced).
>
> This one has a checkered past. I keep a decent sampling of old
> versions of sed around.
> Running one of the two tests that would be affected shows how
> treatment of an invalid byte in a multi-byte locale has differed over
> the years:
>
> $ for i in $(env ls -dv /p/p/sed-*); do printf '%-14s: ' $i; echo a
> |LC_ALL=ja_JP $i/bin/sed 's/a/b\U\xb2c/'|od -tx1|head -1; done|sed
> s/.....//
> sed-1.18 : 0000000 62 55 78 62 32 63 0a
> sed-2.05 : 0000000 62 55 78 62 32 63 0a
> sed-3.01 : 0000000 62 55 78 62 32 63 0a
> sed-3.02 : 0000000 62 55 78 62 32 63 0a
> sed-4.0a : 0000000 62 00 43 0a
> sed-4.0b : 0000000 62 00 43 0a
> sed-4.0c : 0000000 62 00 43 0a
> sed-4.0d : 0000000 62 00 43 0a
> sed-4.0e : 0000000 62 00 43 0a
> sed-4.0.6: 0000000 62 b2 43 0a
> sed-4.0.7: 0000000 62 b2 43 0a
> sed-4.0.8: 0000000 62 b2 43 0a
> sed-4.0.9: 0000000 62 b2 43 0a
> sed-4.1  : 0000000 62 00 43 0a
> sed-4.1c : 0000000 62 00 43 0a
> sed-4.1d : 0000000 62 00 43 0a
> sed-4.1e : 0000000 62 00 43 0a
> sed-4.1f : 0000000 62 00 43 0a
> sed-4.1.1: 0000000 62 00 43 0a
> sed-4.1.2: 0000000 62 00 43 0a
> sed-4.1.3: 0000000 62 00 43 0a
> sed-4.1.4: 0000000 62 00 43 0a
> sed-4.1.5: 0000000 62 00 43 0a
> sed-4.2  : 0000000 62 00 43 0a
> sed-4.2.1: 0000000 62 00 43 0a
> sed-4.2.2: 0000000 62 00 43 0a
>
> Now, it does what 4.0.6..4.0.9 did.

Here's a complete patch:
From 9dd563c897eba7bebf150f2ef6b0db3e72b7d7a2 Mon Sep 17 00:00:00 2001
From: Jim Meyering <[email-masked]>
Date: Sun, 27 Dec 2015 21:56:04 -0800
Subject: [PATCH] sed: do not elide an invalid byte in a substitution RHS

Now, sed copies such bytes into the output.
* sed/execute.c (str_append_modified): Copy each invalid byte
into the result string, as the comment suggests.
* testsuite/invalid-mb-seq-UMR.sh: Adjust this test to conform.
Perl does the same thing.  Also, remove an unnecessary "-e" option.
* NEWS (Bug fixes): Mention it.
Bug introduced prior to the switch to git in 2004,
between the releases of sed-4.0.9 and sed-4.1.
This addresses http://debbugs.gnu.org/22254.
---
 NEWS                            | 4 ++++
 sed/execute.c                   | 1 +
 testsuite/invalid-mb-seq-UMR.sh | 4 ++--
 3 files changed, 7 insertions(+), 2 deletions(-)

diff --git a/NEWS b/NEWS
index bc2f96f..7b5c6f9 100644
--- a/NEWS
+++ b/NEWS
@@ -45,6 +45,10 @@ GNU sed NEWS                                    -*- outline -*-
   sed --follow-symlinks now works again for stdin.
   [bug introduced in sed-4.2.2]

+  sed no longer elides invalid bytes in a substitution RHS.
+  Now, sed copies such bytes into the output, just as Perl does.
+  [bug introduced in sed-4.1 -- it was also present prior to 4.0.6]
+
 ** Feature removal

   The "L" command (format a paragraph like the fmt(1) command would)
diff --git a/sed/execute.c b/sed/execute.c
index 5bebfc2..93b5650 100644
--- a/sed/execute.c
+++ b/sed/execute.c
@@ -225,6 +225,7 @@ str_append_modified(struct line *to, const char *string, size_t length,
               return;
             }

+          str_append (to, string, 1);
           memset (&to->mbstate, 0, sizeof (from_stat));
           n = 1;
           string += n, length -= n;
diff --git a/testsuite/invalid-mb-seq-UMR.sh b/testsuite/invalid-mb-seq-UMR.sh
index f631e24..0c17b9a 100755
--- a/testsuite/invalid-mb-seq-UMR.sh
+++ b/testsuite/invalid-mb-seq-UMR.sh
@@ -24,9 +24,9 @@ require_valgrind_
 test "$LOCALE_JA" = none && skip_ found no Japanese EUC locale

 echo a > in || framework_failure_
-echo bC > exp || framework_failure_
+printf 'b\262C\n' > exp || framework_failure_
 LC_ALL=$LOCALE_JA valgrind --quiet --error-exitcode=1 \
-  sed -e 's/a/b\U\xb2c/' in > out 2> err || fail=1
+  sed 's/a/b\U\xb2c/' in > out 2> err || fail=1

 # Work around a bug in CentOS 5.10's valgrind
 # FIXME: remove in 2018 or when CentOS 5 is no longer officially supported
-- 
2.6.2