Subject: bug#22254: closed (Re: bug#22254: [PATCH] sed: do not elide an invalid byte in a substitution RHS)

From: [email-masked] (GNU bug Tracking System)
Date: Tue, 29 Dec 2015 06:19:02 +0000

Your bug report

#22254: [PATCH] sed: do not elide an invalid byte in a substitution RHS

which was filed against the sed package, has been closed.

The explanation is attached below, along with your original report.
If you require more details, please reply to [email-masked].

-- 
22254: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=22254
GNU Bug Tracking System
Contact [email-masked] with problems

Subject: Re: bug#22254: [PATCH] sed: do not elide an invalid byte in a substitution RHS
From: Jim Meyering <[email-masked]>
To: [email-masked]
Date: Mon, 28 Dec 2015 22:18:22 -0800
List-Id: <debbugs-submit.debbugs.gnu.org>

On Sun, Dec 27, 2015 at 10:42 PM, Jim Meyering <[email-masked]> wrote:
> On Sun, Dec 27, 2015 at 10:24 PM, Jim Meyering <[email-masked]> wrote:
>> On Sun, Dec 27, 2015 at 10:02 PM, Jim Meyering <[email-masked]> wrote:
>>> Just after pushing the fix for that heap-clobbering bug,
>>> http://debbugs.gnu.org/22127, I noticed there was another minor
>>> problem: sed would silently elide any invalid byte specified in a
>>> substitution replacement string.
>>>
>>> Here's a patch that's not quite complete (no NEWS entry, and I haven't
>>> yet tried to determine when it was introduced).
>>
>> This one has a checkered past. I keep a decent sampling of old
>> versions of sed around.
>> Running one of the two tests that would be affected shows how
>> treatment of an invalid byte in a multi-byte locale has differed over
>> the years:
>>
>> $ for i in $(env ls -dv /p/p/sed-*); do printf '%-14s: ' $i; echo a
>> |LC_ALL=ja_JP $i/bin/sed 's/a/b\U\xb2c/'|od -tx1|head -1; done|sed
>> s/.....//
>> sed-1.18 : 0000000 62 55 78 62 32 63 0a
>> sed-2.05 : 0000000 62 55 78 62 32 63 0a
>> sed-3.01 : 0000000 62 55 78 62 32 63 0a
>> sed-3.02 : 0000000 62 55 78 62 32 63 0a
>> sed-4.0a : 0000000 62 00 43 0a
...
>> sed-4.0e : 0000000 62 00 43 0a
>> sed-4.0.6: 0000000 62 b2 43 0a
>> sed-4.0.7: 0000000 62 b2 43 0a
>> sed-4.0.8: 0000000 62 b2 43 0a
>> sed-4.0.9: 0000000 62 b2 43 0a
>> sed-4.1  : 0000000 62 00 43 0a
...
>> sed-4.2.2: 0000000 62 00 43 0a
>>
>> Now, it does what 4.0.6..4.0.9 did.
>
> Here's a complete patch:

Pushed.

Subject: [PATCH] sed: do not elide an invalid byte in a substitution RHS
From: Jim Meyering <[email-masked]>
To: [email-masked]
Date: Sun, 27 Dec 2015 22:02:24 -0800
List-Id: <debbugs-submit.debbugs.gnu.org>

Just after pushing the fix for that heap-clobbering bug,
http://debbugs.gnu.org/22127, I noticed there was another minor
problem: sed would silently elide any invalid byte specified in a
substitution replacement string.

Here's a patch that's not quite complete (no NEWS entry, and I haven't
yet tried to determine when it was introduced).
From 74f6081358bfd271f679b87600367975111c7436 Mon Sep 17 00:00:00 2001
From: Jim Meyering <[email-masked]>
Date: Sun, 27 Dec 2015 21:56:04 -0800
Subject: [PATCH] sed: do not elide an invalid byte in a substitution RHS

Now, sed copies such bytes into the output.
* sed/execute.c (str_append_modified): Copy each invalid byte
into the result string, as the comment suggests.
* testsuite/invalid-mb-seq-UMR.sh: Adjust this test to conform.
Perl does the same thing.  Also, remove an unnecessary "-e" option.
---
 sed/execute.c                   | 1 +
 testsuite/invalid-mb-seq-UMR.sh | 4 ++--
 2 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/sed/execute.c b/sed/execute.c
index 5bebfc2..93b5650 100644
--- a/sed/execute.c
+++ b/sed/execute.c
@@ -225,6 +225,7 @@ str_append_modified(struct line *to, const char *string, size_t length,
               return;
             }

+          str_append (to, string, 1);
           memset (&to->mbstate, 0, sizeof (from_stat));
           n = 1;
           string += n, length -= n;
diff --git a/testsuite/invalid-mb-seq-UMR.sh b/testsuite/invalid-mb-seq-UMR.sh
index f631e24..0c17b9a 100755
--- a/testsuite/invalid-mb-seq-UMR.sh
+++ b/testsuite/invalid-mb-seq-UMR.sh
@@ -24,9 +24,9 @@ require_valgrind_
 test "$LOCALE_JA" = none && skip_ found no Japanese EUC locale

 echo a > in || framework_failure_
-echo bC > exp || framework_failure_
+printf 'b\262C\n' > exp || framework_failure_
 LC_ALL=$LOCALE_JA valgrind --quiet --error-exitcode=1 \
-  sed -e 's/a/b\U\xb2c/' in > out 2> err || fail=1
+  sed 's/a/b\U\xb2c/' in > out 2> err || fail=1

 # Work around a bug in CentOS 5.10's valgrind
 # FIXME: remove in 2018 or when CentOS 5 is no longer officially supported
-- 
2.6.2