GNU bug report logs - #30794
Cannot produce backslash through hexadecimal - sed (GNU sed) 4.2.2

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 30794 in the body.
You can then email your comments to 30794 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: 0xddaa <0xddaa <at> gmail.com>
To: bug-sed <at> gnu.org
Subject: Cannot produce backslash through hexadecimal - sed (GNU sed) 4.2.2
Date: Tue, 13 Mar 2018 18:02:05 +0800

[Message part 1 (text/plain, inline)]

Hi,

I use sed to modify the field of size in file header for some binaries, and
I found it will failed when the file size included *\x5c* (backslash) after
converted into hexadecimal.

There is the Proof-of-Concept:

# echo z | sed 's/z/\x5c/' # \x5c == '\'
\ # success
# echo z | sed 's/z/\x5c\x61/'
a # backslash is gone

According the manual, *\xxx* should produce an ascii character. sed should
not treat *\x5c* as an escape symbol.

I also try the replacement with '&' and it's work fine:

# echo z | sed 's/z/\x26\x61/' # \x26 == '&'
&a // replace to &
# echo z | sed 's/z/&\x61/'
za  // execute & feature

I try to fix the problem and I hope it will be useful. Sorry for my bad
English.
https://github.com/0xddaa/sed/commit/723fa224c771ecd16913aea8117aa917877bbd38

Sincerely,
ddaa

[Message part 2 (text/html, inline)]

Message #8 received at 30794 <at> debbugs.gnu.org (full text, mbox):

From: Assaf Gordon <assafgordon <at> gmail.com>
To: 0xddaa <0xddaa <at> gmail.com>
Cc: 30794 <at> debbugs.gnu.org
Subject: Re: bug#30794: Cannot produce backslash through hexadecimal - sed
 (GNU sed) 4.2.2
Date: Tue, 13 Mar 2018 20:45:32 -0600

[Message part 1 (text/plain, inline)]

Hello,

On Tue, Mar 13, 2018 at 06:02:05PM +0800, 0xddaa wrote:
> There is the Proof-of-Concept:
> 
> # echo z | sed 's/z/\x5c/' # \x5c == '\'
> \ # success
> # echo z | sed 's/z/\x5c\x61/'
> a # backslash is gone
> 
> According the manual, *\xxx* should produce an ascii character. sed should
> not treat *\x5c* as an escape symbol.

Thank you for reporting this issue with such clear details
and providing a bug fix - much appreciated!

I expanded your fix by adding tests and news item, see attached patch.

There is a small issue that this is a backwards incompatible change,
and might break some exotic (bad) scripts.

I'll wait for other opinions before pushing this.

Jim,
What do you think?

regards,
 - assaf

[0001-sed-treat-x5c-as-literal-backslash.patch (text/x-diff, attachment)]

Message #11 received at 30794 <at> debbugs.gnu.org (full text, mbox):

From: 0xddaa <0xddaa <at> gmail.com>
To: Assaf Gordon <assafgordon <at> gmail.com>
Cc: 0xddaa <0xddaa <at> gmail.com>, 30794 <at> debbugs.gnu.org
Subject: Re: bug#30794: Cannot produce backslash through hexadecimal - sed
 (GNU sed) 4.2.2
Date: Wed, 14 Mar 2018 11:50:31 +0800

[Message part 1 (text/plain, inline)]

I am glad that my report is helpful.
Thanks for your praise. :)

2018-03-14 10:45 GMT+08:00 Assaf Gordon <assafgordon <at> gmail.com>:

> Hello,
>
> On Tue, Mar 13, 2018 at 06:02:05PM +0800, 0xddaa wrote:
> > There is the Proof-of-Concept:
> >
> > # echo z | sed 's/z/\x5c/' # \x5c == '\'
> > \ # success
> > # echo z | sed 's/z/\x5c\x61/'
> > a # backslash is gone
> >
> > According the manual, *\xxx* should produce an ascii character. sed
> should
> > not treat *\x5c* as an escape symbol.
>
> Thank you for reporting this issue with such clear details
> and providing a bug fix - much appreciated!
>
> I expanded your fix by adding tests and news item, see attached patch.
>
> There is a small issue that this is a backwards incompatible change,
> and might break some exotic (bad) scripts.
>
> I'll wait for other opinions before pushing this.
>
> Jim,
> What do you think?
>
> regards,
>  - assaf
>

[Message part 2 (text/html, inline)]

Message #14 received at 30794 <at> debbugs.gnu.org (full text, mbox):

From: Jim Meyering <jim <at> meyering.net>
To: Assaf Gordon <assafgordon <at> gmail.com>
Cc: 0xddaa <0xddaa <at> gmail.com>, 30794 <at> debbugs.gnu.org
Subject: Re: bug#30794: Cannot produce backslash through hexadecimal - sed
 (GNU sed) 4.2.2
Date: Tue, 13 Mar 2018 21:45:55 -0700

On Tue, Mar 13, 2018 at 7:45 PM, Assaf Gordon <assafgordon <at> gmail.com> wrote:
...
> Thank you for reporting this issue with such clear details
> and providing a bug fix - much appreciated!
>
> I expanded your fix by adding tests and news item, see attached patch.
>
> There is a small issue that this is a backwards incompatible change,
> and might break some exotic (bad) scripts.
>
> I'll wait for other opinions before pushing this.
>
> Jim,
> What do you think?

Thanks for the fine work from both of you.
Assaf, I think it's fair to extend the bound in the NEWS to say this:

  [Bug present since the beginning]

To conclude that, I ran the following on a modern system for which I
have built many old sed binaries.
Note that I dropped the "-E" and instead use a BRE, since -E and -r
were not present in the earlier releases.

$ for i in $(env ls -dv /p/p/sed-*); do printf '%-14s: ' $i; echo z |
$i/bin/sed 's/\(z\)/\x5c1/'; done|sed s/.....//    s:
sed-1.18 : x5c1
sed-2.05 : x5c1
sed-3.01 : x5c1
sed-3.02 : x5c1
sed-4.0a : z
sed-4.0b : z
sed-4.0c : z
sed-4.0d : z
sed-4.0e : z
sed-4.0.6: z
sed-4.0.7: z
sed-4.0.8: z
sed-4.0.9: z
sed-4.1  : z
sed-4.1c : z
sed-4.1d : z
sed-4.1e : z
sed-4.1f : z
sed-4.1.1: z
sed-4.1.2: z
sed-4.1.3: z
sed-4.1.4: z
sed-4.1.5: z
sed-4.2  : z
sed-4.2.1: z
sed-4.2.2: z
sed-4.3  : z
sed-4.4  : z

Message #17 received at 30794 <at> debbugs.gnu.org (full text, mbox):

From: Assaf Gordon <assafgordon <at> gmail.com>
To: Jim Meyering <jim <at> meyering.net>
Cc: 0xddaa <0xddaa <at> gmail.com>, 30794 <at> debbugs.gnu.org
Subject: Re: bug#30794: Cannot produce backslash through hexadecimal - sed
 (GNU sed) 4.2.2
Date: Wed, 14 Mar 2018 16:34:18 -0600

Hello Jim,

On Tue, Mar 13, 2018 at 09:45:55PM -0700, Jim Meyering wrote:
> On Tue, Mar 13, 2018 at 7:45 PM, Assaf Gordon <assafgordon <at> gmail.com> wrote:
> ...
> > I expanded your fix by adding tests and news item, see attached patch.
> >
> Thanks for the fine work from both of you.
> Assaf, I think it's fair to extend the bound in the NEWS to say this:
> 
>   [Bug present since the beginning]
> 
> To conclude that, I ran the following on a modern system for which I
> have built many old sed binaries.
> [...]
> $ for i in $(env ls -dv /p/p/sed-*); do printf '%-14s: ' $i; echo z |
> $i/bin/sed 's/\(z\)/\x5c1/'; done|sed s/.....//    s:
> sed-1.18 : x5c1
> sed-2.05 : x5c1
> sed-3.01 : x5c1
> sed-3.02 : x5c1
> [...]

Sorry for bike-shedding,
but wouldn't you say that if "\x5c1" is printed it means that older seds
do not understand hex characters, and simply treated "\x" as "x" ?

-assaf

Message #20 received at 30794 <at> debbugs.gnu.org (full text, mbox):

From: Assaf Gordon <assafgordon <at> gmail.com>
To: Jim Meyering <jim <at> meyering.net>
Cc: 0xddaa <0xddaa <at> gmail.com>, 30794 <at> debbugs.gnu.org
Subject: Re: bug#30794: Cannot produce backslash through hexadecimal - sed
 (GNU sed) 4.2.2
Date: Wed, 14 Mar 2018 16:43:11 -0600

[Message part 1 (text/plain, inline)]

Replying to myself:

On Wed, Mar 14, 2018 at 04:34:18PM -0600, Assaf Gordon wrote:
> > [...]
> > sed-1.18 : x5c1
> > sed-2.05 : x5c1
> > sed-3.01 : x5c1
> > sed-3.02 : x5c1
> > [...]
 
I checked the NEWS file,
and found that backslash sequences (e.g. \n \t \xNN \dNNN \oNNN)
were added in version 3.02.80 (which I don't think is common and
we don't have the source code).

I changed the NEWS accordingly (attached updated patch).

OK to push?

thanks,
 - assaf

[0001-sed-treat-x5c-as-literal-backslash.patch (text/x-diff, attachment)]

Message #25 received at 30794-done <at> debbugs.gnu.org (full text, mbox):

From: Jim Meyering <jim <at> meyering.net>
To: Assaf Gordon <assafgordon <at> gmail.com>
Cc: 0xddaa <0xddaa <at> gmail.com>, 30794-done <at> debbugs.gnu.org
Subject: Re: bug#30794: Cannot produce backslash through hexadecimal - sed
 (GNU sed) 4.2.2
Date: Wed, 14 Mar 2018 17:18:43 -0700

On Wed, Mar 14, 2018 at 3:43 PM, Assaf Gordon <assafgordon <at> gmail.com> wrote:
> Replying to myself:
>
> On Wed, Mar 14, 2018 at 04:34:18PM -0600, Assaf Gordon wrote:
>> > [...]
>> > sed-1.18 : x5c1
>> > sed-2.05 : x5c1
>> > sed-3.01 : x5c1
>> > sed-3.02 : x5c1
>> > [...]
>
> I checked the NEWS file,
> and found that backslash sequences (e.g. \n \t \xNN \dNNN \oNNN)
> were added in version 3.02.80 (which I don't think is common and
> we don't have the source code).
>
> I changed the NEWS accordingly (attached updated patch).
>
> OK to push?

Sure. Re the "bug present since..." issue, my intent was to convey
that \x5c never worked the way we now consider correct.

Message #28 received at 30794-done <at> debbugs.gnu.org (full text, mbox):

From: Assaf Gordon <assafgordon <at> gmail.com>
To: Jim Meyering <jim <at> meyering.net>
Cc: 0xddaa <0xddaa <at> gmail.com>, 30794-done <at> debbugs.gnu.org
Subject: Re: bug#30794: Cannot produce backslash through hexadecimal - sed
 (GNU sed) 4.2.2
Date: Wed, 14 Mar 2018 22:20:46 -0600

On Wed, Mar 14, 2018 at 05:18:43PM -0700, Jim Meyering wrote:
> On Wed, Mar 14, 2018 at 3:43 PM, Assaf Gordon <assafgordon <at> gmail.com> wrote:
> >
> > OK to push?
> 
> Sure. Re the "bug present since..." issue, my intent was to convey
> that \x5c never worked the way we now consider correct.

Thanks.

pushed here:
https://git.savannah.gnu.org/cgit/sed.git/commit/?id=b5f5236a4b3a2d6c2f89fb99d614486a65a40a24

regards,
 - assaf

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.