GNU bug report logs - #35993
Windows port redirection bug

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 35993 in the body.
You can then email your comments to 35993 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: nlweb <at> sapo.pt
To: bug-sed <at> gnu.org
Subject: Windows port redirection bug
Date: Wed, 29 May 2019 15:53:06 +0100

[Message part 1 (text/plain, inline)]

Hi,

The bug is reproducible with this command: SED 'S/"//G' TESTE.TXT > OUT.TXT

It should remove double quotes and save the result in the out.txt  
file. With a Bash shell it works as expected, but under Windows 10's  
command line it prints the resulting output and issues this error:
 
SED: CAN'T READ >: NO SUCH FILE OR DIRECTORY
SED: CAN'T READ OUT.TXT: NO SUCH FILE OR DIRECTORY

Escaping the double quote doesn't change the result, but if I use  
another character instead, like SED 'S/X//G' TESTE.TXT > OUT.TXT it  
works.

Sed is version 4.5 on both cases.
  sed (GNU sed) 4.5
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later  
<https://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by Jay Fenlason, Tom Lord, Ken Pizzini,
and Paolo Bonzini.
GNU sed home page: <https://www.gnu.org/software/sed/>.
General help using GNU software: <https://www.gnu.org/gethelp/>.
E-mail bug reports to: <bug-sed <at> gnu.org>.

Thanks!

Nuno
 

[Message part 2 (text/html, inline)]

Message #8 received at submit <at> debbugs.gnu.org (full text, mbox):

From: bill-auger <bill-auger <at> peers.community>
To: bug-sed <at> gnu.org
Subject: Re: bug#35993: Windows port redirection bug
Date: Wed, 29 May 2019 12:32:46 -0400

just a wild guess, but all those CAPITAL LETTERS look like the
classic DOS shell - have you tried that in the powershell or
the mingw shell?

Message #11 received at 35993 <at> debbugs.gnu.org (full text, mbox):

From: Assaf Gordon <assafgordon <at> gmail.com>
To: nlweb <at> sapo.pt
Cc: 35993 <at> debbugs.gnu.org
Subject: Re: bug#35993: Windows port redirection bug
Date: Sat, 1 Jun 2019 12:54:52 -0600

tags 35993 notabug
close 35993
stop

Hello,

On Wed, May 29, 2019 at 03:53:06PM +0100, nlweb <at> sapo.pt wrote:
> The bug is reproducible with this command: SED 'S/"//G' TESTE.TXT > OUT.TXT
>
> It should remove double quotes and save the result in the out.txt file. With
> a Bash shell it works as expected, but under Windows 10's command line it
> prints the resulting output and issues this error:
>  
> SED: CAN'T READ >: NO SUCH FILE OR DIRECTORY
> SED: CAN'T READ OUT.TXT: NO SUCH FILE OR DIRECTORY
>
> Escaping the double quote doesn't change the result, but if I use another
> character instead, like SED 'S/X//G' TESTE.TXT > OUT.TXT it works.
>

This is not a bug in sed - just incorrect usage of quotes in the Windows
command prompt (CMD.EXE).

Before going into (long) details, here's the solution:

    c:\Users\gordon\Desktop> type teste.txt
    hello"world

    c:\Users\gordon\Desktop> sed-4.7-64bit.exe "s/\"/XXX/g" teste.txt
    helloXXXworld

or even:

    c:\Users\gordon\Desktop> sed-4.7-64bit.exe s/\"/XXX/g teste.txt
    helloXXXworld


Now some details:

1.
Single-quotes have special meaning AT ALL in cmd.exe.
There's no point using them. In fact, they will just cause more
problems, as they are passed as-is to the sed program, and sed will
complain that a single-quote is not a recognizable sed command:

   c:\Users\gordon\Desktop> sed-4.7-64bit.exe '
   sed-4.7-64bit.exe: -e expression #1, char 1: unknown command: `''


2.
Double-quotes DO NOT behave like you expect if you are
familiar with unix-style shell quoting.

In unix-world, quotes (both single and double) act in pairs (opening
and closing), and we can speak about "text inside single/double quotes" and
"text outside quotes".

A side-effect of this approach is that strings with unbalanced quotes
result in parsing error. E.g. the following is not a complete/valid
command in unix:

     echo hello"world

3.
In windows' CMD.EXE, double-quote character (ONE character) changes an
internal parsing state which controls whether special characters are
important or ignored (surprising/unintuitive if you're coming from unix
world). a SPACE character is the most common example of special
characters.

For example, in CMD.EXE the following is valid command:

    echo foo > hello" world.txt

And it will create a file named HELLO<SPACE>WORLD<DOT>TXT .

The above string is parsed like so:
    1. 'hello' - as is
    2. double-quote - turns on "special character handling" state.
    3. space character - kept (not ignored) because of the new state.
    4. 'world.txt' - as is.

Another example, the following two commands are valid in CMD.EXE.
In the second command, once a double-quote character is encountered,
The PIPE character (loses its special meaning and is just consumed as
part of the string):

    c:\Users\gordon\Desktop\a>echo "hello world" | more
    "hello world"


    c:\Users\gordon\Desktop\a>echo "hello | more
    "hello | more


4.
For more strange cases, try the following:

    c:\Users\gordon\Desktop\c> echo foo > hello.txt
    c:\Users\gordon\Desktop\c> echo foo > "hello world.txt"
    c:\Users\gordon\Desktop\c> echo foo > hello" w o r l d.txt
    c:\Users\gordon\Desktop\c> echo foo > hello"   world.txt
    c:\Users\gordon\Desktop\c> echo foo > 'hello world.txt'

    c:\Users\gordon\Desktop\c> dir
     Volume in drive C is OS
     Volume Serial Number is 4CA3-CC48

     Directory of c:\Users\gordon\Desktop\c

    06/01/2019  12:50 PM    <DIR>          .
    06/01/2019  12:50 PM    <DIR>          ..
    06/01/2019  12:50 PM                17 'hello
    06/01/2019  12:50 PM                 6 hello   world.txt
    06/01/2019  12:50 PM                 6 hello w o r l d.txt
    06/01/2019  12:49 PM                 6 hello world.txt
    06/01/2019  12:49 PM                 6 hello.txt
                   5 File(s)             41 bytes
                   2 Dir(s)  384,140,804,096 bytes free

5.
To go even deeper into the nitty-gritty of CMD.EXE parsing and quoting,
see this interesting blog post:
http://www.windowsinspired.com/understanding-the-command-line-string-and-arguments-received-by-a-windows-program/


As such, I'm closing this as "not a bug", but discussion can continue
by replying to this thread.

regards,
 - assaf


P.S.
A newer version of SED (version 4.7) was released in December 2019,
and it contains few minor fixes/changes to behaviour on windows.
See here on how to build and/or download the binaries:

  https://lists.gnu.org/archive/html/sed-devel/2018-12/msg00031.html

Message #18 received at 35993 <at> debbugs.gnu.org (full text, mbox):

From: Nuno Leal <nlweb <at> sapo.pt>
To: Assaf Gordon <assafgordon <at> gmail.com>
Cc: 35993 <at> debbugs.gnu.org
Subject: Re: bug#35993: Windows port redirection bug
Date: Sat, 1 Jun 2019 20:08:34 +0100

Hi Assaf!

I mentioned that escaping the quote wouldn't work either. Now I can't 
test is, but I recall doing

   SED 'S/\"//G' TESTE.TXT > OUT.TXT

and getting the same error result.

Monday I'll do some more tests and get back to you with my findings, if they're relevant. I'm curious about the newer 4.7 version!

Thanks!

Nuno

On 01/06/19 19:54, Assaf Gordon wrote:
> tags 35993 notabug
> close 35993
> stop
>
> Hello,
>
> On Wed, May 29, 2019 at 03:53:06PM +0100, nlweb <at> sapo.pt wrote:
>> The bug is reproducible with this command: SED 'S/"//G' TESTE.TXT > OUT.TXT
>>
>> It should remove double quotes and save the result in the out.txt file. With
>> a Bash shell it works as expected, but under Windows 10's command line it
>> prints the resulting output and issues this error:
>>   
>> SED: CAN'T READ >: NO SUCH FILE OR DIRECTORY
>> SED: CAN'T READ OUT.TXT: NO SUCH FILE OR DIRECTORY
>>
>> Escaping the double quote doesn't change the result, but if I use another
>> character instead, like SED 'S/X//G' TESTE.TXT > OUT.TXT it works.
>>
> This is not a bug in sed - just incorrect usage of quotes in the Windows
> command prompt (CMD.EXE).
>
> Before going into (long) details, here's the solution:
>
>      c:\Users\gordon\Desktop> type teste.txt
>      hello"world
>
>      c:\Users\gordon\Desktop> sed-4.7-64bit.exe "s/\"/XXX/g" teste.txt
>      helloXXXworld
>
> or even:
>
>      c:\Users\gordon\Desktop> sed-4.7-64bit.exe s/\"/XXX/g teste.txt
>      helloXXXworld
>
>
> Now some details:
>
> 1.
> Single-quotes have special meaning AT ALL in cmd.exe.
> There's no point using them. In fact, they will just cause more
> problems, as they are passed as-is to the sed program, and sed will
> complain that a single-quote is not a recognizable sed command:
>
>     c:\Users\gordon\Desktop> sed-4.7-64bit.exe '
>     sed-4.7-64bit.exe: -e expression #1, char 1: unknown command: `''
>
>
> 2.
> Double-quotes DO NOT behave like you expect if you are
> familiar with unix-style shell quoting.
>
> In unix-world, quotes (both single and double) act in pairs (opening
> and closing), and we can speak about "text inside single/double quotes" and
> "text outside quotes".
>
> A side-effect of this approach is that strings with unbalanced quotes
> result in parsing error. E.g. the following is not a complete/valid
> command in unix:
>
>       echo hello"world
>
> 3.
> In windows' CMD.EXE, double-quote character (ONE character) changes an
> internal parsing state which controls whether special characters are
> important or ignored (surprising/unintuitive if you're coming from unix
> world). a SPACE character is the most common example of special
> characters.
>
> For example, in CMD.EXE the following is valid command:
>
>      echo foo > hello" world.txt
>
> And it will create a file named HELLO<SPACE>WORLD<DOT>TXT .
>
> The above string is parsed like so:
>      1. 'hello' - as is
>      2. double-quote - turns on "special character handling" state.
>      3. space character - kept (not ignored) because of the new state.
>      4. 'world.txt' - as is.
>
> Another example, the following two commands are valid in CMD.EXE.
> In the second command, once a double-quote character is encountered,
> The PIPE character (loses its special meaning and is just consumed as
> part of the string):
>
>      c:\Users\gordon\Desktop\a>echo "hello world" | more
>      "hello world"
>
>
>      c:\Users\gordon\Desktop\a>echo "hello | more
>      "hello | more
>
>
> 4.
> For more strange cases, try the following:
>
>      c:\Users\gordon\Desktop\c> echo foo > hello.txt
>      c:\Users\gordon\Desktop\c> echo foo > "hello world.txt"
>      c:\Users\gordon\Desktop\c> echo foo > hello" w o r l d.txt
>      c:\Users\gordon\Desktop\c> echo foo > hello"   world.txt
>      c:\Users\gordon\Desktop\c> echo foo > 'hello world.txt'
>
>      c:\Users\gordon\Desktop\c> dir
>       Volume in drive C is OS
>       Volume Serial Number is 4CA3-CC48
>
>       Directory of c:\Users\gordon\Desktop\c
>
>      06/01/2019  12:50 PM    <DIR>          .
>      06/01/2019  12:50 PM    <DIR>          ..
>      06/01/2019  12:50 PM                17 'hello
>      06/01/2019  12:50 PM                 6 hello   world.txt
>      06/01/2019  12:50 PM                 6 hello w o r l d.txt
>      06/01/2019  12:49 PM                 6 hello world.txt
>      06/01/2019  12:49 PM                 6 hello.txt
>                     5 File(s)             41 bytes
>                     2 Dir(s)  384,140,804,096 bytes free
>
> 5.
> To go even deeper into the nitty-gritty of CMD.EXE parsing and quoting,
> see this interesting blog post:
> http://www.windowsinspired.com/understanding-the-command-line-string-and-arguments-received-by-a-windows-program/
>
>
> As such, I'm closing this as "not a bug", but discussion can continue
> by replying to this thread.
>
> regards,
>   - assaf
>
>
> P.S.
> A newer version of SED (version 4.7) was released in December 2019,
> and it contains few minor fixes/changes to behaviour on windows.
> See here on how to build and/or download the binaries:
>
>    https://lists.gnu.org/archive/html/sed-devel/2018-12/msg00031.html
>

Message #21 received at 35993 <at> debbugs.gnu.org (full text, mbox):

From: nlweb <at> sapo.pt
To: Assaf Gordon <assafgordon <at> gmail.com>
Cc: 35993 <at> debbugs.gnu.org
Subject: Re: bug#35993: Windows port redirection bug
Date: Wed, 05 Jun 2019 16:27:46 +0100

[Message part 1 (text/plain, inline)]

Hi Assaf,

These are my findings on Windows 10:
 
S:\temp>type teste.txt
hello"world

S:\temp>SED 's/"//g' teste.txt > out.txt
SED: can't read >: No such file or directory
SED: can't read out.txt: No such file or directory
helloworld

S:\temp>SED s/\"//g teste.txt > out.txt
SED: -e expression #1, char 26: unterminated `s' command

S:\temp>SED "s/\"//g" teste.txt > out.txt
SED: can't read >: No such file or directory
SED: can't read out.txt: No such file or directory
helloworld

S:\temp>sed-4.7-64bit s/\"//g teste.txt > out.txt
helloworldsed-4.7-64bit: can't read >: Invalid argument
sed-4.7-64bit: can't read out.txt: No such file or directory

S:\temp>sed-4.7-64bit "s/\"//g" teste.txt > out.txt
helloworldsed-4.7-64bit: can't read >: Invalid argument
sed-4.7-64bit: can't read out.txt: No such file or directory

As you can see, the problem persists. My goal is to remove double  
quotes, so I replace them with nothing. This command line works inside  
a bash shell, but gives the above errors under Windows.

Thanks!

Nuno
Citando Assaf Gordon <assafgordon <at> gmail.com>:

> tags 35993 notabug
> close 35993
> stop
>
> Hello,
>
> On Wed, May 29, 2019 at 03:53:06PM +0100, nlweb <at> sapo.pt wrote:
>> The bug is reproducible with this command: SED 'S/"//G' TESTE.TXT > OUT.TXT
>>
>> It should remove double quotes and save the result in the out.txt file. With
>> a Bash shell it works as expected, but under Windows 10's command line it
>> prints the resulting output and issues this error:
>>  
>> SED: CAN'T READ >: NO SUCH FILE OR DIRECTORY
>> SED: CAN'T READ OUT.TXT: NO SUCH FILE OR DIRECTORY
>>
>> Escaping the double quote doesn't change the result, but if I use another
>> character instead, like SED 'S/X//G' TESTE.TXT > OUT.TXT it works.
>
> This is not a bug in sed - just incorrect usage of quotes in the Windows
> command prompt (CMD.EXE).
>
> Before going into (long) details, here's the solution:
>
>    c:\Users\gordon\Desktop> type teste.txt
>    hello"world
>
>    c:\Users\gordon\Desktop> sed-4.7-64bit.exe "s/\"/XXX/g" teste.txt
>    helloXXXworld
>
> or even:
>
>    c:\Users\gordon\Desktop> sed-4.7-64bit.exe s/\"/XXX/g teste.txt
>    helloXXXworld
>
> Now some details:
>
> 1.
> Single-quotes have special meaning AT ALL in cmd.exe.
> There's no point using them. In fact, they will just cause more
> problems, as they are passed as-is to the sed program, and sed will
> complain that a single-quote is not a recognizable sed command:
>
>   c:\Users\gordon\Desktop> sed-4.7-64bit.exe '
>   sed-4.7-64bit.exe: -e expression #1, char 1: unknown command: `''
>
> 2.
> Double-quotes DO NOT behave like you expect if you are
> familiar with unix-style shell quoting.
>
> In unix-world, quotes (both single and double) act in pairs (opening
> and closing), and we can speak about "text inside single/double quotes" and
> "text outside quotes".
>
> A side-effect of this approach is that strings with unbalanced quotes
> result in parsing error. E.g. the following is not a complete/valid
> command in unix:
>
>     echo hello"world
>
> 3.
> In windows' CMD.EXE, double-quote character (ONE character) changes an
> internal parsing state which controls whether special characters are
> important or ignored (surprising/unintuitive if you're coming from unix
> world). a SPACE character is the most common example of special
> characters.
>
> For example, in CMD.EXE the following is valid command:
>
>    echo foo > hello" world.txt
>
> And it will create a file named HELLO<SPACE>WORLD<DOT>TXT .
>
> The above string is parsed like so:
>    1. 'hello' - as is
>    2. double-quote - turns on "special character handling" state.
>    3. space character - kept (not ignored) because of the new state.
>    4. 'world.txt' - as is.
>
> Another example, the following two commands are valid in CMD.EXE.
> In the second command, once a double-quote character is encountered,
> The PIPE character (loses its special meaning and is just consumed as
> part of the string):
>
>    c:\Users\gordon\Desktop\a>echo "hello world" | more
>    "hello world"
>
>    c:\Users\gordon\Desktop\a>echo "hello | more
>    "hello | more
>
> 4.
> For more strange cases, try the following:
>
>    c:\Users\gordon\Desktop\c> echo foo > hello.txt
>    c:\Users\gordon\Desktop\c> echo foo > "hello world.txt"
>    c:\Users\gordon\Desktop\c> echo foo > hello" w o r l d.txt
>    c:\Users\gordon\Desktop\c> echo foo > hello"   world.txt
>    c:\Users\gordon\Desktop\c> echo foo > 'hello world.txt'
>
>    c:\Users\gordon\Desktop\c> dir
>     Volume in drive C is OS
>     Volume Serial Number is 4CA3-CC48
>
>     Directory of c:\Users\gordon\Desktop\c
>
>    06/01/2019  12:50 PM    <DIR>          .
>    06/01/2019  12:50 PM    <DIR>          ..
>    06/01/2019  12:50 PM                17 'hello
>    06/01/2019  12:50 PM                 6 hello   world.txt
>    06/01/2019  12:50 PM                 6 hello w o r l d.txt
>    06/01/2019  12:49 PM                 6 hello world.txt
>    06/01/2019  12:49 PM                 6 hello.txt
>                   5 File(s)             41 bytes
>                   2 Dir(s)  384,140,804,096 bytes free
>
> 5.
> To go even deeper into the nitty-gritty of CMD.EXE parsing and quoting,
> see this interesting blog post:
> http://www.windowsinspired.com/understanding-the-command-line-string-and-arguments-received-by-a-windows-program/
>
> As such, I'm closing this as "not a bug", but discussion can continue
> by replying to this thread.
>
> regards,
> - assaf
>
> P.S.
> A newer version of SED (version 4.7) was released in December 2019,
> and it contains few minor fixes/changes to behaviour on windows.
> See here on how to build and/or download the binaries:
> https://lists.gnu.org/archive/html/sed-devel/2018-12/msg00031.html

 

[Message part 2 (text/html, inline)]

Message #24 received at 35993 <at> debbugs.gnu.org (full text, mbox):

From: Assaf Gordon <assafgordon <at> gmail.com>
To: nlweb <at> sapo.pt
Cc: 35993 <at> debbugs.gnu.org
Subject: Re: bug#35993: Windows port redirection bug
Date: Thu, 6 Jun 2019 00:48:53 -0600

Hello,

It is worth repeating: none of these are bugs in sed itself
(that is - there is nothing to change/fix in sed to make it work).
These are problems in the shell usage (e.g. CMD.EXE in windows 10).

On Wed, Jun 05, 2019 at 04:27:46PM +0100, nlweb <at> sapo.pt wrote:
> These are my findings on Windows 10:

> S:\temp>sed-4.7-64bit s/\"//g teste.txt > out.txt
> helloworldsed-4.7-64bit: can't read >: Invalid argument
> sed-4.7-64bit: can't read out.txt: No such file or directory
> 
> S:\temp>sed-4.7-64bit "s/\"//g" teste.txt > out.txt
> helloworldsed-4.7-64bit: can't read >: Invalid argument
> sed-4.7-64bit: can't read out.txt: No such file or directory

I do not have access to a Windows 10 machine,
so my results are from Windows 7.

I just double-checked and the above two commands work well for me
in windows 7 with cmd.exe.

To verify: are you using the standard CMD.EXE or something else ?

> As you can see, the problem persists. My goal is to remove double quotes, so
> I replace them with nothing. This command line works inside a bash shell,
> but gives the above errors under Windows.

As you wrote, it works well (and as expected) in 'bash' shell.
Hence it is a problem in the usage of quoting in your terminal
(cmd.exe?), not in sed.
There is not much more we can do as sed developers.

You might want to ask in Microsoft forums about quoting rules in windows
10.

regards,
 - assaf

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.