Subject: Re: bug#35993: Windows port redirection bug

From: Nuno Leal <[email-masked]>
Date: Sat, 1 Jun 2019 20:08:34 +0100

Hi Assaf!

I mentioned that escaping the quote wouldn't work either. Now I can't 
test is, but I recall doing

    SED 'S/\"//G' TESTE.TXT > OUT.TXT

and getting the same error result.

Monday I'll do some more tests and get back to you with my findings, if they're relevant. I'm curious about the newer 4.7 version!

Thanks!

Nuno

On 01/06/19 19:54, Assaf Gordon wrote:
> tags 35993 notabug
> close 35993
> stop
>
> Hello,
>
> On Wed, May 29, 2019 at 03:53:06PM +0100, [email-masked] wrote:
>> The bug is reproducible with this command: SED 'S/"//G' TESTE.TXT > OUT.TXT
>>
>> It should remove double quotes and save the result in the out.txt file. With
>> a Bash shell it works as expected, but under Windows 10's command line it
>> prints the resulting output and issues this error:
>>   
>> SED: CAN'T READ >: NO SUCH FILE OR DIRECTORY
>> SED: CAN'T READ OUT.TXT: NO SUCH FILE OR DIRECTORY
>>
>> Escaping the double quote doesn't change the result, but if I use another
>> character instead, like SED 'S/X//G' TESTE.TXT > OUT.TXT it works.
>>
> This is not a bug in sed - just incorrect usage of quotes in the Windows
> command prompt (CMD.EXE).
>
> Before going into (long) details, here's the solution:
>
>      c:\Users\gordon\Desktop> type teste.txt
>      hello"world
>
>      c:\Users\gordon\Desktop> sed-4.7-64bit.exe "s/\"/XXX/g" teste.txt
>      helloXXXworld
>
> or even:
>
>      c:\Users\gordon\Desktop> sed-4.7-64bit.exe s/\"/XXX/g teste.txt
>      helloXXXworld
>
>
> Now some details:
>
> 1.
> Single-quotes have special meaning AT ALL in cmd.exe.
> There's no point using them. In fact, they will just cause more
> problems, as they are passed as-is to the sed program, and sed will
> complain that a single-quote is not a recognizable sed command:
>
>     c:\Users\gordon\Desktop> sed-4.7-64bit.exe '
>     sed-4.7-64bit.exe: -e expression #1, char 1: unknown command: `''
>
>
> 2.
> Double-quotes DO NOT behave like you expect if you are
> familiar with unix-style shell quoting.
>
> In unix-world, quotes (both single and double) act in pairs (opening
> and closing), and we can speak about "text inside single/double quotes" and
> "text outside quotes".
>
> A side-effect of this approach is that strings with unbalanced quotes
> result in parsing error. E.g. the following is not a complete/valid
> command in unix:
>
>       echo hello"world
>
> 3.
> In windows' CMD.EXE, double-quote character (ONE character) changes an
> internal parsing state which controls whether special characters are
> important or ignored (surprising/unintuitive if you're coming from unix
> world). a SPACE character is the most common example of special
> characters.
>
> For example, in CMD.EXE the following is valid command:
>
>      echo foo > hello" world.txt
>
> And it will create a file named HELLO<SPACE>WORLD<DOT>TXT .
>
> The above string is parsed like so:
>      1. 'hello' - as is
>      2. double-quote - turns on "special character handling" state.
>      3. space character - kept (not ignored) because of the new state.
>      4. 'world.txt' - as is.
>
> Another example, the following two commands are valid in CMD.EXE.
> In the second command, once a double-quote character is encountered,
> The PIPE character (loses its special meaning and is just consumed as
> part of the string):
>
>      c:\Users\gordon\Desktop\a>echo "hello world" | more
>      "hello world"
>
>
>      c:\Users\gordon\Desktop\a>echo "hello | more
>      "hello | more
>
>
> 4.
> For more strange cases, try the following:
>
>      c:\Users\gordon\Desktop\c> echo foo > hello.txt
>      c:\Users\gordon\Desktop\c> echo foo > "hello world.txt"
>      c:\Users\gordon\Desktop\c> echo foo > hello" w o r l d.txt
>      c:\Users\gordon\Desktop\c> echo foo > hello"   world.txt
>      c:\Users\gordon\Desktop\c> echo foo > 'hello world.txt'
>
>      c:\Users\gordon\Desktop\c> dir
>       Volume in drive C is OS
>       Volume Serial Number is 4CA3-CC48
>
>       Directory of c:\Users\gordon\Desktop\c
>
>      06/01/2019  12:50 PM    <DIR>          .
>      06/01/2019  12:50 PM    <DIR>          ..
>      06/01/2019  12:50 PM                17 'hello
>      06/01/2019  12:50 PM                 6 hello   world.txt
>      06/01/2019  12:50 PM                 6 hello w o r l d.txt
>      06/01/2019  12:49 PM                 6 hello world.txt
>      06/01/2019  12:49 PM                 6 hello.txt
>                     5 File(s)             41 bytes
>                     2 Dir(s)  384,140,804,096 bytes free
>
> 5.
> To go even deeper into the nitty-gritty of CMD.EXE parsing and quoting,
> see this interesting blog post:
> http://www.windowsinspired.com/understanding-the-command-line-string-and-arguments-received-by-a-windows-program/
>
>
> As such, I'm closing this as "not a bug", but discussion can continue
> by replying to this thread.
>
> regards,
>   - assaf
>
>
> P.S.
> A newer version of SED (version 4.7) was released in December 2019,
> and it contains few minor fixes/changes to behaviour on windows.
> See here on how to build and/or download the binaries:
>
>    https://lists.gnu.org/archive/html/sed-devel/2018-12/msg00031.html
>