GNU bug report logs - #30251
REMOVE bllank line on DOS file format

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 30251 in the body.
You can then email your comments to 30251 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Mauro Rigoni <mauro.rigoni <at> gmail.com>
To: bug-sed <at> gnu.org
Subject: REMOVE bllank line on DOS file format
Date: Thu, 25 Jan 2018 13:25:35 +0000

[Message part 1 (text/plain, inline)]

Hi,

                with the last x86_32 version of sed (windows platform) the
following pattern



sed '\?^$?D'



don’t remove blank line from file if the file is in DOS format. In the
previous version 2.2 this command working.


In attach the demo test files.


*sed --version :*

*sed (GNU sed) 4.4*

*Packaged by Cygwin (4.4-1)*

*Copyright (C) 2017 Free Software Foundation, Inc.*

*License GPLv3+: GNU GPL version 3 or later
<http://gnu.org/licenses/gpl.html <http://gnu.org/licenses/gpl.html>>.*

*This is free software: you are free to change and redistribute it.*

*There is NO WARRANTY, to the extent permitted by law.*


*Written by Jay Fenlason, Tom Lord, Ken Pizzini,*

*and Paolo Bonzini.*

*GNU sed home page: <http://www.gnu.org/software/sed/
<http://www.gnu.org/software/sed/>>.*

*General help using GNU software: <http://www.gnu.org/gethelp/
<http://www.gnu.org/gethelp/>>.*

*E-mail bug reports to: <bug-sed <at> gnu.org <bug-sed <at> gnu.org>>.*
-- 

Saluti
Mauro Rigoni

[Message part 2 (text/html, inline)]

[test_DOS.txt (text/plain, attachment)]

[test_UNIX.txt (text/plain, attachment)]

Message #8 received at 30251 <at> debbugs.gnu.org (full text, mbox):

From: Assaf Gordon <assafgordon <at> gmail.com>
To: Mauro Rigoni <mauro.rigoni <at> gmail.com>
Cc: 30251 <at> debbugs.gnu.org
Subject: Re: bug#30251: REMOVE bllank line on DOS file format
Date: Thu, 25 Jan 2018 14:59:21 -0700

Hello,

On Thu, Jan 25, 2018 at 01:25:35PM +0000, Mauro Rigoni wrote:
> with the last x86_32 version of sed (windows platform) the
> following pattern
> 
> sed '\?^$?D'
> 
> don’t remove blank line from file if the file is in DOS format. In the
> previous version 2.2 this command working.

First,
Sed version 2.X is more than 20 years old. It is not the previous version.
The previous (and most common) version is version 4.2.2 from 2012.
Can you test your sed program with version 4.2.2 ?


Second,
A question about your sed program:
Does the problem still happen if you use the following?
   sed '/^$/d'

That is, regex with slashes (instead of '\?')
and using lower-case 'd' (simple delete command)
instead of 'D' (which is slightly more complicated
and deals with multilined buffers).


Third,
I do not have access to a windows machine, so a bit more information
is needed:
1. Which windows version are you using?
2. Which cygwin version are you using?
(I assume it is cygwin based on the version string)
3. Can you try the same sed command, but add the '--binary' flag,
   if does it show the same results?

For reference,
there is are two bug reports related to newlines with sed on windows.
https://bugs.gnu.org/25459 and https://bugs.gnu.org/26879 .

Specifically, note here https://debbugs.gnu.org/cgi/bugreport.cgi?bug=26879#12
where Eric Blake mentions that cygwin modified some internal
behaviour related to binary-vs-text files in February 2017.


regards,
 - assaf

Message #11 received at 30251 <at> debbugs.gnu.org (full text, mbox):

From: Eric Blake <eblake <at> redhat.com>
To: Mauro Rigoni <mauro.rigoni <at> gmail.com>, 30251 <at> debbugs.gnu.org
Subject: Re: bug#30251: REMOVE bllank line on DOS file format
Date: Thu, 25 Jan 2018 16:13:04 -0600

[Message part 1 (text/plain, inline)]

On 01/25/2018 07:25 AM, Mauro Rigoni wrote:

> *sed (GNU sed) 4.4*
> 
> *Packaged by Cygwin (4.4-1)*

Questions about the Cygwin port of sed are best directed to the Cygwin
list.  In particular, note that the Cygwin folks have decided that it is
better to have sed operate in binary mode by default (it used to operate
in text mode and silently corrupt binary data); so you HAVE to strip out
line endings yourself as a pre-processing step before handing the data
to sed (the same as you would have to do when processing DOS files on
Linux, where text and binary files have synonymous behaviors).  More
information here:

https://cygwin.com/ml/cygwin-announce/2017-02/msg00036.html

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org

[signature.asc (application/pgp-signature, attachment)]

Message #14 received at 30251 <at> debbugs.gnu.org (full text, mbox):

From: Mauro Rigoni <mauro.rigoni <at> gmail.com>
To: Assaf Gordon <assafgordon <at> gmail.com>
Cc: 30251 <at> debbugs.gnu.org
Subject: Re: bug#30251: REMOVE bllank line on DOS file format
Date: Fri, 26 Jan 2018 12:33:20 +0000

[Message part 1 (text/plain, inline)]

Hello,
  we use CYGWIN in our application, and there are a lot of shell-script
that use this sintax. In Windows environment, is normat that the End-Line
terminator is CRLF e not LF.

Regarding your question:

1) Can you test your sed program with version 4.2.2 ?
    R: Yes, same problems

2) Does the problem still happen if you use the following?   sed '/^$/d'
    R:: same problems

3.1. Which windows version are you using?
       Tested on Windows server 2012, Windows Server 2016, Windows 7 (all
64 bit)

3.2. Which cygwin version are you using?
       uname -a ->  CYGWIN_NT-10.0-WOW ITP-WDK2020 2.9.0(0.318/5/3)
2017-09-12 10:41 i686 Cygwin

3. Can you try the same sed command, but add the '--binary' flag,   if does
it show the same results?
     R:: same problems


Regards

Mauro

Il giorno gio 25 gen 2018 alle ore 22:59 Assaf Gordon <assafgordon <at> gmail.com>
ha scritto:

> Hello,
>
> On Thu, Jan 25, 2018 at 01:25:35PM +0000, Mauro Rigoni wrote:
> > with the last x86_32 version of sed (windows platform) the
> > following pattern
> >
> > sed '\?^$?D'
> >
> > don’t remove blank line from file if the file is in DOS format. In the
> > previous version 2.2 this command working.
>
> First,
> Sed version 2.X is more than 20 years old. It is not the previous version.
> The previous (and most common) version is version 4.2.2 from 2012.
> Can you test your sed program with version 4.2.2 ?
>
>
> Second,
> A question about your sed program:
> Does the problem still happen if you use the following?
>    sed '/^$/d'
>
> That is, regex with slashes (instead of '\?')
> and using lower-case 'd' (simple delete command)
> instead of 'D' (which is slightly more complicated
> and deals with multilined buffers).
>
>
> Third,
> I do not have access to a windows machine, so a bit more information
> is needed:
> 1. Which windows version are you using?
> 2. Which cygwin version are you using?
> (I assume it is cygwin based on the version string)
> 3. Can you try the same sed command, but add the '--binary' flag,
>    if does it show the same results?
>
> For reference,
> there is are two bug reports related to newlines with sed on windows.
> https://bugs.gnu.org/25459 and https://bugs.gnu.org/26879 .
>
> Specifically, note here
> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=26879#12
> where Eric Blake mentions that cygwin modified some internal
> behaviour related to binary-vs-text files in February 2017.
>
>
> regards,
>  - assaf
>
-- 

Saluti
Mauro Rigoni

[Message part 2 (text/html, inline)]

Message #17 received at 30251 <at> debbugs.gnu.org (full text, mbox):

From: Assaf Gordon <assafgordon <at> gmail.com>
To: Mauro Rigoni <mauro.rigoni <at> gmail.com>
Cc: 30251 <at> debbugs.gnu.org
Subject: Re: bug#30251: REMOVE bllank line on DOS file format
Date: Sat, 27 Jan 2018 22:42:36 -0700

tag 30251 notabug
close 30251
stop

Hello,

On 2018-01-26 05:33 AM, Mauro Rigoni wrote:
>    we use CYGWIN in our application, and there are a lot of shell-script 
> that use this sintax. In Windows environment, is normat that the 
> End-Line terminator is CRLF e not LF.

In Windows, yes.
But as the previous emails in this thread mentioned, not in unix,
and Cygwin recently changed its behavior to match unix more closely.

> Regarding your question:
> 
> 1) Can you test your sed program with version 4.2.2 ?
>      R: Yes, same problems
> 
[...]
> 3.2. Which cygwin version are you using?
>         uname -a ->  CYGWIN_NT-10.0-WOW ITP-WDK2020 2.9.0(0.318/5/3) 
> 2017-09-12 10:41 i686 Cygwin

Based on this information, it is likely that the change in behavior
you are experiencing is due to a recent Cygwin version (or the way
Cygwin developers build 'sed') and not due to a bug in sed itself.

Cygwin version 2.9.0 was released in September 2017,
and it already included the change in CR/LF handling.

You can try several things:
1. Build sed from sources, and enable the 'binary' handling by default
(the same option that is now disabled by default on cygwin).

2. Try to downgrade to an older cygwin version (likely time-consuming, 
and not recommended).

3. Accept the new behavior, and add an additional step
of removing the CR characters using 'd2u' as suggested by Eric Blake.


As such, I'm marking this as not-a-bug and closing the ticket,
but discussion can continue by replying to this thread.

regards,
 - assaf

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.