GNU bug report logs - #32228
sed -i does not buffer, causing excessive writes

Previous Next

Full log

Message #11 received at 32228 <at> debbugs.gnu.org (full text, mbox):

From: Vidar Holen <vidar <at> vidarholen.net>
To: Assaf Gordon <assafgordon <at> gmail.com>
Cc: 32228 <at> debbugs.gnu.org
Subject: Re: bug#32228: sed -i does not buffer, causing excessive writes
Date: Fri, 20 Jul 2018 19:26:42 -0700

[Message part 1 (text/plain, inline)]

Thanks for looking into it. I can confirm that this patch significantly
improves the performance of `sed -i` for me.

On Fri, Jul 20, 2018 at 6:48 PM, Assaf Gordon <assafgordon <at> gmail.com> wrote:

> Hello,
>
> On 20/07/18 02:23 PM, Vidar Holen wrote:
>
>> I'm using noticing that `sed -i` does not do any kind of output
>> buffering. While behavior is correct, this causes an excessive number of
>> small writes:
>>
>
> I'm seeing this on Debian with the latest sed commit (c52a676) and libc
>> 2.27-2. The root cause appears to be `ck_mkstemp` using `fdopen`, which
>> unlike `fopen` does not buffer by default.
>>
>
> Thank you for reporting this issue, and providing clear way to reproduce
> it. I can confirm seeing the same behavior on Debian 9.4 with glibc 2.24
> (and latest sed).
>
> I think the technical culprit is slightly different:
> It is not due to 'ck_mkstemp/fdopen', but because sed explicitly flushes
> the output after every line, except if the output is STDOUT.
>
> This happens in execute.c:flush_output():
> https://opengrok.housegordon.com/source/xref/sed/sed/execute
> .c#flush_output
>
> Changing this function enables default buffering with "sed -i"
> (using whatever the buffering default is for glibc).
>
> This change does have a side-effect:
> It also changes the buffering of other write commands such as 'w', 'W'
> and 's///w'.
> It might have other side effects I'm haven't spotted.
> Based on the ChangeLog-2014 file, I see this function was added in 2003.
>
> I wonder if there are good reasons to explicitly flush all output -
> ideas anyone?
>
> If you can give this a quick test and let me know if you also notice
> improvements on your system - would be appreciated.
>
>
> Comments and suggestions welcomed,
>  - assaf
>
>
>
>
>

[Message part 2 (text/html, inline)]

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.