GNU bug report logs - #32228
sed -i does not buffer, causing excessive writes

Previous Next

Full log

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Vidar Holen <vidar <at> vidarholen.net>
To: bug-sed <at> gnu.org
Subject: sed -i does not buffer, causing excessive writes
Date: Fri, 20 Jul 2018 13:23:35 -0700

Hi,

I'm using noticing that `sed -i` does not do any kind of output
buffering. While behavior is correct, this causes an excessive number of
small writes:

    $ strace -e write ./sed -i 's/foo/bar/g' file.txt
    write(4, "                    GNU GENERAL "..., 47) = 47
    write(4, "                       Version 3"..., 47) = 47
    write(4, "\n", 1)                       = 1

Compare this to redirection:

    $ strace -e write ./sed 's/foo/bar/g' file.txt > tmpfile
    write(1, "                    GNU GENERAL "..., 4096) = 4096
    write(1, "om or adapt all or part of the w"..., 4096) = 4096
    write(1, ".\n\n  You may make, run and propa"..., 4096) = 4096

On my system, this makes `sed -i` take 7x longer than redirection+mv:
3.7s vs 0.5s for 100MB, for 675 vs 10 writes on the same input.

I'm seeing this on Debian with the latest sed commit (c52a676) and libc
2.27-2. The root cause appears to be `ck_mkstemp` using `fdopen`, which
unlike `fopen` does not buffer by default.

Enabling buffering should be simple and effective, resulting in a nice
speedup.

Regards,
Vidar Holen

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.