GNU bug report logs - #14752
sort fails to fork() + execlp(compress_program) if overcommit limit is reached

Previous Next

Full log

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Petros Aggelatos <petrosagg <at> gmail.com>
To: bug-coreutils <at> gnu.org
Subject: sort fails to fork() + execlp(compress_program) if overcommit limit
 is reached
Date: Sun, 30 Jun 2013 04:42:34 +0300

I was trying to sort a big file (22GB, 5GB gzipped) with `sort
--compress-program=gzip -S40% data`. My /tmp filesystem is a 6GB tmpfs
and the total system RAM is 16GB. The problem was that after a while
sort would write uncompressed temp files in /tmp causing it to fill up
and then crash for having no free space.

After messing around with sort.c I found out that fork() would fail
after the first batches were written successfully in /tmp with errno
ENOMEM. I found out that I was hitting my kernel's VM overcommit limit
as the tmpfs utilisation grew. And indeed, running `sysctl -w
vm.overcommit=1` fixed the problem and the file was sorted.

As sort is a program with high memory consumption and could be running
in environments where overcommit is either unavailable or disabled I
would expect it to use another method for spawning its
compress_program. I'm not a C developer, so I'm not sure this is
possible, but vfork() and clone() that seem to solve this problem.

--
Petros Angelatos

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.