From unknown Sat Aug 09 15:52:20 2025 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Mailer: MIME-tools 5.509 (Entity 5.509) Content-Type: text/plain; charset=utf-8 From: bug#14752 <14752@debbugs.gnu.org> To: bug#14752 <14752@debbugs.gnu.org> Subject: Status: sort fails to fork() + execlp(compress_program) if overcommit limit is reached Reply-To: bug#14752 <14752@debbugs.gnu.org> Date: Sat, 09 Aug 2025 22:52:20 +0000 retitle 14752 sort fails to fork() + execlp(compress_program) if overcommit= limit is reached reassign 14752 coreutils submitter 14752 Petros Aggelatos severity 14752 normal thanks From debbugs-submit-bounces@debbugs.gnu.org Sun Jun 30 01:22:26 2013 Received: (at submit) by debbugs.gnu.org; 30 Jun 2013 05:22:26 +0000 Received: from localhost ([127.0.0.1]:48809 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1UtA5h-00034m-LO for submit@debbugs.gnu.org; Sun, 30 Jun 2013 01:22:26 -0400 Received: from eggs.gnu.org ([208.118.235.92]:59861) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1Ut6fU-0004ku-5p for submit@debbugs.gnu.org; Sat, 29 Jun 2013 21:43:08 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Ut6fM-0006by-5v for submit@debbugs.gnu.org; Sat, 29 Jun 2013 21:43:02 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=0.8 required=5.0 tests=BAYES_50,FREEMAIL_FROM, T_DKIM_INVALID autolearn=disabled version=3.3.2 Received: from lists.gnu.org ([2001:4830:134:3::11]:35807) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Ut6fM-0006bs-2x for submit@debbugs.gnu.org; Sat, 29 Jun 2013 21:43:00 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:33339) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Ut6fJ-0003Hj-Cn for bug-coreutils@gnu.org; Sat, 29 Jun 2013 21:43:00 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Ut6fG-0006Xr-Oc for bug-coreutils@gnu.org; Sat, 29 Jun 2013 21:42:57 -0400 Received: from mail-qc0-x22b.google.com ([2607:f8b0:400d:c01::22b]:37125) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Ut6fG-0006Xj-Jc for bug-coreutils@gnu.org; Sat, 29 Jun 2013 21:42:54 -0400 Received: by mail-qc0-f171.google.com with SMTP id n1so2165281qcw.16 for ; Sat, 29 Jun 2013 18:42:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:from:date:message-id:subject:to:content-type; bh=eHt5NHavvnAkYi48l4RpY9ACRd4dQ3ZQ6vVePPe5DG8=; b=F8AhkHYyhYzoPMKwDFGFCHKZbGvzKUg5Mf0MRHyUxrqUtuXeY21+CmO7320EinQH+c cApyday1uLWws2rX7ZL83doNiKPPefChaxtjAC2yjck66jMQHNNYH1GZkBEg1QsjWzp1 zOUEWqqjAJTDAiJzT+lq6IFhSD9xEX7Kj7HZccfqmnpBs6CnX7O6PX8KNY8rwl/2mxEq 4J8RauMxr6MF6oCpwLwZJk/jHosyXUl5XUPaGB0nRXSKXZLh8mZKW74+ns89VG/Kwewn K6tpdT9oZuVTiqwn41VKL2FUmlXmR0V2HWyE1HtLHCdmhUJAypf8H7Qgpq7Ks2hMgBOu v05Q== X-Received: by 10.49.84.164 with SMTP id a4mr25214478qez.4.1372556574097; Sat, 29 Jun 2013 18:42:54 -0700 (PDT) MIME-Version: 1.0 Received: by 10.229.205.133 with HTTP; Sat, 29 Jun 2013 18:42:34 -0700 (PDT) From: Petros Aggelatos Date: Sun, 30 Jun 2013 04:42:34 +0300 Message-ID: Subject: sort fails to fork() + execlp(compress_program) if overcommit limit is reached To: bug-coreutils@gnu.org Content-Type: text/plain; charset=UTF-8 X-detected-operating-system: by eggs.gnu.org: Error: Malformed IPv6 address (bad octet value). X-detected-operating-system: by eggs.gnu.org: Error: Malformed IPv6 address (bad octet value). X-Received-From: 2001:4830:134:3::11 X-Spam-Score: -4.3 (----) X-Debbugs-Envelope-To: submit X-Mailman-Approved-At: Sun, 30 Jun 2013 01:22:24 -0400 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -4.3 (----) I was trying to sort a big file (22GB, 5GB gzipped) with `sort --compress-program=gzip -S40% data`. My /tmp filesystem is a 6GB tmpfs and the total system RAM is 16GB. The problem was that after a while sort would write uncompressed temp files in /tmp causing it to fill up and then crash for having no free space. After messing around with sort.c I found out that fork() would fail after the first batches were written successfully in /tmp with errno ENOMEM. I found out that I was hitting my kernel's VM overcommit limit as the tmpfs utilisation grew. And indeed, running `sysctl -w vm.overcommit=1` fixed the problem and the file was sorted. As sort is a program with high memory consumption and could be running in environments where overcommit is either unavailable or disabled I would expect it to use another method for spawning its compress_program. I'm not a C developer, so I'm not sure this is possible, but vfork() and clone() that seem to solve this problem. -- Petros Angelatos From debbugs-submit-bounces@debbugs.gnu.org Mon Jul 01 03:16:48 2013 Received: (at 14752-done) by debbugs.gnu.org; 1 Jul 2013 07:16:48 +0000 Received: from localhost ([127.0.0.1]:49764 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1UtYLw-0002U9-7Z for submit@debbugs.gnu.org; Mon, 01 Jul 2013 03:16:48 -0400 Received: from moutng.kundenserver.de ([212.227.17.8]:63398) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1UtYLt-0002Ts-E4 for 14752-done@debbugs.gnu.org; Mon, 01 Jul 2013 03:16:46 -0400 Received: from [192.168.1.11] (p57A5C935.dip0.t-ipconnect.de [87.165.201.53]) by mrelayeu.kundenserver.de (node=mreu2) with ESMTP (Nemesis) id 0LfFgy-1UR4AA00iG-00pPUn; Mon, 01 Jul 2013 09:16:38 +0200 Message-ID: <51D12CD5.20507@bernhard-voelker.de> Date: Mon, 01 Jul 2013 09:16:37 +0200 From: Bernhard Voelker User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130329 Thunderbird/17.0.5 MIME-Version: 1.0 To: Petros Aggelatos Subject: Re: bug#14752: sort fails to fork() + execlp(compress_program) if overcommit limit is reached References: In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Provags-ID: V02:K0:nf+pZGQi+aloJ9z+zQkYcbCzftLCRINLA1Rs0ktIl7q AQ6GFRK1+HRsB6d1i/DBIcfssQDNsA3wvYNS/paXAr9NctzPa/ xQUiwewsqrFWwH0PprgSerUG2wQ/Z8Lwh0cbnRFew2hAIiPAom KV3CU+lPJLLn5AQuQPrGweEl1KcWzJ0N3LiEm4ggigh7LO0TuY McqEuhoRB9kDsR4XZDNM1nveCahwKYwcYpH4UHmPi49LSoNgb5 THNyWrPvH5gstjSz4cTeuLmCoaENhoVcdYkJqrEUfhmFrJgNrc XVwXfEx4u2exOEvN21+NwgLbtTBZdN+KqIuQVW/Cu0yLC7wOYu DTh9j9/hy9tt28c3viZ7jsIjjERe+7F+GHoByaxhg X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 14752-done Cc: 14752-done@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.0 (/) tag 14752 notabug close 14752 stop On 06/30/2013 03:42 AM, Petros Aggelatos wrote: > I was trying to sort a big file (22GB, 5GB gzipped) with `sort > --compress-program=gzip -S40% data`. My /tmp filesystem is a 6GB tmpfs > and the total system RAM is 16GB. The problem was that after a while > sort would write uncompressed temp files in /tmp causing it to fill up > and then crash for having no free space. Thanks for reporting this. However, I think that your system's memory is just too small for sorting that file (that way, see below). You already recognized yourself that sort(1) was writing huge chunk files into the /tmp directory which is a tmpfs file system, i.e., that all that data is decreasing the memory available for running processes. The overhead for spawning a new process is negligible compared to such an amount of data. In such a case, you're much better off telling sort(1) to use a different directory for the temporary files. Here's an excerpt from the texinfo manual (info coreutils 'sort invocation'): If the environment variable `TMPDIR' is set, `sort' uses its value as the directory for temporary files instead of `/tmp'. The `--temporary-directory' (`-T') option in turn overrides the environment variable. ... `-T TEMPDIR' `--temporary-directory=TEMPDIR' Use directory TEMPDIR to store temporary files, overriding the `TMPDIR' environment variable. If this option is given more than once, temporary files are stored in all the directories given. If you have a large sort or merge that is I/O-bound, you can often improve performance by using this option to specify directories on different disks and controllers. Have a nice day, Berny From debbugs-submit-bounces@debbugs.gnu.org Mon Jul 01 05:52:12 2013 Received: (at 14752) by debbugs.gnu.org; 1 Jul 2013 09:52:13 +0000 Received: from localhost ([127.0.0.1]:49844 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1UtamJ-0006xa-VV for submit@debbugs.gnu.org; Mon, 01 Jul 2013 05:52:12 -0400 Received: from mail1.vodafone.ie ([213.233.128.43]:57042) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1UtamG-0006xG-Vu for 14752@debbugs.gnu.org; Mon, 01 Jul 2013 05:52:09 -0400 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AjkDAFRP0VFtTsAr/2dsb2JhbAANTYM7g1C8DgMBgRSDFwEBAQMBAQIgDwFLCwsNCwICBRYLAgIJAwIBAgEWLwcMCAEBiAUSqWVzkF6BJo0BC01mglGBFgOVA4NuhRuDWoh/gTqBZwEj Received: from unknown (HELO [192.168.1.79]) ([109.78.192.43]) by mail1.vodafone.ie with ESMTP; 01 Jul 2013 10:52:01 +0100 Message-ID: <51D15140.3070208@draigBrady.com> Date: Mon, 01 Jul 2013 10:52:00 +0100 From: =?UTF-8?B?UMOhZHJhaWcgQnJhZHk=?= User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130110 Thunderbird/17.0.2 MIME-Version: 1.0 To: 14752@debbugs.gnu.org, mail@bernhard-voelker.de, petrosagg@gmail.com Subject: Re: bug#14752: sort fails to fork() + execlp(compress_program) if overcommit limit is reached References: <51D12CD5.20507@bernhard-voelker.de> In-Reply-To: <51D12CD5.20507@bernhard-voelker.de> X-Enigmail-Version: 1.5.1 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 14752 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 0.0 (/) On 07/01/2013 08:16 AM, Bernhard Voelker wrote: > tag 14752 notabug > close 14752 > stop > > On 06/30/2013 03:42 AM, Petros Aggelatos wrote: >> I was trying to sort a big file (22GB, 5GB gzipped) with `sort >> --compress-program=gzip -S40% data`. My /tmp filesystem is a 6GB tmpfs >> and the total system RAM is 16GB. The problem was that after a while >> sort would write uncompressed temp files in /tmp causing it to fill up >> and then crash for having no free space. > > Thanks for reporting this. However, I think that your system's memory > is just too small for sorting that file (that way, see below). I'm not so sure there is nothing we might change here. Petros said that the sort succeeded when overcommit was set to "always overcommit". Some notes on fork() overcommit behavior: http://lkml.indiana.edu/hypermail/linux/kernel/0902.1/02389.html Petros, for completeness, what kernel were you using, and was SELinux enabled? > You already recognized yourself that sort(1) was writing huge chunk files > into the /tmp directory which is a tmpfs file system, i.e., that all that > data is decreasing the memory available for running processes. > The overhead for spawning a new process is negligible compared to such > an amount of data. > > In such a case, you're much better off telling sort(1) to use a different > directory for the temporary files. > > Here's an excerpt from the texinfo manual > (info coreutils 'sort invocation'): > > If the environment variable `TMPDIR' is set, `sort' uses its value > as the directory for temporary files instead of `/tmp'. The > `--temporary-directory' (`-T') option in turn overrides the environment > variable. > > ... > > `-T TEMPDIR' > `--temporary-directory=TEMPDIR' > Use directory TEMPDIR to store temporary files, overriding the > `TMPDIR' environment variable. If this option is given more than > once, temporary files are stored in all the directories given. If > you have a large sort or merge that is I/O-bound, you can often > improve performance by using this option to specify directories on > different disks and controllers. Note tmpfs is backed by RAM and swap, so depending on the swapiness settings for the kernel, it will auto migrate to the swap device(s) under RAM pressure. BTW, -S40% may be the root of the issue. Petros have you tried smaller buffers, which would probably avoid the issue on fork(), but also may take advantage of cache locality. I.E. sort currently takes advantage of a 2 level memory hierarchy by allocating large RAM buffers by default, and assuming /tmp is the next storage level down. By reducing the mem buffers down to take advantage of ever increasing cache sizes further up the hierarchy, may increase performance while avoiding the fork() issue. I previously made some notes on this here: http://lists.gnu.org/archive/html/bug-coreutils/2010-03/msg00008.html vfork() might be an option here. One can't rely on it being different to fork(), and it blocks the parent until the exec() in the child, and there are various restrictions on the child, but that might be fine? But I think posix_spawn() is the new standardised equivalent, and I notice the spawn-pipe gnulib module which might be leverged here? cheers, Pádraig. From debbugs-submit-bounces@debbugs.gnu.org Mon Jul 01 06:33:26 2013 Received: (at 14752) by debbugs.gnu.org; 1 Jul 2013 10:33:26 +0000 Received: from localhost ([127.0.0.1]:49874 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1UtbQE-0008Rx-CX for submit@debbugs.gnu.org; Mon, 01 Jul 2013 06:33:26 -0400 Received: from mail-ea0-f178.google.com ([209.85.215.178]:63296) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1UtbQ9-0008Rc-5t for 14752@debbugs.gnu.org; Mon, 01 Jul 2013 06:33:23 -0400 Received: by mail-ea0-f178.google.com with SMTP id l15so1939240eak.23 for <14752@debbugs.gnu.org>; Mon, 01 Jul 2013 03:33:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:x-enigmail-version:content-type :content-transfer-encoding; bh=KPAsdMnUCOMcPsjMrmDhgN70vhW2zVQIt6nTdiWDbXw=; b=KBCDrF9YzoWj64WtSUdXyitTwNegcGPC6OPipVYMT8vkHE3rcnWMYkPrYWQPTTyuum RRxe/9ctWWKCvNVrYvku3DzSPcUPTfjmLpV/oiN8MCQaUvfAe+A782RrbvV0UJu0+r4K rDepqMyA2ipDxx0e2iROYRW1igbe3gnBp0X4lIW6BS6Ia/zM/LfS+gY4IspAiOUKmMMd QRgQLT429e5hJgi4C8eEs4EuxRzvZJVMLTkBT5glY7gzaWff1mza44SJqkSBxQROumtv y3ptP9VzoUEQE5gEJ/bFZ/EnYCPC4EZh1Q6b/F6+trw0gYA2WxPXclRLxNGskEJvq0LU F0Dw== X-Received: by 10.15.50.132 with SMTP id l4mr20170660eew.122.1372674794971; Mon, 01 Jul 2013 03:33:14 -0700 (PDT) Received: from [192.168.1.6] ([88.218.140.53]) by mx.google.com with ESMTPSA id c44sm28868695eeb.8.2013.07.01.03.33.13 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Mon, 01 Jul 2013 03:33:13 -0700 (PDT) Message-ID: <51D15AE7.3010805@gmail.com> Date: Mon, 01 Jul 2013 13:33:11 +0300 From: Petros Aggelatos User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130625 Thunderbird/17.0.7 MIME-Version: 1.0 To: 14752@debbugs.gnu.org Subject: Re: bug#14752: sort fails to fork() + execlp(compress_program) if overcommit limit is reached References: <51D12CD5.20507@bernhard-voelker.de> <51D15140.3070208@draigBrady.com> In-Reply-To: <51D15140.3070208@draigBrady.com> X-Enigmail-Version: 1.5.1 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 14752 Cc: mail@bernhard-voelker.de, =?UTF-8?B?UMOhZHJhaWcgQnJhZHk=?= X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.7 (/) -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 07/01/2013 12:52 PM, Pádraig Brady wrote: > On 07/01/2013 08:16 AM, Bernhard Voelker wrote: Petros, for > completeness, what kernel were you using, and was SELinux enabled? I'm running kernel 3.9.7-1-ARCH and SELinux is disabled. > Note tmpfs is backed by RAM and swap, so depending on the swapiness > settings for the kernel, it will auto migrate to the swap device(s) > under RAM pressure. I have swap disabled on my system as 16GB or RAM seem more than enough. The problem is irrelevant whether it is using tmpfs or not, it just happened to trigger the error on my machine as it grew. But one can reproduce by disabling overcommit and running a sort with compression and -S60%. > BTW, -S40% may be the root of the issue. Petros have you tried > smaller buffers, which would probably avoid the issue on fork(), > but also may take advantage of cache locality. I tried using smaller buffers but I needed more temp space which I didn't have. If the initial cutting of the file results in more than NMERGE chunks then in the worst case it will need: N + NMERGE * P * N / (NMERGE + 1) where N is the input size and P is the number of merge-sorts running in parallel. But by using -S40% the input file would be split in less than 16 chunks so it would be just a single 16-way merge and require just N size of temp space. > vfork() might be an option here. One can't rely on it being > different to fork(), and it blocks the parent until the exec() in > the child, and there are various restrictions on the child, but > that might be fine? But I think posix_spawn() is the new > standardised equivalent, and I notice the spawn-pipe gnulib module > which might be leverged here? I did some further research on this after the initial report and posix_spawn() seems to me that is the standard way too. I could write a patch and test it. Regards, Petros -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.20 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQEcBAEBAgAGBQJR0VrnAAoJEHebBSKmldDLOasH/3iSJzO2AMU2LxKPKviSuMTz 5519trFD83e38C9I0o1ta830hofPXcpjoEHcinbc5VXibcLyvpfigSrcXoJa9FDZ pHvdOVJELFq4Rd4bXfDoVGoFZ79oNAUAZyR/tg2lapL0TKAiLF6hLOdG2WTtbdmM A1bOFIfjOWjC3Ro2aN7IPs4n1QTPYwLlmKCc4zCRWyl2B3m6EeiLN+qdKWNLpf/3 g8XoGv4yHRQENzk2s2o1aBcPyJ9FLtyQy/HlheNHThc9k+ReJe8JJ89h2UXhFKiT xgE62oNpPi3PL0w8SGuziTQgIKOgRbalQEZafUbeW2HB0ZiBKjZiY9o2PWh3UzM= =J1rs -----END PGP SIGNATURE----- From debbugs-submit-bounces@debbugs.gnu.org Mon Jul 01 07:14:05 2013 Received: (at 14752) by debbugs.gnu.org; 1 Jul 2013 11:14:05 +0000 Received: from localhost ([127.0.0.1]:49913 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1Utc3Y-0001mo-Rp for submit@debbugs.gnu.org; Mon, 01 Jul 2013 07:14:05 -0400 Received: from mail1.vodafone.ie ([213.233.128.43]:2348) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1Utc3W-0001m3-GU for 14752@debbugs.gnu.org; Mon, 01 Jul 2013 07:14:03 -0400 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AjkDAN5i0VFtTsAr/2dsb2JhbAANTYM7g1C8DQMBgRSDFwEBAQQjDwFGEAsNCwICBRYLAgIJAwIBAgFFBg0BBwEBiBepbnOQX4EmjQFYXweCUYEWA5hxhRuDWoh/gTqBaA Received: from unknown (HELO [192.168.1.79]) ([109.78.192.43]) by mail1.vodafone.ie with ESMTP; 01 Jul 2013 12:13:56 +0100 Message-ID: <51D16470.5000205@draigBrady.com> Date: Mon, 01 Jul 2013 12:13:52 +0100 From: =?UTF-8?B?UMOhZHJhaWcgQnJhZHk=?= User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130110 Thunderbird/17.0.2 MIME-Version: 1.0 To: Petros Aggelatos Subject: Re: bug#14752: sort fails to fork() + execlp(compress_program) if overcommit limit is reached References: <51D12CD5.20507@bernhard-voelker.de> <51D15140.3070208@draigBrady.com> <51D15AE7.3010805@gmail.com> In-Reply-To: <51D15AE7.3010805@gmail.com> X-Enigmail-Version: 1.5.1 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 14752 Cc: mail@bernhard-voelker.de, 14752@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 0.0 (/) On 07/01/2013 11:33 AM, Petros Aggelatos wrote: > On 07/01/2013 12:52 PM, Pádraig Brady wrote: >> On 07/01/2013 08:16 AM, Bernhard Voelker wrote: Petros, for >> completeness, what kernel were you using, and was SELinux enabled? > I'm running kernel 3.9.7-1-ARCH and SELinux is disabled. > >> Note tmpfs is backed by RAM and swap, so depending on the swapiness >> settings for the kernel, it will auto migrate to the swap device(s) >> under RAM pressure. > I have swap disabled on my system as 16GB or RAM seem more than > enough. The problem is irrelevant whether it is using tmpfs or not, it > just happened to trigger the error on my machine as it grew. But one > can reproduce by disabling overcommit and running a sort with > compression and -S60%. > >> BTW, -S40% may be the root of the issue. Petros have you tried >> smaller buffers, which would probably avoid the issue on fork(), >> but also may take advantage of cache locality. > I tried using smaller buffers but I needed more temp space which I > didn't have. If the initial cutting of the file results in more than > NMERGE chunks then in the worst case it will need: > N + NMERGE * P * N / (NMERGE + 1) > where N is the input size and P is the number of merge-sorts running > in parallel. But by using -S40% the input file would be split in less > than 16 chunks so it would be just a single 16-way merge and require > just N size of temp space. > >> vfork() might be an option here. One can't rely on it being >> different to fork(), and it blocks the parent until the exec() in >> the child, and there are various restrictions on the child, but >> that might be fine? But I think posix_spawn() is the new >> standardised equivalent, and I notice the spawn-pipe gnulib module >> which might be leverged here? > I did some further research on this after the initial report and > posix_spawn() seems to me that is the standard way too. I could write > a patch and test it. Patches would be gladly accepted! The higher level spawn-pipe module might not be appropriate, requiring use of posix_spawn() directly. In any case I notice that both are already incorporated in coreutils, through: posix-shell posix_spawn-internal posix_spawn_file_actions_addclose posix_spawn_file_actions_addclose-tests posix_spawn_file_actions_adddup2 posix_spawn_file_actions_adddup2-tests posix_spawn_file_actions_addopen posix_spawn_file_actions_addopen-tests posix_spawn_file_actions_destroy posix_spawn_file_actions_init posix_spawnattr_destroy posix_spawnattr_init posix_spawnattr_setflags posix_spawnattr_setsigmask posix_spawnp posix_spawnp-tests sigaction spawn spawn-pipe As a side note I notice that there is no specific cygwin implementation, though one seems to be available at http://cygwin.com/ml/cygwin/2012-01/msg00032.html thanks! Pádraig. From debbugs-submit-bounces@debbugs.gnu.org Mon Jul 01 19:45:43 2013 Received: (at 14752) by debbugs.gnu.org; 1 Jul 2013 23:45:43 +0000 Received: from localhost ([127.0.0.1]:51225 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1Utnmw-0007xr-JD for submit@debbugs.gnu.org; Mon, 01 Jul 2013 19:45:42 -0400 Received: from mail-ee0-f49.google.com ([74.125.83.49]:33233) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1Utnmt-0007xb-RI for 14752@debbugs.gnu.org; Mon, 01 Jul 2013 19:45:40 -0400 Received: by mail-ee0-f49.google.com with SMTP id b57so2328598eek.8 for <14752@debbugs.gnu.org>; Mon, 01 Jul 2013 16:45:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:x-enigmail-version:content-type :content-transfer-encoding; bh=iJh0LALpGQb5LbXeHvReg7d+Gf3dQ5nwQ9vAPvQEe7k=; b=ike4vyXYWVF2HHZgMBDFGewnJAO8HI4Qh9suqpRefFwqBZwXnM7bZTwC3/W6cw6EtK VKxw6vvXQErcHG0sW/I2u7sf99zeeFbr3QRHuC/82asWaxcAo0kJ/2SYr4rsPb0tknxQ rmtM1sfYB9WPAY8sS0hvoIQRUx4CaqLeOpHHZVu8WNNUpGhQnojDw1j4WYn8BLnibBTp Mx6kf/aAE6Ephrwibdu7A7HWEkythNkqS6pkVDeR/UmPSykhRKJwxlPKoRXXCUk9t8f7 UkVq55YcIfPxHF0egE4iaCkFbpKWIavQWBcATYIoVt2TV+k6J6dMYX4UGxcgscqBcaTo 9/Gw== X-Received: by 10.15.109.200 with SMTP id cf48mr23489980eeb.46.1372722333633; Mon, 01 Jul 2013 16:45:33 -0700 (PDT) Received: from [192.168.1.7] ([88.218.140.53]) by mx.google.com with ESMTPSA id bj46sm32878002eeb.13.2013.07.01.16.45.32 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Mon, 01 Jul 2013 16:45:32 -0700 (PDT) Message-ID: <51D2149A.6000605@gmail.com> Date: Tue, 02 Jul 2013 02:45:30 +0300 From: Petros Aggelatos User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130625 Thunderbird/17.0.7 MIME-Version: 1.0 To: =?UTF-8?B?UMOhZHJhaWcgQnJhZHk=?= Subject: Re: bug#14752: sort fails to fork() + execlp(compress_program) if overcommit limit is reached References: <51D12CD5.20507@bernhard-voelker.de> <51D15140.3070208@draigBrady.com> <51D15AE7.3010805@gmail.com> <51D16470.5000205@draigBrady.com> In-Reply-To: <51D16470.5000205@draigBrady.com> X-Enigmail-Version: 1.5.1 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 14752 Cc: mail@bernhard-voelker.de, 14752@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.7 (/) Following up on this issue it seems that posix_spawn() cannot be used, at least not trivially. The issue is that posix_spawn decides if it will spawn the new process with fork() of vfork() based on some conditions, one of which is if file_actions is NULL. http://repo.or.cz/w/glibc.git/blob/HEAD:/sysdeps/posix/spawni.c#l105 For the temp compression to work it is nessesary to pass the file descriptors of the pipe from the parent to the child. I'm not sure how to proceed, I found this relevant thread that proposes to relax the restrictions and use vfork more often: http://sourceware.org/bugzilla/show_bug.cgi?id=10354 And this thread http://sourceware.org/ml/libc-help/2010-10/msg00001.html of someone having the same problem and proposing two solutions. Solution #1 seems to me that adds a lot of complexity. Solution #2 is hacky, and I'm not aware if there are unwanted sideffects of using the enviroment to transfer the FDs. From debbugs-submit-bounces@debbugs.gnu.org Mon Jul 01 20:10:41 2013 Received: (at 14752) by debbugs.gnu.org; 2 Jul 2013 00:10:41 +0000 Received: from localhost ([127.0.0.1]:51277 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1UtoB5-00006H-Ql for submit@debbugs.gnu.org; Mon, 01 Jul 2013 20:10:41 -0400 Received: from mail1.vodafone.ie ([213.233.128.43]:9553) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1UtoB1-000060-Hy for 14752@debbugs.gnu.org; Mon, 01 Jul 2013 20:10:37 -0400 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AjkDAAMZ0lFtTsAr/2dsb2JhbAANTYM7g1C8PgMBgRSDFwEBAQMBIw8BRgULCw0LAgIFFgsCAgkDAgECAUUGCgMBBwEBF4duEqk6c5EigSaNAVhfB4JRgRYDmHGFG4NaiH+BOoFo Received: from unknown (HELO [192.168.1.79]) ([109.78.192.43]) by mail1.vodafone.ie with ESMTP; 02 Jul 2013 01:10:29 +0100 Message-ID: <51D21A71.3010008@draigBrady.com> Date: Tue, 02 Jul 2013 01:10:25 +0100 From: =?UTF-8?B?UMOhZHJhaWcgQnJhZHk=?= User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130110 Thunderbird/17.0.2 MIME-Version: 1.0 To: Petros Aggelatos Subject: Re: bug#14752: sort fails to fork() + execlp(compress_program) if overcommit limit is reached References: <51D12CD5.20507@bernhard-voelker.de> <51D15140.3070208@draigBrady.com> <51D15AE7.3010805@gmail.com> <51D16470.5000205@draigBrady.com> <51D2149A.6000605@gmail.com> In-Reply-To: <51D2149A.6000605@gmail.com> X-Enigmail-Version: 1.5.1 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 14752 Cc: mail@bernhard-voelker.de, 14752@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 0.0 (/) On 07/02/2013 12:45 AM, Petros Aggelatos wrote: > Following up on this issue it seems that posix_spawn() cannot be used, > at least not trivially. The issue is that posix_spawn decides if it will > spawn the new process with fork() of vfork() based on some conditions, > one of which is if file_actions is NULL. > > http://repo.or.cz/w/glibc.git/blob/HEAD:/sysdeps/posix/spawni.c#l105 Ugh, that's unfortunate :( That restriction seems a bit too harsh on cursory glance. > For the temp compression to work it is nessesary to pass the file > descriptors of the pipe from the parent to the child. I'm not sure how > to proceed, I found this relevant thread that proposes to relax the > restrictions and use vfork more often: > > http://sourceware.org/bugzilla/show_bug.cgi?id=10354 > > And this thread http://sourceware.org/ml/libc-help/2010-10/msg00001.html > of someone having the same problem and proposing two solutions. Solution > #1 seems to me that adds a lot of complexity. Solution #2 is hacky, and > I'm not aware if there are unwanted sideffects of using the enviroment > to transfer the FDs. Thanks for taking the time to find these previous discussions. Perhaps the best first approach is to use vfork() directly, so see if we actually do hit limitations in practice. If not, then perhaps we could adjust the gnulib spawni.c code accordingly, and add that as input to the above glibc bug for eventual incorporation into glibc. cheers, Pádraig. From unknown Sat Aug 09 15:52:20 2025 Received: (at fakecontrol) by fakecontrolmessage; To: internal_control@debbugs.gnu.org From: Debbugs Internal Request Subject: Internal Control Message-Id: bug archived. Date: Tue, 30 Jul 2013 11:24:04 +0000 User-Agent: Fakemail v42.6.9 # This is a fake control message. # # The action: # bug archived. thanks # This fakemail brought to you by your local debbugs # administrator From debbugs-submit-bounces@debbugs.gnu.org Mon May 26 16:46:11 2014 Received: (at control) by debbugs.gnu.org; 26 May 2014 20:46:11 +0000 Received: from localhost ([127.0.0.1]:60802 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1Wp1mc-00034u-A1 for submit@debbugs.gnu.org; Mon, 26 May 2014 16:46:10 -0400 Received: from mail4.vodafone.ie ([213.233.128.170]:19719) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1Wp1mZ-00034M-QU for control@debbugs.gnu.org; Mon, 26 May 2014 16:46:08 -0400 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AqYBAKamg1NtTKbf/2dsb2JhbAANTIQRgwqqW5RHMoEFgzkKKlQNAgUWCwILAwIBAgE5BgICFQgBAYhDrjR3pWsXgSqNRYJfgUsEoQePTA Received: from unknown (HELO [192.168.1.79]) ([109.76.166.223]) by mail3.vodafone.ie with ESMTP; 26 May 2014 21:46:01 +0100 Message-ID: <5383A809.7010506@draigBrady.com> Date: Mon, 26 May 2014 21:46:01 +0100 From: =?UTF-8?B?UMOhZHJhaWcgQnJhZHk=?= User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130110 Thunderbird/17.0.2 MIME-Version: 1.0 To: control@debbugs.gnu.org X-Enigmail-Version: 1.6 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Spam-Score: 2.0 (++) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has identified this incoming email as possible spam. The original message has been attached to this so you can view it (if it isn't spam) or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: unarchive 14752 [...] Content analysis details: (2.0 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- 1.8 MISSING_SUBJECT Missing Subject: header 0.2 NO_SUBJECT Extra score for no subject X-Debbugs-Envelope-To: control X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 2.0 (++) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has identified this incoming email as possible spam. The original message has been attached to this so you can view it (if it isn't spam) or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: unarchive 14752 [...] Content analysis details: (2.0 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- 1.8 MISSING_SUBJECT Missing Subject: header 0.2 NO_SUBJECT Extra score for no subject 0.0 TVD_SPACE_RATIO TVD_SPACE_RATIO unarchive 14752 From debbugs-submit-bounces@debbugs.gnu.org Mon May 26 16:48:22 2014 Received: (at 14752) by debbugs.gnu.org; 26 May 2014 20:48:22 +0000 Received: from localhost ([127.0.0.1]:60807 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1Wp1ok-00038k-5X for submit@debbugs.gnu.org; Mon, 26 May 2014 16:48:22 -0400 Received: from mail5.vodafone.ie ([213.233.128.176]:34813) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1Wp1oh-00038T-0X for 14752@debbugs.gnu.org; Mon, 26 May 2014 16:48:19 -0400 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AjQDAA6og1NtTKbf/2dsb2JhbAANTINZUa1MlFABgS2DGQEBAQQyAUYQCw0BCgkWDwkDAgECAUUGDQEHAQGIQwMFriqmYheOUgeEQASbMIVXhiSJKA Received: from unknown (HELO [192.168.1.79]) ([109.76.166.223]) by mail3.vodafone.ie with ESMTP; 26 May 2014 21:48:12 +0100 Message-ID: <5383A88C.7090301@draigBrady.com> Date: Mon, 26 May 2014 21:48:12 +0100 From: =?ISO-8859-1?Q?P=E1draig_Brady?= User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130110 Thunderbird/17.0.2 MIME-Version: 1.0 To: Azat Khuzhin Subject: Re: [PATCH] sort: print warning when fork() failed for --compress-program References: <1401135205-19048-1-git-send-email-a3at.mail@gmail.com> In-Reply-To: <1401135205-19048-1-git-send-email-a3at.mail@gmail.com> X-Enigmail-Version: 1.6 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 14752 Cc: 14752@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 0.0 (/) On 05/26/2014 09:13 PM, Azat Khuzhin wrote: > sort already have one when it trying to create decompressor, it is > obvious why it is really required in this case, since sort will read > compressed data as plain otherwise. > But sometimes it is really usefull to know whether sort failed to create > compressor or not, since some users may rely on available free space and > compressor. > > * src/sort.c (create_temp_file): Add a warning when creating of > compressor failed. > --- > There is some old discussion about this > http://osdir.com/ml/bug-coreutils-gnu/2013-07/msg00010.html, but before this > will be fixed(?) we could print a warning on fail at least. > Thanks. > > src/sort.c | 7 +++++++ > 1 file changed, 7 insertions(+) > > diff --git a/src/sort.c b/src/sort.c > index 49caae5..eb1b1f3 100644 > --- a/src/sort.c > +++ b/src/sort.c > @@ -1133,6 +1133,13 @@ maybe_create_temp (FILE **pfp, bool survive_fd_exhaustion) > > async_safe_die (errno, "couldn't execute compress program"); > } > + else > + { > + error (0, errno, > + _("warning: couldn't create process for %s " > + "(try to install overcommit always)"), > + compress_program); > + } > } > > *pfp = fdopen (tempfd, "w"); Thanks for the patch. Note POSIX says that programs shouldn't output to stderr unless they're exiting with a failure code, I guess to avoid gradual accretion of warnings etc. which could impair general usage. So the issue here is that sort is allocating a large buffer up front thus impacting the fork(). Really sort(1) should be trying to avoid this issue in the first place, and the issue is already logged at: http://bugs.gnu.org/14752 thanks, Pádraig. From debbugs-submit-bounces@debbugs.gnu.org Mon May 26 17:00:42 2014 Received: (at 14752) by debbugs.gnu.org; 26 May 2014 21:00:42 +0000 Received: from localhost ([127.0.0.1]:60840 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1Wp20e-00047Q-NZ for submit@debbugs.gnu.org; Mon, 26 May 2014 17:00:41 -0400 Received: from mail-lb0-f175.google.com ([209.85.217.175]:36415) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1Wp20b-0003ww-LB for 14752@debbugs.gnu.org; Mon, 26 May 2014 17:00:38 -0400 Received: by mail-lb0-f175.google.com with SMTP id l4so4411195lbv.6 for <14752@debbugs.gnu.org>; Mon, 26 May 2014 14:00:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:content-transfer-encoding :in-reply-to:user-agent; bh=Mb//HOr4L2ZE2CRj30rIEGaPyi7NMGiGiO1hfkb8LHs=; b=xyFZKjHzceIO95IoCHprSyqo6+sq9lZRvfQ5+9I4lrHDP7Rf43W0Ind1csNseaSHSH rLzLJeM1fL2ENjmF6XhloAclIWlO3S25FVPe4MNRsw8XIZnNnNGoyJIic4lA07FTdI7z Pi2Wnwn/GD89Q6DnxgDKUlxS5Cg6Y+wmnEfD5OqIZ9tomTTZTX7QnnlmVeCU5wyJMfMH eEDNoHBSFs7TlqmXkbIa5mgXnt3R7jAkxZ3rUQem5IBeJvwPrRVGuzobdW+wd+mAmPft YgJaQgJec56HNbU4ZFszIsSGi1qAowcodLz82q+Fl7jE06xki2FqZFFDKnneDzmhhTpY zM3w== X-Received: by 10.152.205.5 with SMTP id lc5mr19739277lac.30.1401138031492; Mon, 26 May 2014 14:00:31 -0700 (PDT) Received: from localhost ([188.134.22.24]) by mx.google.com with ESMTPSA id n3sm11394874lan.3.2014.05.26.14.00.30 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 26 May 2014 14:00:30 -0700 (PDT) Date: Tue, 27 May 2014 01:00:22 +0400 From: Azat Khuzhin To: =?iso-8859-1?Q?P=E1draig?= Brady Subject: Re: [PATCH] sort: print warning when fork() failed for --compress-program Message-ID: <20140526210022.GI26948@azat> References: <1401135205-19048-1-git-send-email-a3at.mail@gmail.com> <5383A76B.1010509@draigBrady.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <5383A76B.1010509@draigBrady.com> User-Agent: Mutt/1.5.23 (2014-03-12) X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 14752 Cc: 14752@debbugs.gnu.org, coreutils@gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.7 (/) On Mon, May 26, 2014 at 09:43:23PM +0100, Pádraig Brady wrote: > On 05/26/2014 09:13 PM, Azat Khuzhin wrote: > > sort already have one when it trying to create decompressor, it is > > obvious why it is really required in this case, since sort will read > > compressed data as plain otherwise. > > But sometimes it is really usefull to know whether sort failed to create > > compressor or not, since some users may rely on available free space and > > compressor. > > > > * src/sort.c (create_temp_file): Add a warning when creating of > > compressor failed. > > --- > > There is some old discussion about this > > http://osdir.com/ml/bug-coreutils-gnu/2013-07/msg00010.html, but before this > > will be fixed(?) we could print a warning on fail at least. > > Thanks. > > > > src/sort.c | 7 +++++++ > > 1 file changed, 7 insertions(+) > > > > diff --git a/src/sort.c b/src/sort.c > > index 49caae5..eb1b1f3 100644 > > --- a/src/sort.c > > +++ b/src/sort.c > > @@ -1133,6 +1133,13 @@ maybe_create_temp (FILE **pfp, bool survive_fd_exhaustion) > > > > async_safe_die (errno, "couldn't execute compress program"); > > } > > + else > > + { > > + error (0, errno, > > + _("warning: couldn't create process for %s " > > + "(try to install overcommit always)"), > > + compress_program); > > + } > > } > > > > *pfp = fdopen (tempfd, "w"); > > Thanks for the patch. > > Note POSIX says that programs shouldn't output to stderr > unless they're exiting with a failure code, Hm, didn't know that, Thanks! (Sometimes POSIX is not so good for human, I suppose.) > I guess to avoid gradual accretion of warnings etc. > which could impair general usage. > > So the issue here is that sort is allocating > a large buffer up front thus impacting the fork(). > Really sort(1) should be trying to avoid this issue > in the first place, and the issue is already logged at: > http://bugs.gnu.org/14752 Yes this is the same as I linked above. Does any body have a patch for this, or should I start working on this? Thanks. > > thanks, > Pádraig. -- Respectfully Azat Khuzhin From debbugs-submit-bounces@debbugs.gnu.org Mon May 26 17:11:07 2014 Received: (at 14752) by debbugs.gnu.org; 26 May 2014 21:11:07 +0000 Received: from localhost ([127.0.0.1]:60846 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1Wp2Ak-0004uL-PT for submit@debbugs.gnu.org; Mon, 26 May 2014 17:11:07 -0400 Received: from mail6.vodafone.ie ([213.233.128.184]:24192) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1Wp2Ai-0004tk-2y for 14752@debbugs.gnu.org; Mon, 26 May 2014 17:11:05 -0400 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AjQDAMqsg1NtTKbf/2dsb2JhbAANTINZUa1MlFABgS2DGQEBAQQyAUYQCw0BCgkWDwkDAgECAUUGDQEHAQGIQwOuNKZlF45SB4RAAQObMIVXhiSJKA Received: from unknown (HELO [192.168.1.79]) ([109.76.166.223]) by mail3.vodafone.ie with ESMTP; 26 May 2014 22:10:57 +0100 Message-ID: <5383ADE1.9070908@draigBrady.com> Date: Mon, 26 May 2014 22:10:57 +0100 From: =?ISO-8859-1?Q?P=E1draig_Brady?= User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130110 Thunderbird/17.0.2 MIME-Version: 1.0 To: Azat Khuzhin Subject: Re: [PATCH] sort: print warning when fork() failed for --compress-program References: <1401135205-19048-1-git-send-email-a3at.mail@gmail.com> <5383A76B.1010509@draigBrady.com> <20140526210022.GI26948@azat> In-Reply-To: <20140526210022.GI26948@azat> X-Enigmail-Version: 1.6 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 14752 Cc: 14752@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 0.0 (/) On 05/26/2014 10:00 PM, Azat Khuzhin wrote: >> So the issue here is that sort is allocating >> a large buffer up front thus impacting the fork(). >> Really sort(1) should be trying to avoid this issue >> in the first place, and the issue is already logged at: >> http://bugs.gnu.org/14752 > > Yes this is the same as I linked above. > Does any body have a patch for this, or should I start working on this? I was waiting for a patch that didn't materialize. I'll have a look myself now. thanks, Pádraig. From debbugs-submit-bounces@debbugs.gnu.org Wed May 28 11:16:09 2014 Received: (at 14752) by debbugs.gnu.org; 28 May 2014 15:16:09 +0000 Received: from localhost ([127.0.0.1]:34668 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1WpfaH-0002Ap-7P for submit@debbugs.gnu.org; Wed, 28 May 2014 11:16:08 -0400 Received: from mail3.vodafone.ie ([213.233.128.45]:54790) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1WpfaA-0002A9-M4 for 14752@debbugs.gnu.org; Wed, 28 May 2014 11:16:02 -0400 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: Ap0CAI78hVNtTK03/2dsb2JhbAANTINZUcFdUQGBH4MZAQEBBDIBRhALDQEKCRYPCQMCAQIBRQYNAQcBAYhDAwWxO6VEF4xYgTFJB4RABIRfApZRhVeGI4dogUFrgQI Received: from unknown (HELO [192.168.1.79]) ([109.76.173.55]) by mail3.vodafone.ie with ESMTP; 28 May 2014 16:15:51 +0100 Message-ID: <5385FDA7.9050405@draigBrady.com> Date: Wed, 28 May 2014 16:15:51 +0100 From: =?ISO-8859-1?Q?P=E1draig_Brady?= User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130110 Thunderbird/17.0.2 MIME-Version: 1.0 To: Azat Khuzhin Subject: Re: bug#14752: [PATCH] sort: print warning when fork() failed for --compress-program References: <1401135205-19048-1-git-send-email-a3at.mail@gmail.com> <5383A76B.1010509@draigBrady.com> <20140526210022.GI26948@azat> <5383ADE1.9070908@draigBrady.com> In-Reply-To: <5383ADE1.9070908@draigBrady.com> X-Enigmail-Version: 1.6 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 14752 Cc: 14752@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 0.0 (/) On 05/26/2014 10:10 PM, Pádraig Brady wrote: > On 05/26/2014 10:00 PM, Azat Khuzhin wrote: >>> So the issue here is that sort is allocating >>> a large buffer up front thus impacting the fork(). >>> Really sort(1) should be trying to avoid this issue >>> in the first place, and the issue is already logged at: >>> http://bugs.gnu.org/14752 >> >> Yes this is the same as I linked above. >> Does any body have a patch for this, or should I start working on this? > > I was waiting for a patch that didn't materialize. > I'll have a look myself now. So I had a look and the change while definitely worth doing is a bit invasive and so probably not appropriate for the impending release, as that's focusing on bug fixes rather than performance characteristics. Some implementation notes for reference... vfork() is portable only when one essentially just does an execve() right after the vfork(). Therefore just for fire and forget processes. Anything where you need to interact with the sub process like setting up files to communicate etc. is going to have portability issues. Even using execvp() is problematic I understand. Also sort is multithreaded which might further complicate things. Leveraging posix_spawn() is promising, as the main reason for that interface is to provide an efficient fork()+exec() mechanism. That can be implemented using vfork() or with clone() on Linux. The implementation in glibc is only a token one however as it just uses fork()+exec() usually. One can override with the POSIX_SPAWN_USEVFORK flag, but there are many non obvious implementation gotchas with doing that. Again being multithreaded may complicate things here. Note the musl, freebsd, osx and solaris posix_spawn() implementations are efficient, which would be another reason to use this assuming the glibc/gnulib implementation is fixed up. Another option would be for sort(1) to start up a helper child process, before we allocate much memory. Then we could communicate descriptors back and forth to that, and it could deal with forking the children. That would be portable too, but a little involved. Ideally we could keep the complication within posix_spawn() instead. Note this is a general issue not just related to sort(1). Many servers for example whether written in java/python/ruby or whatever have this issue when they use lots of memory and would like to popen() something. So fixing up the glibc posix_spawn() implementation would be very useful so that popen() etc. within glibc and the various language runtimes could leverage. Some links for reference: http://www.oracle.com/technetwork/server-storage/solaris10/subprocess-136439.html https://sourceware.org/ml/libc-help/2010-10/msg00001.html https://sourceware.org/bugzilla/show_bug.cgi?id=10354 https://sourceware.org/bugzilla/show_bug.cgi?id=378 http://git.musl-libc.org/cgit/musl/tree/src/process/posix_spawn.c http://ewontfix.com/7/ http://stackoverflow.com/questions/2731531/faster-forking-of-large-processes-on-linux http://stackoverflow.com/questions/8152076/spawn-process-from-multithreaded-application https://github.com/rtomayko/posix-spawn/blob/master/ext/posix-spawn.c http://blog.famzah.net/2009/11/20/a-much-faster-popen-and-system-implementation-for-linux/ http://code.google.com/p/popen-noshell/source/browse/trunk/popen_noshell.c From debbugs-submit-bounces@debbugs.gnu.org Wed May 28 12:23:11 2014 Received: (at 14752) by debbugs.gnu.org; 28 May 2014 16:23:11 +0000 Received: from localhost ([127.0.0.1]:34739 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1Wpgd8-0007N7-CW for submit@debbugs.gnu.org; Wed, 28 May 2014 12:23:11 -0400 Received: from smtp.cs.ucla.edu ([131.179.128.62]:39928) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1Wpgd1-0007MR-Qk for 14752@debbugs.gnu.org; Wed, 28 May 2014 12:23:04 -0400 Received: from localhost (localhost.localdomain [127.0.0.1]) by smtp.cs.ucla.edu (Postfix) with ESMTP id CE48DA60020; Wed, 28 May 2014 09:22:53 -0700 (PDT) X-Virus-Scanned: amavisd-new at smtp.cs.ucla.edu Received: from smtp.cs.ucla.edu ([127.0.0.1]) by localhost (smtp.cs.ucla.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 0S87fYWmDtCL; Wed, 28 May 2014 09:22:45 -0700 (PDT) Received: from [192.168.1.9] (pool-108-0-233-62.lsanca.fios.verizon.net [108.0.233.62]) by smtp.cs.ucla.edu (Postfix) with ESMTPSA id 38F5BA60019; Wed, 28 May 2014 09:22:45 -0700 (PDT) Message-ID: <53860D54.70701@cs.ucla.edu> Date: Wed, 28 May 2014 09:22:44 -0700 From: Paul Eggert Organization: UCLA Computer Science Department User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.5.0 MIME-Version: 1.0 To: =?UTF-8?B?UMOhZHJhaWcgQnJhZHk=?= , Azat Khuzhin Subject: Re: bug#14752: [PATCH] sort: print warning when fork() failed for --compress-program References: <1401135205-19048-1-git-send-email-a3at.mail@gmail.com> <5383A76B.1010509@draigBrady.com> <20140526210022.GI26948@azat> <5383ADE1.9070908@draigBrady.com> <5385FDA7.9050405@draigBrady.com> In-Reply-To: <5385FDA7.9050405@draigBrady.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Spam-Score: -3.0 (---) X-Debbugs-Envelope-To: 14752 Cc: 14752@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.0 (---) Pádraig Brady wrote: > Anything where you need to interact with the sub process like setting up files > to communicate etc. is going to have portability issues. Even using execvp() > is problematic I understand. As long as the child doesn't touch parent memory that the parent needs, it should be OK. There is a memory leak in execvp in old glibc versions, but I expect that isn't something we need to worry about. Last time I checked, vfork was significantly faster than fork when the parent process has a lot of memory, and was still worth using for its performance advantages. From unknown Sat Aug 09 15:52:20 2025 Received: (at fakecontrol) by fakecontrolmessage; To: internal_control@debbugs.gnu.org From: Debbugs Internal Request Subject: Internal Control Message-Id: bug archived. Date: Thu, 26 Jun 2014 11:24:04 +0000 User-Agent: Fakemail v42.6.9 # This is a fake control message. # # The action: # bug archived. thanks # This fakemail brought to you by your local debbugs # administrator From debbugs-submit-bounces@debbugs.gnu.org Wed Jul 15 23:06:59 2015 Received: (at control) by debbugs.gnu.org; 16 Jul 2015 03:06:59 +0000 Received: from localhost ([127.0.0.1]:50503 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1ZFZVj-0007bo-6t for submit@debbugs.gnu.org; Wed, 15 Jul 2015 23:06:59 -0400 Received: from mail2.vodafone.ie ([213.233.128.44]:56749) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1ZFZVh-0007bW-Ue for control@debbugs.gnu.org; Wed, 15 Jul 2015 23:06:58 -0400 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AlcIAD0fp1VtTfsV/2dsb2JhbABagxOBFwiBX4EVTok/tko1gRpMAQEBAQEBgQuEQwoqVA0CBRQCCwILAwIBAgE5BgICCA0IAQGILgGpTo9fhWuQa4Eikh+BQwWUPpRvkBwmgUkBAQEHAQEBAYImPYJ8AgEC Received: from unknown (HELO localhost.localdomain) ([109.77.251.21]) by mail2.vodafone.ie with ESMTP; 16 Jul 2015 04:06:52 +0100 Message-ID: <55A71FCC.6010708@draigBrady.com> Date: Thu, 16 Jul 2015 04:06:52 +0100 From: =?UTF-8?B?UMOhZHJhaWcgQnJhZHk=?= User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.6.0 MIME-Version: 1.0 To: GNU bug tracker automated control server Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Spam-Score: 2.0 (++) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has identified this incoming email as possible spam. The original message has been attached to this so you can view it (if it isn't spam) or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: unarchive 14752 [...] Content analysis details: (2.0 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at http://www.dnswl.org/, no trust [213.233.128.44 listed in list.dnswl.org] 1.8 MISSING_SUBJECT Missing Subject: header 0.2 NO_SUBJECT Extra score for no subject X-Debbugs-Envelope-To: control X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 2.0 (++) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has identified this incoming email as possible spam. The original message has been attached to this so you can view it (if it isn't spam) or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: unarchive 14752 [...] Content analysis details: (2.0 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at http://www.dnswl.org/, no trust [213.233.128.44 listed in list.dnswl.org] 1.8 MISSING_SUBJECT Missing Subject: header 0.2 NO_SUBJECT Extra score for no subject 0.0 TVD_SPACE_RATIO TVD_SPACE_RATIO unarchive 14752 From debbugs-submit-bounces@debbugs.gnu.org Wed Jul 15 23:10:05 2015 Received: (at 14752) by debbugs.gnu.org; 16 Jul 2015 03:10:05 +0000 Received: from localhost ([127.0.0.1]:50508 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1ZFZYf-0007hM-RJ for submit@debbugs.gnu.org; Wed, 15 Jul 2015 23:10:04 -0400 Received: from mail2.vodafone.ie ([213.233.128.44]:26261) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1ZFZYd-0007gy-7e for 14752@debbugs.gnu.org; Wed, 15 Jul 2015 23:10:00 -0400 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AnMFAD0fp1VtTfsV/2dsb2JhbABagxNUYwaCVrpqhStHAQICgUVMAQEBAQEBgQuEIwEBAQMBMgFWCw0LCRQCDwkDAgECAUUTBgIBAYgiDAEDBb8QkD8MAR+LTIJrgVACUIQrBZQ+hG2IZkaGVokAgzuDYSZjgVuBPz0xAYEEgUYBAQE Received: from unknown (HELO localhost.localdomain) ([109.77.251.21]) by mail2.vodafone.ie with ESMTP; 16 Jul 2015 04:09:53 +0100 Message-ID: <55A72081.8010309@draigBrady.com> Date: Thu, 16 Jul 2015 04:09:53 +0100 From: =?windows-1252?Q?P=E1draig_Brady?= User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.6.0 MIME-Version: 1.0 To: 14752@debbugs.gnu.org Subject: Re: bug#14752: [PATCH] sort: print warning when fork() failed for --compress-program References: <1401135205-19048-1-git-send-email-a3at.mail@gmail.com> <5383A76B.1010509@draigBrady.com> <20140526210022.GI26948@azat> <5383ADE1.9070908@draigBrady.com> <5385FDA7.9050405@draigBrady.com> <20150716020028.GN31308@azat> In-Reply-To: <20150716020028.GN31308@azat> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 8bit X-Spam-Score: 1.7 (+) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has identified this incoming email as possible spam. The original message has been attached to this so you can view it (if it isn't spam) or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: On 16/07/15 03:00, Azat Khuzhin wrote: > On Wed, May 28, 2014 at 04:15:51PM +0100, Pádraig Brady wrote: >> On 05/26/2014 10:10 PM, Pádraig Brady wrote: >>> On 05/26/2014 10:00 PM, Azat Khuzhin wrote: >>>>> So the issue here is that sort is allocating >>>>> a large buffer up front thus impacting the fork(). >>>>> Really sort(1) should be trying to avoid this issue >>>>> in the first place, and the issue is already logged at: >>>>> http://bugs.gnu.org/14752 >>>> >>>> Yes this is the same as I linked above. >>>> Does any body have a patch for this, or should I start working on this? >>> >>> I was waiting for a patch that didn't materialize. >>> I'll have a look myself now. >> >> So I had a look and the change while definitely worth doing >> is a bit invasive and so probably not appropriate for the impending release, >> as that's focusing on bug fixes rather than performance characteristics. >> >> Some implementation notes for reference... > > Hi Pádraig, > > Thanks for your vision/thoughts about this problem, they are very > detailed. > >> vfork() is portable only when one essentially just does an >> execve() right after the vfork(). Therefore just for fire and forget processes. >> Anything where you need to interact with the sub process like setting up files >> to communicate etc. is going to have portability issues. Even using execvp() >> is problematic I understand. Also sort is multithreaded which might further >> complicate things. > > Agree. > >> Leveraging posix_spawn() is promising, as the main reason for that >> interface is to provide an efficient fork()+exec() mechanism. >> That can be implemented using vfork() or with clone() on Linux. >> The implementation in glibc is only a token one however as it >> just uses fork()+exec() usually. One can override with the >> POSIX_SPAWN_USEVFORK flag, but there are many non obvious >> implementation gotchas with doing that. Again being multithreaded >> may complicate things here. Note the musl, freebsd, osx and solaris >> posix_spawn() implementations are efficient, [...] Content analysis details: (1.7 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at http://www.dnswl.org/, no trust [213.233.128.44 listed in list.dnswl.org] 1.7 URIBL_BLACK Contains an URL listed in the URIBL blacklist [URIs: musl-libc.org] X-Debbugs-Envelope-To: 14752 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 1.7 (+) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has identified this incoming email as possible spam. The original message has been attached to this so you can view it (if it isn't spam) or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: On 16/07/15 03:00, Azat Khuzhin wrote: > On Wed, May 28, 2014 at 04:15:51PM +0100, Pádraig Brady wrote: >> On 05/26/2014 10:10 PM, Pádraig Brady wrote: >>> On 05/26/2014 10:00 PM, Azat Khuzhin wrote: >>>>> So the issue here is that sort is allocating >>>>> a large buffer up front thus impacting the fork(). >>>>> Really sort(1) should be trying to avoid this issue >>>>> in the first place, and the issue is already logged at: >>>>> http://bugs.gnu.org/14752 >>>> >>>> Yes this is the same as I linked above. >>>> Does any body have a patch for this, or should I start working on this? >>> >>> I was waiting for a patch that didn't materialize. >>> I'll have a look myself now. >> >> So I had a look and the change while definitely worth doing >> is a bit invasive and so probably not appropriate for the impending release, >> as that's focusing on bug fixes rather than performance characteristics. >> >> Some implementation notes for reference... > > Hi Pádraig, > > Thanks for your vision/thoughts about this problem, they are very > detailed. > >> vfork() is portable only when one essentially just does an >> execve() right after the vfork(). Therefore just for fire and forget processes. >> Anything where you need to interact with the sub process like setting up files >> to communicate etc. is going to have portability issues. Even using execvp() >> is problematic I understand. Also sort is multithreaded which might further >> complicate things. > > Agree. > >> Leveraging posix_spawn() is promising, as the main reason for that >> interface is to provide an efficient fork()+exec() mechanism. >> That can be implemented using vfork() or with clone() on Linux. >> The implementation in glibc is only a token one however as it >> just uses fork()+exec() usually. One can override with the >> POSIX_SPAWN_USEVFORK flag, but there are many non obvious >> implementation gotchas with doing that. Again being multithreaded >> may complicate things here. Note the musl, freebsd, osx and solaris >> posix_spawn() implementations are efficient, [...] Content analysis details: (1.7 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at http://www.dnswl.org/, no trust [213.233.128.44 listed in list.dnswl.org] 1.7 URIBL_BLACK Contains an URL listed in the URIBL blacklist [URIs: musl-libc.org] On 16/07/15 03:00, Azat Khuzhin wrote: > On Wed, May 28, 2014 at 04:15:51PM +0100, Pádraig Brady wrote: >> On 05/26/2014 10:10 PM, Pádraig Brady wrote: >>> On 05/26/2014 10:00 PM, Azat Khuzhin wrote: >>>>> So the issue here is that sort is allocating >>>>> a large buffer up front thus impacting the fork(). >>>>> Really sort(1) should be trying to avoid this issue >>>>> in the first place, and the issue is already logged at: >>>>> http://bugs.gnu.org/14752 >>>> >>>> Yes this is the same as I linked above. >>>> Does any body have a patch for this, or should I start working on this? >>> >>> I was waiting for a patch that didn't materialize. >>> I'll have a look myself now. >> >> So I had a look and the change while definitely worth doing >> is a bit invasive and so probably not appropriate for the impending release, >> as that's focusing on bug fixes rather than performance characteristics. >> >> Some implementation notes for reference... > > Hi Pádraig, > > Thanks for your vision/thoughts about this problem, they are very > detailed. > >> vfork() is portable only when one essentially just does an >> execve() right after the vfork(). Therefore just for fire and forget processes. >> Anything where you need to interact with the sub process like setting up files >> to communicate etc. is going to have portability issues. Even using execvp() >> is problematic I understand. Also sort is multithreaded which might further >> complicate things. > > Agree. > >> Leveraging posix_spawn() is promising, as the main reason for that >> interface is to provide an efficient fork()+exec() mechanism. >> That can be implemented using vfork() or with clone() on Linux. >> The implementation in glibc is only a token one however as it >> just uses fork()+exec() usually. One can override with the >> POSIX_SPAWN_USEVFORK flag, but there are many non obvious >> implementation gotchas with doing that. Again being multithreaded >> may complicate things here. Note the musl, freebsd, osx and solaris >> posix_spawn() implementations are efficient, which would be >> another reason to use this assuming the glibc/gnulib implementation >> is fixed up. > > AFAIK posix_spawn() in glibc will be something like fork() if you need > to setup files, IOW it seems that sort can't posix_spawn() for > compress-prog to fix this problem: > > __spawni: > ... > if ((flags & POSIX_SPAWN_USEVFORK) != 0 > /* If no major work is done, allow using vfork. Note that we > might perform the path searching. But this would be done by > a call to execvp(), too, and such a call must be OK according > to POSIX. */ > || ((flags & (POSIX_SPAWN_SETSIGMASK | POSIX_SPAWN_SETSIGDEF > | POSIX_SPAWN_SETSCHEDPARAM | POSIX_SPAWN_SETSCHEDULER > | POSIX_SPAWN_SETPGROUP | POSIX_SPAWN_RESETIDS)) == 0 > && file_actions == NULL)) > new_pid = __vfork (); > else > new_pid = __fork (); > > While sort must use file_actions, no? > > Am I missing something? Right the glibc implementation currently doesn't benefit here. Though it's no worse, and other platforms benefit. There has been a little movement on the associated glibc bug since: https://sourceware.org/bugzilla/show_bug.cgi?id=10354 >> Another option would be for sort(1) to start up a helper child process, >> before we allocate much memory. Then we could communicate descriptors >> back and forth to that, and it could deal with forking the children. >> That would be portable too, but a little involved. Ideally we could >> keep the complication within posix_spawn() instead. > > So I guess that this is the only choice, and since there is no activity > since 2014, I have WIP/RFC patch that implements this, it passes some > basic tests, but not all (I'm working on it) and also needs in cleanup > and prettying, as for now maybe somebody have some comments? > > You could find patches in attachments and via git: > git@github.com:azat/coreutils.git sort-compress-prog-fixes-v2 Thanks for picking this up again. It is quite involved as expected. >> Note this is a general issue not just related to sort(1). >> Many servers for example whether written in java/python/ruby or whatever >> have this issue when they use lots of memory and would like to >> popen() something. So fixing up the glibc posix_spawn() implementation >> would be very useful so that popen() etc. within glibc and the >> various language runtimes could leverage. With the above general benefits in mind, and the complexity of the sort fork server implementation, I'm inclined to think working on posix_spawn() in glibc is more beneficial, and might actually be as easy. Also sort(1) can be moved to using posix_spawn() independently. >> >> Some links for reference: >> >> http://www.oracle.com/technetwork/server-storage/solaris10/subprocess-136439.html >> https://sourceware.org/ml/libc-help/2010-10/msg00001.html >> https://sourceware.org/bugzilla/show_bug.cgi?id=10354 >> https://sourceware.org/bugzilla/show_bug.cgi?id=378 >> http://git.musl-libc.org/cgit/musl/tree/src/process/posix_spawn.c >> http://ewontfix.com/7/ >> http://stackoverflow.com/questions/2731531/faster-forking-of-large-processes-on-linux >> http://stackoverflow.com/questions/8152076/spawn-process-from-multithreaded-application >> https://github.com/rtomayko/posix-spawn/blob/master/ext/posix-spawn.c >> http://blog.famzah.net/2009/11/20/a-much-faster-popen-and-system-implementation-for-linux/ >> http://code.google.com/p/popen-noshell/source/browse/trunk/popen_noshell.c thanks! Pádraig. From debbugs-submit-bounces@debbugs.gnu.org Wed Jul 15 23:12:28 2015 Received: (at 14752) by debbugs.gnu.org; 16 Jul 2015 03:12:28 +0000 Received: from localhost ([127.0.0.1]:50512 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1ZFZb2-0007l7-6Z for submit@debbugs.gnu.org; Wed, 15 Jul 2015 23:12:28 -0400 Received: from mail2.vodafone.ie ([213.233.128.44]:56066) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1ZFZb0-0007ku-Al for 14752@debbugs.gnu.org; Wed, 15 Jul 2015 23:12:26 -0400 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AlcFAGAgp1VtTfsV/2dsb2JhbABagxOEE74MglQCgUZMAQEBAQEBgQuEIwEBAQMBMgFGBQsLDQEKCRQCDwkDAgECAUUGDQEHAQEXiAsMAb8YkD8BAQEBBgIBH4tMhDtLB4QrAQSUPpRvkBwmY4FbgT89gnwBAQE Received: from unknown (HELO localhost.localdomain) ([109.77.251.21]) by mail2.vodafone.ie with ESMTP; 16 Jul 2015 04:12:20 +0100 Message-ID: <55A72114.7020601@draigBrady.com> Date: Thu, 16 Jul 2015 04:12:20 +0100 From: =?windows-1252?Q?P=E1draig_Brady?= User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.6.0 MIME-Version: 1.0 To: Azat Khuzhin Subject: Re: bug#14752: [PATCH] sort: print warning when fork() failed for --compress-program References: <1401135205-19048-1-git-send-email-a3at.mail@gmail.com> <5383A76B.1010509@draigBrady.com> <20140526210022.GI26948@azat> <5383ADE1.9070908@draigBrady.com> <5385FDA7.9050405@draigBrady.com> <20150716020028.GN31308@azat> <55A71B1F.8010208@draigBrady.com> <20150716025636.GV31308@azat> In-Reply-To: <20150716025636.GV31308@azat> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 8bit X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 14752 Cc: 14752@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 0.0 (/) On 16/07/15 03:56, Azat Khuzhin wrote: > On Thu, Jul 16, 2015 at 03:46:55AM +0100, Pádraig Brady wrote: >> With the above general benefits in mind, and the complexity >> of the sort fork server implementation, I'm inclined to >> think working on posix_spawn() in glibc is more beneficial, >> and might actually be as easy. Also sort(1) can be moved >> to using posix_spawn() independently. > > Yes it will be more general benefit, but I don't think (and this is how > it works) that every custom server will switch to it, but I agree that > this shouldn't be a problem for highly used servers. And also after > posix_spawn will be fixed sort will require new glibc to fix this > issue, and hence some distros (like debian) couldn't ship "sort" with > fix for this issue without glibc (but I totally agree with you about how > this must be fixed -- in more generic basis). Note we've some flexibility with gnulib to replace posix_spawn() on platforms where we determine the system version doesn't suffice. > Anyway, let's get back to "sort", any chances that that patches (more > cleaner and less buggy of course) could be picked for upstream? Maybe, though sort(1) is complex enough already, so I'd still prefer at least to look at the more general solution. > And thanks for link to the bug about posix_spawn I will look it more > closely. thanks, Pádraig. From unknown Sat Aug 09 15:52:20 2025 Received: (at fakecontrol) by fakecontrolmessage; To: internal_control@debbugs.gnu.org From: Debbugs Internal Request Subject: Internal Control Message-Id: bug archived. Date: Thu, 13 Aug 2015 11:24:04 +0000 User-Agent: Fakemail v42.6.9 # This is a fake control message. # # The action: # bug archived. thanks # This fakemail brought to you by your local debbugs # administrator From debbugs-submit-bounces@debbugs.gnu.org Wed Feb 03 00:16:01 2016 Received: (at control) by debbugs.gnu.org; 3 Feb 2016 05:16:01 +0000 Received: from localhost ([127.0.0.1]:57333 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84) (envelope-from ) id 1aQpnN-0001L9-7U for submit@debbugs.gnu.org; Wed, 03 Feb 2016 00:16:01 -0500 Received: from mail.magicbluesmoke.com ([82.195.144.49]:45197) by debbugs.gnu.org with esmtp (Exim 4.84) (envelope-from ) id 1aQpnL-0001Kz-Ui for control@debbugs.gnu.org; Wed, 03 Feb 2016 00:16:00 -0500 Received: from localhost.localdomain (c-73-70-29-104.hsd1.ca.comcast.net [73.70.29.104]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.magicbluesmoke.com (Postfix) with ESMTPSA id 49F6E9CCF for ; Wed, 3 Feb 2016 05:15:58 +0000 (GMT) To: GNU bug tracker automated control server From: =?UTF-8?Q?P=c3=a1draig_Brady?= Message-ID: <56B18D0B.3000608@draigBrady.com> Date: Tue, 2 Feb 2016 21:15:55 -0800 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.3.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Spam-Score: 2.0 (++) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: unarchive 14752 [...] Content analysis details: (2.0 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- 1.8 MISSING_SUBJECT Missing Subject: header 0.2 NO_SUBJECT Extra score for no subject X-Debbugs-Envelope-To: control X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 2.0 (++) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: unarchive 14752 [...] Content analysis details: (2.0 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- 1.8 MISSING_SUBJECT Missing Subject: header 0.2 NO_SUBJECT Extra score for no subject 0.0 TVD_SPACE_RATIO No description available. unarchive 14752 From debbugs-submit-bounces@debbugs.gnu.org Wed Feb 03 00:18:01 2016 Received: (at 14752) by debbugs.gnu.org; 3 Feb 2016 05:18:01 +0000 Received: from localhost ([127.0.0.1]:57338 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84) (envelope-from ) id 1aQppB-0001OW-K1 for submit@debbugs.gnu.org; Wed, 03 Feb 2016 00:18:01 -0500 Received: from mail.magicbluesmoke.com ([82.195.144.49]:45199) by debbugs.gnu.org with esmtp (Exim 4.84) (envelope-from ) id 1aQpp4-0001OJ-Kp for 14752@debbugs.gnu.org; Wed, 03 Feb 2016 00:17:52 -0500 Received: from localhost.localdomain (c-73-70-29-104.hsd1.ca.comcast.net [73.70.29.104]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.magicbluesmoke.com (Postfix) with ESMTPSA id BE1749CE5; Wed, 3 Feb 2016 05:17:45 +0000 (GMT) Subject: Re: bug#14752: [PATCH] sort: print warning when fork() failed for --compress-program To: Azat Khuzhin References: <1401135205-19048-1-git-send-email-a3at.mail@gmail.com> <5383A76B.1010509@draigBrady.com> <20140526210022.GI26948@azat> <5383ADE1.9070908@draigBrady.com> <5385FDA7.9050405@draigBrady.com> From: =?UTF-8?Q?P=c3=a1draig_Brady?= Message-ID: <56B18D77.6020106@draigBrady.com> Date: Tue, 2 Feb 2016 21:17:43 -0800 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.3.0 MIME-Version: 1.0 In-Reply-To: <5385FDA7.9050405@draigBrady.com> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 8bit X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 14752 Cc: 14752@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 0.0 (/) unarchive 14752 On 28/05/14 08:15, Pádraig Brady wrote: > On 05/26/2014 10:10 PM, Pádraig Brady wrote: >> On 05/26/2014 10:00 PM, Azat Khuzhin wrote: >>>> So the issue here is that sort is allocating >>>> a large buffer up front thus impacting the fork(). >>>> Really sort(1) should be trying to avoid this issue >>>> in the first place, and the issue is already logged at: >>>> http://bugs.gnu.org/14752 >>> >>> Yes this is the same as I linked above. >>> Does any body have a patch for this, or should I start working on this? >> >> I was waiting for a patch that didn't materialize. >> I'll have a look myself now. > > So I had a look and the change while definitely worth doing > is a bit invasive and so probably not appropriate for the impending release, > as that's focusing on bug fixes rather than performance characteristics. > > Some implementation notes for reference... > > vfork() is portable only when one essentially just does an > execve() right after the vfork(). Therefore just for fire and forget processes. > Anything where you need to interact with the sub process like setting up files > to communicate etc. is going to have portability issues. Even using execvp() > is problematic I understand. Also sort is multithreaded which might further > complicate things. > > Leveraging posix_spawn() is promising, as the main reason for that > interface is to provide an efficient fork()+exec() mechanism. > That can be implemented using vfork() or with clone() on Linux. > The implementation in glibc is only a token one however as it > just uses fork()+exec() usually. One can override with the > POSIX_SPAWN_USEVFORK flag, but there are many non obvious > implementation gotchas with doing that. Again being multithreaded > may complicate things here. Note the musl, freebsd, osx and solaris > posix_spawn() implementations are efficient, which would be > another reason to use this assuming the glibc/gnulib implementation > is fixed up. Progress on the glibc posix_spawn() front: https://sourceware.org/ml/libc-alpha/2016-02/msg00016.html > > Another option would be for sort(1) to start up a helper child process, > before we allocate much memory. Then we could communicate descriptors > back and forth to that, and it could deal with forking the children. > That would be portable too, but a little involved. Ideally we could > keep the complication within posix_spawn() instead. > > Note this is a general issue not just related to sort(1). > Many servers for example whether written in java/python/ruby or whatever > have this issue when they use lots of memory and would like to > popen() something. So fixing up the glibc posix_spawn() implementation > would be very useful so that popen() etc. within glibc and the > various language runtimes could leverage. > > Some links for reference: > > http://www.oracle.com/technetwork/server-storage/solaris10/subprocess-136439.html > https://sourceware.org/ml/libc-help/2010-10/msg00001.html > https://sourceware.org/bugzilla/show_bug.cgi?id=10354 > https://sourceware.org/bugzilla/show_bug.cgi?id=378 > http://git.musl-libc.org/cgit/musl/tree/src/process/posix_spawn.c > http://ewontfix.com/7/ > http://stackoverflow.com/questions/2731531/faster-forking-of-large-processes-on-linux > http://stackoverflow.com/questions/8152076/spawn-process-from-multithreaded-application > https://github.com/rtomayko/posix-spawn/blob/master/ext/posix-spawn.c > http://blog.famzah.net/2009/11/20/a-much-faster-popen-and-system-implementation-for-linux/ > http://code.google.com/p/popen-noshell/source/browse/trunk/popen_noshell.c From unknown Sat Aug 09 15:52:20 2025 Received: (at fakecontrol) by fakecontrolmessage; To: internal_control@debbugs.gnu.org From: Debbugs Internal Request Subject: Internal Control Message-Id: bug archived. Date: Wed, 02 Mar 2016 12:24:04 +0000 User-Agent: Fakemail v42.6.9 # This is a fake control message. # # The action: # bug archived. thanks # This fakemail brought to you by your local debbugs # administrator