From unknown Tue Jun 17 01:33:03 2025 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Mailer: MIME-tools 5.509 (Entity 5.509) Content-Type: text/plain; charset=utf-8 From: bug#36630 <36630@debbugs.gnu.org> To: bug#36630 <36630@debbugs.gnu.org> Subject: Status: [PATCH] guix: parallelize building the manual-database Reply-To: bug#36630 <36630@debbugs.gnu.org> Date: Tue, 17 Jun 2025 08:33:03 +0000 retitle 36630 [PATCH] guix: parallelize building the manual-database reassign 36630 guix-patches submitter 36630 arne_bab@web.de severity 36630 normal tag 36630 patch thanks From debbugs-submit-bounces@debbugs.gnu.org Fri Jul 12 17:43:19 2019 Received: (at submit) by debbugs.gnu.org; 12 Jul 2019 21:43:19 +0000 Received: from localhost ([127.0.0.1]:40861 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hm3K2-0006Ni-Os for submit@debbugs.gnu.org; Fri, 12 Jul 2019 17:43:19 -0400 Received: from lists.gnu.org ([209.51.188.17]:38274) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hm3K0-0006Na-V8 for submit@debbugs.gnu.org; Fri, 12 Jul 2019 17:43:17 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:52943) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hm3Jz-0003qD-Tr for guix-patches@gnu.org; Fri, 12 Jul 2019 17:43:16 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-0.7 required=5.0 tests=BAYES_20,FREEMAIL_FROM, RCVD_IN_DNSWL_LOW autolearn=disabled version=3.3.2 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hm3Jy-0005gA-Sn for guix-patches@gnu.org; Fri, 12 Jul 2019 17:43:15 -0400 Received: from mout.web.de ([217.72.192.78]:33057) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1hm3Jw-0005dr-Po for guix-patches@gnu.org; Fri, 12 Jul 2019 17:43:13 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=web.de; s=dbaedf251592; t=1562967788; bh=JSbsKdEiZ7UE+6euXjjDF1YUCnSrh1qriFSJCajp9WE=; h=X-UI-Sender-Class:From:To:Cc:Subject:Date:Reply-To; b=HoCYmKd1W4wBxRFYm8LJwncDR4fvwHBY4FaLxJ6UXivGXFx3ICocirIVphZV0HNGf 7WHyjjwSmIRZDdGhilPbhVzHrVAGZ+ai/OQK8XTXOfezPVc2jn49H68/x25PLI5eE8 jmISwfqE9tYTvswKN3c5po9m5rK34PTrS7hlafoA= X-UI-Sender-Class: c548c8c5-30a9-4db5-a2e7-cb6cb037b8f9 Received: from fluss.speedport.ip ([84.149.83.171]) by smtp.web.de (mrweb103 [213.165.67.124]) with ESMTPSA (Nemesis) id 0MRlED-1hsoej20IT-00SuF8; Fri, 12 Jul 2019 23:43:08 +0200 From: Arne Babenhauserheide To: guix-patches@gnu.org Subject: [PATCH] guix: parallelize building the manual-database Date: Fri, 12 Jul 2019 23:42:45 +0200 Message-Id: <20190712214245.23857-1-arne_bab@web.de> X-Mailer: git-send-email 2.22.0 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Provags-ID: V03:K1:M6QunW6vh03fYfPLecXtdY1IdXpj1J2+g7gqzxoq5hpplzKG1rj w1WR1Y/83NNBgPaIt91gT4b4JdYf27w5rrPSJDG4Wblhyaypo5MOGGij8a7m6jT1EU4qOnE WxI+RBQBWKUMHIY+GQYAdqbhXSdjKF7/9KZoLJ1LlKfxN+5a7u0PqoVSwNJ0FR/GB6M6DDf xq2cNNdlLw9TB0+MEltRA== X-UI-Out-Filterresults: notjunk:1;V03:K0:Z4qCFm3X6ts=:tE0LYddfyeMyARB0+HN+N7 V4S/ZxVv0r/GKADlYF2UkWqdflqj2kIpgXuApr6PALMxwUe/flEeQZpuNffSUBnoQTZnryq4g 89CNK2YGk8l5lwck+deV+E+fsclC9z6TuVKqCZtLdHwwtjTCh4G955cYctvLKvFLNDlcT3aZ6 lxrQFWxt/aq/gjsgGeDd8z9n8AifbHOMrP62vgQeN/0A4E6ktisuM9YPCJj8D/p+b59/jFOe7 xD1eJAZ6nYwsKE0LeKFTv7DzL266J/x958RUYI5JfyrYiEH7BQebx2Ooelqy+HoZVTUyjNW9T hrreQkWR6AnlAnQ7e7aHV44zumDeZ1Y7OmjERZ8OjobZac3p7emjPX/HNXqzhZYQhLyVVDOI9 JWYIo9s/mVPhw3SwNA/5zbsx4OiKClv/tZfIbCrvOhBwpa5srz4kJfGJNpepUZARPdwBWhFzW qqdS3DkDhmrAdOfp4xRXLxH8P3NQugkxq4S4YVOarXiblg+pvWCp8v2WV5jJKAvn/ExQenBdh p6Vr9HljPxNZVEkXbyoNC9CSq5Ldq0KQOM381bRi0fvajoN68gtNe2WwWgePMqcNOiUejDD0m iZw5yMNrThofgPQMGfUJTaFBwC5Gbs2+XzlzRoAUmyyHsyS+M5xV8HELUB3NrzvSYwTtOu/UQ bfzQ= X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 217.72.192.78 X-Spam-Score: -1.4 (-) X-Debbugs-Envelope-To: submit Cc: Arne Babenhauserheide X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: arne_bab@web.de Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.4 (--) * guix/profiles.scm (manual-database): par-map over the entries. This distributes the load roughly equally over all cores and avoids blocking on I/O. The order of the entries stays the same since write-mandb-database s= orts them. =2D-- guix/profiles.scm | 20 ++++++++++++-------- 1 file changed, 12 insertions(+), 8 deletions(-) diff --git a/guix/profiles.scm b/guix/profiles.scm index f5c863945c..374f0f8a90 100644 =2D-- a/guix/profiles.scm +++ b/guix/profiles.scm @@ -1312,15 +1312,11 @@ the entries in MANIFEST." #~(begin (use-modules (guix man-db) (guix build utils) + (ice-9 threads) (srfi srfi-1) (srfi srfi-19)) - (define (compute-entries) - ;; This is the most expensive part (I/O and CPU, due to - ;; decompression), so report progress as we traverse INPUTS= . - (let* ((inputs '#$(manifest-inputs manifest)) - (total (length inputs))) - (append-map (lambda (directory count) + (define (compute-entry directory count total) (format #t "\r[~3d/~3d] building list of \ man-db entries..." count total) @@ -1330,8 +1326,16 @@ man-db entries..." (if (directory-exists? man) (mandb-entries man) '()))) - inputs - (iota total 1)))) + + (define (compute-entries) + ;; This is the most expensive part (I/O and CPU, due to + ;; decompression), so report progress as we traverse INPUTS= . + (let* ((inputs '#$(manifest-inputs manifest)) + (total (length inputs))) + (apply append (par-map compute-entry + inputs + (iota total 1) + (make-list total total))))) (define man-directory (string-append #$output "/share/man")) =2D- 2.22.0 From debbugs-submit-bounces@debbugs.gnu.org Mon Jul 15 12:12:44 2019 Received: (at 36630) by debbugs.gnu.org; 15 Jul 2019 16:12:44 +0000 Received: from localhost ([127.0.0.1]:48839 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hn3am-0007kC-0Y for submit@debbugs.gnu.org; Mon, 15 Jul 2019 12:12:44 -0400 Received: from eggs.gnu.org ([209.51.188.92]:45191) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hn3ae-0007jj-Sk for 36630@debbugs.gnu.org; Mon, 15 Jul 2019 12:12:41 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:51429) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hn3aZ-00023b-5Z; Mon, 15 Jul 2019 12:12:31 -0400 Received: from [2001:660:6102:320:e120:2c8f:8909:cdfe] (port=40746 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1hn3aY-0007pt-Km; Mon, 15 Jul 2019 12:12:30 -0400 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Arne Babenhauserheide Subject: Re: [bug#36630] [PATCH] guix: parallelize building the manual-database References: <20190712214245.23857-1-arne_bab@web.de> Date: Mon, 15 Jul 2019 18:12:27 +0200 In-Reply-To: <20190712214245.23857-1-arne_bab@web.de> (Arne Babenhauserheide's message of "Fri, 12 Jul 2019 23:42:45 +0200") Message-ID: <878sszl1jo.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 36630 Cc: 36630@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hi Arne, Arne Babenhauserheide skribis: > * guix/profiles.scm (manual-database): par-map over the entries. This > distributes the load roughly equally over all cores and avoids blocking on > I/O. The order of the entries stays the same since write-mandb-database = sorts > them. I would think the whole process is largely I/O-bound. Did you try measuring differences? I picked the manual-database derivation returned for: guix environment --ad-hoc jupyter python-ipython python-ipykernel -n (It has 3,046 entries.) On a SSD and with a hot cache, on my 4-core laptop, I get 74s with =E2=80=98master=E2=80=99, and 53s with this patch. However, it will definitely not scale linearly, so we should probably cap at 2 or 4 threads. WDYT? Another issue with the patch is that the [n/total] counter does not grow monotically now: it might temporally go backwards. Consequently, at -v1, users will see a progress bar that hesitates and occasionally goes backward, which isn=E2=80=99t great. This would need to fix it with a mutex-protected global counter. All in all, I=E2=80=99m not sure this is worth the complexity. WDYT? Thanks, Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Mon Jul 15 19:33:04 2019 Received: (at 36630) by debbugs.gnu.org; 15 Jul 2019 23:33:04 +0000 Received: from localhost ([127.0.0.1]:49391 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hnASu-0008PX-8O for submit@debbugs.gnu.org; Mon, 15 Jul 2019 19:33:04 -0400 Received: from mout.web.de ([217.72.192.78]:53605) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hnASs-0008Ou-Dn for 36630@debbugs.gnu.org; Mon, 15 Jul 2019 19:33:03 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=web.de; s=dbaedf251592; t=1563233573; bh=t2olDhcKTruwjH+4XQInq7SYnJpZsx1v0zsRS5Ck75w=; h=X-UI-Sender-Class:References:From:To:Cc:Subject:In-reply-to:Date; b=mf20X3huvBVfsrsIkWDoU6PQw9oOgUfZ8VIj+AvFvITHA7Zv5tOnl6vc3wClmwEHu vbmWJFbrC9D0h3Rh0FvFVbe1apgMwFOPR2OiJDRof5uehND4x2SK40dBLSYQFCQdnq u9Re55niSI+AdDMdidLLLpRr0pxJCvM7Ta0OCfJQ= X-UI-Sender-Class: c548c8c5-30a9-4db5-a2e7-cb6cb037b8f9 Received: from fluss ([84.149.83.171]) by smtp.web.de (mrweb103 [213.165.67.124]) with ESMTPSA (Nemesis) id 0LmuAa-1iNbkP0b08-00h8uj; Tue, 16 Jul 2019 01:32:53 +0200 References: <20190712214245.23857-1-arne_bab@web.de> <878sszl1jo.fsf@gnu.org> User-agent: mu4e 1.2.0; emacs 26.2 From: Arne Babenhauserheide To: Ludovic =?utf-8?Q?Court=C3=A8s?= Subject: Re: [bug#36630] [PATCH] guix: parallelize building the manual-database In-reply-to: <878sszl1jo.fsf@gnu.org> Date: Tue, 16 Jul 2019 01:32:46 +0200 Message-ID: <877e8ig9gh.fsf@web.de> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" X-Provags-ID: V03:K1:bkrVH6QjhXkXv1Wpnuww+iiwTSjJcnwRnom0S8WaiUT8DVm44Ey 1YNfHajak8BqHsb7l262XinfTmoTUC5SHTNGbrEYQ7sTivyEGHZ9xlwV2Bt+XcLkMerjdJq lnBoUsonVnfzeN1xyTBxEUz3/2mxxJvd8GjZ4L+ycxrKmwCSPLMIQ//IgPuJ3FTAzUCpudu EupC/FfQY/EDBYzdcyCGQ== X-Spam-Flag: NO X-UI-Out-Filterresults: notjunk:1;V03:K0:ONnhTmU/Bqk=:VLTjzf3RwY0r1BmNYkLdzV vCM0rmghWe6sOG0SIqE1vk9Dt6BUBUnbecdyl4ulBpaqQP/eHbWErluXGA7UbGc5ntqNfaEFd ukuarguqqsV62+2gzhaSSTpJ0kFEvACKhvr7M3V4D2V460/4ADR4Li/LwFYPOoj1ozTqiHLs/ PSpVHqNnxrXRdRqaMcSQn+TVjex5huqXl2pA/NpDVLJ0pFkubvSZS17avhHj+h/alb6rpB3R+ R3KkGrHeCDuo0fKklBzP8fgZXQAMaeZ6iuA67CrO9bVAauGWcyUEg/PrmdYg40DSJSxHRgGRd Xl6S3NRU6KYCPH0KQw7R7oONjdcAlR2RqoRcVAyoABsdT4hpZB4CqLHjNPHO6KhJ5+O43OGxm S3rnhFcypqq6FgpHPpC036aq9nBRL0fJJfjz3rsgT6RrUcahbP95K3ubDwnHXhx+ia3PADK2Q VB40betV5x66AaBz5cPrd9cSm9Yqt9hZjHCgL4WYecFNtE1eDWRqsCKahiMWaclgkLf0mutmH uEBBKvfJsMTnk00L2XUHRxuuGfLip+IP9mBo0Kv1VmELu4eXxJDsXTyWgS0l7TEW7Q+pdcYVV dgjQo0dYtVaEUu7NeUOuyUxFj71TehRVK5phKehIKO4FcoRmFlhr60PzQtlXi8UQKLZmwJotM 8tkc= X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 36630 Cc: 36630@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hi Ludo=E2=80=99, Ludovic Court=C3=A8s writes: >> * guix/profiles.scm (manual-database): par-map over the entries. This >> distributes the load roughly equally over all cores and avoids blocking = on >> I/O. The order of the entries stays the same since write-mandb-database= sorts >> them. > > I would think the whole process is largely I/O-bound. Did you try > measuring differences? I did not measure the difference in build-time, but I did check the system load. Without this patch, one of my cores is under full load. With this patch all 12 hyperthreads have a mean load of 50%. > I picked the manual-database derivation returned for: > guix environment --ad-hoc jupyter python-ipython python-ipykernel -n > (It has 3,046 entries.) How exactly did you run the derivation? I=E2=80=99d like to check it if you= can give me the exact commandline to run (a command I can run repeatedly). > On a SSD and with a hot cache, on my 4-core laptop, I get 74s with > =E2=80=98master=E2=80=99, and 53s with this patch. I=E2=80=99m using a machine with 6 physical cores, hyperthreading, and an N= VMe M.2 disk, so it is likely that it would not be disk-bound for me at 4 threads. > However, it will definitely not scale linearly, so we should probably > cap at 2 or 4 threads. WDYT? Looking at the underlying action, this seems to be a task that scales pretty well. It just unpacks files into the disk-cache. It should also not consume much memory, so I don=E2=80=99t see a reason to artificially limit the number of threads. > Another issue with the patch is that the [n/total] counter does not grow > monotically now: it might temporally go backwards. Consequently, at > -v1, users will see a progress bar that hesitates and occasionally goes > backward, which isn=E2=80=99t great. It typically jumps forward in the beginning and then stalls until the first manual page is finished. Since par-map uses a global queue of futures to process, and since the output is the first part of (compute-entry =E2=80=A6), I don=E2=80=99t expe= ct the progress to move backwards in ways a user sees: It could only move backwards during the initial step where all threads start at the same time, and there the initial output should be overwritten fast enough to not be noticeable. > This would need to fix it with a mutex-protected global counter. A global counter would be pretty bad for scaling. As it is, this code needs no communication between processes besides returning the final result, so it behaves exactly like a normal map, aside from being faster. So I=E2=80=99d prefer to accept the forward-jumping. > All in all, I=E2=80=99m not sure this is worth the complexity. > > WDYT? Given that building manual pages is the most timeconsuming part when installing a small tool into my profile, I think it is worth the complexity. Especially because most of the complexity is being taken care of by (ice-9 threads par-map). Best wishes, Arne =2D- Unpolitisch sein hei=C3=9Ft politisch sein ohne es zu merken --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEE801qEjXQSQPNItXAE++NRSQDw+sFAl0tDSAACgkQE++NRSQD w+uuzQ//SIjIPq5kpaNC/3EgntRa5nIt+yKfT23K2juzCaYCAVUjnwHw4IXCCcSi c2vX8heNXMFcLeML65E6ldao72WejeJ5tyXdbwjvexIQgvddUvG5vXpJfVVnrfPb JPsjgaNj3DWdZRdq1lTMghlIpGrnRym+XIlzMB4BFHaiI1V5mBJ6IUk3ICddSqQo QxOvEcI3oHuDh6C55UkQrrClcO0KDYNxG631EXB6/jCLNd7UzSq3DKtQNmHGRiQq JyN2g29XipNyFp5HZ284FlegBoiO/OWseZoJ9s61KaWb9TKHeDeb7j4Yroj+MHxc lpYoD58hVyJHb0Cgy0PtWIaQIa6WwWVG1OGStDKTTkLD5a8aZqwk0I8luuzvftV2 NdZYf047HYpbABQuVR+VLpS+ofkRy5GJCpSgJwpYzko8aI71LWDexipIs34LJTEE RJ2QZ8WM7vUGBFRqjztOi0kxOYArRAkBR+aBY+dTOG+DAU/5Bitm8goVxuq9THob MZQeRRCC4b/KC4lMB3QkREXf2U6FfLOpC0Y0+OHQMV4WBs/5Wf+3Qhqa3oy/NIZW oodu4nAV4sytcdT9lVJuJqXmW6Vqy1UdcDry/FQbUjocJa4HN/BoRg2q5C5TL9Om +B4CEXoxjfa7KLEL1dxEEqt+xcXajX6ys1t14FELqxr/OKJGdYmIswQBAQgAHRYh BN0ovebZh1yrzkqLHdzPDbMLwQVIBQJdLQ0gAAoJENzPDbMLwQVICe0EAJarRMWw leLwx4Q6quii/FUJK85wPqbB3cAPupTyS2v9pso/QIGkd2ku/NkksWEowa1PbhRx V1PhD1AVFZ8aW1eQjz5aeh49g1udv0+IrOzetdm9Bzsv+zuweswzhZUJXgM1MMni ILP1PcMcQEMPcYkC1+BiYOxZMgEip+3U72fN =883H -----END PGP SIGNATURE----- --=-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Tue Jul 16 17:14:57 2019 Received: (at 36630) by debbugs.gnu.org; 16 Jul 2019 21:14:57 +0000 Received: from localhost ([127.0.0.1]:51256 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hnUmn-00088l-9x for submit@debbugs.gnu.org; Tue, 16 Jul 2019 17:14:57 -0400 Received: from eggs.gnu.org ([209.51.188.92]:40415) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hnUml-00088X-W4 for 36630@debbugs.gnu.org; Tue, 16 Jul 2019 17:14:56 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:46432) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hnUmg-0002WG-OS; Tue, 16 Jul 2019 17:14:50 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=60040 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1hnUmg-0002tl-AN; Tue, 16 Jul 2019 17:14:50 -0400 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Arne Babenhauserheide Subject: Re: [bug#36630] [PATCH] guix: parallelize building the manual-database References: <20190712214245.23857-1-arne_bab@web.de> <878sszl1jo.fsf@gnu.org> <877e8ig9gh.fsf@web.de> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 28 Messidor an 227 de la =?utf-8?Q?R=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Tue, 16 Jul 2019 23:14:48 +0200 In-Reply-To: <877e8ig9gh.fsf@web.de> (Arne Babenhauserheide's message of "Tue, 16 Jul 2019 01:32:46 +0200") Message-ID: <87zhldel6f.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 36630 Cc: 36630@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hello, Arne Babenhauserheide skribis: > Ludovic Court=C3=A8s writes: [...] >> I picked the manual-database derivation returned for: >> guix environment --ad-hoc jupyter python-ipython python-ipykernel -n >> (It has 3,046 entries.) > > How exactly did you run the derivation? I=E2=80=99d like to check it if y= ou can > give me the exact commandline to run (a command I can run repeatedly). If you run the command above, it=E2=80=99ll list /gnu/store/=E2=80=A6-manual-database.drv. So you can just run: guix build /gnu/store/=E2=80=A6-manual-database.drv or: guix build /gnu/store/=E2=80=A6-manual-database.drv --check if it had already been built before. >> On a SSD and with a hot cache, on my 4-core laptop, I get 74s with >> =E2=80=98master=E2=80=99, and 53s with this patch. > > I=E2=80=99m using a machine with 6 physical cores, hyperthreading, and an= NVMe > M.2 disk, so it is likely that it would not be disk-bound for me at 4 > threads. The result may be entirely different with a spinning disk. :-) I=E2=80=99m not saying we should optimize for spinning disks, just that wha= t you see is at one end of the spectrum. >> However, it will definitely not scale linearly, so we should probably >> cap at 2 or 4 threads. WDYT? > > Looking at the underlying action, this seems to be a task that scales > pretty well. It just unpacks files into the disk-cache. > > It should also not consume much memory, so I don=E2=80=99t see a reason to > artificially limit the number of threads. On a many-core machine like we have in our build farm, with spinning disks, I believe that using one thread per core would be counterproductive. >> Another issue with the patch is that the [n/total] counter does not grow >> monotically now: it might temporally go backwards. Consequently, at >> -v1, users will see a progress bar that hesitates and occasionally goes >> backward, which isn=E2=80=99t great. > > It typically jumps forward in the beginning and then stalls until the > first manual page is finished. > > Since par-map uses a global queue of futures to process, and since the > output is the first part of (compute-entry =E2=80=A6), I don=E2=80=99t ex= pect the > progress to move backwards in ways a user sees: It could only move > backwards during the initial step where all threads start at the same > time, and there the initial output should be overwritten fast enough to > not be noticeable. Hmm, maybe. I=E2=80=99m sure we=E2=80=99ll get reports saying this looks w= eird and Something Must Absolutely Be Done About It. :-) But anyway, another issue is that we would need to honor =E2=80=98parallel-job-count=E2=80=99, which means using =E2=80=98n-par-map= =E2=80=99, which doesn=E2=80=99t use futures. > Given that building manual pages is the most timeconsuming part when > installing a small tool into my profile, I think it is worth the > complexity. Especially because most of the complexity is being taken > care of by (ice-9 threads par-map). Just today I realized that the example above (with Jupyter) has so many entries because of propagated inputs; in particular libxext along brings 1,000+ man pages. We should definitely do something about these packages. Needs more thought=E2=80=A6 Thanks, Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Wed Jul 17 18:06:54 2019 Received: (at 36630) by debbugs.gnu.org; 17 Jul 2019 22:06:54 +0000 Received: from localhost ([127.0.0.1]:53120 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hns4c-000162-0s for submit@debbugs.gnu.org; Wed, 17 Jul 2019 18:06:54 -0400 Received: from mout.web.de ([212.227.15.4]:51039) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hns4a-00015m-2U for 36630@debbugs.gnu.org; Wed, 17 Jul 2019 18:06:52 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=web.de; s=dbaedf251592; t=1563401203; bh=cDHhpIPNoBGX5FC90euBJCs+zrFX24Gq8Gf+RqRemEY=; h=X-UI-Sender-Class:References:From:To:Cc:Subject:In-reply-to:Date; b=nxlQ8aUsKLuHtp1e2BnYGSqNbfWO2juxNFPTKRyTjMFj2Ps/+4znKyGG/OnN6pTy4 lkrd+HvOJGmIecv7ZRblsX3ZGTlPtl6h/jfRQw0bk+Ow6eCatvfB1Y6liORfg0pAtt 7AxDCFPdMaWbJlvV+EdKR1IdRvGhJFsnjQNss5Dk= X-UI-Sender-Class: c548c8c5-30a9-4db5-a2e7-cb6cb037b8f9 Received: from fluss ([84.149.83.171]) by smtp.web.de (mrweb004 [213.165.67.108]) with ESMTPSA (Nemesis) id 0Lgs1W-1iGtsr1Yc2-00oHlL; Thu, 18 Jul 2019 00:06:43 +0200 References: <20190712214245.23857-1-arne_bab@web.de> <878sszl1jo.fsf@gnu.org> <877e8ig9gh.fsf@web.de> <87zhldel6f.fsf@gnu.org> User-agent: mu4e 1.2.0; emacs 26.1 From: Arne Babenhauserheide To: Ludovic =?utf-8?Q?Court=C3=A8s?= Subject: Re: [bug#36630] [PATCH] guix: parallelize building the manual-database In-reply-to: <87zhldel6f.fsf@gnu.org> Date: Thu, 18 Jul 2019 00:06:36 +0200 Message-ID: <878ssw8ger.fsf@web.de> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" X-Provags-ID: V03:K1:FQNI/0k6ZfN9fqnPlinpQi5oU2M1Gt6S/a132fgqXpbDfR/DvXe KdhQtY11+eYC11M0SnaBZPmc8Lk5kMhmxAo5O1Jw1hocSDAVu+iwgw2aC6gcGSGCKIpvGEW fN0ei5VxFShYjc5zhC1ucm/qJxQo2m3Kem/U2kEYNwlKDyz+hHlJgGS07hFbJN5M4uPl2Ik Xb3lieSW86an0Gb4gAmkw== X-Spam-Flag: NO X-UI-Out-Filterresults: notjunk:1;V03:K0:WVi4h1ny/wk=:ynqdMADmxec9CxQ7fRJTym PoSPtJKJwkGeFU3OgUlBGlDPVc7CqzpArBKg130kM1LSuZMXxwQYrVnTenc77DoC3Kvud5ukl 7vlJdfVBBmNr+IApCJu2ub9Ew4w/L9wUd89Wdqucq0Ol8K1XEMLd82Xivl/W+1PoyaFJM5UPX YSl9+UZPOWNCu0CtlGwTOzU3UkQu26mNcho2RGofCbgnIw7wHC5YkHasGze0CvwwpXdkWZKj1 WSBZwmtORrR/v8q2FutlNtMRvfntfbGo5KrMAmZ8dYEAARYBhJywo/Xaca/ufTRvWWlSvep/g TsAdFJ1C8MjEzqJAhbPoDObyv6YcFFDFBXrbzP7bS71+LyCtyOASVXQFKu52RtlwhsbbTOagf zTbNKkWQX4PI5BVuI4W+KHIdvRUWpJ5BzjrL7cigVRmO14JHsG7zDNgemqUhVwgfoRylq2wmY m5PqOH5zwFRPYx4JSfWwokbVUvnZ49ggtYay68cBc+uAj7m1Ek4NBJNHZiF5ZqTgM69f2WUHc 0TSu+nLJvGxO0zDGve6Css1QkxqyY8mHIxxEezyLTVwomyUCW5TWUEnlRn6UxED2WCcls8ier aBBSWK9bf2IX6HXkkwzXqC32NFlgAMS1uPMW6z9o+oyRtzYe95UrANASJ4TaRPQNrD50jrpDF o3sY= X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 36630 Cc: 36630@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hi, Ludovic Court=C3=A8s skribis: > Arne Babenhauserheide skribis: Offtopic: I love reading Esperanto here! >> Ludovic Court=C3=A8s writes: >>> guix environment --ad-hoc jupyter python-ipython python-ipykernel -n >> How exactly did you run the derivation? > If you run the command above, it=E2=80=99ll list > /gnu/store/=E2=80=A6-manual-database.drv. So you can just run: > > guix build /gnu/store/=E2=80=A6-manual-database.drv > > or: > > guix build /gnu/store/=E2=80=A6-manual-database.drv --check > > if it had already been built before. Somehow I can=E2=80=99t get guix to actually run my changed code with this command, so I=E2=80=99m not sure I tested the right thing. What is the clean approach to run the profile.scm from git? >>> On a SSD and with a hot cache, on my 4-core laptop, I get 74s with >>> =E2=80=98master=E2=80=99, and 53s with this patch. >> >> I=E2=80=99m using a machine with 6 physical cores, hyperthreading, and a= n NVMe >> M.2 disk, so it is likely that it would not be disk-bound for me at 4 >> threads. > > The result may be entirely different with a spinning disk. :-) > > I=E2=80=99m not saying we should optimize for spinning disks, just that w= hat you > see is at one end of the spectrum. That=E2=80=99s right, yes. > But anyway, another issue is that we would need to honor > =E2=80=98parallel-job-count=E2=80=99, which means using =E2=80=98n-par-ma= p=E2=80=99, which doesn=E2=80=99t use > futures. Ouch, yes. That=E2=80=99s an issue =E2=80=A6=20 Thank you for bringing it up! Best wishes, Arne =2D-=20 Unpolitisch sein hei=C3=9Ft politisch sein ohne es zu merken --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEE801qEjXQSQPNItXAE++NRSQDw+sFAl0vm+4ACgkQE++NRSQD w+uICBAAwiQygjoFRmEjJDl6Y8Ny6AdeTGmLXfOVb2FH9s/FEF2zhJtLf0atKWUN EDqoj5mzqQ1IEoyKhcgzBbw/kWdx/eNj4qp/yJpj/Tp5z39Rr5QZFtbgX+oEQUur d1T4zgCb2tpll93W3IC4BKGNtTBbubYM0SlPxlELU+x0yGRKTccYfSaYIHynWgC1 3Ij5tYZoU34m1Tk29H5AGdjlPjMfF5oNapmwwNk/JGlUWtg9Vaquf1S9HFqOcn3d mezgEMz22wyXOdMXpUkuIxjwtvwJ5qvbxb0mlfjejt3eiShl0R8eThYhebzKgDNq DTZ//3XLGaSf6Ih99Up4E03HUXOhZBCdQfM8aucU9LUR0okMGm5N9Nc+C4v0AI/N hWzyuLC0wTKBdDLOqBJx5WTUBvKnuKIGKwOq0n3PQYVzVhye/eLFdTMAojPMXWaI CK7edZP5HzKHnrRWxeTIIGkUrcccfQa1adVDH2d7xF1MzOSTAhkTSPGKGW7EZ2Ce aeu0/DyGY8LGQD55sAoIeSKKfRkt1zkCA/AYsLC4ZkhuwdfoLQ9kvgp/DqvorUKl bDMgFoMcDOo0mBHm409N5HQudca10FYPuaw4LwBIbKKxebY5Iiv75NA3WFehCDOs ub4EX4C5lraADIqf9NY1AYo5CNYHpiegaeuA0jWT7jfEcNXCkXWIswQBAQgAHRYh BN0ovebZh1yrzkqLHdzPDbMLwQVIBQJdL5vuAAoJENzPDbMLwQVIWr8D/1A6Kdyf dQOA+QugrHNqVeYC5MqT/GvisnlikfM0N1mU+KrNX59CnoHZetKzKrYS8dh07eOU 3Ad3tdvSCErAQoUIhP6zACDeSudAu4ofuCxI9b5aQM4hNvDdMRQjO3VV+wSEA6a6 qMI4FopRdngMbwtxqrZu/bE4kt5R+4PCiqrO =Uob2 -----END PGP SIGNATURE----- --=-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Thu Jul 18 04:58:25 2019 Received: (at 36630) by debbugs.gnu.org; 18 Jul 2019 08:58:25 +0000 Received: from localhost ([127.0.0.1]:53377 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ho2F6-0004Jx-Um for submit@debbugs.gnu.org; Thu, 18 Jul 2019 04:58:25 -0400 Received: from eggs.gnu.org ([209.51.188.92]:48106) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ho2F5-0004Jd-JW for 36630@debbugs.gnu.org; Thu, 18 Jul 2019 04:58:23 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:50521) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ho2EG-0008Bt-U8; Thu, 18 Jul 2019 04:58:18 -0400 Received: from nat-eduroam-36-gw-01-bso.bordeaux.inria.fr ([194.199.1.36]:58448 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1ho2EG-00085K-8i; Thu, 18 Jul 2019 04:57:32 -0400 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Arne Babenhauserheide Subject: Re: [bug#36630] [PATCH] guix: parallelize building the manual-database In-Reply-To: <878ssw8ger.fsf@web.de> (Arne Babenhauserheide's message of "Thu, 18 Jul 2019 00:06:36 +0200") References: <20190712214245.23857-1-arne_bab@web.de> <878sszl1jo.fsf@gnu.org> <877e8ig9gh.fsf@web.de> <87zhldel6f.fsf@gnu.org> <878ssw8ger.fsf@web.de> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.2 (gnu/linux) X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 30 Messidor an 227 de la =?utf-8?Q?R=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Thu, 18 Jul 2019 10:57:30 +0200 Message-ID: <875znzhg91.fsf@gnu.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 36630 Cc: 36630@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hello, Arne Babenhauserheide skribis: > Ludovic Court=C3=A8s skribis: >> Arne Babenhauserheide skribis: > Offtopic: I love reading Esperanto here! >>> Ludovic Court=C3=A8s writes: > >>>> guix environment --ad-hoc jupyter python-ipython python-ipykernel -n > >>> How exactly did you run the derivation? > >> If you run the command above, it=E2=80=99ll list >> /gnu/store/=E2=80=A6-manual-database.drv. So you can just run: >> >> guix build /gnu/store/=E2=80=A6-manual-database.drv >> >> or: >> >> guix build /gnu/store/=E2=80=A6-manual-database.drv --check >> >> if it had already been built before. > > Somehow I can=E2=80=99t get guix to actually run my changed code with this > command, so I=E2=80=99m not sure I tested the right thing. Did you try the =E2=80=98guix environment -n=E2=80=99 command above? Doesn= =E2=80=99t it show the manual-database.drv? Alternately, you can also do something like: guix install -p /tmp/foo jupyter python-ipython python-ipykernel -n HTH, Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Thu Jul 18 04:58:25 2019 Received: (at 36630) by debbugs.gnu.org; 18 Jul 2019 08:58:25 +0000 Received: from localhost ([127.0.0.1]:53379 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ho2F7-0004Jz-7L for submit@debbugs.gnu.org; Thu, 18 Jul 2019 04:58:25 -0400 Received: from eggs.gnu.org ([209.51.188.92]:48104) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ho2F5-0004Jc-Jd for 36630@debbugs.gnu.org; Thu, 18 Jul 2019 04:58:23 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:50522) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ho2EM-0008Dd-Q6; Thu, 18 Jul 2019 04:58:18 -0400 Received: from nat-eduroam-36-gw-01-bso.bordeaux.inria.fr ([194.199.1.36]:58410 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1ho2Cs-0007km-BH; Thu, 18 Jul 2019 04:57:38 -0400 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Arne Babenhauserheide Subject: Re: [bug#36630] [PATCH] guix: parallelize building the manual-database References: <20190712214245.23857-1-arne_bab@web.de> <878sszl1jo.fsf@gnu.org> <877e8ig9gh.fsf@web.de> <87zhldel6f.fsf@gnu.org> <878ssw8ger.fsf@web.de> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 30 Messidor an 227 de la =?utf-8?Q?R=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Thu, 18 Jul 2019 10:55:19 +0200 In-Reply-To: <878ssw8ger.fsf@web.de> (Arne Babenhauserheide's message of "Thu, 18 Jul 2019 00:06:36 +0200") Message-ID: <877e8fhgco.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 36630 Cc: 36630@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hello, Arne Babenhauserheide skribis: > Ludovic Court=C3=A8s skribis: >> Arne Babenhauserheide skribis: > Offtopic: I love reading Esperanto here! >>> Ludovic Court=C3=A8s writes: > >>>> guix environment --ad-hoc jupyter python-ipython python-ipykernel -n > >>> How exactly did you run the derivation? > >> If you run the command above, it=E2=80=99ll list >> /gnu/store/=E2=80=A6-manual-database.drv. So you can just run: >> >> guix build /gnu/store/=E2=80=A6-manual-database.drv >> >> or: >> >> guix build /gnu/store/=E2=80=A6-manual-database.drv --check >> >> if it had already been built before. > > Somehow I can=E2=80=99t get guix to actually run my changed code with this > command, so I=E2=80=99m not sure I tested the right thing. Did you try the =E2=80=98guix environment -n=E2=80=99 command above? Doesn= =E2=80=99t it show the manual-database.drv? Alternately, you can also do something like: guix install -p /tmp/foo jupyter python-ipython python-ipykernel -n HTH, Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Thu Jul 18 06:59:42 2019 Received: (at 36630) by debbugs.gnu.org; 18 Jul 2019 10:59:42 +0000 Received: from localhost ([127.0.0.1]:53489 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ho48U-00015s-46 for submit@debbugs.gnu.org; Thu, 18 Jul 2019 06:59:42 -0400 Received: from mout.web.de ([217.72.192.78]:38149) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ho48S-00015f-7q for 36630@debbugs.gnu.org; Thu, 18 Jul 2019 06:59:40 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=web.de; s=dbaedf251592; t=1563447571; bh=HcZzKeJnT1IV+bkjllI29jtpYhBxxhVyP3TCEkCeE/g=; h=X-UI-Sender-Class:References:From:To:Cc:Subject:In-reply-to:Date; b=IcN/L76XUkO9lDMQm0A+CuXHT8nB+VptXyAJEotJRzenWzU0aDRiNOZzeVFVkcLhj XIbvv+2ZDkMZjmuqQVPGgGnd16E8mtOMYZFd5J0rX7DwRnR0N0nRQGf3pSrEXK2XuO qnYguGNe3Gm5KR+OEM2sbp03aqqAqxQyfxUBpU5Y= X-UI-Sender-Class: c548c8c5-30a9-4db5-a2e7-cb6cb037b8f9 Received: from fluss ([84.149.83.171]) by smtp.web.de (mrweb103 [213.165.67.124]) with ESMTPSA (Nemesis) id 0LjJH1-1iRXYL3fbD-00daZu; Thu, 18 Jul 2019 12:59:31 +0200 References: <20190712214245.23857-1-arne_bab@web.de> <878sszl1jo.fsf@gnu.org> <877e8ig9gh.fsf@web.de> <87zhldel6f.fsf@gnu.org> <878ssw8ger.fsf@web.de> <875znzhg91.fsf@gnu.org> User-agent: mu4e 1.2.0; emacs 26.1 From: Arne Babenhauserheide To: Ludovic =?utf-8?Q?Court=C3=A8s?= Subject: Re: [bug#36630] [PATCH] guix: parallelize building the manual-database In-reply-to: <875znzhg91.fsf@gnu.org> Date: Thu, 18 Jul 2019 12:59:23 +0200 Message-ID: <87k1cfpq0k.fsf@web.de> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" X-Provags-ID: V03:K1:aE3uPQTK6sDvqGJ+OnjsPAO6yJqND8GOL2bTawbtuSwWdhCKJc2 HBvsdFMW8TXxt1FKD4GPmc+7fikwa6wb9ap+ggfsp4fn7OCisVdxLOFFa4Y0x5mAYLoTmwY 9T7mBp96sjgmNa04WTIuZDlHFmJjMEC7YRUkWZsrFqjl9Ms/K+8zZNusqcqU+RlXPdhCQLT t9z/lp6j6Fk3J4JWHpOSw== X-Spam-Flag: NO X-UI-Out-Filterresults: notjunk:1;V03:K0:XNLPMwLmP+I=:nH3gyGvhadNSA3BVmMYXKA V2TlV1py/soZJqSR+nbEXYyw5ZRAANH3mReGtf0XxLLVLgb1i+W63j0Lir8X+aIGEablIK/FG esTnPKjYYhqK7vPtRVt8PXvRahWk768P5jggVk9176Hj3A+6ENo6Sx1ttWUYNiBlaWKnD/aoN VImx4eDjQcQpo5qM9AO2nzaNuFr8DnUiB5OkRqpk4+rqV25c9hN3p8NmaXWKjoJXIn6rhMvfz BdLF+ScKUZ73Q1vvcJIdPMPre45vuMsXGwIR/1h4ny5j04seja73MPslqbIWwm/6YouG/eB5H 9pwugb7NEarSVEFCDioa6VVGYN04CLVRo9oAoT55TH0Vt+Lz396HhOSTjIPi9zA9dXtQZ/iuE 31sRX0rIk8mphD84hXtnCWzHN4HLyrW5hNBr8J081/eAbLBoyw4htj/jmwW00cRGVWSw6/xGN MNQdmCrf+QeskMQUZp6LcAewuwmpNUoW+yOXhyMJ6tnUvIdyOMB9ktiqpKZa8ECcUa4QGBmAC 0yZeO8lncUryczRpPZ21GltrQ29bVa6+MS2wFUzgBFDzb/fgwCycCyOSYGngX96dQSb7Mub1J QsUr+DFdaZaykeF1joc6Su/QvzYjEILuEH/7Y3BkDtZUBEs9ty6KoOZhCgdyP26W9NW4W9tY3 cxEo= X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 36630 Cc: 36630@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Ludovic Court=C3=A8s writes: > Did you try the =E2=80=98guix environment -n=E2=80=99 command above? Doe= sn=E2=80=99t it show > the manual-database.drv? It does show the manual database, but then running guix build /gnu/....drv --check does not run my changed code. I=E2=80=99m doing time ./pre-inst-env guix build /gnu/store/jnkxwwxk71n07fs6naa11fxmg3vpnnb3-= manual-database.drv --check But it runs the installed guix, not the local changes to profile.scm. Best wishes, Arne =2D-=20 Unpolitisch sein hei=C3=9Ft politisch sein ohne es zu merken --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEE801qEjXQSQPNItXAE++NRSQDw+sFAl0wUQ4ACgkQE++NRSQD w+upYhAAueygBlMf1kVpYmi/9zrfTCk2KjSv36H4PxlILes3F0KYg//wyFafMQc+ c/gP3PK2+nj2KNLcTrEF7eUwLwXDhVM7LkXVQSZQSBlaRnTRxxlqwEEkq2YMydeC aEMOaIV1egUD6Chriwl29hE0JUxT4gNxQoo+EbKenK64bouhQmtLHzkZDgLiZld3 dF+k1zWeKNUP/r9Iege6ZYrFTjS5sRqlYRLo9h410meP/j8w3kTA7X64oBqBbcp3 9Q+PfldIh7i+2m5N0cnoI3QxXYK+Z9kmfGHI+1djPCPpJD2Kvi4CGQZLSg6VCMJw x132r0BbKFtmNtFFRicCytofx9Y1CKZ3+xwMU9nwxQ73iKU2raiWRSWYtBelbqRE Mzm272aWPZxPN8Ak1YkpukLYaq9SaxZAfmJ0ZzBD5yS8/vcqUPcNzB0rT/FIQ0V8 U8Apxn2TyjZu1hXjUq8nhwkYyxptX/QDiHiKPIPScBlBnGDcmz6BGZW1jP8rB2A4 tbWpecSY+fyf/JE2DPLAgoRksQvlTimISkIn+FptduYKNxSP1fNqx95cM7T46XML omPGk8VAaZVrXKs2C5nw4JhZgxpCczDrwwUqzwXAUx5+XidBE18+lhZrsJ/cMNH/ WDAOXHWgGlRppW3uxTxSFMccZupIT51V+VTSzWizojVigcD0K+CIswQBAQgAHRYh BN0ovebZh1yrzkqLHdzPDbMLwQVIBQJdMFEOAAoJENzPDbMLwQVImLYD/0IO8lOZ mq65FrAx3IFt9ArzcH3/9vYFVmQGcw9z7BsKQLlLAuFkv93koPnQqEvH+wcRwh8f SJH3wVaZ4Y+WJBvR46PV8OLQq2SHFvIDROTIu6UWeXsyDHZLuD8eneQPFekYt2K9 cYFA4x8vd0uOkxJ2zJ/OoU6EudvWYIJOGpag =asMy -----END PGP SIGNATURE----- --=-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Thu Jul 18 09:46:30 2019 Received: (at 36630) by debbugs.gnu.org; 18 Jul 2019 13:46:30 +0000 Received: from localhost ([127.0.0.1]:53638 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ho6ju-000113-Dk for submit@debbugs.gnu.org; Thu, 18 Jul 2019 09:46:30 -0400 Received: from eggs.gnu.org ([209.51.188.92]:55185) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ho6jr-00010p-4V for 36630@debbugs.gnu.org; Thu, 18 Jul 2019 09:46:27 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:55953) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ho6jl-0002KU-JE; Thu, 18 Jul 2019 09:46:21 -0400 Received: from [2001:660:6102:320:e120:2c8f:8909:cdfe] (port=42776 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1ho6jl-0006TE-2r; Thu, 18 Jul 2019 09:46:21 -0400 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Arne Babenhauserheide Subject: Re: [bug#36630] [PATCH] guix: parallelize building the manual-database References: <20190712214245.23857-1-arne_bab@web.de> <878sszl1jo.fsf@gnu.org> <877e8ig9gh.fsf@web.de> <87zhldel6f.fsf@gnu.org> <878ssw8ger.fsf@web.de> <875znzhg91.fsf@gnu.org> <87k1cfpq0k.fsf@web.de> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 30 Messidor an 227 de la =?utf-8?Q?R=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Thu, 18 Jul 2019 15:46:18 +0200 In-Reply-To: <87k1cfpq0k.fsf@web.de> (Arne Babenhauserheide's message of "Thu, 18 Jul 2019 12:59:23 +0200") Message-ID: <87zhlbe9qt.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 36630 Cc: 36630@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Arne Babenhauserheide skribis: > Ludovic Court=C3=A8s writes: > >> Did you try the =E2=80=98guix environment -n=E2=80=99 command above? Do= esn=E2=80=99t it show >> the manual-database.drv? > > It does show the manual database, but then running guix build > /gnu/....drv --check does not run my changed code. > > I=E2=80=99m doing > > time ./pre-inst-env guix build /gnu/store/jnkxwwxk71n07fs6naa11fxmg3vpnnb= 3-manual-database.drv --check > > But it runs the installed guix, not the local changes to profile.scm. Right, sorry for being unclear: you need to run ./pre-inst-env guix environment -n =E2=80=A6 That will run you modified code and thus create a manual-database.drv that uses your code; it=E2=80=99s this manual-database.drv that you should = pass to =E2=80=98guix build=E2=80=99. HTH, Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Thu Jul 18 18:49:22 2019 Received: (at 36630) by debbugs.gnu.org; 18 Jul 2019 22:49:22 +0000 Received: from localhost ([127.0.0.1]:54808 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hoFDG-0002so-FC for submit@debbugs.gnu.org; Thu, 18 Jul 2019 18:49:22 -0400 Received: from mout.web.de ([212.227.15.4]:44253) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hoFDE-0002sZ-U7 for 36630@debbugs.gnu.org; Thu, 18 Jul 2019 18:49:21 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=web.de; s=dbaedf251592; t=1563490150; bh=8rdjocyH0SL+ZjGXhU68dGzMDUtUNxluyicHnNbUpuk=; h=X-UI-Sender-Class:References:From:To:Cc:Subject:In-reply-to:Date; b=kRRg6NkL3JqtsK0jb38vIyzpG4UnREphY1UEXOXOrPXTL6xjnVJDqooA5XaGuIIju M+D0S+d/D+TLjAXV/e88ONUJW4AG3Lw8pS+ddQSW5N/Nz43f4RIaf1FTO131BZCTuS OpOgFkb9qby35oFRHAtx9TTCNvvJ8rZyb6x9wCn8= X-UI-Sender-Class: c548c8c5-30a9-4db5-a2e7-cb6cb037b8f9 Received: from fluss ([84.149.83.171]) by smtp.web.de (mrweb001 [213.165.67.108]) with ESMTPSA (Nemesis) id 0MNc1S-1hmXvw1zPW-007B7z; Thu, 18 Jul 2019 22:03:18 +0200 References: <20190712214245.23857-1-arne_bab@web.de> <878sszl1jo.fsf@gnu.org> <877e8ig9gh.fsf@web.de> <87zhldel6f.fsf@gnu.org> <878ssw8ger.fsf@web.de> <875znzhg91.fsf@gnu.org> <87k1cfpq0k.fsf@web.de> <87zhlbe9qt.fsf@gnu.org> User-agent: mu4e 1.2.0; emacs 26.1 From: Arne Babenhauserheide To: Ludovic =?utf-8?Q?Court=C3=A8s?= Subject: Re: [bug#36630] [PATCH] guix: parallelize building the manual-database In-reply-to: <87zhlbe9qt.fsf@gnu.org> Date: Thu, 18 Jul 2019 22:03:03 +0200 Message-ID: <87d0i7p0ug.fsf@web.de> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" X-Provags-ID: V03:K1:hVHqX+XcTIQtaBVXm++v/uzGbo4/4JtVGhXa3fyVcrUOkQEugxt 2eXM8OvbEM2Wrx+XtvBQ7EVacW91T9PF2RqlW4Ts7AeoMRoVMV2d/rhelUJgWPaPCZt0Lo1 upWYhf1a6srfKcPGsccZXDdHo8dqLGIjzuc0HNtoAuqiA9KbjSRsF9jL6/7NvgW40vBW8YF LSI6PcRbE5ksy8uXY3uFw== X-Spam-Flag: NO X-UI-Out-Filterresults: notjunk:1;V03:K0:/EO14+QPL1c=:nwONDV7YFty9dHgRZ1gcUP piIC6zDIKE+q+IyVTlJosDuLy5zhLbXuOHFMnbIS9tUHwLTp+kndg3pwWN5/BXej/a91dWA02 pY4IP8UO3cTsps6p08TkTkXaLs9uCk/dJ4G5aWc/+1YGLis2LLvcHVoHaR+dvQqr8sPFNpvJZ BLA985aZticvZcMvwLLfWuB8TDUTgBAKr3uhB39NsFL9EU7P75FmhvYZXhFgKKFAKcuGAByzE mSrs8OpAAUBWQPmwCNMl01Yy+iI5qgK32Mb7PbSRZ4iMUc2/J5zOJdDICV2SpbPRTtLpZljiJ EGI/PxPiYjjK1J1tILTsyEY/76SDWDGIu5aCpDj09gsuLOmHOyhuqEMorN5piOFGfli1cQ0mn LFRfVPTX6mtiJmpd4RN46T3b+HF2NT6/h+B/Q9WUutr+JHFnHcUBytb88nkCiOVvzTtTTmJYf sX1BDFWzdi/jL4Ln7f6I4RxEWzebq/AI/Uj8Oi4TzNvauw1vYEPPpf7Wv4p7HI4VSQWtCYItl QsVkYdGmTBCdHaZtC0eF6U2kHgDsLcukM5WxcRKtDWWVo+KTT9ReIbtVVtSFcCO2NKDIMOqd+ V2FfoOTj4OC7BFgmPBaiaks9PVO7lIiOkuZ4x00RxtMwoW/JTLBE4Ma5cmbftK0yOJWo7nCdz dCAQ= X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 36630 Cc: 36630@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Ludovic Court=C3=A8s writes: > Arne Babenhauserheide skribis: > >> Ludovic Court=C3=A8s writes: >> >>> Did you try the =E2=80=98guix environment -n=E2=80=99 command above? D= oesn=E2=80=99t it show >>> the manual-database.drv? >> >> It does show the manual database, but then running guix build >> /gnu/....drv --check does not run my changed code. >> >> I=E2=80=99m doing >> >> time ./pre-inst-env guix build /gnu/store/jnkxwwxk71n07fs6naa11fxmg3vpnn= b3-manual-database.drv --check >> >> But it runs the installed guix, not the local changes to profile.scm. > > Right, sorry for being unclear: you need to run > > ./pre-inst-env guix environment -n =E2=80=A6 > > That will run you modified code and thus create a manual-database.drv > that uses your code; it=E2=80=99s this manual-database.drv that you shoul= d pass > to =E2=80=98guix build=E2=80=99. That works now =E2=80=94 thank you! With the change: 185552 entries processed in 108.2 s Before the change: 185552 entries processed in 220.1 s The exact commands I=E2=80=99m running: cd Dokumente/Guix/guix git checkout master # with the change ./pre-inst-env guix environment --ad-hoc jupyter python-ipython python-ipyk= ernel -n time guix build /gnu/...-manual-database.drv git checkout 97bf46e64c11c64a968fdb833983ede6bdafbc00 ./pre-inst-env guix environment --ad-hoc jupyter python-ipython python-ipyk= ernel -n time guix build /gnu/...-manual-database.drv So I also see roughly factor 2 speedup, which means a limit to 4 threads should work. (what I saw is that I only see the =E2=80=A6manual-database.drv before I bu= ild it, after it=E2=80=99s built, I no longer see it in the environment output) How do I get the defined limit of cores and threads? Best wishes, Arne =2D-=20 Unpolitisch sein hei=C3=9Ft politisch sein ohne es zu merken --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEE801qEjXQSQPNItXAE++NRSQDw+sFAl0w0HkACgkQE++NRSQD w+sPHhAA0Mr9BAzh9hjHgDq+Ji5KQFRXBCEvHyyQV2TYJBs0TZ04oXmNiHZHSWQp coSGnmnPvtx1aGkXI7edHuuy/rDwKBcUpJtWZLmFiZ/89baFL+hJnRteZSJILj9P DFGFFODAg/2MRyDZQhpMcEx+ea9hZKUJ068aGPW66DkpN35hJoSu/w0qetAqD8Pm +o1WyB0z5Y/5vuUPgGcN63EVh+BMMAQABGJLlYBJjEao2n5EsrAh6q+c4yMn5JjO PLybFvT/MLkWfUylNpvlIbRffJaphgwbXhLlZBbzPivyZ/ucZTek3J1439JW65oN u4IziODcVZwsZfjluR0koRNvJx73CAM7eflfQp+A/lwNRLDQf96Qqc6VYJO/2HdJ YhVNX3RqFCzgVAdo6fosjtCkPQUJAkfiq4gmNIPAR6Q9ZMn7jw/hD6GDGRJMlFZ8 aqN9QlWVdkLS4Bn9TJ4mZpNcA7F+YF9xusFhk3yC/ViYe5O9iaCtACml4wGVCRzL q1He4D/ZVjXNusfv734vGjxW89gfRuF9/J+CClEwCwaOA8N0p0lK+MFqRInzjvDI aA/i/Bf20RVtjl8tZA6/lZi/bwubwJGU21D2mEcNrdFZFCV73Zl11vVvEc0bsaLH 2LAp/39saKigMZO3jVGrQ9vnVjCAjIv2fnUB97W51463FR3fKUyIswQBAQgAHRYh BN0ovebZh1yrzkqLHdzPDbMLwQVIBQJdMNB5AAoJENzPDbMLwQVIcBMD/i6C+ujg +M7Y7GC2YXgXEHmnZQGRMQ6MJGdF6nqy9jTQFXx5ZyCALtgGSi0JKy2dExF28a3n xzYs9MNftZ03sMxEKl1sOh9GNbLFEPAZNCkTBX2yxGBM7AnoqHjUn7YHLtdtPfmQ AFu5xtjKSUTYWsPlfaX125ExhcPnKUPLu3uS =UQ9d -----END PGP SIGNATURE----- --=-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Wed Oct 23 15:59:42 2019 Received: (at 36630) by debbugs.gnu.org; 23 Oct 2019 19:59:42 +0000 Received: from localhost ([127.0.0.1]:35055 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1iNMnG-00055w-2u for submit@debbugs.gnu.org; Wed, 23 Oct 2019 15:59:42 -0400 Received: from mout.web.de ([212.227.15.14]:56111) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1iNMnE-00055j-9N for 36630@debbugs.gnu.org; Wed, 23 Oct 2019 15:59:41 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=web.de; s=dbaedf251592; t=1571860769; bh=pqxV/DTGr4V8YDf4swbiSrj5VJ0+V4lecEdV1REM3Mo=; h=X-UI-Sender-Class:From:To:Cc:Subject:Date:Reply-To; b=jzKJIbb5R3S49V1f/Smt6hWyt69z1OjdN3m8gMoz/OWmrVVE02bRKtM1n4fEwDRid ZWkV89fe5hPGqCy4S+dPr5jnzGjD2mDUXWX825Pkllz/XR4jrcdxFsq8yIT6aLSIYx DU8k6N/ePUMs+978xeh0/S20dFu7SsRyXsfNzEMc= X-UI-Sender-Class: c548c8c5-30a9-4db5-a2e7-cb6cb037b8f9 Received: from fluss.speedport.ip ([80.136.28.233]) by smtp.web.de (mrweb004 [213.165.67.108]) with ESMTPSA (Nemesis) id 0Lpyh5-1hlXs62hvj-00fhPa; Wed, 23 Oct 2019 21:59:29 +0200 From: Arne Babenhauserheide To: 36630@debbugs.gnu.org Subject: [PATCH] use two threads to build man-pages and secure output msgs with mutex Date: Wed, 23 Oct 2019 21:59:18 +0200 Message-Id: <20191023195918.26514-1-arne_bab@web.de> X-Mailer: git-send-email 2.23.0 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Provags-ID: V03:K1:QVHHxoGzswf3MQ+89avBnUIW9/QYe8oHGjoaYSHcVY2yqjiEjkz io+0RmEoIGJa/Is3pKfMTq5vFu78sl9No4hR+sW1slm3TsmQhuFQxRN5hyuBO57e4JaEIRv rSlw/qrE0B0OAmiHdj5Or+eNto95reA9OyYuy5f7bVAO45r5xZsE8iM0MeMswE3hkVKRJsx u5LtkRaWXLorUpmSDIeBQ== X-Spam-Flag: NO X-UI-Out-Filterresults: notjunk:1;V03:K0:7htekj3tE2U=:hgN19edpyHpr47v/Q+k+Gn i4MuZ1OLKvt+s6zYia7uQn+U6PV9PQM1cKzR4yXAc3NuOMAb8PxVn7q+kZEnYX7qXfTntKWPk 3eoS7xu0T2YlWxqg3l1cEux73QAs4hWRm9HBRhRRtXg3zCEYRK+G0x1RHO6EzyovdAGw435Aq nrs/ZGvQKkROxGxJ4cN17ZRhwCnOOvE01M70gJDQty3kY2cFjmkl3rGRPE4N1venTnDWp+Xk2 W/ekKwM8+qgv4dc05ky7GXFkGAitAuGARaA6N8QbEEi5J6FeQFB08o9cx40wt3qxUDfI8op3w 6ru94rUC7NCBA41V9gw8+LpIwd7CVV124oIGZUlQtDJQoF1H5UlvR/pkjd4EjKzfGe0lpKrX3 egAFYhCdc6mkP+GegO8osk69lbPTsFUbT18DuDRwjaoKvC/9Ho2pEv4Vl817A1mptN697ONyV g1Dey90SG+3tMafV5KNVMRMZpl+Rs0gxUFlRsXRpV4JY09AbAADtn0UUdGinSsUHCXwin9K0F +6R5ZcIWkMvhntBTy2mrKfcuURIq1J6q/PTUMmKxKjDybtVz+SPl2kQ0O/MuBZ80YQVDbqJIl Rssue75xDg9MkGtOeoBhULQClST+pAMHxNHCT3yr0aQlC1gMuWf9oAZXYHeCLaB2GkIxpaJij fo9kGhKFCSsckbMxY/4oP7pU5BFWVJfPIMaQJ8C7aY2JQAEqlUMuHq0k2zCJMhYW3Z5Yk5JGZ PFtgQcCQ/2zV4t32W4s2KL90zj7esCqowi2dgxOxYIjywGJ3yfjMZrpxHXlWL+jsgNHnd4Ql+ uGOx9KbYikzUDYx7q5riCfuP+Aaxtxbcm4kiBMjV2R2WszBNjP4KrU6kCiQs4zmOMQYdbgI2Z aQ8LU1ja4dF3hQnFCTOdsvxjbBiLLAqAlQLg3246jYGzwdfVDwm49NOlZT97UIUXXb97VwPzF x5a/t8JMsCyvmVja9vFS7tZXkEkmAiaUThGBSZ9ZMrhCGeF5LpP3lo/3axIldhUpFaNskIyv3 zYIkekTE6jtpZarHxbHtonBxFBqj8fzHwsyQD73DwQZpl+Qo1/nkgzrL+OJTwQqxGqkQs53vZ XlbjEkW0AOTpYTtv4jlgvr9PAK4ZFAL+evLaHqfunPbqisNfm2s1AUa5L7DZY4EKHlBRyd6wa mM3pT1DA2oqVSiZDjeD7i6CE/GOtdTyawHf99+A1V4KZa01TVD/wh9xdktwvBSMLXZ6csja38 9JSzOYv+WxK/Be6qTb6rDq/WS65OL4Xb4O0apBg== X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 36630 Cc: Arne Babenhauserheide X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: arne_bab@web.de Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) =2D-- guix/profiles.scm | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/guix/profiles.scm b/guix/profiles.scm index a0ae9a4c7e..9d2133df2d 100644 =2D-- a/guix/profiles.scm +++ b/guix/profiles.scm @@ -1315,11 +1315,18 @@ the entries in MANIFEST." (ice-9 threads) (srfi srfi-1) (srfi srfi-19)) + ;; ensure conflict free user-output + (define status-output-mutex (make-mutex)) + + (define (status-message msg) + (lock-mutex status-output-mutex) + (display msg) + (unlock-mutex status-output-mutex)) (define (compute-entry directory count total) - (format #t "\r[~3d/~3d] building list of \ + (status-message (format #f "\r[~3d/~3d] bui= lding list of \ man-db entries..." - count total) + count total)) (force-output) (let ((man (string-append directory "/share/man"))) @@ -1332,7 +1339,9 @@ man-db entries..." ;; decompression), so report progress as we traverse INPUTS= . (let* ((inputs '#$(manifest-inputs manifest)) (total (length inputs))) - (apply append (par-map compute-entry + ;; experimenting shows that two threads suffice to remove= most + ;; waiting + (apply append (n-par-map 2 compute-entry inputs (iota total 1) (make-list total total))))) =2D- 2.23.0 From debbugs-submit-bounces@debbugs.gnu.org Wed Oct 23 16:02:06 2019 Received: (at 36630) by debbugs.gnu.org; 23 Oct 2019 20:02:06 +0000 Received: from localhost ([127.0.0.1]:35059 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1iNMpY-0005BA-Ki for submit@debbugs.gnu.org; Wed, 23 Oct 2019 16:02:06 -0400 Received: from mout.web.de ([212.227.15.3]:57587) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1iNMpV-0005Ae-0A for 36630@debbugs.gnu.org; Wed, 23 Oct 2019 16:02:01 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=web.de; s=dbaedf251592; t=1571860913; bh=AwhukiMjGjBrM4o5hamKsMz0psS6f+r++4m/7y5Udp8=; h=X-UI-Sender-Class:References:From:To:Cc:Subject:In-reply-to:Date; b=YbzrTehSCG/hQ+IElCmBusw88k9jFrzz6EnHpanukXT3OogVqeL9dtIYvMmpC4Jjs vf+RHKPzvKP72eChF+H1r3vydeWe3NgMJZGedDKkwL08auEtNnTeLyRiFjwUWaLyfF DosQnZ+UsPbAW5akCOv1RzuFUPVirRZZua7HFNlI= X-UI-Sender-Class: c548c8c5-30a9-4db5-a2e7-cb6cb037b8f9 Received: from fluss ([80.136.28.233]) by smtp.web.de (mrweb003 [213.165.67.108]) with ESMTPSA (Nemesis) id 0Lfzwp-1hdoap3C6l-00pZ42; Wed, 23 Oct 2019 22:01:52 +0200 References: <20190712214245.23857-1-arne_bab@web.de> <878sszl1jo.fsf@gnu.org> <877e8ig9gh.fsf@web.de> <87zhldel6f.fsf@gnu.org> <878ssw8ger.fsf@web.de> <875znzhg91.fsf@gnu.org> <87k1cfpq0k.fsf@web.de> <87zhlbe9qt.fsf@gnu.org> <87d0i7p0ug.fsf@web.de> User-agent: mu4e 1.2.0; emacs 26.1 From: Arne Babenhauserheide To: Ludovic =?utf-8?Q?Court=C3=A8s?= Subject: Re: [bug#36630] [PATCH] guix: parallelize building the manual-database In-reply-to: <87d0i7p0ug.fsf@web.de> Date: Wed, 23 Oct 2019 22:01:46 +0200 Message-ID: <87d0entemt.fsf@web.de> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" X-Provags-ID: V03:K1:HDoXsw0bY6XmcR7hDxUrtcNImI6OnXi/pEl8xjEahWDN3wRkYSg eT2++0xIc2MALltMloCmeXLXiBIz7AgVl9y9BqlobDfnOpb57t/+/cQUf4HyoIlpM6slgzK uCiuZJdOR73t7OyrV2lp8HBiyodWh53tD7bebwCtT58GKwA4bxU09c5IKRtqeQsBXGVvfpV p/tqBUX6BUc14K+dT6FrQ== X-Spam-Flag: NO X-UI-Out-Filterresults: notjunk:1;V03:K0:ONibKfXLCrA=:lDnJ2pD0k11OY+1v9ij7wK fqtEkGGLoTKd9GMuwliZZ7pU2+o4FaSbu98upMIIEk1FB4EZ7ZVBZZV9f4M8NOcFIAxvYokz6 oo+Sxr/HVaSSj5DCvj7yXBLXAKEEeHi0aHxpRbzPJba0d9ZzZSD9trG9knYik6aNb78YwxnuY zugaRhcKPhOBwwx2u76HVqlZGMqBJCB63miaj9JdUQKm30hatkyzilZyJ9cTPIICx1Om3EbRc s+TL1A6EcabO+LGnDxpeqoloAHdk8ChKsnD20hRSnFihPVFKSYjt3OUYJa+GPnWLJKfvz8yWa KyJDnbFMd86WQJ2D/TcXMvDVGBVVeHtLzbHNEIg4eFudPVHMwWR67kcRWyy4hqucosWcta/7Z Z+4OELCB5CZRSC1GgELVkCW34FWrf65S1ZctMG3fhuCDJfKRO3z4W/BwqdlyH9cBuh9PvQoFd 8kv+d64ipdL9AELk0zo5sljcS7daGXKGMUy6Zcx7jM0mHPxvJUINJy0QZIELBKPZMx5RiQBpa e5izV14VK/i9hrpkz9wdcbC8D0v/jlernfy6JawaK6I9Pnn0XtR//6qoL2E1HUXz7A/mJyBDy 7zzqCJ3p3Nk0qPzLMPb4aiW+11Zz3PGfuMTO4NUXBz6W+AkyMgrbZkFWfeXv1e580xndh21b7 XyMLdVdREPeMWzcly+8Qqnmg2fwXZ15oP03m3v2kIlDSKURXnB0PjONhdHxliTMeuLgY2a5MC 3jnvfa8jLN/E1UmsdlGFOo7f6/mk6ouf1iSObua6KUxsLlQphc2XSJHwJuaYaL7a7FI7T5RsB 102oy2RfK6eH9OpT6HR6fUjQ8Y3VhXio3AZAdblXH9fdu4Cd9hWkrmt+o1wZctB0L2W80oSAW ml4zPdmGAPft29BdqTVt7LO/3GEvRGfDEuxGyrNODKAs0uRn061tKeFedwjzYJ2vRv0qj8eJP Ncbmy04x2b60dbji9Ib2H2L3lZ38CXkpD9J0I2ZpmgiEL6zYVxGXGsQKGzjwTKcsSBKzf7fiz Ij364ag2AefRz7770vCsEUiypcGxAHJ7lG6nhXkdBJk/UWwanxMhMDrvOk498WnimjJwBLVsf A2q37lbwvf2I0d8/hcUg4SaD32NADZ8Cv4/SY0XGIFpZyhBN9MbYUy5W/CiEHiHghi//efAVV fItGdsYcp8+lUIa97sAqGpS++t3ByLB9FaBKmEHYmXcqjbyCMr811WQBva9f8dhJaD/srIGgd O6NuRyC6CeEjubY5bsb04bgCT0lLFAroJgCZJuQ== X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 36630 Cc: 36630@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Arne Babenhauserheide writes: >> Right, sorry for being unclear: you need to run >> >> ./pre-inst-env guix environment -n =E2=80=A6 >> >> That will run you modified code and thus create a manual-database.drv >> that uses your code; it=E2=80=99s this manual-database.drv that you shou= ld pass >> to =E2=80=98guix build=E2=80=99. > > That works now =E2=80=94 thank you! =E2=80=A6 > With the change: > 185552 entries processed in 108.2 s > Before the change: > 185552 entries processed in 220.1 s > > The exact commands I=E2=80=99m running: > cd Dokumente/Guix/guix > git checkout master # with the change > ./pre-inst-env guix environment --ad-hoc jupyter python-ipython python-ip= ykernel -n > time guix build /gnu/...-manual-database.drv > git checkout 97bf46e64c11c64a968fdb833983ede6bdafbc00 > ./pre-inst-env guix environment --ad-hoc jupyter python-ipython python-ip= ykernel -n > time guix build /gnu/...-manual-database.drv I now reduced the thread count to exactly 2 (to avoid running into resource troubles; I hope that two should be safe) and added a mutex for status messages to ensure that writes don=E2=80=99t overlap. The patch should arrive shortly. Best wishes, Arne =2D- Unpolitisch sein hei=C3=9Ft politisch sein ohne es zu merken --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEE801qEjXQSQPNItXAE++NRSQDw+sFAl2wsa0ACgkQE++NRSQD w+sN7g//fy4HDFaVlUVHxACX9sIYsFi+R6xR654u+Q8UujdtryKizroo4vGCY4Fr QmWPXonFn/npwUMni1IFPgQ7JsWRvyXghZfs4hHG45y3tc8AGIVfs3Z5yC4WqVi1 07cIvTHnmoL0/XQW4U38BGwWfY1MH1eypVTPp3mvwJZgGvSB2yjmRCVKVxxbQwX+ t1KzpdWq/guBBHoMl4qkNv5BBWHmoOO/1KQY/92BwMNorwtL95MJFqjqcwCfOdpo 6HLXg6UgBdAyoxoxmUoGhJqAtJhrkP4sC60syg/vl+Pi3pFL3wvPOP3gzRnIAwU4 31ZurLSsemlVESsI2w5u0zuhakup+k6emHwgXIkbTE4WYMySMgw1I57G8qnPlLOW jsiajBlOx4F0MQnpsWRovhaSfG7M1QFfY8c+FR1MR/XkiKTSKYrDXK0PyX30Orm8 Xy2RWJ/dFaZqp3kSzvWxrIX6GOQdU/wAlzI/1FYQTav0qEInXYNOvIfMeg55RKUV wdneE6d7/CGpZaRGV3SdYZ6roQdCHnldA1xaaUsmC/Xf46yVhUzhhktkkkNA4K6C uOXHgebZUUuslMTEl3DNPXy3EdYmVvtWJoiES64Rq/PY7R1G/mCxNJ1wN5TLsDOu AjOxsLENBJKiaqRDuGM+1arSGIN/HspFvRKbHyLUBT9a3KItfzmIswQBAQgAHRYh BN0ovebZh1yrzkqLHdzPDbMLwQVIBQJdsLGtAAoJENzPDbMLwQVIGKMEAJlgj9AX iJ+O8cUqr60V9ZneVgaNqMmpLc2BuCPFHcAuHmym0tPDvWWp7dcdGsQWbXgWVXKt 01iilzh/dLe1er6TeKWbevfcAZx2ZB5gCn9SSvtP0CwA6HJsDnIht5QzHV4YeuI/ 0tRDgXdtxGjgrgUCdLZh474pd7Ri66bvVL7a =IW/g -----END PGP SIGNATURE----- --=-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Sun Oct 27 12:28:21 2019 Received: (at 36630) by debbugs.gnu.org; 27 Oct 2019 16:28:21 +0000 Received: from localhost ([127.0.0.1]:43430 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1iOlOv-0004F1-Ha for submit@debbugs.gnu.org; Sun, 27 Oct 2019 12:28:21 -0400 Received: from mout.web.de ([212.227.17.12]:40003) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1iOlOt-0004Eo-Gi for 36630@debbugs.gnu.org; Sun, 27 Oct 2019 12:28:20 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=web.de; s=dbaedf251592; t=1572193693; bh=pqxV/DTGr4V8YDf4swbiSrj5VJ0+V4lecEdV1REM3Mo=; h=X-UI-Sender-Class:From:To:Cc:Subject:Date:Reply-To; b=XA9q28MTAeq0sKCZaq01KC9mgJeqFa2ax8MmURNKjLizIPTZLdk+Ic6vRY2m5lTf2 H/6AFo19vwcMg+bqFF0LP02QJTycrLYDWJaX21D40nAfA8mZTH7wRYT962SKK2cgE4 nBgUzW2ktmpqU4fjsLsanpSiZNfGtbo8WIV1Vlh8= X-UI-Sender-Class: c548c8c5-30a9-4db5-a2e7-cb6cb037b8f9 Received: from fluss.speedport.ip ([80.136.28.233]) by smtp.web.de (mrweb103 [213.165.67.124]) with ESMTPSA (Nemesis) id 0Lx7Ir-1hvk7p3dIP-016eGV; Sun, 27 Oct 2019 17:28:12 +0100 From: Arne Babenhauserheide To: 36630@debbugs.gnu.org Subject: [PATCH] use two threads to build man-pages and secure output msgs with mutex Date: Sun, 27 Oct 2019 17:27:57 +0100 Message-Id: <20191027162757.32295-1-arne_bab@web.de> X-Mailer: git-send-email 2.23.0 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Provags-ID: V03:K1:MDzkB42nxTDLwCiU+4hCseRPTZcPet9JXARYgejWT4ldFEZyBXk LwqPKk0GIMusRdgVmx2kovg6bc5EgfXAbETwi8pEr5cjfIX/Cpm3RlsbhFjAn0hlh93hnAe 1O4DAX+mCFnuoqzX71/H38t0PfCDPqnOY7BDUKnU2dIpfQGymOuWG/NJ7Cei84WmW0JSJ6O qNYYaU+BN6OsterZS33Iw== X-Spam-Flag: NO X-UI-Out-Filterresults: notjunk:1;V03:K0:w49cmtcv31Y=:PgFOMq8i7BC3ko1KWCdD/9 Mx0WS7igsNEm7EaJ/GHu5JcocJ9/AWf05sASki8u/J/hf2Zvmb2g7LokNBagKjWTA7q86S7if rkqMqC/tx5sQ6Hm+gMf+rZt1CAPQxlpEvQvQXSs6kJrOsCjIOAGnj8028CZspJ/DyxuXJ8W9w WspVnEewI8bvUWcxTa+jsCt6/5BC2uk+fUmgIbv6Zc+B/gja0ClOkn0Dku+tgw395d83nsO4v ChgVlQ65auK8e71DaivV1paSX/nj3f0QjfocOEUxnwF8hvKAYOVtZMzk9AVXxDzS2ZEL+aQrz KHvvN5Vs+b7uTltOjf2RgM4CfSIY5V3CHbqWNYH2NrKFqH8PjRiE2qRdsnd1wdUR7bnmNRN9S G5o5nD9QNCvIF2aaEw44scDyFYQpGNnzxS0JCJGvuu+0MH0V+O5XBhnFk9Ip8aZGwmvSG0MDR hBMg6fLpqWQZdW10vo3WM9EdoK1skH3ywZbpVh/gMzMuPoMGFF8A0v9FjdJ9LvRvx739KTWIX vIdD2cuSNNU9cxXVADHyNz39OvNBqbdJrYluN1Ve2pmLrD6fpybRI2HFRMaoTp+tJ7LrCItrW sHpU0k9U+ndVPdyUZmf+XZEvMEu/MLDskU0XA3PQs4mZhbBWuZk+W8PppCDtWrhGof8JMzepH Z/ExloHLD00uF5N9UTuUGZiH3nmurAwYFb+icVX3szAIcF2SX3Wb7TbQiYumVVEKpR5P01YSj ypG86hdOejvpMPOe1ukLxfYYj2zHw0UuylE5eqc5io482CmYXq496CQrYphaHeeZD+AIzYi24 oLVtjATgvRUPSmRjUDOg1pI/Q4pxV/uHuF3cicMe/WVElMSOg765g/8PWECTfYgJkZrCPjJsS lwXosPUzYmIHSJ1y2+5p9ugfl8bxStaGi3dcflpR1GWNY0jYHRJZlAzBD16DAADqMzC2wWXNW X7Uv+y5bXz/GvAxYh3UxrXPSIlZVt5Zx7JeYaDjvZBTdtMEQnme6igRqmeu3Nnu1SJkk1M+Kg Mbz0ckNkcIBzwmV8oa49w/pVyvA38hi0nHk60XVVdDasUmOJd/Z2u8YjycG9yao2I0JF9Drwg uD0zgte098oiTrRRKLjLCNyjlvfnLqkn1/GvFr2g02LhJP9tnhCnZfkbmwRxIFN7OG4zwM905 n0skzG743s979YsRApNzGrD7J+y3iq2xiHBl5mhLP8HL0CcUPhX/xMrMhpZL8p2b27R9SXKTt edIzZi13beYdyaWHcHqsEI4+g9TAxMJTl7j22cw== X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 36630 Cc: Arne Babenhauserheide X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: arne_bab@web.de Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) =2D-- guix/profiles.scm | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/guix/profiles.scm b/guix/profiles.scm index a0ae9a4c7e..9d2133df2d 100644 =2D-- a/guix/profiles.scm +++ b/guix/profiles.scm @@ -1315,11 +1315,18 @@ the entries in MANIFEST." (ice-9 threads) (srfi srfi-1) (srfi srfi-19)) + ;; ensure conflict free user-output + (define status-output-mutex (make-mutex)) + + (define (status-message msg) + (lock-mutex status-output-mutex) + (display msg) + (unlock-mutex status-output-mutex)) (define (compute-entry directory count total) - (format #t "\r[~3d/~3d] building list of \ + (status-message (format #f "\r[~3d/~3d] bui= lding list of \ man-db entries..." - count total) + count total)) (force-output) (let ((man (string-append directory "/share/man"))) @@ -1332,7 +1339,9 @@ man-db entries..." ;; decompression), so report progress as we traverse INPUTS= . (let* ((inputs '#$(manifest-inputs manifest)) (total (length inputs))) - (apply append (par-map compute-entry + ;; experimenting shows that two threads suffice to remove= most + ;; waiting + (apply append (n-par-map 2 compute-entry inputs (iota total 1) (make-list total total))))) =2D- 2.23.0 From debbugs-submit-bounces@debbugs.gnu.org Tue Mar 31 09:02:58 2020 Received: (at 36630-done) by debbugs.gnu.org; 31 Mar 2020 13:02:58 +0000 Received: from localhost ([127.0.0.1]:35738 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jJGXi-0002eq-2Y for submit@debbugs.gnu.org; Tue, 31 Mar 2020 09:02:58 -0400 Received: from eggs.gnu.org ([209.51.188.92]:52426) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jJGXf-0002dx-SQ for 36630-done@debbugs.gnu.org; Tue, 31 Mar 2020 09:02:56 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:37571) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1jJGXa-0008Eu-Ht; Tue, 31 Mar 2020 09:02:50 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=48618 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1jJGXY-0001q7-Vq; Tue, 31 Mar 2020 09:02:50 -0400 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Arne Babenhauserheide Subject: Re: [bug#36630] [PATCH] guix: parallelize building the manual-database References: <20190712214245.23857-1-arne_bab@web.de> <878sszl1jo.fsf@gnu.org> <877e8ig9gh.fsf@web.de> <87zhldel6f.fsf@gnu.org> <878ssw8ger.fsf@web.de> <875znzhg91.fsf@gnu.org> <87k1cfpq0k.fsf@web.de> <87zhlbe9qt.fsf@gnu.org> <87d0i7p0ug.fsf@web.de> <87d0entemt.fsf@web.de> Date: Tue, 31 Mar 2020 15:02:47 +0200 In-Reply-To: <87d0entemt.fsf@web.de> (Arne Babenhauserheide's message of "Wed, 23 Oct 2019 22:01:46 +0200") Message-ID: <87zhbwvfrc.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.3 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 36630-done Cc: 36630-done@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) Hi Arne, Arne Babenhauserheide skribis: > I now reduced the thread count to exactly 2 (to avoid running into > resource troubles; I hope that two should be safe) and added a mutex for > status messages to ensure that writes don=E2=80=99t overlap. It=E2=80=99s been 9 months but I finally committed a slightly modified vari= ant as ef4b5f2fed3ca13a0e15a821ba7e561cd4395aa6. It turns out that the mutex was unnecessary as ports are thread-safe. As noted in the log, I see a 36% speedup on my SSD laptop with 4 cores (slightly less with 2 cores). It=E2=80=99s not great, but still an improve= ment! Thanks, Ludo=E2=80=99. From unknown Tue Jun 17 01:33:03 2025 Received: (at fakecontrol) by fakecontrolmessage; To: internal_control@debbugs.gnu.org From: Debbugs Internal Request Subject: Internal Control Message-Id: bug archived. Date: Wed, 29 Apr 2020 11:24:04 +0000 User-Agent: Fakemail v42.6.9 # This is a fake control message. # # The action: # bug archived. thanks # This fakemail brought to you by your local debbugs # administrator