From unknown Sat Jun 14 03:52:08 2025 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Mailer: MIME-tools 5.509 (Entity 5.509) Content-Type: text/plain; charset=utf-8 From: bug#39728 <39728@debbugs.gnu.org> To: bug#39728 <39728@debbugs.gnu.org> Subject: Status: [PATCH] Allow parallel downloads and builds Reply-To: bug#39728 <39728@debbugs.gnu.org> Date: Sat, 14 Jun 2025 10:52:08 +0000 retitle 39728 [PATCH] Allow parallel downloads and builds reassign 39728 guix-patches submitter 39728 Julien Lepiller severity 39728 normal tag 39728 patch thanks From debbugs-submit-bounces@debbugs.gnu.org Fri Feb 21 17:53:34 2020 Received: (at submit) by debbugs.gnu.org; 21 Feb 2020 22:53:34 +0000 Received: from localhost ([127.0.0.1]:48348 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1j5HAs-0008CC-1o for submit@debbugs.gnu.org; Fri, 21 Feb 2020 17:53:34 -0500 Received: from lists.gnu.org ([209.51.188.17]:55942) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1j5HAq-0008C3-30 for submit@debbugs.gnu.org; Fri, 21 Feb 2020 17:53:32 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:44754) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1j5HAn-0000RJ-QH for guix-patches@gnu.org; Fri, 21 Feb 2020 17:53:31 -0500 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=0.8 required=5.0 tests=BAYES_50,URIBL_BLOCKED autolearn=disabled version=3.3.2 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1j5HAi-0008CD-1t for guix-patches@gnu.org; Fri, 21 Feb 2020 17:53:29 -0500 Received: from lepiller.eu ([2a00:5884:8208::1]:39044) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1j5HAh-0008AT-4v for guix-patches@gnu.org; Fri, 21 Feb 2020 17:53:24 -0500 Received: from lepiller.eu (localhost [127.0.0.1]) by lepiller.eu (OpenSMTPD) with ESMTP id cd01c274 for ; Fri, 21 Feb 2020 22:53:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed; d=lepiller.eu; h=date:from :to:subject:message-id:mime-version:content-type; s=dkim; bh=n++ 9yoIChEqgG20cqPGuVAueYEx728KISUJPEfg13k4=; b=Y08S5xrCZ+3Zzo/X308 qvrWmAKKlugH1AmrL5Zk6iKmLk5QgPOXglurseeDqZTxT5L4ee3gX9jpJxjrBWVC KoiCvqv+vrs4GmttN/X3+aY+HWx7vOdYtfwm+xt6YsLMeXFZqFcuuGPvsNNeNWzn CwOeKgsycJmlDRCItsG/y8S+IReKupngF0RoCnFxOzocPgNVu45cxD0KZPZ8UNjS jZaOqOkw8DIYwCNB2fkS3HdWAose8bFYv5mLuimWksGLltXMQWtiVspWwaU2AVao k4tBLAN70dj2K+jOym6M3PaEl93oZmpMXdti4K3b5vwxvKCShDyltPwDYhLHGhm5 JBA== Received: by lepiller.eu (OpenSMTPD) with ESMTPSA id e21dd517 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO) for ; Fri, 21 Feb 2020 22:53:18 +0000 (UTC) Date: Fri, 21 Feb 2020 23:53:07 +0100 From: Julien Lepiller To: guix-patches@gnu.org Subject: [PATCH] Allow parallel downloads and builds Message-ID: <20200221235307.535fb453@tachikoma.lepiller.eu> X-Mailer: Claws Mail 3.17.4 (GTK+ 2.24.32; x86_64-unknown-linux-gnu) MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="MP_/08Zvlxdk=mg4mXHm6u3=if1" X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:5884:8208::1 X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: submit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) --MP_/08Zvlxdk=mg4mXHm6u3=if1 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Content-Disposition: inline Hi guix! This patch allows to count builds and downloads separately. The idea is that downloads need bandwidth, but no CPU, while builds do not need bandwidth, but need CPU. With this patch, guix will be able to download substitutes while building unrelated packages. Currently, guix needs to wait for the download to finish before proceeding to the build. This should reduce the time of guix commands that need to build and download things at the same time. What do you think? --MP_/08Zvlxdk=mg4mXHm6u3=if1 Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=0001-nix-Count-build-and-download-jobs-separately.patch >From 9c059d81ba4f4016f8c400b403f8c5edbdb160c2 Mon Sep 17 00:00:00 2001 From: Julien Lepiller Date: Fri, 21 Feb 2020 23:41:33 +0100 Subject: [PATCH] nix: Count build and download jobs separately. This allows to run downloads (that take bandwith) and builds (that take CPU time) independently from one another. * nix/nix-daemon/guix-daemon.cc: Add a max-download-jobs option. * nix/libstore/globals.hh: Add a maxDownloadJobs setting. * nix/libstore/globals.cc: Add a default value to it. * nix/libstore/build.cc: Manage build and download jobs separately. --- nix/libstore/build.cc | 75 +++++++++++++++++++++++++++-------- nix/libstore/globals.cc | 2 + nix/libstore/globals.hh | 3 ++ nix/nix-daemon/guix-daemon.cc | 5 +++ 4 files changed, 69 insertions(+), 16 deletions(-) diff --git a/nix/libstore/build.cc b/nix/libstore/build.cc index 17e92c68a7..d08904c4ee 100644 --- a/nix/libstore/build.cc +++ b/nix/libstore/build.cc @@ -204,6 +204,7 @@ struct Child set fds; bool respectTimeouts; bool inBuildSlot; + bool inDownloadSlot; time_t lastOutput; /* time we last got output on stdout/stderr */ time_t timeStarted; }; @@ -231,10 +232,14 @@ private: /* Child processes currently running. */ Children children; - /* Number of build slots occupied. This includes local builds and - substitutions but not remote builds via the build hook. */ + /* Number of build slots occupied. This includes local builds but not + substitution, built-ins and remote builds via the build hook. */ unsigned int nrLocalBuilds; + /* Number of download slots occupied. This includes substitution and + built-ins. */ + unsigned int nrDownloads; + /* Maps used to prevent multiple instantiations of a goal for the same derivation / path. */ WeakGoalMap derivationGoals; @@ -275,15 +280,18 @@ public: /* Wake up a goal (i.e., there is something for it to do). */ void wakeUp(GoalPtr goal); - /* Return the number of local build and substitution processes - currently running (but not remote builds via the build - hook). */ + /* Return the number of local build processes currently running (but + not remote builds via the build hook). */ unsigned int getNrLocalBuilds(); + /* Return the number of downloads currently running. */ + unsigned int getNrDownloads(); + /* Registers a running child process. `inBuildSlot' means that the process counts towards the jobs limit. */ void childStarted(GoalPtr goal, pid_t pid, - const set & fds, bool inBuildSlot, bool respectTimeouts); + const set & fds, bool inBuildSlot, bool inDownloadSlot, + bool respectTimeouts); /* Unregisters a running child process. `wakeSleepers' should be false if there is no sense in waking up goals that are sleeping @@ -295,6 +303,10 @@ public: might be right away). */ void waitForBuildSlot(GoalPtr goal); + /* Put `goal' to sleep until a download slot becomes available (which + might be right away). */ + void waitForDownloadSlot(GoalPtr goal); + /* Wait for any goal to finish. Pretty indiscriminate way to wait for some resource that some other goal is holding. */ void waitForAnyGoal(GoalPtr goal); @@ -1359,12 +1371,21 @@ void DerivationGoal::tryToBuild() derivation prefers to be done locally, do it even if maxBuildJobs is 0. */ unsigned int curBuilds = worker.getNrLocalBuilds(); - if (curBuilds >= settings.maxBuildJobs && !(buildLocally && curBuilds == 0)) { + if (curBuilds >= settings.maxBuildJobs && !(buildLocally && curBuilds == 0) && + !fixedOutput) { worker.waitForBuildSlot(shared_from_this()); outputLocks.unlock(); return; } + unsigned int curDownloads = worker.getNrDownloads(); + if (curDownloads >= (settings.maxDownloadJobs==0?1:settings.maxDownloadJobs) && + fixedOutput) { + worker.waitForDownloadSlot(shared_from_this()); + outputLocks.unlock(); + return; + } + try { /* Okay, we have to build. */ @@ -1648,7 +1669,7 @@ HookReply DerivationGoal::tryBuildHook() set fds; fds.insert(hook->fromHook.readSide); fds.insert(hook->builderOut.readSide); - worker.childStarted(shared_from_this(), hook->pid, fds, false, true); + worker.childStarted(shared_from_this(), hook->pid, fds, false, false, true); if (settings.printBuildTrace) printMsg(lvlError, format("@ build-started %1% - %2% %3% %4%") @@ -2030,7 +2051,7 @@ void DerivationGoal::startBuilder() pid.setSeparatePG(true); builderOut.writeSide.close(); worker.childStarted(shared_from_this(), pid, - singleton >(builderOut.readSide), true, true); + singleton >(builderOut.readSide), !fixedOutput, fixedOutput, true); /* Check if setting up the build environment failed. */ string msg = readLine(builderOut.readSide); @@ -3034,11 +3055,11 @@ void SubstitutionGoal::tryToRun() trace("trying to run"); /* Make sure that we are allowed to start a build. Note that even - is maxBuildJobs == 0 (no local builds allowed), we still allow - a substituter to run. This is because substitutions cannot be - distributed to another machine via the build hook. */ - if (worker.getNrLocalBuilds() >= (settings.maxBuildJobs == 0 ? 1 : settings.maxBuildJobs)) { - worker.waitForBuildSlot(shared_from_this()); + is maxDownloadJobs == 0 (no downloads allowed), we still allow + a substituter to run. This is because we always need to download, so + not allowing is meaningless. */ + if (worker.getNrDownloads() >= (settings.maxDownloadJobs == 0 ? 1 : settings.maxDownloadJobs)) { + worker.waitForDownloadSlot(shared_from_this()); return; } @@ -3107,7 +3128,7 @@ void SubstitutionGoal::tryToRun() outPipe.writeSide.close(); logPipe.writeSide.close(); worker.childStarted(shared_from_this(), - pid, singleton >(logPipe.readSide), true, true); + pid, singleton >(logPipe.readSide), false, true, true); state = &SubstitutionGoal::finished; @@ -3242,6 +3263,7 @@ Worker::Worker(LocalStore & store) if (working) abort(); working = true; nrLocalBuilds = 0; + nrDownloads = 0; lastWokenUp = 0; permanentFailure = false; timedOut = false; @@ -3334,9 +3356,14 @@ unsigned Worker::getNrLocalBuilds() return nrLocalBuilds; } +unsigned Worker::getNrDownloads() +{ + return nrDownloads; +} + void Worker::childStarted(GoalPtr goal, - pid_t pid, const set & fds, bool inBuildSlot, + pid_t pid, const set & fds, bool inBuildSlot, bool inDownloadSlot, bool respectTimeouts) { Child child; @@ -3344,9 +3371,11 @@ void Worker::childStarted(GoalPtr goal, child.fds = fds; child.timeStarted = child.lastOutput = time(0); child.inBuildSlot = inBuildSlot; + child.inDownloadSlot = inDownloadSlot; child.respectTimeouts = respectTimeouts; children[pid] = child; if (inBuildSlot) nrLocalBuilds++; + if (inDownloadSlot) nrDownloads++; } @@ -3362,6 +3391,11 @@ void Worker::childTerminated(pid_t pid, bool wakeSleepers) nrLocalBuilds--; } + if (i->second.inDownloadSlot) { + assert(nrDownloads > 0); + nrDownloads--; + } + children.erase(pid); if (wakeSleepers) { @@ -3386,6 +3420,15 @@ void Worker::waitForBuildSlot(GoalPtr goal) addToWeakGoals(wantingToBuild, goal); } +void Worker::waitForDownloadSlot(GoalPtr goal) +{ + debug("wait for download slot"); + if (getNrDownloads() < (settings.maxDownloadJobs==0?1:settings.maxDownloadJobs)) + wakeUp(goal); /* we can do it right away */ + else + addToWeakGoals(wantingToBuild, goal); +} + void Worker::waitForAnyGoal(GoalPtr goal) { diff --git a/nix/libstore/globals.cc b/nix/libstore/globals.cc index 0cc001fbe4..416033718d 100644 --- a/nix/libstore/globals.cc +++ b/nix/libstore/globals.cc @@ -29,6 +29,7 @@ Settings::Settings() tryFallback = false; buildVerbosity = lvlError; maxBuildJobs = 1; + maxDownloadJobs = 1; buildCores = 1; readOnlyMode = false; thisSystem = SYSTEM; @@ -118,6 +119,7 @@ void Settings::update() { _get(tryFallback, "build-fallback"); _get(maxBuildJobs, "build-max-jobs"); + _get(maxDownloadJobs, "download-max-jobs"); _get(buildCores, "build-cores"); _get(thisSystem, "system"); _get(multiplexedBuildOutput, "multiplexed-build-output"); diff --git a/nix/libstore/globals.hh b/nix/libstore/globals.hh index 27616a2283..c033f8ed56 100644 --- a/nix/libstore/globals.hh +++ b/nix/libstore/globals.hh @@ -90,6 +90,9 @@ struct Settings { /* Maximum number of parallel build jobs. 0 means unlimited. */ unsigned int maxBuildJobs; + /* Maximum number of parallel download jobs. 0 means 1. */ + unsigned int maxDownloadJobs; + /* Number of CPU cores to utilize in parallel within a build, i.e. by passing this number to Make via '-j'. 0 means that the number of actual CPU cores on the local host ought to be diff --git a/nix/nix-daemon/guix-daemon.cc b/nix/nix-daemon/guix-daemon.cc index cd949aca67..6997c07f5a 100644 --- a/nix/nix-daemon/guix-daemon.cc +++ b/nix/nix-daemon/guix-daemon.cc @@ -99,6 +99,8 @@ static const struct argp_option options[] = }, { "max-jobs", 'M', n_("N"), 0, n_("allow at most N build jobs") }, + { "max-downloads", 'D', n_("N"), 0, + n_("allow at most N download jobs") }, { "timeout", GUIX_OPT_TIMEOUT, n_("SECONDS"), 0, n_("mark builds as failed after SECONDS of activity") }, { "max-silent-time", GUIX_OPT_MAX_SILENT_TIME, n_("SECONDS"), 0, @@ -276,6 +278,9 @@ parse_opt (int key, char *arg, struct argp_state *state) case 'M': settings.set ("build-max-jobs", arg); break; + case 'D': + settings.set ("download-max-jobs", arg); + break; case GUIX_OPT_TIMEOUT: settings.set ("build-timeout", arg); break; -- 2.24.0 --MP_/08Zvlxdk=mg4mXHm6u3=if1-- From debbugs-submit-bounces@debbugs.gnu.org Mon Feb 24 16:23:56 2020 Received: (at 39728) by debbugs.gnu.org; 24 Feb 2020 21:23:56 +0000 Received: from localhost ([127.0.0.1]:54204 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1j6LCm-0005mT-3e for submit@debbugs.gnu.org; Mon, 24 Feb 2020 16:23:56 -0500 Received: from eggs.gnu.org ([209.51.188.92]:42208) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1j6LCk-0005mB-V0 for 39728@debbugs.gnu.org; Mon, 24 Feb 2020 16:23:55 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]:34891) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1j6LCf-00086z-4p; Mon, 24 Feb 2020 16:23:49 -0500 Received: from [80.215.212.253] (port=17708 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1j6LCe-0007dP-HW; Mon, 24 Feb 2020 16:23:48 -0500 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Julien Lepiller Subject: Re: [bug#39728] [PATCH] Allow parallel downloads and builds References: <20200221235307.535fb453@tachikoma.lepiller.eu> Date: Mon, 24 Feb 2020 22:23:45 +0100 In-Reply-To: <20200221235307.535fb453@tachikoma.lepiller.eu> (Julien Lepiller's message of "Fri, 21 Feb 2020 23:53:07 +0100") Message-ID: <87h7zfel26.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.3 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 39728 Cc: 39728@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) Hi! Julien Lepiller skribis: > This patch allows to count builds and downloads separately. The idea is > that downloads need bandwidth, but no CPU, while builds do not need > bandwidth, but need CPU. With this patch, guix will be able to download > substitutes while building unrelated packages. Currently, guix needs to > wait for the download to finish before proceeding to the build. This > should reduce the time of guix commands that need to build and download > things at the same time. > > What do you think? I think it=E2=80=99s a good idea! I wonder what the UI will look like: (guix status) would no longer display a progress bar when there=E2=80=99s more than on job (build or down= load) taking place at the same time. >>>From 9c059d81ba4f4016f8c400b403f8c5edbdb160c2 Mon Sep 17 00:00:00 2001 > From: Julien Lepiller > Date: Fri, 21 Feb 2020 23:41:33 +0100 > Subject: [PATCH] nix: Count build and download jobs separately. ^ I=E2=80=99d write =E2=80=9Cdaemon:=E2=80=9D here. :-) > This allows to run downloads (that take bandwith) and builds (that take ^ =E2=80=9CThis allows us=E2=80=9D > CPU time) independently from one another. > > * nix/nix-daemon/guix-daemon.cc: Add a max-download-jobs option. > * nix/libstore/globals.hh: Add a maxDownloadJobs setting. > * nix/libstore/globals.cc: Add a default value to it. > * nix/libstore/build.cc: Manage build and download jobs separately. For the final patch, please specify the entities changed (classes, functions, etc.). > + /* Number of download slots occupied. This includes substitution and > + built-ins. */ > + unsigned int nrDownloads; Note that not all builtins are downloads. Fixed-output derivations are (usually) also downloads. (It=E2=80=99d be the first time the daemon gets a notion of =E2=80=9Cdownlo= ad=E2=80=9D. We should make sure it doesn=E2=80=99t conflict with other assumptions.) > /* Registers a running child process. `inBuildSlot' means that > the process counts towards the jobs limit. */ > void childStarted(GoalPtr goal, pid_t pid, > - const set & fds, bool inBuildSlot, bool respectTimeouts); > + const set & fds, bool inBuildSlot, bool inDownloadSlot, > + bool respectTimeouts); How about replacing these two Booleans by a single enum? > + unsigned int curDownloads =3D worker.getNrDownloads(); > + if (curDownloads >=3D (settings.maxDownloadJobs=3D=3D0?1:settings.ma= xDownloadJobs) && > + fixedOutput) { This is hard to parse and lacking spacing. :-) Perhaps make an intermediate function or variable? > +void Worker::waitForDownloadSlot(GoalPtr goal) > +{ > + debug("wait for download slot"); > + if (getNrDownloads() < (settings.maxDownloadJobs=3D=3D0?1:settings.m= axDownloadJobs)) Same here. > @@ -118,6 +119,7 @@ void Settings::update() > { > _get(tryFallback, "build-fallback"); > _get(maxBuildJobs, "build-max-jobs"); > + _get(maxDownloadJobs, "download-max-jobs"); We should also allow =E2=80=98set-build-options=E2=80=99 to set this option= , as well as add it to =E2=80=98%standard-build-options=E2=80=99. That can prolly come = in a separate patch. > + { "max-downloads", 'D', n_("N"), 0, > + n_("allow at most N download jobs") }, We=E2=80=99d need to update doc/guix.texi. It would be great if you could test this patch for your daily usage. I find it surprisingly easy to break things in the daemon. :-) Thank you! Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Tue Feb 25 10:21:43 2020 Received: (at 39728) by debbugs.gnu.org; 25 Feb 2020 15:21:43 +0000 Received: from localhost ([127.0.0.1]:55965 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1j6c1m-0004Mb-P7 for submit@debbugs.gnu.org; Tue, 25 Feb 2020 10:21:42 -0500 Received: from mail-qt1-f195.google.com ([209.85.160.195]:36976) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1j6c1l-0004MP-2V for 39728@debbugs.gnu.org; Tue, 25 Feb 2020 10:21:41 -0500 Received: by mail-qt1-f195.google.com with SMTP id j34so283923qtk.4 for <39728@debbugs.gnu.org>; Tue, 25 Feb 2020 07:21:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=+1/V3oHsEuRIiui8lAj98kNIISS4d4tLP3Rs9F3P1lI=; b=dY2JC989BJI5p8S4030g2j+TSohWLiD5V/f0oMbwVy6Q6Zt+vavIz6PJV7AlkqaJCG 0YXzgVhbORdOKLZWxmE7iW4/2Q7H3/FnOgHqfz2R8Z9m/IPE7gHkLo8RCiarG8CUTB1G Pzp9q5CEtwMZbz8IM1DcfdaMAZIQvSae94UoNvsGHI6//jCFujWq87BUNDVGHuXCh3d5 KmR2AC4FoibadoyDeXhKrsS67RnsB4y1NZRVtaFOeJRXolT5+PsRDhPxjVoOPLa6rY5H 9+F7gJ9Mi6u+V3EoHwErLGDiwUnrEuKy8HBvS4Jf1qD1OCGQ4sZNObEX5YCe4EqTQgax 9TNA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=+1/V3oHsEuRIiui8lAj98kNIISS4d4tLP3Rs9F3P1lI=; b=qI1pd5AkVewGGur9Sb3JH+5IWywsSebC9XcCyDTZiv622s1xkzefXXax1fkMLm6+ss 3B6RpaolPsG8krABcRayCny6jk7TmXHJ86RFGXRU/Sf40V2UuBUwyec5OTOUkp1kPPe1 tiD3X+77iB/sxpnTx1ojNQhD+ALZndGUo4WoMTa26lfhr2W/43+cqAa9+MbFvCzS9/Gz JO5o50w4wwejTuCXtcEAPMQ5cTKepmNPAtkoC4pEH+CwRuvJJ1PVXPLtK608kQuDZ8wv 0Vl9NodWQXWul/rivKKiJkasdpDu1KWJ+FYoA2E47dnC5+DCpm623LQcbGfE64OlfJaP IRcA== X-Gm-Message-State: APjAAAUYod/euVyTodv5tttewnPsZbS3UCb9TFgErkQM7VOyBG6gDQZe Y8zcjFBqpm7mDndsMm5nn21KlA4LP0JQ3KLhYQg= X-Google-Smtp-Source: APXvYqx6bm3ffEwYK3nccAIBnBkPm95NA6PxsWaHzTP2Wmlt9vJOcIN1F9szevxrjboIJcOnvAOk+1EkmX88ZssgLq0= X-Received: by 2002:ac8:550a:: with SMTP id j10mr56073537qtq.354.1582644095513; Tue, 25 Feb 2020 07:21:35 -0800 (PST) MIME-Version: 1.0 References: <20200221235307.535fb453@tachikoma.lepiller.eu> <87h7zfel26.fsf@gnu.org> In-Reply-To: <87h7zfel26.fsf@gnu.org> From: zimoun Date: Tue, 25 Feb 2020 16:21:24 +0100 Message-ID: Subject: Re: [bug#39728] [PATCH] Allow parallel downloads and builds To: =?UTF-8?Q?Ludovic_Court=C3=A8s?= Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 39728 Cc: Julien Lepiller , 39728@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Hi Julien, On Mon, 24 Feb 2020 at 22:43, Ludovic Court=C3=A8s wrote: > Julien Lepiller skribis: > > This patch allows to count builds and downloads separately. The idea is > > that downloads need bandwidth, but no CPU, while builds do not need > > bandwidth, but need CPU. With this patch, guix will be able to download > > substitutes while building unrelated packages. Currently, guix needs to > > wait for the download to finish before proceeding to the build. This > > should reduce the time of guix commands that need to build and download > > things at the same time. > > > > What do you think? > > I think it=E2=80=99s a good idea! > > I wonder what the UI will look like: (guix status) would no longer > display a progress bar when there=E2=80=99s more than on job (build or do= wnload) > taking place at the same time. Speaking about progress bar, it could be nice (as an improvement) to have a concurrent progress bar. As an example, see: http://hackage.haskell.org/package/concurrent-output > It would be great if you could test this patch for your daily usage. I > find it surprisingly easy to break things in the daemon. :-) How can I do that? After the 'make', how can change the daemon? And then revert it again to the default one? Cheers, simon From debbugs-submit-bounces@debbugs.gnu.org Tue Feb 25 10:39:36 2020 Received: (at 39728) by debbugs.gnu.org; 25 Feb 2020 15:39:36 +0000 Received: from localhost ([127.0.0.1]:55995 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1j6cJ5-0004pN-PW for submit@debbugs.gnu.org; Tue, 25 Feb 2020 10:39:35 -0500 Received: from lepiller.eu ([89.234.186.109]:58262) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1j6cJ4-0004pE-0P for 39728@debbugs.gnu.org; Tue, 25 Feb 2020 10:39:35 -0500 Received: from lepiller.eu (localhost [127.0.0.1]) by lepiller.eu (OpenSMTPD) with ESMTP id 965ab102; Tue, 25 Feb 2020 15:39:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed; d=lepiller.eu; h=date :in-reply-to:references:mime-version:content-type :content-transfer-encoding:subject:to:cc:from:message-id; s= dkim; bh=m/swkUwBhV8Ro1mXkZ2+YUjBv4r05WyVXfmkEq/2FRI=; b=Dask9Cs fpchC2kDn+HJGRNW6/CPQWxcc1mCfeqlb9K+xeaGgo1Bgw9u/4XXzVu+M5uQzMoH w0z/UcecFlqFZEw1HXWHe85yTz/d/3Pz45d6f5+QLlvIg/MCwu2mW3wp5XM1VYe1 ffeu1yEmi2y2qUNYXRNoSNyop1+W/e2fiCf3ieJll6YclNwBOTfJpswPHG90xbKF CZRfNTOfS+jRoR3vsQVqQh7TruJEHclny6PGyoJFSJEKZrrZOSysWs8/ZpWXh/Tl xRDIXkSHwZ8fRXuf3qxFHzYjoGjg4L04F8pCy3Q9ZYibVnsSofihAe8f6m2Ok7Ss uTHp4rUtq9aCLnw== Received: by lepiller.eu (OpenSMTPD) with ESMTPSA id d753005e (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256:NO); Tue, 25 Feb 2020 15:39:31 +0000 (UTC) Date: Tue, 25 Feb 2020 10:39:17 -0500 User-Agent: K-9 Mail for Android In-Reply-To: References: <20200221235307.535fb453@tachikoma.lepiller.eu> <87h7zfel26.fsf@gnu.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Subject: Re: [bug#39728] [PATCH] Allow parallel downloads and builds To: zimoun , =?ISO-8859-1?Q?Ludovic_Court=E8s?= From: Julien Lepiller Message-ID: <62A3FE42-52DD-4E51-9AD1-242B86B6A391@lepiller.eu> X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 39728 Cc: 39728@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Le 25 f=C3=A9vrier 2020 10:21:24 GMT-05:00, zimoun a =C3=A9crit : >Hi Julien, > >On Mon, 24 Feb 2020 at 22:43, Ludovic Court=C3=A8s wrote= : >> Julien Lepiller skribis: > >> > This patch allows to count builds and downloads separately=2E The >idea is >> > that downloads need bandwidth, but no CPU, while builds do not need >> > bandwidth, but need CPU=2E With this patch, guix will be able to >download >> > substitutes while building unrelated packages=2E Currently, guix >needs to >> > wait for the download to finish before proceeding to the build=2E >This >> > should reduce the time of guix commands that need to build and >download >> > things at the same time=2E >> > >> > What do you think? >> >> I think it=E2=80=99s a good idea! >> >> I wonder what the UI will look like: (guix status) would no longer >> display a progress bar when there=E2=80=99s more than on job (build or >download) >> taking place at the same time=2E > >Speaking about progress bar, it could be nice (as an improvement) to >have a concurrent progress bar=2E As an example, see: > >http://hackage=2Ehaskell=2Eorg/package/concurrent-output > > >> It would be great if you could test this patch for your daily usage=2E= =20 >I >> find it surprisingly easy to break things in the daemon=2E :-) > >How can I do that? >After the 'make', how can change the daemon? And then revert it again >to the default one? > > >Cheers, >simon On the guix system, try (in a guix environment guix) sudo herd stop guix-d= aemon; sudo =2E/pre-inst-env guix-daemon --build-users-group=3Dguixbuild=20 To revert back, kill this (^C) and sudo herd start guix-daemon=2E From debbugs-submit-bounces@debbugs.gnu.org Wed Feb 26 05:36:41 2020 Received: (at 39728) by debbugs.gnu.org; 26 Feb 2020 10:36:41 +0000 Received: from localhost ([127.0.0.1]:56449 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1j6u3V-0002r8-M8 for submit@debbugs.gnu.org; Wed, 26 Feb 2020 05:36:41 -0500 Received: from relay7-d.mail.gandi.net ([217.70.183.200]:35271) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1j6u3T-0002qu-Qg for 39728@debbugs.gnu.org; Wed, 26 Feb 2020 05:36:40 -0500 X-Originating-IP: 92.169.129.147 Received: from bababa (lfbn-idf2-1-1315-147.w92-169.abo.wanadoo.fr [92.169.129.147]) (Authenticated sender: mail@ambrevar.xyz) by relay7-d.mail.gandi.net (Postfix) with ESMTPSA id 0108320016; Wed, 26 Feb 2020 10:36:32 +0000 (UTC) From: Pierre Neidhardt To: Julien Lepiller Subject: Re: [bug#39728] [PATCH] Allow parallel downloads and builds References: <20200221235307.535fb453@tachikoma.lepiller.eu> <87h7zfel26.fsf@gnu.org> <62A3FE42-52DD-4E51-9AD1-242B86B6A391@lepiller.eu> Date: Wed, 26 Feb 2020 11:36:32 +0100 In-Reply-To: <62A3FE42-52DD-4E51-9AD1-242B86B6A391@lepiller.eu> (Julien Lepiller's message of "Tue, 25 Feb 2020 10:39:17 -0500") Message-ID: <87zhd58wjz.fsf@ambrevar.xyz> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.3 (gnu/linux) MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" X-Spam-Score: 1.6 (+) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: Thank you so much for this, Julien! There was a thread on this topic on the mailing list: https://lists.gnu.org/archive/html/guix-devel/2019-11/msg00002.html (could not find the first email :p). There were a couple other issues that were mentioned there. This patch would be a first step towards more parallelization! Content analysis details: (1.6 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- 0.0 URIBL_BLOCKED ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [URIs: gnu.org] -0.7 RCVD_IN_DNSWL_LOW RBL: Sender listed at https://www.dnswl.org/, low trust [217.70.183.200 listed in list.dnswl.org] 1.8 PDS_OTHER_BAD_TLD Untrustworthy TLDs [URI: ambrevar.xyz (xyz)] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.0 SPF_PASS SPF: sender matches SPF record 0.5 FROM_SUSPICIOUS_NTLD From abused NTLD X-Debbugs-Envelope-To: 39728 Cc: Ludovic =?utf-8?Q?Court=C3=A8s?= , 39728@debbugs.gnu.org, zimoun X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 1.6 (+) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: Thank you so much for this, Julien! There was a thread on this topic on the mailing list: https://lists.gnu.org/archive/html/guix-devel/2019-11/msg00002.html (could not find the first email :p). There were a couple other issues that were mentioned there. This patch would be a first step towards more parallelization! Content analysis details: (1.6 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- 0.0 URIBL_BLOCKED ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [URIs: ambrevar.xyz] -0.7 RCVD_IN_DNSWL_LOW RBL: Sender listed at https://www.dnswl.org/, low trust [217.70.183.200 listed in list.dnswl.org] 1.8 PDS_OTHER_BAD_TLD Untrustworthy TLDs [URI: ambrevar.xyz (xyz)] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.0 SPF_PASS SPF: sender matches SPF record 1.0 BULK_RE_SUSP_NTLD Precedence bulk and RE: from a suspicious TLD 0.5 FROM_SUSPICIOUS_NTLD From abused NTLD -1.0 MAILING_LIST_MULTI Multiple indicators imply a widely-seen list manager --=-=-= Content-Type: text/plain Content-Transfer-Encoding: quoted-printable Thank you so much for this, Julien! There was a thread on this topic on the mailing list: https://lists.gnu.org/archive/html/guix-devel/2019-11/msg00002.html (could not find the first email :p). There were a couple other issues that were mentioned there. This patch would be a first step towards more parallelization! =2D-=20 Pierre Neidhardt https://ambrevar.xyz/ --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQEzBAEBCAAdFiEEUPM+LlsMPZAEJKvom9z0l6S7zH8FAl5WSjAACgkQm9z0l6S7 zH8IqwgArM56uM65zPsqx/L/OPSJGWRxh9a2nvKlnjwfn5OrksFiiDZYBZ+EDUxd lgZHHr2oChbBgmKxEP3pDQN9d62Ikze5G9oFgQulXkmOEfjvePDUEbcVWcjxTclG z8m1cbef2AoIzc8ITVyp9yND1HWTgFGSximmN2Y2NJGTDfOy+1WSSXdNil8kZQWC 1TgiJNzFg6nIfjeX2p+wVM64w6x1NeT64p+blNUpDj8/ReHY69sbr156PC1rzTB6 X3dkdGIIQX2vQJ5wsl90MVT7AMlW1bMPc2nUn37qhzb8fvpwiExt5NMcAMGdYN1K C2lg+3jlzs62QkOfmUUVuGplfDilFA== =fs5l -----END PGP SIGNATURE----- --=-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Tue Jul 13 22:49:12 2021 Received: (at 39728) by debbugs.gnu.org; 14 Jul 2021 02:49:12 +0000 Received: from localhost ([127.0.0.1]:42508 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1m3UxU-0007FR-EX for submit@debbugs.gnu.org; Tue, 13 Jul 2021 22:49:12 -0400 Received: from mail-qv1-f49.google.com ([209.85.219.49]:43854) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1m3UxR-0007F7-5I for 39728@debbugs.gnu.org; Tue, 13 Jul 2021 22:49:10 -0400 Received: by mail-qv1-f49.google.com with SMTP id i4so313219qvq.10 for <39728@debbugs.gnu.org>; Tue, 13 Jul 2021 19:49:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version; bh=tHw0CbrfuNo6JCyO3VrTK3BdQXrHqG/U8ZGrpG0WNdk=; b=HowAAhIrt1H3GU3QDQKhdTcdQ7CLMjM7SBjfyoDspVqrqEFDXHXKFP4bqdcIpS+nNx v1A4H1FAQujipiCn9GYuCgMpQaoTvzHZuo3bvFpTjOTaJVNL9nMK+lbGep4zdrS56olc 0j9z0h7unhN0RYpamPEQB4h2mDQ8Z/Itpg8dz2j3JdFsMpUnqKoz8VIJ5Fd1/DLJj6vZ RnLVYYwATyLC13jXmE3pOJeCjyz1rd3y4vcTjPBoPsigCe0N04gfIaogqwArkyeFrWTk Q3R9JiK1ZVTo4C72f6UPdh9XrF7YOOEvKjlgbPWTDAvoXJuvdpqNfTZeZwAtwf1BXpAl PmGA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to :message-id:user-agent:mime-version; bh=tHw0CbrfuNo6JCyO3VrTK3BdQXrHqG/U8ZGrpG0WNdk=; b=Egdp/O03wwMDYqf3W3yIvQZJyArAeVlrnmW/dDLxU/hzzcdoQFhINzBcDy2jaOVapE HpoSNshGrcvpPlY/1UzmvlevCdrBjI4XSE/5XezSkd2ckmmnT5BCrzWZsZiNvTqiS6qy qJdXU2TrTbL/Nf9A8Dvb6LPHUnO4SMF1qk0iqABb+4D2UslPXLxGJja8qb+yLFR5WPC8 CNmT83Sk1hf6+vIpB8JwUx51xRtWeLVuj2WE/IaT2CDtDt1es1/pD75vuRDcommwg18c 7IA0epU9UBe8S6T3E1jxcCppEoQ1mh0QS8lv/i/KvfOriaVPqhLSrFvnWaadnKyr4n7L XzeA== X-Gm-Message-State: AOAM532fW/N64XgV/nVOF50ASmXBeuV5tz6gymyhve9SGivzEvuv6iHw adVqAKK4Q+smvQBmUWPepaIFMLMJQLapf6Ip X-Google-Smtp-Source: ABdhPJxaRO1wBOdQ94LLRRtM6LOHUH7LXxxGOvsrZ4p/VpUVjYoGnjDHi/gjpzid/zX64lKUQxPUwA== X-Received: by 2002:a0c:d605:: with SMTP id c5mr8540737qvj.7.1626230943518; Tue, 13 Jul 2021 19:49:03 -0700 (PDT) Received: from hurd (dsl-10-136-19.b2b2c.ca. [72.10.136.19]) by smtp.gmail.com with ESMTPSA id y2sm416856qkl.58.2021.07.13.19.49.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Jul 2021 19:49:03 -0700 (PDT) From: Maxim Cournoyer To: Julien Lepiller Subject: Re: bug#39728: [PATCH] Allow parallel downloads and builds References: <20200221235307.535fb453@tachikoma.lepiller.eu> Date: Tue, 13 Jul 2021 22:49:02 -0400 In-Reply-To: <20200221235307.535fb453@tachikoma.lepiller.eu> (Julien Lepiller's message of "Fri, 21 Feb 2020 23:53:07 +0100") Message-ID: <874kcxr4o1.fsf@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 39728 Cc: 39728@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Hello! Julien Lepiller writes: > Hi guix! > > This patch allows to count builds and downloads separately. The idea is > that downloads need bandwidth, but no CPU, while builds do not need > bandwidth, but need CPU. With this patch, guix will be able to download > substitutes while building unrelated packages. Currently, guix needs to > wait for the download to finish before proceeding to the build. This > should reduce the time of guix commands that need to build and download > things at the same time. > > What do you think? Looks like a neat improvement! Could you provide a follow-up to Ludovic's review? It seems not much is missing (minor cosmetic changes to commit messages + code and more importantly, the accompanying documentation update). Thank you! Maxim From debbugs-submit-bounces@debbugs.gnu.org Sun Nov 21 17:50:01 2021 Received: (at 39728) by debbugs.gnu.org; 21 Nov 2021 22:50:01 +0000 Received: from localhost ([127.0.0.1]:46285 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1movel-0001MF-IR for submit@debbugs.gnu.org; Sun, 21 Nov 2021 17:50:01 -0500 Received: from lepiller.eu ([89.234.186.109]:39638) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1movei-0001M4-Sr for 39728@debbugs.gnu.org; Sun, 21 Nov 2021 17:49:54 -0500 Received: from lepiller.eu (localhost [127.0.0.1]) by lepiller.eu (OpenSMTPD) with ESMTP id f305089e; Sun, 21 Nov 2021 22:49:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed; d=lepiller.eu; h=date:from :to:cc:subject:message-id:in-reply-to:references:mime-version :content-type; s=dkim; bh=IsYhBHenT5UmJSqrlIR0fSyMLjS2/+gRmIoF6m MOaFU=; b=F9TAWIhF9oi5030Je8p5hoS0L9/TPs/tKwQZdRdNQsHfC0e7wUNNs2 AieE3bgorJzgU7PZv13XYwYDpEe/G8wHT2i1BXV1TfJZipQ3ezLioBq1oS8Gsi7m 7KXrsbj7AphftkKhZ9si9AWeVSkgyORYMFSpA3Z6//pPHFJ0deksZ08vn73gJTs8 XAxGsf+fQdRztC3TW+mG1i3M31rfz3gP8QRbChSwVoJfEf5CKrsdNinaAl8ux2jd igGeCzr6ePwEjMkcMaWyWFqb8lt9iP7LixEbz8AbJhbmOk0kFAbX2ZCRNLQc4wmt XkHR9GAkNJcO6rXcSTAbiwmXpu7bcqPw== Received: by lepiller.eu (OpenSMTPD) with ESMTPSA id cc2b7bcb (TLSv1.3:AEAD-AES256-GCM-SHA384:256:NO); Sun, 21 Nov 2021 22:49:48 +0000 (UTC) Date: Sun, 21 Nov 2021 23:49:41 +0100 From: Julien Lepiller To: Pierre Neidhardt Subject: Re: [bug#39728] [PATCH v2] Allow parallel downloads and builds Message-ID: <20211121234941.32724b68@tachikoma.lepiller.eu> In-Reply-To: <87zhd58wjz.fsf@ambrevar.xyz> References: <20200221235307.535fb453@tachikoma.lepiller.eu> <87h7zfel26.fsf@gnu.org> <62A3FE42-52DD-4E51-9AD1-242B86B6A391@lepiller.eu> <87zhd58wjz.fsf@ambrevar.xyz> X-Mailer: Claws Mail 4.0.0 (GTK+ 3.24.24; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="MP_/CmFwiqPWXairFzvp0vfW.vs" X-Spam-Score: 2.0 (++) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: Le Wed, 26 Feb 2020 11:36:32 +0100, Pierre Neidhardt a écrit : > Thank you so much for this, Julien! > > There was a thread on this topic on the mailing list: > https://lists.gnu.org/archive/html/guix-devel/2019-11/msg00002.html > (could not find the first email [...] Content analysis details: (2.0 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 SPF_PASS SPF: sender matches SPF record 2.0 PDS_OTHER_BAD_TLD Untrustworthy TLDs [URI: ambrevar.xyz (xyz)] -0.0 SPF_HELO_PASS SPF: HELO matches SPF record X-Debbugs-Envelope-To: 39728 Cc: Ludovic =?UTF-8?B?Q291cnTDqHM=?= , 39728@debbugs.gnu.org, zimoun X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 1.0 (+) --MP_/CmFwiqPWXairFzvp0vfW.vs Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline Le Wed, 26 Feb 2020 11:36:32 +0100, Pierre Neidhardt a =C3=A9crit : > Thank you so much for this, Julien! >=20 > There was a thread on this topic on the mailing list: > https://lists.gnu.org/archive/html/guix-devel/2019-11/msg00002.html > (could not find the first email :p). >=20 > There were a couple other issues that were mentioned there. > This patch would be a first step towards more parallelization! >=20 Hi! After so long, I managed to find the time to go over the comments and improve my patches. I tested the new daemon for a bit, and it's working as expected so far :D --MP_/CmFwiqPWXairFzvp0vfW.vs Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=0001-daemon-Count-build-and-download-jobs-separately.patch >From 3dee3844b6da1d70b883391f5b2d23e331d6a5ad Mon Sep 17 00:00:00 2001 Message-Id: <3dee3844b6da1d70b883391f5b2d23e331d6a5ad.1637534792.git.julien@lepiller.eu> From: Julien Lepiller Date: Sun, 21 Nov 2021 22:35:09 +0100 Subject: [PATCH 1/2] daemon: Count build and download jobs separately. This allows us to run downloads (that take bandwith) and builds (that take CPU time) independently from one another. * nix/nix-daemon/guix-daemon.cc: Add a max-download-jobs option. * nix/nix-daemon/nix-daemon.cc: Add a max-download-jobs option. * nix/libstore/globals.hh (Settings): Add a maxDownloadJobs setting. * nix/libstore/globals.cc (Settings::Settings): Add a default value to it. (Settings::update): Add it. * nix/libstore/build.cc (Slot): New enum type. (Child): Use `Slot` enum to record the job slot of the task, if any. (Goal): Separate build and download slot members and methods. (DerivationGoal::tryToBuild): Check number of available slots in build or download tasks. (DerivationGoal::tryBuildHook, SubstitutionGoal::tryToRun): Check number of download slots instead of build slots. (Worker::Worker, Worker::childStarted, Worker::childTerminated): Manage build and download slots separately. * doc/guix.texi (Invoking guix-daemon): Document `--max-downloads`. --- doc/guix.texi | 6 ++ nix/libstore/build.cc | 105 ++++++++++++++++++++++++++-------- nix/libstore/globals.cc | 2 + nix/libstore/globals.hh | 3 + nix/nix-daemon/guix-daemon.cc | 5 ++ nix/nix-daemon/nix-daemon.cc | 2 +- 6 files changed, 97 insertions(+), 26 deletions(-) diff --git a/doc/guix.texi b/doc/guix.texi index 59ceb4477a..b8de53c53b 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -1619,6 +1619,12 @@ Invoking guix-daemon locally; instead, the daemon will offload builds (@pxref{Daemon Offload Setup}), or simply fail. +@item --max-downloads=@var{n} +@itemx -D @var{n} +Allow at most @var{n} download jobs in parallel. The default value is +@code{1}. Since some downloads always need to happen, setting it to +@code{0} does not make any sense and will be ignored. + @item --max-silent-time=@var{seconds} When the build or substitution process remains silent for more than @var{seconds}, terminate it and report a build failure. diff --git a/nix/libstore/build.cc b/nix/libstore/build.cc index 5697ae5a43..2fbbd81af2 100644 --- a/nix/libstore/build.cc +++ b/nix/libstore/build.cc @@ -196,6 +196,9 @@ bool CompareGoalPtrs::operator() (const GoalPtr & a, const GoalPtr & b) { } +/* A list of possible slots */ +typedef enum { slotNone, slotBuild, slotDownload } Slot; + /* A mapping used to remember for each child process to what goal it belongs, and file descriptors for receiving log data and output path creation commands. */ @@ -204,7 +207,7 @@ struct Child WeakGoalPtr goal; set fds; bool respectTimeouts; - bool inBuildSlot; + Slot inSlot; time_t lastOutput; /* time we last got output on stdout/stderr */ time_t timeStarted; }; @@ -232,10 +235,15 @@ private: /* Child processes currently running. */ Children children; - /* Number of build slots occupied. This includes local builds and - substitutions but not remote builds via the build hook. */ + /* Number of build slots occupied. This includes local builds but not + fixed-output derivations, substitution, built-ins and remote builds + via the build hook. */ unsigned int nrLocalBuilds; + /* Number of download slots occupied. This includes fixed-output + derivations, substitution and built-ins. */ + unsigned int nrDownloads; + /* Maps used to prevent multiple instantiations of a goal for the same derivation / path. */ WeakGoalMap derivationGoals; @@ -277,15 +285,18 @@ public: /* Wake up a goal (i.e., there is something for it to do). */ void wakeUp(GoalPtr goal); - /* Return the number of local build and substitution processes - currently running (but not remote builds via the build - hook). */ + /* Return the number of local build processes currently running (but + not remote builds via the build hook). */ unsigned int getNrLocalBuilds(); - /* Registers a running child process. `inBuildSlot' means that - the process counts towards the jobs limit. */ + /* Return the number of downloads currently running. */ + unsigned int getNrDownloads(); + + /* Registers a running child process. `inSlot' indicates whether + the process counts towards the jobs limit, and which one. */ void childStarted(GoalPtr goal, pid_t pid, - const set & fds, bool inBuildSlot, bool respectTimeouts); + const set & fds, Slot inSlot, + bool respectTimeouts); /* Unregisters a running child process. `wakeSleepers' should be false if there is no sense in waking up goals that are sleeping @@ -297,6 +308,10 @@ public: might be right away). */ void waitForBuildSlot(GoalPtr goal); + /* Put `goal' to sleep until a download slot becomes available (which + might be right away). */ + void waitForDownloadSlot(GoalPtr goal); + /* Wait for any goal to finish. Pretty indiscriminate way to wait for some resource that some other goal is holding. */ void waitForAnyGoal(GoalPtr goal); @@ -1249,16 +1264,28 @@ void DerivationGoal::tryToBuild() } } - /* Make sure that we are allowed to start a build. If this - derivation prefers to be done locally, do it even if - maxBuildJobs is 0. */ + /* Make sure that, if this is a build (not a download), we are allowed to + start a build. If this derivation prefers to be done locally, do it + even if maxBuildJobs is 0. */ unsigned int curBuilds = worker.getNrLocalBuilds(); - if (curBuilds >= settings.maxBuildJobs && !(buildLocally && curBuilds == 0)) { + if (curBuilds >= settings.maxBuildJobs && !(buildLocally && curBuilds == 0) && + !fixedOutput) { worker.waitForBuildSlot(shared_from_this()); outputLocks.unlock(); return; } + /* Make sure that, if this is a download (not a build), we are allowed to + start a download. Even if maxDownloadJobs is 0, we always allow + downloads, so it is set to 1 */ + unsigned int curDownloads = worker.getNrDownloads(); + unsigned int maxDownloads = settings.maxDownloadJobs == 0 ? 1 : settings.maxDownloadJobs; + if (curDownloads >= maxDownloads && fixedOutput) { + worker.waitForDownloadSlot(shared_from_this()); + outputLocks.unlock(); + return; + } + try { /* Okay, we have to build. */ @@ -1546,7 +1573,7 @@ HookReply DerivationGoal::tryBuildHook() set fds; fds.insert(hook->fromAgent.readSide); fds.insert(hook->builderOut.readSide); - worker.childStarted(shared_from_this(), hook->pid, fds, false, true); + worker.childStarted(shared_from_this(), hook->pid, fds, slotNone, true); if (settings.printBuildTrace) printMsg(lvlError, format("@ build-started %1% - %2% %3% %4%") @@ -1940,7 +1967,8 @@ void DerivationGoal::startBuilder() pid.setSeparatePG(true); builderOut.writeSide.close(); worker.childStarted(shared_from_this(), pid, - singleton >(builderOut.readSide), true, true); + singleton >(builderOut.readSide), + fixedOutput? slotDownload: slotBuild, true); /* Check if setting up the build environment failed. */ string msg = readLine(builderOut.readSide); @@ -2979,11 +3007,11 @@ void SubstitutionGoal::tryToRun() trace("trying to run"); /* Make sure that we are allowed to start a build. Note that even - is maxBuildJobs == 0 (no local builds allowed), we still allow - a substituter to run. This is because substitutions cannot be - distributed to another machine via the build hook. */ - if (worker.getNrLocalBuilds() >= (settings.maxBuildJobs == 0 ? 1 : settings.maxBuildJobs)) { - worker.waitForBuildSlot(shared_from_this()); + is maxDownloadJobs == 0 (no downloads allowed), we still allow + a substituter to run. This is because we always need to download, so + not allowing is meaningless. */ + if (worker.getNrDownloads() >= (settings.maxDownloadJobs == 0 ? 1 : settings.maxDownloadJobs)) { + worker.waitForDownloadSlot(shared_from_this()); return; } @@ -3042,7 +3070,8 @@ void SubstitutionGoal::tryToRun() set fds; fds.insert(substituter->fromAgent.readSide); fds.insert(substituter->builderOut.readSide); - worker.childStarted(shared_from_this(), substituter->pid, fds, true, true); + worker.childStarted(shared_from_this(), substituter->pid, fds, + slotDownload, true); state = &SubstitutionGoal::finished; @@ -3206,6 +3235,7 @@ Worker::Worker(LocalStore & store) if (working) abort(); working = true; nrLocalBuilds = 0; + nrDownloads = 0; lastWokenUp = 0; permanentFailure = false; timedOut = false; @@ -3299,18 +3329,25 @@ unsigned Worker::getNrLocalBuilds() } +unsigned Worker::getNrDownloads() +{ + return nrDownloads; +} + + void Worker::childStarted(GoalPtr goal, - pid_t pid, const set & fds, bool inBuildSlot, + pid_t pid, const set & fds, Slot inSlot, bool respectTimeouts) { Child child; child.goal = goal; child.fds = fds; child.timeStarted = child.lastOutput = time(0); - child.inBuildSlot = inBuildSlot; + child.inSlot = inSlot; child.respectTimeouts = respectTimeouts; children[pid] = child; - if (inBuildSlot) nrLocalBuilds++; + if (inSlot == slotBuild) nrLocalBuilds++; + if (inSlot == slotDownload) nrDownloads++; } @@ -3321,11 +3358,16 @@ void Worker::childTerminated(pid_t pid, bool wakeSleepers) Children::iterator i = children.find(pid); assert(i != children.end()); - if (i->second.inBuildSlot) { + if (i->second.inSlot == slotBuild) { assert(nrLocalBuilds > 0); nrLocalBuilds--; } + if (i->second.inSlot == slotDownload) { + assert(nrDownloads > 0); + nrDownloads--; + } + children.erase(pid); if (wakeSleepers) { @@ -3351,6 +3393,19 @@ void Worker::waitForBuildSlot(GoalPtr goal) } +void Worker::waitForDownloadSlot(GoalPtr goal) +{ + debug("wait for download slot"); + /* Disabling all downloads does not make any sense, so we make sure we + can download with at least one job. */ + unsigned int maxDownloads = settings.maxDownloadJobs == 0 ? 1 : settings.maxDownloadJobs; + if (getNrDownloads() < maxDownloads) + wakeUp(goal); /* we can do it right away */ + else + addToWeakGoals(wantingToBuild, goal); +} + + void Worker::waitForAnyGoal(GoalPtr goal) { debug("wait for any goal"); diff --git a/nix/libstore/globals.cc b/nix/libstore/globals.cc index 0cc001fbe4..416033718d 100644 --- a/nix/libstore/globals.cc +++ b/nix/libstore/globals.cc @@ -29,6 +29,7 @@ Settings::Settings() tryFallback = false; buildVerbosity = lvlError; maxBuildJobs = 1; + maxDownloadJobs = 1; buildCores = 1; readOnlyMode = false; thisSystem = SYSTEM; @@ -118,6 +119,7 @@ void Settings::update() { _get(tryFallback, "build-fallback"); _get(maxBuildJobs, "build-max-jobs"); + _get(maxDownloadJobs, "download-max-jobs"); _get(buildCores, "build-cores"); _get(thisSystem, "system"); _get(multiplexedBuildOutput, "multiplexed-build-output"); diff --git a/nix/libstore/globals.hh b/nix/libstore/globals.hh index 27616a2283..c033f8ed56 100644 --- a/nix/libstore/globals.hh +++ b/nix/libstore/globals.hh @@ -90,6 +90,9 @@ struct Settings { /* Maximum number of parallel build jobs. 0 means unlimited. */ unsigned int maxBuildJobs; + /* Maximum number of parallel download jobs. 0 means 1. */ + unsigned int maxDownloadJobs; + /* Number of CPU cores to utilize in parallel within a build, i.e. by passing this number to Make via '-j'. 0 means that the number of actual CPU cores on the local host ought to be diff --git a/nix/nix-daemon/guix-daemon.cc b/nix/nix-daemon/guix-daemon.cc index 36a06a3fae..f1b60b0050 100644 --- a/nix/nix-daemon/guix-daemon.cc +++ b/nix/nix-daemon/guix-daemon.cc @@ -100,6 +100,8 @@ static const struct argp_option options[] = }, { "max-jobs", 'M', n_("N"), 0, n_("allow at most N build jobs") }, + { "max-downloads", 'D', n_("N"), 0, + n_("allow at most N download jobs") }, { "timeout", GUIX_OPT_TIMEOUT, n_("SECONDS"), 0, n_("mark builds as failed after SECONDS of activity") }, { "max-silent-time", GUIX_OPT_MAX_SILENT_TIME, n_("SECONDS"), 0, @@ -285,6 +287,9 @@ parse_opt (int key, char *arg, struct argp_state *state) case 'M': settings.set ("build-max-jobs", arg); break; + case 'D': + settings.set ("download-max-jobs", arg); + break; case GUIX_OPT_TIMEOUT: settings.set ("build-timeout", arg); break; diff --git a/nix/nix-daemon/nix-daemon.cc b/nix/nix-daemon/nix-daemon.cc index 497de11a04..4b2f8bcf22 100644 --- a/nix/nix-daemon/nix-daemon.cc +++ b/nix/nix-daemon/nix-daemon.cc @@ -610,7 +610,7 @@ static void performOp(bool trusted, unsigned int clientVersion, string value = readString(from); if (name == "build-timeout" || name == "build-max-silent-time" || name == "build-max-jobs" || name == "build-cores" - || name == "build-repeat" + || name == "build-repeat" || name == "download-max-jobs" || name == "multiplexed-build-output") settings.set(name, value); else if (name == "user-name" -- 2.33.1 --MP_/CmFwiqPWXairFzvp0vfW.vs Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=0002-guix-Support-specifying-max-download-jobs.patch >From d8b39af21d97dc3f0d3057ceb7ba91a93ff8d3ec Mon Sep 17 00:00:00 2001 Message-Id: In-Reply-To: <3dee3844b6da1d70b883391f5b2d23e331d6a5ad.1637534792.git.julien@lepiller.eu> References: <3dee3844b6da1d70b883391f5b2d23e331d6a5ad.1637534792.git.julien@lepiller.eu> From: Julien Lepiller Date: Sun, 21 Nov 2021 23:26:11 +0100 Subject: [PATCH 2/2] guix: Support specifying max download jobs. * guix/store.scm (set-build-options): Optionally send max-download-jobs. * guix/scripts/build.scm (set-build-options-from-command-line) (%standard-build-options): Support `--max-downloads` * doc/guix.texi (Common Build Options): Document it. --- doc/guix.texi | 6 ++++++ guix/scripts/build.scm | 8 ++++++++ guix/store.scm | 5 +++++ 3 files changed, 19 insertions(+) diff --git a/doc/guix.texi b/doc/guix.texi index b8de53c53b..0afefbd416 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -10846,6 +10846,12 @@ Common Build Options guix-daemon, @option{--max-jobs}}, for details about this option and the equivalent @command{guix-daemon} option. +@item --max-downloads=@var{n} +@itemx -D @var{n} +Allow at most @var{n} download jobs in parallel. @xref{Invoking +guix-daemon, @option{--max-downloads}}, for details about this option and the +equivalent @command{guix-daemon} option. + @item --debug=@var{level} Produce debugging output coming from the build daemon. @var{level} must be an integer between 0 and 5; higher means more verbose output. Setting a level of diff --git a/guix/scripts/build.scm b/guix/scripts/build.scm index 97e2f5a167..0d0199eccd 100644 --- a/guix/scripts/build.scm +++ b/guix/scripts/build.scm @@ -207,6 +207,7 @@ (define (set-build-options-from-command-line store opts) #:rounds (assoc-ref opts 'rounds) #:build-cores (assoc-ref opts 'cores) #:max-build-jobs (assoc-ref opts 'max-jobs) + #:max-download-jobs (assoc-ref opts 'max-downloads) #:fallback? (assoc-ref opts 'fallback?) #:use-substitutes? (assoc-ref opts 'substitutes?) #:substitute-urls (assoc-ref opts 'substitute-urls) @@ -316,6 +317,13 @@ (define %standard-build-options (let ((c (false-if-exception (string->number arg)))) (if c (apply values (alist-cons 'max-jobs c result) rest) + (leave (G_ "not a number: '~a' option argument: ~a~%") + name arg))))) + (option '(#\D "max-downloads") #t #f + (lambda (opt name arg result . rest) + (let ((c (false-if-exception (string->number arg)))) + (if c + (apply values (alist-cons 'max-downloads c result) rest) (leave (G_ "not a number: '~a' option argument: ~a~%") name arg))))))) diff --git a/guix/store.scm b/guix/store.scm index 7388953d15..6b5b9262b1 100644 --- a/guix/store.scm +++ b/guix/store.scm @@ -803,6 +803,7 @@ (define* (set-build-options server (verbosity 0) rounds ;number of build rounds max-build-jobs + max-download-jobs timeout max-silent-time (offload? #t) @@ -896,6 +897,10 @@ (define* (set-build-options server `(("build-max-jobs" . ,(number->string max-build-jobs))) '()) + ,@(if max-download-jobs + `(("download-max-jobs" + . ,(number->string max-download-jobs))) + '()) ,@(if build-cores `(("build-cores" . ,(number->string build-cores))) '()) -- 2.33.1 --MP_/CmFwiqPWXairFzvp0vfW.vs-- From debbugs-submit-bounces@debbugs.gnu.org Thu Nov 25 07:53:46 2021 Received: (at 39728) by debbugs.gnu.org; 25 Nov 2021 12:53:46 +0000 Received: from localhost ([127.0.0.1]:56116 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mqEG1-0001Hz-Ha for submit@debbugs.gnu.org; Thu, 25 Nov 2021 07:53:46 -0500 Received: from eggs.gnu.org ([209.51.188.92]:56966) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mqEFx-0001Hj-Mm for 39728@debbugs.gnu.org; Thu, 25 Nov 2021 07:53:45 -0500 Received: from [2001:470:142:3::e] (port=39630 helo=fencepost.gnu.org) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mqEFo-0006ls-K2; Thu, 25 Nov 2021 07:53:32 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:In-Reply-To:Date:References:Subject:To: From; bh=rSsv+GmSBgxR9+IphpgR2oym+1HHEVKjxajDTXRPunU=; b=KLcWDNSfACDY0nE2VUgp y6XbSODWsAcDnVdfrMyeAQ1nvx4DAkfSmT7cKDg4Ky5hdo+oRYxlDbGjOACTxJCAEFQObj4xwn87l ZoRDjJuOV+1PlJSzhaqjpZ4MQFl5BUPLqf/pqJhF5BZo/hnnSRz/hq7K7P5OKPegOLpKPGfGwdv9R 3lkNsILWB9xqZe8D0AH3/iz9W1ehzsv+433tksYl5hVBuUJ+8IV2gS+ZQhwpn5tyIb9VzO7izH4Pg ir+gZKqCRn03jTkzlIf4MOTBiDik83F08NvbrD67MPmBSEKLuvdbf3FU8BRquWKFkdZA6qzC16olF OsPZZ/e3oJ+jZA==; Received: from nat-eduroam-36-gw-01-bso.bordeaux.inria.fr ([194.199.1.36]:56190 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mqEFo-0004UP-8G; Thu, 25 Nov 2021 07:53:32 -0500 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Julien Lepiller Subject: Re: [bug#39728] [PATCH v2] Allow parallel downloads and builds References: <20200221235307.535fb453@tachikoma.lepiller.eu> <87h7zfel26.fsf@gnu.org> <62A3FE42-52DD-4E51-9AD1-242B86B6A391@lepiller.eu> <87zhd58wjz.fsf@ambrevar.xyz> <20211121234941.32724b68@tachikoma.lepiller.eu> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 5 Frimaire an 230 de la =?utf-8?Q?R=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Thu, 25 Nov 2021 13:53:25 +0100 In-Reply-To: <20211121234941.32724b68@tachikoma.lepiller.eu> (Julien Lepiller's message of "Sun, 21 Nov 2021 23:49:41 +0100") Message-ID: <87ilwge6re.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 39728 Cc: 39728@debbugs.gnu.org, Pierre Neidhardt , zimoun X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hello, Julien Lepiller skribis: > After so long, I managed to find the time to go over the comments and > improve my patches. I tested the new daemon for a bit, and it's working > as expected so far :D On a recent daemon, have you seen cases where having multiple downloads in parallel speeds things up? The analysis in suggests that at the time you first submitted this patch, substitution speed (which is different from raw download speed) was often CPU-bound. This is no longer the case, meaning that downloads should now be network-bound or almost. Thanks, Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Thu Nov 25 08:02:01 2021 Received: (at 39728) by debbugs.gnu.org; 25 Nov 2021 13:02:01 +0000 Received: from localhost ([127.0.0.1]:56129 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mqEO1-0001Wa-7F for submit@debbugs.gnu.org; Thu, 25 Nov 2021 08:02:01 -0500 Received: from lepiller.eu ([89.234.186.109]:54130) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mqENx-0001WP-Al for 39728@debbugs.gnu.org; Thu, 25 Nov 2021 08:01:59 -0500 Received: from lepiller.eu (localhost [127.0.0.1]) by lepiller.eu (OpenSMTPD) with ESMTP id 06274774; Thu, 25 Nov 2021 13:01:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed; d=lepiller.eu; h=date:from :to:cc:subject:message-id:in-reply-to:references:mime-version :content-type:content-transfer-encoding; s=dkim; bh=TD2ARY0iD0xl +aAO/Fi5uIXnhD23vtOdh6Gg8T921VY=; b=mF6MWhAo9QB4BA5A0CtplGEMHLGQ vfC2fJ+L2nTXkr4EiczF9KkpGDMFTwZEwBKyor4P1MrlFW2FgoIuZ/7AllUaLDsq TUF8butOTYPcWnt0mxFge1eDUMUeHA3liglEI4Unzt2UWv7nZmTxjSFWyC0YdRxy PnpgJmuiQxy7UPZPC4IvoBVfwUkScn3pDQX4MvUiiCIgwdDkcbsWCtXkO2/V9I8g cbkPP6ZUhPHP+xsAQ9hVBVIO7rGcoqKZDjUQJqP8+SoCufGLWrhoGe9R7u7ultRs JKVuEmNt6A8tGpmliJXgKkhQXkuhGC5ZBYJvvq7leAVECj4dyJbcig2ruw== Received: by lepiller.eu (OpenSMTPD) with ESMTPSA id ba4f2205 (TLSv1.3:AEAD-AES256-GCM-SHA384:256:NO); Thu, 25 Nov 2021 13:01:54 +0000 (UTC) Date: Thu, 25 Nov 2021 14:01:45 +0100 From: Julien Lepiller To: Ludovic =?UTF-8?B?Q291cnTDqHM=?= Subject: Re: [bug#39728] [PATCH v2] Allow parallel downloads and builds Message-ID: <20211125140145.1c00decc@tachikoma.lepiller.eu> In-Reply-To: <87ilwge6re.fsf@gnu.org> References: <20200221235307.535fb453@tachikoma.lepiller.eu> <87h7zfel26.fsf@gnu.org> <62A3FE42-52DD-4E51-9AD1-242B86B6A391@lepiller.eu> <87zhd58wjz.fsf@ambrevar.xyz> <20211121234941.32724b68@tachikoma.lepiller.eu> <87ilwge6re.fsf@gnu.org> X-Mailer: Claws Mail 4.0.0 (GTK+ 3.24.24; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 39728 Cc: 39728@debbugs.gnu.org, Pierre Neidhardt , zimoun X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Le Thu, 25 Nov 2021 13:53:25 +0100, Ludovic Court=C3=A8s a =C3=A9crit : > Hello, >=20 > Julien Lepiller skribis: >=20 > > After so long, I managed to find the time to go over the comments > > and improve my patches. I tested the new daemon for a bit, and it's > > working as expected so far :D =20 >=20 > On a recent daemon, have you seen cases where having multiple > downloads in parallel speeds things up? >=20 > The analysis in > > suggests that at the time you first submitted this patch, substitution > speed (which is different from raw download speed) was often > CPU-bound. This is no longer the case, meaning that downloads should > now be network-bound or almost. >=20 > Thanks, > Ludo=E2=80=99. I would still say yes, because the output from berlin is often much less than my throughput. With multiple downloads in parallel it at least feels quicker, probably because I can download at full speed. In any case, I see often a build start while downloads are in progress, so I think it's still a win if you can get a few derivations built while waiting for a big download to finish at the same time :) At some point we might want to prioritize builds/downloads that help unlock as much builds as possible early, so we don't have builds waiting for downloads. From debbugs-submit-bounces@debbugs.gnu.org Fri Nov 26 05:16:27 2021 Received: (at 39728) by debbugs.gnu.org; 26 Nov 2021 10:16:27 +0000 Received: from localhost ([127.0.0.1]:58744 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mqYHK-0002Q0-Oc for submit@debbugs.gnu.org; Fri, 26 Nov 2021 05:16:26 -0500 Received: from eggs.gnu.org ([209.51.188.92]:37922) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mqYHJ-0002Pn-DO for 39728@debbugs.gnu.org; Fri, 26 Nov 2021 05:16:25 -0500 Received: from [2001:470:142:3::e] (port=53614 helo=fencepost.gnu.org) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mqYHD-0002wE-DA; Fri, 26 Nov 2021 05:16:19 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:In-Reply-To:Date:References:Subject:To: From; bh=Bqga2G5FYambx/++7B/qMahni2nVvsRC+QFQxJdLNz0=; b=sb0Ro9FfSxCvX7reu6Vp bRSpU9u/qHeBk90voKC0DEZ6zMsqKyXtRT8Rh3aPyVpGV9uN66cCmpd6rBnZJsTULB/LjanEEKqye izHEKvg1EqxyBGDCHCGVbZRKfOzc+djRAF54qWcUsl0HqqKaavK0d6+K5eRFxlAbOMicWbSoS8AlN 9FMp2gPNgaV7dZCzsl9O+uylce015wAprr0AtMs0v1qb/3LaN5jlNH91/lM0G9Pvq0gwariWr9+JG pnwHnFUxEW+uThicTXVgAcaKub4cQm7Qp26wmh6XABNRhqYkyIj2mLrCCdZknuJ78E+vtOUjMNeHq TAsXp3NS5ObI8w==; Received: from [2001:660:6102:320:e120:2c8f:8909:cdfe] (port=49512 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mqYHD-0002j6-6v; Fri, 26 Nov 2021 05:16:19 -0500 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Julien Lepiller Subject: Re: [bug#39728] [PATCH v2] Allow parallel downloads and builds References: <20200221235307.535fb453@tachikoma.lepiller.eu> <87h7zfel26.fsf@gnu.org> <62A3FE42-52DD-4E51-9AD1-242B86B6A391@lepiller.eu> <87zhd58wjz.fsf@ambrevar.xyz> <20211121234941.32724b68@tachikoma.lepiller.eu> <87ilwge6re.fsf@gnu.org> <20211125140145.1c00decc@tachikoma.lepiller.eu> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 6 Frimaire an 230 de la =?utf-8?Q?R=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Fri, 26 Nov 2021 11:16:16 +0100 In-Reply-To: <20211125140145.1c00decc@tachikoma.lepiller.eu> (Julien Lepiller's message of "Thu, 25 Nov 2021 14:01:45 +0100") Message-ID: <87r1b39q8f.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 39728 Cc: 39728@debbugs.gnu.org, Pierre Neidhardt , zimoun X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hi! Julien Lepiller skribis: > I would still say yes, because the output from berlin is often much > less than my throughput. With multiple downloads in parallel it at > least feels quicker, probably because I can download at full speed. It would be nice to measure that because like I wrote, I think we=E2=80=99re pretty much network-bound these days, at least with zstd and uncompressed downloads. > In any case, I see often a build start while downloads are in progress, > so I think it's still a win if you can get a few derivations built > while waiting for a big download to finish at the same time :) True! Overlapping downloads and builds sounds like a good idea. > At some point we might want to prioritize builds/downloads that help > unlock as much builds as possible early, so we don't have builds > waiting for downloads. Right now the daemon starts with substitutes and builds afterwards. BTW, we=E2=80=99re assuming downloads =3D substitutes in this whole discuss= ion, but we could/should take fixed-output derivations into account too. I=E2=80=99ll take a closer look later on=E2=80=A6 Thanks, Ludo=E2=80=99.