From unknown Mon Jun 16 23:51:53 2025 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Mailer: MIME-tools 5.509 (Entity 5.509) Content-Type: text/plain; charset=utf-8 From: bug#38649 <38649@debbugs.gnu.org> To: bug#38649 <38649@debbugs.gnu.org> Subject: Status: [PATCH] Parallelize `guix package` Reply-To: bug#38649 <38649@debbugs.gnu.org> Date: Tue, 17 Jun 2025 06:51:53 +0000 retitle 38649 [PATCH] Parallelize `guix package` reassign 38649 guix-patches submitter 38649 Leo Prikler severity 38649 normal tag 38649 patch thanks From debbugs-submit-bounces@debbugs.gnu.org Tue Dec 17 09:18:46 2019 Received: (at submit) by debbugs.gnu.org; 17 Dec 2019 14:18:46 +0000 Received: from localhost ([127.0.0.1]:40875 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ihDgU-0003nA-GQ for submit@debbugs.gnu.org; Tue, 17 Dec 2019 09:18:46 -0500 Received: from lists.gnu.org ([209.51.188.17]:46184) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ihDgT-0003n3-56 for submit@debbugs.gnu.org; Tue, 17 Dec 2019 09:18:45 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:53851) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ihDgR-0008L2-4W for guix-patches@gnu.org; Tue, 17 Dec 2019 09:18:44 -0500 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.5 required=5.0 tests=BAYES_50,RCVD_IN_DNSWL_MED, URIBL_BLOCKED autolearn=disabled version=3.3.2 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ihDgP-00080h-7B for guix-patches@gnu.org; Tue, 17 Dec 2019 09:18:42 -0500 Received: from mailrelay.tugraz.at ([129.27.2.202]:37029) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1ihDgO-0007hW-HQ for guix-patches@gnu.org; Tue, 17 Dec 2019 09:18:41 -0500 Received: from nijino.local (213-240-64-42.hdsl.highway.telekom.at [213.240.64.42]) by mailrelay.tugraz.at (Postfix) with ESMTPSA id 47cgGz5fPWz3wYB for ; Tue, 17 Dec 2019 15:18:35 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tugraz.at; s=mailrelay; t=1576592315; bh=/rn7p/S+N8X15E0RfwSAPCkzZjwUMtV5cWFUk5UwMU0=; h=Subject:From:To:Date; b=lBlXXp/+k+TBodNm/xg9QERqepvyERRebB5E6NAN5lnOBSGsYmEXud1KJa6N0Gf1e O/D8vhpsU0JceSdAD0wH69izKc5CaGZxc2A9PJ5zxHCA96Ou9e3O0itsweieEdRzzT AwGznt3JXq6pAYoDvo/EVQhj4V625QG5O7AoOri4= Message-ID: Subject: [PATCH] Parallelize `guix package` From: Leo Prikler To: guix-patches@gnu.org Date: Tue, 17 Dec 2019 15:18:44 +0100 Content-Type: multipart/mixed; boundary="=-MKoVmQLaz2hzdn/NcZsK" User-Agent: Evolution 3.32.4 MIME-Version: 1.0 X-TUG-Backscatter-control: bt4lQm5Tva3SBgCuw0EnZw X-Scanned-By: MIMEDefang 2.74 on 129.27.10.117 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 129.27.2.202 X-Spam-Score: -1.3 (-) X-Debbugs-Envelope-To: submit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.3 (--) --=-MKoVmQLaz2hzdn/NcZsK Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit Hi Guix! Yesterday I had an interesting conversation on IRC about the behaviour of multiple `guix package` processes running in parallel. Specifically, when two transactions target the same profile (usually /var/guix/profiles/per-user/$USER/guix-profile) at the same time, one of them will fail to claim the lock and abort. 0001 makes it so that the process waits for the lock. 0002 makes it so that packages specified via -i can be built in parallel. Regards, Leo --=-MKoVmQLaz2hzdn/NcZsK Content-Disposition: attachment; filename="0001-guix-Wait-for-file-lock.patch" Content-Type: text/x-patch; name="0001-guix-Wait-for-file-lock.patch"; charset="UTF-8" Content-Transfer-Encoding: base64 RnJvbSBmOTkwZmVlMjVlNGQ2MmZmZmY0ZDZlZTg5YmUwNWIyNTYzODY1MzI0IE1vbiBTZXAgMTcg MDA6MDA6MDAgMjAwMQpGcm9tOiBMZW8gUHJpa2xlciA8bGVvLnByaWtsZXJAc3R1ZGVudC50dWdy YXouYXQ+CkRhdGU6IFR1ZSwgMTcgRGVjIDIwMTkgMTI6NTM6MjEgKzAxMDAKU3ViamVjdDogW1BB VENIIDEvMl0gZ3VpeDogV2FpdCBmb3IgZmlsZSBsb2NrLgoKKiBndWl4L3VpLnNjbTogQWRqdXN0 IGltcG9ydHMuCih3aXRoLXByb2ZpbGUtbG9jayk6IFVzZSBgd2l0aC1maWxlLWxvY2snIGluc3Rl YWQgb2YgYHdpdGgtZmlsZS1sb2NrL25vLXdhaXQnLgoqIGd1aXgvc2NyaXB0cy9wYWNrYWdlLnNj bTogRHJvcCB1bnVzZWQgaW1wb3J0LgoocHJvY2Vzcy1hY3Rpb25zKTogUHJpbnQgbWVzc2FnZSB3 aGlsZSB3YWl0aW5nIG9uIGxvY2suCi0tLQogZ3VpeC9zY3JpcHRzL3BhY2thZ2Uuc2NtIHwgMyAr LS0KIGd1aXgvdWkuc2NtICAgICAgICAgICAgICB8IDQgKystLQogMiBmaWxlcyBjaGFuZ2VkLCAz IGluc2VydGlvbnMoKyksIDQgZGVsZXRpb25zKC0pCgpkaWZmIC0tZ2l0IGEvZ3VpeC9zY3JpcHRz L3BhY2thZ2Uuc2NtIGIvZ3VpeC9zY3JpcHRzL3BhY2thZ2Uuc2NtCmluZGV4IDkyYzZlMzQxOTQu LjIwMmE2ZDY0NzAgMTAwNjQ0Ci0tLSBhL2d1aXgvc2NyaXB0cy9wYWNrYWdlLnNjbQorKysgYi9n dWl4L3NjcmlwdHMvcGFja2FnZS5zY20KQEAgLTQyLDggKzQyLDYgQEAKICAgIzphdXRvbG9hZCAg IChndWl4IHN0b3JlIHJvb3RzKSAoZ2Mtcm9vdHMpCiAgICM6dXNlLW1vZHVsZSAoKGd1aXggYnVp bGQgdXRpbHMpCiAgICAgICAgICAgICAgICAgIzpzZWxlY3QgKGRpcmVjdG9yeS1leGlzdHM/IG1r ZGlyLXApKQotICAjOnVzZS1tb2R1bGUgKChndWl4IGJ1aWxkIHN5c2NhbGxzKQotICAgICAgICAg ICAgICAgICM6c2VsZWN0ICh3aXRoLWZpbGUtbG9jay9uby13YWl0KSkKICAgIzp1c2UtbW9kdWxl IChpY2UtOSBmb3JtYXQpCiAgICM6dXNlLW1vZHVsZSAoaWNlLTkgbWF0Y2gpCiAgICM6dXNlLW1v ZHVsZSAoaWNlLTkgcmVnZXgpCkBAIC04NjYsNiArODY0LDcgQEAgcHJvY2Vzc2VkLCAjZiBvdGhl cndpc2UuIgogCiAgIDs7IEZpcnN0LCBhY3F1aXJlIGEgbG9jayBvbiB0aGUgcHJvZmlsZSwgdG8g ZW5zdXJlIG9ubHkgb25lIGd1aXggcHJvY2VzcwogICA7OyBpcyBtb2RpZnlpbmcgaXQgYXQgYSB0 aW1lLgorICAoZm9ybWF0ICN0ICJXYWl0aW5nIGZvciBsb2NrIG9uIH5hLi4ufiUiIHByb2ZpbGUp CiAgICh3aXRoLXByb2ZpbGUtbG9jayBwcm9maWxlCiAgICAgOzsgVGhlbiwgcHJvY2VzcyByb2xs LWJhY2tzLCBnZW5lcmF0aW9uIHJlbW92YWxzLCBldGMuCiAgICAgKGZvci1lYWNoIChtYXRjaC1s YW1iZGEKZGlmZiAtLWdpdCBhL2d1aXgvdWkuc2NtIGIvZ3VpeC91aS5zY20KaW5kZXggNTQwNjcx ZjNkZC4uMDYwYzRlNzNmMSAxMDA2NDQKLS0tIGEvZ3VpeC91aS5zY20KKysrIGIvZ3VpeC91aS5z Y20KQEAgLTQ4LDcgKzQ4LDcgQEAKICAgICAgICAgICAgICAgICAjOnNlbGVjdCAobGljZW5zZT8g bGljZW5zZS1uYW1lIGxpY2Vuc2UtdXJpKSkKICAgIzp1c2UtbW9kdWxlICgoZ3VpeCBidWlsZCBz eXNjYWxscykKICAgICAgICAgICAgICAgICAjOnNlbGVjdCAoZnJlZS1kaXNrLXNwYWNlIHRlcm1p bmFsLWNvbHVtbnMgdGVybWluYWwtcm93cwotICAgICAgICAgICAgICAgICAgICAgICAgICB3aXRo LWZpbGUtbG9jay9uby13YWl0KSkKKyAgICAgICAgICAgICAgICAgICAgICAgICAgd2l0aC1maWxl LWxvY2spKQogICAjOnVzZS1tb2R1bGUgKChndWl4IGJ1aWxkIHV0aWxzKQogICAgICAgICAgICAg ICAgIDs7IFhYWDogQWxsIHdlIG5lZWQgYXJlIHRoZSBiaW5kaW5ncyByZWxhdGVkIHRvCiAgICAg ICAgICAgICAgICAgOzsgJyZpbnZva2UtZXJyb3InLiAgSG93ZXZlciwgdG8gd29yayBhcm91bmQg dGhlIGJ1ZyBkZXNjcmliZWQKQEAgLTE2ODAsNyArMTY4MCw3IEBAIERVUkFUSU9OLVJFTEFUSU9O IHdpdGggdGhlIGN1cnJlbnQgdGltZS4iCiAoZGVmaW5lLXN5bnRheC1ydWxlICh3aXRoLXByb2Zp bGUtbG9jayBwcm9maWxlIGV4cCAuLi4pCiAgICJHcmFiIFBST0ZJTEUncyBsb2NrIGFuZCBldmFs dWF0ZSBFWFAuLi4gIENhbGwgJ2xlYXZlJyBpZiB0aGUgbG9jayBpcwogYWxyZWFkeSB0YWtlbi4i Ci0gICh3aXRoLWZpbGUtbG9jay9uby13YWl0IChwcm9maWxlLWxvY2stZmlsZSBwcm9maWxlKQor ICAod2l0aC1maWxlLWxvY2sgKHByb2ZpbGUtbG9jay1maWxlIHByb2ZpbGUpCiAgICAgKGN1dCBw cm9maWxlLWxvY2staGFuZGxlciBwcm9maWxlIDwuLi4+KQogICAgIGV4cCAuLi4pKQogCi0tIAoy LjI0LjEKCg== --=-MKoVmQLaz2hzdn/NcZsK Content-Disposition: attachment; filename="0002-guix-Build-to-be-installed-packages-in-parallel.patch" Content-Type: text/x-patch; name="0002-guix-Build-to-be-installed-packages-in-parallel.patch"; charset="UTF-8" Content-Transfer-Encoding: base64 RnJvbSAzMzY2OTJkZjE1ZTc3ZjlkOTA2MTlkMGZlNjBlODY0YzRkMmZiMzdhIE1vbiBTZXAgMTcg MDA6MDA6MDAgMjAwMQpGcm9tOiBMZW8gUHJpa2xlciA8bGVvLnByaWtsZXJAc3R1ZGVudC50dWdy YXouYXQ+CkRhdGU6IFR1ZSwgMTcgRGVjIDIwMTkgMTQ6MDQ6MTIgKzAxMDAKU3ViamVjdDogW1BB VENIIDIvMl0gZ3VpeDogQnVpbGQgdG8gYmUgaW5zdGFsbGVkIHBhY2thZ2VzIGluIHBhcmFsbGVs LgoKKiBndWl4L3NjcmlwdHMvcGFja2FnZS5zY20gKG9wdGlvbnMtPmJ1aWxkYWJsZSk6IE5ldyBw cm9jZWR1cmUuCihwcm9jZXNzLWFjdGlvbnMpOiBCdWlsZCBwYWNrYWdlcyBiZWZvcmUgYWNxdWly aW5nIHByb2ZpbGUgbG9jay4KLS0tCiBndWl4L3NjcmlwdHMvcGFja2FnZS5zY20gfCAyMiArKysr KysrKysrKysrKysrKysrKystCiAxIGZpbGUgY2hhbmdlZCwgMjEgaW5zZXJ0aW9ucygrKSwgMSBk ZWxldGlvbigtKQoKZGlmZiAtLWdpdCBhL2d1aXgvc2NyaXB0cy9wYWNrYWdlLnNjbSBiL2d1aXgv c2NyaXB0cy9wYWNrYWdlLnNjbQppbmRleCAyMDJhNmQ2NDcwLi4xMjc4ZDFhNjVmIDEwMDY0NAot LS0gYS9ndWl4L3NjcmlwdHMvcGFja2FnZS5zY20KKysrIGIvZ3VpeC9zY3JpcHRzL3BhY2thZ2Uu c2NtCkBAIC01ODcsNiArNTg3LDE5IEBAIHRoZSByZXN1bHRpbmcgbWFuaWZlc3QgZW50cnkuIgog ICAocGFja2FnZS0+bWFuaWZlc3QtZW50cnkgcGFja2FnZSBvdXRwdXQKICAgICAgICAgICAgICAg ICAgICAgICAgICAgICM6cHJvcGVydGllcyAocHJvdmVuYW5jZS1wcm9wZXJ0aWVzIHBhY2thZ2Up KSkKIAorKGRlZmluZSAob3B0aW9ucy0+YnVpbGRhYmxlIG9wdHMpCisgIChmaWx0ZXItbWFwICht YXRjaC1sYW1iZGEKKyAgICAgICAgICAgICAgICAoKCdpbnN0YWxsIC4gKD8gcGFja2FnZT8gcCkp CisgICAgICAgICAgICAgICAgIHApCisgICAgICAgICAgICAgICAgKCgnaW5zdGFsbCAuICg/IHN0 cmluZz8gc3BlYykpCisgICAgICAgICAgICAgICAgIChpZiAoc3RvcmUtcGF0aD8gc3BlYykKKyAg ICAgICAgICAgICAgICAgICAgICNmIDs7IGFzc3VtZSBhbHJlYWR5IGludGVybmVkCisgICAgICAg ICAgICAgICAgICAgICAoc3BlY2lmaWNhdGlvbi0+cGFja2FnZSBzcGVjKSkpCisgICAgICAgICAg ICAgICAgKCgnaW5zdGFsbCAuIG9iaikKKyAgICAgICAgICAgICAgICAgKGxlYXZlIChHXyAiY2Fu bm90IGJ1aWxkIG5vbi1wYWNrYWdlIG9iamVjdDogfnN+JSIpCisgICAgICAgICAgICAgICAgICAg ICAgICBvYmopKQorICAgICAgICAgICAgICAgIChfICNmKSkKKyAgICAgICAgICAgICAgb3B0cykp CiAKIChkZWZpbmUgKG9wdGlvbnMtPmluc3RhbGxhYmxlIG9wdHMgbWFuaWZlc3QgdHJhbnNhY3Rp b24pCiAgICJHaXZlbiBNQU5JRkVTVCwgdGhlIGN1cnJlbnQgbWFuaWZlc3QsIGFuZCBPUFRTLCB0 aGUgcmVzdWx0IG9mICdhcmdzLWZvbGQnLApAQCAtODYxLDggKzg3NCwxNSBAQCBwcm9jZXNzZWQs ICNmIG90aGVyd2lzZS4iCiAgICAgICAgICAgICAgICAgICAgICAocGFja2FnZS12ZXJzaW9uIGl0 ZW0pCiAgICAgICAgICAgICAgICAgICAgICAobWFuaWZlc3QtZW50cnktdmVyc2lvbiBlbnRyeSkp KSkpKQogCisgIDs7IEZpcnN0LCBwcm9jZXNzIGluc3RhbGxhdGlvbnMsIGFzIHRoZXNlIGNhbiBi ZSBoYW5kbGVkIGluIHBhcmFsbGVsLgorICAodW5sZXNzIGRyeS1ydW4/CisgICAgKGxldCogKChk cnYgKG1hcCAoY29tcG9zZSAobGFtYmRhIChkcnYpIChkcnYgc3RvcmUpKSBwYWNrYWdlLT5kZXJp dmF0aW9uKQorICAgICAgICAgICAgICAgICAgICAgKG9wdGlvbnMtPmJ1aWxkYWJsZSBvcHRzKSkp KQorICAgICAgKHNob3ctd2hhdC10by1idWlsZCBzdG9yZSBkcnYKKyAgICAgICAgICAgICAgICAg ICAgICAgICAgIzp1c2Utc3Vic3RpdHV0ZXM/IHN1YnN0aXR1dGVzPykKKyAgICAgIChidWlsZC1k ZXJpdmF0aW9ucyBzdG9yZSBkcnYpKSkKIAotICA7OyBGaXJzdCwgYWNxdWlyZSBhIGxvY2sgb24g dGhlIHByb2ZpbGUsIHRvIGVuc3VyZSBvbmx5IG9uZSBndWl4IHByb2Nlc3MKKyAgOzsgTm93LCBh Y3F1aXJlIGEgbG9jayBvbiB0aGUgcHJvZmlsZSwgdG8gZW5zdXJlIG9ubHkgb25lIGd1aXggcHJv Y2VzcwogICA7OyBpcyBtb2RpZnlpbmcgaXQgYXQgYSB0aW1lLgogICAoZm9ybWF0ICN0ICJXYWl0 aW5nIGZvciBsb2NrIG9uIH5hLi4ufiUiIHByb2ZpbGUpCiAgICh3aXRoLXByb2ZpbGUtbG9jayBw cm9maWxlCi0tIAoyLjI0LjEKCg== --=-MKoVmQLaz2hzdn/NcZsK-- From debbugs-submit-bounces@debbugs.gnu.org Tue Dec 17 09:20:53 2019 Received: (at 38649) by debbugs.gnu.org; 17 Dec 2019 14:20:53 +0000 Received: from localhost ([127.0.0.1]:40881 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ihDiX-0003qx-23 for submit@debbugs.gnu.org; Tue, 17 Dec 2019 09:20:53 -0500 Received: from mout02.posteo.de ([185.67.36.66]:46873) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ihDiU-0003qj-Oi for 38649@debbugs.gnu.org; Tue, 17 Dec 2019 09:20:51 -0500 Received: from submission (posteo.de [89.146.220.130]) by mout02.posteo.de (Postfix) with ESMTPS id 3D278240103 for <38649@debbugs.gnu.org>; Tue, 17 Dec 2019 15:20:44 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017; t=1576592444; bh=IuHoSMaZSVLzJSZw82mlcKAH3MLkCD9bJ/Hh2Djk/Yg=; h=Date:From:To:Cc:Subject:From; b=lBxKfIzTvqixWdRDTHlW8z8vMTCtN440eyAGjrwWwq/ggHm70HyJsQGIv1pxFADh6 4+of96aZ9TJsT5oITV5IB/M9J8j3kSEGRQ9GCe6u7Ekebd8k7OWWA9mPmStfghcl7P YL+cSQn4JBjWL+CT6c9wdr/fugdmQN9okad4Zag9NvTos7qF0CObKOLtYhyngIrITP lAPYkCrskI3z8G3/lZwVqks2IVttfrGOoPKtDPGBdOU7bwQ0AvIX4Sze+oQgMtT9Po 6HoWw99qLQkjALcVWn+SKKL3KPIwAABpXk+jubT0IxZdt30wxkunjZxYNmFTr+95HN ySGsh5g+87c0A== Received: from customer (localhost [127.0.0.1]) by submission (posteo.de) with ESMTPSA id 47cgKQ6LQGz9rxB; Tue, 17 Dec 2019 15:20:42 +0100 (CET) Date: Tue, 17 Dec 2019 14:20:40 +0000 (UTC) From: Brett Gilio To: Leo Prikler Message-ID: <6dd517ad-639d-4932-be9c-2acbd889d1ed@localhost> In-Reply-To: References: Subject: Re: [bug#38649] [PATCH] Parallelize `guix package` MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Correlation-ID: <6dd517ad-639d-4932-be9c-2acbd889d1ed@localhost> X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 38649 Cc: 38649@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Dec 17, 2019 8:19:14 AM Leo Prikler : > Hi Guix! > > Yesterday I had an interesting conversation on IRC about the behaviour > of multiple `guix package` processes running in parallel. > Specifically, when two transactions target the same profile (usually > /var/guix/profiles/per-user/$USER/guix-profile) at the same time, one > of them will fail to claim the lock and abort. 0001 makes it so that > the process waits for the lock. 0002 makes it so that packages > specified via -i can be built in parallel. > > Regards, > Leo > Can we extend this to include things like environment --ad-hoc? From debbugs-submit-bounces@debbugs.gnu.org Tue Dec 17 09:32:25 2019 Received: (at 38649) by debbugs.gnu.org; 17 Dec 2019 14:32:25 +0000 Received: from localhost ([127.0.0.1]:40890 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ihDth-0004BC-CO for submit@debbugs.gnu.org; Tue, 17 Dec 2019 09:32:25 -0500 Received: from eggs.gnu.org ([209.51.188.92]:48164) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ihDtf-0004Ax-22 for 38649@debbugs.gnu.org; Tue, 17 Dec 2019 09:32:23 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]:50852) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ihDtX-00072k-9j; Tue, 17 Dec 2019 09:32:15 -0500 Received: from [2001:660:6102:320:e120:2c8f:8909:cdfe] (port=50604 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1ihDtV-0001tv-Eg; Tue, 17 Dec 2019 09:32:14 -0500 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Leo Prikler Subject: Re: [bug#38649] [PATCH] Parallelize `guix package` References: Date: Tue, 17 Dec 2019 15:32:10 +0100 In-Reply-To: (Leo Prikler's message of "Tue, 17 Dec 2019 15:18:44 +0100") Message-ID: <87tv5zrpjp.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.3 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 38649 Cc: julien lepiller , 38649@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hi Leo, (Cc: Julien, who worked on this part.) Leo Prikler skribis: > Yesterday I had an interesting conversation on IRC about the behaviour > of multiple `guix package` processes running in parallel.=20 > Specifically, when two transactions target the same profile (usually > /var/guix/profiles/per-user/$USER/guix-profile) at the same time, one > of them will fail to claim the lock and abort. 0001 makes it so that > the process waits for the lock. 0002 makes it so that packages > specified via -i can be built in parallel. I actually like the current behavior, FWIW. Julien came up with this locking mostly so that people do not inadvertently attempt to perform several operations concurrently. The key word here is =E2=80=9Cinadvertently=E2=80=9D: IMO, there=E2=80=99s = no reason to run multiple =E2=80=98guix package=E2=80=99 on the same profile concurrently. = With a wait-for-lock policy, the result would be non-deterministic: you cannot tell which one of the two processes will complete first. WDYT? Thanks, Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Tue Dec 17 09:34:29 2019 Received: (at 38649) by debbugs.gnu.org; 17 Dec 2019 14:34:29 +0000 Received: from localhost ([127.0.0.1]:40894 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ihDvg-0004EA-UA for submit@debbugs.gnu.org; Tue, 17 Dec 2019 09:34:29 -0500 Received: from mailrelay.tugraz.at ([129.27.2.202]:34078) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ihDve-0004E1-Un for 38649@debbugs.gnu.org; Tue, 17 Dec 2019 09:34:27 -0500 Received: from nijino.local (213-240-64-42.hdsl.highway.telekom.at [213.240.64.42]) by mailrelay.tugraz.at (Postfix) with ESMTPSA id 47cgdC108Qz3wFM; Tue, 17 Dec 2019 15:34:23 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tugraz.at; s=mailrelay; t=1576593263; bh=8teBEdH71fh0fR/4YdCYa0zATvGwQmbIxgwnTJ8PMDo=; h=Subject:From:To:Cc:Date:In-Reply-To:References; b=BJF53ze3du39UZHOHR7v9nqoAi+SoPCMofw2e73AuQy8ylqwwst/4OQ0TbT6m/GSC mgVvMSJA8PbspMH6ljb+RIb63ym8r1jSTZGoMFDkGcEDdww/kbk4ouvqhgdDU3Dr7+ h9o46rVNtxPxQT6qazy9en/d2YqXBP+9DP0UTVDo= Message-ID: Subject: Re: [bug#38649] [PATCH] Parallelize `guix package` From: Leo Prikler To: Brett Gilio Date: Tue, 17 Dec 2019 15:34:31 +0100 In-Reply-To: <6dd517ad-639d-4932-be9c-2acbd889d1ed@localhost> References: <6dd517ad-639d-4932-be9c-2acbd889d1ed@localhost> Content-Type: text/plain; charset="UTF-8" User-Agent: Evolution 3.32.4 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-TUG-Backscatter-control: bt4lQm5Tva3SBgCuw0EnZw X-Spam-Scanner: SpamAssassin 3.003001 X-Spam-Score-relay: -1.9 X-Scanned-By: MIMEDefang 2.74 on 129.27.10.116 X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 38649 Cc: 38649@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Am Dienstag, den 17.12.2019, 14:20 +0000 schrieb Brett Gilio: > > Dec 17, 2019 8:19:14 AM Leo Prikler : > > > Hi Guix! > > > > Yesterday I had an interesting conversation on IRC about the > > behaviour > > of multiple `guix package` processes running in parallel. > > Specifically, when two transactions target the same profile > > (usually > > /var/guix/profiles/per-user/$USER/guix-profile) at the same time, > > one > > of them will fail to claim the lock and abort. 0001 makes it so > > that > > the process waits for the lock. 0002 makes it so that packages > > specified via -i can be built in parallel. > > > > Regards, > > Leo > > > > Can we extend this to include things like environment --ad-hoc? `guix environment` does not claim any locks, so it does not suffer from the problem that this patch tries to address. Perhaps my wording was bad: By "can be built in parallel", I meant that if one starts two processes, e.g. `guix install emacs` and `guix install ffmpeg`, emacs and ffmpeg are built in parallel. This does not mean, that dependencies of emacs are built in parallel – for that you'd have to dig closer to the core. Regards, Leo From debbugs-submit-bounces@debbugs.gnu.org Tue Dec 17 09:38:14 2019 Received: (at 38649) by debbugs.gnu.org; 17 Dec 2019 14:38:14 +0000 Received: from localhost ([127.0.0.1]:40908 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ihDzK-0004KK-LN for submit@debbugs.gnu.org; Tue, 17 Dec 2019 09:38:14 -0500 Received: from mout02.posteo.de ([185.67.36.66]:49257) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ihDzI-0004K7-Qj for 38649@debbugs.gnu.org; Tue, 17 Dec 2019 09:38:13 -0500 Received: from submission (posteo.de [89.146.220.130]) by mout02.posteo.de (Postfix) with ESMTPS id 0072A2400FD for <38649@debbugs.gnu.org>; Tue, 17 Dec 2019 15:38:05 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017; t=1576593486; bh=PGiiUrE7JqZuhTYz/kFDfgz3JQXMsViYtK7FqsSfBDQ=; h=Date:From:To:Cc:Subject:From; b=E1dWhesLw7J4Nvq3XpkmpnrUD8iiaD8T3Gk5yEahPHV9FnRwXiRJiCmPwiZY478ou R4s8qXbHM9g94s5skwOl9zock02UULA0uV+onHokoOvkNiuOrwD9j2qvs+PRWjpF4v pjJpUVxIQt/wJSeVxPfUlESGUqqgv9zBPDWVzCODSfbzu1u2bMqi8415CsL6imEIVm CQYNQDAuUYjy0VuN42yvnXgwd5m3bAP5LJUovugud6bQxyePQ5ag0XasM5u1Z9zVuJ eYLOdhgUL8QahhNzGTxdBZKI3rwuvT1co2/68qG7x1zH9l4xtTXcP1Q4f6m/bg7CLD DSqQoOQweKycA== Received: from customer (localhost [127.0.0.1]) by submission (posteo.de) with ESMTPSA id 47cgjS11F2z9rxS; Tue, 17 Dec 2019 15:38:03 +0100 (CET) Date: Tue, 17 Dec 2019 14:38:00 +0000 (UTC) From: Brett Gilio To: Leo Prikler Message-ID: <70d9299d-0d1f-496a-a49c-f6648a721639@localhost> In-Reply-To: References: <6dd517ad-639d-4932-be9c-2acbd889d1ed@localhost> Subject: Re: [bug#38649] [PATCH] Parallelize `guix package` MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Correlation-ID: <70d9299d-0d1f-496a-a49c-f6648a721639@localhost> X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 38649 Cc: 38649@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Dec 17, 2019 8:34:25 AM Leo Prikler : > Am Dienstag, den 17.12.2019, 14:20 +0000 schrieb Brett Gilio: > > > > > Dec 17, 2019 8:19:14 AM Leo Prikler : > > > > > > > Hi Guix! > > > > > > Yesterday I had an interesting conversation on IRC about the > > > behaviour > > > of multiple `guix package` processes running in parallel. > > > Specifically, when two transactions target the same profile > > > (usually > > > /var/guix/profiles/per-user/$USER/guix-profile) at the same time, > > > one > > > of them will fail to claim the lock and abort. 0001 makes it so > > > that > > > the process waits for the lock. 0002 makes it so that packages > > > specified via -i can be built in parallel. > > > > > > Regards, > > > Leo > > > > > > > > > > Can we extend this to include things like environment --ad-hoc? > > > `guix environment` does not claim any locks, so it does not suffer from > the problem that this patch tries to address. Perhaps my wording was > bad: By "can be built in parallel", I meant that if one starts two > processes, e.g. `guix install emacs` and `guix install ffmpeg`, emacs > and ffmpeg are built in parallel. This does not mean, that > dependencies of emacs are built in parallel ? for that you'd have to > dig closer to the core. > > Regards, > Leo > Ah right. My mistake. I just woke up, so I think I need more coffee. :) Brett Gilio From debbugs-submit-bounces@debbugs.gnu.org Tue Dec 17 10:19:31 2019 Received: (at 38649) by debbugs.gnu.org; 17 Dec 2019 15:19:31 +0000 Received: from localhost ([127.0.0.1]:42195 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ihEdG-0005iK-Q0 for submit@debbugs.gnu.org; Tue, 17 Dec 2019 10:19:31 -0500 Received: from mailrelay.tugraz.at ([129.27.2.202]:2415) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ihEdE-0005iB-Bd for 38649@debbugs.gnu.org; Tue, 17 Dec 2019 10:19:30 -0500 Received: from nijino.local (213-240-64-42.hdsl.highway.telekom.at [213.240.64.42]) by mailrelay.tugraz.at (Postfix) with ESMTPSA id 47chd93wsfz1DHSW; Tue, 17 Dec 2019 16:19:25 +0100 (CET) DKIM-Filter: OpenDKIM Filter v2.11.0 mailrelay.tugraz.at 47chd93wsfz1DHSW DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tugraz.at; s=mailrelay; t=1576595965; bh=Atv4Gs33aamzdgxYzt9+pnminQ23GJSon2mNimtpWMY=; h=Subject:From:To:Cc:Date:In-Reply-To:References:From; b=eEmUEmnKEm9qCp1X7jEo85KvkzgiViGMB1JAzoOaEuz4Plpt1YEisPJ7tkM3ObdRQ 2RA9J4tYI5IWwVugOVP2ObrqR3U5iK+T0RskIwrU3gruf91gSADWzgW9L49q5pVPDi tZbbW9S1rn198ZM4T8Uxa5WmFdJBPM2BsrCejeMU= Message-ID: <3d0ca2a8b59dd99e15b55033bc89b2e21aa49814.camel@student.tugraz.at> Subject: Re: [bug#38649] [PATCH] Parallelize `guix package` From: Leo Prikler To: Ludovic =?ISO-8859-1?Q?Court=E8s?= Date: Tue, 17 Dec 2019 16:19:34 +0100 In-Reply-To: <87tv5zrpjp.fsf@gnu.org> References: <87tv5zrpjp.fsf@gnu.org> Content-Type: text/plain; charset="UTF-8" User-Agent: Evolution 3.32.4 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-TUG-Backscatter-control: bt4lQm5Tva3SBgCuw0EnZw X-Spam-Scanner: SpamAssassin 3.003001 X-Spam-Score-relay: -1.9 X-Scanned-By: MIMEDefang 2.74 on 129.27.10.116 X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 38649 Cc: julien lepiller , 38649@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hi Ludo’, Am Dienstag, den 17.12.2019, 15:32 +0100 schrieb Ludovic Courtès: > Hi Leo, > > (Cc: Julien, who worked on this part.) > > Leo Prikler skribis: > > > Yesterday I had an interesting conversation on IRC about the > > behaviour > > of multiple `guix package` processes running in parallel. > > Specifically, when two transactions target the same profile > > (usually > > /var/guix/profiles/per-user/$USER/guix-profile) at the same time, > > one > > of them will fail to claim the lock and abort. 0001 makes it so > > that > > the process waits for the lock. 0002 makes it so that packages > > specified via -i can be built in parallel. > > I actually like the current behavior, FWIW. Julien came up with this > locking mostly so that people do not inadvertently attempt to perform > several operations concurrently. Fair enough and that is an improvement over non-locking behaviour, where one could spawn multiple profile generations from one, neither of which is complete. Perhaps my attempt at doing this in a somewhat controlled manner is equally harmful, but I will still try my best arguing for it, as I believe it can make a positive impact. > The key word here is “inadvertently”: IMO, there’s no reason to run > multiple ‘guix package’ on the same profile concurrently. With a > wait-for-lock policy, the result would be non-deterministic: you > cannot > tell which one of the two processes will complete first. > > WDYT? I think the current policy is wait-for-lock deferred to the user. The user has to let the first task complete before they can start the second. In this setup, the user can simply launch the setup and trust, that it will complete later while taking into account the changes the first one has made. Let's talk about three classes of operations – installations, removals and upgrades – and their interactions. I will not take into account roll-back, switch-generation and delete-generation, as it is nonsensical to perform these in parallel to any other action. Perhaps we could check for their presence first and acquire the lock with no- wait semantics in that case. - any operation on different packages: Either succeeds first and the other builds on the profile it generates. As there is no collision in the packages themselves, there will be no harm. - install same package twice: Either succeeds first, the other will be a no-op. - install vs. remove same package: Non-deterministic, but why would you do that? - install vs. upgrade same package: Upgrade will be a no-op in either case. - remove vs. upgrade same package: Upgrade may inadvertently upgrade the old package if it happens to come first, but in the final package it will be removed either way. Of course, any operation can also fail midway due to some step not succeeding. In that case it would be as if one had issued the other command right after that, which may perhaps not be what one wanted to do (assuming I install package A, and some guide suggests to also build related, but not dependency-connected package B, so I end up installing B without A). However, such cases can easily be fixed by either installing a fixed version of A later, using B on its own if it can be, or rolling back. Of course, both solutions are flawed in the way that they assume user intent either way. Perhaps a better one would be to let the user specify whether they want to wait or not through a command line parameter, using the current behaviour as the default approach. WDYT? Regards, Leo From debbugs-submit-bounces@debbugs.gnu.org Tue Dec 17 10:50:59 2019 Received: (at submit) by debbugs.gnu.org; 17 Dec 2019 15:50:59 +0000 Received: from localhost ([127.0.0.1]:42229 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ihF7j-0006UD-Gr for submit@debbugs.gnu.org; Tue, 17 Dec 2019 10:50:59 -0500 Received: from lists.gnu.org ([209.51.188.17]:36419) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ihF7i-0006U5-Ij for submit@debbugs.gnu.org; Tue, 17 Dec 2019 10:50:58 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:59602) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ihF7h-0007Jz-AX for guix-patches@gnu.org; Tue, 17 Dec 2019 10:50:58 -0500 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=0.8 required=5.0 tests=BAYES_50,URIBL_BLOCKED autolearn=disabled version=3.3.2 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ihF7g-0003ES-4p for guix-patches@gnu.org; Tue, 17 Dec 2019 10:50:57 -0500 Received: from lepiller.eu ([89.234.186.109]:43694) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1ihF7e-00039b-2p; Tue, 17 Dec 2019 10:50:54 -0500 Received: from lepiller.eu (localhost [127.0.0.1]) by lepiller.eu (OpenSMTPD) with ESMTP id bb3d0371; Tue, 17 Dec 2019 15:50:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=lepiller.eu; h=date :in-reply-to:references:mime-version:content-type :content-transfer-encoding:subject:to:cc:from:message-id; s= dkim; bh=7XYQR3v6CSCasMUIQ/0CMrwEZNg=; b=C44YlIYCa+VnNVbuTo4asO+ /LScWy5fZ95H074GtIjHxCDiF8whMmk0x3ux0UP3FS+O2c6TkGvlqsYJ+wJOixe8 i2GwpR/9NTJYp5mY7kb2mSh9zZhS+5iLNHoal7TKsEKTzX8FjIXkChZcSnxcdtYh ly7ey5AzzbA9Fe0s6tXZFRkQL7n/U9gjazkwPQAB8KbUb39CTMZC4zcAtTEd3FST 8olEpVn2IhqF9BGcPzebZkyw3XNoyf0ab9jXCfIyeyQA4Dkmw07Gi/XApae+zutv AUW/Yj6zFWJKhE+5uKzKqp1OnHU4E5OxbS9OPch7PyLdKREPiY01Txfb7YAWmHw= = Received: by lepiller.eu (OpenSMTPD) with ESMTPSA id dcc186d8 (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256:NO); Tue, 17 Dec 2019 15:50:48 +0000 (UTC) Date: Tue, 17 Dec 2019 16:50:30 +0100 User-Agent: K-9 Mail for Android In-Reply-To: <3d0ca2a8b59dd99e15b55033bc89b2e21aa49814.camel@student.tugraz.at> References: <87tv5zrpjp.fsf@gnu.org> <3d0ca2a8b59dd99e15b55033bc89b2e21aa49814.camel@student.tugraz.at> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Subject: Re: [bug#38649] [PATCH] Parallelize `guix package` To: guix-patches@gnu.org, Leo Prikler , =?ISO-8859-1?Q?Ludovic_Court=E8s?= From: Julien Lepiller Message-ID: <09CEFC5C-85EB-4B43-BADD-C4D1920E656A@lepiller.eu> X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 89.234.186.109 X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: submit Cc: 38649@debbugs.gnu.org, julien lepiller X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Le 17 d=C3=A9cembre 2019 16:19:34 GMT+01:00, Leo Prikler a =C3=A9crit : > >Of course, any operation can also fail midway due to some step not >succeeding=2E In that case it would be as if one had issued the other >command right after that, which may perhaps not be what one wanted to >do (assuming I install package A, and some guide suggests to also build >related, but not dependency-connected package B, so I end up installing >B without A)=2E However, such cases can easily be fixed by either >installing a fixed version of A later, using B on its own if it can be, >or rolling back=2E > >Of course, both solutions are flawed in the way that they assume user >intent either way=2E Perhaps a better one would be to let the user >specify whether they want to wait or not through a command line >parameter, using the current behaviour as the default approach=2E > >WDYT? I might be missing something=2E Guix install etc act on a "hidden" descrip= cion of the profile=2E Tgey take the current profile, modify it as specifie= d (adding a package, renovinh another or upgrading some)=2E When you run tw= o guix package in parallel, they both work on the same profile, which creat= es unexpected results=2E The expectation behind tge lock is that users will cancel tge ocher comman= d and fix it before re-running it (e=2Eg=2E instead of guix install foo & g= uix install bar, run guix install foo bar)=2E > >Regards, >Leo From debbugs-submit-bounces@debbugs.gnu.org Tue Dec 17 11:16:13 2019 Received: (at submit) by debbugs.gnu.org; 17 Dec 2019 16:16:13 +0000 Received: from localhost ([127.0.0.1]:42270 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ihFW9-00084q-5Y for submit@debbugs.gnu.org; Tue, 17 Dec 2019 11:16:13 -0500 Received: from lists.gnu.org ([209.51.188.17]:40886) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ihFW6-00082w-Qo for submit@debbugs.gnu.org; Tue, 17 Dec 2019 11:16:11 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:45211) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ihFW3-0002pQ-Jh for guix-patches@gnu.org; Tue, 17 Dec 2019 11:16:10 -0500 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.5 required=5.0 tests=BAYES_50,RCVD_IN_DNSWL_MED, URIBL_BLOCKED autolearn=disabled version=3.3.2 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ihFW1-00033H-Kb for guix-patches@gnu.org; Tue, 17 Dec 2019 11:16:06 -0500 Received: from mailrelay.tugraz.at ([129.27.2.202]:56621) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1ihFW0-0002fm-Ux; Tue, 17 Dec 2019 11:16:05 -0500 Received: from nijino.local (213-240-64-42.hdsl.highway.telekom.at [213.240.64.42]) by mailrelay.tugraz.at (Postfix) with ESMTPSA id 47cjtL5xbFz3xhM; Tue, 17 Dec 2019 17:15:54 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tugraz.at; s=mailrelay; t=1576599355; bh=BaEN7YUaDkXOSuXrvoglbGVWOKNd+tCbCtkOJMMpjkA=; h=Subject:From:To:Cc:Date:In-Reply-To:References; b=BME3hh4n0jj9fAz5fljDOziRHlKaefkN3OumdpfNupvkdYYkItYmFN8OWgSmqDMgW HGOB7rCDTli284iqZ6+r5I7zh++zNJBXFUXu7I1WNcBBdy8kUpxk8q1X8UdNhscywI m0LM5kThKG9aiJaITQYk5kdxM78W++IClaAWG1kU= Message-ID: <303cfa9fb7484874e028e55bb0fb82a9387207a7.camel@student.tugraz.at> Subject: Re: [bug#38649] [PATCH] Parallelize `guix package` From: Leo Prikler To: Julien Lepiller , guix-patches@gnu.org, Ludovic =?ISO-8859-1?Q?Court=E8s?= Date: Tue, 17 Dec 2019 17:16:03 +0100 In-Reply-To: <09CEFC5C-85EB-4B43-BADD-C4D1920E656A@lepiller.eu> References: <87tv5zrpjp.fsf@gnu.org> <3d0ca2a8b59dd99e15b55033bc89b2e21aa49814.camel@student.tugraz.at> <09CEFC5C-85EB-4B43-BADD-C4D1920E656A@lepiller.eu> Content-Type: text/plain; charset="UTF-8" User-Agent: Evolution 3.32.4 MIME-Version: 1.0 X-TUG-Backscatter-control: bt4lQm5Tva3SBgCuw0EnZw X-Scanned-By: MIMEDefang 2.74 on 129.27.10.116 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 129.27.2.202 X-Spam-Score: -1.3 (-) X-Debbugs-Envelope-To: submit Cc: 38649@debbugs.gnu.org, julien lepiller X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.3 (--) Am Dienstag, den 17.12.2019, 16:50 +0100 schrieb Julien Lepiller: > Le 17 d=C3=A9cembre 2019 16:19:34 GMT+01:00, Leo Prikler < > leo.prikler@student.tugraz.at> a =C3=A9crit : > > Of course, any operation can also fail midway due to some step not > > succeeding. In that case it would be as if one had issued the > > other > > command right after that, which may perhaps not be what one wanted > > to > > do (assuming I install package A, and some guide suggests to also > > build > > related, but not dependency-connected package B, so I end up > > installing > > B without A). However, such cases can easily be fixed by either > > installing a fixed version of A later, using B on its own if it can > > be, > > or rolling back. > >=20 > > Of course, both solutions are flawed in the way that they assume > > user > > intent either way. Perhaps a better one would be to let the user > > specify whether they want to wait or not through a command line > > parameter, using the current behaviour as the default approach. > >=20 > > WDYT? >=20 > I might be missing something. Guix install etc act on a "hidden" > descripcion of the profile. Tgey take the current profile, modify it > as specified (adding a package, renovinh another or upgrading some). > When you run two guix package in parallel, they both work on the same > profile, which creates unexpected results. That's why the lock is claimed first. This way, the second process acts on the profile that the first generated. I've tested this by installing cowsay in parallel to lolcat, but it should work for bigger packages in much the same way. > The expectation behind tge lock is that users will cancel tge ocher > command and fix it before re-running it (e.g. instead of guix install > foo & guix install bar, run guix install foo bar). That is perhaps a reasonable expectation in most cases, but may be annoying in others. Take any package with an absurdly long build time (e.g. icecat) and then think "Oh, but I also wanted this" while it is building. Now you have to either actively wait for icecat to complete or stop it, add the other package and suffer the same build time again, (whereas in the other way, you can wait for icecat to complete and still launch a second process). With the parallel builds of 0002, thing become even better, as you can use bar even before foo is completed in case it manages to grab the lock first. With the long build of icecat against a package with a relatively short build, this could very well be the case and might end up being a game changer. Of course, one could abuse ad-hoc environments as well while waiting for the first process to finish, but I don't think that's how people running into this problem expect to be solving it (especially if they do want both foo and bar in their profiles). Regards, Leo From debbugs-submit-bounces@debbugs.gnu.org Wed Dec 18 09:37:58 2019 Received: (at 38649) by debbugs.gnu.org; 18 Dec 2019 14:37:58 +0000 Received: from localhost ([127.0.0.1]:42803 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ihaSc-0002sl-6U for submit@debbugs.gnu.org; Wed, 18 Dec 2019 09:37:58 -0500 Received: from eggs.gnu.org ([209.51.188.92]:47744) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ihaSa-0002sY-Ft for 38649@debbugs.gnu.org; Wed, 18 Dec 2019 09:37:57 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]:43878) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ihaSS-0007vK-8d; Wed, 18 Dec 2019 09:37:48 -0500 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=41322 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1ihaSR-0000nz-QL; Wed, 18 Dec 2019 09:37:48 -0500 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Leo Prikler Subject: Re: [bug#38649] [PATCH] Parallelize `guix package` References: <87tv5zrpjp.fsf@gnu.org> <3d0ca2a8b59dd99e15b55033bc89b2e21aa49814.camel@student.tugraz.at> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 28 Frimaire an 228 de la =?utf-8?Q?R=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Wed, 18 Dec 2019 15:37:45 +0100 In-Reply-To: <3d0ca2a8b59dd99e15b55033bc89b2e21aa49814.camel@student.tugraz.at> (Leo Prikler's message of "Tue, 17 Dec 2019 16:19:34 +0100") Message-ID: <87lfr9pume.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.3 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 38649 Cc: julien lepiller , 38649@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hi Leo, Leo Prikler skribis: > I think the current policy is wait-for-lock deferred to the user. The > user has to let the first task complete before they can start the > second. In this setup, the user can simply launch the setup and trust, > that it will complete later while taking into account the changes the > first one has made. > > Let's talk about three classes of operations =E2=80=93 installations, rem= ovals > and upgrades =E2=80=93 and their interactions. I will not take into acco= unt > roll-back, switch-generation and delete-generation, as it is > nonsensical to perform these in parallel to any other action. Perhaps > we could check for their presence first and acquire the lock with no- > wait semantics in that case. > > - any operation on different packages: Either succeeds first and the > other builds on the profile it generates. As there is no collision in > the packages themselves, there will be no harm. > - install same package twice: Either succeeds first, the other will be > a no-op. > - install vs. remove same package: Non-deterministic, but why would you > do that? > - install vs. upgrade same package: Upgrade will be a no-op in either > case. > - remove vs. upgrade same package: Upgrade may inadvertently upgrade > the old package if it happens to come first, but in the final package > it will be removed either way.=20 > > Of course, any operation can also fail midway due to some step not > succeeding. In that case it would be as if one had issued the other > command right after that, which may perhaps not be what one wanted to > do (assuming I install package A, and some guide suggests to also build > related, but not dependency-connected package B, so I end up installing > B without A). However, such cases can easily be fixed by either > installing a fixed version of A later, using B on its own if it can be, > or rolling back. > > Of course, both solutions are flawed in the way that they assume user > intent either way. Perhaps a better one would be to let the user > specify whether they want to wait or not through a command line > parameter, using the current behaviour as the default approach. I cannot think of a useful behavior if wait-for-lock were implemented. Really, as a user, you=E2=80=99d be unable to know what the end result is. = I don=E2=80=99t see that as very useful. :-) What you describe above as potential mitigation is just that, mitigation, and it could easily become complex (as a maintainer, I woudn=E2=80=99t want to be responsible for this kind of complexity :-)), and again, for very questionable =E2=80=9Cgains=E2=80=9D. Thoughts? Julien? Thanks, Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Tue May 11 10:25:23 2021 Received: (at 38649) by debbugs.gnu.org; 11 May 2021 14:25:23 +0000 Received: from localhost ([127.0.0.1]:37031 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lgTK7-0006Yx-8T for submit@debbugs.gnu.org; Tue, 11 May 2021 10:25:23 -0400 Received: from mailrelay.tugraz.at ([129.27.2.202]:26330) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lgTK4-0006Yk-Ua; Tue, 11 May 2021 10:25:21 -0400 Received: from [10.0.0.4] (91-114-247-246.adsl.highway.telekom.at [91.114.247.246]) by mailrelay.tugraz.at (Postfix) with ESMTPSA id 4FfgDs2lXmz1LLyL; Tue, 11 May 2021 16:25:17 +0200 (CEST) DKIM-Filter: OpenDKIM Filter v2.11.0 mailrelay.tugraz.at 4FfgDs2lXmz1LLyL DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tugraz.at; s=mailrelay; t=1620743117; bh=brt2wMMaFuDiitoIXNsc1i3i4uZvYO3V6NavXGAONKA=; h=Subject:From:To:Cc:Date:In-Reply-To:References:From; b=SHo4W/9DMsJ19L89NAgny0Jz/urJIWcHIyv7GRYI0HM1AEMRIhWTGUHSCtGij8e0b QYR74GrLXJsjJU7bL/x7CiXjGZhcurvp/dcwqtMEMoA5j/hCaDDsD86/w+hGdzajVI pzDN6nt6DONGOg99IBjxJ4xAmd4rj7D32vNBsy14= Message-ID: Subject: Re: [bug#38649] [PATCH] Parallelize `guix package` From: Leo Prikler To: Ludovic =?ISO-8859-1?Q?Court=E8s?= Date: Tue, 11 May 2021 16:24:56 +0200 In-Reply-To: <87lfr9pume.fsf@gnu.org> References: <87tv5zrpjp.fsf@gnu.org> <3d0ca2a8b59dd99e15b55033bc89b2e21aa49814.camel@student.tugraz.at> <87lfr9pume.fsf@gnu.org> Content-Type: text/plain; charset="UTF-8" User-Agent: Evolution 3.34.2 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-TUG-Backscatter-control: bt4lQm5Tva3SBgCuw0EnZw X-Spam-Scanner: SpamAssassin 3.003001 X-Spam-Score-relay: -1.9 X-Scanned-By: MIMEDefang 2.74 on 129.27.10.117 X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 38649 Cc: control@debbugs.gnu.org, julien lepiller , 38649@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) close 38649 thanks Am Mittwoch, den 18.12.2019, 15:37 +0100 schrieb Ludovic Courtès: > I cannot think of a useful behavior if wait-for-lock were > implemented. > Really, as a user, you’d be unable to know what the end result is. I > don’t see that as very useful. :-) It took a while, but I feel you've convinced me. From unknown Mon Jun 16 23:51:53 2025 Received: (at fakecontrol) by fakecontrolmessage; To: internal_control@debbugs.gnu.org From: Debbugs Internal Request Subject: Internal Control Message-Id: bug archived. Date: Wed, 09 Jun 2021 11:24:06 +0000 User-Agent: Fakemail v42.6.9 # This is a fake control message. # # The action: # bug archived. thanks # This fakemail brought to you by your local debbugs # administrator