From debbugs-submit-bounces@debbugs.gnu.org Wed May 18 10:01:38 2022 Received: (at submit) by debbugs.gnu.org; 18 May 2022 14:01:38 +0000 Received: from localhost ([127.0.0.1]:33049 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nrKF8-00034J-3H for submit@debbugs.gnu.org; Wed, 18 May 2022 10:01:38 -0400 Received: from lists.gnu.org ([209.51.188.17]:37554) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nrKF4-00031p-V0 for submit@debbugs.gnu.org; Wed, 18 May 2022 10:01:36 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:44810) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nrKF3-000256-KX for bug-guix@gnu.org; Wed, 18 May 2022 10:01:34 -0400 Received: from sender4-of-o51.zoho.com ([136.143.188.51]:21122) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nrKEt-000846-8Y; Wed, 18 May 2022 10:01:33 -0400 ARC-Seal: i=1; a=rsa-sha256; t=1652882469; cv=none; d=zohomail.com; s=zohoarc; b=Qn5nYNNPyQMIrQm3N3WF8swQ6g+x4RwXS82BcN4pQfJHSxtGfcXiLQxUwwg7Aqf7b/VipTu3ufSr2WR/S94Re0sqPqlkxYB8YjiZwDP18damhEYKcCm8SOPjFoES/ASeXsNClETYua2j71jjn0XTDCnkZaBYU9dvIDArmi3VXYQ= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1652882469; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:MIME-Version:Message-ID:Subject:To; bh=gV5Ob8t7ZvuHDXn8fP33+Py781zwI88RqthA7T62hC4=; b=j/IDLhoiKUwhRvkifMp9QblCjSPkZKw3qtmbYD3u8d9LQVwiZcAWHUUWF96pTDrL1/KPMOkPw14S8hQfwHeov0cgyokV2hTU1+yzHsiFqhobW2W0qkYQwFt3ztrnEpHehbx8N+6Z2yuOGfaZVYH6FXAkp8FB/8jLg4LePWdazDk= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass header.i=elephly.net; spf=pass smtp.mailfrom=rekado@elephly.net; dmarc=pass header.from= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1652882469; s=zoho; d=elephly.net; i=rekado@elephly.net; h=From:From:To:To:Cc:Cc:Subject:Subject:Date:Date:Message-ID:MIME-Version:Content-Type:Content-Transfer-Encoding:Message-Id:Reply-To; bh=gV5Ob8t7ZvuHDXn8fP33+Py781zwI88RqthA7T62hC4=; b=doCotpDYZsDm/q6bDDvufjJrbjDh+VQwUy3c8+kOEo+Sjmf9zgWBWVVazOM51etj KLmtLrNBXKwgebif1JnQMhj2dHdbBTgtxGzNwJxfMQknvXUKM/5tgqOGiV625EDQtJw IRQ9hrLCWnO0KwJJyRODvvT7rmOPGzYM2BBmcSYw= Received: from localhost (118-108-142-46.pool.kielnet.net [46.142.108.118]) by mx.zohomail.com with SMTPS id 1652882468127716.8637033216065; Wed, 18 May 2022 07:01:08 -0700 (PDT) User-agent: mu4e 1.6.10; emacs 28.0.50 From: Ricardo Wurmus To: bug-guix@gnu.org Subject: excessively large manifests due to propagation Date: Wed, 18 May 2022 15:53:06 +0200 X-URL: https://elephly.net X-PGP-Key: https://elephly.net/rekado.pubkey X-PGP-Fingerprint: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC Message-ID: <87sfp7kkim.fsf@elephly.net> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-ZohoMailClient: External Received-SPF: pass client-ip=136.143.188.51; envelope-from=rekado@elephly.net; helo=sender4-of-o51.zoho.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-Spam-Score: -1.3 (-) X-Debbugs-Envelope-To: submit Cc: Ludovic =?utf-8?Q?Court=C3=A8s?= X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.3 (--) Packages of some languages rely heavily on propagation. R is one example. Since the generated =E2=80=9Cmanifest=E2=80=9D file of a Guix pro= file records entries for all propagated packages, this can get really big really quickly. A profile consisting only of four R packages (r-seurat, r-cistopic, r-monocle3, and r-cicero-monocle3) results in a =E2=80=9Cmanifest=E2=80=9D = file that weighs 7.1MB. At the MDC I repeatedly encountered manifest files that are exceeding 24MB. Simply reading that big a file with (call-with-input-file "huge-manifest" read) takes several seconds. On the MDC cluster I observed a delay of about 27 seconds. Disabling read positions with (read-disable 'positions) significantly speeds this up (18s vs 27s), but it=E2=80=99s still very slow. We may be able to speed things up by supporting definitions or references in manifest files, so that we don=E2=80=99t need to repeat a sub= -tree when generating the file. --=20 Ricardo From debbugs-submit-bounces@debbugs.gnu.org Mon May 23 11:56:20 2022 Received: (at control) by debbugs.gnu.org; 23 May 2022 15:56:20 +0000 Received: from localhost ([127.0.0.1]:50074 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ntAPs-0007sY-BJ for submit@debbugs.gnu.org; Mon, 23 May 2022 11:56:20 -0400 Received: from mail3-relais-sop.national.inria.fr ([192.134.164.104]:52208) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ntAPr-0007sG-Bu for control@debbugs.gnu.org; Mon, 23 May 2022 11:56:19 -0400 Authentication-Results: mail3-relais-sop.national.inria.fr; dkim=none (message not signed) header.i=none; spf=SoftFail smtp.mailfrom=ludo@gnu.org; dmarc=fail (p=none dis=none) d=gnu.org X-IronPort-AV: E=Sophos;i="5.91,246,1647298800"; d="scan'208";a="14873652" Received: from unknown (HELO ribbon) ([193.50.110.143]) by mail3-relais-sop.national.inria.fr with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 May 2022 17:56:13 +0200 Date: Mon, 23 May 2022 17:56:12 +0200 Message-Id: <87o7zo2qg3.fsf@gnu.org> To: control@debbugs.gnu.org From: =?utf-8?Q?Ludovic_Court=C3=A8s?= Subject: control message for bug #55499 MIME-version: 1.0 Content-type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Spam-Score: -1.3 (-) X-Debbugs-Envelope-To: control X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.3 (--) severity 55499 important quit From debbugs-submit-bounces@debbugs.gnu.org Tue May 24 11:30:45 2022 Received: (at 55499) by debbugs.gnu.org; 24 May 2022 15:30:45 +0000 Received: from localhost ([127.0.0.1]:53121 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ntWUf-0001up-9Q for submit@debbugs.gnu.org; Tue, 24 May 2022 11:30:45 -0400 Received: from eggs.gnu.org ([209.51.188.92]:46144) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ntWUb-0001lH-Ts for 55499@debbugs.gnu.org; Tue, 24 May 2022 11:30:43 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:38904) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ntWUW-0003Gw-I1; Tue, 24 May 2022 11:30:36 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:In-Reply-To:Date:References:Subject:To: From; bh=K9wyVp3pyucdrwo4KnP6/AI/FPufQurGPbQhFES8bXU=; b=Zal/Zwdc1yOHFrjOIqVV CT0cl5ghaCkt87yCIOKnqsR6DFbZ93CNwW784TWkD324ByR1b2MATABZApla1AXyIKupPTlyHKYRP zewzmEKQtilKU615C7xVa7waV71FOfkaR+BeJju/et7hIgBHsIbKMd0MBrho1BFph5kNiCTRu+oOq zxbuRYyn5lfm5WUKbEL2VOOP1kAEKKnqKjeakDXVl87pmTNDy5HkkhlA7qjI1O8MgsmH5yQw3jRM0 0Cfv1wxLYG54H2RAKJFTRDMzan+DLTVjjz84J7op3GvB4LONm0E6BRL3CztCfPTkgDf3gLH5/aGyh GIQLFoHKzZo2Vg==; Received: from [2001:660:6102:320:e120:2c8f:8909:cdfe] (port=40714 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ntWUV-0007Om-QN; Tue, 24 May 2022 11:30:36 -0400 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Ricardo Wurmus Subject: Re: bug#55499: excessively large manifests due to propagation References: <87sfp7kkim.fsf@elephly.net> Date: Tue, 24 May 2022 17:30:33 +0200 In-Reply-To: <87sfp7kkim.fsf@elephly.net> (Ricardo Wurmus's message of "Wed, 18 May 2022 15:53:06 +0200") Message-ID: <87a6b7j6cm.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 55499 Cc: 55499@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hi! Ricardo Wurmus skribis: > A profile consisting only of four R packages (r-seurat, r-cistopic, > r-monocle3, and r-cicero-monocle3) results in a =E2=80=9Cmanifest=E2=80= =9D file that > weighs 7.1MB. At the MDC I repeatedly encountered manifest files that > are exceeding 24MB. Commit 93f601d97ca2d9b82c41afeb86879ee37eae39e6 provides a 12% size reduction on this example, and it=E2=80=99s backward-compatible and cheap. I=E2=80=99ll try and follow up with changes along the lines you describe. Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Wed May 25 00:36:12 2022 Received: (at 55499) by debbugs.gnu.org; 25 May 2022 04:36:13 +0000 Received: from localhost ([127.0.0.1]:54081 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ntikm-00073k-KD for submit@debbugs.gnu.org; Wed, 25 May 2022 00:36:12 -0400 Received: from sender4-of-o50.zoho.com ([136.143.188.50]:21027) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ntiki-00073Z-GW for 55499@debbugs.gnu.org; Wed, 25 May 2022 00:36:10 -0400 ARC-Seal: i=1; a=rsa-sha256; t=1653453358; cv=none; d=zohomail.com; s=zohoarc; b=my7b2kJp/obHnFg7+ptUdrTHijMq5XrAsKtIX0Zd3N1wTwVyCZB0gumdm3cTdwVzlEZM9JNopTL1gXv1XLHQWRV+4sMu5goJkusneqzL5GUas5W84YaouEajB8uku2fFAREf/hVYsiV7htgUFSHlGXa39xeeBmzMEmxTa6/62pU= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1653453358; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:MIME-Version:Message-ID:References:Subject:To; bh=bsj0EMNmrk+uSqfw/tzrPwy3o7z5hMHAWvU2Pj6y314=; b=IzDGTABpZ/qiCcd7+dU3na1h+yCqexlafIhSsRk2GfhN6SlAIssv4WBq6wNdSYjLeUj6T1mGYvceBKj0aeXVmG/46edPhT1b0/tScOwSBmCbQKQsbqNxLuotERkLUhD6jEcKPLoAQFjeiYbqrsQp4hQKpXOy7I2lwAGV6DtHIlA= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass header.i=elephly.net; spf=pass smtp.mailfrom=rekado@elephly.net; dmarc=pass header.from= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1653453358; s=zoho; d=elephly.net; i=rekado@elephly.net; h=References:From:From:To:To:Cc:Cc:Subject:Subject:Date:Date:In-reply-to:Message-ID:MIME-Version:Content-Type:Content-Transfer-Encoding:Message-Id:Reply-To; bh=bsj0EMNmrk+uSqfw/tzrPwy3o7z5hMHAWvU2Pj6y314=; b=Leh0KvLZcqSDoBaci4LZGkt4LVMnA5jSAgcHyFE5k6nj0cUNUrYYh4uIDKL6UCxC Ird4ygZRRULSmgCNYo9P3lYU1qlC8FQxsj0pYTP5FtNx8wnAD/6bRmypbCp+bsLbJ1i EiLoAVQIHvLfyZZc4hSDye1Cflgwz5dOv6bh1wYI= Received: from localhost (187-110-142-46.pool.kielnet.net [46.142.110.187]) by mx.zohomail.com with SMTPS id 1653453354334395.39543278007943; Tue, 24 May 2022 21:35:54 -0700 (PDT) References: <87sfp7kkim.fsf@elephly.net> <87a6b7j6cm.fsf@gnu.org> User-agent: mu4e 1.6.10; emacs 28.0.50 From: Ricardo Wurmus To: Ludovic =?utf-8?Q?Court=C3=A8s?= Subject: Re: bug#55499: excessively large manifests due to propagation Date: Wed, 25 May 2022 06:35:32 +0200 In-reply-to: <87a6b7j6cm.fsf@gnu.org> X-URL: https://elephly.net X-PGP-Key: https://elephly.net/rekado.pubkey X-PGP-Fingerprint: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC Message-ID: <87sfoygrfd.fsf@elephly.net> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-ZohoMailClient: External X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 55499 Cc: 55499@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Ludovic Court=C3=A8s writes: > Hi! > > Ricardo Wurmus skribis: > >> A profile consisting only of four R packages (r-seurat, r-cistopic, >> r-monocle3, and r-cicero-monocle3) results in a =E2=80=9Cmanifest=E2=80= =9D file that >> weighs 7.1MB. At the MDC I repeatedly encountered manifest files that >> are exceeding 24MB. > > Commit 93f601d97ca2d9b82c41afeb86879ee37eae39e6 provides a 12% size > reduction on this example, and it=E2=80=99s backward-compatible and cheap. Excellent! This is a great first step. > I=E2=80=99ll try and follow up with changes along the lines you describe. Thank you! --=20 Ricardo From debbugs-submit-bounces@debbugs.gnu.org Tue May 31 12:09:53 2022 Received: (at 55499) by debbugs.gnu.org; 31 May 2022 16:09:53 +0000 Received: from localhost ([127.0.0.1]:48727 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nw4RL-0004NQ-G7 for submit@debbugs.gnu.org; Tue, 31 May 2022 12:09:53 -0400 Received: from eggs.gnu.org ([209.51.188.92]:49668) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nw4RJ-0004N8-2f for 55499@debbugs.gnu.org; Tue, 31 May 2022 12:09:49 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:41042) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nw4RD-0002LP-Qf; Tue, 31 May 2022 12:09:43 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:References:In-Reply-To:Date:Subject:To: From; bh=4FzyPDQ2/wKppshfTjMndtEDbgCepbz97k16beuPG+Q=; b=ODRWGHVaFZ4SZMYOd2qM /UyhfoZct0dVVyKIvtiG+ycKZuznZs9YSdxXH06C5c98ALXaHIbUOb/5vEKio99rK6ft6SLOpqsMt yYQURFweTot6dkB2m6Zn9SyZVCQl4803zO3miEPpxiDgYnOybOzeuaXXQ+6Nm9USMSgzThXwmoYoZ ApujuYiFWzXFBzEs+rDX9qRhysga9esRhJ5SPO+CTyo0fmvyZtHZYa4ouo3mPl8j+nHsFHfHkbmwu VRXl3M1pjHvjPJUPGjxdStWbPySLXQtD+sTKk3kZj17PTQj/zn7DLfblIs5QGlzZJP94qvwDyFhqm WAx2sBilXsI/5w==; Received: from [2001:660:6102:320:e120:2c8f:8909:cdfe] (port=52502 helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1nw4R9-00067m-FK; Tue, 31 May 2022 12:09:42 -0400 From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= To: 55499@debbugs.gnu.org Subject: [PATCH 0/3] Make 'manifest' files more compact Date: Tue, 31 May 2022 18:09:13 +0200 Message-Id: <20220531160916.21508-1-ludo@gnu.org> X-Mailer: git-send-email 2.36.1 In-Reply-To: <87sfp7kkim.fsf@elephly.net> References: <87sfp7kkim.fsf@elephly.net> X-Debbugs-Cc: Ricardo Wurmus MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 55499 Cc: =?UTF-8?q?Ludovic=20Court=C3=A8s?= X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hello, These patches implement what you suggested on IRC: not repeating entire manifest entries and their propagated inputs. This has a dramatic impact on the size of the ‘manifest’ file and on the memory and processing time to read it for the the use case you gave. The second patch goes a tiny bit further by making the ‘search-paths’ and ‘propagated-inputs’ fields optional, shaving another ~10% on the size of ‘manifest’ in this example. The second patch should be squashed with the first one (so we don’t bump version formats a second time and duplicate code). It’s kinda optional because it doesn’t bring much compared to the first patch and causes a bit of extra complexity, but maybe it’s still worth keeping? Could you try this on your larger use cases and tell me how it goes? Thanks, Ludo’. Ludovic Courtès (3): tests: Augment profile collision test. profiles: Do not repeat entries in 'manifest' file. squash! profiles: Make all entry fields optional. guix/build/profiles.scm | 32 ++++++++-- guix/profiles.scm | 137 ++++++++++++++++++++++++++++++++-------- tests/profiles.scm | 52 ++++++++++++++- 3 files changed, 187 insertions(+), 34 deletions(-) base-commit: fed51b26141548a5bae349a5e1d8d6f681320f4f -- 2.36.1 From debbugs-submit-bounces@debbugs.gnu.org Tue May 31 12:09:57 2022 Received: (at 55499) by debbugs.gnu.org; 31 May 2022 16:09:57 +0000 Received: from localhost ([127.0.0.1]:48729 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nw4RR-0004Nv-4h for submit@debbugs.gnu.org; Tue, 31 May 2022 12:09:57 -0400 Received: from eggs.gnu.org ([209.51.188.92]:49678) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nw4RJ-0004NA-Sc for 55499@debbugs.gnu.org; Tue, 31 May 2022 12:09:52 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:41050) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nw4RE-0002MJ-MG; Tue, 31 May 2022 12:09:44 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:References:In-Reply-To:Date:Subject:To: From; bh=sjABqRdlHF6jZWwr0em6ySoXDazNqLZd5o7Q+B4UjUg=; b=oeURduUKgE224HqIkW1t 5ztMgWgStRRCp9wHL7qlJPMd6jP2zSmYCBQvCqwnewAeboHWQvlV3aEAVt2PHZvmxDBWHwme3BOLF OewO8rzIhi42Yo5UiyYkSLidbwl/SZZnsWqKQzBTlS0uFR5x3mcPFi5PTBLBKTT0O+LT9NwjKmuFg lB0Pg+PY9yiL+CGS7jotOGptzbxYqwNRC7UhibQzCQrlJot2VffmbpFV/K/7aFQyGxwxo6nyyh8Jj hnIshF+TfFwH94ximAJTdalHOyp71cHYRF1osCYm6gmnvqsBxv2p4FdTSaSS1GretDs7Qo3N58a5o zbE+DrKVmHcM6A==; Received: from [2001:660:6102:320:e120:2c8f:8909:cdfe] (port=52502 helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1nw4RE-00067m-0p; Tue, 31 May 2022 12:09:44 -0400 From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= To: 55499@debbugs.gnu.org Subject: [PATCH 2/3] profiles: Do not repeat entries in 'manifest' file. Date: Tue, 31 May 2022 18:09:15 +0200 Message-Id: <20220531160916.21508-3-ludo@gnu.org> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20220531160916.21508-1-ludo@gnu.org> References: <87sfp7kkim.fsf@elephly.net> <20220531160916.21508-1-ludo@gnu.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 55499 Cc: =?UTF-8?q?Ludovic=20Court=C3=A8s?= X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Fixes . Reported by Ricardo Wurmus . With this change, the manifest file created for: guix install r r-seurat r-cistopic r-monocle3 r-cicero-monocle3 r-assertthat goes from 5.6M to 192K. Likewise, on this profile, wall-clock time of: GUIX_PROFILING=gc guix package -I goes from 0.7s to 0.1s, with heap usage going from 55M to 9M. * guix/profiles.scm (manifest->gexp)[entry->gexp]: Turn into a monadic procedure. Return a 'repeated' sexp if ENTRY was already visited before. Adjust caller accordingly. Bump manifest version. (sexp->manifest)[sexp->manifest-entry]: Turn into a monadic procedure. Add case for 'repeated' nodes. Add each entry to the current state vhash. Add clause for version 4 manifests. * tests/profiles.scm ("deduplication of repeated entries"): New test. * guix/build/profiles.scm (manifest-sexp->inputs+search-paths): Expect version 4. Add clause for 'repeated' nodes. --- guix/build/profiles.scm | 4 +- guix/profiles.scm | 127 ++++++++++++++++++++++++++++------------ tests/profiles.scm | 42 +++++++++++++ 3 files changed, 134 insertions(+), 39 deletions(-) diff --git a/guix/build/profiles.scm b/guix/build/profiles.scm index f9875ca92e..c4460f624b 100644 --- a/guix/build/profiles.scm +++ b/guix/build/profiles.scm @@ -150,7 +150,7 @@ (define (manifest-sexp->inputs+search-paths manifest) values: the list of store items of its manifest entries, and the list of search path specifications." (match manifest ;this must match 'manifest->gexp' - (('manifest ('version 3) + (('manifest ('version 4) ('packages (entries ...))) (let loop ((entries entries) (inputs '()) @@ -162,6 +162,8 @@ (define (manifest-sexp->inputs+search-paths manifest) (loop (append rest deps) ;breadth-first traversal (cons item inputs) (append paths search-paths))) + ((('repeated name version item) . rest) + (loop rest inputs search-paths)) (() (values (reverse inputs) (delete-duplicates diff --git a/guix/profiles.scm b/guix/profiles.scm index bf50c00a1e..44ff37e75b 100644 --- a/guix/profiles.scm +++ b/guix/profiles.scm @@ -455,31 +455,53 @@ (define (inferior->entry) (define (manifest->gexp manifest) "Return a representation of MANIFEST as a gexp." (define (entry->gexp entry) - (match entry - (($ name version output (? string? path) - (deps ...) (search-paths ...) _ (properties ...)) - #~(#$name #$version #$output #$path - (propagated-inputs #$(map entry->gexp deps)) - (search-paths #$(map search-path-specification->sexp - search-paths)) - #$@(if (null? properties) - #~() - #~((properties . #$properties))))) - (($ name version output package - (deps ...) (search-paths ...) _ (properties ...)) - #~(#$name #$version #$output - (ungexp package (or output "out")) - (propagated-inputs #$(map entry->gexp deps)) - (search-paths #$(map search-path-specification->sexp - search-paths)) - #$@(if (null? properties) - #~() - #~((properties . #$properties))))))) + ;; Maintain in state monad a vhash of visited entries, indexed by their + ;; item, usually package objects (we cannot use the entry itself as an + ;; index since identical entries are usually not 'eq?'). Use that vhash + ;; to avoid repeating duplicate entries. This is particularly useful in + ;; the presence of propagated inputs, where we could otherwise end up + ;; repeating large trees. + (mlet %state-monad ((visited (current-state))) + (if (match (vhash-assq (manifest-entry-item entry) visited) + ((_ . previous-entry) + (manifest-entry=? previous-entry entry)) + (#f #f)) + (return #~(repeated #$(manifest-entry-name entry) + #$(manifest-entry-version entry) + #$(manifest-entry-item entry))) + (mbegin %state-monad + (set-current-state (vhash-consq (manifest-entry-item entry) + entry visited)) + (mlet %state-monad ((deps (mapm %state-monad entry->gexp + (manifest-entry-dependencies entry)))) + (return + (match entry + (($ name version output (? string? path) + (_deps ...) (search-paths ...) _ (properties ...)) + #~(#$name #$version #$output #$path + (propagated-inputs #$deps) + (search-paths #$(map search-path-specification->sexp + search-paths)) + #$@(if (null? properties) + #~() + #~((properties . #$properties))))) + (($ name version output package + (_deps ...) (search-paths ...) _ (properties ...)) + #~(#$name #$version #$output + (ungexp package (or output "out")) + (propagated-inputs #$deps) + (search-paths #$(map search-path-specification->sexp + search-paths)) + #$@(if (null? properties) + #~() + #~((properties . #$properties)))))))))))) (match manifest (($ (entries ...)) - #~(manifest (version 3) - (packages #$(map entry->gexp entries)))))) + #~(manifest (version 4) + (packages #$(run-with-state + (mapm %state-monad entry->gexp entries) + vlist-null)))))) (define (find-package name version) "Return a package from the distro matching NAME and possibly VERSION. This @@ -522,25 +544,44 @@ (define (infer-dependency item parent) (define* (sexp->manifest-entry sexp #:optional (parent (delay #f))) (match sexp + (('repeated name version path) + ;; This entry is the same as another one encountered earlier; look it + ;; up and return it. + (mlet %state-monad ((visited (current-state)) + (key -> (list name version path))) + (match (vhash-assoc key visited) + (#f + (raise (formatted-message + (G_ "invalid repeated entry in profile: ~s") + sexp))) + ((_ . entry) + (return entry))))) ((name version output path ('propagated-inputs deps) ('search-paths search-paths) extra-stuff ...) - ;; For each of DEPS, keep a promise pointing to ENTRY. - (letrec* ((deps* (map (cut sexp->manifest-entry <> (delay entry)) - deps)) - (entry (manifest-entry - (name name) - (version version) - (output output) - (item path) - (dependencies deps*) - (search-paths (map sexp->search-path-specification - search-paths)) - (parent parent) - (properties (or (assoc-ref extra-stuff 'properties) - '()))))) - entry)))) + (mlet* %state-monad + ((entry -> #f) + (deps* (mapm %state-monad + (cut sexp->manifest-entry <> (delay entry)) + deps)) + (visited (current-state)) + (key -> (list name version path))) + (set! entry ;XXX: emulate 'letrec*' + (manifest-entry + (name name) + (version version) + (output output) + (item path) + (dependencies deps*) + (search-paths (map sexp->search-path-specification + search-paths)) + (parent parent) + (properties (or (assoc-ref extra-stuff 'properties) + '())))) + (mbegin %state-monad + (set-current-state (vhash-cons key entry visited)) + (return entry)))))) (match sexp (('manifest ('version 0) @@ -608,7 +649,17 @@ (define* (sexp->manifest-entry sexp #:optional (parent (delay #f))) ;; Version 3 represents DEPS as full-blown manifest entries. (('manifest ('version 3 minor-version ...) ('packages (entries ...))) - (manifest (map sexp->manifest-entry entries))) + (manifest (run-with-state + (mapm %state-monad sexp->manifest-entry entries) + vlist-null))) + + ;; Version 4 deduplicates repeated entries, as can happen with deep + ;; propagated input trees. + (('manifest ('version 4 minor-version ...) + ('packages (entries ...))) + (manifest (run-with-state + (mapm %state-monad sexp->manifest-entry entries) + vlist-null))) (_ (raise (condition (&message (message "unsupported manifest format"))))))) diff --git a/tests/profiles.scm b/tests/profiles.scm index 7e51d37ab9..3838d971c9 100644 --- a/tests/profiles.scm +++ b/tests/profiles.scm @@ -586,6 +586,48 @@ (define (entry->sexp entry) #:locales? #f))) (return #f))))) +(test-assertm "deduplication of repeated entries" + ;; Make sure the 'manifest' file does not duplicate identical entries. + ;; See . + (mlet* %store-monad ((p0 -> (dummy-package "p0" + (build-system trivial-build-system) + (arguments + `(#:guile ,%bootstrap-guile + #:builder (mkdir (assoc-ref %outputs "out")))) + (propagated-inputs + `(("guile" ,%bootstrap-guile))))) + (p1 -> (package + (inherit p0) + (name "p1"))) + (drv (profile-derivation (packages->manifest + (list p0 p1)) + #:hooks '() + #:locales? #f))) + (mbegin %store-monad + (built-derivations (list drv)) + (let ((file (string-append (derivation->output-path drv) + "/manifest")) + (manifest (profile-manifest (derivation->output-path drv)))) + (define (contains-repeated? sexp) + (match sexp + (('repeated _ ...) #t) + ((lst ...) (any contains-repeated? sexp)) + (_ #f))) + + (return (and (contains-repeated? (call-with-input-file file read)) + + ;; MANIFEST has two entries for %BOOTSTRAP-GUILE since + ;; it's propagated both from P0 and from P1. When + ;; reading a 'repeated' node, 'read-manifest' should + ;; reuse the previously-read entry so the two + ;; %BOOTSTRAP-GUILE entries must be 'eq?'. + (match (manifest-entries manifest) + (((= manifest-entry-dependencies (dep0)) + (= manifest-entry-dependencies (dep1))) + (and (string=? (manifest-entry-name dep0) + (package-name %bootstrap-guile)) + (eq? dep0 dep1)))))))))) + (test-assertm "no collision" ;; Here we have an entry that is "lowered" (its 'item' field is a store file ;; name) and another entry (its 'item' field is a package) that is -- 2.36.1 From debbugs-submit-bounces@debbugs.gnu.org Tue May 31 12:09:58 2022 Received: (at 55499) by debbugs.gnu.org; 31 May 2022 16:09:58 +0000 Received: from localhost ([127.0.0.1]:48731 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nw4RR-0004O0-PP for submit@debbugs.gnu.org; Tue, 31 May 2022 12:09:58 -0400 Received: from eggs.gnu.org ([209.51.188.92]:49664) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nw4RJ-0004N7-0b for 55499@debbugs.gnu.org; Tue, 31 May 2022 12:09:52 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:41048) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nw4RD-0002Ll-PV; Tue, 31 May 2022 12:09:43 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:References:In-Reply-To:Date:Subject:To: From; bh=qihHKTPRSaOMn8OoQQhfdaq7/RB4JCTYhzo1ERcbFXM=; b=bsh+6zZIM2yExtEm3Hgi CsUra3JSEnAJ7CPOBw/80iNJrbBxI3SkNNgnKPw6t1HFGW6qAlTaaQ5bGMr2QUjwfV3uBfReBIjah gFVEEMBMg5KxMhgvpOVR9hXJ5LkWeezzvD6WDQU7/TyKmfLumYioClDIGb0fvYMswxP5lO6vHQQTZ /KAbsB2xvqjlG105PA0HY8CpDRuigNKCmBhYciobgdr6iiN0WXn24M5SyY4UgtLTLEc9TRyOXNKuK DrcrFF/vvtfPNCnuftwC50SLllGyek1G7lSmgNgv+MIzo8xuH9WlQ46EPOIKzcDyM9Kosx0Ei1mbF 8/CWOzJWmjFvPQ==; Received: from [2001:660:6102:320:e120:2c8f:8909:cdfe] (port=52502 helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1nw4RD-00067m-0X; Tue, 31 May 2022 12:09:43 -0400 From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= To: 55499@debbugs.gnu.org Subject: [PATCH 1/3] tests: Augment profile collision test. Date: Tue, 31 May 2022 18:09:14 +0200 Message-Id: <20220531160916.21508-2-ludo@gnu.org> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20220531160916.21508-1-ludo@gnu.org> References: <87sfp7kkim.fsf@elephly.net> <20220531160916.21508-1-ludo@gnu.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 55499 Cc: =?UTF-8?q?Ludovic=20Court=C3=A8s?= X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) * tests/profiles.scm ("collision of propagated inputs"): Check the parents of ENTRY1 and ENTRY2. --- tests/profiles.scm | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/tests/profiles.scm b/tests/profiles.scm index d59d75985f..7e51d37ab9 100644 --- a/tests/profiles.scm +++ b/tests/profiles.scm @@ -556,14 +556,20 @@ (define (entry->sexp entry) (return #f))))) (test-equal "collision of propagated inputs" - '(("guile-bootstrap" "2.0") ("guile-bootstrap" "42")) + '(("guile-bootstrap" "2.0") "p1" + <> ("guile-bootstrap" "42") "p2") (guard (c ((profile-collision-error? c) (let ((entry1 (profile-collision-error-entry c)) (entry2 (profile-collision-error-conflict c))) (list (list (manifest-entry-name entry1) (manifest-entry-version entry1)) + (manifest-entry-name + (force (manifest-entry-parent entry1))) + '<> (list (manifest-entry-name entry2) - (manifest-entry-version entry2)))))) + (manifest-entry-version entry2)) + (manifest-entry-name + (force (manifest-entry-parent entry2))))))) (run-with-store %store (mlet* %store-monad ((p0 -> (package (inherit %bootstrap-guile) -- 2.36.1 From debbugs-submit-bounces@debbugs.gnu.org Tue May 31 12:09:58 2022 Received: (at 55499) by debbugs.gnu.org; 31 May 2022 16:09:58 +0000 Received: from localhost ([127.0.0.1]:48733 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nw4RS-0004O7-29 for submit@debbugs.gnu.org; Tue, 31 May 2022 12:09:58 -0400 Received: from eggs.gnu.org ([209.51.188.92]:49682) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nw4RK-0004NC-LN for 55499@debbugs.gnu.org; Tue, 31 May 2022 12:09:53 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:41052) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nw4RF-0002NY-FF; Tue, 31 May 2022 12:09:45 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:References:In-Reply-To:Date:Subject:To: From; bh=zHG6lv3j3qj9u1jTW+jp2e4+mbnFpOAS0kjcgD0CEGQ=; b=UaVcu5gKsIltBrA/7R8n e8h5mpja5285xNhDnXfZV1SCL2/RSsvi86QsyHrhiKdMGXDmBR2jUzDCHTXQNFmcRVYJVQSemRqlI GyhW9VXTASgv8FzSarP9aFP7myRWZRYhxkml0e93B0BPvLAaKFNY1dpJSBBuxj/YijodTmxWoye5M DQWBNlh3I3N9NCIZ2Bon2liuzAVgIkkP18/rkdfD7ILi8YabbFpZ3GLNU/J4zO6w6aMo+R/e+ha4E KtackxtJoPqDxuhHohndHJ0Ik7gc1c2zXFeMLDDojPB6B2cIBbxmxZNedwdDSS81+CZqArBWUTe5a wxbNsYw9OOTufg==; Received: from [2001:660:6102:320:e120:2c8f:8909:cdfe] (port=52502 helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1nw4RE-00067m-UZ; Tue, 31 May 2022 12:09:45 -0400 From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= To: 55499@debbugs.gnu.org Subject: [PATCH 3/3] squash! profiles: Make all entry fields optional. Date: Tue, 31 May 2022 18:09:16 +0200 Message-Id: <20220531160916.21508-4-ludo@gnu.org> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20220531160916.21508-1-ludo@gnu.org> References: <87sfp7kkim.fsf@elephly.net> <20220531160916.21508-1-ludo@gnu.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 55499 Cc: =?UTF-8?q?Ludovic=20Court=C3=A8s?= X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) This is meant to be squashed with the previous patch. This makes the 'search-paths' and 'propagated-inputs' fields of each entry optional, shaving a bit more space and reading time, down to 180K instead of 192K. * guix/build/profiles.scm (manifest-sexp->inputs+search-paths)[let-fields]: New macro. Use it. * guix/profiles.scm (manifest->gexp)[optional]: New procedure. Use it. [sexp->manifest-entry]: Rename to... [sexp->manifest-entry/v3]: ... this. --- guix/build/profiles.scm | 28 ++++++++-- guix/profiles.scm | 120 ++++++++++++++++++++++++++-------------- 2 files changed, 100 insertions(+), 48 deletions(-) diff --git a/guix/build/profiles.scm b/guix/build/profiles.scm index c4460f624b..2ab76bde74 100644 --- a/guix/build/profiles.scm +++ b/guix/build/profiles.scm @@ -149,6 +149,18 @@ (define (manifest-sexp->inputs+search-paths manifest) "Parse MANIFEST, an sexp as produced by 'manifest->gexp', and return two values: the list of store items of its manifest entries, and the list of search path specifications." + (define-syntax let-fields + (syntax-rules () + ;; Bind the fields NAME of LST to same-named variables in the lexical + ;; scope of BODY. + ((_ lst (name rest ...) body ...) + (let ((name (match (assq 'name lst) + ((_ value) value) + (#f '())))) + (let-fields lst (rest ...) body ...))) + ((_ lst () body ...) + (begin body ...)))) + (match manifest ;this must match 'manifest->gexp' (('manifest ('version 4) ('packages (entries ...))) @@ -156,12 +168,12 @@ (define (manifest-sexp->inputs+search-paths manifest) (inputs '()) (search-paths '())) (match entries - (((name version output item - ('propagated-inputs deps) - ('search-paths paths) _ ...) . rest) - (loop (append rest deps) ;breadth-first traversal - (cons item inputs) - (append paths search-paths))) + (((name version output item fields ...) . rest) + (let ((paths search-paths)) + (let-fields fields (propagated-inputs search-paths properties) + (loop (append rest propagated-inputs) ;breadth-first traversal + (cons item inputs) + (append search-paths paths))))) ((('repeated name version item) . rest) (loop rest inputs search-paths)) (() @@ -214,4 +226,8 @@ (define manifest-file ;; Write 'OUTPUT/etc/profile'. (build-etc/profile output search-paths))) +;;; Local Variables: +;;; eval: (put 'let-fields 'scheme-indent-function 2) +;;; End: + ;;; profile.scm ends here diff --git a/guix/profiles.scm b/guix/profiles.scm index 44ff37e75b..d694ac07da 100644 --- a/guix/profiles.scm +++ b/guix/profiles.scm @@ -454,6 +454,11 @@ (define (inferior->entry) (define (manifest->gexp manifest) "Return a representation of MANIFEST as a gexp." + (define (optional name value) + (if (null? value) + #~() + #~((#$name #$value)))) + (define (entry->gexp entry) ;; Maintain in state monad a vhash of visited entries, indexed by their ;; item, usually package objects (we cannot use the entry itself as an @@ -477,24 +482,22 @@ (define (entry->gexp entry) (return (match entry (($ name version output (? string? path) - (_deps ...) (search-paths ...) _ (properties ...)) + (_ ...) (search-paths ...) _ (properties ...)) #~(#$name #$version #$output #$path - (propagated-inputs #$deps) - (search-paths #$(map search-path-specification->sexp - search-paths)) - #$@(if (null? properties) - #~() - #~((properties . #$properties))))) + #$@(optional 'propagated-inputs deps) + #$@(optional 'search-paths + (map search-path-specification->sexp + search-paths)) + #$@(optional 'properties properties))) (($ name version output package (_deps ...) (search-paths ...) _ (properties ...)) #~(#$name #$version #$output (ungexp package (or output "out")) - (propagated-inputs #$deps) - (search-paths #$(map search-path-specification->sexp - search-paths)) - #$@(if (null? properties) - #~() - #~((properties . #$properties)))))))))))) + #$@(optional 'propagated-inputs deps) + #$@(optional 'search-paths + (map search-path-specification->sexp + search-paths)) + #$@(optional 'properties properties)))))))))) (match manifest (($ (entries ...)) @@ -542,6 +545,40 @@ (define (infer-dependency item parent) (item item) (parent parent)))) + (define* (sexp->manifest-entry/v3 sexp #:optional (parent (delay #f))) + (match sexp + ((name version output path + ('propagated-inputs deps) + ('search-paths search-paths) + extra-stuff ...) + ;; For each of DEPS, keep a promise pointing to ENTRY. + (letrec* ((deps* (map (cut sexp->manifest-entry/v3 <> (delay entry)) + deps)) + (entry (manifest-entry + (name name) + (version version) + (output output) + (item path) + (dependencies deps*) + (search-paths (map sexp->search-path-specification + search-paths)) + (parent parent) + (properties (or (assoc-ref extra-stuff 'properties) + '()))))) + entry)))) + + (define-syntax let-fields + (syntax-rules () + ;; Bind the fields NAME of LST to same-named variables in the lexical + ;; scope of BODY. + ((_ lst (name rest ...) body ...) + (let ((name (match (assq 'name lst) + ((_ value) value) + (#f '())))) + (let-fields lst (rest ...) body ...))) + ((_ lst () body ...) + (begin body ...)))) + (define* (sexp->manifest-entry sexp #:optional (parent (delay #f))) (match sexp (('repeated name version path) @@ -556,32 +593,29 @@ (define* (sexp->manifest-entry sexp #:optional (parent (delay #f))) sexp))) ((_ . entry) (return entry))))) - ((name version output path - ('propagated-inputs deps) - ('search-paths search-paths) - extra-stuff ...) - (mlet* %state-monad - ((entry -> #f) - (deps* (mapm %state-monad - (cut sexp->manifest-entry <> (delay entry)) - deps)) - (visited (current-state)) - (key -> (list name version path))) - (set! entry ;XXX: emulate 'letrec*' - (manifest-entry - (name name) - (version version) - (output output) - (item path) - (dependencies deps*) - (search-paths (map sexp->search-path-specification - search-paths)) - (parent parent) - (properties (or (assoc-ref extra-stuff 'properties) - '())))) - (mbegin %state-monad - (set-current-state (vhash-cons key entry visited)) - (return entry)))))) + ((name version output path fields ...) + (let-fields fields (propagated-inputs search-paths properties) + (mlet* %state-monad + ((entry -> #f) + (deps (mapm %state-monad + (cut sexp->manifest-entry <> (delay entry)) + propagated-inputs)) + (visited (current-state)) + (key -> (list name version path))) + (set! entry ;XXX: emulate 'letrec*' + (manifest-entry + (name name) + (version version) + (output output) + (item path) + (dependencies deps) + (search-paths (map sexp->search-path-specification + search-paths)) + (parent parent) + (properties properties))) + (mbegin %state-monad + (set-current-state (vhash-cons key entry visited)) + (return entry))))))) (match sexp (('manifest ('version 0) @@ -649,9 +683,7 @@ (define* (sexp->manifest-entry sexp #:optional (parent (delay #f))) ;; Version 3 represents DEPS as full-blown manifest entries. (('manifest ('version 3 minor-version ...) ('packages (entries ...))) - (manifest (run-with-state - (mapm %state-monad sexp->manifest-entry entries) - vlist-null))) + (manifest (map sexp->manifest-entry/v3 entries))) ;; Version 4 deduplicates repeated entries, as can happen with deep ;; propagated input trees. @@ -2368,4 +2400,8 @@ (define (user-friendly-profile profile) %known-shorthand-profiles) profile)) +;;; Local Variables: +;;; eval: (put 'let-fields 'scheme-indent-function 2) +;;; End: + ;;; profiles.scm ends here -- 2.36.1 From debbugs-submit-bounces@debbugs.gnu.org Tue May 31 13:36:02 2022 Received: (at 55499) by debbugs.gnu.org; 31 May 2022 17:36:02 +0000 Received: from localhost ([127.0.0.1]:48810 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nw5mk-0006oU-5T for submit@debbugs.gnu.org; Tue, 31 May 2022 13:36:02 -0400 Received: from baptiste.telenet-ops.be ([195.130.132.51]:42812) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nw5mg-0006oB-3x for 55499@debbugs.gnu.org; Tue, 31 May 2022 13:36:01 -0400 Received: from [172.20.10.5] ([188.188.13.110]) by baptiste.telenet-ops.be with bizsmtp id dVbv2700C2NU7E401Vbwki; Tue, 31 May 2022 19:35:56 +0200 Message-ID: <10a79d0be70467d59fffd225b18f0bbd17ccdba3.camel@telenet.be> Subject: Re: bug#55499: [PATCH 2/3] profiles: Do not repeat entries in 'manifest' file. From: Maxime Devos To: Ludovic =?ISO-8859-1?Q?Court=E8s?= , 55499@debbugs.gnu.org Date: Tue, 31 May 2022 19:35:50 +0200 In-Reply-To: <20220531160916.21508-3-ludo@gnu.org> References: <87sfp7kkim.fsf@elephly.net> <20220531160916.21508-1-ludo@gnu.org> <20220531160916.21508-3-ludo@gnu.org> Content-Type: multipart/signed; micalg="pgp-sha512"; protocol="application/pgp-signature"; boundary="=-hv+1HRlus1vfA5LAtgjs" User-Agent: Evolution 3.38.3-1 MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=telenet.be; s=r22; t=1654018556; bh=fwIpiDaAZ1YH3dsF1wNVAfx+Tnji3ChRoPTz4DKhsws=; h=Subject:From:To:Date:In-Reply-To:References; b=h6z9BWRq2HJN233a+8kD0BvY4n6zXADx8twZLA2urwv28RBq3A5dcrTd9wLe6EUHe +U2H/XQ0m5MTS3rsXLOwfRl2vfMgJpLGyD3xk0tc8Qi0LG+5LmzR25iKOIz+Pi3B0w ELodaR/IFBLubZEB+9+tymDT7NDrnCtF4jF6o3PlX5YbA+z6Wn66wYr8R44gyBRLxC ZEndJR2QCWh7TwwzE/OG7Qf3pYt1u27GyB4WqBpyn+iKKAedIXmtOxcWI7h7WjRQp1 qtcFYRVO+OrS2zoHvvF3BL1OADw3BJY2JkGAi3vdMqoWQVl0qBrlLn2D+sDgE9aJAu q8fgYtA4s9nbg== X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 55499 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) --=-hv+1HRlus1vfA5LAtgjs Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Ludovic Court=C3=A8s schreef op di 31-05-2022 om 18:09 [+0200]: > With this change, the manifest file created for: >=20 > =C2=A0 guix install r r-seurat r-cistopic r-monocle3 r-cicero-monocle3 r-= assertthat >=20 > goes from 5.6M to 192K.=C2=A0 Likewise, on this profile, wall-clock time = of: >=20 > =C2=A0 GUIX_PROFILING=3Dgc guix package -I >=20 > goes from 0.7s to 0.1s, with heap usage going from 55M to 9M. I'm not familiar enough with this part of Guix to evaluate the patches, but the time, disk memory and heap memory decreases sound great! Greetings, Maxime. --=-hv+1HRlus1vfA5LAtgjs Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYpZR9hccbWF4aW1lZGV2 b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7gAyAP9m0h53t04kE/RAWjFgbx5e1rKs mOeRM17mlIlgkSgJewD/QLShv6NeQQpbUu9TAKW8FDtq8p/f2CBAtoRIldkwOA0= =O9+S -----END PGP SIGNATURE----- --=-hv+1HRlus1vfA5LAtgjs-- From debbugs-submit-bounces@debbugs.gnu.org Wed Jun 01 05:39:09 2022 Received: (at 55499) by debbugs.gnu.org; 1 Jun 2022 09:39:09 +0000 Received: from localhost ([127.0.0.1]:49751 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nwKon-0006L0-K9 for submit@debbugs.gnu.org; Wed, 01 Jun 2022 05:39:09 -0400 Received: from eggs.gnu.org ([209.51.188.92]:45102) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nwKoi-0006KP-Iv for 55499@debbugs.gnu.org; Wed, 01 Jun 2022 05:39:07 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:58746) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nwKob-0000zm-Qp; Wed, 01 Jun 2022 05:38:58 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:In-Reply-To:Date:References:Subject:To: From; bh=NZuJ008OqUDMWavsG0QgkjzCxpdT8/63bRqMAdnjyWc=; b=ns5AHlUSfXcIl1VdNSy1 967vxfCJeQjfZuGKAyqbwSmOiaRO/qTR/t/oyQ+mxdWDlPrxuedwORuo8KQ5JBipvZRIsdZ/WVOGR S0CsI87slKvdsyq8IqY6h8rUKNZyFZLHzK6F68bawsLFQrbzGfMPHXJTSRiCKY81YC5kqFfqu/FcH vpFjcaD8cbTd420RbO2vQSeUf02yxklbdaV3eDlE5/+PH35B/Xt+R5CIJikbRaRA8J51cs3iqjVoI cMrWlPrOJgLyUMKhhcKj+V5LAfgBAFPRecfwpqsxVXbSuILZi3/fTnzxUJD9Rz7a9VCC6hjLrd0Lm Fh5jkPN9QPXYQA==; Received: from 91-160-117-201.subs.proxad.net ([91.160.117.201]:61113 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nwKob-0005az-Dp; Wed, 01 Jun 2022 05:38:57 -0400 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Maxime Devos Subject: Re: bug#55499: [PATCH 2/3] profiles: Do not repeat entries in 'manifest' file. References: <87sfp7kkim.fsf@elephly.net> <20220531160916.21508-1-ludo@gnu.org> <20220531160916.21508-3-ludo@gnu.org> <10a79d0be70467d59fffd225b18f0bbd17ccdba3.camel@telenet.be> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: Tridi 13 Prairial an 230 de la =?utf-8?Q?R=C3=A9volu?= =?utf-8?Q?tion=2C?= jour du Pois X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Wed, 01 Jun 2022 11:38:54 +0200 In-Reply-To: <10a79d0be70467d59fffd225b18f0bbd17ccdba3.camel@telenet.be> (Maxime Devos's message of "Tue, 31 May 2022 19:35:50 +0200") Message-ID: <871qw8yb81.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 55499 Cc: 55499@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hi, Maxime Devos skribis: > Ludovic Court=C3=A8s schreef op di 31-05-2022 om 18:09 [+0200]: >> With this change, the manifest file created for: >>=20 >> =C2=A0 guix install r r-seurat r-cistopic r-monocle3 r-cicero-monocle3 r= -assertthat >>=20 >> goes from 5.6M to 192K.=C2=A0 Likewise, on this profile, wall-clock time= of: >>=20 >> =C2=A0 GUIX_PROFILING=3Dgc guix package -I >>=20 >> goes from 0.7s to 0.1s, with heap usage going from 55M to 9M. > > I'm not familiar enough with this part of Guix to evaluate the patches, > but the time, disk memory and heap memory decreases sound great! Yup! The difference is significant primarily for profiles with lots of propagated inputs, so typically profiles with R or Python packages. For my home profile (300+ packages but no R and no Python), it goes from 316K to 112K, heap usage for =E2=80=98guix package -I=E2=80=99 goes from 12= M to 9M and wall-clock time is almost unchanged. Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Tue Jun 14 03:07:01 2022 Received: (at 55499) by debbugs.gnu.org; 14 Jun 2022 07:07:01 +0000 Received: from localhost ([127.0.0.1]:32774 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1o10dh-0007Jh-0b for submit@debbugs.gnu.org; Tue, 14 Jun 2022 03:07:01 -0400 Received: from eggs.gnu.org ([209.51.188.92]:58058) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1o10df-0007JU-4D for 55499@debbugs.gnu.org; Tue, 14 Jun 2022 03:06:59 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:58482) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1o10dY-00005z-WD; Tue, 14 Jun 2022 03:06:53 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:In-Reply-To:Date:References:Subject:To: From; bh=jDpfROzEYheJaOVGi0nyi3px9d3DsBuYS4KymzQcAM0=; b=orNusn2IWrRS1X3+IRsr hdkcKHDjRT6690O1xGi0u//mKSS1F5wZ/sC6uMW2hiGuBn5QQwZwTo170+htvLzaYxtt8+6TyBde0 9c0GpfCvUvtKoZ7IxCx2ynOlDgNC8jmmEbc/MSs/YDmmlrkUj1damqtcErx5K7PJQv9fjbANUzb8K sEfzN9FF4QUx+hvF0jDohQS+Q9HaAG7u6mZiySoe1uLjcgNoYXwfDqrlqceKdF8Gxr3D3qQUtsYZA MGuRb+aqIRW8QAiqb54MG/bXmISpqRTifBva8vLBLyEwmoo7dQjDcIENvmmfD79pbGmJduWrCFr+w crZbRDyfoRSmyA==; Received: from [2001:660:6102:320:e120:2c8f:8909:cdfe] (port=51876 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1o10dX-0000TY-09; Tue, 14 Jun 2022 03:06:52 -0400 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: 55499@debbugs.gnu.org Subject: Re: bug#55499: excessively large manifests due to propagation References: <87sfp7kkim.fsf@elephly.net> <20220531160916.21508-1-ludo@gnu.org> Date: Tue, 14 Jun 2022 09:06:48 +0200 In-Reply-To: <20220531160916.21508-1-ludo@gnu.org> ("Ludovic =?utf-8?Q?Cou?= =?utf-8?Q?rt=C3=A8s=22's?= message of "Tue, 31 May 2022 18:09:13 +0200") Message-ID: <87a6afoh9z.fsf_-_@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 55499 Cc: Ricardo Wurmus X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hey Ricardo, Ludovic Court=C3=A8s skribis: > These patches implement what you suggested on IRC: not repeating > entire manifest entries and their propagated inputs. This has a > dramatic impact on the size of the =E2=80=98manifest=E2=80=99 file and on= the memory > and processing time to read it for the the use case you gave. > > The second patch goes a tiny bit further by making the =E2=80=98search-pa= ths=E2=80=99 > and =E2=80=98propagated-inputs=E2=80=99 fields optional, shaving another = ~10% on the > size of =E2=80=98manifest=E2=80=99 in this example. > > The second patch should be squashed with the first one (so we don=E2=80= =99t > bump version formats a second time and duplicate code). It=E2=80=99s kin= da > optional because it doesn=E2=80=99t bring much compared to the first patc= h and > causes a bit of extra complexity, but maybe it=E2=80=99s still worth keep= ing? > > Could you try this on your larger use cases and tell me how it goes? Did you have a chance to give it a try? Maybe I can double-check that everything=E2=80=99s alright and go ahead. Thanks, Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Fri Jul 01 17:54:52 2022 Received: (at 55499-done) by debbugs.gnu.org; 1 Jul 2022 21:54:52 +0000 Received: from localhost ([127.0.0.1]:39260 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1o7ObE-0008FS-CU for submit@debbugs.gnu.org; Fri, 01 Jul 2022 17:54:52 -0400 Received: from eggs.gnu.org ([209.51.188.92]:41272) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1o7ObA-0008F8-M3 for 55499-done@debbugs.gnu.org; Fri, 01 Jul 2022 17:54:51 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:40730) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1o7Ob5-0002Ce-8A; Fri, 01 Jul 2022 17:54:43 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:In-Reply-To:Date:References:Subject:To: From; bh=+Ya8mNl/iW1A+Ohmqk4BwiKcDG/7Qw+WySdGnyIx0Kg=; b=eH9gRW5wZ1IIe9qn+pxG 04TiVUOkNRdJnnEzUaRyE0N8HnqbXaeBdPk8VwUH/A5wjYDEl0/iLX1fd3N7v5kffIHUh18FFfRH9 q2vhWf53vJyNBSP7SkW0Tfix8IXtiRkpqsER8gTxuvTIoxB54y1sA0LdUdAX2/4ReI2izoorSkydJ oGZc5XbYwBaytVZqwvWiZQEqgJ12t3JNziRysCAIXlexORSpcjKxMwatRRnwkpoxCmJukXB0efLhI adxMECu7AvPgNd9/+geiaOWgoVF5VntHu9oYg/n4jJn1hVNiVlqCtsF5EpclZ7gBQhttDuEzJEE/8 TJJSEgJMbShfiQ==; Received: from 91-160-117-201.subs.proxad.net ([91.160.117.201]:60994 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1o7Ob4-0005tZ-Rk; Fri, 01 Jul 2022 17:54:43 -0400 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: 55499-done@debbugs.gnu.org Subject: Re: bug#55499: excessively large manifests due to propagation References: <87sfp7kkim.fsf@elephly.net> <20220531160916.21508-1-ludo@gnu.org> <87a6afoh9z.fsf_-_@gnu.org> Date: Fri, 01 Jul 2022 23:54:41 +0200 In-Reply-To: <87a6afoh9z.fsf_-_@gnu.org> ("Ludovic =?utf-8?Q?Court=C3=A8s?= =?utf-8?Q?=22's?= message of "Tue, 14 Jun 2022 09:06:48 +0200") Message-ID: <878rpc33dq.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 55499-done Cc: Ricardo Wurmus X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hi! I went ahead and pushed it as 4ff12d1de7cd617b791996ee7ca1240660b4c20e. Ludo=E2=80=99. From unknown Sat Jun 21 10:44:17 2025 Received: (at fakecontrol) by fakecontrolmessage; To: internal_control@debbugs.gnu.org From: Debbugs Internal Request Subject: Internal Control Message-Id: bug archived. Date: Sat, 30 Jul 2022 11:24:09 +0000 User-Agent: Fakemail v42.6.9 # This is a fake control message. # # The action: # bug archived. thanks # This fakemail brought to you by your local debbugs # administrator