From debbugs-submit-bounces@debbugs.gnu.org Sun Nov 06 08:55:45 2022 Received: (at submit) by debbugs.gnu.org; 6 Nov 2022 13:55:46 +0000 Received: from localhost ([127.0.0.1]:58966 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1org7l-0004kr-3O for submit@debbugs.gnu.org; Sun, 06 Nov 2022 08:55:45 -0500 Received: from lists.gnu.org ([209.51.188.17]:46312) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1org7i-0004kh-0G for submit@debbugs.gnu.org; Sun, 06 Nov 2022 08:55:42 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1org7h-000693-Qk for guix-patches@gnu.org; Sun, 06 Nov 2022 08:55:41 -0500 Received: from mira.cbaines.net ([212.71.252.8]) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1org7f-0003fv-BX for guix-patches@gnu.org; Sun, 06 Nov 2022 08:55:41 -0500 Received: from localhost (host-82-184-251-30.business.telecomitalia.it [82.184.251.30]) by mira.cbaines.net (Postfix) with ESMTPSA id EDA3827BBE9 for ; Sun, 6 Nov 2022 13:55:35 +0000 (GMT) Received: from localhost (localhost [local]) by localhost (OpenSMTPD) with ESMTPA id 61b65099 for ; Sun, 6 Nov 2022 13:55:32 +0000 (UTC) From: Christopher Baines To: guix-patches@gnu.org Subject: [PATCH] lint: Split the derivation lint checker by system. Date: Sun, 6 Nov 2022 14:55:32 +0100 Message-Id: <20221106135532.5724-1-mail@cbaines.net> X-Mailer: git-send-email 2.37.3 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Received-SPF: pass client-ip=212.71.252.8; envelope-from=mail@cbaines.net; helo=mira.cbaines.net X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, UNPARSEABLE_RELAY=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-Spam-Score: -1.4 (-) X-Debbugs-Envelope-To: submit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.4 (--) Currently, if you attempt to run the derivation checker on all packages, the Guile process will run out of memory. I think a contributing factor to this is that the checker forces an inefficient order when you want to generate derivations for all the supported systems of each package, constantly switching system then package. This problem also impacts the Guix Data Service, since it tries to run the derivation checker for all packages. The changes in this commit to split the derivation lint checker in to several, one for each system, means that you can now treat each system separately, which should be better for caching purposes. If it's desirable to keep some notion of checking all supported systems for a single package, I think lint checker groups could be added, so that you could ask for the "derivation" checker, and this would run all the derivation checkers. * guix/lint.scm (check-derivation): Adapt to make-check-derivation-for-system. (%derivation-checkers): New variable. (%local-checkers): Include all %derivation-checkers. * doc/guix.texi (Invoking guix lint): Update. --- doc/guix.texi | 4 +- guix/lint.scm | 139 +++++++++++++++++++++++++++++--------------------- 2 files changed, 83 insertions(+), 60 deletions(-) diff --git a/doc/guix.texi b/doc/guix.texi index 7806b21a0f..8d4989a60c 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -14422,9 +14422,9 @@ Parse the @code{source} URL to determine if a tarball from GitHub is autogenerated or if it is a release tarball. Unfortunately GitHub's autogenerated tarballs are sometimes regenerated. -@item derivation +@item derivation/SYSTEM Check that the derivation of the given packages can be successfully -computed for all the supported systems (@pxref{Derivations}). +computed for the specified system (@pxref{Derivations}). @item profile-collisions Check whether installing the given packages in a profile would lead to diff --git a/guix/lint.scm b/guix/lint.scm index 8e3976171f..f692856f42 100644 --- a/guix/lint.scm +++ b/guix/lint.scm @@ -52,6 +52,7 @@ (define-module (guix lint) #:use-module (guix memoization) #:use-module (guix profiles) #:use-module (guix monads) + #:use-module (guix platform) #:use-module (guix scripts) #:use-module ((guix ui) #:select (texi->plain-text fill-paragraph)) #:use-module (guix gnu-maintenance) @@ -98,7 +99,6 @@ (define-module (guix lint) check-patch-file-names check-patch-headers check-synopsis-style - check-derivation check-home-page check-name check-source @@ -116,6 +116,8 @@ (define-module (guix lint) check-haskell-stackage check-tests-true + make-check-derivation-for-system + lint-warning lint-warning? lint-warning-package @@ -1369,56 +1371,6 @@ (define (check-phases-deltas deltas) (append-map check-phases-delta deltas)) (find-phase-deltas package check-phases-deltas)) -(define* (check-derivation package #:key store) - "Emit a warning if we fail to compile PACKAGE to a derivation." - (define (try store system) - (guard (c ((store-protocol-error? c) - (make-warning package - (G_ "failed to create ~a derivation: ~a") - (list system - (store-protocol-error-message c)))) - ((exception-with-kind-and-args? c) - (make-warning package - (G_ "failed to create ~a derivation: ~s") - (list system - (cons (exception-kind c) - (exception-args c))))) - ((message-condition? c) - (make-warning package - (G_ "failed to create ~a derivation: ~a") - (list system - (condition-message c)))) - ((formatted-message? c) - (let ((str (apply format #f - (formatted-message-string c) - (formatted-message-arguments c)))) - (make-warning package - (G_ "failed to create ~a derivation: ~a") - (list system str)))) - (else - (make-warning package - (G_ "failed to create ~a derivation: ~a") - (list system c)))) - (parameterize ((%graft? #f)) - (package-derivation store package system #:graft? #f) - - ;; If there's a replacement, make sure we can compute its - ;; derivation. - (match (package-replacement package) - (#f #t) - (replacement - (package-derivation store replacement system - #:graft? #f)))))) - - (define (check-with-store store) - (filter lint-warning? - (map (cut try store <>) (package-supported-systems package)))) - - ;; For backwards compatability, don't rely on store being set - (or (and=> store check-with-store) - (with-store store - (check-with-store store)))) - (define* (check-profile-collisions package #:key store) "Check for collisions that would occur when installing PACKAGE as a result of the propagated inputs it pulls in." @@ -1843,13 +1795,88 @@ (define (check-formatting package) (G_ "source file not found")))))))) '()))) +(define (make-check-derivation-for-system system) + (define (try package proc) + (guard (c ((store-protocol-error? c) + (make-warning package + (G_ "failed to create ~a derivation: ~a") + (list system + (store-protocol-error-message c)))) + ((exception-with-kind-and-args? c) + (make-warning package + (G_ "failed to create ~a derivation: ~s") + (list system + (cons (exception-kind c) + (exception-args c))))) + ((message-condition? c) + (make-warning package + (G_ "failed to create ~a derivation: ~a") + (list system + (condition-message c)))) + ((formatted-message? c) + (let ((str (apply format #f + (formatted-message-string c) + (formatted-message-arguments c)))) + (make-warning package + (G_ "failed to create ~a derivation: ~a") + (list system str)))) + (else + (make-warning package + (G_ "failed to create ~a derivation: ~a") + (list system c)))) + (proc))) + + + + (lambda* (package #:key store) + "Emit a warning if we fail to compile PACKAGE to a derivation." + + (define (check-with-store store) + (if (member system (package-supported-systems package)) + (filter + lint-warning? + (map (cut try package <>) + (list + (lambda () + (parameterize ((%graft? #f)) + (package-derivation store package system #:graft? #f))) + (lambda () + ;; If there's a replacement, make sure we can compute its + ;; derivation. + (match (package-replacement package) + (#f #t) + (replacement + (parameterize ((%graft? #f)) + (package-derivation store replacement system + #:graft? #f)))))))) + '())) + + ;; For backwards compatability, don't rely on store being set + (or (and=> store check-with-store) + (with-store store + (check-with-store store))))) + ;;; ;;; List of checkers. ;;; +(define %derivation-checkers + (map (lambda (system) + (lint-checker + (name (string->symbol + (simple-format #f "derivation/~A" system))) + (description + (simple-format + #f + "Report failure to compile a package to a derivation for ~A" + system)) + (check (make-check-derivation-for-system system)) + (requires-store? #t))) + (systems))) + (define %local-checkers - (list + (cons* (lint-checker (name 'name) (description "Validate package names") @@ -1901,11 +1928,6 @@ (define %local-checkers (name 'source-unstable-tarball) (description "Check for autogenerated tarballs") (check check-source-unstable-tarball)) - (lint-checker - (name 'derivation) - (description "Report failure to compile a package to a derivation") - (check check-derivation) - (requires-store? #t)) (lint-checker (name 'profile-collisions) (description "Report collisions that would occur due to propagated inputs") @@ -1922,7 +1944,8 @@ (define %local-checkers (lint-checker (name 'formatting) (description "Look for formatting issues in the source") - (check check-formatting)))) + (check check-formatting)) + %derivation-checkers)) (define %network-dependent-checkers (list -- 2.37.3 From debbugs-submit-bounces@debbugs.gnu.org Mon Nov 07 12:46:59 2022 Received: (at 59078) by debbugs.gnu.org; 7 Nov 2022 17:46:59 +0000 Received: from localhost ([127.0.0.1]:35274 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1os6D5-0002w6-Du for submit@debbugs.gnu.org; Mon, 07 Nov 2022 12:46:59 -0500 Received: from mira.cbaines.net ([212.71.252.8]:41800) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1os6D3-0002vx-E1 for 59078@debbugs.gnu.org; Mon, 07 Nov 2022 12:46:57 -0500 Received: from localhost (93-42-6-17.ip84.fastwebnet.it [93.42.6.17]) by mira.cbaines.net (Postfix) with ESMTPSA id D3C1027BBE9; Mon, 7 Nov 2022 17:46:55 +0000 (GMT) Received: from felis (localhost [127.0.0.1]) by localhost (OpenSMTPD) with ESMTP id a456c9c1; Mon, 7 Nov 2022 17:46:54 +0000 (UTC) References: <20221106135532.5724-1-mail@cbaines.net> User-agent: mu4e 1.8.9; emacs 28.1 From: Christopher Baines To: Christopher Baines Subject: Re: [bug#59078] [PATCH] lint: Split the derivation lint checker by system. Date: Mon, 07 Nov 2022 18:37:32 +0100 In-reply-to: <20221106135532.5724-1-mail@cbaines.net> Message-ID: <87cz9ypshf.fsf@cbaines.net> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 59078 Cc: 59078@debbugs.gnu.org, guix-patches@gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) --=-=-= Content-Type: text/plain Christopher Baines writes: > This problem also impacts the Guix Data Service, since it tries to run the > derivation checker for all packages. This patch has now been processed by qa.guix.gnu.org. Looking at the logs for the Guix Data Service processing the base and target revision, and the change is more significant than I'd imagined: Base: inferior heap after cleanup: 1739.0 MiB used (5160.0 MiB heap) debug: Finished getting formatting lint warnings, took 349 seconds debug: Finished fetching inferior lint warnings, took 3782 seconds Target: inferior heap after cleanup: 1152.0 MiB used (1778.0 MiB heap) debug: Finished getting derivation/aarch64-linux lint warnings, took 334 seconds debug: Finished fetching inferior lint warnings, took 3285 seconds So with the changes, it's a little faster, but the main difference is that the heap ~3GiB smaller, so ~34% of what it was previously. I did notice that this also subtly differs from how the linter behaved previously, since some packages define support for systems not defined through the platform module. https://data.qa.guix.gnu.org/compare?base_commit=a60dc46c2bb5de196858594b72b00d5f86ca7e98&target_commit=4e152714f55337015991e62e51e8dea15e889b9f Personally, I think this change is still a good one. Maybe we can add a separate linter to go round and check that packages don't declare support for systems that aren't in the platform module. Unless anyone objects, I'll like to push this sooner rather than later, as I think the excessive heap size in the inferior process is not ideal. Thanks, Chris --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmNpRIxfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh aW5lcy5uZXQACgkQXiijOwuE9XflUg//bzvSQ+wbaAL3FO9EQz6zGmVeDOgqbkEe rX+iCqLk0ZvYAzTMUYmz+z55ozfJChkhB5TwA71s68hOKZQFEvkI6lwUylaryy4a ovkttrcn6hEtr5E0KglToltSj5F8IHlQ3BT4uc0re2/zWIfrsFUEbjI2HLH99xyA CKxH1ENy0GboVZBH8SHYRLTTWefMRkMUCq6c2hWpZeOSYHuqf4Gk6KXc6oEln6js FBkFeH9N1V2607/KakrCY851zevbjXc5LD+0OXXN6ALR9qXzrb+gp00U/5gJ54zd pNlzZSanRC8R+LM3bAKOQxe1womVv3/QIczJCTclXL7NTVT/5a3WYlrbIldoPMhN ET2XVjRzHz4WheprtTzkm/F2JdcbFPNsra9K+uDzRHX2AH98JzklDhq58AzsxF+s hXge0BCv3rsJpjgQHtp8sW26VdyBssuZYMqEE9XbX+Jzvr++DtN4rgDu2nXWJOX/ RMns3KiK6U5yP3Gq6dI8YCtdxreHPbAPjfH829H5/gPCqE4wDcyinb1YCCAwsy7K 4ELkFjg/L7CEFfiuB5N+wuGq2bpxF2YNwWMs5QMq/tlGGSSXkGVWDwsaxKS+Y0XI GBzO77Hsjxm1fNpuBowP0ZaQKYnBtEwl5gbUYkeAnIDBqgdZECcGPTGOEh/jEQFN KQljNj7kxtE= =ALR6 -----END PGP SIGNATURE----- --=-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Fri Nov 11 16:57:48 2022 Received: (at 59078) by debbugs.gnu.org; 11 Nov 2022 21:57:48 +0000 Received: from localhost ([127.0.0.1]:46914 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1otc20-0003Fm-3G for submit@debbugs.gnu.org; Fri, 11 Nov 2022 16:57:48 -0500 Received: from eggs.gnu.org ([209.51.188.92]:32934) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1otc1x-0003FY-Mb for 59078@debbugs.gnu.org; Fri, 11 Nov 2022 16:57:46 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1otc1s-0000f9-Fd; Fri, 11 Nov 2022 16:57:40 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:In-Reply-To:Date:References:Subject:To: From; bh=cuVrb0nyfLdB1+jaQibg2Rr8Wzu0A8bSFxThvMvT2nc=; b=BAFpAcm0760ubEHe91NV dvQN8JFS+pGM0LYBU6Nx1mqrd+zKiJdP4P+AWGJ5Wql78UCQt7W6SRdscROqhyiRtTi1HhL91bM0d FeSU6Cva/s9hUnHEYtC8G4I2QTl7kCT8xYyPQsX5piUl5ri9NYm9oBqXoyeVRFjNX678Hce7Upxs+ mtVvXbhm+Zb722mjqBpwIEXMYfg8DTNY58MicdPClTubqxMssGFcIQanWr5ea1o11CNiMGRRjZved XgC8KsDVRhAioyZzB0vXCsDVEQtrSBUIPeQDprBrhAwFtMiP4+8Mbtg1xv9u8UEc5xXMVuqK1wTpY PHxCCljAdHgG5g==; Received: from 91-160-117-201.subs.proxad.net ([91.160.117.201] helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1otc1q-0007vv-MM; Fri, 11 Nov 2022 16:57:40 -0500 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Christopher Baines Subject: Re: bug#59078: [PATCH] lint: Split the derivation lint checker by system. References: <20221106135532.5724-1-mail@cbaines.net> Date: Fri, 11 Nov 2022 22:57:36 +0100 In-Reply-To: <20221106135532.5724-1-mail@cbaines.net> (Christopher Baines's message of "Sun, 6 Nov 2022 14:55:32 +0100") Message-ID: <87k041dui7.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (gnu/linux) MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 59078 Cc: 59078@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hi, Christopher Baines skribis: > Currently, if you attempt to run the derivation checker on all packages, = the > Guile process will run out of memory. I think a contributing factor to th= is is > that the checker forces an inefficient order when you want to generate > derivations for all the supported systems of each package, constantly > switching system then package. > > This problem also impacts the Guix Data Service, since it tries to run the > derivation checker for all packages. > > The changes in this commit to split the derivation lint checker in to sev= eral, > one for each system, means that you can now treat each system separately, > which should be better for caching purposes. > > If it's desirable to keep some notion of checking all supported systems f= or a > single package, I think lint checker groups could be added, so that you c= ould > ask for the "derivation" checker, and this would run all the derivation > checkers. The =E2=80=98derivation=E2=80=99 checker was added for this purpose: making= sure that a package=E2=80=99s derivation can be computed for all the supported systems. Previously it was easy to overlook that kind of breakage. I think it=E2=80=99s important to keep a =E2=80=98derivation=E2=80=99 check= er that does this. Now, the memory consumption you report is unacceptable and this needs to be addressed. Most (all?) caches are now per-session (associated with ). Since =E2=80=98guix lint=E2=80=99 uses a single sessi= on, those caches keep growing because there=E2=80=99s no eviction mechanism in place. A hack like the one below should work around that. Could you check how well it works for you? It can also be helpful to set: GUIX_PROFILING=3Dgc GUIX_PROFILING_EVENTS=3Dafter-gc Longer-term we should have a proper cache eviction mechanism in place. It=E2=80=99s not been a priority because =E2=80=9Cnormal=E2=80=9D applicati= ons such as =E2=80=98guix package=E2=80=99 don=E2=80=99t need it. Thanks, Ludo=E2=80=99. --=-=-= Content-Type: text/x-patch Content-Disposition: inline diff --git a/guix/scripts/lint.scm b/guix/scripts/lint.scm index 9920c3ee62..6d9b9e96a9 100644 --- a/guix/scripts/lint.scm +++ b/guix/scripts/lint.scm @@ -209,23 +209,31 @@ (define (parse-options) (list-checkers-and-exit checkers)) (with-error-handling - (let ((any-lint-checker-requires-store? - (any lint-checker-requires-store? checkers))) + (let ((store-needed? (any lint-checker-requires-store? checkers))) - (define (call-maybe-with-store proc) - (if any-lint-checker-requires-store? - (with-store store - (proc store)) - (proc #f))) - - (call-maybe-with-store - (lambda (store) - (cond - ((null? args) - (fold-packages (lambda (p r) (run-checkers p checkers - #:store store)) '())) - (else - (for-each (lambda (package) - (run-checkers package checkers - #:store store)) - args))))))))) + (cond + ((null? args) + (let loop ((packages (fold-packages cons '())) + (processed 0) + (store #f)) + (match packages + ((package . rest) + (let ((store (and store-needed? + (if store + ;; XXX: Periodically start a new session + ;; with empty caches to avoid unbounded + ;; heap growth caused by the 'derivation' + ;; checker. + (if (zero? (modulo processed 1000)) + (begin + (close-connection store) + (open-connection)) + store) + (open-connection))))) + (run-checkers package checkers #:store store) + (loop rest (+ 1 processed) store)))))) + (else + (for-each (lambda (package) + (run-checkers package checkers + #:store store)) + args))))))) --=-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Sun Nov 13 12:50:27 2022 Received: (at 59078) by debbugs.gnu.org; 13 Nov 2022 17:50:27 +0000 Received: from localhost ([127.0.0.1]:52462 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ouH7j-0004he-9G for submit@debbugs.gnu.org; Sun, 13 Nov 2022 12:50:27 -0500 Received: from mira.cbaines.net ([212.71.252.8]:41834) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ouH7g-0004hU-3d for 59078@debbugs.gnu.org; Sun, 13 Nov 2022 12:50:26 -0500 Received: from localhost (unknown [IPv6:2a02:8010:68c1:0:3a91:a0a4:ecee:f157]) by mira.cbaines.net (Postfix) with ESMTPSA id A205B27BBE9; Sun, 13 Nov 2022 17:50:22 +0000 (GMT) Received: from felis (localhost [127.0.0.1]) by localhost (OpenSMTPD) with ESMTP id 94e9cc66; Sun, 13 Nov 2022 17:50:21 +0000 (UTC) References: <20221106135532.5724-1-mail@cbaines.net> <87k041dui7.fsf@gnu.org> User-agent: mu4e 1.8.9; emacs 28.1 From: Christopher Baines To: Ludovic =?utf-8?Q?Court=C3=A8s?= Subject: Re: bug#59078: [PATCH] lint: Split the derivation lint checker by system. Date: Sun, 13 Nov 2022 17:27:48 +0000 In-reply-to: <87k041dui7.fsf@gnu.org> Message-ID: <87bkpau4kl.fsf@cbaines.net> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 59078 Cc: 59078@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Ludovic Court=C3=A8s writes: > Christopher Baines skribis: > >> Currently, if you attempt to run the derivation checker on all packages,= the >> Guile process will run out of memory. I think a contributing factor to t= his is >> that the checker forces an inefficient order when you want to generate >> derivations for all the supported systems of each package, constantly >> switching system then package. >> >> This problem also impacts the Guix Data Service, since it tries to run t= he >> derivation checker for all packages. >> >> The changes in this commit to split the derivation lint checker in to se= veral, >> one for each system, means that you can now treat each system separately, >> which should be better for caching purposes. >> >> If it's desirable to keep some notion of checking all supported systems = for a >> single package, I think lint checker groups could be added, so that you = could >> ask for the "derivation" checker, and this would run all the derivation >> checkers. > > The =E2=80=98derivation=E2=80=99 checker was added for this purpose: maki= ng sure that a > package=E2=80=99s derivation can be computed for all the supported system= s. > Previously it was easy to overlook that kind of breakage. > > I think it=E2=80=99s important to keep a =E2=80=98derivation=E2=80=99 che= cker that does this. What aspect of it do you think is important? I realise just splitting the checker in to several doesn't quite pick up on all the same problems, but an additional checker could be added that flags when packages support systems that don't appear as platforms. I think that would catch everything that the derivation checker currently does. If it's a "we need a singular checker" kind of problem, I was thinking of having some kind of lint checker groups. So you could have a group called "derivation", which runs all of the relevant checkers. > Now, the memory consumption you report is unacceptable and this needs to > be addressed. > > Most (all?) caches are now per-session (associated with > ). Since =E2=80=98guix lint=E2=80=99 uses a single ses= sion, those > caches keep growing because there=E2=80=99s no eviction mechanism in plac= e. > > A hack like the one below should work around that. Could you check how > well it works for you? I tried in the Guix Data Service processing packages in chunks of 1000 plus closing the store connection after each batch, and that led to a heap size of 3090MiB. But this is still higher than 1778MiB heap usage I got just by splitting the derivation linter. Fundamentally, I think it's still going to be more memory/time efficient to move towards processing the derivations by system, rather than by package. Thanks, Chris --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmNxLlpfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh aW5lcy5uZXQACgkQXiijOwuE9XezfA/9FHf6b7aHZTFe1eu4v2YibxOX8g0s87O1 2GVyMEG5qc3ome2130/430dATjtdQu20SIyswavSHLTQTPchFuwTi3AE0z5un15l doeMEHtM+PHDgTl0alxE9UAfWIRuliUnGzbtjJco921zbVr+pPQ9qT+//XFlAfZx X64XVcvlvGELVww0i5mrLVbukeI8rErcsHW0EcLkBw5El4JxN+54CDr3t6KnF+2O yh63HVgfWj9GOZUDQH1iuEktAoRccZl5CWKHO8qHVaeW4nt/0wrvAj/XO3XI4xXp elqg7WRZdj5fPFHqvwBtrFnMa/QI2+vBsJ7qvN1XDLWhxfeSgiDF7Hoh20gVowHt 3KcEWp0tN092ciyTFr1UrQeLeXxz+rxmo55gytQzINqJxzVAOWDQKgTq2v1bx4+w /DGS32Br7aZQiTBEhVaoQQVGX7rZHhFEKLL1algNHCKKdAhROkKIANBV43Go54SP M2/lRY2j0CvSRcbwxAo0nWNcwM1/zMK/knjenQ1GUpd1W9ysKlfcXw7emjwAR1TU SmxlMUKqlnoMTYj0lt/HG+ueAOpvpmGUaGSM5311us0CdJ7x5c8VqQm8lNfAfBGB RspmDeyYm0cO3lgngq4SD5aCMPLtDRK6S5lVQ9SFumXPBfkNmYe21jOO9fpOo814 xXLfKg+C/LI= =/WZa -----END PGP SIGNATURE----- --=-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Mon Nov 14 07:51:49 2022 Received: (at 59078) by debbugs.gnu.org; 14 Nov 2022 12:51:49 +0000 Received: from localhost ([127.0.0.1]:49103 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ouYwG-0007Wc-TX for submit@debbugs.gnu.org; Mon, 14 Nov 2022 07:51:49 -0500 Received: from eggs.gnu.org ([209.51.188.92]:49212) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ouYwE-0007WP-PA for 59078@debbugs.gnu.org; Mon, 14 Nov 2022 07:51:47 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ouYw9-0006Pa-Eb; Mon, 14 Nov 2022 07:51:41 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:In-Reply-To:Date:References:Subject:To: From; bh=4382XRtYEFJmfpv8Tfpz6I76GVbBRulb0pTF2habUqI=; b=pAWgu2IktBeA8eEjJ44h YYzKzO9D4T0P7eax+Ngz925rrB4k4XiUi7mfpZhx/UmM9kRkq9/e9OLSqafwz4lp2ZKijmNN114P1 78rrsoF3ycyqRE56OAfQKH/2CvC1dOaGv1h/JqKDhEZwHSnLhV0+7KOJYyk1lxp0wPELKN8NWeEE0 O2D2C/62V2J7fPsmJ65ILyiGFkOrAGxpqhwaXEo8bTgilZZoDMAEg1MDlbNgUsHMJOzx5VUTPH2yp 8gug8ZsVQwtlXgnu9InOe8n2hfjGd0AQaWy3k07KP7oV1x0rYffeG2lGcfY/1e5kqs9lSwM30PWG/ I1vtpAijOn01Aw==; Received: from [2001:660:6102:320:e120:2c8f:8909:cdfe] (helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ouYw9-00040V-1G; Mon, 14 Nov 2022 07:51:41 -0500 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Christopher Baines Subject: Re: bug#59078: [PATCH] lint: Split the derivation lint checker by system. References: <20221106135532.5724-1-mail@cbaines.net> <87k041dui7.fsf@gnu.org> <87bkpau4kl.fsf@cbaines.net> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: Quartidi 24 Brumaire an 231 de la =?utf-8?Q?R=C3=A9v?= =?utf-8?Q?olution=2C?= jour de l'Orange X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Mon, 14 Nov 2022 13:51:37 +0100 In-Reply-To: <87bkpau4kl.fsf@cbaines.net> (Christopher Baines's message of "Sun, 13 Nov 2022 17:27:48 +0000") Message-ID: <87h6z1puli.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 59078 Cc: 59078@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hi! Christopher Baines skribis: > Ludovic Court=C3=A8s writes: [...] >> The =E2=80=98derivation=E2=80=99 checker was added for this purpose: mak= ing sure that a >> package=E2=80=99s derivation can be computed for all the supported syste= ms. >> Previously it was easy to overlook that kind of breakage. >> >> I think it=E2=80=99s important to keep a =E2=80=98derivation=E2=80=99 ch= ecker that does this. > > What aspect of it do you think is important? I meant that it=E2=80=99s important to have a single =E2=80=98derivation=E2= =80=99 checker that checks derivations for all the supported systems. Packagers should be able to run =E2=80=98guix lint -c derivation PKG=E2=80=99 and be confident = that it=E2=80=99s fine for all systems. >> Now, the memory consumption you report is unacceptable and this needs to >> be addressed. >> >> Most (all?) caches are now per-session (associated with >> ). Since =E2=80=98guix lint=E2=80=99 uses a single se= ssion, those >> caches keep growing because there=E2=80=99s no eviction mechanism in pla= ce. >> >> A hack like the one below should work around that. Could you check how >> well it works for you? > > I tried in the Guix Data Service processing packages in chunks of 1000 > plus closing the store connection after each batch, How was it implemented? Was it after the caches came into ? > and that led to a heap size of 3090MiB. But this is still higher than > 1778MiB heap usage I got just by splitting the derivation linter. I didn=E2=80=99t take the time to do it, but it would be nice to see, with = the patch I gave, how =E2=80=98guix lint -c derivations=E2=80=99 behaves. Thanks, Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Tue Nov 15 04:56:09 2022 Received: (at 59078) by debbugs.gnu.org; 15 Nov 2022 09:56:09 +0000 Received: from localhost ([127.0.0.1]:52998 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ousfo-0002TC-J9 for submit@debbugs.gnu.org; Tue, 15 Nov 2022 04:56:09 -0500 Received: from mira.cbaines.net ([212.71.252.8]:41836) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ousfi-0002Sy-U1 for 59078@debbugs.gnu.org; Tue, 15 Nov 2022 04:56:06 -0500 Received: from localhost (unknown [IPv6:2a02:8010:68c1:0:54d1:d5d4:280e:f699]) by mira.cbaines.net (Postfix) with ESMTPSA id 6472327BBE9; Tue, 15 Nov 2022 09:56:01 +0000 (GMT) Received: from felis (localhost [127.0.0.1]) by localhost (OpenSMTPD) with ESMTP id b72db4c8; Tue, 15 Nov 2022 09:55:59 +0000 (UTC) References: <20221106135532.5724-1-mail@cbaines.net> <87k041dui7.fsf@gnu.org> <87bkpau4kl.fsf@cbaines.net> <87h6z1puli.fsf@gnu.org> User-agent: mu4e 1.8.11; emacs 28.2 From: Christopher Baines To: Ludovic =?utf-8?Q?Court=C3=A8s?= Subject: Re: bug#59078: [PATCH] lint: Split the derivation lint checker by system. Date: Tue, 15 Nov 2022 08:35:42 +0000 In-reply-to: <87h6z1puli.fsf@gnu.org> Message-ID: <87leocfsnm.fsf@cbaines.net> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 59078 Cc: 59078@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Ludovic Court=C3=A8s writes: > Hi! > > Christopher Baines skribis: > >> Ludovic Court=C3=A8s writes: > > [...] > >>> The =E2=80=98derivation=E2=80=99 checker was added for this purpose: ma= king sure that a >>> package=E2=80=99s derivation can be computed for all the supported syst= ems. >>> Previously it was easy to overlook that kind of breakage. >>> >>> I think it=E2=80=99s important to keep a =E2=80=98derivation=E2=80=99 c= hecker that does this. >> >> What aspect of it do you think is important? > > I meant that it=E2=80=99s important to have a single =E2=80=98derivation= =E2=80=99 checker that > checks derivations for all the supported systems. Packagers should be > able to run =E2=80=98guix lint -c derivation PKG=E2=80=99 and be confiden= t that it=E2=80=99s > fine for all systems. I think we can still keep that by adding support for grouping lint checkers. So have a derivation group, instead of a single checker. Maybe that'll change the command slightly to 'guix lint -g derivation PKG', but I think that can be equivalent. >>> Now, the memory consumption you report is unacceptable and this needs to >>> be addressed. >>> >>> Most (all?) caches are now per-session (associated with >>> ). Since =E2=80=98guix lint=E2=80=99 uses a single s= ession, those >>> caches keep growing because there=E2=80=99s no eviction mechanism in pl= ace. >>> >>> A hack like the one below should work around that. Could you check how >>> well it works for you? >> >> I tried in the Guix Data Service processing packages in chunks of 1000 >> plus closing the store connection after each batch, > > How was it implemented? Was it after the caches came into > ? I'm not sure what aspect of the implementation is important, but I think it's working correctly. Closing the store connection wasn't very easy. Previously there was a fresh store connection with each call to inferior-eval-with-store, but for this test I close the connections in %store-table and then clear the hash table between the inferior-eval-with-store calls. >> and that led to a heap size of 3090MiB. But this is still higher than >> 1778MiB heap usage I got just by splitting the derivation linter. > > I didn=E2=80=99t take the time to do it, but it would be nice to see, wit= h the > patch I gave, how =E2=80=98guix lint -c derivations=E2=80=99 behaves. I've put some numbers below, with no changes the last batch to finish processing leaves the heap at 7755MiB [1], then Guile crashes after that. With the patch you sent, the heap size seems to stabilise at 4042MiB [2]. It also crashes at the end due to the match block not matching '(), but that's not important. I also hacked the lint script to run the checkers in the same manor as the Guix Data Service, so one checker at a time rather than one package at a time. That led to a max heap size of 3505MiB [3]. By adding in batching (as the Guix Data Service already does), I think it's possible to further reduce this to the 1778MiB number I give above. Reducing the memory usage helps reduce the cost/improve the throughput of loading data in to the Guix Data Service which is my primary motivation here. I'm also not only concerned with reducing the peak memory usage, but trying to have an implementation that'll gracefully handle more systems being supported in the future. It's for that second point that I think arranging the derivation linting so that it's possible to process each system in turn is important for the Guix Data Service, so that when new platforms are added, the memory usage won't grow as much. 1: no batching, one derivation checker 1065.0 MiB 1409.0 MiB 2089.0 MiB 2297.0 MiB 2513.0 MiB 2705.0 MiB 3077.0 MiB 3373.0 MiB 3557.0 MiB 3661.0 MiB 3901.0 MiB 3997.0 MiB 4147.0 MiB 4491.0 MiB 4635.0 MiB 5899.0 MiB 7755.0 MiB 2: batches of 1000 with fresh store connection, one derivation checker 1057.0 MiB 1137.0 MiB 1481.0 MiB 1481.0 MiB 1481.0 MiB 1481.0 MiB 1697.0 MiB 1697.0 MiB 1697.0 MiB 1761.0 MiB 1761.0 MiB 1761.0 MiB 1937.0 MiB 1977.0 MiB 1985.0 MiB 3065.0 MiB 3633.0 MiB 4041.0 MiB 4042.0 MiB 4042.0 MiB 4042.0 MiB 3: multiple derivation checkers, batched by system 1297.0 MiB 1545.0 MiB 1873.0 MiB 2113.0 MiB 2353.0 MiB 2609.0 MiB 2961.0 MiB 3193.0 MiB 3505.0 MiB --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmNzYi1fFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh aW5lcy5uZXQACgkQXiijOwuE9Xf2Tg/+MEKPBt1Ibu6sWQD9Xia/nR7MTimBHcOw dA8/WR13He3MwjxwaBjSI+T09RVYokukCYbpf4UjkxgU3np+bDyAUj7XA/C01m6b 67jYpuZ0+11Jgm4iLbZWh8O7YVX+K8hilnLynFTEU4X5ns0way4L1pGrN19LmnPt dS0Dh7Ul3j5rCHMMWjCrrK9/p51snm12lZCV1uk9bvhYrkoEC/ByO/1T9o3FS7h7 WD3dmBN3eqwtfe4VsPVBf+6XREBDlxDTBPlxsGd+RsmghFkR5k34WQP5otyAP+R7 CpeKZ2K/BoU1CJduz5fMQQUtAumASGBS+symysyIUyRIp2hcZyHTp4YbvbkjxwIe 9CS/Oq/qIQFoOpFfX8qGb/23EJc761YnO+73Pjk42ggWwO+nl45tN20T4jwUTblJ 7L+RoKBhK+pdn/ojJfFdk1CoSnwM0sihkhkuEAaNJ6lDr8wYSb2q5O73ZzKoBCUT lw1PrPDNEv3D2BQ1w1mO1dste6X0Jxb7ZbIGUJRGLkU+p4+RII3N59/EXceiq69w WOaFaYvo/Cij5y2ZPGocIgZofHaFImd3A1oBoa3YgaAA2hopQqugbDBLz6WLnVMR +gIMgzd0UhHoPGIXI3ZsQkGsYW3eA1sY++hBxzkciBMG4n/+etsfk9SJswWz7je9 QpTiSyazp4M= =m6Vq -----END PGP SIGNATURE----- --=-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Tue Nov 15 09:54:57 2022 Received: (at 59078) by debbugs.gnu.org; 15 Nov 2022 14:54:57 +0000 Received: from localhost ([127.0.0.1]:53647 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ouxKz-0004jh-82 for submit@debbugs.gnu.org; Tue, 15 Nov 2022 09:54:57 -0500 Received: from mail-wr1-f48.google.com ([209.85.221.48]:35829) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ouxKu-0004iv-Ks for 59078@debbugs.gnu.org; Tue, 15 Nov 2022 09:54:52 -0500 Received: by mail-wr1-f48.google.com with SMTP id l14so24703654wrw.2 for <59078@debbugs.gnu.org>; Tue, 15 Nov 2022 06:54:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:message-id:date:references :in-reply-to:subject:cc:to:from:from:to:cc:subject:date:message-id :reply-to; bh=VJc3yY62OvqpdFMSCWmq7uKfyuwRl4ujz8tlaKQO8eo=; b=IfDIQMoakdJvv8zv4hzwPgcaAtuVwVAC/+moaYFnFlrXAF1jz68tCMepfwxB+DtgD/ 2aYGArDRPKX5seiT+k50gpBoEKNBJPVJgOtEEWqwPtWHnC68zhD/ziskslCLxF+EK3Um +iuct2xpdxEEL5SmPLL6a7qIqGCgi8fSXSgPlc4wbdva6u0dkdeC08rcEeY5uA5rYo1i je6rDCXqkwj2ikTWfCtOIh93oOF9LP9MQB45J6Zm1F79PD94TAoGmj6B1ICd4kDZq9jW RM26/Pgz1VpHkkKjF7Jmq72FhTzV7HJzCS9LTiJMWKuNE5VhVxj11ipMBitNN7Pd4Eaq JE8A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:message-id:date:references :in-reply-to:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=VJc3yY62OvqpdFMSCWmq7uKfyuwRl4ujz8tlaKQO8eo=; b=dVo7+/eGdPbGaGjw3lclnIpyjkMOdOBGaGixZI77ZFXCv0ObMP71loZzFpRsUmSMKq Ur8Ps0BQ+gcAuaroIQIwqLnRhlD7dP5igw4+NINY76dpO2JGgJpEdXs3mq2IhsrYERzw a0NOfNHYoK+Fgm0Pr5ih3CoBwv8pkLkhM9nyjXO7pPIABqLwEaIdZMlnJ6uUagf+4CKu rI0kUkdcF3y+VDKtEwGTMypJv5JArEWP1XsKgAJ6lb8XsUFH0WbOv7AMxMk9PdaVgVGO OK6pMW0mQF76+yBsW4WSov+UOgN3HgeJm1SJfcjLoG46lPwQgQ5XXY8bPdHA90VjqUtF Qpvw== X-Gm-Message-State: ANoB5pmnIh/c3FhEIGnFVfcPNDWcPqZ5Fh1lwq7BiAqbkI4jS/DHMD4F W1iBFBKNU+WzrQKhtvo7SmwJqgtjQPY= X-Google-Smtp-Source: AA0mqf6uPwkKLWVhgMW6MCBY2ceNxZf9YYJGH+lKthvA0GIKqfP4MDFCXP4L0psy1nEYB3XVeJTgow== X-Received: by 2002:a5d:52c6:0:b0:236:754e:f8b4 with SMTP id r6-20020a5d52c6000000b00236754ef8b4mr10992895wrv.478.1668524087104; Tue, 15 Nov 2022 06:54:47 -0800 (PST) Received: from lili (roam-nat-fw-prg-194-254-61-46.net.univ-paris-diderot.fr. [194.254.61.46]) by smtp.gmail.com with ESMTPSA id bp11-20020a5d5a8b000000b002366eb01e07sm12717059wrb.114.2022.11.15.06.54.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 15 Nov 2022 06:54:46 -0800 (PST) From: zimoun To: Ludovic =?utf-8?Q?Court=C3=A8s?= , Christopher Baines Subject: Re: [bug#59078] [PATCH] lint: Split the derivation lint checker by system. In-Reply-To: <87h6z1puli.fsf@gnu.org> References: <20221106135532.5724-1-mail@cbaines.net> <87k041dui7.fsf@gnu.org> <87bkpau4kl.fsf@cbaines.net> <87h6z1puli.fsf@gnu.org> Date: Tue, 15 Nov 2022 10:03:37 +0100 Message-ID: <86r0y4io7q.fsf@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: 1.1 (+) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: Hi, On Mon, 14 Nov 2022 at 13:51, Ludovic Courtès wrote: > I meant that it’s important to have a single ‘derivation’ checker that > checks derivations for all the supported systems. Packagers should be > able to run ‘guix lint -c derivation PKG’ a [...] Content analysis details: (1.1 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [209.85.221.48 listed in list.dnswl.org] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record 1.1 DATE_IN_PAST_03_06 Date: is 3 to 6 hours before Received: date 0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider (zimon.toutoune[at]gmail.com) -0.0 SPF_PASS SPF: sender matches SPF record -0.0 RCVD_IN_MSPIKE_H2 RBL: Average reputation (+2) [209.85.221.48 listed in wl.mailspike.net] X-Debbugs-Envelope-To: 59078 Cc: 59078@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 0.1 (/) Hi, On Mon, 14 Nov 2022 at 13:51, Ludovic Court=C3=A8s wrote: > I meant that it=E2=80=99s important to have a single =E2=80=98derivation= =E2=80=99 checker that > checks derivations for all the supported systems. Packagers should be > able to run =E2=80=98guix lint -c derivation PKG=E2=80=99 and be confiden= t that it=E2=80=99s > fine for all systems. The CLI invokation is unrelated to the invoked checkers, no? As Chris is proposing, it seems being worth to group some checkers. For instance, we already have the option =E2=80=99-n, --no-network=E2=80=99 whi= ch does that but probably at the wrong level. Maybe we could have another command line option, guix lint --group=3Dno-network guix lint --group=3Dderivation guix lint --group=3Dno-network,derivation I do not know=E2=80=A6 Cheers, simon From debbugs-submit-bounces@debbugs.gnu.org Thu Nov 17 12:22:53 2022 Received: (at 59078) by debbugs.gnu.org; 17 Nov 2022 17:22:53 +0000 Received: from localhost ([127.0.0.1]:34224 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ovibE-00050M-Mw for submit@debbugs.gnu.org; Thu, 17 Nov 2022 12:22:53 -0500 Received: from eggs.gnu.org ([209.51.188.92]:42648) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ovibB-000506-Ho for 59078@debbugs.gnu.org; Thu, 17 Nov 2022 12:22:52 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ovib6-0005gx-7f; Thu, 17 Nov 2022 12:22:44 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:In-Reply-To:Date:References:Subject:To: From; bh=IDEz3EsCy9TPnfsPZVcnWX7/J8OlFq8kBCh15P1Pa54=; b=NFCZucqDYuVq6MbGlb3y CkSKbPsfAakFhjRiQ3LgkBkg/7p2Jq12UnvNnFiczh98Te+ri3JiH2CmumBnMVPxdqWxnUMsczV/o G1O3buMDKUeN54cH3aCfYLRAYfRQ4htqKbpHK52qzNm6wvVZWnUpF4oCE+6y9G5jRJ0QsVDsH5EEG Ff5A8YD0UPhjAMKvScdCB72nExIcQ4UwkWkVl69ORvrb/QydcsPxd/BO9MgfNMvtQJgxZ0o05I2aB mE23d4KF698Iwh57Iz5ysIjcy3iw0cPIQqIJ9d88pjp7AxwIa2ymjAC4zYb8ujppfj+8KuV2IxpYm ZWZTL1wm/8VbwQ==; Received: from 91-160-117-201.subs.proxad.net ([91.160.117.201] helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ovib5-0008EA-Rq; Thu, 17 Nov 2022 12:22:44 -0500 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Christopher Baines Subject: Re: bug#59078: [PATCH] lint: Split the derivation lint checker by system. References: <20221106135532.5724-1-mail@cbaines.net> <87k041dui7.fsf@gnu.org> <87bkpau4kl.fsf@cbaines.net> <87h6z1puli.fsf@gnu.org> <87leocfsnm.fsf@cbaines.net> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: Septidi 27 Brumaire an 231 de la =?utf-8?Q?R=C3=A9vo?= =?utf-8?Q?lution=2C?= jour du Macjon X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Thu, 17 Nov 2022 18:22:41 +0100 In-Reply-To: <87leocfsnm.fsf@cbaines.net> (Christopher Baines's message of "Tue, 15 Nov 2022 08:35:42 +0000") Message-ID: <87a64ph4wu.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 59078 Cc: 59078@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hi, Christopher Baines skribis: > I've put some numbers below, with no changes the last batch to finish > processing leaves the heap at 7755MiB [1], then Guile crashes after > that. > > With the patch you sent, the heap size seems to stabilise at 4042MiB > [2]. It also crashes at the end due to the match block not matching '(), > but that's not important. > > I also hacked the lint script to run the checkers in the same manor as > the Guix Data Service, so one checker at a time rather than one package > at a time. That led to a max heap size of 3505MiB [3]. Thanks for testing and reporting back! > Reducing the memory usage helps reduce the cost/improve the throughput > of loading data in to the Guix Data Service which is my primary > motivation here. I'm also not only concerned with reducing the peak > memory usage, but trying to have an implementation that'll gracefully > handle more systems being supported in the future. Understood. From the viewpoint of core Guix, I=E2=80=99d like to understand where all this memory is going; this is unreasonable. The heap profiler I posted recently=C2=B9, coupled with =E2=80=98gcprof=E2=80=99, might help = us understand what=E2=80=99s going on. =C2=B9 https://lists.gnu.org/archive/html/guile-user/2022-11/msg00012.html > 1: no batching, one derivation checker [...] > 7755.0 MiB > > 2: batches of 1000 with fresh store connection, one derivation checker > 1057.0 MiB > 1137.0 MiB > 1481.0 MiB > 1481.0 MiB > 1481.0 MiB > 1481.0 MiB > 1697.0 MiB > 1697.0 MiB > 1697.0 MiB > 1761.0 MiB > 1761.0 MiB > 1761.0 MiB > 1937.0 MiB > 1977.0 MiB > 1985.0 MiB > 3065.0 MiB > 3633.0 MiB > 4041.0 MiB > 4042.0 MiB > 4042.0 MiB > 4042.0 MiB This seems to indicate a memory leak outside , such as a static cache. =E2=80=9CGUIX_PROFILING=3Dmemoization=E2=80=9D may give a = hint. I=E2=80=99d really like to get it solved before we come up with workarounds= like those we=E2=80=99re talking about. (However I=E2=80=99ll postpone this one= because I=E2=80=99d like to focus on the release for now.) Thanks, Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Fri Jan 27 13:56:29 2023 Received: (at 59078) by debbugs.gnu.org; 27 Jan 2023 18:56:29 +0000 Received: from localhost ([127.0.0.1]:38461 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pLTtl-0003gm-F5 for submit@debbugs.gnu.org; Fri, 27 Jan 2023 13:56:29 -0500 Received: from mail-wm1-f48.google.com ([209.85.128.48]:52978) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pLTtf-0003fz-MC for 59078@debbugs.gnu.org; Fri, 27 Jan 2023 13:56:24 -0500 Received: by mail-wm1-f48.google.com with SMTP id k16so4104730wms.2 for <59078@debbugs.gnu.org>; Fri, 27 Jan 2023 10:56:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:user-agent:message-id :in-reply-to:date:references:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=xlY1wNKzo0n8fEKupj7+n5LYmMDRitrY70U3OuM7cBw=; b=cuYK6gdaLo2Ouj5GYdAOrT6nlIlayBDf5s8g6hHsA8T5fSuHKAKWSVWCGCQRtIuaGD uflN47gZIkFA+vnCFS6XpB3cbgFYWuMv6X193G8ksIzxahsRHJZDuZcxo4CGKXRT4yEj hBjdbrnO3ogi6Tthjs6osaeL2nl2M88tcrpAB1jT8I058ED1XaCE4yXgMjKa4HFFj2NX jZ95Bby5tfwmwOHAzvNOKRpRRrHCa6BtYfvIQbggofG+Ehh/rZ7boQa0yNuALIFOhGv6 VMJY2tq6BelnWgCqYMw5xyfyjpZhw9kYWDJZIl958TyanXXrfA1U5rvzM6cecZtMuo/G 8VQQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:user-agent:message-id :in-reply-to:date:references:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=xlY1wNKzo0n8fEKupj7+n5LYmMDRitrY70U3OuM7cBw=; b=EkY2W4r85PWV10NCr9V2FM51yPGO/UbZQK3OLXq5vLg4AjEiljDV4VIu4i8O+4onrv cnHNi21Kn4S8MMF0AJd5CeUDexgy9ev8+15Gv/Pjw3r7rrMtDYx6KCO0gyeX2mKX6kNB VWIlvPOrlwsZHZXsz4Dgxymd1jhZI0YmByW9fbaMVtD4SRUW2Y3eEzH0UEB5yyyY5K2o aNkusKmOqgTbaEtK1GIdBTN15ize6phgs1NevYSjM4SDKXe/mR7XJIQe+zUZ/hngndBR iGCU1ZCY1OrPjLRuCdlRArNlGags4wf+iBtcABRcb/XpCfs1dexyVY1ZwCGc4htpiNCa xM2w== X-Gm-Message-State: AFqh2kp8hml/vI5wesSu5wU1q0hotyTgJZ6bIXPv41Np1LTcFVhe1LiH DaZ/nzXqMlmYVYl9HuT42e/POARIHoY= X-Google-Smtp-Source: AMrXdXs3Xl7EtwEDVBllnQczhB+/Ri1wzpwgL367U5VXZg+SBSdod/GntEfdzqrFtxA10RabdJsa7w== X-Received: by 2002:a05:600c:1d22:b0:3da:29b8:8980 with SMTP id l34-20020a05600c1d2200b003da29b88980mr10974782wms.1.1674845778124; Fri, 27 Jan 2023 10:56:18 -0800 (PST) Received: from pfiuh07 ([193.48.40.241]) by smtp.gmail.com with ESMTPSA id f12-20020adfdb4c000000b002bfb1de74absm4759870wrj.114.2023.01.27.10.56.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 27 Jan 2023 10:56:17 -0800 (PST) From: Simon Tournier To: Ludovic =?utf-8?Q?Court=C3=A8s?= Subject: Re: bug#59078: [PATCH] lint: Split the derivation lint checker by system. References: <20221106135532.5724-1-mail@cbaines.net> <87k041dui7.fsf@gnu.org> <87bkpau4kl.fsf@cbaines.net> <87h6z1puli.fsf@gnu.org> <86r0y4io7q.fsf@gmail.com> Date: Fri, 27 Jan 2023 18:48:19 +0100 In-Reply-To: <86r0y4io7q.fsf@gmail.com> (zimoun's message of "Tue, 15 Nov 2022 10:03:37 +0100") Message-ID: <875ycroom4.fsf_-_@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 59078 Cc: Christopher Baines , 59078@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Hi, On mar., 15 nov. 2022 at 10:03, zimoun wrote: > On Mon, 14 Nov 2022 at 13:51, Ludovic Court=C3=A8s wrote: > >> I meant that it=E2=80=99s important to have a single =E2=80=98derivation= =E2=80=99 checker that >> checks derivations for all the supported systems. Packagers should be >> able to run =E2=80=98guix lint -c derivation PKG=E2=80=99 and be confide= nt that it=E2=80=99s >> fine for all systems. > > The CLI invokation is unrelated to the invoked checkers, no? As Chris > is proposing, it seems being worth to group some checkers. For > instance, we already have the option =E2=80=99-n, --no-network=E2=80=99 w= hich does that > but probably at the wrong level. > > Maybe we could have another command line option, > > guix lint --group=3Dno-network > guix lint --group=3Dderivation > guix lint --group=3Dno-network,derivation What about this? Cheers, simon From debbugs-submit-bounces@debbugs.gnu.org Tue Jan 31 11:33:33 2023 Received: (at 59078) by debbugs.gnu.org; 31 Jan 2023 16:33:33 +0000 Received: from localhost ([127.0.0.1]:54832 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pMtZd-0004aA-DM for submit@debbugs.gnu.org; Tue, 31 Jan 2023 11:33:33 -0500 Received: from eggs.gnu.org ([209.51.188.92]:38828) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pMtZc-0004Zp-C2 for 59078@debbugs.gnu.org; Tue, 31 Jan 2023 11:33:32 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pMtZW-00073s-Tu; Tue, 31 Jan 2023 11:33:27 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:In-Reply-To:Date:References:Subject:To: From; bh=amDMNArAXgCJ3gFrQQnzofdX3IkOOg+qYnSeCHog2n0=; b=Ys1sP4WWn6GQRsZHEOqA HiDgLYiK217jbrWht+/2HHYJ/b6gP5BSJAsLLcnHA5MFiuLgArMHDfq+hvRmoy11OeqYGcevDAnSH Ju3XQOjArsuvPfRAWZFuX4QcjgcsuRxV27VkparhVjteAsUyQ/eRXC32rV+o9a/pwz1Tc2cYrYQId A8khXqdjXt1h+reUpIfNJsvd05Jvaizr+ED7ctwjEy+0SSnkq4FKkL3TQxK5/7l5eORqp/p5Usb+b U1sZH9m8Y60Z0aXF78CisuQn1Av34H0JKjrHOVZcHymmKlAxHhNKU6AlXmplfdKeZpS+tTIt3Fzff Xha0NG+Hblcs3A==; Received: from [2001:660:6102:320:e120:2c8f:8909:cdfe] (helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pMtZW-0006xV-29; Tue, 31 Jan 2023 11:33:26 -0500 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Simon Tournier Subject: Re: bug#59078: [PATCH] lint: Split the derivation lint checker by system. References: <20221106135532.5724-1-mail@cbaines.net> <87k041dui7.fsf@gnu.org> <87bkpau4kl.fsf@cbaines.net> <87h6z1puli.fsf@gnu.org> <86r0y4io7q.fsf@gmail.com> <875ycroom4.fsf_-_@gmail.com> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: Duodi 12 =?utf-8?Q?Pluvi=C3=B4se?= an 231 de la =?utf-8?Q?R=C3=A9volution=2C?= jour du Brocoli X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Tue, 31 Jan 2023 17:33:24 +0100 In-Reply-To: <875ycroom4.fsf_-_@gmail.com> (Simon Tournier's message of "Fri, 27 Jan 2023 18:48:19 +0100") Message-ID: <87tu06u0iz.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 59078 Cc: Christopher Baines , 59078@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hi, Simon Tournier skribis: > On mar., 15 nov. 2022 at 10:03, zimoun wrote: >> On Mon, 14 Nov 2022 at 13:51, Ludovic Court=C3=A8s wrote: >> >>> I meant that it=E2=80=99s important to have a single =E2=80=98derivatio= n=E2=80=99 checker that >>> checks derivations for all the supported systems. Packagers should be >>> able to run =E2=80=98guix lint -c derivation PKG=E2=80=99 and be confid= ent that it=E2=80=99s >>> fine for all systems. >> >> The CLI invokation is unrelated to the invoked checkers, no? As Chris >> is proposing, it seems being worth to group some checkers. For >> instance, we already have the option =E2=80=99-n, --no-network=E2=80=99 = which does that >> but probably at the wrong level. >> >> Maybe we could have another command line option, >> >> guix lint --group=3Dno-network >> guix lint --group=3Dderivation >> guix lint --group=3Dno-network,derivation > > What about this? In general, being able to tell which category a checker belongs to, and then being able to select checkers by categories sounds like a useful improvement to me. I don=E2=80=99t think it solves the problem Christopher initially reported though (about memory consumption of the =E2=80=98derivation=E2=80=99 checke= r.) Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Wed Feb 01 05:14:50 2023 Received: (at 59078) by debbugs.gnu.org; 1 Feb 2023 10:14:50 +0000 Received: from localhost ([127.0.0.1]:56556 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pNA8f-0007Da-T2 for submit@debbugs.gnu.org; Wed, 01 Feb 2023 05:14:50 -0500 Received: from mail-wm1-f53.google.com ([209.85.128.53]:56059) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pNA8c-0007Cz-BL for 59078@debbugs.gnu.org; Wed, 01 Feb 2023 05:14:48 -0500 Received: by mail-wm1-f53.google.com with SMTP id q8so12244015wmo.5 for <59078@debbugs.gnu.org>; Wed, 01 Feb 2023 02:14:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:message-id:date:references :in-reply-to:subject:cc:to:from:from:to:cc:subject:date:message-id :reply-to; bh=lxxSaUzGsGxDk0oNqhtBfuOtN3RVdc/7LRRYzmWG/AY=; b=MWtHD/LMm9FzYf8nkoIWs+TvdfUH9jG6+LIFMUY57lAdngS/2E00GxG911/IR8S3G9 erqUmKP293xWx3NKUdNjLkPCDaf+8BrBMwdv60KvLQTYx+WnU2LCrWv4Y3dRn8Vh78MU tza3hFTlc3udOSoHAprHiBtXVloZhsXfT2MJTPt8UQIqtVV/37ZOP4zT32dHt7Omd0pG bEjgX0qE7beaOR5dZJWfUhPjHmLsnkfLWDOe4n4iYDdimsiZoAv45QOY7t2Baudow34l mmD9dpjeeEbBSBZ1UadeqTP7Pp8cnFFmfwCV9DkAbohTxzMTOrISuNW4zqz0/J3hZvC5 v8LA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:message-id:date:references :in-reply-to:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=lxxSaUzGsGxDk0oNqhtBfuOtN3RVdc/7LRRYzmWG/AY=; b=dV86BweAslU7BMeXubbH+scTvHT68ra/AELxTzbCE8uvYCmJsTeSbCMJf9OQdsaZJd AY6KFB/W6xuWOwRnsvzO+WIXwiearmZZFqZu3El2vx7snTkGfMj/WeZN9iIJazsFT6Dd xThzX2KnGTiY9ml7QDAhNiDbT1xYoaWa8CgptnDC39nPqxaVEergC0qlYelb1kyD7/9A qumijeBWS4gGyJMIWE9EJVLkVRUYKLc2Mn+WjkziYqJKAYSktmgp6WJkTarv9HAEokY+ P5+cYbchP8WofqWVCKN2FIYPu1WUpkUSHty0did4obohnB8PVOOZU11eAWqmA/ZFkF5n TXug== X-Gm-Message-State: AO0yUKXYf1XrbslGfWCMUUGY3MOu+ZqRqfczcSD+VqfdknVzf+k5wlNK zluQ3l8OQdp3lF4JRQknWht9Qdc7vT0= X-Google-Smtp-Source: AK7set8IkGS0FLYBL80vlxgoHeXYG4KioTMGKlyAneN2LUkGcaqIn+brjUqeGm34s59d9bTNx87OpQ== X-Received: by 2002:a05:600c:1d05:b0:3dc:57e8:1d10 with SMTP id l5-20020a05600c1d0500b003dc57e81d10mr1445846wms.1.1675246480636; Wed, 01 Feb 2023 02:14:40 -0800 (PST) Received: from lili ([2a01:e0a:59b:9120:65d2:2476:f637:db1e]) by smtp.gmail.com with ESMTPSA id bg21-20020a05600c3c9500b003db06493ee7sm1374977wmb.47.2023.02.01.02.14.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 01 Feb 2023 02:14:40 -0800 (PST) From: zimoun To: Ludovic =?utf-8?Q?Court=C3=A8s?= Subject: Re: [bug#59078] [PATCH] lint: Split the derivation lint checker by system. In-Reply-To: <87tu06u0iz.fsf@gnu.org> References: <20221106135532.5724-1-mail@cbaines.net> <87k041dui7.fsf@gnu.org> <87bkpau4kl.fsf@cbaines.net> <87h6z1puli.fsf@gnu.org> <86r0y4io7q.fsf@gmail.com> <875ycroom4.fsf_-_@gmail.com> <87tu06u0iz.fsf@gnu.org> Date: Wed, 01 Feb 2023 10:47:08 +0100 Message-ID: <868rhhvhsz.fsf@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 59078 Cc: Christopher Baines , 59078@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Hi Ludo, On Tue, 31 Jan 2023 at 17:33, Ludovic Court=C3=A8s wrote: > In general, being able to tell which category a checker belongs to, and > then being able to select checkers by categories sounds like a useful > improvement to me. > > I don=E2=80=99t think it solves the problem Christopher initially reported > though (about memory consumption of the =E2=80=98derivation=E2=80=99 chec= ker.) Indeed. My understanding of the proposal is to split some checkers (as derivation) and then group them to have the expected behaviour. You wrote [1]: I meant that it=E2=80=99s important to have a single =E2=80=98deriv= ation=E2=80=99 checker that checks derivations for all the supported systems. Packagers should be able to run =E2=80=98guix lint -c derivation PK= G=E2=80=99 and be confident that it=E2=80=99s fine for all systems. and Chris answered [2]: I think we can still keep that by adding support for grouping lint checkers. So have a derivation group, instead of a single checker. Maybe that'll change the command slightly to 'guix lint -g derivati= on PKG', but I think that can be equivalent. where I try to feed Chris=E2=80=99s proposal. :-) I do not know if it is the correct level but being able to run =E2=80=9Csma= ller=E2=80=9C checkers appears to me worth to try. 1: http://issues.guix.gnu.org/msgid/87h6z1puli.fsf@gnu.org 2: http://issues.guix.gnu.org/msgid/87leocfsnm.fsf@cbaines.net Cheers, simon