From debbugs-submit-bounces@debbugs.gnu.org Tue Jul 16 05:11:21 2024 Received: (at submit) by debbugs.gnu.org; 16 Jul 2024 09:11:21 +0000 Received: from localhost ([127.0.0.1]:60958 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1sTeDR-0004q0-5j for submit@debbugs.gnu.org; Tue, 16 Jul 2024 05:11:21 -0400 Received: from lists.gnu.org ([209.51.188.17]:50406) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1sTeDP-0004pr-DG for submit@debbugs.gnu.org; Tue, 16 Jul 2024 05:11:20 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sTeDM-0005kG-Ds for guix-patches@gnu.org; Tue, 16 Jul 2024 05:11:16 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sTeDL-0002RD-Gb; Tue, 16 Jul 2024 05:11:15 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:Date:Subject:To:From:in-reply-to: references; bh=uCG46G1Ium+J47Lxs59VUVhRX82cof1jfl19Fka//Ng=; b=a8qMM4QfC2a+t9 yxF6cHiEavFjCtFEpOm2T5XQSPZ1Z1Lc+d7RQKeBA97v2krewy0Seempgt4ggghUkmWCc488Y0HIv O8OcD9+u/gFvXUaYmNI26+vduHsTHazuyDM/c3zwmknWZJKvakmBS1PBit1R/B2ftxIHXkGlJGhDs 7d/2+d8CJGe1YEhSZz4PlhG9XGj9jqJcCiukjpVERgyes9RrCnyognHVT1+dxr2ZMg2dwi2b2vmCj M4Q5uoNkCY9TO2dZL+3Js29s95Shv4EgEslBgNhBq+ypwtIJ/4UYcYUxY3DyhAbryLBvoptI0VQE/ llcpcjReDalhkFYXR8YQ==; From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= To: guix-patches@gnu.org Subject: [PATCH 0/2] Avoid cache cleanup storms Date: Tue, 16 Jul 2024 11:10:56 +0200 Message-ID: X-Mailer: git-send-email 2.45.2 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 X-Debbugs-Cc: Christopher Baines , Josselin Poiret , Ludovic Courtès , Mathieu Othacehe , Simon Tournier , Tobias Geerinckx-Rice Content-Transfer-Encoding: 8bit X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: submit Cc: =?UTF-8?q?Ludovic=20Court=C3=A8s?= X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hello! This fixes “cache cleanup storms” reported by Chris Baines whereby typically multiple ‘guix substitute’ processes would start cleaning up /var/guix/substitute/cache concurrently, leading to poor performance (in particular on build farms with many such processes running in parallel, even worse when on spinning disks). Thoughts? Ludo’. Ludovic Courtès (2): syscalls: Add ‘mode’ parameter to ‘lock-file’. cache: Avoid cache cleanup storms from concurrent processes. guix/build/syscalls.scm | 14 +++++++++----- guix/cache.scm | 27 ++++++++++++++++++--------- tests/cache.scm | 30 +++++++++++++++++++++++++++++- tests/syscalls.scm | 13 +++++++++++++ 4 files changed, 69 insertions(+), 15 deletions(-) base-commit: eb508e32d2d359c94d2cabebfe90dc32ca5dcf4f -- 2.45.2 From debbugs-submit-bounces@debbugs.gnu.org Tue Jul 16 05:18:15 2024 Received: (at 72137) by debbugs.gnu.org; 16 Jul 2024 09:18:15 +0000 Received: from localhost ([127.0.0.1]:60968 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1sTeK0-00052O-1w for submit@debbugs.gnu.org; Tue, 16 Jul 2024 05:18:15 -0400 Received: from eggs.gnu.org ([209.51.188.92]:53248) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1sTeJx-00051p-Pa for 72137@debbugs.gnu.org; Tue, 16 Jul 2024 05:18:06 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sTeHk-0003TP-I9; Tue, 16 Jul 2024 05:15:48 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:References:In-Reply-To:Date:Subject:To: From; bh=zymSuu05wUsaPkj+VZXqKU0K5ZcilZM+hbGBPPkqzNs=; b=TMxYjNXVaiIV/YSijzSU sbh6GesukxRXJILCCmwg4LXkOHnamS2s20gifJAJAWwIpYnONOrsSsczXBc/Y82tswD+4aUJgxJbA Kqx5aWfWErGuenxL5M6TCnQxXieq/dwJJv8JUZUTsgufWUnRG6lE1AdXcoNZiPUdE9DNQE6D7RNl3 C5QofR9Fn0XhdtLBgZHODp5oLv07gpUl3kbVIya58LZ9+OA2NSxlbWga4vXLyuwWvnWfBqMeLtBXc oky+u2fTskgLxis2SVw8k9Jfs42wqHWYrvVWmITca/qg9RsGDPmgSOoO5SruJN6MPgFW6gZoPxf9Y RaT0eIEJL+y5hA==; From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= To: 72137@debbugs.gnu.org Subject: [PATCH 1/2] =?UTF-8?q?syscalls:=20Add=20=E2=80=98mode=E2=80=99=20?= =?UTF-8?q?parameter=20to=20=E2=80=98lock-file=E2=80=99.?= Date: Tue, 16 Jul 2024 11:15:36 +0200 Message-ID: X-Mailer: git-send-email 2.45.2 In-Reply-To: References: MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 72137 Cc: =?UTF-8?q?Ludovic=20Court=C3=A8s?= X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) * guix/build/syscalls.scm (lock-file): Add ‘mode’ parameter and honor it. * tests/syscalls.scm ("lock-file + unlock-file"): New test. Change-Id: I113fb4a8b35dd8782b9c0991574e39a4b4393333 --- guix/build/syscalls.scm | 14 +++++++++----- tests/syscalls.scm | 13 +++++++++++++ 2 files changed, 22 insertions(+), 5 deletions(-) diff --git a/guix/build/syscalls.scm b/guix/build/syscalls.scm index 39bcffd516..2c20edf058 100644 --- a/guix/build/syscalls.scm +++ b/guix/build/syscalls.scm @@ -1398,14 +1398,18 @@ (define fcntl-flock ;; Presumably we got EAGAIN or so. (throw 'flock-error err)))))) -(define* (lock-file file #:key (wait? #t)) - "Wait and acquire an exclusive lock on FILE. Return an open port." - (let ((port (open-file file "w0"))) - (fcntl-flock port 'write-lock #:wait? wait?) +(define* (lock-file file #:optional (mode "w0") + #:key (wait? #t)) + "Wait and acquire an exclusive lock on FILE. Return an open port according +to MODE." + (let ((port (open-file file mode))) + (fcntl-flock port + (if (output-port? port) 'write-lock 'read-lock) + #:wait? wait?) port)) (define (unlock-file port) - "Unlock PORT, a port returned by 'lock-file'." + "Unlock PORT, a port returned by 'lock-file', and close it." (fcntl-flock port 'unlock) (close-port port) #t) diff --git a/tests/syscalls.scm b/tests/syscalls.scm index 7cf67c060d..13f4f11721 100644 --- a/tests/syscalls.scm +++ b/tests/syscalls.scm @@ -383,6 +383,19 @@ (define perform-container-tests? (close-port file) result))))))))) +(test-equal "lock-file + unlock-file" + 'hello + (call-with-temporary-directory + (lambda (directory) + (let* ((file (in-vicinity directory "lock")) + (out (lock-file file #:wait? #f))) + (display "hello" out) + (unlock-file out) + (let* ((in (lock-file file "r0")) + (content (read in))) + (unlock-file in) + content))))) + (test-equal "set-thread-name" "Syscall Test" (let ((name (thread-name))) -- 2.45.2 From debbugs-submit-bounces@debbugs.gnu.org Tue Jul 16 05:18:30 2024 Received: (at 72137) by debbugs.gnu.org; 16 Jul 2024 09:18:31 +0000 Received: from localhost ([127.0.0.1]:60972 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1sTeK8-00052d-5s for submit@debbugs.gnu.org; Tue, 16 Jul 2024 05:18:30 -0400 Received: from eggs.gnu.org ([209.51.188.92]:43328) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1sTeK1-00052E-T8 for 72137@debbugs.gnu.org; Tue, 16 Jul 2024 05:18:10 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sTeHm-0003Th-5t; Tue, 16 Jul 2024 05:15:50 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:References:In-Reply-To:Date:Subject:To: From; bh=fftxX9awFlRpeAR97JvG8Rfi9msKGGI6CMneubSk01o=; b=Vum5DuPdUl5LqzeQWVLi F13ZgiArr+iRHRNPKxfGUapgP+oGFwRmaUupsUd4BfXAeX4+1oS3LPE3iEAdpXAYVVf2b4R91EDni AvRMsP5zMB+/O3fqx9tUI69QzO49FjThnkBNMOFijNS4rM9G7UjeYWBZp89FlChpyKzoDt8gywHv7 fwnv9ILjTwMWsBO2/uQwEAbeX5xkIItgmPe1nFdnjk88aWKtHpwZfRYRMfUASWerY0kU6CSe7d+dG 9lYGREncTtAlqvltZkeTvWV84inNQnFzumPVjGDNS8BXnhPsbb8G0x7Zj8UB+MTNmAeTAJhwdoGPU H1NpG1X6WWG3tQ==; From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= To: 72137@debbugs.gnu.org Subject: [PATCH 2/2] cache: Avoid cache cleanup storms from concurrent processes. Date: Tue, 16 Jul 2024 11:15:37 +0200 Message-ID: <26f82a475d4ccd776c64c0a14628597dd65d8908.1721120884.git.ludo@gnu.org> X-Mailer: git-send-email 2.45.2 In-Reply-To: References: MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 X-Debbugs-Cc: Christopher Baines , Josselin Poiret , Ludovic Courtès , Mathieu Othacehe , Simon Tournier , Tobias Geerinckx-Rice Content-Transfer-Encoding: 8bit X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 72137 Cc: =?UTF-8?q?Ludovic=20Court=C3=A8s?= X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Reported by Christopher Baines . * guix/cache.scm (maybe-remove-expired-cache-entries): Define ‘expiry-port’; create it with ‘lock-file’. Change ‘last-expiry-date’ accordingly. Write timestamp straight to ‘expiry-port’. * tests/cache.scm ("maybe-remove-expired-cache-entries, cleanup needed but lock taken"): New test. Change-Id: I22441d9d2c4a339d3d3878de131864db5a0ae826 --- guix/cache.scm | 27 ++++++++++++++++++--------- tests/cache.scm | 30 +++++++++++++++++++++++++++++- 2 files changed, 47 insertions(+), 10 deletions(-) diff --git a/guix/cache.scm b/guix/cache.scm index 6a91c7d3ef..8b12312c77 100644 --- a/guix/cache.scm +++ b/guix/cache.scm @@ -1,5 +1,5 @@ ;;; GNU Guix --- Functional package management for GNU -;;; Copyright © 2013-2017, 2020-2021, 2023 Ludovic Courtès +;;; Copyright © 2013-2017, 2020-2021, 2023-2024 Ludovic Courtès ;;; Copyright © 2022 Simon Tournier ;;; ;;; This file is part of GNU Guix. @@ -19,6 +19,7 @@ (define-module (guix cache) #:use-module ((guix utils) #:select (with-atomic-file-output)) + #:autoload (guix build syscalls) (lock-file unlock-file) #:use-module (srfi srfi-19) #:use-module (srfi srfi-26) #:use-module (ice-9 match) @@ -93,13 +94,19 @@ (define* (maybe-remove-expired-cache-entries cache (define expiry-file (string-append cache "/last-expiry-cleanup")) + (define expiry-port + ;; Get exclusive access to EXPIRY-FILE to avoid "cleanup storms" where + ;; several processes would concurrently decide that time has come to clean + ;; up the same cache. 'lock-file' might throw to 'system-error' or to + ;; 'flock-error'; in either case, assume that we lost the race. + (false-if-exception + (lock-file expiry-file "a+0" #:wait? #f))) + (define last-expiry-date - (catch 'system-error - (lambda () - (or (string->number - (call-with-input-file expiry-file get-string-all)) - 0)) - (const 0))) + (if expiry-port + (or (string->number (get-string-all expiry-port)) + 0) + +inf.0)) (when (obsolete? last-expiry-date now cleanup-period) (remove-expired-cache-entries (cache-entries cache) @@ -108,8 +115,10 @@ (define* (maybe-remove-expired-cache-entries cache #:delete-entry delete-entry) (catch 'system-error (lambda () - (with-atomic-file-output expiry-file - (cute write (time-second now) <>))) + (seek expiry-port 0 SEEK_SET) + (truncate-file expiry-port 0) + (write (time-second now) expiry-port) + (unlock-file expiry-port)) (lambda args ;; ENOENT means CACHE does not exist. (unless (= ENOENT (system-error-errno args)) diff --git a/tests/cache.scm b/tests/cache.scm index d495ace2bd..e8ad083d40 100644 --- a/tests/cache.scm +++ b/tests/cache.scm @@ -1,5 +1,5 @@ ;;; GNU Guix --- Functional package management for GNU -;;; Copyright © 2017, 2020 Ludovic Courtès +;;; Copyright © 2017, 2020, 2024 Ludovic Courtès ;;; Copyright © 2022 Simon Tournier ;;; ;;; This file is part of GNU Guix. @@ -22,7 +22,9 @@ (define-module (test-cache) #:use-module (srfi srfi-1) #:use-module (srfi srfi-19) #:use-module (srfi srfi-64) + #:use-module ((guix build syscalls) #:select (lock-file)) #:use-module ((guix utils) #:select (call-with-temporary-directory)) + #:use-module ((rnrs io ports) #:select (get-string-all)) #:use-module (ice-9 match)) (test-begin "cache") @@ -75,6 +77,32 @@ (define-syntax-rule (test-cache-cleanup cache exp ...) (lambda (port) (display 0 port))))) +(let ((pid #f)) + (test-equal "maybe-remove-expired-cache-entries, cleanup needed but lock taken" + '() + (test-cache-cleanup cache + (let ((in+out (pipe))) + (match (primitive-fork) + (0 (dynamic-wind + (const #t) + (lambda () + (close-port (car in+out)) + (let ((port (lock-file + (string-append cache "/last-expiry-cleanup")))) + (display 0 port) + (display "done!\n" (cdr in+out)) + (close-port (cdr in+out)) + (sleep 100))) + (lambda () + (primitive-exit 0)))) + (n + (set! pid n) + (close-port (cdr in+out)) + (pk 'chr (get-string-all (car in+out))) + (close-port (car in+out))))))) + + (when pid (kill pid SIGKILL))) + (test-equal "maybe-remove-expired-cache-entries, empty cache" '("a" "b" "c") (test-cache-cleanup cache -- 2.45.2 From debbugs-submit-bounces@debbugs.gnu.org Tue Aug 20 18:55:41 2024 Received: (at 72137-done) by debbugs.gnu.org; 20 Aug 2024 22:55:41 +0000 Received: from localhost ([127.0.0.1]:33790 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1sgXlM-0000uk-Of for submit@debbugs.gnu.org; Tue, 20 Aug 2024 18:55:41 -0400 Received: from eggs.gnu.org ([209.51.188.92]:58600) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1sgXlK-0000uN-MV for 72137-done@debbugs.gnu.org; Tue, 20 Aug 2024 18:55:39 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sgXkW-00020C-Iz; Tue, 20 Aug 2024 18:54:48 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:Date:References:In-Reply-To:Subject:To: From; bh=ZUaZo+m7z+SrT8Cg954ywJHVnaDMwGQxYpU+4rfttT0=; b=LGKbaC3ynFGRuWEcHFjV y8O6st1rUS3s0NdID0Wh69fF7KoZm3ltguty0Co/MeCFJ/ZNEjKjqGYzlhyIOxF43MkdfaNPBMdoc Wiv1rZgvIfaaRbDxKN8ZnjZgeHqED/BxjRFy/7OHGvQLIxhuPXb8kdBvQ7uvO3C1NHh4xjs1586wI IG3lRgxSG1jP9AwMBBPTZyMzASlmcEo1Wz1ZMQCx+1DV1wuFtZNAs5NPzkaQlicc+RfECFCuRZjGV +czOBHGmZOlMA2fRTw6iN9XIaazNkuIk8e+VL6QuMCZKYRw6hbnK0XILPwx72W4UW5E2wvXUyRQo8 TOWqpxNL8mOYEA==; From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: 72137-done@debbugs.gnu.org Subject: Re: [bug#72137] [PATCH 0/2] Avoid cache cleanup storms In-Reply-To: ("Ludovic =?utf-8?Q?Court?= =?utf-8?Q?=C3=A8s=22's?= message of "Tue, 16 Jul 2024 11:10:56 +0200") References: Date: Wed, 21 Aug 2024 00:54:45 +0200 Message-ID: <87seuyst3e.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 72137-done Cc: Tobias Geerinckx-Rice , Christopher Baines , Josselin Poiret , Simon Tournier , Mathieu Othacehe X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hi, Ludovic Court=C3=A8s skribis: > This fixes =E2=80=9Ccache cleanup storms=E2=80=9D reported by Chris Baine= s whereby > typically multiple =E2=80=98guix substitute=E2=80=99 processes would star= t cleaning > up /var/guix/substitute/cache concurrently, leading to poor performance > (in particular on build farms with many such processes running in > parallel, even worse when on spinning disks). > > Thoughts? > > Ludo=E2=80=99. > > Ludovic Court=C3=A8s (2): > syscalls: Add =E2=80=98mode=E2=80=99 parameter to =E2=80=98lock-file=E2= =80=99. > cache: Avoid cache cleanup storms from concurrent processes. I went ahead and pushed it as d921c742b774a9f0a016f3db6442d5c58a330c92. Ludo=E2=80=99. From unknown Sun Aug 10 07:34:20 2025 Received: (at fakecontrol) by fakecontrolmessage; To: internal_control@debbugs.gnu.org From: Debbugs Internal Request Subject: Internal Control Message-Id: bug archived. Date: Wed, 18 Sep 2024 11:24:09 +0000 User-Agent: Fakemail v42.6.9 # This is a fake control message. # # The action: # bug archived. thanks # This fakemail brought to you by your local debbugs # administrator