From unknown Sun Jun 22 08:01:49 2025 X-Loop: help-debbugs@gnu.org Subject: bug#70877: guix-daemon fails to copy 4+GB file to store Resent-From: Ricardo Wurmus Original-Sender: "Debbugs-submit" Resent-CC: ludo@gnu.org, bug-guix@gnu.org Resent-Date: Sat, 11 May 2024 10:54:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 70877 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: 70877@debbugs.gnu.org Cc: ludo@gnu.org X-Debbugs-Original-To: bug-guix@gnu.org X-Debbugs-Original-Xcc: ludo@gnu.org Received: via spool by submit@debbugs.gnu.org id=B.17154247899460 (code B ref -1); Sat, 11 May 2024 10:54:01 +0000 Received: (at submit) by debbugs.gnu.org; 11 May 2024 10:53:09 +0000 Received: from localhost ([127.0.0.1]:47688 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1s5kLk-0002SW-Hu for submit@debbugs.gnu.org; Sat, 11 May 2024 06:53:08 -0400 Received: from lists.gnu.org ([209.51.188.17]:42620) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1s5kLi-0002SQ-6i for submit@debbugs.gnu.org; Sat, 11 May 2024 06:53:07 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1s5kLi-0004jv-Go for bug-guix@gnu.org; Sat, 11 May 2024 06:53:06 -0400 Received: from sender4-of-o51.zoho.com ([136.143.188.51]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1s5kLg-0002SY-Gn for bug-guix@gnu.org; Sat, 11 May 2024 06:53:06 -0400 ARC-Seal: i=1; a=rsa-sha256; t=1715424779; cv=none; d=zohomail.com; s=zohoarc; b=j04xfDC+Am47I5gycTn6agPY1Oa0wSWfVwsKoTZYZCx7hhf9bcmOCjsVmF9Eh7vtE2n1lcufrB8G6V96PZZPS8yUkqs1F/HrcZP/OUYpoSHa/GM5OR1uqcqoYw0+xpqmAhu1QMLBUik5sO3usNJZvClfiMMRJ5yGA6kLtmFhbRA= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1715424779; h=Content-Type:Date:Date:From:From:MIME-Version:Message-ID:Subject:Subject:To:To:Message-Id:Reply-To:Cc; bh=t9J9EHFuaKrYrMFPWsIYaTbSQKBJj55dZTfJaFBTTUk=; b=HaqO77xNk6t0bOJU7977dkeiUHrUCUy4lQUU7jG25Qwybfzd5Tq45M0vmHOa5cXkFvGSOFYeErQ08lzOa6lik+TQ+N7DksMVE43hqD82ZMAjba/ihN4cxcILxTAq8w9ArVrXzanhOtw93PvBiKUlcvpwlwAJyN6PUJdCMcJHDtM= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass header.i=elephly.net; spf=pass smtp.mailfrom=rekado@elephly.net; dmarc=pass header.from= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1715424779; s=zoho; d=elephly.net; i=rekado@elephly.net; h=From:From:To:To:Subject:Subject:Date:Date:Message-ID:MIME-Version:Content-Type:Message-Id:Reply-To:Cc; bh=t9J9EHFuaKrYrMFPWsIYaTbSQKBJj55dZTfJaFBTTUk=; b=gvw3mk8unx/8yp05/IUlDrRh+hJ+c6HRjmoK0GxdQhP1VW8ecBVQsJQKYi4yx1xB Y5L9Mo2zSuUbwG7QbKTHVhZICjtdEJn/5+Tv6MzjtdyfcWsFuO+HZBHzCuqrOLk4IIK 0ttw+TLbGF7me8fNj/z841rGuQvtLuH43A7J6Mc4= Received: by mx.zohomail.com with SMTPS id 1715424777121760.7577937280839; Sat, 11 May 2024 03:52:57 -0700 (PDT) From: Ricardo Wurmus User-Agent: mu4e 1.12.4; emacs 29.3 Date: Sat, 11 May 2024 12:52:53 +0200 Message-ID: <87msowpqm2.fsf@elephly.net> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" X-ZohoMailClient: External X-Zoho-Virus-Status: 1 Received-SPF: pass client-ip=136.143.188.51; envelope-from=rekado@elephly.net; helo=sender4-of-o51.zoho.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-Spam-Score: -1.3 (-) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.3 (--) --=-=-= Content-Type: text/plain The guix-daemon's libutil/util.cc uses copy_file_range to copy a downloaded file into the store. copy_file_range fails on files larger than 4GB with an error like this: guix build: error: short write in copy_file_range `15' to `16': No such file or directory The man page for copy_file_range says that it could return EFBIG when the range exceeds the maximum range. The daemon code does not check any limits and will attempt to copy the whole file. I believe our code ought to check the value of st.size and fall back to a boring copy if it exceeds some "reasonable" value. This is where copy_file_range is used: https://git.savannah.gnu.org/cgit/guix.git/tree/nix/libutil/util.cc#n382 Here is a little reproducer: --=-=-= Content-Type: text/plain Content-Disposition: inline; filename=bug.scm (use-modules (guix download) (guix packages) (guix build-system trivial)) (package (name "chungus") (version "1") (source (origin (method url-fetch) (uri "http://localhost:1111/chungus") (sha256 (base32 "0nx67d4ls2nfwcfdmg81vf240z6lpwpdqypssr1wzn3hyz4szci4")))) (build-system trivial-build-system) (home-page "") (synopsis "") (description "") (license #f)) --=-=-= Content-Type: text/plain --8<---------------cut here---------------start------------->8--- # generate a big file dd bs=1M count=4096 if=/dev/zero of=/tmp/chungus # serve it guix shell woof -- woof -i 127.0.0.1 -p 1111 -c 1 /tmp/chungus # build the source derivation guix build --no-grafts -Sf bug.scm # observe the error # guix build: error: short write in copy_file_range `15' to `16': No such file or directory --8<---------------cut here---------------end--------------->8--- -- Ricardo --=-=-=-- From unknown Sun Jun 22 08:01:49 2025 X-Loop: help-debbugs@gnu.org Subject: bug#70877: guix-daemon fails to copy 4+GB file to store Resent-From: Efraim Flashner Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Sun, 12 May 2024 07:14:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 70877 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: Ricardo Wurmus Cc: ludo@gnu.org, 70877@debbugs.gnu.org Received: via spool by 70877-submit@debbugs.gnu.org id=B70877.171549800416319 (code B ref 70877); Sun, 12 May 2024 07:14:01 +0000 Received: (at 70877) by debbugs.gnu.org; 12 May 2024 07:13:24 +0000 Received: from localhost ([127.0.0.1]:52880 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1s63Oe-0004F9-5G for submit@debbugs.gnu.org; Sun, 12 May 2024 03:13:24 -0400 Received: from mail-wm1-f45.google.com ([209.85.128.45]:45438) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1s63Oa-0004F3-Gj for 70877@debbugs.gnu.org; Sun, 12 May 2024 03:13:21 -0400 Received: by mail-wm1-f45.google.com with SMTP id 5b1f17b1804b1-41b79451145so24214585e9.3 for <70877@debbugs.gnu.org>; Sun, 12 May 2024 00:13:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1715497934; x=1716102734; darn=debbugs.gnu.org; h=in-reply-to:content-disposition:mime-version:references :mail-followup-to:message-id:subject:cc:to:from:date:sender:from:to :cc:subject:date:message-id:reply-to; bh=sy+4crbrCA7izAVrXQkoruxwqyozKistZX5a2QCRSxI=; b=JjCPMZNwjJqS3Afze1zTHtSWATAKLS367VwXLK3my6LgVdxFPIv/1TbO13E247J9hH P7qpPDzGvhLdDsgPq1h82Q6TeWaTIPvKOHAd18kUozpRZixnRaP5OxB4rRPPMgaDAh3p vF6ptC0DenEDsv6YJ+jdidrprkz/ipBoxxPU5ocbdGHmKpJMJ0E5R94/UZvocznjo8Pm 90YDVQCt5PDp3EyhpNYTN9ypyhtaxCRoqvem/EBKOF1Cksyz4yFjfZ98Vv9Nz0vLoj9V WyGKs74O96Y4NQVOnjWQ6kIIqtry1Y8kuIDjfY1TfTIXMRAFb/4zf/uxBhkdqywSU/NB K/+w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715497934; x=1716102734; h=in-reply-to:content-disposition:mime-version:references :mail-followup-to:message-id:subject:cc:to:from:date:sender :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=sy+4crbrCA7izAVrXQkoruxwqyozKistZX5a2QCRSxI=; b=b+dXRY6rc8o7F696HGKpvzcjlwk1bG1+Y6B/2FyUzBAEENRL8nhhfgBhe5p0buxmo7 akRizSiD2P/FCSHhoLtb8v41y3g7nUMQLGiWp8NQdgoUD2xxxwqqYM9QJY3vTPfU58D/ zvW9vIwt0sos5hoBFWXirst6iAz2CwJBRz6OODjgGwjWhIA2HF87iYuugpbTD7JdR3jf noelf52kkr0ntVMBVpaNnFcCO5CE6xLwqsNvgOPvuZ0KtuXhxdDCfWDvyPvef/p6njpN O3mhPLevMaov9TjvPD6z34vRcd7wr1mwD/MNI7tT1YtjFVNa63CEFs6fyXt0ab0+WdUt kJNQ== X-Gm-Message-State: AOJu0YzVOl+jGcbL8APoh4vlZ3bTU3JEAWjB0gOffhH5ISJcP80r9x4C kbnPKIMrjNPsS8uNlajG/8/kLHzRswjOMeDB+UjFBnsi1dogljB4 X-Google-Smtp-Source: AGHT+IHj5W6k25aNWZ9kgnBKAISiAY9lCVmxoVhzrr13l/QqGmcQspmGoW20dUyBSMn0HCOA1I3BKQ== X-Received: by 2002:a05:600c:4fcd:b0:41c:2931:e670 with SMTP id 5b1f17b1804b1-41fea9320acmr47614675e9.2.1715497933376; Sun, 12 May 2024 00:12:13 -0700 (PDT) Received: from localhost ([37.46.46.122]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-41fccbe8f9bsm119694925e9.6.2024.05.12.00.12.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 12 May 2024 00:12:12 -0700 (PDT) Date: Sun, 12 May 2024 10:12:11 +0300 From: Efraim Flashner Message-ID: Mail-Followup-To: Efraim Flashner , Ricardo Wurmus , 70877@debbugs.gnu.org, ludo@gnu.org References: <87msowpqm2.fsf@elephly.net> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="WY049zgeJ+b4bTKr" Content-Disposition: inline In-Reply-To: <87msowpqm2.fsf@elephly.net> X-PGP-Key-ID: 0x41AAE7DCCA3D8351 X-PGP-Key: https://flashner.co.il/~efraim/efraim_flashner.asc X-PGP-Fingerprint: A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351 X-Spam-Score: 0.3 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.7 (/) --WY049zgeJ+b4bTKr Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sat, May 11, 2024 at 12:52:53PM +0200, Ricardo Wurmus wrote: > The guix-daemon's libutil/util.cc uses copy_file_range to copy a > downloaded file into the store. copy_file_range fails on files larger > than 4GB with an error like this: >=20 > guix build: error: short write in copy_file_range `15' to `16': No su= ch file or directory >=20 > The man page for copy_file_range says that it could return EFBIG when > the range exceeds the maximum range. The daemon code does not check any > limits and will attempt to copy the whole file. >=20 > I believe our code ought to check the value of st.size and fall back to > a boring copy if it exceeds some "reasonable" value. >=20 > This is where copy_file_range is used: > https://git.savannah.gnu.org/cgit/guix.git/tree/nix/libutil/util.cc#n382 >=20 > Here is a little reproducer: >=20 > (use-modules (guix download) > (guix packages) > (guix build-system trivial)) >=20 > (package > (name "chungus") > (version "1") > (source > (origin > (method url-fetch) > (uri "http://localhost:1111/chungus") > (sha256 > (base32 "0nx67d4ls2nfwcfdmg81vf240z6lpwpdqypssr1wzn3hyz4szci4")))) > (build-system trivial-build-system) > (home-page "") > (synopsis "") > (description "") > (license #f)) >=20 > --8<---------------cut here---------------start------------->8--- > # generate a big file > dd bs=3D1M count=3D4096 if=3D/dev/zero of=3D/tmp/chungus > # serve it > guix shell woof -- woof -i 127.0.0.1 -p 1111 -c 1 /tmp/chungus > # build the source derivation > guix build --no-grafts -Sf bug.scm > # observe the error > # guix build: error: short write in copy_file_range `15' to `16': No such= file or directory > --8<---------------cut here---------------end--------------->8--- >=20 This sounds like a similar failure to bug 65714 that I ran into with guix copy, but I wasn't able to diagnose it. https://issues.guix.gnu.org/65714 --=20 Efraim Flashner =D7=A8=D7=A0=D7=A9=D7=9C=D7=A4 = =D7=9D=D7=99=D7=A8=D7=A4=D7=90 GPG key =3D A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351 Confidentiality cannot be guaranteed on emails sent or received unencrypted --WY049zgeJ+b4bTKr Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEEoov0DD5VE3JmLRT3Qarn3Mo9g1EFAmZAa8gACgkQQarn3Mo9 g1GBJw//VqKkLgiTUONuYFYrM7TiCSmxrt3wLT3GkxxF+EDEyaf6Lye9W3uSFYek Moy/5kWSSvjfyiJA6UCAGelcao90ogylfH88oHYt6zqa4h5ArwimAmYTSmrRr3b6 Pvr6QypDT1tflwe5x2R+YbigFswCgXYPXtM85IDwQzOcwffl5rePz3HB9YKCJCVx +54xhbY8LL7aBmG68jyXS4G1XsIGWnzBVret7hLZ++i/HTwC9lX2Kn9lSXUSfPev hiS8290J3Vz7pES5jp2/VsTZpw2zBnhNOwK+rbEpQMcOYHycfTjDTlZ4x2POPXSg HpcMIsyo34nOWsVtYYbMl3wYSukF3mIkZx2lpb7ZNNsOTvGqX3IIgADslqoiaCtB ENaXGLO3BLJc3AXUEwrOe6a5b0APdDy4RDNqZXZYP0RMDR5M13YREepazHcb0If7 IExQp3S9Eo9DA6Jnbb5wcNAVHta4ytvE3qYrYMp4+J0aYFETlLyyFQqbTpPiqHxN Gp/NLT8VravsGf9r/UJYZHEz12wC4XrSHR1tvw/yyj0T4cARgv27kU/8myGPXWtC ys90UhU2hs2MoCcgtW5cML2epH/+5ycgXmPgWM9pS5ukMDKktcPnJCwMUO/2DawC UocY2aWvPStnddMkCfdT+QqaKXukWYENo+QAk1fEBNvplJMDAX8= =ZUuL -----END PGP SIGNATURE----- --WY049zgeJ+b4bTKr-- From debbugs-submit-bounces@debbugs.gnu.org Mon May 13 05:05:21 2024 Received: (at control) by debbugs.gnu.org; 13 May 2024 09:05:21 +0000 Received: from localhost ([127.0.0.1]:59698 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1s6RcX-0004lp-6R for submit@debbugs.gnu.org; Mon, 13 May 2024 05:05:21 -0400 Received: from eggs.gnu.org ([209.51.188.92]:59830) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1s6RcV-0004le-4T for control@debbugs.gnu.org; Mon, 13 May 2024 05:05:19 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1s6RcO-0001jQ-Rq for control@debbugs.gnu.org; Mon, 13 May 2024 05:05:13 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-version:Subject:From:To:Date:in-reply-to: references; bh=wx8oKnmeowrmSZgQBAzpFdGVorEPUVyqKrx+3Zheiz8=; b=rwlhJbtNHv6Lo8 Q0SdDdkYdpw9paafvJNE1yBttvXevk++BR2DWdubJo8sH8I+w2nantj4FbxnAzG7h4Q0gL4hu+bXV bhKy8I5MtldB02LJ7nX9nZrDqpZ1l2vh6fL/RCrO0YQKcmNr2fCAXuqkwBE2X/Cac6/lskljYf5lw 1yreE9g8zjojZcdRnXxaf4IMFfmqVc9O5T1Cbz/4cxQUddF20lmAmvmZfdUJaw6Bt7+4GRLj5cpfy UAnTNJfsp43jpkxXBTUmGaO419R2W7oEQ1ijgPlATKK35oJbGYs6HY+v8XDm5xmiX0ON9G4RV94Od cBKuLGFkuvN93QDsepEQ==; Date: Mon, 13 May 2024 11:05:09 +0200 Message-Id: <87frumdquy.fsf@gnu.org> To: control@debbugs.gnu.org From: =?utf-8?Q?Ludovic_Court=C3=A8s?= Subject: control message for bug #70877 MIME-version: 1.0 Content-type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: control X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) severity 70877 important quit From unknown Sun Jun 22 08:01:49 2025 X-Loop: help-debbugs@gnu.org Subject: bug#70877: guix-daemon fails to copy 4+GB file to store Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Mon, 13 May 2024 10:11:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 70877 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: Ricardo Wurmus Cc: 70877@debbugs.gnu.org Received: via spool by 70877-submit@debbugs.gnu.org id=B70877.171559504731463 (code B ref 70877); Mon, 13 May 2024 10:11:01 +0000 Received: (at 70877) by debbugs.gnu.org; 13 May 2024 10:10:47 +0000 Received: from localhost ([127.0.0.1]:60017 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1s6Sdq-0008BP-UP for submit@debbugs.gnu.org; Mon, 13 May 2024 06:10:47 -0400 Received: from eggs.gnu.org ([209.51.188.92]:50268) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1s6Sdo-0008BF-Ld for 70877@debbugs.gnu.org; Mon, 13 May 2024 06:10:45 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1s6Sdi-000665-1c; Mon, 13 May 2024 06:10:38 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:Date:References:In-Reply-To:Subject:To: From; bh=MA8XKeP6+jP6u3AJSD+5h+T8XN14wKaetTkznCASlQ4=; b=frsWhmhwbaFMfE1wcpNc lurQLXmmek3Y7865LyNEe2JcGraREgVRRrT3oJUBvPiaQomKTaIjVBKVlgfh23Yw4MMpeE/PYsZpJ g8uun1CCmpgVmIxtpUROrG6pXCEPI4yQXBdWp5HgQcUQhYnTbe1ZTD83vuj8FE8241U7/ghGRJ4n5 DY6fEZxw701QQTfZsI/4BTat/Ci/WLZa9vxS6g5jAF2lDACLaK45RBk9Os/5aCFU3FAFdneCWkV1E CsNR4uVx7VXpzqJftaJKT0Iq5kNuUsJX/svuqN0E5NeCJixBHUpyOQ/hzz+NOnC6p2acmVmiBB7P7 xZ+ELhCQf3eNvQ==; From: Ludovic =?UTF-8?Q?Court=C3=A8s?= In-Reply-To: <87msowpqm2.fsf@elephly.net> (Ricardo Wurmus's message of "Sat, 11 May 2024 12:52:53 +0200") References: <87msowpqm2.fsf@elephly.net> Date: Mon, 13 May 2024 12:10:34 +0200 Message-ID: <87v83ic99h.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" X-Spam-Score: -2.3 (--) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hi, Thanks for the bug report and nice reproducer! Ricardo Wurmus skribis: > The guix-daemon's libutil/util.cc uses copy_file_range to copy a > downloaded file into the store. copy_file_range fails on files larger > than 4GB with an error like this: > > guix build: error: short write in copy_file_range `15' to `16': No su= ch file or directory > > The man page for copy_file_range says that it could return EFBIG when > the range exceeds the maximum range. The daemon code does not check any > limits and will attempt to copy the whole file. > > I believe our code ought to check the value of st.size and fall back to > a boring copy if it exceeds some "reasonable" value. The goal leading to this error message looks like this: copy_file_range(15, NULL, 16, NULL, 4294967297, 0) =3D 2147479552 =E2=80=A6 which is precisely 2 GiB - 4 KiB. Reading the man page, it=E2=80=99s entirely fine: like =E2=80=98write=E2=80= =99, =E2=80=98copy_file_range=E2=80=99 might copy less than asked for, so it=E2= =80=99s really a mistake of mine to assume that short writes can=E2=80=99t happen. Presumab= ly there=E2=80=99s an internal limit here we=E2=80=99re reaching that explains= why it won=E2=80=99t copy more than 2 GiB at once. With the following change, we get: newfstatat(15, "", {st_mode=3DS_IFREG|0644, st_size=3D4294967297, ...}, A= T_EMPTY_PATH) =3D 0 copy_file_range(15, NULL, 16, NULL, 4294967297, 0) =3D 2147479552 copy_file_range(15, NULL, 16, NULL, 2147487745, 0) =3D 2147479552 copy_file_range(15, NULL, 16, NULL, 8193, 0) =3D 8193 fchown(16, 30001, 30000) =3D 0 Could you confirm that it works for you? Thanks, Ludo=E2=80=99. --=-=-= Content-Type: text/x-patch; charset=utf-8 Content-Disposition: inline; filename=0001-daemon-Loop-over-copy_file_range-upon-short-writes.patch Content-Transfer-Encoding: quoted-printable Content-Description: the patch >From efd9f3383756df9959651125c0f2e2e769630851 Mon Sep 17 00:00:00 2001 Message-ID: From: =3D?UTF-8?q?Ludovic=3D20Court=3DC3=3DA8s?=3D Date: Mon, 13 May 2024 12:02:30 +0200 Subject: [PATCH] =3D?UTF-8?q?daemon:=3D20Loop=3D20over=3D20=3DE2=3D80=3D98c= opy=3D5Ffile=3D5F?=3D =3D?UTF-8?q?range=3DE2=3D80=3D99=3D20upon=3D20short=3D20writes.?=3D MIME-Version: 1.0 Content-Type: text/plain; charset=3DUTF-8 Content-Transfer-Encoding: 8bit Fixes . * nix/libutil/util.cc (copyFile): Loop over =E2=80=98copy_file_range=E2=80= =99 instead of throwing upon short write. Reported-by: Ricardo Wurmus Change-Id: Id7b8a65ea59006c2d91bc23732309a68665b9ca0 --- nix/libutil/util.cc | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/nix/libutil/util.cc b/nix/libutil/util.cc index 578d6572934..3206dea11b1 100644 --- a/nix/libutil/util.cc +++ b/nix/libutil/util.cc @@ -397,9 +397,14 @@ static void copyFile(int sourceFd, int destinationFd) } else { if (result < 0) throw SysError(format("copy_file_range `%1%' to `%2%'") % sourceFd % = destinationFd); - if (result < st.st_size) - throw SysError(format("short write in copy_file_range `%1%' to `%2%'") - % sourceFd % destinationFd); + + /* If 'copy_file_range' copied less than requested, try again. */ + for (ssize_t copied =3D result; copied < st.st_size; copied +=3D result) { + result =3D copy_file_range(sourceFd, NULL, destinationFd, NULL, + st.st_size - copied, 0); + if (result < 0) + throw SysError(format("copy_file_range `%1%' to `%2%'") % sourceFd % des= tinationFd); + } } } =20 base-commit: 89cd778f6a45cd9b43a4dc1f236dcd0a87af955c --=20 2.41.0 --=-=-=-- From unknown Sun Jun 22 08:01:49 2025 X-Loop: help-debbugs@gnu.org Subject: bug#70877: guix-daemon fails to copy 4+GB file to store Resent-From: Ricardo Wurmus Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Mon, 13 May 2024 12:11:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 70877 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: Ludovic =?UTF-8?Q?Court=C3=A8s?= Cc: 70877@debbugs.gnu.org Received: via spool by 70877-submit@debbugs.gnu.org id=B70877.171560220725044 (code B ref 70877); Mon, 13 May 2024 12:11:02 +0000 Received: (at 70877) by debbugs.gnu.org; 13 May 2024 12:10:07 +0000 Received: from localhost ([127.0.0.1]:60590 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1s6UVL-0006Vs-00 for submit@debbugs.gnu.org; Mon, 13 May 2024 08:10:07 -0400 Received: from sender4-of-o51.zoho.com ([136.143.188.51]:21175) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1s6UVG-0006VZ-Vs for 70877@debbugs.gnu.org; Mon, 13 May 2024 08:10:06 -0400 ARC-Seal: i=1; a=rsa-sha256; t=1715602192; cv=none; d=zohomail.com; s=zohoarc; b=V7YTYCKJ4TrOkZGrnDrsrwzBTc7CMYia5fAGthSVqAp2IKUue00EdjckoVVxtJNw7UPK0TNuEy1hYPZA9g3nmyGq1unDBS2NJ9si3IcUTuhCfsahfoL/JBN8iRNtpWB8HmbraLZe9ayWF6AcvPA8n1xyXjESUoaKyIBV+ZlHC60= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1715602192; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:MIME-Version:Message-ID:References:Subject:Subject:To:To:Message-Id:Reply-To; bh=fCZVBBDqS1tbu9an6j1PdvTaceOGl55XVlhASdBm9ao=; b=C/bzwsqmh8W2IvPoBKJininXnWhI0PdnE1k4b6/27pgisRD5YeCaQwSH6XT2FDBfHAXBbLrqRbEgPd+HSKmADsmip3i6AI4x/NJWtTK0y/p3oJR+idtHF9JzUUE1EANISq3O3MGXL8qBWc2vHX+3IsFsjJGjeFSLC/5igoHImwo= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass header.i=elephly.net; spf=pass smtp.mailfrom=rekado@elephly.net; dmarc=pass header.from= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1715602192; s=zoho; d=elephly.net; i=rekado@elephly.net; h=From:From:To:To:Cc:Cc:Subject:Subject:In-Reply-To:References:Date:Date:Message-ID:MIME-Version:Content-Type:Content-Transfer-Encoding:Message-Id:Reply-To; bh=fCZVBBDqS1tbu9an6j1PdvTaceOGl55XVlhASdBm9ao=; b=E0zstX+Qv3J6nA1B+Ksy+XlA7ZfRY920iXFXJPhR7PX0g82ytOqhTjsdUIuLG/+a Qg1HNcnaBwKqRzEEVFFY4O8xJD5SiRj8LLO7rLNzX8VVAPlyz4eTzvi/XJ6cSAbh+4N dVkfudGw6YlebYbipRTmuxdYPWFPdC98+OOWIpm0= Received: by mx.zohomail.com with SMTPS id 1715602191172316.70731816187197; Mon, 13 May 2024 05:09:51 -0700 (PDT) From: Ricardo Wurmus In-Reply-To: <87v83ic99h.fsf@gnu.org> ("Ludovic =?UTF-8?Q?Court=C3=A8s?="'s message of "Mon, 13 May 2024 12:10:34 +0200") References: <87msowpqm2.fsf@elephly.net> <87v83ic99h.fsf@gnu.org> User-Agent: mu4e 1.12.4; emacs 29.3 Date: Mon, 13 May 2024 14:09:47 +0200 Message-ID: <87wmnxoqus.fsf@elephly.net> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-ZohoMailClient: External X-Spam-Score: 0.0 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Ludovic Court=C3=A8s writes: > Could you confirm that it works for you? I've applied this locally, started the new daemon, and used it to build the 4+GB source code derivation of a big package that used to fail before. It works now. Thank you! --=20 Ricardo From unknown Sun Jun 22 08:01:49 2025 X-Loop: help-debbugs@gnu.org Subject: bug#70877: guix-daemon fails to copy 4+GB file to store Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Mon, 13 May 2024 14:35:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 70877 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: Ricardo Wurmus Cc: 70877@debbugs.gnu.org Received: via spool by 70877-submit@debbugs.gnu.org id=B70877.171561089019634 (code B ref 70877); Mon, 13 May 2024 14:35:02 +0000 Received: (at 70877) by debbugs.gnu.org; 13 May 2024 14:34:50 +0000 Received: from localhost ([127.0.0.1]:33072 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1s6WlN-00056c-2g for submit@debbugs.gnu.org; Mon, 13 May 2024 10:34:50 -0400 Received: from eggs.gnu.org ([209.51.188.92]:59172) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1s6WlI-00056W-4T for 70877@debbugs.gnu.org; Mon, 13 May 2024 10:34:47 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1s6Wl9-0002wC-1Y; Mon, 13 May 2024 10:34:35 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:Date:References:In-Reply-To:Subject:To: From; bh=fSi7TlMtlI1wArdjQ57lRW2JuttJDe2LTdbQ5S1FaoI=; b=hYdvTAh/bwcWDRPTStAC uMEAldQSyXhUbI2dNou2D0TewJU9eSN04FYuRXRW8vSah7qjMrncmL0fwRGB/nbUzcRP7+jMEc7uP 6ltqnAgzqsL7gSnPX9v1GalvGoG+nJiUIT3GAUp4x04MYxxizt1wU48/GqHiLRo3H+ZTwdVV9uGMv 7dU/BMWu5wx/WVfqT5FPZSvMNn2xap2F4F0fWivuG9AL2YJaiK/YEFrd5qgOLb+auzUrE+g8ngDxH bb77fpRQK4YHDhpA2JIpm7prY8JmkvgTK3Hhvu14gIJdD4uIJ6ODtBiywcGoewnwXhRt7nULB1PhT Qr41VjTLdXFZMQ==; From: Ludovic =?UTF-8?Q?Court=C3=A8s?= In-Reply-To: <87wmnxoqus.fsf@elephly.net> (Ricardo Wurmus's message of "Mon, 13 May 2024 14:09:47 +0200") References: <87msowpqm2.fsf@elephly.net> <87v83ic99h.fsf@gnu.org> <87wmnxoqus.fsf@elephly.net> Date: Mon, 13 May 2024 16:34:26 +0200 Message-ID: <874jb1aih9.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Ricardo Wurmus skribis: > Ludovic Court=C3=A8s writes: > >> Could you confirm that it works for you? > > I've applied this locally, started the new daemon, and used it to build > the 4+GB source code derivation of a big package that used to fail > before. It works now. Thank you! Pushed as 7757fdd491862fa5c33f1f894503346b89898a01. I=E2=80=99ll update the =E2=80=98guix=E2=80=99 package to make the fix avai= lable. Thanks for testing! Ludo=E2=80=99. From unknown Sun Jun 22 08:01:49 2025 MIME-Version: 1.0 X-Mailer: MIME-tools 5.505 (Entity 5.505) X-Loop: help-debbugs@gnu.org From: help-debbugs@gnu.org (GNU bug Tracking System) To: Ricardo Wurmus Subject: bug#70877: closed (Re: bug#70877: guix-daemon fails to copy 4+GB file to store) Message-ID: References: <87ikzh7k95.fsf@gnu.org> <87msowpqm2.fsf@elephly.net> X-Gnu-PR-Message: they-closed 70877 X-Gnu-PR-Package: guix Reply-To: 70877@debbugs.gnu.org Date: Mon, 13 May 2024 16:25:02 +0000 Content-Type: multipart/mixed; boundary="----------=_1715617502-13286-1" This is a multi-part message in MIME format... ------------=_1715617502-13286-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Your bug report #70877: guix-daemon fails to copy 4+GB file to store which was filed against the guix package, has been closed. The explanation is attached below, along with your original report. If you require more details, please reply to 70877@debbugs.gnu.org. --=20 70877: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D70877 GNU Bug Tracking System Contact help-debbugs@gnu.org with problems ------------=_1715617502-13286-1 Content-Type: message/rfc822 Content-Disposition: inline Content-Transfer-Encoding: 7bit Received: (at 70877-done) by debbugs.gnu.org; 13 May 2024 16:24:38 +0000 Received: from localhost ([127.0.0.1]:33578 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1s6YTe-0003Rz-DG for submit@debbugs.gnu.org; Mon, 13 May 2024 12:24:38 -0400 Received: from eggs.gnu.org ([209.51.188.92]:44454) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1s6YTY-0003Rr-Tp for 70877-done@debbugs.gnu.org; Mon, 13 May 2024 12:24:36 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1s6YTR-0008Sv-Qz; Mon, 13 May 2024 12:24:25 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:Date:References:In-Reply-To:Subject:To: From; bh=W7ZUJYOeQHUiKd04OJPLtllRi0XjLyu+rKnkfthbufQ=; b=JAVXXxgfvqXJmY3HTPIa k6NBu1ix9HLtS/eGCuWWFzyXYK6HLCcgY/ME0AYfuGPtUa46MxJBqlD5rUfnLCFwkFDhEVPkFCgpH f4CqShNBHh8aUym8zp3cr1Y3Dlzc7Z82xVaaQH1SXl47V0TIa2n3xVQVH4l1LIJcQMBOCHLA886tY BqQVnYQ1hO8+eOCZN3T++LE+WFbJ5t01EDKYjSo2Duh+LKJ0hvIH+Txzblu/5KBJWqE7uhkzX4RS2 zEg80PtVTiPolylmX6r9cLKKbVKLKMCkeviwVbf3+ixoteBUxFlo/62EVXjzpSS96vzTDhZJ2r4rW 2K1+GGd6ONj2JA==; From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Ricardo Wurmus Subject: Re: bug#70877: guix-daemon fails to copy 4+GB file to store In-Reply-To: <874jb1aih9.fsf@gnu.org> ("Ludovic =?utf-8?Q?Court=C3=A8s=22'?= =?utf-8?Q?s?= message of "Mon, 13 May 2024 16:34:26 +0200") References: <87msowpqm2.fsf@elephly.net> <87v83ic99h.fsf@gnu.org> <87wmnxoqus.fsf@elephly.net> <874jb1aih9.fsf@gnu.org> Date: Mon, 13 May 2024 18:24:22 +0200 Message-ID: <87ikzh7k95.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 70877-done Cc: 70877-done@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Ludovic Court=C3=A8s skribis: > Pushed as 7757fdd491862fa5c33f1f894503346b89898a01. > > I=E2=80=99ll update the =E2=80=98guix=E2=80=99 package to make the fix av= ailable. Done in 58be9a79e2862d5fa9842d73f498ce2e5442b9ce. Ludo'. ------------=_1715617502-13286-1 Content-Type: message/rfc822 Content-Disposition: inline Content-Transfer-Encoding: 7bit Received: (at submit) by debbugs.gnu.org; 11 May 2024 10:53:09 +0000 Received: from localhost ([127.0.0.1]:47688 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1s5kLk-0002SW-Hu for submit@debbugs.gnu.org; Sat, 11 May 2024 06:53:08 -0400 Received: from lists.gnu.org ([209.51.188.17]:42620) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1s5kLi-0002SQ-6i for submit@debbugs.gnu.org; Sat, 11 May 2024 06:53:07 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1s5kLi-0004jv-Go for bug-guix@gnu.org; Sat, 11 May 2024 06:53:06 -0400 Received: from sender4-of-o51.zoho.com ([136.143.188.51]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1s5kLg-0002SY-Gn for bug-guix@gnu.org; Sat, 11 May 2024 06:53:06 -0400 ARC-Seal: i=1; a=rsa-sha256; t=1715424779; cv=none; d=zohomail.com; s=zohoarc; b=j04xfDC+Am47I5gycTn6agPY1Oa0wSWfVwsKoTZYZCx7hhf9bcmOCjsVmF9Eh7vtE2n1lcufrB8G6V96PZZPS8yUkqs1F/HrcZP/OUYpoSHa/GM5OR1uqcqoYw0+xpqmAhu1QMLBUik5sO3usNJZvClfiMMRJ5yGA6kLtmFhbRA= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1715424779; h=Content-Type:Date:Date:From:From:MIME-Version:Message-ID:Subject:Subject:To:To:Message-Id:Reply-To:Cc; bh=t9J9EHFuaKrYrMFPWsIYaTbSQKBJj55dZTfJaFBTTUk=; b=HaqO77xNk6t0bOJU7977dkeiUHrUCUy4lQUU7jG25Qwybfzd5Tq45M0vmHOa5cXkFvGSOFYeErQ08lzOa6lik+TQ+N7DksMVE43hqD82ZMAjba/ihN4cxcILxTAq8w9ArVrXzanhOtw93PvBiKUlcvpwlwAJyN6PUJdCMcJHDtM= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass header.i=elephly.net; spf=pass smtp.mailfrom=rekado@elephly.net; dmarc=pass header.from= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1715424779; s=zoho; d=elephly.net; i=rekado@elephly.net; h=From:From:To:To:Subject:Subject:Date:Date:Message-ID:MIME-Version:Content-Type:Message-Id:Reply-To:Cc; bh=t9J9EHFuaKrYrMFPWsIYaTbSQKBJj55dZTfJaFBTTUk=; b=gvw3mk8unx/8yp05/IUlDrRh+hJ+c6HRjmoK0GxdQhP1VW8ecBVQsJQKYi4yx1xB Y5L9Mo2zSuUbwG7QbKTHVhZICjtdEJn/5+Tv6MzjtdyfcWsFuO+HZBHzCuqrOLk4IIK 0ttw+TLbGF7me8fNj/z841rGuQvtLuH43A7J6Mc4= Received: by mx.zohomail.com with SMTPS id 1715424777121760.7577937280839; Sat, 11 May 2024 03:52:57 -0700 (PDT) From: Ricardo Wurmus To: bug-guix@gnu.org Subject: guix-daemon fails to copy 4+GB file to store User-Agent: mu4e 1.12.4; emacs 29.3 X-Debbugs-Cc: ludo@gnu.org Date: Sat, 11 May 2024 12:52:53 +0200 Message-ID: <87msowpqm2.fsf@elephly.net> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" X-ZohoMailClient: External X-Zoho-Virus-Status: 1 Received-SPF: pass client-ip=136.143.188.51; envelope-from=rekado@elephly.net; helo=sender4-of-o51.zoho.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-Spam-Score: -1.3 (-) X-Debbugs-Envelope-To: submit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.3 (--) --=-=-= Content-Type: text/plain The guix-daemon's libutil/util.cc uses copy_file_range to copy a downloaded file into the store. copy_file_range fails on files larger than 4GB with an error like this: guix build: error: short write in copy_file_range `15' to `16': No such file or directory The man page for copy_file_range says that it could return EFBIG when the range exceeds the maximum range. The daemon code does not check any limits and will attempt to copy the whole file. I believe our code ought to check the value of st.size and fall back to a boring copy if it exceeds some "reasonable" value. This is where copy_file_range is used: https://git.savannah.gnu.org/cgit/guix.git/tree/nix/libutil/util.cc#n382 Here is a little reproducer: --=-=-= Content-Type: text/plain Content-Disposition: inline; filename=bug.scm (use-modules (guix download) (guix packages) (guix build-system trivial)) (package (name "chungus") (version "1") (source (origin (method url-fetch) (uri "http://localhost:1111/chungus") (sha256 (base32 "0nx67d4ls2nfwcfdmg81vf240z6lpwpdqypssr1wzn3hyz4szci4")))) (build-system trivial-build-system) (home-page "") (synopsis "") (description "") (license #f)) --=-=-= Content-Type: text/plain --8<---------------cut here---------------start------------->8--- # generate a big file dd bs=1M count=4096 if=/dev/zero of=/tmp/chungus # serve it guix shell woof -- woof -i 127.0.0.1 -p 1111 -c 1 /tmp/chungus # build the source derivation guix build --no-grafts -Sf bug.scm # observe the error # guix build: error: short write in copy_file_range `15' to `16': No such file or directory --8<---------------cut here---------------end--------------->8--- -- Ricardo --=-=-=-- ------------=_1715617502-13286-1-- From unknown Sun Jun 22 08:01:49 2025 X-Loop: help-debbugs@gnu.org Subject: bug#70877: guix-daemon fails to copy 4+GB file to store Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Tue, 14 May 2024 22:27:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 70877 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: 70877@debbugs.gnu.org, 70398@debbugs.gnu.org Received: via spool by 70877-submit@debbugs.gnu.org id=B70877.171572561023867 (code B ref 70877); Tue, 14 May 2024 22:27:02 +0000 Received: (at 70877) by debbugs.gnu.org; 14 May 2024 22:26:50 +0000 Received: from localhost ([127.0.0.1]:41512 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1s70bi-0006Co-1I for submit@debbugs.gnu.org; Tue, 14 May 2024 18:26:50 -0400 Received: from eggs.gnu.org ([209.51.188.92]:49324) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1s70bd-0006Cb-6f; Tue, 14 May 2024 18:26:47 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1s70bV-0003TQ-Uo; Tue, 14 May 2024 18:26:37 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:Date:References:In-Reply-To:Subject:To: From; bh=bs6dohsQOFaG4/Nnim3fabYfPKB+7KJarcFioXwNPM0=; b=brq/UbVFFurwNJO9IvSA ZdEHaVCGBOp406trR1lEbW6uggIUP1PHxfJ9SSidxmfBNmU5qrb9Rp8y9Ws+n79fVzw+OTI1Lgb8d cNi0nJCuzHCQcxfN7BT1nitnUVwny5Ewv25sxcfvO6en2fS29UzWpi+SZua0EDZAqokDHFC8dt6dU Vl7Ts90tSdA9GhEBtJs2mlIvQ09gjjUH8igyaCpBV71uzkxai1PO9NkSUZYhTQCC+xvAVYcrHML80 wn2D7JbyXuCjY3EQPG3ltPVLSxNHnyt+AEQl+S+wKW3ZZcB9tWszgIpHvqZXXig7VVntJy590v3WL DRiow1eR1kZKmA==; From: Ludovic =?UTF-8?Q?Court=C3=A8s?= In-Reply-To: <874jb1aih9.fsf@gnu.org> ("Ludovic =?UTF-8?Q?Court=C3=A8s?="'s message of "Mon, 13 May 2024 16:34:26 +0200") References: <87msowpqm2.fsf@elephly.net> <87v83ic99h.fsf@gnu.org> <87wmnxoqus.fsf@elephly.net> <874jb1aih9.fsf@gnu.org> Date: Wed, 15 May 2024 00:26:34 +0200 Message-ID: <874jb02fol.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) BTW, the newly updated =E2=80=98guix=E2=80=99 package is 8% smaller, as a r= esult of : --8<---------------cut here---------------start------------->8--- $ guix describe=20 Generation 302 May 12 2024 23:29:11 (current) guix 89cd778 repository URL: https://git.savannah.gnu.org/git/guix.git branch: master commit: 89cd778f6a45cd9b43a4dc1f236dcd0a87af955c $ guix size guix |head -2 store item total s= elf /gnu/store/r96xq0064nqf43ygcr7z9lgb18vrd1wa-guix-1.4.0-18.4c94b9e 705.8 = 400.6 56.8% $ ./pre-inst-env guix size guix |head -2 store item total s= elf /gnu/store/mcw1d2zy96is5ymjj903i3bi5a0qdwr5-guix-1.4.0-19.7ca9809 673.8 = 368.7 54.7% $ git log |head -1 commit 58be9a79e2862d5fa9842d73f498ce2e5442b9ce --8<---------------cut here---------------end--------------->8--- Ludo=E2=80=99.