From debbugs-submit-bounces@debbugs.gnu.org Sat Nov 16 07:58:34 2024 Received: (at submit) by debbugs.gnu.org; 16 Nov 2024 12:58:34 +0000 Received: from localhost ([127.0.0.1]:52439 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1tCINl-00019k-OU for submit@debbugs.gnu.org; Sat, 16 Nov 2024 07:58:34 -0500 Received: from lists.gnu.org ([209.51.188.17]:52516) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1tCINj-00019c-Gi for submit@debbugs.gnu.org; Sat, 16 Nov 2024 07:58:32 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tCINj-00079x-3a for bug-guix@gnu.org; Sat, 16 Nov 2024 07:58:31 -0500 Received: from mout02.posteo.de ([185.67.36.66]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tCINg-0005UZ-Fj for bug-guix@gnu.org; Sat, 16 Nov 2024 07:58:30 -0500 Received: from submission (posteo.de [185.67.36.169]) by mout02.posteo.de (Postfix) with ESMTPS id 8E08B240101 for ; Sat, 16 Nov 2024 13:58:24 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017; t=1731761904; bh=VpYofyjWuA2w7HCm/zgJ1TWnUfAeiQ1C/6pQy8H+3cY=; h=Message-ID:Date:MIME-Version:To:From:Subject:Content-Type: Content-Transfer-Encoding:From; b=LT5oWMKWBiKZHzUurqkBHirRSA8orT+i/YdmSSTXeprwWHCfj3EpOqqiJrJ564bUH SvhaY8l76FCDg5WoT2LwM8M1MgX/l2KFcJi3Bh9iwhYTdtX2U69Whc2ND74xRlAxt+ zLpliByZRSBZrkVljkgunG3rZMduml1uINv+qSU+J5j9lxbMhYnFzyimvVlxu5htJV BREYAS4rGtI4zqm7ZKXSqj+mImZIYy9HCvn3mw9DeClvEdPbR2o9qeUCARZE18ouN5 gspFMpy0+vMcIjey8lWQRYPR3f4QCoDSvvAH6UnbnKViRlPEXaVI4t9ZwTNBU4OURX I1dOX0D+IaXHg== Received: from customer (localhost [127.0.0.1]) by submission (posteo.de) with ESMTPSA id 4XrDTX0dWbz6twx for ; Sat, 16 Nov 2024 13:58:23 +0100 (CET) Message-ID: <3faad437-c87c-44d0-ad77-9e42418bf2dd@posteo.net> Date: Sat, 16 Nov 2024 12:58:14 +0000 MIME-Version: 1.0 Content-Language: en-US To: bug-guix@gnu.org From: Dariqq Subject: High memory usage during guix pull (i686-linux, guile jit) Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Received-SPF: pass client-ip=185.67.36.66; envelope-from=dariqq@posteo.net; helo=mout02.posteo.de X-Spam_score_int: -43 X-Spam_score: -4.4 X-Spam_bar: ---- X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_CERTIFIED_BLOCKED=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-Spam-Score: -1.3 (-) X-Debbugs-Envelope-To: submit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.3 (--) Hi, Using 'guix pull' on i686-linux regularly fails when building 'guix-packages-base': E.g this recent run on ci.guix.gnu.org: https://ci.guix.gnu.org/build/6472396/details Log from me trying to pull 3e8d3d80f41e016cdfe80e488a78c2351c94fef8: Warning: Failed to expand heap by 8192 KiB GC Warning: Failed to expand heap by 8192 KiB GC Warning: Failed to expand heap by 8192 KiB GC Warning: Failed to expand heap by 8192 KiB GC Warning: Failed to expand heap by 8192 KiB GC Warning: Failed to expand heap by 8192 KiB GC Warning: Failed to expand heap by 8192 KiB GC Warning: Failed to expand heap by 8192 KiB GC Warning: Failed to expand heap by 8192 KiB GC Warning: Failed to expand heap by 8192 KiB GC Warning: Failed to expand heap by 8192 KiB GC Warning: Failed to expand heap by 8192 KiB GC Warning: Failed to expand heap by 64 KiB GC Warning: Out of Memory! Heap size: 3436 MiB. Returning NULL! Warning: Unwind-only out of memory exception; skipping pre-unwind handler. Warning: Unwind-only out of memory exception; skipping pre-unwind handler. [ 19/ 20] compiling... 90.0% of 10 filesallocating JIT code buffer failed: Cannot allocate memory JIT failed due to resource exhaustion disabling automatic JIT compilation allocate_stack failed: Cannot allocate memory Warning: Unwind-only stack overflow exception; skipping pre-unwind handler. [...] The error suggests this might be related to guile jit compiling things Disabling it by adding GUILE_JIT_THRESHOLD=-1 to the guix-daemon environment got things working for me again. From debbugs-submit-bounces@debbugs.gnu.org Thu Dec 19 11:54:23 2024 Received: (at 74381) by debbugs.gnu.org; 19 Dec 2024 16:54:23 +0000 Received: from localhost ([127.0.0.1]:39864 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1tOJn4-0001NH-RX for submit@debbugs.gnu.org; Thu, 19 Dec 2024 11:54:23 -0500 Received: from mout01.posteo.de ([185.67.36.65]:51087) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1tOJmz-0001Mw-Qk for 74381@debbugs.gnu.org; Thu, 19 Dec 2024 11:54:21 -0500 Received: from submission (posteo.de [185.67.36.169]) by mout01.posteo.de (Postfix) with ESMTPS id A5C63240027 for <74381@debbugs.gnu.org>; Thu, 19 Dec 2024 17:54:10 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017; t=1734627251; bh=dYoQ9O5VT60PtWyZm1bW9q6rE83vqFxWAXTCjUnmkGU=; h=Message-ID:Date:MIME-Version:To:Subject:From:Content-Type: Content-Transfer-Encoding:From; b=C9v9OpKmARpR6PaoIKKMyCakRtYO0kKJ85HEoWOlgpL0gVe0sa+N7hH/itpVeL+Bm d5m9CCfQPO57K97PMRvxBZ4kkI2U+XECbmcMRricJ2Z1oOurckMX5mpc4YwrS9lMu9 HcTzq+398OX2lf6l9QH0z2Yp+oyZpJaCMVAgJwRhUAp7wrAmopXK5C5c0an+w/IKYG xF1IN+pxGxo2hikNvVFTTw1xQygBy8uOXVUbxOnj4ZjNN+lphEVdp39weizeOrat89 CD87Tg4U4WW3PNFQa4wlgwaZe2n676/SmQAeouFdLMpsopYo0fUTqaRA//mxrsGuYL EFcP4glt3S/ww== Received: from customer (localhost [127.0.0.1]) by submission (posteo.de) with ESMTPSA id 4YDc8L4PvPz9rxV for <74381@debbugs.gnu.org>; Thu, 19 Dec 2024 17:54:10 +0100 (CET) Message-ID: Date: Thu, 19 Dec 2024 16:54:04 +0000 MIME-Version: 1.0 To: 74381@debbugs.gnu.org Subject: High memory usage during guix pull (i686-linux, guile jit) Content-Language: en-US From: Dariqq Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 74381 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) It turns out disabling guile jit does not work reliably and even without it guix pull may oom building guix-packages-base using 3GB+ of RAM. My workaround for now is to run 'guix pull -s i686-linux -p /tmp/whatever' on an x86_64 machine (to have more RAM available) with guile jit disabled multiple times until it succeeds (not exactly sure why that works, is something getting cached somewhere?) and use that machine to serve substitutes. After mentioning this on irc today janneke mentioned this commit https://git.savannah.gnu.org/cgit/guix.git/commit/?id=15c5f1a2c20b21de0f19f42db1ccab4c42117ebb which reduces the number of files per chunk from 25 to 10 to get guix pull working on the 32 bit hurd. I tried reducing that even further to 5 in my local checkout and the max RAM usage went to (a little bit more reasonable) 1.5-2 GB (both on i686 natively and when emulating with -s i686 with and without guile jit). This would be still not enough for my little machine but might be enough for the substitute servers to build it reliably. I also wanted to try this in a i586-gnu childhurd but guix pull immediately segfaulted. From debbugs-submit-bounces@debbugs.gnu.org Mon Jan 13 06:03:54 2025 Received: (at 74381) by debbugs.gnu.org; 13 Jan 2025 11:03:54 +0000 Received: from localhost ([127.0.0.1]:50502 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1tXIEc-0002r0-4D for submit@debbugs.gnu.org; Mon, 13 Jan 2025 06:03:54 -0500 Received: from mout02.posteo.de ([185.67.36.66]:50151) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1tXIEY-0002qg-Ph for 74381@debbugs.gnu.org; Mon, 13 Jan 2025 06:03:52 -0500 Received: from submission (posteo.de [185.67.36.169]) by mout02.posteo.de (Postfix) with ESMTPS id 052B0240101 for <74381@debbugs.gnu.org>; Mon, 13 Jan 2025 12:03:40 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017; t=1736766222; bh=/rZOy2duSAuMZmTzli7PE82U/R6C+bdAhxJT8BFQdmY=; h=Message-ID:Date:MIME-Version:Subject:From:To:Content-Type: Content-Transfer-Encoding:From; b=THlNN3+/B3arCJotd67Mzj3uIchXofxlnNqn2P09zQnFXDBNMCzYEJpSgKyDnjV5t lNv4ncoTs+j4BpYAZrThwIIUqWyApQthrJ2sR9/a8BZ21Dd7YnPWAPLfIVvvem5f37 blDoHn2tYqEmxwFSJe1U7qZyPRkmKaBvvPDptAK0PMsd26P8WOpKyixSqGkq2ulGJt X/o/KsWsRv7tqQZMYnic2Pyl5oAckyqZjGbX5gzq9wrremBBh7adaWy29dk/FkwXCB jj547I6uZjRnOY+umWS23kA79mGowgZxgVfIgpnJRHRsHkzaxWoEmEqj/AZmLwU40c UveNrvth5O4dQ== Received: from customer (localhost [127.0.0.1]) by submission (posteo.de) with ESMTPSA id 4YWqBN0WLMz6twS for <74381@debbugs.gnu.org>; Mon, 13 Jan 2025 12:03:38 +0100 (CET) Message-ID: <242cf4e9-120f-47f6-b975-d7f270e78665@posteo.net> Date: Mon, 13 Jan 2025 11:03:27 +0000 MIME-Version: 1.0 Subject: Re: High memory usage during guix pull (i686-linux, guile jit) From: Dariqq To: 74381@debbugs.gnu.org References: Content-Language: en-US In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 74381 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Yesterday I had a very weird situation, I tried to pull f365d6c6365b823bae2ae06d53088088ee07fba7 which failed on i686 natively with the 'Failed to expand Heap by 8192 KiB' warning. Rerun things on an x86_64 machine with -s i686-linux and building packages-base succededd with the guile process memory maxing out at 3.5 GB (+ another 500 MB for the guix process) However when I then pulled on the i686 machine substituting the packages-base guix was then left in a halfbroken state claiming that rust-criterion-0.5 is unbound which led to an error for almost anything useful(i think determining librsvg-for-system). I checked the guix (gnu packages crates-check) of that generation output and the scm file seemed file, so something must have gone horrible wrong when that one got (jit) - compiled. Unfortunately I had to gc the evidence to continue. I then rolled back, deleted and gced the broken generation and repulled (now 666f484fcfb03d801dc5c78a15501e01eb78567d) this time disabling guile-jit on the x86-64 machine as well, which drastically reduced memory consumption. After pulling that one on i686 aswell I had no further issues