From debbugs-submit-bounces@debbugs.gnu.org Tue Jun 22 14:50:49 2021 Received: (at submit) by debbugs.gnu.org; 22 Jun 2021 18:50:49 +0000 Received: from localhost ([127.0.0.1]:39882 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lvlU1-0007he-9E for submit@debbugs.gnu.org; Tue, 22 Jun 2021 14:50:49 -0400 Received: from lists.gnu.org ([209.51.188.17]:58646) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lvlTz-0007hW-Nr for submit@debbugs.gnu.org; Tue, 22 Jun 2021 14:50:48 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:44130) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lvlTz-00037z-DF for bug-guix@gnu.org; Tue, 22 Jun 2021 14:50:47 -0400 Received: from mail-qt1-x831.google.com ([2607:f8b0:4864:20::831]:39775) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lvlTx-0005oB-Oq for bug-guix@gnu.org; Tue, 22 Jun 2021 14:50:47 -0400 Received: by mail-qt1-x831.google.com with SMTP id 17so213411qtt.6 for ; Tue, 22 Jun 2021 11:50:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:subject:date:message-id:mime-version; bh=o2RRDeRl+CBINwa9K6J/YA6WHkrgnD5KAggvM6kcAW4=; b=WtAIf7c5u/MG0Z+xUnrNjjPDHQyzD9k7yKj0Qv+dJ8roq9B9hoJWDgDJLJ12a2s3Pp zCXrawuUjjnBBn8uv/wj/c/s0ApBlGHAAmZGOrICw1MiDTBJLdN98SnH9+FMida9kS5E dhCO45FlqF9/bELJTTIRFO00jeYVlMQBSmlmvSO+YVZ2Q0Khk6xqbirQwa4ou1hA3Mse 848HYBECanGWsr6xytLGffBa1TjyRxFZ2iTbRnglrVMw9pXKJeWa3H61fwd28dXcCUKh 92PtpTrnzgLyRTwuRJsV8INVezBp40RdmyUQyn1iFg3afMWGJKHo+DQsQCUM79TCS7i0 cN7w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:mime-version; bh=o2RRDeRl+CBINwa9K6J/YA6WHkrgnD5KAggvM6kcAW4=; b=D2WN5N/+3fWTgz2+f5JpQhale15TAv/4auInTcG73hmUh0ro2VQRrmXTdSF43ErlUw ESTB5VBJjxD8naota6I4okDcx+Dmpe5DCMkGYVYAerrHBS4BK0YvgkXIstY3yRlAAZi0 GikvDY8jhFLXdzhyaN5kVBQhG5nZhqZDFaaNkjMEjbK4Sfat4qKF7Qb7QHK31ER4r+Vp 5baWaGGaT/0iavpeKHciRVlYhYFhriHYBbPOSlfszyd644M27hq6/ENVYzy5hYKTKEKC +sVdV4ufB/9+rGzzY0aP44G1eA9NZ+4OCHObCG1mZX1Dvyv3oKAo0MvVv+0W5iQ6f8V5 LAbQ== X-Gm-Message-State: AOAM530AF/rILdL4TntbjRh2irHkfc7zMMCVcL0Pm+qwJtalWVzuYq1L 2p2acMD7GagtkIwA67vLfay0xSqkfIRgiA== X-Google-Smtp-Source: ABdhPJyIz3N//jlnEYm5InkAZaT0dbds848gMsrFbijbTb9GNrwc2p7mF0nwWO2+qQ+GHX4BnGWonA== X-Received: by 2002:ac8:4682:: with SMTP id g2mr221010qto.274.1624387843674; Tue, 22 Jun 2021 11:50:43 -0700 (PDT) Received: from hurd (dsl-205-236-230-118.b2b2c.ca. [205.236.230.118]) by smtp.gmail.com with ESMTPSA id p6sm10151984qkp.80.2021.06.22.11.50.43 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Jun 2021 11:50:43 -0700 (PDT) From: Maxim Cournoyer To: bug-guix Subject: Poor 'guix substitute' performance when receiving Zstd-compressed substitutes Date: Tue, 22 Jun 2021 14:50:42 -0400 Message-ID: <87fsx93f99.fsf@gmail.com> MIME-Version: 1.0 Content-Type: text/plain Received-SPF: pass client-ip=2607:f8b0:4864:20::831; envelope-from=maxim.cournoyer@gmail.com; helo=mail-qt1-x831.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-Spam-Score: -1.3 (-) X-Debbugs-Envelope-To: submit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.3 (--) Hello, It's something I've been observing for a while, but substitutes are very IO intensive (as can be seen in iotop, the substitute process is waiting on IO > 99% of the time) and is much slower than expected (3 minutes to transfer 100 MiB uncompressed over a 50 mbps downstream link): --8<---------------cut here---------------start------------->8--- TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND 13934 be/4 root 1033.09 K/s 1485.06 K/s 0.00 % 93.36 % guile \ /gnu/store/vphx2839xv0qj9xwcwrb95592lzrrnx7-guix-1.3.0-3.50dfbbf/bin/guix substitute --substitute --8<---------------cut here---------------end--------------->8--- The publisher (remote machine) is has its guix-daemon configured via: --8<---------------cut here---------------start------------->8--- (service guix-publish-service-type (guix-publish-configuration (advertise? #t) (compression '(("zstd" 3))) (host "0.0.0.0"))) ;listen on all interfaces --8<---------------cut here---------------end--------------->8--- CPU is idling during on both sides during the transfer (all the work appears to be in the IO on the receiving end). The above example was observed downloading /gnu/store/4fcwwlv9bzfrraxiz41b4vcv131930fx-libstdc++-doc-9.4.0, but I do not think it is substitute-specific. Maxim From debbugs-submit-bounces@debbugs.gnu.org Wed Jun 23 09:26:53 2021 Received: (at 49174) by debbugs.gnu.org; 23 Jun 2021 13:26:53 +0000 Received: from localhost ([127.0.0.1]:40603 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lw2u5-0007vG-7a for submit@debbugs.gnu.org; Wed, 23 Jun 2021 09:26:53 -0400 Received: from eggs.gnu.org ([209.51.188.92]:35924) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lw2u3-0007v3-H1 for 49174@debbugs.gnu.org; Wed, 23 Jun 2021 09:26:51 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:37170) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lw2ty-0002oP-6m; Wed, 23 Jun 2021 09:26:46 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=33526 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lw2tx-0004kj-Uq; Wed, 23 Jun 2021 09:26:46 -0400 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Maxim Cournoyer Subject: Re: bug#49174: Poor 'guix substitute' performance when receiving Zstd-compressed substitutes References: <87fsx93f99.fsf@gmail.com> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 5 Messidor an 229 de la =?utf-8?Q?R=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Wed, 23 Jun 2021 15:26:43 +0200 In-Reply-To: <87fsx93f99.fsf@gmail.com> (Maxim Cournoyer's message of "Tue, 22 Jun 2021 14:50:42 -0400") Message-ID: <87eecsyang.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 49174 Cc: 49174@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hi, Maxim Cournoyer skribis: > It's something I've been observing for a while, but substitutes are very > IO intensive (as can be seen in iotop, the substitute process is waiting > on IO > 99% of the time) and is much slower than expected (3 minutes to > transfer 100 MiB uncompressed over a 50 mbps downstream link): > > TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND > 13934 be/4 root 1033.09 K/s 1485.06 K/s 0.00 % 93.36 % guile \ /gnu/= store/vphx2839xv0qj9xwcwrb95592lzrrnx7-guix-1.3.0-3.50dfbbf/bin/guix substi= tute --substitute > > > The publisher (remote machine) is has its guix-daemon configured via: > > (service guix-publish-service-type > (guix-publish-configuration > (advertise? #t) > (compression '(("zstd" 3))) > (host "0.0.0.0"))) ;listen on all interfaces Note that in this case nars are built and compressed on the fly on the server side, which puts an upper bound on the bandwidth you can achieve. I showed earlier how I profiled these things: https://guix.gnu.org/en/blog/2021/getting-bytes-to-disk-more-quickly/ If the client is I/O-bound, that=E2=80=99s good: it means we can=E2=80=99t = do any better (unless we skip unpacking as demonstrated by distri). If you can provide detailed profiles of either the server side or the client side (but in that case, make sure the server is caching things), that=E2=80=99d be great! Otherwise I=E2=80=99m afraid this is not actionable. :-) Thanks, Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Sat Jul 03 12:10:23 2021 Received: (at control) by debbugs.gnu.org; 3 Jul 2021 16:10:23 +0000 Received: from localhost ([127.0.0.1]:39720 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lziDn-00038K-6I for submit@debbugs.gnu.org; Sat, 03 Jul 2021 12:10:23 -0400 Received: from mail3-relais-sop.national.inria.fr ([192.134.164.104]:30142) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lziDk-000385-IQ for control@debbugs.gnu.org; Sat, 03 Jul 2021 12:10:21 -0400 IronPort-HdrOrdr: =?us-ascii?q?A9a23=3AvXTMoKrJLDUzIDy+y9porygaV5pPeYIsimQD?= =?us-ascii?q?101hICG9Ffb5qyl0ppUmPHDP4wr5NEtNpThSUJPvfZqjz/RICOAqVN+ftWLd11?= =?us-ascii?q?dAQrsO0bff?= X-IronPort-AV: E=Sophos;i="5.83,322,1616454000"; d="scan'208";a="386960110" Received: from 91-160-117-201.subs.proxad.net (HELO ribbon) ([91.160.117.201]) by mail3-relais-sop.national.inria.fr with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 03 Jul 2021 18:10:13 +0200 Date: Sat, 03 Jul 2021 18:10:13 +0200 Message-Id: <87czrzz8d6.fsf@gnu.org> To: control@debbugs.gnu.org From: =?utf-8?Q?Ludovic_Court=C3=A8s?= Subject: control message for bug #49174 MIME-version: 1.0 Content-type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Spam-Score: -1.3 (-) X-Debbugs-Envelope-To: control X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.3 (--) tags 49174 + moreinfo quit From debbugs-submit-bounces@debbugs.gnu.org Thu Oct 06 21:11:39 2022 Received: (at 49174-done) by debbugs.gnu.org; 7 Oct 2022 01:11:39 +0000 Received: from localhost ([127.0.0.1]:33895 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ogbtr-0003RD-C8 for submit@debbugs.gnu.org; Thu, 06 Oct 2022 21:11:39 -0400 Received: from mail-qt1-f177.google.com ([209.85.160.177]:37887) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ogbtm-0003Qq-Cr for 49174-done@debbugs.gnu.org; Thu, 06 Oct 2022 21:11:38 -0400 Received: by mail-qt1-f177.google.com with SMTP id l27so2064636qtv.4 for <49174-done@debbugs.gnu.org>; Thu, 06 Oct 2022 18:11:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:user-agent:message-id :in-reply-to:date:references:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=IM4x3rx5PZeqJqnoO06acx5ag93Yd/qVaNtE7ojViP0=; b=EgnGCSwK+QgNEIvPgkpI2wEWBkoics0APDd6YaGP8l3H5wSgzZZFbkjkOCjqeGDY75 0riHOnbRO30zIZA5G2YB4Hm1Bix72kNZW62UNql31cCDUsTBWAGaSpKP1CrjQlJfMsSt FqsHmPEwr2WBNmZmL4nidReT5Q8x/1sx2OACWMKslCeoFIcZXN4Sy8IjEm+DYZbyTNO2 U25+MwckjXYwMevjf7t0Q0Aq7ETIX13BCgUXSPuxJwTX8sZQBsS86Oqi9IzR1jUpxQKb pBOFA/LsFFvG3DwzRGivY62DRUAl1bLxJOwhfgfjWVfdoKmlxnMa5Rhxqv2Ba96kDc23 gqGw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:user-agent:message-id :in-reply-to:date:references:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=IM4x3rx5PZeqJqnoO06acx5ag93Yd/qVaNtE7ojViP0=; b=lvVyyDCjsA30XMmqqi9NyInnxYQXce3IjTXkMzNF6Y3U5FE/I7WTAOoUtax71mSdeh xF1JbjoYpmfGotu0mdbAKGEUdTA0QM7pyg7v6sSZunL9kbE8MPdImk7zWVz1OFcmp4mx hXVZdWX5jxIy/0AyVTd90WAQP6eennxY/DQa7IePViNpPrZBeyf+c6rcp9Yi2yZURzMI pglV3alWe8Ybg+OfXAAQNYIt7pkr/2Wsz8qme0ZP8zqbMo1gZYxLuSBlMAC55Xr0nm8E nW1h+5h7RWTLf5ckvkTbv9Q1IUK9if7KYDMPPxHKG8L4pa2ETI3LtJZc7SawtaCmADJL Cgmw== X-Gm-Message-State: ACrzQf1dqWUdC5CTki5jfvtzhMIH9IGpENqdVGHHgXLJRHxEbv+GRNRT L8U9mO11If1vRHzzJJv9zchD2NavUhA= X-Google-Smtp-Source: AMsMyM7OPWYVX2r8spH6+E6SU4tp1fuMn2gQETSvU8ZZhN1Uxdx9NOqBVolhGUZG0BGYvcOHpHgZDQ== X-Received: by 2002:ac8:5745:0:b0:35c:ca39:cc9f with SMTP id 5-20020ac85745000000b0035cca39cc9fmr2588524qtx.670.1665105087574; Thu, 06 Oct 2022 18:11:27 -0700 (PDT) Received: from hurd ([2607:fad8:4:3::1005]) by smtp.gmail.com with ESMTPSA id o19-20020a05620a2a1300b006cddf59a600sm718759qkp.34.2022.10.06.18.11.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 06 Oct 2022 18:11:27 -0700 (PDT) From: Maxim Cournoyer To: Ludovic =?utf-8?Q?Court=C3=A8s?= Subject: Re: bug#49174: Poor 'guix substitute' performance when receiving Zstd-compressed substitutes References: <87fsx93f99.fsf@gmail.com> <87eecsyang.fsf@gnu.org> Date: Thu, 06 Oct 2022 21:11:24 -0400 In-Reply-To: <87eecsyang.fsf@gnu.org> ("Ludovic =?utf-8?Q?Court=C3=A8s=22'?= =?utf-8?Q?s?= message of "Wed, 23 Jun 2021 15:26:43 +0200") Message-ID: <87ilkw7803.fsf@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 49174-done Cc: 49174-done@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Hi, Ludovic Court=C3=A8s writes: > Hi, > > Maxim Cournoyer skribis: > >> It's something I've been observing for a while, but substitutes are very >> IO intensive (as can be seen in iotop, the substitute process is waiting >> on IO > 99% of the time) and is much slower than expected (3 minutes to >> transfer 100 MiB uncompressed over a 50 mbps downstream link): >> >> TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND >> 13934 be/4 root 1033.09 K/s 1485.06 K/s 0.00 % 93.36 % guile \ /gnu= /store/vphx2839xv0qj9xwcwrb95592lzrrnx7-guix-1.3.0-3.50dfbbf/bin/guix subst= itute --substitute >> >> >> The publisher (remote machine) is has its guix-daemon configured via: >> >> (service guix-publish-service-type >> (guix-publish-configuration >> (advertise? #t) >> (compression '(("zstd" 3))) >> (host "0.0.0.0"))) ;listen on all interfaces > > Note that in this case nars are built and compressed on the fly on the > server side, which puts an upper bound on the bandwidth you can achieve. > > I showed earlier how I profiled these things: > > https://guix.gnu.org/en/blog/2021/getting-bytes-to-disk-more-quickly/ > > If the client is I/O-bound, that=E2=80=99s good: it means we can=E2=80=99= t do any better > (unless we skip unpacking as demonstrated by distri). > > If you can provide detailed profiles of either the server side or the > client side (but in that case, make sure the server is caching things), > that=E2=80=99d be great! > > Otherwise I=E2=80=99m afraid this is not actionable. :-) Since moving from HDDs to SSDs, I haven't seen this problem, so I suspect the poor IO of the HDDs was really the culprit rather than something to do with guile-zstd (and we had also benchmarked the late some when I experimented with using zstd-compressed man pages). Closing! --=20 Thanks, Maxim From unknown Fri Sep 05 11:52:33 2025 Received: (at fakecontrol) by fakecontrolmessage; To: internal_control@debbugs.gnu.org From: Debbugs Internal Request Subject: Internal Control Message-Id: bug archived. Date: Fri, 04 Nov 2022 11:24:12 +0000 User-Agent: Fakemail v42.6.9 # This is a fake control message. # # The action: # bug archived. thanks # This fakemail brought to you by your local debbugs # administrator