From unknown Sun Jul 27 05:17:00 2025 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Mailer: MIME-tools 5.509 (Entity 5.509) Content-Type: text/plain; charset=utf-8 From: bug#40628 <40628@debbugs.gnu.org> To: bug#40628 <40628@debbugs.gnu.org> Subject: Status: Optimize admin/nt dependency computation Reply-To: bug#40628 <40628@debbugs.gnu.org> Date: Sun, 27 Jul 2025 12:17:00 +0000 retitle 40628 Optimize admin/nt dependency computation reassign 40628 emacs submitter 40628 Noam Postavsky severity 40628 wishlist tag 40628 patch thanks From debbugs-submit-bounces@debbugs.gnu.org Tue Apr 14 12:41:53 2020 Received: (at submit) by debbugs.gnu.org; 14 Apr 2020 16:41:53 +0000 Received: from localhost ([127.0.0.1]:34611 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jOOdF-0001SW-D0 for submit@debbugs.gnu.org; Tue, 14 Apr 2020 12:41:53 -0400 Received: from lists.gnu.org ([209.51.188.17]:34038) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jOOdE-0001SP-46 for submit@debbugs.gnu.org; Tue, 14 Apr 2020 12:41:52 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:57386) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jOOdC-0000BR-2I for bug-gnu-emacs@gnu.org; Tue, 14 Apr 2020 12:41:51 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=0.8 required=5.0 tests=BAYES_50,FREEMAIL_FROM, URIBL_BLOCKED autolearn=disabled version=3.3.2 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1jOOdA-0006Ax-Kw for bug-gnu-emacs@gnu.org; Tue, 14 Apr 2020 12:41:49 -0400 Received: from mail-qv1-xf2f.google.com ([2607:f8b0:4864:20::f2f]:32832) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1jOOdA-0006AZ-Ef for bug-gnu-emacs@gnu.org; Tue, 14 Apr 2020 12:41:48 -0400 Received: by mail-qv1-xf2f.google.com with SMTP id p19so169847qve.0 for ; Tue, 14 Apr 2020 09:41:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:subject:date:message-id:mime-version; bh=KpWw2VQWkuWbTHN9yy2qkLWICjE/RHWddZ9XZdDMO+E=; b=txixhTyB2NGKvulai/QixH7yn3ph+XMDDE8pAK6FAzye3/bCHgQIbUATPqcPnnno1J SU8OrUzXr7PBdHTaUacj9RagIhmemw8wBY2OHwUbn/Nf6muf8YKqM8roXj6buxaOffLs U2Sr5gQrRd/9k1ecWbyndhbLuWidDVAOhybEK7fnShRGn28bwUbNWXUvYtciEnJnGf4W 1B4UUcaPKxGAFiiYBTgGSzySnKm9/oGohOAbSWff+ymWyOAj2vtQ9NUWET95Z+IPPVL/ Da/j0OarxbtBOcYmkEYKUibCXU79nnT7LkNOumPL006fslWvJoCF/upuRCHQNB3vZjny Eg+A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:mime-version; bh=KpWw2VQWkuWbTHN9yy2qkLWICjE/RHWddZ9XZdDMO+E=; b=SxGvGldukW4Hs8zk754HHFAMoCHavAYHIpzNsm/vRyzmffkKTAsRbCUBJjVcXc11A0 qE/IY16E6zJf2qsamy7nQFesXgwf9QDvVHQRVRP/0cPVjQ8966HxII7vEOevlIqfHL5i SJN+tL7vF1/9a7GkGVYY1xHkTfcJlxRr7HAZkUuxHjy6hVZDQicGc9K5IsCbwWMJbSzL xZRHZ/qlyYEravNnlB5ARgEQ2dqy2lHPyH8P3Y94DdIykGSysr1OduqhXi4pc3630urw L13zPnvf0HQV/gaRQAZ7sVJhKEAmASFy1PlzQasZfJ0w9i0m0lMRC1dFGIoqzx/Mmr58 rjzg== X-Gm-Message-State: AGi0PuYD8v4uUAX1k8SR9jIM6gz6rJb7dA1LaHpeyLToeRGhEm3xu9IR q8yEev6vMuI3mGzmR64q34Oic4Id0LE= X-Google-Smtp-Source: APiQypKe+/GHgvprOfyleCmaOMXf9YlffpVW0x4RWoAtK61xFIB3EGwOGtGrZCVNvfx3EC+FD+oAgg== X-Received: by 2002:a0c:fa08:: with SMTP id q8mr855492qvn.132.1586882504646; Tue, 14 Apr 2020 09:41:44 -0700 (PDT) Received: from vhost2 (CPE001143542e1f-CMf81d0f809fa0.cpe.net.cable.rogers.com. [99.230.38.42]) by smtp.gmail.com with ESMTPSA id v62sm1773278qkb.85.2020.04.14.09.41.43 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Tue, 14 Apr 2020 09:41:43 -0700 (PDT) From: Noam Postavsky To: bug-gnu-emacs@gnu.org Subject: Optimize admin/nt dependency computation Date: Tue, 14 Apr 2020 12:41:43 -0400 Message-ID: <854ktmyq7s.fsf@gmail.com> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::f2f X-Spam-Score: 2.3 (++) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: Severity: wishlist Tags: patch X-Debbugs-CC: Phillip Lord While checking the patch I suggested in https://debbugs.gnu.org/40003#11, I noticed the dependency extraction seemed pretty slow. With the patch attached below it's much faster (1m20.437s to 0m3.695s [...] Content analysis details: (2.3 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- 0.0 URIBL_BLOCKED ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [URIs: build-dep-zips.py] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record 0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider (npostavs[at]gmail.com) 1.0 SPF_SOFTFAIL SPF: sender does not match SPF record (softfail) -0.7 RCVD_IN_DNSWL_LOW RBL: Sender listed at https://www.dnswl.org/, low trust [209.51.188.17 listed in list.dnswl.org] 2.0 SPOOFED_FREEMAIL No description available. X-Debbugs-Envelope-To: submit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.7 (/) --=-=-= Content-Type: text/plain Severity: wishlist Tags: patch X-Debbugs-CC: Phillip Lord While checking the patch I suggested in https://debbugs.gnu.org/40003#11, I noticed the dependency extraction seemed pretty slow. With the patch attached below it's much faster (1m20.437s to 0m3.695s according to 'time ./build-dep-zips.py -l'). --=-=-= Content-Type: text/plain Content-Disposition: attachment; filename=0001-Optimize-admin-nt-dependency-computation.patch Content-Description: patch >From 5312f51e315cc147ca901d499aa3c3cc64aa3340 Mon Sep 17 00:00:00 2001 From: Noam Postavsky Date: Tue, 14 Apr 2020 12:27:34 -0400 Subject: [PATCH] Optimize admin/nt dependency computation admin/nt/dist-build/build-dep-zips.py (immediate_deps) (extract_deps): Gather package dependency info in batches, rather than one at a time. This reduces the number of invocations of 'pacman -Si ...' to the depth of the dependency tree, rather than the number of dependent packages. (top-level): Don't call 'extract_deps' when given the '-l' option. --- admin/nt/dist-build/build-dep-zips.py | 45 ++++++++++++++------------- 1 file changed, 23 insertions(+), 22 deletions(-) diff --git a/admin/nt/dist-build/build-dep-zips.py b/admin/nt/dist-build/build-dep-zips.py index b0345a42cf3..dd7dc5e99dc 100755 --- a/admin/nt/dist-build/build-dep-zips.py +++ b/admin/nt/dist-build/build-dep-zips.py @@ -22,6 +22,8 @@ import os import shutil import re +import functools +import operator from subprocess import check_output @@ -47,7 +49,7 @@ ## Packages to fiddle with ## Source for gcc-libs is part of gcc SKIP_SRC_PKGS=["mingw-w64-gcc-libs"] -SKIP_DEP_PKGS=["mingw-w64-x86_64-glib2"] +SKIP_DEP_PKGS=frozenset(["mingw-w64-x86_64-glib2"]) MUNGE_SRC_PKGS={"mingw-w64-libwinpthread-git":"mingw-w64-winpthreads-git"} MUNGE_DEP_PKGS={ "mingw-w64-i686-libwinpthread":"mingw-w64-i686-libwinpthread-git", @@ -68,16 +70,14 @@ def check_output_maybe(*args,**kwargs): else: return check_output(*args,**kwargs) -def immediate_deps(pkg): - package_info = check_output(["pacman", "-Si", pkg]).decode("utf-8").split("\n") +def immediate_deps(pkgs): + package_info = check_output(["pacman", "-Si"] + pkgs).decode("utf-8").splitlines() - ## Extract the "Depends On" line - depends_on = [x for x in package_info if x.startswith("Depends On")][0] - ## Remove "Depends On" prefix - dependencies = depends_on.split(":")[1] - - ## Split into dependencies - dependencies = dependencies.strip().split(" ") + ## Extract the packages listed for "Depends On:" lines. + dependencies = [line.split(":")[1].split() for line in package_info + if line.startswith("Depends On")] + ## Flatten dependency lists from multiple packages into one list. + dependencies = functools.reduce(operator.iconcat, dependencies, []) ## Remove > signs TODO can we get any other punctuation here? dependencies = [d.split(">")[0] for d in dependencies if d] @@ -92,18 +92,19 @@ def extract_deps(): print( "Extracting deps" ) # Get a list of all dependencies needed for packages mentioned above. - pkgs = PKG_REQ[:] + pkgs = set(PKG_REQ) print("Initial pkgs", pkgs) - n = 0 - while n < len(pkgs): - subdeps = immediate_deps(pkgs[n]) - for p in subdeps: - if not (p in pkgs or p in SKIP_DEP_PKGS): - print("adding", p) - pkgs.append(p) - n = n + 1 - - return sorted(pkgs) + newdeps = pkgs + print("adding...") + while True: + subdeps = frozenset(immediate_deps(list(newdeps))) + newdeps = subdeps - SKIP_DEP_PKGS - pkgs + if not newdeps: + break + print('\n'.join(newdeps)) + pkgs |= newdeps + + return list(pkgs) def gather_deps(deps, arch, directory): @@ -261,7 +262,7 @@ def clean(): if( args.l ): print("List of dependencies") - print( extract_deps() ) + print( deps ) exit(0) if args.s: -- 2.23.0.windows.1 --=-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Sat Aug 08 08:57:13 2020 Received: (at 40628) by debbugs.gnu.org; 8 Aug 2020 12:57:13 +0000 Received: from localhost ([127.0.0.1]:58476 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1k4OPQ-0004xC-Sa for submit@debbugs.gnu.org; Sat, 08 Aug 2020 08:57:13 -0400 Received: from quimby.gnus.org ([95.216.78.240]:51436) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1k4OPP-0004wz-EC for 40628@debbugs.gnu.org; Sat, 08 Aug 2020 08:57:11 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnus.org; s=20200322; h=Content-Type:MIME-Version:Message-ID:In-Reply-To:Date: References:Subject:Cc:To:From:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender: Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe: List-Subscribe:List-Post:List-Owner:List-Archive; bh=SPBHvh7N4kgRMXmVaZ2VfhpyYHeUCYSr/XTnc2VF3uQ=; b=BxJKVXzwkWH0rxi8VUhnR0mZgt IZa4re9MQw6FUTlidwfMszwnl4ZAS0xC3R5LrD7sgcAdQbCmWlBJ7wyC3vh3JMokvLjMnZ0OMG/Zl rttHqmy8I32TlFyoVffKPoWtiHFUvIZummO1Xu03HUHfp0GFWGOhnkdI87X5VvB4CR9E=; Received: from cm-84.212.202.86.getinternet.no ([84.212.202.86] helo=xo) by quimby with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1k4OPG-0004YC-Il; Sat, 08 Aug 2020 14:57:05 +0200 From: Lars Ingebrigtsen To: Noam Postavsky Subject: Re: bug#40628: Optimize admin/nt dependency computation References: <854ktmyq7s.fsf@gmail.com> Date: Sat, 08 Aug 2020 14:57:00 +0200 In-Reply-To: <854ktmyq7s.fsf@gmail.com> (Noam Postavsky's message of "Tue, 14 Apr 2020 12:41:43 -0400") Message-ID: <87k0y91ejn.fsf@gnus.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.0.50 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Report: Spam detection software, running on the system "quimby.gnus.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see @@CONTACT_ADDRESS@@ for details. Content preview: Noam Postavsky writes: > While checking the patch I suggested in > https://debbugs.gnu.org/40003#11, I noticed the dependency extraction > seemed pretty slow. With the patch attached below it's much faster > (1m20.437s to 0 [...] Content analysis details: (-2.9 points, 5.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -1.0 ALL_TRUSTED Passed through trusted hosts only via SMTP -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1% [score: 0.0000] X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 40628 Cc: 40628@debbugs.gnu.org, phillip lord X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Noam Postavsky writes: > While checking the patch I suggested in > https://debbugs.gnu.org/40003#11, I noticed the dependency extraction > seemed pretty slow. With the patch attached below it's much faster > (1m20.437s to 0m3.695s according to 'time ./build-dep-zips.py -l'). > >>>From 5312f51e315cc147ca901d499aa3c3cc64aa3340 Mon Sep 17 00:00:00 2001 > From: Noam Postavsky > Date: Tue, 14 Apr 2020 12:27:34 -0400 > Subject: [PATCH] Optimize admin/nt dependency computation > > admin/nt/dist-build/build-dep-zips.py (immediate_deps) > (extract_deps): Gather package dependency info in batches, rather than > one at a time. This reduces the number of invocations of 'pacman -Si > ...' to the depth of the dependency tree, rather than the number of > dependent packages. > (top-level): Don't call 'extract_deps' when given the '-l' option. This was four months ago, but the patch was apparently not applied? -- (domestic pets only, the antidote for overdose, milk.) bloggy blog: http://lars.ingebrigtsen.no From debbugs-submit-bounces@debbugs.gnu.org Tue Aug 18 09:58:44 2020 Received: (at 40628) by debbugs.gnu.org; 18 Aug 2020 13:58:44 +0000 Received: from localhost ([127.0.0.1]:36532 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1k828S-0007ta-BL for submit@debbugs.gnu.org; Tue, 18 Aug 2020 09:58:44 -0400 Received: from quimby.gnus.org ([95.216.78.240]:32952) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1k828Q-0007tK-RD for 40628@debbugs.gnu.org; Tue, 18 Aug 2020 09:58:43 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnus.org; s=20200322; h=Content-Type:MIME-Version:Message-ID:In-Reply-To:Date: References:Subject:Cc:To:From:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender: Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe: List-Subscribe:List-Post:List-Owner:List-Archive; bh=8gJ+aKfsS3EgM5C6pY8MW5YdeHH9mKXSAP3LO/NASZw=; b=alKf79Cc1fsMh12/2NgiSgCWCW nNPUDdaZEZesTvOIl4zvzzLqMts3z9lkko0Byp9Dbece25atCqQiw+QBS/3AKPDL7KblMcgRsSP41 mCjQIy0+Fwo4HKmjNLvWZwGqHNbMPYBDlk7ReXqzsqlv2E5sMFpw68SWx0W662iGds6k=; Received: from cm-84.212.202.86.getinternet.no ([84.212.202.86] helo=xo) by quimby with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1k828I-0007bs-3b; Tue, 18 Aug 2020 15:58:36 +0200 From: Lars Ingebrigtsen To: Noam Postavsky Subject: Re: bug#40628: Optimize admin/nt dependency computation References: <854ktmyq7s.fsf@gmail.com> <87k0y91ejn.fsf@gnus.org> X-Now-Playing: Dead Can Dance's _Spleen and Ideal_: "De Profundis (Out of the Depths of Sorrow)" Date: Tue, 18 Aug 2020 15:58:32 +0200 In-Reply-To: <87k0y91ejn.fsf@gnus.org> (Lars Ingebrigtsen's message of "Sat, 08 Aug 2020 14:57:00 +0200") Message-ID: <87lfickqd3.fsf@gnus.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.0.50 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Report: Spam detection software, running on the system "quimby.gnus.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see @@CONTACT_ADDRESS@@ for details. Content preview: Lars Ingebrigtsen writes: > Noam Postavsky writes: > >> While checking the patch I suggested in >> https://debbugs.gnu.org/40003#11, I noticed the dependency extraction >> seemed pretty slow. With the patc [...] Content analysis details: (-2.9 points, 5.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -1.0 ALL_TRUSTED Passed through trusted hosts only via SMTP -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1% [score: 0.0000] X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 40628 Cc: 40628@debbugs.gnu.org, phillip lord X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Lars Ingebrigtsen writes: > Noam Postavsky writes: > >> While checking the patch I suggested in >> https://debbugs.gnu.org/40003#11, I noticed the dependency extraction >> seemed pretty slow. With the patch attached below it's much faster >> (1m20.437s to 0m3.695s according to 'time ./build-dep-zips.py -l'). [...] > This was four months ago, but the patch was apparently not applied? I don't have any Windows machines -- has anybody given this patch a test to see whether it introduces any problems? -- (domestic pets only, the antidote for overdose, milk.) bloggy blog: http://lars.ingebrigtsen.no From debbugs-submit-bounces@debbugs.gnu.org Fri Nov 12 04:23:33 2021 Received: (at 40628) by debbugs.gnu.org; 12 Nov 2021 09:23:33 +0000 Received: from localhost ([127.0.0.1]:43445 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mlSmT-0005r1-6L for submit@debbugs.gnu.org; Fri, 12 Nov 2021 04:23:33 -0500 Received: from quimby.gnus.org ([95.216.78.240]:35400) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mlSmR-0005ql-4E for 40628@debbugs.gnu.org; Fri, 12 Nov 2021 04:23:32 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnus.org; s=20200322; h=Content-Type:MIME-Version:Message-ID:In-Reply-To:Date: References:Subject:Cc:To:From:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender: Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe: List-Subscribe:List-Post:List-Owner:List-Archive; bh=sZpSZGr6nQXvCchb5Q8bE3poCjU1EILcExjiPHiFDyk=; b=O5lnsiuUJBOg2EAOFxfUu0Tpx4 iA4+Vm/VfdUJn2WtcRYMX9jJRRX3ifS3YS73uKgQ6lMzUbYePZQWk/MDuAqIhU+gSjxittv37lrys zftgBSi0+jDEh5SBXWlEcarw9ys40b9pLDJmhMDsNnK53zx15E+fvQ2gWhcsRfbs9HDk=; Received: from [84.212.220.105] (helo=xo) by quimby.gnus.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1mlSmE-00065P-Uj; Fri, 12 Nov 2021 10:23:22 +0100 From: Lars Ingebrigtsen To: Noam Postavsky Subject: Re: bug#40628: Optimize admin/nt dependency computation References: <854ktmyq7s.fsf@gmail.com> <87k0y91ejn.fsf@gnus.org> <87lfickqd3.fsf@gnus.org> X-Now-Playing: Sonic Youth's _Sonic Nurse_: "Pattern Recognition" Date: Fri, 12 Nov 2021 10:23:17 +0100 In-Reply-To: <87lfickqd3.fsf@gnus.org> (Lars Ingebrigtsen's message of "Tue, 18 Aug 2020 15:58:32 +0200") Message-ID: <87czn5u3qy.fsf@gnus.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/29.0.50 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Report: Spam detection software, running on the system "quimby.gnus.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see @@CONTACT_ADDRESS@@ for details. Content preview: Lars Ingebrigtsen writes: > Lars Ingebrigtsen writes: > >> Noam Postavsky writes: >> >>> While checking the patch I suggested in >>> https://debbugs.gnu.org/40003#11, I noticed the depende [...] Content analysis details: (-2.9 points, 5.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -1.0 ALL_TRUSTED Passed through trusted hosts only via SMTP -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1% [score: 0.0000] X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 40628 Cc: 40628@debbugs.gnu.org, phillip lord X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Lars Ingebrigtsen writes: > Lars Ingebrigtsen writes: > >> Noam Postavsky writes: >> >>> While checking the patch I suggested in >>> https://debbugs.gnu.org/40003#11, I noticed the dependency extraction >>> seemed pretty slow. With the patch attached below it's much faster >>> (1m20.437s to 0m3.695s according to 'time ./build-dep-zips.py -l'). > > [...] > >> This was four months ago, but the patch was apparently not applied? > > I don't have any Windows machines -- has anybody given this patch a test > to see whether it introduces any problems? I tested it myself, and it brought the generation time down from ~3 minutes to about 20 seconds, so I've applied the patch. I don't see any difference in the results, but then again, if failed with this both before and after the change: subprocess.CalledProcessError: Command 'wget -a ../download.log -O ../emacs-src-cache/mingw-w64-libtiff-4.3.0-6.src.tar.gz https://sourceforge.net/projects/msys2/files/REPOS/MINGW/Sources/mingw-w64-libtiff-4.3.0-6.src.tar.gz/download' returned non-zero exit status 8. -- (domestic pets only, the antidote for overdose, milk.) bloggy blog: http://lars.ingebrigtsen.no From debbugs-submit-bounces@debbugs.gnu.org Fri Nov 12 04:23:38 2021 Received: (at control) by debbugs.gnu.org; 12 Nov 2021 09:23:38 +0000 Received: from localhost ([127.0.0.1]:43448 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mlSmY-0005rJ-FA for submit@debbugs.gnu.org; Fri, 12 Nov 2021 04:23:38 -0500 Received: from quimby.gnus.org ([95.216.78.240]:35412) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mlSmX-0005qu-AK for control@debbugs.gnu.org; Fri, 12 Nov 2021 04:23:37 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnus.org; s=20200322; h=Subject:From:To:Message-Id:Date:Sender:Reply-To:Cc: MIME-Version:Content-Type:Content-Transfer-Encoding:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:In-Reply-To:References:List-Id:List-Help:List-Unsubscribe: List-Subscribe:List-Post:List-Owner:List-Archive; bh=934cLy399ur7lYTlMHkc5YUaKTi6zVZvscwORdrikcc=; b=fJcxZVs8uRdUUqcAJQSGRKowmK laH+1IQs5YaGxrwTHNuDviIxYuIJMRcCbq/hO4tTYUmCtuql8wz1aRkd6rnARHaxRIj9ef+HENxRm 8nsGshGoMT9n1JhQA13ZtcBFUS/PUsE85h3e2To+5ZePIQgPhmy4SbMofs74gFPJnyS0=; Received: from [84.212.220.105] (helo=xo) by quimby.gnus.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1mlSmP-00065X-3x for control@debbugs.gnu.org; Fri, 12 Nov 2021 10:23:31 +0100 Date: Fri, 12 Nov 2021 10:23:26 +0100 Message-Id: <87bl2pu3qp.fsf@gnus.org> To: control@debbugs.gnu.org From: Lars Ingebrigtsen Subject: control message for bug #40628 X-Spam-Report: Spam detection software, running on the system "quimby.gnus.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see @@CONTACT_ADDRESS@@ for details. Content preview: close 40628 29.1 quit Content analysis details: (-2.9 points, 5.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -1.0 ALL_TRUSTED Passed through trusted hosts only via SMTP -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1% [score: 0.0000] X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: control X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) close 40628 29.1 quit From debbugs-submit-bounces@debbugs.gnu.org Fri Nov 12 04:55:20 2021 Received: (at submit) by debbugs.gnu.org; 12 Nov 2021 09:55:20 +0000 Received: from localhost ([127.0.0.1]:43497 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mlTHD-0000Rs-RW for submit@debbugs.gnu.org; Fri, 12 Nov 2021 04:55:20 -0500 Received: from lists.gnu.org ([209.51.188.17]:56622) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mlTHB-0000Rk-LQ for submit@debbugs.gnu.org; Fri, 12 Nov 2021 04:55:18 -0500 Received: from eggs.gnu.org ([209.51.188.92]:53142) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mlTHB-0004rv-Bz for bug-gnu-emacs@gnu.org; Fri, 12 Nov 2021 04:55:17 -0500 Received: from ciao.gmane.io ([116.202.254.214]:56156) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mlTH9-0005ts-DA for bug-gnu-emacs@gnu.org; Fri, 12 Nov 2021 04:55:17 -0500 Received: from list by ciao.gmane.io with local (Exim 4.92) (envelope-from ) id 1mlTH5-0008H8-M6 for bug-gnu-emacs@gnu.org; Fri, 12 Nov 2021 10:55:11 +0100 X-Injected-Via-Gmane: http://gmane.org/ To: bug-gnu-emacs@gnu.org From: Andy Moreton Subject: Re: bug#40628: Optimize admin/nt dependency computation Date: Fri, 12 Nov 2021 09:55:02 +0000 Message-ID: <86wnldd7gp.fsf@gmail.com> References: <854ktmyq7s.fsf@gmail.com> <87k0y91ejn.fsf@gnus.org> <87lfickqd3.fsf@gnus.org> <87czn5u3qy.fsf@gnus.org> Mime-Version: 1.0 Content-Type: text/plain User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.0.60 (windows-nt) Cancel-Lock: sha1:dhj48TPnwyTsfUscK53Wb3/B4ww= Received-SPF: pass client-ip=116.202.254.214; envelope-from=geb-bug-gnu-emacs@m.gmane-mx.org; helo=ciao.gmane.io X-Spam_score_int: 5 X-Spam_score: 0.5 X-Spam_bar: / X-Spam_report: (0.5 / 5.0 requ) BAYES_00=-1.9, DKIM_ADSP_CUSTOM_MED=0.001, FORGED_GMAIL_RCVD=1, FREEMAIL_FORGED_FROMDOMAIN=0.248, FREEMAIL_FROM=0.001, HEADER_FROM_DIFFERENT_DOMAINS=0.248, NML_ADSP_CUSTOM_MED=0.9, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=no autolearn_force=no X-Spam_action: no action X-Spam-Score: 0.1 (/) X-Debbugs-Envelope-To: submit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.9 (/) On Fri 12 Nov 2021, Lars Ingebrigtsen wrote: > Lars Ingebrigtsen writes: > >> Lars Ingebrigtsen writes: >> >>> Noam Postavsky writes: >>> >>>> While checking the patch I suggested in >>>> https://debbugs.gnu.org/40003#11, I noticed the dependency extraction >>>> seemed pretty slow. With the patch attached below it's much faster >>>> (1m20.437s to 0m3.695s according to 'time ./build-dep-zips.py -l'). >> >> [...] >> >>> This was four months ago, but the patch was apparently not applied? >> >> I don't have any Windows machines -- has anybody given this patch a test >> to see whether it introduces any problems? > > I tested it myself, and it brought the generation time down from ~3 > minutes to about 20 seconds, so I've applied the patch. I don't see any > difference in the results, but then again, if failed with this both > before and after the change: > > subprocess.CalledProcessError: Command 'wget -a ../download.log -O > ../emacs-src-cache/mingw-w64-libtiff-4.3.0-6.src.tar.gz > https://sourceforge.net/projects/msys2/files/REPOS/MINGW/Sources/mingw-w64-libtiff-4.3.0-6.src.tar.gz/download' > returned non-zero exit status 8. The MSYS2 project has moved some stuff away from sourceforge to its own site. The sources are now at: https://repo.msys2.org/mingw/sources/ For the libtiff download above, that results in: https://repo.msys2.org/mingw/sources/mingw-w64-libtiff-4.3.0-6.src.tar.gz The script will need adjusting to change the URL, and drop the "/download" suffix. AndyM From debbugs-submit-bounces@debbugs.gnu.org Fri Nov 12 07:03:47 2021 Received: (at 40628) by debbugs.gnu.org; 12 Nov 2021 12:03:47 +0000 Received: from localhost ([127.0.0.1]:43632 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mlVHW-00040T-Nf for submit@debbugs.gnu.org; Fri, 12 Nov 2021 07:03:47 -0500 Received: from eggs.gnu.org ([209.51.188.92]:47446) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mlVHQ-00040D-MZ for 40628@debbugs.gnu.org; Fri, 12 Nov 2021 07:03:45 -0500 Received: from [2001:470:142:3::e] (port=55064 helo=fencepost.gnu.org) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mlVHL-00052S-4c; Fri, 12 Nov 2021 07:03:35 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date: mime-version; bh=ugmvDuPSW6TgM2gUceKAq5yTHo+E1EEp9Y431Ep80Yg=; b=KPshZgVunIzD x7l6e22JAloVklwmQPY98p2nmTWQnI7mf/4rYinF0pEqLqwc84NmpcDtvnKgalaa4RF6/EfyPrWPM zzrmGZIqea84q3BxbOCzDdj87BqHuBY7+eIwp++YkQ+Du+Hs5m/9MmV1SD/LFa6UnkVFJ+2g/QKNZ BdgkqJuhJVrLgCqToXA+YVH/7GkCH7Tg9iOmAzSlsZuYdFDKeHHMsbiUKh4h6oHFbmBab8bK6eaTh gxX/q9vJ65GIHQGefxaAQMKSKVogy9MEnBEs41Yy+fxyyq1HcY/sPHNfPe1odYmlkwNxXEnXdIW63 NDaZ/2PgNlL4c4zabgGWCw==; Received: from [87.69.77.57] (port=3562 helo=home-c4e4a596f7) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mlVHK-0003eH-Mt; Fri, 12 Nov 2021 07:03:35 -0500 Date: Fri, 12 Nov 2021 14:03:16 +0200 Message-Id: <83czn5zim3.fsf@gnu.org> From: Eli Zaretskii To: Lars Ingebrigtsen In-Reply-To: <87czn5u3qy.fsf@gnus.org> (message from Lars Ingebrigtsen on Fri, 12 Nov 2021 10:23:17 +0100) Subject: Re: bug#40628: Optimize admin/nt dependency computation References: <854ktmyq7s.fsf@gmail.com> <87k0y91ejn.fsf@gnus.org> <87lfickqd3.fsf@gnus.org> <87czn5u3qy.fsf@gnus.org> X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 40628 Cc: 40628@debbugs.gnu.org, npostavs@gmail.com, phillip.lord@russet.org.uk X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) > From: Lars Ingebrigtsen > Date: Fri, 12 Nov 2021 10:23:17 +0100 > Cc: 40628@debbugs.gnu.org, phillip lord > > I tested it myself, and it brought the generation time down from ~3 > minutes to about 20 seconds, so I've applied the patch. I don't see any > difference in the results, but then again, if failed with this both > before and after the change: > > subprocess.CalledProcessError: Command 'wget -a ../download.log -O ../emacs-src-cache/mingw-w64-libtiff-4.3.0-6.src.tar.gz https://sourceforge.net/projects/msys2/files/REPOS/MINGW/Sources/mingw-w64-libtiff-4.3.0-6.src.tar.gz/download' returned non-zero exit status 8. According to my records, 8 means "Server issued an error response". Whatever that means. From debbugs-submit-bounces@debbugs.gnu.org Sat Nov 13 20:07:07 2021 Received: (at 40628) by debbugs.gnu.org; 14 Nov 2021 01:07:07 +0000 Received: from localhost ([127.0.0.1]:48522 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mm3z9-0005LV-Hp for submit@debbugs.gnu.org; Sat, 13 Nov 2021 20:07:07 -0500 Received: from quimby.gnus.org ([95.216.78.240]:52920) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mm3z7-0005Ky-Jz for 40628@debbugs.gnu.org; Sat, 13 Nov 2021 20:07:06 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnus.org; s=20200322; h=Content-Type:MIME-Version:Message-ID:In-Reply-To:Date: References:Subject:Cc:To:From:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender: Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe: List-Subscribe:List-Post:List-Owner:List-Archive; bh=pNi3SK06MlqDJ5cm5So17sFq5jSHdGOE5/oH2tT0QRc=; b=XxAhfzIbxLKw0OWYkrYvDMq+Ej eJMdbN0jII0FDqdG//ZIgE9FswKoWI3ZrBE6HY8+oZo1DmDkrJ5dIpI2GdB8nZE8ZAM+3hFV7Os1b /CYbtYIsNTidULA0iIVHsRP7N9J3X9XjqqA+enh9ZpdWkxObMZEtgaPDmG4zHdgCXlTo=; Received: from [84.212.220.105] (helo=xo) by quimby.gnus.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1mm3yy-0005kK-OH; Sun, 14 Nov 2021 02:06:59 +0100 From: Lars Ingebrigtsen To: Andy Moreton Subject: Re: bug#40628: Optimize admin/nt dependency computation References: <854ktmyq7s.fsf@gmail.com> <87k0y91ejn.fsf@gnus.org> <87lfickqd3.fsf@gnus.org> <87czn5u3qy.fsf@gnus.org> <86wnldd7gp.fsf@gmail.com> X-Now-Playing: Anna B Savage's _A Common Turn_: "A Steady Warmth" Date: Sun, 14 Nov 2021 02:06:56 +0100 In-Reply-To: <86wnldd7gp.fsf@gmail.com> (Andy Moreton's message of "Fri, 12 Nov 2021 09:55:02 +0000") Message-ID: <875ysvh7f3.fsf@gnus.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/29.0.50 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Report: Spam detection software, running on the system "quimby.gnus.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see @@CONTACT_ADDRESS@@ for details. Content preview: Andy Moreton writes: > The MSYS2 project has moved some stuff away from sourceforge to its own > site. The sources are now at: > > https://repo.msys2.org/mingw/sources/ > > For the libtiff download above, that results in: [...] Content analysis details: (-2.9 points, 5.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -1.0 ALL_TRUSTED Passed through trusted hosts only via SMTP -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1% [score: 0.0000] X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 40628 Cc: 40628@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Andy Moreton writes: > The MSYS2 project has moved some stuff away from sourceforge to its own > site. The sources are now at: > > https://repo.msys2.org/mingw/sources/ > > For the libtiff download above, that results in: > > https://repo.msys2.org/mingw/sources/mingw-w64-libtiff-4.3.0-6.src.tar.gz > > The script will need adjusting to change the URL, and drop the > "/download" suffix. Thanks; I've now updated the script with the new URL (and dropped the /download). -- (domestic pets only, the antidote for overdose, milk.) bloggy blog: http://lars.ingebrigtsen.no From unknown Sun Jul 27 05:17:00 2025 Received: (at fakecontrol) by fakecontrolmessage; To: internal_control@debbugs.gnu.org From: Debbugs Internal Request Subject: Internal Control Message-Id: bug archived. Date: Sun, 12 Dec 2021 12:24:07 +0000 User-Agent: Fakemail v42.6.9 # This is a fake control message. # # The action: # bug archived. thanks # This fakemail brought to you by your local debbugs # administrator