From unknown Wed Jun 18 23:13:34 2025 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Mailer: MIME-tools 5.509 (Entity 5.509) Content-Type: text/plain; charset=utf-8 From: bug#42044 <42044@debbugs.gnu.org> To: bug#42044 <42044@debbugs.gnu.org> Subject: Status: nproc says "1" on mobile processors Reply-To: bug#42044 <42044@debbugs.gnu.org> Date: Thu, 19 Jun 2025 06:13:34 +0000 retitle 42044 nproc says "1" on mobile processors reassign 42044 coreutils submitter 42044 Adam Borowski severity 42044 normal thanks From debbugs-submit-bounces@debbugs.gnu.org Thu Jun 25 14:05:24 2020 Received: (at submit) by debbugs.gnu.org; 25 Jun 2020 18:05:24 +0000 Received: from localhost ([127.0.0.1]:41117 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1joWFX-0005o6-0L for submit@debbugs.gnu.org; Thu, 25 Jun 2020 14:05:24 -0400 Received: from lists.gnu.org ([209.51.188.17]:44080) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1joVos-00056v-GK for submit@debbugs.gnu.org; Thu, 25 Jun 2020 13:37:51 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:49870) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1joVos-0004dY-7h for bug-coreutils@gnu.org; Thu, 25 Jun 2020 13:37:50 -0400 Received: from tartarus.angband.pl ([2001:41d0:602:dbe::8]:45492) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1joVoq-0004N9-2h for bug-coreutils@gnu.org; Thu, 25 Jun 2020 13:37:49 -0400 Received: from kilobyte by tartarus.angband.pl with local (Exim 4.92) (envelope-from ) id 1joVoh-0002r0-0n for bug-coreutils@gnu.org; Thu, 25 Jun 2020 19:37:39 +0200 Date: Thu, 25 Jun 2020 19:37:39 +0200 From: Adam Borowski To: bug-coreutils@gnu.org Subject: nproc says "1" on mobile processors Message-ID: <20200625173739.GA9648@angband.pl> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit X-Junkbait: aaron@angband.pl, zzyx@angband.pl User-Agent: Mutt/1.10.1 (2018-07-13) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: kilobyte@angband.pl X-SA-Exim-Scanned: No (on tartarus.angband.pl); SAEximRunCond expanded to false Received-SPF: pass client-ip=2001:41d0:602:dbe::8; envelope-from=kilobyte@angband.pl; helo=tartarus.angband.pl X-detected-operating-system: by eggs.gnu.org: No matching host in p0f cache. That's all we know. X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-Spam-Score: -1.3 (-) X-Debbugs-Envelope-To: submit X-Mailman-Approved-At: Thu, 25 Jun 2020 14:05:22 -0400 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.3 (--) Hi! I'm afraid that `nproc` shows only the number of _currently_ online CPUs, which on mobile processors tends to be 1 when starting a job. As there's a need to conserve power, holding cores online when they have nothing to do would be a waste, thus they constantly get onlined and offlined. For example, on a 10-core box where the processor consists of three clusters, 4+4+2 cores, topline graph shows: (oooo▁▁▁ oo) (oooo▁ ▁ oo) (▃ooooooooo) (oooo▁▁▁▁oo) (▂ooooooooo) (oooo▁▁▁ oo) (oooo▁ ▁ oo) (oooo▁▁ oo) (oooo▁ ▁oo) (oooo▁▁ oo) (oooo▁ oo) (1 line = 1 second, "o" means offline core, otherwise it's utilization level.) Because of noisy GUI (that eg. draws this very graph), the machine keeps flipping between onlining just CPU 0, or a cluster of CPU4..7. Thus, `nproc` will randomly says either "1" or "4", while the user expects to run her compile with all 10 cores. I found out that numa_num_task_cpus() from libnuma gets the right answer, handling both affinity mask and CPUs present-but-currently-offline. But alas, it doesn't provide a command-line tool, and thousands of scripts already use nproc, thus switching a tool would be a waste of effort. Ie, could you please make nproc include all available CPUs rather than only online ones? Meow! -- ⢀⣴⠾⠻⢶⣦⠀ ⣾⠁⢠⠒⠀⣿⡁ in the beginning was the boot and root floppies and they were good. ⢿⡄⠘⠷⠚⠋⠀ -- on #linux-sunxi ⠈⠳⣄⠀⠀⠀⠀ From debbugs-submit-bounces@debbugs.gnu.org Fri Jun 26 14:26:29 2020 Received: (at 42044) by debbugs.gnu.org; 26 Jun 2020 18:26:29 +0000 Received: from localhost ([127.0.0.1]:43024 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jot3V-0002gf-8s for submit@debbugs.gnu.org; Fri, 26 Jun 2020 14:26:29 -0400 Received: from mail-ed1-f50.google.com ([209.85.208.50]:44486) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jot3R-0002gP-5r for 42044@debbugs.gnu.org; Fri, 26 Jun 2020 14:26:28 -0400 Received: by mail-ed1-f50.google.com with SMTP id s28so7616587edw.11 for <42044@debbugs.gnu.org>; Fri, 26 Jun 2020 11:26:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:to:references:from:message-id:date:user-agent :mime-version:in-reply-to:content-language:content-transfer-encoding; bh=G4xw0MLd9DrhY5OhFWK3FNaw+z8uH4ZndqJHg3GptwQ=; b=kj/VPDgcJ8cCAhpnQZj1e787BX9c/LdiEpDXQc33l0HCg5EbC68ifHBplyirnbkoD3 R8GBnow4uh/RPq7DGPZsIsrcKSYxHkj0qzuH423VUqXKb58rzRkKL3YwMwtpOV0/FyJL lRM2TGvyUf+x+zCiRlWThY7bC+Z+1M80EzMoR/NtjbWs5Aqq3KN6jSGjq3ZC2rPEzaWu TH3TtWDRzUUbuReM9NQ720FD1u6pv+qi+IlPf44EYPGJZ8S5TFrzijbmGW88C9pYwndk PrjaIKtPAIn2KRq2bel6DeHUV3lPWQz6LzU7oRUXFZ73ORqEiBkudXF/YIKIDqaUixyk m3bg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:to:references:from:message-id :date:user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=G4xw0MLd9DrhY5OhFWK3FNaw+z8uH4ZndqJHg3GptwQ=; b=nn6EAx30ZT/7JAryWzqkP30j5UqA6TBtZ3ts+T2FQUKsL+j9Z5LatbIoXrXnBpZF3L Ia2j0iBrT/HYUgZ4CakFZ6EF/pppBYVYNBtPBDxQlSbFoPkA9Hjwjv8pJy589sRDfeyT X0joYOQA0SCG/Qp5cjXy1el/+FnewSlj0LQm6xH1C5KUG8KUM6NKjPkPYj0B4NLRiHfs gSRE9RkiDemgAMKf2ED6Ac+BQY86gWColEma3TOXsDUPuwWrPZFeyXWUTj5FKwDpCSMz 8QpDdcgK/H+7ydTX+zj/vq4M0nuz+BvBLVg9vBxRSAd7BIIlZXIKa3ZvcWR7fY7HxofQ r6Pg== X-Gm-Message-State: AOAM533BFyM+Z69cL7hrcNPU/CRZApGgBfDdrLfYEsPVb+wYazoPCxAJ 5Y+5K466OQNqCZN1HH7oAkLrbx1o X-Google-Smtp-Source: ABdhPJyeyI5KC3mW1p0v9xQW13kawtVTq1o9JM5OmJB9L46vWN1olnotQLoL0HXfB0ygLbCGrbhRoA== X-Received: by 2002:a50:e604:: with SMTP id y4mr4493287edm.373.1593195978981; Fri, 26 Jun 2020 11:26:18 -0700 (PDT) Received: from localhost.localdomain (86-42-14-227-dynamic.agg2.lod.rsl-rtd.eircom.net. [86.42.14.227]) by smtp.googlemail.com with ESMTPSA id 92sm5756712edg.78.2020.06.26.11.26.17 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 26 Jun 2020 11:26:18 -0700 (PDT) Subject: Re: bug#42044: nproc says "1" on mobile processors To: Adam Borowski , 42044@debbugs.gnu.org References: <20200625173739.GA9648@angband.pl> From: =?UTF-8?Q?P=c3=a1draig_Brady?= Message-ID: Date: Fri, 26 Jun 2020 19:26:16 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.0 MIME-Version: 1.0 In-Reply-To: <20200625173739.GA9648@angband.pl> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 42044 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) On 25/06/2020 18:37, Adam Borowski wrote: > Hi! > I'm afraid that `nproc` shows only the number of _currently_ online CPUs, > which on mobile processors tends to be 1 when starting a job. As there's > a need to conserve power, holding cores online when they have nothing to > do would be a waste, thus they constantly get onlined and offlined. > > For example, on a 10-core box where the processor consists of three > clusters, 4+4+2 cores, topline graph shows: > > (oooo▁▁▁ oo) > (oooo▁ ▁ oo) > (▃ooooooooo) > (oooo▁▁▁▁oo) > (▂ooooooooo) > (oooo▁▁▁ oo) > (oooo▁ ▁ oo) > (oooo▁▁ oo) > (oooo▁ ▁oo) > (oooo▁▁ oo) > (oooo▁ oo) > > (1 line = 1 second, "o" means offline core, otherwise it's utilization > level.) > > Because of noisy GUI (that eg. draws this very graph), the machine keeps > flipping between onlining just CPU 0, or a cluster of CPU4..7. Thus, > `nproc` will randomly says either "1" or "4", while the user expects to > run her compile with all 10 cores. > > I found out that numa_num_task_cpus() from libnuma gets the right answer, > handling both affinity mask and CPUs present-but-currently-offline. But > alas, it doesn't provide a command-line tool, and thousands of scripts > already use nproc, thus switching a tool would be a waste of effort. > > Ie, could you please make nproc include all available CPUs rather than > only online ones? Does `nproc --all` suffice for your use case? From debbugs-submit-bounces@debbugs.gnu.org Fri Jun 26 15:19:03 2020 Received: (at 42044) by debbugs.gnu.org; 26 Jun 2020 19:19:04 +0000 Received: from localhost ([127.0.0.1]:43052 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jotsN-00040b-O4 for submit@debbugs.gnu.org; Fri, 26 Jun 2020 15:19:03 -0400 Received: from tartarus.angband.pl ([54.37.238.230]:34304) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jotsJ-000408-TQ for 42044@debbugs.gnu.org; Fri, 26 Jun 2020 15:19:01 -0400 Received: from kilobyte by tartarus.angband.pl with local (Exim 4.92) (envelope-from ) id 1jotsH-0002c8-VZ; Fri, 26 Jun 2020 21:18:57 +0200 Date: Fri, 26 Jun 2020 21:18:57 +0200 From: Adam Borowski To: =?iso-8859-1?Q?P=E1draig?= Brady Subject: Re: bug#42044: nproc says "1" on mobile processors Message-ID: <20200626191857.GA9719@angband.pl> References: <20200625173739.GA9648@angband.pl> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-Junkbait: aaron@angband.pl, zzyx@angband.pl User-Agent: Mutt/1.10.1 (2018-07-13) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: kilobyte@angband.pl X-SA-Exim-Scanned: No (on tartarus.angband.pl); SAEximRunCond expanded to false X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 42044 Cc: 42044@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) On Fri, Jun 26, 2020 at 07:26:16PM +0100, Pádraig Brady wrote: > On 25/06/2020 18:37, Adam Borowski wrote: > > Hi! > > I'm afraid that `nproc` shows only the number of _currently_ online CPUs, > > which on mobile processors tends to be 1 when starting a job. As there's > > a need to conserve power, holding cores online when they have nothing to > > do would be a waste, thus they constantly get onlined and offlined. > > Ie, could you please make nproc include all available CPUs rather than > > only online ones? > > Does `nproc --all` suffice for your use case? It works on the mobile box, but doesn't obey affinity mask anymore: [~]$ numactl -N 0 nproc --all 64 [~]$ numactl -N 0 nproc 16 I'd wish for fitting all uses, big and small. Ie, the answer "on this machine, with currently imposed limits, what's the fastest level of parallelism?". Meow. -- ⢀⣴⠾⠻⢶⣦⠀ ⣾⠁⢠⠒⠀⣿⡁ in the beginning was the boot and root floppies and they were good. ⢿⡄⠘⠷⠚⠋⠀ -- on #linux-sunxi ⠈⠳⣄⠀⠀⠀⠀