From unknown Mon Jun 23 20:16:37 2025 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Mailer: MIME-tools 5.509 (Entity 5.509) Content-Type: text/plain; charset=utf-8 From: bug#45705 <45705@debbugs.gnu.org> To: bug#45705 <45705@debbugs.gnu.org> Subject: Status: [feature/native-comp] Excessive memory consumption on windows 10 Reply-To: bug#45705 <45705@debbugs.gnu.org> Date: Tue, 24 Jun 2025 03:16:37 +0000 retitle 45705 [feature/native-comp] Excessive memory consumption on windows= 10 reassign 45705 emacs submitter 45705 =C3=89douard Debry severity 45705 normal thanks From debbugs-submit-bounces@debbugs.gnu.org Wed Jan 06 15:48:49 2021 Received: (at submit) by debbugs.gnu.org; 6 Jan 2021 20:48:49 +0000 Received: from localhost ([127.0.0.1]:45706 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kxFjd-0002vy-0E for submit@debbugs.gnu.org; Wed, 06 Jan 2021 15:48:49 -0500 Received: from lists.gnu.org ([209.51.188.17]:34184) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kxFjb-0002vr-Ou for submit@debbugs.gnu.org; Wed, 06 Jan 2021 15:48:48 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:50698) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kxFjb-0005DJ-HQ for bug-gnu-emacs@gnu.org; Wed, 06 Jan 2021 15:48:47 -0500 Received: from mail-wm1-x32e.google.com ([2a00:1450:4864:20::32e]:39723) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1kxFja-0004O7-2v for bug-gnu-emacs@gnu.org; Wed, 06 Jan 2021 15:48:47 -0500 Received: by mail-wm1-x32e.google.com with SMTP id 3so3737379wmg.4 for ; Wed, 06 Jan 2021 12:48:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=user-agent:from:to:subject:date:message-id:mime-version; bh=tY/UZzscwIDfBI7wI1fWTW6rfdgzKGQS5zg4Z2Omtlw=; b=LOO50Lr2jFJpGg1XruVCP+KZ3rZlzm5Zd2vZ1U0pwTx9LXeqZU8CM4Gh4f4xIvqlhi WnbUX0ijLQqX7mfGnTZ1py02UgETkTk0Xv3vCPCJ9KR+2xaLKTHGqCTHhpRwQHLISgE6 vWpxQ8VHxy1NlrwOUtKx4X4gHvc2OH4+kkR8Dfi8FexJbaij/blmJ3awhXOR0rBNB8CP TtIeKUrFq5bSOyIrnkCz+UZXajeL5f/65nVlkETZWiVO1f7+z1umydokwRSZqXkSnq0Z be7o202eSTl5MH1XBh3UYwhKvczFz7kKJYJalo/bKLfJhNRLqvhvPPy3VZFBh1i1+BE3 ZdMw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:user-agent:from:to:subject:date:message-id :mime-version; bh=tY/UZzscwIDfBI7wI1fWTW6rfdgzKGQS5zg4Z2Omtlw=; b=d6jUWOM2ZdadDgzRck2BSTE7KRaWtOv2491bOn0uSi20qbCcFKVdpMjhMQkgaH4WX7 fTM6BgKHhTXt3FLpjFrLzB7ziYYJQ/xG6laNZZsKp+Nl0H8wydQgQ40MK2JuqyhmuwCL YTl7FY5tjzyUD/SUy0+bHfn+m7waVxpcZ7Gu7lN/pmUCp9IJ1pb27hd6R89LSLNGrGEW LAs9413Ncv2+MhB8fHoB8HLlxEHC8MXQI6isk98O9uSUiOcQiJTeN7NInsM1QeC88a1s 075hFBPLdG12QPGBbNWMM6p3VaqC71A1V/gj1X76wu3ckt01ZRIB1LqWvWTNLpUKRmFP AkPQ== X-Gm-Message-State: AOAM531FxAlH6uuE/dhn6fgt7mDZIrWVmnP31QZfa22GgPJmptgq4Yc+ TkVtxa0frCucCu9qkq8kxcxUO8+FLnk= X-Google-Smtp-Source: ABdhPJxXGQf4cqKhiQ+xBqrTgSU3kTbjsnTE0KsoNRbxQGqVg2VrDPru4VLjwABGLGlrkK2Ocy+iew== X-Received: by 2002:a1c:6208:: with SMTP id w8mr5235133wmb.96.1609966123769; Wed, 06 Jan 2021 12:48:43 -0800 (PST) Received: from paquerette (176-142-29-14.abo.bbox.fr. [176.142.29.14]) by smtp.gmail.com with ESMTPSA id z15sm4614422wrv.67.2021.01.06.12.48.43 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 06 Jan 2021 12:48:43 -0800 (PST) User-agent: mu4e 1.0; emacs 28.0.50 From: =?utf-8?Q?=C3=89douard?= Debry To: bug-gnu-emacs@gnu.org Subject: [feature/native-comp] Excessive memory consumption on windows 10 Date: Wed, 06 Jan 2021 21:48:37 +0100 Message-ID: <87y2h5hjve.fsf@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; format=flowed Received-SPF: pass client-ip=2a00:1450:4864:20::32e; envelope-from=edouard.debry@gmail.com; helo=mail-wm1-x32e.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-Spam-Score: -1.3 (-) X-Debbugs-Envelope-To: submit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.3 (--) Hello, On windows 10, I noticed that emacs could use a lot of memory, up to 6Go and sometimes more. The memory consumption goes up and down from time to time while I am not running any specific program in emacs, apart `lsp-java` As my personal computer has 16Go of RAM, I can afford it. Unfortunately, my work computer has much less and the whole compute completely freezes at one point when using emacs. I did not notice that behavior on linux. I do not know if the master branch has the same problem. What could be the problem ? From debbugs-submit-bounces@debbugs.gnu.org Wed Jan 06 15:55:39 2021 Received: (at 45705) by debbugs.gnu.org; 6 Jan 2021 20:55:39 +0000 Received: from localhost ([127.0.0.1]:45724 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kxFqF-00036n-LR for submit@debbugs.gnu.org; Wed, 06 Jan 2021 15:55:39 -0500 Received: from mx.sdf.org ([205.166.94.24]:61564) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kxFqD-00036e-7b for 45705@debbugs.gnu.org; Wed, 06 Jan 2021 15:55:38 -0500 Received: from mab (ma.sdf.org [205.166.94.33]) by mx.sdf.org (8.15.2/8.14.5) with ESMTP id 106KtaUb005111; Wed, 6 Jan 2021 20:55:36 GMT From: Andrea Corallo To: =?utf-8?Q?=C3=89douard?= Debry Subject: Re: bug#45705: [feature/native-comp] Excessive memory consumption on windows 10 References: <87y2h5hjve.fsf@gmail.com> Date: Wed, 06 Jan 2021 20:55:35 +0000 In-Reply-To: <87y2h5hjve.fsf@gmail.com> (=?utf-8?Q?=22=C3=89douard?= Debry"'s message of "Wed, 06 Jan 2021 21:48:37 +0100") Message-ID: User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.0.50 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 45705 Cc: 45705@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) =C3=89douard Debry writes: > Hello, > > On windows 10, I noticed that emacs could use a lot of memory, up to > 6Go and > sometimes more. The memory consumption goes up and down from time to > time while > I am not running any specific program in emacs, apart `lsp-java` > > As my personal computer has 16Go of RAM, I can afford > it. Unfortunately, my work > computer has much less and the whole compute completely freezes at one > point > when using emacs. > > I did not notice that behavior on linux. I do not know if the master > branch has > the same problem. What could be the problem ? Hi =C3=89douard, AFAIK was never proved recently Emacs garbage collector is failing to recall memory, so I guess this is just some Lisp program that is allocating a lot of memory keeping then those objects referenced. Others may have more detaliled info about. Andrea From debbugs-submit-bounces@debbugs.gnu.org Thu Jan 07 09:25:40 2021 Received: (at submit) by debbugs.gnu.org; 7 Jan 2021 14:25:40 +0000 Received: from localhost ([127.0.0.1]:46913 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kxWEO-00064f-1F for submit@debbugs.gnu.org; Thu, 07 Jan 2021 09:25:40 -0500 Received: from lists.gnu.org ([209.51.188.17]:57372) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kxWEM-00064X-2t for submit@debbugs.gnu.org; Thu, 07 Jan 2021 09:25:38 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:51234) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kxWEL-0005OH-SS for bug-gnu-emacs@gnu.org; Thu, 07 Jan 2021 09:25:37 -0500 Received: from mx.sdf.org ([205.166.94.24]:53194) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kxWEI-00029k-Vw for bug-gnu-emacs@gnu.org; Thu, 07 Jan 2021 09:25:37 -0500 Received: from mab (ma.sdf.org [205.166.94.33]) by mx.sdf.org (8.15.2/8.14.5) with ESMTP id 107EPQ0g026194; Thu, 7 Jan 2021 14:25:27 GMT From: Andrea Corallo To: Andrea Corallo via "Bug reports for GNU Emacs, the Swiss army knife of text editors" Subject: Re: bug#45705: [feature/native-comp] Excessive memory consumption on windows 10 References: <87y2h5hjve.fsf@gmail.com> Date: Thu, 07 Jan 2021 14:25:26 +0000 In-Reply-To: (Andrea Corallo via's message of "Wed, 06 Jan 2021 20:55:35 +0000") Message-ID: User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.0.50 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Received-SPF: pass client-ip=205.166.94.24; envelope-from=akrl@sdf.org; helo=mx.sdf.org X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-Spam-Score: -1.4 (-) X-Debbugs-Envelope-To: submit Cc: =?utf-8?Q?=C3=89douard?= Debry , 45705@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.4 (--) Andrea Corallo via "Bug reports for GNU Emacs, the Swiss army knife of text editors" writes: > =C3=89douard Debry writes: > >> Hello, >> >> On windows 10, I noticed that emacs could use a lot of memory, up to >> 6Go and >> sometimes more. The memory consumption goes up and down from time to >> time while >> I am not running any specific program in emacs, apart `lsp-java` >> >> As my personal computer has 16Go of RAM, I can afford >> it. Unfortunately, my work >> computer has much less and the whole compute completely freezes at one >> point >> when using emacs. >> >> I did not notice that behavior on linux. I do not know if the master >> branch has >> the same problem. What could be the problem ? > > Hi =C3=89douard, > > AFAIK was never proved recently Emacs garbage collector is failing to > recall memory, so I guess this is just some Lisp program that is > allocating a lot of memory keeping then those objects referenced. > > Others may have more detaliled info about. > > Andrea > IOW I believe this is a duplicate of bug#43389. Should we merge these? Andrea From debbugs-submit-bounces@debbugs.gnu.org Fri Jan 08 09:25:57 2021 Received: (at 45705) by debbugs.gnu.org; 8 Jan 2021 14:25:57 +0000 Received: from localhost ([127.0.0.1]:49532 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kxsiC-0000Ha-UX for submit@debbugs.gnu.org; Fri, 08 Jan 2021 09:25:57 -0500 Received: from eggs.gnu.org ([209.51.188.92]:35948) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kxsiA-0000HG-VY for 45705@debbugs.gnu.org; Fri, 08 Jan 2021 09:25:55 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]:37370) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kxsi5-00033n-H1; Fri, 08 Jan 2021 09:25:49 -0500 Received: from 84.94.185.95.cable.012.net.il ([84.94.185.95]:1631 helo=home-c4e4a596f7) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1kxsi0-0007Lj-Iz; Fri, 08 Jan 2021 09:25:46 -0500 Date: Fri, 08 Jan 2021 16:25:45 +0200 Message-Id: <831revjyja.fsf@gnu.org> From: Eli Zaretskii To: Andrea Corallo In-Reply-To: (bug-gnu-emacs@gnu.org) Subject: Re: bug#45705: [feature/native-comp] Excessive memory consumption on windows 10 References: <87y2h5hjve.fsf@gmail.com> MIME-version: 1.0 Content-type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 45705 Cc: edouard.debry@gmail.com, 45705@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) > Date: Wed, 06 Jan 2021 20:55:35 +0000 > Cc: 45705@debbugs.gnu.org > From: Andrea Corallo via "Bug reports for GNU Emacs, > the Swiss army knife of text editors" > > > On windows 10, I noticed that emacs could use a lot of memory, up to > > 6Go and > > sometimes more. The memory consumption goes up and down from time to > > time while > > I am not running any specific program in emacs, apart `lsp-java` > > > > As my personal computer has 16Go of RAM, I can afford > > it. Unfortunately, my work > > computer has much less and the whole compute completely freezes at one > > point > > when using emacs. > > > > I did not notice that behavior on linux. I do not know if the master > > branch has > > the same problem. What could be the problem ? > > Hi Édouard, > > AFAIK was never proved recently Emacs garbage collector is failing to > recall memory, so I guess this is just some Lisp program that is > allocating a lot of memory keeping then those objects referenced. IME, 6 GiB is too much for any Lisp program to explain away. Also, the memory allocator we use on Windows is known to return memory to the OS more than glibc on GNU/Linux. I have never seen my Emacs session get anywhere near 1 GiB, let alone more, and my sessions run for many weeks without restarting. Can you tell what kind of memory footprints you see on your system with the native-comp branch, after running the session for several days or more? From debbugs-submit-bounces@debbugs.gnu.org Fri Jan 08 10:50:33 2021 Received: (at 45705) by debbugs.gnu.org; 8 Jan 2021 15:50:33 +0000 Received: from localhost ([127.0.0.1]:50371 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kxu25-0002g1-79 for submit@debbugs.gnu.org; Fri, 08 Jan 2021 10:50:33 -0500 Received: from mx.sdf.org ([205.166.94.24]:53012) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kxu23-0002fp-C8 for 45705@debbugs.gnu.org; Fri, 08 Jan 2021 10:50:32 -0500 Received: from mab (ma.sdf.org [205.166.94.33]) by mx.sdf.org (8.15.2/8.14.5) with ESMTP id 108FoSdS020904; Fri, 8 Jan 2021 15:50:28 GMT From: Andrea Corallo To: Eli Zaretskii Subject: Re: bug#45705: [feature/native-comp] Excessive memory consumption on windows 10 References: <87y2h5hjve.fsf@gmail.com> <831revjyja.fsf@gnu.org> Date: Fri, 08 Jan 2021 15:50:28 +0000 In-Reply-To: <831revjyja.fsf@gnu.org> (Eli Zaretskii's message of "Fri, 08 Jan 2021 16:25:45 +0200") Message-ID: User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.0.50 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 45705 Cc: edouard.debry@gmail.com, 45705@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Eli Zaretskii writes: >> Date: Wed, 06 Jan 2021 20:55:35 +0000 >> Cc: 45705@debbugs.gnu.org >> From: Andrea Corallo via "Bug reports for GNU Emacs, >> the Swiss army knife of text editors" >>=20 >> > On windows 10, I noticed that emacs could use a lot of memory, up to >> > 6Go and >> > sometimes more. The memory consumption goes up and down from time to >> > time while >> > I am not running any specific program in emacs, apart `lsp-java` >> > >> > As my personal computer has 16Go of RAM, I can afford >> > it. Unfortunately, my work >> > computer has much less and the whole compute completely freezes at one >> > point >> > when using emacs. >> > >> > I did not notice that behavior on linux. I do not know if the master >> > branch has >> > the same problem. What could be the problem ? >>=20 >> Hi =C3=89douard, >>=20 >> AFAIK was never proved recently Emacs garbage collector is failing to >> recall memory, so I guess this is just some Lisp program that is >> allocating a lot of memory keeping then those objects referenced. > > IME, 6 GiB is too much for any Lisp program to explain away. > > Also, the memory allocator we use on Windows is known to return memory > to the OS more than glibc on GNU/Linux. I have never seen my Emacs > session get anywhere near 1 GiB, let alone more, and my sessions run > for many weeks without restarting. > > Can you tell what kind of memory footprints you see on your system > with the native-comp branch, after running the session for several > days or more? Hi Eli, consumptions of few GiB is something I've seen more then once for long standing sessions. You might be right in this being a memory leak, indeed I've no prove of that (I think we have none for the other direction either). Assuming it's a bug I don't see a priori why this should be a different one respect the one reported for master. Regards Andrea From debbugs-submit-bounces@debbugs.gnu.org Fri Jan 08 11:10:05 2021 Received: (at 45705) by debbugs.gnu.org; 8 Jan 2021 16:10:05 +0000 Received: from localhost ([127.0.0.1]:50394 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kxuKz-0003CN-Cx for submit@debbugs.gnu.org; Fri, 08 Jan 2021 11:10:05 -0500 Received: from eggs.gnu.org ([209.51.188.92]:35170) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kxuKx-0003Bn-Q0 for 45705@debbugs.gnu.org; Fri, 08 Jan 2021 11:10:04 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]:39308) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kxuKs-0006th-By; Fri, 08 Jan 2021 11:09:58 -0500 Received: from 84.94.185.95.cable.012.net.il ([84.94.185.95]:4158 helo=home-c4e4a596f7) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1kxuKr-00048v-IH; Fri, 08 Jan 2021 11:09:58 -0500 Date: Fri, 08 Jan 2021 18:10:00 +0200 Message-Id: <83turrif53.fsf@gnu.org> From: Eli Zaretskii To: Andrea Corallo In-Reply-To: (message from Andrea Corallo on Fri, 08 Jan 2021 15:50:28 +0000) Subject: Re: bug#45705: [feature/native-comp] Excessive memory consumption on windows 10 References: <87y2h5hjve.fsf@gmail.com> <831revjyja.fsf@gnu.org> X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 45705 Cc: edouard.debry@gmail.com, 45705@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) > From: Andrea Corallo > Cc: edouard.debry@gmail.com, 45705@debbugs.gnu.org > Date: Fri, 08 Jan 2021 15:50:28 +0000 > > consumptions of few GiB is something I've seen more then once for long > standing sessions. You might be right in this being a memory leak, > indeed I've no prove of that (I think we have none for the other > direction either). I'm not yet worried about memory leaks, I'm more worried that there's no memory leaks, and instead using libgccjit indeed requires such large memory amounts. Are you sure this is not the case? From debbugs-submit-bounces@debbugs.gnu.org Fri Jan 08 17:02:43 2021 Received: (at 45705) by debbugs.gnu.org; 8 Jan 2021 22:02:43 +0000 Received: from localhost ([127.0.0.1]:50840 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kxzqE-0005bY-US for submit@debbugs.gnu.org; Fri, 08 Jan 2021 17:02:43 -0500 Received: from mx.sdf.org ([205.166.94.24]:62187) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kxzqD-0005bM-56 for 45705@debbugs.gnu.org; Fri, 08 Jan 2021 17:02:41 -0500 Received: from mab (ma.sdf.org [205.166.94.33]) by mx.sdf.org (8.15.2/8.14.5) with ESMTP id 108M2cYP003945; Fri, 8 Jan 2021 22:02:38 GMT From: Andrea Corallo To: Eli Zaretskii Subject: Re: bug#45705: [feature/native-comp] Excessive memory consumption on windows 10 References: <87y2h5hjve.fsf@gmail.com> <831revjyja.fsf@gnu.org> <83turrif53.fsf@gnu.org> Date: Fri, 08 Jan 2021 22:02:38 +0000 In-Reply-To: <83turrif53.fsf@gnu.org> (Eli Zaretskii's message of "Fri, 08 Jan 2021 18:10:00 +0200") Message-ID: User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.0.50 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 45705 Cc: edouard.debry@gmail.com, 45705@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Eli Zaretskii writes: >> From: Andrea Corallo >> Cc: edouard.debry@gmail.com, 45705@debbugs.gnu.org >> Date: Fri, 08 Jan 2021 15:50:28 +0000 >> >> consumptions of few GiB is something I've seen more then once for long >> standing sessions. You might be right in this being a memory leak, >> indeed I've no prove of that (I think we have none for the other >> direction either). > > I'm not yet worried about memory leaks, I'm more worried that there's > no memory leaks, and instead using libgccjit indeed requires such > large memory amounts. Are you sure this is not the case? I see, if we are interested in comparing the memory footprint of using shared libraries for functions vs stock byte-code I think we can "statically" compare two sessions after the startup. I've compiled current native-comp with and without --with-nativecomp repeating the experiment with and without X. These are the data-points: | | --without-x | --without-x --with-nativecomp | | |---------+-------------+-------------------------------+------| | -Q | 49M | 92M | 1.9x | | my-conf | 92M | 179M | 1.9x | | | | --with-nativecomp | | |---------+------+-------------------+------| | -Q | 536M | 756M | 1.4x | | my-conf | 585M | 1453M | 2.4x | So yes shared are using considerably more memory, I think this is expected as also the file footprint suggests native code is less dense that byte-code (actually with a quite similar relative results). Indeed *with use the delta should decay as data are the same and there's no difference in its representation*, so this picture should be more on the worst case side than on the average. Also if we want to see a positive side, multiple Emacs sessions will share the majority the pages allocated for the shared libraries. Andrea From debbugs-submit-bounces@debbugs.gnu.org Sat Jan 09 02:56:25 2021 Received: (at 45705) by debbugs.gnu.org; 9 Jan 2021 07:56:25 +0000 Received: from localhost ([127.0.0.1]:51174 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ky96n-0003JP-2v for submit@debbugs.gnu.org; Sat, 09 Jan 2021 02:56:25 -0500 Received: from eggs.gnu.org ([209.51.188.92]:46500) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ky96g-0003J5-KX for 45705@debbugs.gnu.org; Sat, 09 Jan 2021 02:56:23 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]:56212) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ky96a-0004vH-Ng; Sat, 09 Jan 2021 02:56:12 -0500 Received: from 84.94.185.95.cable.012.net.il ([84.94.185.95]:2798 helo=home-c4e4a596f7) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1ky96a-0003Zq-3O; Sat, 09 Jan 2021 02:56:12 -0500 Date: Sat, 09 Jan 2021 09:56:16 +0200 Message-Id: <83lfd2ilwf.fsf@gnu.org> From: Eli Zaretskii To: Andrea Corallo In-Reply-To: (message from Andrea Corallo on Fri, 08 Jan 2021 22:02:38 +0000) Subject: Re: bug#45705: [feature/native-comp] Excessive memory consumption on windows 10 References: <87y2h5hjve.fsf@gmail.com> <831revjyja.fsf@gnu.org> <83turrif53.fsf@gnu.org> X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 45705 Cc: edouard.debry@gmail.com, 45705@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) > From: Andrea Corallo > Cc: edouard.debry@gmail.com, 45705@debbugs.gnu.org > Date: Fri, 08 Jan 2021 22:02:38 +0000 > > I've compiled current native-comp with and without --with-nativecomp > repeating the experiment with and without X. These are the data-points: > > | | --without-x | --without-x --with-nativecomp | | > |---------+-------------+-------------------------------+------| > | -Q | 49M | 92M | 1.9x | > | my-conf | 92M | 179M | 1.9x | > > > | | | --with-nativecomp | | > |---------+------+-------------------+------| > | -Q | 536M | 756M | 1.4x | > | my-conf | 585M | 1453M | 2.4x | > > So yes shared are using considerably more memory, I think this is > expected as also the file footprint suggests native code is less dense > that byte-code (actually with a quite similar relative results). Thanks for the data points. What about memory usage when there's a background compilation of Lisp going on? GCC is known to be a memory hog in some cases, so I wonder what happens in this case with libgccjit. (Do we allow multiple async compilations, btw? if so, how many concurrent compilations can be running, and how do we or the user control that?) Also, what are the numbers for a session that has been running for several days? I understand that it would be hard for you to collect such numbers about all the configurations, but could you show the growth of the configuration you are routinely using, which I presume is --with-x --with-nativecomp and with your config? As your numbers above show, it starts at 1.5 GiB, but what is the footprint after a day or a week? > Indeed *with use the delta should decay as data are the same and there's > no difference in its representation*, so this picture should be more on > the worst case side than on the average. That's why I asked to see the memory footprint after the session has run for some time, yes. > Also if we want to see a positive side, multiple Emacs sessions will > share the majority the pages allocated for the shared libraries. Assuming those sessions run the same binary of Emacs, yes. Otherwise, only some of them will be shared, the ones that are in common public directories, and even that only if the running Emacsen are of the same version. Most of the shared code is in the pdumper file, btw, which is outside of the picture for the purposes of this discussion. Thanks. From debbugs-submit-bounces@debbugs.gnu.org Sat Jan 09 05:55:27 2021 Received: (at 45705) by debbugs.gnu.org; 9 Jan 2021 10:55:27 +0000 Received: from localhost ([127.0.0.1]:51261 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kyBu3-0007gw-1S for submit@debbugs.gnu.org; Sat, 09 Jan 2021 05:55:27 -0500 Received: from mx.sdf.org ([205.166.94.24]:58233) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kyBu0-0007gm-Vt for 45705@debbugs.gnu.org; Sat, 09 Jan 2021 05:55:26 -0500 Received: from mab (ma.sdf.org [205.166.94.33]) by mx.sdf.org (8.15.2/8.14.5) with ESMTP id 109AtNS0024509; Sat, 9 Jan 2021 10:55:23 GMT From: Andrea Corallo To: Eli Zaretskii Subject: Re: bug#45705: [feature/native-comp] Excessive memory consumption on windows 10 References: <87y2h5hjve.fsf@gmail.com> <831revjyja.fsf@gnu.org> <83turrif53.fsf@gnu.org> <83lfd2ilwf.fsf@gnu.org> Date: Sat, 09 Jan 2021 10:55:23 +0000 In-Reply-To: <83lfd2ilwf.fsf@gnu.org> (Eli Zaretskii's message of "Sat, 09 Jan 2021 09:56:16 +0200") Message-ID: User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.0.50 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 45705 Cc: edouard.debry@gmail.com, 45705@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Eli Zaretskii writes: >> From: Andrea Corallo >> Cc: edouard.debry@gmail.com, 45705@debbugs.gnu.org >> Date: Fri, 08 Jan 2021 22:02:38 +0000 >> >> I've compiled current native-comp with and without --with-nativecomp >> repeating the experiment with and without X. These are the data-points: >> >> | | --without-x | --without-x --with-nativecomp | | >> |---------+-------------+-------------------------------+------| >> | -Q | 49M | 92M | 1.9x | >> | my-conf | 92M | 179M | 1.9x | >> >> >> | | | --with-nativecomp | | >> |---------+------+-------------------+------| >> | -Q | 536M | 756M | 1.4x | >> | my-conf | 585M | 1453M | 2.4x | >> >> So yes shared are using considerably more memory, I think this is >> expected as also the file footprint suggests native code is less dense >> that byte-code (actually with a quite similar relative results). > > Thanks for the data points. > > What about memory usage when there's a background compilation of Lisp > going on? GCC is known to be a memory hog in some cases, so I wonder > what happens in this case with libgccjit. In June we changed the way we store immediate objects in the shared and this makes the compilation way lighter on the GCC side (both in time and memory). I've no precise data on this other than the experimental observation that compiling all Elisp files in Emacs on 32bit systems is not anymore an issue. This IIUC implies that the memory footprint for each compilation is always < 2GB. As a note: in all cases except bootstrap the final pass (the one driving libgccjit) is executed as a sub-process, this to protect us from eventual GCC leaks and not to generate unnecessary fragmentation. In async compilation we indeed run all the compilation (also the Lisp computation) in the child process, so compiling should not have impact on the memory footprint of the main Emacs session. > (Do we allow multiple async compilations, btw? if so, how many > concurrent compilations can be running, and how do we or the user > control that?) Yes see > Also, what are the numbers for a session that has been running for > several days? I understand that it would be hard for you to collect > such numbers about all the configurations, but could you show the > growth of the configuration you are routinely using, which I presume > is --with-x --with-nativecomp and with your config? As your numbers > above show, it starts at 1.5 GiB, but what is the footprint after a > day or a week? ATM I can provide this number, this is an Aarch64 daemon compiled with '--without-x' with an up-time of 25 days and is showing a footprint of 765M. >> Indeed *with use the delta should decay as data are the same and there's >> no difference in its representation*, so this picture should be more on >> the worst case side than on the average. > > That's why I asked to see the memory footprint after the session has > run for some time, yes. The hard part is to have a reference to compare against as the memory footprint is strictly connected to the usage. One with very regular working habits should work like one week on vanilla and one week on native-comp to make a comparison. I've no regular working habits so I fear I'm not the best fit for this comparison. Thanks Andrea From debbugs-submit-bounces@debbugs.gnu.org Sat Jan 09 06:55:36 2021 Received: (at 45705) by debbugs.gnu.org; 9 Jan 2021 11:55:36 +0000 Received: from localhost ([127.0.0.1]:51291 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kyCqG-0000yb-DD for submit@debbugs.gnu.org; Sat, 09 Jan 2021 06:55:36 -0500 Received: from eggs.gnu.org ([209.51.188.92]:50274) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kyCqE-0000yO-VT for 45705@debbugs.gnu.org; Sat, 09 Jan 2021 06:55:35 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]:58564) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kyCq8-0000bt-5g; Sat, 09 Jan 2021 06:55:28 -0500 Received: from 84.94.185.95.cable.012.net.il ([84.94.185.95]:1912 helo=home-c4e4a596f7) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1kyCpk-0008Rr-8a; Sat, 09 Jan 2021 06:55:06 -0500 Date: Sat, 09 Jan 2021 13:55:08 +0200 Message-Id: <83y2h2gw9v.fsf@gnu.org> From: Eli Zaretskii To: Andrea Corallo In-Reply-To: (message from Andrea Corallo on Sat, 09 Jan 2021 10:55:23 +0000) Subject: Re: bug#45705: [feature/native-comp] Excessive memory consumption on windows 10 References: <87y2h5hjve.fsf@gmail.com> <831revjyja.fsf@gnu.org> <83turrif53.fsf@gnu.org> <83lfd2ilwf.fsf@gnu.org> X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 45705 Cc: edouard.debry@gmail.com, 45705@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) > From: Andrea Corallo > Cc: edouard.debry@gmail.com, 45705@debbugs.gnu.org > Date: Sat, 09 Jan 2021 10:55:23 +0000 > > > What about memory usage when there's a background compilation of Lisp > > going on? GCC is known to be a memory hog in some cases, so I wonder > > what happens in this case with libgccjit. > > In June we changed the way we store immediate objects in the shared and > this makes the compilation way lighter on the GCC side (both in time and > memory). I've no precise data on this other than the experimental > observation that compiling all Elisp files in Emacs on 32bit systems is > not anymore an issue. This IIUC implies that the memory footprint for > each compilation is always < 2GB. You assume that the compilations are all done serially? AFAIK, most people build Emacs with "make -jN", so parallel compilation is an important use case. I guess we will have to collect the information about that, if you say we don't have it now. > As a note: in all cases except bootstrap the final pass (the one driving > libgccjit) is executed as a sub-process, this to protect us from > eventual GCC leaks and not to generate unnecessary fragmentation. In > async compilation we indeed run all the compilation (also the Lisp > computation) in the child process, so compiling should not have impact > on the memory footprint of the main Emacs session. That's fine, but the memory footprint of such a subprocess is also of interest, as it could be relevant to the overall memory pressure of the OS, and thus indirectly on the parent Emacs process as well. > > (Do we allow multiple async compilations, btw? if so, how many > > concurrent compilations can be running, and how do we or the user > > control that?) > > Yes see Thanks. This needs further tuning, IMO, both per the FIXME (i.e. provide a primitive to return the number of execution units), and wrt the default value being half of the available units. We should pay attention to the system's load average as well, I think. > > Also, what are the numbers for a session that has been running for > > several days? I understand that it would be hard for you to collect > > such numbers about all the configurations, but could you show the > > growth of the configuration you are routinely using, which I presume > > is --with-x --with-nativecomp and with your config? As your numbers > > above show, it starts at 1.5 GiB, but what is the footprint after a > > day or a week? > > ATM I can provide this number, this is an Aarch64 daemon compiled with > '--without-x' with an up-time of 25 days and is showing a footprint of > 765M. OK, thanks. > The hard part is to have a reference to compare against as the memory > footprint is strictly connected to the usage. One with very regular > working habits should work like one week on vanilla and one week on > native-comp to make a comparison. I've no regular working habits so I > fear I'm not the best fit for this comparison. I agree, these numbers still need to be collected. Maybe we should ask on emacs-devel that people who use the branch report their numbers? From debbugs-submit-bounces@debbugs.gnu.org Sat Jan 09 07:37:59 2021 Received: (at 45705) by debbugs.gnu.org; 9 Jan 2021 12:37:59 +0000 Received: from localhost ([127.0.0.1]:51348 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kyDVH-00047k-Fa for submit@debbugs.gnu.org; Sat, 09 Jan 2021 07:37:59 -0500 Received: from mx.sdf.org ([205.166.94.24]:52261) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kyDVE-00047a-9D for 45705@debbugs.gnu.org; Sat, 09 Jan 2021 07:37:58 -0500 Received: from mab (ma.sdf.org [205.166.94.33]) by mx.sdf.org (8.15.2/8.14.5) with ESMTP id 109CbsmP011267; Sat, 9 Jan 2021 12:37:54 GMT From: Andrea Corallo To: Eli Zaretskii Subject: Re: bug#45705: [feature/native-comp] Excessive memory consumption on windows 10 References: <87y2h5hjve.fsf@gmail.com> <831revjyja.fsf@gnu.org> <83turrif53.fsf@gnu.org> <83lfd2ilwf.fsf@gnu.org> <83y2h2gw9v.fsf@gnu.org> Date: Sat, 09 Jan 2021 12:37:54 +0000 In-Reply-To: <83y2h2gw9v.fsf@gnu.org> (Eli Zaretskii's message of "Sat, 09 Jan 2021 13:55:08 +0200") Message-ID: User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.0.50 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 45705 Cc: edouard.debry@gmail.com, 45705@debbugs.gnu.org, =?utf-8?Q?K=C3=A9vin?= Le Gouguec X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Eli Zaretskii writes: >> From: Andrea Corallo >> Cc: edouard.debry@gmail.com, 45705@debbugs.gnu.org >> Date: Sat, 09 Jan 2021 10:55:23 +0000 >> >> > What about memory usage when there's a background compilation of Lisp >> > going on? GCC is known to be a memory hog in some cases, so I wonder >> > what happens in this case with libgccjit. >> >> In June we changed the way we store immediate objects in the shared and >> this makes the compilation way lighter on the GCC side (both in time and >> memory). I've no precise data on this other than the experimental >> observation that compiling all Elisp files in Emacs on 32bit systems is >> not anymore an issue. This IIUC implies that the memory footprint for >> each compilation is always < 2GB. > > You assume that the compilations are all done serially? AFAIK, most > people build Emacs with "make -jN", so parallel compilation is an > important use case. Yeah, we can say this loose information is only a per compilation unit upper-bound. > I guess we will have to collect the information about that, if you say > we don't have it now. I'm adding in CC Kevin, IIRC for bug#41077 he used a nice setup to produce quite accurate results on memory footprint during the compilation process. Perhaps he has time and he's so kind to gather some data on the current state, that would be extremely helpful. >> As a note: in all cases except bootstrap the final pass (the one driving >> libgccjit) is executed as a sub-process, this to protect us from >> eventual GCC leaks and not to generate unnecessary fragmentation. In >> async compilation we indeed run all the compilation (also the Lisp >> computation) in the child process, so compiling should not have impact >> on the memory footprint of the main Emacs session. > > That's fine, but the memory footprint of such a subprocess is also of > interest, as it could be relevant to the overall memory pressure of > the OS, and thus indirectly on the parent Emacs process as well. Indeed, I thought was relevant to make you aware of this mechanism. >> > (Do we allow multiple async compilations, btw? if so, how many >> > concurrent compilations can be running, and how do we or the user >> > control that?) >> >> Yes see > > Thanks. This needs further tuning, IMO, both per the FIXME > (i.e. provide a primitive to return the number of execution units), > and wrt the default value being half of the available units. We > should pay attention to the system's load average as well, I think. Agree, I guess we'll be able to tune it better when we'll have more data. We could even think of using the memory pressure on the system to make such a decision. >> > Also, what are the numbers for a session that has been running for >> > several days? I understand that it would be hard for you to collect >> > such numbers about all the configurations, but could you show the >> > growth of the configuration you are routinely using, which I presume >> > is --with-x --with-nativecomp and with your config? As your numbers >> > above show, it starts at 1.5 GiB, but what is the footprint after a >> > day or a week? >> >> ATM I can provide this number, this is an Aarch64 daemon compiled with >> '--without-x' with an up-time of 25 days and is showing a footprint of >> 765M. > > OK, thanks. > >> The hard part is to have a reference to compare against as the memory >> footprint is strictly connected to the usage. One with very regular >> working habits should work like one week on vanilla and one week on >> native-comp to make a comparison. I've no regular working habits so I >> fear I'm not the best fit for this comparison. > > I agree, these numbers still need to be collected. Maybe we should > ask on emacs-devel that people who use the branch report their > numbers? Is a good idea, my fear would be only to have very noisy or hard to interpret results. A priori I think would be nice to collect such data also for master too. Thanks Andrea From debbugs-submit-bounces@debbugs.gnu.org Sat Jan 09 12:26:59 2021 Received: (at 45705) by debbugs.gnu.org; 9 Jan 2021 17:26:59 +0000 Received: from localhost ([127.0.0.1]:52536 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kyI0x-00034t-8z for submit@debbugs.gnu.org; Sat, 09 Jan 2021 12:26:59 -0500 Received: from mail-wm1-f43.google.com ([209.85.128.43]:55201) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kyI0s-00034S-Kg for 45705@debbugs.gnu.org; Sat, 09 Jan 2021 12:26:58 -0500 Received: by mail-wm1-f43.google.com with SMTP id c133so10288370wme.4 for <45705@debbugs.gnu.org>; Sat, 09 Jan 2021 09:26:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version; bh=p3RbSEpgcVqcaRM+sho17m+rQzwp9qQmVHEaG0tV+8E=; b=LzikEmPfaMA8FZph1WMiCwbnuHdUccG9Osi4SAkQaLRiNPLYZh2cH7xdTZ6SR1gphP V/tU6jwvmHeqH6/cX/+5WMiwk9TNTHl63LPnHNuTcvRlnebfj8y0B5Ec/3as6i9td1Jp A21HlDgKmJ84whmWyiIzYCbOzsKHyS3tX8n6KS9sQAR9Odq9EPneVk/mqcbDemR9TEpe 8DtYoVTS38WnmzWG3azYZQAzRRXFzzOR+lq/3M9DSew7ouXcu8uiqVYrnkjKGsAe01iE j13I+KJiuec66mrCDq1gZERByun7GbeN6jxbpoHDg6/ywN0aFh2x8ifX7BmU6uj1bhpP TuCg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to :message-id:user-agent:mime-version; bh=p3RbSEpgcVqcaRM+sho17m+rQzwp9qQmVHEaG0tV+8E=; b=eEwVXQuw5yhEzx0JpSva6VXy8/sooksTwjxHoaJBTdaSNozNlUHbW3q5ax/f9ZzwmN z3cOqL+W0ztDd/RFqbYH77qWhctvYqmvUOFHXZKnaMMYDfhkz/kLdXSw3ImTId5wRS9k fQyacal8GaaQLsyvcBd6QFcf8hkUxqKlUDB3Ss+fzCmeQgzhoST8Xp8vThwC1+QHh+fe EPEg7do45jvoLHvX+4XrWaP0lHbWhrmkEsQqL+gX28xeKPxXh1hO70N998gRN5chd3QM OsLsfB6bsoreQuzYiVUNfE/naddefQ3n4xV3nLEpIl43aP3NmAViOJlgHVDDZhBDZ4cY 1dnQ== X-Gm-Message-State: AOAM531REnwJZYkaG+h01g6qX1t0T5Ed1cjPlx9jio09aDSCy2r3wrka tKf8TpbZsmacO1198EyvTXRCQ0iDkPLFH8in X-Google-Smtp-Source: ABdhPJz380Pp/cGH6Baq4VxLoEuRkjsiPsSYZEAXsYq2laUiz20L3FAwnY2rPzQ9X/U5s4Pa1m2jdw== X-Received: by 2002:a1c:c254:: with SMTP id s81mr8048005wmf.132.1610213208414; Sat, 09 Jan 2021 09:26:48 -0800 (PST) Received: from my-little-tumbleweed ([2a01:e0a:20e:d340:922b:34ff:fe95:9aed]) by smtp.gmail.com with ESMTPSA id m21sm15979328wml.13.2021.01.09.09.26.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 09 Jan 2021 09:26:47 -0800 (PST) From: =?utf-8?Q?K=C3=A9vin_Le_Gouguec?= To: Andrea Corallo Subject: Re: bug#45705: [feature/native-comp] Excessive memory consumption on windows 10 References: <87y2h5hjve.fsf@gmail.com> <831revjyja.fsf@gnu.org> <83turrif53.fsf@gnu.org> <83lfd2ilwf.fsf@gnu.org> <83y2h2gw9v.fsf@gnu.org> Date: Sat, 09 Jan 2021 18:26:46 +0100 In-Reply-To: (Andrea Corallo's message of "Sat, 09 Jan 2021 12:37:54 +0000") Message-ID: <87im86c97t.fsf@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.0.50 (gnu/linux) MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 45705 Cc: Eli Zaretskii , edouard.debry@gmail.com, 45705@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Andrea Corallo writes: > Eli Zaretskii writes: > >>> From: Andrea Corallo >>>=20 >>> In June we changed the way we store immediate objects in the shared and >>> this makes the compilation way lighter on the GCC side (both in time and >>> memory). I've no precise data on this other than the experimental >>> observation that compiling all Elisp files in Emacs on 32bit systems is >>> not anymore an issue. This IIUC implies that the memory footprint for >>> each compilation is always < 2GB. >> >> You assume that the compilations are all done serially? AFAIK, most >> people build Emacs with "make -jN", so parallel compilation is an >> important use case. > >> I guess we will have to collect the information about that, if you say >> we don't have it now. > > I'm adding in CC Kevin, IIRC for bug#41077 he used a nice setup to > produce quite accurate results on memory footprint during the > compilation process. Perhaps he has time and he's so kind to gather > some data on the current state, that would be extremely helpful. See also bug#41194#20 and bug#41194#28 where I outlined how the improvements reduced compilation time and memory usage. I've dusted off my 32-bit laptop; unfortunately the fan sounds like it's in need of=E2=80=A6 something (probably exorcism, given the noise). Until I figure that out, here are the (very hacky) scripts I used to measure and plot the RAM usage, in case someone else wants to take some measurements: - ./monitor.sh $PID finds the most RAM-consuming process among $PID and its children, and logs its memory usage (VSZ and RSS) and its command-line. (Logs are collected every 10 seconds; this probably needs to be reduced for faster machines) - ./plot.py uses matplotlib to make graphs out of these measurements; it attempts to replace the command line with the less-verbose diagnostics from "make". - My workflow was to start an emacs session, run M-x compile RET make, then ./monitor.sh $PID_OF_EMACS_SESSION. (PARENT_RE in plot.py should match the command-line of this parent session; its RAM consumption is then labeled as "noise floor" on the graph. This serves no real purpose and should be removed; monitor.sh should be amended to filter the parent session out of monitored PIDs, with some error control to handle the lack of child processes when compilation is finished.) - There are some hardcoded things to tweak at the bottom of plot.py, e.g. how long should a child process last for it to have a label on the graph. --=-=-= Content-Type: application/x-shellscript Content-Disposition: attachment; filename=monitor.sh Content-Transfer-Encoding: base64 IyEvYmluL2Jhc2gKCnNldCAtZXUKCm91dD0kKGRpcm5hbWUgJDApL21vbml0b3IubG9nCnBhcmVu dD0kMQoKcmVjdXJzZSAoKQp7CiAgICBlY2hvICQxCiAgICBmb3IgY2hpbGQgaW4gJChwcyAtbyBw aWQ9IC0tcHBpZCAkMSkKICAgIGRvCiAgICAgICAgcmVjdXJzZSAkY2hpbGQKICAgIGRvbmUKfQoK d2hpbGUgc2xlZXAgMTAKZG8KICAgIGRhdGUgKyIlRi0lVCIgPj4gJHtvdXR9CiAgICBwcyBvIGNw dXRpbWVzPSx2c3o9LHJzcz0sYXJncz0gLS1zb3J0PS12c3ogcCAkKHJlY3Vyc2UgJHtwYXJlbnR9 KSB8IGhlYWQgLTEgPj4gJHtvdXR9CiAgICBmcmVlID4+ICR7b3V0fQogICAgZWNobyA+PiAke291 dH0KZG9uZQo= --=-=-= Content-Type: text/x-python; charset=utf-8 Content-Disposition: attachment; filename=plot.py Content-Transfer-Encoding: quoted-printable #!/usr/bin/env python3 from datetime import datetime, timedelta from pathlib import Path import re import matplotlib from matplotlib import pyplot from matplotlib.dates import DateFormatter, HourLocator, MinuteLocator from matplotlib.ticker import EngFormatter MONITOR_RE =3D re.compile('\n'.join(( '(?P