From debbugs-submit-bounces@debbugs.gnu.org Tue Apr 03 10:06:39 2018 Received: (at submit) by debbugs.gnu.org; 3 Apr 2018 14:06:39 +0000 Received: from localhost ([127.0.0.1]:37129 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1f3Ma7-0003IC-0S for submit@debbugs.gnu.org; Tue, 03 Apr 2018 10:06:39 -0400 Received: from eggs.gnu.org ([208.118.235.92]:44602) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1f3HTo-0008IL-BQ for submit@debbugs.gnu.org; Tue, 03 Apr 2018 04:39:48 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1f3HTf-0004Ej-T8 for submit@debbugs.gnu.org; Tue, 03 Apr 2018 04:39:43 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=0.8 required=5.0 tests=BAYES_50,T_RP_MATCHES_RCVD autolearn=disabled version=3.3.2 Received: from lists.gnu.org ([2001:4830:134:3::11]:49872) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1f3HTf-0004Ed-QB for submit@debbugs.gnu.org; Tue, 03 Apr 2018 04:39:39 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:35553) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1f3HTe-0006uq-0f for bug-gnu-emacs@gnu.org; Tue, 03 Apr 2018 04:39:39 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1f3HTc-0004CR-IS for bug-gnu-emacs@gnu.org; Tue, 03 Apr 2018 04:39:38 -0400 Received: from linux-libre.fsfla.org ([208.118.235.54]:40224) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1f3HTc-0004CL-Dp for bug-gnu-emacs@gnu.org; Tue, 03 Apr 2018 04:39:36 -0400 Received: from free.home (home.lxoliva.fsfla.org [172.31.160.164]) by linux-libre.fsfla.org (8.14.4/8.14.4/Debian-4.1ubuntu1) with ESMTP id w338dWMS018198 for ; Tue, 3 Apr 2018 08:39:34 GMT Received: from livre (livre.home [172.31.160.2]) by free.home (8.15.2/8.15.2) with ESMTP id w338dPMU160630; Tue, 3 Apr 2018 05:39:26 -0300 From: Alexandre Oliva To: bug-gnu-emacs@gnu.org Subject: 25.3; Memory use explodes splitting email into gnus nnfolders Date: Tue, 03 Apr 2018 05:39:25 -0300 Message-ID: MIME-Version: 1.0 Content-Type: text/plain X-Scanned-By: MIMEDefang 2.84 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6.x X-Received-From: 2001:4830:134:3::11 X-Spam-Score: -5.0 (-----) X-Debbugs-Envelope-To: submit X-Mailman-Approved-At: Tue, 03 Apr 2018 10:06:38 -0400 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -5.0 (-----) I have some 5 GB of active gnus nnfolder files. Emacs memory use, in a fresh -nw remote session, skyrocketed to over 55GB while splitting some 150MB of email. This happened to me today. However, this has not been unusual. Smaller batches of email have often used up over 20GB of RAM; my poor x200, limited to 8GB of RAM, hasn't been able to split my mail any more for months; I can't promise this started with 25.3, it may have been before that, but some older version of Emacs had little trouble splitting similar aomunts of email onto pretty much the same set of folders with 8GB, and even 4GB of RAM, though the latter was already pushing it a longer while ago. I'm clueless as to how to debug this, or even to identify the source of such outrageous memory consumption. All I know is that memory use grows while gnus is splitting email, keeping multiple unsaved folders open until the splitting is complete and it goes about saving them all, at which point memory use goes back down. While splitting is ongoing, it doesn't look like the working set is that large, and, given enough swap space, it will complete, slowed down more by the swapping out of inactive pages than by actually bringing stuff back in. Pointers about debugging Emacs memory use while it's active would be appreciated (interrupting mail splitting in the middle is not such a great idea). I'm using a Fedora build of GNU Emacs, if that matters, but I could build custom versions if that would help. I can do some Emacs Lisp too, though not much. Thanks in advance for any advice, In GNU Emacs 25.3.1 (x86_64-redhat-linux-gnu, GTK+ Version 3.22.19) of 2017-09-14 built on buildvm-31.phx2.fedoraproject.org Windowing system distributor 'Fedora Project', version 11.0.11906000 System Description: Generic release 27 (Generic) Configured using: 'configure --build=x86_64-redhat-linux-gnu --host=x86_64-redhat-linux-gnu --program-prefix= --disable-dependency-tracking --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc --datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib64 --libexecdir=/usr/libexec --localstatedir=/var --sharedstatedir=/var/lib --mandir=/usr/share/man --infodir=/usr/share/info --with-dbus --with-gif --with-jpeg --with-png --with-rsvg --with-tiff --with-xft --with-xpm --with-x-toolkit=gtk3 --with-gpm=no --with-xwidgets --with-modules build_alias=x86_64-redhat-linux-gnu host_alias=x86_64-redhat-linux-gnu 'CFLAGS=-DMAIL_USE_LOCKF -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic' LDFLAGS=-Wl,-z,relro PKG_CONFIG_PATH=:/usr/lib64/pkgconfig:/usr/share/pkgconfig' Configured features: XPM JPEG TIFF GIF PNG RSVG IMAGEMAGICK SOUND DBUS GCONF GSETTINGS NOTIFY ACL LIBSELINUX GNUTLS LIBXML2 FREETYPE M17N_FLT LIBOTF XFT ZLIB TOOLKIT_SCROLL_BARS GTK3 X11 MODULES XWIDGETS Important settings: value of $LANG: en_US.UTF-8 value of $XMODIFIERS: @im=none locale-coding-system: utf-8-unix Major mode: Group Minor modes in effect: diff-auto-refine-mode: t shell-dirtrack-mode: t gnus-topic-mode: t cursor-sensor-mode: t gnus-undo-mode: t delete-selection-mode: t tooltip-mode: t global-eldoc-mode: t electric-indent-mode: t mouse-wheel-mode: t tool-bar-mode: t menu-bar-mode: t file-name-shadow-mode: t global-font-lock-mode: t font-lock-mode: t blink-cursor-mode: t auto-composition-mode: t auto-encryption-mode: t auto-compression-mode: t buffer-read-only: t line-number-mode: t transient-mark-mode: t Recent messages: Deleting article 17615 in redhat.tools.team... Deleting articles...done Saving file /home/aoliva/.../mail/Mail/nnfolder/redhat/tools/team... Wrote /home/aoliva/.../mail/Mail/nnfolder/redhat/tools/team Expiring articles...done Saving /home/aoliva/.newsrc.eld... Saving file /home/aoliva/.newsrc.eld... Wrote /home/aoliva/.newsrc.eld Saving /home/aoliva/.newsrc.eld...done Making completion list... [2 times] Load-path shadows: /home/aoliva/src/elsp/site-start hides /usr/share/emacs/site-lisp/site-start /usr/share/emacs/site-lisp/lilypond-init hides /usr/share/emacs/site-lisp/site-start.d/lilypond-init Features: (shadow warnings emacsbug diff-mode flow-fill tramp-cmds tramp-cache tramp-sh tramp tramp-compat tramp-loaddefs trampver ucs-normalize shell pcomplete comint ring custom advice misearch multi-isearch gnus-dup sgml-mode shr-color color shr seq dom subr-x browse-url network-stream nsm starttls mailalias smtpmail sendmail supercite easy-mmode regi nnir mail-extr qp sort ansi-color gnus-cite mm-archive gnus-async gnus-bcklg gnus-ml disp-table gnus-topic cursor-sensor nndraft nnrss xml mm-url url url-proxy url-privacy url-expand url-methods url-history url-cookie url-domsuf url-util url-parse auth-source cl-seq eieio byte-opt bytecomp byte-compile cl-extra cconv eieio-core cl-macs gv url-vars nnml nnfolder nnmh gnus-agent gnus-srvr gnus-score score-mode nnvirtual gnus-msg gnus-art mm-uu mml2015 mm-view mml-smime smime dig mailcap gnus-cache gnus-demon nntp gnus-mlspl gnus-sum gnus-group gnus-undo gnus-start gnus-cloud nnimap nnmail mail-source tls gnutls utf7 netrc nnoo parse-time gnus-spec gnus-int gnus-range message idna dired format-spec rfc822 mml mml-sec password-cache epg epg-config mm-decode mm-bodies mm-encode mail-parse rfc2231 rfc2047 rfc2045 ietf-drums mailabbrev gmm-utils mailheader gnus-win gnus gnus-ems nnheader gnus-util mail-utils mm-util help-fns help-mode easymenu mail-prsvr wid-edit cl-loaddefs pcase cl-lib delsel cus-start cus-load flyspell ispell iso-transl jka-compr dot-emacs preview-latex tex-site auto-loads time-date mule-util tooltip eldoc electric uniquify ediff-hook vc-hooks lisp-float-type mwheel x-win term/common-win x-dnd tool-bar dnd fontset image regexp-opt fringe tabulated-list newcomment elisp-mode lisp-mode prog-mode register page menu-bar rfn-eshadow timer select scroll-bar mouse jit-lock font-lock syntax facemenu font-core frame cl-generic cham georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek romanian slovak czech european ethiopic indian cyrillic chinese charscript case-table epa-hook jka-cmpr-hook help simple abbrev minibuffer cl-preloaded nadvice loaddefs button faces cus-face macroexp files text-properties overlay sha1 md5 base64 format env code-pages mule widget hashtable-print-readable backquote dbusbind inotify dynamic-setting system-font-setting font-render-setting xwidget-internal move-toolbar gtk x-toolkit x multi-tty make-network-process emacs) Memory information: ((conses 16 536826 110508) (symbols 48 37634 64) (miscs 40 330 1684) (strings 32 70356 15142) (string-bytes 1 2305445) (vectors 16 56895) (vector-slots 8 2174791 108123) (floats 8 1059 549) (intervals 56 6371 615) (buffers 976 65)) -- Alexandre Oliva, freedom fighter http://FSFLA.org/~lxoliva/ You must be the change you wish to see in the world. -- Gandhi Be Free! -- http://FSFLA.org/ FSF Latin America board member Free Software Evangelist|Red Hat Brasil GNU Toolchain Engineer From debbugs-submit-bounces@debbugs.gnu.org Tue Apr 03 10:17:04 2018 Received: (at 31044) by debbugs.gnu.org; 3 Apr 2018 14:17:04 +0000 Received: from localhost ([127.0.0.1]:37144 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1f3MkC-0003YT-0l for submit@debbugs.gnu.org; Tue, 03 Apr 2018 10:17:04 -0400 Received: from eggs.gnu.org ([208.118.235.92]:48983) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1f3MkA-0003Xw-TA for 31044@debbugs.gnu.org; Tue, 03 Apr 2018 10:17:03 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1f3Mk4-0004sc-Sw for 31044@debbugs.gnu.org; Tue, 03 Apr 2018 10:16:57 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=0.8 required=5.0 tests=BAYES_50,T_RP_MATCHES_RCVD autolearn=disabled version=3.3.2 Received: from fencepost.gnu.org ([2001:4830:134:3::e]:43081) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1f3Mjz-0004k7-44; Tue, 03 Apr 2018 10:16:51 -0400 Received: from [176.228.60.248] (port=2764 helo=home-c4e4a596f7) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1f3Mjy-00061c-Ig; Tue, 03 Apr 2018 10:16:50 -0400 Date: Tue, 03 Apr 2018 17:16:58 +0300 Message-Id: <83zi2k4bol.fsf@gnu.org> From: Eli Zaretskii To: Alexandre Oliva In-reply-to: (message from Alexandre Oliva on Tue, 03 Apr 2018 05:39:25 -0300) Subject: Re: bug#31044: 25.3; Memory use explodes splitting email into gnus nnfolders References: X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 2001:4830:134:3::e X-Spam-Score: -5.0 (-----) X-Debbugs-Envelope-To: 31044 Cc: 31044@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Eli Zaretskii Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -5.0 (-----) > From: Alexandre Oliva > Date: Tue, 03 Apr 2018 05:39:25 -0300 > > > I have some 5 GB of active gnus nnfolder files. Emacs memory use, in a > fresh -nw remote session, skyrocketed to over 55GB while splitting some > 150MB of email. This happened to me today. > > However, this has not been unusual. Smaller batches of email have often > used up over 20GB of RAM; my poor x200, limited to 8GB of RAM, hasn't > been able to split my mail any more for months; I can't promise this > started with 25.3, it may have been before that, but some older version > of Emacs had little trouble splitting similar aomunts of email onto > pretty much the same set of folders with 8GB, and even 4GB of RAM, > though the latter was already pushing it a longer while ago. Can you try the latest pretest of Emacs 26.1 (it's available from alpha.gnu), or building the emacs-26 branch of the Emacs Git repository? This could be due to some memory related issues that are supposed to be fixed in Emacs 26. Thanks. From debbugs-submit-bounces@debbugs.gnu.org Wed Apr 04 13:06:39 2018 Received: (at 31044) by debbugs.gnu.org; 4 Apr 2018 17:06:39 +0000 Received: from localhost ([127.0.0.1]:38532 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1f3lrq-0006DJ-CO for submit@debbugs.gnu.org; Wed, 04 Apr 2018 13:06:38 -0400 Received: from linux-libre.fsfla.org ([208.118.235.54]:34092) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1f3kyz-0004rn-D6 for 31044@debbugs.gnu.org; Wed, 04 Apr 2018 12:09:58 -0400 Received: from free.home (home.lxoliva.fsfla.org [172.31.160.164]) by linux-libre.fsfla.org (8.14.4/8.14.4/Debian-4.1ubuntu1) with ESMTP id w34G9qUe002503; Wed, 4 Apr 2018 16:09:54 GMT Received: from livre (livre.home [172.31.160.2]) by free.home (8.15.2/8.15.2) with ESMTP id w34G9klj275685; Wed, 4 Apr 2018 13:09:46 -0300 From: Alexandre Oliva To: Eli Zaretskii Subject: Re: bug#31044: 25.3; Memory use explodes splitting email into gnus nnfolders Organization: Free thinker, not speaking for the GNU Project References: <83zi2k4bol.fsf@gnu.org> Date: Wed, 04 Apr 2018 13:09:46 -0300 In-Reply-To: <83zi2k4bol.fsf@gnu.org> (Eli Zaretskii's message of "Tue, 03 Apr 2018 17:16:58 +0300") Message-ID: User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.0.91 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Scanned-By: MIMEDefang 2.84 X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 31044 X-Mailman-Approved-At: Wed, 04 Apr 2018 13:06:36 -0400 Cc: 31044@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.0 (/) On Apr 3, 2018, Eli Zaretskii wrote: > Can you try the latest pretest of Emacs 26.1 Oh, sure, yeah, sorry I didn't think of that. I grabbed 26.0.91 from alpha and I'm now pretty sure it fixes the problem. Yay! Thanks! -- Alexandre Oliva, freedom fighter http://FSFLA.org/~lxoliva/ You must be the change you wish to see in the world. -- Gandhi Be Free! -- http://FSFLA.org/ FSF Latin America board member Free Software Evangelist|Red Hat Brasil GNU Toolchain Engineer From debbugs-submit-bounces@debbugs.gnu.org Wed Apr 04 15:02:43 2018 Received: (at 31044-done) by debbugs.gnu.org; 4 Apr 2018 19:02:43 +0000 Received: from localhost ([127.0.0.1]:38637 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1f3ngB-0000Rp-Hj for submit@debbugs.gnu.org; Wed, 04 Apr 2018 15:02:43 -0400 Received: from eggs.gnu.org ([208.118.235.92]:54970) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1f3ng8-0000Rb-Gp for 31044-done@debbugs.gnu.org; Wed, 04 Apr 2018 15:02:40 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1f3ng2-00028Y-Lc for 31044-done@debbugs.gnu.org; Wed, 04 Apr 2018 15:02:35 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,T_RP_MATCHES_RCVD autolearn=disabled version=3.3.2 Received: from fencepost.gnu.org ([2001:4830:134:3::e]:47196) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1f3nfw-000226-7v; Wed, 04 Apr 2018 15:02:28 -0400 Received: from [176.228.60.248] (port=1519 helo=home-c4e4a596f7) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1f3nfv-0001kO-Ll; Wed, 04 Apr 2018 15:02:28 -0400 Date: Wed, 04 Apr 2018 22:02:39 +0300 Message-Id: <83woxm3icw.fsf@gnu.org> From: Eli Zaretskii To: Alexandre Oliva In-reply-to: (message from Alexandre Oliva on Wed, 04 Apr 2018 13:09:46 -0300) Subject: Re: bug#31044: 25.3; Memory use explodes splitting email into gnus nnfolders References: <83zi2k4bol.fsf@gnu.org> X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 2001:4830:134:3::e X-Spam-Score: -5.0 (-----) X-Debbugs-Envelope-To: 31044-done Cc: 31044-done@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Eli Zaretskii Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -5.0 (-----) > From: Alexandre Oliva > Cc: 31044@debbugs.gnu.org > Date: Wed, 04 Apr 2018 13:09:46 -0300 > > On Apr 3, 2018, Eli Zaretskii wrote: > > > Can you try the latest pretest of Emacs 26.1 > > Oh, sure, yeah, sorry I didn't think of that. I grabbed 26.0.91 from > alpha and I'm now pretty sure it fixes the problem. Yay! Thanks! OK, thanks for testing. I'm therefore closing the bug; feel free to reopen if the problem comes back. From unknown Sun Jun 22 03:57:01 2025 Received: (at fakecontrol) by fakecontrolmessage; To: internal_control@debbugs.gnu.org From: Debbugs Internal Request Subject: Internal Control Message-Id: bug archived. Date: Thu, 03 May 2018 11:24:06 +0000 User-Agent: Fakemail v42.6.9 # This is a fake control message. # # The action: # bug archived. thanks # This fakemail brought to you by your local debbugs # administrator