From debbugs-submit-bounces@debbugs.gnu.org Sat Apr 27 02:29:04 2019 Received: (at submit) by debbugs.gnu.org; 27 Apr 2019 06:29:04 +0000 Received: from localhost ([127.0.0.1]:34145 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hKGpc-0004xu-66 for submit@debbugs.gnu.org; Sat, 27 Apr 2019 02:29:04 -0400 Received: from eggs.gnu.org ([209.51.188.92]:43889) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hKGpY-0004xP-V4 for submit@debbugs.gnu.org; Sat, 27 Apr 2019 02:29:02 -0400 Received: from lists.gnu.org ([209.51.188.17]:56766) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1hKGpS-0007bX-Jy for submit@debbugs.gnu.org; Sat, 27 Apr 2019 02:28:54 -0400 Received: from eggs.gnu.org ([209.51.188.92]:58206) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hKGpR-0005EF-Cz for bug-gnu-emacs@gnu.org; Sat, 27 Apr 2019 02:28:54 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=0.8 required=5.0 tests=BAYES_50,URIBL_BLOCKED autolearn=disabled version=3.3.2 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hKGjw-0005cc-HB for bug-gnu-emacs@gnu.org; Sat, 27 Apr 2019 02:23:13 -0400 Received: from dev.gentoo.org ([2001:470:ea4a:1:5054:ff:fec7:86e4]:53583 helo=smtp.gentoo.org) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1hKGjv-0005aq-Vo for bug-gnu-emacs@gnu.org; Sat, 27 Apr 2019 02:23:12 -0400 Received: from a1i15 (host2092.kph.uni-mainz.de [134.93.134.92]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: ulm) by smtp.gentoo.org (Postfix) with ESMTPSA id 4FF86342896; Sat, 27 Apr 2019 06:23:08 +0000 (UTC) From: Ulrich Mueller To: bug-gnu-emacs@gnu.org Subject: 27.0.50; Gnus (nnimap) shows "ghost" messages in summary buffer Date: Sat, 27 Apr 2019 08:23:03 +0200 Message-ID: User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.0.50 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2001:470:ea4a:1:5054:ff:fec7:86e4 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6.x X-Spam-Score: -1.6 (-) X-Debbugs-Envelope-To: submit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.6 (--) In GNU Emacs 27.0.50 (build 1, x86_64-pc-linux-gnu, X toolkit) of 2019-04-21 built on a1i15 Windowing system distributor 'The X.Org Foundation', version 11.0.11905000 System Description: Gentoo/Linux When fetching e-mail from a dovecot-2.3.5.1 IMAP server, Gnus/nnimap gets confused and displays ghost messages with address "nobody" and subject "(none)" in the summary buffer, like in this example: *Summary nnimap+dev.gentoo.org:INBOX* ---------------------------------------------------------------------- R. [ 1: Ulrich Mueller ] test 1 . [ ?: nobody ] (none) . [ 1: Ulrich Mueller ] test 2 ---------------------------------------------------------------------- My gnus-secondary-select-methods looks like this: '((nnimap "dev.gentoo.org" (nnimap-stream tls) (nnimap-user "ulm")) ;; ... further methods omitted ... ) For debugging, I have enabled logging with nnimap-record-commands, and set debug-on-entry for function nnimap-transform-headers. I then get the following "*imap log*" buffer: ---------------------------------------------------------------------- 07:41:57 [dev.gentoo.org] (inhibited) 07:41:59 [dev.gentoo.org] 10105 CAPABILITY 07:41:59 [dev.gentoo.org] 10106 ENABLE QRESYNC 07:41:59 [dev.gentoo.org] 10107 LIST "" "*" [... other servers ...] 07:42:00 [dev.gentoo.org] 10147 EXAMINE "INBOX" (QRESYNC (1364468255 63360)) [... other servers ...] 07:42:08 [dev.gentoo.org] 10192 SELECT "INBOX" 07:42:08 [dev.gentoo.org] 10193 SELECT "INBOX" 07:42:08 [dev.gentoo.org] 10194 UID FETCH 32409:32410 (UID RFC822.SIZE BODYSTRUCTURE BODY.PEEK[HEADER.FIELDS (Subject From Date Message-Id References In-Reply-To Xref X-Diary-Time-Zone X-Diary-Dow X-Diary-Year X-Diary-Month X-Diary-Dom X-Diary-Hour X-Diary-Minute To Newsgroups Cc)]) ---------------------------------------------------------------------- Buffer " *nnimap dev.gentoo.org nil *nntpd**" looks like this, upon entering nnimap-transform-headers: ---------------------------------------------------------------------- * 583 FETCH (UID 32409 RFC822.SIZE 698 BODYSTRUCTURE ("text" "plain" ("charset" "us-ascii") NIL NIL "7bit" 8 1 NIL NIL NIL NIL) BODY[HEADER.FIELDS (SUBJECT FROM DATE MESSAGE-ID REFERENCES IN-REPLY-TO XREF X-DIARY-TIME-ZONE X-DIARY-DOW X-DIARY-YEAR X-DIARY-MONTH X-DIARY-DOM X-DIARY-HOUR X-DIARY-MINUTE TO NEWSGROUPS CC)] {165} From: Ulrich Mueller To: ulm@gentoo.org Subject: test 1 Date: Sat, 27 Apr 2019 07:39:56 +0200 Message-ID: ) * 584 FETCH (UID 32410 RFC822.SIZE 698 BODYSTRUCTURE ("text" "plain" ("charset" "us-ascii") NIL NIL "7bit" 8 1 NIL NIL NIL NIL) BODY[HEADER.FIELDS (SUBJECT FROM DATE MESSAGE-ID REFERENCES IN-REPLY-TO XREF X-DIARY-TIME-ZONE X-DIARY-DOW X-DIARY-YEAR X-DIARY-MONTH X-DIARY-DOM X-DIARY-HOUR X-DIARY-MINUTE TO NEWSGROUPS CC)] {165} From: Ulrich Mueller To: ulm@gentoo.org Subject: test 2 Date: Sat, 27 Apr 2019 07:40:15 +0200 Message-ID: ) * 583 FETCH (UID 32409 MODSEQ (63364) FLAGS ($HasNoAttachment)) * 584 FETCH (UID 32410 MODSEQ (63364) FLAGS ($HasNoAttachment)) 10194 OK Fetch completed (0.003 + 0.000 + 0.002 secs). ---------------------------------------------------------------------- From debbugs-submit-bounces@debbugs.gnu.org Sat Apr 27 11:03:42 2019 Received: (at 35443) by debbugs.gnu.org; 27 Apr 2019 15:03:42 +0000 Received: from localhost ([127.0.0.1]:35734 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hKOrd-0005GG-R7 for submit@debbugs.gnu.org; Sat, 27 Apr 2019 11:03:42 -0400 Received: from ericabrahamsen.net ([52.70.2.18]:38238 helo=mail.ericabrahamsen.net) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hKOrb-0005G1-Po for 35443@debbugs.gnu.org; Sat, 27 Apr 2019 11:03:40 -0400 Received: from localhost (75-172-115-9.tukw.qwest.net [75.172.115.9]) (Authenticated sender: eric@ericabrahamsen.net) by mail.ericabrahamsen.net (Postfix) with ESMTPSA id EF23AFA59D; Sat, 27 Apr 2019 15:03:32 +0000 (UTC) From: Eric Abrahamsen To: Ulrich Mueller Subject: Re: bug#35443: 27.0.50; Gnus (nnimap) shows "ghost" messages in summary buffer References: Date: Sat, 27 Apr 2019 08:03:29 -0700 In-Reply-To: (Ulrich Mueller's message of "Sat, 27 Apr 2019 08:23:03 +0200") Message-ID: <87wojfjxry.fsf@ericabrahamsen.net> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.0.50 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 35443 Cc: 35443@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Ulrich Mueller writes: > In GNU Emacs 27.0.50 (build 1, x86_64-pc-linux-gnu, X toolkit) > of 2019-04-21 built on a1i15 > Windowing system distributor 'The X.Org Foundation', version 11.0.11905000 > System Description: Gentoo/Linux > > When fetching e-mail from a dovecot-2.3.5.1 IMAP server, Gnus/nnimap > gets confused and displays ghost messages with address "nobody" and > subject "(none)" in the summary buffer, like in this example: Are you able to build an earlier version of Emacs (say, from a couple of months ago) and test if you see the same thing? I made some fairly deep changes to Gnus over the past month, and while I don't see any reason why that would cause this behavior, it would be nice to rule it out. Thanks, Eric From debbugs-submit-bounces@debbugs.gnu.org Sat Apr 27 11:50:37 2019 Received: (at 35443) by debbugs.gnu.org; 27 Apr 2019 15:50:37 +0000 Received: from localhost ([127.0.0.1]:35788 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hKPb3-0006Wh-4J for submit@debbugs.gnu.org; Sat, 27 Apr 2019 11:50:37 -0400 Received: from smtp.gentoo.org ([140.211.166.183]:53792) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hKPb0-0006WS-D0 for 35443@debbugs.gnu.org; Sat, 27 Apr 2019 11:50:35 -0400 Received: from a1i15 (host2092.kph.uni-mainz.de [134.93.134.92]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: ulm) by smtp.gentoo.org (Postfix) with ESMTPSA id 5E46E342960; Sat, 27 Apr 2019 15:50:26 +0000 (UTC) From: Ulrich Mueller To: Eric Abrahamsen Subject: Re: bug#35443: 27.0.50; Gnus (nnimap) shows "ghost" messages in summary buffer References: <87wojfjxry.fsf@ericabrahamsen.net> Date: Sat, 27 Apr 2019 17:49:17 +0200 In-Reply-To: <87wojfjxry.fsf@ericabrahamsen.net> (Eric Abrahamsen's message of "Sat, 27 Apr 2019 08:03:29 -0700") Message-ID: User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.0.50 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: -5.0 (-----) X-Debbugs-Envelope-To: 35443 Cc: 35443@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -6.0 (------) >>>>> On Sat, 27 Apr 2019, Eric Abrahamsen wrote: > Are you able to build an earlier version of Emacs (say, from a couple > of months ago) and test if you see the same thing? I made some fairly > deep changes to Gnus over the past month, and while I don't see any > reason why that would cause this behavior, it would be nice to rule it > out. Yes, I had tried with Emacs 26.2 and it has the same problem. From debbugs-submit-bounces@debbugs.gnu.org Sat Apr 27 16:47:29 2019 Received: (at 35443) by debbugs.gnu.org; 27 Apr 2019 20:47:29 +0000 Received: from localhost ([127.0.0.1]:36107 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hKUEL-0003mX-9R for submit@debbugs.gnu.org; Sat, 27 Apr 2019 16:47:29 -0400 Received: from ericabrahamsen.net ([52.70.2.18]:45664 helo=mail.ericabrahamsen.net) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hKUEJ-0003mJ-NB for 35443@debbugs.gnu.org; Sat, 27 Apr 2019 16:47:28 -0400 Received: from localhost (75-172-115-9.tukw.qwest.net [75.172.115.9]) (Authenticated sender: eric@ericabrahamsen.net) by mail.ericabrahamsen.net (Postfix) with ESMTPSA id 4D4C3FA026; Sat, 27 Apr 2019 20:47:21 +0000 (UTC) From: Eric Abrahamsen To: Ulrich Mueller Subject: Re: bug#35443: 27.0.50; Gnus (nnimap) shows "ghost" messages in summary buffer References: <87wojfjxry.fsf@ericabrahamsen.net> Date: Sat, 27 Apr 2019 13:47:18 -0700 In-Reply-To: (Ulrich Mueller's message of "Sat, 27 Apr 2019 17:49:17 +0200") Message-ID: <87h8ajjhux.fsf@ericabrahamsen.net> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.0.50 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 35443 Cc: 35443@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Ulrich Mueller writes: >>>>>> On Sat, 27 Apr 2019, Eric Abrahamsen wrote: > >> Are you able to build an earlier version of Emacs (say, from a couple >> of months ago) and test if you see the same thing? I made some fairly >> deep changes to Gnus over the past month, and while I don't see any >> reason why that would cause this behavior, it would be nice to rule it >> out. > > Yes, I had tried with Emacs 26.2 and it has the same problem. Huh. Can you show us the value of gnus-newsgroup-data and gnus-newsgroup-headers in this group? Does the dummy article always show up? Only in this group? Is this a newly-added IMAP server, and did it display this problem right from the beginning? From debbugs-submit-bounces@debbugs.gnu.org Sat Apr 27 23:53:22 2019 Received: (at 35443) by debbugs.gnu.org; 28 Apr 2019 03:53:22 +0000 Received: from localhost ([127.0.0.1]:36520 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hKasU-0007na-5O for submit@debbugs.gnu.org; Sat, 27 Apr 2019 23:53:22 -0400 Received: from smtp.gentoo.org ([140.211.166.183]:36820) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hKasS-0007nL-99 for 35443@debbugs.gnu.org; Sat, 27 Apr 2019 23:53:20 -0400 Received: from a1i15 (host2092.kph.uni-mainz.de [134.93.134.92]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: ulm) by smtp.gentoo.org (Postfix) with ESMTPSA id 2EAD2342AA5; Sun, 28 Apr 2019 03:53:13 +0000 (UTC) From: Ulrich Mueller To: Eric Abrahamsen Subject: Re: bug#35443: 27.0.50; Gnus (nnimap) shows "ghost" messages in summary buffer References: <87wojfjxry.fsf@ericabrahamsen.net> <87h8ajjhux.fsf@ericabrahamsen.net> Date: Sun, 28 Apr 2019 05:53:09 +0200 In-Reply-To: <87h8ajjhux.fsf@ericabrahamsen.net> (Eric Abrahamsen's message of "Sat, 27 Apr 2019 13:47:18 -0700") Message-ID: User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.0.50 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -5.0 (-----) X-Debbugs-Envelope-To: 35443 Cc: 35443@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -6.0 (------) >>>>> On Sat, 27 Apr 2019, Eric Abrahamsen wrote: > Huh. Can you show us the value of gnus-newsgroup-data and > gnus-newsgroup-headers in this group? See below, for the current group, whose summary buffer shows your article and a dummy article. > Does the dummy article always show up? It always shows up when new articles have been fetched. It doesn't show up when entering a group that doesn't have new mail (i.e., if a group previously had a dummy article, it will be gone when reentering that group). > Only in this group? All groups for that particular IMAP server.=20 > Is this a newly-added IMAP server, and did it display this problem > right from the beginning? AFAICS, the problem appeared after dovecot on the server side was upgraded from version 2.2.34 to 2.3.5.1. But I see it only with Gnus. I don't have any issues with Mozilla Thunderbird (on GNU/Linux) nor with K-9 Mail (on Android/Linux). ---------------------------------------------------------------------- gnus-newsgroup-data is a variable defined in =E2=80=98gnus-sum.el=E2=80=99. Its value is shown below. Documentation: Not documented as a variable. Value: ((32415 82 2 [32415 "Re: bug#35443: 27.0.50; Gnus (nnimap) shows \"ghost\" messages in = summary buffer" "Eric Abrahamsen " "Sat, 27 Apr 20= 19 13:47:18 -0700" "<87h8ajjhux.fsf@ericabrahamsen.net>" " <87wojfjxry.fsf@ericabrahamsen.net> " 2609 16 nil ((Cc . "35443@debbugs.gnu.org") (To . "Ulrich Mueller "))] 0) (32415 32 116 [32415 "(none)" "(nobody)" "" "fake+none+nnimap+dev.gentoo.org:INBOX+32415= " nil -1 -1 nil nil] 0)) Local in buffer *Summary nnimap+dev.gentoo.org:INBOX*; global value is the = same. ---------------------------------------------------------------------- gnus-newsgroup-headers is a variable defined in =E2=80=98gnus-sum.el=E2=80= =99. Its value is ([32415 "Re: bug#35443: 27.0.50; Gnus (nnimap) shows \"ghost\" messages in = summary buffer" "Eric Abrahamsen " "Sat, 27 Apr 20= 19 13:47:18 -0700" "<87h8ajjhux.fsf@ericabrahamsen.net>" " <87wojfjxry.fsf@ericabrahamsen.net> " 2609 16 nil ((Cc . "35443@debbugs.gnu.org") (To . "Ulrich Mueller "))] [32415 "(none)" "(nobody)" "" "fake+none+nnimap+dev.gentoo.org:INBOX+32415= " nil -1 -1 nil nil]) Local in buffer *Summary nnimap+dev.gentoo.org:INBOX*; global value is nil Documentation: List of article headers in the current newsgroup. ---------------------------------------------------------------------- From debbugs-submit-bounces@debbugs.gnu.org Sun Apr 28 20:43:29 2019 Received: (at 35443) by debbugs.gnu.org; 29 Apr 2019 00:43:29 +0000 Received: from localhost ([127.0.0.1]:38079 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hKuOG-00078O-Nk for submit@debbugs.gnu.org; Sun, 28 Apr 2019 20:43:28 -0400 Received: from ericabrahamsen.net ([52.70.2.18]:54520 helo=mail.ericabrahamsen.net) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hKuOE-00078B-CZ for 35443@debbugs.gnu.org; Sun, 28 Apr 2019 20:43:27 -0400 Received: from localhost (97-113-61-17.tukw.qwest.net [97.113.61.17]) (Authenticated sender: eric@ericabrahamsen.net) by mail.ericabrahamsen.net (Postfix) with ESMTPSA id E64CBFA023; Mon, 29 Apr 2019 00:43:19 +0000 (UTC) From: Eric Abrahamsen To: Ulrich Mueller Subject: Re: bug#35443: 27.0.50; Gnus (nnimap) shows "ghost" messages in summary buffer References: <87wojfjxry.fsf@ericabrahamsen.net> <87h8ajjhux.fsf@ericabrahamsen.net> Date: Sun, 28 Apr 2019 17:43:16 -0700 In-Reply-To: (Ulrich Mueller's message of "Sun, 28 Apr 2019 05:53:09 +0200") Message-ID: <87lfztiqu3.fsf@ericabrahamsen.net> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.0.50 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 35443 Cc: 35443@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Ulrich Mueller writes: >>>>>> On Sat, 27 Apr 2019, Eric Abrahamsen wrote: > >> Huh. Can you show us the value of gnus-newsgroup-data and >> gnus-newsgroup-headers in this group? > > See below, for the current group, whose summary buffer shows your > article and a dummy article. > >> Does the dummy article always show up? > > It always shows up when new articles have been fetched. It doesn't > show up when entering a group that doesn't have new mail (i.e., if a > group previously had a dummy article, it will be gone when reentering > that group). > >> Only in this group? > > All groups for that particular IMAP server. > >> Is this a newly-added IMAP server, and did it display this problem >> right from the beginning? > > AFAICS, the problem appeared after dovecot on the server side was > upgraded from version 2.2.34 to 2.3.5.1. But I see it only with Gnus. > I don't have any issues with Mozilla Thunderbird (on GNU/Linux) nor with > K-9 Mail (on Android/Linux). Thanks, this should be enough to make some progress in tracking the problem down. Give me a couple of days... From debbugs-submit-bounces@debbugs.gnu.org Tue May 07 16:34:37 2019 Received: (at 35443) by debbugs.gnu.org; 7 May 2019 20:34:37 +0000 Received: from localhost ([127.0.0.1]:59750 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hO6nM-0003n0-V7 for submit@debbugs.gnu.org; Tue, 07 May 2019 16:34:37 -0400 Received: from ericabrahamsen.net ([52.70.2.18]:39818 helo=mail.ericabrahamsen.net) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hO6nK-0003mh-Ax for 35443@debbugs.gnu.org; Tue, 07 May 2019 16:34:35 -0400 Received: from localhost (50-251-205-17-static.hfc.comcastbusiness.net [50.251.205.17]) (Authenticated sender: eric@ericabrahamsen.net) by mail.ericabrahamsen.net (Postfix) with ESMTPSA id 5B11EFA026; Tue, 7 May 2019 20:34:27 +0000 (UTC) From: Eric Abrahamsen To: Ulrich Mueller Subject: Re: bug#35443: 27.0.50; Gnus (nnimap) shows "ghost" messages in summary buffer References: Date: Tue, 07 May 2019 13:34:24 -0700 In-Reply-To: (Ulrich Mueller's message of "Sat, 27 Apr 2019 08:23:03 +0200") Message-ID: <87d0kugg1b.fsf@ericabrahamsen.net> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.0.50 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 35443 Cc: 35443@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Ulrich Mueller writes: > In GNU Emacs 27.0.50 (build 1, x86_64-pc-linux-gnu, X toolkit) > of 2019-04-21 built on a1i15 > Windowing system distributor 'The X.Org Foundation', version 11.0.11905000 > System Description: Gentoo/Linux > > When fetching e-mail from a dovecot-2.3.5.1 IMAP server, Gnus/nnimap > gets confused and displays ghost messages with address "nobody" and > subject "(none)" in the summary buffer, like in this example: > > *Summary nnimap+dev.gentoo.org:INBOX* > ---------------------------------------------------------------------- > R. [ 1: Ulrich Mueller ] test 1 > . [ ?: nobody ] (none) > . [ 1: Ulrich Mueller ] test 2 > ---------------------------------------------------------------------- [...] > Buffer " *nnimap dev.gentoo.org nil *nntpd**" looks like this, upon > entering nnimap-transform-headers: >---------------------------------------------------------------------- > * 583 FETCH (UID 32409 RFC822.SIZE 698 BODYSTRUCTURE ("text" "plain" ("charset" "us-ascii") NIL NIL "7bit" 8 1 NIL NIL NIL NIL) BODY[HEADER.FIELDS (SUBJECT FROM DATE MESSAGE-ID REFERENCES IN-REPLY-TO XREF X-DIARY-TIME-ZONE X-DIARY-DOW X-DIARY-YEAR X-DIARY-MONTH X-DIARY-DOM X-DIARY-HOUR X-DIARY-MINUTE TO NEWSGROUPS CC)] {165} > From: Ulrich Mueller > To: ulm@gentoo.org > Subject: test 1 > Date: Sat, 27 Apr 2019 07:39:56 +0200 > Message-ID: > > ) > * 584 FETCH (UID 32410 RFC822.SIZE 698 BODYSTRUCTURE ("text" "plain" ("charset" "us-ascii") NIL NIL "7bit" 8 1 NIL NIL NIL NIL) BODY[HEADER.FIELDS (SUBJECT FROM DATE MESSAGE-ID REFERENCES IN-REPLY-TO XREF X-DIARY-TIME-ZONE X-DIARY-DOW X-DIARY-YEAR X-DIARY-MONTH X-DIARY-DOM X-DIARY-HOUR X-DIARY-MINUTE TO NEWSGROUPS CC)] {165} > From: Ulrich Mueller > To: ulm@gentoo.org > Subject: test 2 > Date: Sat, 27 Apr 2019 07:40:15 +0200 > Message-ID: > > ) > * 583 FETCH (UID 32409 MODSEQ (63364) FLAGS ($HasNoAttachment)) > * 584 FETCH (UID 32410 MODSEQ (63364) FLAGS ($HasNoAttachment)) > 10194 OK Fetch completed (0.003 + 0.000 + 0.002 secs). > ---------------------------------------------------------------------- Okay, I've made a bit of progress on this. Locally I'm using Dovecot 2.3.6, and it does not output those last two lines before the OK. `nnimap-transform-headers' is not expecting those two lines -- it deletes all but the last, leaving a buffer that looks like: 211 32409 Article retrieved. Chars: 698 Lines: 1 From: Ulrich Mueller To: ulm@gentoo.org Subject: test 1 Date: Sat, 27 Apr 2019 07:39:56 +0200 Message-ID: . 211 32410 Article retrieved. Chars: 698 Lines: 1 From: Ulrich Mueller To: ulm@gentoo.org Subject: test 2 Date: Sat, 27 Apr 2019 07:40:15 +0200 Message-ID: . 211 32409 Article retrieved. * 584 FETCH (UID 32410 MODSEQ (63364) FLAGS ($HasNoAttachment)) 10194 OK Fetch completed (0.003 + 0.000 + 0.002 secs). . Which Gnus then parses as an _extra_ article 32409, but then there's no header data for it, which is why you get all the "nobody" "none" nonsense. Essentially, we're not set up to parse this particular return value. I guess what I'll do is ask on the dovecot mailing list under what circumstances/versions we'd get a string like that, and try to help Gnus parse it correctly. Thanks for your patience, Eric From debbugs-submit-bounces@debbugs.gnu.org Tue May 07 17:14:30 2019 Received: (at 35443) by debbugs.gnu.org; 7 May 2019 21:14:30 +0000 Received: from localhost ([127.0.0.1]:59806 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hO7Py-0004hi-IF for submit@debbugs.gnu.org; Tue, 07 May 2019 17:14:30 -0400 Received: from ericabrahamsen.net ([52.70.2.18]:40802 helo=mail.ericabrahamsen.net) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hO7Pv-0004hS-V1 for 35443@debbugs.gnu.org; Tue, 07 May 2019 17:14:29 -0400 Received: from localhost (50-251-205-17-static.hfc.comcastbusiness.net [50.251.205.17]) (Authenticated sender: eric@ericabrahamsen.net) by mail.ericabrahamsen.net (Postfix) with ESMTPSA id D27DBFA026; Tue, 7 May 2019 21:14:20 +0000 (UTC) From: Eric Abrahamsen To: Ulrich Mueller Subject: Re: bug#35443: 27.0.50; Gnus (nnimap) shows "ghost" messages in summary buffer References: <87wojfjxry.fsf@ericabrahamsen.net> <87h8ajjhux.fsf@ericabrahamsen.net> Date: Tue, 07 May 2019 14:14:18 -0700 In-Reply-To: (Ulrich Mueller's message of "Sun, 28 Apr 2019 05:53:09 +0200") Message-ID: <87woj2ezmd.fsf@ericabrahamsen.net> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.0.50 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 35443 Cc: 35443@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) On 04/28/19 05:53 AM, Ulrich Mueller wrote: >>>>>> On Sat, 27 Apr 2019, Eric Abrahamsen wrote: > >> Huh. Can you show us the value of gnus-newsgroup-data and >> gnus-newsgroup-headers in this group? > > See below, for the current group, whose summary buffer shows your > article and a dummy article. Can you also tell me your value for `nnmail-extra-headers'? Thanks, Eric From debbugs-submit-bounces@debbugs.gnu.org Wed May 08 03:07:21 2019 Received: (at 35443) by debbugs.gnu.org; 8 May 2019 07:07:21 +0000 Received: from localhost ([127.0.0.1]:60229 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hOGfh-0005El-HA for submit@debbugs.gnu.org; Wed, 08 May 2019 03:07:21 -0400 Received: from smtp.gentoo.org ([140.211.166.183]:58670) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hOGfg-0005EZ-6t for 35443@debbugs.gnu.org; Wed, 08 May 2019 03:07:20 -0400 Received: from a1i15 (host2092.kph.uni-mainz.de [134.93.134.92]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: ulm) by smtp.gentoo.org (Postfix) with ESMTPSA id 1FE06343BD0; Wed, 8 May 2019 07:07:12 +0000 (UTC) From: Ulrich Mueller To: Eric Abrahamsen Subject: Re: bug#35443: 27.0.50; Gnus (nnimap) shows "ghost" messages in summary buffer References: <87wojfjxry.fsf@ericabrahamsen.net> <87h8ajjhux.fsf@ericabrahamsen.net> <87woj2ezmd.fsf@ericabrahamsen.net> Date: Wed, 08 May 2019 09:07:08 +0200 In-Reply-To: <87woj2ezmd.fsf@ericabrahamsen.net> (Eric Abrahamsen's message of "Tue, 07 May 2019 14:14:18 -0700") Message-ID: User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.0.50 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: -5.0 (-----) X-Debbugs-Envelope-To: 35443 Cc: 35443@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -6.0 (------) >>>>> On Tue, 07 May 2019, Eric Abrahamsen wrote: > Can you also tell me your value for `nnmail-extra-headers'? (X-Diary-Time-Zone X-Diary-Dow X-Diary-Year X-Diary-Month X-Diary-Dom X-Diary-Hour X-Diary-Minute To Newsgroups Cc) Not sure where the X-Diary-* ones come from, since I don't have that variable in my config files. From debbugs-submit-bounces@debbugs.gnu.org Wed May 08 11:18:40 2019 Received: (at 35443) by debbugs.gnu.org; 8 May 2019 15:18:40 +0000 Received: from localhost ([127.0.0.1]:33783 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hOOL9-0003XV-Na for submit@debbugs.gnu.org; Wed, 08 May 2019 11:18:39 -0400 Received: from smtp.gentoo.org ([140.211.166.183]:59626) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hOOL6-0003X1-QC for 35443@debbugs.gnu.org; Wed, 08 May 2019 11:18:37 -0400 Received: from a1i15 (host2092.kph.uni-mainz.de [134.93.134.92]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: ulm) by smtp.gentoo.org (Postfix) with ESMTPSA id 79A0E343C5A; Wed, 8 May 2019 15:18:29 +0000 (UTC) From: Ulrich Mueller To: Eric Abrahamsen Subject: Re: bug#35443: 27.0.50; Gnus (nnimap) shows "ghost" messages in summary buffer References: <87d0kugg1b.fsf@ericabrahamsen.net> Date: Wed, 08 May 2019 17:18:26 +0200 In-Reply-To: <87d0kugg1b.fsf@ericabrahamsen.net> (Eric Abrahamsen's message of "Tue, 07 May 2019 13:34:24 -0700") Message-ID: User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.0.50 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: -5.0 (-----) X-Debbugs-Envelope-To: 35443 Cc: 35443@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -6.0 (------) >>>>> On Tue, 07 May 2019, Eric Abrahamsen wrote: > Locally I'm using Dovecot 2.3.6, and it does not output those last two > lines before the OK. Hm, on the server side here dovecot was upgraded to 2.3.6 today, but the problem persists. Upon entering nnimap-transform-headers, the " *nnimap dev.gentoo.org nil *nntpd**" buffer shows the same pattern as in my original report: * 590 FETCH (UID 32471 RFC822.SIZE 694 BODYSTRUCTURE ("text" "plain" ("charset" "us-ascii") NIL NIL "7bit" 6 1 NIL NIL NIL NIL) BODY[HEADER.FIELDS (SUBJECT FROM DATE MESSAGE-ID REFERENCES IN-REPLY-TO XREF X-DIARY-TIME-ZONE X-DIARY-DOW X-DIARY-YEAR X-DIARY-MONTH X-DIARY-DOM X-DIARY-HOUR X-DIARY-MINUTE TO NEWSGROUPS CC)] {163} From: Ulrich Mueller To: ulm@gentoo.org Subject: test Date: Wed, 08 May 2019 13:59:32 +0200 Message-ID: ) * 590 FETCH (UID 32471 MODSEQ (63547) FLAGS ($HasNoAttachment)) 16742 OK Fetch completed (0.003 + 0.000 + 0.002 secs). From debbugs-submit-bounces@debbugs.gnu.org Thu May 09 13:27:17 2019 Received: (at 35443) by debbugs.gnu.org; 9 May 2019 17:27:17 +0000 Received: from localhost ([127.0.0.1]:36286 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hOmpB-0006X6-68 for submit@debbugs.gnu.org; Thu, 09 May 2019 13:27:17 -0400 Received: from ericabrahamsen.net ([52.70.2.18]:48370 helo=mail.ericabrahamsen.net) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hOmp9-0006Wo-Hq for 35443@debbugs.gnu.org; Thu, 09 May 2019 13:27:16 -0400 Received: from localhost (97-113-41-132.tukw.qwest.net [97.113.41.132]) (Authenticated sender: eric@ericabrahamsen.net) by mail.ericabrahamsen.net (Postfix) with ESMTPSA id D8B55FA621; Thu, 9 May 2019 17:27:08 +0000 (UTC) From: Eric Abrahamsen To: Ulrich Mueller Subject: Re: bug#35443: 27.0.50; Gnus (nnimap) shows "ghost" messages in summary buffer References: <87wojfjxry.fsf@ericabrahamsen.net> <87h8ajjhux.fsf@ericabrahamsen.net> <87lfztiqu3.fsf@ericabrahamsen.net> Date: Thu, 09 May 2019 10:27:07 -0700 In-Reply-To: <87lfztiqu3.fsf@ericabrahamsen.net> (Eric Abrahamsen's message of "Sun, 28 Apr 2019 17:43:16 -0700") Message-ID: <871s1735ec.fsf@ericabrahamsen.net> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.0.50 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 35443 Cc: 35443@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Eric Abrahamsen writes: --8<---------------cut here---------------start------------->8--- > Ulrich Mueller writes: > >>>>>>> On Sat, 27 Apr 2019, Eric Abrahamsen wrote: >> >>> Huh. Can you show us the value of gnus-newsgroup-data and >>> gnus-newsgroup-headers in this group? >> >> See below, for the current group, whose summary buffer shows your >> article and a dummy article. >> >>> Does the dummy article always show up? >> >> It always shows up when new articles have been fetched. It doesn't >> show up when entering a group that doesn't have new mail (i.e., if a >> group previously had a dummy article, it will be gone when reentering >> that group). >> >>> Only in this group? >> >> All groups for that particular IMAP server. >> >>> Is this a newly-added IMAP server, and did it display this problem >>> right from the beginning? >> >> AFAICS, the problem appeared after dovecot on the server side was >> upgraded from version 2.2.34 to 2.3.5.1. But I see it only with Gnus. >> I don't have any issues with Mozilla Thunderbird (on GNU/Linux) nor with >> K-9 Mail (on Android/Linux). > > Thanks, this should be enough to make some progress in tracking the > problem down. Give me a couple of days... Okay, here's what I was told on irc: --8<---------------cut here---------------start------------->8--- looks like it's the new feature that adds $HasAttachment or $HasNoAttachment flags to mails. they're actually supposed to be added only during mail delivery, and there's a setting needed to enable them: mail_attachment_detection_options = add-flags-on-save but .. there's an "unintentional feature" :) that they also get added during some FETCH commands if they're not already there but even without this, imap protocol allows sending FETCH FLAGS updates (or any updates really) as a response to any command. and with concurrent imap access dovecot does this. for example if another client adds a \Seen flag, that same thing could happen either way, a client should really be able to process these "unexpeceted" FETCH updates, that's what the standard is about --8<---------------cut here---------------end--------------->8--- Ultimately, the "real" fix would be to teach Gnus to handle all possible responses from IMAP servers, probably using a proper parser. But if you have access to the dovecot config in your case, you might try removing the mail_attachment_detection_options setting above, if it's set. If it's not set, then you might be getting the "unintentional feature". I suppose in the interim I could mess with `nnimap-transform-headers' and try to add some bookkeeping so that multiple FETCH responses for the same article UID get merged together. On the other hand, this feature only seems to relate to has/hasnoattachment flags, which Gnus doesn't handle anyway -- we could safely drop those lines altogether. (Though it sure would be nice to handle hasattachment flags, that's something that users have requested in the past...) Well, that's where we're at. Eric From debbugs-submit-bounces@debbugs.gnu.org Thu May 09 15:03:51 2019 Received: (at 35443) by debbugs.gnu.org; 9 May 2019 19:03:51 +0000 Received: from localhost ([127.0.0.1]:36443 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hOoKc-0004k3-Q8 for submit@debbugs.gnu.org; Thu, 09 May 2019 15:03:51 -0400 Received: from smtp.gentoo.org ([140.211.166.183]:43310) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hOoKa-0004jn-MB for 35443@debbugs.gnu.org; Thu, 09 May 2019 15:03:49 -0400 Received: from a1i15 (host2092.kph.uni-mainz.de [134.93.134.92]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: ulm) by smtp.gentoo.org (Postfix) with ESMTPSA id 8ACCB343E7C; Thu, 9 May 2019 19:03:41 +0000 (UTC) From: Ulrich Mueller To: Eric Abrahamsen Subject: Re: bug#35443: 27.0.50; Gnus (nnimap) shows "ghost" messages in summary buffer References: <87wojfjxry.fsf@ericabrahamsen.net> <87h8ajjhux.fsf@ericabrahamsen.net> <87lfztiqu3.fsf@ericabrahamsen.net> <871s1735ec.fsf@ericabrahamsen.net> Date: Thu, 09 May 2019 21:03:38 +0200 In-Reply-To: <871s1735ec.fsf@ericabrahamsen.net> (Eric Abrahamsen's message of "Thu, 09 May 2019 10:27:07 -0700") Message-ID: User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.0.50 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: -5.0 (-----) X-Debbugs-Envelope-To: 35443 Cc: 35443@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -6.0 (------) >>>>> On Thu, 09 May 2019, Eric Abrahamsen wrote: > Okay, here's what I was told on irc: > looks like it's the new feature that adds $HasAttachment or > $HasNoAttachment flags to mails. they're actually supposed to be added > only during mail delivery, and there's a setting needed to enable them: > mail_attachment_detection_options = add-flags-on-save > but .. there's an "unintentional feature" :) that they also get added > during some FETCH commands if they're not already there > but even without this, imap protocol allows sending FETCH FLAGS updates > (or any updates really) as a response to any command. and with > concurrent imap access dovecot does this. > for example if another client adds a \Seen flag, that same thing could > happen > either way, a client should really be able to process these > "unexpeceted" FETCH updates, that's what the standard is about > --8<---------------cut here---------------end--------------->8--- > Ultimately, the "real" fix would be to teach Gnus to handle all possible > responses from IMAP servers, probably using a proper parser. > But if you have access to the dovecot config in your case, you might try > removing the mail_attachment_detection_options setting above, if it's > set. If it's not set, then you might be getting the "unintentional > feature". Thank you very much for your help. Indeed the add-flags-on-save option was enabled in the config file. Gentoo's infrastructure team has agreed to disable it for now, which made the problem go away. > I suppose in the interim I could mess with `nnimap-transform-headers' > and try to add some bookkeeping so that multiple FETCH responses for > the same article UID get merged together. On the other hand, this > feature only seems to relate to has/hasnoattachment flags, which Gnus > doesn't handle anyway -- we could safely drop those lines altogether. > (Though it sure would be nice to handle hasattachment flags, that's > something that users have requested in the past...) Maybe drop the extra FETCH responses for now, and add proper bookkeeping later when implementing the requested feature? From debbugs-submit-bounces@debbugs.gnu.org Thu May 09 16:15:10 2019 Received: (at 35443) by debbugs.gnu.org; 9 May 2019 20:15:10 +0000 Received: from localhost ([127.0.0.1]:36545 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hOpRe-00009B-21 for submit@debbugs.gnu.org; Thu, 09 May 2019 16:15:10 -0400 Received: from ericabrahamsen.net ([52.70.2.18]:51636 helo=mail.ericabrahamsen.net) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hOpRc-00008u-Vk for 35443@debbugs.gnu.org; Thu, 09 May 2019 16:15:09 -0400 Received: from localhost (50-251-205-17-static.hfc.comcastbusiness.net [50.251.205.17]) (Authenticated sender: eric@ericabrahamsen.net) by mail.ericabrahamsen.net (Postfix) with ESMTPSA id 9FC7BFA621; Thu, 9 May 2019 20:15:02 +0000 (UTC) From: Eric Abrahamsen To: Ulrich Mueller Subject: Re: bug#35443: 27.0.50; Gnus (nnimap) shows "ghost" messages in summary buffer References: <87wojfjxry.fsf@ericabrahamsen.net> <87h8ajjhux.fsf@ericabrahamsen.net> <87lfztiqu3.fsf@ericabrahamsen.net> <871s1735ec.fsf@ericabrahamsen.net> Date: Thu, 09 May 2019 13:15:00 -0700 In-Reply-To: (Ulrich Mueller's message of "Thu, 09 May 2019 21:03:38 +0200") Message-ID: <87ftpn75bv.fsf@ericabrahamsen.net> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.0.50 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 35443 Cc: 35443@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Ulrich Mueller writes: >>>>>> On Thu, 09 May 2019, Eric Abrahamsen wrote: > >> Okay, here's what I was told on irc: > >> looks like it's the new feature that adds $HasAttachment or >> $HasNoAttachment flags to mails. they're actually supposed to be added >> only during mail delivery, and there's a setting needed to enable them: >> mail_attachment_detection_options = add-flags-on-save >> but .. there's an "unintentional feature" :) that they also get added >> during some FETCH commands if they're not already there >> but even without this, imap protocol allows sending FETCH FLAGS updates >> (or any updates really) as a response to any command. and with >> concurrent imap access dovecot does this. >> for example if another client adds a \Seen flag, that same thing could >> happen >> either way, a client should really be able to process these >> "unexpeceted" FETCH updates, that's what the standard is about >> --8<---------------cut here---------------end--------------->8--- > >> Ultimately, the "real" fix would be to teach Gnus to handle all possible >> responses from IMAP servers, probably using a proper parser. > >> But if you have access to the dovecot config in your case, you might try >> removing the mail_attachment_detection_options setting above, if it's >> set. If it's not set, then you might be getting the "unintentional >> feature". > > Thank you very much for your help. Indeed the add-flags-on-save option > was enabled in the config file. Gentoo's infrastructure team has agreed > to disable it for now, which made the problem go away. Well that was lucky. >> I suppose in the interim I could mess with `nnimap-transform-headers' >> and try to add some bookkeeping so that multiple FETCH responses for >> the same article UID get merged together. On the other hand, this >> feature only seems to relate to has/hasnoattachment flags, which Gnus >> doesn't handle anyway -- we could safely drop those lines altogether. > >> (Though it sure would be nice to handle hasattachment flags, that's >> something that users have requested in the past...) > > Maybe drop the extra FETCH responses for now, and add proper bookkeeping > later when implementing the requested feature? Okay, I'll work something up in the next week that goes halfway with this. Glad it's partially fixed. Eric From debbugs-submit-bounces@debbugs.gnu.org Sat Jun 22 08:26:11 2019 Received: (at 35443) by debbugs.gnu.org; 22 Jun 2019 12:26:11 +0000 Received: from localhost ([127.0.0.1]:51094 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hef5u-0007A8-Ps for submit@debbugs.gnu.org; Sat, 22 Jun 2019 08:26:10 -0400 Received: from quimby.gnus.org ([80.91.231.51]:39152) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hef5r-00079z-VY for 35443@debbugs.gnu.org; Sat, 22 Jun 2019 08:26:09 -0400 Received: from cm-84.212.202.86.getinternet.no ([84.212.202.86] helo=stories) by quimby.gnus.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.89) (envelope-from ) id 1hef5o-0005m0-LS; Sat, 22 Jun 2019 14:26:06 +0200 From: Lars Ingebrigtsen To: Eric Abrahamsen Subject: Re: bug#35443: 27.0.50; Gnus (nnimap) shows "ghost" messages in summary buffer References: <87wojfjxry.fsf@ericabrahamsen.net> <87h8ajjhux.fsf@ericabrahamsen.net> <87lfztiqu3.fsf@ericabrahamsen.net> <871s1735ec.fsf@ericabrahamsen.net> <87ftpn75bv.fsf@ericabrahamsen.net> Date: Sat, 22 Jun 2019 14:26:04 +0200 In-Reply-To: <87ftpn75bv.fsf@ericabrahamsen.net> (Eric Abrahamsen's message of "Thu, 09 May 2019 13:15:00 -0700") Message-ID: User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.0.50 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Report: Spam detection software, running on the system "quimby.gnus.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see @@CONTACT_ADDRESS@@ for details. Content preview: Eric Abrahamsen writes: >> Maybe drop the extra FETCH responses for now, and add proper bookkeeping >> later when implementing the requested feature? > > Okay, I'll work something up in the next week that goes halfway with > [...] Content analysis details: (-2.9 points, 5.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -1.0 ALL_TRUSTED Passed through trusted hosts only via SMTP -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1% [score: 0.0000] X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 35443 Cc: 35443@debbugs.gnu.org, Ulrich Mueller X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Eric Abrahamsen writes: >> Maybe drop the extra FETCH responses for now, and add proper bookkeeping >> later when implementing the requested feature? > > Okay, I'll work something up in the next week that goes halfway with > this. Did you get any further in fixing this nnimap parsing bug? -- (domestic pets only, the antidote for overdose, milk.) bloggy blog: http://lars.ingebrigtsen.no From debbugs-submit-bounces@debbugs.gnu.org Sat Jun 22 12:27:28 2019 Received: (at 35443) by debbugs.gnu.org; 22 Jun 2019 16:27:28 +0000 Received: from localhost ([127.0.0.1]:52457 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1heirQ-0005m9-Ce for submit@debbugs.gnu.org; Sat, 22 Jun 2019 12:27:28 -0400 Received: from ericabrahamsen.net ([52.70.2.18]:38866 helo=mail.ericabrahamsen.net) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1heirO-0005lr-6K for 35443@debbugs.gnu.org; Sat, 22 Jun 2019 12:27:27 -0400 Received: from localhost (unknown [172.92.212.120]) (Authenticated sender: eric@ericabrahamsen.net) by mail.ericabrahamsen.net (Postfix) with ESMTPSA id 8ED00FA648; Sat, 22 Jun 2019 16:27:16 +0000 (UTC) From: Eric Abrahamsen To: Lars Ingebrigtsen Subject: Re: bug#35443: 27.0.50; Gnus (nnimap) shows "ghost" messages in summary buffer References: <87wojfjxry.fsf@ericabrahamsen.net> <87h8ajjhux.fsf@ericabrahamsen.net> <87lfztiqu3.fsf@ericabrahamsen.net> <871s1735ec.fsf@ericabrahamsen.net> <87ftpn75bv.fsf@ericabrahamsen.net> Date: Sat, 22 Jun 2019 09:27:10 -0700 In-Reply-To: (Lars Ingebrigtsen's message of "Sat, 22 Jun 2019 14:26:04 +0200") Message-ID: <87r27ly49t.fsf@ericabrahamsen.net> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.0.50 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 35443 Cc: 35443@debbugs.gnu.org, Ulrich Mueller X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Lars Ingebrigtsen writes: > Eric Abrahamsen writes: > >>> Maybe drop the extra FETCH responses for now, and add proper bookkeeping >>> later when implementing the requested feature? >> >> Okay, I'll work something up in the next week that goes halfway with >> this. > > Did you get any further in fixing this nnimap parsing bug? No I didn't! I looked at it for a while, had an internal conflict about just dropping the lines vs actually doing something with them, imagined all the work that would go into a more fully-featured parser, then got distracted and forgot about it. I still hope that in some distant future we could have a real parser consuming these buffers. So far as I know the only in-emacs options are in cedet -- wisent and the other one -- but I've never been able to make them work. And now it sounds like cedet might not even stay in-tree? Anyway, I'll make a real todo for dropping the lines, though it seems a shame... Eric From debbugs-submit-bounces@debbugs.gnu.org Sat Jun 22 17:36:32 2019 Received: (at 35443) by debbugs.gnu.org; 22 Jun 2019 21:36:32 +0000 Received: from localhost ([127.0.0.1]:52613 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hengV-0005oG-N8 for submit@debbugs.gnu.org; Sat, 22 Jun 2019 17:36:32 -0400 Received: from ericabrahamsen.net ([52.70.2.18]:50910 helo=mail.ericabrahamsen.net) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hengR-0005o1-K0 for 35443@debbugs.gnu.org; Sat, 22 Jun 2019 17:36:29 -0400 Received: from localhost (unknown [172.92.212.120]) (Authenticated sender: eric@ericabrahamsen.net) by mail.ericabrahamsen.net (Postfix) with ESMTPSA id 9A456FA05D; Sat, 22 Jun 2019 21:36:19 +0000 (UTC) From: Eric Abrahamsen To: Lars Ingebrigtsen Subject: Re: bug#35443: 27.0.50; Gnus (nnimap) shows "ghost" messages in summary buffer References: <87wojfjxry.fsf@ericabrahamsen.net> <87h8ajjhux.fsf@ericabrahamsen.net> <87lfztiqu3.fsf@ericabrahamsen.net> <871s1735ec.fsf@ericabrahamsen.net> <87ftpn75bv.fsf@ericabrahamsen.net> Date: Sat, 22 Jun 2019 14:36:13 -0700 Message-ID: <87pnn5wbea.fsf@ericabrahamsen.net> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.0.50 (gnu/linux) MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 35443 Cc: 35443@debbugs.gnu.org, Ulrich Mueller X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) --=-=-= Content-Type: text/plain Lars Ingebrigtsen writes: > Eric Abrahamsen writes: > >>> Maybe drop the extra FETCH responses for now, and add proper bookkeeping >>> later when implementing the requested feature? >> >> Okay, I'll work something up in the next week that goes halfway with >> this. > > Did you get any further in fixing this nnimap parsing bug? Here's a whack at it. I tried to make sure that it would handle unwanted FETCH responses whether they came before or after (or in the middle of) the wanted FETCH responses, but I'm not in love with checking the header regexp this way. Because this IMAP server feature is very closely focused on adding a flag in case of attachment (and because Gnus never explicitly requests this flag, though I'd sure like to in the future), another more targeted approach would be to simply delete any lines containing $Has\(No\)?Attachment, assuming that these FETCH responses will only take up one line. I've configured my local Dovecot with the offending setting, and will test it out for a bit. Eric --=-=-= Content-Type: text/x-patch Content-Disposition: attachment; filename=imap-header-transform-fix.diff diff --git a/lisp/gnus/nnimap.el b/lisp/gnus/nnimap.el index 9e52abc1ca..7478fcb3e5 100644 --- a/lisp/gnus/nnimap.el +++ b/lisp/gnus/nnimap.el @@ -232,7 +232,7 @@ nnimap-retrieve-headers (defun nnimap-transform-headers () (goto-char (point-min)) - (let (article lines size string labels) + (let (seen-articles article lines size string labels) (cl-block nil (while (not (eobp)) (while (not (looking-at "\\* [0-9]+ FETCH")) @@ -261,45 +261,56 @@ nnimap-transform-headers (and (re-search-forward "UID \\([0-9]+\\)" (line-end-position) t) (match-string 1))) - (setq lines nil) - (beginning-of-line) - (setq size - (and (re-search-forward "RFC822.SIZE \\([0-9]+\\)" - (line-end-position) - t) - (match-string 1))) - (beginning-of-line) - (when (search-forward "X-GM-LABELS" (line-end-position) t) - (setq labels (ignore-errors (read (current-buffer))))) - (beginning-of-line) - (when (search-forward "BODYSTRUCTURE" (line-end-position) t) - (let ((structure (ignore-errors - (read (current-buffer))))) - (while (and (consp structure) - (not (atom (car structure)))) - (setq structure (car structure))) - (setq lines (if (and - (stringp (car structure)) - (equal (upcase (nth 0 structure)) "MESSAGE") - (equal (upcase (nth 1 structure)) "RFC822")) - (nth 9 structure) - (nth 7 structure))))) - (delete-region (line-beginning-position) (line-end-position)) - (insert (format "211 %s Article retrieved." article)) - (forward-line 1) - (when size - (insert (format "Chars: %s\n" size))) - (when lines - (insert (format "Lines: %s\n" lines))) - (when labels - (insert (format "X-GM-LABELS: %s\n" labels))) - ;; Most servers have a blank line after the headers, but - ;; Davmail doesn't. - (unless (re-search-forward "^\r$\\|^)\r?$" nil t) - (goto-char (point-max))) - (delete-region (line-beginning-position) (line-end-position)) - (insert ".") - (forward-line 1))))) + ;; If we've already got headers for this article, or this + ;; FETCH line doesn't provide headers for the article, skip + ;; it. See bug#35433. + (if (or (member article seen-articles) + (save-excursion + (forward-line) + (null (looking-at-p + "\\(From\\|To\\|Subject\\|Date\\|Message-ID\\)")))) + (delete-region (line-beginning-position) + (1+ (line-end-position))) + (setq lines nil) + (beginning-of-line) + (setq size + (and (re-search-forward "RFC822.SIZE \\([0-9]+\\)" + (line-end-position) + t) + (match-string 1))) + (beginning-of-line) + (when (search-forward "X-GM-LABELS" (line-end-position) t) + (setq labels (ignore-errors (read (current-buffer))))) + (beginning-of-line) + (when (search-forward "BODYSTRUCTURE" (line-end-position) t) + (let ((structure (ignore-errors + (read (current-buffer))))) + (while (and (consp structure) + (not (atom (car structure)))) + (setq structure (car structure))) + (setq lines (if (and + (stringp (car structure)) + (equal (upcase (nth 0 structure)) "MESSAGE") + (equal (upcase (nth 1 structure)) "RFC822")) + (nth 9 structure) + (nth 7 structure))))) + (delete-region (line-beginning-position) (line-end-position)) + (insert (format "211 %s Article retrieved." article)) + (forward-line 1) + (when size + (insert (format "Chars: %s\n" size))) + (when lines + (insert (format "Lines: %s\n" lines))) + (when labels + (insert (format "X-GM-LABELS: %s\n" labels))) + ;; Most servers have a blank line after the headers, but + ;; Davmail doesn't. + (unless (re-search-forward "^\r$\\|^)\r?$" nil t) + (goto-char (point-max))) + (delete-region (line-beginning-position) (line-end-position)) + (insert ".") + (forward-line 1) + (push article seen-articles)))))) (defun nnimap-unfold-quoted-lines () ;; Unfold quoted {number} strings. --=-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Sun Jun 23 08:13:35 2019 Received: (at 35443) by debbugs.gnu.org; 23 Jun 2019 12:13:35 +0000 Received: from localhost ([127.0.0.1]:52964 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hf1NH-0006Ku-4Q for submit@debbugs.gnu.org; Sun, 23 Jun 2019 08:13:35 -0400 Received: from quimby.gnus.org ([80.91.231.51]:39940) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hf1NF-0006Km-BZ for 35443@debbugs.gnu.org; Sun, 23 Jun 2019 08:13:34 -0400 Received: from cm-84.212.202.86.getinternet.no ([84.212.202.86] helo=stories) by quimby.gnus.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.89) (envelope-from ) id 1hf1NB-0005sS-Ut; Sun, 23 Jun 2019 14:13:32 +0200 From: Lars Ingebrigtsen To: Eric Abrahamsen Subject: Re: bug#35443: 27.0.50; Gnus (nnimap) shows "ghost" messages in summary buffer References: <87wojfjxry.fsf@ericabrahamsen.net> <87h8ajjhux.fsf@ericabrahamsen.net> <87lfztiqu3.fsf@ericabrahamsen.net> <871s1735ec.fsf@ericabrahamsen.net> <87ftpn75bv.fsf@ericabrahamsen.net> <87r27ly49t.fsf@ericabrahamsen.net> Face: iVBORw0KGgoAAAANSUhEUgAAADAAAAAwBAMAAAClLOS0AAAAGFBMVEX89eny2brdwaJqQCST ZDomFQq3hVLNp3oYgMsHAAACVUlEQVQ4jXXUsXKjMBAGYOxL4taiMK3RZEyfFXZ7k5XxA2AlbcYS UssBg17/Vgic5G6iGRfWp38FiyBJfhrrkbOMc8a5LwrvvfMj/fg+2RhPUMgwMVoaKCVWvk3WuiBo atELTXMAoAEuBBujpeRlqRowoAVoLcwA4iPZ4AQD9A2I3sBZY4+DoITWIPhQHfsejz1SKexBHttk tcvC5vY0DqMsKn8zl1NVyQ+ClO14YS7hcnmXciNeBODvGbypOM8ya8fiMEpjqnaCbLR47YriIAX2 A0pEMZVKdtbWr/KWpTlnLOX84GxMHKx9E/I1szxn22TFaOwnoFYAypc0TSmxJUhnMEeLDQbIWbb/ TGQEZyBgAdrtHXYEDeKJKuUs3X7CQVe2OeO7913OGD2jGbKxrKwu8eoWWC2AttJlWV9mWN0T2shG lVfn/JdS612WSWtKVcpxglUeNt8mmzFj1jZKXe1wHGly3a4meLIuH8xZKVWPLkCyT2awzmuaV5X3 EeLmT3Y6MwThSP0h2MY9HqSsSAzWATqWb5N4578ASCSZC0Bnkt0BhJQxZ72PUESgIxPVTCu69Lls ZwAoVR2yMBUA2CeP4T+or4Nm3pdSpfo+bgTif6gdvR9PAj9L1SKsOHJOQDeIYpqUjprzTN3kC8hK XRtqVXfwvgD1mt9BqrcAjt5Bjqd8SWAArRwtJyjycEjnBE4JS4nYq3sCQyI0qvsGcoFuBv4vxETc o00eYC4VOkufADddFA/dxbkhNcjwvLoF5jZdpYjPyroIj2GWMnH5Xdofv0p/AeFwBBYh7n1LAAAA AElFTkSuQmCC Date: Sun, 23 Jun 2019 14:13:29 +0200 In-Reply-To: <87r27ly49t.fsf@ericabrahamsen.net> (Eric Abrahamsen's message of "Sat, 22 Jun 2019 09:27:10 -0700") Message-ID: User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.0.50 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Report: Spam detection software, running on the system "quimby.gnus.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see @@CONTACT_ADDRESS@@ for details. Content preview: Eric Abrahamsen writes: > No I didn't! I looked at it for a while, had an internal conflict about > just dropping the lines vs actually doing something with them, imagined > all the work that would go into a more fully-featu [...] Content analysis details: (-2.9 points, 5.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -1.0 ALL_TRUSTED Passed through trusted hosts only via SMTP -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1% [score: 0.0000] X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 35443 Cc: 35443@debbugs.gnu.org, Ulrich Mueller X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Eric Abrahamsen writes: > No I didn't! I looked at it for a while, had an internal conflict about > just dropping the lines vs actually doing something with them, imagined > all the work that would go into a more fully-featured parser, then got > distracted and forgot about it. OK, I'll take a look at it... > I still hope that in some distant future we could have a real parser > consuming these buffers. So far as I know the only in-emacs options are > in cedet -- wisent and the other one -- but I've never been able to make > them work. And now it sounds like cedet might not even stay in-tree? I had to add a parsing feature to wisent, and that was kinda more painful than you'd expect. :-/ But I haven't heard anything about dumping cedet from the tree. Is that the plan? -- (domestic pets only, the antidote for overdose, milk.) bloggy blog: http://lars.ingebrigtsen.no From debbugs-submit-bounces@debbugs.gnu.org Sun Jun 23 08:23:59 2019 Received: (at 35443) by debbugs.gnu.org; 23 Jun 2019 12:23:59 +0000 Received: from localhost ([127.0.0.1]:52995 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hf1XL-0000Ij-3H for submit@debbugs.gnu.org; Sun, 23 Jun 2019 08:23:59 -0400 Received: from quimby.gnus.org ([80.91.231.51]:40066) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hf1XG-0000IX-LU for 35443@debbugs.gnu.org; Sun, 23 Jun 2019 08:23:57 -0400 Received: from cm-84.212.202.86.getinternet.no ([84.212.202.86] helo=stories) by quimby.gnus.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.89) (envelope-from ) id 1hf1XC-0005yc-Mc; Sun, 23 Jun 2019 14:23:53 +0200 From: Lars Ingebrigtsen To: Eric Abrahamsen Subject: Re: bug#35443: 27.0.50; Gnus (nnimap) shows "ghost" messages in summary buffer References: <87wojfjxry.fsf@ericabrahamsen.net> <87h8ajjhux.fsf@ericabrahamsen.net> <87lfztiqu3.fsf@ericabrahamsen.net> <871s1735ec.fsf@ericabrahamsen.net> <87ftpn75bv.fsf@ericabrahamsen.net> <87pnn5wbea.fsf@ericabrahamsen.net> Face: iVBORw0KGgoAAAANSUhEUgAAADAAAAAwBAMAAAClLOS0AAAAGFBMVEX89eny2brdwaJqQCST ZDomFQq3hVLNp3oYgMsHAAACVUlEQVQ4jXXUsXKjMBAGYOxL4taiMK3RZEyfFXZ7k5XxA2AlbcYS UssBg17/Vgic5G6iGRfWp38FiyBJfhrrkbOMc8a5LwrvvfMj/fg+2RhPUMgwMVoaKCVWvk3WuiBo atELTXMAoAEuBBujpeRlqRowoAVoLcwA4iPZ4AQD9A2I3sBZY4+DoITWIPhQHfsejz1SKexBHttk tcvC5vY0DqMsKn8zl1NVyQ+ClO14YS7hcnmXciNeBODvGbypOM8ya8fiMEpjqnaCbLR47YriIAX2 A0pEMZVKdtbWr/KWpTlnLOX84GxMHKx9E/I1szxn22TFaOwnoFYAypc0TSmxJUhnMEeLDQbIWbb/ TGQEZyBgAdrtHXYEDeKJKuUs3X7CQVe2OeO7913OGD2jGbKxrKwu8eoWWC2AttJlWV9mWN0T2shG lVfn/JdS612WSWtKVcpxglUeNt8mmzFj1jZKXe1wHGly3a4meLIuH8xZKVWPLkCyT2awzmuaV5X3 EeLmT3Y6MwThSP0h2MY9HqSsSAzWATqWb5N4578ASCSZC0Bnkt0BhJQxZ72PUESgIxPVTCu69Lls ZwAoVR2yMBUA2CeP4T+or4Nm3pdSpfo+bgTif6gdvR9PAj9L1SKsOHJOQDeIYpqUjprzTN3kC8hK XRtqVXfwvgD1mt9BqrcAjt5Bjqd8SWAArRwtJyjycEjnBE4JS4nYq3sCQyI0qvsGcoFuBv4vxETc o00eYC4VOkufADddFA/dxbkhNcjwvLoF5jZdpYjPyroIj2GWMnH5Xdofv0p/AeFwBBYh7n1LAAAA AElFTkSuQmCC Date: Sun, 23 Jun 2019 14:23:50 +0200 In-Reply-To: <87pnn5wbea.fsf@ericabrahamsen.net> (Eric Abrahamsen's message of "Sat, 22 Jun 2019 14:36:13 -0700") Message-ID: User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.0.50 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Report: Spam detection software, running on the system "quimby.gnus.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see @@CONTACT_ADDRESS@@ for details. Content preview: Eric Abrahamsen writes: >> Did you get any further in fixing this nnimap parsing bug? > > Here's a whack at it. I tried to make sure that it would handle unwanted > FETCH responses whether they came before or after (or in th [...] Content analysis details: (-2.9 points, 5.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -1.0 ALL_TRUSTED Passed through trusted hosts only via SMTP -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1% [score: 0.0000] X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 35443 Cc: 35443@debbugs.gnu.org, Ulrich Mueller X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Eric Abrahamsen writes: >> Did you get any further in fixing this nnimap parsing bug? > > Here's a whack at it. I tried to make sure that it would handle unwanted > FETCH responses whether they came before or after (or in the middle of) > the wanted FETCH responses, but I'm not in love with checking the header > regexp this way. Well, I think it's OK... > Because this IMAP server feature is very closely focused on adding a > flag in case of attachment (and because Gnus never explicitly requests > this flag, though I'd sure like to in the future), another more targeted > approach would be to simply delete any lines containing > $Has\(No\)?Attachment, assuming that these FETCH responses will only > take up one line. That sounds a bit brittle -- I'm sure there'll be other extensions like this in the future to the IMAP protocol. > I've configured my local Dovecot with the offending setting, and will > test it out for a bit. Great! -- (domestic pets only, the antidote for overdose, milk.) bloggy blog: http://lars.ingebrigtsen.no From debbugs-submit-bounces@debbugs.gnu.org Sun Jun 23 12:56:06 2019 Received: (at 35443) by debbugs.gnu.org; 23 Jun 2019 16:56:06 +0000 Received: from localhost ([127.0.0.1]:54370 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hf5mf-0003Ly-P1 for submit@debbugs.gnu.org; Sun, 23 Jun 2019 12:56:06 -0400 Received: from ericabrahamsen.net ([52.70.2.18]:48538 helo=mail.ericabrahamsen.net) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hf5me-0003LK-NI for 35443@debbugs.gnu.org; Sun, 23 Jun 2019 12:56:05 -0400 Received: from localhost (unknown [172.92.212.120]) (Authenticated sender: eric@ericabrahamsen.net) by mail.ericabrahamsen.net (Postfix) with ESMTPSA id 7811AFA648; Sun, 23 Jun 2019 16:55:57 +0000 (UTC) From: Eric Abrahamsen To: Lars Ingebrigtsen Subject: Re: bug#35443: 27.0.50; Gnus (nnimap) shows "ghost" messages in summary buffer References: <87wojfjxry.fsf@ericabrahamsen.net> <87h8ajjhux.fsf@ericabrahamsen.net> <87lfztiqu3.fsf@ericabrahamsen.net> <871s1735ec.fsf@ericabrahamsen.net> <87ftpn75bv.fsf@ericabrahamsen.net> <87r27ly49t.fsf@ericabrahamsen.net> Date: Sun, 23 Jun 2019 09:55:51 -0700 In-Reply-To: (Lars Ingebrigtsen's message of "Sun, 23 Jun 2019 14:13:29 +0200") Message-ID: <87ef3kutpk.fsf@ericabrahamsen.net> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.0.50 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 35443 Cc: 35443@debbugs.gnu.org, Ulrich Mueller X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) On 06/23/19 14:13 PM, Lars Ingebrigtsen wrote: > Eric Abrahamsen writes: > >> No I didn't! I looked at it for a while, had an internal conflict about >> just dropping the lines vs actually doing something with them, imagined >> all the work that would go into a more fully-featured parser, then got >> distracted and forgot about it. > > OK, I'll take a look at it... > >> I still hope that in some distant future we could have a real parser >> consuming these buffers. So far as I know the only in-emacs options are >> in cedet -- wisent and the other one -- but I've never been able to make >> them work. And now it sounds like cedet might not even stay in-tree? > > I had to add a parsing feature to wisent, and that was kinda more > painful than you'd expect. :-/ Oh, I thought that was most of the point of wisent to begin with. No wonder I couldn't get it to do anything. In theory, do you have any recommendations in the parsing direction. > But I haven't heard anything about dumping cedet from the tree. Is that > the plan? No, I was skimming your threads about cleaning up the build process and got the impression that it was unmaintained. I probably misinterpreted. On 06/23/19 14:23 PM, Lars Ingebrigtsen wrote: > Eric Abrahamsen writes: > >>> Did you get any further in fixing this nnimap parsing bug? >> >> Here's a whack at it. I tried to make sure that it would handle unwanted >> FETCH responses whether they came before or after (or in the middle of) >> the wanted FETCH responses, but I'm not in love with checking the header >> regexp this way. > > Well, I think it's OK... Cool. >> Because this IMAP server feature is very closely focused on adding a >> flag in case of attachment (and because Gnus never explicitly requests >> this flag, though I'd sure like to in the future), another more targeted >> approach would be to simply delete any lines containing >> $Has\(No\)?Attachment, assuming that these FETCH responses will only >> take up one line. > > That sounds a bit brittle -- I'm sure there'll be other extensions like > this in the future to the IMAP protocol. Sure, I'll stick with this. From debbugs-submit-bounces@debbugs.gnu.org Sun Jun 23 12:58:55 2019 Received: (at 35443) by debbugs.gnu.org; 23 Jun 2019 16:58:56 +0000 Received: from localhost ([127.0.0.1]:54378 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hf5pP-0003TC-LY for submit@debbugs.gnu.org; Sun, 23 Jun 2019 12:58:55 -0400 Received: from quimby.gnus.org ([80.91.231.51]:43648) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hf5pO-0003T3-Bv for 35443@debbugs.gnu.org; Sun, 23 Jun 2019 12:58:54 -0400 Received: from cm-84.212.202.86.getinternet.no ([84.212.202.86] helo=stories) by quimby.gnus.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.89) (envelope-from ) id 1hf5pL-000892-0g; Sun, 23 Jun 2019 18:58:53 +0200 From: Lars Ingebrigtsen To: Eric Abrahamsen Subject: Re: bug#35443: 27.0.50; Gnus (nnimap) shows "ghost" messages in summary buffer References: <87wojfjxry.fsf@ericabrahamsen.net> <87h8ajjhux.fsf@ericabrahamsen.net> <87lfztiqu3.fsf@ericabrahamsen.net> <871s1735ec.fsf@ericabrahamsen.net> <87ftpn75bv.fsf@ericabrahamsen.net> <87r27ly49t.fsf@ericabrahamsen.net> <87ef3kutpk.fsf@ericabrahamsen.net> Face: iVBORw0KGgoAAAANSUhEUgAAADAAAAAwBAMAAAClLOS0AAAAGFBMVEULCwsHBwc0MjNRTU+A fH0ZGRrSzc8QEBDiLqT7AAACS0lEQVQ4jV2UQW/cIBCFsaLKV0gPubZglGvL4Oy1iunmbhv5Gqkr crW8sfj7fQO73TTPF4uPNzMMHosv8aY9LhlqV5F30d7Wc9EqIIAHvC87sRzRRPTtA7iqbIYQKgpZ 19o1t0I0jRCqOKJQa8YjsCpk0zRyVaItDiXyegnRqOJQslS16BwtTDnvyL7R5sNh9yEITVqbuKDa MQzH93Q+prf5mJJwpONUjjEzOG3H9DoHAO2tnSwD4+npnADeuuLwzo49A/LkZ4BjBX1HeurRpnGA Y745ekPWOBjORMb/B4zWBIA20QWYVADqIhsXbmN3PnG5UwVkXHeIpcF+C+yY6jkg08eRu+5ruTfg +jg5zSBwqLEAAzBcQPd++geey+UBaEemm09bSq/V4WEY+uVOO1MAehUr8H4Ih2Vj4Eu5v8Zr8hB6 ADux4xP4nTdcC1XHSy1Xc5kDuajNxVGB1Xb0aRic1Vr7dz55BTFaEwLud4TjaSo5qiM6fxqw2yLH E06eLi3p47M/EYOo+Wqnj+APAwQzoUtzunwlADRxKB31hKgFpAq6x20guzPzQ3cFdnfnweaRUHDe jfYAXXXQYsLgljwacst9COGlgmV7zJmPgjGLegQ4VICPGWQfPZAQ7UjBHBl8UKNYmITM81FGhiXV FbSrAni4TJ4USl0djWrhuIGmejBUkkNdRxVDpxoJBCvnaHPmPGt1SdHAKtu9gKUWcNfRjxJQtnnn HLyKfeLO008pkaQCCcBhlfxq3PdSF/8TYsn6SRV8Xq1gj38BCRUOP4pOIfkAAAAASUVORK5CYII= Date: Sun, 23 Jun 2019 18:58:50 +0200 In-Reply-To: <87ef3kutpk.fsf@ericabrahamsen.net> (Eric Abrahamsen's message of "Sun, 23 Jun 2019 09:55:51 -0700") Message-ID: User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.0.50 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Report: Spam detection software, running on the system "quimby.gnus.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see @@CONTACT_ADDRESS@@ for details. Content preview: Eric Abrahamsen writes: >> I had to add a parsing feature to wisent, and that was kinda more >> painful than you'd expect. :-/ > > Oh, I thought that was most of the point of wisent to begin with. No > wonder I couldn't get [...] Content analysis details: (-2.9 points, 5.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -1.0 ALL_TRUSTED Passed through trusted hosts only via SMTP -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1% [score: 0.0000] X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 35443 Cc: 35443@debbugs.gnu.org, Ulrich Mueller X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Eric Abrahamsen writes: >> I had to add a parsing feature to wisent, and that was kinda more >> painful than you'd expect. :-/ > > Oh, I thought that was most of the point of wisent to begin with. No > wonder I couldn't get it to do anything. In theory, do you have any > recommendations in the parsing direction. Yes, that's the point of wisent, but as with any lex-like thing I've dealt with, it's... painful. :-) At least to me. >> But I haven't heard anything about dumping cedet from the tree. Is that >> the plan? > > No, I was skimming your threads about cleaning up the build process and > got the impression that it was unmaintained. I probably misinterpreted. We are all the maintainer of cedet now. :-) -- (domestic pets only, the antidote for overdose, milk.) bloggy blog: http://lars.ingebrigtsen.no From debbugs-submit-bounces@debbugs.gnu.org Sun Jun 23 13:23:17 2019 Received: (at 35443) by debbugs.gnu.org; 23 Jun 2019 17:23:17 +0000 Received: from localhost ([127.0.0.1]:54414 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hf6Cy-00072y-Mw for submit@debbugs.gnu.org; Sun, 23 Jun 2019 13:23:16 -0400 Received: from ericabrahamsen.net ([52.70.2.18]:49436 helo=mail.ericabrahamsen.net) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hf6Cw-00072I-Lz for 35443@debbugs.gnu.org; Sun, 23 Jun 2019 13:23:15 -0400 Received: from localhost (unknown [172.92.212.120]) (Authenticated sender: eric@ericabrahamsen.net) by mail.ericabrahamsen.net (Postfix) with ESMTPSA id B489AFA648; Sun, 23 Jun 2019 17:23:07 +0000 (UTC) From: Eric Abrahamsen To: Lars Ingebrigtsen Subject: Re: bug#35443: 27.0.50; Gnus (nnimap) shows "ghost" messages in summary buffer References: <87wojfjxry.fsf@ericabrahamsen.net> <87h8ajjhux.fsf@ericabrahamsen.net> <87lfztiqu3.fsf@ericabrahamsen.net> <871s1735ec.fsf@ericabrahamsen.net> <87ftpn75bv.fsf@ericabrahamsen.net> <87r27ly49t.fsf@ericabrahamsen.net> <87ef3kutpk.fsf@ericabrahamsen.net> Date: Sun, 23 Jun 2019 10:23:01 -0700 In-Reply-To: (Lars Ingebrigtsen's message of "Sun, 23 Jun 2019 18:58:50 +0200") Message-ID: <871rzkusga.fsf@ericabrahamsen.net> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.0.50 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 35443 Cc: 35443@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) On 06/23/19 18:58 PM, Lars Ingebrigtsen wrote: > Eric Abrahamsen writes: > >>> I had to add a parsing feature to wisent, and that was kinda more >>> painful than you'd expect. :-/ >> >> Oh, I thought that was most of the point of wisent to begin with. No >> wonder I couldn't get it to do anything. In theory, do you have any >> recommendations in the parsing direction. > > Yes, that's the point of wisent, but as with any lex-like thing I've > dealt with, it's... painful. :-) At least to me. Huh. There's a package in the repos called parsec that's pretty easy to use, but nnimap couldn't rely on that.... >>> But I haven't heard anything about dumping cedet from the tree. Is that >>> the plan? >> >> No, I was skimming your threads about cleaning up the build process and >> got the impression that it was unmaintained. I probably misinterpreted. > > We are all the maintainer of cedet now. :-) Ugh. From debbugs-submit-bounces@debbugs.gnu.org Thu Jul 11 13:38:49 2019 Received: (at 35443) by debbugs.gnu.org; 11 Jul 2019 17:38:50 +0000 Received: from localhost ([127.0.0.1]:38574 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hld1t-0005PH-Ks for submit@debbugs.gnu.org; Thu, 11 Jul 2019 13:38:49 -0400 Received: from ericabrahamsen.net ([52.70.2.18]:60594 helo=mail.ericabrahamsen.net) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hld1q-0005P1-LI for 35443@debbugs.gnu.org; Thu, 11 Jul 2019 13:38:47 -0400 Received: from localhost (c-73-254-86-141.hsd1.wa.comcast.net [73.254.86.141]) (Authenticated sender: eric@ericabrahamsen.net) by mail.ericabrahamsen.net (Postfix) with ESMTPSA id A0C09FA01E; Thu, 11 Jul 2019 17:38:39 +0000 (UTC) From: Eric Abrahamsen To: Lars Ingebrigtsen Subject: Re: bug#35443: 27.0.50; Gnus (nnimap) shows "ghost" messages in summary buffer References: <87wojfjxry.fsf@ericabrahamsen.net> <87h8ajjhux.fsf@ericabrahamsen.net> <87lfztiqu3.fsf@ericabrahamsen.net> <871s1735ec.fsf@ericabrahamsen.net> <87ftpn75bv.fsf@ericabrahamsen.net> <87pnn5wbea.fsf@ericabrahamsen.net> Date: Thu, 11 Jul 2019 10:38:38 -0700 In-Reply-To: (Lars Ingebrigtsen's message of "Sun, 23 Jun 2019 14:23:50 +0200") Message-ID: <87a7dkv5cx.fsf@ericabrahamsen.net> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.0.50 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 35443 Cc: 35443@debbugs.gnu.org, Ulrich Mueller X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) On 06/23/19 14:23 PM, Lars Ingebrigtsen wrote: > Eric Abrahamsen writes: > >>> Did you get any further in fixing this nnimap parsing bug? >> >> Here's a whack at it. I tried to make sure that it would handle unwanted >> FETCH responses whether they came before or after (or in the middle of) >> the wanted FETCH responses, but I'm not in love with checking the header >> regexp this way. > > Well, I think it's OK... > >> Because this IMAP server feature is very closely focused on adding a >> flag in case of attachment (and because Gnus never explicitly requests >> this flag, though I'd sure like to in the future), another more targeted >> approach would be to simply delete any lines containing >> $Has\(No\)?Attachment, assuming that these FETCH responses will only >> take up one line. > > That sounds a bit brittle -- I'm sure there'll be other extensions like > this in the future to the IMAP protocol. > >> I've configured my local Dovecot with the offending setting, and will >> test it out for a bit. > > Great! (NB This patch is very broken and no one should use it; I'm still working on it. I want a proper parser!) From debbugs-submit-bounces@debbugs.gnu.org Thu Jul 11 16:28:54 2019 Received: (at submit) by debbugs.gnu.org; 11 Jul 2019 20:28:54 +0000 Received: from localhost ([127.0.0.1]:38782 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hlfgT-000724-Va for submit@debbugs.gnu.org; Thu, 11 Jul 2019 16:28:54 -0400 Received: from lists.gnu.org ([209.51.188.17]:38356) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hlfgS-00071x-1B for submit@debbugs.gnu.org; Thu, 11 Jul 2019 16:28:52 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:33564) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hlfgQ-0007IG-Qg for bug-gnu-emacs@gnu.org; Thu, 11 Jul 2019 16:28:51 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: * X-Spam-Status: No, score=1.8 required=5.0 tests=BAYES_50,RDNS_DYNAMIC, URIBL_BLOCKED autolearn=disabled version=3.3.2 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hlfgP-0006tX-OV for bug-gnu-emacs@gnu.org; Thu, 11 Jul 2019 16:28:50 -0400 Received: from 195-159-176-226.customer.powertech.no ([195.159.176.226]:38430 helo=blaine.gmane.org) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1hlfgP-0006rp-Gn for bug-gnu-emacs@gnu.org; Thu, 11 Jul 2019 16:28:49 -0400 Received: from list by blaine.gmane.org with local (Exim 4.89) (envelope-from ) id 1hlfgL-000xvK-8F for bug-gnu-emacs@gnu.org; Thu, 11 Jul 2019 22:28:45 +0200 X-Injected-Via-Gmane: http://gmane.org/ To: bug-gnu-emacs@gnu.org From: Eric Abrahamsen Subject: Re: bug#35443: 27.0.50; Gnus (nnimap) shows "ghost" messages in summary buffer Date: Thu, 11 Jul 2019 13:28:37 -0700 Message-ID: <87k1cotix6.fsf@ericabrahamsen.net> References: <87wojfjxry.fsf@ericabrahamsen.net> <87h8ajjhux.fsf@ericabrahamsen.net> <87lfztiqu3.fsf@ericabrahamsen.net> <871s1735ec.fsf@ericabrahamsen.net> <87ftpn75bv.fsf@ericabrahamsen.net> <87pnn5wbea.fsf@ericabrahamsen.net> <87a7dkv5cx.fsf@ericabrahamsen.net> Mime-Version: 1.0 Content-Type: text/plain User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.0.50 (gnu/linux) Cancel-Lock: sha1:XstAmgY8S4xviNOHzHHGs5/HN1I= X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 195.159.176.226 X-Spam-Score: -2.0 (--) X-Debbugs-Envelope-To: submit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.0 (---) Eric Abrahamsen writes: > On 06/23/19 14:23 PM, Lars Ingebrigtsen wrote: >> Eric Abrahamsen writes: >> >>>> Did you get any further in fixing this nnimap parsing bug? >>> >>> Here's a whack at it. I tried to make sure that it would handle unwanted >>> FETCH responses whether they came before or after (or in the middle of) >>> the wanted FETCH responses, but I'm not in love with checking the header >>> regexp this way. >> >> Well, I think it's OK... >> >>> Because this IMAP server feature is very closely focused on adding a >>> flag in case of attachment (and because Gnus never explicitly requests >>> this flag, though I'd sure like to in the future), another more targeted >>> approach would be to simply delete any lines containing >>> $Has\(No\)?Attachment, assuming that these FETCH responses will only >>> take up one line. >> >> That sounds a bit brittle -- I'm sure there'll be other extensions like >> this in the future to the IMAP protocol. >> >>> I've configured my local Dovecot with the offending setting, and will >>> test it out for a bit. >> >> Great! > > (NB This patch is very broken and no one should use it; I'm still > working on it. I want a proper parser!) Turns out the dumb thing was simple: I was testing if the FETCH line was followed by mail headers by looking for: "\\(From\\|To\\|Subject\\|Date\\|Message-ID\\)" Which is obviously insufficient. I've since changed it to: "\\([A-Z][[:ascii:]-]+: \\)" Which works better, but also seems susceptible to breakage due to ignorance. A quick scan doesn't turn up any `mail-header-regexp' variables or `this-is-a-mail-header-p' functions -- does the above regexp seem reasonable? From debbugs-submit-bounces@debbugs.gnu.org Fri Jul 12 10:18:08 2019 Received: (at 35443) by debbugs.gnu.org; 12 Jul 2019 14:18:08 +0000 Received: from localhost ([127.0.0.1]:40446 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hlwNE-0007W6-03 for submit@debbugs.gnu.org; Fri, 12 Jul 2019 10:18:08 -0400 Received: from quimby.gnus.org ([80.91.231.51]:45070) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hlwNC-0007Vw-PC for 35443@debbugs.gnu.org; Fri, 12 Jul 2019 10:18:07 -0400 Received: from cm-84.212.202.86.getinternet.no ([84.212.202.86] helo=stories) by quimby.gnus.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.89) (envelope-from ) id 1hlwN8-0006mi-Kd; Fri, 12 Jul 2019 16:18:05 +0200 From: Lars Ingebrigtsen To: Eric Abrahamsen Subject: Re: bug#35443: 27.0.50; Gnus (nnimap) shows "ghost" messages in summary buffer References: <87wojfjxry.fsf@ericabrahamsen.net> <87h8ajjhux.fsf@ericabrahamsen.net> <87lfztiqu3.fsf@ericabrahamsen.net> <871s1735ec.fsf@ericabrahamsen.net> <87ftpn75bv.fsf@ericabrahamsen.net> <87pnn5wbea.fsf@ericabrahamsen.net> <87a7dkv5cx.fsf@ericabrahamsen.net> <87k1cotix6.fsf@ericabrahamsen.net> Face: iVBORw0KGgoAAAANSUhEUgAAADAAAAAwBAMAAAClLOS0AAAAD1BMVEX+/gfj2Qr//whAOgT/ /wkKSUWSAAACfklEQVQ4jU2U2xmkIAyFI04BXCwgRAogxgIU6b+mPXFm91teUH5zOwmStFvnfLY5 47xPPGkqPfNCInPdHNzbHMAWRKK2F+DDJ0vbZORZVeaQNF4gOvcc7kMktP0AGHIt1GSytj3TcjCF zx7DnLMhRmtCRfbIoWSisMbPnMMBPFKhPVKolimUSENEAIawA5xJNbegazQRDy5shHesPYubSruQ 1RQi5dX9cz3G2olFylbJzfByUCKpGc8BwMaN4Aw/9WBlhisxmIqKg1sFADlLIeFVufbPrJTMs1wO WjX1EIjMrDeAYhvyQeKmCaUXpRTFgZlGB9ViyYFMySAaggNYCIjSZe0cfuBrYZ0kAEDyqv0Lbsqs pgzVOuqT2h1wa5fXYQZ1ag9lk/0FCO5AiqEwaGsiazzFTplfsH5BACghih3TgbQBEGRRnIrJEux8 wQxcLIqntOZmvGQ7m4PBYiiEAPZ8Q6i/gARqoVkAz1EBjp8rL1C9yX3U/uDt/A9g2qQe7dLnnLdC Xvu8IAnAfrRm6waXDqC4p8TUJmMsrpKZ0wsu2jAmkBCJ4SR9oF+BD7g6hDmEvWTxL9UFwobgBwdm SUvGBEwpudrP1QlHIvrk5hfgPssLFACDxU2fo+FiydRiHVEMYMCinmiSDgArhvLMIh0+3Pt56YRF m6Vgg0n0GDKebejXVSku4YMxQOelrdsoTdHgATC/4BQeoyDRD6R8XbWvK4zOGNZwoQFme4NfHryH EaalVE7/9E33Wj1dTSn5INvp6dz9X4Hv1j+QW1vjEpe/krx7D3Sp+1Pvw1fE1xTzfnkffPBxAnzR dy3pt+Jvv35oWRbcTkIevvBzmH8AXd+kTBu5Ba4AAAAASUVORK5CYII= Date: Fri, 12 Jul 2019 16:18:02 +0200 In-Reply-To: <87k1cotix6.fsf@ericabrahamsen.net> (Eric Abrahamsen's message of "Thu, 11 Jul 2019 13:28:37 -0700") Message-ID: User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.0.50 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Report: Spam detection software, running on the system "quimby.gnus.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see @@CONTACT_ADDRESS@@ for details. Content preview: Eric Abrahamsen writes: > Which is obviously insufficient. I've since changed it to: > > "\\([A-Z][[:ascii:]-]+: \\)" > > Which works better, but also seems susceptible to breakage due to > ignorance. A quick scan doesn't tu [...] Content analysis details: (-2.9 points, 5.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -1.0 ALL_TRUSTED Passed through trusted hosts only via SMTP -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1% [score: 0.0000] X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 35443 Cc: 35443@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Eric Abrahamsen writes: > Which is obviously insufficient. I've since changed it to: > > "\\([A-Z][[:ascii:]-]+: \\)" > > Which works better, but also seems susceptible to breakage due to > ignorance. A quick scan doesn't turn up any `mail-header-regexp' > variables or `this-is-a-mail-header-p' functions -- does the above > regexp seem reasonable? Headers are case insensitive... but not all characters in the [:ascii:] set are allowed; far from it. [:alnum:] is closer, but with - and... er... stuff... One of the RFCs probably has the definition. :-) -- (domestic pets only, the antidote for overdose, milk.) bloggy blog: http://lars.ingebrigtsen.no From debbugs-submit-bounces@debbugs.gnu.org Fri Jul 12 11:02:30 2019 Received: (at 35443) by debbugs.gnu.org; 12 Jul 2019 15:02:30 +0000 Received: from localhost ([127.0.0.1]:40535 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hlx49-0002Q4-PP for submit@debbugs.gnu.org; Fri, 12 Jul 2019 11:02:29 -0400 Received: from quimby.gnus.org ([80.91.231.51]:45934) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hlx46-0002Pv-3Y for 35443@debbugs.gnu.org; Fri, 12 Jul 2019 11:02:28 -0400 Received: from cm-84.212.202.86.getinternet.no ([84.212.202.86] helo=stories) by quimby.gnus.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.89) (envelope-from ) id 1hlx42-000790-Oh; Fri, 12 Jul 2019 17:02:25 +0200 From: Lars Ingebrigtsen To: Eric Abrahamsen Subject: Re: bug#35443: 27.0.50; Gnus (nnimap) shows "ghost" messages in summary buffer References: <87wojfjxry.fsf@ericabrahamsen.net> <87h8ajjhux.fsf@ericabrahamsen.net> <87lfztiqu3.fsf@ericabrahamsen.net> <871s1735ec.fsf@ericabrahamsen.net> <87ftpn75bv.fsf@ericabrahamsen.net> <87pnn5wbea.fsf@ericabrahamsen.net> <87a7dkv5cx.fsf@ericabrahamsen.net> Face: iVBORw0KGgoAAAANSUhEUgAAADAAAAAwBAMAAAClLOS0AAAAD1BMVEX+/gfj2Qr//whAOgT/ /wkKSUWSAAACfklEQVQ4jU2U2xmkIAyFI04BXCwgRAogxgIU6b+mPXFm91teUH5zOwmStFvnfLY5 47xPPGkqPfNCInPdHNzbHMAWRKK2F+DDJ0vbZORZVeaQNF4gOvcc7kMktP0AGHIt1GSytj3TcjCF zx7DnLMhRmtCRfbIoWSisMbPnMMBPFKhPVKolimUSENEAIawA5xJNbegazQRDy5shHesPYubSruQ 1RQi5dX9cz3G2olFylbJzfByUCKpGc8BwMaN4Aw/9WBlhisxmIqKg1sFADlLIeFVufbPrJTMs1wO WjX1EIjMrDeAYhvyQeKmCaUXpRTFgZlGB9ViyYFMySAaggNYCIjSZe0cfuBrYZ0kAEDyqv0Lbsqs pgzVOuqT2h1wa5fXYQZ1ag9lk/0FCO5AiqEwaGsiazzFTplfsH5BACghih3TgbQBEGRRnIrJEux8 wQxcLIqntOZmvGQ7m4PBYiiEAPZ8Q6i/gARqoVkAz1EBjp8rL1C9yX3U/uDt/A9g2qQe7dLnnLdC Xvu8IAnAfrRm6waXDqC4p8TUJmMsrpKZ0wsu2jAmkBCJ4SR9oF+BD7g6hDmEvWTxL9UFwobgBwdm SUvGBEwpudrP1QlHIvrk5hfgPssLFACDxU2fo+FiydRiHVEMYMCinmiSDgArhvLMIh0+3Pt56YRF m6Vgg0n0GDKebejXVSku4YMxQOelrdsoTdHgATC/4BQeoyDRD6R8XbWvK4zOGNZwoQFme4NfHryH EaalVE7/9E33Wj1dTSn5INvp6dz9X4Hv1j+QW1vjEpe/krx7D3Sp+1Pvw1fE1xTzfnkffPBxAnzR dy3pt+Jvv35oWRbcTkIevvBzmH8AXd+kTBu5Ba4AAAAASUVORK5CYII= Date: Fri, 12 Jul 2019 17:02:22 +0200 In-Reply-To: <87a7dkv5cx.fsf@ericabrahamsen.net> (Eric Abrahamsen's message of "Thu, 11 Jul 2019 10:38:38 -0700") Message-ID: User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.0.50 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Report: Spam detection software, running on the system "quimby.gnus.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see @@CONTACT_ADDRESS@@ for details. Content preview: Eric Abrahamsen writes: > (NB This patch is very broken and no one should use it; I'm still > working on it. I want a proper parser!) Yes, that would be nice. But there wasn't any patch attached. :-) Content analysis details: (-2.9 points, 5.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -1.0 ALL_TRUSTED Passed through trusted hosts only via SMTP -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1% [score: 0.0000] X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 35443 Cc: 35443@debbugs.gnu.org, Ulrich Mueller X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Eric Abrahamsen writes: > (NB This patch is very broken and no one should use it; I'm still > working on it. I want a proper parser!) Yes, that would be nice. But there wasn't any patch attached. :-) -- (domestic pets only, the antidote for overdose, milk.) bloggy blog: http://lars.ingebrigtsen.no From debbugs-submit-bounces@debbugs.gnu.org Fri Jul 12 12:50:16 2019 Received: (at 35443) by debbugs.gnu.org; 12 Jul 2019 16:50:16 +0000 Received: from localhost ([127.0.0.1]:40623 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hlykR-0007Ct-N0 for submit@debbugs.gnu.org; Fri, 12 Jul 2019 12:50:16 -0400 Received: from ericabrahamsen.net ([52.70.2.18]:58878 helo=mail.ericabrahamsen.net) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hlykP-0007CY-C2 for 35443@debbugs.gnu.org; Fri, 12 Jul 2019 12:50:14 -0400 Received: from localhost (c-73-254-86-141.hsd1.wa.comcast.net [73.254.86.141]) (Authenticated sender: eric@ericabrahamsen.net) by mail.ericabrahamsen.net (Postfix) with ESMTPSA id 40F6CFA02D; Fri, 12 Jul 2019 16:50:03 +0000 (UTC) From: Eric Abrahamsen To: Lars Ingebrigtsen Subject: Re: bug#35443: 27.0.50; Gnus (nnimap) shows "ghost" messages in summary buffer References: <87wojfjxry.fsf@ericabrahamsen.net> <87h8ajjhux.fsf@ericabrahamsen.net> <87lfztiqu3.fsf@ericabrahamsen.net> <871s1735ec.fsf@ericabrahamsen.net> <87ftpn75bv.fsf@ericabrahamsen.net> <87pnn5wbea.fsf@ericabrahamsen.net> <87a7dkv5cx.fsf@ericabrahamsen.net> <87k1cotix6.fsf@ericabrahamsen.net> Date: Fri, 12 Jul 2019 09:50:01 -0700 In-Reply-To: (Lars Ingebrigtsen's message of "Fri, 12 Jul 2019 16:18:02 +0200") Message-ID: <875zo7w62u.fsf@ericabrahamsen.net> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.0.50 (gnu/linux) MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 35443 Cc: 35443@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) --=-=-= Content-Type: text/plain Lars Ingebrigtsen writes: > Eric Abrahamsen writes: > >> Which is obviously insufficient. I've since changed it to: >> >> "\\([A-Z][[:ascii:]-]+: \\)" >> >> Which works better, but also seems susceptible to breakage due to >> ignorance. A quick scan doesn't turn up any `mail-header-regexp' >> variables or `this-is-a-mail-header-p' functions -- does the above >> regexp seem reasonable? > > Headers are case insensitive... but not all characters in the [:ascii:] > set are allowed; far from it. [:alnum:] is closer, but with - and... > er... stuff... One of the RFCs probably has the definition. :-) Oh, good point. RFC2822 says chars 33 to 126, inclusive, minus the colon. So I guess it's "[!-9,;-~]+: ", unlikely as some of those characters seem. Now I'm working with the attached diff. Eric --=-=-= Content-Type: text/x-patch Content-Disposition: attachment; filename=imap-header-transform-fix.diff diff --git a/lisp/gnus/nnimap.el b/lisp/gnus/nnimap.el index 9e52abc1ca..802254fc14 100644 --- a/lisp/gnus/nnimap.el +++ b/lisp/gnus/nnimap.el @@ -232,7 +232,7 @@ nnimap-retrieve-headers (defun nnimap-transform-headers () (goto-char (point-min)) - (let (article lines size string labels) + (let (seen-articles article lines size string labels) (cl-block nil (while (not (eobp)) (while (not (looking-at "\\* [0-9]+ FETCH")) @@ -261,45 +261,56 @@ nnimap-transform-headers (and (re-search-forward "UID \\([0-9]+\\)" (line-end-position) t) (match-string 1))) - (setq lines nil) - (beginning-of-line) - (setq size - (and (re-search-forward "RFC822.SIZE \\([0-9]+\\)" - (line-end-position) - t) - (match-string 1))) - (beginning-of-line) - (when (search-forward "X-GM-LABELS" (line-end-position) t) - (setq labels (ignore-errors (read (current-buffer))))) - (beginning-of-line) - (when (search-forward "BODYSTRUCTURE" (line-end-position) t) - (let ((structure (ignore-errors - (read (current-buffer))))) - (while (and (consp structure) - (not (atom (car structure)))) - (setq structure (car structure))) - (setq lines (if (and - (stringp (car structure)) - (equal (upcase (nth 0 structure)) "MESSAGE") - (equal (upcase (nth 1 structure)) "RFC822")) - (nth 9 structure) - (nth 7 structure))))) - (delete-region (line-beginning-position) (line-end-position)) - (insert (format "211 %s Article retrieved." article)) - (forward-line 1) - (when size - (insert (format "Chars: %s\n" size))) - (when lines - (insert (format "Lines: %s\n" lines))) - (when labels - (insert (format "X-GM-LABELS: %s\n" labels))) - ;; Most servers have a blank line after the headers, but - ;; Davmail doesn't. - (unless (re-search-forward "^\r$\\|^)\r?$" nil t) - (goto-char (point-max))) - (delete-region (line-beginning-position) (line-end-position)) - (insert ".") - (forward-line 1))))) + ;; If we've already got headers for this article, or this + ;; FETCH line doesn't provide headers for the article, skip + ;; it. See bug#35433. + (if (or (member article seen-articles) + (save-excursion + (forward-line) + (null (looking-at-p + "[!-9,;-~]+: ")))) + (delete-region (line-beginning-position) + (1+ (line-end-position))) + (setq lines nil) + (beginning-of-line) + (setq size + (and (re-search-forward "RFC822.SIZE \\([0-9]+\\)" + (line-end-position) + t) + (match-string 1))) + (beginning-of-line) + (when (search-forward "X-GM-LABELS" (line-end-position) t) + (setq labels (ignore-errors (read (current-buffer))))) + (beginning-of-line) + (when (search-forward "BODYSTRUCTURE" (line-end-position) t) + (let ((structure (ignore-errors + (read (current-buffer))))) + (while (and (consp structure) + (not (atom (car structure)))) + (setq structure (car structure))) + (setq lines (if (and + (stringp (car structure)) + (equal (upcase (nth 0 structure)) "MESSAGE") + (equal (upcase (nth 1 structure)) "RFC822")) + (nth 9 structure) + (nth 7 structure))))) + (delete-region (line-beginning-position) (line-end-position)) + (insert (format "211 %s Article retrieved." article)) + (forward-line 1) + (when size + (insert (format "Chars: %s\n" size))) + (when lines + (insert (format "Lines: %s\n" lines))) + (when labels + (insert (format "X-GM-LABELS: %s\n" labels))) + ;; Most servers have a blank line after the headers, but + ;; Davmail doesn't. + (unless (re-search-forward "^\r$\\|^)\r?$" nil t) + (goto-char (point-max))) + (delete-region (line-beginning-position) (line-end-position)) + (insert ".") + (forward-line 1) + (push article seen-articles)))))) (defun nnimap-unfold-quoted-lines () ;; Unfold quoted {number} strings. --=-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Fri Jul 12 12:59:10 2019 Received: (at submit) by debbugs.gnu.org; 12 Jul 2019 16:59:10 +0000 Received: from localhost ([127.0.0.1]:40631 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hlyt4-0007Ue-1a for submit@debbugs.gnu.org; Fri, 12 Jul 2019 12:59:10 -0400 Received: from lists.gnu.org ([209.51.188.17]:34497) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hlyt2-0007UT-Cg for submit@debbugs.gnu.org; Fri, 12 Jul 2019 12:59:08 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:57405) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hlyt0-0005Uu-S5 for bug-gnu-emacs@gnu.org; Fri, 12 Jul 2019 12:59:08 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: * X-Spam-Status: No, score=1.8 required=5.0 tests=BAYES_50,RDNS_DYNAMIC, URIBL_BLOCKED autolearn=disabled version=3.3.2 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hlysz-0006bZ-KH for bug-gnu-emacs@gnu.org; Fri, 12 Jul 2019 12:59:06 -0400 Received: from 195-159-176-226.customer.powertech.no ([195.159.176.226]:57448 helo=blaine.gmane.org) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1hlysz-0006aT-BU for bug-gnu-emacs@gnu.org; Fri, 12 Jul 2019 12:59:05 -0400 Received: from list by blaine.gmane.org with local (Exim 4.89) (envelope-from ) id 1hlysv-000RAZ-So for bug-gnu-emacs@gnu.org; Fri, 12 Jul 2019 18:59:01 +0200 X-Injected-Via-Gmane: http://gmane.org/ To: bug-gnu-emacs@gnu.org From: Eric Abrahamsen Subject: Re: bug#35443: 27.0.50; Gnus (nnimap) shows "ghost" messages in summary buffer Date: Fri, 12 Jul 2019 09:58:49 -0700 Message-ID: <871ryvw5o6.fsf@ericabrahamsen.net> References: <87wojfjxry.fsf@ericabrahamsen.net> <87h8ajjhux.fsf@ericabrahamsen.net> <87lfztiqu3.fsf@ericabrahamsen.net> <871s1735ec.fsf@ericabrahamsen.net> <87ftpn75bv.fsf@ericabrahamsen.net> <87pnn5wbea.fsf@ericabrahamsen.net> <87a7dkv5cx.fsf@ericabrahamsen.net> <87k1cotix6.fsf@ericabrahamsen.net> <875zo7w62u.fsf@ericabrahamsen.net> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.0.50 (gnu/linux) Cancel-Lock: sha1:9/RfmDyhE4dmf5P54g4qcguQf5M= X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 195.159.176.226 X-Spam-Score: -2.0 (--) X-Debbugs-Envelope-To: submit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.0 (---) --=-=-= Content-Type: text/plain Eric Abrahamsen writes: > Lars Ingebrigtsen writes: > >> Eric Abrahamsen writes: >> >>> Which is obviously insufficient. I've since changed it to: >>> >>> "\\([A-Z][[:ascii:]-]+: \\)" >>> >>> Which works better, but also seems susceptible to breakage due to >>> ignorance. A quick scan doesn't turn up any `mail-header-regexp' >>> variables or `this-is-a-mail-header-p' functions -- does the above >>> regexp seem reasonable? >> >> Headers are case insensitive... but not all characters in the [:ascii:] >> set are allowed; far from it. [:alnum:] is closer, but with - and... >> er... stuff... One of the RFCs probably has the definition. :-) > > Oh, good point. RFC2822 says chars 33 to 126, inclusive, minus the > colon. So I guess it's "[!-9,;-~]+: ", unlikely as some of those > characters seem. > > Now I'm working with the attached diff. Sorry, this one... --=-=-= Content-Type: text/x-patch Content-Disposition: attachment; filename=imap-header-transform-fix.diff diff --git a/lisp/gnus/nnimap.el b/lisp/gnus/nnimap.el index 9e52abc1ca..1fe6d32c4f 100644 --- a/lisp/gnus/nnimap.el +++ b/lisp/gnus/nnimap.el @@ -232,7 +232,7 @@ nnimap-retrieve-headers (defun nnimap-transform-headers () (goto-char (point-min)) - (let (article lines size string labels) + (let (seen-articles article lines size string labels) (cl-block nil (while (not (eobp)) (while (not (looking-at "\\* [0-9]+ FETCH")) @@ -261,45 +261,56 @@ nnimap-transform-headers (and (re-search-forward "UID \\([0-9]+\\)" (line-end-position) t) (match-string 1))) - (setq lines nil) - (beginning-of-line) - (setq size - (and (re-search-forward "RFC822.SIZE \\([0-9]+\\)" - (line-end-position) - t) - (match-string 1))) - (beginning-of-line) - (when (search-forward "X-GM-LABELS" (line-end-position) t) - (setq labels (ignore-errors (read (current-buffer))))) - (beginning-of-line) - (when (search-forward "BODYSTRUCTURE" (line-end-position) t) - (let ((structure (ignore-errors - (read (current-buffer))))) - (while (and (consp structure) - (not (atom (car structure)))) - (setq structure (car structure))) - (setq lines (if (and - (stringp (car structure)) - (equal (upcase (nth 0 structure)) "MESSAGE") - (equal (upcase (nth 1 structure)) "RFC822")) - (nth 9 structure) - (nth 7 structure))))) - (delete-region (line-beginning-position) (line-end-position)) - (insert (format "211 %s Article retrieved." article)) - (forward-line 1) - (when size - (insert (format "Chars: %s\n" size))) - (when lines - (insert (format "Lines: %s\n" lines))) - (when labels - (insert (format "X-GM-LABELS: %s\n" labels))) - ;; Most servers have a blank line after the headers, but - ;; Davmail doesn't. - (unless (re-search-forward "^\r$\\|^)\r?$" nil t) - (goto-char (point-max))) - (delete-region (line-beginning-position) (line-end-position)) - (insert ".") - (forward-line 1))))) + ;; If we've already got headers for this article, or this + ;; FETCH line doesn't provide headers for the article, skip + ;; it. See bug#35433. + (if (or (member article seen-articles) + (save-excursion + (forward-line) + (null (looking-at-p + "^[!-9;-~]+: ")))) + (delete-region (line-beginning-position) + (1+ (line-end-position))) + (setq lines nil) + (beginning-of-line) + (setq size + (and (re-search-forward "RFC822.SIZE \\([0-9]+\\)" + (line-end-position) + t) + (match-string 1))) + (beginning-of-line) + (when (search-forward "X-GM-LABELS" (line-end-position) t) + (setq labels (ignore-errors (read (current-buffer))))) + (beginning-of-line) + (when (search-forward "BODYSTRUCTURE" (line-end-position) t) + (let ((structure (ignore-errors + (read (current-buffer))))) + (while (and (consp structure) + (not (atom (car structure)))) + (setq structure (car structure))) + (setq lines (if (and + (stringp (car structure)) + (equal (upcase (nth 0 structure)) "MESSAGE") + (equal (upcase (nth 1 structure)) "RFC822")) + (nth 9 structure) + (nth 7 structure))))) + (delete-region (line-beginning-position) (line-end-position)) + (insert (format "211 %s Article retrieved." article)) + (forward-line 1) + (when size + (insert (format "Chars: %s\n" size))) + (when lines + (insert (format "Lines: %s\n" lines))) + (when labels + (insert (format "X-GM-LABELS: %s\n" labels))) + ;; Most servers have a blank line after the headers, but + ;; Davmail doesn't. + (unless (re-search-forward "^\r$\\|^)\r?$" nil t) + (goto-char (point-max))) + (delete-region (line-beginning-position) (line-end-position)) + (insert ".") + (forward-line 1) + (push article seen-articles)))))) (defun nnimap-unfold-quoted-lines () ;; Unfold quoted {number} strings. --=-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Sun Jul 14 12:19:43 2019 Received: (at 35443) by debbugs.gnu.org; 14 Jul 2019 16:19:43 +0000 Received: from localhost ([127.0.0.1]:45707 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hmhDy-0005YD-SA for submit@debbugs.gnu.org; Sun, 14 Jul 2019 12:19:43 -0400 Received: from ericabrahamsen.net ([52.70.2.18]:53938 helo=mail.ericabrahamsen.net) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hmhDx-0005Y1-AV for 35443@debbugs.gnu.org; Sun, 14 Jul 2019 12:19:42 -0400 Received: from localhost (c-73-254-86-141.hsd1.wa.comcast.net [73.254.86.141]) (Authenticated sender: eric@ericabrahamsen.net) by mail.ericabrahamsen.net (Postfix) with ESMTPSA id 18A55FA01E; Sun, 14 Jul 2019 16:19:32 +0000 (UTC) From: Eric Abrahamsen To: Lars Ingebrigtsen Subject: Re: bug#35443: 27.0.50; Gnus (nnimap) shows "ghost" messages in summary buffer References: <87wojfjxry.fsf@ericabrahamsen.net> <87h8ajjhux.fsf@ericabrahamsen.net> <87lfztiqu3.fsf@ericabrahamsen.net> <871s1735ec.fsf@ericabrahamsen.net> <87ftpn75bv.fsf@ericabrahamsen.net> <87pnn5wbea.fsf@ericabrahamsen.net> <87a7dkv5cx.fsf@ericabrahamsen.net> <87k1cotix6.fsf@ericabrahamsen.net> <875zo7w62u.fsf@ericabrahamsen.net> Date: Sun, 14 Jul 2019 09:19:31 -0700 In-Reply-To: <875zo7w62u.fsf@ericabrahamsen.net> (Eric Abrahamsen's message of "Fri, 12 Jul 2019 09:50:01 -0700") Message-ID: <871rysk2r0.fsf@ericabrahamsen.net> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.0.50 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 35443 Cc: 35443@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) On 07/12/19 09:50 AM, Eric Abrahamsen wrote: > Lars Ingebrigtsen writes: > >> Eric Abrahamsen writes: >> >>> Which is obviously insufficient. I've since changed it to: >>> >>> "\\([A-Z][[:ascii:]-]+: \\)" >>> >>> Which works better, but also seems susceptible to breakage due to >>> ignorance. A quick scan doesn't turn up any `mail-header-regexp' >>> variables or `this-is-a-mail-header-p' functions -- does the above >>> regexp seem reasonable? >> >> Headers are case insensitive... but not all characters in the [:ascii:] >> set are allowed; far from it. [:alnum:] is closer, but with - and... >> er... stuff... One of the RFCs probably has the definition. :-) > > Oh, good point. RFC2822 says chars 33 to 126, inclusive, minus the > colon. So I guess it's "[!-9,;-~]+: ", unlikely as some of those > characters seem. > > Now I'm working with the attached diff. I think this is the one. I'll push in a bit. From debbugs-submit-bounces@debbugs.gnu.org Mon Jul 15 13:45:55 2019 Received: (at 35443-done) by debbugs.gnu.org; 15 Jul 2019 17:45:55 +0000 Received: from localhost ([127.0.0.1]:48983 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hn52w-000744-VS for submit@debbugs.gnu.org; Mon, 15 Jul 2019 13:45:55 -0400 Received: from ericabrahamsen.net ([52.70.2.18]:55582 helo=mail.ericabrahamsen.net) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hn52o-0006s2-LH for 35443-done@debbugs.gnu.org; Mon, 15 Jul 2019 13:45:51 -0400 Received: from localhost (c-73-254-86-141.hsd1.wa.comcast.net [73.254.86.141]) (Authenticated sender: eric@ericabrahamsen.net) by mail.ericabrahamsen.net (Postfix) with ESMTPSA id 0A3ABFA02C; Mon, 15 Jul 2019 17:45:39 +0000 (UTC) From: Eric Abrahamsen To: Lars Ingebrigtsen Subject: Re: bug#35443: 27.0.50; Gnus (nnimap) shows "ghost" messages in summary buffer References: <87wojfjxry.fsf@ericabrahamsen.net> <87h8ajjhux.fsf@ericabrahamsen.net> <87lfztiqu3.fsf@ericabrahamsen.net> <871s1735ec.fsf@ericabrahamsen.net> <87ftpn75bv.fsf@ericabrahamsen.net> <87pnn5wbea.fsf@ericabrahamsen.net> <87a7dkv5cx.fsf@ericabrahamsen.net> Date: Mon, 15 Jul 2019 10:45:38 -0700 In-Reply-To: (Lars Ingebrigtsen's message of "Fri, 12 Jul 2019 17:02:22 +0200") Message-ID: <87k1cji43h.fsf@ericabrahamsen.net> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.0.50 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 35443-done Cc: 35443-done@debbugs.gnu.org, Ulrich Mueller X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) There goes... From debbugs-submit-bounces@debbugs.gnu.org Mon Jul 15 14:16:07 2019 Received: (at 35443) by debbugs.gnu.org; 15 Jul 2019 18:16:07 +0000 Received: from localhost ([127.0.0.1]:49041 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hn5WA-00034E-LA for submit@debbugs.gnu.org; Mon, 15 Jul 2019 14:16:06 -0400 Received: from quimby.gnus.org ([80.91.231.51]:46602) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hn5W8-00033z-7h for 35443@debbugs.gnu.org; Mon, 15 Jul 2019 14:16:04 -0400 Received: from cm-84.212.202.86.getinternet.no ([84.212.202.86] helo=sandy) by quimby.gnus.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.89) (envelope-from ) id 1hn5W4-0001yg-Q7; Mon, 15 Jul 2019 20:16:03 +0200 From: Lars Ingebrigtsen To: Eric Abrahamsen Subject: Re: bug#35443: 27.0.50; Gnus (nnimap) shows "ghost" messages in summary buffer References: <87wojfjxry.fsf@ericabrahamsen.net> <87h8ajjhux.fsf@ericabrahamsen.net> <87lfztiqu3.fsf@ericabrahamsen.net> <871s1735ec.fsf@ericabrahamsen.net> <87ftpn75bv.fsf@ericabrahamsen.net> <87pnn5wbea.fsf@ericabrahamsen.net> <87a7dkv5cx.fsf@ericabrahamsen.net> <87k1cji43h.fsf@ericabrahamsen.net> Date: Mon, 15 Jul 2019 20:16:00 +0200 In-Reply-To: <87k1cji43h.fsf@ericabrahamsen.net> (Eric Abrahamsen's message of "Mon, 15 Jul 2019 10:45:38 -0700") Message-ID: <87blxv9na7.fsf@mouse.gnus.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.0.50 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Report: Spam detection software, running on the system "quimby.gnus.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see @@CONTACT_ADDRESS@@ for details. Content preview: Eric Abrahamsen writes: > There goes... :-) Content analysis details: (-2.9 points, 5.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -1.0 ALL_TRUSTED Passed through trusted hosts only via SMTP -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1% [score: 0.0000] X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 35443 Cc: 35443@debbugs.gnu.org, Ulrich Mueller X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Eric Abrahamsen writes: > There goes... :-) Works for me. -- (domestic pets only, the antidote for overdose, milk.) bloggy blog: http://lars.ingebrigtsen.no From unknown Fri Jun 20 07:29:13 2025 Received: (at fakecontrol) by fakecontrolmessage; To: internal_control@debbugs.gnu.org From: Debbugs Internal Request Subject: Internal Control Message-Id: bug archived. Date: Tue, 13 Aug 2019 11:24:07 +0000 User-Agent: Fakemail v42.6.9 # This is a fake control message. # # The action: # bug archived. thanks # This fakemail brought to you by your local debbugs # administrator