GNU bug report logs - #56332
29.0.50; Large gnus imap groups; articles incorrectly marked as read (old)

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 56332 in the body.
You can then email your comments to 56332 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Michael Welsh Duggan <md5i <at> md5i.com>
To: bug-gnu-emacs <at> gnu.org
Subject: 29.0.50; Large gnus imap groups; articles incorrectly marked as
 read (old)
Date: Fri, 01 Jul 2022 02:47:10 -0400

[Message part 1 (text/plain, inline)]

I have a large, sparse imap folder with read and unread articles.  In
recent versions of Emacs, when I visit this group, a set of articles in
the middle (when sorted by number) are marked as read with the
`O' mark.  I have bisected this to the following commit:

commit ea640581bad1c596f657ca405f6c97e1b4fc4b11 (refs/bisect/bad)
Author: Lars Ingebrigtsen <larsi <at> gnus.org>
Date:   Thu Jun 23 11:22:29 2022 +0200

    Ensure that nnimap doesn't send too long lines to the server
    
    * lisp/gnus/nnimap.el (nnimap-retrieve-headers): Don't send
    too-long lines to the server (bug#56138).

I've attached the FETCH command used by last "good" version and the ones
issued by a build with the offending commit.  In the "bad" version, the
articles from 60795 until 63651 are marked as read with an `O'.

Good:

[Message part 2 (text/plain, attachment)]

[Message part 3 (text/plain, inline)]

Bad:

[Message part 4 (text/plain, attachment)]

[Message part 5 (text/plain, inline)]

In GNU Emacs 29.0.50 (build 1, x86_64-pc-linux-gnu, X toolkit, cairo version 1.16.0)
 of 2022-07-01 built on miko
Repository revision: efc2a878de6368eebb1ba73f5131eec563ca9b56
Repository branch: master
Windowing system distributor 'The X.Org Foundation', version 11.0.12101003
System Description: Debian GNU/Linux bookworm/sid

Configured using:
 'configure --without-toolkit-scroll-bars --with-x-toolkit=lucid
 --with-native-compilation --with-xinput2 'CFLAGS=-Og -ggdb''

Configured features:
CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GPM GSETTINGS HARFBUZZ JPEG JSON
LCMS2 LIBOTF LIBSELINUX LIBSYSTEMD LIBXML2 M17N_FLT MODULES NATIVE_COMP
NOTIFY INOTIFY PDUMPER PNG RSVG SECCOMP SOUND SQLITE3 THREADS TIFF X11
XAW3D XDBE XIM XINPUT2 XPM LUCID ZLIB

Important settings:
  value of $LANG: en_US.UTF-8
  value of $XMODIFIERS: @im=ibus
  locale-coding-system: utf-8-unix

Major mode: Fundamental

Minor modes in effect:
  display-time-mode: t
  magit-wip-initial-backup-mode: t
  magit-wip-before-change-mode: t
  magit-wip-after-apply-mode: t
  magit-wip-after-save-mode: t
  magit-wip-mode: t
  global-git-commit-mode: t
  magit-auto-revert-mode: t
  shell-dirtrack-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  line-number-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t

Load-path shadows:
/home/md5i/.config/emacs/elpa/transient-20220514.945/transient hides /home/md5i/src/emacs/master/lisp/transient

Features:
(shadow emacsbug noutline outline pcase so-long advice misearch
multi-isearch mule-util sort gnus-cite mail-extr textsec uni-scripts
idna-mapping ucs-normalize uni-confusable textsec-check gnus-async
gnus-bcklg gnus-ml disp-table gnus-topic nndraft nnmh nnfolder utf-7
epa-file network-stream gnus-agent gnus-srvr gnus-score score-mode
nnvirtual gnus-msg gnus-art mm-uu mml2015 mm-view mml-smime smime dig
gnus-cache gnus-sum shr pixel-fill kinsoku url-file url-dired svg
gnus-demon nntp gnus-group gnus-undo gnutls gnus-start gnus-dbus
gnus-cloud gnus-spec gnus-win flyspell ispell view pacproxy descr-text
tramp tramp-loaddefs trampver tramp-integration cus-edit pp cus-load
files-x tramp-compat parse-time iso8601 ls-lisp time sieve-manage sasl
sasl-anonymous sasl-login sasl-plain rng-loc rng-uri rng-parse rng-match
rng-dt rng-util rng-pttrn nxml-parse nxml-ns nxml-enc xmltok nxml-util
sgml-mode facemenu dom python ps-print ps-print-loaddefs ps-def lpr
picture nm dbus xml magit-submodule magit-obsolete magit-blame
magit-stash magit-reflog magit-bisect magit-push magit-pull magit-fetch
magit-clone magit-remote magit-commit magit-sequence magit-notes
magit-worktree magit-tag magit-merge magit-branch magit-reset
magit-files magit-refs magit-status magit magit-repos magit-apply
magit-wip magit-log which-func imenu magit-diff smerge-mode diff
diff-mode easy-mmode git-commit log-edit pcvs-util add-log magit-core
magit-autorevert autorevert filenotify magit-margin magit-transient
magit-process with-editor shell pcomplete server magit-mode transient
comp comp-cstr warnings rx cl-extra edmacro kmacro help-mode magit-git
magit-base magit-section format-spec crm dash compat-27 compat-26 compat
nnimap nnmail gnus-int mail-source gnus-range message sendmail
yank-media rfc822 mml mml-sec epa mm-decode mm-bodies mm-encode
mailabbrev gmm-utils mailheader utf7 netrc nnoo gnus wid-edit nnheader
gnus-util time-date mail-utils range gnus-o365-oauth2 oauth2 url-http
url-auth mail-parse rfc2231 rfc2047 rfc2045 mm-util ietf-drums
mail-prsvr url-gw nsm puny plstore generated generic-x epg rfc6068
epg-config ediff ediff-merg ediff-mult ediff-wind ediff-diff ediff-help
ediff-init ediff-util dired-x dired dired-loaddefs compile
text-property-search comint ring ansi-color cc-mode cc-fonts cc-guess
cc-menus cc-cmds cc-styles cc-align cc-engine cc-vars cc-defs derived
debian-el rainbow-delimiters-autoloads info package browse-url url
url-proxy url-privacy url-expand url-methods url-history url-cookie
generate-lisp-file url-domsuf url-util mailcap url-handlers url-parse
auth-source cl-seq eieio eieio-core cl-macs eieio-loaddefs
password-cache json subr-x map byte-opt gv bytecomp byte-compile cconv
url-vars cl-loaddefs cl-lib rmc iso-transl tooltip eldoc paren electric
uniquify ediff-hook vc-hooks lisp-float-type elisp-mode mwheel
term/x-win x-win term/common-win x-dnd tool-bar dnd fontset image
regexp-opt fringe tabulated-list replace newcomment text-mode lisp-mode
prog-mode register page tab-bar menu-bar rfn-eshadow isearch easymenu
timer select scroll-bar mouse jit-lock font-lock syntax font-core
term/tty-colors frame minibuffer nadvice seq simple cl-generic
indonesian philippine cham georgian utf-8-lang misc-lang vietnamese
tibetan thai tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek
romanian slovak czech european ethiopic indian cyrillic chinese
composite emoji-zwj charscript charprop case-table epa-hook
jka-cmpr-hook help abbrev obarray oclosure cl-preloaded button loaddefs
faces cus-face macroexp files window text-properties overlay sha1 md5
base64 format env code-pages mule custom widget keymap
hashtable-print-readable backquote threads dbusbind inotify lcms2
dynamic-setting system-font-setting font-render-setting cairo x-toolkit
xinput2 x multi-tty make-network-process native-compile emacs)

Memory information:
((conses 16 727404 164291)
 (symbols 48 29486 22)
 (strings 32 161584 25563)
 (string-bytes 1 5310091)
 (vectors 16 112893)
 (vector-slots 8 1650392 153295)
 (floats 8 1114 281)
 (intervals 56 777 216)
 (buffers 992 18))

-- 
Michael Welsh Duggan
(md5i <at> md5i.com)

Message #8 received at 56332 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Michael Welsh Duggan <md5i <at> md5i.com>
Cc: 56332 <at> debbugs.gnu.org
Subject: Re: bug#56332: 29.0.50; Large gnus imap groups; articles
 incorrectly marked as read (old)
Date: Fri, 01 Jul 2022 11:36:30 +0200

Michael Welsh Duggan <md5i <at> md5i.com> writes:

> I have a large, sparse imap folder with read and unread articles.  In
> recent versions of Emacs, when I visit this group, a set of articles in
> the middle (when sorted by number) are marked as read with the
> `O' mark.

And they should be unread?

> I've attached the FETCH command used by last "good" version and the ones
> issued by a build with the offending commit.  In the "bad" version, the
> articles from 60795 until 63651 are marked as read with an `O'.

Hm.  Those are pretty much in the middle...  If nnimap fails to parse
the headers, I'd expect everything after the failure to be missing in
the Summary buffer, not be marked as read.

Do you use the Gnus cache or the Gnus Agent?

Hm...  reading `nnimap-transform-headers', it kinda seems like it might
be working (mostly) by accident in the multiple-FETCH situation.  I'll
try to reproduce this here and see what I can find out.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #11 received at 56332 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Michael Welsh Duggan <md5i <at> md5i.com>
Cc: 56332 <at> debbugs.gnu.org
Subject: Re: bug#56332: 29.0.50; Large gnus imap groups; articles
 incorrectly marked as read (old)
Date: Fri, 01 Jul 2022 11:50:31 +0200

Lars Ingebrigtsen <larsi <at> gnus.org> writes:

> Hm...  reading `nnimap-transform-headers', it kinda seems like it might
> be working (mostly) by accident in the multiple-FETCH situation.  I'll
> try to reproduce this here and see what I can find out.

No, it seems like it should handle this fine.

Can you do a `M-x debug-on-entry RET nnimap-split-incoming-mail RET' and
then look in the " *nnimap ...*" (note leading space) buffer and see
whether the headers for the articles are there or whether anything looks
obviously messed-up?

I've now introduced a new nnimap--max-retrieve-headers variable that can
be used to debug this easier -- set it to a very high number to get the
old behaviour, and compare the contents of that buffer with a lower
value.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #14 received at 56332 <at> debbugs.gnu.org (full text, mbox):

From: Michael Welsh Duggan <mwd <at> md5i.com>
To: Lars Ingebrigtsen <larsi <at> gnus.org>
Cc: 56332 <at> debbugs.gnu.org
Subject: Re: bug#56332: 29.0.50; Large gnus imap groups; articles
 incorrectly marked as read (old)
Date: Fri, 01 Jul 2022 11:54:56 -0400

Lars Ingebrigtsen <larsi <at> gnus.org> writes:

> Lars Ingebrigtsen <larsi <at> gnus.org> writes:
>
>> Hm...  reading `nnimap-transform-headers', it kinda seems like it might
>> be working (mostly) by accident in the multiple-FETCH situation.  I'll
>> try to reproduce this here and see what I can find out.
>
> No, it seems like it should handle this fine.
>
> Can you do a `M-x debug-on-entry RET nnimap-split-incoming-mail RET' and
> then look in the " *nnimap ...*" (note leading space) buffer and see
> whether the headers for the articles are there or whether anything looks
> obviously messed-up?

I think you must have meant a different function?  Because I am not
doing any Gnus-side splitting right now (I'm using server-side sieve
splitting instead).

> I've now introduced a new nnimap--max-retrieve-headers variable that can
> be used to debug this easier -- set it to a very high number to get the
> old behaviour, and compare the contents of that buffer with a lower
> value.

I can, at least, verify that setting this variable to a higher value
does "solve" the problem, though we still don't know why the problem
happens.

-- 
Michael Welsh Duggan
(md5i <at> md5i.com)

Message #17 received at 56332 <at> debbugs.gnu.org (full text, mbox):

From: Michael Welsh Duggan <mwd <at> md5i.com>
To: Michael Welsh Duggan <mwd <at> md5i.com>
Cc: Lars Ingebrigtsen <larsi <at> gnus.org>, 56332 <at> debbugs.gnu.org
Subject: Re: bug#56332: 29.0.50; Large gnus imap groups; articles
 incorrectly marked as read (old)
Date: Fri, 01 Jul 2022 11:56:20 -0400

Michael Welsh Duggan <mwd <at> md5i.com> writes:

> Lars Ingebrigtsen <larsi <at> gnus.org> writes:
>
>> Lars Ingebrigtsen <larsi <at> gnus.org> writes:
>>
>>> Hm...  reading `nnimap-transform-headers', it kinda seems like it might
>>> be working (mostly) by accident in the multiple-FETCH situation.  I'll
>>> try to reproduce this here and see what I can find out.
>>
>> No, it seems like it should handle this fine.
>>
>> Can you do a `M-x debug-on-entry RET nnimap-split-incoming-mail RET' and
>> then look in the " *nnimap ...*" (note leading space) buffer and see
>> whether the headers for the articles are there or whether anything looks
>> obviously messed-up?
>
> I think you must have meant a different function?  Because I am not
> doing any Gnus-side splitting right now (I'm using server-side sieve
> splitting instead).

I should clarify: setting a breakpoint on that function does not trigger
when entering the group in question.

-- 
Michael Welsh Duggan
(md5i <at> md5i.com)

Message #20 received at 56332 <at> debbugs.gnu.org (full text, mbox):

From: Michael Welsh Duggan <mwd <at> md5i.com>
To: Lars Ingebrigtsen <larsi <at> gnus.org>
Cc: 56332 <at> debbugs.gnu.org
Subject: Re: bug#56332: 29.0.50; Large gnus imap groups; articles
 incorrectly marked as read (old)
Date: Fri, 01 Jul 2022 11:56:58 -0400

Lars Ingebrigtsen <larsi <at> gnus.org> writes:

> Michael Welsh Duggan <md5i <at> md5i.com> writes:
>
>> I have a large, sparse imap folder with read and unread articles.  In
>> recent versions of Emacs, when I visit this group, a set of articles in
>> the middle (when sorted by number) are marked as read with the
>> `O' mark.
>
> And they should be unread?
>
>> I've attached the FETCH command used by last "good" version and the ones
>> issued by a build with the offending commit.  In the "bad" version, the
>> articles from 60795 until 63651 are marked as read with an `O'.
>
> Hm.  Those are pretty much in the middle...  If nnimap fails to parse
> the headers, I'd expect everything after the failure to be missing in
> the Summary buffer, not be marked as read.
>
> Do you use the Gnus cache or the Gnus Agent?

I do not.

-- 
Michael Welsh Duggan
(md5i <at> md5i.com)

Message #23 received at 56332 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Michael Welsh Duggan <mwd <at> md5i.com>
Cc: 56332 <at> debbugs.gnu.org
Subject: Re: bug#56332: 29.0.50; Large gnus imap groups; articles
 incorrectly marked as read (old)
Date: Sat, 02 Jul 2022 14:00:21 +0200

Michael Welsh Duggan <mwd <at> md5i.com> writes:

>> Can you do a `M-x debug-on-entry RET nnimap-split-incoming-mail RET' and
>> then look in the " *nnimap ...*" (note leading space) buffer and see
>> whether the headers for the articles are there or whether anything looks
>> obviously messed-up?
>
> I think you must have meant a different function?  Because I am not
> doing any Gnus-side splitting right now (I'm using server-side sieve
> splitting instead).

Yes, sorry -- I meant nnimap-transform-headers.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #26 received at 56332 <at> debbugs.gnu.org (full text, mbox):

From: Michael Welsh Duggan <mwd <at> md5i.com>
To: Lars Ingebrigtsen <larsi <at> gnus.org>
Cc: Michael Welsh Duggan <mwd <at> md5i.com>, 56332 <at> debbugs.gnu.org
Subject: Re: bug#56332: 29.0.50; Large gnus imap groups; articles
 incorrectly marked as read (old)
Date: Sat, 02 Jul 2022 10:30:05 -0400

[Message part 1 (text/plain, inline)]

Lars Ingebrigtsen <larsi <at> gnus.org> writes:

> Michael Welsh Duggan <mwd <at> md5i.com> writes:
>
>>> Can you do a `M-x debug-on-entry RET nnimap-split-incoming-mail RET' and
>>> then look in the " *nnimap ...*" (note leading space) buffer and see
>>> whether the headers for the articles are there or whether anything looks
>>> obviously messed-up?
>>
>> I think you must have meant a different function?  Because I am not
>> doing any Gnus-side splitting right now (I'm using server-side sieve
>> splitting instead).
>
> Yes, sorry -- I meant nnimap-transform-headers.

Right.  Should have guessed that.  Here's what I have been able to
determine:

1) The problem is non-determinant.  The articles that are marked read
   don't appear as read in every run, and from run to run the list of
   articles that appear as read is not consistent.

2) When the problem happens, it seems to happen because the order of the
   fetch responses do not appear in the order of the fetches themselves.
   I attach a buffer where I have removed all of the responses except
   for the fetch response sequence numbers along with the message UIDs
   and the FETCH OK responses.  All the sequence numbers are in order
   through 7394 (UID 65348), after which the next response is 7889 (UID
   67026).  That sequence runs until 8048 (UID 67483).  The next set is
   7559 (UID 66026) through 7888 (UID 67024).  This is followed by 7395
   (UID 67024) through 7558 (UID 66024).

   The articles that are marked as "read" are the articles in the
   out-of-sequence sets, those represented by sequence numbers 7395
   through 7888.

Here's the stripped down buffer of responses:

[Message part 2 (text/plain, attachment)]

[Message part 3 (text/plain, inline)]

-- 
Michael Welsh Duggan
(md5i <at> md5i.com)

Message #29 received at 56332 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Michael Welsh Duggan <mwd <at> md5i.com>
Cc: 56332 <at> debbugs.gnu.org
Subject: Re: bug#56332: 29.0.50; Large gnus imap groups; articles
 incorrectly marked as read (old)
Date: Sat, 02 Jul 2022 17:23:49 +0200

Michael Welsh Duggan <mwd <at> md5i.com> writes:

> 2) When the problem happens, it seems to happen because the order of the
>    fetch responses do not appear in the order of the fetches themselves.

Oh, that's interesting.  IMAP is a streaming protocol, so if you send
two commands after one another, you should first get the responses from
the first, and then from the last.  It sounds like UID FETCH doesn't
respect that?

In which case the simple solution would be to wait until the first
command has ended before issuing a new one, but that would make things a
bit slower (depending on the latency of the connection).

Hm...  OK, I've tried this myself now, and I can definitely see
something odd here.  I don't see shuffled headers, but I see

OUTPUT FROM FIRST
OUTPUT FROM SECOND
26166 OK Fetch completed (0.002 + 0.000 + 0.001 secs).
26167 OK Fetch completed (0.002 + 0.000 + 0.001 secs).

So that seems to support this -- UID FETCH is not really streamable (at
least not with this IMAP server).  I'm using Dovecot -- are you using
the same?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #32 received at 56332 <at> debbugs.gnu.org (full text, mbox):

From: Michael Welsh Duggan <mwd <at> md5i.com>
To: Lars Ingebrigtsen <larsi <at> gnus.org>
Cc: Michael Welsh Duggan <mwd <at> md5i.com>, 56332 <at> debbugs.gnu.org
Subject: Re: bug#56332: 29.0.50; Large gnus imap groups; articles
 incorrectly marked as read (old)
Date: Sat, 02 Jul 2022 12:00:30 -0400

Lars Ingebrigtsen <larsi <at> gnus.org> writes:

> Michael Welsh Duggan <mwd <at> md5i.com> writes:
>
>> 2) When the problem happens, it seems to happen because the order of the
>>    fetch responses do not appear in the order of the fetches themselves.
>
> Oh, that's interesting.  IMAP is a streaming protocol, so if you send
> two commands after one another, you should first get the responses from
> the first, and then from the last.  It sounds like UID FETCH doesn't
> respect that?
>
> In which case the simple solution would be to wait until the first
> command has ended before issuing a new one, but that would make things a
> bit slower (depending on the latency of the connection).
>
> Hm...  OK, I've tried this myself now, and I can definitely see
> something odd here.  I don't see shuffled headers, but I see
>
> OUTPUT FROM FIRST
> OUTPUT FROM SECOND
> 26166 OK Fetch completed (0.002 + 0.000 + 0.001 secs).
> 26167 OK Fetch completed (0.002 + 0.000 + 0.001 secs).
> 
> So that seems to support this -- UID FETCH is not really streamable (at
> least not with this IMAP server).  I'm using Dovecot -- are you using
> the same?

I am using Dovecot.  I am betting that the only reason I am seeing
shuffled headers is due to the larger group, and maybe due to the
sparsity of the UIDs.  More fetches grant more opportunity for random
re-ordering.

I will also note that, though the fetch data responses are not in order,
the fetch completion messages are in order.  Though I'm not certain they
have to be.  Here's some data from the Internet, though I can't find
anything in the standard that seems to either confirm or refute this
data:

https://stackoverflow.com/questions/26034086/does-imap-guarantee-that-servers-send-responses-in-order

Wouldn't another solution be to sort the results by UID?  They are being
requested in UID order, after all.

-- 
Michael Welsh Duggan
(md5i <at> md5i.com)

Message #35 received at 56332 <at> debbugs.gnu.org (full text, mbox):

From: Michael Welsh Duggan <mwd <at> md5i.com>
To: Michael Welsh Duggan <mwd <at> md5i.com>
Cc: Lars Ingebrigtsen <larsi <at> gnus.org>, 56332 <at> debbugs.gnu.org
Subject: Re: bug#56332: 29.0.50; Large gnus imap groups; articles
 incorrectly marked as read (old)
Date: Sat, 02 Jul 2022 12:40:04 -0400

Michael Welsh Duggan <mwd <at> md5i.com> writes:

> Lars Ingebrigtsen <larsi <at> gnus.org> writes:
>
>> Michael Welsh Duggan <mwd <at> md5i.com> writes:
>>
>>> 2) When the problem happens, it seems to happen because the order of the
>>>    fetch responses do not appear in the order of the fetches themselves.
>>
>> Oh, that's interesting.  IMAP is a streaming protocol, so if you send
>> two commands after one another, you should first get the responses from
>> the first, and then from the last.  It sounds like UID FETCH doesn't
>> respect that?
>>
>> In which case the simple solution would be to wait until the first
>> command has ended before issuing a new one, but that would make things a
>> bit slower (depending on the latency of the connection).
>>
>> Hm...  OK, I've tried this myself now, and I can definitely see
>> something odd here.  I don't see shuffled headers, but I see
>>
>> OUTPUT FROM FIRST
>> OUTPUT FROM SECOND
>> 26166 OK Fetch completed (0.002 + 0.000 + 0.001 secs).
>> 26167 OK Fetch completed (0.002 + 0.000 + 0.001 secs).
>> 
>> So that seems to support this -- UID FETCH is not really streamable (at
>> least not with this IMAP server).  I'm using Dovecot -- are you using
>> the same?
>
> I am using Dovecot.  I am betting that the only reason I am seeing
> shuffled headers is due to the larger group, and maybe due to the
> sparsity of the UIDs.  More fetches grant more opportunity for random
> re-ordering.
>
> I will also note that, though the fetch data responses are not in order,
> the fetch completion messages are in order.  Though I'm not certain they
> have to be.  Here's some data from the Internet, though I can't find
> anything in the standard that seems to either confirm or refute this
> data:
>
> https://stackoverflow.com/questions/26034086/does-imap-guarantee-that-servers-send-responses-in-order
>
> Wouldn't another solution be to sort the results by UID?  They are being
> requested in UID order, after all.

You should probably read this section of the RFC, especially the
"Note:".

https://datatracker.ietf.org/doc/html/rfc3501#section-5.5

-- 
Michael Welsh Duggan
(md5i <at> md5i.com)

Message #38 received at 56332 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Michael Welsh Duggan <mwd <at> md5i.com>
Cc: 56332 <at> debbugs.gnu.org
Subject: Re: bug#56332: 29.0.50; Large gnus imap groups; articles
 incorrectly marked as read (old)
Date: Sun, 03 Jul 2022 12:59:31 +0200

Michael Welsh Duggan <mwd <at> md5i.com> writes:

>> I will also note that, though the fetch data responses are not in order,
>> the fetch completion messages are in order.  Though I'm not certain they
>> have to be.  Here's some data from the Internet, though I can't find
>> anything in the standard that seems to either confirm or refute this
>> data:
>>
>> https://stackoverflow.com/questions/26034086/does-imap-guarantee-that-servers-send-responses-in-order
>>
>> Wouldn't another solution be to sort the results by UID?  They are being
>> requested in UID order, after all.
>
> You should probably read this section of the RFC, especially the
> "Note:".
>
> https://datatracker.ietf.org/doc/html/rfc3501#section-5.5

Reading that, I'm not sure whether the completion messages are
guaranteed to be in order, either, so I've now changed the code to avoid
streaming altogether.  Can you check whether that fixes the problem?

(If the completion messages are guaranteed to be in order, we could
change it back to using streaming and then just reorder the results, as
you suggest, but I'm not sure it's worth it even if it is guaranteed.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #41 received at 56332 <at> debbugs.gnu.org (full text, mbox):

From: Michael Welsh Duggan <mwd <at> md5i.com>
To: Lars Ingebrigtsen <larsi <at> gnus.org>
Cc: Michael Welsh Duggan <mwd <at> md5i.com>, 56332 <at> debbugs.gnu.org
Subject: Re: bug#56332: 29.0.50; Large gnus imap groups; articles
 incorrectly marked as read (old)
Date: Sun, 03 Jul 2022 10:50:46 -0400

Lars Ingebrigtsen <larsi <at> gnus.org> writes:

> Michael Welsh Duggan <mwd <at> md5i.com> writes:
>
>>> I will also note that, though the fetch data responses are not in order,
>>> the fetch completion messages are in order.  Though I'm not certain they
>>> have to be.  Here's some data from the Internet, though I can't find
>>> anything in the standard that seems to either confirm or refute this
>>> data:
>>>
>>> https://stackoverflow.com/questions/26034086/does-imap-guarantee-that-servers-send-responses-in-order
>>>
>>> Wouldn't another solution be to sort the results by UID?  They are being
>>> requested in UID order, after all.
>>
>> You should probably read this section of the RFC, especially the
>> "Note:".
>>
>> https://datatracker.ietf.org/doc/html/rfc3501#section-5.5
>
> Reading that, I'm not sure whether the completion messages are
> guaranteed to be in order, either, so I've now changed the code to avoid
> streaming altogether.  Can you check whether that fixes the problem?
>
> (If the completion messages are guaranteed to be in order, we could
> change it back to using streaming and then just reorder the results, as
> you suggest, but I'm not sure it's worth it even if it is guaranteed.)

As expected, I cannot cause a failure using this.  Though I will note
that there is no guarantee that a server will send its untagged fetch
responses in any particular order.  I will admit that it normally makes
very little sense for a server to actually do this in another order, but
I can conceive of two reasons that a server might do this, one of which
would probably not affect Gnus, but the other of which could:

1) If the messages were not requested in a sorted order, I could see
a server generating a sorted order anyway for internal implementation
reasons. (Not a problem for Gnus, since it will always request the
messages in sorted order.)

2) If messages were being retrieved from some sort of big-data
map-reduce store, the messages could conceivable come back in any order
due to messages being stored on different nodes with different
performance characteristics.  The server is not obligated to sort the
results before returning them.

That said, this fix will work for me for Dovecot and may work on all
existent or future IMAP servers, but I don't believe the standard
requires that this will be the case.

-- 
Michael Welsh Duggan
(md5i <at> md5i.com)

Message #44 received at 56332 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Michael Welsh Duggan <mwd <at> md5i.com>
Cc: 56332 <at> debbugs.gnu.org
Subject: Re: bug#56332: 29.0.50; Large gnus imap groups; articles
 incorrectly marked as read (old)
Date: Mon, 04 Jul 2022 12:49:25 +0200

Michael Welsh Duggan <mwd <at> md5i.com> writes:

> As expected, I cannot cause a failure using this.

Thanks for testing.

> Though I will note that there is no guarantee that a server will send
> its untagged fetch responses in any particular order.

That's true, but I think it's highly unlikely that any servers would do
that, so it doesn't seem worth adding code to sort the headers.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.