GNU bug report logs - #12375
Broken matching of regexps in fancy splitting

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 12375 in the body.
You can then email your comments to 12375 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: jathd <jathdr <at> gmail.com>
To: submit <at> debbugs.gnu.org (The Gnus Bugfixing Girls + Boys)
Subject: Broken matching of regexps in fancy splitting
Date: Fri, 07 Sep 2012 05:41:45 +0200

Hi.

I'm not sure if this counts as a bug, but it's certainly an unexpected
behaviour (to me) to which I have found no reference in the manual.

The situation:

 * In .gnus.el, specify splitting with
    
       (setq nnmail-split-methods 'nnmail-split-fancy)
       (setq nnmail-split-fancy
             '(to "\\(\\w+\\)-devel@.*" "list.devel.\\1"))

 * Visit a message whose only relevant header (relative to the "to"
   abbreviation) is

       To: Batteries-devel <batteries-devel <at> lists.forge.ocamlcore.org>
   
 * Press B q and read the message in the minibuffer.

Current message: This message would go to list.devel.s
Expected message: This message would go to list.devel.batteries


I believe the problem comes from the way the regexp value I gave (the
VALUE field in the split) is matched against the contents of the header,
as defined in the function nnmail-split-it in nnmail.el. Once the
interesting bit of the header has been found, the match is done with
re-search-backward, which explains why the "\\w+" bit only matches the
last letter of "batteries".

This conflicts with what is usually meant by a regexp matching a
string. I know I was expecting the match to be done as if by
(string-match regexp relevant-header-part). Here's a simple patch that
changes re-search-backward to re-search-forward:


========================================================================
--- a/lisp/nnmail.el
+++ b/lisp/nnmail.el
@@ -1463,13 +1463,13 @@ See the documentation for the variable `nnmail-split-fancy' for details."
 		  (setq split-rest nil)
 		(setq split-rest (cddr split-rest))))
 	    (when split-rest
-	      (goto-char end)
+	      (goto-char start-of-value)
 	      (let ((value (nth 1 split)))
 		(if (symbolp value)
 		    (setq value (cdr (assq value nnmail-split-abbrev-alist))))
 		;; Someone might want to do a \N sub on this match, so get the
 		;; correct match positions.
-		(re-search-backward value start-of-value))
+		(re-search-forward value end))
 	      (dolist (sp (nnmail-split-it (car split-rest)))
 		(unless (member sp split-result)
 		  (push sp split-result))))))
========================================================================


This produces the expected message, and I don't think it should break
anything.

[As a side note: is there a particular reason why the file nnmail.el
uses re-search-backward everywhere and not re-search-forward, or does it
come down to taste?]


Ma Gnus v0.6
GNU Emacs 24.2.50.1 (x86_64-apple-darwin10.8.0, NS apple-appkit-1038.36)
 of 2012-09-07 on mbk.local
200 news-1.free.fr (5-1) NNRP Service Ready - newsmaster <at> proxad.net (posting ok)
500 What?

-- 
jathd

Message #8 received at 12375 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: jathd <jathdr <at> gmail.com>
Cc: 12375 <at> debbugs.gnu.org
Subject: Re: bug#12375: Broken matching of regexps in fancy splitting
Date: Tue, 25 Dec 2012 16:13:03 +0100

jathd <jathdr <at> gmail.com> writes:

> I'm not sure if this counts as a bug, but it's certainly an unexpected
> behaviour (to me) to which I have found no reference in the manual.
>
> The situation:
>
>  * In .gnus.el, specify splitting with
>
>        (setq nnmail-split-methods 'nnmail-split-fancy)
>        (setq nnmail-split-fancy
>              '(to "\\(\\w+\\)-devel@.*" "list.devel.\\1"))
>
>  * Visit a message whose only relevant header (relative to the "to"
>    abbreviation) is
>
>        To: Batteries-devel <batteries-devel <at> lists.forge.ocamlcore.org>
>
>  * Press B q and read the message in the minibuffer.
>
> Current message: This message would go to list.devel.s
> Expected message: This message would go to list.devel.batteries
>
> I believe the problem comes from the way the regexp value I gave (the
> VALUE field in the split) is matched against the contents of the header,
> as defined in the function nnmail-split-it in nnmail.el. Once the
> interesting bit of the header has been found, the match is done with
> re-search-backward, which explains why the "\\w+" bit only matches the
> last letter of "batteries".

Do you have `nnmail-split-fancy-match-partial-words' set?  If not, the
splitting machinery will put a \\< in front of the regexp, leading to
"batteries" being matched.

-- 
(domestic pets only, the antidote for overdose, milk.)
  http://lars.ingebrigtsen.no  *  Lars Magne Ingebrigtsen

Message #11 received at 12375 <at> debbugs.gnu.org (full text, mbox):

From: jathd <jathdr <at> gmail.com>
To: Lars Ingebrigtsen <larsi <at> gnus.org>
Cc: 12375 <at> debbugs.gnu.org
Subject: Re: bug#12375: Broken matching of regexps in fancy splitting
Date: Sat, 02 Feb 2013 20:02:30 +0100

Lars Ingebrigtsen <larsi <at> gnus.org> writes:

> Do you have `nnmail-split-fancy-match-partial-words' set?  If not, the
> splitting machinery will put a \\< in front of the regexp, leading to
> "batteries" being matched.

I have `nnmail-split-fancy-match-partial-words' set to `nil', and I
checked that the splitting process does put a "\\<" in the regexp.

The problem is that this regexp is used to decide whether a message
follows a rule (in other words, that regexp is matched against the
message header), but when extracting the value with which to replace
"\\1" in the group name "list.devel.\\1" I gave, it matches a second
time with the *original regexp* I gave (and not the whole header this
time, it narrows the search to the interesting part of the relevant
header field). So this time, it uses the regexp without the "\\<", which
explains why it replaces "\\1" with "s" instead of "batteries".

So there's a simple workaround: I just put a "\\<" in the regexp in
`nnmail-split-fancy', and it works fine. So I don't know if this is a
bug or not; the workaround is simple, but I still find the behaviour of
the matcher a bit unexpected. Namely, I would expect it to match as if
using `(string-match regexp-I-gave contents-of-relevant-header-field)'.
Maybe the manual should have a word about this?

-- 
jathd

Message #14 received at 12375 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: jathd <jathdr <at> gmail.com>
Cc: 12375 <at> debbugs.gnu.org
Subject: Re: bug#12375: Broken matching of regexps in fancy splitting
Date: Sat, 06 Jul 2013 19:53:46 +0200

jathd <jathdr <at> gmail.com> writes:

> The problem is that this regexp is used to decide whether a message
> follows a rule (in other words, that regexp is matched against the
> message header), but when extracting the value with which to replace
> "\\1" in the group name "list.devel.\\1" I gave, it matches a second
> time with the *original regexp* I gave (and not the whole header this
> time, it narrows the search to the interesting part of the relevant
> header field). So this time, it uses the regexp without the "\\<", which
> explains why it replaces "\\1" with "s" instead of "batteries".

Hm.  Looking at the code, I can't see where it's re-matching against the
original regexp.  `nnmail-expand-newtext' just uses the result from
`match-beginning'/`match-end', which should be the right thing to do.

If I've reading the code correctly, which I may not be doing.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Message #17 received at 12375 <at> debbugs.gnu.org (full text, mbox):

From: jathd <jathdr <at> gmail.com>
To: Lars Ingebrigtsen <larsi <at> gnus.org>
Cc: 12375 <at> debbugs.gnu.org
Subject: Re: bug#12375: Broken matching of regexps in fancy splitting
Date: Mon, 12 Aug 2013 00:07:00 +0200

Lars Ingebrigtsen <larsi <at> gnus.org> writes:

> Hm.  Looking at the code, I can't see where it's re-matching against the
> original regexp.

It's around line 1470:

    (let ((value (nth 1 split)))
      (if (symbolp value)
        (setq value (cdr (assq value nnmail-split-abbrev-alist))))
      ;; Someone might want to do a \N sub on this match, so get the
      ;; correct match positions.
      (re-search-backward value start-of-value))

The `value' is the regexp I specified in `nnmail-split-fancy'. Before
that, to see whether a header matches, the function does

    (re-search-backward (cdr cached-pair) nil t)

where (cdr cached-pair) is a regexp it constructed the first time
around. Since it prepended a "\\<" to *that* regexp, the header is
correctly identified; but it didn't prepend anything to `value', so the
extraction goes wrong.

-- 
jathd

Message #20 received at 12375 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: jathd <jathdr <at> gmail.com>
Cc: 12375 <at> debbugs.gnu.org
Subject: Re: bug#12375: Broken matching of regexps in fancy splitting
Date: Thu, 30 Jan 2014 15:37:49 -0800

jathd <jathdr <at> gmail.com> writes:

> The `value' is the regexp I specified in `nnmail-split-fancy'. Before
> that, to see whether a header matches, the function does
>
>     (re-search-backward (cdr cached-pair) nil t)
>
> where (cdr cached-pair) is a regexp it constructed the first time
> around. Since it prepended a "\\<" to *that* regexp, the header is
> correctly identified; but it didn't prepend anything to `value', so the
> extraction goes wrong.

Aha.

I've now rewritten that function to save the match data explicitly after
doing the first match so that we don't have to redo the search.  It
should be faster, too.

Does this fix the problem for you?  It's in git Gnus, but should be in
bzr Emacs soon...

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Message #23 received at 12375 <at> debbugs.gnu.org (full text, mbox):

From: Sven Joachim <svenjoac <at> gmx.de>
To: Lars Ingebrigtsen <larsi <at> gnus.org>
Cc: jathd <jathdr <at> gmail.com>, 12375 <at> debbugs.gnu.org
Subject: Re: bug#12375: Broken matching of regexps in fancy splitting
Date: Mon, 10 Feb 2014 19:21:24 +0100

On 2014-01-31 00:37 +0100, Lars Ingebrigtsen wrote:

> jathd <jathdr <at> gmail.com> writes:
>
>> The `value' is the regexp I specified in `nnmail-split-fancy'. Before
>> that, to see whether a header matches, the function does
>>
>>     (re-search-backward (cdr cached-pair) nil t)
>>
>> where (cdr cached-pair) is a regexp it constructed the first time
>> around. Since it prepended a "\\<" to *that* regexp, the header is
>> correctly identified; but it didn't prepend anything to `value', so the
>> extraction goes wrong.
>
> Aha.
>
> I've now rewritten that function to save the match data explicitly after
> doing the first match so that we don't have to redo the search.  It
> should be faster, too.
>
> Does this fix the problem for you?  It's in git Gnus, but should be in
> bzr Emacs soon...

It seems this happened recently, and it apparently broke something for
me: I'm subscribed to Debian bug 715194[1], and messages coming from
that bug have the following header line:

,----
| Resent-CC: Debian OpenSSH Maintainers <debian-ssh <at> lists.debian.org>
`----

Now I'm using the following split rule, copied from the Gnus manual:

,----
| (to "debian-\\b\\(\\w+\\)@lists.debian.org" "mail.debian.\\1")
`----

With that rule, messages to the above bug used to appear under
mail.debian.ssh, but since my latest Emacs upgrade yesterday, they show
up under "mail.debian.resent-cc: debian openssh maintainers <" instead.
Huh?

Cheers,
       Sven


1. http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=715194

Message #26 received at 12375 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Sven Joachim <svenjoac <at> gmx.de>
Cc: jathd <jathdr <at> gmail.com>, 12375 <at> debbugs.gnu.org
Subject: Re: bug#12375: Broken matching of regexps in fancy splitting
Date: Tue, 11 Feb 2014 21:52:10 -0800

Sven Joachim <svenjoac <at> gmx.de> writes:

> With that rule, messages to the above bug used to appear under
> mail.debian.ssh, but since my latest Emacs upgrade yesterday, they show
> up under "mail.debian.resent-cc: debian openssh maintainers <" instead.
> Huh?

Oops.  Fixed in bzr Emacs now, so it'll be fixed in git Gnus soon.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Message #33 received at 12375 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Sven Joachim <svenjoac <at> gmx.de>
Cc: jathd <jathdr <at> gmail.com>, 12375 <at> debbugs.gnu.org
Subject: Re: bug#12375: Broken matching of regexps in fancy splitting
Date: Wed, 25 Jan 2017 15:50:56 +0100

Lars Ingebrigtsen <larsi <at> gnus.org> writes:

> Sven Joachim <svenjoac <at> gmx.de> writes:
>
>> With that rule, messages to the above bug used to appear under
>> mail.debian.ssh, but since my latest Emacs upgrade yesterday, they show
>> up under "mail.debian.resent-cc: debian openssh maintainers <" instead.
>> Huh?
>
> Oops.  Fixed in bzr Emacs now, so it'll be fixed in git Gnus soon.

I guess this can be closed now...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.