From debbugs-submit-bounces@debbugs.gnu.org Tue Dec 08 18:33:58 2020 Received: (at submit) by debbugs.gnu.org; 8 Dec 2020 23:33:58 +0000 Received: from localhost ([127.0.0.1]:60211 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kmmUX-0002tr-1B for submit@debbugs.gnu.org; Tue, 08 Dec 2020 18:33:58 -0500 Received: from lists.gnu.org ([209.51.188.17]:46546) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kmmJZ-0000TY-E3 for submit@debbugs.gnu.org; Tue, 08 Dec 2020 18:22:37 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:47656) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kmmJZ-0005ZH-8h for bug-gnu-emacs@gnu.org; Tue, 08 Dec 2020 18:22:37 -0500 Received: from amoxicillin.media.mit.edu ([18.27.72.70]:34632) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kmmJV-0004a9-Gp for bug-gnu-emacs@gnu.org; Tue, 08 Dec 2020 18:22:35 -0500 Received: from bella.media.mit.edu (bella.media.mit.edu [18.27.127.33]) by amoxicillin.media.mit.edu (8.15.2/8.15.2/MEDIA) with ESMTP id 0B8NMV6a006547; Tue, 8 Dec 2020 18:22:31 -0500 Received: from bella.media.mit.edu (bella.media.mit.edu [18.27.127.33]) by bella.media.mit.edu (Postfix) with ESMTP id DC32F32604; Tue, 8 Dec 2020 18:22:30 -0500 (EST) Received: by bella.media.mit.edu (Postfix, from userid 1000) id B10B7123E20; Tue, 8 Dec 2020 18:22:30 -0500 (EST) From: emacs-f@media.mit.edu To: bug-gnu-emacs@gnu.org Subject: [PATCH] rmail-spam-filter can lose mail Date: Tue, 08 Dec 2020 18:22:30 -0500 Message-ID: <87czzjnart.fsf@media.mit.edu> MIME-Version: 1.0 Content-Type: text/plain Received-SPF: pass client-ip=18.27.72.70; envelope-from=foner@media.mit.edu; helo=amoxicillin.media.mit.edu X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-Spam-Score: -1.3 (-) X-Debbugs-Envelope-To: submit X-Mailman-Approved-At: Tue, 08 Dec 2020 18:33:55 -0500 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.3 (--) [See after description for patch and reproducer.] I just started using rmail-spam-filter to flush a couple of trolls, and discovered that it's got bugs which will cause mail to be spuriously deleted and/or overlooked. The bugs probably date back to the package's introduction; I'm running 24.3.1 and the code is unchanged in 27.1 except for tiny stylistic changes to three comments. There are two bugs here: (1) If an output-and-delete action fires, and rsf-file does not already exist, -the wrong message will be filed and deleted-! Specifically, the affected message will be the first unseen message, and -not- the spam. (This message will be "stealth-deleted"---it won't show up with a D in the summary window, but it -will- have a label of "deleted". Expunging or saving doesn't seem to actually delete it, but if you save, kill the RMAIL buffer, and reload it, -now- the message has a D and -then- it will be expunged on save. Also, it was simply deleted without the kill & reload in my main RMAIL file when I first hit this, though I haven't reproduced that. See below. It's buggy either way.) (2) Anytime rsf decides to prompt, it spuriously moves to the first unseen message in order to make the rest of the screen look better during the prompt. But that means that when control leaves rsf and reverts back to the usual mailreading flow, the user will entirely miss seeing that message, since RMAIL will have been incorrectly told the user has seen it. (This is because RMAIL marks the message as seen while rsf is displaying an entirely unrelated prompt about a piece of -spam-, so the user is likely looking at the prompt, not an entirely unrelated non-spam message. Then, when rsf exits, the message -after- what was the first unseen message is made current. This will cause users to simply miss mail unless they notice (perhaps from the summary buffer) that they've skipped one. And if there are two prompts (such as for an expunge), yet -another- message will be skipped.) Both of these bugs are caused by use of rmail-first-unseen-message in an undisciplined fashion---showing the message it denotes will side-effect RMAIL's state, but rsf's author didn't seem to know that. Note that bug #1 is caused by a failure to reset the current message (from the first unseen message to the target message) before carrying out the file & delete. This may have been overlooked because it will only occur if the output file doesn't already exist, so it'll happen on the first use and then the bug will seem to vanish, if it's even noticed at all. [Note: Even though I didn't do the kill-and-reload-the-buffer thing for my main RMAIL file, which is where I first noticed this bug, the wrong message -was- deleted. I could verify that this did in fact happen, because it wound up in rsf-file, and no line of that message could be found in my RMAIL file via grep, even though it -could- be found in a backup taken the day before. So the message was clearly deleted, and it was clearly filed---in fact, I wouldn't have caught the deletion at all if I hadn't immediately looked at rsf-file (after all, I was just starting to use rsf and was thus testing it out). Someone who was less paranoid could easily have simply lost an important message. My test case clearly shows the misfiling of the wrong message, through, and the stealth-deletion of it as well.] [Note: There are a lot of FIXME's scattered through the original code; I've removed one of them, but there are several left that I haven't looked at. In my own version (not in this patch), I've also added a few small features, such as (a) auto-expunge without prompting and/or (b) not even trying to offer to expunge and not expunging (to be left to the user later), and I've also tightened up the prompts and feedback, but those are -not- in this patch; if there's interest, I'll post a second patch. This patch is solely a bugfix.] Patch: --- rsf.original.el 2020-12-07 18:36:10.597449493 -0500 +++ rsf.patched.el 2020-12-07 18:40:29.811489011 -0500 @@ -216,0 +217,5 @@ +;; Don't spuriously advance to the next unseen message while prompting, because that causes it to then +;; be -missed- while actually reading mail afterwards! Call this instead of rmail-first-unseen-message. +(defun rsf-rmail-last-seen-message () + (max 1 (1- (or (rmail-first-unseen-message) 1)))) ; r-f-u-m can return nil in a completely empty buffer. + @@ -330,2 +335 @@ - (let ((rmail-current-message msg) ; FIXME does this do anything? - (action (cdr (assq 'action + (let ((action (cdr (assq 'action @@ -340 +344,2 @@ - (rmail-show-message (rmail-first-unseen-message) t)) + (rmail-show-message (rsf-rmail-last-seen-message) t)) + (setq rmail-current-message msg) ; Reset to spammy message! @@ -380 +385 @@ - (rmail-show-message (or (rmail-first-unseen-message) 1) t) + (rmail-show-message (rsf-rmail-last-seen-message) t) To reproduce: Create a file (let's call it test-rmail) with these contents: [BEGIN] >From foo@example.com Mon Dec 7 19:05:49 2020 Return-Path: From: foo@example.com To: foo@example.com Subject: 1 Date: Mon, 07 Dec 2020 19:05:49 -0500 Message-ID: <87r1ozlffh.fsf@example.com> MIME-Version: 1.0 Content-Type: text/plain X-RMAIL-ATTRIBUTES: -------- foo. >From foo@example.com Mon Dec 7 19:05:49 2020 Return-Path: From: foo@example.com To: foo@example.com Subject: 2 Date: Mon, 07 Dec 2020 19:05:49 -0500 Message-ID: <87r1ozlffh.fsf@example.com> MIME-Version: 1.0 Content-Type: text/plain X-RMAIL-ATTRIBUTES: -------- foo. >From foo@example.com Mon Dec 7 19:05:49 2020 Return-Path: From: foo@example.com To: foo@example.com Subject: 3 Date: Mon, 07 Dec 2020 19:05:49 -0500 Message-ID: <87r1ozlffh.fsf@example.com> MIME-Version: 1.0 Content-Type: text/plain X-RMAIL-ATTRIBUTES: -------- foo. >From foo@example.com Mon Dec 7 19:05:49 2020 Return-Path: From: foo@example.com To: foo@example.com Subject: 4 Date: Mon, 07 Dec 2020 19:05:49 -0500 Message-ID: <87r1ozlffh.fsf@example.com> MIME-Version: 1.0 Content-Type: text/plain X-RMAIL-ATTRIBUTES: ------U- foo. >From foo@example.com Mon Dec 7 19:05:49 2020 Return-Path: From: foo@example.com To: foo@example.com Subject: 5 Date: Mon, 07 Dec 2020 19:05:49 -0500 Message-ID: <87r1ozlffh.fsf@example.com> MIME-Version: 1.0 Content-Type: text/plain X-RMAIL-ATTRIBUTES: ------U- foo. >From foo@example.com Mon Dec 7 19:05:49 2020 Return-Path: From: foo@example.com To: foo@example.com Subject: 6 Date: Mon, 07 Dec 2020 19:05:49 -0500 Message-ID: <87r1ozlffh.fsf@example.com> MIME-Version: 1.0 Content-Type: text/plain X-RMAIL-ATTRIBUTES: ------U- foo. [END] Create a second file (let's call it test-spam) with these contents: [BEGIN] >From spam@example.com Mon Dec 7 19:05:49 2020 Return-Path: From: spam@example.com To: spam@example.com Subject: spammy Date: Mon, 07 Dec 2020 19:05:49 -0500 Message-ID: <87r1ozlffh.fsf@example.com> MIME-Version: 1.0 Content-Type: text/plain X-RMAIL-ATTRIBUTES: ------U- spam. [END] To reproduce in the unpatched code, make a second copy of "test-rmail" (it'll get destructively modified, and you want to be able to put it back), ensure that "spam-output" does not exist, and: (require 'rmail-spam-filter) (setq rsf-file "spam-output") (setq rsf-definitions-alist '( ((from . "spam@example.com") (action . output-and-delete)) )) Type: M-X rmail-input test-rmail If you type h, note that only message #1 is marked seen. Type: C-U g test-spam You'll be prompted to create spam-output; answer yes. Note that the message whose subject is "2" is now in spam-output (and potentially also deleted from rmail-test), and -not- the spam! Note also that you're magically on message #4, whereas typing g should have only incremented the current message by one, so you should be on message #2. To verify the patch: delete spam-output, regenerate test-rmail from its backup, kill the test-rmail buffer, load the patched code, and try the above steps again. None of the bugs should happen any more---the current message stays current (except for advancing by 1 only if g was used), and the spam message---and not the first unread message---winds up in spam-output and is deleted (and the deletion shows up in the summary buffer as a D, instead of being stealthed via just its label). From debbugs-submit-bounces@debbugs.gnu.org Tue Dec 08 18:50:24 2020 Received: (at control) by debbugs.gnu.org; 8 Dec 2020 23:50:24 +0000 Received: from localhost ([127.0.0.1]:60234 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kmmkS-0003K1-9R for submit@debbugs.gnu.org; Tue, 08 Dec 2020 18:50:24 -0500 Received: from quimby.gnus.org ([95.216.78.240]:58020) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kmmkQ-0003Ji-8c for control@debbugs.gnu.org; Tue, 08 Dec 2020 18:50:23 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnus.org; s=20200322; h=Subject:From:To:Message-Id:Date:Sender:Reply-To:Cc: MIME-Version:Content-Type:Content-Transfer-Encoding:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:In-Reply-To:References:List-Id:List-Help:List-Unsubscribe: List-Subscribe:List-Post:List-Owner:List-Archive; bh=OPl7fgjcZz4GH1KlPTDA1wFk/MZvAJtEgZwf0cAfcag=; b=nC5/vdq3zW5/pfpxVNKjL4XR4y TzqjpbkrGe3owGMCiBu9GVGD1WhJeEKycqOt3VeyNeCUwFzqPK2zrtrF2X/tMpwifS4ihiu574gUT Jqv0wG/wdDF6YMz62OA5Q7tEYUyZHabLavrZ5GEZIU4dsx9jwz5eH1uQJMUtPJnEtDUg=; Received: from cm-84.212.202.86.getinternet.no ([84.212.202.86] helo=xo) by quimby.gnus.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1kmmkH-0001t2-R8 for control@debbugs.gnu.org; Wed, 09 Dec 2020 00:50:15 +0100 Date: Wed, 09 Dec 2020 00:50:12 +0100 Message-Id: <87pn3j7t8r.fsf@gnus.org> To: control@debbugs.gnu.org From: Lars Ingebrigtsen Subject: control message for bug #45129 X-Spam-Report: Spam detection software, running on the system "quimby.gnus.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see @@CONTACT_ADDRESS@@ for details. Content preview: forcemerge 45129 45128 quit Content analysis details: (-2.9 points, 5.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -1.0 ALL_TRUSTED Passed through trusted hosts only via SMTP -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1% [score: 0.0000] X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: control X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) forcemerge 45129 45128 quit From debbugs-submit-bounces@debbugs.gnu.org Sun May 16 11:47:50 2021 Received: (at 45128) by debbugs.gnu.org; 16 May 2021 15:47:50 +0000 Received: from localhost ([127.0.0.1]:51341 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1liIzd-0000m8-TD for submit@debbugs.gnu.org; Sun, 16 May 2021 11:47:50 -0400 Received: from quimby.gnus.org ([95.216.78.240]:48150) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1liIzb-0000lm-QQ; Sun, 16 May 2021 11:47:48 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnus.org; s=20200322; h=Content-Type:MIME-Version:Message-ID:In-Reply-To:Date: References:Subject:Cc:To:From:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender: Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe: List-Subscribe:List-Post:List-Owner:List-Archive; bh=YtKjB2F2f0PYCysCsI0kE7AtXKMafvS/sAr0zTrjp5w=; b=XevltkYq+c8FE1KTMAU/Glb2kx z1Of6HGzHxF+0CEQ8FYr+7ZB0f1cfpFJhC+2C2i3Sts08AR8GdeFVF8m+pYAWBBC/DUwi6OcSMfCg im+QZr7z1bzzVhtSHFZLaC7IPPmm5hNvumavcG73GIqiCLrGgmFvGtathQrNQPWpQhuk=; Received: from cm-84.212.220.105.getinternet.no ([84.212.220.105] helo=xo) by quimby.gnus.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1liIzQ-00006V-Mz; Sun, 16 May 2021 17:47:41 +0200 From: Lars Ingebrigtsen To: emacs-f@media.mit.edu Subject: Re: bug#45128: [PATCH] rmail-spam-filter can lose mail References: <875z5bn9sk.fsf@media.mit.edu> X-Now-Playing: Doubting Thomas's _Father Don't Cry_: "Turn A New Leaf" Date: Sun, 16 May 2021 17:47:36 +0200 In-Reply-To: <875z5bn9sk.fsf@media.mit.edu> (emacs-f@media.mit.edu's message of "Tue, 08 Dec 2020 18:43:39 -0500") Message-ID: <877djyn0iv.fsf@gnus.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.0.50 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Report: Spam detection software, running on the system "quimby.gnus.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see @@CONTACT_ADDRESS@@ for details. Content preview: It looks like this patch was applied in: commit f9264a3878ba6366aacdf80e3ebba3ba799415a6 Author: emacs-f AuthorDate: Tue Dec 8 18:22:30 2020 -0500 Commit: Eli Zaretskii CommitDate: Sat Dec 26 11:46:11 2 [...] Content analysis details: (-2.9 points, 5.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -1.0 ALL_TRUSTED Passed through trusted hosts only via SMTP -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1% [score: 0.0000] X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 45128 Cc: 45129@debbugs.gnu.org, 45128@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) It looks like this patch was applied in: commit f9264a3878ba6366aacdf80e3ebba3ba799415a6 Author: emacs-f AuthorDate: Tue Dec 8 18:22:30 2020 -0500 Commit: Eli Zaretskii CommitDate: Sat Dec 26 11:46:11 2020 +0200 But this bug report was left open, so I'm closing it now. -- (domestic pets only, the antidote for overdose, milk.) bloggy blog: http://lars.ingebrigtsen.no From debbugs-submit-bounces@debbugs.gnu.org Sun May 16 11:47:54 2021 Received: (at control) by debbugs.gnu.org; 16 May 2021 15:47:54 +0000 Received: from localhost ([127.0.0.1]:51346 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1liIzi-0000mV-By for submit@debbugs.gnu.org; Sun, 16 May 2021 11:47:54 -0400 Received: from quimby.gnus.org ([95.216.78.240]:48164) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1liIzg-0000lt-4Y for control@debbugs.gnu.org; Sun, 16 May 2021 11:47:52 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnus.org; s=20200322; h=Subject:From:To:Message-Id:Date:Sender:Reply-To:Cc: MIME-Version:Content-Type:Content-Transfer-Encoding:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:In-Reply-To:References:List-Id:List-Help:List-Unsubscribe: List-Subscribe:List-Post:List-Owner:List-Archive; bh=e9hz0gLuVFDGi2Kef0TIwXhTSdcG38AWlCdSKD4f7hI=; b=XttewTRoR4zk+HFPQPRgklj7pY Vpd3ts8dlbL8FTeuusp6K2+Dqa4yhsZ+gmKZR/1mEgxz0NNOYXDn73UEcURWzf/wikPRcw4gd20nh f4PXcvmRjnWi0D55gw2ekBE/5hJQdiavT4eFUMWjxTTuZZIrq1TvoKUhY1SeIsxn2KdQ=; Received: from cm-84.212.220.105.getinternet.no ([84.212.220.105] helo=xo) by quimby.gnus.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1liIzY-00006f-Km for control@debbugs.gnu.org; Sun, 16 May 2021 17:47:46 +0200 Date: Sun, 16 May 2021 17:47:44 +0200 Message-Id: <875yzin0in.fsf@gnus.org> To: control@debbugs.gnu.org From: Lars Ingebrigtsen Subject: control message for bug #45128 X-Spam-Report: Spam detection software, running on the system "quimby.gnus.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see @@CONTACT_ADDRESS@@ for details. Content preview: tags 45128 fixed close 45128 28.1 quit Content analysis details: (-2.9 points, 5.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -1.0 ALL_TRUSTED Passed through trusted hosts only via SMTP -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1% [score: 0.0000] X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: control X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) tags 45128 fixed close 45128 28.1 quit From unknown Thu Aug 21 12:11:43 2025 Received: (at fakecontrol) by fakecontrolmessage; To: internal_control@debbugs.gnu.org From: Debbugs Internal Request Subject: Internal Control Message-Id: bug archived. Date: Mon, 14 Jun 2021 11:24:04 +0000 User-Agent: Fakemail v42.6.9 # This is a fake control message. # # The action: # bug archived. thanks # This fakemail brought to you by your local debbugs # administrator