Subject: 30.0.91; In Gnus, when yanking an article, all its newlines become soft

From: Mekeor Melire <[email-masked]>
Date: Tue, 20 May 2025 22:50:21 +0000

GNU Emacs and Gnus support RFC 3676 (obsoleting RFC 2646) “The
Text/Plain Format and DelSp Parameters” as described in (info
"(emacs-mime) Flowed text"). But when replying to an article, or
more specifically, when yanking an article (no matter whether it's
Fixed or Flowed) while composing a message, all yanked newlines
become soft newlines, or more specifically, no newline has the
text-property `hard' set to t. As a consequence, when the composed
message is sent with Format=Flowed, it will be encoded wrongly and
its readers will see it terribly displayed.

How to arrive at this misbehavior?
==================================

1. Open an article of content-type text/plain with Gnus. Gnus
decodes and inlines the article. If the article has the Format
MIME-parameter set to Flowed, the function `mm-inline-text'
inlines the article and refills it to the column specified per
`fill-flowed-display-column' variable by calling
`fill-flowed'. `fill-flowed' itself can tell which newlines are
soft and which are hard because they are respectively present as
either as the sequence SPC CRLF, or just CRLF without preceding
SPC. But after `fill-flowed' is called, both types of newlines are
just CRLF and even without any text-property.

2. Start composing a reply to the article, e.g. by pressing R,
bound `gnus-article-reply-with-original' in
`gnus-article-mode-map'. Gnus will set up a compose buffer and
yank the original article with citation prefix. Yanking happens by
first calling `gnus-copy-article-buffer', which copies the
original article, from the buffer where it is displayed and read,
to the new composing message buffer while removing all
text-properties. (Then, yanking will continue by adding the
citation prefixes.) The resulting buffer has only soft newlines.

A. Note that this behavior is only a bug if the composed reply has
Format parameter set to Flowed, which Emacs decides to do if at
least once in the buffer, the text-property `hard' is set to
t. See “On encoding text, regardless of ‘use-hard-newlines’, lines
terminated by soft newline characters are filled together and
wrapped” at (info "(emacs-mime) Flowed text") and the form (setq
use-hard-newlines (text-property-any (point-min) (point-max) 'hard
't)) in definition of function `mml-generate-mime-1' [2].

What would be the correct behavior?
===================================

X. In my opinion, we should retain the property that it makes no
difference in effect between first turning on `use-hard-newlines'
and then yanking, or the other way around.

Y. In my opinion, we should continue to use the `hard'
text-property on hard newlines in the composing buffer, because
most filling related commands know about it.

Z. In my opinion, when yanking a Fixed formatted article, all
yanked newlines should become hard or rather be interpreted as
hard, i.e. they should get the `hard' text-property set to t. This
is because it's hard to tell if we can interpret a newline as soft
newline, or if it's really meant to be hard, e.g. because it's a
poem. [3]

Thus, I think that yanking hard newlines from Format=Flowed
articles should have `hard' set to t in the compose
buffer. Similarly, any newline yanked from a Fixed [4] article
should have `hard' set to t as well. And the detection described
at (A.), whether Format parameter should be set to Fixed for the
composing message, should be changed, so that it only checks
whether `use-hard-newlines' is non-nil rather than looking for any
`hard' property set to t, because this would conflict with (X.).

How can we approach the proposed correct behavior?
==================================================

α. Let's fix `fill-flowed' so that it sets `hard' to t on hard
newlines after refilling. Please find attached a patch for this.

β. Let's fix `gnus-copy-article-buffer' so that it keeps `hard'
text-properties on newlines. Please find attached a patch for
this. I'm not sure if it suffices to fix this function, because we
want the `hard' propertized hard newlines in many cases:

  - Content-Type text/plain, without any Format specification; or
    with Format specification set to Fixed or Flowed
  - Content-Type multipart/* where for each text/plain part we
    want proper hard newlines.

γ. Let's fix (Z) but I'm yet sure how.

Footnotes
=========

[2] Oops, in my init.el, I have configured
    `message-citation-line-format' to contain a `hard-newline. I
    guess, all my sent reply-messages have Format=Flowed?

[3] There might be approximating heuristics for this. But what'd
    be more important is to provide users commands so that they
    can convert hard and soft newlines within a region or
    paragraph back and forth, as desired. But that's future work.

[4] Be it Fixed by explicitly setting Format=Fixed, or by omitting
    the Format parameter completely.

diff --git a/lisp/mail/flow-fill.el b/lisp/mail/flow-fill.el
index 683aa981864..2c22d245ee9 100644
--- a/lisp/mail/flow-fill.el
+++ b/lisp/mail/flow-fill.el
@@ -170,7 +170,8 @@ fill-flowed
 		  (fill-region (line-beginning-position)
                                (line-end-position)
 			       'left 'nosqueeze))))))
-        (forward-line 1)))))
+	(forward-line 1)
+        (put-text-property (1- (point)) (point) 'hard t)))))
 
 (make-obsolete-variable 'fill-flowed-encode-tests nil "27.1")
 (defvar fill-flowed-encode-tests)
diff --git a/lisp/gnus/gnus-msg.el b/lisp/gnus/gnus-msg.el
index 807c4e17a33..e4370b43eb3 100644
--- a/lisp/gnus/gnus-msg.el
+++ b/lisp/gnus/gnus-msg.el
@@ -851,10 +851,21 @@ gnus-copy-article-buffer
 	    (gnus-remove-text-with-property 'gnus-prev)
 	    (gnus-remove-text-with-property 'gnus-next)
 	    (gnus-remove-text-with-property 'gnus-decoration)
-	    (insert
-	     (prog1
-		 (buffer-substring-no-properties (point-min) (point-max))
-	       (erase-buffer)))
+            ;; Remove all text-properties except for non-nil `hard' on
+            ;; newlines:
+            (let ((pt (point-min)) (end (point-max)))
+	      (goto-char pt)
+	      (while (search-forward "\n" end 'move)
+		;; Remove all text properties before newline.
+		(set-text-properties pt (match-beginning 0) nil)
+		;; Replace newline with hard or soft newline.
+		(replace-match
+		 (if (get-text-property (match-beginning 0) 'hard)
+		     hard-newline "\n"))
+		;; Start next loop with point after newline.
+		(setq pt (goto-char (match-end 0))))
+	      ;; Remove all text properties after last newline.
+	      (set-text-properties pt (point) nil))
 	    ;; Find the original headers.
 	    (set-buffer gnus-original-article-buffer)
 	    (goto-char (point-min))