GNU bug report logs - #78519
30.0.91; In Gnus, when yanking an article, all its newlines become soft

Previous Next

To reply to this bug, email your comments to 78519 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Mekeor Melire <mekeor <at> posteo.de>
To: bug-gnu-emacs <at> gnu.org
Subject: 30.0.91; In Gnus, when yanking an article, all its newlines become
 soft
Date: Tue, 20 May 2025 22:50:21 +0000

[Message part 1 (text/plain, inline)]

GNU Emacs and Gnus support RFC 3676 (obsoleting RFC 2646) “The
Text/Plain Format and DelSp Parameters” as described in (info
"(emacs-mime) Flowed text"). But when replying to an article, or
more specifically, when yanking an article (no matter whether it's
Fixed or Flowed) while composing a message, all yanked newlines
become soft newlines, or more specifically, no newline has the
text-property `hard' set to t. As a consequence, when the composed
message is sent with Format=Flowed, it will be encoded wrongly and
its readers will see it terribly displayed.


How to arrive at this misbehavior?
==================================

1. Open an article of content-type text/plain with Gnus. Gnus
decodes and inlines the article. If the article has the Format
MIME-parameter set to Flowed, the function `mm-inline-text'
inlines the article and refills it to the column specified per
`fill-flowed-display-column' variable by calling
`fill-flowed'. `fill-flowed' itself can tell which newlines are
soft and which are hard because they are respectively present as
either as the sequence SPC CRLF, or just CRLF without preceding
SPC. But after `fill-flowed' is called, both types of newlines are
just CRLF and even without any text-property.

2. Start composing a reply to the article, e.g. by pressing R,
bound `gnus-article-reply-with-original' in
`gnus-article-mode-map'. Gnus will set up a compose buffer and
yank the original article with citation prefix. Yanking happens by
first calling `gnus-copy-article-buffer', which copies the
original article, from the buffer where it is displayed and read,
to the new composing message buffer while removing all
text-properties. (Then, yanking will continue by adding the
citation prefixes.) The resulting buffer has only soft newlines.

A. Note that this behavior is only a bug if the composed reply has
Format parameter set to Flowed, which Emacs decides to do if at
least once in the buffer, the text-property `hard' is set to
t. See “On encoding text, regardless of ‘use-hard-newlines’, lines
terminated by soft newline characters are filled together and
wrapped” at (info "(emacs-mime) Flowed text") and the form (setq
use-hard-newlines (text-property-any (point-min) (point-max) 'hard
't)) in definition of function `mml-generate-mime-1' [2].


What would be the correct behavior?
===================================

X. In my opinion, we should retain the property that it makes no
difference in effect between first turning on `use-hard-newlines'
and then yanking, or the other way around.

Y. In my opinion, we should continue to use the `hard'
text-property on hard newlines in the composing buffer, because
most filling related commands know about it.

Z. In my opinion, when yanking a Fixed formatted article, all
yanked newlines should become hard or rather be interpreted as
hard, i.e. they should get the `hard' text-property set to t. This
is because it's hard to tell if we can interpret a newline as soft
newline, or if it's really meant to be hard, e.g. because it's a
poem. [3]

Thus, I think that yanking hard newlines from Format=Flowed
articles should have `hard' set to t in the compose
buffer. Similarly, any newline yanked from a Fixed [4] article
should have `hard' set to t as well. And the detection described
at (A.), whether Format parameter should be set to Fixed for the
composing message, should be changed, so that it only checks
whether `use-hard-newlines' is non-nil rather than looking for any
`hard' property set to t, because this would conflict with (X.).


How can we approach the proposed correct behavior?
==================================================

α. Let's fix `fill-flowed' so that it sets `hard' to t on hard
newlines after refilling. Please find attached a patch for this.

β. Let's fix `gnus-copy-article-buffer' so that it keeps `hard'
text-properties on newlines. Please find attached a patch for
this. I'm not sure if it suffices to fix this function, because we
want the `hard' propertized hard newlines in many cases:

  - Content-Type text/plain, without any Format specification; or
    with Format specification set to Fixed or Flowed
  - Content-Type multipart/* where for each text/plain part we
    want proper hard newlines.

γ. Let's fix (Z) but I'm yet sure how.


Footnotes
=========

[2] Oops, in my init.el, I have configured
    `message-citation-line-format' to contain a `hard-newline. I
    guess, all my sent reply-messages have Format=Flowed?

[3] There might be approximating heuristics for this. But what'd
    be more important is to provide users commands so that they
    can convert hard and soft newlines within a region or
    paragraph back and forth, as desired. But that's future work.

[4] Be it Fixed by explicitly setting Format=Fixed, or by omitting
    the Format parameter completely.

[fill-flowed.diff (text/x-patch, attachment)]

[gnus-copy-article-buffer.diff (text/x-patch, attachment)]

Message #8 received at 78519 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Mekeor Melire <mekeor <at> posteo.de>, Eric Abrahamsen <eric <at> ericabrahamsen.net>
Cc: 78519 <at> debbugs.gnu.org
Subject: Re: bug#78519: 30.0.91;
 In Gnus, when yanking an article, all its newlines become soft
Date: Sat, 24 May 2025 12:37:36 +0300

> From: Mekeor Melire <mekeor <at> posteo.de>
> Date: Tue, 20 May 2025 22:50:21 +0000
> 
> GNU Emacs and Gnus support RFC 3676 (obsoleting RFC 2646) “The
> Text/Plain Format and DelSp Parameters” as described in (info
> "(emacs-mime) Flowed text"). But when replying to an article, or
> more specifically, when yanking an article (no matter whether it's
> Fixed or Flowed) while composing a message, all yanked newlines
> become soft newlines, or more specifically, no newline has the
> text-property `hard' set to t. As a consequence, when the composed
> message is sent with Format=Flowed, it will be encoded wrongly and
> its readers will see it terribly displayed.
> 
> 
> How to arrive at this misbehavior?
> ==================================
> 
> 1. Open an article of content-type text/plain with Gnus. Gnus
> decodes and inlines the article. If the article has the Format
> MIME-parameter set to Flowed, the function `mm-inline-text'
> inlines the article and refills it to the column specified per
> `fill-flowed-display-column' variable by calling
> `fill-flowed'. `fill-flowed' itself can tell which newlines are
> soft and which are hard because they are respectively present as
> either as the sequence SPC CRLF, or just CRLF without preceding
> SPC. But after `fill-flowed' is called, both types of newlines are
> just CRLF and even without any text-property.
> 
> 2. Start composing a reply to the article, e.g. by pressing R,
> bound `gnus-article-reply-with-original' in
> `gnus-article-mode-map'. Gnus will set up a compose buffer and
> yank the original article with citation prefix. Yanking happens by
> first calling `gnus-copy-article-buffer', which copies the
> original article, from the buffer where it is displayed and read,
> to the new composing message buffer while removing all
> text-properties. (Then, yanking will continue by adding the
> citation prefixes.) The resulting buffer has only soft newlines.
> 
> A. Note that this behavior is only a bug if the composed reply has
> Format parameter set to Flowed, which Emacs decides to do if at
> least once in the buffer, the text-property `hard' is set to
> t. See “On encoding text, regardless of ‘use-hard-newlines’, lines
> terminated by soft newline characters are filled together and
> wrapped” at (info "(emacs-mime) Flowed text") and the form (setq
> use-hard-newlines (text-property-any (point-min) (point-max) 'hard
> 't)) in definition of function `mml-generate-mime-1' [2].
> 
> 
> What would be the correct behavior?
> ===================================
> 
> X. In my opinion, we should retain the property that it makes no
> difference in effect between first turning on `use-hard-newlines'
> and then yanking, or the other way around.
> 
> Y. In my opinion, we should continue to use the `hard'
> text-property on hard newlines in the composing buffer, because
> most filling related commands know about it.
> 
> Z. In my opinion, when yanking a Fixed formatted article, all
> yanked newlines should become hard or rather be interpreted as
> hard, i.e. they should get the `hard' text-property set to t. This
> is because it's hard to tell if we can interpret a newline as soft
> newline, or if it's really meant to be hard, e.g. because it's a
> poem. [3]
> 
> Thus, I think that yanking hard newlines from Format=Flowed
> articles should have `hard' set to t in the compose
> buffer. Similarly, any newline yanked from a Fixed [4] article
> should have `hard' set to t as well. And the detection described
> at (A.), whether Format parameter should be set to Fixed for the
> composing message, should be changed, so that it only checks
> whether `use-hard-newlines' is non-nil rather than looking for any
> `hard' property set to t, because this would conflict with (X.).
> 
> 
> How can we approach the proposed correct behavior?
> ==================================================
> 
> α. Let's fix `fill-flowed' so that it sets `hard' to t on hard
> newlines after refilling. Please find attached a patch for this.
> 
> β. Let's fix `gnus-copy-article-buffer' so that it keeps `hard'
> text-properties on newlines. Please find attached a patch for
> this. I'm not sure if it suffices to fix this function, because we
> want the `hard' propertized hard newlines in many cases:
> 
>   - Content-Type text/plain, without any Format specification; or
>     with Format specification set to Fixed or Flowed
>   - Content-Type multipart/* where for each text/plain part we
>     want proper hard newlines.
> 
> γ. Let's fix (Z) but I'm yet sure how.
> 
> 
> Footnotes
> =========
> 
> [2] Oops, in my init.el, I have configured
>     `message-citation-line-format' to contain a `hard-newline. I
>     guess, all my sent reply-messages have Format=Flowed?
> 
> [3] There might be approximating heuristics for this. But what'd
>     be more important is to provide users commands so that they
>     can convert hard and soft newlines within a region or
>     paragraph back and forth, as desired. But that's future work.
> 
> [4] Be it Fixed by explicitly setting Format=Fixed, or by omitting
>     the Format parameter completely.

Eric, any comments?

Message #11 received at 78519 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: eric <at> ericabrahamsen.net
Cc: mekeor <at> posteo.de, 78519 <at> debbugs.gnu.org
Subject: Re: bug#78519: 30.0.91;
 In Gnus, when yanking an article, all its newlines become soft
Date: Sat, 07 Jun 2025 11:21:34 +0300

Ping!  Eric, could you please chime in?

> Cc: 78519 <at> debbugs.gnu.org
> Date: Sat, 24 May 2025 12:37:36 +0300
> From: Eli Zaretskii <eliz <at> gnu.org>
> 
> > From: Mekeor Melire <mekeor <at> posteo.de>
> > Date: Tue, 20 May 2025 22:50:21 +0000
> > 
> > GNU Emacs and Gnus support RFC 3676 (obsoleting RFC 2646) “The
> > Text/Plain Format and DelSp Parameters” as described in (info
> > "(emacs-mime) Flowed text"). But when replying to an article, or
> > more specifically, when yanking an article (no matter whether it's
> > Fixed or Flowed) while composing a message, all yanked newlines
> > become soft newlines, or more specifically, no newline has the
> > text-property `hard' set to t. As a consequence, when the composed
> > message is sent with Format=Flowed, it will be encoded wrongly and
> > its readers will see it terribly displayed.
> > 
> > 
> > How to arrive at this misbehavior?
> > ==================================
> > 
> > 1. Open an article of content-type text/plain with Gnus. Gnus
> > decodes and inlines the article. If the article has the Format
> > MIME-parameter set to Flowed, the function `mm-inline-text'
> > inlines the article and refills it to the column specified per
> > `fill-flowed-display-column' variable by calling
> > `fill-flowed'. `fill-flowed' itself can tell which newlines are
> > soft and which are hard because they are respectively present as
> > either as the sequence SPC CRLF, or just CRLF without preceding
> > SPC. But after `fill-flowed' is called, both types of newlines are
> > just CRLF and even without any text-property.
> > 
> > 2. Start composing a reply to the article, e.g. by pressing R,
> > bound `gnus-article-reply-with-original' in
> > `gnus-article-mode-map'. Gnus will set up a compose buffer and
> > yank the original article with citation prefix. Yanking happens by
> > first calling `gnus-copy-article-buffer', which copies the
> > original article, from the buffer where it is displayed and read,
> > to the new composing message buffer while removing all
> > text-properties. (Then, yanking will continue by adding the
> > citation prefixes.) The resulting buffer has only soft newlines.
> > 
> > A. Note that this behavior is only a bug if the composed reply has
> > Format parameter set to Flowed, which Emacs decides to do if at
> > least once in the buffer, the text-property `hard' is set to
> > t. See “On encoding text, regardless of ‘use-hard-newlines’, lines
> > terminated by soft newline characters are filled together and
> > wrapped” at (info "(emacs-mime) Flowed text") and the form (setq
> > use-hard-newlines (text-property-any (point-min) (point-max) 'hard
> > 't)) in definition of function `mml-generate-mime-1' [2].
> > 
> > 
> > What would be the correct behavior?
> > ===================================
> > 
> > X. In my opinion, we should retain the property that it makes no
> > difference in effect between first turning on `use-hard-newlines'
> > and then yanking, or the other way around.
> > 
> > Y. In my opinion, we should continue to use the `hard'
> > text-property on hard newlines in the composing buffer, because
> > most filling related commands know about it.
> > 
> > Z. In my opinion, when yanking a Fixed formatted article, all
> > yanked newlines should become hard or rather be interpreted as
> > hard, i.e. they should get the `hard' text-property set to t. This
> > is because it's hard to tell if we can interpret a newline as soft
> > newline, or if it's really meant to be hard, e.g. because it's a
> > poem. [3]
> > 
> > Thus, I think that yanking hard newlines from Format=Flowed
> > articles should have `hard' set to t in the compose
> > buffer. Similarly, any newline yanked from a Fixed [4] article
> > should have `hard' set to t as well. And the detection described
> > at (A.), whether Format parameter should be set to Fixed for the
> > composing message, should be changed, so that it only checks
> > whether `use-hard-newlines' is non-nil rather than looking for any
> > `hard' property set to t, because this would conflict with (X.).
> > 
> > 
> > How can we approach the proposed correct behavior?
> > ==================================================
> > 
> > α. Let's fix `fill-flowed' so that it sets `hard' to t on hard
> > newlines after refilling. Please find attached a patch for this.
> > 
> > β. Let's fix `gnus-copy-article-buffer' so that it keeps `hard'
> > text-properties on newlines. Please find attached a patch for
> > this. I'm not sure if it suffices to fix this function, because we
> > want the `hard' propertized hard newlines in many cases:
> > 
> >   - Content-Type text/plain, without any Format specification; or
> >     with Format specification set to Fixed or Flowed
> >   - Content-Type multipart/* where for each text/plain part we
> >     want proper hard newlines.
> > 
> > γ. Let's fix (Z) but I'm yet sure how.
> > 
> > 
> > Footnotes
> > =========
> > 
> > [2] Oops, in my init.el, I have configured
> >     `message-citation-line-format' to contain a `hard-newline. I
> >     guess, all my sent reply-messages have Format=Flowed?
> > 
> > [3] There might be approximating heuristics for this. But what'd
> >     be more important is to provide users commands so that they
> >     can convert hard and soft newlines within a region or
> >     paragraph back and forth, as desired. But that's future work.
> > 
> > [4] Be it Fixed by explicitly setting Format=Fixed, or by omitting
> >     the Format parameter completely.
> 
> Eric, any comments?
> 
> 
> 
> 

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.