GNU bug report logs - #75628
31.0.50; Error sending mail using SMTP - base64 encoding fails

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 75628 in the body.
You can then email your comments to 75628 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Ingo Brunberg <ingo_brunberg <at> web.de>
To: bug-gnu-emacs <at> gnu.org
Subject: 31.0.50; Error sending mail using SMTP - base64 encoding fails
Date: Fri, 17 Jan 2025 10:18:52 +0000

When I try to send mail with emacs using SMTP, I get error "Multibyte character
in data for base64 encoding".

In the current code I have narrowed it down to line 637 in smtpmail.el, where an
attempt is made to base64 encode a string containing the password which, of
course, can contain multibyte characters.

With a simple password some time ago, I could send mail.

Message #8 received at 75628 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Ingo Brunberg <ingo_brunberg <at> web.de>
Cc: 75628 <at> debbugs.gnu.org
Subject: Re: bug#75628: 31.0.50;
 Error sending mail using SMTP - base64 encoding fails
Date: Fri, 17 Jan 2025 14:55:55 +0200

> Date: Fri, 17 Jan 2025 10:18:52 +0000
> From:  Ingo Brunberg via "Bug reports for GNU Emacs,
>  the Swiss army knife of text editors" <bug-gnu-emacs <at> gnu.org>
> 
> When I try to send mail with emacs using SMTP, I get error "Multibyte character
> in data for base64 encoding".
> 
> In the current code I have narrowed it down to line 637 in smtpmail.el, where an
> attempt is made to base64 encode a string containing the password which, of
> course, can contain multibyte characters.
> 
> With a simple password some time ago, I could send mail.

Thanks, but please show a complete recipe, preferably starting from
"emacs -Q".

base64 encoding indeed requires its input to be a unibyte string, but
it is impossible to guess how this failed to happen without seeing the
sequence of commands you typed to trigger the problem.

Message #11 received at 75628 <at> debbugs.gnu.org (full text, mbox):

From: Robert Pluim <rpluim <at> gmail.com>
To: 75628 <at> debbugs.gnu.org
Cc: Ingo Brunberg <ingo_brunberg <at> web.de>
Subject: Re: bug#75628: 31.0.50; Error sending mail using SMTP - base64
 encoding fails
Date: Fri, 17 Jan 2025 14:15:26 +0100

>>>>> On Fri, 17 Jan 2025 10:18:52 +0000, Ingo Brunberg via "Bug reports for GNU Emacs, the Swiss army knife of text editors" <bug-gnu-emacs <at> gnu.org> said:

    Ingo> When I try to send mail with emacs using SMTP, I get error "Multibyte character
    Ingo> in data for base64 encoding".

    Ingo> In the current code I have narrowed it down to line 637 in smtpmail.el, where an
    Ingo> attempt is made to base64 encode a string containing the password which, of
    Ingo> course, can contain multibyte characters.

    Ingo> With a simple password some time ago, I could send mail.

Does this fix it? I guess at least "AUTH LOGIN" needs similar
treatment, and probably "AUTH CRAM-MD5". I donʼt think XOAUTH2 needs
it, but Iʼll check.

diff --git a/lisp/mail/smtpmail.el b/lisp/mail/smtpmail.el
index 6cb576fe72b..5aebbec14d0 100644
--- a/lisp/mail/smtpmail.el
+++ b/lisp/mail/smtpmail.el
@@ -631,11 +631,12 @@ smtpmail-try-auth-method
   ;; violate a SHOULD in RFC 2222 paragraph 5.1.  Note that this
   ;; is not sent if the server did not advertise AUTH PLAIN in
   ;; the EHLO response.  See RFC 2554 for more info.
+  (let ((password (encode-coding-string password 'utf-8)))
   (smtpmail-command-or-throw
    process
    (concat "AUTH PLAIN "
 	   (base64-encode-string (concat "\0" user "\0" password) t))
-   235))
+   235)))
 
 (cl-defmethod smtpmail-try-auth-method
   (process (_mech (eql 'xoauth2)) user password)






Robert
-- 

Message #14 received at 75628 <at> debbugs.gnu.org (full text, mbox):

From: Robert Pluim <rpluim <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: Ingo Brunberg <ingo_brunberg <at> web.de>, 75628 <at> debbugs.gnu.org
Subject: Re: bug#75628: 31.0.50; Error sending mail using SMTP - base64
 encoding fails
Date: Fri, 17 Jan 2025 14:26:53 +0100

>>>>> On Fri, 17 Jan 2025 14:55:55 +0200, Eli Zaretskii <eliz <at> gnu.org> said:

    >> Date: Fri, 17 Jan 2025 10:18:52 +0000
    >> From:  Ingo Brunberg via "Bug reports for GNU Emacs,
    >> the Swiss army knife of text editors" <bug-gnu-emacs <at> gnu.org>
    >> 
    >> When I try to send mail with emacs using SMTP, I get error "Multibyte character
    >> in data for base64 encoding".
    >> 
    >> In the current code I have narrowed it down to line 637 in smtpmail.el, where an
    >> attempt is made to base64 encode a string containing the password which, of
    >> course, can contain multibyte characters.
    >> 
    >> With a simple password some time ago, I could send mail.

    Eli> Thanks, but please show a complete recipe, preferably starting from
    Eli> "emacs -Q".

    Eli> base64 encoding indeed requires its input to be a unibyte string, but
    Eli> it is impossible to guess how this failed to happen without seeing the
    Eli> sequence of commands you typed to trigger the problem.

If the password is coming from 'auth-source', it will be a multibyte
string:

ELISP> (auth-info-password (car (auth-source-search :host "test")))
"bär"

ELISP> (multibyte-string-p (auth-info-password (car (auth-source-search :host "test"))))
t

Robert
-- 

Message #17 received at 75628 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Robert Pluim <rpluim <at> gmail.com>
Cc: ingo_brunberg <at> web.de, 75628 <at> debbugs.gnu.org
Subject: Re: bug#75628: 31.0.50;
 Error sending mail using SMTP - base64 encoding fails
Date: Fri, 17 Jan 2025 15:36:59 +0200

> Cc: Ingo Brunberg <ingo_brunberg <at> web.de>
> From: Robert Pluim <rpluim <at> gmail.com>
> Date: Fri, 17 Jan 2025 14:15:26 +0100
> 
>     Ingo> With a simple password some time ago, I could send mail.
> 
> Does this fix it? I guess at least "AUTH LOGIN" needs similar
> treatment, and probably "AUTH CRAM-MD5". I donʼt think XOAUTH2 needs
> it, but Iʼll check.

If this is the fix, then how do you explain what the OP says, that
"some time ago" he could do this without problems?  Do you assume that
the "simple password" was plain-ASCII or something?

The OP didn't show the data collected by report-emacs-bug, so we don't
know the locale's encoding in his case.  That could perhaps explain
the difference in behavior.

Also, smtpmail-command-or-throw calls process-send-string, which
should already encode the command.  So the call to
encode-coding-string is almost certainly incorrect; if anything, we
should bind coding-system-for-write to 'utf-8'.  (Btw, how do we know
UTF-8 is the right encoding in this case? is that in some relevant
RFC?)

Thanks.

Message #20 received at 75628 <at> debbugs.gnu.org (full text, mbox):

From: Robert Pluim <rpluim <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: ingo_brunberg <at> web.de, 75628 <at> debbugs.gnu.org
Subject: Re: bug#75628: 31.0.50; Error sending mail using SMTP - base64
 encoding fails
Date: Fri, 17 Jan 2025 14:59:59 +0100

>>>>> On Fri, 17 Jan 2025 15:36:59 +0200, Eli Zaretskii <eliz <at> gnu.org> said:

    >> Cc: Ingo Brunberg <ingo_brunberg <at> web.de>
    >> From: Robert Pluim <rpluim <at> gmail.com>
    >> Date: Fri, 17 Jan 2025 14:15:26 +0100
    >> 
    Ingo> With a simple password some time ago, I could send mail.
    >> 
    >> Does this fix it? I guess at least "AUTH LOGIN" needs similar
    >> treatment, and probably "AUTH CRAM-MD5". I donʼt think XOAUTH2 needs
    >> it, but Iʼll check.

    Eli> If this is the fix, then how do you explain what the OP says, that
    Eli> "some time ago" he could do this without problems?  Do you assume that
    Eli> the "simple password" was plain-ASCII or something?

Yes

    Eli> The OP didn't show the data collected by report-emacs-bug, so we don't
    Eli> know the locale's encoding in his case.  That could perhaps explain
    Eli> the difference in behavior.

    Eli> Also, smtpmail-command-or-throw calls process-send-string, which
    Eli> should already encode the command.  So the call to
    Eli> encode-coding-string is almost certainly incorrect; if anything, we
    Eli> should bind coding-system-for-write to 'utf-8'.  (Btw, how do we know
    Eli> UTF-8 is the right encoding in this case? is that in some relevant
    Eli> RFC?)

The encoding done by process-send-string is done after the base64
encoding of the password used by the "AUTH PLAIN" command, so it
doesnʼt help.

RFC 4616 specifies that the arguments to "AUTH PLAIN" are in UTF-8
(prior to encoding with base64, see RFC 4954), so we should be
encoding 'user' as well, strictly speaking.

Robert
-- 

Message #23 received at 75628 <at> debbugs.gnu.org (full text, mbox):

From: Ingo Brunberg <ingo_brunberg <at> web.de>
To: Robert Pluim <rpluim <at> gmail.com>
Cc: 75628 <at> debbugs.gnu.org
Subject: Re: bug#75628: 31.0.50; Error sending mail using SMTP - base64
 encoding fails
Date: Fri, 17 Jan 2025 16:51:15 +0100

Hello Robert,

yes, your patch does indeed fix the issue. I also guess there are more
places which need that treatment. Glad, it works for me again.

Thanks
Ingo

Robert Pluim <rpluim <at> gmail.com> writes:

>>>>>> On Fri, 17 Jan 2025 10:18:52 +0000, Ingo Brunberg via "Bug reports for GNU Emacs, the Swiss army knife of text editors" <bug-gnu-emacs <at> gnu.org> said:
>
>     Ingo> When I try to send mail with emacs using SMTP, I get error "Multibyte character
>     Ingo> in data for base64 encoding".
>
>     Ingo> In the current code I have narrowed it down to line 637 in smtpmail.el, where an
>     Ingo> attempt is made to base64 encode a string containing the password which, of
>     Ingo> course, can contain multibyte characters.
>
>     Ingo> With a simple password some time ago, I could send mail.
>
> Does this fix it? I guess at least "AUTH LOGIN" needs similar
> treatment, and probably "AUTH CRAM-MD5". I donʼt think XOAUTH2 needs
> it, but Iʼll check.
>
> diff --git a/lisp/mail/smtpmail.el b/lisp/mail/smtpmail.el
> index 6cb576fe72b..5aebbec14d0 100644
> --- a/lisp/mail/smtpmail.el
> +++ b/lisp/mail/smtpmail.el
> @@ -631,11 +631,12 @@ smtpmail-try-auth-method
>    ;; violate a SHOULD in RFC 2222 paragraph 5.1.  Note that this
>    ;; is not sent if the server did not advertise AUTH PLAIN in
>    ;; the EHLO response.  See RFC 2554 for more info.
> +  (let ((password (encode-coding-string password 'utf-8)))
>    (smtpmail-command-or-throw
>     process
>     (concat "AUTH PLAIN "
>  	   (base64-encode-string (concat "\0" user "\0" password) t))
> -   235))
> +   235)))
>  
>  (cl-defmethod smtpmail-try-auth-method
>    (process (_mech (eql 'xoauth2)) user password)
>
>
>
>
>
>
> Robert

Message #26 received at 75628 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Robert Pluim <rpluim <at> gmail.com>
Cc: ingo_brunberg <at> web.de, 75628 <at> debbugs.gnu.org
Subject: Re: bug#75628: 31.0.50; Error sending mail using SMTP - base64
 encoding fails
Date: Fri, 17 Jan 2025 18:43:57 +0200

> From: Robert Pluim <rpluim <at> gmail.com>
> Cc: 75628 <at> debbugs.gnu.org,  ingo_brunberg <at> web.de
> Date: Fri, 17 Jan 2025 14:59:59 +0100
> 
> >>>>> On Fri, 17 Jan 2025 15:36:59 +0200, Eli Zaretskii <eliz <at> gnu.org> said:
> 
>     >> Cc: Ingo Brunberg <ingo_brunberg <at> web.de>
>     >> From: Robert Pluim <rpluim <at> gmail.com>
>     >> Date: Fri, 17 Jan 2025 14:15:26 +0100
>     >> 
>     Ingo> With a simple password some time ago, I could send mail.
>     >> 
>     >> Does this fix it? I guess at least "AUTH LOGIN" needs similar
>     >> treatment, and probably "AUTH CRAM-MD5". I donʼt think XOAUTH2 needs
>     >> it, but Iʼll check.
> 
>     Eli> If this is the fix, then how do you explain what the OP says, that
>     Eli> "some time ago" he could do this without problems?  Do you assume that
>     Eli> the "simple password" was plain-ASCII or something?
> 
> Yes
> 
>     Eli> The OP didn't show the data collected by report-emacs-bug, so we don't
>     Eli> know the locale's encoding in his case.  That could perhaps explain
>     Eli> the difference in behavior.
> 
>     Eli> Also, smtpmail-command-or-throw calls process-send-string, which
>     Eli> should already encode the command.  So the call to
>     Eli> encode-coding-string is almost certainly incorrect; if anything, we
>     Eli> should bind coding-system-for-write to 'utf-8'.  (Btw, how do we know
>     Eli> UTF-8 is the right encoding in this case? is that in some relevant
>     Eli> RFC?)
> 
> The encoding done by process-send-string is done after the base64
> encoding of the password used by the "AUTH PLAIN" command, so it
> doesnʼt help.
> 
> RFC 4616 specifies that the arguments to "AUTH PLAIN" are in UTF-8
> (prior to encoding with base64, see RFC 4954), so we should be
> encoding 'user' as well, strictly speaking.

OK, but then I think we need to bind coding-system-for-write to
no-conversion when calling smtpmail-command-or-throw in this
case. because the string is already encoded.

Message #29 received at 75628 <at> debbugs.gnu.org (full text, mbox):

From: Robert Pluim <rpluim <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: ingo_brunberg <at> web.de, 75628 <at> debbugs.gnu.org
Subject: Re: bug#75628: 31.0.50; Error sending mail using SMTP - base64
 encoding fails
Date: Fri, 17 Jan 2025 18:04:12 +0100

>>>>> On Fri, 17 Jan 2025 18:43:57 +0200, Eli Zaretskii <eliz <at> gnu.org> said:

    Eli> OK, but then I think we need to bind coding-system-for-write to
    Eli> no-conversion when calling smtpmail-command-or-throw in this
    Eli> case. because the string is already encoded.

No need, the password is utf-8 encoded, then itʼs base64 encoded, then
itʼs concatenated with "AUTH PLAIN", and the result is then passed to
smtpmail-command-or-thow. So we can continue to use the value for
coding-system-for-write used by 'smtpmail-via-smtp', which is 'binary.

Robert
-- 

Message #32 received at 75628 <at> debbugs.gnu.org (full text, mbox):

From: Robert Pluim <rpluim <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: ingo_brunberg <at> web.de, 75628 <at> debbugs.gnu.org
Subject: Re: bug#75628: 31.0.50; Error sending mail using SMTP - base64
 encoding fails
Date: Thu, 13 Feb 2025 14:35:21 +0100

tags 75628 fixed
close 75628 31.1
quit

I forgot to push this one.

Closing.
Committed as 745847ba8ec

Robert
-- 

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.