GNU bug report logs - #22081
24.5; netrc.el fails parsing authinfo items spread over multiple lines

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 22081 in the body.
You can then email your comments to 22081 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Vasilij Schneidermann <v.schneidermann <at> gmail.com>
To: bug-gnu-emacs <at> gnu.org
Subject: 24.5; netrc.el fails parsing authinfo items spread over multiple lines
Date: Thu, 3 Dec 2015 11:20:29 +0100

[Message part 1 (text/plain, inline)]

A .netrc or .authinfo file typically contains the credentials for a
specific machine on a single line, but according to netrc(5)
<http://linux.die.net/man/5/netrc> it's allowed to use newlines between
the tokens as well.  Using (netrc-parse) leaves such items out which is
clearly an error in the implementation.



In GNU Emacs 24.5.1 (x86_64-unknown-linux-gnu, GTK+ Version 3.16.6)
 of 2015-09-10 on foutrelis
Windowing system distributor `The X.Org Foundation', version 11.0.11704000
System Description: Arch Linux

Configured using:
 `configure --prefix=/usr --sysconfdir=/etc --libexecdir=/usr/lib
 --localstatedir=/var --with-x-toolkit=gtk3 --with-xft
 'CFLAGS=-march=x86-64 -mtune=generic -O2 -pipe -fstack-protector-strong
 --param=ssp-buffer-size=4' CPPFLAGS=-D_FORTIFY_SOURCE=2
 LDFLAGS=-Wl,-O1,--sort-common,--as-needed,-z,relro'

Important settings:
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Lisp Interaction

Minor modes in effect:
  tooltip-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent messages:
For information about GNU Emacs and the GNU system, type C-h C-a.

Load-path shadows:
None found.

Features:
(shadow sort gnus-util mail-extr emacsbug message format-spec rfc822 mml
easymenu mml-sec mm-decode mm-bodies mm-encode mail-parse rfc2231
mailabbrev gmm-utils mailheader sendmail rfc2047 rfc2045 ietf-drums
mm-util help-fns mail-prsvr mail-utils time-date tooltip electric
uniquify ediff-hook vc-hooks lisp-float-type mwheel x-win x-dnd tool-bar
dnd fontset image regexp-opt fringe tabulated-list newcomment lisp-mode
prog-mode register page menu-bar rfn-eshadow timer select scroll-bar
mouse jit-lock font-lock syntax facemenu font-core frame cham georgian
utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean
japanese hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese case-table epa-hook jka-cmpr-hook help simple abbrev
minibuffer nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote make-network-process
dbusbind gfilenotify dynamic-setting system-font-setting
font-render-setting move-toolbar gtk x-toolkit x multi-tty emacs)

Memory information:
((conses 16 70945 6811)
 (symbols 48 17553 0)
 (miscs 40 35 138)
 (strings 32 9020 4341)
 (string-bytes 1 247698)
 (vectors 16 8908)
 (vector-slots 8 383094 17840)
 (floats 8 63 74)
 (intervals 56 177 0)
 (buffers 960 11)
 (heap 1024 49544 969))

[Message part 2 (text/html, inline)]

Message #8 received at 22081 <at> debbugs.gnu.org (full text, mbox):

From: Ted Zlatanov <tzz <at> lifelogs.com>
To: Vasilij Schneidermann <v.schneidermann <at> gmail.com>,
 Lars Magne Ingebrigtsen <larsi <at> gnus.org>
Cc: 22081 <at> debbugs.gnu.org
Subject: Re: bug#22081: 24.5;
 netrc.el fails parsing authinfo items spread over multiple lines
Date: Thu, 03 Dec 2015 08:55:50 -0500

On Thu, 3 Dec 2015 11:20:29 +0100 Vasilij Schneidermann <v.schneidermann <at> gmail.com> wrote: 

VS> A .netrc or .authinfo file typically contains the credentials for a
VS> specific machine on a single line, but according to netrc(5)
VS> <http://linux.die.net/man/5/netrc> it's allowed to use newlines between
VS> the tokens as well.  Using (netrc-parse) leaves such items out which is
VS> clearly an error in the implementation.

The Emacs implementation in auth-source.el `auth-source-netrc-parse',
which is used by default for netrc files today, is not a
reimplementation of that semi-standard:

* there is no RFC for the netrc format as far as I know

* Emacs supports both single and double quotes around strings (note the
  man page doesn't indicate how tokens are quoted!)

* macdef and default are not explicitly supported

* arbitrary keywords are supported

* the "machine" keyword is not required for an entry

The last two, especially, make it hard to support multi-line entries.
Since there is no RFC to my knowledge and no functional need to spread
entries across multiple lines, I'd like to close this as a WONTFIX. Any
opinions?

(The netrc.el `netrc-parse' code is not actively used nowadays in Emacs,
as far as I know, but hasn't been removed due to my laziness. I see a
single reference to it in gnus/nntp.el. It has some deficiencies that
were improved in `auth-source-netrc-parse', for instance supporting
single-quoted strings. Any objections to removing it? CC to Lars since
he wrote it originally.)

Ted

Message #11 received at 22081 <at> debbugs.gnu.org (full text, mbox):

From: Glenn Morris <rgm <at> gnu.org>
To: Vasilij Schneidermann <v.schneidermann <at> gmail.com>
Cc: Lars Magne Ingebrigtsen <larsi <at> gnus.org>, 22081 <at> debbugs.gnu.org
Subject: Re: bug#22081: 24.5;
 netrc.el fails parsing authinfo items spread over multiple lines
Date: Thu, 03 Dec 2015 13:28:24 -0500

Ted Zlatanov wrote:

> (The netrc.el `netrc-parse' code is not actively used nowadays in Emacs,

Just to point out that there are even more functions for netrc parsing
in Emacs, as pointed out three years ago in http://debbugs.gnu.org/12511
(of which this report is a duplicate).
At the time it seemed to me that ange-ftp-parse-netrc looked
like it might handle newlines.

Message #16 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Andy Moreton <andrewjmoreton <at> gmail.com>
To: bug-gnu-emacs <at> gnu.org
Subject: Re: bug#22081: 24.5;
 netrc.el fails parsing authinfo items spread over multiple lines
Date: Fri, 04 Dec 2015 14:25:00 +0000

On Thu 03 Dec 2015, Ted Zlatanov wrote:

> On Thu, 3 Dec 2015 11:20:29 +0100 Vasilij Schneidermann <v.schneidermann <at> gmail.com> wrote: 
>
> VS> A .netrc or .authinfo file typically contains the credentials for a
> VS> specific machine on a single line, but according to netrc(5)
> VS> <http://linux.die.net/man/5/netrc> it's allowed to use newlines between
> VS> the tokens as well.  Using (netrc-parse) leaves such items out which is
> VS> clearly an error in the implementation.
>
> The Emacs implementation in auth-source.el `auth-source-netrc-parse',
> which is used by default for netrc files today, is not a
> reimplementation of that semi-standard:
>
> * there is no RFC for the netrc format as far as I know
>
> * Emacs supports both single and double quotes around strings (note the
>   man page doesn't indicate how tokens are quoted!)
>
> * macdef and default are not explicitly supported
>
> * arbitrary keywords are supported
>
> * the "machine" keyword is not required for an entry
>
> The last two, especially, make it hard to support multi-line entries.
> Since there is no RFC to my knowledge and no functional need to spread
> entries across multiple lines, I'd like to close this as a WONTFIX. Any
> opinions?

RFCs describe wire-protocols and interchange data formats, so I would
not expect an RFC for this. The .netrc file format is from BSD UNIX.

Support for arbitrary keywords, and omission of the machine token mean
that emacs will parse a .netrc file that other programs reject as
ill-formed. That is unhelpful: either be compatible with other programs
that read .netrc, or define  a new file format with a different name.

    AndyM

Message #19 received at 22081 <at> debbugs.gnu.org (full text, mbox):

From: Ted Zlatanov <tzz <at> lifelogs.com>
To: Andy Moreton <andrewjmoreton <at> gmail.com>
Cc: Glenn Morris <rgm <at> gnu.org>, 22081 <at> debbugs.gnu.org,
 Vasilij Schneidermann <v.schneidermann <at> gmail.com>,
 Lars Magne Ingebrigtsen <larsi <at> gnus.org>
Subject: Re: bug#22081: 24.5;
 netrc.el fails parsing authinfo items spread over multiple lines
Date: Sat, 05 Dec 2015 18:43:38 -0500

On Fri, 04 Dec 2015 14:25:00 +0000 Andy Moreton <andrewjmoreton <at> gmail.com> wrote: 

AM> RFCs describe wire-protocols and interchange data formats, so I would
AM> not expect an RFC for this. The .netrc file format is from BSD UNIX.

For instance, https://tools.ietf.org/html/rfc1035 defines the format of
master zone files, and https://tools.ietf.org/html/rfc4180 defines CSV
files. The closest RFC in spirit is https://tools.ietf.org/html/rfc4155
which defines the mbox format but says

    "The mbox database format is not documented in an
    authoritative specification, but instead exists as a well-known output
    format that is anecdotally documented, or which is only authoritatively
    documented for a specific platform or tool."

Sounds familiar? :) I'm sure we can argue for a while about the nuances
of RFCs, but I hope this is enough to explain why I looked for an RFC.

My more important point was that there's no standard (or library) for
parsing netrc files. In particular, quoting is not specified as I
mentioned. The BSD spec is ambiguous and allowing multi-line entries
would not serve any practical purpose.

I did a quick survey. I first remember using netrc files with ncftp. The
documentation at http://cng.seas.rochester.edu/CNG/docs/man/ncftp.html
shows examples of the "macdef" keyword that make it clear that it just
doesn't make sense across multiple programs:

       machine wuarchive.wustl.edu
                macdef init
                      cd /graphics/gif
                      ls -lt
                (mandatory blank line to end the macro)

Note that quoting is not shown, so it's up to the user to figure it out.

The most used implementation is probably
http://curl.haxx.se/docs/manual.html but they don't specify the syntax
and from looking at their netrc.c don't support "default" or "macdef"
keywords either.

wget just says to consult "your system manuals" but does support
"default" and "macdef" in its netrc.c so it's a bit more compliant.

I hope this was useful to illustrate my original statements.

AM> Support for arbitrary keywords, and omission of the machine token mean
AM> that emacs will parse a .netrc file that other programs reject as
AM> ill-formed. That is unhelpful: either be compatible with other programs
AM> that read .netrc, or define  a new file format with a different name.

Do you have an actual example? Can you be more specific about what broke
and how? Otherwise I think it's OK to leave things as they are.

Ted

Message #22 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Andy Moreton <andrewjmoreton <at> gmail.com>
To: bug-gnu-emacs <at> gnu.org
Subject: Re: bug#22081: 24.5;
 netrc.el fails parsing authinfo items spread over multiple lines
Date: Mon, 07 Dec 2015 14:14:13 +0000

On Sat 05 Dec 2015, Ted Zlatanov wrote:

> On Fri, 04 Dec 2015 14:25:00 +0000 Andy Moreton <andrewjmoreton <at> gmail.com> wrote: 
> My more important point was that there's no standard (or library) for
> parsing netrc files. In particular, quoting is not specified as I
> mentioned. The BSD spec is ambiguous and allowing multi-line entries
> would not serve any practical purpose.

The spec clearly defines that tokens are separated with SPACE TAB or
NEWLINE, which seems unambiguous to me. Multi-line entries are an
inherent part of the file format, and are useful for readable layout.
For an example, see:
    http://www.catb.org/esr/writings/taoup/html/ch10s03.html

It's perfectly ok to parse an extended version of the file format which
allows for quoting and escaping mechanisms, but not if it is done is a way
which is incompatible with reasonable and well formed existing files.

Disallowing NEWLINE as a token separator means you are not parsing an
extension of the existing format, but something incompatible with it.
That's fine as it offers useful features, but it should use a different
file name to make it clear that it is not the same format.

    AndyM

Message #25 received at 22081 <at> debbugs.gnu.org (full text, mbox):

From: Ted Zlatanov <tzz <at> lifelogs.com>
To: Andy Moreton <andrewjmoreton <at> gmail.com>
Cc: 22081 <at> debbugs.gnu.org
Subject: Re: bug#22081: 24.5;
 netrc.el fails parsing authinfo items spread over multiple lines
Date: Mon, 07 Dec 2015 13:15:59 -0500

On Mon, 07 Dec 2015 14:14:13 +0000 Andy Moreton <andrewjmoreton <at> gmail.com> wrote: 

AM> On Sat 05 Dec 2015, Ted Zlatanov wrote:
>> On Fri, 04 Dec 2015 14:25:00 +0000 Andy Moreton <andrewjmoreton <at> gmail.com> wrote: 
>> My more important point was that there's no standard (or library) for
>> parsing netrc files. In particular, quoting is not specified as I
>> mentioned. The BSD spec is ambiguous and allowing multi-line entries
>> would not serve any practical purpose.

AM> The spec clearly defines that tokens are separated with SPACE TAB or
AM> NEWLINE, which seems unambiguous to me.

Without a quoting standard, it's not clear whether strings inside quotes
can be multiline and whether the newline itself can be escaped.

AM> Disallowing NEWLINE as a token separator means you are not parsing an
AM> extension of the existing format, but something incompatible with it.
AM> That's fine as it offers useful features, but it should use a different
AM> file name to make it clear that it is not the same format.

Fortunately `auth-sources' defaults to ~/.authinfo and ~/.authinfo.gpg
before ~/.netrc right now.

Ted

Message #28 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Andy Moreton <andrewjmoreton <at> gmail.com>
To: bug-gnu-emacs <at> gnu.org
Subject: Re: bug#22081: 24.5;
 netrc.el fails parsing authinfo items spread over multiple lines
Date: Tue, 08 Dec 2015 09:35:22 +0000

On Mon 07 Dec 2015, Ted Zlatanov wrote:

> On Mon, 07 Dec 2015 14:14:13 +0000 Andy Moreton <andrewjmoreton <at> gmail.com> wrote: 
>
> AM> On Sat 05 Dec 2015, Ted Zlatanov wrote:
>>> On Fri, 04 Dec 2015 14:25:00 +0000 Andy Moreton <andrewjmoreton <at> gmail.com> wrote: 
>>> My more important point was that there's no standard (or library) for
>>> parsing netrc files. In particular, quoting is not specified as I
>>> mentioned. The BSD spec is ambiguous and allowing multi-line entries
>>> would not serve any practical purpose.
>
> AM> The spec clearly defines that tokens are separated with SPACE TAB or
> AM> NEWLINE, which seems unambiguous to me.
>
> Without a quoting standard, it's not clear whether strings inside quotes
> can be multiline and whether the newline itself can be escaped.

The original spec is archaic - there is no quoting standard, only tokens
separated by whitespace. As such, a quote is simply another character.

> AM> Disallowing NEWLINE as a token separator means you are not parsing an
> AM> extension of the existing format, but something incompatible with it.
> AM> That's fine as it offers useful features, but it should use a different
> AM> file name to make it clear that it is not the same format.
>
> Fortunately `auth-sources' defaults to ~/.authinfo and ~/.authinfo.gpg
> before ~/.netrc right now.

Indeed.

    AndyM

Message #33 received at 22081-done <at> debbugs.gnu.org (full text, mbox):

From: Ted Zlatanov <tzz <at> lifelogs.com>
To: Andy Moreton <andrewjmoreton <at> gmail.com>
Cc: 22081-done <at> debbugs.gnu.org
Subject: Re: bug#22081: 24.5;
 netrc.el fails parsing authinfo items spread over multiple lines
Date: Tue, 08 Dec 2015 09:06:33 -0500

On Tue, 08 Dec 2015 09:35:22 +0000 Andy Moreton <andrewjmoreton <at> gmail.com> wrote: 

AM> On Mon 07 Dec 2015, Ted Zlatanov wrote:

>> Without a quoting standard, it's not clear whether strings inside quotes
>> can be multiline and whether the newline itself can be escaped.

AM> The original spec is archaic - there is no quoting standard, only tokens
AM> separated by whitespace. As such, a quote is simply another character.

Right, and therefore we can't support the original spec as long as we
also support the more important single-quoting and double-quoting of strings.

I'm closing this without prejudice because I think we've stated our
positions clearly and it's not a bug, but if you or anyone else want to
propose a patch that can support multiline or the other keywords without
breaking the current functionality, we can reopen.

I would also welcome patches to remove the netrc.el usage.

Thanks
Ted

Message #41 received at 22081 <at> debbugs.gnu.org (full text, mbox):

From: Lars Magne Ingebrigtsen <larsi <at> gnus.org>
To: Vasilij Schneidermann <v.schneidermann <at> gmail.com>
Cc: 22081 <at> debbugs.gnu.org
Subject: Re: bug#22081: 24.5;
 netrc.el fails parsing authinfo items spread over multiple lines
Date: Mon, 14 Dec 2015 18:57:08 +0100

Ted Zlatanov <tzz <at> lifelogs.com> writes:

> (The netrc.el `netrc-parse' code is not actively used nowadays in Emacs,
> as far as I know, but hasn't been removed due to my laziness. I see a
> single reference to it in gnus/nntp.el. It has some deficiencies that
> were improved in `auth-source-netrc-parse', for instance supporting
> single-quoted strings. Any objections to removing it? CC to Lars since
> he wrote it originally.)

Let's see...

Geez, that's some convoluted code.  I think what it's meant to do is to
allow the user to define `nntp-authinfo-file' and allow that to override
the other Emacs authinfo choices.

I think it would be nice to not break people's setups, but I think it
would be clearer just to do something like

(let ((auth-sources (cons nntp-authinfo-file auth-sources)))
  ...)

before calling auth-source, and delete the explicit calls to netrc
completely.  And then you could move netrc.el to obsolete/.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.