GNU bug report logs - #33441
reading and printing Lisp Objects - what changed from 25.3.1 to 26.1?

Previous Next

To reply to this bug, email your comments to 33441 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Peter Milliken <peter.milliken <at> gmail.com>
To: bug-gnu-emacs <at> gnu.org
Subject: reading and printing Lisp Objects - what changed from 25.3.1 to 26.1?
Date: Tue, 20 Nov 2018 18:21:34 +1100

[Message part 1 (text/plain, inline)]

I'm not sure if this is a bug as such, but I can't find any changes to the
Elisp Manual (for 26.1) to explain the behaviour.

I have a long-standing Emacs extension that, to save load time, saves Lisp
Objects in a file and then reads them at Emacs start-up (if the file is
available). This works up to Emacs 25.3, but fails on the Read process, in
Emacs 26.1, with an error message in *Messages* buffer of:

setq: Invalid read syntax: "#"

I have looked at the Lisp Object files produced under 25.3.1 and 26.1 and
they look completely different. The "same" Lisp objects file written by
26.1 is approximately 15 times the size of the Lisp objects written by
25.3.1.

I use a "print" command to write the information to file and a "read"
command to read back as a Lisp Object. The objects written are Lisp class
instances created from the eieio class package library.

I have attached the two files containing the produced Lisp Objects - as
stated, the larger file is produced using 26.1, the smaller file contains
the same Lisp Objects produced by 25.3.1. With 26.1, the "read" operation
fails at line 16 (the first eieio class object instance).

Any help on this would be appreciated. My initial thought is to just trap
the error and bypass reading the Lisp Objects from file and take them from
the original text source instead - but that is a slower process, but would
allow my extension to work with both Emacs versions.

Thanks
Peter

P.S. If you want to look at the code that reads and writes these Lisp
Objects, it is on GitHub at:

https://github.com/peter-milliken/ELSE/else-structs.el

The defuns to write/read the Lisp Objects are dump-language-to-file (line
96) and restore-language-from-file (line 119) respectively.

[Message part 2 (text/html, inline)]

[C-25.3.1.esl (application/octet-stream, attachment)]

[C-26.1.esl (application/octet-stream, attachment)]

Message #8 received at 33441 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Peter Milliken <peter.milliken <at> gmail.com>
Cc: 33441 <at> debbugs.gnu.org
Subject: Re: reading and printing Lisp Objects - what changed from 25.3.1 to
 26.1?
Date: Tue, 20 Nov 2018 12:56:16 -0800

> https://github.com/peter-milliken/ELSE/else-structs.el

That gives me a 404 Not Found.

The syntax error occurs at the "#5" here:

... #s(#5 "" "" "" nil nil nil nil nil) (:custom-groups nil 
:documentation "Placeholder Class base." ...

and you might want to investigate what is outputting "#5".

Message #11 received at 33441 <at> debbugs.gnu.org (full text, mbox):

From: Noam Postavsky <npostavs <at> gmail.com>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: 33441 <at> debbugs.gnu.org, Peter Milliken <peter.milliken <at> gmail.com>
Subject: Re: bug#33441: reading and printing Lisp Objects - what changed from
 25.3.1 to 26.1?
Date: Tue, 20 Nov 2018 16:02:27 -0500

On Tue, 20 Nov 2018 at 15:57, Paul Eggert <eggert <at> cs.ucla.edu> wrote:

> The syntax error occurs at the "#5" here:
>
> ... #s(#5 "" "" "" nil nil nil nil nil) (:custom-groups nil
> :documentation "Placeholder Class base." ...
>
> and you might want to investigate what is outputting "#5".

Standard Emacs printing functions will produce that for circular
structures if print-circle is nil (otherwise they use #5#).

Message #14 received at 33441 <at> debbugs.gnu.org (full text, mbox):

From: Peter Milliken <peter.milliken <at> gmail.com>
To: npostavs <at> gmail.com
Cc: 33441 <at> debbugs.gnu.org, eggert <at> cs.ucla.edu
Subject: Re: bug#33441: reading and printing Lisp Objects - what changed from
 25.3.1 to 26.1?
Date: Wed, 21 Nov 2018 08:09:38 +1100

[Message part 1 (text/plain, inline)]

Not sure what the reference to circular structures means in this context.
As I said, all versions of Emacs less than 26 it worked fine with these
same Lisp Objects.

As I said: what changed?  Are you trying to say something in 26.1 is now
"fixed" i.e. it should never have worked in versions less than 26? and what
I have been doing for years should have always failed?

As for the file reference, try
https://github.com/peter-milliken/ELSE
<https://github.com/peter-milliken/ELSE/else-structs.el>

and look for else-structs.e
<https://github.com/peter-milliken/ELSE/else-structs.el>l



On Wed, Nov 21, 2018 at 8:02 AM Noam Postavsky <npostavs <at> gmail.com> wrote:

> On Tue, 20 Nov 2018 at 15:57, Paul Eggert <eggert <at> cs.ucla.edu> wrote:
>
> > The syntax error occurs at the "#5" here:
> >
> > ... #s(#5 "" "" "" nil nil nil nil nil) (:custom-groups nil
> > :documentation "Placeholder Class base." ...
> >
> > and you might want to investigate what is outputting "#5".
>
> Standard Emacs printing functions will produce that for circular
> structures if print-circle is nil (otherwise they use #5#).
>

[Message part 2 (text/html, inline)]

Message #17 received at 33441 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Peter Milliken <peter.milliken <at> gmail.com>, npostavs <at> gmail.com
Cc: 33441 <at> debbugs.gnu.org
Subject: Re: bug#33441: reading and printing Lisp Objects - what changed from
 25.3.1 to 26.1?
Date: Tue, 20 Nov 2018 13:17:46 -0800

On 11/20/18 1:09 PM, Peter Milliken wrote:
> Not sure what the reference to circular structures means in this 
> context. As I said, all versions of Emacs less than 26 it worked fine 
> with these same Lisp Objects.

How do we know the Lisp objects are actually the same?


> As for the file reference, try
> https://github.com/peter-milliken/ELSE 
> <https://github.com/peter-milliken/ELSE/else-structs.el>
>
> and look for else-structs.e 
> <https://github.com/peter-milliken/ELSE/else-structs.el>l
>
Thanks. Could you give a complete, self-contained recipe for reproducing 
the problem?

Message #20 received at 33441 <at> debbugs.gnu.org (full text, mbox):

From: Noam Postavsky <npostavs <at> gmail.com>
To: Peter Milliken <peter.milliken <at> gmail.com>
Cc: 33441 <at> debbugs.gnu.org, Paul Eggert <eggert <at> cs.ucla.edu>
Subject: Re: bug#33441: reading and printing Lisp Objects - what changed from
 25.3.1 to 26.1?
Date: Tue, 20 Nov 2018 16:19:38 -0500

On Tue, 20 Nov 2018 at 16:09, Peter Milliken <peter.milliken <at> gmail.com> wrote:
>
> Not sure what the reference to circular structures means in this context. As I said, all versions of Emacs less than 26 it worked fine with these same Lisp Objects.
>
> As I said: what changed?  Are you trying to say something in 26.1 is now "fixed" i.e. it should never have worked in versions less than 26? and what I have been doing for years should have always failed?

I think the difference is that in Emacs 25 and earlier the type of a
struct (or eieio class?) was encoded as a symbol, whereas in Emacs 26
and later it is a record object with circular references.

See also https://debbugs.gnu.org/cgi/bugreport.cgi?bug=29220#47 and replies.

Message #23 received at 33441 <at> debbugs.gnu.org (full text, mbox):

From: Peter Milliken <peter.milliken <at> gmail.com>
To: npostavs <at> gmail.com
Cc: 33441 <at> debbugs.gnu.org, eggert <at> cs.ucla.edu
Subject: Re: bug#33441: reading and printing Lisp Objects - what changed from
 25.3.1 to 26.1?
Date: Wed, 21 Nov 2018 12:52:57 +1100

[Message part 1 (text/plain, inline)]

OK, I skimmed through the trail on 29220 and it appears to me as though
there was no definitive fix implemented - even though there was an
acknowledgment that a problem existed. The annotation at the end indicates
there have been no changes for 268 days, so I guess that is it. My object
instances are definitely complicated and probably do have circularities.

I think perhaps you might want to put more information/warnings in section
19 of the Elisp manual because it is obviously not true anymore that you
can simply Print/Read any arbitrary Lisp Object.

I'm going to trap the error in my extension and just force
reading/"compiling" from the original text file instead of reading from the
intermediate file created by the print process. This will just cause a
small delay to the user the first time that a set of language templates get
loaded in an edit session - probably not a big deal. Certainly, it is
better than telling people my extension is "broken" and won't work with
26.1 and onwards.

Thanks for the help and information.

Regards
Peter

On Wed, Nov 21, 2018 at 8:19 AM Noam Postavsky <npostavs <at> gmail.com> wrote:

> On Tue, 20 Nov 2018 at 16:09, Peter Milliken <peter.milliken <at> gmail.com>
> wrote:
> >
> > Not sure what the reference to circular structures means in this
> context. As I said, all versions of Emacs less than 26 it worked fine with
> these same Lisp Objects.
> >
> > As I said: what changed?  Are you trying to say something in 26.1 is now
> "fixed" i.e. it should never have worked in versions less than 26? and what
> I have been doing for years should have always failed?
>
> I think the difference is that in Emacs 25 and earlier the type of a
> struct (or eieio class?) was encoded as a symbol, whereas in Emacs 26
> and later it is a record object with circular references.
>
> See also https://debbugs.gnu.org/cgi/bugreport.cgi?bug=29220#47 and
> replies.
>

[Message part 2 (text/html, inline)]

Message #26 received at 33441 <at> debbugs.gnu.org (full text, mbox):

From: "Basil L. Contovounesios" <contovob <at> tcd.ie>
To: Peter Milliken <peter.milliken <at> gmail.com>
Cc: 33441 <at> debbugs.gnu.org, eggert <at> cs.ucla.edu, npostavs <at> gmail.com
Subject: Re: bug#33441: reading and printing Lisp Objects - what changed from
 25.3.1 to 26.1?
Date: Wed, 21 Nov 2018 03:27:23 +0000

Peter Milliken <peter.milliken <at> gmail.com> writes:

> I think perhaps you might want to put more information/warnings in
> section 19 of the Elisp manual because it is obviously not true
> anymore that you can simply Print/Read any arbitrary Lisp Object.

FWIW, this has been outlined in (info "(elisp) Streams Intro") since
before Emacs 24.  Having said that, more information never hurts.

-- 
Basil

Message #29 received at 33441 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Peter Milliken <peter.milliken <at> gmail.com>
Cc: 33441 <at> debbugs.gnu.org, eggert <at> cs.ucla.edu, npostavs <at> gmail.com
Subject: Re: bug#33441: reading and printing Lisp Objects - what changed from
 25.3.1 to 26.1?
Date: Wed, 21 Nov 2018 05:46:40 +0200

> From: Peter Milliken <peter.milliken <at> gmail.com>
> Date: Wed, 21 Nov 2018 12:52:57 +1100
> Cc: 33441 <at> debbugs.gnu.org, eggert <at> cs.ucla.edu
> 
> I think perhaps you might want to put more information/warnings in section 19 of the Elisp manual because it
> is obviously not true anymore that you can simply Print/Read any arbitrary Lisp Object.
> 
> I'm going to trap the error in my extension and just force reading/"compiling" from the original text file instead
> of reading from the intermediate file created by the print process. This will just cause a small delay to the user
> the first time that a set of language templates get loaded in an edit session - probably not a big deal. Certainly,
> it is better than telling people my extension is "broken" and won't work with 26.1 and onwards.

I'd urge you not to give up on Emacs maintenance so quickly, and
provide the reproducer requested by Paul.  Please give us a chance to
analyze the problem you are having and consider whether and how it
could or should be fixed.  Leaving the problems unfixed because we
failed to look into them is not a good paradigm in maintaining such a
flexible and powerful software package as Emacs is.

Thanks in advance.

Message #32 received at 33441 <at> debbugs.gnu.org (full text, mbox):

From: Peter Milliken <peter.milliken <at> gmail.com>
To: eliz <at> gnu.org
Cc: 33441 <at> debbugs.gnu.org, eggert <at> cs.ucla.edu, npostavs <at> gmail.com
Subject: Re: bug#33441: reading and printing Lisp Objects - what changed from
 25.3.1 to 26.1?
Date: Wed, 21 Nov 2018 15:21:43 +1100

[Message part 1 (text/plain, inline)]

I'm not sure that I am giving up on Emacs maintenance - the reference Noam
supplied indicates this is a "well known" issue and somebody, has some
intention, at some stage, to do something about it, but given the last
update to referenced bug/email stream was 268 days ago, that intention may
be on the back burner :-)

Faking up a code snippet is not necessarily that easy - and, again, the
problem described in 29220 sounds exactly like the issue I am having. I
have already supplied the GitHub reference to my package and indicated the
file that contains the functions that use/invoke the read/print processes.
This is a working package, so, in my opinion, that satisfies that request
:-) Using the package is not difficult, it is fully documented with its own
user manual and the writing/reading process of the Lisp Objects is
completely transparent to the user and happens automatically the first time
the minor mode is invoked i.e. any maintainer investigating the issue
shouldn't need to spend too much time getting to a point where they can
debug the issue. The file(s) are written to the "user-emacs-directory".

regards
Peter



On Wed, Nov 21, 2018 at 2:46 PM Eli Zaretskii <eliz <at> gnu.org> wrote:

> > From: Peter Milliken <peter.milliken <at> gmail.com>
> > Date: Wed, 21 Nov 2018 12:52:57 +1100
> > Cc: 33441 <at> debbugs.gnu.org, eggert <at> cs.ucla.edu
> >
> > I think perhaps you might want to put more information/warnings in
> section 19 of the Elisp manual because it
> > is obviously not true anymore that you can simply Print/Read any
> arbitrary Lisp Object.
> >
> > I'm going to trap the error in my extension and just force
> reading/"compiling" from the original text file instead
> > of reading from the intermediate file created by the print process. This
> will just cause a small delay to the user
> > the first time that a set of language templates get loaded in an edit
> session - probably not a big deal. Certainly,
> > it is better than telling people my extension is "broken" and won't work
> with 26.1 and onwards.
>
> I'd urge you not to give up on Emacs maintenance so quickly, and
> provide the reproducer requested by Paul.  Please give us a chance to
> analyze the problem you are having and consider whether and how it
> could or should be fixed.  Leaving the problems unfixed because we
> failed to look into them is not a good paradigm in maintaining such a
> flexible and powerful software package as Emacs is.
>
> Thanks in advance.
>

[Message part 2 (text/html, inline)]

Message #37 received at 33441 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Peter Milliken <peter.milliken <at> gmail.com>
Cc: 33441 <at> debbugs.gnu.org
Subject: Re: bug#33441: reading and printing Lisp Objects - what changed from
 25.3.1 to 26.1?
Date: Thu, 22 Nov 2018 23:12:41 -0800

Peter Milliken wrote:
> Faking up a code snippet is not necessarily that easy - and, again, the
> problem described in 29220 sounds exactly like the issue I am having.

If it's a pain to reproduce the bug and if you have a different workaround which 
means the bug is low priority for you, then perhaps the best we can do for now 
is to merge bugs 29220 and 33441. I'll do that next.

Message #42 received at 33441 <at> debbugs.gnu.org (full text, mbox):

From: Noam Postavsky <npostavs <at> gmail.com>
To: Peter Milliken <peter.milliken <at> gmail.com>
Cc: 33441 <at> debbugs.gnu.org, eliz <at> gnu.org, eggert <at> cs.ucla.edu
Subject: Re: bug#33441: reading and printing Lisp Objects - what changed from
 25.3.1 to 26.1?
Date: Mon, 01 Apr 2019 19:55:15 -0400

Peter Milliken <peter.milliken <at> gmail.com> writes:

> I'm not sure that I am giving up on Emacs maintenance - the reference Noam
> supplied indicates this is a "well known" issue and somebody, has some
> intention, at some stage, to do something about it, but given the last
> update to referenced bug/email stream was 268 days ago, that intention may
> be on the back burner :-)

Hi, I'm reviewing some old bugs, and I think I missed explicitly giving
the obvious suggestion to you here the first time around.  Does it help
to let-bind print-circle to t in dump-language-to-file?  Bug#29220 has a
lot more going on because it concerns eieio-persist* functions, which as
far as I can tell, your package is not using.

> Faking up a code snippet is not necessarily that easy

Possibly I'm missing something, but is there more to your problem than
this?

    (defclass foo ()
      ())

    (read (prin1-to-string (make-instance 'foo))) ;=> (invalid-read-syntax "#")

    (read (let ((print-circle t))
            (prin1-to-string (make-instance 'foo)))) ;=> #s(#s(eieio--class foo ...))

Message #45 received at 33441 <at> debbugs.gnu.org (full text, mbox):

From: Peter Milliken <peter.milliken <at> gmail.com>
To: Noam Postavsky <npostavs <at> gmail.com>
Cc: 33441 <at> debbugs.gnu.org, eliz <at> gnu.org, eggert <at> cs.ucla.edu
Subject: Re: bug#33441: reading and printing Lisp Objects - what changed from
 25.3.1 to 26.1?
Date: Tue, 2 Apr 2019 13:51:31 +1100

[Message part 1 (text/plain, inline)]

Hi Noam,

Thanks for the reply. If I understand correctly, you intend a change like
this:

(cl-defmethod dump-language-to-file ((obj else-repository) name to-file)
  (let ((language-to-dump (access-language else-Language-Repository name))
        (write-marker nil)
        (language-output-buffer nil)
        (print-circle t))
<-------------------- new line
    (save-excursion
      (unless (file-directory-p (file-name-directory to-file))
        (make-directory (file-name-directory to-file)))
.
.
      (mapatoms (lambda (e)
                  (print (symbol-plist e) write-marker))
                (oref language-to-dump placeholders))
      (save-buffer)
      (kill-buffer))))

I tried this and the read code still throws an error when reading the
objects back from the file.

Peter



On Tue, Apr 2, 2019 at 10:55 AM Noam Postavsky <npostavs <at> gmail.com> wrote:

> Peter Milliken <peter.milliken <at> gmail.com> writes:
>
> > I'm not sure that I am giving up on Emacs maintenance - the reference
> Noam
> > supplied indicates this is a "well known" issue and somebody, has some
> > intention, at some stage, to do something about it, but given the last
> > update to referenced bug/email stream was 268 days ago, that intention
> may
> > be on the back burner :-)
>
> Hi, I'm reviewing some old bugs, and I think I missed explicitly giving
> the obvious suggestion to you here the first time around.  Does it help
> to let-bind print-circle to t in dump-language-to-file?  Bug#29220 has a
> lot more going on because it concerns eieio-persist* functions, which as
> far as I can tell, your package is not using.
>
> > Faking up a code snippet is not necessarily that easy
>
> Possibly I'm missing something, but is there more to your problem than
> this?
>
>     (defclass foo ()
>       ())
>
>     (read (prin1-to-string (make-instance 'foo))) ;=> (invalid-read-syntax
> "#")
>
>     (read (let ((print-circle t))
>             (prin1-to-string (make-instance 'foo)))) ;=>
> #s(#s(eieio--class foo ...))
>

[Message part 2 (text/html, inline)]

Message #48 received at 33441 <at> debbugs.gnu.org (full text, mbox):

From: Noam Postavsky <npostavs <at> gmail.com>
To: Peter Milliken <peter.milliken <at> gmail.com>
Cc: 33441 <at> debbugs.gnu.org, eliz <at> gnu.org, eggert <at> cs.ucla.edu
Subject: Re: bug#33441: reading and printing Lisp Objects - what changed from
 25.3.1 to 26.1?
Date: Tue, 02 Apr 2019 09:29:53 -0400

Peter Milliken <peter.milliken <at> gmail.com> writes:

> (cl-defmethod dump-language-to-file ((obj else-repository) name to-file)
>   (let ((language-to-dump (access-language else-Language-Repository name))
>         (write-marker nil)
>         (language-output-buffer nil)
>         (print-circle t))  <-------------------- new line

> I tried this and the read code still throws an error when reading the
> objects back from the file.

I tried it here; this time the error isn't from reading, but from
add-element "Attempting to add an illegal object to language".  So this
means it really is the same problem as in bug#29220: that newly read
objects don't have the right type.

A quick demo:

    (require 'eieio)

    (defclass foo ()
      ())

    (let ((print-circle t)
          (foo-obj (make-instance 'foo)))
      (cl-assert (foo-p foo-obj))
      ;; This assertion fails!
      (cl-assert (foo-p (read (prin1-to-string foo-obj)))))

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.