GNU bug report logs - #10132
Help lilypond interleave scheme and lilypond code in guile 2.x

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 10132 in the body.
You can then email your comments to 10132 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Andy Wingo <wingo <at> pobox.com>
To: bug-guile <bug-guile <at> gnu.org>
Cc: Ian Hulin <ian <at> hulin.org.uk>, David Kastrup <dak <at> gnu.org>
Subject: Help lilypond interleave scheme and lilypond code in guile 2.x
Date: Fri, 25 Nov 2011 12:15:14 +0100

Hi David,

This bug was forked from bug 10099, where David has a longer
explanation.

On Fri 25 Nov 2011 11:37, David Kastrup <dak <at> gnu.org> writes:

> So much for that.  The next quote is for a totally different issue, the
> availability of local environments and evaluation in them.  Lilypond has
> an input syntax of its own, and it allows interspersing Scheme code. $
> or # switches to the Scheme interpreter (for one sexp) when in Lilypond
> syntax, and #{ ... #} switches to Lilypond inside.

Aaah.  Thanks for this explanation; I had never seen this code before.

Do you use a read-hash-extend reader for #{#} ?

What do you use to parse the lilypond code?  What does it parse to?

I agree that the-environment and local-eval were nice solutions for
this.  In Guile 2.0 it's not as nice for you, because if you implement
another evaluator, you don't get backtraces that are as nice.

> As I said: for this particular application, I have coded a rather
> inelegant and resource-grabbing workaround that really is not going to
> help performance since the intertwined Lilypond interpreter does not
> benefit from precompilation of mostly trivial lambda functions when the
> actual procedure-environment is unlikely to ever reference more than
> five variables.

Understood.  Let's work to find a good solution in 2.0.

Andy
-- 
http://wingolog.org/

Message #8 received at submit <at> debbugs.gnu.org (full text, mbox):

From: David Kastrup <dak <at> gnu.org>
To: Andy Wingo <wingo <at> pobox.com>
Cc: bug-guile <bug-guile <at> gnu.org>, Ian Hulin <ian <at> hulin.org.uk>
Subject: Re: Help lilypond interleave scheme and lilypond code in guile 2.x
Date: Fri, 25 Nov 2011 14:35:28 +0100

Andy Wingo <wingo <at> pobox.com> writes:

> Hi David,
>
> This bug was forked from bug 10099, where David has a longer
> explanation.
>
> On Fri 25 Nov 2011 11:37, David Kastrup <dak <at> gnu.org> writes:
>
>> So much for that.  The next quote is for a totally different issue, the
>> availability of local environments and evaluation in them.  Lilypond has
>> an input syntax of its own, and it allows interspersing Scheme code. $
>> or # switches to the Scheme interpreter (for one sexp) when in Lilypond
>> syntax, and #{ ... #} switches to Lilypond inside.
>
> Aaah.  Thanks for this explanation; I had never seen this code before.
>
> Do you use a read-hash-extend reader for #{#} ?

Yes:

<URL:http://git.savannah.gnu.org/cgit/lilypond.git/tree/scm/parser-ly-from-scheme.scm>

This link is likely a bit of a moving target since I have several
patches in the queue to make the behavior more predictable (and produce
better quality error messages when things go wrong).

> What do you use to parse the lilypond code?  What does it parse to?

Classical Bison/Flex parser/scanner.  There is no "what does it parse
to" since the Bison rules execute the actions on the fly: it is a
classical interpreter.  With a number of lexical and semantical tie-ins,
it would be non-trivial to actually create an intermediate
representation.

<URL:http://git.savannah.gnu.org/cgit/lilypond.git/tree/lily/parser.yy>

<URL:http://git.savannah.gnu.org/cgit/lilypond.git/tree/lily/lexer.ll>

The file responsible for reading and evaluating embedded Scheme
expressions is

<URL:http://git.savannah.gnu.org/cgit/lilypond.git/tree/lily/parse-scm.cc>

The proposed robustifying changes currently in the queue are in
<URL:http://codereview.appspot.com/5437055>.

The reading is done by the lexer.  Evaluation of $ forms is done in the
lexer, evaluation of # forms is delayed and happens in the parser in
order to avoid timing problems (lookahead tokens should preferably not
be evaluated while they are still lookahead since they might depend on
the actions of the commands before them).

> I agree that the-environment and local-eval were nice solutions for
> this.  In Guile 2.0 it's not as nice for you, because if you implement
> another evaluator, you don't get backtraces that are as nice.

We don't really do anything in the line of backtraces.  Until the
proposed patch gets through, we don't really do anything sensible in the
line of error messages either.

>> As I said: for this particular application, I have coded a rather
>> inelegant and resource-grabbing workaround that really is not going
>> to help performance since the intertwined Lilypond interpreter does
>> not benefit from precompilation of mostly trivial lambda functions
>> when the actual procedure-environment is unlikely to ever reference
>> more than five variables.
>
> Understood.  Let's work to find a good solution in 2.0.

If you follow the history of parser-ly-from-scheme.scm, you'll see that
there has been a flurry of activity recently.  Before I had to cater for
GuileV2, the code just sweeped up the procedure-environment of a
basically empty lambda function and left all rereading and evaluation to
runtime.  Even earlier than that, there was a complicated interpretation
of # and $ where the respective expressions were all evaluated in a
let-form at runtime before the Lilypond parser went to work.

The current approach of wrapping everything in lambda as compared to the
historic implementation of evaluating everything Scheme before starting
the Lilypond interpreter has the advantage that the timing of evaluation
(if any) is determined by the Lilypond interpreter.

The procedure-environment approach was elegant and minimally complex.
The question is how feasible it is for the Guile compiler to capture an
environment in a form that can be used even after compilation.  Like
taking the address of a variable in C, the export of such an environment
interferes with a number of static optimizations.  For our particular
application, readonly access to the symbols in the environment should be
quite sufficient, but of course I can't vouch for other potential uses.

-- 
David Kastrup

Message #11 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Andy Wingo <wingo <at> pobox.com>
To: David Kastrup <dak <at> gnu.org>
Cc: bug-guile <bug-guile <at> gnu.org>, Ian Hulin <ian <at> hulin.org.uk>
Subject: Re: Help lilypond interleave scheme and lilypond code in guile 2.x
Date: Fri, 25 Nov 2011 15:26:00 +0100

Hi,

I am going to be away from the machine for the weekend, but before I
headed out, I just wanted to put out one idea:

On Fri 25 Nov 2011 14:35, David Kastrup <dak <at> gnu.org> writes:

>> What do you use to parse the lilypond code?  What does it parse to?
>
> Classical Bison/Flex parser/scanner.  There is no "what does it parse
> to" since the Bison rules execute the actions on the fly: it is a
> classical interpreter.  With a number of lexical and semantical tie-ins,
> it would be non-trivial to actually create an intermediate
> representation.

Have you considered using silex or some other tokenizer in scheme,
combined with the lalr parser from (system base lalr)?  See "LALR(1)
Parsing" in the manual for Guile 2.0.

> The procedure-environment approach was elegant and minimally complex.
> The question is how feasible it is for the Guile compiler to capture an
> environment in a form that can be used even after compilation.  Like
> taking the address of a variable in C, the export of such an environment
> interferes with a number of static optimizations.  For our particular
> application, readonly access to the symbols in the environment should be
> quite sufficient, but of course I can't vouch for other potential uses.

If this is the answer, then we can figure out a way to implement it in
Guile 2.0.x as well.  But if you are amenable to it, implementing the
parser in Scheme would be another attractive option -- though, it would
be a change, and that has costs.

Cheers,

Andy
-- 
http://wingolog.org/

Message #14 received at submit <at> debbugs.gnu.org (full text, mbox):

From: David Kastrup <dak <at> gnu.org>
To: Andy Wingo <wingo <at> pobox.com>
Cc: bug-guile <bug-guile <at> gnu.org>, Ian Hulin <ian <at> hulin.org.uk>
Subject: Re: Help lilypond interleave scheme and lilypond code in guile 2.x
Date: Fri, 25 Nov 2011 15:44:27 +0100

Andy Wingo <wingo <at> pobox.com> writes:

> I am going to be away from the machine for the weekend, but before I
> headed out, I just wanted to put out one idea:
>
> On Fri 25 Nov 2011 14:35, David Kastrup <dak <at> gnu.org> writes:
>
>>> What do you use to parse the lilypond code?  What does it parse to?
>>
>> Classical Bison/Flex parser/scanner.  There is no "what does it parse
>> to" since the Bison rules execute the actions on the fly: it is a
>> classical interpreter.  With a number of lexical and semantical tie-ins,
>> it would be non-trivial to actually create an intermediate
>> representation.
>
> Have you considered using silex or some other tokenizer in scheme,
> combined with the lalr parser from (system base lalr)?  See "LALR(1)
> Parsing" in the manual for Guile 2.0.

Lilypond is not yet capable of running under Guile 2.0 (and needs to
stay 1.8-compatible for a considerable time span), so it makes no sense
to think about using 2.0 features for core parts of it.

The core of Lilypond is quite C++-centric.  Parsing takes a
non-negligible amount of runtime already.  Switching to a different
system likely to be slower and less directly interfacing with C++ is not
going to be on the agenda anytime soon, and I am less than convinced
that this total change of playground would actually make a qualitative
difference regarding the implementation of this particular scoping task.

>> The procedure-environment approach was elegant and minimally complex.
>> The question is how feasible it is for the Guile compiler to capture
>> an environment in a form that can be used even after compilation.
>> Like taking the address of a variable in C, the export of such an
>> environment interferes with a number of static optimizations.  For
>> our particular application, readonly access to the symbols in the
>> environment should be quite sufficient, but of course I can't vouch
>> for other potential uses.
>
> If this is the answer, then we can figure out a way to implement it in
> Guile 2.0.x as well.

"Readonly access" in this context does not mean "separate copies" but
rather "access to the original variables without the need to change them
via this access path".  If such a closure variable is changed by a
function "properly" compiled in the lexical closure (rather than
artificially relying on process-environment), we would likely still need
to see this change.

> But if you are amenable to it, implementing the parser in Scheme would
> be another attractive option -- though, it would be a change, and that
> has costs.

That's not on the table right now in my opinion, and I actually don't
see that it would help to a relevant degree since, as I said, lexical
and semantic tie-ins require the parser to work _progressively_ instead
of being able to compile an intermediate Scheme representation, and
without an intermediate representation, I don't see how we could exploit
the "natural" implicit implementation of closures in the Scheme
compiler.

-- 
David Kastrup

Message #19 received at 10132-done <at> debbugs.gnu.org (full text, mbox):

From: Andy Wingo <wingo <at> pobox.com>
To: 10132-done <at> debbugs.gnu.org
Cc: Ian Hulin <ian <at> hulin.org.uk>, David Kastrup <dak <at> gnu.org>
Subject: Re: bug#10132: Help lilypond interleave scheme and lilypond code in
	guile 2.x
Date: Fri, 27 Jan 2012 16:48:22 +0100

Hi Ian and David,

On Fri 25 Nov 2011 12:15, Andy Wingo <wingo <at> pobox.com> writes:

> On Fri 25 Nov 2011 11:37, David Kastrup <dak <at> gnu.org> writes:
>
>> As I said: for this particular application, I have coded a rather
>> inelegant and resource-grabbing workaround that really is not going to
>> help performance since the intertwined Lilypond interpreter does not
>> benefit from precompilation of mostly trivial lambda functions when the
>> actual procedure-environment is unlikely to ever reference more than
>> five variables.
>
> Understood.  Let's work to find a good solution in 2.0.

2.0 now has the-environment and local-eval again (though not
procedure-environment).  Import the (ice-9 local-eval) module to have
access to these forms.  Hopefully at this point we have solved this
issue; please give it a try, and open a new bug if something comes up.

Cheers,

Andy
-- 
http://wingolog.org/

Message #20 received at 10132-done <at> debbugs.gnu.org (full text, mbox):

From: David Kastrup <dak <at> gnu.org>
To: Andy Wingo <wingo <at> pobox.com>
Cc: Ian Hulin <ian <at> hulin.org.uk>, 10132-done <at> debbugs.gnu.org
Subject: Re: bug#10132: Help lilypond interleave scheme and lilypond code in
	guile 2.x
Date: Fri, 27 Jan 2012 17:32:26 +0100

Andy Wingo <wingo <at> pobox.com> writes:

> Hi Ian and David,
>
> On Fri 25 Nov 2011 12:15, Andy Wingo <wingo <at> pobox.com> writes:
>
>> On Fri 25 Nov 2011 11:37, David Kastrup <dak <at> gnu.org> writes:
>>
>>> As I said: for this particular application, I have coded a rather
>>> inelegant and resource-grabbing workaround that really is not going to
>>> help performance since the intertwined Lilypond interpreter does not
>>> benefit from precompilation of mostly trivial lambda functions when the
>>> actual procedure-environment is unlikely to ever reference more than
>>> five variables.
>>
>> Understood.  Let's work to find a good solution in 2.0.
>
> 2.0 now has the-environment and local-eval again (though not
> procedure-environment).  Import the (ice-9 local-eval) module to have
> access to these forms.  Hopefully at this point we have solved this
> issue; please give it a try, and open a new bug if something comes up.

Currently the stable LilyPond release 2.16 is imminent (meaning that
we'll hopefully get it out within the month).  We don't support Guilev2
yet (there have been serious problems with macro dependencies for
precompiling .go files as far as I understand because interspersedly
defined macros were impossible to deal with in the apparently required
per-file independent compilation: I am not convinced of that
requirement, but the people actually working on it are, and the
corresponding large source rearrangements and occasionally discovered
new dependencies so far have not been able to get completed before being
obsoleted a few times already).

There is no clear time frame on when that will happen and whether it
will be feasible to require a version of 2.0 that supports local-eval
right away, but it will likely happen some time in the next half year.

And it will definitely be helpful for the long-term maintainability of
LilyPond to be able working with the-environment again and phase out the
current hack.

Thank you very much!

-- 
David Kastrup

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.