GNU bug report logs - #53518
29.0.50; em-extpipe breaks input of sharp-quoted Lisp symbols

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 53518 in the body.
You can then email your comments to 53518 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Sean Whitton <spwhitton <at> spwhitton.name>
To: bug-gnu-emacs <at> gnu.org
Subject: 29.0.50; em-extpipe breaks input of sharp-quoted Lisp symbols
Date: Mon, 24 Jan 2022 22:33:10 -0700

Hello Jim,

On Mon 24 Jan 2022 at 06:39pm -08, Jim Porter wrote:

> I just noticed a small bit of breakage with this. It's no longer 
> possible to refer to Lisp functions in Eshell like so:
>
>    #'upcase
>
> Eshell explicitly supports this construct (see `eshell-lisp-regexp'), 
> though it doesn't appear to be documented in the manual. Currently, this 
> syntax is only occasionally useful, but I'm working on a patch series 
> where it'll likely become a lot more common. My patches will add support 
> for piping to Lisp functions, so that you can do the following, for example:
>
>    $ echo hi | #'upcase
>    HI
>
> It looks like the breakage in parsing #'upcase is the result of 
> `eshell-parse-external-pipeline' trying to skip over args like '*|' 
> (with the quotes). However, it sees the single-quote in #'upcase and 
> then calls `eshell-parse-literal-quote', resulting in the error message:
>
>    Expecting completion of delimiter ' ...

Thanks for the report.  I have a fix; incoming once I have a bug number.

-- 
Sean Whitton

Message #8 received at 53518 <at> debbugs.gnu.org (full text, mbox):

From: Sean Whitton <spwhitton <at> spwhitton.name>
To: 53518 <at> debbugs.gnu.org, Jim Porter <jporterbugs <at> gmail.com>
Cc: Lars Ingebrigtsen <larsi <at> gnus.org>,
 Michael Albinus <michael.albinus <at> gmx.de>
Subject: Re: bug#53518: 29.0.50; em-extpipe breaks input of sharp-quoted
 Lisp symbols
Date: Tue, 25 Jan 2022 01:50:35 -0700

[Message part 1 (text/plain, inline)]

Hello again,

On Mon 24 Jan 2022 at 06:39pm -08, Jim Porter wrote:

> I just noticed a small bit of breakage with this. It's no longer 
> possible to refer to Lisp functions in Eshell like so:
>
>    #'upcase
>
> Eshell explicitly supports this construct (see `eshell-lisp-regexp'), 
> though it doesn't appear to be documented in the manual. Currently, this 
> syntax is only occasionally useful, but I'm working on a patch series 
> where it'll likely become a lot more common. My patches will add support 
> for piping to Lisp functions, so that you can do the following, for example:
>
>    $ echo hi | #'upcase
>    HI

Out of curiosity, why is there a need for the sharpquote?  Why not just
'echo hi | upcase'?  Is it to do with requesting the new piping?

> It looks like the breakage in parsing #'upcase is the result of
> `eshell-parse-external-pipeline' trying to skip over args like '*|' 
> (with the quotes). However, it sees the single-quote in #'upcase and 
> then calls `eshell-parse-literal-quote', resulting in the error message:
>
>    Expecting completion of delimiter ' ...

Here is a patch which has `eshell-parse-external-pipeline' try skipping
over sharp-quoted symbols before trying to skip over single-quoted
strings.

-- 
Sean Whitton

[0001-Fix-input-of-sharp-quoted-symbols-in-Eshell-with-em-.patch (text/x-diff, attachment)]

Message #11 received at 53518 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Sean Whitton <spwhitton <at> spwhitton.name>
Cc: 53518 <at> debbugs.gnu.org, Michael Albinus <michael.albinus <at> gmx.de>,
 Jim Porter <jporterbugs <at> gmail.com>
Subject: Re: bug#53518: 29.0.50; em-extpipe breaks input of sharp-quoted
 Lisp symbols
Date: Tue, 25 Jan 2022 13:26:59 +0100

Sean Whitton <spwhitton <at> spwhitton.name> writes:

> Here is a patch which has `eshell-parse-external-pipeline' try skipping
> over sharp-quoted symbols before trying to skip over single-quoted
> strings.

This leads to the following compilation warning:

  ELC      eshell/em-extpipe.elc
In end of data:
eshell/em-extpipe.el:107:38: Warning: the function `eshell-parse-lisp-argument' is not known to be defined.


-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #14 received at 53518 <at> debbugs.gnu.org (full text, mbox):

From: Sean Whitton <spwhitton <at> spwhitton.name>
To: Lars Ingebrigtsen <larsi <at> gnus.org>
Cc: 53518 <at> debbugs.gnu.org, Michael Albinus <michael.albinus <at> gmx.de>,
 Jim Porter <jporterbugs <at> gmail.com>
Subject: Re: bug#53518: 29.0.50; em-extpipe breaks input of sharp-quoted
 Lisp symbols
Date: Tue, 25 Jan 2022 09:48:16 -0700

[Message part 1 (text/plain, inline)]

Hello,

On Tue 25 Jan 2022 at 01:26pm +01, Lars Ingebrigtsen wrote:

> This leads to the following compilation warning:
>
>   ELC      eshell/em-extpipe.elc
> In end of data:
> eshell/em-extpipe.el:107:38: Warning: the function `eshell-parse-lisp-argument' is not known to be defined.

Apologies for this.  Here is a fixed patch.

-- 
Sean Whitton

[v2-0001-Fix-input-of-sharp-quoted-symbols-in-Eshell-with-.patch (text/x-diff, attachment)]

Message #17 received at 53518 <at> debbugs.gnu.org (full text, mbox):

From: Jim Porter <jporterbugs <at> gmail.com>
To: Sean Whitton <spwhitton <at> spwhitton.name>, 53518 <at> debbugs.gnu.org
Cc: Lars Ingebrigtsen <larsi <at> gnus.org>,
 Michael Albinus <michael.albinus <at> gmx.de>
Subject: Re: bug#53518: 29.0.50; em-extpipe breaks input of sharp-quoted Lisp
 symbols
Date: Tue, 25 Jan 2022 10:14:52 -0800

On 1/25/2022 12:50 AM, Sean Whitton wrote:
> Hello again,
> 
> On Mon 24 Jan 2022 at 06:39pm -08, Jim Porter wrote:
> 
>> I just noticed a small bit of breakage with this. It's no longer
>> possible to refer to Lisp functions in Eshell like so:
>>
>>     #'upcase
>>
>> Eshell explicitly supports this construct (see `eshell-lisp-regexp'),
>> though it doesn't appear to be documented in the manual. Currently, this
>> syntax is only occasionally useful, but I'm working on a patch series
>> where it'll likely become a lot more common. My patches will add support
>> for piping to Lisp functions, so that you can do the following, for example:
>>
>>     $ echo hi | #'upcase
>>     HI
> 
> Out of curiosity, why is there a need for the sharpquote?  Why not just
> 'echo hi | upcase'?  Is it to do with requesting the new piping?

It becomes more relevant with my WIP patches to support piping to Lisp 
functions, but it means something different today too. "upcase" *calls* 
the function `upcase', whereas "#'upcase" evaluates to the function 
object itself. For example:

  $ upcase
  Wrong number of arguments: #<subr upcase>, 0
  $ #'upcase
  upcase

Or, for a slightly different, but more practical example today:

  $ mapcar upcase $(list "foo" "bar")
  Invalid function: "upcase"
  $ mapcar #'upcase $(list "foo" "bar")
  ("FOO" "BAR")

In this case, "upcase" in the first command is just the string "upcase", 
whereas "#'upcase" refers to the function as in the other case. The 
shortest other way I'm aware of to spell that using shell-like 
invocation would be:

  $ mapcar $(quote upcase) $(list "foo" "bar")
  ("FOO" "BAR")

For testing purposes, it would probably also be useful to ensure that 
Lisp syntax works too:

  $ (mapcar #'upcase '("foo" "bar"))
  ("FOO" "BAR")

Just for the sake of completeness, in my WIP patches, "echo hi | 
#'upcase" and "echo hi | upcase" will also do different things, 
following the above precedent. The former pipes the output of echo to 
the function upcase. The latter pipes the output of echo to the *result* 
of calling the function upcase with no arguments. In addition to being 
consistent with how Eshell currently works, this allows you to do things 
like "echo hi | less -N", where "less -N" is evaluated as an Eshell 
command and then returns a pseudo-pipe for echo to connect to.

Message #20 received at 53518 <at> debbugs.gnu.org (full text, mbox):

From: Sean Whitton <spwhitton <at> spwhitton.name>
To: Jim Porter <jporterbugs <at> gmail.com>, 53518 <at> debbugs.gnu.org
Cc: Michael Albinus <michael.albinus <at> gmx.de>
Subject: Re: bug#53518: 29.0.50; em-extpipe breaks input of sharp-quoted
 Lisp symbols
Date: Tue, 25 Jan 2022 13:01:51 -0700

Hello,

On Tue 25 Jan 2022 at 10:14AM -08, Jim Porter wrote:

> It becomes more relevant with my WIP patches to support piping to Lisp
> functions, but it means something different today too.

Indeed.

> Just for the sake of completeness, in my WIP patches, "echo hi |
> #'upcase" and "echo hi | upcase" will also do different things,
> following the above precedent. The former pipes the output of echo to
> the function upcase. The latter pipes the output of echo to the *result*
> of calling the function upcase with no arguments.

Okay, thanks.  I asked because I thought you were only planning support
the former case.

> In addition to being consistent with how Eshell currently works, this
> allows you to do things like "echo hi | less -N", where "less -N" is
> evaluated as an Eshell command and then returns a pseudo-pipe for echo
> to connect to.

It's not quite clear to me how this is consistent with how Eshell
currently works.  When you type "echo hi | cat" is it right to say that
"cat" is evaluated to produce the thing to which the output of the first
command is piped?  Perhaps I'm not thinking broadly enough.

-- 
Sean Whitton

Message #23 received at 53518 <at> debbugs.gnu.org (full text, mbox):

From: Jim Porter <jporterbugs <at> gmail.com>
To: Sean Whitton <spwhitton <at> spwhitton.name>, 53518 <at> debbugs.gnu.org
Cc: Michael Albinus <michael.albinus <at> gmx.de>
Subject: Re: bug#53518: 29.0.50; em-extpipe breaks input of sharp-quoted Lisp
 symbols
Date: Tue, 25 Jan 2022 12:52:36 -0800

On 1/25/2022 12:01 PM, Sean Whitton wrote:
>> In addition to being consistent with how Eshell currently works, this
>> allows you to do things like "echo hi | less -N", where "less -N" is
>> evaluated as an Eshell command and then returns a pseudo-pipe for echo
>> to connect to.
> 
> It's not quite clear to me how this is consistent with how Eshell
> currently works.  When you type "echo hi | cat" is it right to say that
> "cat" is evaluated to produce the thing to which the output of the first
> command is piped?  Perhaps I'm not thinking broadly enough.

In general, I just mean that when running a command "foo" (in a pipeline 
or not), if Eshell finds a Lisp function for "foo", it always calls that 
function. On the other hand, "#'foo" always evaluates to the function 
object itself.

So, since "cat" points to a Lisp function, your summary is pretty close, 
I'd say. However, I think I'd describe it as "cat is evaluated to hook 
up the pipeline into what the user meant". Currently, it works like so:

1. `eshell-parse-command' turns "echo hi | cat" into:
   (eshell-commands
    (progn
      (run-hooks 'eshell-pre-command-hook)
      (catch 'top-level
        (progn
          (eshell-trap-errors
           (eshell-execute-pipeline
            '((eshell-named-command "echo" (list "hi"))
              (eshell-named-command "cat"))))))
      (run-hooks 'eshell-post-command-hook)))

2. Eshell iteratively evaluates that, eventually getting to to the "cat"
   part

3. `eshell-named-command' looks for "cat" and calls the function
   `eshell/cat'

4. `eshell/cat' notices that it's being called in a pipeline and bails
   out, throwing `eshell-replace-command' to convert "cat" into
   "/bin/cat" (or wherever your cat program lives)

5. This gets reevaluated in `eshell-do-eval' and calls
   `eshell-external-command' for "/bin/cat"

6. This calls `eshell-gather-process-output', returning a process
   object with the pipes connected

That is, Eshell evaluates `eshell/cat' (which could do anything, really) 
with the ultimate goal of hooking up the pipeline. Currently that just 
means "call /bin/cat and connect pipes to the external process", but 
`eshell/cat' can do whatever it wants to achieve the result.

For the "pipe to a Lisp function" case, like "echo hi | #'upcase", 
what's actually happening under the hood in my patch it that "#'upcase" 
is evaluated, Eshell sees that the result is a function, and then the 
function is converted into a pseudo-pipe that gets properly hooked up to 
echo.

For a case like "echo hi | less -N", where "less" has a Lisp 
implementation, Eshell evaluates `(eshell/less "-N")', which could 
return a function that gets converted into a pseudo-pipe like above. Or 
(and this is how `eshell/less' will actually work), it returns its own 
pseudo-pipe that's hooked up properly; this case is analogous to 
`eshell-gather-process-output' returning a properly-connected process 
object.

The subtleties of Lisp functions in pipelines will probably be a bit 
easier to understand once I post patches, which should hopefully be this 
week or next. :) The patches work already, but I'm still implementing a 
few more pipeable Eshell commands in Lisp so that I can be sure the core 
implementation is flexible enough to be used for practical cases.

Message #26 received at 53518 <at> debbugs.gnu.org (full text, mbox):

From: Sean Whitton <spwhitton <at> spwhitton.name>
To: Jim Porter <jporterbugs <at> gmail.com>, 53518 <at> debbugs.gnu.org
Cc: Michael Albinus <michael.albinus <at> gmx.de>
Subject: Re: bug#53518: 29.0.50; em-extpipe breaks input of sharp-quoted
 Lisp symbols
Date: Tue, 25 Jan 2022 15:38:58 -0700

Hello,

On Tue 25 Jan 2022 at 12:52PM -08, Jim Porter wrote:

> The subtleties of Lisp functions in pipelines will probably be a bit
> easier to understand once I post patches, which should hopefully be this
> week or next. :) The patches work already, but I'm still implementing a
> few more pipeable Eshell commands in Lisp so that I can be sure the core
> implementation is flexible enough to be used for practical cases.

Thank you for the explanation.  I still don't see how your proposal is a
natural extension of what Eshell already does, to be honest, but I'm
sure things will become clearer once you post your docs and tests, as
you say.

-- 
Sean Whitton

Message #29 received at 53518 <at> debbugs.gnu.org (full text, mbox):

From: Jim Porter <jporterbugs <at> gmail.com>
To: Sean Whitton <spwhitton <at> spwhitton.name>, Lars Ingebrigtsen <larsi <at> gnus.org>
Cc: 53518 <at> debbugs.gnu.org, Michael Albinus <michael.albinus <at> gmx.de>
Subject: Re: bug#53518: 29.0.50; em-extpipe breaks input of sharp-quoted Lisp
 symbols
Date: Tue, 25 Jan 2022 21:38:01 -0800

On 1/25/2022 8:48 AM, Sean Whitton wrote:
> On Tue 25 Jan 2022 at 01:26pm +01, Lars Ingebrigtsen wrote:
> 
>> This leads to the following compilation warning:
>>
>>    ELC      eshell/em-extpipe.elc
>> In end of data:
>> eshell/em-extpipe.el:107:38: Warning: the function `eshell-parse-lisp-argument' is not known to be defined.
> 
> Apologies for this.  Here is a fixed patch.

Thanks for the quick fix, this resolves everything on my end. I applied 
the patch to my local branch for Lisp function pipelining and all the 
tests in test/lisp/eshell/ now pass, including the new ones I'm working on.

Message #32 received at 53518 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Sean Whitton <spwhitton <at> spwhitton.name>
Cc: 53518 <at> debbugs.gnu.org, Michael Albinus <michael.albinus <at> gmx.de>,
 Jim Porter <jporterbugs <at> gmail.com>
Subject: Re: bug#53518: 29.0.50; em-extpipe breaks input of sharp-quoted
 Lisp symbols
Date: Wed, 26 Jan 2022 14:13:18 +0100

Sean Whitton <spwhitton <at> spwhitton.name> writes:

> Apologies for this.  Here is a fixed patch.

Thanks; applied to Emacs 29.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.