GNU bug report logs - #4708
23.1; completion-try-completion adds an extra $: $$HOMj

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 4708 in the body.
You can then email your comments to 4708 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> emacsbugs.donarmstrong.com (full text, mbox):

From: "Drew Adams" <drew.adams <at> oracle.com>
To: <bug-gnu-emacs <at> gnu.org>
Subject: 23.1; completion-try-completion adds an extra $:  $$HOMj
Date: Mon, 12 Oct 2009 12:05:32 -0700

emacs -Q
 
M-: (completion-try-completion "c:/some-dir/$HOMj" nil 16)
 
correctly returns: (c:/some-dir/$HOMEj" . 17)
 
M-: (completion-try-completion "c:/some-dir/$HOMj" nil 17)
 
returns: ("c:/some-dir/$$HOMj" . 18)
 
That doesn't seem correct. Is it correct to have $$ here? If so, can
you please explain it a bit (why)?
 
Also, if this is correct behavior, then please explain this in the doc
string of `completion-try-completion'. The doc string currently says
that STRING, in the return value of (STRING . NEWPOINT), is "the
completed result string". But "result string" cannot be right, if
we're talking about the result of completion. There is no completion
that contains $$HOM.
 
In GNU Emacs 23.1.1 (i386-mingw-nt5.1.2600)
 of 2009-07-29 on SOFT-MJASON
Windowing system distributor `Microsoft Corp.', version 5.1.2600
configured using `configure --with-gcc (4.4)'
 

Message #10 received at submit <at> emacsbugs.donarmstrong.com (full text, mbox):

From: Stefan Monnier <monnier <at> IRO.UMontreal.CA>
To: Drew Adams <drew.adams <at> oracle.com>
Cc: 4708 <at> debbugs.gnu.org, <bug-gnu-emacs <at> gnu.org>
Subject: Re: bug#4708: 23.1; completion-try-completion adds an extra $:  $$HOMj
Date: Tue, 13 Oct 2009 16:13:25 -0400

>>>>> "Drew" == Drew Adams <drew.adams <at> oracle.com> writes:

> emacs -Q
> M-: (completion-try-completion "c:/some-dir/$HOMj" nil 16)
 
> correctly returns: (c:/some-dir/$HOMEj" . 17)
 
> M-: (completion-try-completion "c:/some-dir/$HOMj" nil 17)
 
> returns: ("c:/some-dir/$$HOMj" . 18)
 
> That doesn't seem correct. Is it correct to have $$ here? If so, can
> you please explain it a bit (why)?

File name entry assigns a special meaning to $ for envvars, but in order
to be able to refer to the file "$HOME", it offers an escape system
where you write "$$HOME".  Since Emacs-21 or so, I've made this escape
unnecessary in the case where the $ is used unambiguously (because
there is no envvar of this name), so if you enter "$HOMj", Emacs know
you don't refer to the file name contained in the envvar `HOMj', but to
the file named $HOMj.

The completion code need to do such unescape/reescaping in order to work
properly and it currently re-escapes in a conservative way (i.e. it
re-escapes $HOMj into $$HOMj even tho it's not strictly necessary).

File name completion is designed to result a string appropriate for
substitute-in-file-name, and indeed

   (substitute-in-file-name "c:/some-dir/$$HOMj")
   ->
   "c:/some-dir/$HOMj"

> Also, if this is correct behavior, then please explain this in the doc
> string of `completion-try-completion'.

It can't be described there, because that function applies to any kind
of completion, whereas the above only applies to completions using
read-file-name-internal.


        Stefan

Message #20 received at submit <at> emacsbugs.donarmstrong.com (full text, mbox):

From: "Drew Adams" <drew.adams <at> oracle.com>
To: "'Stefan Monnier'" <monnier <at> IRO.UMontreal.CA>
Cc: <4708 <at> debbugs.gnu.org>, <bug-gnu-emacs <at> gnu.org>
Subject: RE: bug#4708: 23.1; completion-try-completion adds an extra $:  $$HOMj
Date: Tue, 13 Oct 2009 15:20:38 -0700

> > emacs -Q
> > M-: (completion-try-completion "c:/some-dir/$HOMj" nil 16)
> > correctly returns: (c:/some-dir/$HOMEj" . 17)
> > M-: (completion-try-completion "c:/some-dir/$HOMj" nil 17)
> > returns: ("c:/some-dir/$$HOMj" . 18)
>  
> > That doesn't seem correct. Is it correct to have $$ here? If so, can
> > you please explain it a bit (why)?
> 
> File name entry assigns a special meaning to $ for envvars, 
> but in order to be able to refer to the file "$HOME", it offers
> an escape system where you write "$$HOME".  Since Emacs-21 or so,
> I've made this escape unnecessary in the case where the $ is
> used unambiguously (because there is no envvar of this name),
> so if you enter "$HOMj", Emacs know you don't refer to the file
> name contained in the envvar `HOMj', but to
> the file named $HOMj.

But there is no such file either. There is neither an envar that starts with
`$HOMj' nor a file whose name starts with `$HOMj'. No matter how you look at it,
there is no possible completion for `$HOMj'. Based on the doc string, I would
expect nil, to indicate that there is no possible completion.

> The completion code need to do such unescape/reescaping in 
> order to work properly and it currently re-escapes in a
> conservative way (i.e. it re-escapes $HOMj into $$HOMj even
> tho it's not strictly necessary).
> 
> File name completion is designed to result a string appropriate for
> substitute-in-file-name, and indeed
> 
>    (substitute-in-file-name "c:/some-dir/$$HOMj") ->
>    "c:/some-dir/$HOMj"
> 
> > Also, if this is correct behavior, then please explain this 
> > in the doc string of `completion-try-completion'.
> 
> It can't be described there, because that function applies to any kind
> of completion, whereas the above only applies to completions using
> read-file-name-internal.

You describe some of the foibles of the implementation, but you don't really
address the bug. It's not clear whether you intend to do that later or you are
trying to say that there is in fact no bug.

Code that calls `completion-try-completion' does not necessarily know what the
value of the TABLE arg is (i.e. whether it is equivalent to
`read-file-name-internal'), so it does not necessarily know whether this is
file-name completion or not.

`completion-try-completion' needs to have an unambiguous way to indicate that
there is no completion possible for the given input, even if it is the TABLE arg
that ultimately gives it that info. The doc says that nil is the way that
no-possible-completion is indicated.

But in this case nil is not returned. Instead, the return value can only be
interpreted, based on the doc string, as indicating that there is one
completion, "c:/some-dir/$$HOMj" (however that is to be intepreted: as a file
name that includes two `$' or as a file name that includes one `$' followed by
an env var). But there is in fact no legitimate completion.

There is no way that "c:/some-dir/$$HOMj" is a valid completion for
"c:/some-dir/$HOMj" - whether you regard the input as a file name that includes
a `$' or you regard it as an attempt to complete an env var. There simply is no
way that "c:/some-dir/$$HOMj" completes.

You can decide where the bug is: `read-file-name-internal',
`completion-try-completion', or somewhere else, but there is a bug if a value
claimed to represent a successful completion is returned when there is no valid
completion.

Or else there must be some way to tell from that returned value that it in fact
somehow represents "there is no completion" (the same as nil does). In that
case, let me know what that way is, and I'll test the return value using it. And
in that case the doc string also needs to document this how-to-tell info.

IOW, either the return value is correct and somehow indicates that there is no
completion - and in that case how to tell that? Or the return value is
incorrect, because it indicates a completion when there is none. Either way,
there is a bug.

I don't know how to handle this return value. My code currently checks for nil
as the (only) indicator of non-completion, and so it incorrectly interprets this
as successful completion.

Message #30 received at submit <at> emacsbugs.donarmstrong.com (full text, mbox):

From: Stefan Monnier <monnier <at> iro.umontreal.ca>
To: "Drew Adams" <drew.adams <at> oracle.com>
Cc: <4708 <at> debbugs.gnu.org>, <bug-gnu-emacs <at> gnu.org>
Subject: Re: bug#4708: 23.1; completion-try-completion adds an extra $:  $$HOMj
Date: Tue, 13 Oct 2009 22:40:12 -0400

>> File name entry assigns a special meaning to $ for envvars, 
>> but in order to be able to refer to the file "$HOME", it offers
>> an escape system where you write "$$HOME".  Since Emacs-21 or so,
>> I've made this escape unnecessary in the case where the $ is
>> used unambiguously (because there is no envvar of this name),
>> so if you enter "$HOMj", Emacs know you don't refer to the file
>> name contained in the envvar `HOMj', but to
>> the file named $HOMj.

> But there is no such file either. There is neither an envar that
> starts with `$HOMj' nor a file whose name starts with `$HOMj'.
> No matter how you look at it, there is no possible completion for
> `$HOMj'.  Based on the doc string, I would expect nil, to indicate
> that there is no possible completion.

Good point, I missed it, sorry.

That try-completion will return nil when applied to "/foo/bar/$$HOMj",
so basically the $HOMj -> $$HOMj is considered to be a form of completion.

That decision is taken by read-file-name-internal.  Can you try the
patch below to see if it improves the behavior?

>> > Also, if this is correct behavior, then please explain this 
>> > in the doc string of `completion-try-completion'.
>> It can't be described there, because that function applies to any kind
>> of completion, whereas the above only applies to completions using
>> read-file-name-internal.
> You describe some of the foibles of the implementation, but you don't
> really address the bug.

No it's not just a detail of the implementation.  The completion is
split into "completion code" and "completion tables".

> It's not clear whether you intend to do that later or you are trying
> to say that there is in fact no bug.

There is no bug w.r.t the docstring of completion-try-completion not
mentioning anything about $-escaping.

> Code that calls `completion-try-completion' does not necessarily know
> what the value of the TABLE arg is (i.e. whether it is equivalent to
> `read-file-name-internal'), so it does not necessarily know whether
> this is file-name completion or not.

Code that calls completion-try-completion has to assume that if it
returns a string, some expansion took place.  It's as simple as that.

> `completion-try-completion' needs to have an unambiguous way to
> indicate that there is no completion possible for the given input,

There is, it's the nil return value.  Note that some completion tables
may never be able to determine it and may hence never return nil.

Note that a completion table might very well allow completion from /u to
/usr/ even if none of the files in /usr/ are acceptable completion
candidates.  I.e. try-completion may sometimes return a string even if
there is no real valid completion with that prefix.  This is simply
because it may be too costly for the function to verify it (e.g. having
to traverse the whole /usr/ subtree).


        Stefan


--- minibuffer.el.~1.84.~	2009-09-24 10:49:09.000000000 -0400
+++ minibuffer.el	2009-10-13 22:36:42.000000000 -0400
@@ -1078,16 +1078,18 @@
        ((null action)
         (let ((comp (file-name-completion name realdir
                                           read-file-name-predicate)))
-          (if (stringp comp)
+          (cond
+           ((stringp comp)
               ;; Requote the $s before returning the completion.
-              (minibuffer--double-dollars (concat specdir comp))
+            (minibuffer--double-dollars (concat specdir comp)))
+           (comp
             ;; Requote the $s before checking for changes.
             (setq str (minibuffer--double-dollars str))
             (if (string-equal string str)
                 comp
               ;; If there's no real completion, but substitute-in-file-name
               ;; changed the string, then return the new string.
-              str))))
+              str)))))
 
        ((eq action t)
         (let ((all (file-name-all-completions name realdir)))

Message #40 received at submit <at> emacsbugs.donarmstrong.com (full text, mbox):

From: "Drew Adams" <drew.adams <at> oracle.com>
To: "'Stefan Monnier'" <monnier <at> iro.umontreal.ca>
Cc: <4708 <at> debbugs.gnu.org>, <bug-gnu-emacs <at> gnu.org>
Subject: RE: bug#4708: 23.1; completion-try-completion adds an extra $:  $$HOMj
Date: Tue, 13 Oct 2009 20:36:08 -0700

> That try-completion will return nil when applied to "/foo/bar/$$HOMj",
> so basically the $HOMj -> $$HOMj is considered to be a form 
> of completion.

Sorry, I don't follow that. Could you say it differently? And did you mean
try-completion or completion-try-completion? Sorry, I don't know what you mean
here.

> That decision is taken by read-file-name-internal.  Can you try the
> patch below to see if it improves the behavior?

Yes, it seems to fix the problem. Thanks.

> Note that a completion table might very well allow completion 
> from /u to /usr/ even if none of the files in /usr/ are
> acceptable completion candidates.  I.e. try-completion may
> sometimes return a string even if there is no real valid
> completion with that prefix.  This is simply because it may
> be too costly for the function to verify it (e.g. having
> to traverse the whole /usr/ subtree).

I understand.

The problem is not so much what is done and the given trade-offs chosen for a
case like that. The problem is to keep users in control - at the very least
giving them knowledge of what the story is. If the command in question tells
users just what you said, then they can know what to expect and can act en
connaissance de cause.

What's not good is to surprise users or keep them guessing about what's
happening and why. This is a problem with DWIM generally. It's no big deal for
something inconsequential, but it can matter where user actions depend on the
result.

Users need to have a reasonable mental model of the behavior (which need not
reflect how things are actually implemented, of course).

Part of helping them in this respect is providing feedback. I've said before
that partial completion can be surprising and confusing - that's nothing new.
But what's worse is that we slip silently from one kind of completion attempt to
another, and another.

We don't say, "No prefix completions found. Do you want to try now for partial
completions?". After running down an unknown number of completion methods under
the covers, we don't even let the user know at the end which method was
ultimately successful.

The user has no way of knowing what kind of completion was actually used, which
means no good way of determining how the result relates to his input.

This can make him lose confidence in his mental model, his general understanding
of the program, and ultimately himself. We should be empowering users, not
making them feel like they're not in control or have no idea what's going on.

Sure, sometimes it doesn't really matter. You don't need to know the details
about how Google finds what it finds - as long as the results seem sensible to
you most of the time. I believe that with the latest completion enhancements
Emacs users are too often confused and surprised, and that we could do a better
job of helping them understand - at least what happened and why.

Anyway, this bug seems to have been fixed - thx.

Message #50 received at 4708-done <at> emacsbugs.donarmstrong.com (full text, mbox):

From: Stefan Monnier <monnier <at> iro.umontreal.ca>
To: "Drew Adams" <drew.adams <at> oracle.com>
Subject: Re: bug#4708: 23.1; completion-try-completion adds an extra $:  $$HOMj
Date: Wed, 14 Oct 2009 00:02:18 -0400

>> That try-completion will return nil when applied to "/foo/bar/$$HOMj",
>> so basically the $HOMj -> $$HOMj is considered to be a form 
>> of completion.
> Sorry, I don't follow that.

Just what I said: read-file-name-internal considered that changing
"$HOMj" into "$$HOMj" is a form of completion.  Just like hitting TAB
might change Foo into FOO in some completion cases.

> And did you mean try-completion or completion-try-completion?

Makes no difference, it's done in the completion-table, i.e. at
a lower level.


        Stefan

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.