GNU bug report logs - #11339
24.1.50; read-{buffer,file-name}-completion-ignore-case fails on non-ascii

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 11339 in the body.
You can then email your comments to 11339 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Stephen Berman <stephen.berman <at> gmx.net>
To: bug-gnu-emacs <at> gnu.org
Subject: 24.1.50;
	read-{buffer,file-name}-completion-ignore-case fails on non-ascii
Date: Wed, 25 Apr 2012 17:37:38 +0200

-1. cd /tmp; touch bäh bÄh bla
0. emacs -Q
1. M-x customize-option RET read-buffer-completion-ignore-case RET.
   Toggle to "on", set for the current session.
   M-x customize-option RET read-file-name-completion-ignore-case RET.
   Toggle to "on", set for the current session.
2. C-x C-f b TAB => *Completions* buffer contains, as expected: bla bÄh bäh.         
3. Visit files `bla', `bäh' and `bÄh', switch to buffer *scratch* and
   repeat step 2, replacing `C-x C-f' with `C-x b' => same results.
4. C-x C-f bä TAB => Emacs completes the input to "bäh".               
5. TAB => Emacs incorrectly reports: "[Sole completion]".                 
6. C-x C-f bÄ TAB => Emacs completes the input to "bÄh".
7. TAB => Emacs incorrectly reports: "[Sole completion]".                 
8. Switch to buffer *scratch* and repeat step 4-7, replacing `C-x C-f'
   with `C-x b' => same results.


In GNU Emacs 24.1.50.1 (i686-suse-linux-gnu, GTK+ Version 2.24.7)
 of 2012-04-18 on escher
Bzr revision: 107959 cyd <at> gnu.org-20120418070052-e6bm11h52qja62ue
Windowing system distributor `The X.Org Foundation', version 11.0.11004000
Configured using:
 `configure '--without-toolkit-scroll-bars' 'CFLAGS=-g''

Important settings:
  value of $LC_ALL: nil
  value of $LC_COLLATE: nil
  value of $LC_CTYPE: nil
  value of $LC_MESSAGES: nil
  value of $LC_MONETARY: nil
  value of $LC_NUMERIC: nil
  value of $LC_TIME: nil
  value of $LANG: en_US.UTF-8
  value of $XMODIFIERS: @im=local
  locale-coding-system: utf-8-unix
  default enable-multibyte-characters: t

Message #8 received at 11339 <at> debbugs.gnu.org (full text, mbox):

From: Stephen Berman <stephen.berman <at> gmx.net>
To: 11339 <at> debbugs.gnu.org
Subject: Re: bug#11339: 24.1.50;
 read-{buffer,file-name}-completion-ignore-case fails on non-ascii
Date: Tue, 09 Jul 2013 00:08:29 +0200

On Wed, 25 Apr 2012 17:37:38 +0200 Stephen Berman <stephen.berman <at> gmx.net> wrote:

> -1. cd /tmp; touch bäh bÄh bla
> 0. emacs -Q
> 1. M-x customize-option RET read-buffer-completion-ignore-case RET.
>    Toggle to "on", set for the current session.
>    M-x customize-option RET read-file-name-completion-ignore-case RET.
>    Toggle to "on", set for the current session.
> 2. C-x C-f b TAB => *Completions* buffer contains, as expected: bla bÄh bäh.         
> 3. Visit files `bla', `bäh' and `bÄh', switch to buffer *scratch* and
>    repeat step 2, replacing `C-x C-f' with `C-x b' => same results.
> 4. C-x C-f bä TAB => Emacs completes the input to "bäh".               
> 5. TAB => Emacs incorrectly reports: "[Sole completion]".                 
> 6. C-x C-f bÄ TAB => Emacs completes the input to "bÄh".
> 7. TAB => Emacs incorrectly reports: "[Sole completion]".                 
> 8. Switch to buffer *scratch* and repeat step 4-7, replacing `C-x C-f'
>    with `C-x b' => same results.

This bug still exists in the current trunk.  I have an additional
observation, which I don't know if I just overlooked in my OP or if
there's been a code change since then that's responsible for this
observation: if only one of the files `bäh' and `bÄh' is being visited,
then `C-x b' completes correctly regardless of the case of the input;
i.e., if only `bÄh' is in a buffer `C-x b bä TAB' completes to it, and
likewise, if only `bäh' is in a buffer `C-x b bÄ TAB' completes to it.
But if only one of the files `bäh' and `bÄh' exists, `C-x C-f' still
only completes if the case of the input matches the case of the file
name, as above.

Here is part of the call chain of find-file:

find-file (files.el) -> find-file-read-args -> read-file-name
(minibuffer.el) -> read-file-name-default -> completing-read ->
read-file-name-internal -> completion-table-in-turn ->
completion--file-name-table -> completion-file-name-table ->
file-name-completion (dired.c) -> file_name_completion ->
fast_string_match_ignore_case (search.c) -> compile_pattern with argument
Vascii_canon_table

The argument Vascii_canon_table looks suspicious, but maybe it's already
been overridden earlier, though I don't see where; or maybe I just don't
understand its import here.  Anyway, the call chain of switch-to-buffer
is quite different:

switch-to-buffer (window.el) -> read-buffer-to-switch ->
internal-complete-buffer-except (minibuffer.el) -> internal-complete-buffer
(minibuf.c) -> (try-completion | all-completions | test-completion) ->
Fcompare_strings

(file_name_completion also calls Fcompare_strings later, but there's no
use of Vascii_canon_table there.)

I don't know how to debug this.

Steve Berman

Message #11 received at 11339 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Kangas <stefan <at> marxist.se>
To: Stephen Berman <stephen.berman <at> gmx.net>
Cc: 11339 <at> debbugs.gnu.org
Subject: Re: 24.1.50; read-{buffer,file-name}-completion-ignore-case fails
 on non-ascii
Date: Fri, 01 Nov 2019 20:57:30 +0100

found 11339 27.0.50
thanks

Stephen Berman <stephen.berman <at> gmx.net> writes:

> -1. cd /tmp; touch bäh bÄh bla
> 0. emacs -Q
> 1. M-x customize-option RET read-buffer-completion-ignore-case RET.
>    Toggle to "on", set for the current session.
>    M-x customize-option RET read-file-name-completion-ignore-case RET.
>    Toggle to "on", set for the current session.
> 2. C-x C-f b TAB => *Completions* buffer contains, as expected: bla bÄh bäh.         
> 3. Visit files `bla', `bäh' and `bÄh', switch to buffer *scratch* and
>    repeat step 2, replacing `C-x C-f' with `C-x b' => same results.
> 4. C-x C-f bä TAB => Emacs completes the input to "bäh".               
> 5. TAB => Emacs incorrectly reports: "[Sole completion]".                 
> 6. C-x C-f bÄ TAB => Emacs completes the input to "bÄh".
> 7. TAB => Emacs incorrectly reports: "[Sole completion]".                 
> 8. Switch to buffer *scratch* and repeat step 4-7, replacing `C-x C-f'
>    with `C-x b' => same results.

I can reproduce this on current master using the simpler recipe:

0. emacs -Q
1. Evaluate (setq read-buffer-completion-ignore-case t)
2. C-x b bäh RET
3. C-x b bÄh RET
4. C-x b bä TAB TAB

Result: Completes to "bäh" on first TAB, and on second TAB [Sole
completion] is displayed in minibuffer.

Expected result: The "*Completion*" buffer is displayed.

Best regards,
Stefan Kangas

Message #16 received at 11339 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Stefan Kangas <stefan <at> marxist.se>
Cc: 11339 <at> debbugs.gnu.org, stephen.berman <at> gmx.net
Subject: Re: bug#11339: 24.1.50;
 read-{buffer,file-name}-completion-ignore-case fails on non-ascii
Date: Fri, 01 Nov 2019 22:20:42 +0200

> From: Stefan Kangas <stefan <at> marxist.se>
> Date: Fri, 01 Nov 2019 20:57:30 +0100
> Cc: 11339 <at> debbugs.gnu.org
> 
> > 0. emacs -Q
> > 1. M-x customize-option RET read-buffer-completion-ignore-case RET.
> >    Toggle to "on", set for the current session.
> >    M-x customize-option RET read-file-name-completion-ignore-case RET.
> >    Toggle to "on", set for the current session.
> > 2. C-x C-f b TAB => *Completions* buffer contains, as expected: bla bÄh bäh.         
> > 3. Visit files `bla', `bäh' and `bÄh', switch to buffer *scratch* and
> >    repeat step 2, replacing `C-x C-f' with `C-x b' => same results.
> > 4. C-x C-f bä TAB => Emacs completes the input to "bäh".               
> > 5. TAB => Emacs incorrectly reports: "[Sole completion]".                 
> > 6. C-x C-f bÄ TAB => Emacs completes the input to "bÄh".
> > 7. TAB => Emacs incorrectly reports: "[Sole completion]".                 
> > 8. Switch to buffer *scratch* and repeat step 4-7, replacing `C-x C-f'
> >    with `C-x b' => same results.
> 
> I can reproduce this on current master using the simpler recipe:
> 
> 0. emacs -Q
> 1. Evaluate (setq read-buffer-completion-ignore-case t)
> 2. C-x b bäh RET
> 3. C-x b bÄh RET
> 4. C-x b bä TAB TAB
> 
> Result: Completes to "bäh" on first TAB, and on second TAB [Sole
> completion] is displayed in minibuffer.
> 
> Expected result: The "*Completion*" buffer is displayed.

I think your expectations are incorrect in this case.  Try having
buffer names that differ (not in letter-case) in the 3rd letter.

What I see here is entirely expected and reasonable.  I see no bug.

Message #19 received at 11339 <at> debbugs.gnu.org (full text, mbox):

From: Stephen Berman <stephen.berman <at> gmx.net>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 11339 <at> debbugs.gnu.org, Stefan Kangas <stefan <at> marxist.se>
Subject: Re: bug#11339: 24.1.50;
 read-{buffer,file-name}-completion-ignore-case fails on non-ascii
Date: Fri, 01 Nov 2019 23:37:54 +0100

On Fri, 01 Nov 2019 22:20:42 +0200 Eli Zaretskii <eliz <at> gnu.org> wrote:

>> From: Stefan Kangas <stefan <at> marxist.se>
>> Date: Fri, 01 Nov 2019 20:57:30 +0100
>> Cc: 11339 <at> debbugs.gnu.org
>> 
>> > 0. emacs -Q
>> > 1. M-x customize-option RET read-buffer-completion-ignore-case RET.
>> >    Toggle to "on", set for the current session.
>> >    M-x customize-option RET read-file-name-completion-ignore-case RET.
>> >    Toggle to "on", set for the current session.
>> > 2. C-x C-f b TAB => *Completions* buffer contains, as expected: bla bÄh
>> > bäh.
>> > 3. Visit files `bla', `bäh' and `bÄh', switch to buffer *scratch* and
>> >    repeat step 2, replacing `C-x C-f' with `C-x b' => same results.
>> > 4. C-x C-f bä TAB => Emacs completes the input to "bäh".               
>> > 5. TAB => Emacs incorrectly reports: "[Sole completion]".                 
>> > 6. C-x C-f bÄ TAB => Emacs completes the input to "bÄh".
>> > 7. TAB => Emacs incorrectly reports: "[Sole completion]".                 
>> > 8. Switch to buffer *scratch* and repeat step 4-7, replacing `C-x C-f'
>> >    with `C-x b' => same results.
>> 
>> I can reproduce this on current master using the simpler recipe:
>> 
>> 0. emacs -Q
>> 1. Evaluate (setq read-buffer-completion-ignore-case t)
>> 2. C-x b bäh RET
>> 3. C-x b bÄh RET
>> 4. C-x b bä TAB TAB
>> 
>> Result: Completes to "bäh" on first TAB, and on second TAB [Sole
>> completion] is displayed in minibuffer.
>> 
>> Expected result: The "*Completion*" buffer is displayed.
>
> I think your expectations are incorrect in this case.  Try having
> buffer names that differ (not in letter-case) in the 3rd letter.
>
> What I see here is entirely expected and reasonable.  I see no bug.

I think the expectation I had about the variables
read-{buffer,file-name}-completion-ignore-case was that they allow
getting information with less effort, in that by typing either a lower
or an upper case character, you would see names that have either or both
cases.  And that is apparently what I (thought I) had experienced with
ASCII characters but not with non-ASCII characters, which I assume is
what prompted my OP.  Unfortunately, the recipe in my OP did not include
the `bah' and `bAh' cases as a sanity check, and now I find the same
behavior with them as reported above for the non-ASCII cases.  So it
seems either the handling of the ASCII cases has changed since my OP, or
I was mistaken at the time in thinking there was a difference.  I also
followed up my OP more than a year later, reporting that the bug still
existed and adding a further observation
(https://lists.gnu.org/archive/html/bug-gnu-emacs/2013-07/msg00340.html).
And regarding the latter, on recent master that does indeed show a
difference between ASCII and non-ASCII characters:

0. touch /tmp/{bah,bAG,bäh,bÄg}
1. emacs-master -Q --eval '(setq read-buffer-completion-ignore-case t read-file-name-completion-ignore-case t)' /tmp/{bah,bAg,bäh,bÄg}
2. C-x b *scratch*
3a. C-x b ba TAB
   => *Completions* pops up showing `bAg' and `bah'
3b. C-x b bA TAB
   => *Completions* pops up showing `bAg' and `bah'
4a. C-g C-x b bä TAB
   => *Completions* pops up showing `bÄg' and `bäh'
4b. C-g C-x b bÄ TAB
   => *Completions* pops up showing `bÄg' and `bäh'
5a. C-g C-x C-f /tmp/ba TAB
   => *Completions* pops up showing `bAg' and `bah'
5b. C-g C-x C-f /tmp/bA TAB
   => *Completions* pops up showing `bAg' and `bah'
6a. C-g C-x C-f /tmp/bä TAB
  => completes to /tmp/bäh
6a. C-g C-x C-f /tmp/bÄ TAB
  => completes to /tmp/bÄg

Steve Berman

Message #24 received at 11339 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Stephen Berman <stephen.berman <at> gmx.net>
Cc: 11339 <at> debbugs.gnu.org, stefan <at> marxist.se
Subject: Re: bug#11339: 24.1.50;
 read-{buffer,file-name}-completion-ignore-case fails on non-ascii
Date: Sat, 02 Nov 2019 09:11:54 +0200

> From: Stephen Berman <stephen.berman <at> gmx.net>
> Cc: Stefan Kangas <stefan <at> marxist.se>,  11339 <at> debbugs.gnu.org
> Date: Fri, 01 Nov 2019 23:37:54 +0100
> 
> I think the expectation I had about the variables
> read-{buffer,file-name}-completion-ignore-case was that they allow
> getting information with less effort, in that by typing either a lower
> or an upper case character, you would see names that have either or both
> cases.  And that is apparently what I (thought I) had experienced with
> ASCII characters but not with non-ASCII characters, which I assume is
> what prompted my OP.  Unfortunately, the recipe in my OP did not include
> the `bah' and `bAh' cases as a sanity check, and now I find the same
> behavior with them as reported above for the non-ASCII cases.  So it
> seems either the handling of the ASCII cases has changed since my OP, or
> I was mistaken at the time in thinking there was a difference.  I also
> followed up my OP more than a year later, reporting that the bug still
> existed and adding a further observation
> (https://lists.gnu.org/archive/html/bug-gnu-emacs/2013-07/msg00340.html).
> And regarding the latter, on recent master that does indeed show a
> difference between ASCII and non-ASCII characters:
> 
> 0. touch /tmp/{bah,bAG,bäh,bÄg}
> 1. emacs-master -Q --eval '(setq read-buffer-completion-ignore-case t read-file-name-completion-ignore-case t)' /tmp/{bah,bAg,bäh,bÄg}
> 2. C-x b *scratch*
> 3a. C-x b ba TAB
>    => *Completions* pops up showing `bAg' and `bah'
> 3b. C-x b bA TAB
>    => *Completions* pops up showing `bAg' and `bah'
> 4a. C-g C-x b bä TAB
>    => *Completions* pops up showing `bÄg' and `bäh'
> 4b. C-g C-x b bÄ TAB
>    => *Completions* pops up showing `bÄg' and `bäh'
> 5a. C-g C-x C-f /tmp/ba TAB
>    => *Completions* pops up showing `bAg' and `bah'
> 5b. C-g C-x C-f /tmp/bA TAB
>    => *Completions* pops up showing `bAg' and `bah'
> 6a. C-g C-x C-f /tmp/bä TAB
>   => completes to /tmp/bäh
> 6a. C-g C-x C-f /tmp/bÄ TAB
>   => completes to /tmp/bÄg

I'm confused wrt what you are saying here.  Your original report
seemed to be about read-buffer-completion-ignore-case not working as
expected with non-ASCII buffer names.  Are you now saying that the
problem is actually with read-file-name-completion-ignore-case
instead?  Or are you saying something else?

(May I suggest in the future to state explicitly the conclusion(s)
from the evidence you present, so as to make the communications more
efficient?  TIA)

Message #27 received at 11339 <at> debbugs.gnu.org (full text, mbox):

From: Stephen Berman <stephen.berman <at> gmx.net>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 11339 <at> debbugs.gnu.org, stefan <at> marxist.se
Subject: Re: bug#11339: 24.1.50;
 read-{buffer,file-name}-completion-ignore-case fails on non-ascii
Date: Sat, 02 Nov 2019 16:09:31 +0100

On Sat, 02 Nov 2019 09:11:54 +0200 Eli Zaretskii <eliz <at> gnu.org> wrote:

>> From: Stephen Berman <stephen.berman <at> gmx.net>
>> Cc: Stefan Kangas <stefan <at> marxist.se>,  11339 <at> debbugs.gnu.org
>> Date: Fri, 01 Nov 2019 23:37:54 +0100
>> 
>> I think the expectation I had about the variables
>> read-{buffer,file-name}-completion-ignore-case was that they allow
>> getting information with less effort, in that by typing either a lower
>> or an upper case character, you would see names that have either or both
>> cases.  And that is apparently what I (thought I) had experienced with
>> ASCII characters but not with non-ASCII characters, which I assume is
>> what prompted my OP.  Unfortunately, the recipe in my OP did not include
>> the `bah' and `bAh' cases as a sanity check, and now I find the same
>> behavior with them as reported above for the non-ASCII cases.  So it
>> seems either the handling of the ASCII cases has changed since my OP, or
>> I was mistaken at the time in thinking there was a difference.  I also
>> followed up my OP more than a year later, reporting that the bug still
>> existed and adding a further observation
>> (https://lists.gnu.org/archive/html/bug-gnu-emacs/2013-07/msg00340.html).
>> And regarding the latter, on recent master that does indeed show a
>> difference between ASCII and non-ASCII characters:
>> 
>> 0. touch /tmp/{bah,bAG,bäh,bÄg}
>> 1. emacs-master -Q --eval '(setq read-buffer-completion-ignore-case t read-file-name-completion-ignore-case t)' /tmp/{bah,bAg,bäh,bÄg}
>> 2. C-x b *scratch*
>> 3a. C-x b ba TAB
>>    => *Completions* pops up showing `bAg' and `bah'
>> 3b. C-x b bA TAB
>>    => *Completions* pops up showing `bAg' and `bah'
>> 4a. C-g C-x b bä TAB
>>    => *Completions* pops up showing `bÄg' and `bäh'
>> 4b. C-g C-x b bÄ TAB
>>    => *Completions* pops up showing `bÄg' and `bäh'
>> 5a. C-g C-x C-f /tmp/ba TAB
>>    => *Completions* pops up showing `bAg' and `bah'
>> 5b. C-g C-x C-f /tmp/bA TAB
>>    => *Completions* pops up showing `bAg' and `bah'
>> 6a. C-g C-x C-f /tmp/bä TAB
>>   => completes to /tmp/bäh
>> 6a. C-g C-x C-f /tmp/bÄ TAB
>>   => completes to /tmp/bÄg
>
> I'm confused wrt what you are saying here.  Your original report
> seemed to be about read-buffer-completion-ignore-case not working as
> expected with non-ASCII buffer names.  Are you now saying that the
> problem is actually with read-file-name-completion-ignore-case
> instead?  Or are you saying something else?

There are two issues: what was reported in my OP, and what was in my
followup more than a year later.

Concerning my OP, I reported what I took to be a bug in how completion
works on names containing non-ASCII characters when either
read-buffer-completion-ignore-case or
read-file-name-completion-ignore-case (or both) are non-nil.  My
expectation was that with e.g. the names `bäh' and `bÄh' typing either
`bä' or `bÄ' and then TAB should shows both names (for both buffer and
file name completion), but in both cases TAB only completed to `bäh' and
`bÄh', respectively.  Since the OP explicitly says this is a problem
with non-ASCII characters, I assume I thought that with ASCII-only
characters the behavior was different, i.e., with names `bah' and `bAh',
typing either `ba' or `bA' followed by TAB did show both names as
completions.  However, what I now see is that only the respective
completions `bah' and `bAh' are shown after TAB.  I usually try to test
what I consider to be all relevant cases before making a bug report, and
if I did so in this case, that means completion with ASCII-only
characters has changed since my OP, and if so, this seems to me to be a
regression, since I had regarded similar behavior with non-ASCII
characters as bug; but since I failed to include the ASCII-only cases in
the bug report, perhaps I was mistaken and the ASCII-only cases were
always like they are now.  In any case, though the current behavior is
consistent between ASCII-only and non-ASCII characters (unlike what I
thought was the case when I made my OP), so I suppose it may be
considered not to be a bug, it is not what I would prefer.

Concerning my followup to my OP, that involved another difference I
observed in completion behavior between buffer names and file names with
non-ASCII characters when *-completion-ignore-case is non-nil, and
that's what the recipe above shows in more detail than in my followup,
which didn't explicitly contrast the behavior with ASCII-only characters
(the above recipe also uses names that differ beyond the case
differences, to avoid the question of expectations where only case
differs): with buffer names, all completions are shown regardless of the
case of the input, but with file names, completion is done according to
the case of the input, i.e. read-file-name-completion-ignore-case
appears to have no effect (again, only when the names contain non-ASCII
characters).  This seems to be a clear bug.

> (May I suggest in the future to state explicitly the conclusion(s)
> from the evidence you present, so as to make the communications more
> efficient?  TIA)

I'll try to, and I hope the above is now clear enough.

Steve Berman

Message #30 received at 11339 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Stephen Berman <stephen.berman <at> gmx.net>
Cc: 11339 <at> debbugs.gnu.org, stefan <at> marxist.se
Subject: Re: bug#11339: 24.1.50;
 read-{buffer,file-name}-completion-ignore-case fails on non-ascii
Date: Sun, 03 Nov 2019 18:16:01 +0200

> From: Stephen Berman <stephen.berman <at> gmx.net>
> Cc: stefan <at> marxist.se,  11339 <at> debbugs.gnu.org
> Date: Sat, 02 Nov 2019 16:09:31 +0100
> 
> Concerning my followup to my OP, that involved another difference I
> observed in completion behavior between buffer names and file names with
> non-ASCII characters when *-completion-ignore-case is non-nil, and
> that's what the recipe above shows in more detail than in my followup,
> which didn't explicitly contrast the behavior with ASCII-only characters
> (the above recipe also uses names that differ beyond the case
> differences, to avoid the question of expectations where only case
> differs): with buffer names, all completions are shown regardless of the
> case of the input, but with file names, completion is done according to
> the case of the input, i.e. read-file-name-completion-ignore-case
> appears to have no effect (again, only when the names contain non-ASCII
> characters).  This seems to be a clear bug.

Yes, it was a bug, and a very old one: our case-insensitive comparison
of file names worked by bytes, which is only TRT for pure-ASCII
strings.

I hope I fixed this now, please try the latest master.

Message #33 received at 11339 <at> debbugs.gnu.org (full text, mbox):

From: Stephen Berman <stephen.berman <at> gmx.net>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 11339 <at> debbugs.gnu.org, stefan <at> marxist.se
Subject: Re: bug#11339: 24.1.50;
 read-{buffer,file-name}-completion-ignore-case fails on non-ascii
Date: Mon, 04 Nov 2019 14:39:25 +0100

[Message part 1 (text/plain, inline)]

On Sun, 03 Nov 2019 18:16:01 +0200 Eli Zaretskii <eliz <at> gnu.org> wrote:

>> From: Stephen Berman <stephen.berman <at> gmx.net>
>> Cc: stefan <at> marxist.se,  11339 <at> debbugs.gnu.org
>> Date: Sat, 02 Nov 2019 16:09:31 +0100
>>
>> Concerning my followup to my OP, that involved another difference I
>> observed in completion behavior between buffer names and file names with
>> non-ASCII characters when *-completion-ignore-case is non-nil, and
>> that's what the recipe above shows in more detail than in my followup,
>> which didn't explicitly contrast the behavior with ASCII-only characters
>> (the above recipe also uses names that differ beyond the case
>> differences, to avoid the question of expectations where only case
>> differs): with buffer names, all completions are shown regardless of the
>> case of the input, but with file names, completion is done according to
>> the case of the input, i.e. read-file-name-completion-ignore-case
>> appears to have no effect (again, only when the names contain non-ASCII
>> characters).  This seems to be a clear bug.
>
> Yes, it was a bug, and a very old one: our case-insensitive comparison
> of file names worked by bytes, which is only TRT for pure-ASCII
> strings.
>
> I hope I fixed this now, please try the latest master.

I've now updated and, the above bug is now fixed; thanks.

However, there seems to be some unintended fallout:

0. emacs -Q
1. M-: (setq read-buffer-completion-ignore-case t read-file-name-completion-ignore-case t)
2. M-x cd RET RET => args-out-of-range error (backtrace attached)

This only seems to happen when trying to cd to a non-empty directory.
Judging by the backtrace, it also seems to be the same as bug#38056,
except that I cannot reproduce the error initially using the recipe of
that bug, but only with the one I gave above.  But once the error
occurs, all subsequent uses of TAB or RET for completion appear to raise
the error.

Steve Berman

[Message part 2 (text/plain, attachment)]

Message #38 received at 11339-done <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Stephen Berman <stephen.berman <at> gmx.net>
Cc: 11339-done <at> debbugs.gnu.org, stefan <at> marxist.se
Subject: Re: bug#11339: 24.1.50;
 read-{buffer,file-name}-completion-ignore-case fails on non-ascii
Date: Mon, 04 Nov 2019 18:21:31 +0200

> From: Stephen Berman <stephen.berman <at> gmx.net>
> Cc: stefan <at> marxist.se,  11339 <at> debbugs.gnu.org
> Date: Mon, 04 Nov 2019 14:39:25 +0100
> 
> > I hope I fixed this now, please try the latest master.
> 
> I've now updated and, the above bug is now fixed; thanks.

Thanks, I'm closing this bug.

> However, there seems to be some unintended fallout:
> 
> 0. emacs -Q
> 1. M-: (setq read-buffer-completion-ignore-case t read-file-name-completion-ignore-case t)
> 2. M-x cd RET RET => args-out-of-range error (backtrace attached)

This is bug#38056, which I just fixed.  It was caused by a thinko on
my part.

Message #46 received at 11339 <at> debbugs.gnu.org (full text, mbox):

From: Stephen Berman <stephen.berman <at> gmx.net>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 11339 <at> debbugs.gnu.org, stefan <at> marxist.se
Subject: Re: bug#11339: 24.1.50;
 read-{buffer,file-name}-completion-ignore-case fails on non-ascii
Date: Mon, 04 Nov 2019 17:22:30 +0100

On Mon, 04 Nov 2019 14:39:25 +0100 Stephen Berman <stephen.berman <at> gmx.net> wrote:

> On Sun, 03 Nov 2019 18:16:01 +0200 Eli Zaretskii <eliz <at> gnu.org> wrote:
>
>>> From: Stephen Berman <stephen.berman <at> gmx.net>
>>> Cc: stefan <at> marxist.se,  11339 <at> debbugs.gnu.org
>>> Date: Sat, 02 Nov 2019 16:09:31 +0100
>>> 
>>> Concerning my followup to my OP, that involved another difference I
>>> observed in completion behavior between buffer names and file names with
>>> non-ASCII characters when *-completion-ignore-case is non-nil, and
>>> that's what the recipe above shows in more detail than in my followup,
>>> which didn't explicitly contrast the behavior with ASCII-only characters
>>> (the above recipe also uses names that differ beyond the case
>>> differences, to avoid the question of expectations where only case
>>> differs): with buffer names, all completions are shown regardless of the
>>> case of the input, but with file names, completion is done according to
>>> the case of the input, i.e. read-file-name-completion-ignore-case
>>> appears to have no effect (again, only when the names contain non-ASCII
>>> characters).  This seems to be a clear bug.
>>
>> Yes, it was a bug, and a very old one: our case-insensitive comparison
>> of file names worked by bytes, which is only TRT for pure-ASCII
>> strings.
>>
>> I hope I fixed this now, please try the latest master.
>
> I've now updated and, the above bug is now fixed; thanks.

I'd like to return to the bevahior I originally reported.  I think my
previous recapitulation of it was inaccurate, so here are explicit
recipes comparing current emacs-26 (representing the situation prior to
your fix) and current emacs-master (including your fix).

Add test files (crucially differing only in case and "asciiness"):

0. $ touch /tmp/{bah,bAh,bäh,bÄh}

Before your fix:

1. $ emacs-26 -Q  --eval '(setq read-buffer-completion-ignore-case tread-file-name-completion-ignore-case t)' /tmp/{bah,bAh,bäh,bÄh}
(*Buffer List* show the buffers bah,bAh,bäh,bÄh)
2. C-x b *scratch*
3a. C-x b ba TAB
   => completes to bah and after TAB displays [Sole completion]
3b. C-g C-x b bA TAB
   => completes to bAh and after TAB displays [Sole completion]
4a. C-g C-x b bä TAB
   => completes to bäh and after TAB displays [Sole completion]
4b. C-g C-x b bÄ TAB
   => completes to bÄh and after TAB displays [Sole completion]
5a. C-g C-x C-f /tmp/ba TAB
   => completes to bah and after TAB displays [Complete, but not unique]
      and *Completions* pops up showing `bAh' and `bah'
5b. C-g C-x C-f /tmp/bA TAB
   => completes to bAh and after TAB displays [Complete, but not unique]
      and *Completions* pops up showing `bAh' and `bah'
6a. C-g C-x C-f /tmp/bä TAB
   => completes to /tmp/bäh and after TAB displays [Sole completion]
6a. C-g C-x C-f /tmp/bÄ TAB
   => completes to /tmp/bÄh and after TAB displays [Sole completion]

After your fix:

1. $ emacs-master -Q  --eval '(setq read-buffer-completion-ignore-case tread-file-name-completion-ignore-case t)' /tmp/{bah,bAh,bäh,bÄh}
(*Buffer List* show the buffers bah,bAh,bäh,bÄh)
2. C-x b *scratch*
3a. C-x b ba TAB
   => completes to bah and after TAB displays [Sole completion]
3b. C-g C-x b bA TAB
   => completes to bAh and after TAB displays [Sole completion]
4a. C-g C-x b bä TAB
   => completes to bäh and after TAB displays [Sole completion]
4b. C-g C-x b bÄ TAB
   => completes to bÄh and after TAB displays [Sole completion]
5a. C-g C-x C-f /tmp/ba TAB
   => completes to bah and after TAB displays [Complete, but not unique]
      and *Completions* pops up showing `bAh' and `bah'
5b. C-g C-x C-f /tmp/bA TAB
   => completes to bAh and after TAB displays [Complete, but not unique]
      and *Completions* pops up showing `bAh' and `bah'
6a. C-g C-x C-f /tmp/bä TAB
   => completes to /tmp/bäh and after TAB displays [Complete, but not unique]
      and *Completions* pops up showing `bAh' and `bah'
6a. C-g C-x C-f /tmp/bÄ TAB
   => completes to /tmp/bÄh and after TAB displays [Complete, but not unique]
      and *Completions* pops up showing `bAh' and `bah'

So, before your fix file name completion involving non-ASCII strings
behaved the same as buffer name completion involving both ASCII-only and
non-ASCII strings, but file name completion involving ASCII-only strings
differed.  Now, after your fix, file name completion involving non-ASCII
strings works the same as file name completion involving ASCII-only
strings did before your fix and still does after, but all the cases of
file name completion differ from the corresponding cases of buffer name
completion.  I would prefer buffer name completion to work the way file
name completion uniformly works after your fix.

Steve Berman

Message #49 received at 11339 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Stephen Berman <stephen.berman <at> gmx.net>
Cc: 11339 <at> debbugs.gnu.org, stefan <at> marxist.se
Subject: Re: bug#11339: 24.1.50;
 read-{buffer,file-name}-completion-ignore-case fails on non-ascii
Date: Mon, 04 Nov 2019 19:12:03 +0200

> From: Stephen Berman <stephen.berman <at> gmx.net>
> Cc: stefan <at> marxist.se,  11339 <at> debbugs.gnu.org
> Date: Mon, 04 Nov 2019 17:22:30 +0100
> 
> After your fix:
> 
> 1. $ emacs-master -Q  --eval '(setq read-buffer-completion-ignore-case tread-file-name-completion-ignore-case t)' /tmp/{bah,bAh,bäh,bÄh}
> (*Buffer List* show the buffers bah,bAh,bäh,bÄh)
> 2. C-x b *scratch*
> 3a. C-x b ba TAB
>    => completes to bah and after TAB displays [Sole completion]
> 3b. C-g C-x b bA TAB
>    => completes to bAh and after TAB displays [Sole completion]
> 4a. C-g C-x b bä TAB
>    => completes to bäh and after TAB displays [Sole completion]
> 4b. C-g C-x b bÄ TAB
>    => completes to bÄh and after TAB displays [Sole completion]
> 5a. C-g C-x C-f /tmp/ba TAB
>    => completes to bah and after TAB displays [Complete, but not unique]
>       and *Completions* pops up showing `bAh' and `bah'
> 5b. C-g C-x C-f /tmp/bA TAB
>    => completes to bAh and after TAB displays [Complete, but not unique]
>       and *Completions* pops up showing `bAh' and `bah'
> 6a. C-g C-x C-f /tmp/bä TAB
>    => completes to /tmp/bäh and after TAB displays [Complete, but not unique]
>       and *Completions* pops up showing `bAh' and `bah'
> 6a. C-g C-x C-f /tmp/bÄ TAB
>    => completes to /tmp/bÄh and after TAB displays [Complete, but not unique]
>       and *Completions* pops up showing `bAh' and `bah'
> 
> So, before your fix file name completion involving non-ASCII strings
> behaved the same as buffer name completion involving both ASCII-only and
> non-ASCII strings, but file name completion involving ASCII-only strings
> differed.  Now, after your fix, file name completion involving non-ASCII
> strings works the same as file name completion involving ASCII-only
> strings did before your fix and still does after, but all the cases of
> file name completion differ from the corresponding cases of buffer name
> completion.  I would prefer buffer name completion to work the way file
> name completion uniformly works after your fix.

"You are in a maze of twisty little passages, all alike."

(I'm confused by a dozen of similar examples which don't say which
result is deemed to be correct and which isn't.)

Previously, you said that buffer-name completion works correctly, so I
never looked at that (which explains why there's no change in that
behavior before and after my changes).  The changes I made are in code
that is used only for completing file names, which again explains why
completion of buffer names didn't change.

Now you are saying that buffer-name completion works incorrectly?  Or
are you saying something else?  I'd like to be sure what is it that
you consider in the need of fixing, and why.

Thanks.

Message #52 received at 11339 <at> debbugs.gnu.org (full text, mbox):

From: Stephen Berman <stephen.berman <at> gmx.net>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 11339 <at> debbugs.gnu.org, stefan <at> marxist.se
Subject: Re: bug#11339: 24.1.50;
 read-{buffer,file-name}-completion-ignore-case fails on non-ascii
Date: Mon, 04 Nov 2019 19:02:15 +0100

On Mon, 04 Nov 2019 19:12:03 +0200 Eli Zaretskii <eliz <at> gnu.org> wrote:

>> From: Stephen Berman <stephen.berman <at> gmx.net>
>> Cc: stefan <at> marxist.se,  11339 <at> debbugs.gnu.org
>> Date: Mon, 04 Nov 2019 17:22:30 +0100
>> 
>> After your fix:
>> 
>> 1. $ emacs-master -Q  --eval '(setq read-buffer-completion-ignore-case tread-file-name-completion-ignore-case t)' /tmp/{bah,bAh,bäh,bÄh}
>> (*Buffer List* show the buffers bah,bAh,bäh,bÄh)
>> 2. C-x b *scratch*
>> 3a. C-x b ba TAB
>>    => completes to bah and after TAB displays [Sole completion]
>> 3b. C-g C-x b bA TAB
>>    => completes to bAh and after TAB displays [Sole completion]
>> 4a. C-g C-x b bä TAB
>>    => completes to bäh and after TAB displays [Sole completion]
>> 4b. C-g C-x b bÄ TAB
>>    => completes to bÄh and after TAB displays [Sole completion]
>> 5a. C-g C-x C-f /tmp/ba TAB
>>    => completes to bah and after TAB displays [Complete, but not unique]
>>       and *Completions* pops up showing `bAh' and `bah'
>> 5b. C-g C-x C-f /tmp/bA TAB
>>    => completes to bAh and after TAB displays [Complete, but not unique]
>>       and *Completions* pops up showing `bAh' and `bah'
>> 6a. C-g C-x C-f /tmp/bä TAB
>>    => completes to /tmp/bäh and after TAB displays [Complete, but not unique]
>>       and *Completions* pops up showing `bAh' and `bah'
>> 6a. C-g C-x C-f /tmp/bÄ TAB
>>    => completes to /tmp/bÄh and after TAB displays [Complete, but not unique]
>>       and *Completions* pops up showing `bAh' and `bah'
>> 
>> So, before your fix file name completion involving non-ASCII strings
>> behaved the same as buffer name completion involving both ASCII-only and
>> non-ASCII strings, but file name completion involving ASCII-only strings
>> differed.  Now, after your fix, file name completion involving non-ASCII
>> strings works the same as file name completion involving ASCII-only
>> strings did before your fix and still does after, but all the cases of
>> file name completion differ from the corresponding cases of buffer name
>> completion.  I would prefer buffer name completion to work the way file
>> name completion uniformly works after your fix.
>
> "You are in a maze of twisty little passages, all alike."
>
> (I'm confused by a dozen of similar examples which don't say which
> result is deemed to be correct and which isn't.)
>
> Previously, you said that buffer-name completion works correctly, 

I don't think so.  In my OP of this bug I reported the results of `C-x
b' as incorrect (step 8 of the recipe there).  In my followup a year
later, I reiterated that the "bug still exists" but added a different
observation of correct behavior involving `C-x b' but incorrect behavior
involving `C-x C-f'.  After Stefan Kangas's post revisiting this bug,
which only looked at the `C-x b' behavior of my OP, you implied the
behavior is correct and said you see no bug, to which I replied with
what I intended to be a clarification of my OP, but it seems to have
confused you; my followup to that unfortunately seems to have added to
the confusion concerning the behavior of `C-x b', for which I apologize.
But I hope and think my last reply quoted above is clear: "all the cases
of file name completion differ from the corresponding cases of buffer
name completion. I would prefer buffer name completion to work the way
file name completion uniformly works after your fix."  To phrase it in
terms of correctness: the results of `C-x b' in steps 3-4 quoted above
are incorrect; they should be the same as the results of `C-x C-f' in
steps 5-6, which are correct (IMO).

Steve Berman

Message #55 received at 11339 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Stephen Berman <stephen.berman <at> gmx.net>,
 Stefan Monnier <monnier <at> iro.umontreal.ca>
Cc: 11339 <at> debbugs.gnu.org, stefan <at> marxist.se
Subject: Re: bug#11339: 24.1.50;
 read-{buffer,file-name}-completion-ignore-case fails on non-ascii
Date: Wed, 06 Nov 2019 18:58:36 +0200

> From: Stephen Berman <stephen.berman <at> gmx.net>
> Cc: stefan <at> marxist.se,  11339 <at> debbugs.gnu.org
> Date: Mon, 04 Nov 2019 19:02:15 +0100
> 
> >> 1. $ emacs-master -Q  --eval '(setq read-buffer-completion-ignore-case tread-file-name-completion-ignore-case t)' /tmp/{bah,bAh,bäh,bÄh}
> >> (*Buffer List* show the buffers bah,bAh,bäh,bÄh)
> >> 2. C-x b *scratch*
> >> 3a. C-x b ba TAB
> >>    => completes to bah and after TAB displays [Sole completion]
> >> 3b. C-g C-x b bA TAB
> >>    => completes to bAh and after TAB displays [Sole completion]
> >> 4a. C-g C-x b bä TAB
> >>    => completes to bäh and after TAB displays [Sole completion]
> >> 4b. C-g C-x b bÄ TAB
> >>    => completes to bÄh and after TAB displays [Sole completion]
> >> 5a. C-g C-x C-f /tmp/ba TAB
> >>    => completes to bah and after TAB displays [Complete, but not unique]
> >>       and *Completions* pops up showing `bAh' and `bah'
> >> 5b. C-g C-x C-f /tmp/bA TAB
> >>    => completes to bAh and after TAB displays [Complete, but not unique]
> >>       and *Completions* pops up showing `bAh' and `bah'
> >> 6a. C-g C-x C-f /tmp/bä TAB
> >>    => completes to /tmp/bäh and after TAB displays [Complete, but not unique]
> >>       and *Completions* pops up showing `bAh' and `bah'
> >> 6a. C-g C-x C-f /tmp/bÄ TAB
> >>    => completes to /tmp/bÄh and after TAB displays [Complete, but not unique]
> >>       and *Completions* pops up showing `bAh' and `bah'
> >> 
> >> So, before your fix file name completion involving non-ASCII strings
> >> behaved the same as buffer name completion involving both ASCII-only and
> >> non-ASCII strings, but file name completion involving ASCII-only strings
> >> differed.  Now, after your fix, file name completion involving non-ASCII
> >> strings works the same as file name completion involving ASCII-only
> >> strings did before your fix and still does after, but all the cases of
> >> file name completion differ from the corresponding cases of buffer name
> >> completion.  I would prefer buffer name completion to work the way file
> >> name completion uniformly works after your fix.
> >
> > "You are in a maze of twisty little passages, all alike."
> >
> > (I'm confused by a dozen of similar examples which don't say which
> > result is deemed to be correct and which isn't.)
> >
> > Previously, you said that buffer-name completion works correctly, 
> 
> I don't think so.  In my OP of this bug I reported the results of `C-x
> b' as incorrect (step 8 of the recipe there).  In my followup a year
> later, I reiterated that the "bug still exists" but added a different
> observation of correct behavior involving `C-x b' but incorrect behavior
> involving `C-x C-f'.  After Stefan Kangas's post revisiting this bug,
> which only looked at the `C-x b' behavior of my OP, you implied the
> behavior is correct and said you see no bug, to which I replied with
> what I intended to be a clarification of my OP, but it seems to have
> confused you; my followup to that unfortunately seems to have added to
> the confusion concerning the behavior of `C-x b', for which I apologize.
> But I hope and think my last reply quoted above is clear: "all the cases
> of file name completion differ from the corresponding cases of buffer
> name completion. I would prefer buffer name completion to work the way
> file name completion uniformly works after your fix."  To phrase it in
> terms of correctness: the results of `C-x b' in steps 3-4 quoted above
> are incorrect; they should be the same as the results of `C-x C-f' in
> steps 5-6, which are correct (IMO).

I'm sorry, but someone else will have to dig into this.  I tried, but
got lost.  I always do when I try to debug non-trivial issues with
completion, what with the endless ping-pong of calls from
minibuffer.el to minibuf.c and back, both directly and indirectly via
variables whose values are functions.

The best I could see (unless I'm mistaken) is that in the "C-x b" case
the second TAB sees that the string in the minibuffer is a possible
completion, and declares success; it doesn't call all-completions as
I'd expect.  If this observation is correct, then relying on
try-completions in this case is what trips us, because try-completions
has special heuristics when the candidates are all identical but for
the letter-case, the result being that only one candidate is returned.

CC'ing Stefan who might have a better idea of what is going on here.

Message #58 received at 11339 <at> debbugs.gnu.org (full text, mbox):

From: Stephen Berman <stephen.berman <at> gmx.net>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 11339 <at> debbugs.gnu.org, stefan <at> marxist.se,
 Stefan Monnier <monnier <at> iro.umontreal.ca>
Subject: Re: bug#11339: 24.1.50;
 read-{buffer,file-name}-completion-ignore-case fails on non-ascii
Date: Thu, 07 Nov 2019 15:34:15 +0100

On Wed, 06 Nov 2019 18:58:36 +0200 Eli Zaretskii <eliz <at> gnu.org> wrote:

>> From: Stephen Berman <stephen.berman <at> gmx.net>
>> Cc: stefan <at> marxist.se,  11339 <at> debbugs.gnu.org
>> Date: Mon, 04 Nov 2019 19:02:15 +0100
>> 
>> >> 1. $ emacs-master -Q  --eval '(setq read-buffer-completion-ignore-case tread-file-name-completion-ignore-case t)' /tmp/{bah,bAh,bäh,bÄh}
>> >> (*Buffer List* show the buffers bah,bAh,bäh,bÄh)
>> >> 2. C-x b *scratch*
>> >> 3a. C-x b ba TAB
>> >>    => completes to bah and after TAB displays [Sole completion]
>> >> 3b. C-g C-x b bA TAB
>> >>    => completes to bAh and after TAB displays [Sole completion]
>> >> 4a. C-g C-x b bä TAB
>> >>    => completes to bäh and after TAB displays [Sole completion]
>> >> 4b. C-g C-x b bÄ TAB
>> >>    => completes to bÄh and after TAB displays [Sole completion]
>> >> 5a. C-g C-x C-f /tmp/ba TAB
>> >>    => completes to bah and after TAB displays [Complete, but not unique]
>> >>       and *Completions* pops up showing `bAh' and `bah'
>> >> 5b. C-g C-x C-f /tmp/bA TAB
>> >>    => completes to bAh and after TAB displays [Complete, but not unique]
>> >>       and *Completions* pops up showing `bAh' and `bah'
>> >> 6a. C-g C-x C-f /tmp/bä TAB
>> >>    => completes to /tmp/bäh and after TAB displays [Complete, but not unique]
>> >>       and *Completions* pops up showing `bAh' and `bah'
>> >> 6a. C-g C-x C-f /tmp/bÄ TAB
>> >>    => completes to /tmp/bÄh and after TAB displays [Complete, but not unique]
>> >>       and *Completions* pops up showing `bAh' and `bah'
>> >> 
[...]
>> the results of `C-x b' in steps 3-4 quoted above
>> are incorrect; they should be the same as the results of `C-x C-f' in
>> steps 5-6, which are correct (IMO).
>
> I'm sorry, but someone else will have to dig into this.  I tried, but
> got lost.  I always do when I try to debug non-trivial issues with
> completion, what with the endless ping-pong of calls from
> minibuffer.el to minibuf.c and back, both directly and indirectly via
> variables whose values are functions.
>
> The best I could see (unless I'm mistaken) is that in the "C-x b" case
> the second TAB sees that the string in the minibuffer is a possible
> completion, and declares success; it doesn't call all-completions as
> I'd expect.  If this observation is correct, then relying on
> try-completions in this case is what trips us, because try-completions
> has special heuristics when the candidates are all identical but for
> the letter-case, the result being that only one candidate is returned.
>
> CC'ing Stefan who might have a better idea of what is going on here.

So why do we have to rely on `try-completion' here?  Emacs built with
the patch below shows the behavior I want with `C-x b' using the above
recipe.  I assume that are bad side effects elsewhere, but I can't think
of any offhand.

Steve Berman

diff --git a/src/minibuf.c b/src/minibuf.c
index f6cf47f1f2..4faf2f856d 100644
--- a/src/minibuf.c
+++ b/src/minibuf.c
@@ -1777,9 +1777,7 @@ If FLAG is nil, invoke `try-completion'; if it is t, invoke
 `all-completions'; otherwise invoke `test-completion'.  */)
   (Lisp_Object string, Lisp_Object predicate, Lisp_Object flag)
 {
-  if (NILP (flag))
-    return Ftry_completion (string, Vbuffer_alist, predicate);
-  else if (EQ (flag, Qt))
+  if (EQ (flag, Qt))
     {
       Lisp_Object res = Fall_completions (string, Vbuffer_alist, predicate, Qnil);
       if (SCHARS (string) > 0)

Message #61 received at 11339 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Stephen Berman <stephen.berman <at> gmx.net>
Cc: 11339 <at> debbugs.gnu.org, stefan <at> marxist.se, monnier <at> iro.umontreal.ca
Subject: Re: bug#11339: 24.1.50;
 read-{buffer,file-name}-completion-ignore-case fails on non-ascii
Date: Thu, 07 Nov 2019 16:49:22 +0200

> From: Stephen Berman <stephen.berman <at> gmx.net>
> Cc: Stefan Monnier <monnier <at> iro.umontreal.ca>,  stefan <at> marxist.se,
>   11339 <at> debbugs.gnu.org
> Date: Thu, 07 Nov 2019 15:34:15 +0100
> 
> > The best I could see (unless I'm mistaken) is that in the "C-x b" case
> > the second TAB sees that the string in the minibuffer is a possible
> > completion, and declares success; it doesn't call all-completions as
> > I'd expect.  If this observation is correct, then relying on
> > try-completions in this case is what trips us, because try-completions
> > has special heuristics when the candidates are all identical but for
> > the letter-case, the result being that only one candidate is returned.
> >
> > CC'ing Stefan who might have a better idea of what is going on here.
> 
> So why do we have to rely on `try-completion' here?

Because that's how completion--do-completion was coded, I suppose.

> Emacs built with the patch below shows the behavior I want with `C-x
> b' using the above recipe.  I assume that are bad side effects
> elsewhere

Of course.  We cannot make such a change, I think.

Message #64 received at 11339 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Monnier <monnier <at> iro.umontreal.ca>
To: Stephen Berman <stephen.berman <at> gmx.net>
Cc: Eli Zaretskii <eliz <at> gnu.org>, stefan <at> marxist.se, 11339 <at> debbugs.gnu.org
Subject: Re: bug#11339: 24.1.50;
 read-{buffer,file-name}-completion-ignore-case fails on non-ascii
Date: Thu, 07 Nov 2019 11:01:01 -0500

[Message part 1 (text/x-markdown, inline)]

> So why do we have to rely on `try-completion' here?  Emacs built with
> the patch below shows the behavior I want with `C-x b' using the above
> recipe.  I assume that are bad side effects elsewhere, but I can't think
> of any offhand.

I think the core of the bug is reproduced below:

    M-: (let ((completion-ignore-case t)) (try-completion "bah" '("bah" "bAh")))

this returns t instead of returning "bah".
Probably an error in the handling of `matchcount` in `try-completion`.


        Stefan

Message #67 received at 11339 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Stefan Monnier <monnier <at> iro.umontreal.ca>
Cc: 11339 <at> debbugs.gnu.org, stephen.berman <at> gmx.net, stefan <at> marxist.se
Subject: Re: bug#11339: 24.1.50;
 read-{buffer,file-name}-completion-ignore-case fails on non-ascii
Date: Thu, 07 Nov 2019 18:08:00 +0200

> From: Stefan Monnier <monnier <at> iro.umontreal.ca>
> Cc: Eli Zaretskii <eliz <at> gnu.org>,  11339 <at> debbugs.gnu.org,  stefan <at> marxist.se
> Date: Thu, 07 Nov 2019 11:01:01 -0500
> 
> I think the core of the bug is reproduced below:
> 
>     M-: (let ((completion-ignore-case t)) (try-completion "bah" '("bah" "bAh")))
> 
> this returns t instead of returning "bah".
> Probably an error in the handling of `matchcount` in `try-completion`.

Are you sure?  I think the return value of t is expected, per these
comments in try-completion:

	      if (completion_ignore_case)
		{
		  /* If this is an exact match except for case,
		     use it as the best match rather than one that is not an
		     exact match.  This way, we get the case pattern
		     of the actual match.  */
		  if ((matchsize == SCHARS (eltstring)
		       && matchsize < SCHARS (bestmatch))
		      ||
		      /* If there is more than one exact match ignoring case,
			 and one of them is exact including case,
			 prefer that one.  */
		      /* If there is no exact match ignoring case,
			 prefer a match that does not change the case
			 of the input.  */

Message #70 received at 11339 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Monnier <monnier <at> iro.umontreal.ca>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 11339 <at> debbugs.gnu.org, stephen.berman <at> gmx.net, stefan <at> marxist.se
Subject: Re: bug#11339: 24.1.50;
 read-{buffer,file-name}-completion-ignore-case fails on non-ascii
Date: Thu, 07 Nov 2019 11:27:50 -0500

>> I think the core of the bug is reproduced below:
>> 
>>     M-: (let ((completion-ignore-case t)) (try-completion "bah" '("bah" "bAh")))
>> 
>> this returns t instead of returning "bah".
>> Probably an error in the handling of `matchcount` in `try-completion`.
>
> Are you sure?

I think so, yes.

> I think the return value of t is expected, per these
> comments in try-completion:

I don't think so:

> 		  /* If this is an exact match except for case,
> 		     use it as the best match rather than one that is not an
> 		     exact match.  This way, we get the case pattern
> 		     of the actual match.  */

This says that (try-completion "bah" '("bAh" "bahfoo")) should return "bAh"

> 		      /* If there is more than one exact match ignoring case,
> 			 and one of them is exact including case,
> 			 prefer that one.  */

This says (try-completion "bah" '("bah" "bAh")) should return "bah" rather
than "bAh", which also agrees with I said above (it should not return t).

> 		      /* If there is no exact match ignoring case,
> 			 prefer a match that does not change the case
> 			 of the input.  */

This says that (try-completion "bah" '("bAhbar" "bahfoo")) should return
"bah" rather than "bAh".


        Stefan

Message #73 received at 11339 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Stefan Monnier <monnier <at> iro.umontreal.ca>
Cc: 11339 <at> debbugs.gnu.org, stephen.berman <at> gmx.net, stefan <at> marxist.se
Subject: Re: bug#11339: 24.1.50;
 read-{buffer,file-name}-completion-ignore-case fails on non-ascii
Date: Thu, 07 Nov 2019 20:12:31 +0200

> From: Stefan Monnier <monnier <at> iro.umontreal.ca>
> Cc: stephen.berman <at> gmx.net,  11339 <at> debbugs.gnu.org,  stefan <at> marxist.se
> Date: Thu, 07 Nov 2019 11:27:50 -0500
> 
> >> I think the core of the bug is reproduced below:
> >> 
> >>     M-: (let ((completion-ignore-case t)) (try-completion "bah" '("bah" "bAh")))
> >> 
> >> this returns t instead of returning "bah".
> >> Probably an error in the handling of `matchcount` in `try-completion`.
> >
> > Are you sure?
> 
> I think so, yes.

The patch below fixes Stefan's simplified test case, but the original
problem is still unsolved: "C-x b ba TAB TAB" says "Sole completion".

What did I miss?

diff --git a/src/minibuf.c b/src/minibuf.c
index f6cf47f..2670211 100644
--- a/src/minibuf.c
+++ b/src/minibuf.c
@@ -1363,7 +1363,13 @@ DEFUN ("try-completion", Ftry_completion, Stry_completion, 2, 3, 0,
 		    bestmatch = eltstring;
 		}
 	      if (bestmatchsize != SCHARS (eltstring)
-		  || bestmatchsize != matchsize)
+		  || bestmatchsize != matchsize
+		  || (completion_ignore_case
+		      && !EQ (Fcompare_strings (bestmatch, zero,
+						make_fixnum (compare),
+						eltstring, zero,
+						make_fixnum (compare), Qnil),
+			      Qt)))
 		/* Don't count the same string multiple times.  */
 		matchcount += matchcount <= 1;
 	      bestmatchsize = matchsize;

Message #76 received at 11339 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Monnier <monnier <at> iro.umontreal.ca>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 11339 <at> debbugs.gnu.org, stephen.berman <at> gmx.net, stefan <at> marxist.se
Subject: Re: bug#11339: 24.1.50;
 read-{buffer,file-name}-completion-ignore-case fails on non-ascii
Date: Fri, 08 Nov 2019 17:36:08 -0500

> The patch below fixes Stefan's simplified test case, but the original
> problem is still unsolved: "C-x b ba TAB TAB" says "Sole completion".
>
> What did I miss?

It still fails when you swap the two elements in the list :-(
I don't have time to dig into the code now, tho,


        Stefan


(ert-deftest try-completion-ignore-case ()
  (let ((completion-ignore-case t))
    (should (equal (try-completion "bar" '("bAr" "barfoo")) "bAr"))
    (should (equal (try-completion "bar" '("bArfoo" "barbaz")) "bar"))
    (should (equal (try-completion "bar" '("bArfoo" "barbaz"))
                   (try-completion "bar" '("barbaz" "bArfoo"))))
    ;; bug#11339
    (should (equal (try-completion "baz" '("baz" "bAz")) "baz")) ;And not `t'!
    (should (equal (try-completion "baz" '("bAz" "baz"))
                   (try-completion "baz" '("baz" "bAz"))))))

Message #79 received at 11339 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Stefan Monnier <monnier <at> iro.umontreal.ca>
Cc: 11339 <at> debbugs.gnu.org, stephen.berman <at> gmx.net, stefan <at> marxist.se
Subject: Re: bug#11339: 24.1.50;
 read-{buffer,file-name}-completion-ignore-case fails on non-ascii
Date: Sat, 09 Nov 2019 11:20:12 +0200

> From: Stefan Monnier <monnier <at> iro.umontreal.ca>
> Cc: stephen.berman <at> gmx.net,  11339 <at> debbugs.gnu.org,  stefan <at> marxist.se
> Date: Fri, 08 Nov 2019 17:36:08 -0500
> 
> > The patch below fixes Stefan's simplified test case, but the original
> > problem is still unsolved: "C-x b ba TAB TAB" says "Sole completion".
> >
> > What did I miss?
> 
> It still fails when you swap the two elements in the list :-(
> I don't have time to dig into the code now, tho,

What about the below?  It passes all the tests, and also fixes the
original use case.

diff --git a/src/minibuf.c b/src/minibuf.c
index f6cf47f..1e87c50 100644
--- a/src/minibuf.c
+++ b/src/minibuf.c
@@ -1323,13 +1323,13 @@ DEFUN ("try-completion", Ftry_completion, Stry_completion, 2, 3, 0,
 	  else
 	    {
 	      compare = min (bestmatchsize, SCHARS (eltstring));
-	      tem = Fcompare_strings (bestmatch, zero,
-				      make_fixnum (compare),
-				      eltstring, zero,
-				      make_fixnum (compare),
+	      Lisp_Object lcompare = make_fixnum (compare);
+	      tem = Fcompare_strings (bestmatch, zero, lcompare,
+				      eltstring, zero, lcompare,
 				      completion_ignore_case ? Qt : Qnil);
 	      matchsize = EQ (tem, Qt) ? compare : eabs (XFIXNUM (tem)) - 1;
 
+	      Lisp_Object old_bestmatch = bestmatch;
 	      if (completion_ignore_case)
 		{
 		  /* If this is an exact match except for case,
@@ -1363,7 +1363,12 @@ DEFUN ("try-completion", Ftry_completion, Stry_completion, 2, 3, 0,
 		    bestmatch = eltstring;
 		}
 	      if (bestmatchsize != SCHARS (eltstring)
-		  || bestmatchsize != matchsize)
+		  || bestmatchsize != matchsize
+		  || (completion_ignore_case
+		      && !EQ (Fcompare_strings (old_bestmatch, zero, lcompare,
+						eltstring, zero, lcompare,
+						Qnil),
+			      Qt)))
 		/* Don't count the same string multiple times.  */
 		matchcount += matchcount <= 1;
 	      bestmatchsize = matchsize;

Message #82 received at 11339 <at> debbugs.gnu.org (full text, mbox):

From: Stephen Berman <stephen.berman <at> gmx.net>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 11339 <at> debbugs.gnu.org, stefan <at> marxist.se,
 Stefan Monnier <monnier <at> iro.umontreal.ca>
Subject: Re: bug#11339: 24.1.50;
 read-{buffer,file-name}-completion-ignore-case fails on non-ascii
Date: Sat, 09 Nov 2019 18:56:23 +0100

On Sat, 09 Nov 2019 11:20:12 +0200 Eli Zaretskii <eliz <at> gnu.org> wrote:

>> From: Stefan Monnier <monnier <at> iro.umontreal.ca>
>> Cc: stephen.berman <at> gmx.net,  11339 <at> debbugs.gnu.org,  stefan <at> marxist.se
>> Date: Fri, 08 Nov 2019 17:36:08 -0500
>>
>> > The patch below fixes Stefan's simplified test case, but the original
>> > problem is still unsolved: "C-x b ba TAB TAB" says "Sole completion".
>> >
>> > What did I miss?
>>
>> It still fails when you swap the two elements in the list :-(
>> I don't have time to dig into the code now, tho,
>
> What about the below?  It passes all the tests, and also fixes the
> original use case.

I confirm it makes completion with `C-x b' work the way I want when case
is ignored, so I am in favor of installing it.  Thanks.

Steve Berman

Message #85 received at 11339 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Monnier <monnier <at> iro.umontreal.ca>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 11339 <at> debbugs.gnu.org, stephen.berman <at> gmx.net, stefan <at> marxist.se
Subject: Re: bug#11339: 24.1.50;
 read-{buffer,file-name}-completion-ignore-case fails on non-ascii
Date: Sat, 09 Nov 2019 13:24:06 -0500

> What about the below?  It passes all the tests, and also fixes the
> original use case.

Sounds good.  I have the impression that I was the original author of
that "early exit" optimization (or at least some of it), but my brain
must be decaying because I have a hard time understanding the code.
I also wonder if it's really worth the effort (of the optimization,
I mean).


        Stefan

Message #88 received at 11339-done <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Stefan Monnier <monnier <at> iro.umontreal.ca>
Cc: 11339-done <at> debbugs.gnu.org, stephen.berman <at> gmx.net, stefan <at> marxist.se
Subject: Re: bug#11339: 24.1.50;
 read-{buffer,file-name}-completion-ignore-case fails on non-ascii
Date: Sat, 09 Nov 2019 21:57:36 +0200

> From: Stefan Monnier <monnier <at> iro.umontreal.ca>
> Cc: stephen.berman <at> gmx.net,  11339 <at> debbugs.gnu.org,  stefan <at> marxist.se
> Date: Sat, 09 Nov 2019 13:24:06 -0500
> 
> > What about the below?  It passes all the tests, and also fixes the
> > original use case.
> 
> Sounds good.

Thanks, installed.

> I have the impression that I was the original author of that "early
> exit" optimization (or at least some of it), but my brain must be
> decaying because I have a hard time understanding the code.  I also
> wonder if it's really worth the effort (of the optimization, I
> mean).

AFAICT, you were the author of the "don't count the same string more
than once" fragment.

I'm closing the bug report.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.