GNU bug report logs - #1183
23.0.60; ediff-buffers is broken

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 1183 in the body.
You can then email your comments to 1183 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> emacsbugs.donarmstrong.com (full text, mbox):

From: "Drew Adams" <drew.adams <at> oracle.com>
To: <emacs-pretest-bug <at> gnu.org>
Subject: 23.0.60; ediff-buffers is broken
Date: Thu, 16 Oct 2008 11:47:11 -0700

emacs -Q
 
Visit both the bookmark.el file from this installation (see below) and
a copy of bookmark.el from CVS for today, 2008-10-16.
 
The only difference between the two files is in fact this text which
was added to the CVS version near the end of the file, just before
run-hooks:
 
(defun bookmark-unload-function ()
  "Unload the Bookmark library."
  (when bookmark-save-flag (bookmark-save))
  ;; continue standard unloading
  nil)
 
You can see this by using ediff in Emacs 22, 21, or 20 - or by using
diff.
 
However, ediff-buffers shows the entire buffers as a single diff, and
hitting `*' to refine that diff has no effect at all. IOW,
ediff-buffers is now useless for seeing the differences between these
two files.
 

In GNU Emacs 23.0.60.1 (i386-mingw-nt5.1.2600)
 of 2008-10-03 on LENNART-69DE564
Windowing system distributor `Microsoft Corp.', version 5.1.2600
configured using `configure --with-gcc (3.4) --no-opt --cflags -Ic:/g/include
-fno-crossjumping'
 

Message #10 received at submit <at> emacsbugs.donarmstrong.com (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Drew Adams <drew.adams <at> oracle.com>, 1183 <at> debbugs.gnu.org
Cc: emacs-pretest-bug <at> gnu.org, bug-gnu-emacs <at> gnu.org
Subject: Re: bug#1183: 23.0.60; ediff-buffers is broken
Date: Thu, 16 Oct 2008 22:25:25 +0200

> From: "Drew Adams" <drew.adams <at> oracle.com>
> Date: Thu, 16 Oct 2008 11:47:11 -0700
> Cc: 
> 
> emacs -Q
>  
> Visit both the bookmark.el file from this installation (see below) and
> a copy of bookmark.el from CVS for today, 2008-10-16.
>  
> The only difference between the two files is in fact this text which
> was added to the CVS version near the end of the file, just before
> run-hooks:
>  
> (defun bookmark-unload-function ()
>   "Unload the Bookmark library."
>   (when bookmark-save-flag (bookmark-save))
>   ;; continue standard unloading
>   nil)
>  
> You can see this by using ediff in Emacs 22, 21, or 20 - or by using
> diff.
>  
> However, ediff-buffers shows the entire buffers as a single diff, and
> hitting `*' to refine that diff has no effect at all. IOW,
> ediff-buffers is now useless for seeing the differences between these
> two files.

I don't have the ``bookmark.el file from this installation'' (you
didn't attach it), so I produced it manually by copying today's
bookmark.el and then removing the function bookmark-unload-function.

With these two files, I can only reproduce this with Emacs built on
Windows from today's CVS if one of the files has Unix end-of-line
format, while the other has DOS/Windows EOL format.  Is that your
case?

If so, this is expected: Ediff on Windows invokes the `diff' program
with the --binary option (see ediff-diff-options), which makes all
lines compare not equal due to the different line endings.  You can
see this by using `diff' directly from the shell's prompt, if you pass
it the --binary option.

If both files have identical EOL format, Ediff produces the output
you'd expect.

(The reason for the --binary option is to allow comparison of buffers
and files with non-ASCII text, which IMO is a much more important
use-case than two almost identical files with different line endings.)

Message #25 received at submit <at> emacsbugs.donarmstrong.com (full text, mbox):

From: "Drew Adams" <drew.adams <at> oracle.com>
To: "'Eli Zaretskii'" <eliz <at> gnu.org>, <1183 <at> debbugs.gnu.org>
Cc: <emacs-pretest-bug <at> gnu.org>, <bug-gnu-emacs <at> gnu.org>
Subject: RE: bug#1183: 23.0.60; ediff-buffers is broken
Date: Thu, 16 Oct 2008 13:45:21 -0700

[Message part 1 (text/plain, inline)]

> From: Eli Zaretskii Sent: Thursday, October 16, 2008 1:25 PM
> > From: "Drew Adams" <drew.adams <at> oracle.com>
> > emacs -Q
> >  
> > Visit both the bookmark.el file from this installation (see 
> > below) and a copy of bookmark.el from CVS for today, 2008-10-16.
> >  
> > The only difference between the two files is in fact this text which
> > was added to the CVS version near the end of the file, just before
> > run-hooks:
> >  
> > (defun bookmark-unload-function ()
> >   "Unload the Bookmark library."
> >   (when bookmark-save-flag (bookmark-save))
> >   ;; continue standard unloading
> >   nil)
> >  
> > You can see this by using ediff in Emacs 22, 21, or 20 - or by using
> > diff.
> >  
> > However, ediff-buffers shows the entire buffers as a single 
> > diff, and hitting `*' to refine that diff has no effect at all. IOW,
> > ediff-buffers is now useless for seeing the differences 
> > between these two files.
> 
> I don't have the ``bookmark.el file from this installation'' (you
> didn't attach it),

Attached now. I didn't think I had to attach it since I referenced its build -
sorry.

> so I produced it manually by copying today's
> bookmark.el and then removing the function bookmark-unload-function.

Same diff. ;-)

> With these two files, I can only reproduce this with Emacs built on
> Windows from today's CVS if one of the files has Unix end-of-line
> format, while the other has DOS/Windows EOL format.  Is that your
> case?

I downloaded the CVS version (with Unix format) from
http://cvs.savannah.gnu.org/viewvc/emacs/emacs/lisp/bookmark.el?view=log by
clicking the first "download" link on the page. The other file was in the
distribution.

Yes, I guess it is my case: The CVS file has "(Unix)" in the mode line; the
installation file has nothing.

> If so, this is expected:

Well it certainly isn't expected in Emacs 20, 21, or 22, where ediff-buffers
works perfectly for these two files. Why this change for Emacs 23? What's the
gain?

And how to change the buffers (e.g. line endings) so ediff-buffers will compare
them correctly? And why wouldn't ediff-buffers itself do that automatically
(perhaps after user confirmation)?

> Ediff on Windows invokes the `diff' program
> with the --binary option (see ediff-diff-options), which makes all
> lines compare not equal due to the different line endings.

Not very useful in situations like this. Comparing source code files where the
only difference is line endings should not produce a totally unusable diff: a
single, whole-file diff for which `*' has no effect (cannot be refined). 

It should show just the line endings as differences, if they are the only
differences. And in this case, where several text lines were different, it
should pick those lines up as a difference. Definitely. 

To me, this seems horribly broken.

> You can see this by using `diff' directly from the shell's prompt,
> if you pass it the --binary option.
> 
> If both files have identical EOL format, Ediff produces the output
> you'd expect.

And how to get that identical EOL format (I should know that, but I never bother
to change the format).

And why shouldn't ediff at least pick that up and let you know how to get to a
useful diff. Better yet, it should do that for you, perhaps after asking for
confirmation to change line endings or whatever.

What was wrong with the Emacs 22, 21, 20,... behavior?

> (The reason for the --binary option is to allow comparison of buffers
> and files with non-ASCII text, which IMO is a much more important
> use-case than two almost identical files with different line endings.)

Apples and oranges. They are probably both important.

Certainly it's important to be able to diff a file in the installation against a
CVS version of the file in a simple way.

Line-ending differences are not the most important differences to show, and in
fact even those differences are not apparent. You cannot tell anything at all
from the ediff-buffers diff - it tells you nothing about the differences except
that there are differences. Pretty useless.

I can't believe that one would argue that this behavior represents progress, at
least in its present form. Something should be done to fix this, IMO.

[bookmark.el (application/octet-stream, attachment)]

Message #40 received at submit <at> emacsbugs.donarmstrong.com (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Drew Adams <drew.adams <at> oracle.com>
Cc: 1183 <at> debbugs.gnu.org, emacs-pretest-bug <at> gnu.org,
        bug-gnu-emacs <at> gnu.org
Subject: Re: bug#1183: 23.0.60; ediff-buffers is broken
Date: Thu, 16 Oct 2008 23:15:45 +0200

> From: "Drew Adams" <drew.adams <at> oracle.com>
> Cc: <emacs-pretest-bug <at> gnu.org>, <bug-gnu-emacs <at> gnu.org>
> Date: Thu, 16 Oct 2008 13:45:21 -0700
> 
> > If so, this is expected:
> 
> Well it certainly isn't expected in Emacs 20, 21, or 22, where ediff-buffers
> works perfectly for these two files. Why this change for Emacs 23? What's the
> gain?

Sorry, you are right: Emacs 22 also uses --binary, but doesn't expose
this problem.  So I guess something else is at work here.  I'll look
closer when I have time.

> And how to change the buffers (e.g. line endings) so ediff-buffers will compare
> them correctly?

"C-x RET f unix RET C-x C-s".

Message #55 received at submit <at> emacsbugs.donarmstrong.com (full text, mbox):

From: "Drew Adams" <drew.adams <at> oracle.com>
To: "'Eli Zaretskii'" <eliz <at> gnu.org>
Cc: <1183 <at> debbugs.gnu.org>, <emacs-pretest-bug <at> gnu.org>,
        <bug-gnu-emacs <at> gnu.org>
Subject: RE: bug#1183: 23.0.60; ediff-buffers is broken
Date: Thu, 16 Oct 2008 14:58:23 -0700

> From: Eli Zaretskii Sent: Thursday, October 16, 2008 2:16 PM
> > From: "Drew Adams" Date: Thu, 16 Oct 2008 13:45:21 -0700
> > 
> > > If so, this is expected:
> > 
> > Well it certainly isn't expected in Emacs 20, 21, or 22, 
> > where ediff-buffers works perfectly for these two files.
> > Why this change for Emacs 23? What's the gain?
> 
> Sorry, you are right: Emacs 22 also uses --binary, but doesn't expose
> this problem.  So I guess something else is at work here.  I'll look
> closer when I have time.

Thanks, Eli. 

BTW, I don't know if it's related, but I just filed bug #1187, which has to do
with reading characters differently in Emacs 23 from Emacs 22. Maybe the two are
related in some way.

> > And how to change the buffers (e.g. line endings) so 
> > ediff-buffers will compare them correctly?
> 
> "C-x RET f unix RET C-x C-s".

But that would save the file with the new line endings. What if I don't want to
do that? ediff-buffers should not require you to save a file differently or even
to have a file associated with the buffer.

Message #70 received at submit <at> emacsbugs.donarmstrong.com (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: 1183 <at> debbugs.gnu.org
Cc: drew.adams <at> oracle.com, bug-gnu-emacs <at> gnu.org,
        Michael Kifer <kifer <at> cs.stonybrook.edu>
Subject: Re: bug#1183: 23.0.60; ediff-buffers is broken
Date: Fri, 17 Oct 2008 14:38:46 +0200

> Date: Thu, 16 Oct 2008 23:15:45 +0200
> From: Eli Zaretskii <eliz <at> gnu.org>
> Cc: 1183 <at> emacsbugs.donarmstrong.com, bug-gnu-emacs <at> gnu.org,
> 	emacs-pretest-bug <at> gnu.org
> 
> > From: "Drew Adams" <drew.adams <at> oracle.com>
> > Cc: <emacs-pretest-bug <at> gnu.org>, <bug-gnu-emacs <at> gnu.org>
> > Date: Thu, 16 Oct 2008 13:45:21 -0700
> > 
> > > If so, this is expected:
> > 
> > Well it certainly isn't expected in Emacs 20, 21, or 22, where ediff-buffers
> > works perfectly for these two files. Why this change for Emacs 23? What's the
> > gain?
> 
> Sorry, you are right: Emacs 22 also uses --binary, but doesn't expose
> this problem.  So I guess something else is at work here.  I'll look
> closer when I have time.

Found the culprit.  It's in ediff-make-temp-file:

  (let* ( ...
  	 (coding-system-for-write
	  (ediff-with-current-buffer buff
	    (if (boundp 'buffer-file-coding-system)
		buffer-file-coding-system
	      ediff-coding-system-for-write)))

This let-binds the coding-system with which we will write the two
buffers to temporary files, to the original encoding of the files read
into the buffers being compared.  The temporary files are then
submitted to Diff, and that makes each line compare different because
of the different EOL format.

Emacs 22 unconditionally uses ediff-coding-system-for-write here,
which is set to `no-conversion', so this problem does not happen in
Emacs 22.

This change was made in Aug 2007, and Michael Kifer wrote a bit later
on emacs-devel that using no-conversion was screwing some other
use-case (I couldn't find the description of that use-case, although
Michael said it was reported earlier on emacs-devel).

I could easily fix this particular problem by forcing the EOL
conversion to -unix, but I think there's a broader issue here, and we
might as well handle it now.  The issue is this: suppose we have two
buffers whose text is identical in the internal Emacs representation
of characters, but whose values of buffer-file-coding-system differ--
do we want such buffers to compare equal or not?  For example, the
buffers could contain Latin-2 text, with one buffer coming from a file
encoded in ISO-8859-2, the other coming from a UTF-8 file.

The current Ediff code will compare these buffers different.  If we
want them to compare equal instead, this means that `ediff' and
`ediff-buffers' will produce different results for the same two files.

The different EOL format is just a special case of this general
problem.

In a discussion in Oct 2007, Stefan said that using the buffer's
encoding is wrong, see:

  http://lists.gnu.org/archive/html/emacs-devel/2007-10/msg01381.html

Stefan wanted to use the equivalent of emacs-mule for Emacs 23 instead
of buffer's encoding, but do we have such an encoding now?  Is
no-conversion-multibyte it?  Or maybe utf-8 is good enough?

But first, we should decide whether we want such buffers to compare
equal or not.

We could also let them compare equal, but display a message to the
effect that the buffers define different encoding for saving them to
files.

Opinions?

Message #80 received at submit <at> emacsbugs.donarmstrong.com (full text, mbox):

From: "Drew Adams" <drew.adams <at> oracle.com>
To: "'Eli Zaretskii'" <eliz <at> gnu.org>, <1183 <at> debbugs.gnu.org>
Cc: <bug-gnu-emacs <at> gnu.org>, "'Michael Kifer'" <kifer <at> cs.stonybrook.edu>
Subject: RE: bug#1183: 23.0.60; ediff-buffers is broken
Date: Fri, 17 Oct 2008 07:36:33 -0700

> From: Eli Zaretskii Sent: Friday, October 17, 2008 5:39 AM
> > > > If so, this is expected:
> > > 
> > > Well it certainly isn't expected in Emacs 20, 21, or 22, 
> > > where ediff-buffers works perfectly for these two files.
> > > Why this change for Emacs 23? What's the gain?
> > 
> > Sorry, you are right: Emacs 22 also uses --binary, but 
> > doesn't expose this problem.  So I guess something else is
> > at work here.  I'll look closer when I have time.
> 
> Found the culprit.  It's in ediff-make-temp-file:
> 
>   (let* ( ...
>   	 (coding-system-for-write
> 	  (ediff-with-current-buffer buff
> 	    (if (boundp 'buffer-file-coding-system)
> 		buffer-file-coding-system
> 	      ediff-coding-system-for-write)))
> 
> This let-binds the coding-system with which we will write the two
> buffers to temporary files, to the original encoding of the files read
> into the buffers being compared.  The temporary files are then
> submitted to Diff, and that makes each line compare different because
> of the different EOL format.
> 
> Emacs 22 unconditionally uses ediff-coding-system-for-write here,
> which is set to `no-conversion', so this problem does not happen in
> Emacs 22.
> 
> This change was made in Aug 2007, and Michael Kifer wrote a bit later
> on emacs-devel that using no-conversion was screwing some other
> use-case (I couldn't find the description of that use-case, although
> Michael said it was reported earlier on emacs-devel).
> 
> I could easily fix this particular problem by forcing the EOL
> conversion to -unix, but I think there's a broader issue here, and we
> might as well handle it now.  The issue is this: suppose we have two
> buffers whose text is identical in the internal Emacs representation
> of characters, but whose values of buffer-file-coding-system differ--
> do we want such buffers to compare equal or not?  For example, the
> buffers could contain Latin-2 text, with one buffer coming from a file
> encoded in ISO-8859-2, the other coming from a UTF-8 file.
> 
> The current Ediff code will compare these buffers different.  If we
> want them to compare equal instead, this means that `ediff' and
> `ediff-buffers' will produce different results for the same two files.
> 
> The different EOL format is just a special case of this general
> problem.
> 
> In a discussion in Oct 2007, Stefan said that using the buffer's
> encoding is wrong, see:
> 
>   http://lists.gnu.org/archive/html/emacs-devel/2007-10/msg01381.html
> 
> Stefan wanted to use the equivalent of emacs-mule for Emacs 23 instead
> of buffer's encoding, but do we have such an encoding now?  Is
> no-conversion-multibyte it?  Or maybe utf-8 is good enough?
> 
> But first, we should decide whether we want such buffers to compare
> equal or not.
> 
> We could also let them compare equal, but display a message to the
> effect that the buffers define different encoding for saving them to
> files.
> 
> Opinions?

I don't have much of an opinion because of ignorance in this area. If you are
asking whether ediff should treat buffers as different when they differ only by
encoding or line endings, I'd say probably yes and know: It would be good to
give the user all knowledge that ediff has, and then let the user control how to
compare. A message could let the user know about line-ending differences, for
example, and ask if you want to compare using only Emacs's internal encoding (or
whatever the correct terminology is) - that is, ignoring just the line-ending
difference.

IOW, give the user the knowledge and control. What's unacceptable is for Emacs
to throw up its hands and just say the two are different, without saying/showing
how they differ. The user should be able to drill down and see all possible
differences, perhaps showing different kinds of differences in different ways.

I'll leave the rest to you guys who know about this stuff. You get the drift of
what's missing/wrong. Thx.

Message #90 received at submit <at> emacsbugs.donarmstrong.com (full text, mbox):

From: Stefan Monnier <monnier <at> iro.umontreal.ca>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 1183 <at> debbugs.gnu.org, bug-gnu-emacs <at> gnu.org,
        Michael Kifer <kifer <at> cs.stonybrook.edu>
Subject: Re: bug#1183: 23.0.60; ediff-buffers is broken
Date: Fri, 17 Oct 2008 12:02:11 -0400

> In a discussion in Oct 2007, Stefan said that using the buffer's
> encoding is wrong, see:

>   http://lists.gnu.org/archive/html/emacs-devel/2007-10/msg01381.html

> Stefan wanted to use the equivalent of emacs-mule for Emacs 23 instead
> of buffer's encoding, but do we have such an encoding now?  Is
> no-conversion-multibyte it?  Or maybe utf-8 is good enough?

As mentioned in the past, I think `no-conversion' should be killed
because it's confusing.  As for the problem at hand, utf-8-emacs-unix is
what we want to use.

> But first, we should decide whether we want such buffers to compare
> equal or not.

I believe we do, because it's called ediff-buffers.  There's ediff-files
for when you want to compare the files.

> We could also let them compare equal, but display a message to the
> effect that the buffers define different encoding for saving them to
> files.
> Opinions?

That would be fine, indeed.


        Stefan

Message #100 received at submit <at> emacsbugs.donarmstrong.com (full text, mbox):

From: "Drew Adams" <drew.adams <at> oracle.com>
To: "'Stefan Monnier'" <monnier <at> iro.umontreal.ca>,
        <1183 <at> debbugs.gnu.org>, "'Eli Zaretskii'" <eliz <at> gnu.org>
Cc: <bug-gnu-emacs <at> gnu.org>, "'Michael Kifer'" <kifer <at> cs.stonybrook.edu>
Subject: RE: bug#1183: 23.0.60; ediff-buffers is broken
Date: Fri, 17 Oct 2008 09:48:06 -0700

> > But first, we should decide whether we want such buffers to compare
> > equal or not.
> 
> I believe we do, because it's called ediff-buffers.  There's 
> ediff-files for when you want to compare the files.

That's terrible. Ediff-buffers has always been usable directly for buffers
visiting files also. 

It's OK for ediff-buffers to be more refined than before, to be able to take
into account current encodings etc. for the buffers, but it should inform the
user of the situation and let the user, if s?he wants, proceed to compare the
buffers using the same encodings etc. - or whatever is necessary to see the
actual textual differences, beyond encoding etc. differences.

The same behavior as previously (Emacs 22) should be available as a user choice
if the only differences are line endings, encodings, etc. And such differences
as line endings should at least be treated as differences and shown as such.
It's no good to just say the buffers are different, without offering more info
than that.

IOW, ediff-buffers should be at least as useful as it was before. Adding coding
diffs should be a plus, not a minus. Simply punting, showing a single giant diff
with no possible refinement and no explanation, is not helpful.

> > We could also let them compare equal, but display a message to the
> > effect that the buffers define different encoding for saving them to
> > files. Opinions?
> 
> That would be fine, indeed.

Fine, but not enough. If a user wants to see the textual differences between the
two buffers, the info that the encodings are different is not helpful enough to
get the job done. In the case described, there are real textual differences (an
added Lisp sexp), and ediff-buffers is not at all helpful in showing them.

Message #110 received at submit <at> emacsbugs.donarmstrong.com (full text, mbox):

From: Michael Kifer <kifer <at> cs.sunysb.edu>
To: "Drew Adams" <drew.adams <at> oracle.com>
Cc: "'Stefan Monnier'" <monnier <at> iro.umontreal.ca>,
        <1183 <at> debbugs.gnu.org>, "'Eli Zaretskii'" <eliz <at> gnu.org>,
        <bug-gnu-emacs <at> gnu.org>, "'Michael Kifer'" <kifer <at> cs.stonybrook.edu>
Subject: Re: bug#1183: 23.0.60; ediff-buffers is broken
Date: Fri, 17 Oct 2008 13:05:33 -0400

On Fri, 17 Oct 2008 09:48:06 -0700
"Drew Adams" <drew.adams <at> oracle.com> wrote:

> > > But first, we should decide whether we want such buffers to compare
> > > equal or not.
> > 
> > I believe we do, because it's called ediff-buffers.  There's 
> > ediff-files for when you want to compare the files.
> 
> That's terrible. Ediff-buffers has always been usable directly for buffers
> visiting files also. 

I didn't see the original post, but the general idea was that whenever things
look the same in Emacs they should be treated as equal (or equal module spaces).
I do not think the user should be bothered with encodings. Copying from buffer
to buffer should also be transparent. (And ediff-files and ediff-buffers should
produce the same results.)

Unfortunately, I have not been following the developments in the last few
years, and my knowledge of the mechanics became rusty.


	--michael  


> It's OK for ediff-buffers to be more refined than before, to be able to take
> into account current encodings etc. for the buffers, but it should inform the
> user of the situation and let the user, if s?he wants, proceed to compare the
> buffers using the same encodings etc. - or whatever is necessary to see the
> actual textual differences, beyond encoding etc. differences.
> 
> The same behavior as previously (Emacs 22) should be available as a user choice
> if the only differences are line endings, encodings, etc. And such differences
> as line endings should at least be treated as differences and shown as such.
> It's no good to just say the buffers are different, without offering more info
> than that.
> 
> IOW, ediff-buffers should be at least as useful as it was before. Adding coding
> diffs should be a plus, not a minus. Simply punting, showing a single giant diff
> with no possible refinement and no explanation, is not helpful.
> 
> > > We could also let them compare equal, but display a message to the
> > > effect that the buffers define different encoding for saving them to
> > > files. Opinions?
> > 
> > That would be fine, indeed.
> 
> Fine, but not enough. If a user wants to see the textual differences between the
> two buffers, the info that the encodings are different is not helpful enough to
> get the job done. In the case described, there are real textual differences (an
> added Lisp sexp), and ediff-buffers is not at all helpful in showing them.
> 
> 
> 

Message #120 received at submit <at> emacsbugs.donarmstrong.com (full text, mbox):

From: "Drew Adams" <drew.adams <at> oracle.com>
To: <kifer <at> cs.sunysb.edu>
Cc: "'Stefan Monnier'" <monnier <at> iro.umontreal.ca>,
        <1183 <at> debbugs.gnu.org>, "'Eli Zaretskii'" <eliz <at> gnu.org>,
        <bug-gnu-emacs <at> gnu.org>, "'Michael Kifer'" <kifer <at> cs.stonybrook.edu>
Subject: RE: bug#1183: 23.0.60; ediff-buffers is broken
Date: Fri, 17 Oct 2008 10:17:27 -0700

> > > > But first, we should decide whether we want such 
> > > > buffers to compare equal or not.
> > > 
> > > I believe we do, because it's called ediff-buffers.  There's 
> > > ediff-files for when you want to compare the files.
> > 
> > That's terrible. Ediff-buffers has always been usable 
> > directly for buffers visiting files also. 
> 
> I didn't see the original post, but the general idea was that 
> whenever things look the same in Emacs they should be treated
> as equal (or equal module spaces). I do not think the user
> should be bothered with encodings. Copying from buffer
> to buffer should also be transparent. (And ediff-files and 
> ediff-buffers should produce the same results.)
> 
> Unfortunately, I have not been following the developments in 
> the last few years, and my knowledge of the mechanics became rusty.

Everything Michael said sounds right to me.

IMO, it would be OK for ediff-buffers, perhaps optionally, to report encoding
differences *also*, but that's not what ediff-buffers is really about.

It's about detecting textual differences: `foo' vs `fog'. If it cannot tell you
that `foo' is different from `fog', and it only tells you that the two buffers
are (somehow) different, then it is being less helpful than before, not more.

Message #130 received at submit <at> emacsbugs.donarmstrong.com (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Drew Adams <drew.adams <at> oracle.com>
Cc: kifer <at> cs.sunysb.edu, monnier <at> iro.umontreal.ca,
        1183 <at> debbugs.gnu.org, bug-gnu-emacs <at> gnu.org,
        kifer <at> cs.stonybrook.edu
Subject: Re: bug#1183: 23.0.60; ediff-buffers is broken
Date: Fri, 17 Oct 2008 20:15:04 +0200

> From: "Drew Adams" <drew.adams <at> oracle.com>
> Cc: "'Stefan Monnier'" <monnier <at> iro.umontreal.ca>,
>         <1183 <at> emacsbugs.donarmstrong.com>, "'Eli Zaretskii'" <eliz <at> gnu.org>,
>         <bug-gnu-emacs <at> gnu.org>, "'Michael Kifer'" <kifer <at> cs.stonybrook.edu>
> Date: Fri, 17 Oct 2008 10:17:27 -0700
> 
> > > > > But first, we should decide whether we want such 
> > > > > buffers to compare equal or not.
> > > > 
> > > > I believe we do, because it's called ediff-buffers.  There's 
> > > > ediff-files for when you want to compare the files.
> > > 
> > > That's terrible. Ediff-buffers has always been usable 
> > > directly for buffers visiting files also. 
> > 
> > I didn't see the original post, but the general idea was that 
> > whenever things look the same in Emacs they should be treated
> > as equal (or equal module spaces). I do not think the user
> > should be bothered with encodings. Copying from buffer
> > to buffer should also be transparent. (And ediff-files and 
> > ediff-buffers should produce the same results.)
> > 
> > Unfortunately, I have not been following the developments in 
> > the last few years, and my knowledge of the mechanics became rusty.
> 
> Everything Michael said sounds right to me.

Then why did you say "that's terrible" in response to Stefan who
expressed the same view as Michael?  They both say that what is
_displayed_ the same in Emacs should compare equal in ediff-buffers.

OTOH, "M-x ediff" that compares _files_ will still show differences
for identical text encoded differently in each of the files.

Message #140 received at 1183 <at> emacsbugs.donarmstrong.com (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Drew Adams <drew.adams <at> oracle.com>
Cc: monnier <at> iro.umontreal.ca, 1183 <at> debbugs.gnu.org,
        bug-gnu-emacs <at> gnu.org, kifer <at> cs.stonybrook.edu
Subject: Re: bug#1183: 23.0.60; ediff-buffers is broken
Date: Fri, 17 Oct 2008 20:19:28 +0200

> From: "Drew Adams" <drew.adams <at> oracle.com>
> Cc: <bug-gnu-emacs <at> gnu.org>, "'Michael Kifer'" <kifer <at> cs.stonybrook.edu>
> Date: Fri, 17 Oct 2008 09:48:06 -0700
> 
> > > We could also let them compare equal, but display a message to the
> > > effect that the buffers define different encoding for saving them to
> > > files. Opinions?
> > 
> > That would be fine, indeed.
> 
> Fine, but not enough. If a user wants to see the textual differences between the
> two buffers, the info that the encodings are different is not helpful enough to
> get the job done.

You misunderstood.  My suggestion, to which Stefan agreed, was that
_in_addition_ to showing any textual differences, ediff-buffers would
display a warning saying that the buffers _also_ differ in the way
they will be encoded on their disk files.

In the extreme case that the text is identical, only the warning will
remain as the single sign of a difference.

> In the case described, there are real textual differences (an
> added Lisp sexp), and ediff-buffers is not at all helpful in showing them.

After Ediff is fixed, you will see the diffs you see in Emacs 22, plus
a message saying that the EOL format is different.

Message #145 received at submit <at> emacsbugs.donarmstrong.com (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Stefan Monnier <monnier <at> iro.umontreal.ca>, Kenichi Handa <handa <at> m17n.org>
Cc: 1183 <at> debbugs.gnu.org, bug-gnu-emacs <at> gnu.org,
        kifer <at> cs.stonybrook.edu
Subject: Re: bug#1183: 23.0.60; ediff-buffers is broken
Date: Fri, 17 Oct 2008 20:34:27 +0200

> From: Stefan Monnier <monnier <at> iro.umontreal.ca>
> Cc: 1183 <at> emacsbugs.donarmstrong.com,  bug-gnu-emacs <at> gnu.org,  Michael Kifer <kifer <at> cs.stonybrook.edu>
> Date: Fri, 17 Oct 2008 12:02:11 -0400
> 
> > In a discussion in Oct 2007, Stefan said that using the buffer's
> > encoding is wrong, see:
> 
> >   http://lists.gnu.org/archive/html/emacs-devel/2007-10/msg01381.html
> 
> > Stefan wanted to use the equivalent of emacs-mule for Emacs 23 instead
> > of buffer's encoding, but do we have such an encoding now?  Is
> > no-conversion-multibyte it?  Or maybe utf-8 is good enough?
> 
> As mentioned in the past, I think `no-conversion' should be killed
> because it's confusing.  As for the problem at hand, utf-8-emacs-unix is
> what we want to use.

Suppose we write the temp files with utf-8-emacs-unix encoding--won't
that bite us when the output from Diff is then read with raw-text (see
ediff-exec-process)?

If raw-text won't work for utf-8-emacs output from Diff, is there a
better way than overriding the coding-system priorities with
`with-coding-priority'?

> > But first, we should decide whether we want such buffers to compare
> > equal or not.
> 
> I believe we do, because it's called ediff-buffers.  There's ediff-files
> for when you want to compare the files.

Don't you think having direct file comparison yield results that are
different from comparing buffers that visit those same files will be
confusing?

Message #155 received at submit <at> emacsbugs.donarmstrong.com (full text, mbox):

From: "Drew Adams" <drew.adams <at> oracle.com>
To: "'Eli Zaretskii'" <eliz <at> gnu.org>
Cc: <monnier <at> iro.umontreal.ca>, <1183 <at> debbugs.gnu.org>,
        <bug-gnu-emacs <at> gnu.org>, <kifer <at> cs.stonybrook.edu>
Subject: RE: bug#1183: 23.0.60; ediff-buffers is broken
Date: Fri, 17 Oct 2008 11:35:43 -0700

> > > > We could also let them compare equal, but display a 
> > > > message to the effect that the buffers define different
> > > > encoding for saving them to files. Opinions?
> > > 
> > > That would be fine, indeed.
> > 
> > Fine, but not enough. If a user wants to see the textual 
> > differences between the two buffers, the info that the
> > encodings are different is not helpful enough to get the
> > job done.
> 
> You misunderstood.  My suggestion, to which Stefan agreed, was that
> _in_addition_ to showing any textual differences, ediff-buffers would
> display a warning saying that the buffers _also_ differ in the way
> they will be encoded on their disk files.
> 
> In the extreme case that the text is identical, only the warning will
> remain as the single sign of a difference.
> 
> > In the case described, there are real textual differences (an
> > added Lisp sexp), and ediff-buffers is not at all helpful 
> > in showing them.
> 
> After Ediff is fixed, you will see the diffs you see in Emacs 22, plus
> a message saying that the EOL format is different.

Sounds good. Sorry for misunderstanding. "Let them compare equal" threw me off;
I thought you were saying that the two buffers I reported on *should* be
considered equal by ediff-buffers.

Message #160 received at submit <at> emacsbugs.donarmstrong.com (full text, mbox):

From: "Drew Adams" <drew.adams <at> oracle.com>
To: "'Eli Zaretskii'" <eliz <at> gnu.org>
Cc: <kifer <at> cs.sunysb.edu>, <monnier <at> iro.umontreal.ca>,
        <1183 <at> debbugs.gnu.org>, <bug-gnu-emacs <at> gnu.org>,
        <kifer <at> cs.stonybrook.edu>
Subject: RE: bug#1183: 23.0.60; ediff-buffers is broken
Date: Fri, 17 Oct 2008 11:35:46 -0700

> > > > > > But first, we should decide whether we want such 
> > > > > > buffers to compare equal or not.
> > > > > 
> > > > > I believe we do, because it's called ediff-buffers.  There's 
> > > > > ediff-files for when you want to compare the files.
> > > > 
> > > > That's terrible. Ediff-buffers has always been usable 
> > > > directly for buffers visiting files also. 
> > > 
> > > I didn't see the original post, but the general idea was that 
> > > whenever things look the same in Emacs they should be treated
> > > as equal (or equal module spaces). I do not think the user
> > > should be bothered with encodings. Copying from buffer
> > > to buffer should also be transparent. (And ediff-files and 
> > > ediff-buffers should produce the same results.)
> > > 
> > > Unfortunately, I have not been following the developments in 
> > > the last few years, and my knowledge of the mechanics 
> became rusty.
> > 
> > Everything Michael said sounds right to me.
> 
> Then why did you say "that's terrible" in response to Stefan who
> expressed the same view as Michael?  They both say that what is
> _displayed_ the same in Emacs should compare equal in ediff-buffers.

I guess I misunderstood. I thought that Stefan was saying that ediff-buffers
should continue to do what it is doing now: just show one big difference, with
no distinction of textual differences, and force users to use ediff-files to see
the textual differences. If he meant the same thing as Michael, then we agree.

The two buffers I reported on should *not* compare equal, but neither should
ediff-buffers just throw up its hands and say only "they're different somehow".
I mistakenly thought that Stefan was saying that ediff-buffers should not
distinguish their textual differences but should just report that they are
different (one big diff). IOW, I thought he was saying that the current behavior
is correct for ediff-buffers but that ediff-files should, as always, show the
textual differences.

> OTOH, "M-x ediff" that compares _files_ will still show differences
> for identical text encoded differently in each of the files.

Again, I have no problem with ediff commands also showing or otherwise
identifying encoding differences. 

What I objected to was that textual differences were being effectively lost,
because ediff-buffers just displays one big diff with identical, full-buffer
highlighting - it doesn't show the textual differences at all.

Sorry for any misunderstanding. It sounds now like we're all in more or less
agreement.

Message #180 received at submit <at> emacsbugs.donarmstrong.com (full text, mbox):

From: Michael Kifer <kifer <at> cs.sunysb.edu>
To: "Drew Adams" <drew.adams <at> oracle.com>
Cc: "'Eli Zaretskii'" <eliz <at> gnu.org>, <monnier <at> iro.umontreal.ca>,
        <1183 <at> debbugs.gnu.org>, <bug-gnu-emacs <at> gnu.org>,
        <kifer <at> cs.stonybrook.edu>
Subject: Re: bug#1183: 23.0.60; ediff-buffers is broken
Date: Fri, 17 Oct 2008 23:17:31 -0400

On Fri, 17 Oct 2008 11:35:46 -0700
"Drew Adams" <drew.adams <at> oracle.com> wrote:

> > > > > > > But first, we should decide whether we want such 
> > > > > > > buffers to compare equal or not.
> > > > > > 
> > > > > > I believe we do, because it's called ediff-buffers.  There's 
> > > > > > ediff-files for when you want to compare the files.
> > > > > 
> > > > > That's terrible. Ediff-buffers has always been usable 
> > > > > directly for buffers visiting files also. 
> > > > 
> > > > I didn't see the original post, but the general idea was that 
> > > > whenever things look the same in Emacs they should be treated
> > > > as equal (or equal module spaces). I do not think the user
> > > > should be bothered with encodings. Copying from buffer
> > > > to buffer should also be transparent. (And ediff-files and 
> > > > ediff-buffers should produce the same results.)
> > > > 
> > > > Unfortunately, I have not been following the developments in 
> > > > the last few years, and my knowledge of the mechanics 
> > became rusty.
> > > 
> > > Everything Michael said sounds right to me.
> > 
> > Then why did you say "that's terrible" in response to Stefan who
> > expressed the same view as Michael?  They both say that what is
> > _displayed_ the same in Emacs should compare equal in ediff-buffers.
> 
> I guess I misunderstood. I thought that Stefan was saying that ediff-buffers
> should continue to do what it is doing now: just show one big difference, with
> no distinction of textual differences, and force users to use ediff-files to see
> the textual differences. If he meant the same thing as Michael, then we agree.
> 
> The two buffers I reported on should *not* compare equal, but neither should
> ediff-buffers just throw up its hands and say only "they're different somehow".

To make it clear, nobody was saying that the two buffers should be compared
equal. The issue is: how do we make diff (NOT ediff) to recognize that these
buffers are nearly identical modulo some minor stuff?

Ediff-buffers does almost the right thing (at least, was doing until things
changed in emacs). It would save the buffers in temp files using the *same*
encoding, so all that crap is pushed out of the way. Then it would run diff on
the temp files.  Since the encodings are the same, diff would find what is
different and then ediff will display that. (With all its complexity, ediff is
just a front-end for diff.) So, for ediff-buffers, the question is which
encoding to use.

For ediff-files things seem to be worse: it runs diff on the original files, so
if one has DOS line endings and the other does not then it all depends on what
diff does.  This is why sometimes you run ediff files on 2 files that are
nearly identical and get one big diff region equal to the entire file.
This is a bit annoying, but not too bad, since hitting * should fix the
problem: it would save the diff regions using the same encoding and will run
diff over them.

So, basically, it boils down to choosing the right encoding.
I am not sure which is right.
Long time ago, it was decided to use no-conversion. Then someone found a bad
case and suggested to use the buffer coding system *if* it is set.
This seemed to work better, but still had some problems.

Back then Stefan suggested emacs-mule instead of no-conversion, but for some
reason this was not done--don't remember why. He also said that things will
change in emacs 23, but I did not follow that development.

What has changed in emacs 23 with respect to this issue?

> I mistakenly thought that Stefan was saying that ediff-buffers should not
> distinguish their textual differences but should just report that they are
> different (one big diff). IOW, I thought he was saying that the current
> behavior
> is correct for ediff-buffers but that ediff-files should, as always, show the
> textual differences.
> 
> > OTOH, "M-x ediff" that compares _files_ will still show differences
> > for identical text encoded differently in each of the files.
> 
> Again, I have no problem with ediff commands also showing or otherwise
> identifying encoding differences. 

See above. The point was not that textual diffs should be lost, but that it
should be made possible for the diff program to recognize identical regions
(modulo dos line endings, etc) as such.


> What I objected to was that textual differences were being effectively lost,
> because ediff-buffers just displays one big diff with identical, full-buffer
> highlighting - it doesn't show the textual differences at all.


This was a misunderstanding.

Message #190 received at submit <at> emacsbugs.donarmstrong.com (full text, mbox):

From: "Drew Adams" <drew.adams <at> oracle.com>
To: <kifer <at> cs.sunysb.edu>
Cc: "'Eli Zaretskii'" <eliz <at> gnu.org>, <monnier <at> iro.umontreal.ca>,
        <1183 <at> debbugs.gnu.org>, <bug-gnu-emacs <at> gnu.org>,
        <kifer <at> cs.stonybrook.edu>
Subject: RE: bug#1183: 23.0.60; ediff-buffers is broken
Date: Fri, 17 Oct 2008 20:43:27 -0700

> To make it clear, nobody was saying that the two buffers 
> should be compared
> equal. The issue is: how do we make diff (NOT ediff) to 
> recognize that these
> buffers are nearly identical modulo some minor stuff?
> 
> Ediff-buffers does almost the right thing (at least, was 
> doing until things
> changed in emacs). It would save the buffers in temp files 
> using the *same*
> encoding, so all that crap is pushed out of the way. Then it 
> would run diff on
> the temp files.  Since the encodings are the same, diff would 
> find what is
> different and then ediff will display that. (With all its 
> complexity, ediff is
> just a front-end for diff.) So, for ediff-buffers, the 
> question is which
> encoding to use.
> 
> For ediff-files things seem to be worse: it runs diff on the 
> original files, so
> if one has DOS line endings and the other does not then it 
> all depends on what
> diff does.  This is why sometimes you run ediff files on 2 
> files that are
> nearly identical and get one big diff region equal to the entire file.
> This is a bit annoying, but not too bad, since hitting * 
> should fix the
> problem: it would save the diff regions using the same 
> encoding and will run
> diff over them.
> 
> So, basically, it boils down to choosing the right encoding.
> I am not sure which is right.
> Long time ago, it was decided to use no-conversion. Then 
> someone found a bad
> case and suggested to use the buffer coding system *if* it is set.
> This seemed to work better, but still had some problems.
> 
> Back then Stefan suggested emacs-mule instead of 
> no-conversion, but for some
> reason this was not done--don't remember why. He also said 
> that things will
> change in emacs 23, but I did not follow that development.
> 
> What has changed in emacs 23 with respect to this issue?
> 
> > Again, I have no problem with ediff commands also showing 
> > or otherwise identifying encoding differences. 
> 
> See above. The point was not that textual diffs should be 
> lost, but that it should be made possible for the diff program
> to recognize identical regions
> (modulo dos line endings, etc) as such.

Got it. Thanks for a clear description. Sorry for having misunderstood.

FWIW, the need to hit `*' to refine for ediff in the situation you describe has
been true for a while, and it is no big deal. Generally, highlighting just the
differences is easier to read, but highlighting everything and then refining is
OK too, if need be.

Thanks to all for working on this.

Message #200 received at submit <at> emacsbugs.donarmstrong.com (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: kifer <at> cs.sunysb.edu, handa <at> m17n.org
Cc: drew.adams <at> oracle.com, monnier <at> iro.umontreal.ca,
        1183 <at> debbugs.gnu.org, bug-gnu-emacs <at> gnu.org,
        kifer <at> cs.stonybrook.edu
Subject: Re: bug#1183: 23.0.60; ediff-buffers is broken
Date: Sat, 18 Oct 2008 11:07:34 +0200

> Date: Fri, 17 Oct 2008 23:17:31 -0400
> From: Michael Kifer <kifer <at> cs.sunysb.edu>
> Cc: "'Eli Zaretskii'" <eliz <at> gnu.org>, <monnier <at> iro.umontreal.ca>,
>         <1183 <at> emacsbugs.donarmstrong.com>, <bug-gnu-emacs <at> gnu.org>,
>         <kifer <at> cs.stonybrook.edu>
> 
> Ediff-buffers does almost the right thing (at least, was doing until things
> changed in emacs). It would save the buffers in temp files using the *same*
> encoding, so all that crap is pushed out of the way. Then it would run diff on
> the temp files.  Since the encodings are the same, diff would find what is
> different and then ediff will display that. (With all its complexity, ediff is
> just a front-end for diff.) So, for ediff-buffers, the question is which
> encoding to use.

The right encoding in Emacs 23 is utf-8-emacs-unix.  The problem with
that is that ediff-exec-process then uses raw-text to read the output
from Diff back into Emacs.  While raw-text is probably OK for reading
Diff output from comparing _files_, I'm afraid it will not be TRT for
reading output from comparing 2 temporary files encoded in
utf-8-emacs-unix.  If my fears are justified, I guess we will have to
modify ediff-exec-process so as to use utf-8-emacs-unix when
ediff-job-name has "buffers" in it.

> For ediff-files things seem to be worse: it runs diff on the original files, so
> if one has DOS line endings and the other does not then it all depends on what
> diff does.  This is why sometimes you run ediff files on 2 files that are
> nearly identical and get one big diff region equal to the entire file.
> This is a bit annoying, but not too bad, since hitting * should fix the
> problem: it would save the diff regions using the same encoding and will run
> diff over them.

Yes, but will hitting "*" help for Drew's use-case?  AFAIU, it will
"magically" show only the textual diffs, with no real explanation how
come the original display shows that every line is different, is that
right?

(Btw, "M-x ediff" actually does not pass the --binary switch to Diff,
so the original Ediff display is actually what Drew wanted, but let's
say we are doing the same on Unix, where changes in the EOL format are
reported by Diff as differences by default.)

> Back then Stefan suggested emacs-mule instead of no-conversion, but for some
> reason this was not done--don't remember why.

No special reason.

> He also said that things will change in emacs 23, but I did not
> follow that development.

See above.  I see that Stefan introduced emacs-internal into the Emacs
22 branch, but I don't see it on the trunk yet.  If and when it
arrives, we should use that one instead.

Message #210 received at submit <at> emacsbugs.donarmstrong.com (full text, mbox):

From: Stefan Monnier <monnier <at> iro.umontreal.ca>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: kifer <at> cs.sunysb.edu, handa <at> m17n.org, drew.adams <at> oracle.com,
        1183 <at> debbugs.gnu.org, bug-gnu-emacs <at> gnu.org,
        kifer <at> cs.stonybrook.edu
Subject: Re: bug#1183: 23.0.60; ediff-buffers is broken
Date: Sat, 18 Oct 2008 22:17:06 -0400

>> Ediff-buffers does almost the right thing (at least, was doing until
>> things changed in emacs). It would save the buffers in temp files
>> using the *same* encoding, so all that crap is pushed out of the
>> way. Then it would run diff on the temp files.  Since the encodings
>> are the same, diff would find what is different and then ediff will
>> display that. (With all its complexity, ediff is just a front-end for
>> diff.) So, for ediff-buffers, the question is which encoding to use.

> The right encoding in Emacs 23 is utf-8-emacs-unix.

Indeed.  This encoding is now also available under the new name
`emacs-internal'. [ As you may have seen, I also added it to the
Emacs-22 branch, although I doubt we'll release anything from that
branch. ]

> The problem with that is that ediff-exec-process then uses raw-text to
> read the output from Diff back into Emacs.

Yes, raw-text is wrong.  It should probably use undecided at least, or
otherwise try to be more clever and use the encoding used by the
source files.

> While raw-text is probably OK for reading Diff output from comparing
> _files_,

There's worse, but for Unicode files (or for latin-1 files, or for ...)
it's kind of ugly.  I suspect that for utf-16 and Chinese it's
even worse.

> I'm afraid it will not be TRT for reading output from
> comparing 2 temporary files encoded in utf-8-emacs-unix.  If my fears
> are justified, I guess we will have to modify ediff-exec-process so as
> to use utf-8-emacs-unix when ediff-job-name has "buffers" in it.

Yes, that's part of the problem.

>> Back then Stefan suggested emacs-mule instead of no-conversion, but for some
>> reason this was not done--don't remember why.

> No special reason.

Actually, after looking at it a little bit, there are some reasons: the
encoding is currently set in ediff-make-temp-file which is used in many
different circumstances, some of which should use emacs-internal, but
maybe not all.  So there's more work to do in ediff to handle
encoding issues reliably.


        Stefan

Message #220 received at submit <at> emacsbugs.donarmstrong.com (full text, mbox):

From: Stefan Monnier <monnier <at> iro.umontreal.ca>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: Kenichi Handa <handa <at> m17n.org>, 1183 <at> debbugs.gnu.org,
        bug-gnu-emacs <at> gnu.org, kifer <at> cs.stonybrook.edu
Subject: Re: bug#1183: 23.0.60; ediff-buffers is broken
Date: Sat, 18 Oct 2008 22:21:39 -0400

> Suppose we write the temp files with utf-8-emacs-unix encoding--won't
> that bite us when the output from Diff is then read with raw-text (see
> ediff-exec-process)?

Yes, tho not much worse than what we have now (where we write in
utf-8/latin-1/younameit and read it back as raw-text aka binary).
I.e. it's not a new problem.

>> > But first, we should decide whether we want such buffers to compare
>> > equal or not.
>> I believe we do, because it's called ediff-buffers.  There's ediff-files
>> for when you want to compare the files.
> Don't you think having direct file comparison yield results that are
> different from comparing buffers that visit those same files will be
> confusing?

Yes.  But I don't know of any better alternative.  Your suggestion to
emit a warning about the different encoding should hopefully make things
more clear.


        Stefan

Message #230 received at submit <at> emacsbugs.donarmstrong.com (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Stefan Monnier <monnier <at> iro.umontreal.ca>
Cc: kifer <at> cs.sunysb.edu, handa <at> m17n.org, drew.adams <at> oracle.com,
        1183 <at> debbugs.gnu.org, bug-gnu-emacs <at> gnu.org,
        kifer <at> cs.stonybrook.edu
Subject: Re: bug#1183: 23.0.60; ediff-buffers is broken
Date: Sun, 19 Oct 2008 09:17:58 +0200

> From: Stefan Monnier <monnier <at> iro.umontreal.ca>
> Cc: kifer <at> cs.sunysb.edu,  handa <at> m17n.org,  drew.adams <at> oracle.com,  1183 <at> emacsbugs.donarmstrong.com,  bug-gnu-emacs <at> gnu.org,  kifer <at> cs.stonybrook.edu
> Date: Sat, 18 Oct 2008 22:17:06 -0400
> 
> >> Back then Stefan suggested emacs-mule instead of no-conversion, but for some
> >> reason this was not done--don't remember why.
> 
> > No special reason.
> 
> Actually, after looking at it a little bit, there are some reasons: the
> encoding is currently set in ediff-make-temp-file which is used in many
> different circumstances, some of which should use emacs-internal, but
> maybe not all.

We have ediff-job-name to distinguish between the different
use-cases.  Ediff already does that in several places, e.g., to pass
the "--binary" option to Diff.

Message #240 received at submit <at> emacsbugs.donarmstrong.com (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Stefan Monnier <monnier <at> iro.umontreal.ca>
Cc: kifer <at> cs.sunysb.edu, handa <at> m17n.org, drew.adams <at> oracle.com,
        1183 <at> debbugs.gnu.org, bug-gnu-emacs <at> gnu.org,
        kifer <at> cs.stonybrook.edu
Subject: Re: bug#1183: 23.0.60; ediff-buffers is broken
Date: Sun, 19 Oct 2008 09:23:43 +0200

> From: Stefan Monnier <monnier <at> iro.umontreal.ca>
> Cc: kifer <at> cs.sunysb.edu,  handa <at> m17n.org,  drew.adams <at> oracle.com,  1183 <at> emacsbugs.donarmstrong.com,  bug-gnu-emacs <at> gnu.org,  kifer <at> cs.stonybrook.edu
> Date: Sat, 18 Oct 2008 22:17:06 -0400
> 
> > The problem with that is that ediff-exec-process then uses raw-text to
> > read the output from Diff back into Emacs.
> 
> Yes, raw-text is wrong.  It should probably use undecided at least, or
> otherwise try to be more clever and use the encoding used by the
> source files.

There are 2 files involved in this, and each one can be in a different
encoding.  Thus, undecided is also not quite correct, since it will at
best most probably pick up at random one of the 2 encodings.

Message #250 received at submit <at> emacsbugs.donarmstrong.com (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: drew.adams <at> oracle.com
Cc: kifer <at> cs.sunysb.edu, 1183 <at> debbugs.gnu.org,
        bug-gnu-emacs <at> gnu.org, kifer <at> cs.stonybrook.edu
Subject: Re: bug#1183: 23.0.60; ediff-buffers is broken
Date: Sun, 19 Oct 2008 10:32:53 +0200

Drew, please see if the patch below fixes the problem for you.

If you are not using a very recent CVS code, you will probably need to
use utf-8-emacs-unix instead of emacs-internal in the ediff-init.el
change, because emacs-internal was only introduced yesterday.

Michael, could you please add a warning message in buffer jobs about
differences in the values of buffer-file-coding-system between the
buffers being compared?  In particular, if there are no differences in
a region, but the above values are different, it would be good if
Ediff would say something like "only character-encoding differences"
instead of "only white-space differences".

Thanks.

2008-10-19  Eli Zaretskii  <eliz <at> gnu.org>

	Fix Bug #1183:

	* ediff-diff.el (ediff-exec-process): For buffer jobs, bind
	coding-system-for-read to ediff-coding-system-for-write.

	* ediff-util.el (ediff-make-temp-file): Unconditionally bind
	coding-system-for-write to ediff-coding-system-for-write.

	* ediff-init.el (ediff-coding-system-for-read): Doc fix.
	(ediff-coding-system-for-write): Set to emacs-internal.


Index: lisp/ediff-init.el
===================================================================
RCS file: /cvsroot/emacs/emacs/lisp/ediff-init.el,v
retrieving revision 1.93
diff -u -r1.93 ediff-init.el
--- lisp/ediff-init.el	31 Jul 2008 05:33:43 -0000	1.93
+++ lisp/ediff-init.el	19 Oct 2008 08:20:30 -0000
@@ -719,17 +719,17 @@
 
 (defcustom ediff-coding-system-for-read 'raw-text
   "*The coding system for read to use when running the diff program as a subprocess.
-In most cases, the default will do. However, under certain circumstances in
-Windows NT/98/95 you might need to use something like 'raw-text-dos here.
+In most cases, the default will do.  However, under certain circumstances in
+MS-Windows you might need to use something like 'raw-text-dos here.
 So, if the output that your diff program sends to Emacs contains extra ^M's,
 you might need to experiment here, if the default or 'raw-text-dos doesn't
 work."
   :type 'symbol
   :group 'ediff)
 
-(defcustom ediff-coding-system-for-write 'no-conversion
+(defcustom ediff-coding-system-for-write 'emacs-internal
   "*The coding system for write to use when writing out difference regions
-to temp files when Ediff needs to find fine differences."
+to temp files in buffer jobs and when Ediff needs to find fine differences."
   :type 'symbol
   :group 'ediff)
 
Index: lisp/ediff-util.el
===================================================================
RCS file: /cvsroot/emacs/emacs/lisp/ediff-util.el,v
retrieving revision 1.93
diff -u -r1.93 ediff-util.el
--- lisp/ediff-util.el	31 Jul 2008 05:33:43 -0000	1.93
+++ lisp/ediff-util.el	19 Oct 2008 08:20:55 -0000
@@ -3146,11 +3146,7 @@
 (defun ediff-make-temp-file (buff &optional prefix given-file start end)
   (let* ((p (ediff-convert-standard-filename (or prefix "ediff")))
 	 (short-p p)
-	 (coding-system-for-write
-	  (ediff-with-current-buffer buff
-	    (if (boundp 'buffer-file-coding-system)
-		buffer-file-coding-system
-	      ediff-coding-system-for-write)))
+	 (coding-system-for-write ediff-coding-system-for-write)
 	 f short-f)
     (if (and (fboundp 'msdos-long-file-names)
 	     (not (msdos-long-file-names))
Index: lisp/ediff-diff.el
===================================================================
RCS file: /cvsroot/emacs/emacs/lisp/ediff-diff.el,v
retrieving revision 1.72
diff -u -r1.72 ediff-diff.el
--- lisp/ediff-diff.el	31 Jul 2008 05:33:42 -0000	1.72
+++ lisp/ediff-diff.el	19 Oct 2008 08:21:10 -0000
@@ -1207,7 +1207,13 @@
 ;; args.
 (defun ediff-exec-process (program buffer synch options &rest files)
   (let ((data (match-data))
-	(coding-system-for-read ediff-coding-system-for-read)
+	;; If this is a buffer job, we are diffing temporary files
+	;; produced by Emacs with ediff-coding-system-for-write, so
+	;; use the same encoding to read the results.
+	(coding-system-for-read
+	 (if (string-match "buffer" (symbol-name ediff-job-name))
+	     ediff-coding-system-for-write
+	   ediff-coding-system-for-read))
 	args)
     (setq args (append (split-string options) files))
     (setq args (delete "" (delq nil args))) ; delete nil and "" from arguments

Message #260 received at submit <at> emacsbugs.donarmstrong.com (full text, mbox):

From: "Drew Adams" <drew.adams <at> oracle.com>
To: "'Eli Zaretskii'" <eliz <at> gnu.org>
Cc: <kifer <at> cs.sunysb.edu>, <1183 <at> debbugs.gnu.org>,
        <bug-gnu-emacs <at> gnu.org>, <kifer <at> cs.stonybrook.edu>
Subject: RE: bug#1183: 23.0.60; ediff-buffers is broken
Date: Sun, 19 Oct 2008 08:07:17 -0700

> From: Eli Zaretskii Sent: Sunday, October 19, 2008 1:33 AM
> Drew, please see if the patch below fixes the problem for you.
> 
> If you are not using a very recent CVS code, you will probably need to
> use utf-8-emacs-unix instead of emacs-internal in the ediff-init.el
> change, because emacs-internal was only introduced yesterday.
> 
> Michael, could you please add a warning message in buffer jobs about
> differences in the values of buffer-file-coding-system between the
> buffers being compared?  In particular, if there are no differences in
> a region, but the above values are different, it would be good if
> Ediff would say something like "only character-encoding differences"
> instead of "only white-space differences". Thanks.
> 2008-10-19  Eli Zaretskii  <eliz <at> gnu.org>
>
> 	Fix Bug #1183:
> 	* ediff-diff.el (ediff-exec-process): For buffer jobs, bind
> 	coding-system-for-read to ediff-coding-system-for-write.
> 	* ediff-util.el (ediff-make-temp-file): Unconditionally bind
> 	coding-system-for-write to ediff-coding-system-for-write.
> 	* ediff-init.el (ediff-coding-system-for-read): Doc fix.
> 	(ediff-coding-system-for-write): Set to emacs-internal.


Yes, it appears to be fixed, judging by the test case I sent.

Thank you! 

Message #270 received at submit <at> emacsbugs.donarmstrong.com (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Drew Adams <drew.adams <at> oracle.com>
Cc: kifer <at> cs.sunysb.edu, 1183 <at> debbugs.gnu.org,
        bug-gnu-emacs <at> gnu.org, kifer <at> cs.stonybrook.edu
Subject: Re: bug#1183: 23.0.60; ediff-buffers is broken
Date: Sun, 19 Oct 2008 17:32:36 +0200

> From: "Drew Adams" <drew.adams <at> oracle.com>
> Cc: <kifer <at> cs.sunysb.edu>, <1183 <at> emacsbugs.donarmstrong.com>,
>         <bug-gnu-emacs <at> gnu.org>, <kifer <at> cs.stonybrook.edu>
> Date: Sun, 19 Oct 2008 08:07:17 -0700
> 
> > 	Fix Bug #1183:
> > 	* ediff-diff.el (ediff-exec-process): For buffer jobs, bind
> > 	coding-system-for-read to ediff-coding-system-for-write.
> > 	* ediff-util.el (ediff-make-temp-file): Unconditionally bind
> > 	coding-system-for-write to ediff-coding-system-for-write.
> > 	* ediff-init.el (ediff-coding-system-for-read): Doc fix.
> > 	(ediff-coding-system-for-write): Set to emacs-internal.
> 
> 
> Yes, it appears to be fixed, judging by the test case I sent.

Thanks for testing.  I closed the bug.

Message #280 received at 1183-done <at> emacsbugs.donarmstrong.com (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: 1183-done <at> debbugs.gnu.org
Subject: RE: bug#1183: 23.0.60; ediff-buffers is broken
Date: Sun, 19 Oct 2008 17:32:07 +0200

Fixed by using the emacs-internal encoding when writing buffers to
temporary files.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.