GNU bug report logs - #18520
string ports should not have an encoding

Previous Next

Full log

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: David Kastrup <dak <at> gnu.org>
To: bug-guile <at> gnu.org
Subject: string ports should not have an encoding
Date: Mon, 22 Sep 2014 01:34:39 +0200

In Guile 2.0, at the time a string port is opened, the value of the
fluid %default-port-encoding is used for deciding how to encode the
string into a byte stream, and set-port-encoding! may then be used for
deciding how to decode that byte stream back into characters.

This does not make sense as ports deliver characters, and strings
contain characters.  There is no point in going through bytes.

Guile-2.2 does not consult %default-port-encoding but uses UTF-8
consistently (I guess, overriding set-port-encoding! will again change
that).

That still is not satisfactory.  For example, using ftell on the input
port will not report the string index of the string connected to the
string port but rather a byte index into a UTF-8 encoded version of the
string.  This is a number that has nothing to do with the original
string and cannot be used for correlating string and port.

Ports fundamentally deliver characters, and so reading and writing from
a string source/sink should not involve _any_ coding system.

Files fundamentally deliver bytes, a conversion is required.  The same
would be the case when opening a port on a _bytevector_.  Here an
encoding would make equally make sense, and ftell/fseek offsets would
naturally be in bytes.  But a port on a string delivers and consumes
characters.  Any conversion, even a fixed UTF-8 conversion, will destroy
the predictable nature of with-output-to-string and
with-input-from-string and the respective uses of string ports.

In code like the following, the results should not depend on either the
fluid-set! or the set-port-encoding!, and the ftell should always output
successive integers independent from either fluid-set! or
set-port-encoding!.  set-port-encoding! should probably flag an error,
like an fseek on an unseekable device.

(fluid-set! %default-port-encoding "UTF-8")
(define s (list->string (map integer->char '(20 200 2000 20000))))
(with-input-from-string s
  (lambda ()
    (set-port-encoding! (current-input-port) "ISO-8859-1")
    (let loop ((ch (read-char (current-input-port))))
      (if (not (eof-object? ch))
	  (begin
	    (format #t "~d, pos=~d\n" (char->integer ch) (ftell (current-input-port)))
	    (loop (read-char (current-input-port))))))))



Again, things are quite different from bytevectors which could be
accepted instead of a string for opening ports with the string-port
commands, or could have their own port open/close commands, and the
respective ports then definitely would want to obey set-port-encoding!
(defaulting to %default-port-encoding) for _decoding_ the bytevector.

I don't know what r7rs might think here.  But for me, associating
encodings for connecting strings to ports does not make sense.  The
relation is one of characters to characters.

-- 
David Kastrup

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.