GNU bug report logs - #35350
Some compile output still leaks through with --verbosity=1

Previous Next

Full log

View this message in rfc822 format

From: Ludovic Courtès <ludo <at> gnu.org>
To: Mark H Weaver <mhw <at> netris.org>
Cc: 35350 <at> debbugs.gnu.org
Subject: bug#35350: Some compile output still leaks through with --verbosity=1
Date: Sat, 27 Apr 2019 18:36:34 +0200

Hi Mark,

Mark H Weaver <mhw <at> netris.org> skribis:

> Ludovic Courtès <ludo <at> gnu.org> writes:
>
>> The third read(2) call here ends on a partial UTF-8 sequence for LEFT
>> SINGLE QUOTATION MARK (we get the first two bytes of a three byte
>> sequence.)
>>
>> What happens is that ‘process-stderr’ in (guix store) gets that byte
>> string from the daemon, passes it through ‘read-maybe-utf8-string’,
>> which replaces the last two bytes with REPLACEMENT CHARACTER, which is
>> itself a 3-byte sequence.
>
> It seems to me that what's needed here is to save the UTF-8 decoder
> state between calls to 'process-stderr'.

So there are two things.  To fix the issue you reported (build output
that goes through), I think we must simply turn off UTF-8 decoding from
‘process-stderr’ and leave that entirely to ‘build-event-output-port’.

However, ‘build-event-output-port’ would still fail to properly decode
split UTF-8 sequences, and for that we’d need to preserve decoder state
as you describe.

> Coincidentally, I also needed something like this a week ago, when I
> tried implementing R6RS custom textual input/output ports on top of
> R6RS custom binary input/output ports.
>
> To meet these needs, I've implemented a fairly efficient, purely
> functional UTF-8 decoder in Scheme that accepts a decoder state and an
> arbitrary range from a bytevector, and returns a new decoder state.
> There's a macro that allows arbitrary actions to be performed when a
> code point (or maximal subpart in the case of errors) is found.
>
> This macro is then used to implement a decoder (utf8->string!) that
> writes into an arbitrary range of an existing string.  Of course, it's
> not purely functional, but it avoids heap allocation when compiled with
> Guile.  On my Thinkpad X200, it can process around 10 megabytes per
> second.
>
> The state is represented as an exact integer between 0 and #xF48FBF
> inclusive, which are simply the bytes that have been seen so far in the
> current code sequence, in big-endian order, or 0 for the start state.
> For example, #xF48FBF represents the state where the bytes (F4 8F BF)
> have been read.  The state is always either 0 or a proper prefix of a
> valid UTF-8 byte sequence.

Awesome!  I think that’s something we should definitely add to Guile
proper.  We can use it in Guix before or after it’s included in Guile.

Thank you!

Ludo’.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.