From unknown Sat Jun 14 03:48:44 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#35880] [PATCH 0/7] Lzip support for 'guix publish' and 'guix substitute' Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Fri, 24 May 2019 13:34:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 35880 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 35880@debbugs.gnu.org Cc: Ludovic =?UTF-8?Q?Court=C3=A8s?= , Pierre Neidhardt X-Debbugs-Original-To: guix-patches@gnu.org Received: via spool by submit@debbugs.gnu.org id=B.15587047948036 (code B ref -1); Fri, 24 May 2019 13:34:02 +0000 Received: (at submit) by debbugs.gnu.org; 24 May 2019 13:33:14 +0000 Received: from localhost ([127.0.0.1]:46267 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hUAJu-00025X-8w for submit@debbugs.gnu.org; Fri, 24 May 2019 09:33:14 -0400 Received: from eggs.gnu.org ([209.51.188.92]:47788) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hUAJt-00025E-0T for submit@debbugs.gnu.org; Fri, 24 May 2019 09:33:13 -0400 Received: from lists.gnu.org ([209.51.188.17]:49366) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1hUAJn-00087p-SW for submit@debbugs.gnu.org; Fri, 24 May 2019 09:33:07 -0400 Received: from eggs.gnu.org ([209.51.188.92]:33887) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hUAJm-0001gR-S9 for guix-patches@gnu.org; Fri, 24 May 2019 09:33:07 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_40 autolearn=disabled version=3.3.2 Received: from fencepost.gnu.org ([2001:470:142:3::e]:38653) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hUAIn-0007aC-3W; Fri, 24 May 2019 09:32:05 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=36416 helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:DHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1hUAIm-0005a1-At; Fri, 24 May 2019 09:32:04 -0400 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Date: Fri, 24 May 2019 15:31:59 +0200 Message-Id: <20190524133159.22568-1-ludo@gnu.org> X-Mailer: git-send-email 2.21.0 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6.x X-Spam-Score: -2.3 (--) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hi! As a followup to Pierre’s work on (guix lzlib), these patches implement ‘lzip’ support for ‘guix substitute’ and ‘guix publish’. With these, you can now run: ./pre-inst-env guix publish -Clzip … on one side, and on another machine: ./pre-inst-env guix-daemon --build-users-group=guixbuild and from there the client machine should be able to fetch lzip-compressed substitutes. These patches do not address the transitioning issue that we discussed earlier, where we have clients lacking lzip support talking to an lzip-capable server. As discussed earlier, clients will have to send a special HTTP header, ‘X-Guix-Accept-Encoding’. For the server-side, I’m still hesitating between implementing it in ‘guix publish’ or simply running two instances of ‘guix publish’ side-by-side (one gzip and one lzip) and letting nginx dispatch between the two. :-) Note that we’ll have to adjust our nginx mirror configs to take that header into account! Comments? Ludo’. Ludovic Courtès (7): lzlib: Add 'make-lzip-input-port/compressed'. utils: Test 'compressed-port' and 'decompressed-port' for both gzip and xz. utils: Support compression and decompression with lzip. publish: Add support for lzip. self: Add dependency on lzlib. gnu: guix: Add dependency on lzlib. lzlib: 'lzread!' never returns more than it was asked for. .dir-locals.el | 2 + doc/guix.texi | 25 +++++-- gnu/packages/package-management.scm | 1 + guix/lzlib.scm | 101 ++++++++++++++++++++++------ guix/scripts/publish.scm | 84 +++++++++++++++++------ guix/self.scm | 13 +++- guix/tests.scm | 1 + guix/utils.scm | 27 ++++++-- tests/lzlib.scm | 10 +++ tests/publish.scm | 36 ++++++++++ tests/utils.scm | 62 +++++++++++------ 11 files changed, 284 insertions(+), 78 deletions(-) -- 2.21.0 From unknown Sat Jun 14 03:48:44 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#35880] [PATCH 2/7] utils: Test 'compressed-port' and 'decompressed-port' for both gzip and xz. Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Fri, 24 May 2019 13:43:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 35880 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 35880@debbugs.gnu.org Cc: Ludovic =?UTF-8?Q?Court=C3=A8s?= , Pierre Neidhardt Received: via spool by 35880-submit@debbugs.gnu.org id=B35880.15587053759208 (code B ref 35880); Fri, 24 May 2019 13:43:01 +0000 Received: (at 35880) by debbugs.gnu.org; 24 May 2019 13:42:55 +0000 Received: from localhost ([127.0.0.1]:46293 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hUATH-0002OQ-EB for submit@debbugs.gnu.org; Fri, 24 May 2019 09:42:55 -0400 Received: from eggs.gnu.org ([209.51.188.92]:50660) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hUATE-0002Nr-Sz for 35880@debbugs.gnu.org; Fri, 24 May 2019 09:42:54 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:38747) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hUAT9-0008Cl-Nn; Fri, 24 May 2019 09:42:47 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=36484 helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:DHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1hUAT9-0007iN-8Z; Fri, 24 May 2019 09:42:47 -0400 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Date: Fri, 24 May 2019 15:42:33 +0200 Message-Id: <20190524134238.22802-2-ludo@gnu.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190524134238.22802-1-ludo@gnu.org> References: <20190524134238.22802-1-ludo@gnu.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Spam-Score: -2.3 (--) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) * tests/utils.scm (test-compression/decompression): New procedure. : Call it for both 'xz and 'gzip. --- tests/utils.scm | 61 +++++++++++++++++++++++++++++++------------------ 1 file changed, 39 insertions(+), 22 deletions(-) diff --git a/tests/utils.scm b/tests/utils.scm index 3015b21b23..7d55107fda 100644 --- a/tests/utils.scm +++ b/tests/utils.scm @@ -174,30 +174,47 @@ (any (compose (negate zero?) cdr waitpid) pids)))) -(test-assert "compressed-port, decompressed-port, non-file" - (let ((data (call-with-input-file (search-path %load-path "guix.scm") - get-bytevector-all))) - (let*-values (((compressed pids1) - (compressed-port 'xz (open-bytevector-input-port data))) - ((decompressed pids2) - (decompressed-port 'xz compressed))) - (and (every (compose zero? cdr waitpid) - (append pids1 pids2)) - (equal? (get-bytevector-all decompressed) data))))) +(define (test-compression/decompression method run?) + "Test METHOD, a symbol such as 'gzip. Call RUN? to determine whether to +skip these tests." + (unless (run?) (test-skip 1)) + (test-assert (format #f "compressed-port, decompressed-port, non-file [~a]" + method) + (let ((data (call-with-input-file (search-path %load-path "guix.scm") + get-bytevector-all))) + (let*-values (((compressed pids1) + (compressed-port method (open-bytevector-input-port data))) + ((decompressed pids2) + (decompressed-port method compressed))) + (and (every (compose zero? cdr waitpid) + (pk 'pids method (append pids1 pids2))) + (let ((result (get-bytevector-all decompressed))) + (pk 'len method + (if (bytevector? result) + (bytevector-length result) + result) + (bytevector-length data)) + (equal? result data)))))) -(false-if-exception (delete-file temp-file)) -(test-assert "compressed-output-port + decompressed-port" - (let* ((file (search-path %load-path "guix/derivations.scm")) - (data (call-with-input-file file get-bytevector-all)) - (port (open-file temp-file "w0b"))) - (call-with-compressed-output-port 'xz port - (lambda (compressed) - (put-bytevector compressed data))) - (close-port port) + (false-if-exception (delete-file temp-file)) + (unless (run?) (test-skip 1)) + (test-assert (format #f "compressed-output-port + decompressed-port [~a]" + method) + (let* ((file (search-path %load-path "guix/derivations.scm")) + (data (call-with-input-file file get-bytevector-all)) + (port (open-file temp-file "w0b"))) + (call-with-compressed-output-port method port + (lambda (compressed) + (put-bytevector compressed data))) + (close-port port) - (bytevector=? data - (call-with-decompressed-port 'xz (open-file temp-file "r0b") - get-bytevector-all)))) + (bytevector=? data + (call-with-decompressed-port method (open-file temp-file "r0b") + get-bytevector-all))))) + +(for-each test-compression/decompression + '(gzip xz) + (list (const #t) (const #f))) ;; This is actually in (guix store). (test-equal "store-path-package-name" -- 2.21.0 From unknown Sat Jun 14 03:48:44 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#35880] [PATCH 1/7] lzlib: Add 'make-lzip-input-port/compressed'. References: <20190524133159.22568-1-ludo@gnu.org> In-Reply-To: <20190524133159.22568-1-ludo@gnu.org> Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Fri, 24 May 2019 13:43:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 35880 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 35880@debbugs.gnu.org Cc: Ludovic =?UTF-8?Q?Court=C3=A8s?= , Pierre Neidhardt Received: via spool by 35880-submit@debbugs.gnu.org id=B35880.15587053769220 (code B ref 35880); Fri, 24 May 2019 13:43:02 +0000 Received: (at 35880) by debbugs.gnu.org; 24 May 2019 13:42:56 +0000 Received: from localhost ([127.0.0.1]:46295 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hUATH-0002OT-Ou for submit@debbugs.gnu.org; Fri, 24 May 2019 09:42:56 -0400 Received: from eggs.gnu.org ([209.51.188.92]:50652) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hUATE-0002Nn-JF for 35880@debbugs.gnu.org; Fri, 24 May 2019 09:42:54 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:38746) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hUAT8-0008C6-Oc; Fri, 24 May 2019 09:42:46 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=36484 helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:DHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1hUAT8-0007iN-AZ; Fri, 24 May 2019 09:42:46 -0400 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Date: Fri, 24 May 2019 15:42:32 +0200 Message-Id: <20190524134238.22802-1-ludo@gnu.org> X-Mailer: git-send-email 2.21.0 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Spam-Score: -2.3 (--) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) * guix/lzlib.scm (lzwrite!, make-lzip-input-port/compressed): New procedures. * tests/lzlib.scm ("make-lzip-input-port/compressed"): New test. * guix/tests.scm (%seed): Export. --- guix/lzlib.scm | 62 +++++++++++++++++++++++++++++++++++++++++++++++++ guix/tests.scm | 1 + tests/lzlib.scm | 10 ++++++++ 3 files changed, 73 insertions(+) diff --git a/guix/lzlib.scm b/guix/lzlib.scm index a6dac46049..48927c6262 100644 --- a/guix/lzlib.scm +++ b/guix/lzlib.scm @@ -1,5 +1,6 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2019 Pierre Neidhardt +;;; Copyright © 2019 Ludovic Courtès ;;; ;;; This file is part of GNU Guix. ;;; @@ -23,9 +24,11 @@ #:use-module (ice-9 match) #:use-module (system foreign) #:use-module (guix config) + #:use-module (srfi srfi-11) #:export (lzlib-available? make-lzip-input-port make-lzip-output-port + make-lzip-input-port/compressed call-with-lzip-input-port call-with-lzip-output-port %default-member-length-limit @@ -515,6 +518,23 @@ the end-of-stream has been reached." (loop rd))) read)) +(define (lzwrite! encoder source source-offset source-count + target target-offset target-count) + "Write up to SOURCE-COUNT bytes from SOURCE to ENCODER, and read up to +TARGET-COUNT bytes into TARGET at TARGET-OFFSET. Return two values: the +number of bytes read from SOURCE, and the number of bytes written to TARGET." + (define read + (if (< 0 (lz-compress-write-size encoder)) + (match (lz-compress-write encoder source source-offset source-count) + (0 (lz-compress-finish encoder) 0) + (n n)) + 0)) + + (let loop () + (match (lz-compress-read encoder target target-offset target-count) + (0 (loop)) + (written (values read written))))) + (define* (lzwrite encoder bv lz-port #:optional (start 0) (count (bytevector-length bv))) "Write up to COUNT bytes from BV at offset START into LZ-PORT. Return @@ -597,6 +617,48 @@ port is closed." (lz-compress-close encoder) (close-port port)))) +(define* (make-lzip-input-port/compressed port + #:key + (level %default-compression-level)) + "Return an input port that compresses data read from PORT, with the given LEVEL. +PORT is automatically closed when the resulting port is closed." + (define encoder (apply lz-compress-open + (car (assoc-ref %compression-levels level)))) + + (define input-buffer (make-bytevector 8192)) + (define input-len 0) + (define input-offset 0) + + (define input-eof? #f) + + (define (read! bv start count) + (cond + (input-eof? + (lz-compress-read encoder bv start count)) + ((= input-offset input-len) + (match (get-bytevector-n! port input-buffer 0 + (bytevector-length input-buffer)) + ((? eof-object?) + (set! input-eof? #t) + (lz-compress-finish encoder)) + (count + (set! input-offset 0) + (set! input-len count))) + (read! bv start count)) + (else + (let-values (((read written) + (lzwrite! encoder + input-buffer input-offset + (- input-len input-offset) + bv start count))) + (set! input-offset (+ input-offset read)) + written)))) + + (make-custom-binary-input-port "lzip-input/compressed" + read! #f #f + (lambda () + (close-port port)))) + (define* (call-with-lzip-input-port port proc) "Call PROC with a port that wraps PORT and decompresses data read from it. PORT is closed upon completion." diff --git a/guix/tests.scm b/guix/tests.scm index 35ebf8464d..66d60e964e 100644 --- a/guix/tests.scm +++ b/guix/tests.scm @@ -33,6 +33,7 @@ #:use-module (web uri) #:export (open-connection-for-tests with-external-store + %seed random-text random-bytevector file=? diff --git a/tests/lzlib.scm b/tests/lzlib.scm index cf53a9417d..543622bb45 100644 --- a/tests/lzlib.scm +++ b/tests/lzlib.scm @@ -108,4 +108,14 @@ (test-assert* "Bytevector of size relative to Lzip internal buffers (1MiB+1)" (compress-and-decompress (random-bytevector (1+ (* 1024 1024))))) +(test-assert "make-lzip-input-port/compressed" + (let* ((len (pk 'len (+ 10 (random 4000 %seed)))) + (data (random-bytevector len)) + (compressed (make-lzip-input-port/compressed + (open-bytevector-input-port data))) + (result (call-with-lzip-input-port compressed + get-bytevector-all))) + (pk (bytevector-length result) (bytevector-length data)) + (bytevector=? result data))) + (test-end) -- 2.21.0 From unknown Sat Jun 14 03:48:44 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#35880] [PATCH 3/7] utils: Support compression and decompression with lzip. Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Fri, 24 May 2019 13:43:03 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 35880 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 35880@debbugs.gnu.org Cc: Ludovic =?UTF-8?Q?Court=C3=A8s?= , Pierre Neidhardt Received: via spool by 35880-submit@debbugs.gnu.org id=B35880.15587053769233 (code B ref 35880); Fri, 24 May 2019 13:43:03 +0000 Received: (at 35880) by debbugs.gnu.org; 24 May 2019 13:42:56 +0000 Received: from localhost ([127.0.0.1]:46298 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hUATI-0002Of-89 for submit@debbugs.gnu.org; Fri, 24 May 2019 09:42:56 -0400 Received: from eggs.gnu.org ([209.51.188.92]:50666) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hUATG-0002Nv-4F for 35880@debbugs.gnu.org; Fri, 24 May 2019 09:42:55 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:38748) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hUATA-0008Db-V2; Fri, 24 May 2019 09:42:49 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=36484 helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:DHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1hUATA-0007iN-7s; Fri, 24 May 2019 09:42:48 -0400 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Date: Fri, 24 May 2019 15:42:34 +0200 Message-Id: <20190524134238.22802-3-ludo@gnu.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190524134238.22802-1-ludo@gnu.org> References: <20190524134238.22802-1-ludo@gnu.org> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Spam-Score: -2.3 (--) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) * guix/utils.scm (lzip-port): New procedure. (decompressed-port, compressed-port, compressed-output-port): Add 'lzip case. * tests/utils.scm : Call 'test-compression/decompression' for 'lzip as well. --- guix/utils.scm | 27 ++++++++++++++++++++++----- tests/utils.scm | 5 +++-- 2 files changed, 25 insertions(+), 7 deletions(-) diff --git a/guix/utils.scm b/guix/utils.scm index ed1a418cca..709cdf9353 100644 --- a/guix/utils.scm +++ b/guix/utils.scm @@ -1,5 +1,5 @@ ;;; GNU Guix --- Functional package management for GNU -;;; Copyright © 2012, 2013, 2014, 2015, 2016, 2017, 2018 Ludovic Courtès +;;; Copyright © 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019 Ludovic Courtès ;;; Copyright © 2013, 2014, 2015 Mark H Weaver ;;; Copyright © 2014 Eric Bavier ;;; Copyright © 2014 Ian Denhardt @@ -169,6 +169,17 @@ buffered data is lost." (close-port out) (loop in (cons child pids))))))))) +(define (lzip-port proc port . args) + "Return the lzip port produced by calling PROC (a symbol) on PORT and ARGS. +Raise an error if lzlib support is missing." + (let* ((lzlib (false-if-exception (resolve-interface '(guix lzlib)))) + (supported? (and lzlib + ((module-ref lzlib 'lzlib-available?))))) + (if supported? + (let ((make-port (module-ref lzlib proc))) + (values (make-port port) '())) + (error "lzip compression not supported" lzlib)))) + (define (decompressed-port compression input) "Return an input port where INPUT is decompressed according to COMPRESSION, a symbol such as 'xz." @@ -177,17 +188,21 @@ a symbol such as 'xz." ('bzip2 (filtered-port `(,%bzip2 "-dc") input)) ('xz (filtered-port `(,%xz "-dc") input)) ('gzip (filtered-port `(,%gzip "-dc") input)) - (else (error "unsupported compression scheme" compression)))) + ('lzip (values (lzip-port 'make-lzip-input-port input) + '())) + (_ (error "unsupported compression scheme" compression)))) (define (compressed-port compression input) - "Return an input port where INPUT is decompressed according to COMPRESSION, + "Return an input port where INPUT is compressed according to COMPRESSION, a symbol such as 'xz." (match compression ((or #f 'none) (values input '())) ('bzip2 (filtered-port `(,%bzip2 "-c") input)) ('xz (filtered-port `(,%xz "-c") input)) ('gzip (filtered-port `(,%gzip "-c") input)) - (else (error "unsupported compression scheme" compression)))) + ('lzip (values (lzip-port 'make-lzip-input-port/compressed input) + '())) + (_ (error "unsupported compression scheme" compression)))) (define (call-with-decompressed-port compression port proc) "Call PROC with a wrapper around PORT, a file port, that decompresses data @@ -244,7 +259,9 @@ program--e.g., '(\"--fast\")." ('bzip2 (filtered-output-port `(,%bzip2 "-c" ,@options) output)) ('xz (filtered-output-port `(,%xz "-c" ,@options) output)) ('gzip (filtered-output-port `(,%gzip "-c" ,@options) output)) - (else (error "unsupported compression scheme" compression)))) + ('lzip (values (lzip-port 'make-lzip-output-port output) + '())) + (_ (error "unsupported compression scheme" compression)))) (define* (call-with-compressed-output-port compression port proc #:key (options '())) diff --git a/tests/utils.scm b/tests/utils.scm index 7d55107fda..7c8f7c09d0 100644 --- a/tests/utils.scm +++ b/tests/utils.scm @@ -23,6 +23,7 @@ #:use-module (guix utils) #:use-module ((guix store) #:select (%store-prefix store-path-package-name)) #:use-module ((guix search-paths) #:select (string-tokenize*)) + #:use-module ((guix lzlib) #:select (lzlib-available?)) #:use-module (srfi srfi-1) #:use-module (srfi srfi-11) #:use-module (srfi srfi-64) @@ -213,8 +214,8 @@ skip these tests." get-bytevector-all))))) (for-each test-compression/decompression - '(gzip xz) - (list (const #t) (const #f))) + '(gzip xz lzip) + (list (const #t) (const #f) lzlib-available?)) ;; This is actually in (guix store). (test-equal "store-path-package-name" -- 2.21.0 From unknown Sat Jun 14 03:48:44 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#35880] [PATCH 4/7] publish: Add support for lzip. Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Fri, 24 May 2019 13:44:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 35880 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 35880@debbugs.gnu.org Cc: Ludovic =?UTF-8?Q?Court=C3=A8s?= , Pierre Neidhardt Received: via spool by 35880-submit@debbugs.gnu.org id=B35880.15587053849297 (code B ref 35880); Fri, 24 May 2019 13:44:02 +0000 Received: (at 35880) by debbugs.gnu.org; 24 May 2019 13:43:04 +0000 Received: from localhost ([127.0.0.1]:46304 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hUATO-0002PZ-U8 for submit@debbugs.gnu.org; Fri, 24 May 2019 09:43:03 -0400 Received: from eggs.gnu.org ([209.51.188.92]:50670) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hUATH-0002Nz-64 for 35880@debbugs.gnu.org; Fri, 24 May 2019 09:42:55 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:38750) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hUATC-0008EC-1C; Fri, 24 May 2019 09:42:50 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=36484 helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:DHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1hUATB-0007iN-Ea; Fri, 24 May 2019 09:42:49 -0400 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Date: Fri, 24 May 2019 15:42:35 +0200 Message-Id: <20190524134238.22802-4-ludo@gnu.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190524134238.22802-1-ludo@gnu.org> References: <20190524134238.22802-1-ludo@gnu.org> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Spam-Score: -1.6 (-) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.6 (--) * guix/scripts/publish.scm (show-help, %options): Support '-C METHOD' and '-C METHOD:LEVEL'. (default-compression): New procedure. (bake-narinfo+nar): Add lzip. (nar-response-port): Likewise. (string->compression-type): New procedure. (make-request-handler): Generalize /nar/gzip handler to handle /nar/lzip as well. * tests/publish.scm ("/nar/lzip/*"): New test. ("/*.narinfo with lzip compression"): New test. * doc/guix.texi (Invoking guix publish): Document it. (Requirements): Mention lzlib. --- .dir-locals.el | 2 + doc/guix.texi | 25 +++++++++--- guix/scripts/publish.scm | 84 +++++++++++++++++++++++++++++----------- tests/publish.scm | 36 +++++++++++++++++ 4 files changed, 119 insertions(+), 28 deletions(-) diff --git a/.dir-locals.el b/.dir-locals.el index 550e06ef09..f1196fd781 100644 --- a/.dir-locals.el +++ b/.dir-locals.el @@ -53,6 +53,8 @@ (eval . (put 'call-with-decompressed-port 'scheme-indent-function 2)) (eval . (put 'call-with-gzip-input-port 'scheme-indent-function 1)) (eval . (put 'call-with-gzip-output-port 'scheme-indent-function 1)) + (eval . (put 'call-with-lzip-input-port 'scheme-indent-function 1)) + (eval . (put 'call-with-lzip-output-port 'scheme-indent-function 1)) (eval . (put 'signature-case 'scheme-indent-function 1)) (eval . (put 'emacs-batch-eval 'scheme-indent-function 0)) (eval . (put 'emacs-batch-edit-file 'scheme-indent-function 1)) diff --git a/doc/guix.texi b/doc/guix.texi index f176bb0163..b0de5632e7 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -757,6 +757,11 @@ Support for build offloading (@pxref{Daemon Offload Setup}) and @uref{https://github.com/artyom-poptsov/guile-ssh, Guile-SSH}, version 0.10.2 or later. +@item +When @url{https://www.nongnu.org/lzip/lzlib.html, lzlib} is available, lzlib +substitutes can be used and @command{guix publish} can compress substitutes +with lzlib. + @item When @url{http://www.bzip.org, libbz2} is available, @command{guix-daemon} can use it to compress build logs. @@ -9656,12 +9661,20 @@ accept connections from any interface. Change privileges to @var{user} as soon as possible---i.e., once the server socket is open and the signing key has been read. -@item --compression[=@var{level}] -@itemx -C [@var{level}] -Compress data using the given @var{level}. When @var{level} is zero, -disable compression. The range 1 to 9 corresponds to different gzip -compression levels: 1 is the fastest, and 9 is the best (CPU-intensive). -The default is 3. +@item --compression[=@var{method}[:@var{level}]] +@itemx -C [@var{method}[:@var{level}]] +Compress data using the given @var{method} and @var{level}. @var{method} is +one of @code{lzip} and @code{gzip}; when @var{method} is omitted, @code{gzip} +is used. + +When @var{level} is zero, disable compression. The range 1 to 9 corresponds +to different compression levels: 1 is the fastest, and 9 is the best +(CPU-intensive). The default is 3. + +Usually, @code{lzip} compresses noticeably better than @code{gzip} for a small +increase in CPU usage; see +@uref{https://nongnu.org/lzip/lzip_benchmark.html,benchmarks on the lzip Web +page}. Unless @option{--cache} is used, compression occurs on the fly and the compressed streams are not diff --git a/guix/scripts/publish.scm b/guix/scripts/publish.scm index a236f3e45c..9e74d789ce 100644 --- a/guix/scripts/publish.scm +++ b/guix/scripts/publish.scm @@ -1,6 +1,6 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2015 David Thompson -;;; Copyright © 2015, 2016, 2017, 2018 Ludovic Courtès +;;; Copyright © 2015, 2016, 2017, 2018, 2019 Ludovic Courtès ;;; ;;; This file is part of GNU Guix. ;;; @@ -51,6 +51,7 @@ #:use-module (guix store) #:use-module ((guix serialization) #:select (write-file)) #:use-module (guix zlib) + #:autoload (guix lzlib) (lzlib-available?) #:use-module (guix cache) #:use-module (guix ui) #:use-module (guix scripts) @@ -74,8 +75,8 @@ Publish ~a over HTTP.\n") %store-directory) (display (G_ " -u, --user=USER change privileges to USER as soon as possible")) (display (G_ " - -C, --compression[=LEVEL] - compress archives at LEVEL")) + -C, --compression[=METHOD:LEVEL] + compress archives with METHOD at LEVEL")) (display (G_ " -c, --cache=DIRECTORY cache published items to DIRECTORY")) (display (G_ " @@ -121,6 +122,9 @@ Publish ~a over HTTP.\n") %store-directory) ;; Since we compress on the fly, default to fast compression. (compression 'gzip 3)) +(define (default-compression type) + (compression type 3)) + (define (actual-compression item requested) "Return the actual compression used for ITEM, which may be %NO-COMPRESSION if ITEM is already compressed." @@ -153,18 +157,28 @@ if ITEM is already compressed." name))))) (option '(#\C "compression") #f #t (lambda (opt name arg result) - (match (if arg (string->number* arg) 3) - (0 - (alist-cons 'compression %no-compression result)) - (level - (if (zlib-available?) - (alist-cons 'compression - (compression 'gzip level) - result) - (begin - (warning (G_ "zlib support is missing; \ -compression disabled~%")) - result)))))) + (let* ((colon (string-index arg #\:)) + (type (cond + (colon (string-take arg colon)) + ((string->number arg) "gzip") + (else arg))) + (level (if colon + (string->number* + (string-drop arg (+ 1 colon))) + (or (string->number arg) 3)))) + (match level + (0 + (alist-cons 'compression %no-compression result)) + (level + (match (string->compression-type type) + ((? symbol? type) + (alist-cons 'compression + (compression type level) + result)) + (_ + (warning (G_ "~a: unsupported compression type~%") + type) + result))))))) (option '(#\c "cache") #t #f (lambda (opt name arg result) (alist-cons 'cache arg result))) @@ -483,6 +497,13 @@ requested using POOL." #:level (compression-level compression) #:buffer-size (* 128 1024)) (rename-file (string-append nar ".tmp") nar)) + ('lzip + ;; Note: the file port gets closed along with the lzip port. + (call-with-lzip-output-port (open-output-file (string-append nar ".tmp")) + (lambda (port) + (write-file item port)) + #:level (compression-level compression)) + (rename-file (string-append nar ".tmp") nar)) ('none ;; Cache nars even when compression is disabled so that we can ;; guarantee the TTL (see .) @@ -687,6 +708,9 @@ example: \"/foo/bar\" yields '(\"foo\" \"bar\")." (make-gzip-output-port (response-port response) #:level level #:buffer-size (* 64 1024))) + (($ 'lzip level) + (make-lzip-output-port (response-port response) + #:level level)) (($ 'none) (response-port response)) (#f @@ -761,12 +785,23 @@ blocking." http-write (@@ (web server http) http-close)) +(define (string->compression-type string) + "Return a symbol denoting the compression method expressed by STRING; return +#f if STRING doesn't match any supported method." + (match string + ("gzip" (and (zlib-available?) 'gzip)) + ("lzip" (and (lzlib-available?) 'lzip)) + (_ #f))) + (define* (make-request-handler store #:key cache pool narinfo-ttl (nar-path "nar") (compression %no-compression)) + (define compression-type? + string->compression-type) + (define nar-path? (let ((expected (split-and-decode-uri-path nar-path))) (cut equal? expected <>))) @@ -815,13 +850,18 @@ blocking." ;; is restarted with different compression parameters. ;; /nar/gzip/ - ((components ... "gzip" store-item) - (if (and (nar-path? components) (zlib-available?)) - (let ((compression (match compression - (($ 'gzip) - compression) - (_ - %default-gzip-compression)))) + ((components ... (? compression-type? type) store-item) + (if (nar-path? components) + (let* ((compression-type (string->compression-type type)) + (compression (match compression + (($ type) + (if (eq? type compression-type) + compression + (default-compression + compression-type))) + (_ + (default-compression + compression-type))))) (if cache (render-nar/cached store cache request store-item #:ttl narinfo-ttl diff --git a/tests/publish.scm b/tests/publish.scm index 097ac036e0..10bc859695 100644 --- a/tests/publish.scm +++ b/tests/publish.scm @@ -36,6 +36,7 @@ #:use-module (gcrypt pk-crypto) #:use-module ((guix pki) #:select (%public-key-file %private-key-file)) #:use-module (guix zlib) + #:use-module (guix lzlib) #:use-module (web uri) #:use-module (web client) #:use-module (web response) @@ -229,6 +230,19 @@ FileSize: ~a~%" (string-append "/nar/gzip/" (basename %item)))))) (get-bytevector-n nar (bytevector-length %gzip-magic-bytes)))) +(unless (lzlib-available?) + (test-skip 1)) +(test-equal "/nar/lzip/*" + "bar" + (call-with-temporary-output-file + (lambda (temp port) + (let ((nar (http-get-port + (publish-uri + (string-append "/nar/lzip/" (basename %item)))))) + (call-with-lzip-input-port nar + (cut restore-file <> temp))) + (call-with-input-file temp read-string)))) + (unless (zlib-available?) (test-skip 1)) (test-equal "/*.narinfo with compression" @@ -251,6 +265,28 @@ FileSize: ~a~%" (_ #f))) (recutils->alist body))))) +(unless (lzlib-available?) + (test-skip 1)) +(test-equal "/*.narinfo with lzip compression" + `(("StorePath" . ,%item) + ("URL" . ,(string-append "nar/lzip/" (basename %item))) + ("Compression" . "lzip")) + (let ((thread (with-separate-output-ports + (call-with-new-thread + (lambda () + (guix-publish "--port=6790" "-Clzip")))))) + (wait-until-ready 6790) + (let* ((url (string-append "http://localhost:6790/" + (store-path-hash-part %item) ".narinfo")) + (body (http-get-port url))) + (filter (lambda (item) + (match item + (("Compression" . _) #t) + (("StorePath" . _) #t) + (("URL" . _) #t) + (_ #f))) + (recutils->alist body))))) + (unless (zlib-available?) (test-skip 1)) (test-equal "/*.narinfo for a compressed file" -- 2.21.0 From unknown Sat Jun 14 03:48:44 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#35880] [PATCH 5/7] self: Add dependency on lzlib. Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Fri, 24 May 2019 13:44:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 35880 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 35880@debbugs.gnu.org Cc: Ludovic =?UTF-8?Q?Court=C3=A8s?= , Pierre Neidhardt Received: via spool by 35880-submit@debbugs.gnu.org id=B35880.15587053849305 (code B ref 35880); Fri, 24 May 2019 13:44:02 +0000 Received: (at 35880) by debbugs.gnu.org; 24 May 2019 13:43:04 +0000 Received: from localhost ([127.0.0.1]:46307 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hUATP-0002Pq-SV for submit@debbugs.gnu.org; Fri, 24 May 2019 09:43:04 -0400 Received: from eggs.gnu.org ([209.51.188.92]:50678) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hUATI-0002O1-5B for 35880@debbugs.gnu.org; Fri, 24 May 2019 09:42:56 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:38751) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hUATD-0008Ep-0C; Fri, 24 May 2019 09:42:51 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=36484 helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:DHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1hUATC-0007iN-Hd; Fri, 24 May 2019 09:42:50 -0400 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Date: Fri, 24 May 2019 15:42:36 +0200 Message-Id: <20190524134238.22802-5-ludo@gnu.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190524134238.22802-1-ludo@gnu.org> References: <20190524134238.22802-1-ludo@gnu.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Spam-Score: -2.3 (--) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) * guix/self.scm (compiled-guix): Pass #:lzlib to 'make-config.scm'. (make-config.scm): Add #:lzlib and honor it. (specification->package): Add "lzlib". --- guix/self.scm | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/guix/self.scm b/guix/self.scm index 6d7569ec19..17d289e486 100644 --- a/guix/self.scm +++ b/guix/self.scm @@ -57,6 +57,7 @@ ("guile-gcrypt" (ref '(gnu packages gnupg) 'guile-gcrypt)) ("gnutls" (ref '(gnu packages tls) 'gnutls)) ("zlib" (ref '(gnu packages compression) 'zlib)) + ("lzlib" (ref '(gnu packages compression) 'lzlib)) ("gzip" (ref '(gnu packages compression) 'gzip)) ("bzip2" (ref '(gnu packages compression) 'bzip2)) ("xz" (ref '(gnu packages compression) 'xz)) @@ -646,6 +647,7 @@ Info manual." (guile-version (effective-version)) (guile-for-build (default-guile)) (zlib (specification->package "zlib")) + (lzlib (specification->package "lzlib")) (gzip (specification->package "gzip")) (bzip2 (specification->package "bzip2")) (xz (specification->package "xz")) @@ -800,6 +802,7 @@ Info manual." #:extra-modules `(((guix config) => ,(make-config.scm #:zlib zlib + #:lzlib lzlib #:gzip gzip #:bzip2 bzip2 #:xz xz @@ -897,7 +900,7 @@ Info manual." (variables rest ...)))))) (variables %localstatedir %storedir %sysconfdir))) -(define* (make-config.scm #:key zlib gzip xz bzip2 +(define* (make-config.scm #:key zlib lzlib gzip xz bzip2 (package-name "GNU Guix") (package-version "0") (bug-report-address "bug-guix@gnu.org") @@ -919,7 +922,7 @@ Info manual." %store-database-directory %config-directory %libz - ;; TODO: %liblz + %liblz %gzip %bzip2 %xz)) @@ -966,7 +969,11 @@ Info manual." (define %libz #+(and zlib - (file-append zlib "/lib/libz")))) + (file-append zlib "/lib/libz"))) + + (define %liblz + #+(and lzlib + (file-append lzlib "/lib/liblz")))) ;; Guile 2.0 *requires* the 'define-module' to be at the ;; top-level or the 'toplevel-ref' in the resulting .go file are -- 2.21.0 From unknown Sat Jun 14 03:48:44 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#35880] [PATCH 6/7] gnu: guix: Add dependency on lzlib. Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Fri, 24 May 2019 13:44:03 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 35880 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 35880@debbugs.gnu.org Cc: Ludovic =?UTF-8?Q?Court=C3=A8s?= , Pierre Neidhardt Received: via spool by 35880-submit@debbugs.gnu.org id=B35880.15587053849312 (code B ref 35880); Fri, 24 May 2019 13:44:03 +0000 Received: (at 35880) by debbugs.gnu.org; 24 May 2019 13:43:04 +0000 Received: from localhost ([127.0.0.1]:46309 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hUATQ-0002Q2-Fq for submit@debbugs.gnu.org; Fri, 24 May 2019 09:43:04 -0400 Received: from eggs.gnu.org ([209.51.188.92]:50681) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hUATJ-0002O2-4F for 35880@debbugs.gnu.org; Fri, 24 May 2019 09:42:57 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:38752) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hUATD-0008FM-Vd; Fri, 24 May 2019 09:42:52 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=36484 helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:DHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1hUATD-0007iN-Gc; Fri, 24 May 2019 09:42:51 -0400 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Date: Fri, 24 May 2019 15:42:37 +0200 Message-Id: <20190524134238.22802-6-ludo@gnu.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190524134238.22802-1-ludo@gnu.org> References: <20190524134238.22802-1-ludo@gnu.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Spam-Score: -2.3 (--) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) * gnu/packages/package-management.scm (guix)[inputs]: Add LZLIB. --- gnu/packages/package-management.scm | 1 + 1 file changed, 1 insertion(+) diff --git a/gnu/packages/package-management.scm b/gnu/packages/package-management.scm index a356a6dab7..985fa801b1 100644 --- a/gnu/packages/package-management.scm +++ b/gnu/packages/package-management.scm @@ -272,6 +272,7 @@ `(("bzip2" ,bzip2) ("gzip" ,gzip) ("zlib" ,zlib) ;for 'guix publish' + ("lzlib" ,lzlib) ;for 'guix publish' and 'guix substitute' ("sqlite" ,sqlite) ("libgcrypt" ,libgcrypt) -- 2.21.0 From unknown Sat Jun 14 03:48:44 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#35880] [PATCH 7/7] lzlib: 'lzread!' never returns more than it was asked for. Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Fri, 24 May 2019 13:44:03 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 35880 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 35880@debbugs.gnu.org Cc: Ludovic =?UTF-8?Q?Court=C3=A8s?= , Pierre Neidhardt Received: via spool by 35880-submit@debbugs.gnu.org id=B35880.15587053859318 (code B ref 35880); Fri, 24 May 2019 13:44:03 +0000 Received: (at 35880) by debbugs.gnu.org; 24 May 2019 13:43:05 +0000 Received: from localhost ([127.0.0.1]:46311 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hUATQ-0002Q9-PA for submit@debbugs.gnu.org; Fri, 24 May 2019 09:43:05 -0400 Received: from eggs.gnu.org ([209.51.188.92]:50684) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hUATK-0002O4-3Y for 35880@debbugs.gnu.org; Fri, 24 May 2019 09:42:58 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:38753) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hUATE-0008Fs-UO; Fri, 24 May 2019 09:42:52 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=36484 helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:DHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1hUATE-0007iN-Fs; Fri, 24 May 2019 09:42:52 -0400 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Date: Fri, 24 May 2019 15:42:38 +0200 Message-Id: <20190524134238.22802-7-ludo@gnu.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190524134238.22802-1-ludo@gnu.org> References: <20190524134238.22802-1-ludo@gnu.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Spam-Score: -2.3 (--) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Fixes a bug whereby 'lzread!' could return more than COUNT. * guix/lzlib.scm (lzread!): Rewrite in a semi-functional style. --- guix/lzlib.scm | 39 +++++++++++++++++++-------------------- 1 file changed, 19 insertions(+), 20 deletions(-) diff --git a/guix/lzlib.scm b/guix/lzlib.scm index 48927c6262..0e384f6cb8 100644 --- a/guix/lzlib.scm +++ b/guix/lzlib.scm @@ -494,29 +494,28 @@ perhaps not yet read." ;; High level functions. -(define* (lzread! decoder file-port bv + +(define* (lzread! decoder port bv #:optional (start 0) (count (bytevector-length bv))) - "Read up to COUNT bytes from FILE-PORT into BV at offset START. Return the + "Read up to COUNT bytes from PORT into BV at offset START. Return the number of uncompressed bytes actually read; it is zero if COUNT is zero or if the end-of-stream has been reached." - ;; WARNING: Because we don't alternate between lz-reads and lz-writes, we can't - ;; process more than lz-decompress-write-size from the file-port. - (when (> count (lz-decompress-write-size decoder)) - (set! count (lz-decompress-write-size decoder))) - (let ((file-bv (get-bytevector-n file-port count))) - (unless (eof-object? file-bv) - (lz-decompress-write decoder file-bv 0 (bytevector-length file-bv)))) - (let ((read 0)) - (let loop ((rd 0)) - (if (< start (bytevector-length bv)) - (begin - (set! rd (lz-decompress-read decoder bv start (- (bytevector-length bv) start))) - (set! start (+ start rd)) - (set! read (+ read rd))) - (set! rd 0)) - (unless (= rd 0) - (loop rd))) - read)) + (define (feed-decoder! decoder) + ;; Feed DECODER with data read from PORT. + (match (get-bytevector-n port (lz-decompress-write-size decoder)) + ((? eof-object? eof) eof) + (bv (lz-decompress-write decoder bv)))) + + (let loop ((read 0) + (start start)) + (cond ((< read count) + (match (lz-decompress-read decoder bv start (- count read)) + (0 (if (eof-object? (feed-decoder! decoder)) + read + (loop read start))) + (n (loop (+ read n) (+ start n))))) + (else + read)))) (define (lzwrite! encoder source source-offset source-count target target-offset target-count) -- 2.21.0 From unknown Sat Jun 14 03:48:44 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#35880] [PATCH 1/7] lzlib: Add 'make-lzip-input-port/compressed'. Resent-From: Pierre Neidhardt Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sat, 25 May 2019 17:25:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 35880 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Ludovic =?UTF-8?Q?Court=C3=A8s?= , 35880@debbugs.gnu.org Cc: Ludovic =?UTF-8?Q?Court=C3=A8s?= Received: via spool by 35880-submit@debbugs.gnu.org id=B35880.155880508310755 (code B ref 35880); Sat, 25 May 2019 17:25:02 +0000 Received: (at 35880) by debbugs.gnu.org; 25 May 2019 17:24:43 +0000 Received: from localhost ([127.0.0.1]:49523 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hUaPT-0002nP-4f for submit@debbugs.gnu.org; Sat, 25 May 2019 13:24:43 -0400 Received: from relay5-d.mail.gandi.net ([217.70.183.197]:34309) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hUaPQ-0002nD-Hy for 35880@debbugs.gnu.org; Sat, 25 May 2019 13:24:41 -0400 X-Originating-IP: 92.169.116.19 Received: from bababa (lfbn-1-4117-19.w92-169.abo.wanadoo.fr [92.169.116.19]) (Authenticated sender: pierre@atlas.engineer) by relay5-d.mail.gandi.net (Postfix) with ESMTPSA id 73F871C0006; Sat, 25 May 2019 17:24:34 +0000 (UTC) From: Pierre Neidhardt In-Reply-To: <20190524134238.22802-1-ludo@gnu.org> References: <20190524134238.22802-1-ludo@gnu.org> Date: Sat, 25 May 2019 19:24:33 +0200 Message-ID: <87d0k6o3am.fsf@ambrevar.xyz> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" X-Spam-Score: -0.2 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.2 (/) --=-=-= Content-Type: text/plain Content-Transfer-Encoding: quoted-printable As an Lzip enthusiast, I have some questions ;) I see you are using make-lzip-input-port/compressed in a subsequent patch, but this does not map how it's done for gzip et al., the latter being invoked via it's system command "gzip -c ...". Why did you decide to do it differently for lzip? Much of the code induced by make-lzip-input-port/compressed seems to repeat the lzread! / lzwrite business, maybe there is a way to factor some of it? > +(define (lzwrite! encoder source source-offset source-count > + target target-offset target-count) > + "Write up to SOURCE-COUNT bytes from SOURCE to ENCODER, and read up to > +TARGET-COUNT bytes into TARGET at TARGET-OFFSET. Return two values: the > +number of bytes read from SOURCE, and the number of bytes written to TAR= GET." > + (define read > + (if (< 0 (lz-compress-write-size encoder)) > + (match (lz-compress-write encoder source source-offset source-co= unt) > + (0 (lz-compress-finish encoder) 0) > + (n n)) > + 0)) > + > + (let loop () > + (match (lz-compress-read encoder target target-offset target-count) > + (0 (loop)) > + (written (values read written))))) Why looping on 0? If there is no byte to read, wouldn't this loop indefini= tely? =2D-=20 Pierre Neidhardt https://ambrevar.xyz/ --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQEzBAEBCAAdFiEEUPM+LlsMPZAEJKvom9z0l6S7zH8FAlzpelEACgkQm9z0l6S7 zH9khggAjW9GVirY7yhqei+pj2GkmLzi7E5bYh5+zCPJtyW8o36VHR93I0/sxM/w wnD3raOmx479JbUmj1QDYiZw+bm056ZWwjLeD8bS9Xk167MhWubAiTbgIbkwjSrv tEtg+TuAUJUGOFbKopksSJhTbykHqpdHKLbMaEW/h2PhkIAGlTjixz6xirZyWpdp oJQTYXqo8eAFOZtUC9IlGvZoeaSBL/1NSpihV5zL99RhRkhxXoGiMUCv8maPgv3b J1ImSuYuhJBkdp8D6znMBQcDHTRFD8Pv3F/hSySIiBz8YpCNZkucrbuK2omKxLIK bVnYrFPjbwzYQjx2/6zisJqIpvmZhw== =T8SE -----END PGP SIGNATURE----- --=-=-=-- From unknown Sat Jun 14 03:48:44 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#35880] [PATCH 3/7] utils: Support compression and decompression with lzip. Resent-From: Pierre Neidhardt Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sat, 25 May 2019 17:28:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 35880 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Ludovic =?UTF-8?Q?Court=C3=A8s?= , 35880@debbugs.gnu.org Cc: Ludovic =?UTF-8?Q?Court=C3=A8s?= Received: via spool by 35880-submit@debbugs.gnu.org id=B35880.155880524911105 (code B ref 35880); Sat, 25 May 2019 17:28:02 +0000 Received: (at 35880) by debbugs.gnu.org; 25 May 2019 17:27:29 +0000 Received: from localhost ([127.0.0.1]:49532 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hUaS8-0002t3-Om for submit@debbugs.gnu.org; Sat, 25 May 2019 13:27:28 -0400 Received: from relay4-d.mail.gandi.net ([217.70.183.196]:37311) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hUaS6-0002su-GT for 35880@debbugs.gnu.org; Sat, 25 May 2019 13:27:27 -0400 X-Originating-IP: 92.169.116.19 Received: from bababa (lfbn-1-4117-19.w92-169.abo.wanadoo.fr [92.169.116.19]) (Authenticated sender: pierre@atlas.engineer) by relay4-d.mail.gandi.net (Postfix) with ESMTPSA id B44E6E0002; Sat, 25 May 2019 17:27:20 +0000 (UTC) From: Pierre Neidhardt In-Reply-To: <20190524134238.22802-3-ludo@gnu.org> References: <20190524134238.22802-1-ludo@gnu.org> <20190524134238.22802-3-ludo@gnu.org> Date: Sat, 25 May 2019 19:27:19 +0200 Message-ID: <87a7fao360.fsf@ambrevar.xyz> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" X-Spam-Score: -0.2 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.2 (/) --=-=-= Content-Type: text/plain Content-Transfer-Encoding: quoted-printable This is the part where you use make-lzip-input-port/compressed: > (define (compressed-port compression input) > - "Return an input port where INPUT is decompressed according to COMPRES= SION, > + "Return an input port where INPUT is compressed according to COMPRESSI= ON, > a symbol such as 'xz." > (match compression > ((or #f 'none) (values input '())) > ('bzip2 (filtered-port `(,%bzip2 "-c") input)) > ('xz (filtered-port `(,%xz "-c") input)) > ('gzip (filtered-port `(,%gzip "-c") input)) > - (else (error "unsupported compression scheme" compression))= )) > + ('lzip (values (lzip-port 'make-lzip-input-port/compressed i= nput) > + '())) > + (_ (error "unsupported compression scheme" compression))= )) So why not doing like for the others? =2D-=20 Pierre Neidhardt https://ambrevar.xyz/ --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQEzBAEBCAAdFiEEUPM+LlsMPZAEJKvom9z0l6S7zH8FAlzpevcACgkQm9z0l6S7 zH9j9ggArez+7dphmivF/ECuibiscKyrWC49iygtu/k68eCTE2kEin9dR453Nio1 g+AI2Xcz6M45BpJ0bp+mHut5MEo/Xs2Se1dApb0/bBBD7YztT1pTMMdDAKVH9DEm XbeT5xxtWX+5Lby4brIwdnkeSzVuIxEAl4CwhatSSD1JqTQ1233K5XIZ13S0abYi VT8cl+Wli3kofKivA/2HaBg6C1/oJXeOmW+5qzBe9mS4ZyxxxbfqA+kWYolcw9VC 5yFX0CKWSSG/EOAcB50MK87E5UlaLui33so7UIa0zZ0T3BO5OyVjDGzH5TdATf4Q 68+IqnvG2AH+vZhBRMxKFysxxCL7gg== =d7zR -----END PGP SIGNATURE----- --=-=-=-- From unknown Sat Jun 14 03:48:44 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#35880] [PATCH 7/7] lzlib: 'lzread!' never returns more than it was asked for. Resent-From: Pierre Neidhardt Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sat, 25 May 2019 17:32:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 35880 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Ludovic =?UTF-8?Q?Court=C3=A8s?= , 35880@debbugs.gnu.org Cc: Ludovic =?UTF-8?Q?Court=C3=A8s?= Received: via spool by 35880-submit@debbugs.gnu.org id=B35880.155880547315911 (code B ref 35880); Sat, 25 May 2019 17:32:02 +0000 Received: (at 35880) by debbugs.gnu.org; 25 May 2019 17:31:13 +0000 Received: from localhost ([127.0.0.1]:49544 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hUaVk-00048K-NU for submit@debbugs.gnu.org; Sat, 25 May 2019 13:31:12 -0400 Received: from relay6-d.mail.gandi.net ([217.70.183.198]:45877) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hUaVj-000464-By for 35880@debbugs.gnu.org; Sat, 25 May 2019 13:31:12 -0400 X-Originating-IP: 92.169.116.19 Received: from bababa (lfbn-1-4117-19.w92-169.abo.wanadoo.fr [92.169.116.19]) (Authenticated sender: pierre@atlas.engineer) by relay6-d.mail.gandi.net (Postfix) with ESMTPSA id 7023FC0007; Sat, 25 May 2019 17:31:04 +0000 (UTC) From: Pierre Neidhardt In-Reply-To: <20190524134238.22802-7-ludo@gnu.org> References: <20190524134238.22802-1-ludo@gnu.org> <20190524134238.22802-7-ludo@gnu.org> Date: Sat, 25 May 2019 19:31:04 +0200 Message-ID: <877eaeo2zr.fsf@ambrevar.xyz> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" X-Spam-Score: -0.6 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.6 (/) --=-=-= Content-Type: text/plain Content-Transfer-Encoding: quoted-printable > Fixes a bug whereby 'lzread!' could return more than COUNT. Hmm... But why is this a bug? lzread! returns the number of _uncompressed_ bytes, while COUNT is number of _compressed_ bytes written, so it's often expected that the former is more than the latter. (By the way, nice rewrite, I like it much better than my C->Scheme translation ;p) =2D-=20 Pierre Neidhardt https://ambrevar.xyz/ --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQEzBAEBCAAdFiEEUPM+LlsMPZAEJKvom9z0l6S7zH8FAlzpe9gACgkQm9z0l6S7 zH+AhQgAk/zszEZtpg/9chO9ywyweduRP7r6nFabLqaqtamgNhfcRqVi0MyEykGY n9nX8nG2mhR/Ips4RNXSe9LNXOYGGbSOwcqrACpiny0WAQ8Ftnmk3lv6i7FDSrFo bUEusFOxwt1nX0oMLG1vbkNJEksICL53f5ey7yF+8xTqSHPxJz2IF4RTnFXCb5I4 jvu+KbETUO8YGwk176y9EwfZErWNKAT44FPbg9gxbrexMbaUbJaEJX6aSfMTM1zL 1At3sWDYYeVSHC5CTE+KcvhVCXung66VFA16dXPReUa1Bh2niZT4/iyggw6Nwpvs 39VgF3Q/7sHSLWSSljmGFOQxotLdTA== =wMKY -----END PGP SIGNATURE----- --=-=-=-- From unknown Sat Jun 14 03:48:44 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#35880] [PATCH 1/7] lzlib: Add 'make-lzip-input-port/compressed'. Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sun, 26 May 2019 19:52:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 35880 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Pierre Neidhardt Cc: 35880@debbugs.gnu.org Received: via spool by 35880-submit@debbugs.gnu.org id=B35880.15589003125065 (code B ref 35880); Sun, 26 May 2019 19:52:01 +0000 Received: (at 35880) by debbugs.gnu.org; 26 May 2019 19:51:52 +0000 Received: from localhost ([127.0.0.1]:51815 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hUzBQ-0001Jd-Hc for submit@debbugs.gnu.org; Sun, 26 May 2019 15:51:52 -0400 Received: from eggs.gnu.org ([209.51.188.92]:59299) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hUzBN-0001JP-Lx for 35880@debbugs.gnu.org; Sun, 26 May 2019 15:51:50 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:50139) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hUzBH-0001qs-Gd; Sun, 26 May 2019 15:51:43 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=54842 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1hUzBG-0004DY-Qv; Sun, 26 May 2019 15:51:43 -0400 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= References: <20190524134238.22802-1-ludo@gnu.org> <87d0k6o3am.fsf@ambrevar.xyz> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 7 Prairial an 227 de la =?UTF-8?Q?R=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Sun, 26 May 2019 21:51:41 +0200 In-Reply-To: <87d0k6o3am.fsf@ambrevar.xyz> (Pierre Neidhardt's message of "Sat, 25 May 2019 19:24:33 +0200") Message-ID: <87ftp1m1te.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Spam-Score: -2.3 (--) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hi! Pierre Neidhardt skribis: > As an Lzip enthusiast, I have some questions ;) > > I see you are using make-lzip-input-port/compressed in a subsequent > patch, but this does not map how it's done for gzip et al., the latter > being invoked via it's system command "gzip -c ...". Why did you decide > to do it differently for lzip? > > Much of the code induced by make-lzip-input-port/compressed seems to > repeat the lzread! / lzwrite business, maybe there is a way to factor > some of it? Actually, =E2=80=98make-lzip-input-port/compressed=E2=80=99 exists solely s= o we can have =E2=80=98compressed-port=E2=80=99 for lzip, which in turn allows us to writ= e tests. It uses (guix lzlib) instead of invoking the =E2=80=98lzip=E2=80=99 command= because we can. :-) Invoking commands is not as nice, because it=E2=80=99s more expensive, requires us to spawn an additional process when the input is not a file port (e.g., it=E2=80=99s a string port), and it=E2=80=99s forkin= g is not possible in multi-threaded programs like =E2=80=98guix publish=E2=80=99. >> +(define (lzwrite! encoder source source-offset source-count >> + target target-offset target-count) >> + "Write up to SOURCE-COUNT bytes from SOURCE to ENCODER, and read up to >> +TARGET-COUNT bytes into TARGET at TARGET-OFFSET. Return two values: the >> +number of bytes read from SOURCE, and the number of bytes written to TA= RGET." >> + (define read >> + (if (< 0 (lz-compress-write-size encoder)) >> + (match (lz-compress-write encoder source source-offset source-c= ount) >> + (0 (lz-compress-finish encoder) 0) >> + (n n)) >> + 0)) >> + >> + (let loop () >> + (match (lz-compress-read encoder target target-offset target-count) >> + (0 (loop)) >> + (written (values read written))))) > > Why looping on 0? If there is no byte to read, wouldn't this loop indefi= nitely? Hmm, good point. The idea is that =E2=80=98lzwrite!=E2=80=99 should return= 0 only on end-of-file, but then the loop should include reading more from SOURCE. I=E2=80=99ll follow up on this one. Thanks! Ludo=E2=80=99. From unknown Sat Jun 14 03:48:44 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#35880] [PATCH 3/7] utils: Support compression and decompression with lzip. Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sun, 26 May 2019 19:53:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 35880 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Pierre Neidhardt Cc: 35880@debbugs.gnu.org Received: via spool by 35880-submit@debbugs.gnu.org id=B35880.15589003475144 (code B ref 35880); Sun, 26 May 2019 19:53:02 +0000 Received: (at 35880) by debbugs.gnu.org; 26 May 2019 19:52:27 +0000 Received: from localhost ([127.0.0.1]:51821 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hUzBx-0001Ks-Ue for submit@debbugs.gnu.org; Sun, 26 May 2019 15:52:26 -0400 Received: from eggs.gnu.org ([209.51.188.92]:59405) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hUzBw-0001Kb-7U for 35880@debbugs.gnu.org; Sun, 26 May 2019 15:52:24 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:50158) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hUzBr-0002B3-2P; Sun, 26 May 2019 15:52:19 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=54844 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1hUzBo-0004GB-SY; Sun, 26 May 2019 15:52:18 -0400 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= References: <20190524134238.22802-1-ludo@gnu.org> <20190524134238.22802-3-ludo@gnu.org> <87a7fao360.fsf@ambrevar.xyz> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 7 Prairial an 227 de la =?UTF-8?Q?R=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Sun, 26 May 2019 21:52:15 +0200 In-Reply-To: <87a7fao360.fsf@ambrevar.xyz> (Pierre Neidhardt's message of "Sat, 25 May 2019 19:27:19 +0200") Message-ID: <878sutm1sg.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Spam-Score: -2.3 (--) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Pierre Neidhardt skribis: > This is the part where you use make-lzip-input-port/compressed: > >> (define (compressed-port compression input) >> - "Return an input port where INPUT is decompressed according to COMPRESSION, >> + "Return an input port where INPUT is compressed according to COMPRESSION, >> a symbol such as 'xz." >> (match compression >> ((or #f 'none) (values input '())) >> ('bzip2 (filtered-port `(,%bzip2 "-c") input)) >> ('xz (filtered-port `(,%xz "-c") input)) >> ('gzip (filtered-port `(,%gzip "-c") input)) >> - (else (error "unsupported compression scheme" compression)))) >> + ('lzip (values (lzip-port 'make-lzip-input-port/compressed input) >> + '())) >> + (_ (error "unsupported compression scheme" compression)))) > > So why not doing like for the others? See my other previous reply. :-) From unknown Sat Jun 14 03:48:44 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#35880] [PATCH 7/7] lzlib: 'lzread!' never returns more than it was asked for. Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sun, 26 May 2019 19:55:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 35880 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Pierre Neidhardt Cc: 35880@debbugs.gnu.org Received: via spool by 35880-submit@debbugs.gnu.org id=B35880.15589005005390 (code B ref 35880); Sun, 26 May 2019 19:55:01 +0000 Received: (at 35880) by debbugs.gnu.org; 26 May 2019 19:55:00 +0000 Received: from localhost ([127.0.0.1]:51828 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hUzES-0001Or-Dq for submit@debbugs.gnu.org; Sun, 26 May 2019 15:55:00 -0400 Received: from eggs.gnu.org ([209.51.188.92]:59590) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hUzEQ-0001Of-VY for 35880@debbugs.gnu.org; Sun, 26 May 2019 15:54:59 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:50207) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hUzEK-0003c3-My; Sun, 26 May 2019 15:54:53 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=54850 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1hUzEK-0004PM-6y; Sun, 26 May 2019 15:54:52 -0400 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= References: <20190524134238.22802-1-ludo@gnu.org> <20190524134238.22802-7-ludo@gnu.org> <877eaeo2zr.fsf@ambrevar.xyz> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 7 Prairial an 227 de la =?UTF-8?Q?R=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Sun, 26 May 2019 21:54:50 +0200 In-Reply-To: <877eaeo2zr.fsf@ambrevar.xyz> (Pierre Neidhardt's message of "Sat, 25 May 2019 19:31:04 +0200") Message-ID: <87zhn9kn3p.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Spam-Score: -2.3 (--) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Pierre Neidhardt skribis: >> Fixes a bug whereby 'lzread!' could return more than COUNT. > > Hmm... But why is this a bug? Because then the =E2=80=98read!=E2=80=99 method of the custom binary input = port could return more than =E2=80=98count=E2=80=99, which is understandably not permi= tted. > (By the way, nice rewrite, I like it much better than my C->Scheme > translation ;p) Heheh, I initially tried to minimize changes but I found it easier to reason about this version. Ludo=E2=80=99. From unknown Sat Jun 14 03:48:44 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#35880] [PATCH 7/7] lzlib: 'lzread!' never returns more than it was asked for. Resent-From: Pierre Neidhardt Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sun, 26 May 2019 20:59:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 35880 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Ludovic =?UTF-8?Q?Court=C3=A8s?= Cc: 35880@debbugs.gnu.org Received: via spool by 35880-submit@debbugs.gnu.org id=B35880.155890428319932 (code B ref 35880); Sun, 26 May 2019 20:59:02 +0000 Received: (at 35880) by debbugs.gnu.org; 26 May 2019 20:58:03 +0000 Received: from localhost ([127.0.0.1]:51909 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hV0DT-0005BQ-3j for submit@debbugs.gnu.org; Sun, 26 May 2019 16:58:03 -0400 Received: from relay7-d.mail.gandi.net ([217.70.183.200]:42859) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hV0DP-0005Aw-Ub for 35880@debbugs.gnu.org; Sun, 26 May 2019 16:58:01 -0400 X-Originating-IP: 92.169.116.19 Received: from bababa (lfbn-1-4117-19.w92-169.abo.wanadoo.fr [92.169.116.19]) (Authenticated sender: pierre@atlas.engineer) by relay7-d.mail.gandi.net (Postfix) with ESMTPSA id F10C020002; Sun, 26 May 2019 20:57:52 +0000 (UTC) From: Pierre Neidhardt In-Reply-To: <87zhn9kn3p.fsf@gnu.org> References: <20190524134238.22802-1-ludo@gnu.org> <20190524134238.22802-7-ludo@gnu.org> <877eaeo2zr.fsf@ambrevar.xyz> <87zhn9kn3p.fsf@gnu.org> Date: Sun, 26 May 2019 22:57:51 +0200 Message-ID: <87r28llyr4.fsf@ambrevar.xyz> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" X-Spam-Score: -0.2 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.2 (/) --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Ludovic Court=C3=A8s writes: > Pierre Neidhardt skribis: > >>> Fixes a bug whereby 'lzread!' could return more than COUNT. >> >> Hmm... But why is this a bug? > > Because then the =E2=80=98read!=E2=80=99 method of the custom binary inpu= t port could > return more than =E2=80=98count=E2=80=99, which is understandably not per= mitted. That's the part where I'm a bit confused because we deal with compressed data here. So when we say "(read count)", does COUNT refer to the compressed or uncompressed data? =2D-=20 Pierre Neidhardt https://ambrevar.xyz/ --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQEzBAEBCAAdFiEEUPM+LlsMPZAEJKvom9z0l6S7zH8FAlzq/c8ACgkQm9z0l6S7 zH8ojgf+I93bYdoP1uDQc78DwZ0LFwkbK5Hp/aWhrWM39/SVANv5LiEpaOtZYWzm eUNbPcqy0gMJJ84EB6afMfjwqM2jOJwc9WDBGUzprFKOqLWl976ZGDmLS6Fo2B50 LrheiEmsmHMs/JRiyN5AbFGXqIYHF1Mg/fxQgLTgx9W+14mI1Kk2FvtX0Cxjd7/X f6ovZefsPAwbVjqGFwHAHGTlQsodHOGGXV+9sFXFXRQVe8tSbMkK25ehJS0jNpwM yM6PcccrkEX5bjM9bkLqxqgGfvhmxWsByKeIJvZx0Ty5qK+xTU5BxDjyk5w0TpGa NwmswBCF45qHpsDAkMntWuj32EhOeQ== =8MXv -----END PGP SIGNATURE----- --=-=-=-- From unknown Sat Jun 14 03:48:44 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#35880] [PATCH 7/7] lzlib: 'lzread!' never returns more than it was asked for. Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sun, 26 May 2019 21:29:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 35880 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Pierre Neidhardt Cc: 35880@debbugs.gnu.org Received: via spool by 35880-submit@debbugs.gnu.org id=B35880.155890613422937 (code B ref 35880); Sun, 26 May 2019 21:29:02 +0000 Received: (at 35880) by debbugs.gnu.org; 26 May 2019 21:28:54 +0000 Received: from localhost ([127.0.0.1]:51967 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hV0hK-0005xt-5f for submit@debbugs.gnu.org; Sun, 26 May 2019 17:28:54 -0400 Received: from eggs.gnu.org ([209.51.188.92]:42369) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hV0hG-0005xd-6s for 35880@debbugs.gnu.org; Sun, 26 May 2019 17:28:50 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:52830) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hV0h9-0003uJ-UH; Sun, 26 May 2019 17:28:44 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=49916 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1hV0h6-00006j-CB; Sun, 26 May 2019 17:28:42 -0400 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= References: <20190524134238.22802-1-ludo@gnu.org> <20190524134238.22802-7-ludo@gnu.org> <877eaeo2zr.fsf@ambrevar.xyz> <87zhn9kn3p.fsf@gnu.org> <87r28llyr4.fsf@ambrevar.xyz> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 7 Prairial an 227 de la =?UTF-8?Q?R=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Sun, 26 May 2019 23:28:38 +0200 In-Reply-To: <87r28llyr4.fsf@ambrevar.xyz> (Pierre Neidhardt's message of "Sun, 26 May 2019 22:57:51 +0200") Message-ID: <87a7f8kird.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Spam-Score: -2.3 (--) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Pierre Neidhardt skribis: > Ludovic Court=C3=A8s writes: > >> Pierre Neidhardt skribis: >> >>>> Fixes a bug whereby 'lzread!' could return more than COUNT. >>> >>> Hmm... But why is this a bug? >> >> Because then the =E2=80=98read!=E2=80=99 method of the custom binary inp= ut port could >> return more than =E2=80=98count=E2=80=99, which is understandably not pe= rmitted. > > That's the part where I'm a bit confused because we deal with compressed > data here. > > So when we say "(read count)", does COUNT refer to the compressed or > uncompressed data? We have this: (define* (make-lzip-input-port port) (define decoder (lz-decompress-open)) (define (read! bv start count) (lzread! decoder port bv start count)) (make-custom-binary-input-port "lzip-input" read! #f #f (lambda () =E2=80=A6))) Here =E2=80=98read!=E2=80=99 must return an integer between 1 and COUNT; it= must return 0 if and only if the end-of-file is reached. IOW, =E2=80=98lzread!=E2=80=99 must return the number of uncompressed bytes= of BV that it consumed, and that number is necessarily <=3D COUNT. Does that make sense? Thanks, Ludo=E2=80=99. From unknown Sat Jun 14 03:48:44 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#35880] [PATCH 7/7] lzlib: 'lzread!' never returns more than it was asked for. Resent-From: Pierre Neidhardt Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Mon, 27 May 2019 07:01:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 35880 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Ludovic =?UTF-8?Q?Court=C3=A8s?= Cc: 35880@debbugs.gnu.org Received: via spool by 35880-submit@debbugs.gnu.org id=B35880.155894041318356 (code B ref 35880); Mon, 27 May 2019 07:01:01 +0000 Received: (at 35880) by debbugs.gnu.org; 27 May 2019 07:00:13 +0000 Received: from localhost ([127.0.0.1]:52509 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hV9cD-0004lz-Ee for submit@debbugs.gnu.org; Mon, 27 May 2019 03:00:13 -0400 Received: from relay3-d.mail.gandi.net ([217.70.183.195]:36053) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hV9cA-0004lo-WB for 35880@debbugs.gnu.org; Mon, 27 May 2019 03:00:12 -0400 X-Originating-IP: 92.169.116.19 Received: from bababa (lfbn-1-4117-19.w92-169.abo.wanadoo.fr [92.169.116.19]) (Authenticated sender: pierre@atlas.engineer) by relay3-d.mail.gandi.net (Postfix) with ESMTPSA id 750DE60012; Mon, 27 May 2019 07:00:05 +0000 (UTC) From: Pierre Neidhardt In-Reply-To: <87a7f8kird.fsf@gnu.org> References: <20190524134238.22802-1-ludo@gnu.org> <20190524134238.22802-7-ludo@gnu.org> <877eaeo2zr.fsf@ambrevar.xyz> <87zhn9kn3p.fsf@gnu.org> <87r28llyr4.fsf@ambrevar.xyz> <87a7f8kird.fsf@gnu.org> Date: Mon, 27 May 2019 09:00:03 +0200 Message-ID: <87a7f8mlfw.fsf@ambrevar.xyz> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" X-Spam-Score: -0.6 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.6 (/) --=-=-= Content-Type: text/plain Content-Transfer-Encoding: quoted-printable It does make sense, but then don't we have the same issue with zlib.scm: =2D-8<---------------cut here---------------start------------->8--- (define gzread! (let ((proc (zlib-procedure int "gzread" (list '* '* unsigned-int)))) (lambda* (gzfile bv #:optional (start 0) (count (bytevector-length bv))) "Read up to COUNT bytes from GZFILE into BV at offset START. Return = the number of uncompressed bytes actually read; it is zero if COUNT is zero or = if the end-of-stream has been reached." ... =2D-8<---------------cut here---------------end--------------->8--- I initially tried to mimic zlib.scm and this part confused me a lot back th= en. =2D-=20 Pierre Neidhardt https://ambrevar.xyz/ --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQEzBAEBCAAdFiEEUPM+LlsMPZAEJKvom9z0l6S7zH8FAlzrivMACgkQm9z0l6S7 zH/k8AgAnuANtYKcJPMxwlex95fPtVn/zOHGgWsz9ZbNoL/svuDgQgUYHHoUE7Rb WP/8jO1gzWmcD/kN0yoxqxjDsJ6l4S49ZAG+C6Z8CXurAJdoFqTMc3hXS4himnop 4ExpModEGqwrI2IQ6U7g58/hz7F5Mj2LrjgxTrVTZsdK7RD1gM0Fnrulih3uScQx 5VXTJpqyASsXcovdN23cCNPT2k9IzJcVHkdwj8ok2bDGOCoLcjeQBdehALY4JIKw HTuZ9/vEkPAnoHel9H7PXLWH6Id8Pqs1pTWjhYkvCOXPMnqLjhdwoL+LS4WpwGNG YcwrTVahk0+v14BxJstzXtkddJNVlA== =UXvg -----END PGP SIGNATURE----- --=-=-=-- From unknown Sat Jun 14 03:48:44 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#35880] [PATCH 7/7] lzlib: 'lzread!' never returns more than it was asked for. Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Mon, 27 May 2019 10:02:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 35880 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Pierre Neidhardt Cc: 35880@debbugs.gnu.org Received: via spool by 35880-submit@debbugs.gnu.org id=B35880.15589512687015 (code B ref 35880); Mon, 27 May 2019 10:02:01 +0000 Received: (at 35880) by debbugs.gnu.org; 27 May 2019 10:01:08 +0000 Received: from localhost ([127.0.0.1]:52670 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hVCRI-0001p5-BO for submit@debbugs.gnu.org; Mon, 27 May 2019 06:01:08 -0400 Received: from eggs.gnu.org ([209.51.188.92]:60079) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hVCRF-0001oF-6F for 35880@debbugs.gnu.org; Mon, 27 May 2019 06:01:06 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:34521) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hVCR8-000394-Hx; Mon, 27 May 2019 06:00:58 -0400 Received: from [2001:660:6102:320:e120:2c8f:8909:cdfe] (port=57956 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1hVCR6-0000EE-WE; Mon, 27 May 2019 06:00:58 -0400 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= References: <20190524134238.22802-1-ludo@gnu.org> <20190524134238.22802-7-ludo@gnu.org> <877eaeo2zr.fsf@ambrevar.xyz> <87zhn9kn3p.fsf@gnu.org> <87r28llyr4.fsf@ambrevar.xyz> <87a7f8kird.fsf@gnu.org> <87a7f8mlfw.fsf@ambrevar.xyz> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 8 Prairial an 227 de la =?UTF-8?Q?R=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Mon, 27 May 2019 12:00:54 +0200 In-Reply-To: <87a7f8mlfw.fsf@ambrevar.xyz> (Pierre Neidhardt's message of "Mon, 27 May 2019 09:00:03 +0200") Message-ID: <87pno4z06h.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Spam-Score: -2.3 (--) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hi Pierre, Pierre Neidhardt skribis: > It does make sense, but then don't we have the same issue with zlib.scm: Which issue? > (define gzread! > (let ((proc (zlib-procedure int "gzread" (list '* '* unsigned-int)))) > (lambda* (gzfile bv #:optional (start 0) (count (bytevector-length bv= ))) > "Read up to COUNT bytes from GZFILE into BV at offset START. Retur= n the > number of uncompressed bytes actually read; it is zero if COUNT is zero o= r if > the end-of-stream has been reached." > ... > > I initially tried to mimic zlib.scm and this part confused me a lot back = then. There=E2=80=99s a key difference: the =E2=80=98gzread=E2=80=99 etc. API is = high-level and easy to use, but it wants a file descriptor to read from (thus a file port in Scheme land.) That=E2=80=99s enough for =E2=80=98guix publish=E2=80=99, which writes gzip= ped data to files, but that=E2=80=99s not enough for =E2=80=98guix substitute=E2=80=99, which = can read data from non-file ports (e.g., chunked-encoding ports or TLS ports from the HTTP client.) HTH, Ludo=E2=80=99. From unknown Sat Jun 14 03:48:44 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#35880] [PATCH 1/7] lzlib: Add 'make-lzip-input-port/compressed'. Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Mon, 27 May 2019 15:46:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 35880 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Pierre Neidhardt Cc: 35880@debbugs.gnu.org Received: via spool by 35880-submit@debbugs.gnu.org id=B35880.15589719485187 (code B ref 35880); Mon, 27 May 2019 15:46:02 +0000 Received: (at 35880) by debbugs.gnu.org; 27 May 2019 15:45:48 +0000 Received: from localhost ([127.0.0.1]:53828 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hVHoq-0001Lb-7K for submit@debbugs.gnu.org; Mon, 27 May 2019 11:45:48 -0400 Received: from eggs.gnu.org ([209.51.188.92]:43058) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hVHoo-0001LG-Bc for 35880@debbugs.gnu.org; Mon, 27 May 2019 11:45:46 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:38937) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hVHoX-0003OG-8w; Mon, 27 May 2019 11:45:32 -0400 Received: from [2001:660:6102:320:e120:2c8f:8909:cdfe] (port=37538 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1hVHoT-00004v-E9; Mon, 27 May 2019 11:45:25 -0400 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= References: <20190524134238.22802-1-ludo@gnu.org> <87d0k6o3am.fsf@ambrevar.xyz> <87ftp1m1te.fsf@gnu.org> Date: Mon, 27 May 2019 17:45:21 +0200 In-Reply-To: <87ftp1m1te.fsf@gnu.org> ("Ludovic \=\?utf-8\?Q\?Court\=C3\=A8s\=22'\?\= \=\?utf-8\?Q\?s\?\= message of "Sun, 26 May 2019 21:51:41 +0200") Message-ID: <87blznsxym.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Spam-Score: -2.3 (--) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Ludovic Court=C3=A8s skribis: > Pierre Neidhardt skribis: [...] >>> +(define (lzwrite! encoder source source-offset source-count >>> + target target-offset target-count) >>> + "Write up to SOURCE-COUNT bytes from SOURCE to ENCODER, and read up = to >>> +TARGET-COUNT bytes into TARGET at TARGET-OFFSET. Return two values: t= he >>> +number of bytes read from SOURCE, and the number of bytes written to T= ARGET." >>> + (define read >>> + (if (< 0 (lz-compress-write-size encoder)) >>> + (match (lz-compress-write encoder source source-offset source-= count) >>> + (0 (lz-compress-finish encoder) 0) >>> + (n n)) >>> + 0)) >>> + >>> + (let loop () >>> + (match (lz-compress-read encoder target target-offset target-count) >>> + (0 (loop)) >>> + (written (values read written))))) >> >> Why looping on 0? If there is no byte to read, wouldn't this loop indef= initely? > > Hmm, good point. The idea is that =E2=80=98lzwrite!=E2=80=99 should retu= rn 0 only on > end-of-file, but then the loop should include reading more from SOURCE. > I=E2=80=99ll follow up on this one. I noticed that =E2=80=98lz-compress-read=E2=80=99 is documented to return a= =E2=80=9Cstrictly positive integer=E2=80=9D, so I=E2=80=99m changing it to this: --8<---------------cut here---------------start------------->8--- (define (lzwrite! encoder source source-offset source-count target target-offset target-count) "Write up to SOURCE-COUNT bytes from SOURCE to ENCODER, and read up to TARGET-COUNT bytes into TARGET at TARGET-OFFSET. Return two values: the number of bytes read from SOURCE, and the non-zero number of bytes written = to TARGET." (define read (if (< 0 (lz-compress-write-size encoder)) (match (lz-compress-write encoder source source-offset source-count) (0 (lz-compress-finish encoder) 0) (n n)) 0)) (define written ;; Note: 'lz-compress-read' promises to return a non-zero integer. (lz-compress-read encoder target target-offset target-count)) (values read written)) --8<---------------cut here---------------end--------------->8--- Let me know what you think! Ludo=E2=80=99. From unknown Sat Jun 14 03:48:44 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#35880] [PATCH 1/7] lzlib: Add 'make-lzip-input-port/compressed'. Resent-From: Pierre Neidhardt Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Mon, 27 May 2019 16:25:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 35880 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Ludovic =?UTF-8?Q?Court=C3=A8s?= Cc: 35880@debbugs.gnu.org Received: via spool by 35880-submit@debbugs.gnu.org id=B35880.15589742549549 (code B ref 35880); Mon, 27 May 2019 16:25:02 +0000 Received: (at 35880) by debbugs.gnu.org; 27 May 2019 16:24:14 +0000 Received: from localhost ([127.0.0.1]:53892 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hVIQ1-0002Tw-OP for submit@debbugs.gnu.org; Mon, 27 May 2019 12:24:13 -0400 Received: from relay1-d.mail.gandi.net ([217.70.183.193]:53187) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hVIPz-0002Tk-Cz for 35880@debbugs.gnu.org; Mon, 27 May 2019 12:24:12 -0400 X-Originating-IP: 92.169.116.19 Received: from bababa (lfbn-1-4117-19.w92-169.abo.wanadoo.fr [92.169.116.19]) (Authenticated sender: pierre@atlas.engineer) by relay1-d.mail.gandi.net (Postfix) with ESMTPSA id 4C56F240005; Mon, 27 May 2019 16:24:08 +0000 (UTC) From: Pierre Neidhardt In-Reply-To: <87blznsxym.fsf@gnu.org> References: <20190524134238.22802-1-ludo@gnu.org> <87d0k6o3am.fsf@ambrevar.xyz> <87ftp1m1te.fsf@gnu.org> <87blznsxym.fsf@gnu.org> Date: Mon, 27 May 2019 18:24:08 +0200 Message-ID: <87ef4jlvbr.fsf@ambrevar.xyz> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" X-Spam-Score: -0.2 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.2 (/) --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Ludovic Court=C3=A8s writes: > I noticed that =E2=80=98lz-compress-read=E2=80=99 is documented to return= a =E2=80=9Cstrictly > positive integer=E2=80=9D, so I=E2=80=99m changing it to this: The docstring (that I wrote) says "non-negative positive integer." "positive" is a typo (sorry about that), it should read "non-negative integer" since the return value can be zero. In general, lzlib's "reads" and "writes" don't give any guarantee about the size of bytes that are actually processed. You need to loop over the calls until some condition is met, see the "finish(ed)" functions. Here in particular, it's not clear that lz-compress-read is going to read all the bytes in the encoder buffer. Maybe that's OK for this particular functions if we don't expect TARGET-COUNT to be reached. See http://www.nongnu.org/lzip/manual/lzlib_manual.html#Buffering That said, if the encoder buffer is not empty, I think lz-compress-read should always return something >0. Yup, lzlib is very low-level! :p =2D-=20 Pierre Neidhardt https://ambrevar.xyz/ --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQEzBAEBCAAdFiEEUPM+LlsMPZAEJKvom9z0l6S7zH8FAlzsDygACgkQm9z0l6S7 zH+djAf6AjSgBiegQaqjfWkGc5AAWo++7B/p06iLMWtKOpgshqdpvy+0Qt0ZkjSN S3GJ5d5ZImZNO6jEq+iDN3KmvZ0+1VV87kHR0vtTeiWZfwNjIcv+xVzfw0835SS+ srRyhzZA4PbUvKVq11SOk8S7WJbpA2beeIMwBewv1LkehBojz4MZ40e8D6/PJuvB Lr23dfLKqrLzQQ5NvXiK1qSuuMysAkEPUfaRbYMHNIvvwNUjYBqehbJ8+K70/Sla mEhZFaxg+2WIO6+5PHcrnVfE1Wg00TMjXKYSkKIGmNTV91jQQ/kLvlz/gts9XXZh DLu3iyjBStonPCQOh+/PT2rNqYsqIg== =JO4j -----END PGP SIGNATURE----- --=-=-=-- From unknown Sat Jun 14 03:48:44 2025 MIME-Version: 1.0 X-Mailer: MIME-tools 5.505 (Entity 5.505) X-Loop: help-debbugs@gnu.org From: help-debbugs@gnu.org (GNU bug Tracking System) To: Ludovic =?UTF-8?Q?Court=C3=A8s?= Subject: bug#35880: closed (Re: [bug#35880] [PATCH 1/7] lzlib: Add 'make-lzip-input-port/compressed'.) Message-ID: References: <87lfyrr554.fsf@gnu.org> <20190524133159.22568-1-ludo@gnu.org> X-Gnu-PR-Message: they-closed 35880 X-Gnu-PR-Package: guix-patches X-Gnu-PR-Keywords: patch Reply-To: 35880@debbugs.gnu.org Date: Mon, 27 May 2019 20:54:04 +0000 Content-Type: multipart/mixed; boundary="----------=_1558990444-13130-1" This is a multi-part message in MIME format... ------------=_1558990444-13130-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Your bug report #35880: [PATCH 0/7] Lzip support for 'guix publish' and 'guix substitute' which was filed against the guix-patches package, has been closed. The explanation is attached below, along with your original report. If you require more details, please reply to 35880@debbugs.gnu.org. --=20 35880: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D35880 GNU Bug Tracking System Contact help-debbugs@gnu.org with problems ------------=_1558990444-13130-1 Content-Type: message/rfc822 Content-Disposition: inline Content-Transfer-Encoding: 7bit Received: (at 35880-done) by debbugs.gnu.org; 27 May 2019 20:53:23 +0000 Received: from localhost ([127.0.0.1]:54307 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hVMcV-0003Oq-4i for submit@debbugs.gnu.org; Mon, 27 May 2019 16:53:23 -0400 Received: from eggs.gnu.org ([209.51.188.92]:44688) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hVMcS-0003Od-Ky for 35880-done@debbugs.gnu.org; Mon, 27 May 2019 16:53:21 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:44788) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hVMcM-0006dB-Je; Mon, 27 May 2019 16:53:14 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=55254 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1hVMcM-0004ks-6R; Mon, 27 May 2019 16:53:14 -0400 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Pierre Neidhardt Subject: Re: [bug#35880] [PATCH 1/7] lzlib: Add 'make-lzip-input-port/compressed'. References: <20190524134238.22802-1-ludo@gnu.org> <87d0k6o3am.fsf@ambrevar.xyz> <87ftp1m1te.fsf@gnu.org> <87blznsxym.fsf@gnu.org> <87ef4jlvbr.fsf@ambrevar.xyz> Date: Mon, 27 May 2019 22:53:11 +0200 In-Reply-To: <87ef4jlvbr.fsf@ambrevar.xyz> (Pierre Neidhardt's message of "Mon, 27 May 2019 18:24:08 +0200") Message-ID: <87lfyrr554.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 35880-done Cc: 35880-done@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hi! Pierre Neidhardt skribis: > That said, if the encoder buffer is not empty, I think lz-compress-read > should always return something >0. Yes, probably. The docstring for =E2=80=98lz-compress-read=E2=80=99 says: "Read up to COUNT bytes from the encoder stream, storing the results in= LZFILE-BV. Return the number of uncompressed bytes written, a strictly positive inte= ger." ^~~~~~~~~~~~~~~~~ However, the lzlib manual doesn=E2=80=99t say that for =E2=80=98LZ_compress= _read=E2=80=99 (info "(lzlib) Compression functions"). But that=E2=80=99s OK: the =E2=80=98read!=E2=80=99 method in =E2=80=98make-= lzip-input-port/compressed=E2=80=99 can just call =E2=80=98lzwrite!=E2=80=99 again with more data when that hap= pens, so I=E2=80=99ve done that. And I pushed the whole thing! :-) I think it=E2=80=99d be good to let people play with it in their personal setups. Next up: multi-compression support in =E2=80=98guix publish=E2=80=99 (possi= bly?) so we can smoothly transition on our build farms. Thanks! Ludo=E2=80=99. ------------=_1558990444-13130-1 Content-Type: message/rfc822 Content-Disposition: inline Content-Transfer-Encoding: 7bit Received: (at submit) by debbugs.gnu.org; 24 May 2019 13:33:14 +0000 Received: from localhost ([127.0.0.1]:46267 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hUAJu-00025X-8w for submit@debbugs.gnu.org; Fri, 24 May 2019 09:33:14 -0400 Received: from eggs.gnu.org ([209.51.188.92]:47788) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hUAJt-00025E-0T for submit@debbugs.gnu.org; Fri, 24 May 2019 09:33:13 -0400 Received: from lists.gnu.org ([209.51.188.17]:49366) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1hUAJn-00087p-SW for submit@debbugs.gnu.org; Fri, 24 May 2019 09:33:07 -0400 Received: from eggs.gnu.org ([209.51.188.92]:33887) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hUAJm-0001gR-S9 for guix-patches@gnu.org; Fri, 24 May 2019 09:33:07 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_40 autolearn=disabled version=3.3.2 Received: from fencepost.gnu.org ([2001:470:142:3::e]:38653) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hUAIn-0007aC-3W; Fri, 24 May 2019 09:32:05 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=36416 helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:DHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1hUAIm-0005a1-At; Fri, 24 May 2019 09:32:04 -0400 From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= To: guix-patches@gnu.org Subject: [PATCH 0/7] Lzip support for 'guix publish' and 'guix substitute' Date: Fri, 24 May 2019 15:31:59 +0200 Message-Id: <20190524133159.22568-1-ludo@gnu.org> X-Mailer: git-send-email 2.21.0 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6.x X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: submit Cc: =?UTF-8?q?Ludovic=20Court=C3=A8s?= , Pierre Neidhardt X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hi! As a followup to Pierre’s work on (guix lzlib), these patches implement ‘lzip’ support for ‘guix substitute’ and ‘guix publish’. With these, you can now run: ./pre-inst-env guix publish -Clzip … on one side, and on another machine: ./pre-inst-env guix-daemon --build-users-group=guixbuild and from there the client machine should be able to fetch lzip-compressed substitutes. These patches do not address the transitioning issue that we discussed earlier, where we have clients lacking lzip support talking to an lzip-capable server. As discussed earlier, clients will have to send a special HTTP header, ‘X-Guix-Accept-Encoding’. For the server-side, I’m still hesitating between implementing it in ‘guix publish’ or simply running two instances of ‘guix publish’ side-by-side (one gzip and one lzip) and letting nginx dispatch between the two. :-) Note that we’ll have to adjust our nginx mirror configs to take that header into account! Comments? Ludo’. Ludovic Courtès (7): lzlib: Add 'make-lzip-input-port/compressed'. utils: Test 'compressed-port' and 'decompressed-port' for both gzip and xz. utils: Support compression and decompression with lzip. publish: Add support for lzip. self: Add dependency on lzlib. gnu: guix: Add dependency on lzlib. lzlib: 'lzread!' never returns more than it was asked for. .dir-locals.el | 2 + doc/guix.texi | 25 +++++-- gnu/packages/package-management.scm | 1 + guix/lzlib.scm | 101 ++++++++++++++++++++++------ guix/scripts/publish.scm | 84 +++++++++++++++++------ guix/self.scm | 13 +++- guix/tests.scm | 1 + guix/utils.scm | 27 ++++++-- tests/lzlib.scm | 10 +++ tests/publish.scm | 36 ++++++++++ tests/utils.scm | 62 +++++++++++------ 11 files changed, 284 insertions(+), 78 deletions(-) -- 2.21.0 ------------=_1558990444-13130-1-- From unknown Sat Jun 14 03:48:44 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#35880] [PATCH 1/7] lzlib: Add 'make-lzip-input-port/compressed'. Resent-From: Pierre Neidhardt Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Mon, 27 May 2019 21:13:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 35880 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Ludovic =?UTF-8?Q?Court=C3=A8s?= Cc: 35880-done@debbugs.gnu.org Received: via spool by 35880-done@debbugs.gnu.org id=D35880.155899155214924 (code D ref 35880); Mon, 27 May 2019 21:13:02 +0000 Received: (at 35880-done) by debbugs.gnu.org; 27 May 2019 21:12:32 +0000 Received: from localhost ([127.0.0.1]:54328 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hVMv2-0003se-7k for submit@debbugs.gnu.org; Mon, 27 May 2019 17:12:32 -0400 Received: from relay11.mail.gandi.net ([217.70.178.231]:41109) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hVMv0-0003sT-EC for 35880-done@debbugs.gnu.org; Mon, 27 May 2019 17:12:31 -0400 Received: from bababa (lfbn-1-4117-19.w92-169.abo.wanadoo.fr [92.169.116.19]) (Authenticated sender: pierre@atlas.engineer) by relay11.mail.gandi.net (Postfix) with ESMTPSA id A865B100004; Mon, 27 May 2019 21:12:24 +0000 (UTC) From: Pierre Neidhardt In-Reply-To: <87lfyrr554.fsf@gnu.org> References: <20190524134238.22802-1-ludo@gnu.org> <87d0k6o3am.fsf@ambrevar.xyz> <87ftp1m1te.fsf@gnu.org> <87blznsxym.fsf@gnu.org> <87ef4jlvbr.fsf@ambrevar.xyz> <87lfyrr554.fsf@gnu.org> Date: Mon, 27 May 2019 23:12:24 +0200 Message-ID: <87sgszk3ev.fsf@ambrevar.xyz> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" X-Spam-Score: -0.2 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.2 (/) --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Ludovic Court=C3=A8s writes: >> That said, if the encoder buffer is not empty, I think lz-compress-read >> should always return something >0. > > Yes, probably. The docstring for =E2=80=98lz-compress-read=E2=80=99 says: Oops, I read the docstring of lz-DEcompress-read. My bad. > "Read up to COUNT bytes from the encoder stream, storing the results = in LZFILE-BV. > Return the number of uncompressed bytes written, a strictly positive in= teger." > ^~~~~~~~~~~~~~~~~ Bigger oops! This comes from a copy-paste of the gzip docstring which I forgot to update properly (I did for the decompression functions, but not for the compression functions). The docstrings should be fixed. > But that=E2=80=99s OK: the =E2=80=98read!=E2=80=99 method in =E2=80=98mak= e-lzip-input-port/compressed=E2=80=99 > can just call =E2=80=98lzwrite!=E2=80=99 again with more data when that h= appens, so I=E2=80=99ve > done that. This could work, but I've had some headaches on such assumptions before. Tests are very necessary here to validate those assumptions ;) The thing is that we are not using lzlib as it is meant to be used (i.e. with the finish* functions) because of the functional approach of the binary ports which don't really play well with the procedural approach of the C library. > And I pushed the whole thing! :-) Hurray! Can't wait to say lz-compressed archives coming to Guix! :) =2D-=20 Pierre Neidhardt https://ambrevar.xyz/ --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQEzBAEBCAAdFiEEUPM+LlsMPZAEJKvom9z0l6S7zH8FAlzsUrgACgkQm9z0l6S7 zH9BpQgAgeO5F0FkIbtYDUQmkBAvQmrZL0M3Y+pQ8gQggvmXYodUTJWr85e6OVF9 F3IqTQ+H022vWuiFmrFklNRSCORy6KWAnNNW4uJpOQ61mPMGZVbrznlBEz+eVts+ tgmYJYmiAzh91o/O+qSev6efN1Yxe5KAWbV1idsuwiTTshQjzCW+vVeUaVabjGqV 5/I9Bfd2jJX+QPid2knvsoSNoq8AvhIPvAy/OeJ5ezV7S0ZuSHmY/DAUiLBzPvmW 0qACL/4OK9d/BhE49UlVZl5+BoXpIsCMcajtVsnVh2nWZjXjrfC06w+a9wxuvPlS mjVk5vhisUrE7+zpPHNASy9iTNz5wQ== =La/B -----END PGP SIGNATURE----- --=-=-=-- From unknown Sat Jun 14 03:48:44 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#35880] [PATCH 1/7] lzlib: Add 'make-lzip-input-port/compressed'. Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Tue, 28 May 2019 07:53:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 35880 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Pierre Neidhardt Cc: 35880-done@debbugs.gnu.org Received: via spool by 35880-done@debbugs.gnu.org id=D35880.15590299438736 (code D ref 35880); Tue, 28 May 2019 07:53:01 +0000 Received: (at 35880-done) by debbugs.gnu.org; 28 May 2019 07:52:23 +0000 Received: from localhost ([127.0.0.1]:54964 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hVWuE-0002Gp-KJ for submit@debbugs.gnu.org; Tue, 28 May 2019 03:52:22 -0400 Received: from eggs.gnu.org ([209.51.188.92]:33444) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hVWuC-0002GX-Ey for 35880-done@debbugs.gnu.org; Tue, 28 May 2019 03:52:20 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:54479) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hVWu6-0008Fx-LH; Tue, 28 May 2019 03:52:14 -0400 Received: from [2001:660:6102:320:e120:2c8f:8909:cdfe] (port=36326 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1hVWu5-0006g5-E6; Tue, 28 May 2019 03:52:14 -0400 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= References: <20190524134238.22802-1-ludo@gnu.org> <87d0k6o3am.fsf@ambrevar.xyz> <87ftp1m1te.fsf@gnu.org> <87blznsxym.fsf@gnu.org> <87ef4jlvbr.fsf@ambrevar.xyz> <87lfyrr554.fsf@gnu.org> <87sgszk3ev.fsf@ambrevar.xyz> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 9 Prairial an 227 de la =?UTF-8?Q?R=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Tue, 28 May 2019 09:52:09 +0200 In-Reply-To: <87sgszk3ev.fsf@ambrevar.xyz> (Pierre Neidhardt's message of "Mon, 27 May 2019 23:12:24 +0200") Message-ID: <877eab3tjq.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Spam-Score: -2.3 (--) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Pierre Neidhardt skribis: > Ludovic Court=C3=A8s writes: > >>> That said, if the encoder buffer is not empty, I think lz-compress-read >>> should always return something >0. >> >> Yes, probably. The docstring for =E2=80=98lz-compress-read=E2=80=99 say= s: > > Oops, I read the docstring of lz-DEcompress-read. My bad. > >> "Read up to COUNT bytes from the encoder stream, storing the results= in LZFILE-BV. >> Return the number of uncompressed bytes written, a strictly positive i= nteger." >> ^~~~~~~~~~~~~~~~~ > > Bigger oops! This comes from a copy-paste of the gzip docstring which I > forgot to update properly (I did for the decompression functions, but > not for the compression functions). The docstrings should be fixed. I fixed this one in e13354a7ca5a0d5e28e02c4cfce6fecb1ab770e4. >> But that=E2=80=99s OK: the =E2=80=98read!=E2=80=99 method in =E2=80=98ma= ke-lzip-input-port/compressed=E2=80=99 >> can just call =E2=80=98lzwrite!=E2=80=99 again with more data when that = happens, so I=E2=80=99ve >> done that. > > This could work, but I've had some headaches on such assumptions > before. Tests are very necessary here to validate those assumptions ;) Definitely! > The thing is that we are not using lzlib as it is meant to be used > (i.e. with the finish* functions) because of the functional approach of > the binary ports which don't really play well with the procedural > approach of the C library. I think we=E2=80=99re using it the way it=E2=80=99s meant to be used, rough= ly along the lines of the examples of its manual (info "(lzlib) Examples"). (I/O ports are not very =E2=80=9Cfunctional=E2=80=9D.) >> And I pushed the whole thing! :-) > > Hurray! Can't wait to say lz-compressed archives coming to Guix! :) I=E2=80=99ve updated the =E2=80=98guix=E2=80=99 package so people can start= using =E2=80=98guix publish -C lzip=E2=80=99 and fetch substitute from there. Thanks for making it possible! Ludo=E2=80=99. From unknown Sat Jun 14 03:48:44 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#35880] [PATCH 1/7] lzlib: Add 'make-lzip-input-port/compressed'. Resent-From: Pierre Neidhardt Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Tue, 28 May 2019 08:47:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 35880 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Ludovic =?UTF-8?Q?Court=C3=A8s?= Cc: 35880-done@debbugs.gnu.org Received: via spool by 35880-done@debbugs.gnu.org id=D35880.155903319514642 (code D ref 35880); Tue, 28 May 2019 08:47:02 +0000 Received: (at 35880-done) by debbugs.gnu.org; 28 May 2019 08:46:35 +0000 Received: from localhost ([127.0.0.1]:55030 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hVXkg-0003o6-S8 for submit@debbugs.gnu.org; Tue, 28 May 2019 04:46:35 -0400 Received: from relay5-d.mail.gandi.net ([217.70.183.197]:57179) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hVXkd-0003np-5O for 35880-done@debbugs.gnu.org; Tue, 28 May 2019 04:46:32 -0400 X-Originating-IP: 92.169.116.19 Received: from bababa (lfbn-1-4117-19.w92-169.abo.wanadoo.fr [92.169.116.19]) (Authenticated sender: pierre@atlas.engineer) by relay5-d.mail.gandi.net (Postfix) with ESMTPSA id D30271C0004; Tue, 28 May 2019 08:46:24 +0000 (UTC) From: Pierre Neidhardt In-Reply-To: <877eab3tjq.fsf@gnu.org> References: <20190524134238.22802-1-ludo@gnu.org> <87d0k6o3am.fsf@ambrevar.xyz> <87ftp1m1te.fsf@gnu.org> <87blznsxym.fsf@gnu.org> <87ef4jlvbr.fsf@ambrevar.xyz> <87lfyrr554.fsf@gnu.org> <87sgszk3ev.fsf@ambrevar.xyz> <877eab3tjq.fsf@gnu.org> Date: Tue, 28 May 2019 10:46:23 +0200 Message-ID: <87k1ebj7a8.fsf@ambrevar.xyz> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" X-Spam-Score: -0.2 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.2 (/) --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Ludovic Court=C3=A8s writes: >> Bigger oops! This comes from a copy-paste of the gzip docstring which I >> forgot to update properly (I did for the decompression functions, but >> not for the compression functions). The docstrings should be fixed. > > I fixed this one in e13354a7ca5a0d5e28e02c4cfce6fecb1ab770e4. Thanks! >> The thing is that we are not using lzlib as it is meant to be used >> (i.e. with the finish* functions) because of the functional approach of >> the binary ports which don't really play well with the procedural >> approach of the C library. > > I think we=E2=80=99re using it the way it=E2=80=99s meant to be used, rou= ghly along the > lines of the examples of its manual (info "(lzlib) Examples"). > > (I/O ports are not very =E2=80=9Cfunctional=E2=80=9D.) In the sense that we define "read!" and "write!" functions which don't allow us to call the "finish" functions properly. So maybe we are following the doc too "roughly" :p This has multiple implications, e.g. it impedes support for multiple members, (which is OK as long as we don't accept archives produced by a third-party) and more importantly it lifts the guarantee that the library is going to work as per the documentation. >>> And I pushed the whole thing! :-) >> >> Hurray! Can't wait to say lz-compressed archives coming to Guix! :) > > I=E2=80=99ve updated the =E2=80=98guix=E2=80=99 package so people can sta= rt using > =E2=80=98guix publish -C lzip=E2=80=99 and fetch substitute from there. > > Thanks for making it possible! And thanks for doing most of the work! :D =2D-=20 Pierre Neidhardt https://ambrevar.xyz/ --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQEzBAEBCAAdFiEEUPM+LlsMPZAEJKvom9z0l6S7zH8FAlzs9V8ACgkQm9z0l6S7 zH9g/wf/W8poG5k/FJeEiNVfNW/FLFHiYB+NB5gv3IE0ElpgOe3Y/GyX0J4J5D5E UjHqng7N+uy1eTS6BU2PEE4G4w8JQdXGC4pcHkZzdT+rfT8l/GQfI79fcyfRQE/d 491T8CN3AzWnJz4a/WcZct58RIsML4WRyaL52gnmKJzFfCc6RJd5wb1rqBO5LUjK rGgBdPsbnqpImrJlINDKxPu9mSfwUEI5wLiDB7uVyywsZTlT7nhb0dOCkXbdgqWZ +7oxKlIsYawKjdCfZKE6cb60baYeOCjBvVWtMxuhVKTU1dFVKJT26tIc0tOGcSKL ga2s26/cBDZIx6xxLDjzhoLVfe1tvQ== =U6r2 -----END PGP SIGNATURE----- --=-=-=-- From unknown Sat Jun 14 03:48:44 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#35880] [PATCH 1/7] lzlib: Add 'make-lzip-input-port/compressed'. Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Tue, 28 May 2019 13:48:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 35880 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Pierre Neidhardt Cc: 35880-done@debbugs.gnu.org Received: via spool by 35880-done@debbugs.gnu.org id=D35880.155905124723157 (code D ref 35880); Tue, 28 May 2019 13:48:01 +0000 Received: (at 35880-done) by debbugs.gnu.org; 28 May 2019 13:47:27 +0000 Received: from localhost ([127.0.0.1]:55323 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hVcRr-00061R-B8 for submit@debbugs.gnu.org; Tue, 28 May 2019 09:47:27 -0400 Received: from eggs.gnu.org ([209.51.188.92]:53113) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hVcRo-000619-G4 for 35880-done@debbugs.gnu.org; Tue, 28 May 2019 09:47:25 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:59455) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hVcRh-0005Ju-Vg; Tue, 28 May 2019 09:47:17 -0400 Received: from [2001:660:6102:320:e120:2c8f:8909:cdfe] (port=38164 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1hVcRh-0002BU-G6; Tue, 28 May 2019 09:47:17 -0400 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= References: <20190524134238.22802-1-ludo@gnu.org> <87d0k6o3am.fsf@ambrevar.xyz> <87ftp1m1te.fsf@gnu.org> <87blznsxym.fsf@gnu.org> <87ef4jlvbr.fsf@ambrevar.xyz> <87lfyrr554.fsf@gnu.org> <87sgszk3ev.fsf@ambrevar.xyz> <877eab3tjq.fsf@gnu.org> <87k1ebj7a8.fsf@ambrevar.xyz> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 9 Prairial an 227 de la =?UTF-8?Q?R=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Tue, 28 May 2019 15:47:13 +0200 In-Reply-To: <87k1ebj7a8.fsf@ambrevar.xyz> (Pierre Neidhardt's message of "Tue, 28 May 2019 10:46:23 +0200") Message-ID: <871s0i1yji.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Spam-Score: -2.3 (--) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Pierre Neidhardt skribis: > Ludovic Court=C3=A8s writes: [...] >>> The thing is that we are not using lzlib as it is meant to be used >>> (i.e. with the finish* functions) because of the functional approach of >>> the binary ports which don't really play well with the procedural >>> approach of the C library. >> >> I think we=E2=80=99re using it the way it=E2=80=99s meant to be used, ro= ughly along the >> lines of the examples of its manual (info "(lzlib) Examples"). >> >> (I/O ports are not very =E2=80=9Cfunctional=E2=80=9D.) > > In the sense that we define "read!" and "write!" functions which don't > allow us to call the "finish" functions properly. > > So maybe we are following the doc too "roughly" :p > > This has multiple implications, e.g. it impedes support for multiple > members, (which is OK as long as we don't accept archives produced by a > third-party) and more importantly it lifts the guarantee that the > library is going to work as per the documentation. I=E2=80=99m not sure I follow. I think =E2=80=98make-lzip-input-port/compr= essed=E2=80=99 corresponds to Example 2 in the manual (info "(lzlib) Examples"), =E2=80=98make-lzip-output-port=E2=80=99 corresponds to Example 1, and =E2=80=98make-lzip-input-port=E2=80=99 corresponds to Example 4 (with the e= xception that =E2=80=98lzread!=E2=80=99 doesn=E2=80=99t call =E2=80=98lz-decompress-finis= hed?=E2=80=99, but it has other means to tell whether we=E2=80=99re done processing input.) What do you think is not done as documented? Thanks, Ludo=E2=80=99. From unknown Sat Jun 14 03:48:44 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#35880] [PATCH 1/7] lzlib: Add 'make-lzip-input-port/compressed'. Resent-From: Pierre Neidhardt Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Wed, 29 May 2019 14:58:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 35880 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Ludovic =?UTF-8?Q?Court=C3=A8s?= Cc: 35880-done@debbugs.gnu.org Received: via spool by 35880-done@debbugs.gnu.org id=D35880.155914185228575 (code D ref 35880); Wed, 29 May 2019 14:58:01 +0000 Received: (at 35880-done) by debbugs.gnu.org; 29 May 2019 14:57:32 +0000 Received: from localhost ([127.0.0.1]:59556 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hW01D-0007Qp-ML for submit@debbugs.gnu.org; Wed, 29 May 2019 10:57:32 -0400 Received: from relay5-d.mail.gandi.net ([217.70.183.197]:59477) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hW01B-0007Qf-Na for 35880-done@debbugs.gnu.org; Wed, 29 May 2019 10:57:30 -0400 X-Originating-IP: 92.169.116.19 Received: from bababa (lfbn-1-4117-19.w92-169.abo.wanadoo.fr [92.169.116.19]) (Authenticated sender: pierre@atlas.engineer) by relay5-d.mail.gandi.net (Postfix) with ESMTPSA id 5E2191C0011; Wed, 29 May 2019 14:57:19 +0000 (UTC) From: Pierre Neidhardt In-Reply-To: <871s0i1yji.fsf@gnu.org> References: <20190524134238.22802-1-ludo@gnu.org> <87d0k6o3am.fsf@ambrevar.xyz> <87ftp1m1te.fsf@gnu.org> <87blznsxym.fsf@gnu.org> <87ef4jlvbr.fsf@ambrevar.xyz> <87lfyrr554.fsf@gnu.org> <87sgszk3ev.fsf@ambrevar.xyz> <877eab3tjq.fsf@gnu.org> <87k1ebj7a8.fsf@ambrevar.xyz> <871s0i1yji.fsf@gnu.org> Date: Wed, 29 May 2019 16:57:19 +0200 Message-ID: <87a7f5wbow.fsf@ambrevar.xyz> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" X-Spam-Score: -0.2 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.2 (/) --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Ludovic Court=C3=A8s writes: > I=E2=80=99m not sure I follow. I think =E2=80=98make-lzip-input-port/com= pressed=E2=80=99 > corresponds to Example 2 in the manual (info "(lzlib) Examples"), > =E2=80=98make-lzip-output-port=E2=80=99 corresponds to Example 1, and > =E2=80=98make-lzip-input-port=E2=80=99 corresponds to Example 4 (with the= exception that > =E2=80=98lzread!=E2=80=99 doesn=E2=80=99t call =E2=80=98lz-decompress-fin= ished?=E2=80=99, but it has other means > to tell whether we=E2=80=99re done processing input.) Example 4 is: 1) LZ_decompress_open 2) go to step 5 if LZ_decompress_write_size returns 0 3) LZ_decompress_write 4) if no more data to write, call LZ_decompress_finish 5) LZ_decompress_read 5a) optionally, if LZ_decompress_member_finished returns 1, read final values for member with LZ_decompress_data_crc, etc. 6) go back to step 2 until LZ_decompress_finished returns 1 7) LZ_decompress_close In `lzread!', we don't call lz-decompress-finished? nor do we loop on lz-decompress-finished. This only works for decompression of single-member archive, but the documentation does not say that. =2D-8<---------------cut here---------------start------------->8--- (match (get-bytevector-n port (lz-decompress-write-size decoder)) ((? eof-object? eof) eof) (bv (lz-decompress-write decoder bv))) =2D-8<---------------cut here---------------end--------------->8--- In the above if lz-decompress-write-size returns 0, we won't be reading anything (infinite loop?). While I understand this should not happen in practice, the documentation of the library does not give such guarantees. Antonio told me that explicitly. =2D-8<---------------cut here---------------start------------->8--- (match (lz-decompress-read decoder bv start (- count read)) (0 (if (eof-object? (feed-decoder! decoder)) read (loop read start))) =2D-8<---------------cut here---------------end--------------->8--- I'm not sure I understand the above: if we read nothing, then we try again? This might loop forever. What do you think? =2D-=20 Pierre Neidhardt https://ambrevar.xyz/ --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQEzBAEBCAAdFiEEUPM+LlsMPZAEJKvom9z0l6S7zH8FAlzunc8ACgkQm9z0l6S7 zH/cAQf7BzqS51ztbb8c8tRkM6zYlI2t5t3oYsv/FAjD6vb3k/CimQhHOTPD8d3s 3PAybM2rimZTkp4xSh06+L66uRh+t0Vw+UnPJonct9irhq8lFOFiZwDbwXYqEgia I+OpHwNETZSO66ALflO4E5cbFmKdeO1IWJALisOUzHMgmnKRCZUo3qriQFb0IZ4u yYAy4rSt4aBj/EE7GaZdFwLQUM0Ya2i3Hemq+ZW8TtmaD6J09w6ffV+o/l0mjGIK u3KscXc7aVGC9BPxec34qAyvvFh8lPJDpD9ffKXyjqDFQj1NOVzM/WmRHDbAeELt wufjlFt6c+2BnJ1Z7Vw7/tM+FC/JCQ== =LbIL -----END PGP SIGNATURE----- --=-=-=-- From unknown Sat Jun 14 03:48:44 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#35880] [PATCH 1/7] lzlib: Add 'make-lzip-input-port/compressed'. Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Fri, 31 May 2019 20:55:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 35880 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Pierre Neidhardt Cc: 35880-done@debbugs.gnu.org Received: via spool by 35880-done@debbugs.gnu.org id=D35880.155933606016326 (code D ref 35880); Fri, 31 May 2019 20:55:01 +0000 Received: (at 35880-done) by debbugs.gnu.org; 31 May 2019 20:54:20 +0000 Received: from localhost ([127.0.0.1]:36406 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hWoXc-0004FG-3J for submit@debbugs.gnu.org; Fri, 31 May 2019 16:54:20 -0400 Received: from eggs.gnu.org ([209.51.188.92]:43467) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hWoXZ-0004Ex-W8 for 35880-done@debbugs.gnu.org; Fri, 31 May 2019 16:54:18 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:53081) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hWoXT-0004WE-PX; Fri, 31 May 2019 16:54:11 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=53250 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1hWoXT-0000Xv-8w; Fri, 31 May 2019 16:54:11 -0400 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= References: <20190524134238.22802-1-ludo@gnu.org> <87d0k6o3am.fsf@ambrevar.xyz> <87ftp1m1te.fsf@gnu.org> <87blznsxym.fsf@gnu.org> <87ef4jlvbr.fsf@ambrevar.xyz> <87lfyrr554.fsf@gnu.org> <87sgszk3ev.fsf@ambrevar.xyz> <877eab3tjq.fsf@gnu.org> <87k1ebj7a8.fsf@ambrevar.xyz> <871s0i1yji.fsf@gnu.org> <87a7f5wbow.fsf@ambrevar.xyz> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 12 Prairial an 227 de la =?UTF-8?Q?R=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Fri, 31 May 2019 22:54:09 +0200 In-Reply-To: <87a7f5wbow.fsf@ambrevar.xyz> (Pierre Neidhardt's message of "Wed, 29 May 2019 16:57:19 +0200") Message-ID: <87v9xqny4u.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Spam-Score: -2.3 (--) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hello, Pierre Neidhardt skribis: > Ludovic Court=C3=A8s writes: > >> I=E2=80=99m not sure I follow. I think =E2=80=98make-lzip-input-port/co= mpressed=E2=80=99 >> corresponds to Example 2 in the manual (info "(lzlib) Examples"), >> =E2=80=98make-lzip-output-port=E2=80=99 corresponds to Example 1, and >> =E2=80=98make-lzip-input-port=E2=80=99 corresponds to Example 4 (with th= e exception that >> =E2=80=98lzread!=E2=80=99 doesn=E2=80=99t call =E2=80=98lz-decompress-fi= nished?=E2=80=99, but it has other means >> to tell whether we=E2=80=99re done processing input.) > > Example 4 is: > > 1) LZ_decompress_open > 2) go to step 5 if LZ_decompress_write_size returns 0 > 3) LZ_decompress_write > 4) if no more data to write, call LZ_decompress_finish > 5) LZ_decompress_read > 5a) optionally, if LZ_decompress_member_finished returns 1, read > final values for member with LZ_decompress_data_crc, etc. > 6) go back to step 2 until LZ_decompress_finished returns 1 > 7) LZ_decompress_close > > In `lzread!', we don't call lz-decompress-finished? nor do we loop on > lz-decompress-finished. Indeed, we=E2=80=99re missing a call to =E2=80=98lz-decompress-finish=E2=80= =99, so lzlib could in theory think there=E2=80=99s still data coming, and so fail to produce m= ore output, possibly leading to an infinite loop. I think the =E2=80=98lzread!=E2=80=99 loop should look like this (the tests= still pass with this): --8<---------------cut here---------------start------------->8--- (let loop ((read 0) (start start)) (cond ((< read count) (match (lz-decompress-read decoder bv start (- count read)) (0 (cond ((lz-decompress-finished? decoder) read) ((eof-object? (feed-decoder! decoder)) (lz-decompress-finish decoder) (loop read start)) (else ;read again (loop read start)))) (n (loop (+ read n) (+ start n))))) (else read))) --8<---------------cut here---------------end--------------->8--- That way, I believe all cases are correctly handled. WDYT? > This only works for decompression of single-member archive, but the > documentation does not say that. > > (match (get-bytevector-n port (lz-decompress-write-size decoder)) > ((? eof-object? eof) eof) > (bv (lz-decompress-write decoder bv))) > > > In the above if lz-decompress-write-size returns 0, we won't be reading > anything (infinite loop?). We=E2=80=99re calling =E2=80=98feed-decoder!=E2=80=99 iff =E2=80=98lz-decom= press-read=E2=80=99 returned 0; when that happens =E2=80=98lz-decompress-write-size=E2=80=99 must return a stric= tly positive number. Otherwise there=E2=80=99s an inconsistency. > (match (lz-decompress-read decoder bv start (- count read)) > (0 (if (eof-object? (feed-decoder! decoder)) > read > (loop read start))) > > I'm not sure I understand the above: if we read nothing, then we try > again? No: if we read *something*, we try again; if we read nothing, we return. Thanks for your careful review, much appreciated! Ludo=E2=80=99. From unknown Sat Jun 14 03:48:44 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#35880] [PATCH 1/7] lzlib: Add 'make-lzip-input-port/compressed'. Resent-From: Pierre Neidhardt Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sat, 01 Jun 2019 06:03:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 35880 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Ludovic =?UTF-8?Q?Court=C3=A8s?= Cc: 35880-done@debbugs.gnu.org Received: via spool by 35880-done@debbugs.gnu.org id=D35880.155936896422432 (code D ref 35880); Sat, 01 Jun 2019 06:03:02 +0000 Received: (at 35880-done) by debbugs.gnu.org; 1 Jun 2019 06:02:44 +0000 Received: from localhost ([127.0.0.1]:36829 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hWx6J-0005pi-Vz for submit@debbugs.gnu.org; Sat, 01 Jun 2019 02:02:44 -0400 Received: from relay7-d.mail.gandi.net ([217.70.183.200]:60043) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hWx6H-0005pY-FU for 35880-done@debbugs.gnu.org; Sat, 01 Jun 2019 02:02:42 -0400 X-Originating-IP: 92.169.116.19 Received: from bababa (lfbn-1-4117-19.w92-169.abo.wanadoo.fr [92.169.116.19]) (Authenticated sender: pierre@atlas.engineer) by relay7-d.mail.gandi.net (Postfix) with ESMTPSA id ECBBF20003; Sat, 1 Jun 2019 06:02:32 +0000 (UTC) From: Pierre Neidhardt In-Reply-To: <87v9xqny4u.fsf@gnu.org> References: <20190524134238.22802-1-ludo@gnu.org> <87d0k6o3am.fsf@ambrevar.xyz> <87ftp1m1te.fsf@gnu.org> <87blznsxym.fsf@gnu.org> <87ef4jlvbr.fsf@ambrevar.xyz> <87lfyrr554.fsf@gnu.org> <87sgszk3ev.fsf@ambrevar.xyz> <877eab3tjq.fsf@gnu.org> <87k1ebj7a8.fsf@ambrevar.xyz> <871s0i1yji.fsf@gnu.org> <87a7f5wbow.fsf@ambrevar.xyz> <87v9xqny4u.fsf@gnu.org> Date: Sat, 01 Jun 2019 08:02:29 +0200 Message-ID: <87ef4dzvuy.fsf@ambrevar.xyz> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" X-Spam-Score: -0.2 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.2 (/) --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Ludovic Court=C3=A8s writes: > I think the =E2=80=98lzread!=E2=80=99 loop should look like this (the tes= ts still pass > with this): > > --8<---------------cut here---------------start------------->8--- > (let loop ((read 0) > (start start)) > (cond ((< read count) > (match (lz-decompress-read decoder bv start (- count read)) > (0 (cond ((lz-decompress-finished? decoder) > read) > ((eof-object? (feed-decoder! decoder)) > (lz-decompress-finish decoder) > (loop read start)) > (else ;read again > (loop read start)))) > (n (loop (+ read n) (+ start n))))) > (else > read))) > --8<---------------cut here---------------end--------------->8--- Looks good to me! >> (match (lz-decompress-read decoder bv start (- count read)) >> (0 (if (eof-object? (feed-decoder! decoder)) >> read >> (loop read start))) >> >> I'm not sure I understand the above: if we read nothing, then we try >> again? > > No: if we read *something*, we try again; if we read nothing, we return. If we read nothing _and_ it is not an EOF (it can be an empty vector), then we loop indefinitely, no? > Thanks for your careful review, much appreciated! You are welcome, thanks for your invaluable work! =2D-=20 Pierre Neidhardt https://ambrevar.xyz/ --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQEzBAEBCAAdFiEEUPM+LlsMPZAEJKvom9z0l6S7zH8FAlzyFPUACgkQm9z0l6S7 zH+nnAgAmoDB/EnDyp3MzWTeDa/rao+g9dFIb8cgEP1vEw6S7Gnc7UgBu29PLGku F4/h2PvuxcWNorl/6eZhqUEeKWgoaznkr0V4mfO/UBWB56r4VuUlc86UvbQrtEeL yALwEfE2z4abJdlNoUnT1Ojl+qBHJIWTzFgVmdp8LVzG8C7SSgsH/ZJ7GEu1NBGQ jxaJ9LsO+7JTJs/8SP4qgwrV5LfLF6rl6TjhMGf5yt6dmcGGVpaHKLItsPDxT+iP +ABTZXwG4z+ZN+Df4mzSuORt38CMNapAG3D30PE9dvPgiszW3985g9hKlT17Gpg1 uayGoEN2t6XBOPqZ/5cKUMWJBVrs0A== =Bu/7 -----END PGP SIGNATURE----- --=-=-=-- From unknown Sat Jun 14 03:48:44 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#35880] [PATCH 1/7] lzlib: Add 'make-lzip-input-port/compressed'. Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sat, 01 Jun 2019 09:43:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 35880 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Pierre Neidhardt Cc: 35880-done@debbugs.gnu.org Received: via spool by 35880-done@debbugs.gnu.org id=D35880.155938213218312 (code D ref 35880); Sat, 01 Jun 2019 09:43:02 +0000 Received: (at 35880-done) by debbugs.gnu.org; 1 Jun 2019 09:42:12 +0000 Received: from localhost ([127.0.0.1]:37102 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hX0Wh-0004lI-O2 for submit@debbugs.gnu.org; Sat, 01 Jun 2019 05:42:11 -0400 Received: from eggs.gnu.org ([209.51.188.92]:60283) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hX0Wf-0004l6-Uj for 35880-done@debbugs.gnu.org; Sat, 01 Jun 2019 05:42:10 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:36014) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hX0WZ-0004GG-LW; Sat, 01 Jun 2019 05:42:03 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=54368 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1hX0WZ-0008Q8-2g; Sat, 01 Jun 2019 05:42:03 -0400 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= References: <20190524134238.22802-1-ludo@gnu.org> <87d0k6o3am.fsf@ambrevar.xyz> <87ftp1m1te.fsf@gnu.org> <87blznsxym.fsf@gnu.org> <87ef4jlvbr.fsf@ambrevar.xyz> <87lfyrr554.fsf@gnu.org> <87sgszk3ev.fsf@ambrevar.xyz> <877eab3tjq.fsf@gnu.org> <87k1ebj7a8.fsf@ambrevar.xyz> <871s0i1yji.fsf@gnu.org> <87a7f5wbow.fsf@ambrevar.xyz> <87v9xqny4u.fsf@gnu.org> <87ef4dzvuy.fsf@ambrevar.xyz> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 13 Prairial an 227 de la =?UTF-8?Q?R=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Sat, 01 Jun 2019 11:41:59 +0200 In-Reply-To: <87ef4dzvuy.fsf@ambrevar.xyz> (Pierre Neidhardt's message of "Sat, 01 Jun 2019 08:02:29 +0200") Message-ID: <87ef4dk5g8.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Spam-Score: -2.3 (--) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hi! Pierre Neidhardt skribis: > Ludovic Court=C3=A8s writes: > >> I think the =E2=80=98lzread!=E2=80=99 loop should look like this (the te= sts still pass >> with this): >> >> --8<---------------cut here---------------start------------->8--- >> (let loop ((read 0) >> (start start)) >> (cond ((< read count) >> (match (lz-decompress-read decoder bv start (- count read)) >> (0 (cond ((lz-decompress-finished? decoder) >> read) >> ((eof-object? (feed-decoder! decoder)) >> (lz-decompress-finish decoder) >> (loop read start)) >> (else ;read again >> (loop read start)))) >> (n (loop (+ read n) (+ start n))))) >> (else >> read))) >> --8<---------------cut here---------------end--------------->8--- > > Looks good to me! OK, committed! >>> (match (lz-decompress-read decoder bv start (- count read)) >>> (0 (if (eof-object? (feed-decoder! decoder)) >>> read >>> (loop read start))) >>> >>> I'm not sure I understand the above: if we read nothing, then we try >>> again? >> >> No: if we read *something*, we try again; if we read nothing, we return. > > If we read nothing _and_ it is not an EOF (it can be an empty vector), > then we loop indefinitely, no? =E2=80=98feed-decoder!=E2=80=99 cannot return an empty bytevector because =E2=80=98lz-decompress-write-size=E2=80=99 necessarily returns a strictly p= ositive integer at this point. (Imperative programming is hard! :-)) Thanks, Ludo=E2=80=99. From unknown Sat Jun 14 03:48:44 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#35880] [PATCH 1/7] lzlib: Add 'make-lzip-input-port/compressed'. Resent-From: Pierre Neidhardt Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sat, 01 Jun 2019 09:59:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 35880 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Ludovic =?UTF-8?Q?Court=C3=A8s?= Cc: 35880-done@debbugs.gnu.org Received: via spool by 35880-done@debbugs.gnu.org id=D35880.155938311019804 (code D ref 35880); Sat, 01 Jun 2019 09:59:02 +0000 Received: (at 35880-done) by debbugs.gnu.org; 1 Jun 2019 09:58:30 +0000 Received: from localhost ([127.0.0.1]:37123 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hX0mU-00059M-Hz for submit@debbugs.gnu.org; Sat, 01 Jun 2019 05:58:30 -0400 Received: from relay5-d.mail.gandi.net ([217.70.183.197]:40805) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hX0mS-00059C-JS for 35880-done@debbugs.gnu.org; Sat, 01 Jun 2019 05:58:29 -0400 X-Originating-IP: 92.169.116.19 Received: from bababa (lfbn-1-4117-19.w92-169.abo.wanadoo.fr [92.169.116.19]) (Authenticated sender: pierre@atlas.engineer) by relay5-d.mail.gandi.net (Postfix) with ESMTPSA id 4914A1C000C; Sat, 1 Jun 2019 09:58:24 +0000 (UTC) From: Pierre Neidhardt In-Reply-To: <87ef4dk5g8.fsf@gnu.org> References: <20190524134238.22802-1-ludo@gnu.org> <87d0k6o3am.fsf@ambrevar.xyz> <87ftp1m1te.fsf@gnu.org> <87blznsxym.fsf@gnu.org> <87ef4jlvbr.fsf@ambrevar.xyz> <87lfyrr554.fsf@gnu.org> <87sgszk3ev.fsf@ambrevar.xyz> <877eab3tjq.fsf@gnu.org> <87k1ebj7a8.fsf@ambrevar.xyz> <871s0i1yji.fsf@gnu.org> <87a7f5wbow.fsf@ambrevar.xyz> <87v9xqny4u.fsf@gnu.org> <87ef4dzvuy.fsf@ambrevar.xyz> <87ef4dk5g8.fsf@gnu.org> Date: Sat, 01 Jun 2019 11:58:23 +0200 Message-ID: <8736ktzkxs.fsf@ambrevar.xyz> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" X-Spam-Score: -0.2 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.2 (/) --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Ludovic Court=C3=A8s writes: > =E2=80=98feed-decoder!=E2=80=99 cannot return an empty bytevector because > =E2=80=98lz-decompress-write-size=E2=80=99 necessarily returns a strictly= positive > integer at this point. I'm not sure that's true: if the buffer is full and the next lz-decompress-read does not read anything, then the buffer will still be full and lz-decompress-write-size will return 0. The specs don't guarantee that lz-decompress-read will always read something. But that's the only assumption we are making I believe, and it's fair :) =2D-=20 Pierre Neidhardt https://ambrevar.xyz/ --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQEzBAEBCAAdFiEEUPM+LlsMPZAEJKvom9z0l6S7zH8FAlzyTD8ACgkQm9z0l6S7 zH/W+gf+McCVLjkNT3Eb6wBG6GQEdkUfMrxGBL5F7aX4wW8npk+KcuS05g1V4xIS pMnHcUYcC6TnofKMr2k1udK7xYyV+3AScN7NfopPHjFlxUzdrSR5GtcDgoIqb9gP bKoKdBMklfSqUk160C5976qE8x3NrOyc5t7K/LPLWrbKMmjuI4P0Qu6zybQH9kyY c4A8lKE79HnPbdarPdKikCqETDq+iYrUgCOvDLRCB+p2uQ2s8Y4fADoSxfBxcnrC 35Do7kK19MJbRP3YW8d/Q0TLky2XN0dqIu2quoSaGUW17Af/NEfer53SUxy67CzP QfHw9jpc06a5T6ZMT4r0h1NK6KILZg== =iidi -----END PGP SIGNATURE----- --=-=-=-- From unknown Sat Jun 14 03:48:44 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#35880] [PATCH 1/7] lzlib: Add 'make-lzip-input-port/compressed'. Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sat, 01 Jun 2019 12:23:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 35880 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Pierre Neidhardt Cc: 35880-done@debbugs.gnu.org Received: via spool by 35880-done@debbugs.gnu.org id=D35880.155939172517523 (code D ref 35880); Sat, 01 Jun 2019 12:23:02 +0000 Received: (at 35880-done) by debbugs.gnu.org; 1 Jun 2019 12:22:05 +0000 Received: from localhost ([127.0.0.1]:37380 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hX31R-0004YZ-Ci for submit@debbugs.gnu.org; Sat, 01 Jun 2019 08:22:05 -0400 Received: from eggs.gnu.org ([209.51.188.92]:49615) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hX31P-0004Y6-M4 for 35880-done@debbugs.gnu.org; Sat, 01 Jun 2019 08:22:04 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:37859) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hX31J-0002z7-48; Sat, 01 Jun 2019 08:21:57 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=54498 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1hX31I-00061N-IS; Sat, 01 Jun 2019 08:21:56 -0400 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= References: <20190524134238.22802-1-ludo@gnu.org> <87d0k6o3am.fsf@ambrevar.xyz> <87ftp1m1te.fsf@gnu.org> <87blznsxym.fsf@gnu.org> <87ef4jlvbr.fsf@ambrevar.xyz> <87lfyrr554.fsf@gnu.org> <87sgszk3ev.fsf@ambrevar.xyz> <877eab3tjq.fsf@gnu.org> <87k1ebj7a8.fsf@ambrevar.xyz> <871s0i1yji.fsf@gnu.org> <87a7f5wbow.fsf@ambrevar.xyz> <87v9xqny4u.fsf@gnu.org> <87ef4dzvuy.fsf@ambrevar.xyz> <87ef4dk5g8.fsf@gnu.org> <8736ktzkxs.fsf@ambrevar.xyz> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 13 Prairial an 227 de la =?UTF-8?Q?R=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Sat, 01 Jun 2019 14:21:53 +0200 In-Reply-To: <8736ktzkxs.fsf@ambrevar.xyz> (Pierre Neidhardt's message of "Sat, 01 Jun 2019 11:58:23 +0200") Message-ID: <8736ktijha.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Spam-Score: -2.3 (--) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Pierre Neidhardt skribis: > Ludovic Court=C3=A8s writes: > >> =E2=80=98feed-decoder!=E2=80=99 cannot return an empty bytevector because >> =E2=80=98lz-decompress-write-size=E2=80=99 necessarily returns a strictl= y positive >> integer at this point. > > I'm not sure that's true: if the buffer is full and the next > lz-decompress-read does not read anything, then the buffer will still be > full and lz-decompress-write-size will return 0. Hmm I don=E2=80=99t think that=E2=80=99s a reasonable scenario, otherwise i= t would mean that you=E2=80=99re in a deadlock anyway (decoder buffer is already full ye= t it does not contain enough data to actually decode anything.) Dunno, I=E2=80=99m rather confident here but we=E2=80=99ll see if that caus= es any troubles. Thanks, Ludo=E2=80=99.