GNU bug report logs - #29714
Lzip variations

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 29714 in the body.
You can then email your comments to 29714 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Tobias Geerinckx-Rice <me <at> tobias.gr>
To: guix-patches <at> gnu.org
Subject: Lzip variations
Date: Thu, 14 Dec 2017 22:35:17 +0100

Guix,

This adds some more lzip implementations (“imps”) by the same author.

Kind regards,

T G-R

Message #8 received at 29714 <at> debbugs.gnu.org (full text, mbox):

From: Tobias Geerinckx-Rice <me <at> tobias.gr>
To: 29714 <at> debbugs.gnu.org
Subject: [PATCH 1/4] gnu: Add lunzip.
Date: Thu, 14 Dec 2017 22:37:21 +0100

* gnu/packages/compression.scm (lunzip): New public variable.
---
 gnu/packages/compression.scm | 27 +++++++++++++++++++++++++++
 1 file changed, 27 insertions(+)

diff --git a/gnu/packages/compression.scm b/gnu/packages/compression.scm
index 6b8b28b9f..6bd0e257e 100644
--- a/gnu/packages/compression.scm
+++ b/gnu/packages/compression.scm
@@ -1937,3 +1937,30 @@ type by using either Perl modules, or command-line tools on your system.")
     (description "Tukaani-xz is an implementation of xz compression/decompression
 algorithms in Java.")
     (license license:public-domain)))
+
+(define-public lunzip
+  (package
+    (name "lunzip")
+    (version "1.9")
+    (source
+     (origin
+       (method url-fetch)
+       (uri (string-append "http://download.savannah.gnu.org/releases/lzip/"
+                           name "/" name "-" version ".tar.gz"))
+       (sha256
+        (base32 "1ax3d9cp66z1qb9q7lfzg5bpx9630xrxgq9a5sw569wm0qqgpg2q"))))
+    (build-system gnu-build-system)
+    (arguments
+     `(#:configure-flags
+       (list "CC=gcc")))
+    (home-page "http://www.nongnu.org/lzip/lunzip.html")
+    (synopsis "Small, stand-alone lzip decompressor")
+    (description
+     "Lunzip is a decompressor for files in the lzip compression format (.lz),
+written as a single small C tool with no dependencies.  This makes it
+well-suited to embedded and other systems without a C++ compiler, or for use in
+applications such as software installers that need only to decompress files,
+not compress them.
+Lunzip is intended to be fully compatible with the regular lzip package.")
+    (license (list license:bsd-2        ; carg_parser.[ch]
+                   license:gpl2+))))    ; everything else
-- 
2.15.0

Message #11 received at 29714 <at> debbugs.gnu.org (full text, mbox):

From: Tobias Geerinckx-Rice <me <at> tobias.gr>
To: 29714 <at> debbugs.gnu.org
Subject: [PATCH 2/4] gnu: Add clzip.
Date: Thu, 14 Dec 2017 22:37:22 +0100

* gnu/packages/compression.scm (clzip): New public variable.
---
 gnu/packages/compression.scm | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)

diff --git a/gnu/packages/compression.scm b/gnu/packages/compression.scm
index 6bd0e257e..f4c90c25c 100644
--- a/gnu/packages/compression.scm
+++ b/gnu/packages/compression.scm
@@ -1964,3 +1964,28 @@ not compress them.
 Lunzip is intended to be fully compatible with the regular lzip package.")
     (license (list license:bsd-2        ; carg_parser.[ch]
                    license:gpl2+))))    ; everything else
+
+(define-public clzip
+  (package
+    (name "clzip")
+    (version "1.9")
+    (source
+     (origin
+       (method url-fetch)
+       (uri (string-append "http://download.savannah.gnu.org/releases/lzip/"
+                           name "/" name "-" version ".tar.gz"))
+       (sha256
+        (base32 "1brvsnpihzj81cf4wk2x5bnr2qldlq0wncpdbzxmzvxapm1cq2yc"))))
+    (build-system gnu-build-system)
+    (arguments
+     `(#:configure-flags
+       (list "CC=gcc")))
+    (home-page "http://www.nongnu.org/lzip/clzip.html")
+    (synopsis "Small, stand-alone lzip compressor and decompressor")
+    (description
+     "Lunzip is a compressor and decompressor for files in the lzip compression
+format (.lz), written as a single small C tool with no dependencies.  This makes
+it well-suited to embedded and other systems without a C++ compiler, or for use
+in other applications like package managers.
+Clzip is intended to be fully compatible with the regular lzip package.")
+    (license license:gpl2+)))
-- 
2.15.0

Message #14 received at 29714 <at> debbugs.gnu.org (full text, mbox):

From: Tobias Geerinckx-Rice <me <at> tobias.gr>
To: 29714 <at> debbugs.gnu.org
Subject: [PATCH 3/4] gnu: Add lzlib.
Date: Thu, 14 Dec 2017 22:37:23 +0100

* gnu/packages/compression.scm (lzlib): New public variable.
---
 gnu/packages/compression.scm | 28 ++++++++++++++++++++++++++++
 1 file changed, 28 insertions(+)

diff --git a/gnu/packages/compression.scm b/gnu/packages/compression.scm
index f4c90c25c..42af4cd51 100644
--- a/gnu/packages/compression.scm
+++ b/gnu/packages/compression.scm
@@ -1989,3 +1989,31 @@ it well-suited to embedded and other systems without a C++ compiler, or for use
 in other applications like package managers.
 Clzip is intended to be fully compatible with the regular lzip package.")
     (license license:gpl2+)))
+
+(define-public lzlib
+  (package
+    (name "lzlib")
+    (version "1.9")
+    (source
+     (origin
+       (method url-fetch)
+       (uri (string-append "http://download.savannah.gnu.org/releases/lzip/"
+                           name "/" name "-" version ".tar.gz"))
+       (sha256
+        (base32 "13mssf3hrcnmd4ijbqnxfk0zgj1q5lvpxxkm1hmrbl1h73czhwi4"))))
+    (build-system gnu-build-system)
+    ;; The included minilzip binary is only ~16 smaller than the ‘real’ lzip.
+    ;; It's used during the test suite, but don't be tempted to install it.
+    (arguments
+     `(#:configure-flags
+       (list "CC=gcc"
+             "--enable-shared")))       ; only static (.a) is built by default
+    (home-page "http://www.nongnu.org/lzip/lzlib.html")
+    (synopsis "Lzip data compression C library")
+    (description
+     "Lzlib is a C library for in-memory LZMA compression and decompression in
+the lzip format.  It supports integrity checking of the decompressed data, and
+all functions are thread-safe.  The library should never crash, even in case of
+corrupted input.")
+    (license (list license:bsd-2        ; the library itself
+                   license:gpl2+))))    ; main.c (i.e. minilzip, used by tests)
-- 
2.15.0

Message #17 received at 29714 <at> debbugs.gnu.org (full text, mbox):

From: Tobias Geerinckx-Rice <me <at> tobias.gr>
To: 29714 <at> debbugs.gnu.org
Subject: [PATCH 4/4] gnu: Add plzip.
Date: Thu, 14 Dec 2017 22:37:24 +0100

* gnu/packages/compression.scm (plzip): New public variable.
---
 gnu/packages/compression.scm | 30 ++++++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)

diff --git a/gnu/packages/compression.scm b/gnu/packages/compression.scm
index 42af4cd51..c3a6345b1 100644
--- a/gnu/packages/compression.scm
+++ b/gnu/packages/compression.scm
@@ -2017,3 +2017,33 @@ all functions are thread-safe.  The library should never crash, even in case of
 corrupted input.")
     (license (list license:bsd-2        ; the library itself
                    license:gpl2+))))    ; main.c (i.e. minilzip, used by tests)
+
+(define-public plzip
+  (package
+    (name "plzip")
+    (version "1.6")
+    (source
+     (origin
+       (method url-fetch)
+       (uri (string-append "http://download.savannah.gnu.org/releases/lzip/"
+                           name "/" name "-" version ".tar.gz"))
+       (sha256
+        (base32 "0z2cs6vn4xl65wakd013xl3sdfpg8dr0cvcjwc2slh8y9bz7j7ax"))))
+    (build-system gnu-build-system)
+    (inputs
+     `(("lzlib" ,lzlib)))
+    (home-page "http://www.nongnu.org/lzip/plzip.html")
+    (synopsis "Parallel lossless data compressor for the lzip format")
+    (description
+     "Plzip is a massively parallel (multi-threaded) lossless data compressor
+and decompressor that uses the lzip file format (.lz).  Files produced by plzip
+are fully compatible with lzip and can be rescued with lziprecover.
+On multiprocessor machines, plzip can compress and decompress large files much
+faster than lzip, at the cost of a slightly reduced compression ratio (0.4% to
+2%).  The number of usable threads is limited by file size: on files of only a
+few MiB, plzip is no faster than lzip.
+Files that were compressed with regular lzip will also not be decompressed
+faster by plzip, unless the @code{-b} option was used: lzip usually produces
+single-member files which can't be decompressed in parallel.")
+    (license (list license:bsd-2        ; arg_parser.{cc,h}
+                   license:gpl2+))))    ; everything else
-- 
2.15.0

Message #20 received at 29714 <at> debbugs.gnu.org (full text, mbox):

From: Leo Famulari <leo <at> famulari.name>
To: Tobias Geerinckx-Rice <me <at> tobias.gr>
Cc: 29714 <at> debbugs.gnu.org
Subject: Re: [bug#29714] [PATCH 2/4] gnu: Add clzip.
Date: Fri, 15 Dec 2017 14:14:57 -0500

[Message part 1 (text/plain, inline)]

On Thu, Dec 14, 2017 at 10:37:22PM +0100, Tobias Geerinckx-Rice wrote:
> * gnu/packages/compression.scm (clzip): New public variable.

> +    (synopsis "Small, stand-alone lzip compressor and decompressor")
> +    (description
> +     "Lunzip is a compressor and decompressor for files in the lzip compression
> +format (.lz), written as a single small C tool with no dependencies.  This makes
> +it well-suited to embedded and other systems without a C++ compiler, or for use
> +in other applications like package managers.
> +Clzip is intended to be fully compatible with the regular lzip package.")

This description is mistakenly copy-pasted from the lunzip package,
right?

[signature.asc (application/pgp-signature, inline)]

Message #23 received at 29714 <at> debbugs.gnu.org (full text, mbox):

From: Leo Famulari <leo <at> famulari.name>
To: Tobias Geerinckx-Rice <me <at> tobias.gr>
Cc: 29714 <at> debbugs.gnu.org
Subject: Re: [bug#29714] [PATCH 1/4] gnu: Add lunzip.
Date: Fri, 15 Dec 2017 14:17:04 -0500

[Message part 1 (text/plain, inline)]

On Thu, Dec 14, 2017 at 10:37:21PM +0100, Tobias Geerinckx-Rice wrote:
> * gnu/packages/compression.scm (lunzip): New public variable.

If the closure of this package is smaller than that of lzip, perhaps we
could use it instead as the default LZ decompressor when building
packages.

[signature.asc (application/pgp-signature, inline)]

Message #26 received at 29714 <at> debbugs.gnu.org (full text, mbox):

From: Tobias Geerinckx-Rice <me <at> tobias.gr>
To: leo <at> famulari.name
Cc: 29714 <at> debbugs.gnu.org
Subject: Re: [bug#29714] [PATCH 2/4] gnu: Add clzip.
Date: Fri, 15 Dec 2017 20:23:36 +0100

[Message part 1 (text/plain, inline)]

Leo,

Leo Famulari wrote on 15/12/17 at 20:14:
> On Thu, Dec 14, 2017 at 10:37:22PM +0100, Tobias Geerinckx-Rice wrote:
>> * gnu/packages/compression.scm (clzip): New public variable.
> 
>> +    (synopsis "Small, stand-alone lzip compressor and decompressor")
>> +    (description
>> +     "Lunzip is a compressor and decompressor for files in the lzip compression
>> +format (.lz), written as a single small C tool with no dependencies.  This makes
>> +it well-suited to embedded and other systems without a C++ compiler, or for use
>> +in other applications like package managers.
>> +Clzip is intended to be fully compatible with the regular lzip package.")
> 
> This description is mistakenly copy-pasted from the lunzip package,
> right?

Partly. The left-over ‘Lunzip’ is indeed a mistake, but the rest is
intentional. The upstream home pages are equally similar, up to the
example of package managers (here without the ‘that only need to
decompress’ caveat).

They're certainly & confusingly similar; thanks for noticing the error!

Kind regards,

T G-R

[signature.asc (application/pgp-signature, attachment)]

Message #29 received at 29714 <at> debbugs.gnu.org (full text, mbox):

From: Leo Famulari <leo <at> famulari.name>
To: Tobias Geerinckx-Rice <me <at> tobias.gr>
Cc: 29714 <at> debbugs.gnu.org
Subject: Re: [bug#29714] [PATCH 4/4] gnu: Add plzip.
Date: Fri, 15 Dec 2017 14:22:57 -0500

[Message part 1 (text/plain, inline)]

On Thu, Dec 14, 2017 at 10:37:24PM +0100, Tobias Geerinckx-Rice wrote:
> * gnu/packages/compression.scm (plzip): New public variable.

> +    (synopsis "Parallel lossless data compressor for the lzip format")
> +    (description
> +     "Plzip is a massively parallel (multi-threaded) lossless data compressor
> +and decompressor that uses the lzip file format (.lz).  Files produced by plzip
> +are fully compatible with lzip and can be rescued with lziprecover.
> +On multiprocessor machines, plzip can compress and decompress large files much
> +faster than lzip, at the cost of a slightly reduced compression ratio (0.4% to
> +2%).  The number of usable threads is limited by file size: on files of only a
> +few MiB, plzip is no faster than lzip.
> +Files that were compressed with regular lzip will also not be decompressed
> +faster by plzip, unless the @code{-b} option was used: lzip usually produces
> +single-member files which can't be decompressed in parallel.")

Maybe this would be useful in ((guix packages) patch-and-repack), (guix
utils) and (guix scripts pack). In those modules, we compress XZ
archives in parallel to speed it up.

[signature.asc (application/pgp-signature, inline)]

Message #32 received at 29714 <at> debbugs.gnu.org (full text, mbox):

From: Tobias Geerinckx-Rice <me <at> tobias.gr>
To: leo <at> famulari.name
Cc: 29714 <at> debbugs.gnu.org
Subject: Re: [bug#29714] [PATCH 1/4] gnu: Add lunzip.
Date: Fri, 15 Dec 2017 20:27:11 +0100

[Message part 1 (text/plain, inline)]

Leo,

Leo Famulari wrote on 15/12/17 at 20:17:
> On Thu, Dec 14, 2017 at 10:37:21PM +0100, Tobias Geerinckx-Rice wrote:
>> * gnu/packages/compression.scm (lunzip): New public variable.
> 
> If the closure of this package is smaller than that of lzip, perhaps we
> could use it instead as the default LZ decompressor when building
> packages.

Indeed! I was thinking the same thing :-)

A performance comparison of all these implementations was on my to-do
list for the weekend, but might be delayed by mandatory holiday cheer.

Kind regards,

T G-R

[signature.asc (application/pgp-signature, attachment)]

Message #35 received at 29714 <at> debbugs.gnu.org (full text, mbox):

From: Tobias Geerinckx-Rice <me <at> tobias.gr>
To: leo <at> famulari.name
Cc: 29714 <at> debbugs.gnu.org
Subject: Re: [bug#29714] [PATCH 4/4] gnu: Add plzip.
Date: Fri, 15 Dec 2017 20:28:47 +0100

[Message part 1 (text/plain, inline)]

Leo Famulari wrote on 15/12/17 at 20:22:
> On Thu, Dec 14, 2017 at 10:37:24PM +0100, Tobias Geerinckx-Rice wrote:
>> * gnu/packages/compression.scm (plzip): New public variable.
> 
> Maybe this would be useful in ((guix packages) patch-and-repack), (guix
> utils) and (guix scripts pack). In those modules, we compress XZ
> archives in parallel to speed it up.

Stop exposing my secret plan.

Kind regards,

T G-R

[signature.asc (application/pgp-signature, attachment)]

Message #38 received at 29714 <at> debbugs.gnu.org (full text, mbox):

From: Tobias Geerinckx-Rice <me <at> tobias.gr>
To: leo <at> famulari.name
Cc: 29714 <at> debbugs.gnu.org
Subject: Re: [bug#29714] [PATCH 1/4] gnu: Add lunzip.
Date: Fri, 22 Dec 2017 02:10:38 +0100

[Message part 1 (text/plain, inline)]

Tobias Geerinckx-Rice wrote on 15/12/17 at 20:27:
>> * gnu/packages/compression.scm (lunzip): New public variable.
>
> A performance comparison of all these implementations was on my to-do
> list for the weekend, but might be delayed by mandatory holiday cheer.

...or, apparently, dental surgery.

It's about as much fun, but you're not allowed to drink.

Leo Famulari wrote on 15/12/17 at 20:17:
> If the closure of this package is smaller than that of lzip, perhaps we
> could use it instead as the default LZ decompressor when building
> packages.

If I were disappointed, it'd be by a surprisingly lean lzip:

  $ du -hs /gnu/store/wbww96afpdsfpnwhq4kxd6q70nz478jn-lunzip-1.9
  72K      ... # the lunzip binary is 48K
  $ guix size /gnu/store/wbww96afpdsfpnwhq4kxd6q70nz478jn-lunzip-1.9
  store item                          total    self
  /gnu/store/...-glibc-2.25              38.5    37.1  54.0%
  /gnu/store/...-gcc-5.4.0-lib           68.6    30.1  43.8%
  /gnu/store/...-bash-static-4.4.12       1.4     1.4   2.1%
  /gnu/store/...-lunzip-1.9              68.7     0.0   0.1%
  total: 68.7 MiB

  $ du -hs /gnu/store/8abzs1c39klm0d2sh8bwv6xn7924blam-lzip-1.18
  172K     ... # the lzip binary is 124K
  $ guix size /gnu/store/8abzs1c39klm0d2sh8bwv6xn7924blam-lzip-1.18
  store item                          total    self
  /gnu/store/...-glibc-2.25              38.5    37.1  54.0%
  /gnu/store/...-gcc-5.4.0-lib           68.6    30.1  43.8%
  /gnu/store/...-bash-static-4.4.12       1.4     1.4   2.1%
  /gnu/store/...-lzip-1.18               68.8     0.1   0.2%
  total: 68.8 MiB

Lzip: too cool to fit in 100%.

Both variants are equally fast. Switching to lunzip saves 100K.
I'm tempted to try it, but that might be the opiates talking.

Kind regards,

T G-R

[signature.asc (application/pgp-signature, attachment)]

Message #43 received at 29714-done <at> debbugs.gnu.org (full text, mbox):

From: Tobias Geerinckx-Rice <me <at> tobias.gr>
To: 29714-done <at> debbugs.gnu.org
Subject: Re: [PATCH 1/4] gnu: Add lunzip.
Date: Fri, 22 Dec 2017 02:30:00 +0100

Pushed with minor fixes as 34e8cf225cc380c2f248eecd43151f6751eeda97 et al.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.