GNU bug report logs - #21022
see how much cloudflare's zlib-improving techniques can help gzip

Previous Next

To reply to this bug, email your comments to 21022 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Jim Meyering <jim <at> meyering.net>
To: bug-gzip <at> gnu.org
Subject: see how much cloudflare's zlib-improving techniques can help gzip
Date: Thu, 9 Jul 2015 16:31:00 -0700

https://blog.cloudflare.com/cloudflare-fights-cancer/

Message #8 received at 21022 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Jim Meyering <jim <at> meyering.net>, 21022 <at> debbugs.gnu.org
Subject: Re: bug#21022: see how much cloudflare's zlib-improving techniques
 can help gzip
Date: Thu, 09 Jul 2015 22:51:24 -0700

On 07/09/2015 04:31 PM, Jim Meyering wrote:
> https://blog.cloudflare.com/cloudflare-fights-cancer/
>

A few years ago I started to rewrite gzip to use zlib rather than to do 
its own compression and decompression.  If we finished that, we could 
use cloudfare's zlib implementation trivially.

Message #11 received at 21022 <at> debbugs.gnu.org (full text, mbox):

From: Mark Adler <madler <at> alumni.caltech.edu>
To: Paul Eggert <eggert <at> CS.UCLA.EDU>
Cc: 21022 <at> debbugs.gnu.org, Jim Meyering <jim <at> meyering.net>
Subject: Re: bug#21022: see how much cloudflare's zlib-improving techniques
 can help gzip
Date: Thu, 9 Jul 2015 22:56:36 -0700

As a side note, there already is a gzip replacement that I wrote that uses zlib and that also provides parallel compression called pigz.

> On Jul 9, 2015, at 10:51 PM, Paul Eggert <eggert <at> CS.UCLA.EDU> wrote:
> 
> On 07/09/2015 04:31 PM, Jim Meyering wrote:
>> https://blog.cloudflare.com/cloudflare-fights-cancer/
>> 
> 
> A few years ago I started to rewrite gzip to use zlib rather than to do its own compression and decompression.  If we finished that, we could use cloudfare's zlib implementation trivially.
> 
> 
> 

Message #14 received at 21022 <at> debbugs.gnu.org (full text, mbox):

From: Jim Meyering <jim <at> meyering.net>
To: Mark Adler <madler <at> alumni.caltech.edu>
Cc: 21022 <at> debbugs.gnu.org, Paul Eggert <eggert <at> cs.ucla.edu>
Subject: Re: bug#21022: see how much cloudflare's zlib-improving techniques
 can help gzip
Date: Fri, 10 Jul 2015 09:00:53 -0700

On Thu, Jul 9, 2015 at 10:56 PM, Mark Adler <madler <at> alumni.caltech.edu> wrote:
> As a side note, there already is a gzip replacement that I wrote that uses zlib and that also provides parallel compression called pigz.
>
>> On Jul 9, 2015, at 10:51 PM, Paul Eggert <eggert <at> CS.UCLA.EDU> wrote:
>>
>> On 07/09/2015 04:31 PM, Jim Meyering wrote:
>>> https://blog.cloudflare.com/cloudflare-fights-cancer/
>>>
>>
>> A few years ago I started to rewrite gzip to use zlib rather than to do its own compression and decompression.  If we finished that, we could use cloudfare's zlib implementation trivially.

Hi Mark,

How close is pigz to being a strict superset of the functionality of gzip?
I.e., would we lose anything if we were to install pigz with the name gzip?

Message #17 received at 21022 <at> debbugs.gnu.org (full text, mbox):

From: Mark Adler <madler <at> alumni.caltech.edu>
To: Jim Meyering <jim <at> meyering.net>
Cc: 21022 <at> debbugs.gnu.org, Paul Eggert <eggert <at> cs.ucla.edu>
Subject: Re: bug#21022: see how much cloudflare's zlib-improving techniques
 can help gzip
Date: Fri, 10 Jul 2015 10:25:48 -0700

Jim,

On Jul 10, 2015, at 9:00 AM, Jim Meyering <jim <at> meyering.net> wrote:
> How close is pigz to being a strict superset of the functionality of gzip?
> I.e., would we lose anything if we were to install pigz with the name gzip?

It should be very close.  I wrote it to be a drop-in replacement.

As I recall there is a difference in the way arguments are processed.  pigz considers the arguments to be an ordered series of instructions, whereas I think gzip looks at all of the options first and them applies them to all of the names.  Someone reported that as an issue when they tried to drop it in where the gzip options followed the name.

pigz does not have unlzh or unpack.  pigz does decompress Unix compress files.

pigz currently only compiles on Unix-ish systems with pthreads.

The super part of the superset includes parallel compression with a speedup of about n for n cores (and limited parallel improvement on decompression), rsyncable compression, compression and decompression of zlib and single-entry zip format, and zopfli maximal, but really slow, compression.

Mark



gzip124 1.2.4 (18 Aug 93)
usage: gzip124 [-cdfhlLnNrtvV19] [-S suffix] [file ...]
 -c --stdout      write on standard output, keep original files unchanged
 -d --decompress  decompress
 -f --force       force overwrite of output file and compress links
 -h --help        give this help
 -l --list        list compressed file contents
 -L --license     display software license
 -n --no-name     do not save or restore the original name and time stamp
 -N --name        save or restore the original name and time stamp
 -q --quiet       suppress all warnings
 -r --recursive   operate recursively on directories
 -S .suf  --suffix .suf     use suffix .suf on compressed files
 -t --test        test compressed file integrity
 -v --verbose     verbose mode
 -V --version     display version number
 -1 --fast        compress faster
 -9 --best        compress better
 file...          files to (de)compress. If none given, use standard input.



Usage: gzip16 [OPTION]... [FILE]...
Compress or uncompress FILEs (by default, compress FILES in-place).

Mandatory arguments to long options are mandatory for short options too.

  -c, --stdout      write on standard output, keep original files unchanged
  -d, --decompress  decompress
  -f, --force       force overwrite of output file and compress links
  -h, --help        give this help
  -k, --keep        keep (don't delete) input files
  -l, --list        list compressed file contents
  -L, --license     display software license
  -n, --no-name     do not save or restore the original name and time stamp
  -N, --name        save or restore the original name and time stamp
  -q, --quiet       suppress all warnings
  -r, --recursive   operate recursively on directories
  -S, --suffix=SUF  use suffix SUF on compressed files
  -t, --test        test compressed file integrity
  -v, --verbose     verbose mode
  -V, --version     display version number
  -1, --fast        compress faster
  -9, --best        compress better

With no FILE, or when FILE is -, read standard input.

Report bugs to <bug-gzip <at> gnu.org>.



Usage: pigz [options] [files ...]
  will compress files in place, adding the suffix '.gz'.  If no files are
  specified, stdin will be compressed to stdout.  pigz does what gzip does,
  but spreads the work over multiple processors and cores when compressing.

Options:
  -0 to -9, -11        Compression level (11 is much slower, a few % better)
  --fast, --best       Compression levels 1 and 9 respectively
  -b, --blocksize mmm  Set compression block size to mmmK (default 128K)
  -c, --stdout         Write all processed output to stdout (won't delete)
  -d, --decompress     Decompress the compressed input
  -f, --force          Force overwrite, compress .gz, links, and to terminal
  -F  --first          Do iterations first, before block split for -11
  -h, --help           Display a help screen and quit
  -i, --independent    Compress blocks independently for damage recovery
  -I, --iterations n   Number of iterations for -11 optimization
  -k, --keep           Do not delete original file after processing
  -K, --zip            Compress to PKWare zip (.zip) single entry format
  -l, --list           List the contents of the compressed input
  -L, --license        Display the pigz license and quit
  -M, --maxsplits n    Maximum number of split blocks for -11
  -n, --no-name        Do not store or restore file name in/from header
  -N, --name           Store/restore file name and mod time in/from header
  -O  --oneblock       Do not split into smaller blocks for -11
  -p, --processes n    Allow up to n compression threads (default is the
                       number of online processors, or 8 if unknown)
  -q, --quiet          Print no messages, even on error
  -r, --recursive      Process the contents of all subdirectories
  -R, --rsyncable      Input-determined block locations for rsync
  -S, --suffix .sss    Use suffix .sss instead of .gz (for compression)
  -t, --test           Test the integrity of the compressed input
  -T, --no-time        Do not store or restore mod time in/from header
  -v, --verbose        Provide more verbose output
  -V  --version        Show the version of pigz
  -z, --zlib           Compress to zlib (.zz) instead of gzip format
  --                   All arguments after "--" are treated as files

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.