GNU bug report logs - #21270
gzip huge filesize problem

Previous Next

Full log

Message #8 received at 21270 <at> debbugs.gnu.org (full text, mbox):

From: Mark Adler <madler <at> alumni.caltech.edu>
To: Alexander Kleinsorge <aleks <at> physik.tu-berlin.de>
Cc: 21270 <at> debbugs.gnu.org
Subject: Re: bug#21270: gzip huge filesize problem
Date: Sun, 16 Aug 2015 00:58:19 -0700

Alexander,

Thank you for your report.  This is a well-known limitation of the gzip format.  The -l function makes use of the uncompressed length stored in the last four bytes of a gzip stream.  Therein lies the rub, since four bytes can represent no more than 4 GB - 1.

There is another problem with that approach, in that a valid gzip file may consist of a series of concatenated gzip streams, in which case -l will report only on the last one.  In that case, even if the entire stream decompresses to less than 4 GB, the result will still be incorrect.

The only reliable way to determine the uncompressed size of a gzip file is to decompress the entire file (which can be done without storing the result).  This in fact is what "pigz -lt file.gz" does.  It will correctly report the uncompressed length, but takes much longer than "gzip -l".

-l remains useful however in most cases, so it remains a gzip and pigz option.

Mark


> On Aug 15, 2015, at 2:42 PM, Alexander Kleinsorge <aleks <at> physik.tu-berlin.de> wrote:
> 
> Hi Gzip team,
> 
> I compressed a 500 GB file (raw hdd image) using gzip 1.6 under Ubuntu 14.10 (64 bit). uncompressing the file gives a file with 500 gb (checked).
> But "gzip -l" shows bad (small) uncompressed_size and bad ratio (-5167%).
> 
> Below you can see some details, but I think it is a general bug.
> Thanks for help, Alexander
> 
> 
> gzip -l asus.gz
> compressed        uncompressed  ratio uncompressed_name     99630975185          1891655680 -5166.9% asus
> 
> gzip --version
> gzip 1.6
> 
> Linux myname 3.16.0-43-generic #58-Ubuntu SMP Fri Jun 19 11:04:02 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
> 
> the 2 files (compressed 93gb + uncompressed 500gb)
> 
> -rwxrwx--- 1 root plugdev 99630975185 Aug 15 21:39 asus.gz
> -rwxrwx--- 1 root plugdev 500107862016 Aug 14 09:00 sdc.raw
> -rwxrwx--- 1 root plugdev 93G Aug 15 21:39 asus.gz
> -rwxrwx--- 1 root plugdev 466G Aug 14 09:00 sdc.raw
> 
> 
> 
> 

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.