GNU bug report logs - #21270
gzip huge filesize problem

Previous Next

Full log

View this message in rfc822 format

From: help-debbugs <at> gnu.org (GNU bug Tracking System)
To: Alexander Kleinsorge <aleks <at> physik.tu-berlin.de>
Subject: bug#21270: closed (Re: bug#21270: gzip huge filesize problem)
Date: Mon, 17 Aug 2015 03:46:03 +0000

[Message part 1 (text/plain, inline)]

Your bug report

#21270: gzip huge filesize problem

which was filed against the gzip package, has been closed.

The explanation is attached below, along with your original report.
If you require more details, please reply to 21270 <at> debbugs.gnu.org.

-- 
21270: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=21270
GNU Bug Tracking System
Contact help-debbugs <at> gnu.org with problems

[Message part 2 (message/rfc822, inline)]

From: Jim Meyering <jim <at> meyering.net>
To: Mark Adler <madler <at> alumni.caltech.edu>
Cc: Alexander Kleinsorge <aleks <at> physik.tu-berlin.de>,
 21270-done <at> debbugs.gnu.org
Subject: Re: bug#21270: gzip huge filesize problem
Date: Sun, 16 Aug 2015 21:44:54 -0600

tags 21270 notabug
thanks

On Sun, Aug 16, 2015 at 1:58 AM, Mark Adler <madler <at> alumni.caltech.edu> wrote:
> Alexander,
>
> Thank you for your report.  This is a well-known limitation of the gzip format.  The -l function makes use of the uncompressed length stored in the last four bytes of a gzip stream.  Therein lies the rub, since four bytes can represent no more than 4 GB - 1.
>
> There is another problem with that approach, in that a valid gzip file may consist of a series of concatenated gzip streams, in which case -l will report only on the last one.  In that case, even if the entire stream decompresses to less than 4 GB, the result will still be incorrect.
>
> The only reliable way to determine the uncompressed size of a gzip file is to decompress the entire file (which can be done without storing the result).  This in fact is what "pigz -lt file.gz" does.  It will correctly report the uncompressed length, but takes much longer than "gzip -l".
>
> -l remains useful however in most cases, so it remains a gzip and pigz option.

Thank you for replying Mark.
I've marked this as "notabug" with the in-line comment above, and am
closing the auto-created issue with the "-done" part of the debbugs
email recipient address.

[Message part 3 (message/rfc822, inline)]

From: Alexander Kleinsorge <aleks <at> physik.tu-berlin.de>
To: bug-gzip <at> gnu.org
Subject: gzip huge filesize problem
Date: Sat, 15 Aug 2015 23:42:20 +0200

Hi Gzip team,

I compressed a 500 GB file (raw hdd image) using gzip 1.6 under Ubuntu 
14.10 (64 bit). uncompressing the file gives a file with 500 gb 
(checked).
But "gzip -l" shows bad (small) uncompressed_size and bad ratio 
(-5167%).

Below you can see some details, but I think it is a general bug.
Thanks for help, Alexander


gzip -l asus.gz
compressed        uncompressed  ratio uncompressed_name     99630975185  
        1891655680 -5166.9% asus

gzip --version
gzip 1.6

Linux myname 3.16.0-43-generic #58-Ubuntu SMP Fri Jun 19 11:04:02 UTC 
2015 x86_64 x86_64 x86_64 GNU/Linux

the 2 files (compressed 93gb + uncompressed 500gb)

-rwxrwx--- 1 root plugdev 99630975185 Aug 15 21:39 asus.gz
-rwxrwx--- 1 root plugdev 500107862016 Aug 14 09:00 sdc.raw
-rwxrwx--- 1 root plugdev 93G Aug 15 21:39 asus.gz
-rwxrwx--- 1 root plugdev 466G Aug 14 09:00 sdc.raw

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.