GNU bug report logs - #20104
[PATCH] gzip: make the GZIP env var obsolescent

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 20104 in the body.
You can then email your comments to 20104 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: gzip bugs <bug-gzip <at> gnu.org>
Subject: [PATCH] gzip: make the GZIP env var obsolescent
Date: Fri, 13 Mar 2015 19:20:11 -0700

[Message part 1 (text/plain, inline)]

Attached is a proposed patch to make the GZIP environment variable obsolescent, 
for the same reason we're making GREP_OPTIONS obsolescent: it's too much 
opportunity for trouble.  For example, with a suitably crafted GZIP environment 
variable I can cause 'gzip' to remove files.

[0001-gzip-make-the-GZIP-env-var-obsolescent.patch (text/x-patch, attachment)]

Message #8 received at 20104 <at> debbugs.gnu.org (full text, mbox):

From: Jim Meyering <jim <at> meyering.net>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: 20104 <at> debbugs.gnu.org
Subject: Re: bug#20104: [PATCH] gzip: make the GZIP env var obsolescent
Date: Sun, 15 Mar 2015 09:04:40 -0700

On Fri, Mar 13, 2015 at 7:20 PM, Paul Eggert <eggert <at> cs.ucla.edu> wrote:
> Attached is a proposed patch to make the GZIP environment variable
> obsolescent, for the same reason we're making GREP_OPTIONS obsolescent: it's
> too much opportunity for trouble.  For example, with a suitably crafted GZIP
> environment variable I can cause 'gzip' to remove files.

Hi Paul,
Thanks for getting the ball rolling.
That patch makes it so use of the GZIP envvar makes gzip fail with
exit status 2:

  $ echo |./gzip > k
  $ echo |GZIP=-1 ./gzip > k
  gzip: warning: GZIP environ var is deprecated; please use an alias or script
  [Exit 2]

Making that clear feels like sufficient justification for adding a test.
If making GZIP use provoke failure is deliberate, the NEWS should say that,
for those who don't recall that a gzip warning implies nonzero exit status.
Also, you may want to spell out "environment variable" in that diagnostic.
Finally, for something prescriptive like that, I would drop the "please".

Message #11 received at 20104 <at> debbugs.gnu.org (full text, mbox):

From: Mark Adler <madler <at> alumni.caltech.edu>
To: Paul Eggert <eggert <at> CS.UCLA.EDU>
Cc: 20104 <at> debbugs.gnu.org
Subject: Re: bug#20104: [PATCH] gzip: make the GZIP env var obsolescent
Date: Sun, 15 Mar 2015 09:39:34 -0700

All,

Might it be better to protect against the vulnerability, instead of deep-sixing the entire capability out of fear?  You could allow only compression level options in the environment variable, which I think was its main intent in the first place.

Mark


On Mar 13, 2015, at 7:20 PM, Paul Eggert <eggert <at> CS.UCLA.EDU> wrote:
> Attached is a proposed patch to make the GZIP environment variable obsolescent, for the same reason we're making GREP_OPTIONS obsolescent: it's too much opportunity for trouble.  For example, with a suitably crafted GZIP environment variable I can cause 'gzip' to remove files.
> <0001-gzip-make-the-GZIP-env-var-obsolescent.patch>

Message #14 received at 20104 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Mark Adler <madler <at> alumni.caltech.edu>
Cc: 20104 <at> debbugs.gnu.org
Subject: Re: bug#20104: [PATCH] gzip: make the GZIP env var obsolescent
Date: Sun, 15 Mar 2015 17:54:24 -0700

Mark Adler wrote:
> You could allow only compression level options in the environment variable, which I think was its main intent in the first place.

The documentation currently suggests using GZIP to specify -q, -v, and --name 
(-N), so it appears that this intent was lost long ago.

Allowing "harmless" options in GZIP raises the question: where does one draw the 
line between "harmless" and "harmful"?  Clearly options like -d and -t are 
typically harmful.  In contrast, -q, -v, -n, -N, and -1 through -9 might be 
harmless in many cases, though they could still cause problems in some.

Any rule that says that GZIP is OK for some options but not others would add 
some complexity to the spec, and would be confusing in the long run for users.

I'm inclined to implement your suggestion immediately (i.e., for the next 
release of gzip), while documenting that the GZIP environment variable is 
obsolescent and that we plan to remove it entirely later.  Following Jim's 
suggestion, gzip should exit with status 0 rather than status 2 when it warns 
about the nontrivial use of the GZIP environment variable.

Message #17 received at 20104 <at> debbugs.gnu.org (full text, mbox):

From: Jim Meyering <jim <at> meyering.net>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: 20104 <at> debbugs.gnu.org, Mark Adler <madler <at> alumni.caltech.edu>
Subject: Re: bug#20104: [PATCH] gzip: make the GZIP env var obsolescent
Date: Sun, 15 Mar 2015 20:40:27 -0700

On Sun, Mar 15, 2015 at 5:54 PM, Paul Eggert <eggert <at> cs.ucla.edu> wrote:
> Mark Adler wrote:
>>
>> You could allow only compression level options in the environment
>> variable, which I think was its main intent in the first place.
>
>
> The documentation currently suggests using GZIP to specify -q, -v, and
> --name (-N), so it appears that this intent was lost long ago.
>
> Allowing "harmless" options in GZIP raises the question: where does one draw
> the line between "harmless" and "harmful"?  Clearly options like -d and -t
> are typically harmful.  In contrast, -q, -v, -n, -N, and -1 through -9 might
> be harmless in many cases, though they could still cause problems in some.
>
> Any rule that says that GZIP is OK for some options but not others would add
> some complexity to the spec, and would be confusing in the long run for
> users.
>
> I'm inclined to implement your suggestion immediately (i.e., for the next
> release of gzip), while documenting that the GZIP environment variable is
> obsolescent and that we plan to remove it entirely later.

Sounds good.
I too prefer to deprecate, and then (a release or two later) to remove entirely.
There is just too much room for "surprise", otherwise.

Message #20 received at 20104 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Jim Meyering <jim <at> meyering.net>
Cc: 20104 <at> debbugs.gnu.org, Mark Adler <madler <at> alumni.caltech.edu>
Subject: Re: bug#20104: [PATCH] gzip: make the GZIP env var obsolescent
Date: Mon, 16 Mar 2015 14:41:18 -0700

[Message part 1 (text/plain, inline)]

On 03/15/2015 08:40 PM, Jim Meyering wrote:
> Sounds good.
> I too prefer to deprecate, and then (a release or two later) to remove entirely.
OK, thanks, here's a draft patch to do that.

[0001-gzip-make-the-GZIP-env-var-obsolescent.patch (text/x-patch, attachment)]

Message #23 received at 20104 <at> debbugs.gnu.org (full text, mbox):

From: Jim Meyering <jim <at> meyering.net>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: 20104 <at> debbugs.gnu.org, Mark Adler <madler <at> alumni.caltech.edu>
Subject: Re: bug#20104: [PATCH] gzip: make the GZIP env var obsolescent
Date: Tue, 17 Mar 2015 07:53:31 -0700

On Mon, Mar 16, 2015 at 2:41 PM, Paul Eggert <eggert <at> cs.ucla.edu> wrote:
> On 03/15/2015 08:40 PM, Jim Meyering wrote:
>>
>> Sounds good.
>> I too prefer to deprecate, and then (a release or two later) to remove
>> entirely.
>
> OK, thanks, here's a draft patch to do that.

Very nice.
I found not even a nit to pick.  Thank you!

Message #26 received at 20104 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Jim Meyering <jim <at> meyering.net>
Cc: 20104 <at> debbugs.gnu.org, Mark Adler <madler <at> alumni.caltech.edu>
Subject: Re: bug#20104: [PATCH] gzip: make the GZIP env var obsolescent
Date: Tue, 17 Mar 2015 11:19:35 -0700

On 03/17/2015 07:53 AM, Jim Meyering wrote:
> Very nice.

Thanks for checking it.  I changed one minor nit (an if-then-else which 
was a tiny bit clearer with the if-condition negated and the then- and 
else- parts swapped) and installed the result.

Hmm, after doing that, though, I discovered that Automake generates a 
dist-gzip rule that by default sets GZIP='--best' in the environment.  I 
think this is partly to work around the problem that GZIP in the 
environment can mess up how 'gzip' works, and partly to generate 
more-compressed output.  So this behavior by Automake is *both* a 
workaround for the longstanding gzip misfeature, *and* a use of the 
misfeature.  What a mess.

I suppose we could fix Automake to supply --best in a more-portable 
way.  Or perhaps we could unbend a bit and allow GZIP to supply -1 
through -9, --fast, --best.

Message #29 received at 20104 <at> debbugs.gnu.org (full text, mbox):

From: Jim Meyering <jim <at> meyering.net>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: 20104 <at> debbugs.gnu.org, Mark Adler <madler <at> alumni.caltech.edu>
Subject: Re: bug#20104: [PATCH] gzip: make the GZIP env var obsolescent
Date: Wed, 18 Mar 2015 04:34:55 +0000

[Message part 1 (text/plain, inline)]

Glad you caught that.

That reminds me:  I added the following to the GNUmakefile
we find in projects like coreutils, grep, diff, etc. (it comes from
gnulib/top/GNUmakefile), and once/if GNU tar deprecates
that envvar, we'll have to adjust that, too:

  # Make tar archive easier to reproduce.
  export TAR_OPTIONS = --owner=0 --group=0 --numeric-owner

The intent was to work (albeit degraded) even with non-GNU tar.
Now that I think about it, I dislike the idea of supporting inferior
tools when running commands to create a release tarball. That
process should be as repeatable as possible. Perhaps we should
try harder to find a tar-creating program that has options to do the
above, and simply abort when it is not found.

On Tue, Mar 17, 2015 at 11:20 AM Paul Eggert <eggert <at> cs.ucla.edu> wrote:

> On 03/17/2015 07:53 AM, Jim Meyering wrote:
> > Very nice.
>
> Thanks for checking it.  I changed one minor nit (an if-then-else which
> was a tiny bit clearer with the if-condition negated and the then- and
> else- parts swapped) and installed the result.
>
> Hmm, after doing that, though, I discovered that Automake generates a
> dist-gzip rule that by default sets GZIP='--best' in the environment.  I
> think this is partly to work around the problem that GZIP in the
> environment can mess up how 'gzip' works, and partly to generate
> more-compressed output.  So this behavior by Automake is *both* a
> workaround for the longstanding gzip misfeature, *and* a use of the
> misfeature.  What a mess.
>
> I suppose we could fix Automake to supply --best in a more-portable
> way.  Or perhaps we could unbend a bit and allow GZIP to supply -1
> through -9, --fast, --best.
>
>

[Message part 2 (text/html, inline)]

Message #32 received at 20104 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Jim Meyering <jim <at> meyering.net>
Cc: 20104 <at> debbugs.gnu.org, Mark Adler <madler <at> alumni.caltech.edu>
Subject: Re: bug#20104: [PATCH] gzip: make the GZIP env var obsolescent
Date: Tue, 17 Mar 2015 22:38:34 -0700

Jim Meyering wrote:
> Perhaps we should
> try harder to find a tar-creating program that has options to do the
> above, and simply abort when it is not found.

For creating tarballs, I think it'd be enough to simply specify the options on 
the command line.  Traditional 'tar' will reject them (with "tar: =: unknown 
function modifier") and will fail.  By and large, only developers want to create 
tarballs, and it's OK to ask them to use development-quality tools.

I did propose an Automake patch, here:

http://bugs.gnu.org/20132

This patch misspells "automake" as "autoconf"; I'll fix that before it goes in.

It's just a quality-of-implementation issue, as unpatched Automake will work OK 
with bleeding-edge gzip.  The main upside of the patch is that even with 
bleeding-edge gzip Automake would by default generate .gz tarballs no larger 
than the tarballs generated by unpatched Automake with older gzip.

Message #35 received at 20104 <at> debbugs.gnu.org (full text, mbox):

From: Jim Meyering <jim <at> meyering.net>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: 20104 <at> debbugs.gnu.org, Mark Adler <madler <at> alumni.caltech.edu>
Subject: Re: bug#20104: [PATCH] gzip: make the GZIP env var obsolescent
Date: Wed, 18 Mar 2015 14:40:29 -0700

On Tue, Mar 17, 2015 at 10:38 PM, Paul Eggert <eggert <at> cs.ucla.edu> wrote:
> Jim Meyering wrote:
>>
>> Perhaps we should
>> try harder to find a tar-creating program that has options to do the
>> above, and simply abort when it is not found.
>
>
> For creating tarballs, I think it'd be enough to simply specify the options
> on the command line.  Traditional 'tar' will reject them (with "tar: =:
> unknown function modifier") and will fail.  By and large, only developers
> want to create tarballs, and it's OK to ask them to use development-quality
> tools.
>
> I did propose an Automake patch, here:
>
> http://bugs.gnu.org/20132

Glanced through, but didn't have time for a thorough review.

> This patch misspells "automake" as "autoconf"; I'll fix that before it goes
> in.
>
> It's just a quality-of-implementation issue, as unpatched Automake will work
> OK with bleeding-edge gzip.  The main upside of the patch is that even with
> bleeding-edge gzip Automake would by default generate .gz tarballs no larger
> than the tarballs generated by unpatched Automake with older gzip.

That works for me.
Thanks for dealing with all of this.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.