GNU bug report logs - #13324
Improvements to "dist" targets (was: Re: EXTRA_DIST, directories, tar --exclude-vcs)

Previous Next

To reply to this bug, email your comments to 13324 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Stefano Lattarini <stefano.lattarini <at> gmail.com>
To: Karl Berry <karl <at> freefriends.org>
Cc: bug-automake <at> gnu.org, automake <at> gnu.org
Subject: Improvements to "dist" targets (was: Re: EXTRA_DIST, directories,
	tar --exclude-vcs)
Date: Tue, 01 Jan 2013 02:09:24 +0100

Severity: wishlist

Hi Karl.

I'm adding bug-automake in CC, so that this discussion will remain visible
and archived in a singe place, and we won't forget about the issue.

On 01/01/2013 01:38 AM, Karl Berry wrote:
> Stefano and all,
> 
> It would be nice to able to list directories in EXTRA_DIST. 
>
That is already possible:
<http://www.gnu.org/software/automake/manual/automake.html#The-dist-Hook>

> It is painful and unnecessary overhead to list every file in a contrib
> directory individually just to avoid including VC files.  (As explained at
> http://www.gnu.org/software/automake/manual/automake.html#Basics-of-Distribution.)
> 
> I don't propose any major surgery to make it work in every conceivable
> circumstance.  All that is really necessary is to provide a way to pass
> --exclude-vcs to tar.  It would only work with GNU tar, but that is ok.
>
Not if that would be done unconditionally (we don't want to require GNU tar);
but if you are proposing we should make it possible to pass extra arguments
to tar invocations, or to customize them in other ways, than I agree that
would be a nice feature to have.

OTOH, what about distribution "tarballs" in '.zip' format?  They don't
use tar at all ...  Time to deprecate them maybe?  Is anybody actually
using them?  And while at it, what about the even more obscure 'shar'
format?

> What worked for me is to override the definition of am__tar, like this in
> Makefile.am:
>   am__tar = tar --exclude-vcs --format=ustar -chf - "$$tardir"
> 
> Except I don't really want to put this into Makefile.am.  Happily, this
> also worked from the command line:
>   $ env am__tar='tar --exclude-vcs --format=ustar -chf - "$tardir"' make dist
> 
> For that matter, it would be nice if am__tar provided a way to pass
> arbitrary extra options.  Perhaps include something like
> $(AM_TAR_USER_OPTIONS) in the definition? 
> 
> Equivalently, provide a way to override the name "tar", as in
> $(AM_TAR_PROGRAM)?  That might be simplest of all.
>
Yes.  Maybe (just thinking aloud here) it could also help us to eventually
dispense with the horrible "tar-v7", "tar-ustar" and "tar-pax" options.
More stuff pushed at 'make dist' time rather than at configure time; and
less need to be "automagical", since 'dist' is a maintainer target ...

> Along those lines, it would be nice if Automake always used the -chf
> style above.  In some distributions' Makefiles I've seen "chf" (didn't
> look into why);
>
"Historical reasons" would be my first guess.

> that prevents adding options though, due to the vagaries
> of tar command line parsing which we all know.  I have the impression
> that every tar (except maybe really ancient ones, can't remember, but we
> don't care) supports the -style.
>
It would be nice to verify this claim on as much systems as possible
(ignoring museum pieces, of course).

> Sorry for the brain dump of all these possibilities, but wdyt?
> 
See above :-)

Thanks,
  Stefano

Message #8 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Bob Friesenhahn <bfriesen <at> simple.dallas.tx.us>
To: Stefano Lattarini <stefano.lattarini <at> gmail.com>
Cc: automake <at> gnu.org, bug-automake <at> gnu.org, Karl Berry <karl <at> freefriends.org>
Subject: Re: Improvements to "dist" targets (was: Re: EXTRA_DIST, directories, 
	tar --exclude-vcs)
Date: Tue, 1 Jan 2013 15:09:31 -0600 (CST)

On Tue, 1 Jan 2013, Stefano Lattarini wrote:
>>
>> I don't propose any major surgery to make it work in every conceivable
>> circumstance.  All that is really necessary is to provide a way to pass
>> --exclude-vcs to tar.  It would only work with GNU tar, but that is ok.

In GraphicsMagick I use find+cpio to copy directories.  This allows 
using egrep to filter out files which should not be distributed.

Is this combination more portable than GNU tar?

DISTDIRS = locale scripts www PerlMagick TclMagick
dist-hook:
	( \
	  builddir=`pwd` ; \
	  cd $(srcdir) && \
	  ( \
	    for dir in $(DISTDIRS) ; do \
	      find $$dir -depth -print | egrep -v '(~$$)|(/\.hg)|(/\.#)|(/\.deps)|(\.pyc)' \
	        | cpio -pdum $$builddir/$(distdir) 2> /dev/null ; \
	    done \
	  ) \
	)

Bob
-- 
Bob Friesenhahn
bfriesen <at> simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/

Message #11 received at 13324 <at> debbugs.gnu.org (full text, mbox):

From: Stefano Lattarini <stefano.lattarini <at> gmail.com>
To: Daniel Herring <dherring <at> tentpost.com>
Cc: 13324 <at> debbugs.gnu.org, automake <at> gnu.org
Subject: Re: Improvements to "dist" targets
Date: Wed, 02 Jan 2013 14:04:52 +0100

On 01/02/2013 02:01 PM, Stefano Lattarini wrote:
> On 01/02/2013 02:58 AM, Daniel Herring wrote:
>> On Tue, 1 Jan 2013, Stefano Lattarini wrote:
>>
>>> OTOH, what about distribution "tarballs" in '.zip' format?  They don't
>>> use tar at all ...  Time to deprecate them maybe?  Is anybody actually
>>> using them?  And while at it, what about the even more obscure 'shar'
>>> format?
>>
>> While I haven't manipulated a shar file in years, but zip is still
>> the dominant archive format on MS platforms.
>>
> While this is absolutely true, my point is that it's not a format truly
> used or required for distribution tarballs.  If you are going to compile
> an Automake-based package from source on MS Windows, you'll need either
> MinGW/MSYS or Cygwin, and AFAICS both those environment comes with
> working tar and gzip programs.
> 
> Or is there something that I'm missing?
> 
>> It is quite common (and a good practice) for a project to distribute
>> \n newlines in a tarball and \r\n newlines in a zip archive.
>>
> But the Automake "dist-*" recipes don't do this, so you'd need to roll
> your own rule if you want to support this use case (such a rule could
> of course leverage on the "distdir" Automake rule to do much of the
> work).
> 
I'm re-copying your message and my reply to the relevant discussion on
GNU debbugs, so that they will remain registered in our bug tracker.
Please do not drop the debbugs address in your further replies.

Thanks,
  Stefano

Message #14 received at 13324 <at> debbugs.gnu.org (full text, mbox):

From: Stefano Lattarini <stefano.lattarini <at> gmail.com>
To: Bob Friesenhahn <bfriesen <at> simple.dallas.tx.us>
Cc: 13324 <at> debbugs.gnu.org, Daniel Herring <dherring <at> tentpost.com>,
	automake <at> gnu.org
Subject: Re: Improvements to "dist" targets
Date: Wed, 02 Jan 2013 14:09:12 +0100

I'm re-copying this message to the relevant discussion on GNU
debbugs, so that it will remain registered in our bug tracker.
Please keep that address in CC in any further reply.

On 01/02/2013 04:10 AM, Bob Friesenhahn wrote:
> On Tue, 1 Jan 2013, Daniel Herring wrote:
> 
>> On Tue, 1 Jan 2013, Stefano Lattarini wrote:
>>
>>> OTOH, what about distribution "tarballs" in '.zip' format?  They don't
>>> use tar at all ...  Time to deprecate them maybe?  Is anybody actually
>>> using them?  And while at it, what about the even more obscure 'shar'
>>> format?
>>
>> While I haven't manipulated a shar file in years, but zip is still
>> the dominant archive format on MS platforms.  It is quite common
>> (and a good practice) for a project to distribute \n newlines in a
>> tarball and \r\n newlines in a zip archive.
> 
> The unzip from Info-ZIP supports the -a option to auto-convert text
> files to use MS-DOS line terminations.  It is not really necessary
> to convert to MS-DOS format while packaging.
> 
> Also, it seems that 'notepad' (default for opening .TXT files) is
> virtually the only Windows text editing program which fails to deal
> with Unix line terminations.  Windows development tools have no
> problems with Unix line terminations.
>
This is good news.

> For my own project, we have ditched zip and switched to 7-zip
> instead because it compresses much better.
> 
> BZip is reaching the point of so little value, it looks like it
> should be tossed.
>
But it appears it is still being used (also by Autoconf tarballs, for
exaple), so we should be more gentle and careful in deprecating and
removing it, of we decide to go down that road.

OTOH, I'm not aware of any real use of 'dist-tarZ', 'dist-zip' or
'dist-shar' these days, so we might start to deprecate (but nor
remove yes) them from the next 1.13.2 release.

> Gzip files are not much larger and can serve
> for the case where the most universal format is needed.
> 
I agree that ever deprecating or tossing gzip would be a bad idea.

Regards,
  Stefano

Message #17 received at 13324 <at> debbugs.gnu.org (full text, mbox):

From: Peter Rosin <peda <at> lysator.liu.se>
To: Stefano Lattarini <stefano.lattarini <at> gmail.com>
Cc: 13324 <at> debbugs.gnu.org, Daniel Herring <dherring <at> tentpost.com>,
	automake <at> gnu.org
Subject: Re: bug#13324: Improvements to "dist" targets
Date: Wed, 02 Jan 2013 19:51:49 +0100

On 2013-01-02 14:04, Stefano Lattarini wrote:
> On 01/02/2013 02:01 PM, Stefano Lattarini wrote:
>> On 01/02/2013 02:58 AM, Daniel Herring wrote:
>>> On Tue, 1 Jan 2013, Stefano Lattarini wrote:
>>>
>>>> OTOH, what about distribution "tarballs" in '.zip' format?  They don't
>>>> use tar at all ...  Time to deprecate them maybe?  Is anybody actually
>>>> using them?  And while at it, what about the even more obscure 'shar'
>>>> format?
>>>
>>> While I haven't manipulated a shar file in years, but zip is still
>>> the dominant archive format on MS platforms.
>>>
>> While this is absolutely true, my point is that it's not a format truly
>> used or required for distribution tarballs.  If you are going to compile
>> an Automake-based package from source on MS Windows, you'll need either
>> MinGW/MSYS or Cygwin, and AFAICS both those environment comes with
>> working tar and gzip programs.
>>
>> Or is there something that I'm missing?

Yes, I believe quite a few projects have a separately maintained Visual
Studio solution, seeded with handwritten config.h etc, meaning that they
don't require Autotools to build from source on Windows.

I can't give you an example off the top of my head though, but I think
that e.g. ntp is like that (and I don't know if they also provide the
source as a .zip-file...)

Cheers,
Peter

Message #20 received at submit <at> debbugs.gnu.org (full text, mbox):

From: karl <at> freefriends.org (Karl Berry)
To: stefano.lattarini <at> gmail.com
Cc: bug-automake <at> gnu.org, automake <at> gnu.org
Subject: Re: Improvements to "dist" targets (was: Re: EXTRA_DIST, directories, 
	tar --exclude-vcs)
Date: Thu, 3 Jan 2013 00:57:08 GMT

    That is already possible:
    <http://www.gnu.org/software/automake/manual/automake.html#The-dist-Hook>

I see.  Given this, I propose merely changing the definition of am__tar
to use variables.  Something like:

am__tar = $(TAR) $(TAR_OPTIONS) - "$$tardir"

where the actual definitions of TAR and TAR_OPTIONS remain as they are
now (except possibly always using -options, as I'm replying to
separately).  That would make it easy for people to change or add
anything they want to change, in various ways.

I'm not sure if "TAR" and "TAR_OPTIONS" would be the conventional
names; any names would do, as far as I'm concerned.

k

Message #23 received at submit <at> debbugs.gnu.org (full text, mbox):

From: karl <at> freefriends.org (Karl Berry)
To: stefano.lattarini <at> gmail.com
Cc: bug-automake <at> gnu.org, automake <at> gnu.org
Subject: Re: Improvements to "dist" targets (was: Re: EXTRA_DIST, directories, 
	tar --exclude-vcs)
Date: Thu, 3 Jan 2013 00:57:08 GMT

    OTOH, what about distribution "tarballs" in '.zip' format?  They don't
    use tar at all ...  Time to deprecate them maybe?  Is anybody actually
    using them?  And while at it, what about the even more obscure 'shar'
    format?

FWIW, I think they should still be supported.  I see recent
distributions on ftp.gnu.org using both -- gzip and tar make shar
archives for the sake of bootstrapping, and some packages use zip to
make things easier for Windows users.

One could certainly make arguments about getting rid of them (especially
shar), nevertheless.  I personally wouldn't want to spend time engaging
in that debate :).

k

Message #26 received at submit <at> debbugs.gnu.org (full text, mbox):

From: karl <at> freefriends.org (Karl Berry)
To: stefano.lattarini <at> gmail.com
Cc: bug-automake <at> gnu.org, automake <at> gnu.org
Subject: Re: Improvements to "dist" targets (was: Re: EXTRA_DIST, directories, 
	tar --exclude-vcs)
Date: Thu, 3 Jan 2013 00:57:07 GMT

    > that every tar (except maybe really ancient ones, can't remember, but we
    > don't care) supports the -style.

    It would be nice to verify this claim on as much systems as possible

Certainly POSIX has always required supporting -options, which is some
15 years old at least.  Or do I mean 25?  Either way ...  And in
practice, every system I've ever used has a system tar supporting
-options, not just "old-style" options without the -.

Thus I would still propose that Automake always use -options to tar in
its output, never old-style options.

As for tar-v7, tar-ustar, tar-pax ... I don't know.

k

Message #29 received at 13324 <at> debbugs.gnu.org (full text, mbox):

From: Stefano Lattarini <stefano.lattarini <at> gmail.com>
To: Karl Berry <karl <at> freefriends.org>
Cc: 13324 <at> debbugs.gnu.org, automake <at> gnu.org
Subject: Re: bug#13324: Improvements to "dist" targets (was: Re: EXTRA_DIST,
	directories, tar --exclude-vcs)
Date: Thu, 03 Jan 2013 10:16:01 +0100

On 01/03/2013 01:57 AM, Karl Berry wrote:
>     > that every tar (except maybe really ancient ones, can't remember, but we
>     > don't care) supports the -style.
>
>     It would be nice to verify this claim on as much systems as possible
>
> Certainly POSIX has always required supporting -options, which is some
> 15 years old at least.  Or do I mean 25?  Either way ...  And in
> practice, every system I've ever used has a system tar supporting
> -options, not just "old-style" options without the -.
>
> Thus I would still propose that Automake always use -options to tar in
> its output, never old-style options.
>
If nobody objects, I can do that for 1.13.2.

> As for tar-v7, tar-ustar, tar-pax ... I don't know.
>
Normalizing that little mess is a long-term work anyway; first, we need
to either determine that the features offered by those options are not
really relevant today, or have an alternative approach that works in
subsuming all the features offered by those options (albeit likely with
a much different API); then we can go to the route of

   deprecation -> runtime warnings -> runtime error -> removal

Process likely to span several releases.

Regards,
  Stefano

Message #32 received at 13324 <at> debbugs.gnu.org (full text, mbox):

From: Stefano Lattarini <stefano.lattarini <at> gmail.com>
To: Karl Berry <karl <at> freefriends.org>
Cc: 13324 <at> debbugs.gnu.org, automake <at> gnu.org
Subject: Re: bug#13324: Improvements to "dist" targets (was: Re: EXTRA_DIST,
	directories, tar --exclude-vcs)
Date: Thu, 03 Jan 2013 10:20:41 +0100

On 01/03/2013 01:57 AM, Karl Berry wrote:
>     That is already possible:
>     <http://www.gnu.org/software/automake/manual/automake.html#The-dist-Hook>
> 
> I see.  Given this, I propose merely changing the definition of am__tar
> to use variables.  Something like:
> 
> am__tar = $(TAR) $(TAR_OPTIONS) - "$$tardir"
> 
This might not always work, since, when the tar-pax or tar-ustar are used,
the a $(am__tar) is defined at configure time, and might even not invoke
tar at all, going through 'cpio' or 'pax' instead.

> where the actual definitions of TAR and TAR_OPTIONS remain as they are
> now (except possibly always using -options, as I'm replying to
> separately).
>
If we go down the route of using -chf instead of, one could simply
override $TAR altogether, bundling the required options in there, no?
Which is already possible as of today (when 'tar-pax' and 'tar-ustar'
aren't used, of course).  So less changes to implement and to document.

> That would make it easy for people to change or add
> anything they want to change, in various ways.
> 
> I'm not sure if "TAR" and "TAR_OPTIONS" would be the conventional
> names; any names would do, as far as I'm concerned.


Regards,
  Stefano

Message #35 received at 13324 <at> debbugs.gnu.org (full text, mbox):

From: Stefano Lattarini <stefano.lattarini <at> gmail.com>
To: Karl Berry <karl <at> freefriends.org>
Cc: 13324 <at> debbugs.gnu.org, automake <at> gnu.org
Subject: Re: bug#13324: Improvements to "dist" targets (was: Re: EXTRA_DIST,
	directories, tar --exclude-vcs)
Date: Thu, 03 Jan 2013 10:24:39 +0100

On 01/03/2013 01:57 AM, Karl Berry wrote:
>     OTOH, what about distribution "tarballs" in '.zip' format?  They don't
>     use tar at all ...  Time to deprecate them maybe?  Is anybody actually
>     using them?  And while at it, what about the even more obscure 'shar'
>     format?
> 
> FWIW, I think they should still be supported.  I see recent
> distributions on ftp.gnu.org using both -- gzip and tar make shar
> archives for the sake of bootstrapping,
>
Given how "special" particular these packages are, they could easily
implement the shar compression by themselves; Automake will still do
the "heavy lifting" of preparing the distdir (that is the tricky part).

I think it's a no-brainer that tarZ and shar format can be deprecated
ASAP (that is, in 1.13.2) and removed ASAP (that is, in 1.14).

> and some packages use zip to make things easier for Windows users.
>
This might be a valid point.  In a previous mail, I wrote:

  [zip] not a format truly used or required for distribution tarballs.
  If you are going to compile an Automake-based package from source on
  MS Windows, you'll need either MinGW/MSYS or Cygwin, and AFAICS both
  those environment comes with working tar and gzip programs.

but Peter Rosin replied:

  Yes, I believe quite a few projects have a separately maintained Visual
  Studio solution, seeded with handwritten config.h etc, meaning that they
  don't require Autotools to build from source on Windows.

implying that the generated tarball might after all not require a UNIX
emulator to build on windows.  So let's keep zip :-)

> One could certainly make arguments about getting rid of them (especially
> shar), nevertheless.  I personally wouldn't want to spend time engaging
> in that debate :).
>
I see no real debate coming up about the removal of shar and tarZ.
They are really obsolete.  I might be wrong though, and that's why
I'll deprecate them long before actually removing them ;-)

Regards,
  Stefano

Message #38 received at 13324 <at> debbugs.gnu.org (full text, mbox):

From: Stefano Lattarini <stefano.lattarini <at> gmail.com>
To: Peter Rosin <peda <at> lysator.liu.se>
Cc: 13324 <at> debbugs.gnu.org, Daniel Herring <dherring <at> tentpost.com>,
	automake <at> gnu.org
Subject: Re: bug#13324: Improvements to "dist" targets
Date: Thu, 03 Jan 2013 10:25:30 +0100

On 01/02/2013 07:51 PM, Peter Rosin wrote:
>
> Yes, I believe quite a few projects have a separately maintained Visual
> Studio solution, seeded with handwritten config.h etc, meaning that they
> don't require Autotools to build from source on Windows.
>
Right, I didn't think about that possibility.  Glad I asked.

Let's keep zip then ;-)

Stefano

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.