GNU bug report logs - #19616
dist tarball contains hardlinks

Previous Next

To reply to this bug, email your comments to 19616 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Dimitrios Apostolou <jimis <at> gmx.net>
To: bug-automake <at> gnu.org, bug-tar <at> gnu.org
Subject: dist tarball contains hardlinks
Date: Fri, 16 Jan 2015 16:43:20 +0100 (CET)

(Cross posting to both bug-automake and bug-tar as I can see both projects 
being affected.)
(Please keep me CC'd as I'm not subscribed.)

Hello,

this bug report is because of the following messages when extracting a 
source tarball generated with "make dist". You can read the full bug 
report at https://dev.cfengine.com/issues/6925.

# tar -xzf core/cfengine-3.7.0a1.5ffcc54.tar.gz
tar: cfengine-3.7.0a1.5ffcc54/tests/acceptance/17_users/unsafe/10_modify_user_with_same_password.cf: Cannot hard link to `cfengine-3.7.0a1.5ffcc54/tests/acceptance/17_users/unsafe/20_modify_user_with_same_password_hpux_tru': No such file or directory
tar: cfengine-3.7.0a1.5ffcc54/tests/acceptance/17_users/unsafe/10_modify_user_with_many_attributes_warn.cf: Cannot hard link to `cfengine-3.7.0a1.5ffcc54/tests/acceptance/17_users/unsafe/20_modify_user_with_many_attributes_warn_h': No such file or directory
tar: cfengine-3.7.0a1.5ffcc54/tests/acceptance/17_users/unsafe/10_add_user_with_many_attributes_warn.cf: Cannot hard link to `cfengine-3.7.0a1.5ffcc54/tests/acceptance/17_users/unsafe/20_add_user_with_many_attributes_warn_hpux': No such file or directory
tar: cfengine-3.7.0a1.5ffcc54/tests/acceptance/17_users/unsafe/10_modify_user_lock_with_password.cf: Cannot hard link to `cfengine-3.7.0a1.5ffcc54/tests/acceptance/17_users/unsafe/20_modify_user_lock_with_password_hpux_tru': No such file or directory
tar: cfengine-3.7.0a1.5ffcc54/tests/acceptance/17_users/unsafe/20_newly_created_account_should_not_count_as_locked_hpux_trusted.cf: Cannot hard link to `cfengine-3.7.0a1.5ffcc54/tests/acceptance/17_users/unsafe/10_newly_created_account_should_not_count_': No such file or directory
tar: cfengine-3.7.0a1.5ffcc54/tests/acceptance/17_users/unsafe/10_modify_user_unlock_with_password.cf: Cannot hard link to `cfengine-3.7.0a1.5ffcc54/tests/acceptance/17_users/unsafe/20_modify_user_unlock_with_password_hpux_t': No such file or directory
tar: cfengine-3.7.0a1.5ffcc54/tests/acceptance/17_users/unsafe/10_add_user_with_hashed_password.cf: Cannot hard link to `cfengine-3.7.0a1.5ffcc54/tests/acceptance/17_users/unsafe/20_add_user_with_hashed_password_hpux_trus': No such file or directory
tar: Error exit delayed from previous errors


Regading automake, it implements "make dist" using "tar -chf" where -h 
stands for --dereference, i.e. it dereferences all symlinks. I believe 
I've read that it is for portability, because symlinks are handled in a 
variety of manners in different tar implementations. However as seen 
above, what actually happens is that GNU tar stores the symlinks as 
hardlinks. Because the total path length of the hardlinks is more than 100 
characters, they get truncated due to "ustar" format limitations (ustar 
format is chosen using the "tar-ustar" automake option).

Please notice that removing "-h" fixes the problem. Symlinks are stored 
properly as their length is much smaller than 100 chars (because they are 
relative symlinks). Another workaround (the one I chose to use) is to pass 
the --hard-dereference to GNU tar in order to store the content of the 
files itself, avoiding all kind of links, and maintaining best 
portability. So I've added the following lines in my Makefile.am:

TAR_OPTIONS = --hard-dereference
export TAR_OPTIONS


Regarding GNU tar, I've read at [1] and [2] that this behaviour was 
introduced recently, but is the intended behaviour.

[1] https://bugs.launchpad.net/ubuntu/+source/tar/+bug/887107
[2] http://lists.gnu.org/archive/html/bug-tar/2011-11/msg00025.html

My opinion is that it's really deceiving, given what the man page 
mentions: "follow symlinks; archive and dump the files they point to". It 
took me a while to figure out what those "hard link" messages mean, 
looking if I introduced hard links by accident. And since the hard links 
have no direction, some messages show the linking direction reversed, 
which made the issue even stranger.

Why is such behaviour desirable? Are the reasons important enough to break 
expectations of users?


Thank you in advance,
Dimitris

Message #8 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Dimitrios Apostolou <jimis <at> gmx.net>, bug-automake <at> gnu.org, bug-tar <at> gnu.org
Subject: Re: [Bug-tar] dist tarball contains hardlinks
Date: Fri, 16 Jan 2015 18:47:36 -0800

Dimitrios Apostolou wrote:
> Why is such behaviour desirable?

It's more logical, since it causes tar to behave as if the symlink were not 
there, and the pointed-to file was there instead.

Using -hard-dereference bloats the tar image, but if that's a price you're 
willing to pay then you have a solution to the problem.

Message #11 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Dimitrios Apostolou <jimis <at> gmx.net>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: bug-automake <at> gnu.org, bug-tar <at> gnu.org
Subject: Re: [Bug-tar] dist tarball contains hardlinks
Date: Sun, 18 Jan 2015 01:10:14 +0100 (CET)

On Fri, 16 Jan 2015, Paul Eggert wrote:

> Dimitrios Apostolou wrote:
>>  Why is such behaviour desirable?
>
> It's more logical, since it causes tar to behave as if the symlink were not 
> there, and the pointed-to file was there instead.

But when the tarball is extracted, two files with same inode are created, 
which is kind of unexpected behaviour - at least for me - after creating 
the tarball with --dereference, i.e. I've explicitly asked to follow 
symlinks. And it beats the purpose some users have been using 
--dereference - to avoid storing links which have more restrictions (e.g. 
100 char limit).

> Using -hard-dereference bloats the tar image, but if that's a price you're 
> willing to pay then you have a solution to the problem.

Indeed that's what I chose as workaround, compression deals well with 
duplicated information so size bloating was minimal.


Thanks,
Dimitris

Message #14 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Dimitrios Apostolou <jimis <at> gmx.net>
Cc: bug-automake <at> gnu.org, bug-tar <at> gnu.org
Subject: Re: [Bug-tar] dist tarball contains hardlinks
Date: Sun, 18 Jan 2015 08:04:25 -0800

Dimitrios Apostolou wrote:
> But when the tarball is extracted, two files with same inode are created, which
> is kind of unexpected behaviour - at least for me

Other utilities have similar behavior (e.g., ls, cp, du), in that they pretend 
the symlink isn't there and behave as if the pointed-to inode is there directly. 
 Tar's current behavior is compatible with these other utilities, whereas its 
old behavior was incompatible.

Message #17 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Joerg Schilling <Joerg.Schilling <at> fokus.fraunhofer.de>
To: <jimis <at> gmx.net>, <eggert <at> cs.ucla.edu>
Cc: bug-automake <at> gnu.org, bug-tar <at> gnu.org
Subject: Re: [Bug-tar] dist tarball contains hardlinks
Date: Mon, 19 Jan 2015 10:38:08 +0100

Dimitrios Apostolou <jimis <at> gmx.net> wrote:

> On Fri, 16 Jan 2015, Paul Eggert wrote:
>
> > Dimitrios Apostolou wrote:
> >>  Why is such behaviour desirable?
> >
> > It's more logical, since it causes tar to behave as if the symlink were not 
> > there, and the pointed-to file was there instead.
>
> But when the tarball is extracted, two files with same inode are created, 
> which is kind of unexpected behaviour - at least for me - after creating 

The problem is not the tarball itself but the fact that someone called

	tar ch ...

This is not the right way to go for collecting files for a distribution.

In former times, I distributed symlinks with my distro tarballs but now I 
include shell scripts that create the symlinks if needed and on platforms that 
do not support symlinks, these shell scripts automatically try to create 
hardlinks.

Maybe you should contact the respobsible person for your distro tarball.

Jörg

-- 
 EMail:joerg <at> schily.net                    (home) Jörg Schilling D-13353 Berlin
       joerg.schilling <at> fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/
 URL:  http://cdrecord.org/private/ http://sourceforge.net/projects/schilytools/files/'

Message #20 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Pavel Raiskup <praiskup <at> redhat.com>
To: bug-tar <at> gnu.org
Cc: Dimitrios Apostolou <jimis <at> gmx.net>, bug-automake <at> gnu.org
Subject: Re: [Bug-tar] dist tarball contains hardlinks
Date: Wed, 21 Jan 2015 14:35:26 +0100

On Friday 16 of January 2015 16:43:20 Dimitrios Apostolou wrote:
> (Cross posting to both bug-automake and bug-tar as I can see both projects
> being affected.)
> (Please keep me CC'd as I'm not subscribed.)
>
> Hello,
>
> this bug report is because of the following messages when extracting a
> source tarball generated with "make dist". You can read the full bug
> report at https://dev.cfengine.com/issues/6925.
>
> # tar -xzf core/cfengine-3.7.0a1.5ffcc54.tar.gz
> tar: cfengine-3.7.0a1.5ffcc54/tests/acceptance/17_users/unsafe/10_modify_user_with_same_password.cf: Cannot hard link to `cfengine-3.7.0a1.5ffcc54/tests/acceptance/17_users/unsafe/20_modify_user_with_same_password_hpux_tru': No such file or directory
> [...]
>
> Regading automake, it implements "make dist" using "tar -chf" where -h
> stands for --dereference, i.e. it dereferences all symlinks. I believe
> I've read that it is for portability, because symlinks are handled in a
> variety of manners in different tar implementations. However as seen
> above, what actually happens is that GNU tar stores the symlinks as
> hardlinks. Because the total path length of the hardlinks is more than 100
> characters, they get truncated due to "ustar" format limitations (ustar
> format is chosen using the "tar-ustar" automake option).

When using automake, try to use:

  AM_INIT_AUTOMAKE([filename-length-max=99])

That breaks 'make dist' as you would expect.  Maybe this could be done by
default for certain archive formats (or make dist could be silenced a
little to see tar's warnings).

> Please notice that removing "-h" fixes the problem. Symlinks are stored
> properly as their length is much smaller than 100 chars (because they are
> relative symlinks).

Not much - when 'tar chf -' (and in your example) the symlinks are
probably just (with a bit of luck) a bit smaller than not archived files.

> Another workaround (the one I chose to use) is to pass the
> --hard-dereference to GNU tar in order to store the content of the files
> itself, avoiding all kind of links, and maintaining best portability. So
> I've added the following lines in my Makefile.am:

Also, when fighting with 'tar chf -', you may AC_SUBST([am__tar]) to whatever
you want.  This would replace what is automatically detected by m4/tar.m4.

Pavel

Message #23 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Dimitrios Apostolou <jimis <at> gmx.net>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: bug-automake <at> gnu.org, bug-tar <at> gnu.org
Subject: Re: [Bug-tar] dist tarball contains hardlinks
Date: Fri, 23 Jan 2015 15:44:26 +0100 (CET)

On Sun, 18 Jan 2015, Paul Eggert wrote:

> Dimitrios Apostolou wrote:
>> But when the tarball is extracted, two files with same inode are created, 
>> which
>> is kind of unexpected behaviour - at least for me
>
> Other utilities have similar behavior (e.g., ls, cp, du), in that they 
> pretend the symlink isn't there and behave as if the pointed-to inode is 
> there directly.  Tar's current behavior is compatible with these other 
> utilities, whereas its old behavior was incompatible.

Well out of the utilities you mentioned, only "cp" actually writes 
something, so it's related to the issue with tar. So I tried the behaviour 
of --dereference in cp:

$ touch blah
$ ln -s blah link_to_blah
$ cp --dereference link_to_blah blah2

The result is that "blah" and "blah2" have different inode! So 
--dereference causes the symlink to be dereferenced to a separate file, 
*not* to a hardlink of the source.

But in tar, --dereference causes a symlink to be dereferenced as a 
hardlink to the source file. So how is their behaviour consistent? BTW, my 
version of "cp" is:

$ cp --version
cp (GNU coreutils) 8.21


Thanks,
Dimitris

Message #26 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Dimitrios Apostolou <jimis <at> gmx.net>
To: Joerg Schilling <Joerg.Schilling <at> fokus.fraunhofer.de>
Cc: eggert <at> cs.ucla.edu, bug-automake <at> gnu.org, bug-tar <at> gnu.org
Subject: Re: [Bug-tar] dist tarball contains hardlinks
Date: Fri, 23 Jan 2015 15:45:57 +0100 (CET)

On Mon, 19 Jan 2015, Joerg Schilling wrote:
> Dimitrios Apostolou <jimis <at> gmx.net> wrote:
>
>> On Fri, 16 Jan 2015, Paul Eggert wrote:
>>
>>> Dimitrios Apostolou wrote:
>>>>  Why is such behaviour desirable?
>>>
>>> It's more logical, since it causes tar to behave as if the symlink were not
>>> there, and the pointed-to file was there instead.
>>
>> But when the tarball is extracted, two files with same inode are created,
>> which is kind of unexpected behaviour - at least for me - after creating
>
> The problem is not the tarball itself but the fact that someone called
>
> 	tar ch ...
>
> This is not the right way to go for collecting files for a distribution.
>
> In former times, I distributed symlinks with my distro tarballs but now I
> include shell scripts that create the symlinks if needed and on platforms that
> do not support symlinks, these shell scripts automatically try to create
> hardlinks.
>
> Maybe you should contact the respobsible person for your distro tarball.

Thank you Joerg. If so then it is an issue that must be fixed in automake, 
which is the reason I cross-posted to both projects, because I am not sure 
which one should be changed!

Dimitris

Message #29 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Dimitrios Apostolou <jimis <at> gmx.net>
Cc: bug-automake <at> gnu.org, bug-tar <at> gnu.org
Subject: Re: [Bug-tar] dist tarball contains hardlinks
Date: Fri, 23 Jan 2015 08:19:55 -0800

Dimitrios Apostolou wrote:
> $ touch blah
> $ ln -s blah link_to_blah
> $ cp --dereference link_to_blah blah2
>
> The result is that "blah" and "blah2" have different inode!

Of course, and that's what one would expect, cp makes copies by default, and the 
copies have a different inode from the original.

I was referring to something like this:

touch blah
ln -s blah link_to_blah
mkdir dir
cp -a --dereference blah link_to_blah dir

This creates two hard links in dir, to the same new file.  ("-a" attempts to 
make "cp" act as much like "tar" as it can.)

Message #32 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Pavel Raiskup <praiskup <at> redhat.com>
To: bug-tar <at> gnu.org
Cc: Joerg Schilling <Joerg.Schilling <at> fokus.fraunhofer.de>,
 Dimitrios Apostolou <jimis <at> gmx.net>, eggert <at> cs.ucla.edu, bug-automake <at> gnu.org
Subject: Re: [Bug-tar] dist tarball contains hardlinks
Date: Fri, 23 Jan 2015 19:39:22 +0100

On Friday 23 of January 2015 15:45:57 Dimitrios Apostolou wrote:
> Thank you Joerg. If so then it is an issue that must be fixed in
> automake, which is the reason I cross-posted to both projects, because I
> am not sure which one should be changed!

What solution you exactly propose?  Note that 'h' tar option is used for
dist from Y1996 commit [1] and its removal is unlikely.  It is quite
portable and it, as the "default", works for many packages.

What is not clear to me also is your motivation to change GNU tar to
behave differently, or to change something so old in automake.  IMO, the
reason why you hit this issue is not the "hardlinking" change itself, but
rather just small mistake of the maintainer who did not read STDERR tar's
warnings while doing 'make dist'.  We could make those warnings more
obvious..  is that what you want?

[1] http://git.savannah.gnu.org/cgit/automake.git/commit/?id=7930983ab1c620

Pavel

Message #35 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Dimitrios Apostolou <jimis <at> gmx.net>
To: Pavel Raiskup <praiskup <at> redhat.com>
Cc: Joerg Schilling <Joerg.Schilling <at> fokus.fraunhofer.de>, eggert <at> cs.ucla.edu,
 bug-automake <at> gnu.org, bug-tar <at> gnu.org
Subject: Re: [Bug-tar] dist tarball contains hardlinks
Date: Mon, 26 Jan 2015 12:12:09 +0100 (CET)

On Fri, 23 Jan 2015, Pavel Raiskup wrote:

> On Friday 23 of January 2015 15:45:57 Dimitrios Apostolou wrote:
>> Thank you Joerg. If so then it is an issue that must be fixed in
>> automake, which is the reason I cross-posted to both projects, because I
>> am not sure which one should be changed!
>
> What solution you exactly propose?  Note that 'h' tar option is used for

Maybe adding --hard-dereference? That's how I worked around the issue at 
least.

> dist from Y1996 commit [1] and its removal is unlikely.  It is quite
> portable and it, as the "default", works for many packages.

It has been there since 1996, so the legacy behaviour of tar has been 
battle tested. However the new behaviour (hard links in place of symlinks) 
is probably only tested for a couple of years, and I've seen complaints 
online.

> What is not clear to me also is your motivation to change GNU tar to
> behave differently, or to change something so old in automake.  IMO, the

Using "tar-ustar" format for distribution you are supposed to raise some 
of the limits of the "v7" tar format but retain portability. Indeed, the 
length limit for regular files used to be 256 characters [2], and since 
all symlinks were stored as files that was universal. Now that symlinks 
are stored as hard links, the limit is 100 characters for those, thus I've 
hit this problem.

[2] http://www.gnu.org/software/tar/manual/html_section/tar_68.html

> reason why you hit this issue is not the "hardlinking" change itself, but
> rather just small mistake of the maintainer who did not read STDERR tar's
> warnings while doing 'make dist'.  We could make those warnings more
> obvious..  is that what you want?

I see this as a separate issue, and my opinion is that automake should 
fail "make dist" when tar returns an error. I've filed a separate bug 
report for that [3].

[3] https://lists.gnu.org/archive/html/bug-automake/2015-01/msg00019.html


Dimitris

>
> [1] http://git.savannah.gnu.org/cgit/automake.git/commit/?id=7930983ab1c620
>
> Pavel
>
>

Message #38 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Pavel Raiskup <praiskup <at> redhat.com>
To: bug-tar <at> gnu.org
Cc: Joerg Schilling <Joerg.Schilling <at> fokus.fraunhofer.de>,
 Dimitrios Apostolou <jimis <at> gmx.net>, eggert <at> cs.ucla.edu, bug-automake <at> gnu.org
Subject: Re: [Bug-tar] dist tarball contains hardlinks
Date: Tue, 27 Jan 2015 21:49:37 +0100

On Monday 26 of January 2015 12:12:09 Dimitrios Apostolou wrote:
> On Fri, 23 Jan 2015, Pavel Raiskup wrote:
>
> > On Friday 23 of January 2015 15:45:57 Dimitrios Apostolou wrote:
> >> Thank you Joerg. If so then it is an issue that must be fixed in
> >> automake, which is the reason I cross-posted to both projects, because I
> >> am not sure which one should be changed!
> >
> > What solution you exactly propose?  Note that 'h' tar option is used for
>
> Maybe adding --hard-dereference? That's how I worked around the issue at
> least.

From the automake POV, this is not standardized option (could be used only
for GNU tar if present though).

> > dist from Y1996 commit [1] and its removal is unlikely.  It is quite
> > portable and it, as the "default", works for many packages.
>
> It has been there since 1996, so the legacy behaviour of tar has been
> battle tested. However the new behaviour (hard links in place of
> symlinks) is probably only tested for a couple of years,

Truth.  The new behavior is in tar from 1.24.  However, if we fixed this
in tar, GNU tar v1.24 to v1.28 still would not able to mimic the old
behavior.

> and I've seen complaints online.

There were posted some complaints here on the list, but these seem to be
not so serious to revert that behavior or add yet another option (like
mentioned in [1] thread).

> > What is not clear to me also is your motivation to change GNU tar to
> > behave differently, or to change something so old in automake.  IMO,
> > the
>
> Using "tar-ustar" format for distribution you are supposed to raise some
> of the limits of the "v7" tar format but retain portability. Indeed, the
> length limit for regular files used to be 256 characters [2], and since
> all symlinks were stored as files that was universal. Now that symlinks
> are stored as hard links, the limit is 100 characters for those, thus
> I've hit this problem.
>
> [2] http://www.gnu.org/software/tar/manual/html_section/tar_68.html

Yes, thanks - I see now.  In ustar link name must fit into 100 characters
and the file path can be theoretically max 256 (if it is splittable
correctly into prefix/name).

You seem to have *directory* in some distribution sensitive variable,
like:

  # Directory contains 'file' and 'link -> file'.
  EXTRA_DIST = directory

.. Because if you had:

  EXTRA_DIST = directory/file directory/link

then automake (via 'cp -p') would follow the link for you.  In case of
directories (before the actual 'tar ch' happens) automake copies the
directory recursively by 'cp -fpR' [2] into temporary directory.

Could we add '-L' option to this cp call (dist target is not supposed to
be portable to everywhere and -L is POSIX)?  Obviously, the archive bloat
is not something automake tries to avoid.

Anyway, I don't feel the handling of symbolic-links this way is correct.
Don't we wan't to use something like $LN_S [autoconf manual] in such
cases?

[1] https://lists.gnu.org/archive/html/bug-tar/2011-01/msg00000.html
[2] http://git.savannah.gnu.org/cgit/automake.git/tree/lib/am/distdir.am?id=6357a630dc#n193

Pavel

Message #41 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Joerg Schilling <Joerg.Schilling <at> fokus.fraunhofer.de>
To: <praiskup <at> redhat.com>, <bug-tar <at> gnu.org>
Cc: jimis <at> gmx.net, eggert <at> cs.ucla.edu, bug-automake <at> gnu.org
Subject: Re: [Bug-tar] dist tarball contains hardlinks
Date: Wed, 28 Jan 2015 12:44:22 +0100

Pavel Raiskup <praiskup <at> redhat.com> wrote:

> On Monday 26 of January 2015 12:12:09 Dimitrios Apostolou wrote:
> > On Fri, 23 Jan 2015, Pavel Raiskup wrote:
> >
> > > On Friday 23 of January 2015 15:45:57 Dimitrios Apostolou wrote:
> > >> Thank you Joerg. If so then it is an issue that must be fixed in
> > >> automake, which is the reason I cross-posted to both projects, because I
> > >> am not sure which one should be changed!
> > >
> > > What solution you exactly propose?  Note that 'h' tar option is used for
> >
> > Maybe adding --hard-dereference? That's how I worked around the issue at
> > least.
>
> From the automake POV, this is not standardized option (could be used only
> for GNU tar if present though).

BTW: the german translation of the help output for that gtar option is rubbish.

Jörg

-- 
 EMail:joerg <at> schily.net                    (home) Jörg Schilling D-13353 Berlin
       joerg.schilling <at> fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/
 URL:  http://cdrecord.org/private/ http://sourceforge.net/projects/schilytools/files/'

Message #44 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Dimitrios Apostolou <jimis <at> gmx.net>
To: Pavel Raiskup <praiskup <at> redhat.com>
Cc: Joerg Schilling <Joerg.Schilling <at> fokus.fraunhofer.de>, eggert <at> cs.ucla.edu,
 bug-automake <at> gnu.org, bug-tar <at> gnu.org
Subject: Re: [Bug-tar] dist tarball contains hardlinks
Date: Fri, 13 Mar 2015 16:33:37 +0100 (CET)

On Tue, 27 Jan 2015, Pavel Raiskup wrote:
> On Monday 26 of January 2015 12:12:09 Dimitrios Apostolou wrote:
>> On Fri, 23 Jan 2015, Pavel Raiskup wrote:
>>
>>> On Friday 23 of January 2015 15:45:57 Dimitrios Apostolou wrote:
>>>> Thank you Joerg. If so then it is an issue that must be fixed in
>>>> automake, which is the reason I cross-posted to both projects, because I
>>>> am not sure which one should be changed!
>>>
>>> What solution you exactly propose?  Note that 'h' tar option is used for
>>
>> Maybe adding --hard-dereference? That's how I worked around the issue at
>> least.
>
> From the automake POV, this is not standardized option (could be used only
> for GNU tar if present though).

You are right. I /thought/ I had resolved the problem by exporting 
TAR_OPTIONS=--hard-dereference. Unfortunately the problem persists, "make 
dist" fails on RHEL4 for example because tar sees this unrecognized 
option.

>
>>> dist from Y1996 commit [1] and its removal is unlikely.  It is quite
>>> portable and it, as the "default", works for many packages.
>>
>> It has been there since 1996, so the legacy behaviour of tar has been
>> battle tested. However the new behaviour (hard links in place of
>> symlinks) is probably only tested for a couple of years,
>
> Truth.  The new behavior is in tar from 1.24.  However, if we fixed this
> in tar, GNU tar v1.24 to v1.28 still would not able to mimic the old
> behavior.

I believe my only choice at this point is to forbid "make dist" if tar is 
detected to be old (less than 1.29), and keep doing --hard-dereference. I 
open to new ideas though. :-)

>>
>> Using "tar-ustar" format for distribution you are supposed to raise some
>> of the limits of the "v7" tar format but retain portability. Indeed, the
>> length limit for regular files used to be 256 characters [2], and since
>> all symlinks were stored as files that was universal. Now that symlinks
>> are stored as hard links, the limit is 100 characters for those, thus
>> I've hit this problem.
>>
>> [2] http://www.gnu.org/software/tar/manual/html_section/tar_68.html
>
> Yes, thanks - I see now.  In ustar link name must fit into 100 characters
> and the file path can be theoretically max 256 (if it is splittable
> correctly into prefix/name).
>
> You seem to have *directory* in some distribution sensitive variable,
> like:
>
>  # Directory contains 'file' and 'link -> file'.
>  EXTRA_DIST = directory
>
> .. Because if you had:
>
>  EXTRA_DIST = directory/file directory/link
>
> then automake (via 'cp -p') would follow the link for you.  In case of
> directories (before the actual 'tar ch' happens) automake copies the
> directory recursively by 'cp -fpR' [2] into temporary directory.

I don't think this solution is worth the trouble in my case, since the 
directory included has a deep and complex structure (hence some files 
surpass 100 characters).


Thanks,
Dimitris


>
> Could we add '-L' option to this cp call (dist target is not supposed to
> be portable to everywhere and -L is POSIX)?  Obviously, the archive bloat
> is not something automake tries to avoid.
>
> Anyway, I don't feel the handling of symbolic-links this way is correct.
> Don't we wan't to use something like $LN_S [autoconf manual] in such
> cases?
>
> [1] https://lists.gnu.org/archive/html/bug-tar/2011-01/msg00000.html
> [2] http://git.savannah.gnu.org/cgit/automake.git/tree/lib/am/distdir.am?id=6357a630dc#n193
>
> Pavel
>
>

Message #47 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Dimitrios Apostolou <jimis <at> gmx.net>
To: bug-automake <at> gnu.org
Cc: Joerg Schilling <Joerg.Schilling <at> fokus.fraunhofer.de>,
 Pavel Raiskup <praiskup <at> redhat.com>, eggert <at> cs.ucla.edu, bug-tar <at> gnu.org
Subject: Re: [Bug-tar] dist tarball contains hardlinks
Date: Mon, 16 Mar 2015 13:29:21 +0100 (CET)

[Message part 1 (text/plain, inline)]

On Fri, 13 Mar 2015, Dimitrios Apostolou wrote:
>
> I believe my only choice at this point is to forbid "make dist" if tar is 
> detected to be old (less than 1.29), and keep doing --hard-dereference. I 
> open to new ideas though. :-)

The attached patch solves the problem in a more elegant manner IMHO, 
so I plan to include this modified tar.m4 in my project.

Review is welcome, and feel free to include to upstream automake.


Dimitris

[0001-GNU-Tar-now-uses-hard-dereference-when-possible.patch (text/x-diff, attachment)]

Message #50 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Dimitrios Apostolou <jimis <at> gmx.net>
To: bug-automake <at> gnu.org
Cc: Joerg Schilling <Joerg.Schilling <at> fokus.fraunhofer.de>,
 Pavel Raiskup <praiskup <at> redhat.com>, eggert <at> cs.ucla.edu, bug-tar <at> gnu.org
Subject: [automake patch] use --hard-dereference with GNU Tar (was: dist
 tarball contains hardlinks)
Date: Mon, 16 Mar 2015 16:46:24 +0100 (CET)

[Message part 1 (text/plain, inline)]

On Mon, 16 Mar 2015, Dimitrios Apostolou wrote:
> On Fri, 13 Mar 2015, Dimitrios Apostolou wrote:
>> 
>> I believe my only choice at this point is to forbid "make dist" if tar is 
>> detected to be old (less than 1.29), and keep doing --hard-dereference. I 
>> open to new ideas though. :-)
>
> The attached patch solves the problem in a more elegant manner IMHO, so I 
> plan to include this modified tar.m4 in my project.
>
> Review is welcome, and feel free to include to upstream automake.

Please ignore the previous patch. The one attached here has prettier 
printing and also works with GNU tar versions that don't return 64 in case 
of unrecognized option.


Thanks,
Dimitris

[0001-GNU-Tar-now-uses-hard-dereference-when-possible.patch (text/x-diff, attachment)]

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.