GNU bug report logs - #8362
make install prefix inserted in source code with generated python files

Previous Next

To reply to this bug, email your comments to 8362 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: "A.T.Hofkamp" <a.t.hofkamp <at> tue.nl>
To: bug-automake <at> gnu.org
Subject: make install prefix inserted in source code with generated python
	files
Date: Mon, 28 Mar 2011 09:36:45 +0200

[Message part 1 (text/plain, inline)]

Hello,

As far as I know, "make install prefix=/path/to/writable/dir" should only change the place where 
files are copied to, instead of injecting that new prefix into the source code. However, for Python 
source files that are generated/modified during installation, this does not seem to hold:

------------- Makefile.am
prefix=@prefix@
exec_prefix=@exec_prefix@
libexecdir = @libexecdir@
libdir = @libdir@

PYTHON_VERSION=@PYTHON_VERSION@
SITEPACKAGES=$(libdir)/python$(PYTHON_VERSION)/site-packages
SED=@sed@

prog_PYTHON=prog/__init__.py prog/x.py
progdir = $(SITEPACKAGES)/prog

prog/x.py: x.src
        $(SED) -e "s,%LIBEXEC%,$(libexecdir)," < x.src > prog/x.py
-------------

prog/x.py is generated from x.src by inserting the libexecdir path into it.

During install, the following happens at my system:
(/somewhere is a fake path for demonstration purposes only,
 also replaced irrelevant lengthy output lines by ... )

+ ./configure --prefix=/somewhere
checking for a BSD-compatible install... /usr/bin/install -c
...

+ make
make: Nothing to be done for `all'.

+ make prefix=/home/hat/tmp/p/installed install
...
sed -e "s,%LIBEXEC%,/home/hat/tmp/p/installed/libexec," < x.src > prog/x.py
...


The prog/x.py rule was run during 'make install' rather than 'make'. As a result, the wrong 
libexecdir got included.

Tested with autoconf 2.59, automake 1.9.6
and (by temporary install) autoconf 2.68, automake 1.11.1


Attached a demo directory 'p' that demonstrates the issue.

[p.tar.gz (application/x-tar, inline)]

Message #8 received at 8362 <at> debbugs.gnu.org (full text, mbox):

From: Ralf Wildenhues <Ralf.Wildenhues <at> gmx.de>
To: "A.T.Hofkamp" <a.t.hofkamp <at> tue.nl>
Cc: 8362 <at> debbugs.gnu.org
Subject: Re: bug#8362: make install prefix inserted in source code with
	generated python files
Date: Mon, 28 Mar 2011 19:51:51 +0200

Hello,

* A.T.Hofkamp wrote on Mon, Mar 28, 2011 at 09:36:45AM CEST:
> As far as I know, "make install prefix=/path/to/writable/dir" should
> only change the place where files are copied to, instead of
> injecting that new prefix into the source code. However, for Python
> source files that are generated/modified during installation, this
> does not seem to hold:

> ------------- Makefile.am
> prefix=@prefix@
> exec_prefix=@exec_prefix@
> libexecdir = @libexecdir@
> libdir = @libdir@
> 
> PYTHON_VERSION=@PYTHON_VERSION@
> SITEPACKAGES=$(libdir)/python$(PYTHON_VERSION)/site-packages
> SED=@sed@
> 
> prog_PYTHON=prog/__init__.py prog/x.py
> progdir = $(SITEPACKAGES)/prog
> 
> prog/x.py: x.src
>         $(SED) -e "s,%LIBEXEC%,$(libexecdir)," < x.src > prog/x.py

Looks like it.  Can you work around this issue by adding

all-local: $(prog_PYTHON)

to the Makefile.am?

Thanks for the report,
Ralf

Message #11 received at 8362 <at> debbugs.gnu.org (full text, mbox):

From: "A.T.Hofkamp" <a.t.hofkamp <at> tue.nl>
To: Ralf Wildenhues <Ralf.Wildenhues <at> gmx.de>
Cc: "8362 <at> debbugs.gnu.org" <8362 <at> debbugs.gnu.org>
Subject: Re: bug#8362: make install prefix inserted in source code with
	generated python files
Date: Tue, 29 Mar 2011 08:17:53 +0200

Ralf Wildenhues wrote:
> Looks like it.  Can you work around this issue by adding
> 
> all-local: $(prog_PYTHON)
> 
> to the Makefile.am?

Confirmed that this addition fixes the problem for me, both with the 'p' example and the real case.
Thanks for the quick fix.

Albert

Message #14 received at 8362 <at> debbugs.gnu.org (full text, mbox):

From: Mike Frysinger <vapier <at> gentoo.org>
To: 8362 <at> debbugs.gnu.org
Subject: Re: make install prefix inserted in source code with generated
 python	files
Date: Sun, 20 Feb 2022 19:35:17 +0000 (UTC)

On Mon, 28 Mar 2011 09:36:45 +0200, A.T.Hofkamp wrote:
> As far as I know, "make install prefix=/path/to/writable/dir" should only change the place where 
> files are copied to, instead of injecting that new prefix into the source code. However, for Python 
> source files that are generated/modified during installation, this does not seem to hold:

really looks like you want to be using DESTDIR= instead of prefix=.
i don't think the behavior described here in general is a bug in
Automake.  that said, we should be compiling the script as part
of `all`.
-mike

Message #21 received at 8362 <at> debbugs.gnu.org (full text, mbox):

From: Bogdan <bogdro_rep <at> gmx.us>
To: 8362 <at> debbugs.gnu.org, Mike Frysinger <vapier <at> gentoo.org>
Subject: [bug#8362] make install prefix inserted in source code with generated
 python files
Date: Tue, 21 Mar 2023 23:05:10 +0100

[Message part 1 (text/plain, inline)]

Hi all.

 A small contribution to the discussion.

 First of all, it's not an Automake defect that the user is allowed 
to override variables using the command line. This may be used e.g. 
for changing compile flags at 'make' time:

	make CFLAGS=-Wall

 Second, in the provided example, the 'install' target indeed depends 
on prog_PYTHON, while 'all' doesn't. Somebody made a decision to 
compile Python files at install time and not at compile time. Maybe 
because it is not known at compile time where the files will end up, 
or they have conflicting names, or for some other good reason.

 Third, and most important (I think) is that we need to note that 
"prog/x.py" is GENERATED, but is NOT marked so. Adding

	BUILT_SOURCES = prog/x.py

to Makefile.am fixes the issue immediately. So, sorry to say this, but 
it looks like this defect is invalid. The file prog/x.py is meant to 
be built/generated at build time and must be marked as being built.

 Another point is that one needs to 'touch' x.src each time to test, 
which also is signalling that something may be wrong. But, this may be 
just a side effect of the minimalistic example.

 By the way, if the real code also needs to substitute just 
"$(libexecdir)", which is known at configure time, it should actually 
be 'configure' which generates prog/x.py. This would probably stop 
this issue from appearing in the first place...

 If you really, really want to fix something in Automake, you can use 
the attached patch, it seems to fix the issue. But, this is something 
I don't recommend, because next we'll be adding each and every 
language/script group to the 'all' target and that's probably not the 
point. That's why I'm not formatting the patch as I should be.


-- 
Regards - Bogdan ('bogdro') D.                 (GNU/Linux & FreeDOS)
X86 assembly (DOS, GNU/Linux):    http://bogdro.evai.pl/index-en.php
Soft(EN): http://bogdro.evai.pl/soft  http://bogdro.evai.pl/soft4asm
www.Xiph.org  www.TorProject.org  www.LibreOffice.org  www.GnuPG.org

[automake-python-in-all.diff (text/x-patch, attachment)]

Message #24 received at 8362 <at> debbugs.gnu.org (full text, mbox):

From: Mike Frysinger <vapier <at> gentoo.org>
To: Bogdan <bogdro_rep <at> gmx.us>
Cc: 8362 <at> debbugs.gnu.org
Subject: Re: [bug#8362] make install prefix inserted in source code with
 generated python files
Date: Sat, 13 Jan 2024 01:26:53 -0500

[Message part 1 (text/plain, inline)]

On 21 Mar 2023 23:05, Bogdan wrote:
>   Third, and most important (I think) is that we need to note that 
> "prog/x.py" is GENERATED, but is NOT marked so. Adding
> 
> 	BUILT_SOURCES = prog/x.py

i don't think this is a correct use of BUILT_SOURCES.
https://www.gnu.org/software/automake/manual/automake.html#Sources

that var is meant for generating headers before Automake's depgen logic
can kick in.  that sounds minor, but in practice it means that every
built source is generated before anything else is compiled.  which can
insert a bubble into parallel build pipelines.  so we would prefer it
only be used for things that matter to header dependency generation.
-mike

[signature.asc (application/pgp-signature, inline)]

Message #27 received at 8362 <at> debbugs.gnu.org (full text, mbox):

From: Bogdan <bogdro_rep <at> gmx.us>
To: Mike Frysinger <vapier <at> gentoo.org>
Cc: 8362 <at> debbugs.gnu.org
Subject: Re: [bug#8362] make install prefix inserted in source code with
 generated python files
Date: Sat, 13 Jan 2024 22:29:48 +0100

Mike Frysinger <vapier <at> gentoo.org>, 2024-01-13 07:26:
> On 21 Mar 2023 23:05, Bogdan wrote:
>>    Third, and most important (I think) is that we need to note that
>> "prog/x.py" is GENERATED, but is NOT marked so. Adding
>>
>> 	BUILT_SOURCES = prog/x.py
>
> i don't think this is a correct use of BUILT_SOURCES.
> https://www.gnu.org/software/automake/manual/automake.html#Sources
>
> that var is meant for generating headers before Automake's depgen logic
> can kick in.  that sounds minor, but in practice it means that every
> built source is generated before anything else is compiled.  which can
> insert a bubble into parallel build pipelines.  so we would prefer it
> only be used for things that matter to header dependency generation.
> -mike


 You're probably right. Perhaps not meant for this purpose, but I
think BUILT_SOURCES is the only way to make generated code files
visible to the build system, no matter if for dependencies or not. How
do I e.g. have a .c file generated by 'configure' to be properly built
if I can't list it in Makefile.am (because it's a .c.in file in the
beginning)?
 This was just my simple hack to help the author. There are better
solutions in this case (the ones I wrote later about), and perhaps the
author's build system/Makefile.am's/dependencies should be fixed in
the first place.

--
Regards - Bogdan ('bogdro') D.                 (GNU/Linux & FreeDOS)
X86 assembly (DOS, GNU/Linux):    http://bogdro.evai.pl/index-en.php
Soft(EN): http://bogdro.evai.pl/soft  http://bogdro.evai.pl/soft4asm
www.Xiph.org  www.TorProject.org  www.LibreOffice.org  www.GnuPG.org

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.