GNU bug report logs - #15831
Absolute library paths with Cygwin + MSVC cl

Previous Next

To reply to this bug, email your comments to 15831 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: "Tony Kelman" <kelman <at> berkeley.edu>
To: <bug-libtool <at> gnu.org>
Subject: Absolute library paths with Cygwin + MSVC cl
Date: Thu, 7 Nov 2013 20:12:38 -0800

Hi all,

I'm working with a set of projects that save their inter-dependencies in
AC_SUBST variables as absolute paths to the relevant .la files. In Cygwin
using the MSVC cl compiler and building only static libraries, I'm seeing
libtool replace the Unix-style absolute path to .la files with the
Unix-style absolute path to the corresponding .libs/<libname>.lib file.
Unsurprisingly, the cl compiler doesn't understand the Unix-style path.
If I modify the Makefile to add `cygpath -w ...` around the absolute path
to the .la file, then libtool instead replaces the Windows-style absolute
path to the .la file with just ./.libs/<libname>.lib, which isn't right
since I'm in a different directory than that library was built in.

My test case here is the COIN-OR Open Solver Interface project
(https://projects.coin-or.org/Osi/wiki) release 0.106.2, but updated to a
recent set of autotools by svn switch'ing the BuildTools directory to
https://projects.coin-or.org/BuildTools/browser/branches/autotools-update.
The same behavior described below happens with the very old libtool
(1.5.22) distributed with the release, but I updated to test whether the
problem still exists in recent libtool. Started Cygwin (64-bit) from the
"VS Command Prompt" to set the right environment variables, and configured
Osi with --enable-doscompile=msvc to use cl compiler.

The problem comes at make test. With the Unix-style absolute path to the
.la file in the variable OSILIB_LIBS, the problem invocation looks like:

/bin/sh ../libtool --tag=CXX -n -v  --mode=link cl  -MT -O2 -nologo -EHsc
 -GR -wd4996 -D_CRT_SECURE_NO_DEPRECATE -DNDEBUG    -DOSI_BUILD   -o
 unitTest.exe unitTest.obj OsiTestSolver.obj OsiTestSolverInterface.obj
 OsiTestSolverInterfaceIO.obj OsiTestSolverInterfaceTest.obj
 ../src/OsiCommonTest/libOsiCommonTests.la       ../src/Osi/libOsi.la
 /home/Tony/Osi-0.106.2/build02/CoinUtils/src/libCoinUtils.la
libtool: link: cl -MT -O2 -nologo -EHsc -GR -wd4996
 -D_CRT_SECURE_NO_DEPRECATE -DNDEBUG -DOSI_BUILD -o unitTest.exe
 unitTest.obj OsiTestSolver.obj OsiTestSolverInterface.obj
 OsiTestSolverInterfaceIO.obj OsiTestSolverInterfaceTest.obj
 ../src/OsiCommonTest/.libs/OsiCommonTests.lib ../src/Osi/.libs/Osi.lib
 /home/Tony/Osi-0.106.2/build02/CoinUtils/src/.libs/libCoinUtils.lib
libtool: link: lt_outputfile="unitTest.exe"
libtool: link:  lt_tool_outputfile="unitTest.exe"
libtool: link:  case  in *.exe|*.EXE) ;; *) lt_outputfile=".exe"
 lt_tool_outputfile=".exe" ;; esac
libtool: link:  func_to_tool_file ""
libtool: link:  if test "mt" != ":" && test -f ".manifest"; then mt
 -manifest ".manifest" -outputresource:"" || exit 1; rm -f ".manifest"; fi

If I take out -n from AM_LIBTOOLFLAGS then I get hundreds of unresolved
externals to everything from libCoinUtils, since cl ignored the "unknown
option" of the Unix-style absolute path to libCoinUtils.la.

Adding `cygpath -w ...` around the absolute path to libCoinUtils.la in the
Osi/test/Makefile variable OSILIB_LIBS results in the following:

/bin/sh ../libtool --tag=CXX -v  --mode=link cl  -MT -O2 -nologo -EHsc
 -GR -wd4996 -D_CRT_SECURE_NO_DEPRECATE -DNDEBUG    -DOSI_BUILD   -o
 unitTest.exe unitTest.obj OsiTestSolver.obj OsiTestSolverInterface.obj
 OsiTestSolverInterfaceIO.obj OsiTestSolverInterfaceTest.obj
 ../src/OsiCommonTest/libOsiCommonTests.la       ../src/Osi/libOsi.la
 `cygpath -w /home/Tony/Osi-0.106.2/build02/CoinUtils/src/libCoinUtils.la`
libtool: link: cl -MT -O2 -nologo -EHsc -GR -wd4996
 -D_CRT_SECURE_NO_DEPRECATE -DNDEBUG -DOSI_BUILD -o unitTest.exe
 unitTest.obj OsiTestSolver.obj OsiTestSolverInterface.obj
 OsiTestSolverInterfaceIO.obj OsiTestSolverInterfaceTest.obj
 ../src/OsiCommonTest/.libs/OsiCommonTests.lib ../src/Osi/.libs/Osi.lib
 ./.libs/libCoinUtils.lib
cl : Command line warning D9035 : option 'o' has been deprecated and will
 be removed in a future release
LINK : fatal error LNK1181: cannot open input file 
'./.libs/libCoinUtils.lib'
libtool: link: lt_outputfile="unitTest.exe"
libtool: link:  lt_tool_outputfile="unitTest.exe"
libtool: link:  case unitTest.exe in *.exe|*.EXE) ;; *)
 lt_outputfile="unitTest.exe.exe" lt_tool_outputfile="unitTest.exe.exe" ;; 
esac
libtool: link:  func_to_tool_file "unitTest.exe"
libtool: link:  if test "mt" != ":" && test -f "unitTest.exe.manifest";
 then mt -manifest "unitTest.exe.manifest" -outputresource:"unitTest.exe"
 || exit 1; rm -f "unitTest.exe.manifest"; fi
Makefile:476: recipe for target 'unitTest.exe' failed
make: *** [unitTest.exe] Error 2

Is this a bug in libtool, or am I doing something wrong/unsupported? MSYS
works better here, but it would be ideal to be able to use Cygwin as well.

Relevant info from the end of libtool --help:

host-triplet:    x86_64-unknown-cygwin
shell:        /bin/sh
compiler:        cl
compiler 
flags:        -MT -O2 -nologo -wd4996 -D_CRT_SECURE_NO_DEPRECATE -DNDEBUG
linker:        link (gnu? no)
libtool:    (GNU libtool) 2.4.2
automake:
autoconf:    autoconf (GNU Autoconf) 2.69

Automake was version 1.12, not sure why libtool didn't see that.

Appreciate any advice here, let me know if there's any more information I
can provide that would be useful.

-Tony

Message #8 received at 15831 <at> debbugs.gnu.org (full text, mbox):

From: Peter Rosin <peda <at> lysator.liu.se>
To: Tony Kelman <kelman <at> berkeley.edu>, 15831 <at> debbugs.gnu.org
Subject: Re: bug#15831: Absolute library paths with Cygwin + MSVC cl
Date: Fri, 08 Nov 2013 15:46:53 +0100

On 2013-11-08 05:12, Tony Kelman wrote:
> Hi all,
> 
> I'm working with a set of projects that save their inter-dependencies in
> AC_SUBST variables as absolute paths to the relevant .la files. In Cygwin
> using the MSVC cl compiler and building only static libraries, I'm seeing
> libtool replace the Unix-style absolute path to .la files with the
> Unix-style absolute path to the corresponding .libs/<libname>.lib file.
> Unsurprisingly, the cl compiler doesn't understand the Unix-style path.
> If I modify the Makefile to add `cygpath -w ...` around the absolute path
> to the .la file, then libtool instead replaces the Windows-style absolute
> path to the .la file with just ./.libs/<libname>.lib, which isn't right
> since I'm in a different directory than that library was built in.
> 
> My test case here is the COIN-OR Open Solver Interface project
> (https://projects.coin-or.org/Osi/wiki) release 0.106.2, but updated to a
> recent set of autotools by svn switch'ing the BuildTools directory to
> https://projects.coin-or.org/BuildTools/browser/branches/autotools-update.
> The same behavior described below happens with the very old libtool
> (1.5.22) distributed with the release, but I updated to test whether the
> problem still exists in recent libtool. Started Cygwin (64-bit) from the
> "VS Command Prompt" to set the right environment variables, and configured
> Osi with --enable-doscompile=msvc to use cl compiler.
> 
> The problem comes at make test. With the Unix-style absolute path to the
> .la file in the variable OSILIB_LIBS, the problem invocation looks like:
> 
> /bin/sh ../libtool --tag=CXX -n -v  --mode=link cl  -MT -O2 -nologo -EHsc
>  -GR -wd4996 -D_CRT_SECURE_NO_DEPRECATE -DNDEBUG    -DOSI_BUILD   -o
>  unitTest.exe unitTest.obj OsiTestSolver.obj OsiTestSolverInterface.obj
>  OsiTestSolverInterfaceIO.obj OsiTestSolverInterfaceTest.obj
>  ../src/OsiCommonTest/libOsiCommonTests.la       ../src/Osi/libOsi.la
>  /home/Tony/Osi-0.106.2/build02/CoinUtils/src/libCoinUtils.la
> libtool: link: cl -MT -O2 -nologo -EHsc -GR -wd4996
>  -D_CRT_SECURE_NO_DEPRECATE -DNDEBUG -DOSI_BUILD -o unitTest.exe
>  unitTest.obj OsiTestSolver.obj OsiTestSolverInterface.obj
>  OsiTestSolverInterfaceIO.obj OsiTestSolverInterfaceTest.obj
>  ../src/OsiCommonTest/.libs/OsiCommonTests.lib ../src/Osi/.libs/Osi.lib
>  /home/Tony/Osi-0.106.2/build02/CoinUtils/src/.libs/libCoinUtils.lib
> libtool: link: lt_outputfile="unitTest.exe"
> libtool: link:  lt_tool_outputfile="unitTest.exe"
> libtool: link:  case  in *.exe|*.EXE) ;; *) lt_outputfile=".exe"
>  lt_tool_outputfile=".exe" ;; esac
> libtool: link:  func_to_tool_file ""
> libtool: link:  if test "mt" != ":" && test -f ".manifest"; then mt
>  -manifest ".manifest" -outputresource:"" || exit 1; rm -f ".manifest"; fi
> 
> If I take out -n from AM_LIBTOOLFLAGS then I get hundreds of unresolved
> externals to everything from libCoinUtils, since cl ignored the "unknown
> option" of the Unix-style absolute path to libCoinUtils.la.
> 
> Adding `cygpath -w ...` around the absolute path to libCoinUtils.la in the
> Osi/test/Makefile variable OSILIB_LIBS results in the following:
> 
> /bin/sh ../libtool --tag=CXX -v  --mode=link cl  -MT -O2 -nologo -EHsc
>  -GR -wd4996 -D_CRT_SECURE_NO_DEPRECATE -DNDEBUG    -DOSI_BUILD   -o
>  unitTest.exe unitTest.obj OsiTestSolver.obj OsiTestSolverInterface.obj
>  OsiTestSolverInterfaceIO.obj OsiTestSolverInterfaceTest.obj
>  ../src/OsiCommonTest/libOsiCommonTests.la       ../src/Osi/libOsi.la
>  `cygpath -w /home/Tony/Osi-0.106.2/build02/CoinUtils/src/libCoinUtils.la`
> libtool: link: cl -MT -O2 -nologo -EHsc -GR -wd4996
>  -D_CRT_SECURE_NO_DEPRECATE -DNDEBUG -DOSI_BUILD -o unitTest.exe
>  unitTest.obj OsiTestSolver.obj OsiTestSolverInterface.obj
>  OsiTestSolverInterfaceIO.obj OsiTestSolverInterfaceTest.obj
>  ../src/OsiCommonTest/.libs/OsiCommonTests.lib ../src/Osi/.libs/Osi.lib
>  ./.libs/libCoinUtils.lib
> cl : Command line warning D9035 : option 'o' has been deprecated and will
>  be removed in a future release
> LINK : fatal error LNK1181: cannot open input file './.libs/libCoinUtils.lib'
> libtool: link: lt_outputfile="unitTest.exe"
> libtool: link:  lt_tool_outputfile="unitTest.exe"
> libtool: link:  case unitTest.exe in *.exe|*.EXE) ;; *)
>  lt_outputfile="unitTest.exe.exe" lt_tool_outputfile="unitTest.exe.exe" ;; esac
> libtool: link:  func_to_tool_file "unitTest.exe"
> libtool: link:  if test "mt" != ":" && test -f "unitTest.exe.manifest";
>  then mt -manifest "unitTest.exe.manifest" -outputresource:"unitTest.exe"
>  || exit 1; rm -f "unitTest.exe.manifest"; fi
> Makefile:476: recipe for target 'unitTest.exe' failed
> make: *** [unitTest.exe] Error 2
> 
> Is this a bug in libtool, or am I doing something wrong/unsupported? MSYS
> works better here, but it would be ideal to be able to use Cygwin as well.
> 
> Relevant info from the end of libtool --help:
> 
> host-triplet:    x86_64-unknown-cygwin
> shell:        /bin/sh
> compiler:        cl
> compiler flags:        -MT -O2 -nologo -wd4996 -D_CRT_SECURE_NO_DEPRECATE -DNDEBUG
> linker:        link (gnu? no)
> libtool:    (GNU libtool) 2.4.2
> automake:
> autoconf:    autoconf (GNU Autoconf) 2.69
> 
> Automake was version 1.12, not sure why libtool didn't see that.
> 
> Appreciate any advice here, let me know if there's any more information I
> can provide that would be useful.

Using the terminology from [1], there are two options for building with
cl on Cygwin. You either fake it or you lie. I recommend faking it.

For that to have any chance to work with absolute file names, you have to
include lt_cv_to_tool_file_cmd=func_convert_file_cygwin_to_w32 as one of
your configure arguments (or do as [1] suggests and
   export lt_cv_to_tool_file_cmd=func_convert_file_cygwin_to_w32
but I prefer to add it as a configure arg, it is then preserved even if
the project happens to be reconfigured from some other environment w/o
that export).

If you instead lie, you also need
  lt_cv_to_host_file_cmd=func_convert_file_cygwin_to_w32

It was not clear from your message if you had taken care of that detail?

Cheers,
Peter

[1] http://www.gnu.org/software/libtool/manual/html_node/Cygwin-to-MinGW-Cross.html

Message #11 received at 15831 <at> debbugs.gnu.org (full text, mbox):

From: "Tony Kelman" <kelman <at> berkeley.edu>
To: "Peter Rosin" <peda <at> lysator.liu.se>,
	<15831 <at> debbugs.gnu.org>
Subject: Re: bug#15831: Absolute library paths with Cygwin + MSVC cl
Date: Fri, 8 Nov 2013 16:42:21 -0800

> Using the terminology from [1], there are two options for building with
> cl on Cygwin. You either fake it or you lie. I recommend faking it.
> 
> For that to have any chance to work with absolute file names, you have to
> include lt_cv_to_tool_file_cmd=func_convert_file_cygwin_to_w32 as one of
> your configure arguments (or do as [1] suggests and
>    export lt_cv_to_tool_file_cmd=func_convert_file_cygwin_to_w32
> but I prefer to add it as a configure arg, it is then preserved even if
> the project happens to be reconfigured from some other environment w/o
> that export).
> 
> If you instead lie, you also need
>   lt_cv_to_host_file_cmd=func_convert_file_cygwin_to_w32
> 
> It was not clear from your message if you had taken care of that detail?

Thanks very much for the info. I tried providing the lt_cv_to_tool_file_cmd
to configure, either with and without --host=x86_64-w64-mingw32, but did
not manage to get libtool to send cl the Windows path to libCoinUtils.lib.
Manually replacing the absolute Cygwin path in the Makefile with either a
relative path or a `cygpath -m ...` mixed path with forward slashes worked.

While it would be great if libtool could do this for me in a simple way,
it seems it's a bit challenging to get working. Compiling with cl is
already heavily special-cased in the COIN-OR build system, so I can modify
one of their autoconf macros that populate the Makefile variables in the
first place to handle this. They're already doing a cygpath -m type path
substitution on any absolute paths to .lib files in the _LIBS variables,
and in initial tests extending that to (.lib|.la) seems to be working.

-Tony

Message #14 received at 15831 <at> debbugs.gnu.org (full text, mbox):

From: Peter Rosin <peda <at> lysator.liu.se>
To: Tony Kelman <kelman <at> berkeley.edu>, 15831 <at> debbugs.gnu.org
Subject: Re: bug#15831: Absolute library paths with Cygwin + MSVC cl
Date: Sat, 09 Nov 2013 02:13:50 +0100

On 2013-11-09 01:42, Tony Kelman wrote:
>> Using the terminology from [1], there are two options for building with
>> cl on Cygwin. You either fake it or you lie. I recommend faking it.
>>
>> For that to have any chance to work with absolute file names, you have to
>> include lt_cv_to_tool_file_cmd=func_convert_file_cygwin_to_w32 as one of
>> your configure arguments (or do as [1] suggests and
>>    export lt_cv_to_tool_file_cmd=func_convert_file_cygwin_to_w32
>> but I prefer to add it as a configure arg, it is then preserved even if
>> the project happens to be reconfigured from some other environment w/o
>> that export).
>>
>> If you instead lie, you also need
>>   lt_cv_to_host_file_cmd=func_convert_file_cygwin_to_w32
>>
>> It was not clear from your message if you had taken care of that detail?
> 
> Thanks very much for the info. I tried providing the lt_cv_to_tool_file_cmd
> to configure, either with and without --host=x86_64-w64-mingw32, but did
> not manage to get libtool to send cl the Windows path to libCoinUtils.lib.
> Manually replacing the absolute Cygwin path in the Makefile with either a
> relative path or a `cygpath -m ...` mixed path with forward slashes worked.
> 
> While it would be great if libtool could do this for me in a simple way,
> it seems it's a bit challenging to get working. Compiling with cl is
> already heavily special-cased in the COIN-OR build system, so I can modify
> one of their autoconf macros that populate the Makefile variables in the
> first place to handle this. They're already doing a cygpath -m type path
> substitution on any absolute paths to .lib files in the _LIBS variables,
> and in initial tests extending that to (.lib|.la) seems to be working.

Hmm, looking a bit closer to your command and the output, I now notice
one thing. Why do you specified an absolute path to libCoinUtils.la in
the first place? I beleive that this isn't according to best practices.
You should either use a -L flag or a relative file name. Does it
work to say
"-L/home/Tony/Osi-0.106.2/build02/CoinUtils/src -lCoinUtils" instead?

I also notice that libCoinUtils.la appears to unfold into
libCoinUtils.lib which seems a bit strange, I would have expected a
straight CoinUtils.lib instead (no lib prefix), not that I think it
matters as long as it's consistent. But how was libCoinUtils.la
produced? Is it a libtool convenience library, or an ordinary library?

Screeech. Stop. Wait a minute. You have not wrapped cl with the
'compile' script (from Automake). The above suggestion will not
work if you don't, I just assumed that it already was in the mix.
My bad. The easiest way to add that script to the project is to add
AM_PROG_CC_C_O to configure.ac (after AC_PROG_CC). Since this is
C++, you will then have to manually say CXX="/path/to/compile cl"
when you configure. Sorry for not noticing earlier.

Cheers,
Peter

Message #17 received at 15831 <at> debbugs.gnu.org (full text, mbox):

From: "Tony Kelman" <kelman <at> berkeley.edu>
To: "Peter Rosin" <peda <at> lysator.liu.se>,
	<15831 <at> debbugs.gnu.org>
Subject: Re: bug#15831: Absolute library paths with Cygwin + MSVC cl
Date: Fri, 8 Nov 2013 19:12:15 -0800

> Hmm, looking a bit closer to your command and the output, I now notice
> one thing. Why do you specified an absolute path to libCoinUtils.la in
> the first place? I beleive that this isn't according to best practices.
> You should either use a -L flag or a relative file name. Does it
> work to say
> "-L/home/Tony/Osi-0.106.2/build02/CoinUtils/src -lCoinUtils" instead?

The makefile variables with these absolute paths are populated by parsing
<proj>-uninstalled.pc files (even on systems without pkg-config). The
justification for why COIN-OR does it that way is described here
https://projects.coin-or.org/BuildTools/wiki/pm-pcfile#The.pcfileofanuninstalledCOIN-ORprojectlibrary
It's used to populate dependencies, evidently the standard way automake
does that would not work for them since they have dependencies that can
change at configure time.

> I also notice that libCoinUtils.la appears to unfold into
> libCoinUtils.lib which seems a bit strange, I would have expected a
> straight CoinUtils.lib instead (no lib prefix), not that I think it
> matters as long as it's consistent. But how was libCoinUtils.la
> produced? Is it a libtool convenience library, or an ordinary library?

It's a libtool convenience library. Produced by usual automake process.
https://projects.coin-or.org/CoinUtils/browser/trunk/CoinUtils/src/Makefile.am#L11

> Screeech. Stop. Wait a minute. You have not wrapped cl with the
> 'compile' script (from Automake). The above suggestion will not
> work if you don't, I just assumed that it already was in the mix.
> My bad. The easiest way to add that script to the project is to add
> AM_PROG_CC_C_O to configure.ac (after AC_PROG_CC). Since this is
> C++, you will then have to manually say CXX="/path/to/compile cl"
> when you configure. Sorry for not noticing earlier.

Aha, that compile script seems much more useful these days than it was
back in automake 1.9.6, which is what COIN-OR has been using for too long.
That works in my tests so far, sorry for wanting libtool to do a job that
a different tool already handles. It looks like using the compile script
should simplify matters and allow the COIN-OR folks to get rid of many of
their custom macros. I'll let them know they should transition to using
it as part of moving to newer autotools.

-Tony

Message #20 received at 15831 <at> debbugs.gnu.org (full text, mbox):

From: Peter Rosin <peda <at> lysator.liu.se>
To: Tony Kelman <kelman <at> berkeley.edu>, 15831 <at> debbugs.gnu.org
Subject: Re: bug#15831: Absolute library paths with Cygwin + MSVC cl
Date: Sat, 09 Nov 2013 11:16:48 +0100

On 2013-11-09 04:12, Tony Kelman wrote:
>> I also notice that libCoinUtils.la appears to unfold into
>> libCoinUtils.lib which seems a bit strange, I would have expected a
>> straight CoinUtils.lib instead (no lib prefix), not that I think it
>> matters as long as it's consistent. But how was libCoinUtils.la
>> produced? Is it a libtool convenience library, or an ordinary library?
> 
> It's a libtool convenience library. Produced by usual automake process.
> https://projects.coin-or.org/CoinUtils/browser/trunk/CoinUtils/src/Makefile.am#L11

Ok.

>> Screeech. Stop. Wait a minute. You have not wrapped cl with the
>> 'compile' script (from Automake). The above suggestion will not
>> work if you don't, I just assumed that it already was in the mix.
>> My bad. The easiest way to add that script to the project is to add
>> AM_PROG_CC_C_O to configure.ac (after AC_PROG_CC). Since this is
>> C++, you will then have to manually say CXX="/path/to/compile cl"
>> when you configure. Sorry for not noticing earlier.
> 
> Aha, that compile script seems much more useful these days than it was
> back in automake 1.9.6, which is what COIN-OR has been using for too long.
> That works in my tests so far, sorry for wanting libtool to do a job that
> a different tool already handles. It looks like using the compile script
> should simplify matters and allow the COIN-OR folks to get rid of many of
> their custom macros. I'll let them know they should transition to using
> it as part of moving to newer autotools.

Yes, the cl side of 'compile' is new since Automake 1.9.6 (8 years old)...
One thing that's still missing is that there is no trigger for it when
using C++.

I'm mostly responding here to hint that there is nothing stopping you
from shipping a recent 'compile' script even if you bootstrap with
Automake 1.9.6, in case there are other reasons that prevent an
Automake upgrade. Or you can just have a private copy of a recent
'compile' script in case it's not feasible to ship a recent 'compile'
script within the project for some political reason.

You might also be interested in the 'ar-lib' script and the AM_PROG_AR
macro that does to lib.exe what 'compile' does to cl, sort of. But
I don't know if you build any old static libraries w/o help from libtool.
Probably not, but just in case...

Cheers,
Peter

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.