GNU bug report logs - #33620
Golang programs keeping references [gnu: go: Update default to 1.11.]

Previous Next

Full log

View this message in rfc822 format

From: help-debbugs <at> gnu.org (GNU bug Tracking System)
To: Leo Famulari <leo <at> famulari.name>
Cc: tracker <at> debbugs.gnu.org
Subject: bug#33741: closed (Packages built with Go 1.10+ keep unnecessary
 references to Go inputs)
Date: Thu, 14 Mar 2019 19:45:02 +0000

[Message part 1 (text/plain, inline)]

Your message dated Thu, 14 Mar 2019 15:44:15 -0400
with message-id <20190314194415.GB17982 <at> jasmine.lan>
and subject line Re: [mail <at> ambrevar.xyz: Re: Golang programs keeping references [gnu: go: Update default to 1.11.]]
has caused the debbugs.gnu.org bug report #33620,
regarding Packages built with Go 1.10+ keep unnecessary references to Go inputs
to be marked as done.

(If you believe you have received this mail in error, please contact
help-debbugs <at> gnu.org.)


-- 
33620: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=33620
GNU Bug Tracking System
Contact help-debbugs <at> gnu.org with problems

[Message part 2 (message/rfc822, inline)]

From: Pierre Neidhardt <mail <at> ambrevar.xyz>
To: bug-guix <at> gnu.org
Subject: Packages built with Go 1.10+ keep unnecessary references to Go inputs
Date: Fri, 14 Dec 2018 16:28:07 +0100

[Message part 3 (text/plain, inline)]

See https://lists.gnu.org/archive/html/guix-devel/2018-11/msg00223.html.

Go binaries are statically build, but since Go 1.10, the full paths to
the Go libraries are kept in the binaries, which results in the Go
dependencies being part of the package closure:

$ guix gc --references $(./pre-inst-env guix build --no-grafts kurly)
/gnu/store/2b2md66fbzyspsmd5dj6zkj9hilac40r-tzdata-2018e
/gnu/store/4iwksvq53rlzphfp3xvp63ihlw226c0n-go-github-com-aki237-nscjar-0.0.0-0.e2df936
/gnu/store/5rxdjbk8h0bh1hbaan8y8ib13va2bcmw-net-base-5.3
/gnu/store/ahvdlp6y44qj6kx63rmx1sq8r61x3zc2-go-github-com-alsm-ioprogress-0.0.0-0.063c372
/gnu/store/f8yps0l8p371jgzh6cki0z5n2kgfjiwy-go-github-com-urfave-cli-1.19.1-0.934abfb
/gnu/store/l4lr0f5cjd0nbsaaf8b5dmcw1a1yypr3-glibc-2.27
/gnu/store/pp0bakrbyv9xmp1kyv2114l19s11b74z-gcc-6.4.0-lib

Previously, they did not:

$ guix gc --references $(guix build --no-grafts kurly)
/gnu/store/2b2md66fbzyspsmd5dj6zkj9hilac40r-tzdata-2018e
/gnu/store/5rxdjbk8h0bh1hbaan8y8ib13va2bcmw-net-base-5.3
/gnu/store/l4lr0f5cjd0nbsaaf8b5dmcw1a1yypr3-glibc-2.27
/gnu/store/pp0bakrbyv9xmp1kyv2114l19s11b74z-gcc-6.4.0-lib

It seems to be an upstream bug:

https://github.com/golang/go/issues/16860

It's still unresolved and only planned for Go 1.13.

Note that adding

              "-asmflags=all=-trimpath=/gnu/store"
              "-gcflags=all=-trimpath=/gnu/store"

to the build system does not work, because we need to trim the hash too.
-trimpath supports only one parameter, so we can't use it for this purpose.

We could use Boyer-Moore to replace the hashes of all Go dependencies.
We must be careful to only replace paths to Go libraries.

--
Pierre Neidhardt
https://ambrevar.xyz/

[signature.asc (application/pgp-signature, inline)]

[Message part 5 (message/rfc822, inline)]

From: Leo Famulari <leo <at> famulari.name>
To: 33620-done <at> debbugs.gnu.org
Subject: Re: [mail <at> ambrevar.xyz: Re: Golang programs keeping references [gnu:
 go: Update default to 1.11.]]
Date: Thu, 14 Mar 2019 15:44:15 -0400

[Message part 6 (text/plain, inline)]

The issue of Go packages keeping references to all their Go-language is
inputs is resolved with commit e3900a4d64e4bf6f426809d6bff058e5a2ae9bc8.

This commit basically avoids bringing the store paths into the build
environment at all by symlinking them into a union directory in the
TMPDIR. This is a bit of a hack but it's actually more idiomatic in Go
to have all the inputs in a single directory ("workspace" in Go). The
previous technique of passing a list of directories in the GOPATH
variable is officially supported but is definitely a 2nd-class technique
in practice.

----- Forwarded message from Pierre Neidhardt <mail <at> ambrevar.xyz> -----
> I've added this to Go's (build) function:
> 
> --8<---------------cut here---------------start------------->8---
>               "-asmflags=all=-trimpath=/gnu/store"
>               "-gcflags=all=-trimpath=/gnu/store"
> --8<---------------cut here---------------end--------------->8---
> 
> The resulting binary is indeed trimmed, but that's not enough: it seems that
> Guix detects the remaining part of the path as a store item and includes it in
> the list of requisites.  Is this really how Guix works?  It does not need the
> full path?

Not having read the reference scanner code carefully, I expect that it
ignores the '/gnu/store/' since this path is actually configurable.

> Regarding Boyer-Moore over the binary, we would have to apply the changes for
> all recursive Go libraries.  Now is there a reliable way to detect what's a Go
> library and what is not?  We don't want to patch non-Go libraries, right?
> (Let's not forget that there is CGo...)

In (guix build-system go), I'd like to construct of list of inputs that
use the go-build-system and pass this list to the biuld side. This would
be useful for other things, but could also be used to detect which
inputs are Go libraries.

> If we can't think of a way to detect a Go library, Boyer-Moore does not seem to
> be a solution either.  And we might have to wait for Go 1.13...

Waiting for upstream is always the easiest way :)

But it would still be nice to have Boyer-Moore available in (guix build
utils) for whenever we want to use it. Or even in Guile itself...

[signature.asc (application/pgp-signature, inline)]

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.