Package: guix-patches;
Reported by: Jørgen Kvalsvik <j <at> lambda.is>
Date: Fri, 15 Nov 2024 21:12:02 UTC
Severity: normal
Tags: patch
Merged with 74371, 74372, 74373
Done: Steve George <steve <at> futurile.net>
Bug is archived. No further changes may be made.
View this message in rfc822 format
From: help-debbugs <at> gnu.org (GNU bug Tracking System) To: Steve George <steve <at> futurile.net> Cc: tracker <at> debbugs.gnu.org Subject: bug#74372: closed ([PATCH 1/4] guix: Add go module fetcher) Date: Tue, 08 Apr 2025 13:51:03 +0000
[Message part 1 (text/plain, inline)]
Your message dated Tue, 8 Apr 2025 14:49:48 +0100 with message-id <Z_UpfF8epIVhnBam <at> t25sg> and subject line Re: bug#74370: Module aware go build system (for Guix) has caused the debbugs.gnu.org bug report #74370, regarding [PATCH 1/4] guix: Add go module fetcher to be marked as done. (If you believe you have received this mail in error, please contact help-debbugs <at> gnu.org.) -- 74370: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=74370 GNU Bug Tracking System Contact help-debbugs <at> gnu.org with problems
[Message part 2 (message/rfc822, inline)]
From: Jørgen Kvalsvik <j <at> lambda.is> To: guix-patches <at> gnu.org Cc: Jørgen Kvalsvik <j <at> lambda.is> Subject: [PATCH 1/4] guix: Add go module fetcher Date: Fri, 15 Nov 2024 22:11:03 +0100Add a new fetcher for go programs, based on go mod download. This is largely how go programs *want* to be fetched, and some pseudo-private dependencies [1] are difficult to wrangle without letting go do the work. This patch only adds support for git as the root module fetch. The approach is conceptually simple: 1. Fetch the root package to build (usually a git repo) 2. Find all the go.mod files in this tree and run go mod download 3. Store the sources of the package, its direct- and transitive dependencies as the source. The source is in source/, the dependencies in go/pkg as encouraged by the toolchain. Go tooling is generally very opinionated on how things are supposed to be done, and fighting it usually brings pain. This approach is not without drawbacks. Go programs tend to be very liberal with dependencies and specific with versions, which will in practice leads to a **large** set of sources. At a system level there will be a lot of duplicated sources, and a lot of slightly different versions which could be compatible. This is not guix specific but rather how go programs design their environments. Another is libraries. Go wants to statically link everything, and generally likes to work with source, and every program pins dependencies to different revisions. The go-mod-fetch getting the source of all libraries the libraries breaks the general packaing approach of packaging libraries separately, but this was never really leveraged by go anyway. This switches the focus to packaging applications rather than libraries. Finally, the package.source needs two hashes - one for the direct origin (e.g. git), and one for the sum of libraries fetched by go.mod. [1] github.com/bufbuild/buf 1.46 requires buf.build/gen/go/bufbuild/bufplugin/protocolbuffers/go v1.35.1-20241031151143-70f632351282.1 which is only available as a .zip and has a layout that go mod understands, but is alien to the go-build-system. * guix/go-mod-download.scm: New file. * gnu/local.mk: Register it. Change-Id: I84c00df07393a9978124667e3e2497aec7009252 --- Makefile.am | 1 + guix/go-mod-download.scm | 146 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 147 insertions(+) create mode 100644 guix/go-mod-download.scm diff --git a/Makefile.am b/Makefile.am index 3a35b7becd..fc00947f4f 100644 --- a/Makefile.am +++ b/Makefile.am @@ -102,6 +102,7 @@ MODULES = \ guix/android-repo-download.scm \ guix/bzr-download.scm \ guix/git-download.scm \ + guix/go-mod-download.scm \ guix/hg-download.scm \ guix/hash.scm \ guix/swh.scm \ diff --git a/guix/go-mod-download.scm b/guix/go-mod-download.scm new file mode 100644 index 0000000000..1a7ffbb6ac --- /dev/null +++ b/guix/go-mod-download.scm @@ -0,0 +1,146 @@ +;;; GNU Guix --- Functional package management for GNU +;;; +;;; This file is part of GNU Guix. +;;; +;;; GNU Guix is free software; you can redistribute it and/or modify it +;;; under the terms of the GNU General Public License as published by +;;; the Free Software Foundation; either version 3 of the License, or (at +;;; your option) any later version. +;;; +;;; GNU Guix is distributed in the hope that it will be useful, but +;;; WITHOUT ANY WARRANTY; without even the implied warranty of +;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +;;; GNU General Public License for more details. +;;; +;;; You should have received a copy of the GNU General Public License +;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>. + +;;; TODOs: +;;; 1. Support non-git root repositories +;;; 2. Store/cache individual module downloads + +(define-module (guix go-mod-download) + #:use-module (guix build utils) + #:use-module (guix derivations) + #:use-module (guix packages) + #:use-module (guix gexp) + #:use-module (guix modules) + #:use-module (guix monads) + #:use-module (guix records) + #:use-module (guix store) + #:use-module (guix git-download) + + #:export (go-mod-reference + go-mod-reference? + go-mod-source + go-mod-go + + go-mod-fetch)) + +(define (go-package) + "Return the default Go package." + (let ((distro (resolve-interface '(gnu packages golang)))) + (module-ref distro 'go))) + +(define (nss-certs-package) + "Return the default nss-certs package." + (let ((distro (resolve-interface '(gnu packages certs)))) + (module-ref distro 'nss-certs))) + +;; The source key accepts the same kinds as the package record, but mostly go +;; use git. The go key specifies which go version to use, which might be +;; necessary if any module sets a newer toolchain in go.mod +(define-record-type* <go-mod-reference> + go-mod-reference make-go-mod-reference + go-mod-reference? + (source go-mod-source) + (go go-mod-go (default (go-package)) (thunked))) + +;; Fetch all direct and indirect dependencies of the go modules in the source +;; tree (usually a git repo) using go mod download. +(define* (go-mod-fetch source hash-algo hash + #:optional name + #:key (system (%current-system)) + (guile (default-guile)) + (go (go-package)) + (nss-certs (nss-certs-package))) + (define guile-json + (module-ref (resolve-interface '(gnu packages guile)) 'guile-json-4)) + + (define* (build source go) + (with-imported-modules (source-module-closure + '((guix build utils) + (guix build download) + (srfi srfi-34))) + (with-extensions (list guile-json) + #~(begin + (use-modules + (guix build download) + (guix build utils) + (srfi srfi-34)) + (let* ((cert-dir (string-append #$nss-certs "/etc/ssl/certs")) + (src-dir (string-append #$output "/source")) + (mod-cache (string-append #$output "/go/pkg")) + (xgo (string-append #$go "/bin/go"))) + ;; go.mod files can specify a minimum required toolchain which could + ;; cause go mod download to fetch and install a newer compiler + ;; if invoked with an older one. + (setenv "GOTOOLCHAIN" (string-append "go" (getenv "go toolchain"))) + (setenv "SSL_CERT_DIR" cert-dir) + (setenv "GOCACHE" "/homeless-shelter") + (setenv "GOPATH" "/homeless-shelter") + (setenv "GOMODCACHE" mod-cache) + + (mkdir-p src-dir) + (mkdir-p mod-cache) + (copy-recursively #$source src-dir) + (let* ((go-mods (find-files src-dir "go.mod"))) + ;; go mod will update the go.mod with transitive dependencies if + ;; they are not set, and fail with an error if the file is not + ;; writable. + (for-each make-file-writable go-mods) + (with-throw-handler + #t + (lambda _ + (for-each + (lambda (go-mod) + (with-directory-excursion (dirname go-mod) + ;; go mod download must be run twice - the first + ;; fetches direct dependencies and *records* + ;; transitive dependencies, the second run fetches + ;; the transitive dependencies. + (and + (invoke xgo "mod" "download") + (invoke xgo "mod" "download")))) + go-mods) + #t) + (lambda (key . args) + (display (string-append "Fetching modules failed.\n" + "Here are the results of `go env`:\n")) + (invoke xgo "env"))))))))) + + (let* ((mod-source (go-mod-source source)) + (mod-ref (origin-uri mod-source)) + (mod-go (go-mod-go source)) + (mod-hash (origin-hash mod-source)) + (mod-hash-value (content-hash-value mod-hash)) + (mod-hash-algo (content-hash-algorithm mod-hash))) + + (mlet* %store-monad ((guile-for-build (package->derivation guile system)) + (git-source (git-fetch mod-ref mod-hash-algo + mod-hash-value + #:system system + #:guile guile-for-build))) + (gexp->derivation (or name "go-mod-fetch") (build git-source mod-go) + #:script-name "go-mod-fetch" + #:env-vars + `(("go toolchain" . ,(package-version mod-go))) + #:leaked-env-vars '("http_proxy" "https_proxy" + "LC_ALL" "LC_MESSAGES" "LANG" + "COLUMNS") + #:system system + #:local-build? #t ;don't offload repo cloning + #:recursive? #t + #:hash-algo hash-algo + #:hash hash + #:guile-for-build guile-for-build)))) -- 2.39.5
[Message part 3 (message/rfc822, inline)]
From: Steve George <steve <at> futurile.net> To: J??rgen Kvalsvik <j <at> lambda.is>, sharlatanus <at> gmail.com Cc: 74370-done <at> debbugs.gnu.org Subject: Re: bug#74370: Module aware go build system (for Guix) Date: Tue, 8 Apr 2025 14:49:48 +0100I think there's a common misunderstanding that if you comment on a bug it won't actually notify the submitter (you have to email NNNN-submitter). It's because debbugs was modelled on the idea that there's an "end-user" who's only interest is the initial submission and a final nnnn-done@ email to tell them it's all solved now! OK, I'm closing #74370. Then you can create a new one when you submit you're new and improved one. Hopefully you and Sharlatan can collaborate on this! Futurile / Steve On 8 Apr, J??rgen Kvalsvik wrote: > Hi, > > I don't know what happened, but as you point out I never received > Sharlatan's emails. > > Anyways, there is still interest, but the approach I proposed is not great. > I have since (re)written the module aware build system with much better > results. I am currently adding a bunch of packages (for my own needs) in > order to get some experience with it and sand down some rough edges, before > I plan to resubmit the new build system upstream. > > Thanks, > J??rgen > > On 4/8/25 15:03, Steve George wrote: > > Hi J??rgen, > > > > You submitted a module aware go build system, there were some queries about it from Sharlatan as there's been a range of attempts at improving this area. Just making you aware so you can reply if you're still interested: > > > > https://debbugs.gnu.org/cgi/bugreport.cgi?bug=74370 > > https://debbugs.gnu.org/cgi/bugreport.cgi?bug=74374 > > > > I don't think you were notified as responses have to go to NNNN-submitter for the original reporter to be send the email. > > > > Thanks, > > > > Futurile / Steve > > > > >
GNU bug tracking system
Copyright (C) 1999 Darren O. Benham,
1997,2003 nCipher Corporation Ltd,
1994-97 Ian Jackson.