Subject: Re: [PATCH rust-team v2 00/17] New Rust packaging workflow based on lockfile importer.

From: Efraim Flashner <[email-masked]>
Date: Wed, 19 Mar 2025 08:52:19 +0200

+CC the other rust-team members, not sure how they got left out.

guix/import/crate/cargo-lock.scm should be added to etc/teams.scm.

I was thinking some more about the rust-crates.tmpl file and I think it
makes sense to keep it since it'll likely be useful for people who want
rust packages in their own channel or for working into a guix.scm file.

Some of the patches, specifically some of the importer ones, seem like
they could go straight to master.

On Tue, Mar 18, 2025 at 10:26:43PM +0800, Hilton Chain wrote:
> V1 -> V2:
>   * Drop jemalloc environment variable patch.
>   * Remove use of RUSTC_BOOTSTRAP.
>   * Adjust documentation.
>   Diff inserted at end.
> 
> Efraim Flashner (1):
>   build/cargo: Don't try to unpack sanity-check.py.
> 
> Hilton Chain (16):
>   build/cargo: Pass ‘--offline’ to cargo.
>   build/cargo: Print out all non-empty binary files.
>   build-system: cargo: Support packaging Cargo workspace.
>   build-system: cargo: Support installing Cargo workspace.
>   build/cargo: Set default value of arguments for build phases.
>   build/cargo: Support non-workspace directory source inputs.
>   scripts: import: Document argument for ‘--insert’ option in help
>     message.
>   scripts: import: Add two newlines for ‘--insert’ option.
>   scripts: import: Support expressions defined by 'define.
>   scripts: import: Pass "--insert" to importers.
>   scripts: import: Skip existing definition for ‘--insert’ option.
>   import: crate: crate-name->package-name: Move to (guix build-system
>     cargo).
>   build-system: cargo: Add ‘cargo-inputs’.
>   import: crate: Add Cargo.lock parser.
>   import: crate: Add ‘--lockfile’ option.
>   doc: Document lockfile importer based Rust packaging workflow.
> 
>  Makefile.am                       |   1 +
>  doc/contributing.texi             | 415 ++++++++++++++++++++++++++++--
>  doc/guix.texi                     |  24 ++
>  etc/teams/rust/cleanup-crates.sh  |  38 +++
>  etc/teams/rust/rust-crates.tmpl   |  42 +++
>  gnu/local.mk                      |   2 +
>  gnu/packages/rust-crates.scm      |  42 +++
>  gnu/packages/rust-sources.scm     |  29 +++
>  guix/build-system/cargo.scm       |  57 +++-
>  guix/build/cargo-build-system.scm | 151 ++++++++---
>  guix/import/crate.scm             |  81 +++++-
>  guix/import/crate/cargo-lock.scm  | 105 ++++++++
>  guix/scripts/import.scm           |  53 ++--
>  guix/scripts/import/crate.scm     |  58 ++++-
>  guix/utils.scm                    |  27 +-
>  tests/crate.scm                   |  88 +++++++
>  16 files changed, 1107 insertions(+), 106 deletions(-)
>  create mode 100755 etc/teams/rust/cleanup-crates.sh
>  create mode 100644 etc/teams/rust/rust-crates.tmpl
>  create mode 100644 gnu/packages/rust-crates.scm
>  create mode 100644 gnu/packages/rust-sources.scm
>  create mode 100644 guix/import/crate/cargo-lock.scm
> 
> 
> base-commit: b54a9ca849f013300c633fb79d80bc754f6b28a2
> --
> 2.48.1
> 
> --8<---------------cut here---------------start------------->8---
> diff --git a/doc/contributing.texi b/doc/contributing.texi
> index e7c5116a3d..837074dead 100644
> --- a/doc/contributing.texi
> +++ b/doc/contributing.texi
> @@ -1607,15 +1607,15 @@ Rust Crates
>  have switched to a different packaging model.
> 
>  Rust programs (binary crates) and dependencies (library crates) are treated
> -separately.  We put main efforts into programs and only package Rust
> -dependencies as sources, utilizing automation with manual focus on unbundling
> +separately.  We put our main efforts into programs and only package Rust
> +dependencies as sources, utilizing automation with a manual focus on unbundling
>  vendored dependencies.  The following paragraphs will explain them and give
>  several examples.
> 
>  Rust programs are treated like any other package and named using the lowercase
>  upstream name.  When using the Cargo build system (@pxref{Build Systems,
>  @code{cargo-build-system}}), Rust programs should have @code{#:install-source?}
> -argument set to @code{#f}, as this argument only makes sense to dependencies.
> +argument set to @code{#f}, as this argument only makes sense for dependencies.
>  When the package source is a Cargo workspace, @code{#:cargo-install-paths} must
>  be set to enable relevant support.
> 
> @@ -1631,21 +1631,21 @@ Rust Crates
>  contributing to Guix.  Naturally, this module serves as a store for both sources
>  and unbundling strategies.
> 
> -This module is managed by Rust team (@pxref{Teams}) to ensure there's always one
> -version containing all changes from other branches, so that the maintained
> +This module is managed by the Rust team (@pxref{Teams}) to ensure there's always
> +one version containing all changes from other branches, so that the maintained
>  version can be used directly in case of merge conflicts, thus coordination is
>  required for other committers to modify it.
> 
>  Guix source ships template @file{etc/teams/rust/rust-crates.tmpl} and cleanup
> -script @file{etc/teams/rust/cleanup-crates.sh} for this moudle.
> +script @file{etc/teams/rust/cleanup-crates.sh} for this module.
> 
>  @item
>  @code{(gnu packages rust-sources)}, storing more complex definitions that need
>  to be full packages.  This includes Rust dependencies requiring external inputs
>  to unbundle and Cargo workspaces.
> 
> -These dependencies should have @code{#:skip-build?} argument set to @code{#t}.
> -For Cargo workspaces, @code{#:cargo-package-crates} must be set.
> +These dependencies should have the @code{#:skip-build?} argument set to
> +@code{#t}.  For Cargo workspaces, @code{#:cargo-package-crates} must be set.
> 
>  Since they are added manually, the following naming convention applies:
> 
> @@ -1668,7 +1668,7 @@ Rust Crates
>  Let's demonstrate the packaging workflow by examples, note that package-specific
>  issues are not involved here.
> 
> -In preparation, we'll adding the following packages to our environment:
> +In preparation, we'll add the following packages to our environment:
> 
>  @example
>  guix shell rust rust:cargo cargo-audit cargo-license
> @@ -1747,7 +1747,7 @@ Rust Crates
>  @code{check-for-pregenerated-files} phase:
> 
>  @example
> -$ guix build cargo-audit
> +$ ./pre-inst-env guix build cargo-audit
>  @dots{}
>  starting phase `check-for-pregenerated-files'
>  Searching for binary files...
> diff --git a/doc/guix.texi b/doc/guix.texi
> index 029c8cf59c..218c2ba630 100644
> --- a/doc/guix.texi
> +++ b/doc/guix.texi
> @@ -9365,14 +9365,14 @@ Build Systems
>  sources, to ease in future hacking on rust packages.
> 
>  This build system supports cargo workspaces.  Parameter
> -@code{#:cargo-package-crates} (default: @code{''()}) allows specifying
> -names of library crates to package in @code{package} phase.  Specified
> -crates are packaged from left to right, in case there's dependency among
> -them.  For example, specifying @code{''("pcre2-sys" "pcre2")} will
> -package @code{"pcre2-sys"} first and then @code{"pcre2"}.  Parameter
> -@code{#:cargo-install-paths} (default: @code{''()}) allows specifying
> -paths of binary crates to install in @code{install} phase,
> -@code{''("crates/atuin")}, for example.
> +@code{#:cargo-package-crates} (default: @code{''()}) allows specifying names of
> +library crates to package in the @code{package} phase.  Specified crates are
> +packaged from left to right, in case there's dependency among them.  For
> +example, specifying @code{''("pcre2-sys" "pcre2")} will package
> +@code{"pcre2-sys"} first and then @code{"pcre2"}.  Parameter
> +@code{#:cargo-install-paths} (default: @code{''()}) allows specifying paths of
> +binary crates to install in the @code{install} phase, @code{''("crates/atuin")},
> +for example.
>  @end defvar
> 
>  @defvar chicken-build-system
> @@ -14705,19 +14705,18 @@ Invoking guix import
>  a comment.
>  @item --lockfile=@var{file}
>  @itemx -f @var{file}
> -When @option{--lockfile} is specified, importer will ignore other
> -options and won't output package definitions, instead generating source
> -definition for all dependencies in @var{file}, a @file{Cargo.lock} file.
> -For example:
> +When @option{--lockfile} is specified, the importer will ignore other options
> +and won't output package definitions, instead generating source definitions for
> +all dependencies in @var{file}, a @file{Cargo.lock} file.  For example:
> 
>  @example
>  guix import crate --lockfile=/path/to/Cargo.lock my-package
>  @end example
> 
> -generates sources from @file{/path/to/Cargo.lock}, and a list
> -@code{my-package-cargo-inputs} referencing these sources.  The generated
> -list is intended for package's base @code{inputs}, replacing
> -@code{#:cargo-inputs} and @code{#:cargo-development-inputs}.
> +generates sources from @file{/path/to/Cargo.lock} and a list
> +@code{my-package-cargo-inputs} referencing these sources.  The generated list is
> +intended for the package's @code{inputs}, replacing @code{#:cargo-inputs} and
> +@code{#:cargo-development-inputs}.
>  @end table
> 
>  @item elm
> diff --git a/guix/build/cargo-build-system.scm b/guix/build/cargo-build-system.scm
> index b547421b88..41adc03752 100644
> --- a/guix/build/cargo-build-system.scm
> +++ b/guix/build/cargo-build-system.scm
> @@ -248,12 +248,6 @@ (define* (configure #:key inputs
>    (setenv "LIBSSH2_SYS_USE_PKG_CONFIG" "1")
>    (setenv "SODIUM_USE_PKG_CONFIG" "1")
>    (setenv "ZSTD_SYS_USE_PKG_CONFIG" "1")
> -  ;; This flag is needed when not using the bundled jemalloc.
> -  ;; https://github.com/tikv/jemallocator/issues/19
> -  (setenv "CARGO_FEATURE_UNPREFIXED_MALLOC_ON_SUPPORTED_PLATFORMS" "1")
> -  (when (assoc-ref inputs "jemalloc")
> -    (setenv "JEMALLOC_OVERRIDE"
> -            (string-append (assoc-ref inputs "jemalloc") "/lib/libjemalloc.so")))
>    (when (assoc-ref inputs "openssl")
>      (setenv "OPENSSL_DIR" (assoc-ref inputs "openssl")))
>    (when (assoc-ref inputs "gettext")
> @@ -356,23 +350,17 @@ (define* (package #:key
> 
>          (if (null? cargo-package-crates)
>              (apply invoke `("cargo" "package" "--offline" ,@cargo-package-flags))
> -            ;; Use unstable feature ‘--registry’.
>              (begin
> -              (setenv "RUSTC_BOOTSTRAP" "1")
>                (for-each
>                 (lambda (pkg)
>                   (apply invoke "cargo" "package" "--offline" "--package" pkg
> -                        "--registry" "crates-io" "-Z" "package-workspace"
>                          cargo-package-flags)
>                   (for-each
>                    (lambda (crate)
>                      (invoke "tar" "xzf" crate "-C" vendor-dir))
> -                  (begin
> -                    (delete-file-recursively "target/package/tmp-registry")
> -                    (find-files "target/package" "\\.crate$")))
> +                  (find-files "target/package" "\\.crate$"))
>                   (patch-cargo-checksums #:vendor-dir vendor-dir))
> -               cargo-package-crates)
> -              (unsetenv "RUSTC_BOOTSTRAP")))
> +               cargo-package-crates)))
> 
>          ;; Then unpack the crate, reset the timestamp of all contained files, and
>          ;; repack them.  This is necessary to ensure that they are reproducible.
> diff --git a/guix/scripts/import.scm b/guix/scripts/import.scm
> index d8e1ed3e6f..f6e2985ed9 100644
> --- a/guix/scripts/import.scm
> +++ b/guix/scripts/import.scm
> @@ -126,13 +126,12 @@ (define-command (guix-import . args)
>       (show-version-and-exit "guix import"))
>      ((or ("-i" file importer args ...)
>           ("--insert" file importer args ...))
> -     (let* ((definer?
> -              (cut member
> -                   <>
> -                   `(,@(if (member importer '("crate"))
> -                           '(define)
> -                           '())
> -                     define-public)))
> +     (let* ((importer-definers
> +             `(,@(if (member importer '("crate"))
> +                     '(define)
> +                     '())
> +               define-public))
> +            (definer? (cut member <> importer-definers))
>              (find-and-insert
>               (lambda (expr)
>                 (match expr
> diff --git a/tests/crate.scm b/tests/crate.scm
> index d46f753f9c..9b7066c3b1 100644
> --- a/tests/crate.scm
> +++ b/tests/crate.scm
> @@ -478,7 +478,7 @@ (define rust-leaf-bob-3.0.2-yanked
>      (license #f)))
> 
>  (define temp-file
> -  (string-append "t-utils-" (number->string (getpid))))
> +  (string-append "t-crate-" (number->string (getpid))))
> 
>  
>  (test-begin "crate")
> --8<---------------cut here---------------end--------------->8---

-- 
Efraim Flashner   <[email-masked]>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted