Package: guix-patches;
Reported by: Hilton Chain <hako <at> ultrarare.space>
Date: Tue, 18 Mar 2025 07:18:02 UTC
Severity: normal
Tags: patch
View this message in rfc822 format
From: Efraim Flashner <efraim <at> flashner.co.il> To: Hilton Chain <hako <at> ultrarare.space> Cc: 77093 <at> debbugs.gnu.org Subject: [bug#77093] [PATCH rust-team v2 17/17] doc: Document lockfile importer based Rust packaging workflow. Date: Wed, 19 Mar 2025 08:52:32 +0200
[Message part 1 (text/plain, inline)]
On Tue, Mar 18, 2025 at 10:27:00PM +0800, Hilton Chain wrote:
> * doc/contributing.texi (Packaging Guidelines)[Rust Crates]: Update
> documentation.
>
> Change-Id: Ic0c6378cf5f5df97d6f8bdd040b486be62c7bddc
> ---
> doc/contributing.texi | 415 +++++++++++++++++++++++++++++++++++++++---
> 1 file changed, 390 insertions(+), 25 deletions(-)
>
> diff --git a/doc/contributing.texi b/doc/contributing.texi
> index ab4f30d54b..837074dead 100644
> --- a/doc/contributing.texi
> +++ b/doc/contributing.texi
> @@ -1600,34 +1600,399 @@ Rust Crates
> @subsection Rust Crates
>
> @cindex rust
> -Rust programs standing for themselves are named as any other package, using the
> -lowercase upstream name.
> +As currently there's no achievable way to reuse Rust packages as pre-compiled
> +inputs for other packages at a distribution scale, and our previous approach on
> +making all Rust dependencies as full packages doesn't cope well with Rust
> +ecosystem thus causing issues on both contribution and maintenance sides, we
> +have switched to a different packaging model.
> +
> +Rust programs (binary crates) and dependencies (library crates) are treated
> +separately. We put our main efforts into programs and only package Rust
> +dependencies as sources, utilizing automation with a manual focus on unbundling
> +vendored dependencies. The following paragraphs will explain them and give
> +several examples.
> +
> +Rust programs are treated like any other package and named using the lowercase
> +upstream name. When using the Cargo build system (@pxref{Build Systems,
> +@code{cargo-build-system}}), Rust programs should have @code{#:install-source?}
> +argument set to @code{#f}, as this argument only makes sense for dependencies.
> +When the package source is a Cargo workspace, @code{#:cargo-install-paths} must
> +be set to enable relevant support.
> +
> +Rust dependencies are managed in two modules:
>
> -To prevent namespace collisions we prefix all other Rust packages with the
> -@code{rust-} prefix. The name should be changed to lowercase as appropriate and
> -dashes should remain in place.
> +@enumerate
> +@item
> +@code{(gnu packages rust-crates)}, storing source definitions imported from Rust
> +programs' @file{Cargo.lock} via the lockfile importer (@pxref{Invoking guix
> +import, crate, @code{--lockfile=@var{file}}}).
> +
> +Imported definitions must be checked and have vendored sources unbundled before
> +contributing to Guix. Naturally, this module serves as a store for both sources
> +and unbundling strategies.
> +
> +This module is managed by the Rust team (@pxref{Teams}) to ensure there's always
> +one version containing all changes from other branches, so that the maintained
> +version can be used directly in case of merge conflicts, thus coordination is
> +required for other committers to modify it.
> +
> +Guix source ships template @file{etc/teams/rust/rust-crates.tmpl} and cleanup
> +script @file{etc/teams/rust/cleanup-crates.sh} for this module.
> +
> +@item
> +@code{(gnu packages rust-sources)}, storing more complex definitions that need
> +to be full packages. This includes Rust dependencies requiring external inputs
> +to unbundle and Cargo workspaces.
> +
> +These dependencies should have the @code{#:skip-build?} argument set to
> +@code{#t}. For Cargo workspaces, @code{#:cargo-package-crates} must be set.
> +
> +Since they are added manually, the following naming convention applies:
> +
> +To prevent namespace collisions they are named with @code{rust-} prefix. The
> +name should be changed to lowercase as appropriate and dashes should remain in
> +place.
>
> In the rust ecosystem it is common for multiple incompatible versions of a
> -package to be used at any given time, so all package definitions should have a
> -versioned suffix. The versioned suffix is the left-most non-zero digit (and
> -any leading zeros, of course). This follows the ``caret'' version scheme
> -intended by Cargo. Examples@: @code{rust-clap-2}, @code{rust-rand-0.6}.
> -
> -Because of the difficulty in reusing rust packages as pre-compiled inputs for
> -other packages the Cargo build system (@pxref{Build Systems,
> -@code{cargo-build-system}}) presents the @code{#:cargo-inputs} and
> -@code{cargo-development-inputs} keywords as build system arguments. It would be
> -helpful to think of these as similar to @code{propagated-inputs} and
> -@code{native-inputs}. Rust @code{dependencies} and @code{build-dependencies}
> -should go in @code{#:cargo-inputs}, and @code{dev-dependencies} should go in
> -@code{#:cargo-development-inputs}. If a Rust package links to other libraries
> -then the standard placement in @code{inputs} and the like should be used.
> -
> -Care should be taken to ensure the correct version of dependencies are used; to
> -this end we try to refrain from skipping the tests or using @code{#:skip-build?}
> -when possible. Of course this is not always possible, as the package may be
> -developed for a different Operating System, depend on features from the Nightly
> -Rust compiler, or the test suite may have atrophied since it was released.
> +package to be used at any given time, so all dependencies should have a
> +versioned suffix. The versioned suffix is the left-most non-zero digit (and any
> +leading zeros, of course). This follows the ``caret'' version scheme intended
> +by Cargo. Examples@: @code{rust-clap-2}, @code{rust-rand-0.6}.
> +
> +In practice we are usually packaging development snapshots of Rust dependencies
> +specifically for some Rust programs, and can't simply identity them by version.
> +In this case we can use a @code{for-@var{program}} suffix. Examples@:
> +@code{rust-pipewire-for-niri}, @code{rust-pubgrub-for-uv}.
> +@end enumerate
> +
> +Let's demonstrate the packaging workflow by examples, note that package-specific
> +issues are not involved here.
> +
> +In preparation, we'll add the following packages to our environment:
> +
> +@example
> +guix shell rust rust:cargo cargo-audit cargo-license
> +@end example
I was able to run cargo-audit as 'cargo-audit audit ...' but I wouldn't
actually recommend that to anyone. I'm not sure we actually need rust
in the shell but I don't have a strong preference. More thinking aloud,
perhaps we should adjust rust:cargo to automatically pull in rust:out?
I also find it amusing to suggest using cargo-audit when it's not (yet)
in guix and we immediately follow up with importing it.
I could see an argument for putting this part in the cookbook instead of
here but I feel strongly that it should be here.
> +
> +Example 1: @code{cargo-audit}, which is published on the crates.io Rust package
> +repository @uref{https://crates.io, crates.io}.
> +
> +@enumerate
> +@item
> +We can generate a draft definition via the crates.io importer (@pxref{Invoking
> +guix import, crate}). In the end we'll have the following definiton:
> +
> +@lisp
> +(define-public cargo-audit
> + (package
> + (name "cargo-audit")
> + (version "0.21.2")
> + (source
> + (origin
> + (method url-fetch)
> + (uri (crate-uri "cargo-audit" version))
> + (file-name (string-append name "-" version ".tar.gz"))
> + (sha256
> + (base32 "1a00yqpckkw86zh2hg7ra82c5fx0ird5766dyynimbvqiwg2ps0n"))))
> + (build-system cargo-build-system)
> + (arguments (list #:install-source? #f))
> + (inputs (cargo-inputs 'cargo-audit))
> + (home-page "https://rustsec.org/")
> + (synopsis "Audit Cargo.lock for crates with security vulnerabilities")
> + (description
> + "This package provides a Cargo subcommand, @@command@{cargo audit@}, to
> +audit @@file@{Cargo.lock@} for crates with security vulnerabilities.")
> + (license (list license:asl2.0 license:expat))))
> +@end lisp
> +
> +@code{cargo-inputs} is a procedure defined in @code{guix build-system cargo},
> +facilitating dependency modification. @code{'cargo-audit} used here must be a
> +unique identifier, usually same to the program's variable name.
> +
> +@item
> +Unpack package source and navigate to the unpacked directory, then execute the
> +following commands:
> +
> +@example
> +cargo generate-lockfile
> +cargo audit
> +cargo license
> +@end example
> +
> +@command{cargo generate-lockfile} updates dependencies to compatible versions,
> +@command{cargo audit} checks known vulnerabilities and @command{cargo license}
> +checks licenses of all dependencies.
> +
> +We must have an acceptable output of @command{cargo audit} and ensure all
> +dependencies are licensed with our supported licenses (@pxref{Defining Packages,
> +@code{license}}).
> +
> +@item
> +Import dependencies from previously generated lockfile:
> +
> +@example
> +guix import --insert=gnu/packages/rust-crates.scm \
> + crate --lockfile=/path/to/Cargo.lock cargo-audit
> +@end example
> +
> +@code{cargo-audit} used here must be consistent with the identifier used for
> +@code{cargo-inputs}.
> +
> +At this stage, @code{cargo-audit} is buildable.
> +
> +@item
> +Finally we'll unbundle vendored sources. The lockfile importer inserts
> +@code{TODO:} comments to dependencies with high probability of bundled sources.
> +@code{cargo-build-system} also performs additional check in its
> +@code{check-for-pregenerated-files} phase:
> +
> +@example
> +$ ./pre-inst-env guix build cargo-audit
> +@dots{}
> +starting phase `check-for-pregenerated-files'
> +Searching for binary files...
> +./guix-vendor/rust-addr2line-0.21.0.tar.gz/rustfmt.toml
> +./guix-vendor/rust-arc-swap-1.7.1.tar.gz/rustfmt.toml
> +./guix-vendor/rust-async-compression-0.4.21.tar.gz/tests/artifacts/dictionary-rust
> +./guix-vendor/rust-async-compression-0.4.21.tar.gz/tests/artifacts/dictionary-rust-other
> +./guix-vendor/rust-async-compression-0.4.21.tar.gz/tests/artifacts/lib.rs.zst
> +./guix-vendor/rust-async-compression-0.4.21.tar.gz/tests/artifacts/long-window-size-lib.rs.zst
> +@dots{}
> +@end example
> +
> +Although dependencies in @code{(gnu packages rust-crates)} are not exported, we
> +can still select them in Guix command-line interface through expression:
> +
> +@example
> +guix build --expression='(@@@@ (gnu packages rust-crates) rust-ring-0.17.14)'
> +@end example
> +
> +For most dependencies, a snippet is sufficient:
> +
> +@lisp
> +(define rust-bzip2-sys-0.1.13+1.0.8
> + (crate-source "bzip2-sys" "0.1.13+1.0.8"
> + "056c39pgjh4272bdslv445f5ry64xvb0f7nph3z7860ln8rzynr2"
> + #:snippet
> + '(begin
> + (delete-file-recursively "bzip2-1.0.8")
> + (delete-file "build.rs")
> + ;; Inspired by Debian's patch.
> + (with-output-to-file "build.rs"
> + (lambda _
> + (format #t "fn main() @{~@@
> + println!(\"cargo:rustc-link-lib=bz2\");~@@
> + @}~%"))))))
> +@end lisp
> +
> +In a more complex case, where unbundling one dependency requires other packages,
> +we should package the dependency in @code{(gnu packages rust-sources)} first and
> +point the imported definition to a symbol with the same name.
> +
> +For example we have defined a @code{rust-ring-0.17} in @code{(gnu packages
> +rust-sources)}, then the imported definition in @code{(gnu packages
> +rust-crates)} should be modified to point to it:
> +
> +@lisp
> +(define rust-ring-0.17.14 'rust-ring-0.17)
> +@end lisp
> +
> +When one dependency can be safely removed, modify it to @code{#f}.
> +
> +@lisp
> +(define rust-openssl-src-300.4.2+3.4.1 #f)
> +@end lisp
> +
> +Such modifications are processed by @code{cargo-inputs} procedure.
> +
> +@code{cargo-inputs} can also be configured by keywoard arguments
> +@code{#:crates-module} (default: @code{(gnu packages rust-crates)}, module to
> +resolve imported definitions) and @code{#:sources-module} (default: @code{(gnu
> +packages rust-sources)}, module to resolve modified symbols), which are useful
> +to maintain an independent channel.
> +@end enumerate
> +
> +Example 2: @code{niri}, which is not available on crates.io and depends on
> +development snapshots (also Cargo workspaces in this example).
> +
> +@enumerate
> +@item
> +As we can't ensure compatibility of a development snapshot, before executing
> +@command{cargo generate-lockfile}, we should modify @file{Cargo.toml} to pin it
> +in a known working revision.
> +
> +To use our packaged development snapshots, it's also necessary to modify
> +@file{Cargo.toml} in build environment, the substitution pattern is
> +package-specific.
> +
> +@code{cargo-inputs} returns a list, all list operations apply to it too.
> +
> +@lisp
> +(define-public niri
> + (package
> + (name "niri")
> + (version "25.02")
> + (source (origin
> + (method git-fetch)
> + (uri (git-reference
> + (url "https://github.com/YaLTeR/niri")
> + (commit (string-append "v" version))))
> + (file-name (git-file-name name version))
> + (sha256
> + (base32
> + "0vzskaalcz6pcml687n54adjddzgf5r07gggc4fhfsa08h1wfd4r"))))
> + (build-system cargo-build-system)
> + (arguments
> + (list #:install-source? #f
> + #:phases
> + #~(modify-phases %standard-phases
> + (add-after 'unpack 'use-guix-vendored-dependencies
> + (lambda _
> + (substitute* "Cargo.toml"
> + (("# version =.*")
> + "version = \"*\"")
> + (("git.*optional")
> + "version = \"*\", optional")
> + (("^git = .*")
> + "")))))))
> + (native-inputs
> + (list pkg-config))
> + (inputs
> + (cons* clang
> + libdisplay-info
> + libinput-minimal
> + libseat
> + libxkbcommon
> + mesa
> + pango
> + pipewire
> + wayland
> + (cargo-inputs 'niri)))
> + (home-page "https://github.com/YaLTeR/niri")
> + (synopsis "Scrollable-tiling Wayland compositor")
> + (description
> + "Niri is a scrollable-tiling Wayland compositor which arranges windows in a
> +scrollable format. It is considered stable for daily use and performs most
> +functions expected of a Wayland compositor.")
> + (license license:gpl3)))
> +@end lisp
> +
> +@item
> +@code{niri} also has Cargo workspace dependencies. When packaging a Cargo
> +workspace, build argument @code{#:cargo-package-crates} is required.
> +
> +@lisp
> +(define-public rust-pipewire-for-niri
> + (let ((commit "fd3d8f7861a29c2eeaa4c393402e013578bb36d9")
> + (revision "0"))
> + (package
> + (name "rust-pipewire")
> + (version (git-version "0.8.0" revision commit))
> + (source
> + (origin
> + (method git-fetch)
> + (uri (git-reference
> + (url "https://gitlab.freedesktop.org/pipewire/pipewire-rs.git")
> + (commit commit)))
> + (file-name (git-file-name name version))
> + (sha256
> + (base32 "1hzyhz7xg0mz8a5y9j6yil513p1m610q3j9pzf6q55vdh5mcn79v"))))
> + (build-system cargo-build-system)
> + (arguments
> + (list #:skip-build? #t
> + #:cargo-package-crates
> + ''("libspa-sys" "libspa" "pipewire-sys" "pipewire")))
> + (inputs (cargo-inputs 'rust-pipewire-for-niri))
should these be inputs or propagated-inputs? in rust-ring-0.17 in
rust-sources they are propagated. I would think inputs would be enough
since the Cargo.lock file will let us know which other crates are
necessary for actually building it as a dependency of that package.
> + (home-page "https://pipewire.org/")
> + (synopsis "Rust bindings for PipeWire")
> + (description "This package provides Rust bindings for PipeWire.")
> + (license license:expat))))
> +@end lisp
> +
> +Don't forget to modify all workspace members in @code{(gnu packages
> +rust-crates)}:
> +
> +@lisp
> +(define rust-pipewire-0.8.0.fd3d8f7 'rust-pipewire-for-niri)
> +(define rust-pipewire-sys-0.8.0.fd3d8f7 'rust-pipewire-for-niri)
> +@dots{}
> +(define rust-libspa-0.8.0.fd3d8f7 'rust-pipewire-for-niri)
> +(define rust-libspa-sys-0.8.0.fd3d8f7 'rust-pipewire-for-niri)
> +@end lisp
> +@end enumerate
> +
> +Example 3: @code{fish}, which combines two build systems.
> +
> +@enumerate
> +@item
> +When building Rust packages in other build systems, we need to add @code{rust}
> +and @code{rust:cargo} to @code{native-inputs}, import and use modules from both
> +build systems, and apply necessary build phases from @code{cargo-build-system}.
> +
> +@lisp
> +(define-public fish
> + (package
> + (name "fish")
> + (version "4.0.0")
> + (source
> + (origin
> + (method url-fetch)
> + (uri (string-append "https://github.com/fish-shell/fish-shell/"
> + "releases/download/" version "/"
> + "fish-" version ".tar.xz"))
> + (sha256
> + (base32 "1wv9kjwg6ax8m2f85i58l9f9cndshn1f15n8skc68w1mf3cmpnig"))))
> + (build-system cmake-build-system)
> + (inputs
> + (cons* fish-foreign-env ncurses pcre2
> + python ;for fish_config and manpage completions
> + (cargo-inputs 'fish)))
> + (native-inputs
> + (list doxygen groff ;for 'fish --help'
> + pkg-config
> + procps ;for the test suite
> + rust
> + `(,rust "cargo")))
> + (arguments
> + (list #:imported-modules
> + (append %cargo-build-system-modules
> + %cmake-build-system-modules)
> + #:modules
> + (((guix build cargo-build-system) #:prefix cargo:)
> + (guix build cmake-build-system)
> + (guix build utils))
> + #:phases
> + #~(modify-phases %standard-phases
> + (add-after 'unpack 'use-guix-vendored-dependencies
> + (lambda _
> + (substitute* "Cargo.toml"
> + (("git.*tag.*,")
> + "version = \"*\","))))
> + (add-after 'unpack 'prepare-cargo-build-system
> + (lambda args
> + (for-each
> + (lambda (phase)
> + (format #t "Running cargo phase: ~a~%" phase)
> + (apply (assoc-ref cargo:%standard-phases phase)
> + args))
> + '(unpack-rust-crates
> + configure
> + check-for-pregenerated-files
> + patch-cargo-checksums)))))))
> + (synopsis "The friendly interactive shell")
> + (description
> + "Fish (friendly interactive shell) is a shell focused on interactive use,
> +discoverability, and friendliness. Fish has very user-friendly and powerful
> +tab-completion, including descriptions of every completion, completion of
> +strings with wildcards, and many completions for specific commands. It also
> +has extensive and discoverable help. A special @@command@{help@} command gives
> +access to all the fish documentation in your web browser. Other features
> +include smart terminal handling based on terminfo, an easy to search history,
> +and syntax highlighting.")
> + (home-page "https://fishshell.com/")
> + (license license:gpl2)))
> +@end lisp
> +@end enumerate
>
>
> @node Elm Packages
> --
> 2.48.1
>
--
Efraim Flashner <efraim <at> flashner.co.il> אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted
[signature.asc (application/pgp-signature, inline)]
GNU bug tracking system
Copyright (C) 1999 Darren O. Benham,
1997,2003 nCipher Corporation Ltd,
1994-97 Ian Jackson.