Package: guix-patches;
Reported by: Hilton Chain <hako <at> ultrarare.space>
Date: Tue, 18 Mar 2025 07:18:02 UTC
Severity: normal
Tags: patch
View this message in rfc822 format
From: Hilton Chain <hako <at> ultrarare.space> To: 77093 <at> debbugs.gnu.org Cc: Hilton Chain <hako <at> ultrarare.space>, Hilton Chain <hako <at> ultrarare.space>, Ludovic Courtès <ludo <at> gnu.org>, Maxim Cournoyer <maxim.cournoyer <at> gmail.com> Subject: [bug#77093] [PATCH rust-team 18/18] doc: Document lockfile importer based Rust packaging workflow. Date: Tue, 18 Mar 2025 15:24:31 +0800
* doc/contributing.texi (Packaging Guidelines)[Rust Crates]: Update
documentation.
Change-Id: Ic0c6378cf5f5df97d6f8bdd040b486be62c7bddc
---
doc/contributing.texi | 415 +++++++++++++++++++++++++++++++++++++++---
1 file changed, 390 insertions(+), 25 deletions(-)
diff --git a/doc/contributing.texi b/doc/contributing.texi
index ab4f30d54b..e7c5116a3d 100644
--- a/doc/contributing.texi
+++ b/doc/contributing.texi
@@ -1600,34 +1600,399 @@ Rust Crates
@subsection Rust Crates
@cindex rust
-Rust programs standing for themselves are named as any other package, using the
-lowercase upstream name.
+As currently there's no achievable way to reuse Rust packages as pre-compiled
+inputs for other packages at a distribution scale, and our previous approach on
+making all Rust dependencies as full packages doesn't cope well with Rust
+ecosystem thus causing issues on both contribution and maintenance sides, we
+have switched to a different packaging model.
+
+Rust programs (binary crates) and dependencies (library crates) are treated
+separately. We put main efforts into programs and only package Rust
+dependencies as sources, utilizing automation with manual focus on unbundling
+vendored dependencies. The following paragraphs will explain them and give
+several examples.
+
+Rust programs are treated like any other package and named using the lowercase
+upstream name. When using the Cargo build system (@pxref{Build Systems,
+@code{cargo-build-system}}), Rust programs should have @code{#:install-source?}
+argument set to @code{#f}, as this argument only makes sense to dependencies.
+When the package source is a Cargo workspace, @code{#:cargo-install-paths} must
+be set to enable relevant support.
+
+Rust dependencies are managed in two modules:
-To prevent namespace collisions we prefix all other Rust packages with the
-@code{rust-} prefix. The name should be changed to lowercase as appropriate and
-dashes should remain in place.
+@enumerate
+@item
+@code{(gnu packages rust-crates)}, storing source definitions imported from Rust
+programs' @file{Cargo.lock} via the lockfile importer (@pxref{Invoking guix
+import, crate, @code{--lockfile=@var{file}}}).
+
+Imported definitions must be checked and have vendored sources unbundled before
+contributing to Guix. Naturally, this module serves as a store for both sources
+and unbundling strategies.
+
+This module is managed by Rust team (@pxref{Teams}) to ensure there's always one
+version containing all changes from other branches, so that the maintained
+version can be used directly in case of merge conflicts, thus coordination is
+required for other committers to modify it.
+
+Guix source ships template @file{etc/teams/rust/rust-crates.tmpl} and cleanup
+script @file{etc/teams/rust/cleanup-crates.sh} for this moudle.
+
+@item
+@code{(gnu packages rust-sources)}, storing more complex definitions that need
+to be full packages. This includes Rust dependencies requiring external inputs
+to unbundle and Cargo workspaces.
+
+These dependencies should have @code{#:skip-build?} argument set to @code{#t}.
+For Cargo workspaces, @code{#:cargo-package-crates} must be set.
+
+Since they are added manually, the following naming convention applies:
+
+To prevent namespace collisions they are named with @code{rust-} prefix. The
+name should be changed to lowercase as appropriate and dashes should remain in
+place.
In the rust ecosystem it is common for multiple incompatible versions of a
-package to be used at any given time, so all package definitions should have a
-versioned suffix. The versioned suffix is the left-most non-zero digit (and
-any leading zeros, of course). This follows the ``caret'' version scheme
-intended by Cargo. Examples@: @code{rust-clap-2}, @code{rust-rand-0.6}.
-
-Because of the difficulty in reusing rust packages as pre-compiled inputs for
-other packages the Cargo build system (@pxref{Build Systems,
-@code{cargo-build-system}}) presents the @code{#:cargo-inputs} and
-@code{cargo-development-inputs} keywords as build system arguments. It would be
-helpful to think of these as similar to @code{propagated-inputs} and
-@code{native-inputs}. Rust @code{dependencies} and @code{build-dependencies}
-should go in @code{#:cargo-inputs}, and @code{dev-dependencies} should go in
-@code{#:cargo-development-inputs}. If a Rust package links to other libraries
-then the standard placement in @code{inputs} and the like should be used.
-
-Care should be taken to ensure the correct version of dependencies are used; to
-this end we try to refrain from skipping the tests or using @code{#:skip-build?}
-when possible. Of course this is not always possible, as the package may be
-developed for a different Operating System, depend on features from the Nightly
-Rust compiler, or the test suite may have atrophied since it was released.
+package to be used at any given time, so all dependencies should have a
+versioned suffix. The versioned suffix is the left-most non-zero digit (and any
+leading zeros, of course). This follows the ``caret'' version scheme intended
+by Cargo. Examples@: @code{rust-clap-2}, @code{rust-rand-0.6}.
+
+In practice we are usually packaging development snapshots of Rust dependencies
+specifically for some Rust programs, and can't simply identity them by version.
+In this case we can use a @code{for-@var{program}} suffix. Examples@:
+@code{rust-pipewire-for-niri}, @code{rust-pubgrub-for-uv}.
+@end enumerate
+
+Let's demonstrate the packaging workflow by examples, note that package-specific
+issues are not involved here.
+
+In preparation, we'll adding the following packages to our environment:
+
+@example
+guix shell rust rust:cargo cargo-audit cargo-license
+@end example
+
+Example 1: @code{cargo-audit}, which is published on the crates.io Rust package
+repository @uref{https://crates.io, crates.io}.
+
+@enumerate
+@item
+We can generate a draft definition via the crates.io importer (@pxref{Invoking
+guix import, crate}). In the end we'll have the following definiton:
+
+@lisp
+(define-public cargo-audit
+ (package
+ (name "cargo-audit")
+ (version "0.21.2")
+ (source
+ (origin
+ (method url-fetch)
+ (uri (crate-uri "cargo-audit" version))
+ (file-name (string-append name "-" version ".tar.gz"))
+ (sha256
+ (base32 "1a00yqpckkw86zh2hg7ra82c5fx0ird5766dyynimbvqiwg2ps0n"))))
+ (build-system cargo-build-system)
+ (arguments (list #:install-source? #f))
+ (inputs (cargo-inputs 'cargo-audit))
+ (home-page "https://rustsec.org/")
+ (synopsis "Audit Cargo.lock for crates with security vulnerabilities")
+ (description
+ "This package provides a Cargo subcommand, @@command@{cargo audit@}, to
+audit @@file@{Cargo.lock@} for crates with security vulnerabilities.")
+ (license (list license:asl2.0 license:expat))))
+@end lisp
+
+@code{cargo-inputs} is a procedure defined in @code{guix build-system cargo},
+facilitating dependency modification. @code{'cargo-audit} used here must be a
+unique identifier, usually same to the program's variable name.
+
+@item
+Unpack package source and navigate to the unpacked directory, then execute the
+following commands:
+
+@example
+cargo generate-lockfile
+cargo audit
+cargo license
+@end example
+
+@command{cargo generate-lockfile} updates dependencies to compatible versions,
+@command{cargo audit} checks known vulnerabilities and @command{cargo license}
+checks licenses of all dependencies.
+
+We must have an acceptable output of @command{cargo audit} and ensure all
+dependencies are licensed with our supported licenses (@pxref{Defining Packages,
+@code{license}}).
+
+@item
+Import dependencies from previously generated lockfile:
+
+@example
+guix import --insert=gnu/packages/rust-crates.scm \
+ crate --lockfile=/path/to/Cargo.lock cargo-audit
+@end example
+
+@code{cargo-audit} used here must be consistent with the identifier used for
+@code{cargo-inputs}.
+
+At this stage, @code{cargo-audit} is buildable.
+
+@item
+Finally we'll unbundle vendored sources. The lockfile importer inserts
+@code{TODO:} comments to dependencies with high probability of bundled sources.
+@code{cargo-build-system} also performs additional check in its
+@code{check-for-pregenerated-files} phase:
+
+@example
+$ guix build cargo-audit
+@dots{}
+starting phase `check-for-pregenerated-files'
+Searching for binary files...
+./guix-vendor/rust-addr2line-0.21.0.tar.gz/rustfmt.toml
+./guix-vendor/rust-arc-swap-1.7.1.tar.gz/rustfmt.toml
+./guix-vendor/rust-async-compression-0.4.21.tar.gz/tests/artifacts/dictionary-rust
+./guix-vendor/rust-async-compression-0.4.21.tar.gz/tests/artifacts/dictionary-rust-other
+./guix-vendor/rust-async-compression-0.4.21.tar.gz/tests/artifacts/lib.rs.zst
+./guix-vendor/rust-async-compression-0.4.21.tar.gz/tests/artifacts/long-window-size-lib.rs.zst
+@dots{}
+@end example
+
+Although dependencies in @code{(gnu packages rust-crates)} are not exported, we
+can still select them in Guix command-line interface through expression:
+
+@example
+guix build --expression='(@@@@ (gnu packages rust-crates) rust-ring-0.17.14)'
+@end example
+
+For most dependencies, a snippet is sufficient:
+
+@lisp
+(define rust-bzip2-sys-0.1.13+1.0.8
+ (crate-source "bzip2-sys" "0.1.13+1.0.8"
+ "056c39pgjh4272bdslv445f5ry64xvb0f7nph3z7860ln8rzynr2"
+ #:snippet
+ '(begin
+ (delete-file-recursively "bzip2-1.0.8")
+ (delete-file "build.rs")
+ ;; Inspired by Debian's patch.
+ (with-output-to-file "build.rs"
+ (lambda _
+ (format #t "fn main() @{~@@
+ println!(\"cargo:rustc-link-lib=bz2\");~@@
+ @}~%"))))))
+@end lisp
+
+In a more complex case, where unbundling one dependency requires other packages,
+we should package the dependency in @code{(gnu packages rust-sources)} first and
+point the imported definition to a symbol with the same name.
+
+For example we have defined a @code{rust-ring-0.17} in @code{(gnu packages
+rust-sources)}, then the imported definition in @code{(gnu packages
+rust-crates)} should be modified to point to it:
+
+@lisp
+(define rust-ring-0.17.14 'rust-ring-0.17)
+@end lisp
+
+When one dependency can be safely removed, modify it to @code{#f}.
+
+@lisp
+(define rust-openssl-src-300.4.2+3.4.1 #f)
+@end lisp
+
+Such modifications are processed by @code{cargo-inputs} procedure.
+
+@code{cargo-inputs} can also be configured by keywoard arguments
+@code{#:crates-module} (default: @code{(gnu packages rust-crates)}, module to
+resolve imported definitions) and @code{#:sources-module} (default: @code{(gnu
+packages rust-sources)}, module to resolve modified symbols), which are useful
+to maintain an independent channel.
+@end enumerate
+
+Example 2: @code{niri}, which is not available on crates.io and depends on
+development snapshots (also Cargo workspaces in this example).
+
+@enumerate
+@item
+As we can't ensure compatibility of a development snapshot, before executing
+@command{cargo generate-lockfile}, we should modify @file{Cargo.toml} to pin it
+in a known working revision.
+
+To use our packaged development snapshots, it's also necessary to modify
+@file{Cargo.toml} in build environment, the substitution pattern is
+package-specific.
+
+@code{cargo-inputs} returns a list, all list operations apply to it too.
+
+@lisp
+(define-public niri
+ (package
+ (name "niri")
+ (version "25.02")
+ (source (origin
+ (method git-fetch)
+ (uri (git-reference
+ (url "https://github.com/YaLTeR/niri")
+ (commit (string-append "v" version))))
+ (file-name (git-file-name name version))
+ (sha256
+ (base32
+ "0vzskaalcz6pcml687n54adjddzgf5r07gggc4fhfsa08h1wfd4r"))))
+ (build-system cargo-build-system)
+ (arguments
+ (list #:install-source? #f
+ #:phases
+ #~(modify-phases %standard-phases
+ (add-after 'unpack 'use-guix-vendored-dependencies
+ (lambda _
+ (substitute* "Cargo.toml"
+ (("# version =.*")
+ "version = \"*\"")
+ (("git.*optional")
+ "version = \"*\", optional")
+ (("^git = .*")
+ "")))))))
+ (native-inputs
+ (list pkg-config))
+ (inputs
+ (cons* clang
+ libdisplay-info
+ libinput-minimal
+ libseat
+ libxkbcommon
+ mesa
+ pango
+ pipewire
+ wayland
+ (cargo-inputs 'niri)))
+ (home-page "https://github.com/YaLTeR/niri")
+ (synopsis "Scrollable-tiling Wayland compositor")
+ (description
+ "Niri is a scrollable-tiling Wayland compositor which arranges windows in a
+scrollable format. It is considered stable for daily use and performs most
+functions expected of a Wayland compositor.")
+ (license license:gpl3)))
+@end lisp
+
+@item
+@code{niri} also has Cargo workspace dependencies. When packaging a Cargo
+workspace, build argument @code{#:cargo-package-crates} is required.
+
+@lisp
+(define-public rust-pipewire-for-niri
+ (let ((commit "fd3d8f7861a29c2eeaa4c393402e013578bb36d9")
+ (revision "0"))
+ (package
+ (name "rust-pipewire")
+ (version (git-version "0.8.0" revision commit))
+ (source
+ (origin
+ (method git-fetch)
+ (uri (git-reference
+ (url "https://gitlab.freedesktop.org/pipewire/pipewire-rs.git")
+ (commit commit)))
+ (file-name (git-file-name name version))
+ (sha256
+ (base32 "1hzyhz7xg0mz8a5y9j6yil513p1m610q3j9pzf6q55vdh5mcn79v"))))
+ (build-system cargo-build-system)
+ (arguments
+ (list #:skip-build? #t
+ #:cargo-package-crates
+ ''("libspa-sys" "libspa" "pipewire-sys" "pipewire")))
+ (inputs (cargo-inputs 'rust-pipewire-for-niri))
+ (home-page "https://pipewire.org/")
+ (synopsis "Rust bindings for PipeWire")
+ (description "This package provides Rust bindings for PipeWire.")
+ (license license:expat))))
+@end lisp
+
+Don't forget to modify all workspace members in @code{(gnu packages
+rust-crates)}:
+
+@lisp
+(define rust-pipewire-0.8.0.fd3d8f7 'rust-pipewire-for-niri)
+(define rust-pipewire-sys-0.8.0.fd3d8f7 'rust-pipewire-for-niri)
+@dots{}
+(define rust-libspa-0.8.0.fd3d8f7 'rust-pipewire-for-niri)
+(define rust-libspa-sys-0.8.0.fd3d8f7 'rust-pipewire-for-niri)
+@end lisp
+@end enumerate
+
+Example 3: @code{fish}, which combines two build systems.
+
+@enumerate
+@item
+When building Rust packages in other build systems, we need to add @code{rust}
+and @code{rust:cargo} to @code{native-inputs}, import and use modules from both
+build systems, and apply necessary build phases from @code{cargo-build-system}.
+
+@lisp
+(define-public fish
+ (package
+ (name "fish")
+ (version "4.0.0")
+ (source
+ (origin
+ (method url-fetch)
+ (uri (string-append "https://github.com/fish-shell/fish-shell/"
+ "releases/download/" version "/"
+ "fish-" version ".tar.xz"))
+ (sha256
+ (base32 "1wv9kjwg6ax8m2f85i58l9f9cndshn1f15n8skc68w1mf3cmpnig"))))
+ (build-system cmake-build-system)
+ (inputs
+ (cons* fish-foreign-env ncurses pcre2
+ python ;for fish_config and manpage completions
+ (cargo-inputs 'fish)))
+ (native-inputs
+ (list doxygen groff ;for 'fish --help'
+ pkg-config
+ procps ;for the test suite
+ rust
+ `(,rust "cargo")))
+ (arguments
+ (list #:imported-modules
+ (append %cargo-build-system-modules
+ %cmake-build-system-modules)
+ #:modules
+ (((guix build cargo-build-system) #:prefix cargo:)
+ (guix build cmake-build-system)
+ (guix build utils))
+ #:phases
+ #~(modify-phases %standard-phases
+ (add-after 'unpack 'use-guix-vendored-dependencies
+ (lambda _
+ (substitute* "Cargo.toml"
+ (("git.*tag.*,")
+ "version = \"*\","))))
+ (add-after 'unpack 'prepare-cargo-build-system
+ (lambda args
+ (for-each
+ (lambda (phase)
+ (format #t "Running cargo phase: ~a~%" phase)
+ (apply (assoc-ref cargo:%standard-phases phase)
+ args))
+ '(unpack-rust-crates
+ configure
+ check-for-pregenerated-files
+ patch-cargo-checksums)))))))
+ (synopsis "The friendly interactive shell")
+ (description
+ "Fish (friendly interactive shell) is a shell focused on interactive use,
+discoverability, and friendliness. Fish has very user-friendly and powerful
+tab-completion, including descriptions of every completion, completion of
+strings with wildcards, and many completions for specific commands. It also
+has extensive and discoverable help. A special @@command@{help@} command gives
+access to all the fish documentation in your web browser. Other features
+include smart terminal handling based on terminfo, an easy to search history,
+and syntax highlighting.")
+ (home-page "https://fishshell.com/")
+ (license license:gpl2)))
+@end lisp
+@end enumerate
@node Elm Packages
--
2.48.1
GNU bug tracking system
Copyright (C) 1999 Darren O. Benham,
1997,2003 nCipher Corporation Ltd,
1994-97 Ian Jackson.