From unknown Fri Jun 20 07:16:26 2025 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Mailer: MIME-tools 5.509 (Entity 5.509) Content-Type: text/plain; charset=utf-8 From: bug#62153 <62153@debbugs.gnu.org> To: bug#62153 <62153@debbugs.gnu.org> Subject: Status: [PATCH 0/2] Add Docker layered image for pack and system Reply-To: bug#62153 <62153@debbugs.gnu.org> Date: Fri, 20 Jun 2025 14:16:26 +0000 retitle 62153 [PATCH 0/2] Add Docker layered image for pack and system reassign 62153 guix-patches submitter 62153 Oleg Pykhalov severity 62153 normal tag 62153 patch thanks From debbugs-submit-bounces@debbugs.gnu.org Sun Mar 12 20:30:42 2023 Received: (at submit) by debbugs.gnu.org; 13 Mar 2023 00:30:42 +0000 Received: from localhost ([127.0.0.1]:33230 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pbW5K-0007H1-33 for submit@debbugs.gnu.org; Sun, 12 Mar 2023 20:30:42 -0400 Received: from lists.gnu.org ([209.51.188.17]:40538) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pbW5H-0007Gp-SA for submit@debbugs.gnu.org; Sun, 12 Mar 2023 20:30:40 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pbW52-0006Nd-TL for guix-patches@gnu.org; Sun, 12 Mar 2023 20:30:29 -0400 Received: from mail-lf1-x12d.google.com ([2a00:1450:4864:20::12d]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pbW50-0004AQ-Ve for guix-patches@gnu.org; Sun, 12 Mar 2023 20:30:24 -0400 Received: by mail-lf1-x12d.google.com with SMTP id i28so13692524lfv.0 for ; Sun, 12 Mar 2023 17:30:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1678667420; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=ucOGKy6q5g61a7cNcRVcCF7YuvL0G2HspA1lHquPyoI=; b=dmWQKNdZHnsZZ8YLurOVG4qE+0yOLrPVrdpqNnjg/nQziq9wRMTwmgMLztci2LCJPx 9wWOzRW7Uq29RTipvKoqXc/Ok3rxT+RkTKq2qJCaQibWFnvFg7ACenNpiqRxy2Usihnz EcBXA8sKSAEOGitpTcUXX90TQEFtLlmhBvRgKy5tA6EXoJhWrzOxXP0o7xWeA5elwrIT 58iFPVGzKWqarbDZCGta1u7cYywjM/ru7ojzcsGXNId4bd4kVOCu0K0e9Sku0haz56+v dkPB3yhnf7HbE0m7Kyi2F00eanQcToqFQionf+OQqxn1cfF7zKJQsW3ZpmZiTlFsbAle phkg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678667420; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=ucOGKy6q5g61a7cNcRVcCF7YuvL0G2HspA1lHquPyoI=; b=SJDjwtKKVLGSYQVS8IedI3Kt8Fs71hTmOL0uJ2FEUsIOF3jHjCHfIwwg9cehhynYXL UVf/w30+VljMWEOvVe16S+sUlUaiIbSWo/s7qcZ/E4v/Yk6kmoS7s9i4DSzakGNrb8XE v7uRbCv54ZG72zKqvIYa1K3uYoRSyj6cWaQ4KTBDHCpoznappJVF0RHFBQqsFxuH4Dib GTWIcV+rCnE4O8PL6SHySUa23e6tU48IhxqXvf5pgg4RziwM8nH6jP1NgrBd4FF+aPPm ciXfwpkj3Yw7EhHpzaWmtAaZ0CBp2d05YFUr4Uee2oHu7+KeWfdHaNtbrsswgVosTPNT r5Hw== X-Gm-Message-State: AO0yUKUAaOl6Qo/RBRWLEaYA2hGXLfKT06rxFmUtJSCjUFV7a1vAuzsT 3Lozqa0Bi5nt/peGmRzht8kLS9wZU6E= X-Google-Smtp-Source: AK7set+n2TbEn9QKPt+FAD2Fzf53FWyE8aMVQxaoS7AVZe9d18Bd6RuMKhHCCNqOullwF8mZSy650A== X-Received: by 2002:a19:f51a:0:b0:4b6:e4c8:8a4e with SMTP id j26-20020a19f51a000000b004b6e4c88a4emr2563834lfb.0.1678667419979; Sun, 12 Mar 2023 17:30:19 -0700 (PDT) Received: from guixsd.wugi.info ([88.201.161.72]) by smtp.gmail.com with ESMTPSA id q11-20020ac246eb000000b004db3bee9a32sm785269lfo.283.2023.03.12.17.30.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 12 Mar 2023 17:30:19 -0700 (PDT) From: Oleg Pykhalov To: guix-patches@gnu.org Subject: [PATCH 0/2] Add Docker layered image for pack and system Date: Mon, 13 Mar 2023 03:30:12 +0300 Message-Id: <20230313003012.14325-1-go.wigust@gmail.com> X-Mailer: git-send-email 2.38.0 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Received-SPF: pass client-ip=2a00:1450:4864:20::12d; envelope-from=go.wigust@gmail.com; helo=mail-lf1-x12d.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-Spam-Score: -1.3 (-) X-Debbugs-Envelope-To: submit Cc: Oleg Pykhalov X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.3 (--) Hi Guix, This patch series add to 'guix pack' and 'guix system image' formats with a layered Docker image, which dicreases images size by sharing same layers on a host. The folling commands show an example for new image formats: ./pre-inst-env guix system image --image-type=docker-layered config.scm docker load -i result ./pre-inst-env guix pack -f docker --entry-point=bin/bash -S /bin=bin bash hello docker load -i result The folloing tests passed: make check-channel-news make check TESTS="tests/pack.scm" make check-system TESTS="docker-system docker-layered-system" The gnu/packages/aux-files/python/stream-layered-image.py Python script is a copy of github.com/NixOS/nixpkgs/pkgs/build-support/docker/stream_layered_image.py with only a simple replacement "/nix" to "/gnu" string. Oleg Pykhalov (2): guix: docker: Build layered image. news: Add entry for the new 'docker-layered' distribution format. Makefile.am | 3 +- doc/guix.texi | 16 +- etc/news.scm | 38 ++ gnu/image.scm | 3 +- .../aux-files/python/stream-layered-image.py | 391 ++++++++++++++++++ gnu/system/image.scm | 84 +++- gnu/tests/docker.scm | 20 +- guix/docker.scm | 182 ++++++-- guix/scripts/pack.scm | 103 +++-- guix/scripts/system.scm | 11 +- tests/pack.scm | 48 +++ 11 files changed, 813 insertions(+), 86 deletions(-) create mode 100644 gnu/packages/aux-files/python/stream-layered-image.py base-commit: 60a211ec705ac98483d76da7f2523f2b8966343a -- 2.38.0 From debbugs-submit-bounces@debbugs.gnu.org Sun Mar 12 20:33:52 2023 Received: (at 62153) by debbugs.gnu.org; 13 Mar 2023 00:33:52 +0000 Received: from localhost ([127.0.0.1]:33235 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pbW8L-0007M2-Qx for submit@debbugs.gnu.org; Sun, 12 Mar 2023 20:33:52 -0400 Received: from mail-lf1-f44.google.com ([209.85.167.44]:44645) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pbW8H-0007Lk-R4 for 62153@debbugs.gnu.org; Sun, 12 Mar 2023 20:33:48 -0400 Received: by mail-lf1-f44.google.com with SMTP id s20so13637964lfb.11 for <62153@debbugs.gnu.org>; Sun, 12 Mar 2023 17:33:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1678667619; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=+CrOOOw/ikdlb1M3RlnsszvA+7Dk8TTM/NvG8qc83SI=; b=iHqxYHVEdjrAD3nOfLflNU2o+z6+idJXFxkNPSF2vIKr//DRTjgSBMmjJMHt1yR7/r l7pXr7BGT1CDZKbz3J6QXCRkvpMbX0D+kSYByUFAH3qh9FHb0MFTfIsNHQK2drpQ37Fh 9AOxXP53iL08lzhpDMAcwA9H2Ddzq2eOKcrsqFLnxJUWo0W/DvUGzRLUJWp7iP2oBgVg DfYmTqT77SBNrE1+42iPOGCzhI2fZ7UrLFelrcBg9ZshsZEgCJS52S0UobNIYPymH3A5 1kA0RBMkFt01aV2Mcer7W/K5FKCDC94ZsBKMMUnr1TE0vs3oPUjUlM37wx//wZ9vtuqH UBWQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678667619; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=+CrOOOw/ikdlb1M3RlnsszvA+7Dk8TTM/NvG8qc83SI=; b=ju2e8OOWAGAGcjTR5n+6JknGsD9m+Wz2fDqpwfoMT937YmqzA+/X91a3qTG5sMGFff 5css88xLbjz3YO8h+z4SbsyTdPvuiaXIgDPNc2KtHDVze5FbgdQJBgwxZ04grQFosZnQ +Ph1PFNMrmiCiZnQeerE4xenX7LfyauywhYXsOBhZFiQWl4pNaV9Md9p2WPHuu5TpOZj rfpxKf8pPecicdiFV6D805FHX8ZMpnu0buhhtBlXJapKb6G/ApkjIBpUjtfI6P//DpIJ kM16mTNOfXe2HYmYVm21NtyfrRfs/pUkwZvqGd6C7RkfIe8M/jSmwQI/EL46n25o34HS w9Lg== X-Gm-Message-State: AO0yUKWvk/q9B+lUR50qW6+X4YKsDCL2coAlzZht+vRHpLVwaV7YB2fg /IJ0qTnpN+nollq+stdJh02ZbMltT3E= X-Google-Smtp-Source: AK7set+f4jA6vZNJHp+5bcajjzhECiLfFOjdp+6G4dQQl4YVszRLI571ll0jaQAM/sZPeEc0HK7V2Q== X-Received: by 2002:a19:c20b:0:b0:4dc:807b:9050 with SMTP id l11-20020a19c20b000000b004dc807b9050mr3706795lfc.0.1678667618407; Sun, 12 Mar 2023 17:33:38 -0700 (PDT) Received: from guixsd.wugi.info ([88.201.161.72]) by smtp.gmail.com with ESMTPSA id t22-20020ac24c16000000b004cb41b43c25sm781659lfq.197.2023.03.12.17.33.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 12 Mar 2023 17:33:38 -0700 (PDT) From: Oleg Pykhalov To: 62153@debbugs.gnu.org Subject: [PATCH 1/2] guix: docker: Build layered image. Date: Mon, 13 Mar 2023 03:33:09 +0300 Message-Id: <20230313003310.17129-1-go.wigust@gmail.com> X-Mailer: git-send-email 2.38.0 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 62153 Cc: Oleg Pykhalov X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) * gnu/packages/aux-files/python/stream-layered-image.py: New file. * Makefile.am (AUX_FILES): Add this. * guix/docker.scm (%docker-image-max-layers): New variable. (build-docker-image)[stream-layered-image, root-system]: New arguments. * guix/scripts/pack.scm (stream-layered-image.py): New variable. (docker-image)[layered-image?]: New argument. (docker-layered-image): New procedure. (%formats)[docker-layered]: New format. (show-formats): Document this. * tests/pack.scm: Add docker-layered-image + localstatedir test. * guix/scripts/system.scm (system-derivation-for-action)[docker-layered-image]: New action. (show-help): Document this. (actions)[docker-layered-image]: New action. (process-action): Add this. * gnu/system/image.scm (docker-layered-image, docker-layered-image-type): New variables. (system-docker-image)[layered-image?]: New argument. (stream-layered-image.py): New variable. (system-docker-layered-image): New procedure. (image->root-file-system)[docker-layered]: New image format. * gnu/tests/docker.scm (%test-docker-layered-system): New test. * gnu/image.scm (validate-image-format)[docker-layered]: New image format. * doc/guix.texi (Invoking guix pack): Document docker-layered format. (image-type Reference): Document docker-layered-image-type. --- Makefile.am | 3 +- doc/guix.texi | 16 +- gnu/image.scm | 3 +- .../aux-files/python/stream-layered-image.py | 391 ++++++++++++++++++ gnu/system/image.scm | 84 +++- gnu/tests/docker.scm | 20 +- guix/docker.scm | 182 ++++++-- guix/scripts/pack.scm | 103 +++-- guix/scripts/system.scm | 11 +- tests/pack.scm | 48 +++ 10 files changed, 775 insertions(+), 86 deletions(-) create mode 100644 gnu/packages/aux-files/python/stream-layered-image.py diff --git a/Makefile.am b/Makefile.am index 23b939b674..9aca84f8f8 100644 --- a/Makefile.am +++ b/Makefile.am @@ -11,7 +11,7 @@ # Copyright © 2017 Arun Isaac # Copyright © 2018 Nikita # Copyright © 2018 Julien Lepiller -# Copyright © 2018 Oleg Pykhalov +# Copyright © 2018, 2023 Oleg Pykhalov # Copyright © 2018 Alex Vong # Copyright © 2019 Efraim Flashner # Copyright © 2021 Chris Marusich @@ -435,6 +435,7 @@ AUX_FILES = \ gnu/packages/aux-files/python/sanity-check.py \ gnu/packages/aux-files/python/sanity-check-next.py \ gnu/packages/aux-files/python/sitecustomize.py \ + gnu/packages/aux-files/python/stream-layered-image.py \ gnu/packages/aux-files/renpy/renpy.in \ gnu/packages/aux-files/run-in-namespace.c diff --git a/doc/guix.texi b/doc/guix.texi index b545751e1b..bd0ee126ee 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -56,7 +56,7 @@ Copyright @copyright{} 2017 Andy Wingo@* Copyright @copyright{} 2017, 2018, 2019, 2020 Arun Isaac@* Copyright @copyright{} 2017 nee@* Copyright @copyright{} 2018 Rutger Helling@* -Copyright @copyright{} 2018, 2021 Oleg Pykhalov@* +Copyright @copyright{} 2018, 2021, 2023 Oleg Pykhalov@* Copyright @copyright{} 2018 Mike Gerwitz@* Copyright @copyright{} 2018 Pierre-Antoine Rouby@* Copyright @copyright{} 2018, 2019 Gábor Boskovits@* @@ -6840,9 +6840,15 @@ the following command: guix pack -f docker -S /bin=bin guile guile-readline @end example +or + +@example +guix pack -f docker-layered -S /bin=bin guile guile-readline +@end example + @noindent -The result is a tarball that can be passed to the @command{docker load} -command, followed by @code{docker run}: +The result is a tarball with image or layered image that can be passed +to the @command{docker load} command, followed by @code{docker run}: @example docker load < @var{file} @@ -43631,6 +43637,10 @@ Build an image based on the @code{iso9660-image} image but with the Build an image based on the @code{docker-image} image. @end defvar +@defvar docker-layered-image-type +Build a layered image based on the @code{docker-layered-image} image. +@end defvar + @defvar raw-with-offset-image-type Build an MBR image with a single partition starting at a @code{1024KiB} offset. This is useful to leave some room to install a bootloader in diff --git a/gnu/image.scm b/gnu/image.scm index 523653dd77..8a6a0d8479 100644 --- a/gnu/image.scm +++ b/gnu/image.scm @@ -1,5 +1,6 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2020, 2022 Mathieu Othacehe +;;; Copyright © 2023 Oleg Pykhalov ;;; ;;; This file is part of GNU Guix. ;;; @@ -152,7 +153,7 @@ (define-with-syntax-properties (name (value properties)) ;; The supported image formats. (define-set-sanitizer validate-image-format format - (disk-image compressed-qcow2 docker iso9660 tarball wsl2)) + (disk-image compressed-qcow2 docker docker-layered iso9660 tarball wsl2)) ;; The supported partition table types. (define-set-sanitizer validate-partition-table-type partition-table-type diff --git a/gnu/packages/aux-files/python/stream-layered-image.py b/gnu/packages/aux-files/python/stream-layered-image.py new file mode 100644 index 0000000000..9ad2168c2d --- /dev/null +++ b/gnu/packages/aux-files/python/stream-layered-image.py @@ -0,0 +1,391 @@ +""" +This script generates a Docker image from a set of store paths. Uses +Docker Image Specification v1.2 as reference [1]. + +It expects a JSON file with the following properties and writes the +image as an uncompressed tarball to stdout: + +* "architecture", "config", "os", "created", "repo_tag" correspond to + the fields with the same name on the image spec [2]. +* "created" can be "now". +* "created" is also used as mtime for files added to the image. +* "store_layers" is a list of layers in ascending order, where each + layer is the list of store paths to include in that layer. + +The main challenge for this script to create the final image in a +streaming fashion, without dumping any intermediate data to disk +for performance. + +A docker image has each layer contents archived as separate tarballs, +and they later all get enveloped into a single big tarball in a +content addressed fashion. However, because how "tar" format works, +we have to know about the name (which includes the checksum in our +case) and the size of the tarball before we can start adding it to the +outer tarball. We achieve that by creating the layer tarballs twice; +on the first iteration we calculate the file size and the checksum, +and on the second one we actually stream the contents. 'add_layer_dir' +function does all this. + +[1]: https://github.com/moby/moby/blob/master/image/spec/v1.2.md +[2]: https://github.com/moby/moby/blob/4fb59c20a4fb54f944fe170d0ff1d00eb4a24d6f/image/spec/v1.2.md#image-json-field-descriptions +""" # noqa: E501 + + +import io +import os +import re +import sys +import json +import hashlib +import pathlib +import tarfile +import itertools +import threading +from datetime import datetime, timezone +from collections import namedtuple + + +def archive_paths_to(obj, paths, mtime): + """ + Writes the given store paths as a tar file to the given stream. + + obj: Stream to write to. Should have a 'write' method. + paths: List of store paths. + """ + + # gettarinfo makes the paths relative, this makes them + # absolute again + def append_root(ti): + ti.name = "/" + ti.name + return ti + + def apply_filters(ti): + ti.mtime = mtime + ti.uid = 0 + ti.gid = 0 + ti.uname = "root" + ti.gname = "root" + return ti + + def nix_root(ti): + ti.mode = 0o0555 # r-xr-xr-x + return ti + + def dir(path): + ti = tarfile.TarInfo(path) + ti.type = tarfile.DIRTYPE + return ti + + with tarfile.open(fileobj=obj, mode="w|") as tar: + # To be consistent with the docker utilities, we need to have + # these directories first when building layer tarballs. + tar.addfile(apply_filters(nix_root(dir("/gnu")))) + tar.addfile(apply_filters(nix_root(dir("/gnu/store")))) + + for path in paths: + path = pathlib.Path(path) + if path.is_symlink(): + files = [path] + else: + files = itertools.chain([path], path.rglob("*")) + + for filename in sorted(files): + ti = append_root(tar.gettarinfo(filename)) + + # copy hardlinks as regular files + if ti.islnk(): + ti.type = tarfile.REGTYPE + ti.linkname = "" + ti.size = filename.stat().st_size + + ti = apply_filters(ti) + if ti.isfile(): + with open(filename, "rb") as f: + tar.addfile(ti, f) + else: + tar.addfile(ti) + + +class ExtractChecksum: + """ + A writable stream which only calculates the final file size and + sha256sum, while discarding the actual contents. + """ + + def __init__(self): + self._digest = hashlib.sha256() + self._size = 0 + + def write(self, data): + self._digest.update(data) + self._size += len(data) + + def extract(self): + """ + Returns: Hex-encoded sha256sum and size as a tuple. + """ + return (self._digest.hexdigest(), self._size) + + +FromImage = namedtuple("FromImage", ["tar", "manifest_json", "image_json"]) +# Some metadata for a layer +LayerInfo = namedtuple("LayerInfo", ["size", "checksum", "path", "paths"]) + + +def load_from_image(from_image_str): + """ + Loads the given base image, if any. + + from_image_str: Path to the base image archive. + + Returns: A 'FromImage' object with references to the loaded base image, + or 'None' if no base image was provided. + """ + if from_image_str is None: + return None + + base_tar = tarfile.open(from_image_str) + + manifest_json_tarinfo = base_tar.getmember("manifest.json") + with base_tar.extractfile(manifest_json_tarinfo) as f: + manifest_json = json.load(f) + + image_json_tarinfo = base_tar.getmember(manifest_json[0]["Config"]) + with base_tar.extractfile(image_json_tarinfo) as f: + image_json = json.load(f) + + return FromImage(base_tar, manifest_json, image_json) + + +def add_base_layers(tar, from_image): + """ + Adds the layers from the given base image to the final image. + + tar: 'tarfile.TarFile' object for new layers to be added to. + from_image: 'FromImage' object with references to the loaded base image. + """ + if from_image is None: + print("No 'fromImage' provided", file=sys.stderr) + return [] + + layers = from_image.manifest_json[0]["Layers"] + checksums = from_image.image_json["rootfs"]["diff_ids"] + layers_checksums = zip(layers, checksums) + + for num, (layer, checksum) in enumerate(layers_checksums, start=1): + layer_tarinfo = from_image.tar.getmember(layer) + checksum = re.sub(r"^sha256:", "", checksum) + + tar.addfile(layer_tarinfo, from_image.tar.extractfile(layer_tarinfo)) + path = layer_tarinfo.path + size = layer_tarinfo.size + + print("Adding base layer", num, "from", path, file=sys.stderr) + yield LayerInfo(size=size, checksum=checksum, path=path, paths=[path]) + + from_image.tar.close() + + +def overlay_base_config(from_image, final_config): + """ + Overlays the final image 'config' JSON on top of selected defaults from the + base image 'config' JSON. + + from_image: 'FromImage' object with references to the loaded base image. + final_config: 'dict' object of the final image 'config' JSON. + """ + if from_image is None: + return final_config + + base_config = from_image.image_json["config"] + + # Preserve environment from base image + final_env = base_config.get("Env", []) + final_config.get("Env", []) + if final_env: + # Resolve duplicates (last one wins) and format back as list + resolved_env = {entry.split("=", 1)[0]: entry for entry in final_env} + final_config["Env"] = list(resolved_env.values()) + return final_config + + +def add_layer_dir(tar, paths, store_dir, mtime): + """ + Appends given store paths to a TarFile object as a new layer. + + tar: 'tarfile.TarFile' object for the new layer to be added to. + paths: List of store paths. + store_dir: the root directory of the nix store + mtime: 'mtime' of the added files and the layer tarball. + Should be an integer representing a POSIX time. + + Returns: A 'LayerInfo' object containing some metadata of + the layer added. + """ + + invalid_paths = [i for i in paths if not i.startswith(store_dir)] + assert len(invalid_paths) == 0, \ + f"Expecting absolute paths from {store_dir}, but got: {invalid_paths}" + + # First, calculate the tarball checksum and the size. + extract_checksum = ExtractChecksum() + archive_paths_to( + extract_checksum, + paths, + mtime=mtime, + ) + (checksum, size) = extract_checksum.extract() + + path = f"{checksum}/layer.tar" + layer_tarinfo = tarfile.TarInfo(path) + layer_tarinfo.size = size + layer_tarinfo.mtime = mtime + + # Then actually stream the contents to the outer tarball. + read_fd, write_fd = os.pipe() + with open(read_fd, "rb") as read, open(write_fd, "wb") as write: + def producer(): + archive_paths_to( + write, + paths, + mtime=mtime, + ) + write.close() + + # Closing the write end of the fifo also closes the read end, + # so we don't need to wait until this thread is finished. + # + # Any exception from the thread will get printed by the default + # exception handler, and the 'addfile' call will fail since it + # won't be able to read required amount of bytes. + threading.Thread(target=producer).start() + tar.addfile(layer_tarinfo, read) + + return LayerInfo(size=size, checksum=checksum, path=path, paths=paths) + + +def add_customisation_layer(target_tar, customisation_layer, mtime): + """ + Adds the customisation layer as a new layer. This is layer is structured + differently; given store path has the 'layer.tar' and corresponding + sha256sum ready. + + tar: 'tarfile.TarFile' object for the new layer to be added to. + customisation_layer: Path containing the layer archive. + mtime: 'mtime' of the added layer tarball. + """ + + checksum_path = os.path.join(customisation_layer, "checksum") + with open(checksum_path) as f: + checksum = f.read().strip() + assert len(checksum) == 64, f"Invalid sha256 at ${checksum_path}." + + layer_path = os.path.join(customisation_layer, "layer.tar") + + path = f"{checksum}/layer.tar" + tarinfo = target_tar.gettarinfo(layer_path) + tarinfo.name = path + tarinfo.mtime = mtime + + with open(layer_path, "rb") as f: + target_tar.addfile(tarinfo, f) + + return LayerInfo( + size=None, + checksum=checksum, + path=path, + paths=[customisation_layer] + ) + + +def add_bytes(tar, path, content, mtime): + """ + Adds a file to the tarball with given path and contents. + + tar: 'tarfile.TarFile' object. + path: Path of the file as a string. + content: Contents of the file. + mtime: 'mtime' of the file. Should be an integer representing a POSIX time. + """ + assert type(content) is bytes + + ti = tarfile.TarInfo(path) + ti.size = len(content) + ti.mtime = mtime + tar.addfile(ti, io.BytesIO(content)) + + +def main(): + with open(sys.argv[1], "r") as f: + conf = json.load(f) + + created = ( + datetime.now(tz=timezone.utc) + if conf["created"] == "now" + else datetime.fromisoformat(conf["created"]) + ) + mtime = int(created.timestamp()) + store_dir = conf["store_dir"] + + from_image = load_from_image(conf["from_image"]) + + with tarfile.open(mode="w|", fileobj=sys.stdout.buffer) as tar: + layers = [] + layers.extend(add_base_layers(tar, from_image)) + + start = len(layers) + 1 + for num, store_layer in enumerate(conf["store_layers"], start=start): + print("Creating layer", num, "from paths:", store_layer, + file=sys.stderr) + info = add_layer_dir(tar, store_layer, store_dir, mtime=mtime) + layers.append(info) + + print("Creating layer", len(layers) + 1, "with customisation...", + file=sys.stderr) + layers.append( + add_customisation_layer( + tar, + conf["customisation_layer"], + mtime=mtime + ) + ) + + print("Adding manifests...", file=sys.stderr) + + image_json = { + "created": datetime.isoformat(created), + "architecture": conf["architecture"], + "os": "linux", + "config": overlay_base_config(from_image, conf["config"]), + "rootfs": { + "diff_ids": [f"sha256:{layer.checksum}" for layer in layers], + "type": "layers", + }, + "history": [ + { + "created": datetime.isoformat(created), + "comment": f"store paths: {layer.paths}" + } + for layer in layers + ], + } + + image_json = json.dumps(image_json, indent=4).encode("utf-8") + image_json_checksum = hashlib.sha256(image_json).hexdigest() + image_json_path = f"{image_json_checksum}.json" + add_bytes(tar, image_json_path, image_json, mtime=mtime) + + manifest_json = [ + { + "Config": image_json_path, + "RepoTags": [conf["repo_tag"]], + "Layers": [layer.path for layer in layers], + } + ] + manifest_json = json.dumps(manifest_json, indent=4).encode("utf-8") + add_bytes(tar, "manifest.json", manifest_json, mtime=mtime) + + print("Done.", file=sys.stderr) + + +if __name__ == "__main__": + main() diff --git a/gnu/system/image.scm b/gnu/system/image.scm index afef79185f..0bfd011ad4 100644 --- a/gnu/system/image.scm +++ b/gnu/system/image.scm @@ -4,6 +4,7 @@ ;;; Copyright © 2022 Pavel Shlyak ;;; Copyright © 2022 Denis 'GNUtoo' Carikli ;;; Copyright © 2022 Alex Griffin +;;; Copyright © 2023 Oleg Pykhalov ;;; ;;; This file is part of GNU Guix. ;;; @@ -45,6 +46,7 @@ (define-module (gnu system image) #:use-module (gnu system uuid) #:use-module (gnu system vm) #:use-module (guix packages) + #:use-module ((gnu packages) #:select (search-auxiliary-file)) #:use-module (gnu packages base) #:use-module (gnu packages bash) #:use-module (gnu packages bootloaders) @@ -58,6 +60,7 @@ (define-module (gnu system image) #:use-module (gnu packages hurd) #:use-module (gnu packages linux) #:use-module (gnu packages mtools) + #:use-module (gnu packages python) #:use-module (gnu packages virtualization) #:use-module ((srfi srfi-1) #:prefix srfi-1:) #:use-module (srfi srfi-11) @@ -78,6 +81,7 @@ (define-module (gnu system image) efi-disk-image iso9660-image docker-image + docker-layered-image tarball-image wsl2-image raw-with-offset-disk-image @@ -89,6 +93,7 @@ (define-module (gnu system image) iso-image-type uncompressed-iso-image-type docker-image-type + docker-layered-image-type tarball-image-type wsl2-image-type raw-with-offset-image-type @@ -167,6 +172,10 @@ (define docker-image (image-without-os (format 'docker))) +(define docker-layered-image + (image-without-os + (format 'docker-layered))) + (define tarball-image (image-without-os (format 'tarball))) @@ -237,6 +246,11 @@ (define docker-image-type (name 'docker) (constructor (cut image-with-os docker-image <>)))) +(define docker-layered-image-type + (image-type + (name 'docker-layered) + (constructor (cut image-with-os docker-layered-image <>)))) + (define tarball-image-type (image-type (name 'tarball) @@ -633,9 +647,12 @@ (define (image-with-label base-image label) (define* (system-docker-image image #:key - (name "docker-image")) + (name "docker-image") + (archiver tar) + layered-image?) "Build a docker image for IMAGE. NAME is the base name to use for the -output file." +output file. If LAYERED-IMAGE? is true, the image will with many of the store +paths being on their own layer to improve sharing between images." (define boot-program ;; Program that runs the boot script of OS, which in turn starts shepherd. (program-file "boot-program" @@ -678,9 +695,11 @@ (define builder (use-modules (guix docker) (guix build utils) (gnu build image) + (srfi srfi-1) (srfi srfi-19) (guix build store-copy) - (guix store database)) + (guix store database) + (ice-9 receive)) ;; Set the SQL schema location. (sql-schema #$schema) @@ -700,18 +719,34 @@ (define builder #:register-closures? #$register-closures? #:deduplicate? #f #:system-directory #$os) - (build-docker-image - #$output - (cons* image-root - (map store-info-item - (call-with-input-file #$graph - read-reference-graph))) - #$os - #:entry-point '(#$boot-program #$os) - #:compressor '(#+(file-append gzip "/bin/gzip") "-9n") - #:creation-time (make-time time-utc 0 1) - #:system #$image-target - #:transformations `((,image-root -> "")))))))) + (when #$layered-image? + (setenv "PATH" + (string-join (list #+(file-append archiver "/bin") + #+(file-append coreutils "/bin") + #+(file-append gzip "/bin") + #+(file-append python "/bin")) + ":"))) + (apply build-docker-image + (append (list #$output + (append (if #$layered-image? + '() + (list image-root)) + (map store-info-item + (call-with-input-file #$graph + read-reference-graph))) + #$os + #:entry-point '(#$boot-program #$os) + #:compressor + '(#+(file-append gzip "/bin/gzip") "-9n") + #:creation-time (make-time time-utc 0 1) + #:system #$image-target + #:transformations `((,image-root -> ""))) + (if #$layered-image? + (list #:root-system + image-root + #:stream-layered-image + #$stream-layered-image.py) + '())))))))) (computed-file name builder ;; Allow offloading so that this I/O-intensive process @@ -720,6 +755,21 @@ (define builder #:options `(#:references-graphs ((,graph ,os)) #:substitutable? ,substitutable?)))) +(define stream-layered-image.py + (local-file (search-auxiliary-file "python/stream-layered-image.py"))) + +(define* (system-docker-layered-image image + #:key + (name "docker-image") + (archiver tar) + (layered-image? #t)) + "Build a docker image for IMAGE. NAME is the base name to use for the +output file." + (system-docker-image image + #:name name + #:archiver archiver + #:layered-image? layered-image?)) + ;;; ;;; Tarball image. @@ -811,7 +861,7 @@ (define (image->root-file-system image) "Return the IMAGE root partition file-system type." (case (image-format image) ((iso9660) "iso9660") - ((docker tarball wsl2) "dummy") + ((docker docker-layered tarball wsl2) "dummy") (else (partition-file-system (find-root-partition image))))) @@ -948,6 +998,8 @@ (define target (cond ("bootcfg" ,bootcfg)))) ((memq image-format '(docker)) (system-docker-image image*)) + ((memq image-format '(docker-layered)) + (system-docker-layered-image image*)) ((memq image-format '(tarball)) (system-tarball-image image*)) ((memq image-format '(wsl2)) diff --git a/gnu/tests/docker.scm b/gnu/tests/docker.scm index 0276e398a7..85c5f178b5 100644 --- a/gnu/tests/docker.scm +++ b/gnu/tests/docker.scm @@ -1,6 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2019 Danny Milosavljevic ;;; Copyright © 2019-2022 Ludovic Courtès +;;; Copyright © 2023 Oleg Pykhalov ;;; ;;; This file is part of GNU Guix. ;;; @@ -43,7 +44,8 @@ (define-module (gnu tests docker) #:use-module (guix build-system trivial) #:use-module ((guix licenses) #:prefix license:) #:export (%test-docker - %test-docker-system)) + %test-docker-system + %test-docker-layered-system)) (define %docker-os (simple-operating-system @@ -309,3 +311,19 @@ (define %test-docker-system (locale-libcs (list glibc))) #:type docker-image-type))) run-docker-system-test))))) + +(define %test-docker-layered-system + (system-test + (name "docker-layered-system") + (description "Run a system image as produced by @command{guix system +docker-layered-image} inside Docker.") + (value (with-monad %store-monad + (>>= (lower-object + (system-image (os->image + (operating-system + (inherit (simple-operating-system)) + ;; Use locales for a single libc to + ;; reduce space requirements. + (locale-libcs (list glibc))) + #:type docker-layered-image-type))) + run-docker-system-test))))) diff --git a/guix/docker.scm b/guix/docker.scm index 5e6460f43f..f1adad26dc 100644 --- a/guix/docker.scm +++ b/guix/docker.scm @@ -3,6 +3,7 @@ ;;; Copyright © 2017, 2018, 2019, 2021 Ludovic Courtès ;;; Copyright © 2018 Chris Marusich ;;; Copyright © 2021 Maxim Cournoyer +;;; Copyright © 2023 Oleg Pykhalov ;;; ;;; This file is part of GNU Guix. ;;; @@ -24,10 +25,14 @@ (define-module (guix docker) #:use-module (guix base16) #:use-module (guix build pack) #:use-module ((guix build utils) - #:select (mkdir-p + #:select (%store-directory + mkdir-p delete-file-recursively + dump-port with-directory-excursion invoke)) + #:use-module (guix diagnostics) + #:use-module (guix i18n) #:use-module (gnu build install) #:use-module (json) ;guile-json #:use-module (srfi srfi-1) @@ -38,6 +43,9 @@ (define-module (guix docker) #:use-module (rnrs bytevectors) #:use-module (ice-9 ftw) #:use-module (ice-9 match) + #:use-module (ice-9 popen) + #:use-module (ice-9 rdelim) + #:use-module (ice-9 receive) #:export (build-docker-image)) ;; Generate a 256-bit identifier in hexadecimal encoding for the Docker image. @@ -136,6 +144,9 @@ (define directive-file (('directory name _ ...) (string-trim name #\/)))) +(define %docker-image-max-layers + 100) + (define* (build-docker-image image paths prefix #:key (repository "guix") @@ -146,11 +157,13 @@ (define* (build-docker-image image paths prefix entry-point (environment '()) compressor - (creation-time (current-time time-utc))) - "Write to IMAGE a Docker image archive containing the given PATHS. PREFIX -must be a store path that is a prefix of any store paths in PATHS. REPOSITORY -is a descriptive name that will show up in \"REPOSITORY\" column of the output -of \"docker images\". + (creation-time (current-time time-utc)) + stream-layered-image + root-system) + "Write to IMAGE a layerer Docker image archive containing the given PATHS. +PREFIX must be a store path that is a prefix of any store paths in PATHS. +REPOSITORY is a descriptive name that will show up in \"REPOSITORY\" column of +the output of \"docker images\". When DATABASE is true, copy it to /var/guix/db in the image and create /var/guix/gcroots and friends. @@ -172,7 +185,13 @@ (define* (build-docker-image image paths prefix SYSTEM is a GNU triplet (or prefix thereof) of the system the binaries in PATHS are for; it is used to produce metadata in the image. Use COMPRESSOR, a command such as '(\"gzip\" \"-9n\"), to compress IMAGE. Use CREATION-TIME, a -SRFI-19 time-utc object, as the creation time in metadata." +SRFI-19 time-utc object, as the creation time in metadata. + +STREAM-LAYERED-IMAGE is a Python script which accepts a JSON configuration +file and prints archive to STDOUT. + +ROOT-SYSTEM is a directory with a provisioned root file system, which will be +added to image as a layer." (define (sanitize path-fragment) (escape-special-chars ;; GNU tar strips the leading slash off of absolute paths before applying @@ -183,6 +202,39 @@ (define (sanitize path-fragment) ;; We also need to escape "/" because we use it as a delimiter. "/*.^$[]\\" #\\)) + (define (file-sha256 file-name) + "Calculate the hexdigest of the sha256 checksum of FILE-NAME and return it." + (let ((port (open-pipe* OPEN_READ + "sha256sum" + "--" + file-name))) + (let ((result (read-delimited " " port))) + (close-pipe port) + result))) + (define (paths-split-sort paths) + "Split list of PATHS at %DOCKER-IMAGE-MAX-LAYERS and sort by disk usage." + (let* ((paths-length (length paths)) + (port (apply open-pipe* OPEN_READ + (append '("du" "--summarize") paths))) + (output (read-string port))) + (close-port port) + (receive (head tail) + (split-at + (map (match-lambda ((size . path) path)) + (sort (map (lambda (line) + (match (string-split line #\tab) + ((size path) + (cons (string->number size) path)))) + (string-split + (string-trim-right output #\newline) + #\newline)) + (lambda (path1 path2) + (< (match path2 ((size . _) size)) + (match path1 ((size . _) size)))))) + (if (>= paths-length %docker-image-max-layers) + (- %docker-image-max-layers 2) + (1- paths-length))) + (list head tail)))) (define transformation->replacement (match-lambda ((old '-> new) @@ -205,7 +257,9 @@ (define transformation-options `("--transform" ,(transformations->expression transformations)))) (let* ((directory "/tmp/docker-image") ;temporary working directory (id (docker-id prefix)) - (time (date->string (time-utc->date creation-time) "~4")) + (time ;Workaround for Python datetime.fromisoformat does not parse Z. + (string-append (date->string (time-utc->date creation-time) "~5") + "+00:00")) (arch (let-syntax ((cond* (syntax-rules () ((_ (pattern clause) ...) (cond ((string-prefix? pattern system) @@ -218,7 +272,8 @@ (define transformation-options ("i686" "386") ("arm" "arm") ("aarch64" "arm64") - ("mips64" "mips64le"))))) + ("mips64" "mips64le")))) + (paths (if stream-layered-image (paths-split-sort paths) paths))) ;; Make sure we start with a fresh, empty working directory. (mkdir directory) (with-directory-excursion directory @@ -229,26 +284,38 @@ (define transformation-options (with-output-to-file "json" (lambda () (scm->json (image-description id time)))) - ;; Create a directory for the non-store files that need to go into the - ;; archive. - (mkdir "extra") + (if root-system + (let ((directory (getcwd))) + (with-directory-excursion root-system + (apply invoke "tar" + "-cf" (string-append directory "/layer.tar") + `(,@transformation-options + ,@(tar-base-options) + ,@(scandir "." + (lambda (file) + (not (member file '("." ".."))))))))) + (begin + ;; Create a directory for the non-store files that need to go + ;; into the archive. + (mkdir "extra") - (with-directory-excursion "extra" - ;; Create non-store files. - (for-each (cut evaluate-populate-directive <> "./") - extra-files) + (with-directory-excursion "extra" + ;; Create non-store files. + (for-each (cut evaluate-populate-directive <> "./") + extra-files) - (when database - ;; Initialize /var/guix, assuming PREFIX points to a profile. - (install-database-and-gc-roots "." database prefix)) + (when database + ;; Initialize /var/guix, assuming PREFIX points to a profile. + (install-database-and-gc-roots "." database prefix)) - (apply invoke "tar" "-cf" "../layer.tar" - `(,@transformation-options - ,@(tar-base-options) - ,@paths - ,@(scandir "." - (lambda (file) - (not (member file '("." "..")))))))) + (apply invoke "tar" "-cf" "../layer.tar" + `(,@transformation-options + ,@(tar-base-options) + ,@(if stream-layered-image '() paths) + ,@(scandir "." + (lambda (file) + (not (member file '("." "..")))))))) + (delete-file-recursively "extra"))) ;; It is possible for "/" to show up in the archive, especially when ;; applying transformations. For example, the transformation @@ -263,22 +330,65 @@ (define transformation-options (lambda () (system* "tar" "--delete" "/" "-f" "layer.tar"))) - (delete-file-recursively "extra")) + (when stream-layered-image + (call-with-output-file "checksum" + (lambda (port) + (display (file-sha256 "layer.tar") port))))) (with-output-to-file "config.json" (lambda () - (scm->json (config (string-append id "/layer.tar") - time arch - #:environment environment - #:entry-point entry-point)))) + (scm->json + (if stream-layered-image + `(("created" . ,time) + ("repo_tag" . "guix:latest") + ("customisation_layer" . ,id) + ("store_layers" . ,(match paths + (((head ...) (tail ...)) + (list->vector + (reverse + (cons (list->vector tail) + (fold (lambda (path paths) + (cons (vector path) paths)) + '() + head))))))) + ("store_dir" . ,(%store-directory)) + ("from_image" . #nil) + ("os" . "linux") + ("config" + (env . ,(list->vector (map (match-lambda + ((name . value) + (string-append name "=" value))) + environment))) + ,@(if entry-point + `((entrypoint . ,(list->vector entry-point))) + '())) + ("architecture" . ,arch)) + (config (string-append id "/layer.tar") + time arch + #:environment environment + #:entry-point entry-point))))) (with-output-to-file "manifest.json" (lambda () (scm->json (manifest prefix id repository)))) (with-output-to-file "repositories" (lambda () - (scm->json (repositories prefix id repository))))) - - (apply invoke "tar" "-cf" image "-C" directory - `(,@(tar-base-options #:compressor compressor) - ".")) + (scm->json (repositories prefix id repository)))) + (if stream-layered-image + (let ((input (open-pipe* OPEN_READ "python3" + stream-layered-image + "config.json"))) + (call-with-output-file "image.tar" + (lambda (output) + (dump-port input output))) + (if (eqv? 0 (status:exit-val (close-pipe input))) + (begin + (invoke "gzip" "image.tar") + (copy-file "image.tar.gz" image)) + (error + (formatted-message + (G_ "failed to create ~a image tarball") + image)))) + (apply invoke "tar" "-cf" image + `(,@(tar-base-options #:compressor compressor) + ".")))) (delete-file-recursively directory))) diff --git a/guix/scripts/pack.scm b/guix/scripts/pack.scm index eb41eb5563..3a8f87e850 100644 --- a/guix/scripts/pack.scm +++ b/guix/scripts/pack.scm @@ -8,6 +8,7 @@ ;;; Copyright © 2020, 2021, 2022, 2023 Maxim Cournoyer ;;; Copyright © 2020 Eric Bavier ;;; Copyright © 2022 Alex Griffin +;;; Copyright © 2023 Oleg Pykhalov ;;; ;;; This file is part of GNU Guix. ;;; @@ -54,6 +55,7 @@ (define-module (guix scripts pack) #:use-module ((gnu packages compression) #:hide (zip)) #:use-module (gnu packages guile) #:use-module (gnu packages base) + #:use-module (gnu packages python) #:autoload (gnu packages package-management) (guix) #:autoload (gnu packages gnupg) (guile-gcrypt) #:autoload (gnu packages guile) (guile2.0-json guile-json) @@ -69,6 +71,7 @@ (define-module (guix scripts pack) debian-archive rpm-archive docker-image + docker-layered-image squashfs-image %formats @@ -591,6 +594,10 @@ (define (mksquashfs args) ;;; ;;; Docker image format. ;;; + +(define stream-layered-image.py + (local-file (search-auxiliary-file "python/stream-layered-image.py"))) + (define* (docker-image name profile #:key target (profile-name "guix-profile") @@ -599,12 +606,14 @@ (define* (docker-image name profile localstatedir? (symlinks '()) (archiver tar) - (extra-options '())) + (extra-options '()) + layered-image?) "Return a derivation to construct a Docker image of PROFILE. The image is a tarball conforming to the Docker Image Specification, compressed with COMPRESSOR. It can be passed to 'docker load'. If TARGET is true, it must a be a GNU triplet and it is used to derive the architecture metadata in -the image." +the image. If LAYERED-IMAGE? is true, the image will with many of the +store paths being on their own layer to improve sharing between images." (define database (and localstatedir? (file-append (store-database (list profile)) @@ -655,25 +664,37 @@ (define directives `((directory "/tmp" ,(getuid) ,(getgid) #o1777) ,@(append-map symlink->directives '#$symlinks))) - (setenv "PATH" #+(file-append archiver "/bin")) - - (build-docker-image #$output - (map store-info-item - (call-with-input-file "profile" - read-reference-graph)) - #$profile - #:repository (manifest->friendly-name - (profile-manifest #$profile)) - #:database #+database - #:system (or #$target %host-type) - #:environment environment - #:entry-point - #$(and entry-point - #~(list (string-append #$profile "/" - #$entry-point))) - #:extra-files directives - #:compressor #+(compressor-command compressor) - #:creation-time (make-time time-utc 0 1)))))) + (setenv "PATH" + (string-join `(#+(file-append archiver "/bin") + #+@(if layered-image? + (list (file-append coreutils "/bin") + (file-append gzip "/bin") + (file-append python "/bin")) + '())) + ":")) + + (apply build-docker-image + (append (list #$output + (map store-info-item + (call-with-input-file "profile" + read-reference-graph)) + #$profile + #:repository (manifest->friendly-name + (profile-manifest #$profile)) + #:database #+database + #:system (or #$target %host-type) + #:environment environment + #:entry-point + #$(and entry-point + #~(list (string-append #$profile "/" + #$entry-point))) + #:extra-files directives + #:compressor #+(compressor-command compressor) + #:creation-time (make-time time-utc 0 1)) + (if #$layered-image? + (list #:stream-layered-image + #$stream-layered-image.py) + '()))))))) (gexp->derivation (string-append name ".tar" (compressor-extension compressor)) @@ -681,6 +702,33 @@ (define directives #:target target #:references-graphs `(("profile" ,profile)))) +(define* (docker-layered-image name profile + #:key target + (profile-name "guix-profile") + (compressor (first %compressors)) + entry-point + localstatedir? + (symlinks '()) + (archiver tar) + (extra-options '()) + (layered-image? #t)) + "Return a derivation to construct a Docker image of PROFILE. The image is a +tarball conforming to the Docker Image Specification, compressed with +COMPRESSOR. It can be passed to 'docker load'. If TARGET is true, it must a +be a GNU triplet and it is used to derive the architecture metadata in the +image. If LAYERED-IMAGE? is true, the image will with many of the store paths +being on their own layer to improve sharing between images." + (docker-image name profile + #:target target + #:profile-name profile-name + #:compressor compressor + #:entry-point entry-point + #:localstatedir? localstatedir? + #:symlinks symlinks + #:archiver archiver + #:extra-options extra-options + #:layered-image? layered-image?)) + ;;; ;;; Debian archive format. @@ -1357,6 +1405,7 @@ (define %formats `((tarball . ,self-contained-tarball) (squashfs . ,squashfs-image) (docker . ,docker-image) + (docker-layered . ,docker-layered-image) (deb . ,debian-archive) (rpm . ,rpm-archive))) @@ -1365,15 +1414,17 @@ (define (show-formats) (display (G_ "The supported formats for 'guix pack' are:")) (newline) (display (G_ " - tarball Self-contained tarball, ready to run on another machine")) + tarball Self-contained tarball, ready to run on another machine")) + (display (G_ " + squashfs Squashfs image suitable for Singularity")) (display (G_ " - squashfs Squashfs image suitable for Singularity")) + docker Tarball ready for 'docker load'")) (display (G_ " - docker Tarball ready for 'docker load'")) + docker-layered Tarball with a layered image ready for 'docker load'")) (display (G_ " - deb Debian archive installable via dpkg/apt")) + deb Debian archive installable via dpkg/apt")) (display (G_ " - rpm RPM archive installable via rpm/yum")) + rpm RPM archive installable via rpm/yum")) (newline)) (define (required-option symbol) diff --git a/guix/scripts/system.scm b/guix/scripts/system.scm index c0bc295c00..e9123e679a 100644 --- a/guix/scripts/system.scm +++ b/guix/scripts/system.scm @@ -11,6 +11,7 @@ ;;; Copyright © 2021 Brice Waegeneire ;;; Copyright © 2021 Simon Tournier ;;; Copyright © 2022 Tobias Geerinckx-Rice +;;; Copyright © 2023 Oleg Pykhalov ;;; ;;; This file is part of GNU Guix. ;;; @@ -734,13 +735,15 @@ (define* (system-derivation-for-action image action #:graphic? graphic? #:disk-image-size image-size #:mappings mappings)) - ((image disk-image vm-image docker-image) + ((image disk-image vm-image docker-image docker-layered-image) (when (eq? action 'disk-image) (warning (G_ "'disk-image' is deprecated: use 'image' instead~%"))) (when (eq? action 'vm-image) (warning (G_ "'vm-image' is deprecated: use 'image' instead~%"))) (when (eq? action 'docker-image) (warning (G_ "'docker-image' is deprecated: use 'image' instead~%"))) + (when (eq? action 'docker-layered-image) + (warning (G_ "'docker-layered-image' is deprecated: use 'image' instead~%"))) (lower-object (system-image image)))))) (define (maybe-suggest-running-guix-pull) @@ -987,6 +990,8 @@ (define (show-help) image build a Guix System image\n")) (display (G_ "\ docker-image build a Docker image\n")) + (display (G_ "\ + docker-layered-image build a Docker layered image\n")) (display (G_ "\ init initialize a root file system to run GNU\n")) (display (G_ "\ @@ -1200,7 +1205,7 @@ (define actions '("build" "container" "vm" "vm-image" "image" "disk-image" "list-generations" "describe" "delete-generations" "roll-back" "switch-generation" "search" "edit" - "docker-image")) + "docker-image" "docker-layered-image")) (define (process-action action args opts) "Process ACTION, a sub-command, with the arguments are listed in ARGS. @@ -1249,6 +1254,8 @@ (define save-provenance? (image (let* ((image-type (case action ((vm-image) qcow2-image-type) ((docker-image) docker-image-type) + ((docker-layered-image) + docker-layered-image-type) (else image-type))) (image-size (assoc-ref opts 'image-size)) (volatile? diff --git a/tests/pack.scm b/tests/pack.scm index 87187bb62c..db2208d91c 100644 --- a/tests/pack.scm +++ b/tests/pack.scm @@ -2,6 +2,7 @@ ;;; Copyright © 2017, 2018, 2019, 2020, 2021 Ludovic Courtès ;;; Copyright © 2018 Ricardo Wurmus ;;; Copyright © 2021, 2023 Maxim Cournoyer +;;; Copyright © 2023 Oleg Pykhalov ;;; ;;; This file is part of GNU Guix. ;;; @@ -29,6 +30,7 @@ (define-module (test-pack) #:use-module (guix gexp) #:use-module (guix modules) #:use-module (guix utils) + #:use-module ((guix build utils) #:select (%store-directory)) #:use-module (gnu packages) #:use-module ((gnu packages base) #:select (glibc-utf8-locales)) #:use-module (gnu packages bootstrap) @@ -246,6 +248,52 @@ (define bin (mkdir #$output))))))) (built-derivations (list check)))) + (unless store (test-skip 1)) + (test-assertm "docker-layered-image + localstatedir" store + (mlet* %store-monad + ((guile (set-guile-for-build (default-guile))) + (profile -> (profile + (content (packages->manifest (list %bootstrap-guile))) + (hooks '()) + (locales? #f))) + (tarball (docker-layered-image "docker-pack" profile + #:symlinks '(("/bin/Guile" -> "bin/guile")) + #:localstatedir? #t)) + (check (gexp->derivation "check-tarball" + (with-imported-modules '((guix build utils)) + #~(begin + (use-modules (guix build utils) + (ice-9 match)) + + (define bin + (string-append "." #$profile "/bin")) + + (define store + (string-append "." #$(%store-directory))) + + (setenv "PATH" (string-append #$%tar-bootstrap "/bin")) + (mkdir "base") + (with-directory-excursion "base" + (invoke "tar" "xvf" #$tarball)) + + (match (find-files "base" "layer.tar") + ((layers ...) + (for-each (lambda (layer) + (invoke "tar" "xvf" layer) + (invoke "chmod" "--recursive" "u+w" store)) + layers))) + + (when + (and (file-exists? (string-append bin "/guile")) + (file-exists? "var/guix/db/db.sqlite") + (file-is-directory? "tmp") + (string=? (string-append #$%bootstrap-guile "/bin") + (pk 'binlink (readlink bin))) + (string=? (string-append #$profile "/bin/guile") + (pk 'guilelink (readlink "bin/Guile")))) + (mkdir #$output))))))) + (built-derivations (list check)))) + (unless store (test-skip 1)) (test-assertm "squashfs-image + localstatedir" store (mlet* %store-monad -- 2.38.0 From debbugs-submit-bounces@debbugs.gnu.org Sun Mar 12 20:34:07 2023 Received: (at 62153) by debbugs.gnu.org; 13 Mar 2023 00:34:07 +0000 Received: from localhost ([127.0.0.1]:33239 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pbW8d-0007Mw-DY for submit@debbugs.gnu.org; Sun, 12 Mar 2023 20:34:07 -0400 Received: from mail-lf1-f51.google.com ([209.85.167.51]:35569) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pbW8b-0007MK-Li for 62153@debbugs.gnu.org; Sun, 12 Mar 2023 20:34:06 -0400 Received: by mail-lf1-f51.google.com with SMTP id m4so649537lfj.2 for <62153@debbugs.gnu.org>; Sun, 12 Mar 2023 17:34:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1678667639; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=5+QlkdvlBqADPCTpd7vxMVVl8vnmFTBVbpRShmZs41M=; b=SbrBuDwophAGJbqlqh4Dz1PjBexR0wxgMkvvlpWJC8JOJnKSPzRJMyhn0ThSBt+iLI 5eRWaluLZ/LsmCJPSq1KGaoJEJ60NXCjADUCplDj4NLYIFBkA+pq5n+9yyoD8Ww7I2VS AK7fzCY5KPmlWWMPDE42A/sJAEZSJSiDYo7fHrPUSxbfiOVmp0t11UDzHSylqOV9eX4a Sux8F62rJcAVETY7kdiBMPyJu+lSwspJZa7nAd14Lo8w7r2R9SJ/jpZFNU+1gcp92K1S 1qWYy2Qj8XtyzY1owV2BCEQkFNDVSOrcKU1/7Pip4THRL7Jb6cbQtkOjTkPF8t6yU4zA WiiA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678667639; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=5+QlkdvlBqADPCTpd7vxMVVl8vnmFTBVbpRShmZs41M=; b=EdoznwhIL0nEcUGg6itdHsH3lKKzZ1G9YzwmvjjCZwhkkhEpDpdddD96qz6ym2xtod SLQ7hqUhJxDAP8cHmr26vdRBItaNjzfBwuYBdWnbLxRFpz7lxh/DLQhYzIANKPqfIKe+ uZdTXAOeNRARRpr1gRMiHbX7+G0K6fCUDDc+PrYdPXBXOBrFpavs7QHHBijAYOAp0+VU vT8TMbq7X7W9GK7nk/qmDaXDl/OrFiLlzYSwbQTL7UrRm9lmYKq2S9cU0zGE4vV7TIRl uwMpTe6tz+bT1VDn90Vo5xlcOmaenHgCdXaK/hMQwWgz8j0Yw8wd+qcXaVbwRhfsc+ga xTQg== X-Gm-Message-State: AO0yUKWPevxkfdGu/gUJnC5BvqbnalN1+UG5FIo9Bi7d0a86nGtpXO8u 7+WKVNVrabug/esv3hcQi7FBAX5C5Cs= X-Google-Smtp-Source: AK7set91qXzICAxyGhsoFNwlmQAg6LR3nNkUk0T2iS68CfY2DhRK/GNKHpE4/tN40UncoYcyG/BD8w== X-Received: by 2002:ac2:5a50:0:b0:4dd:a025:d87 with SMTP id r16-20020ac25a50000000b004dda0250d87mr2389941lfn.0.1678667639408; Sun, 12 Mar 2023 17:33:59 -0700 (PDT) Received: from guixsd.wugi.info ([88.201.161.72]) by smtp.gmail.com with ESMTPSA id t22-20020ac24c16000000b004cb41b43c25sm781659lfq.197.2023.03.12.17.33.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 12 Mar 2023 17:33:59 -0700 (PDT) From: Oleg Pykhalov To: 62153@debbugs.gnu.org Subject: [PATCH 2/2] news: Add entry for the new 'docker-layered' distribution format. Date: Mon, 13 Mar 2023 03:33:10 +0300 Message-Id: <20230313003310.17129-2-go.wigust@gmail.com> X-Mailer: git-send-email 2.38.0 In-Reply-To: <20230313003310.17129-1-go.wigust@gmail.com> References: <20230313003310.17129-1-go.wigust@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 62153 Cc: Oleg Pykhalov X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) * etc/news.scm: Add entry. --- etc/news.scm | 38 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+) diff --git a/etc/news.scm b/etc/news.scm index 924c2b35b4..98fb7f536c 100644 --- a/etc/news.scm +++ b/etc/news.scm @@ -18,6 +18,7 @@ ;; Copyright © 2021 Andrew Tropin ;; Copyright © 2021 Jonathan Brielmaier ;; Copyright © 2022 Thiago Jung Bauermann +;; Copyright © 2023 Oleg Pykhalov ;; ;; Copying and distribution of this file, with or without modification, are ;; permitted in any medium without royalty provided the copyright notice and @@ -26,6 +27,43 @@ (channel-news (version 0) + (entry (commit "45777c5b753ce330ad007d4e71189cf3fc627ccc") + (title + (en "New @samp{docker-layered} format for the @command{guix pack} command") + (ru "Новый @samp{docker-layered} формат для @command{guix pack} команды")) + (body + (en "Docker layered image can now be produced via the @command{guix +pack --format=docker-layered} command, providing a Docker image with many of +the store paths being on their own layer to improve sharing between images. +The image is realized into the GNU store as a gzipped tarball. Here is a +simple example that generates a layered Docker image for the @code{hello} +package: + +@example +guix pack --format=docker-layered --symlink=/usr/bin/hello=bin/hello hello +@end example + +See @command{info \"(guix) Invoking guix pack\"} for more information. + +@command{guix system image} can now produce layered Docker image by passing +@code{docker-layered} to @option{--image-type} option. +") + (ru "Появилась команда создания многослойных Docker образов с помощью +@command{guix pack --format=docker-layered}, которая соберет Docker образ с +путями в store расположенными на отдельных слоях, ускоряя таким образом +передачу образов. Образ будет создан в GNU store в качестве gzipped tarball. + +Пример создания Docker layered image с @code{hello} пакетом: +@example +guix pack --format=docker-layered --symlink=/usr/bin/hello=bin/hello hello +@end example + +Смотрите @command{info \"(guix) Invoking guix pack\"} для получения более +детальных сведений. + +@command{guix system image} теперь может создавать layered Docker image путем +указания в опции @option{--image-type} параметра @code{docker-layered}."))) + (entry (commit "598f4c509bbfec2b983a8ee246cce0a0fe45ec7f") (title (de "Neues Format @samp{rpm} für den Befehl @command{guix pack}") -- 2.38.0 From debbugs-submit-bounces@debbugs.gnu.org Sun Mar 12 20:43:21 2023 Received: (at 62153) by debbugs.gnu.org; 13 Mar 2023 00:43:21 +0000 Received: from localhost ([127.0.0.1]:33249 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pbWHZ-0007aZ-4z for submit@debbugs.gnu.org; Sun, 12 Mar 2023 20:43:21 -0400 Received: from mail-lf1-f48.google.com ([209.85.167.48]:38844) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pbWHX-0007aJ-9N for 62153@debbugs.gnu.org; Sun, 12 Mar 2023 20:43:19 -0400 Received: by mail-lf1-f48.google.com with SMTP id m6so13678398lfq.5 for <62153@debbugs.gnu.org>; Sun, 12 Mar 2023 17:43:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1678668193; h=mime-version:message-id:date:subject:to:from:from:to:cc:subject :date:message-id:reply-to; bh=AaFinQliBwH2sLq8qnQXn07Q4rq7CQxUpNm8xr6kqr4=; b=Vmx9YNeQtJ4NtMPhm436rz4D8BjjfHOmJQrSMSOLdEgp4bPBGmhj8uNd96W7teZYSk 71TyZarb+7/ZNs3HIDaYc6/fAE4V7qwSlzEFV4EEjBxFJTPtOUENoj1gg0x2lPo1iMkJ 2HXNUb+NvyWsIcjDfGyg3aEpQZ8GvAmWe2G5buyjFCs12H88VsiGx8AWkM89AsySO4Wm FH1iX/IcppuRbrkSm2xE10XUYWPch6H9B/XQYqgaQm1VymCHpMBl54bu2MKoEA2uqH4v YttRqD8mKdIDD0AO9lnHudKiT1WrivWnD/nAMkKZLLksrIbqeMD5SweiACi3+iUYt6IZ blQQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678668193; h=mime-version:message-id:date:subject:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=AaFinQliBwH2sLq8qnQXn07Q4rq7CQxUpNm8xr6kqr4=; b=r2nBjhqT6RDv0VbT0FTtaJhsRMhzaKf5j93jZzipCz4bYNhrTbPh7NJyDGdPYKeOY2 6muWMRkWwiuN59Dv+mjl92xNEVlM2K+jP7dTkOgtGWQMgIIoYy4m0MOhaVUuvbNN0XYw WlhdVFhO7tATywT9HcqqAbUKdqu64LBJBZctrJL+bCU854stIeDzaZfnIV/OuUqEino3 ehXvO3q39amVMuA4PEqlws+R32GuKV0csdR2drO5wViPHPRZ3YkpUPpO78r05dM9PhV5 5DBPjR7aRoCGMFM60SCY/6xw1v5FMopKKYvBPwXeA7EvrgUw7cA5PhfaGBrsoZgu+3Ze VMNQ== X-Gm-Message-State: AO0yUKVwcG/lbuV7BNEv7PTtqtyC8+vBp8tpyvwsqadmxY0dnE8kmJ89 avcqxTevx58CaPNSoFVGVqIkaXf9UK0= X-Google-Smtp-Source: AK7set8Wj6ofC+NUgTyqCqNlj3qL1GFwHSyWhkdQfxQU/bbR3VrVQ3VGcKj97PXgdATJycy7qjqrsg== X-Received: by 2002:ac2:5a46:0:b0:4b6:f595:cfe1 with SMTP id r6-20020ac25a46000000b004b6f595cfe1mr2540076lfn.5.1678668192865; Sun, 12 Mar 2023 17:43:12 -0700 (PDT) Received: from localhost ([88.201.161.72]) by smtp.gmail.com with ESMTPSA id y28-20020ac2447c000000b004db45648d78sm798402lfl.13.2023.03.12.17.43.12 for <62153@debbugs.gnu.org> (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 12 Mar 2023 17:43:12 -0700 (PDT) From: Oleg Pykhalov To: 62153@debbugs.gnu.org Subject: Cover lever typo in guix pack format example Date: Mon, 13 Mar 2023 03:43:11 +0300 Message-ID: <87wn3l32hs.fsf@gmail.com> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 62153 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) --=-=-= Content-Type: text/plain The cover lever guix pack example should be: ./pre-inst-env guix pack -f docker-layered --entry-point=bin/bash -S /bin=bin bash hello instead of ./pre-inst-env guix pack -f docker --entry-point=bin/bash -S /bin=bin bash hello Apologies, Oleg. --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQJIBAEBCgAyFiEEcjhxI46s62NFSFhXFn+OpQAa+pwFAmQOcZ8UHGdvLndpZ3Vz dEBnbWFpbC5jb20ACgkQFn+OpQAa+pzRkg/9H8+149MGcejnPteSR7C1XKWckEWi xoADRq5qcwFf9JTGGNEDqEoq/z+BFCa5ainGpLKO6jzVGe0gCfzsmmxRA7V5fZY3 udwfPwVmhb3oV4CV8t8Gi3tqTnJ3meUG7CdzNttocYRlwLys2TUoy5t8fJ9xwg/v 6ZSVeKfcXJdM9BP5a+KkLMzfxGOGlfu8HFVIrVlPAMgRmliZLRKFdk3mmZ7jJEQv OepVWlQ5j6BRHfbETEYz6nB7Hw2BYzfL5dwzkrsMBwO2KgtG9sYSgdqIbRVJXcP9 L3R5Z1Z51p4CcByYG35iq0ahwGy9daUL1UiCx8gIXJPEy7RGMFqjOAGekg+lN2PI qOHLMpthqJBKQxKnYBxp7ZdgnvwX4iYs7YrUDDX6I507L1zYABWNM2exLl1N3PUF vlHKUmI2la4ozWDaHNZsTa7qVtFtRwXr5Vs8hfkbqfEhAG3YHQKd0zMBr33Ht6bn WYdT6zKq2Dnp3BjK1dkOKYhIJelmWlwptUYv88h7KiozgEOI84l8RpSOU7IXnEBY CUhhGhPp2xu4w7NCkOA3TAYHGYFT2ebFnlZ+l3fTq6i8Y3p9LVD65Jvz9Pqgt0MQ 9v7+ArClkWYmKxc0XNrqpv2J8mAr5PlSvfXtHM0wTsZk+cpuaC/xIkCf4upHyLyh 1a4qZJxNTV9FjLA= =z9Ac -----END PGP SIGNATURE----- --=-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Mon Mar 13 11:02:15 2023 Received: (at 62153) by debbugs.gnu.org; 13 Mar 2023 15:02:15 +0000 Received: from localhost ([127.0.0.1]:35390 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pbjgl-0002B9-Eb for submit@debbugs.gnu.org; Mon, 13 Mar 2023 11:02:15 -0400 Received: from mail-wm1-f46.google.com ([209.85.128.46]:40910) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pbjgh-0002AN-1m for 62153@debbugs.gnu.org; Mon, 13 Mar 2023 11:02:11 -0400 Received: by mail-wm1-f46.google.com with SMTP id fm20-20020a05600c0c1400b003ead37e6588so11137022wmb.5 for <62153@debbugs.gnu.org>; Mon, 13 Mar 2023 08:02:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1678719725; h=content-transfer-encoding:mime-version:message-id:date:references :in-reply-to:subject:cc:to:from:from:to:cc:subject:date:message-id :reply-to; bh=py+hcS6AbjfqmB3vtixBEB3yE6pURlqTB9FJKmGjyeI=; b=c/kJMTODpAcq1VqQ4yJvrepe5+wSC2cU1RwJToqjmU0GyIT53M4FUZY8I/No8FHZ94 SKMwXpKOdFQVpcv41GF4U60xoEoC3cnkwyuY/I25wZ37w02iR48CfUomx8+691amDXEA 9BqZX2GOwoQdAXocZIcNQuwuvc8na99P1caSS6RTrhNBWnGmYUYsxmk/zxbmqeHgaiGJ dMVT0DJ9UE65EROffR8iVMHpd0tLHhy4/jj4y7r/488SdjmMy3bJ/nD8nIRmHEPRqtdU QBW9bcNE5nfkl7V6fVBec7LH7+UmBWU5bWjvT2APzInEHF/vfE+0/wtVb0cIa7ZNOAZr 2kww== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678719725; h=content-transfer-encoding:mime-version:message-id:date:references :in-reply-to:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=py+hcS6AbjfqmB3vtixBEB3yE6pURlqTB9FJKmGjyeI=; b=NSfzRGLhU2hW4pW+USkoZWXITISJmg/WGrgCjBFzTz7cZip6aWMIBVfhMnYkWTbsnB JzflNRct6jSskIwbmwuZ2nq2ggl40jAubMasatZ9ujwnFN0V/KOoAzqj315yne2Z1rZB MlMfRrU/jrSJm8y0YP/NhuK+3ifDqjdA2AXXVdf3fR1AAOYE4uQ+eWggVbqT444I4k6r 84hdCuceJnHSeMJE4oB/wlew9/y4RKJDpzY4Fqn9nfoYtJjJljf+vd68S4YFka9dGskr Z+G2H5olkF9a/LyK/oNn66jLYuTo4PSOk6BPeoaQdzqraRY+AXSQGNYbGG7QLvndZgRB fhDA== X-Gm-Message-State: AO0yUKVk+wUx0fPITzZdp1aghK0A2dGu2d21kgoEDtATZyH6UbWqK1AA pgNUvL/ezca4xeoHjoHP4bs= X-Google-Smtp-Source: AK7set8CqQTfdll5HPJaOneUJigyvqfg7EZd2wuFaomd4nY6rH8sm2x2OxsjgBXKH0uqhkIkFNYYfw== X-Received: by 2002:a05:600c:3c9d:b0:3e2:17c2:fc8 with SMTP id bg29-20020a05600c3c9d00b003e217c20fc8mr12683713wmb.2.1678719725354; Mon, 13 Mar 2023 08:02:05 -0700 (PDT) Received: from pfiuh07 ([193.48.40.241]) by smtp.gmail.com with ESMTPSA id o13-20020a05600c510d00b003e20970175dsm10468044wms.32.2023.03.13.08.02.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 13 Mar 2023 08:02:05 -0700 (PDT) From: Simon Tournier To: Oleg Pykhalov , 62153@debbugs.gnu.org Subject: Re: [bug#62153] [PATCH 1/2] guix: docker: Build layered image. In-Reply-To: <20230313003310.17129-1-go.wigust@gmail.com> References: <20230313003012.14325-1-go.wigust@gmail.com> <20230313003310.17129-1-go.wigust@gmail.com> Date: Mon, 13 Mar 2023 16:01:04 +0100 Message-ID: <87r0tsk85r.fsf@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 62153 Cc: Oleg Pykhalov X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Hi, Oh cool! Awesome! Thanks for pushing forward. On lun., 13 mars 2023 at 03:33, Oleg Pykhalov wrote: > diff --git a/gnu/packages/aux-files/python/stream-layered-image.py b/gnu/= packages/aux-files/python/stream-layered-image.py > new file mode 100644 > index 0000000000..9ad2168c2d > --- /dev/null > +++ b/gnu/packages/aux-files/python/stream-layered-image.py > @@ -0,0 +1,391 @@ > +""" > +This script generates a Docker image from a set of store paths. Uses > +Docker Image Specification v1.2 as reference [1]. Instead of Python, would it possible to implement in Guile? I mean, does Python have something that is missing in Guile? The facility for manipulating Tar? Something else? Because then, if I understand correctly=E2=80=A6 > diff --git a/guix/docker.scm b/guix/docker.scm > index 5e6460f43f..f1adad26dc 100644 > --- a/guix/docker.scm > +++ b/guix/docker.scm [...] > + (if stream-layered-image > + (let ((input (open-pipe* OPEN_READ "python3" > + stream-layered-image > + "config.json"))) =E2=80=A6it requires to drag Python for building/packing layered Docker. Well, I have not really look yet to the Python script which does most of the job. Do you use a similar strategy as [1]? And I remember something in that direction by Chris but I am unable to find back the patch. )-: 1: https://grahamc.com/blog/nix-and-layered-docker-images/ Cheers, simon From debbugs-submit-bounces@debbugs.gnu.org Mon Mar 13 17:09:20 2023 Received: (at 62153) by debbugs.gnu.org; 13 Mar 2023 21:09:20 +0000 Received: from localhost ([127.0.0.1]:35709 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pbpQ0-0005nF-68 for submit@debbugs.gnu.org; Mon, 13 Mar 2023 17:09:20 -0400 Received: from relay.yourmailgateway.de ([188.68.63.166]:40081) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pbpPy-0005n6-DY for 62153@debbugs.gnu.org; Mon, 13 Mar 2023 17:09:19 -0400 Received: from mors-relay-8202.netcup.net (localhost [127.0.0.1]) by mors-relay-8202.netcup.net (Postfix) with ESMTPS id 4Pb8RJ2Rg4z3qbp; Mon, 13 Mar 2023 22:09:16 +0100 (CET) Authentication-Results: mors-relay-8202.netcup.net; dkim=permerror (bad message/signature format) Received: from policy02-mors.netcup.net (unknown [46.38.225.35]) by mors-relay-8202.netcup.net (Postfix) with ESMTPS id 4Pb8RJ22vfz3qbR; Mon, 13 Mar 2023 22:09:16 +0100 (CET) X-Virus-Scanned: Debian amavisd-new at policy02-mors.netcup.net X-Spam-Flag: NO X-Spam-Score: -2.9 X-Spam-Level: X-Spam-Status: No, score=-2.9 required=6.31 tests=[ALL_TRUSTED=-1, BAYES_00=-1.9] autolearn=ham autolearn_force=no Received: from mxe217.netcup.net (unknown [10.243.12.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by policy02-mors.netcup.net (Postfix) with ESMTPS id 4Pb8RH5qLdz8svP; Mon, 13 Mar 2023 22:09:12 +0100 (CET) Received: from florianrock64 (ip92344de0.dynamic.kabel-deutschland.de [146.52.77.224]) by mxe217.netcup.net (Postfix) with ESMTPSA id 771A880A68; Mon, 13 Mar 2023 22:09:07 +0100 (CET) From: "pelzflorian (Florian Pelz)" To: Oleg Pykhalov Subject: Re: [bug#62153] [PATCH 2/2] news: Add entry for the new 'docker-layered' distribution format. References: <20230313003310.17129-1-go.wigust@gmail.com> <20230313003310.17129-2-go.wigust@gmail.com> Date: Mon, 13 Mar 2023 22:09:06 +0100 In-Reply-To: <20230313003310.17129-2-go.wigust@gmail.com> (Oleg Pykhalov's message of "Mon, 13 Mar 2023 03:33:10 +0300") Message-ID: <87cz5ccqa5.fsf@pelzflorian.de> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 771A880A68 X-Spamd-Result: default: False [-5.60 / 15.00]; BAYES_HAM(-5.50)[100.00%]; MIME_GOOD(-0.10)[text/plain]; RCVD_COUNT_ZERO(0.00)[0]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; ASN(0.00)[asn:3209, ipnet:146.52.0.0/16, country:DE]; MIME_TRACE(0.00)[0:+]; MID_RHS_MATCH_FROM(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; FREEMAIL_ENVRCPT(0.00)[gmail.com]; FROM_HAS_DN(0.00)[]; TAGGED_RCPT(0.00)[]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; ARC_NA(0.00)[] X-Rspamd-Server: rspamd-worker-8404 X-NC-CID: f+VhsuhAjkECKF0AQoX8UIDZUuy1cYPVIF5vvI/JBhnTzVcZuJ+1n/Cd X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 62153 Cc: 62153@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Thank you Oleg for this feature. Could you change the following three things in the news: Change the beginning of the English translation from "Docker layered image can now be produced" to "Docker layered images can now be produced". And at the end, also in Russian, switch around these two paragraphs and add a =E2=80=9Cthe=E2=80=9D and reference the System Images chapter: > @command{guix system image} can now produce layered Docker image by passi= ng > @code{docker-layered} to the @option{--image-type} option. >=20 > See @command{info \"(guix) Invoking guix pack\"} and > @command{info \"(guix) System Images\"} for more information. Lastly, could you then also add a German translation: (title =E2=80=A6 (de "Neues Format @samp{docker-layered} f=C3=BCr den Befehl @command{guix = pack}") (body =E2=80=A6 (de "Sie k=C3=B6nnen jetzt auch mehrschichtige Docker-Abbilder mit dem Bef= ehl @command{guix pack --format=3Ddocker-layered} erzeugen. Damit bekommen Sie = ein Docker-Abbild, bei dem Store-Pfade auf getrennten Schichten (=E2=80=9ELayer= =E2=80=9C) untergebracht sind, die sich mehrere Abbilder teilen k=C3=B6nnen. Das Abbi= ld wird im Store als gzip-komprimierter Tarball erzeugt. Hier ist ein einfaches Beispiel, wo ein mehrschichtiges Docker-Abbild f=C3=BCr das Paket @code{hel= lo} angelegt wird: @example guix pack --format=3Ddocker-layered --symlink=3D/usr/bin/hello=3Dbin/hello = hello @end example @command{guix system image} kann jetzt geschichtete Docker-Abbilder erzeuge= n, indem Sie @code{docker-layered} an die Befehlszeilenoption @option{--image-= type} =C3=BCbergeben. Siehe @command{info \"(guix.de) Aufruf von guix pack\"} und @command{info \"(guix.de) Systemabbilder\"} f=C3=BCr weitere Informationen.= ") Regards, Florian From debbugs-submit-bounces@debbugs.gnu.org Mon Mar 13 17:11:07 2023 Received: (at 62153) by debbugs.gnu.org; 13 Mar 2023 21:11:07 +0000 Received: from localhost ([127.0.0.1]:35713 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pbpRi-0005q9-PH for submit@debbugs.gnu.org; Mon, 13 Mar 2023 17:11:07 -0400 Received: from mail-lj1-f170.google.com ([209.85.208.170]:39707) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pbpRg-0005pe-Dm for 62153@debbugs.gnu.org; Mon, 13 Mar 2023 17:11:05 -0400 Received: by mail-lj1-f170.google.com with SMTP id b13so14031430ljf.6 for <62153@debbugs.gnu.org>; Mon, 13 Mar 2023 14:11:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1678741858; h=mime-version:user-agent:message-id:in-reply-to:date:references :subject:cc:to:from:from:to:cc:subject:date:message-id:reply-to; bh=4c7j0i2EFC6jpMuSPWTkkV+r7WutTPeyJ8Svn9EyFio=; b=STvNpsN6Yhr0ULBDdTC+iHNoUuV5FOUPZtwc2Fx7O4e7TGTlHorknzlL4newlTkcS8 3JmcGmB6ouxAYS9QMsmpCOktFsrIftjr+DV+Y6K0w8Nkj1TSp0h98bCbwmxvIpzEL2Qf +sBEaM1kjc8d64kMLpIoRbwOOhWkeeCzqbLz5D6Yr8CHuKglygffFrfUFmFpXCtsmNer ex3+3MtZh3aF9KIrV+cp8xuMLZ8oVKOBoh5yXJWuODA0oadFrNnJzfkpAPA9Q53rtwYD M7YtiMq5nmSdwYMnKkRaJscuH7zJXh6Gvxu9afIy8JSMPl/eASxFP6mYp8G7w73Qdkui GGiw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678741858; h=mime-version:user-agent:message-id:in-reply-to:date:references :subject:cc:to:from:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=4c7j0i2EFC6jpMuSPWTkkV+r7WutTPeyJ8Svn9EyFio=; b=ngULmHTPCxOBua/MlaeIhsDCLL+xg9TWA/rAGoK9u9shRuVoqspqgrny76mqPc2PMJ HAHlY4stnHQ9uLPKkQs3z9Ix+eQQvG7CBZxHKWBGHDV9SY8R1eBRkSlccdgXLDBZQWto Jdf1Vmr6CsOkz5nFW783a1LJb9E+CvTIf6n5DR4833J2OB6xCoDwWE5Hn7GgmNv8RznO hiLlzu0s8Oj2Z4h0daByLYJMKP6xlHxiN5ItlhIqlPvKYd4amHEmvikFxYHuE+x/8Sqx MkunPo075URMFiHwjPfr1AyZGuPo1X9RNh6yZt85jclgIfQaGDRTTwO2JN8xF3NdDVtm RZYg== X-Gm-Message-State: AO0yUKX24thZo3nei4yph+RXZjl+5JI4KHWwhIiyldpBjdBzE2ybCyWk SnWqv4ECmKPwBnM97fO/GmE= X-Google-Smtp-Source: AK7set9di6UqWrIYphHLNYnfDooguPpk4joROlhw8Ztq8X7KGT+rLkImLSrluTHlajhViTf7ehqgow== X-Received: by 2002:a05:651c:1a2c:b0:295:a08c:12a1 with SMTP id by44-20020a05651c1a2c00b00295a08c12a1mr5160313ljb.0.1678741858001; Mon, 13 Mar 2023 14:10:58 -0700 (PDT) Received: from localhost ([88.201.161.72]) by smtp.gmail.com with ESMTPSA id t6-20020a2e9c46000000b00297dad1a2b0sm143309ljj.103.2023.03.13.14.10.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 13 Mar 2023 14:10:57 -0700 (PDT) From: Oleg Pykhalov To: Simon Tournier Subject: Re: [bug#62153] [PATCH 1/2] guix: docker: Build layered image. References: <20230313003012.14325-1-go.wigust@gmail.com> <20230313003310.17129-1-go.wigust@gmail.com> <87r0tsk85r.fsf@gmail.com> Date: Tue, 14 Mar 2023 00:10:56 +0300 In-Reply-To: <87r0tsk85r.fsf@gmail.com> (Simon Tournier's message of "Mon, 13 Mar 2023 16:01:04 +0100") Message-ID: <87bkkw2w7z.fsf@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (gnu/linux) MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 62153 Cc: 62153@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hi Simon, Thank you for the review. Simon Tournier writes: > On lun., 13 mars 2023 at 03:33, Oleg Pykhalov wrote: > >> diff --git a/gnu/packages/aux-files/python/stream-layered-image.py b/gnu= /packages/aux-files/python/stream-layered-image.py >> new file mode 100644 >> index 0000000000..9ad2168c2d >> --- /dev/null >> +++ b/gnu/packages/aux-files/python/stream-layered-image.py >> @@ -0,0 +1,391 @@ >> +""" >> +This script generates a Docker image from a set of store paths. Uses >> +Docker Image Specification v1.2 as reference [1]. > > Instead of Python, would it possible to implement in Guile? I mean, > does Python have something that is missing in Guile? > > The facility for manipulating Tar? Something else? I think nothing else. As I understand Python implemented Tar inside the language itself in 2500 lines of code by manipulating binary data. /gnu/store/...-python-3.9.9/lib/python3.9/tarfile.py Technically it's probably possible to use tar utility with --append flag instead of opening a new file and streaming to it as the Python script does. To be honest I would like not to write it in this way if the Python script does not block current patch for merge. Also I don't see myself writing Tar implementation in Guile, yet. ;-) The Nix project uses this script heavily to build layered images, so it should be robust in terms of up to date to current Tar and Python implementations. > Because then, if I understand correctly=E2=80=A6 > >> diff --git a/guix/docker.scm b/guix/docker.scm >> index 5e6460f43f..f1adad26dc 100644 >> --- a/guix/docker.scm >> +++ b/guix/docker.scm > > [...] > >> + (if stream-layered-image >> + (let ((input (open-pipe* OPEN_READ "python3" >> + stream-layered-image >> + "config.json"))) > > =E2=80=A6it requires to drag Python for building/packing layered Docker. Correct. > Well, I have not really look yet to the Python script which does most of > the job. Do you use a similar strategy as [1]? > > And I remember something in that direction by Chris but I am unable to > find back the patch. )-: > > 1: https://grahamc.com/blog/nix-and-layered-docker-images/ Not similar. My patch implements a very simple sorting by size, no complex sorting by reference popularity as in [1], which is probably implemented in the following file github.com/NixOS/nixpkgs/pkgs/build-support/references-by-popularity/clo= sure-graph.py From=20https://grahamc.com/blog/nix-and-layered-docker-images/ article: > How Docker really represents an Image > > Docker=E2=80=99s layers are content addressable and aren=E2=80=99t requir= ed to > explicitly reference a parent layer. This means a layer for > readline-7.0p5 doesn=E2=80=99t have to mention that it has any relationsh= ip to > ncurses-6.1 or glibc-2.27 at all. > > Instead each image has a manifest which defines the order: > > { > "Layers": [ > "bash-interactive-4.4-p23", > "bash-4.4p23", > "readline-7.0p5", > ... > ] > } > > If you have only built Docker images using a Dockerfile, then you > would expect the way we flatten our graph to be critically > important. If we sometimes picked readline-7.0p5 to come first and > other times picked bash-4.4p23 then we may never make cache hits. > > However since the Image defines the order, we don=E2=80=99t have to solve= this > impossible problem: we can order the layers in any way we want and the > layer cache will always hit. In case of sorting by size, bigest layers will be on top of a container image, which will produce a cache hit for bigest directories in the GNU store during images transfer with same layers. I would like to say this sorting could binifit more than sorting by popularity during transfer but let's assume I didn't write it. ;-) The following example shows common layers between images, which will be not tranfered if you load image inside Docker as well as pull and push: ./pre-inst-env guix pack -f docker-layered --entry-point=3Dbin/bash -S = /bin=3Dbin bash hello and ./pre-inst-env guix pack -f docker-layered --entry-point=3Dbin/bash -S = /bin=3Dbin bash hello emacs share 6 layers in total =2D-8<---------------cut here---------------start------------->8--- $ f() { docker image inspect "$1" | jq --raw-output '.[0].RootFS.Layers[] |= .' | sort ; } $ comm -1 -2 --total <(f sha256:fb43b32380a5e6a867410721f4ce2917db14d4ae943= c433983afbaf84416c421) <(f sha256:0ce4a11973d1071aeec5441db228d6148dfd09fea= 3ae77b731c750ebfcc2fe1d) sha256:3b3daa2a00f1acd12eeb16698bf1caeb6ba6c436e3dbca6259c3a9c622664e00 sha256:5c2be7469293854257221cb6aa8aa4af1e10e2c550935390dbcfeede3d3fbacd sha256:60317981d94928659389f299e4b86703e5ded420a53537d67627952187fbd3f9 sha256:6d7c8ce5441d4c4c74e0ecff6c203a7b265b37137cca3b0a0ccf10526cfaa6e2 sha256:c2ded2ffe3f46fa7a64a62e0fc6b9d28cb7d4f8d9c64d5a52d137a508cba11fc sha256:fbcad85d7d3c25bd2aa6d95bb3bf3d02c499ee3b3e443ddd3e5b679c2b33c139 5 94 6 total =2D-8<---------------cut here---------------end--------------->8--- Regards, Oleg. --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQJIBAEBCgAyFiEEcjhxI46s62NFSFhXFn+OpQAa+pwFAmQPkWAUHGdvLndpZ3Vz dEBnbWFpbC5jb20ACgkQFn+OpQAa+pzFZBAApU6Atd2czoZgSRir9SG0/V/l7yva zKst6JvXL77QxVU+e0QqVcY/o8rJg9hIGH+cQaOmlxmbYXJExEI1Go+tngk61OgQ dsMrdBBPkaXvqzd7PKQRUtCn221CGfoMMfZGEzCUyQFTYgo6+K7s76Ep6++lODEW nV/nPa/gqKhg6IZ2G+NeuNKeHsFN8YH/U3Si8myvyLSt0B/ZZEb/8eMumhl3lt61 4GXPPhlFXvp/8VebhMbvzN7TUGWl4z8uMpVExYdDjG3BxkcJBqlEfGfJPQlWc18E +8G93/JCFYhtE/ae0d/qCpVs8k7CLRrJEkllQFARau+e0GxNggdA08mACmuxZqTF u1t52FVvg+cgIO5XKN0x0HyR0t+Rv6gkZPA1b4EXF6t8XRCkdh5VXZr/v3Wvwea2 2wbTeX2vDeca3coyCxDNRjka+FgcLrcxxVBZNc33/76sgglXjovQI4g4LDsklkV9 pSa8StEt4/lCSPXRUQYy0g/nUO4Lk2QaIVx7FSefqCaS+EwCz6NkMGi8esKrtHuw l1gFvYhMFtHvQjzbO50edR1QyLgbMTFbyJdFhHJOEI0rAeh4AIde+R/FfXbPFZR4 RCPwCij2+Ng0O8lrd9uQcdo92oms6FGHLN+ndgYm38hzMQdI+ecMpAQtlR09efsW WBhi5/HWNf3372o= =r2xS -----END PGP SIGNATURE----- --=-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Mon Mar 13 20:25:50 2023 Received: (at 62153) by debbugs.gnu.org; 14 Mar 2023 00:25:50 +0000 Received: from localhost ([127.0.0.1]:35829 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pbsUA-000512-9f for submit@debbugs.gnu.org; Mon, 13 Mar 2023 20:25:50 -0400 Received: from mail-lf1-f48.google.com ([209.85.167.48]:40765) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pbsU8-00050a-F0 for 62153@debbugs.gnu.org; Mon, 13 Mar 2023 20:25:48 -0400 Received: by mail-lf1-f48.google.com with SMTP id y15so8393756lfa.7 for <62153@debbugs.gnu.org>; Mon, 13 Mar 2023 17:25:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1678753542; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=UUETRAR9EF2mxkW7JtRCqRxFITfXvOzwdcHHPDQFQyQ=; b=HU0YAmQIRLUG3tTDiIhFs808/ahoTlyPQqyDWa/zh79k0jZ2YMfPJt3+bGNIe9KXfz 9JPjAz5ISrVel/0GvnvlGwMeVYuxCK1FMmZRGKJHgJ5A6URn3VbpuQj67XdlM+rV0dIe B8h0qrNrQls6DPYr8ZdgHAduN8Xj9dgqfIeluO+sRd8P4H9YyKJqA8enk9n21j/ATLpO WKwnFUY0mNsVUBiF5NAQKuxNkA2INuPYCWHrMAUwwH4M+ShzjP5BaxN+Vt6lP6Xm/2vA Hpj03q9NPg4Nv00iIKA+YMpRxHxvvSpePoWxp0x+ayw+1tYn0BvESf7j1O5+hDehFSu/ F4Vw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678753542; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=UUETRAR9EF2mxkW7JtRCqRxFITfXvOzwdcHHPDQFQyQ=; b=thEsfAaAnaaozCcng8VxLvHkBPkCgQscJvYZprhnzLXuQyR/dRxC/tkUbvs84c1Z0o TEe4MzlMDnQz+S0Ir+TsacaMJNdrRTaDoGnULzGxoX9TgMrKCVWTlMB3b4BCo1GcS/CO ZLcsx36C2KFC44/Uvw5da9SHyrXPAUu5MycA0BxwCCh1yXKAgms+iFYikaevOO5EgOVP JSDqOrZBaZnrw3UK9J051RKYabGKcE4h7UE3UTUDA/dSVTsBgR0KahQvCZLJXlasUpMv mdWjjlSS8/sjXJNBbvUX6/55QxQL9fOhXj1/af7NO4VzlEFh1Mp3r1C4+mK7Ou4PviIa KicA== X-Gm-Message-State: AO0yUKVBKMLN08X9rNg/vHhi/fuaYO+6d9WTC021VlS8Y77KjzCG/L7+ wvJU2MNjvOZqI1XVfR4tKxQ= X-Google-Smtp-Source: AK7set8zBOf59IY3DFTuNLjainBfPn4XUrGbsgNOFCTqEKIJGeEALymJWYYQqi+GmWOe7MWTlFOOcQ== X-Received: by 2002:a19:f51a:0:b0:4d2:c70a:fe0a with SMTP id j26-20020a19f51a000000b004d2c70afe0amr3436525lfb.2.1678753542064; Mon, 13 Mar 2023 17:25:42 -0700 (PDT) Received: from guixsd.wugi.info ([88.201.161.72]) by smtp.gmail.com with ESMTPSA id e7-20020ac25467000000b0048afb8b8e53sm157297lfn.80.2023.03.13.17.25.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 13 Mar 2023 17:25:41 -0700 (PDT) From: Oleg Pykhalov To: pelzflorian@pelzflorian.de Subject: [PATCH 0/2] Add Docker layered image for pack and system (v2) Date: Tue, 14 Mar 2023 03:24:51 +0300 Message-Id: <20230314002453.24668-1-go.wigust@gmail.com> X-Mailer: git-send-email 2.38.0 In-Reply-To: <87cz5ccqa5.fsf@pelzflorian.de> References: <87cz5ccqa5.fsf@pelzflorian.de> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 62153 Cc: Oleg Pykhalov , 62153@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Hi, Florian. Thank you for the review. This patch series applies your suggestions. Also it's rebased on origin/master and added a missing documentation for ‘docker-layered’ format in ‘guix system image’ command in doc/guix.texi file (following diff). <#part type="text/x-patch" buffer=m1.txt disposition=inline description="Add missing docker-layered format documentation for guix system image command"> <#/part> The folloing tests passed: make check-channel-news make check TESTS="tests/pack.scm" make check-system TESTS="docker-system docker-layered-system" Oleg Pykhalov (2): guix: docker: Build layered image. news: Add entry for the new 'docker-layered' distribution format. Makefile.am | 3 +- doc/guix.texi | 18 +- etc/news.scm | 58 +++ gnu/image.scm | 3 +- .../aux-files/python/stream-layered-image.py | 391 ++++++++++++++++++ gnu/system/image.scm | 84 +++- gnu/tests/docker.scm | 20 +- guix/docker.scm | 182 ++++++-- guix/scripts/pack.scm | 105 +++-- guix/scripts/system.scm | 11 +- tests/pack.scm | 48 +++ 11 files changed, 837 insertions(+), 86 deletions(-) create mode 100644 gnu/packages/aux-files/python/stream-layered-image.py base-commit: 5312d798ac36a72d8a977325a7c6ff7647be670a -- 2.38.0 From debbugs-submit-bounces@debbugs.gnu.org Mon Mar 13 20:26:27 2023 Received: (at 62153) by debbugs.gnu.org; 14 Mar 2023 00:26:27 +0000 Received: from localhost ([127.0.0.1]:35833 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pbsUi-00052p-Lv for submit@debbugs.gnu.org; Mon, 13 Mar 2023 20:26:27 -0400 Received: from mail-lf1-f48.google.com ([209.85.167.48]:40765) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pbsUe-00050a-9C for 62153@debbugs.gnu.org; Mon, 13 Mar 2023 20:26:23 -0400 Received: by mail-lf1-f48.google.com with SMTP id y15so8395146lfa.7 for <62153@debbugs.gnu.org>; Mon, 13 Mar 2023 17:26:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1678753580; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=YISYd8IIC1KPcQHum8j/kmpGopZe7MxXpzKLcYkKmXk=; b=DH4FY7Ybjd9AyqAZgSX0DRiD2Jy6ms+HRDPOHRTHEXlUBgofTuK/AIKOwKbg6NAKXM XeRTsnbsK94EsmiEO7niUuohXwWHzx2aTOHdC4aSLPppJ0tY5U6iZR9Ey+lte+EuR72B vt1umwBWjxFGMyEXjF9+Tf5cNgHrhViUD27mQEeJbm3XR8NTF2CzAOGcGY2shJeb3cmM evTWUD5DaGDU2MC0mT6z1zVvzHxbuiU8i5+5MikQW7QI2zEocMhyd+e3N1H13S4ggl2i 2P/KIr8o862dV5O+OONUuOsHF9+JHU/WWLvLpVWEwsfpSP9zdiB3Z1bEkCBKpJ0ap39b +xuQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678753580; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=YISYd8IIC1KPcQHum8j/kmpGopZe7MxXpzKLcYkKmXk=; b=gwnIiqwyud/cmmTqakdHTsCGcpYuBN/CCVtgb7XpMCilIO8Y2VhFaE1wxLU2YrLgT0 6KwwNIzI93QdeDt4S4h6rE2W7YOu3r6oCl+Xx3iWUYyg/eHs7uOOcYVQjfG+wNHCctHb WsdT3g4T04Qe8b+BNKYxAS/4ii5R/xDIXSGZRlkoaR/kyuebQynwxwQCfPcy5y7eWVCn CKviwyrjY9vKexp1RtnQgApViPPlt19i35QxmeJOWdUBXV0qaIp82nIMe9sOL1crU3ob DML2ig/xDkDYYXtGL+j6vaSIU6sUQP6qgTnU5cQML7pKx56FB4K50TermkCcnW+3itr8 Jw8Q== X-Gm-Message-State: AO0yUKXR3X3EEKy6Dg51/0Dikv+I0y5DUzTyTg0r8BeM7S5ObvTtiGjU 5+CszjtEMxtR5dqa0jqgFMM= X-Google-Smtp-Source: AK7set+wtnB2hxPFSxpKm1XofkIM+PShrhHAwqsRhVSQmYqd2+221i4Jj8UbzBp2lx8BkIRqiYC0Kg== X-Received: by 2002:a19:5215:0:b0:4e7:ed3c:68ea with SMTP id m21-20020a195215000000b004e7ed3c68eamr3392508lfb.5.1678753579023; Mon, 13 Mar 2023 17:26:19 -0700 (PDT) Received: from guixsd.wugi.info ([88.201.161.72]) by smtp.gmail.com with ESMTPSA id e7-20020ac25467000000b0048afb8b8e53sm157297lfn.80.2023.03.13.17.26.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 13 Mar 2023 17:26:18 -0700 (PDT) From: Oleg Pykhalov To: pelzflorian@pelzflorian.de Subject: [PATCH 1/2] guix: docker: Build layered image. Date: Tue, 14 Mar 2023 03:24:52 +0300 Message-Id: <20230314002453.24668-2-go.wigust@gmail.com> X-Mailer: git-send-email 2.38.0 In-Reply-To: <20230314002453.24668-1-go.wigust@gmail.com> References: <87cz5ccqa5.fsf@pelzflorian.de> <20230314002453.24668-1-go.wigust@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 62153 Cc: Oleg Pykhalov , 62153@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) * gnu/packages/aux-files/python/stream-layered-image.py: New file. * Makefile.am (AUX_FILES): Add this. * guix/docker.scm (%docker-image-max-layers): New variable. (build-docker-image)[stream-layered-image, root-system]: New arguments. * guix/scripts/pack.scm (stream-layered-image.py): New variable. (docker-image)[layered-image?]: New argument. (docker-layered-image): New procedure. (%formats)[docker-layered]: New format. (show-formats): Document this. * tests/pack.scm: Add docker-layered-image + localstatedir test. * guix/scripts/system.scm (system-derivation-for-action)[docker-layered-image]: New action. (show-help): Document this. (actions)[docker-layered-image]: New action. (process-action): Add this. * gnu/system/image.scm (docker-layered-image, docker-layered-image-type): New variables. (system-docker-image)[layered-image?]: New argument. (stream-layered-image.py): New variable. (system-docker-layered-image): New procedure. (image->root-file-system)[docker-layered]: New image format. * gnu/tests/docker.scm (%test-docker-layered-system): New test. * gnu/image.scm (validate-image-format)[docker-layered]: New image format. * doc/guix.texi (Invoking guix pack): Document docker-layered format. (image Reference): Same. (image-type Reference): Document docker-layered-image-type. --- Makefile.am | 3 +- doc/guix.texi | 18 +- gnu/image.scm | 3 +- .../aux-files/python/stream-layered-image.py | 391 ++++++++++++++++++ gnu/system/image.scm | 84 +++- gnu/tests/docker.scm | 20 +- guix/docker.scm | 182 ++++++-- guix/scripts/pack.scm | 105 +++-- guix/scripts/system.scm | 11 +- tests/pack.scm | 48 +++ 10 files changed, 779 insertions(+), 86 deletions(-) create mode 100644 gnu/packages/aux-files/python/stream-layered-image.py diff --git a/Makefile.am b/Makefile.am index 23b939b674..9aca84f8f8 100644 --- a/Makefile.am +++ b/Makefile.am @@ -11,7 +11,7 @@ # Copyright © 2017 Arun Isaac # Copyright © 2018 Nikita # Copyright © 2018 Julien Lepiller -# Copyright © 2018 Oleg Pykhalov +# Copyright © 2018, 2023 Oleg Pykhalov # Copyright © 2018 Alex Vong # Copyright © 2019 Efraim Flashner # Copyright © 2021 Chris Marusich @@ -435,6 +435,7 @@ AUX_FILES = \ gnu/packages/aux-files/python/sanity-check.py \ gnu/packages/aux-files/python/sanity-check-next.py \ gnu/packages/aux-files/python/sitecustomize.py \ + gnu/packages/aux-files/python/stream-layered-image.py \ gnu/packages/aux-files/renpy/renpy.in \ gnu/packages/aux-files/run-in-namespace.c diff --git a/doc/guix.texi b/doc/guix.texi index 39932d5aad..fa4b7586c9 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -56,7 +56,7 @@ Copyright @copyright{} 2017 Andy Wingo@* Copyright @copyright{} 2017, 2018, 2019, 2020 Arun Isaac@* Copyright @copyright{} 2017 nee@* Copyright @copyright{} 2018 Rutger Helling@* -Copyright @copyright{} 2018, 2021 Oleg Pykhalov@* +Copyright @copyright{} 2018, 2021, 2023 Oleg Pykhalov@* Copyright @copyright{} 2018 Mike Gerwitz@* Copyright @copyright{} 2018 Pierre-Antoine Rouby@* Copyright @copyright{} 2018, 2019 Gábor Boskovits@* @@ -6837,9 +6837,15 @@ the following command: guix pack -f docker -S /bin=bin guile guile-readline @end example +or + +@example +guix pack -f docker-layered -S /bin=bin guile guile-readline +@end example + @noindent -The result is a tarball that can be passed to the @command{docker load} -command, followed by @code{docker run}: +The result is a tarball with image or layered image that can be passed +to the @command{docker load} command, followed by @code{docker run}: @example docker load < @var{file} @@ -43274,6 +43280,8 @@ one or multiple partitions. @item @code{docker}, a Docker image. +@item @code{docker-layered}, a layered Docker image. + @item @code{iso9660}, an ISO-9660 image. @item @code{tarball}, a tar.gz image archive. @@ -43605,6 +43613,10 @@ Build an image based on the @code{iso9660-image} image but with the Build an image based on the @code{docker-image} image. @end defvar +@defvar docker-layered-image-type +Build a layered image based on the @code{docker-layered-image} image. +@end defvar + @defvar raw-with-offset-image-type Build an MBR image with a single partition starting at a @code{1024KiB} offset. This is useful to leave some room to install a bootloader in diff --git a/gnu/image.scm b/gnu/image.scm index 523653dd77..8a6a0d8479 100644 --- a/gnu/image.scm +++ b/gnu/image.scm @@ -1,5 +1,6 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2020, 2022 Mathieu Othacehe +;;; Copyright © 2023 Oleg Pykhalov ;;; ;;; This file is part of GNU Guix. ;;; @@ -152,7 +153,7 @@ (define-with-syntax-properties (name (value properties)) ;; The supported image formats. (define-set-sanitizer validate-image-format format - (disk-image compressed-qcow2 docker iso9660 tarball wsl2)) + (disk-image compressed-qcow2 docker docker-layered iso9660 tarball wsl2)) ;; The supported partition table types. (define-set-sanitizer validate-partition-table-type partition-table-type diff --git a/gnu/packages/aux-files/python/stream-layered-image.py b/gnu/packages/aux-files/python/stream-layered-image.py new file mode 100644 index 0000000000..9ad2168c2d --- /dev/null +++ b/gnu/packages/aux-files/python/stream-layered-image.py @@ -0,0 +1,391 @@ +""" +This script generates a Docker image from a set of store paths. Uses +Docker Image Specification v1.2 as reference [1]. + +It expects a JSON file with the following properties and writes the +image as an uncompressed tarball to stdout: + +* "architecture", "config", "os", "created", "repo_tag" correspond to + the fields with the same name on the image spec [2]. +* "created" can be "now". +* "created" is also used as mtime for files added to the image. +* "store_layers" is a list of layers in ascending order, where each + layer is the list of store paths to include in that layer. + +The main challenge for this script to create the final image in a +streaming fashion, without dumping any intermediate data to disk +for performance. + +A docker image has each layer contents archived as separate tarballs, +and they later all get enveloped into a single big tarball in a +content addressed fashion. However, because how "tar" format works, +we have to know about the name (which includes the checksum in our +case) and the size of the tarball before we can start adding it to the +outer tarball. We achieve that by creating the layer tarballs twice; +on the first iteration we calculate the file size and the checksum, +and on the second one we actually stream the contents. 'add_layer_dir' +function does all this. + +[1]: https://github.com/moby/moby/blob/master/image/spec/v1.2.md +[2]: https://github.com/moby/moby/blob/4fb59c20a4fb54f944fe170d0ff1d00eb4a24d6f/image/spec/v1.2.md#image-json-field-descriptions +""" # noqa: E501 + + +import io +import os +import re +import sys +import json +import hashlib +import pathlib +import tarfile +import itertools +import threading +from datetime import datetime, timezone +from collections import namedtuple + + +def archive_paths_to(obj, paths, mtime): + """ + Writes the given store paths as a tar file to the given stream. + + obj: Stream to write to. Should have a 'write' method. + paths: List of store paths. + """ + + # gettarinfo makes the paths relative, this makes them + # absolute again + def append_root(ti): + ti.name = "/" + ti.name + return ti + + def apply_filters(ti): + ti.mtime = mtime + ti.uid = 0 + ti.gid = 0 + ti.uname = "root" + ti.gname = "root" + return ti + + def nix_root(ti): + ti.mode = 0o0555 # r-xr-xr-x + return ti + + def dir(path): + ti = tarfile.TarInfo(path) + ti.type = tarfile.DIRTYPE + return ti + + with tarfile.open(fileobj=obj, mode="w|") as tar: + # To be consistent with the docker utilities, we need to have + # these directories first when building layer tarballs. + tar.addfile(apply_filters(nix_root(dir("/gnu")))) + tar.addfile(apply_filters(nix_root(dir("/gnu/store")))) + + for path in paths: + path = pathlib.Path(path) + if path.is_symlink(): + files = [path] + else: + files = itertools.chain([path], path.rglob("*")) + + for filename in sorted(files): + ti = append_root(tar.gettarinfo(filename)) + + # copy hardlinks as regular files + if ti.islnk(): + ti.type = tarfile.REGTYPE + ti.linkname = "" + ti.size = filename.stat().st_size + + ti = apply_filters(ti) + if ti.isfile(): + with open(filename, "rb") as f: + tar.addfile(ti, f) + else: + tar.addfile(ti) + + +class ExtractChecksum: + """ + A writable stream which only calculates the final file size and + sha256sum, while discarding the actual contents. + """ + + def __init__(self): + self._digest = hashlib.sha256() + self._size = 0 + + def write(self, data): + self._digest.update(data) + self._size += len(data) + + def extract(self): + """ + Returns: Hex-encoded sha256sum and size as a tuple. + """ + return (self._digest.hexdigest(), self._size) + + +FromImage = namedtuple("FromImage", ["tar", "manifest_json", "image_json"]) +# Some metadata for a layer +LayerInfo = namedtuple("LayerInfo", ["size", "checksum", "path", "paths"]) + + +def load_from_image(from_image_str): + """ + Loads the given base image, if any. + + from_image_str: Path to the base image archive. + + Returns: A 'FromImage' object with references to the loaded base image, + or 'None' if no base image was provided. + """ + if from_image_str is None: + return None + + base_tar = tarfile.open(from_image_str) + + manifest_json_tarinfo = base_tar.getmember("manifest.json") + with base_tar.extractfile(manifest_json_tarinfo) as f: + manifest_json = json.load(f) + + image_json_tarinfo = base_tar.getmember(manifest_json[0]["Config"]) + with base_tar.extractfile(image_json_tarinfo) as f: + image_json = json.load(f) + + return FromImage(base_tar, manifest_json, image_json) + + +def add_base_layers(tar, from_image): + """ + Adds the layers from the given base image to the final image. + + tar: 'tarfile.TarFile' object for new layers to be added to. + from_image: 'FromImage' object with references to the loaded base image. + """ + if from_image is None: + print("No 'fromImage' provided", file=sys.stderr) + return [] + + layers = from_image.manifest_json[0]["Layers"] + checksums = from_image.image_json["rootfs"]["diff_ids"] + layers_checksums = zip(layers, checksums) + + for num, (layer, checksum) in enumerate(layers_checksums, start=1): + layer_tarinfo = from_image.tar.getmember(layer) + checksum = re.sub(r"^sha256:", "", checksum) + + tar.addfile(layer_tarinfo, from_image.tar.extractfile(layer_tarinfo)) + path = layer_tarinfo.path + size = layer_tarinfo.size + + print("Adding base layer", num, "from", path, file=sys.stderr) + yield LayerInfo(size=size, checksum=checksum, path=path, paths=[path]) + + from_image.tar.close() + + +def overlay_base_config(from_image, final_config): + """ + Overlays the final image 'config' JSON on top of selected defaults from the + base image 'config' JSON. + + from_image: 'FromImage' object with references to the loaded base image. + final_config: 'dict' object of the final image 'config' JSON. + """ + if from_image is None: + return final_config + + base_config = from_image.image_json["config"] + + # Preserve environment from base image + final_env = base_config.get("Env", []) + final_config.get("Env", []) + if final_env: + # Resolve duplicates (last one wins) and format back as list + resolved_env = {entry.split("=", 1)[0]: entry for entry in final_env} + final_config["Env"] = list(resolved_env.values()) + return final_config + + +def add_layer_dir(tar, paths, store_dir, mtime): + """ + Appends given store paths to a TarFile object as a new layer. + + tar: 'tarfile.TarFile' object for the new layer to be added to. + paths: List of store paths. + store_dir: the root directory of the nix store + mtime: 'mtime' of the added files and the layer tarball. + Should be an integer representing a POSIX time. + + Returns: A 'LayerInfo' object containing some metadata of + the layer added. + """ + + invalid_paths = [i for i in paths if not i.startswith(store_dir)] + assert len(invalid_paths) == 0, \ + f"Expecting absolute paths from {store_dir}, but got: {invalid_paths}" + + # First, calculate the tarball checksum and the size. + extract_checksum = ExtractChecksum() + archive_paths_to( + extract_checksum, + paths, + mtime=mtime, + ) + (checksum, size) = extract_checksum.extract() + + path = f"{checksum}/layer.tar" + layer_tarinfo = tarfile.TarInfo(path) + layer_tarinfo.size = size + layer_tarinfo.mtime = mtime + + # Then actually stream the contents to the outer tarball. + read_fd, write_fd = os.pipe() + with open(read_fd, "rb") as read, open(write_fd, "wb") as write: + def producer(): + archive_paths_to( + write, + paths, + mtime=mtime, + ) + write.close() + + # Closing the write end of the fifo also closes the read end, + # so we don't need to wait until this thread is finished. + # + # Any exception from the thread will get printed by the default + # exception handler, and the 'addfile' call will fail since it + # won't be able to read required amount of bytes. + threading.Thread(target=producer).start() + tar.addfile(layer_tarinfo, read) + + return LayerInfo(size=size, checksum=checksum, path=path, paths=paths) + + +def add_customisation_layer(target_tar, customisation_layer, mtime): + """ + Adds the customisation layer as a new layer. This is layer is structured + differently; given store path has the 'layer.tar' and corresponding + sha256sum ready. + + tar: 'tarfile.TarFile' object for the new layer to be added to. + customisation_layer: Path containing the layer archive. + mtime: 'mtime' of the added layer tarball. + """ + + checksum_path = os.path.join(customisation_layer, "checksum") + with open(checksum_path) as f: + checksum = f.read().strip() + assert len(checksum) == 64, f"Invalid sha256 at ${checksum_path}." + + layer_path = os.path.join(customisation_layer, "layer.tar") + + path = f"{checksum}/layer.tar" + tarinfo = target_tar.gettarinfo(layer_path) + tarinfo.name = path + tarinfo.mtime = mtime + + with open(layer_path, "rb") as f: + target_tar.addfile(tarinfo, f) + + return LayerInfo( + size=None, + checksum=checksum, + path=path, + paths=[customisation_layer] + ) + + +def add_bytes(tar, path, content, mtime): + """ + Adds a file to the tarball with given path and contents. + + tar: 'tarfile.TarFile' object. + path: Path of the file as a string. + content: Contents of the file. + mtime: 'mtime' of the file. Should be an integer representing a POSIX time. + """ + assert type(content) is bytes + + ti = tarfile.TarInfo(path) + ti.size = len(content) + ti.mtime = mtime + tar.addfile(ti, io.BytesIO(content)) + + +def main(): + with open(sys.argv[1], "r") as f: + conf = json.load(f) + + created = ( + datetime.now(tz=timezone.utc) + if conf["created"] == "now" + else datetime.fromisoformat(conf["created"]) + ) + mtime = int(created.timestamp()) + store_dir = conf["store_dir"] + + from_image = load_from_image(conf["from_image"]) + + with tarfile.open(mode="w|", fileobj=sys.stdout.buffer) as tar: + layers = [] + layers.extend(add_base_layers(tar, from_image)) + + start = len(layers) + 1 + for num, store_layer in enumerate(conf["store_layers"], start=start): + print("Creating layer", num, "from paths:", store_layer, + file=sys.stderr) + info = add_layer_dir(tar, store_layer, store_dir, mtime=mtime) + layers.append(info) + + print("Creating layer", len(layers) + 1, "with customisation...", + file=sys.stderr) + layers.append( + add_customisation_layer( + tar, + conf["customisation_layer"], + mtime=mtime + ) + ) + + print("Adding manifests...", file=sys.stderr) + + image_json = { + "created": datetime.isoformat(created), + "architecture": conf["architecture"], + "os": "linux", + "config": overlay_base_config(from_image, conf["config"]), + "rootfs": { + "diff_ids": [f"sha256:{layer.checksum}" for layer in layers], + "type": "layers", + }, + "history": [ + { + "created": datetime.isoformat(created), + "comment": f"store paths: {layer.paths}" + } + for layer in layers + ], + } + + image_json = json.dumps(image_json, indent=4).encode("utf-8") + image_json_checksum = hashlib.sha256(image_json).hexdigest() + image_json_path = f"{image_json_checksum}.json" + add_bytes(tar, image_json_path, image_json, mtime=mtime) + + manifest_json = [ + { + "Config": image_json_path, + "RepoTags": [conf["repo_tag"]], + "Layers": [layer.path for layer in layers], + } + ] + manifest_json = json.dumps(manifest_json, indent=4).encode("utf-8") + add_bytes(tar, "manifest.json", manifest_json, mtime=mtime) + + print("Done.", file=sys.stderr) + + +if __name__ == "__main__": + main() diff --git a/gnu/system/image.scm b/gnu/system/image.scm index afef79185f..0bfd011ad4 100644 --- a/gnu/system/image.scm +++ b/gnu/system/image.scm @@ -4,6 +4,7 @@ ;;; Copyright © 2022 Pavel Shlyak ;;; Copyright © 2022 Denis 'GNUtoo' Carikli ;;; Copyright © 2022 Alex Griffin +;;; Copyright © 2023 Oleg Pykhalov ;;; ;;; This file is part of GNU Guix. ;;; @@ -45,6 +46,7 @@ (define-module (gnu system image) #:use-module (gnu system uuid) #:use-module (gnu system vm) #:use-module (guix packages) + #:use-module ((gnu packages) #:select (search-auxiliary-file)) #:use-module (gnu packages base) #:use-module (gnu packages bash) #:use-module (gnu packages bootloaders) @@ -58,6 +60,7 @@ (define-module (gnu system image) #:use-module (gnu packages hurd) #:use-module (gnu packages linux) #:use-module (gnu packages mtools) + #:use-module (gnu packages python) #:use-module (gnu packages virtualization) #:use-module ((srfi srfi-1) #:prefix srfi-1:) #:use-module (srfi srfi-11) @@ -78,6 +81,7 @@ (define-module (gnu system image) efi-disk-image iso9660-image docker-image + docker-layered-image tarball-image wsl2-image raw-with-offset-disk-image @@ -89,6 +93,7 @@ (define-module (gnu system image) iso-image-type uncompressed-iso-image-type docker-image-type + docker-layered-image-type tarball-image-type wsl2-image-type raw-with-offset-image-type @@ -167,6 +172,10 @@ (define docker-image (image-without-os (format 'docker))) +(define docker-layered-image + (image-without-os + (format 'docker-layered))) + (define tarball-image (image-without-os (format 'tarball))) @@ -237,6 +246,11 @@ (define docker-image-type (name 'docker) (constructor (cut image-with-os docker-image <>)))) +(define docker-layered-image-type + (image-type + (name 'docker-layered) + (constructor (cut image-with-os docker-layered-image <>)))) + (define tarball-image-type (image-type (name 'tarball) @@ -633,9 +647,12 @@ (define (image-with-label base-image label) (define* (system-docker-image image #:key - (name "docker-image")) + (name "docker-image") + (archiver tar) + layered-image?) "Build a docker image for IMAGE. NAME is the base name to use for the -output file." +output file. If LAYERED-IMAGE? is true, the image will with many of the store +paths being on their own layer to improve sharing between images." (define boot-program ;; Program that runs the boot script of OS, which in turn starts shepherd. (program-file "boot-program" @@ -678,9 +695,11 @@ (define builder (use-modules (guix docker) (guix build utils) (gnu build image) + (srfi srfi-1) (srfi srfi-19) (guix build store-copy) - (guix store database)) + (guix store database) + (ice-9 receive)) ;; Set the SQL schema location. (sql-schema #$schema) @@ -700,18 +719,34 @@ (define builder #:register-closures? #$register-closures? #:deduplicate? #f #:system-directory #$os) - (build-docker-image - #$output - (cons* image-root - (map store-info-item - (call-with-input-file #$graph - read-reference-graph))) - #$os - #:entry-point '(#$boot-program #$os) - #:compressor '(#+(file-append gzip "/bin/gzip") "-9n") - #:creation-time (make-time time-utc 0 1) - #:system #$image-target - #:transformations `((,image-root -> "")))))))) + (when #$layered-image? + (setenv "PATH" + (string-join (list #+(file-append archiver "/bin") + #+(file-append coreutils "/bin") + #+(file-append gzip "/bin") + #+(file-append python "/bin")) + ":"))) + (apply build-docker-image + (append (list #$output + (append (if #$layered-image? + '() + (list image-root)) + (map store-info-item + (call-with-input-file #$graph + read-reference-graph))) + #$os + #:entry-point '(#$boot-program #$os) + #:compressor + '(#+(file-append gzip "/bin/gzip") "-9n") + #:creation-time (make-time time-utc 0 1) + #:system #$image-target + #:transformations `((,image-root -> ""))) + (if #$layered-image? + (list #:root-system + image-root + #:stream-layered-image + #$stream-layered-image.py) + '())))))))) (computed-file name builder ;; Allow offloading so that this I/O-intensive process @@ -720,6 +755,21 @@ (define builder #:options `(#:references-graphs ((,graph ,os)) #:substitutable? ,substitutable?)))) +(define stream-layered-image.py + (local-file (search-auxiliary-file "python/stream-layered-image.py"))) + +(define* (system-docker-layered-image image + #:key + (name "docker-image") + (archiver tar) + (layered-image? #t)) + "Build a docker image for IMAGE. NAME is the base name to use for the +output file." + (system-docker-image image + #:name name + #:archiver archiver + #:layered-image? layered-image?)) + ;;; ;;; Tarball image. @@ -811,7 +861,7 @@ (define (image->root-file-system image) "Return the IMAGE root partition file-system type." (case (image-format image) ((iso9660) "iso9660") - ((docker tarball wsl2) "dummy") + ((docker docker-layered tarball wsl2) "dummy") (else (partition-file-system (find-root-partition image))))) @@ -948,6 +998,8 @@ (define target (cond ("bootcfg" ,bootcfg)))) ((memq image-format '(docker)) (system-docker-image image*)) + ((memq image-format '(docker-layered)) + (system-docker-layered-image image*)) ((memq image-format '(tarball)) (system-tarball-image image*)) ((memq image-format '(wsl2)) diff --git a/gnu/tests/docker.scm b/gnu/tests/docker.scm index 0276e398a7..85c5f178b5 100644 --- a/gnu/tests/docker.scm +++ b/gnu/tests/docker.scm @@ -1,6 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2019 Danny Milosavljevic ;;; Copyright © 2019-2022 Ludovic Courtès +;;; Copyright © 2023 Oleg Pykhalov ;;; ;;; This file is part of GNU Guix. ;;; @@ -43,7 +44,8 @@ (define-module (gnu tests docker) #:use-module (guix build-system trivial) #:use-module ((guix licenses) #:prefix license:) #:export (%test-docker - %test-docker-system)) + %test-docker-system + %test-docker-layered-system)) (define %docker-os (simple-operating-system @@ -309,3 +311,19 @@ (define %test-docker-system (locale-libcs (list glibc))) #:type docker-image-type))) run-docker-system-test))))) + +(define %test-docker-layered-system + (system-test + (name "docker-layered-system") + (description "Run a system image as produced by @command{guix system +docker-layered-image} inside Docker.") + (value (with-monad %store-monad + (>>= (lower-object + (system-image (os->image + (operating-system + (inherit (simple-operating-system)) + ;; Use locales for a single libc to + ;; reduce space requirements. + (locale-libcs (list glibc))) + #:type docker-layered-image-type))) + run-docker-system-test))))) diff --git a/guix/docker.scm b/guix/docker.scm index 5e6460f43f..f1adad26dc 100644 --- a/guix/docker.scm +++ b/guix/docker.scm @@ -3,6 +3,7 @@ ;;; Copyright © 2017, 2018, 2019, 2021 Ludovic Courtès ;;; Copyright © 2018 Chris Marusich ;;; Copyright © 2021 Maxim Cournoyer +;;; Copyright © 2023 Oleg Pykhalov ;;; ;;; This file is part of GNU Guix. ;;; @@ -24,10 +25,14 @@ (define-module (guix docker) #:use-module (guix base16) #:use-module (guix build pack) #:use-module ((guix build utils) - #:select (mkdir-p + #:select (%store-directory + mkdir-p delete-file-recursively + dump-port with-directory-excursion invoke)) + #:use-module (guix diagnostics) + #:use-module (guix i18n) #:use-module (gnu build install) #:use-module (json) ;guile-json #:use-module (srfi srfi-1) @@ -38,6 +43,9 @@ (define-module (guix docker) #:use-module (rnrs bytevectors) #:use-module (ice-9 ftw) #:use-module (ice-9 match) + #:use-module (ice-9 popen) + #:use-module (ice-9 rdelim) + #:use-module (ice-9 receive) #:export (build-docker-image)) ;; Generate a 256-bit identifier in hexadecimal encoding for the Docker image. @@ -136,6 +144,9 @@ (define directive-file (('directory name _ ...) (string-trim name #\/)))) +(define %docker-image-max-layers + 100) + (define* (build-docker-image image paths prefix #:key (repository "guix") @@ -146,11 +157,13 @@ (define* (build-docker-image image paths prefix entry-point (environment '()) compressor - (creation-time (current-time time-utc))) - "Write to IMAGE a Docker image archive containing the given PATHS. PREFIX -must be a store path that is a prefix of any store paths in PATHS. REPOSITORY -is a descriptive name that will show up in \"REPOSITORY\" column of the output -of \"docker images\". + (creation-time (current-time time-utc)) + stream-layered-image + root-system) + "Write to IMAGE a layerer Docker image archive containing the given PATHS. +PREFIX must be a store path that is a prefix of any store paths in PATHS. +REPOSITORY is a descriptive name that will show up in \"REPOSITORY\" column of +the output of \"docker images\". When DATABASE is true, copy it to /var/guix/db in the image and create /var/guix/gcroots and friends. @@ -172,7 +185,13 @@ (define* (build-docker-image image paths prefix SYSTEM is a GNU triplet (or prefix thereof) of the system the binaries in PATHS are for; it is used to produce metadata in the image. Use COMPRESSOR, a command such as '(\"gzip\" \"-9n\"), to compress IMAGE. Use CREATION-TIME, a -SRFI-19 time-utc object, as the creation time in metadata." +SRFI-19 time-utc object, as the creation time in metadata. + +STREAM-LAYERED-IMAGE is a Python script which accepts a JSON configuration +file and prints archive to STDOUT. + +ROOT-SYSTEM is a directory with a provisioned root file system, which will be +added to image as a layer." (define (sanitize path-fragment) (escape-special-chars ;; GNU tar strips the leading slash off of absolute paths before applying @@ -183,6 +202,39 @@ (define (sanitize path-fragment) ;; We also need to escape "/" because we use it as a delimiter. "/*.^$[]\\" #\\)) + (define (file-sha256 file-name) + "Calculate the hexdigest of the sha256 checksum of FILE-NAME and return it." + (let ((port (open-pipe* OPEN_READ + "sha256sum" + "--" + file-name))) + (let ((result (read-delimited " " port))) + (close-pipe port) + result))) + (define (paths-split-sort paths) + "Split list of PATHS at %DOCKER-IMAGE-MAX-LAYERS and sort by disk usage." + (let* ((paths-length (length paths)) + (port (apply open-pipe* OPEN_READ + (append '("du" "--summarize") paths))) + (output (read-string port))) + (close-port port) + (receive (head tail) + (split-at + (map (match-lambda ((size . path) path)) + (sort (map (lambda (line) + (match (string-split line #\tab) + ((size path) + (cons (string->number size) path)))) + (string-split + (string-trim-right output #\newline) + #\newline)) + (lambda (path1 path2) + (< (match path2 ((size . _) size)) + (match path1 ((size . _) size)))))) + (if (>= paths-length %docker-image-max-layers) + (- %docker-image-max-layers 2) + (1- paths-length))) + (list head tail)))) (define transformation->replacement (match-lambda ((old '-> new) @@ -205,7 +257,9 @@ (define transformation-options `("--transform" ,(transformations->expression transformations)))) (let* ((directory "/tmp/docker-image") ;temporary working directory (id (docker-id prefix)) - (time (date->string (time-utc->date creation-time) "~4")) + (time ;Workaround for Python datetime.fromisoformat does not parse Z. + (string-append (date->string (time-utc->date creation-time) "~5") + "+00:00")) (arch (let-syntax ((cond* (syntax-rules () ((_ (pattern clause) ...) (cond ((string-prefix? pattern system) @@ -218,7 +272,8 @@ (define transformation-options ("i686" "386") ("arm" "arm") ("aarch64" "arm64") - ("mips64" "mips64le"))))) + ("mips64" "mips64le")))) + (paths (if stream-layered-image (paths-split-sort paths) paths))) ;; Make sure we start with a fresh, empty working directory. (mkdir directory) (with-directory-excursion directory @@ -229,26 +284,38 @@ (define transformation-options (with-output-to-file "json" (lambda () (scm->json (image-description id time)))) - ;; Create a directory for the non-store files that need to go into the - ;; archive. - (mkdir "extra") + (if root-system + (let ((directory (getcwd))) + (with-directory-excursion root-system + (apply invoke "tar" + "-cf" (string-append directory "/layer.tar") + `(,@transformation-options + ,@(tar-base-options) + ,@(scandir "." + (lambda (file) + (not (member file '("." ".."))))))))) + (begin + ;; Create a directory for the non-store files that need to go + ;; into the archive. + (mkdir "extra") - (with-directory-excursion "extra" - ;; Create non-store files. - (for-each (cut evaluate-populate-directive <> "./") - extra-files) + (with-directory-excursion "extra" + ;; Create non-store files. + (for-each (cut evaluate-populate-directive <> "./") + extra-files) - (when database - ;; Initialize /var/guix, assuming PREFIX points to a profile. - (install-database-and-gc-roots "." database prefix)) + (when database + ;; Initialize /var/guix, assuming PREFIX points to a profile. + (install-database-and-gc-roots "." database prefix)) - (apply invoke "tar" "-cf" "../layer.tar" - `(,@transformation-options - ,@(tar-base-options) - ,@paths - ,@(scandir "." - (lambda (file) - (not (member file '("." "..")))))))) + (apply invoke "tar" "-cf" "../layer.tar" + `(,@transformation-options + ,@(tar-base-options) + ,@(if stream-layered-image '() paths) + ,@(scandir "." + (lambda (file) + (not (member file '("." "..")))))))) + (delete-file-recursively "extra"))) ;; It is possible for "/" to show up in the archive, especially when ;; applying transformations. For example, the transformation @@ -263,22 +330,65 @@ (define transformation-options (lambda () (system* "tar" "--delete" "/" "-f" "layer.tar"))) - (delete-file-recursively "extra")) + (when stream-layered-image + (call-with-output-file "checksum" + (lambda (port) + (display (file-sha256 "layer.tar") port))))) (with-output-to-file "config.json" (lambda () - (scm->json (config (string-append id "/layer.tar") - time arch - #:environment environment - #:entry-point entry-point)))) + (scm->json + (if stream-layered-image + `(("created" . ,time) + ("repo_tag" . "guix:latest") + ("customisation_layer" . ,id) + ("store_layers" . ,(match paths + (((head ...) (tail ...)) + (list->vector + (reverse + (cons (list->vector tail) + (fold (lambda (path paths) + (cons (vector path) paths)) + '() + head))))))) + ("store_dir" . ,(%store-directory)) + ("from_image" . #nil) + ("os" . "linux") + ("config" + (env . ,(list->vector (map (match-lambda + ((name . value) + (string-append name "=" value))) + environment))) + ,@(if entry-point + `((entrypoint . ,(list->vector entry-point))) + '())) + ("architecture" . ,arch)) + (config (string-append id "/layer.tar") + time arch + #:environment environment + #:entry-point entry-point))))) (with-output-to-file "manifest.json" (lambda () (scm->json (manifest prefix id repository)))) (with-output-to-file "repositories" (lambda () - (scm->json (repositories prefix id repository))))) - - (apply invoke "tar" "-cf" image "-C" directory - `(,@(tar-base-options #:compressor compressor) - ".")) + (scm->json (repositories prefix id repository)))) + (if stream-layered-image + (let ((input (open-pipe* OPEN_READ "python3" + stream-layered-image + "config.json"))) + (call-with-output-file "image.tar" + (lambda (output) + (dump-port input output))) + (if (eqv? 0 (status:exit-val (close-pipe input))) + (begin + (invoke "gzip" "image.tar") + (copy-file "image.tar.gz" image)) + (error + (formatted-message + (G_ "failed to create ~a image tarball") + image)))) + (apply invoke "tar" "-cf" image + `(,@(tar-base-options #:compressor compressor) + ".")))) (delete-file-recursively directory))) diff --git a/guix/scripts/pack.scm b/guix/scripts/pack.scm index 25ac9d29d0..3a8f87e850 100644 --- a/guix/scripts/pack.scm +++ b/guix/scripts/pack.scm @@ -8,6 +8,7 @@ ;;; Copyright © 2020, 2021, 2022, 2023 Maxim Cournoyer ;;; Copyright © 2020 Eric Bavier ;;; Copyright © 2022 Alex Griffin +;;; Copyright © 2023 Oleg Pykhalov ;;; ;;; This file is part of GNU Guix. ;;; @@ -28,6 +29,7 @@ (define-module (guix scripts pack) #:use-module (guix scripts) #:use-module (guix ui) #:use-module (guix gexp) + #:use-module ((guix build utils) #:select (%xz-parallel-args)) #:use-module (guix utils) #:use-module (guix store) #:use-module ((guix status) #:select (with-status-verbosity)) @@ -53,6 +55,8 @@ (define-module (guix scripts pack) #:use-module ((gnu packages compression) #:hide (zip)) #:use-module (gnu packages guile) #:use-module (gnu packages base) + #:use-module (gnu packages python) + #:autoload (gnu packages package-management) (guix) #:autoload (gnu packages gnupg) (guile-gcrypt) #:autoload (gnu packages guile) (guile2.0-json guile-json) #:use-module (srfi srfi-1) @@ -67,6 +71,7 @@ (define-module (guix scripts pack) debian-archive rpm-archive docker-image + docker-layered-image squashfs-image %formats @@ -589,6 +594,10 @@ (define (mksquashfs args) ;;; ;;; Docker image format. ;;; + +(define stream-layered-image.py + (local-file (search-auxiliary-file "python/stream-layered-image.py"))) + (define* (docker-image name profile #:key target (profile-name "guix-profile") @@ -597,12 +606,14 @@ (define* (docker-image name profile localstatedir? (symlinks '()) (archiver tar) - (extra-options '())) + (extra-options '()) + layered-image?) "Return a derivation to construct a Docker image of PROFILE. The image is a tarball conforming to the Docker Image Specification, compressed with COMPRESSOR. It can be passed to 'docker load'. If TARGET is true, it must a be a GNU triplet and it is used to derive the architecture metadata in -the image." +the image. If LAYERED-IMAGE? is true, the image will with many of the +store paths being on their own layer to improve sharing between images." (define database (and localstatedir? (file-append (store-database (list profile)) @@ -653,25 +664,37 @@ (define directives `((directory "/tmp" ,(getuid) ,(getgid) #o1777) ,@(append-map symlink->directives '#$symlinks))) - (setenv "PATH" #+(file-append archiver "/bin")) - - (build-docker-image #$output - (map store-info-item - (call-with-input-file "profile" - read-reference-graph)) - #$profile - #:repository (manifest->friendly-name - (profile-manifest #$profile)) - #:database #+database - #:system (or #$target %host-type) - #:environment environment - #:entry-point - #$(and entry-point - #~(list (string-append #$profile "/" - #$entry-point))) - #:extra-files directives - #:compressor #+(compressor-command compressor) - #:creation-time (make-time time-utc 0 1)))))) + (setenv "PATH" + (string-join `(#+(file-append archiver "/bin") + #+@(if layered-image? + (list (file-append coreutils "/bin") + (file-append gzip "/bin") + (file-append python "/bin")) + '())) + ":")) + + (apply build-docker-image + (append (list #$output + (map store-info-item + (call-with-input-file "profile" + read-reference-graph)) + #$profile + #:repository (manifest->friendly-name + (profile-manifest #$profile)) + #:database #+database + #:system (or #$target %host-type) + #:environment environment + #:entry-point + #$(and entry-point + #~(list (string-append #$profile "/" + #$entry-point))) + #:extra-files directives + #:compressor #+(compressor-command compressor) + #:creation-time (make-time time-utc 0 1)) + (if #$layered-image? + (list #:stream-layered-image + #$stream-layered-image.py) + '()))))))) (gexp->derivation (string-append name ".tar" (compressor-extension compressor)) @@ -679,6 +702,33 @@ (define directives #:target target #:references-graphs `(("profile" ,profile)))) +(define* (docker-layered-image name profile + #:key target + (profile-name "guix-profile") + (compressor (first %compressors)) + entry-point + localstatedir? + (symlinks '()) + (archiver tar) + (extra-options '()) + (layered-image? #t)) + "Return a derivation to construct a Docker image of PROFILE. The image is a +tarball conforming to the Docker Image Specification, compressed with +COMPRESSOR. It can be passed to 'docker load'. If TARGET is true, it must a +be a GNU triplet and it is used to derive the architecture metadata in the +image. If LAYERED-IMAGE? is true, the image will with many of the store paths +being on their own layer to improve sharing between images." + (docker-image name profile + #:target target + #:profile-name profile-name + #:compressor compressor + #:entry-point entry-point + #:localstatedir? localstatedir? + #:symlinks symlinks + #:archiver archiver + #:extra-options extra-options + #:layered-image? layered-image?)) + ;;; ;;; Debian archive format. @@ -1355,6 +1405,7 @@ (define %formats `((tarball . ,self-contained-tarball) (squashfs . ,squashfs-image) (docker . ,docker-image) + (docker-layered . ,docker-layered-image) (deb . ,debian-archive) (rpm . ,rpm-archive))) @@ -1363,15 +1414,17 @@ (define (show-formats) (display (G_ "The supported formats for 'guix pack' are:")) (newline) (display (G_ " - tarball Self-contained tarball, ready to run on another machine")) + tarball Self-contained tarball, ready to run on another machine")) + (display (G_ " + squashfs Squashfs image suitable for Singularity")) (display (G_ " - squashfs Squashfs image suitable for Singularity")) + docker Tarball ready for 'docker load'")) (display (G_ " - docker Tarball ready for 'docker load'")) + docker-layered Tarball with a layered image ready for 'docker load'")) (display (G_ " - deb Debian archive installable via dpkg/apt")) + deb Debian archive installable via dpkg/apt")) (display (G_ " - rpm RPM archive installable via rpm/yum")) + rpm RPM archive installable via rpm/yum")) (newline)) (define (required-option symbol) diff --git a/guix/scripts/system.scm b/guix/scripts/system.scm index d7163dd3eb..e4bf0347c7 100644 --- a/guix/scripts/system.scm +++ b/guix/scripts/system.scm @@ -11,6 +11,7 @@ ;;; Copyright © 2021 Brice Waegeneire ;;; Copyright © 2021 Simon Tournier ;;; Copyright © 2022 Tobias Geerinckx-Rice +;;; Copyright © 2023 Oleg Pykhalov ;;; ;;; This file is part of GNU Guix. ;;; @@ -727,13 +728,15 @@ (define* (system-derivation-for-action image action #:graphic? graphic? #:disk-image-size image-size #:mappings mappings)) - ((image disk-image vm-image docker-image) + ((image disk-image vm-image docker-image docker-layered-image) (when (eq? action 'disk-image) (warning (G_ "'disk-image' is deprecated: use 'image' instead~%"))) (when (eq? action 'vm-image) (warning (G_ "'vm-image' is deprecated: use 'image' instead~%"))) (when (eq? action 'docker-image) (warning (G_ "'docker-image' is deprecated: use 'image' instead~%"))) + (when (eq? action 'docker-layered-image) + (warning (G_ "'docker-layered-image' is deprecated: use 'image' instead~%"))) (lower-object (system-image image)))))) (define (maybe-suggest-running-guix-pull) @@ -980,6 +983,8 @@ (define (show-help) image build a Guix System image\n")) (display (G_ "\ docker-image build a Docker image\n")) + (display (G_ "\ + docker-layered-image build a Docker layered image\n")) (display (G_ "\ init initialize a root file system to run GNU\n")) (display (G_ "\ @@ -1193,7 +1198,7 @@ (define actions '("build" "container" "vm" "vm-image" "image" "disk-image" "list-generations" "describe" "delete-generations" "roll-back" "switch-generation" "search" "edit" - "docker-image")) + "docker-image" "docker-layered-image")) (define (process-action action args opts) "Process ACTION, a sub-command, with the arguments are listed in ARGS. @@ -1242,6 +1247,8 @@ (define save-provenance? (image (let* ((image-type (case action ((vm-image) qcow2-image-type) ((docker-image) docker-image-type) + ((docker-layered-image) + docker-layered-image-type) (else image-type))) (image-size (assoc-ref opts 'image-size)) (volatile? diff --git a/tests/pack.scm b/tests/pack.scm index 87187bb62c..db2208d91c 100644 --- a/tests/pack.scm +++ b/tests/pack.scm @@ -2,6 +2,7 @@ ;;; Copyright © 2017, 2018, 2019, 2020, 2021 Ludovic Courtès ;;; Copyright © 2018 Ricardo Wurmus ;;; Copyright © 2021, 2023 Maxim Cournoyer +;;; Copyright © 2023 Oleg Pykhalov ;;; ;;; This file is part of GNU Guix. ;;; @@ -29,6 +30,7 @@ (define-module (test-pack) #:use-module (guix gexp) #:use-module (guix modules) #:use-module (guix utils) + #:use-module ((guix build utils) #:select (%store-directory)) #:use-module (gnu packages) #:use-module ((gnu packages base) #:select (glibc-utf8-locales)) #:use-module (gnu packages bootstrap) @@ -246,6 +248,52 @@ (define bin (mkdir #$output))))))) (built-derivations (list check)))) + (unless store (test-skip 1)) + (test-assertm "docker-layered-image + localstatedir" store + (mlet* %store-monad + ((guile (set-guile-for-build (default-guile))) + (profile -> (profile + (content (packages->manifest (list %bootstrap-guile))) + (hooks '()) + (locales? #f))) + (tarball (docker-layered-image "docker-pack" profile + #:symlinks '(("/bin/Guile" -> "bin/guile")) + #:localstatedir? #t)) + (check (gexp->derivation "check-tarball" + (with-imported-modules '((guix build utils)) + #~(begin + (use-modules (guix build utils) + (ice-9 match)) + + (define bin + (string-append "." #$profile "/bin")) + + (define store + (string-append "." #$(%store-directory))) + + (setenv "PATH" (string-append #$%tar-bootstrap "/bin")) + (mkdir "base") + (with-directory-excursion "base" + (invoke "tar" "xvf" #$tarball)) + + (match (find-files "base" "layer.tar") + ((layers ...) + (for-each (lambda (layer) + (invoke "tar" "xvf" layer) + (invoke "chmod" "--recursive" "u+w" store)) + layers))) + + (when + (and (file-exists? (string-append bin "/guile")) + (file-exists? "var/guix/db/db.sqlite") + (file-is-directory? "tmp") + (string=? (string-append #$%bootstrap-guile "/bin") + (pk 'binlink (readlink bin))) + (string=? (string-append #$profile "/bin/guile") + (pk 'guilelink (readlink "bin/Guile")))) + (mkdir #$output))))))) + (built-derivations (list check)))) + (unless store (test-skip 1)) (test-assertm "squashfs-image + localstatedir" store (mlet* %store-monad -- 2.38.0 From debbugs-submit-bounces@debbugs.gnu.org Mon Mar 13 20:26:43 2023 Received: (at 62153) by debbugs.gnu.org; 14 Mar 2023 00:26:43 +0000 Received: from localhost ([127.0.0.1]:35836 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pbsV1-00053Z-7c for submit@debbugs.gnu.org; Mon, 13 Mar 2023 20:26:43 -0400 Received: from mail-lf1-f49.google.com ([209.85.167.49]:38686) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pbsUy-00053H-Ol for 62153@debbugs.gnu.org; Mon, 13 Mar 2023 20:26:42 -0400 Received: by mail-lf1-f49.google.com with SMTP id x17so1774198lfu.5 for <62153@debbugs.gnu.org>; Mon, 13 Mar 2023 17:26:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1678753595; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=G8S8uTM/fM3Q2BYYsNhFLfE+5yUcrfSAF3eAXi2VVPg=; b=nv5QscYNafK+93a4MOxkoUjhA2bVYs6TXA0oLWYtPSxLV5bhwRdQboZ3IH/fVzxuSa P/eVGmyzCARhBaO9Z1PJjGL2gWpvqWSfwBSX+GpUKA/s8SJeWyhXH6EjwM9LIoGnlX/k Mo7GsIIK6B9+NAqZHVgIInQ8SDEaHwl31lLmGB39z/LFvBd+W29aOl6gCveEO4WHTYNJ gSBLwUZN/XypbTGsprJ+1HhxcikWgY6NHAadnOmmcW2XAM7/s1oeCRIj7hREi9WP9qS5 p5b3ZTo5V+VNusrd6hkGm1LecZa6PvmY55yNeNk217tY1cZt2M1+Jvl9ltp5CDCK+0AK a+pw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678753595; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=G8S8uTM/fM3Q2BYYsNhFLfE+5yUcrfSAF3eAXi2VVPg=; b=OrqT7FxMQZ+G30ZcF4ScG+KHM5Fd4EEBcozOxr9ZY9zBLrGjrsVLigtPulfGXZynr9 Gg7LZD5J2a9qPuAaxibHzeKVPsExUY3U5+GMxjiAVrvTMdoJkpwJxYCVZlhG1bOTaMLL oEgG2jEDCqny6JhIjMl/tyrILVml1tKxAu3wfyxNSVfKRdg+GJpguY0KVfAO1dYRdrtZ q5SFGmxd01WFGGgaYOgC8XnE40PYV+S3sWotDG373xfZ46l7+1KoSx2ALdQBIFZsNNSt oJl9hobNSKLa8S0LeZH72zaLSthuFLj1ZK6ZyAK11jBnonNcEvBbezKXA/Qnw6/lYMfM 5DuQ== X-Gm-Message-State: AO0yUKXRNAbyHVg84OFiLEY+rSNEg4uhnTajOQ4DkIdrQv6rtD+xp8si qz9MKOyIV94UfiCEzoK+1gk= X-Google-Smtp-Source: AK7set99lR8FycKWOB7vWEjAyU6Y8+2rIYMzRjlQLyj4m3DeyOO7NeOmKN6lUtkhtr7Xbv+rT67iew== X-Received: by 2002:ac2:43d9:0:b0:4db:3890:cb59 with SMTP id u25-20020ac243d9000000b004db3890cb59mr3425224lfl.1.1678753594883; Mon, 13 Mar 2023 17:26:34 -0700 (PDT) Received: from guixsd.wugi.info ([88.201.161.72]) by smtp.gmail.com with ESMTPSA id e7-20020ac25467000000b0048afb8b8e53sm157297lfn.80.2023.03.13.17.26.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 13 Mar 2023 17:26:34 -0700 (PDT) From: Oleg Pykhalov To: pelzflorian@pelzflorian.de Subject: [PATCH 2/2] news: Add entry for the new 'docker-layered' distribution format. Date: Tue, 14 Mar 2023 03:24:53 +0300 Message-Id: <20230314002453.24668-3-go.wigust@gmail.com> X-Mailer: git-send-email 2.38.0 In-Reply-To: <20230314002453.24668-1-go.wigust@gmail.com> References: <87cz5ccqa5.fsf@pelzflorian.de> <20230314002453.24668-1-go.wigust@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 62153 Cc: Oleg Pykhalov , 62153@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) * etc/news.scm: Add entry. --- etc/news.scm | 58 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 58 insertions(+) diff --git a/etc/news.scm b/etc/news.scm index 55d1218df5..4bbdfd2a59 100644 --- a/etc/news.scm +++ b/etc/news.scm @@ -18,6 +18,7 @@ ;; Copyright © 2021 Andrew Tropin ;; Copyright © 2021 Jonathan Brielmaier ;; Copyright © 2022 Thiago Jung Bauermann +;; Copyright © 2023 Oleg Pykhalov ;; ;; Copying and distribution of this file, with or without modification, are ;; permitted in any medium without royalty provided the copyright notice and @@ -26,6 +27,63 @@ (channel-news (version 0) + (entry (commit "a5c3baf510adab1f5b3bb855b1aa9cafe3cb66b9") + (title + (de "Neues Format @samp{docker-layered} für den Befehl @command{guix pack}") + (en "New @samp{docker-layered} format for the @command{guix pack} command") + (ru "Новый @samp{docker-layered} формат для @command{guix pack} команды")) + (body + (de "Sie können jetzt auch mehrschichtige Docker-Abbilder mit dem Befehl +@command{guix pack --format=docker-layered} erzeugen. Damit bekommen Sie ein +Docker-Abbild, bei dem Store-Pfade auf getrennten Schichten („Layer“) +untergebracht sind, die sich mehrere Abbilder teilen können. Das Abbild wird +im Store als gzip-komprimierter Tarball erzeugt. Hier ist ein einfaches +Beispiel, wo ein mehrschichtiges Docker-Abbild für das Paket @code{hello} +angelegt wird: + +@example +guix pack --format=docker-layered --symlink=/usr/bin/hello=bin/hello hello +@end example + +@command{guix system image} kann jetzt geschichtete Docker-Abbilder erzeugen, +indem Sie @code{docker-layered} an die Befehlszeilenoption @option{--image-type} +übergeben. + +Siehe @command{info \"(guix.de) Aufruf von guix pack\"} und +@command{info \"(guix.de) Systemabbilder\"} für weitere Informationen.") + (en "Docker layered images can now be produced via the @command{guix +pack --format=docker-layered} command, providing a Docker image with many of +the store paths being on their own layer to improve sharing between images. +The image is realized into the GNU store as a gzipped tarball. Here is a +simple example that generates a layered Docker image for the @code{hello} +package: + +@example +guix pack --format=docker-layered --symlink=/usr/bin/hello=bin/hello hello +@end example + +The @command{guix system image} can now produce layered Docker image by passing +@code{docker-layered} to @option{--image-type} option. + +See @command{info \"(guix) Invoking guix pack\"} and +@command{info \"(guix) System Images\"} for more information.") + (ru "Появилась команда создания многослойных Docker образов с помощью +@command{guix pack --format=docker-layered}, которая соберет Docker образ с +путями в store расположенными на отдельных слоях, ускоряя таким образом +передачу образов. Образ будет создан в GNU store в качестве gzipped tarball. + +Пример создания Docker layered образ с @code{hello} пакетом: +@example +guix pack --format=docker-layered --symlink=/usr/bin/hello=bin/hello hello +@end example + +@command{guix system image} теперь может создавать layered Docker образ путем +указания в опции @option{--image-type} параметра @code{docker-layered}. + +Смотрите @command{info \"(guix) Invoking guix pack\"} и +@command{info \"(guix) System Images\"} для получения более детальных +сведений."))) + (entry (commit "0e18c5e5bcb9204c278cfc75493d3b02b746d5c3") (title (en "Linux-libre kernel updated to 6.2") -- 2.38.0 From debbugs-submit-bounces@debbugs.gnu.org Mon Mar 13 20:40:35 2023 Received: (at 62153) by debbugs.gnu.org; 14 Mar 2023 00:40:35 +0000 Received: from localhost ([127.0.0.1]:35855 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pbsiR-0005gg-2w for submit@debbugs.gnu.org; Mon, 13 Mar 2023 20:40:35 -0400 Received: from mail-lf1-f44.google.com ([209.85.167.44]:37734) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pbsiO-0005gG-Kd for 62153@debbugs.gnu.org; Mon, 13 Mar 2023 20:40:33 -0400 Received: by mail-lf1-f44.google.com with SMTP id g17so17952230lfv.4 for <62153@debbugs.gnu.org>; Mon, 13 Mar 2023 17:40:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1678754426; h=mime-version:message-id:date:subject:to:from:from:to:cc:subject :date:message-id:reply-to; bh=xuiWXw0Iim1/LD2BefFvz6FbAw3LlGt18d2zplYiOkc=; b=UKSA0KUPrejcM+M3vacvdarp2GH5tVyMtgbEXBVaSLRbKmmVyrCW6NS6Jy8Wb0DD+k T1yawHZ6C8Zrq7FLWHM18dJDYJ4s2cvvJvPtdugvEvVthngkyslGbSoyqNT7gNeomgog Ti8pyxMqOqpscpbK/pz9Xwdmt6W3bfCpMAzn0JlYnGIRiQZoZiLtHdOErfcaXFTZGQRI AoL7xxCd6quH/DzPZuKMCt4Hm+/F6jDCAzz9EHzsObq/9ITi8chvYdgoqyCEbcthccv3 J4W8s/awa/ZpHY+PR/08ewgoA7rnxmnHBtcxIB0iUKSyfKc883dbHN6qJav03TsRAb+K 4TSw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678754426; h=mime-version:message-id:date:subject:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=xuiWXw0Iim1/LD2BefFvz6FbAw3LlGt18d2zplYiOkc=; b=j3iBNHNQ7MbQVVzoCAtY2zqPsDJ7pprl6rb2pt6+G3ET2AxjOOweWi6QyD/95EoDaF r4+2qO0cACWfmrT1Pr8Y8i4E9otdG1+kde8G7QIXHPetA+wTp3cfvqBAddrIz2Q48aCW CXW8IyIMsvOuV5vK9K/RSy1CwWqO8TXIjCODIqtPK5i+TW9KyVw9VyDL2xtny59bUaie duVVCnbcesA4DtdY0TFq6epmps47yuQB1S0taOlyxc2duJPLz4IWD6g9gJHnOoX0Rx6Z jnwlY5bcK3RvmrCs7nmm4NAemTHuWwG4Wor4QcG/55p4f16jft8MBJGZaMKZfnkNVgWO NrtA== X-Gm-Message-State: AO0yUKWXW0gLc4Q/0ToB0KFq6FqfLwl9XvE0yEVQGBh29iEIfgSfHCMT +77Mzm1LrsVuw87jyGhpSrEShl3MZ/Y= X-Google-Smtp-Source: AK7set+CVdIETYMt9l8lcgg07mMSlGj7pd4NqUFH+SVEMpZyMlj6fC4zNjc4L9JQot7bLLuhpSoRkA== X-Received: by 2002:a19:f51a:0:b0:4d2:c70a:fe0a with SMTP id j26-20020a19f51a000000b004d2c70afe0amr3451014lfb.2.1678754425985; Mon, 13 Mar 2023 17:40:25 -0700 (PDT) Received: from localhost ([88.201.161.72]) by smtp.gmail.com with ESMTPSA id w21-20020ac254b5000000b004dc7fae3cfcsm161126lfk.75.2023.03.13.17.40.25 for <62153@debbugs.gnu.org> (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 13 Mar 2023 17:40:25 -0700 (PDT) From: Oleg Pykhalov To: 62153@debbugs.gnu.org Subject: Missing diff in cover lever for v2 patch Date: Tue, 14 Mar 2023 03:40:24 +0300 Message-ID: <87wn3k17yf.fsf@gmail.com> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 62153 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Missing diff in cover leter for v2 patch attached below. > This patch series applies your suggestions. Also it's rebased on > origin/master and added a missing documentation for =E2=80=98docker-layer= ed=E2=80=99 format in > =E2=80=98guix system image=E2=80=99 command in doc/guix.texi file (follow= ing diff). =2D-8<---------------cut here---------------start------------->8--- diff --git a/doc/guix.texi b/doc/guix.texi index bd0ee126ee..6938743154 100644 =2D-- a/doc/guix.texi +++ b/doc/guix.texi @@ -43306,6 +43306,8 @@ one or multiple partitions. =20 @item @code{docker}, a Docker image. =20 +@item @code{docker-layered}, a layered Docker image. + @item @code{iso9660}, an ISO-9660 image. =20 @item @code{tarball}, a tar.gz image archive. =2D-8<---------------cut here---------------end--------------->8--- --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQJIBAEBCgAyFiEEcjhxI46s62NFSFhXFn+OpQAa+pwFAmQPwngUHGdvLndpZ3Vz dEBnbWFpbC5jb20ACgkQFn+OpQAa+pw5aw/+JeyQ3p6e4cfIX6+RHnDjUgP9HA2Q ESTfjzWxsJcCxleL0LKO3AbOUVby/qq4TrUgNSm1cqL/S4vWDAIo34XEyT5Xerdg oHSNtoaHqD467NsGhSvtdS8TYGe7JLTTr76nkNzwjQAGjDYXzJXvrcr758H4VMw0 6k143Q1uqDYj3+E2uqQWT9wme5oZ4IkxzZW/+XCME3s00yBF19Uti9VqROLAEIhV OnQAuurDYScfZ7in17DMuyHxOfrJm4M2G7UxRSiVnKbBeGjbaQoL4+QSRCRa+/+O 0VABMVnYFYUqQPFKm+Y7csQiFbjE6mojc3nNwqBEd0i3rUwu4vPjad6mirKUC9FQ C3gpBDAIVNJ6I0UWmmBdRH/p/hNRvtiAw0FAhBaV0arTsWoLMeybmZjoxDF533zo wg7ZPbaZtJd8nHowzKsdbXYqYEf1bhio1goBcnraGJdgBFscSsT5eJhk/Tgp9n3a yQB9NyF4E/Uc7WLzSiscwaxU2ksg+W1s9xZVrz5xcbu1yX8+yI9u+dSp6e6fa7qf VLyYOGaD7Qju5alCPc6p4z12/a+DD5ZAEJeBReYoJCfEEPjfOClrwHtnJksQyPY/ PlCL+blYcC4yrMQgQN7UeWbDWTEGzwJPHIWgOMzR/meYOQO0TZ1yFj865E9VmyfF TjXACct5ZaD5Y64= =otdV -----END PGP SIGNATURE----- --=-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Tue Mar 14 04:46:27 2023 Received: (at 62153) by debbugs.gnu.org; 14 Mar 2023 08:46:27 +0000 Received: from localhost ([127.0.0.1]:36212 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pc0Id-0002AK-0d for submit@debbugs.gnu.org; Tue, 14 Mar 2023 04:46:27 -0400 Received: from mail-wm1-f49.google.com ([209.85.128.49]:51993) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pc0Ia-0002A1-PG for 62153@debbugs.gnu.org; Tue, 14 Mar 2023 04:46:25 -0400 Received: by mail-wm1-f49.google.com with SMTP id ay8so4407867wmb.1 for <62153@debbugs.gnu.org>; Tue, 14 Mar 2023 01:46:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1678783578; h=mime-version:message-id:date:references:in-reply-to:subject:cc:to :from:from:to:cc:subject:date:message-id:reply-to; bh=S486eBDbmvoSKhNJhX2qss9v2rMcF0NLI7lr4vI/t+k=; b=Hlc7mhiCWn7cms0N0/EP+OZXQPc18hMyZySRrZgWxx1C6Wou0qxa/okhGnnfFjJx1s 38OFZM6FfWJzuiSgOCO3liiL5bEvjSUwJ912MiqgnN+8kclwDba2jf4mGeYW8kP5dsxT UPy6kNhnvaoVeYPevOsTgmz2BDudEznneGgXcWgj6L3j9amf3s9VL3vyKXbAwYwy8ilI dm60dBx+BrMKA/qfVvMXa59OW+gDMUedZLDXgUuLWJq3ZdxfwWLeF9/zmiW+CDjOyv4S BqZnjWqFKa3a/HRF7+4nqKmHKtlYVZXWnxIreQYs9tqeMOiQKARKjDupxwJRvkInWmCP W5Ng== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678783578; h=mime-version:message-id:date:references:in-reply-to:subject:cc:to :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=S486eBDbmvoSKhNJhX2qss9v2rMcF0NLI7lr4vI/t+k=; b=AjZSlm+zYnNBQCyaCRCEoSOwCgtGJvJYolKW5lNm0uULtrj2lBw1/8VjWzy1YDYufy /ywgGylEob/6yg2SBdkJEh5m+M4HANSWE24NP8aGVsjk7QENapJM2j7gD6qJd7LdAnZV RhBwyrHb12sMNZ4kxMPtlNEtlxO2l/EuNEOWM/cfeiGbANlTKYs/6ciYIgaze9TQfoiC YeRZ0TGa/xqG0e7hYR4vsYKRJBkQ4WrBTuL6J9/ZCHcAQQRKuqRcoHigAUS0BT6CMULu BnndwbYNvyr8TZhzljroG0oP0Uuv6MTxOzkJIJGv3fA6KHEZyHPqj+3Nt02atgqJU0yQ +dmg== X-Gm-Message-State: AO0yUKX/0iCALcefmb5EX+mj9wlhV9BTvT8uAUA716RleO4+GbbSsg6R u9V6CDqZWThwGd+SN2p7CSw= X-Google-Smtp-Source: AK7set8RYZlUWNysqYMHyruhhv+fvdFYr2kWtzPk/OjLAIl0mTKgR15MO2RoauO2ucCOlFycUMOBTw== X-Received: by 2002:a05:600c:3b15:b0:3eb:3986:9c03 with SMTP id m21-20020a05600c3b1500b003eb39869c03mr12067971wms.4.1678783578514; Tue, 14 Mar 2023 01:46:18 -0700 (PDT) Received: from lili ([2a01:e0a:59b:9120:65d2:2476:f637:db1e]) by smtp.gmail.com with ESMTPSA id f24-20020a1cc918000000b003e203681b26sm2140678wmb.29.2023.03.14.01.46.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Mar 2023 01:46:18 -0700 (PDT) From: Simon Tournier To: Oleg Pykhalov Subject: Re: [bug#62153] [PATCH 1/2] guix: docker: Build layered image. In-Reply-To: <87bkkw2w7z.fsf@gmail.com> References: <20230313003012.14325-1-go.wigust@gmail.com> <20230313003310.17129-1-go.wigust@gmail.com> <87r0tsk85r.fsf@gmail.com> <87bkkw2w7z.fsf@gmail.com> Date: Tue, 14 Mar 2023 09:19:51 +0100 Message-ID: <86cz5b923c.fsf@gmail.com> MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 62153 Cc: Josselin Poiret , Tobias Geerinckx-Rice , Simon Tournier , Mathieu Othacehe , Ludovic =?utf-8?Q?Court=C3=A8s?= , Christopher Baines , Ricardo Wurmus , 62153@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Hi Oleg, CC: core teams On Tue, 14 Mar 2023 at 00:10, Oleg Pykhalov wrote: >>> diff --git a/gnu/packages/aux-files/python/stream-layered-image.py b/gnu/packages/aux-files/python/stream-layered-image.py >>> new file mode 100644 >>> index 0000000000..9ad2168c2d >>> --- /dev/null >>> +++ b/gnu/packages/aux-files/python/stream-layered-image.py >>> @@ -0,0 +1,391 @@ >>> +""" >>> +This script generates a Docker image from a set of store paths. Uses >>> +Docker Image Specification v1.2 as reference [1]. >> >> Instead of Python, would it possible to implement in Guile? I mean, >> does Python have something that is missing in Guile? >> >> The facility for manipulating Tar? Something else? > > I think nothing else. As I understand Python implemented Tar inside the > language itself in 2500 lines of code by manipulating binary data. > > /gnu/store/...-python-3.9.9/lib/python3.9/tarfile.py > > Technically it's probably possible to use tar utility with --append flag > instead of opening a new file and streaming to it as the Python script > does. To be honest I would like not to write it in this way if the > Python script does not block current patch for merge. Ok, thanks for explaining. > Also I don't see myself writing Tar implementation in Guile, yet. ;-) Maybe not reimplementing Tar in Guile, maybe just enough for working. Or maybe some Guile bindings. Or maybe something is already around for the bootstrap story. The use of external tools as Python for producing built-in Guix feature will be the first time, no? For what it is worth, I would prefer to consider the options before emitting an opinion about dragging Python building/packing layered Docker. :-) > The Nix project uses this script heavily to build layered images, so it > should be robust in terms of up to date to current Tar and Python > implementations. Do you mean this script is coming from Nix. Well, in all cases, this script is not trivial and so it requires Copyright for authorship. [...] > The following example shows common layers between images, which will be > not tranfered if you load image inside Docker as well as pull and push: Thanks for explaining. Cheers, simon PS: I will be off-line these 2-3 next weeks. So a lack of an answer from me will not be a lack of interest. ;-) From debbugs-submit-bounces@debbugs.gnu.org Tue Mar 14 05:12:14 2023 Received: (at submit) by debbugs.gnu.org; 14 Mar 2023 09:12:14 +0000 Received: from localhost ([127.0.0.1]:36249 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pc0ha-0002qn-HO for submit@debbugs.gnu.org; Tue, 14 Mar 2023 05:12:14 -0400 Received: from lists.gnu.org ([209.51.188.17]:57648) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pc0hW-0002qV-GS for submit@debbugs.gnu.org; Tue, 14 Mar 2023 05:12:10 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pc0hW-00016c-73 for guix-patches@gnu.org; Tue, 14 Mar 2023 05:12:10 -0400 Received: from mira.cbaines.net ([212.71.252.8]) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pc0hU-0000XH-Li for guix-patches@gnu.org; Tue, 14 Mar 2023 05:12:09 -0400 Received: from localhost (unknown [IPv6:2a02:8010:68c1:0:54d1:d5d4:280e:f699]) by mira.cbaines.net (Postfix) with ESMTPSA id 23B5816E4D; Tue, 14 Mar 2023 09:12:07 +0000 (GMT) Received: from felis (localhost [127.0.0.1]) by localhost (OpenSMTPD) with ESMTP id 4af7bfcd; Tue, 14 Mar 2023 09:12:05 +0000 (UTC) References: <20230313003012.14325-1-go.wigust@gmail.com> <20230313003310.17129-1-go.wigust@gmail.com> <87r0tsk85r.fsf@gmail.com> User-agent: mu4e 1.8.13; emacs 28.2 From: Christopher Baines To: Simon Tournier Subject: Re: [bug#62153] [PATCH 1/2] guix: docker: Build layered image. Date: Tue, 14 Mar 2023 09:11:36 +0000 In-reply-to: <87r0tsk85r.fsf@gmail.com> Message-ID: <8735671yu4.fsf@cbaines.net> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" Received-SPF: pass client-ip=212.71.252.8; envelope-from=mail@cbaines.net; helo=mira.cbaines.net X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-Spam-Score: -1.4 (-) X-Debbugs-Envelope-To: submit Cc: Oleg Pykhalov , guix-patches@gnu.org, 62153@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.4 (--) --=-=-= Content-Type: text/plain Simon Tournier writes: > And I remember something in that direction by Chris but I am unable to > find back the patch. )-: This is the thread https://lists.gnu.org/archive/html/guix-devel/2020-03/msg00299.html --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmQQOmNfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh aW5lcy5uZXQACgkQXiijOwuE9XeUkRAApwCv6Zh8MrAuDE5TRebOf0IJflIq98wc iB1i643fUvGZzVsr/pjAXsrkKDqyBdPH9Oee/JHvPPtEcCVGJwwAtodiGC6Tnf1t 8eYzWSXdzYVK7oFReQZ/oHLzmA0zQYVTmcnIoqP4wy+Cgv/zE3fGWlLrpiIau2hV NgPUo4wPdZu8ZwM0B2+gK2H4I5pNXUxhIuWFI8rORU6q0nxL/OqrDMh5G/+B1UPh cuGHBOxoBx3JERUnaHkk0BkgS1TySYpaVDA/9RwtCeqV8cv0bSjznkOqdtusr0vK prM5WJwzOpdlHkAqwkmEEuE93KjdAoMl5gmYWWlUobCdAtwLNLTXCGa+wAlx7VmW kFl0XygIJCeXHICtDhiBkv/cfIVSpKYVbqJExRAhzeH6Ykllww6UUaEHC32Fy0ae WsWUUpgpRGZSgbT70i05D8TaPBjNgIBUV5HfczWTsRgvR1b4fVaqrKaxCooy1gV1 YjIJO7ddNHGCSNp0uxGyMzZMmpEy46gQvSiUNTW46AYgl2c5TEVMaQ/gLNvgmAvt K+PRQhOaHyoZNLLCIZaRhmuPwbwKEh3MLOGcR/AUF7nW5sp2wk9LBm889lDmarpz bHk5ZfhqiZryeo36k9O3L3DE70h+fibm1uKr7zn6jRt74LmvGdWA4lCWaz4A9USE VjZkN4ayEaQ= =9k/e -----END PGP SIGNATURE----- --=-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Tue Mar 14 05:17:26 2023 Received: (at 62153) by debbugs.gnu.org; 14 Mar 2023 09:17:27 +0000 Received: from localhost ([127.0.0.1]:36259 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pc0mc-0002zO-ID for submit@debbugs.gnu.org; Tue, 14 Mar 2023 05:17:26 -0400 Received: from sender4-of-o50.zoho.com ([136.143.188.50]:21037) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pc0ma-0002zG-U4 for 62153@debbugs.gnu.org; Tue, 14 Mar 2023 05:17:25 -0400 ARC-Seal: i=1; a=rsa-sha256; t=1678785436; cv=none; d=zohomail.com; s=zohoarc; b=jjreRIZwJl2TrrlaA0gKpkScZuFP162S8prxCAcyToKMrTg1r6zxIXaVqj/yr0Gm/ShqrsBIvJeBYDfW6EsKi6pnxQrmPbBehjvDknGcgP+G07StaPkbgl12HMDkAKJcs+P0/1nUlhgInLuhkWfQ0+VCh7py8gR8ykXMHRXg6ao= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1678785436; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:MIME-Version:Message-ID:References:Subject:To; bh=OrKts9VUoJ0S3cB6l/6v4yQOPb82NROFl7WOB+bRVSU=; b=CB/Ut4DsgdU5QFbJj+MYG27aU0bzfPw63JH5a7fI7bEbjDD4xNobQ0FnSFujUCr+1gWGeL4013AfaSlr9eNsryV1L++ffCUfRvetdLG3vc1nqz0ceARC12+AZo+lMS7Z6+bAisGLRDUxJElUv4H3BZQm4UAa0ii66aJU2ECDddI= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass header.i=elephly.net; spf=pass smtp.mailfrom=rekado@elephly.net; dmarc=pass header.from= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1678785436; s=zoho; d=elephly.net; i=rekado@elephly.net; h=References:From:From:To:To:Cc:Cc:Subject:Subject:Date:Date:In-reply-to:Message-ID:MIME-Version:Content-Type:Content-Transfer-Encoding:Message-Id:Reply-To; bh=OrKts9VUoJ0S3cB6l/6v4yQOPb82NROFl7WOB+bRVSU=; b=Cnm6bAKy9jtQNFKlt70iTaoOwgeFrf4ykc0MvDl1kclXe6K2XhilFoy+a4+xMgNn 8cmUfLrbnXzhdWqoR4yJ5AEfJ/fwThXOTZdUO2b93s7E95fYIJck2kPD984v8H7UaCw Chz+A0LRRt3KUnk3H/CgQ9xbTBVpDzpApyoPpqSg= Received: from localhost (175-109-142-46.pool.kielnet.net [46.142.109.175]) by mx.zohomail.com with SMTPS id 1678785433975291.00451690492855; Tue, 14 Mar 2023 02:17:13 -0700 (PDT) References: <20230313003012.14325-1-go.wigust@gmail.com> <20230313003310.17129-1-go.wigust@gmail.com> <87r0tsk85r.fsf@gmail.com> <87bkkw2w7z.fsf@gmail.com> <86cz5b923c.fsf@gmail.com> User-agent: mu4e 1.8.13; emacs 28.2 From: Ricardo Wurmus To: Simon Tournier Subject: Re: [bug#62153] [PATCH 1/2] guix: docker: Build layered image. Date: Tue, 14 Mar 2023 10:15:49 +0100 In-reply-to: <86cz5b923c.fsf@gmail.com> X-URL: https://elephly.net X-PGP-Key: https://elephly.net/rekado.pubkey X-PGP-Fingerprint: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC Message-ID: <874jqnzo8a.fsf@elephly.net> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-ZohoMailClient: External X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 62153 Cc: Josselin Poiret , Tobias Geerinckx-Rice , Oleg Pykhalov , Ludovic =?utf-8?Q?Court=C3=A8s?= , Christopher Baines , Mathieu Othacehe , 62153@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Simon Tournier writes: >>> Instead of Python, would it possible to implement in Guile? I mean, >>> does Python have something that is missing in Guile? >>> >>> The facility for manipulating Tar? Something else? >> >> I think nothing else. As I understand Python implemented Tar inside the >> language itself in 2500 lines of code by manipulating binary data. >> >> /gnu/store/...-python-3.9.9/lib/python3.9/tarfile.py >> >> Technically it's probably possible to use tar utility with --append flag >> instead of opening a new file and streaming to it as the Python script >> does. To be honest I would like not to write it in this way if the >> Python script does not block current patch for merge. > > Ok, thanks for explaining. > >> Also I don't see myself writing Tar implementation in Guile, yet. ;-) > > Maybe not reimplementing Tar in Guile, maybe just enough for working. > Or maybe some Guile bindings. Or maybe something is already around for > the bootstrap story. gash-utils has (gash ustar); it=E2=80=99s about 620 lines of code. --=20 Ricardo From debbugs-submit-bounces@debbugs.gnu.org Thu Mar 16 06:38:12 2023 Received: (at 62153) by debbugs.gnu.org; 16 Mar 2023 10:38:12 +0000 Received: from localhost ([127.0.0.1]:41272 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pckzr-0006zS-Ub for submit@debbugs.gnu.org; Thu, 16 Mar 2023 06:38:12 -0400 Received: from eggs.gnu.org ([209.51.188.92]:52042) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pckzp-0006zF-IF for 62153@debbugs.gnu.org; Thu, 16 Mar 2023 06:38:10 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pckzj-0003y1-64; Thu, 16 Mar 2023 06:38:03 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:In-Reply-To:Date:References:Subject:To: From; bh=lDWz9QBW4/evXQhdWCCt4LEX9vNYZy4rDnUceWThBqs=; b=TSIjYALGSkI+Yy2ZsTgf JFhnV9CJSGgEBKlAi9G9D/7vX8JHfKq8uN4c85mRqbCdOch4JSm8SiUtYmuZdGPm7GWzysb3EYwX2 Kv3rbLpIWJfhLF3IyNMOC40igIcDI1f9RJcIsprGhAwwaVgq9OdljjJ9ja2R3S4ZGF6c8HBqHHObA 2veDaaZwtFm2D1wDqMVDtePq74GZwf2yw7G2F+aYhsOukfgh1Fdwt+oeEM/LCjqZfKSh5MPSbJfyF hwYwxP7fX2pLsNKXXyb2agGi0d4tA/yrNK7fmNRm42q5hS79x0EynDLCrfQVSGwFXCzLdNpAtnQgB Kf9n0nguJGXctA==; Received: from 91-160-117-201.subs.proxad.net ([91.160.117.201] helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pckzh-0007SY-EL; Thu, 16 Mar 2023 06:38:02 -0400 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Ricardo Wurmus Subject: Re: [bug#62153] [PATCH 1/2] guix: docker: Build layered image. References: <20230313003012.14325-1-go.wigust@gmail.com> <20230313003310.17129-1-go.wigust@gmail.com> <87r0tsk85r.fsf@gmail.com> <87bkkw2w7z.fsf@gmail.com> <86cz5b923c.fsf@gmail.com> <874jqnzo8a.fsf@elephly.net> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: Sextidi 26 =?utf-8?Q?Vent=C3=B4se?= an 231 de la =?utf-8?Q?R=C3=A9volution=2C?= jour du Pissenlit X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Thu, 16 Mar 2023 11:37:58 +0100 In-Reply-To: <874jqnzo8a.fsf@elephly.net> (Ricardo Wurmus's message of "Tue, 14 Mar 2023 10:15:49 +0100") Message-ID: <877cvhhth5.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 62153 Cc: Josselin Poiret , Tobias Geerinckx-Rice , Simon Tournier , Oleg Pykhalov , Christopher Baines , Mathieu Othacehe , 62153@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hi, Ricardo Wurmus skribis: > Simon Tournier writes: > >>>> Instead of Python, would it possible to implement in Guile? I mean, >>>> does Python have something that is missing in Guile? >>>> >>>> The facility for manipulating Tar? Something else? >>> >>> I think nothing else. As I understand Python implemented Tar inside the >>> language itself in 2500 lines of code by manipulating binary data. >>> >>> /gnu/store/...-python-3.9.9/lib/python3.9/tarfile.py >>> >>> Technically it's probably possible to use tar utility with --append flag >>> instead of opening a new file and streaming to it as the Python script >>> does. To be honest I would like not to write it in this way if the >>> Python script does not block current patch for merge. >> >> Ok, thanks for explaining. >> >>> Also I don't see myself writing Tar implementation in Guile, yet. ;-) >> >> Maybe not reimplementing Tar in Guile, maybe just enough for working. >> Or maybe some Guile bindings. Or maybe something is already around for >> the bootstrap story. > > gash-utils has (gash ustar); it=E2=80=99s about 620 lines of code. Disarchive also has a tar implementation. No excuse! :-) Oleg, could you check which of these would satisfy your needs? I had a plan to improve the tar implementation in Gash-Utils, perhaps it=E2=80=99s a good time to get my act together. Thanks, Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Mon Mar 20 02:38:34 2023 Received: (at 62153) by debbugs.gnu.org; 20 Mar 2023 06:38:34 +0000 Received: from localhost ([127.0.0.1]:53505 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pe9A9-0005Ar-5e for submit@debbugs.gnu.org; Mon, 20 Mar 2023 02:38:33 -0400 Received: from mail-lj1-f177.google.com ([209.85.208.177]:39538) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pe9A6-0005Ae-Vw for 62153@debbugs.gnu.org; Mon, 20 Mar 2023 02:38:32 -0400 Received: by mail-lj1-f177.google.com with SMTP id a11so432064lji.6 for <62153@debbugs.gnu.org>; Sun, 19 Mar 2023 23:38:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1679294304; h=mime-version:user-agent:message-id:date:references:subject:cc:to :from:from:to:cc:subject:date:message-id:reply-to; bh=00BvvjYqQZgIyB1wweqh5Iv4mg8Qc3VbJVG5vJlNXio=; b=NXi0jvwhtUiJc6Sy9wE/XjtAkpd19urDhL95ubTEpPA2uEvTKWUxG6NNUJZNlztQs1 qpER4dxbhTfx+JkbtFrlLRUc5IXspqrw7epQlx2pNnJtKgoQXBm2KdT7aVWM82+dg2Es OHLnpsrPtaOoaAGUHFm/NDj2L+fewU3ajy3umMjREN+8Wz6TrNObLrD+NMPn3hg18abY eUWTwyC5IU49BKdAVNbPrEL8u6n+Ei2VVb7ML3rBejHMXxCwT+VDSSFNIqk6oZxXBkSD VZ7SKqAFOQdwCINNBMYNvbO6I3q+qsMAiycBWBZSo+Pq5egG927LWwFQSPAXPePKThJG 6BRA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679294305; h=mime-version:user-agent:message-id:date:references:subject:cc:to :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=00BvvjYqQZgIyB1wweqh5Iv4mg8Qc3VbJVG5vJlNXio=; b=O+28eDguxaWbm02GQMAyatZC0ojgYOB7E2vOYK+0+uPM+BB1Xx4MhY8KwQGFGar7ZL pK/3NWXWn+KFZI4C/DYDRt7uLkVHR7JLXB3C61hKq9zMywnTFXuxw1vtvnEefWFBUUnN F8gSMD7UDxdzM/N3kVQUBZMxsPLLP0z8IINLSzE0RYwmtAfpxJMEs+GWt7Lp8NyJOO4E SC5wFgSpvR/o/qrE+KxtehHFAxITn5AA3To7Ril3YOnfbn2x9k8QC6vRoTbLiUeBnRvL jwH+LhUVEZpTvyaKv0hpLG8H7nFDeJ4bp3SCqSApS2VXOs8WfKH9UG5aJ1n+t22N+HsO c9Og== X-Gm-Message-State: AO0yUKVzzUtJOU57CbImp2k73DMcE6MD4RzxeqqRH5/emlmt/Tl61hkH MxPBMsyMXYPXfEQ4QbhGvcw= X-Google-Smtp-Source: AK7set+rXJsHP2/IhDio6XgCuH8+/fuOrMlChnoo/MjiTZB/sk7SwpQeWQCkB9Rq3cDmtfDYR7J6hQ== X-Received: by 2002:a2e:b8d5:0:b0:29d:6dd5:d712 with SMTP id s21-20020a2eb8d5000000b0029d6dd5d712mr578495ljp.2.1679294304389; Sun, 19 Mar 2023 23:38:24 -0700 (PDT) Received: from localhost ([88.201.161.72]) by smtp.gmail.com with ESMTPSA id x26-20020a2e9dda000000b0029bd4365c95sm907971ljj.87.2023.03.19.23.38.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 19 Mar 2023 23:38:23 -0700 (PDT) From: Oleg Pykhalov To: Ludovic =?utf-8?Q?Court=C3=A8s?= Subject: Re: [bug#62153] [PATCH 1/2] guix: docker: Build layered image. References: <20230313003012.14325-1-go.wigust@gmail.com> <20230313003310.17129-1-go.wigust@gmail.com> <87r0tsk85r.fsf@gmail.com> <87bkkw2w7z.fsf@gmail.com> <86cz5b923c.fsf@gmail.com> <874jqnzo8a.fsf@elephly.net> <877cvhhth5.fsf@gnu.org> Date: Mon, 20 Mar 2023 09:38:22 +0300 Message-ID: <87h6ugapwh.fsf@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (gnu/linux) MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 62153 Cc: Josselin Poiret , Tobias Geerinckx-Rice , Simon Tournier , Mathieu Othacehe , Christopher Baines , Ricardo Wurmus , 62153@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hi Ludovic, Ludovic Court=C3=A8s writes: > Ricardo Wurmus skribis: > >> Simon Tournier writes: >> >>>>> Instead of Python, would it possible to implement in Guile? I mean, >>>>> does Python have something that is missing in Guile? >>>>> >>>>> The facility for manipulating Tar? Something else? >>>> >>>> I think nothing else. As I understand Python implemented Tar inside t= he >>>> language itself in 2500 lines of code by manipulating binary data. >>>> >>>> /gnu/store/...-python-3.9.9/lib/python3.9/tarfile.py >>>> >>>> Technically it's probably possible to use tar utility with --append fl= ag >>>> instead of opening a new file and streaming to it as the Python script >>>> does. To be honest I would like not to write it in this way if the >>>> Python script does not block current patch for merge. >>> >>> Ok, thanks for explaining. >>> >>>> Also I don't see myself writing Tar implementation in Guile, yet. ;-) >>> >>> Maybe not reimplementing Tar in Guile, maybe just enough for working. >>> Or maybe some Guile bindings. Or maybe something is already around for >>> the bootstrap story. >> >> gash-utils has (gash ustar); it=E2=80=99s about 620 lines of code. > > Disarchive also has a tar implementation. No excuse! :-) > > Oleg, could you check which of these would satisfy your needs? I had a > plan to improve the tar implementation in Gash-Utils, perhaps it=E2=80=99= s a > good time to get my act together. Gash-Utils should work, e.g. [1] script. It's already possible to rewrite with Gash-Utils, but at least write-ustar-file and write-ustar-footer should be exported in ustar.scm. Disarchive requires to write a file specification in case of using disarchive-assemble. And disarchive-assemble does not work, if I don't miss anything [2]. Also gzip compression does not work in a Guile REPL. [1]: =2D-8<---------------cut here---------------start------------->8--- #!/usr/bin/env -S guile --no-auto-compile -e main -s !# (set! %load-path (append '("/gnu/store/...-gash-utils-0.1.0/share/guile/site/3.0" "/gnu/store/...-gash-0.2.0/share/guile/site/3.0") %load-path)) (set! %load-compiled-path (append '("/gnu/store/...-gash-utils-0.1.0/lib/guile/3.0/site-ccache" "/gnu/store/...-gash-0.2.0/lib/guile/3.0/site-ccache") %load-compiled-path)) (use-modules (gash ustar) (srfi srfi-26) (guix build utils)) (define write-ustar-file (@@ (gash ustar) write-ustar-file)) (define write-ustar-footer (@@ (gash ustar) write-ustar-footer)) (define (main . args) (call-with-port (open-file "out.tar.gz" "wb") (lambda (port) (with-directory-excursion "." (call-with-compressed-output-port 'gzip port (cut write-ustar-file <> "Makefile.am" #:verbosity 0))) (with-directory-excursion "doc" (call-with-compressed-output-port 'gzip port (cut write-ustar-file <> "." #:verbosity 0))) (write-ustar-footer port)))) =2D-8<---------------cut here---------------end--------------->8--- [2]: =2D-8<---------------cut here---------------start------------->8--- $ guile ,m(disarchive assemblers tarball) (assemble-tarball (disassemble-tarball "out.tar") "result/out.tar") # Generated tarball: $ tar tf result/out.tar/sha256/1e7100029373723df900712d1191be0bad5beadf752f= 367bb264fab891c36356 ./ ./images/ ./images/bootstrap-graph.dot ./images/service-graph.pdf ./guix-cookbook.zh_Hans.texi tar: Unexpected EOF in archive tar: Error is not recoverable: exiting now # Original tarball: $ tar tf out.tar ./ ./images/ ./images/bootstrap-graph.dot ./images/bootstrap-graph.eps ./images/bootstrap-graph.pdf ./images/bootstrap-graph.png ./images/bootstrap-packages.dot ./images/bootstrap-packages.eps ./images/bootstrap-packages.pdf ./images/bootstrap-packages.png ./images/coreutils-bag-graph.dot ./images/coreutils-bag-graph.eps ./images/coreutils-bag-graph.pdf ./images/coreutils-bag-graph.png ./images/coreutils-graph.dot ./images/coreutils-graph.eps ./images/coreutils-graph.pdf ./images/coreutils-graph.png ./images/coreutils-size-map.eps ./images/coreutils-size-map.png ./images/gcc-core-mesboot0-graph.dot ./images/gcc-core-mesboot0-graph.eps ./images/gcc-core-mesboot0-graph.pdf ./images/gcc-core-mesboot0-graph.png ./images/installer-network.png ./images/installer-partitions.png ./images/installer-resume.png ./images/service-graph.dot ./images/service-graph.eps ./images/service-graph.pdf ./images/service-graph.png ./images/shepherd-graph.dot ./images/shepherd-graph.eps ./images/shepherd-graph.pdf ./images/shepherd-graph.png ./.dirstamp ./environment-gdb.scm ./fdl-1.3.texi ./os-config-bare-bones.texi ./os-config-desktop.texi ./os-config-lightweight-desktop.texi ./package-hello.json ./package-hello.scm ./stamp-1 ./stamp-10 ./stamp-11 ./stamp-2 ./stamp-3 ./stamp-4 ./stamp-5 ./stamp-6 ./stamp-7 ./stamp-8 ./stamp-9 ./stamp-vti ./contributing.fa.texi ./guix.fa.texi ./contributing.fi.texi ./guix.fi.texi ./contributing.it.texi ./guix.it.texi ./contributing.ko.texi ./guix.ko.texi ./contributing.sk.texi ./guix.sk.texi ./guix-cookbook.es.texi ./guix-cookbook.fa.texi ./guix-cookbook.fi.texi ./guix-cookbook.pt_BR.texi ./guix-cookbook.ru.texi ./guix-cookbook.zh_Hans.texi ./version-fa.texi ./guix.fa.info-1 ./guix.fa.info-2 ./guix.fa.info-3 ./guix.fa.info-4 ./guix.fa.info-5 ./guix.fa.info-6 ./guix.fa.info ./version-fi.texi ./guix.fi.info-1 ./guix.fi.info-2 ./guix.fi.info-3 ./guix.fi.info-4 ./guix.fi.info-5 ./guix.fi.info-6 ./guix.fi.info ./version-it.texi ./guix.it.info-1 ./guix.it.info-2 ./guix.it.info-3 ./guix.it.info-4 ./guix.it.info-5 ./guix.it.info-6 ./guix.it.info ./version-ko.texi ./guix.ko.info-1 ./guix.ko.info-2 ./guix.ko.info-3 ./guix.ko.info-4 ./guix.ko.info-5 ./guix.ko.info-6 ./guix.ko.info ./version-sk.texi ./guix.sk.info-1 ./guix.sk.info-2 ./guix.sk.info-3 ./guix.sk.info-4 ./guix.sk.info-5 ./guix.sk.info-6 ./guix.sk.info ./guix-cookbook.es.info ./guix-cookbook.fa.info ./guix-cookbook.fi.info ./guix-cookbook.pt_BR.info ./guix-cookbook.ru.info ./guix-cookbook.zh_Hans.info ./he-config-bare-bones.scm ./guix-lint.1 ./build.scm ./contributing.texi ./guix-cookbook.texi ./htmlxref.cnf ./local.mk ./guix-cookbook.info ./guix-gc.1 ./guix-git.1 ./guix-container.1 ./guix-copy.1 ./guix-describe.1 ./guix-processes.1 ./guix-size.1 ./guix-weather.1 ./guix-shell.1 ./guix-cookbook.ko.texi ./guix-cookbook.fr.texi ./guix-cookbook.de.texi ./guix-cookbook.sk.texi ./contributing.zh_CN.texi ./contributing.ru.texi ./contributing.pt_BR.texi ./contributing.fr.texi ./contributing.es.texi ./contributing.de.texi ./guix.zh_CN.texi ./guix.ru.texi ./guix.pt_BR.texi ./guix.de.texi ./guix.es.texi ./guix.fr.texi ./guix-cookbook.sk.info ./guix-cookbook.de.info ./guix-cookbook.fr.info ./guix-cookbook.ko.info ./version-de.texi ./version-ru.texi ./version-fr.texi ./version-es.texi ./version-pt_BR.texi ./version-zh_CN.texi ./guix-daemon.1 ./guix.pt_BR.info-1 ./guix.pt_BR.info-2 ./guix.fr.info-1 ./guix.fr.info-2 ./guix.de.info-1 ./guix.de.info-2 ./guix.pt_BR.info-3 ./guix.ru.info-1 ./guix.ru.info-2 ./guix.fr.info-3 ./guix.ru.info-3 ./guix.zh_CN.info-1 ./guix.zh_CN.info-2 ./guix.es.info-1 ./guix.es.info-2 ./guix.de.info-3 ./guix.pt_BR.info-4 ./guix.fr.info-4 ./guix.ru.info-4 ./guix.zh_CN.info-3 ./guix.de.info-4 ./guix.es.info-3 ./guix.ru.info-5 ./guix.pt_BR.info-5 ./guix.zh_CN.info-4 ./guix.fr.info-5 ./guix.es.info-4 ./guix.de.info-5 ./guix.pt_BR.info-6 ./guix.ru.info-6 ./guix.fr.info-6 ./guix.pt_BR.info-7 ./guix.es.info-5 ./guix.zh_CN.info-5 ./guix.de.info-6 ./guix.pt_BR.info ./guix.ru.info-7 ./guix.fr.info-7 ./guix.ru.info-8 ./guix.de.info-7 ./guix.es.info-6 ./guix.fr.info-8 ./guix.zh_CN.info-6 ./guix.ru.info ./guix.fr.info ./guix.de.info-8 ./guix.zh_CN.info-7 ./guix.es.info-7 ./guix.de.info ./guix.zh_CN.info ./guix.es.info-8 ./guix.es.info ./guix.texi ./version.texi ./guix.info-1 ./guix.info-2 ./guix.info-3 ./guix.info-4 ./guix.info-5 ./guix.info-6 ./guix.info-7 ./guix.info ./guix-build.1 ./guix-archive.1 ./guix-import.1 ./guix-offload.1 ./guix-graph.1 ./guix-hash.1 ./guix-challenge.1 ./guix-download.1 ./guix-edit.1 ./guix-repl.1 ./guix-publish.1 ./guix-package.1 ./guix-style.1 ./guix-pull.1 ./guix-time-machine.1 ./guix-environment.1 ./guix-pack.1 ./guix-deploy.1 ./guix-refresh.1 ./guix-home.1 ./guix-system.1 ./guix.1 =2D-8<---------------cut here---------------end--------------->8--- Regards, Oleg. --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQJIBAEBCgAyFiEEcjhxI46s62NFSFhXFn+OpQAa+pwFAmQX/18UHGdvLndpZ3Vz dEBnbWFpbC5jb20ACgkQFn+OpQAa+pyHCg//Y+CwbxC5OzVdZFBdSDKYtxOV2w6Z w6X/ZXRt5sorx2Wfjsn0QJIzO9+6CTEBn1RSeOCQxuVvlQqjdeKqxQ0C90YQQG5t /R7PZ5wT2LHhztT1R52O8njqj93bppxkJDMsZVg4lg38WzNt2jVvL94mczS1V3zH 4VpccvDHYjIwK/ECkhTEKHRvRSGFqH7VcGJYbKYwGjmhF1Q6+lWq4XBiBhF9C6G0 +TwsS/GMXi207dFEtXdhit6DcxbEi70EseNj55aLDoM9xUumlQJuVTz5f9FRbL/C QOQmI2xFMPqN2EwZhBX3YL7FjwooA+nyyrj7H0AjWe3IX4RuVyvF9XZNIm1HM4P3 VlcR1foBdKxo1JRCTCU7V/hlD44Yl3e/FXQbKIS6aQyhRahrudYjeeL1gf7jBRya FAzkv2kP/ORoxP/bKj/BoNV372fOz8Sm5sSr5YzAZJz4DJQmctstIw+6WFcgP8Ca az682R4sMLDWteAxhZp/NNSg21irM6S7U4Tu4em3nP69ky2Fs5wUy7m1JzMWx9Vg IrODYJdDN0P5wgSM5hVKAW30J8uLXs3XxwRAl2W+NKCu8l4eonnpKU3KufaSclG+ Qz7cg6xNqTLzvDrYoDaaMhGd2WGJOdQ3wzH3TcWgR9Kh1STNwNnYfXXsZPvYQ5wi QWmUkZU8RZRQ3BI= =rP1f -----END PGP SIGNATURE----- --=-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Mon Mar 20 12:52:05 2023 Received: (at 62153) by debbugs.gnu.org; 20 Mar 2023 16:52:05 +0000 Received: from localhost ([127.0.0.1]:55600 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1peIjs-0002Yo-Rb for submit@debbugs.gnu.org; Mon, 20 Mar 2023 12:52:05 -0400 Received: from mail-lf1-f49.google.com ([209.85.167.49]:36379) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1peIjq-0002YI-Hb for 62153@debbugs.gnu.org; Mon, 20 Mar 2023 12:52:03 -0400 Received: by mail-lf1-f49.google.com with SMTP id f18so15737053lfa.3 for <62153@debbugs.gnu.org>; Mon, 20 Mar 2023 09:52:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1679331116; h=mime-version:user-agent:message-id:in-reply-to:date:references :subject:cc:to:from:from:to:cc:subject:date:message-id:reply-to; bh=vDPLnkJU/NiJrwBR2NdM9z+/Ugjh6HHpTqIecOHaUmI=; b=VFDbLiJCFblomjFlXAU3TV9Ek4QRk8/2D0qsjKVBLKRukS4sN3MXHOOjGHMEMD5sTt 4zVcH8LhqwM5aIA3TZOpkNipLtQXT9jzQDJs3ca0yFiAwa8T+6sD6/1iRM5/lLdO6SGo 2yvZwth6Q4KdCzVM5lsIKpOq5uvN5KIKjr3SA6OZeLm2d58loztH/doA0RtSTG31bD6I w+y+FWgypVtqJBB3jqJfM53kib5qwQE/YUSb0HmG+dhrPcSH3BTonB9D+r6Hb99f1lko LmDMrxVm7wC3Q64pHI2fFF42aW5o5NdR9VWDxdPz9PTbgl7hYPrs7wdXByaQhDomqtcF VPcA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679331116; h=mime-version:user-agent:message-id:in-reply-to:date:references :subject:cc:to:from:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=vDPLnkJU/NiJrwBR2NdM9z+/Ugjh6HHpTqIecOHaUmI=; b=OCrduDoyl4FkbbvVWcR+a1FKaOQtRV10bLjWgepUORrSWMu6aix1AsbzNXZ6+QlbGf l/HiWnQ9OhIi64OJdGz3qGRuaailpetQnoMyJzFPHRiu0n/GMsXwFrs37YSBfxN6EL2A n2mb1dE5S07w3HLwJldUWdMEXHZzNoP8jJ0Hy1SSFAft03jyPGLAgLzVyj5mPJr8ht8c 3ovk6yTnSh5v9KMm6DY1yRl5s9t0aVak7+iNGxXB0Dg8CK6sYC7TyWZR25rjGvWN94DL oUriE/Ltcw1/VHYgWQm1wmtXbgRQuPF7pfJkU0X6PwOm0SBo9hq6poK8W0iNwwOOCAm1 I2Fg== X-Gm-Message-State: AO0yUKXJpfWpsgONE0QS8KYeYDiG+KECjX3IwZhmv0cMDRjybbOUhcWz P/+StPAOgyoVQ/TUffDxCQ4= X-Google-Smtp-Source: AK7set+iSKR28x8ZygGcaJh3EhWKCF+ddXf1BtlI9zDwI059nVlkJCZW+YKZlPMdSKNypqhNzXRk7w== X-Received: by 2002:ac2:5304:0:b0:4e0:2e20:b663 with SMTP id c4-20020ac25304000000b004e02e20b663mr4949917lfh.6.1679331116145; Mon, 20 Mar 2023 09:51:56 -0700 (PDT) Received: from localhost ([88.201.161.72]) by smtp.gmail.com with ESMTPSA id x22-20020a19f616000000b0048a982ad0a8sm1762585lfe.23.2023.03.20.09.51.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 20 Mar 2023 09:51:55 -0700 (PDT) From: Oleg Pykhalov To: Ludovic =?utf-8?Q?Court=C3=A8s?= Subject: Re: bug#62153: [PATCH 0/2] Disarchive vs Gash-Utils for docker-layered References: <20230313003012.14325-1-go.wigust@gmail.com> <20230313003310.17129-1-go.wigust@gmail.com> <87r0tsk85r.fsf@gmail.com> <87bkkw2w7z.fsf@gmail.com> <86cz5b923c.fsf@gmail.com> <874jqnzo8a.fsf@elephly.net> <877cvhhth5.fsf@gnu.org> <87h6ugapwh.fsf@gmail.com> Date: Mon, 20 Mar 2023 19:51:52 +0300 In-Reply-To: <87h6ugapwh.fsf@gmail.com> (Oleg Pykhalov's message of "Mon, 20 Mar 2023 09:38:22 +0300") Message-ID: <87cz53bc2f.fsf_-_@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (gnu/linux) MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 62153 Cc: Josselin Poiret , Christopher Baines , Simon Tournier , Mathieu Othacehe , Tobias Geerinckx-Rice , Ricardo Wurmus , 62153@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Oleg Pykhalov writes: > [=E2=80=A6] > > And disarchive-assemble does not work, if I don't miss anything. I forgot about that input tarball was generated with changing current working directory, that's probably the reason of the broken archive. According to "Assembles from stdin to file" test from git.ngyro.com/disarchive/tests/cli.scm it is possible to generate a tarball for docker-layered with Disarchive, but a file describing spec like git.ngyro.com/disarchive/tests/data/test-archive.da is required. Generation of this file without having a Tar archive beforehand is complicated and because of that IMHO ustar.scm from Gash-Utils is preferred for the task. Regards, Oleg. --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQJIBAEBCgAyFiEEcjhxI46s62NFSFhXFn+OpQAa+pwFAmQYjygUHGdvLndpZ3Vz dEBnbWFpbC5jb20ACgkQFn+OpQAa+pzbEhAAy76WdREdgRRwg4j9h83H8dUa30x7 0+6pjM24kKBrDJYLGmv6aW2pumAr+YVre7n4/n1HdlXLStcdJ86/G0zraPj1wD7+ JvR04l0qGb7LY87lPdU/bGGkyt+YubnGbhZ6XoU0Q4CmHHo2aJzvq1S6+Z98m6YN f/RWl2mkxnrXDau+rLwE2RXQ4wvjZIMg77SSI64lgWoo3IaiwOuhWxcs2Q+ImebS ymirYoVzhFPCBgznkD2otEDNOvkUMxJ/e2caVGZ21F3guHhq0Df2gmYhRvaIqj/S JH8xfguqOTntBSvzhi0zjj50A+pyn976IV2C2Mu5mAqtQB0HpSayF/79Y3kwYXYH bteVg+tsGRrhC4VTPO3Tf5w2PE+SWr7j6ssSNNyRC9wcoYQjeE1y/btYEUeR85vc L3TS5uD/KM9WJmmpzxyAnmAGR/WkXAvFDtMZ7ZNcq6qTzQVQYltNJD/K69UZ55c4 vPU4KrnEx49lvtnMDzxl8BhzZsgVGjG6k26iX3B5ZyGWpGGayellDVwPWP21MLPr 1R4wanmcyykJvLl1iAVkDDNCRY8cJn4QKQDmUd0Y3gkSJi4Ic7zqYHzIFaEHbLda /jJ1XB1vTsadXQTJAIdOIQw0fXbK9NF7Tm68nm+czF6AyjUYU5Nz6NmQ85zAEAH4 1cpPWOSj0xOsoYQ= =uXt9 -----END PGP SIGNATURE----- --=-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Wed May 31 04:45:46 2023 Received: (at 62153) by debbugs.gnu.org; 31 May 2023 08:45:46 +0000 Received: from localhost ([127.0.0.1]:34841 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1q4HSk-0007FC-JJ for submit@debbugs.gnu.org; Wed, 31 May 2023 04:45:46 -0400 Received: from mail-lf1-f51.google.com ([209.85.167.51]:38149) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1q4HSf-0007Ex-Jr for 62153@debbugs.gnu.org; Wed, 31 May 2023 04:45:45 -0400 Received: by mail-lf1-f51.google.com with SMTP id 2adb3069b0e04-4f4453b607eso990738e87.1 for <62153@debbugs.gnu.org>; Wed, 31 May 2023 01:45:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1685522735; x=1688114735; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=q5xwnTwyxZlRIQ+AmsYG7aMw5GGuHuyD8/kAgcEx3XM=; b=A1bs+0+KLzfwhVypG63FxfNEn5DlaqgtbXEDWybhDPSuix2XeC16ixx7BLI8D3k4X5 is70hlVjzt5ByNpJZZTIcnhhseRZaV0m8XOcdjO6W49cWeIa2Setr7cvOC7ZKMUbT6Vy 2IZlfqO6mifjADaqNlNdxp1zqf4z6eu3NS23s9biUTR8Yxdf30Qqr0hkVa9EF0bFxZpd B+K9RttL3R6j+TqZQqhdth9ptUT1MpBVHh6NmlM9n4ahmxJD3i0W2XwzuHAUho8yOQ2R 3H89tlQ/j79MOl48j63e1tnAScEHq/VvIFIbZrNAW5Kc8uoxinpTkU80Y45fxyNVywGW dDWQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685522735; x=1688114735; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=q5xwnTwyxZlRIQ+AmsYG7aMw5GGuHuyD8/kAgcEx3XM=; b=ZgHE9YG5e108LgKLTiTNVgRDjA1tYDeCphYEKBpMGXx/Id/9JSUe1T05jbMdTGbzO/ CjSzXWzbPzBpdr8wBwa27di7Jksiv9D1T9dP5QnQTIIZBjQhFt/zV7uhfHbBr6OjijSc ketnfNhdLyfBArjnz4Gv+FUHyYItM2rGP+dadIQ+7hOW+PqKIxpqfc+zXf5XNwYYceij ZSr5M4SZCBVNYokEHugtwlDWl1TtT5IVpWlKcbBBWAS2S0AlYXu3HFf1uzHNly6rjuuE 7j21Pa/WD8eZd+kk3umDOIUjesiTlYTILWl+QDaEXrMiwTgsJNz4FVoEWcrEfVTX0Qp6 jiYQ== X-Gm-Message-State: AC+VfDw7MrsUb0XNkmQaz3VBg23HlmHjjX0DMaJf7xVH4JIuRwNIuws8 S0SyoJGUSASi135XzpMMasnSo6XSxY8= X-Google-Smtp-Source: ACHHUZ7bwsSFXu6iV1WQ4HLZhhH8GepTaYZ4+NcBRVEl+unwBKlJ2St2MiIblLFowtUCLQaJfLEr2Q== X-Received: by 2002:a19:a415:0:b0:4ea:fafd:e66f with SMTP id q21-20020a19a415000000b004eafafde66fmr478685lfc.0.1685522734478; Wed, 31 May 2023 01:45:34 -0700 (PDT) Received: from guixsd.wugi.info ([88.201.161.72]) by smtp.gmail.com with ESMTPSA id q8-20020ac246e8000000b004f14ae5ded8sm635811lfo.28.2023.05.31.01.45.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 May 2023 01:45:34 -0700 (PDT) From: Oleg Pykhalov To: 62153@debbugs.gnu.org Subject: [PATCH] Add Docker layered image for pack and system (v3) Date: Wed, 31 May 2023 11:45:29 +0300 Message-Id: X-Mailer: git-send-email 2.38.0 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 62153 Cc: Oleg Pykhalov X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Hi, Guix. These patches series is rebased on origin/master. Also, the Python script is replaced in favour of calls to GNU Tar and GNU Gzip programs. Passed tests: make check TESTS="tests/pack.scm" make check-system TESTS="docker-system" make check-system TESTS="docker-layered-system" Oleg Pykhalov (2): guix: docker: Build layered image. news: Add entry for the new 'docker-layered' distribution format. doc/guix.texi | 18 +++- etc/news.scm | 58 ++++++++++++ gnu/image.scm | 3 +- gnu/system/image.scm | 76 +++++++++++---- gnu/tests/docker.scm | 20 +++- guix/docker.scm | 205 +++++++++++++++++++++++++++++++--------- guix/scripts/pack.scm | 62 ++++++++++-- guix/scripts/system.scm | 11 ++- tests/pack.scm | 48 ++++++++++ 9 files changed, 424 insertions(+), 77 deletions(-) base-commit: 77f52db416a13e195d090cad4e9e7658feb2e86b -- 2.38.0 From debbugs-submit-bounces@debbugs.gnu.org Wed May 31 04:48:36 2023 Received: (at 62153) by debbugs.gnu.org; 31 May 2023 08:48:36 +0000 Received: from localhost ([127.0.0.1]:34847 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1q4HVT-0007O2-2l for submit@debbugs.gnu.org; Wed, 31 May 2023 04:48:36 -0400 Received: from mail-lf1-f47.google.com ([209.85.167.47]:39281) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1q4HVN-0007NI-0q for 62153@debbugs.gnu.org; Wed, 31 May 2023 04:48:33 -0400 Received: by mail-lf1-f47.google.com with SMTP id 2adb3069b0e04-4f3b03358e9so896236e87.1 for <62153@debbugs.gnu.org>; Wed, 31 May 2023 01:48:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1685522902; x=1688114902; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=UJJST3rnJWZsshyWjybCclOol2Vy0Lxza5SzzsyuvOs=; b=FOit0MlRYK0qYOVLhVpV1SdhwTxQGI/Hr05W3oxvHGKguveo8AOOb1MEIWqKiv/ArB PFqRcbpELFSJIDE3nxFppbp9acwQGd0EMYTU8Q6mtBHihRWuYSSvx8r+dO3EcKAZLXTZ vZgr3bvoJrDs8RwDD2KQQ/twlYcOqBEBFVS7WGlRWJ99MRA5VWvrSsxC+I+drtqeCHuX rXINnxEw4pzGoPP+1fBGfqMlyPLq2YD9sUTFoUJ6qZPgZO9SKxOPV0jgAKjc+zD8ALAx /F2+enNNtehuEVELfkhVHPHznCyMhwT254hfDXlvaW+iCOIrLAgQl4lADnQhtffBDWwQ ixig== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685522902; x=1688114902; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=UJJST3rnJWZsshyWjybCclOol2Vy0Lxza5SzzsyuvOs=; b=QBNRpV1mUh8E77xcxm3cu+Bm1qwhGdOfjRD6pq7wGMZaqNwbqZJXw0mHM+X/uPYnxZ fQFeK9nLK+cf3vDCBaRBLkAKz5A2/SszPe7K5yXaKHMnEK8RgHZm0mKgrsCgJkmaDS7k dVSwp8YeNGJSW4DRwsKrAGEGHDj2qTCvivr/Z3D4rgpwiUIP882snYMN/FTfnWCQp+v+ JMaLrVw1wBTE0sNFjXY+CLuclDVSsukno3cgBkFsPhG1pW/jBwB26WePRZnGC0kDoQPs W2RjaMK+QEMxI2rKfEEtx/GxaxlSLJcXBTs7trS3vQD/8RhLBfNNRDDea7lnI9OoHQBR nY+g== X-Gm-Message-State: AC+VfDxxhfy8UA0EJLAomsyO+XzFM5Go+p47MSmh+m5/Q1Xlu1kCow3J K8lFJawCwLjZ9RVO/C0GsCHyrpazzUI= X-Google-Smtp-Source: ACHHUZ7wZ8y+6+dgeowGue23Hjexqn+EBU/0YPh7QFjtA50Cgk2su32hAhMFmZeweUhzWg2Pzl2Nlg== X-Received: by 2002:ac2:53b5:0:b0:4f3:a4fc:6283 with SMTP id j21-20020ac253b5000000b004f3a4fc6283mr632826lfh.3.1685522901774; Wed, 31 May 2023 01:48:21 -0700 (PDT) Received: from guixsd.wugi.info ([88.201.161.72]) by smtp.gmail.com with ESMTPSA id m9-20020ac24ac9000000b004f3886a63d1sm620295lfp.298.2023.05.31.01.48.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 May 2023 01:48:21 -0700 (PDT) From: Oleg Pykhalov To: 62153@debbugs.gnu.org Subject: [PATCH] guix: docker: Build layered image. Date: Wed, 31 May 2023 11:47:53 +0300 Message-Id: X-Mailer: git-send-email 2.38.0 In-Reply-To: References: MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 62153 Cc: Oleg Pykhalov X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) * doc/guix.texi (Invoking guix pack): Document docker-layered format. (image Reference): Same. (image-type Reference): Document docker-layered-image-type. * gnu/image.scm (validate-image-format)[docker-layered]: New image format. * gnu/system/image.scm (docker-layered-image, docker-layered-image-type): New variables. (system-docker-image)[layered-image?]: New argument. (system-docker-layered-image): New procedure. (image->root-file-system)[docker-layered]: New image format. * gnu/tests/docker.scm (%test-docker-layered-system): New test. * guix/docker.scm (%docker-image-max-layers): New variable. (build-docker-image)[stream-layered-image, root-system]: New arguments. * guix/scripts/pack.scm (stream-layered-image.py): New variable. (docker-image)[layered-image?]: New argument. (docker-layered-image): New procedure. (%formats)[docker-layered]: New format. (show-formats): Document this. * guix/scripts/system.scm (system-derivation-for-action)[docker-layered-image]: New action. (show-help): Document this. (actions)[docker-layered-image]: New action. (process-action): Add this. * tests/pack.scm: Add "docker-layered-image + localstatedir" test. --- doc/guix.texi | 18 +++- gnu/image.scm | 3 +- gnu/system/image.scm | 76 +++++++++++---- gnu/tests/docker.scm | 20 +++- guix/docker.scm | 205 +++++++++++++++++++++++++++++++--------- guix/scripts/pack.scm | 62 ++++++++++-- guix/scripts/system.scm | 11 ++- tests/pack.scm | 48 ++++++++++ 8 files changed, 366 insertions(+), 77 deletions(-) diff --git a/doc/guix.texi b/doc/guix.texi index 5fd2449ed5..1c95ec4320 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -56,7 +56,7 @@ Copyright @copyright{} 2017, 2018, 2019, 2020 Arun Isaac@* Copyright @copyright{} 2017 nee@* Copyright @copyright{} 2018 Rutger Helling@* -Copyright @copyright{} 2018, 2021 Oleg Pykhalov@* +Copyright @copyright{} 2018, 2021, 2023 Oleg Pykhalov@* Copyright @copyright{} 2018 Mike Gerwitz@* Copyright @copyright{} 2018 Pierre-Antoine Rouby@* Copyright @copyright{} 2018, 2019 Gábor Boskovits@* @@ -6984,9 +6984,15 @@ Invoking guix pack guix pack -f docker -S /bin=bin guile guile-readline @end example +or + +@example +guix pack -f docker-layered -S /bin=bin guile guile-readline +@end example + @noindent -The result is a tarball that can be passed to the @command{docker load} -command, followed by @code{docker run}: +The result is a tarball with image or layered image that can be passed +to the @command{docker load} command, followed by @code{docker run}: @example docker load < @var{file} @@ -44309,6 +44315,8 @@ image Reference @item @code{docker}, a Docker image. +@item @code{docker-layered}, a layered Docker image. + @item @code{iso9660}, an ISO-9660 image. @item @code{tarball}, a tar.gz image archive. @@ -44644,6 +44652,10 @@ image-type Reference Build an image based on the @code{docker-image} image. @end defvar +@defvar docker-layered-image-type +Build a layered image based on the @code{docker-layered-image} image. +@end defvar + @defvar raw-with-offset-image-type Build an MBR image with a single partition starting at a @code{1024KiB} offset. This is useful to leave some room to install a bootloader in diff --git a/gnu/image.scm b/gnu/image.scm index 523653dd77..8a6a0d8479 100644 --- a/gnu/image.scm +++ b/gnu/image.scm @@ -1,5 +1,6 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2020, 2022 Mathieu Othacehe +;;; Copyright © 2023 Oleg Pykhalov ;;; ;;; This file is part of GNU Guix. ;;; @@ -152,7 +153,7 @@ (define-syntax-rule (define-set-sanitizer name field set) ;; The supported image formats. (define-set-sanitizer validate-image-format format - (disk-image compressed-qcow2 docker iso9660 tarball wsl2)) + (disk-image compressed-qcow2 docker docker-layered iso9660 tarball wsl2)) ;; The supported partition table types. (define-set-sanitizer validate-partition-table-type partition-table-type diff --git a/gnu/system/image.scm b/gnu/system/image.scm index afef79185f..3a502f19ec 100644 --- a/gnu/system/image.scm +++ b/gnu/system/image.scm @@ -4,6 +4,7 @@ ;;; Copyright © 2022 Pavel Shlyak ;;; Copyright © 2022 Denis 'GNUtoo' Carikli ;;; Copyright © 2022 Alex Griffin +;;; Copyright © 2023 Oleg Pykhalov ;;; ;;; This file is part of GNU Guix. ;;; @@ -78,6 +79,7 @@ (define-module (gnu system image) efi-disk-image iso9660-image docker-image + docker-layered-image tarball-image wsl2-image raw-with-offset-disk-image @@ -89,6 +91,7 @@ (define-module (gnu system image) iso-image-type uncompressed-iso-image-type docker-image-type + docker-layered-image-type tarball-image-type wsl2-image-type raw-with-offset-image-type @@ -167,6 +170,10 @@ (define docker-image (image-without-os (format 'docker))) +(define docker-layered-image + (image-without-os + (format 'docker-layered))) + (define tarball-image (image-without-os (format 'tarball))) @@ -237,6 +244,11 @@ (define docker-image-type (name 'docker) (constructor (cut image-with-os docker-image <>)))) +(define docker-layered-image-type + (image-type + (name 'docker-layered) + (constructor (cut image-with-os docker-layered-image <>)))) + (define tarball-image-type (image-type (name 'tarball) @@ -633,9 +645,12 @@ (define (image-with-label base-image label) (define* (system-docker-image image #:key - (name "docker-image")) + (name "docker-image") + (archiver tar) + layered-image?) "Build a docker image for IMAGE. NAME is the base name to use for the -output file." +output file. If LAYERED-IMAGE? is true, the image will with many of the store +paths being on their own layer to improve sharing between images." (define boot-program ;; Program that runs the boot script of OS, which in turn starts shepherd. (program-file "boot-program" @@ -678,9 +693,11 @@ (define* (system-docker-image image (use-modules (guix docker) (guix build utils) (gnu build image) + (srfi srfi-1) (srfi srfi-19) (guix build store-copy) - (guix store database)) + (guix store database) + (ice-9 receive)) ;; Set the SQL schema location. (sql-schema #$schema) @@ -700,18 +717,31 @@ (define* (system-docker-image image #:register-closures? #$register-closures? #:deduplicate? #f #:system-directory #$os) - (build-docker-image - #$output - (cons* image-root - (map store-info-item - (call-with-input-file #$graph - read-reference-graph))) - #$os - #:entry-point '(#$boot-program #$os) - #:compressor '(#+(file-append gzip "/bin/gzip") "-9n") - #:creation-time (make-time time-utc 0 1) - #:system #$image-target - #:transformations `((,image-root -> "")))))))) + (when #$layered-image? + (setenv "PATH" + (string-join (list #+(file-append archiver "/bin") + #+(file-append coreutils "/bin") + #+(file-append gzip "/bin")) + ":"))) + (apply build-docker-image + (append (list #$output + (append (if #$layered-image? + '() + (list image-root)) + (map store-info-item + (call-with-input-file #$graph + read-reference-graph))) + #$os + #:entry-point '(#$boot-program #$os) + #:compressor + '(#+(file-append gzip "/bin/gzip") "-9n") + #:creation-time (make-time time-utc 0 1) + #:system #$image-target + #:transformations `((,image-root -> ""))) + (if #$layered-image? + (list #:root-system image-root + #:layered-image? #$layered-image?) + '())))))))) (computed-file name builder ;; Allow offloading so that this I/O-intensive process @@ -720,6 +750,18 @@ (define* (system-docker-image image #:options `(#:references-graphs ((,graph ,os)) #:substitutable? ,substitutable?)))) +(define* (system-docker-layered-image image + #:key + (name "docker-image") + (archiver tar) + (layered-image? #t)) + "Build a docker image for IMAGE. NAME is the base name to use for the +output file." + (system-docker-image image + #:name name + #:archiver archiver + #:layered-image? layered-image?)) + ;;; ;;; Tarball image. @@ -811,7 +853,7 @@ (define (image->root-file-system image) "Return the IMAGE root partition file-system type." (case (image-format image) ((iso9660) "iso9660") - ((docker tarball wsl2) "dummy") + ((docker docker-layered tarball wsl2) "dummy") (else (partition-file-system (find-root-partition image))))) @@ -948,6 +990,8 @@ (define* (system-image image) ("bootcfg" ,bootcfg)))) ((memq image-format '(docker)) (system-docker-image image*)) + ((memq image-format '(docker-layered)) + (system-docker-layered-image image*)) ((memq image-format '(tarball)) (system-tarball-image image*)) ((memq image-format '(wsl2)) diff --git a/gnu/tests/docker.scm b/gnu/tests/docker.scm index edc9804414..0cccc02ad2 100644 --- a/gnu/tests/docker.scm +++ b/gnu/tests/docker.scm @@ -1,6 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2019 Danny Milosavljevic ;;; Copyright © 2019-2023 Ludovic Courtès +;;; Copyright © 2023 Oleg Pykhalov ;;; ;;; This file is part of GNU Guix. ;;; @@ -43,7 +44,8 @@ (define-module (gnu tests docker) #:use-module (guix build-system trivial) #:use-module ((guix licenses) #:prefix license:) #:export (%test-docker - %test-docker-system)) + %test-docker-system + %test-docker-layered-system)) (define %docker-os (simple-operating-system @@ -316,3 +318,19 @@ (define %test-docker-system (locale-libcs (list glibc))) #:type docker-image-type))) run-docker-system-test))))) + +(define %test-docker-layered-system + (system-test + (name "docker-layered-system") + (description "Run a system image as produced by @command{guix system +docker-layered-image} inside Docker.") + (value (with-monad %store-monad + (>>= (lower-object + (system-image (os->image + (operating-system + (inherit (simple-operating-system)) + ;; Use locales for a single libc to + ;; reduce space requirements. + (locale-libcs (list glibc))) + #:type docker-layered-image-type))) + run-docker-system-test))))) diff --git a/guix/docker.scm b/guix/docker.scm index 5e6460f43f..e10b940aa4 100644 --- a/guix/docker.scm +++ b/guix/docker.scm @@ -3,6 +3,7 @@ ;;; Copyright © 2017, 2018, 2019, 2021 Ludovic Courtès ;;; Copyright © 2018 Chris Marusich ;;; Copyright © 2021 Maxim Cournoyer +;;; Copyright © 2023 Oleg Pykhalov ;;; ;;; This file is part of GNU Guix. ;;; @@ -28,6 +29,8 @@ (define-module (guix docker) delete-file-recursively with-directory-excursion invoke)) + #:use-module (guix diagnostics) + #:use-module (guix i18n) #:use-module (gnu build install) #:use-module (json) ;guile-json #:use-module (srfi srfi-1) @@ -38,6 +41,9 @@ (define-module (guix docker) #:use-module (rnrs bytevectors) #:use-module (ice-9 ftw) #:use-module (ice-9 match) + #:use-module (ice-9 popen) + #:use-module (ice-9 rdelim) + #:use-module (ice-9 receive) #:export (build-docker-image)) ;; Generate a 256-bit identifier in hexadecimal encoding for the Docker image. @@ -92,12 +98,12 @@ (define (canonicalize-repository-name name) (make-string (- min-length l) padding-character))) (_ normalized-name)))) -(define* (manifest path id #:optional (tag "guix")) +(define* (manifest path layers #:optional (tag "guix")) "Generate a simple image manifest." (let ((tag (canonicalize-repository-name tag))) `#(((Config . "config.json") (RepoTags . #(,(string-append tag ":latest"))) - (Layers . #(,(string-append id "/layer.tar"))))))) + (Layers . ,(list->vector layers)))))) ;; According to the specifications this is required for backwards ;; compatibility. It duplicates information provided by the manifest. @@ -106,8 +112,8 @@ (define* (repositories path id #:optional (tag "guix")) `((,(canonicalize-repository-name tag) . ((latest . ,id))))) ;; See https://github.com/opencontainers/image-spec/blob/master/config.md -(define* (config layer time arch #:key entry-point (environment '())) - "Generate a minimal image configuration for the given LAYER file." +(define* (config layers-diff-ids time arch #:key entry-point (environment '())) + "Generate a minimal image configuration for the given LAYERS files." ;; "architecture" must be values matching "platform.arch" in the ;; runtime-spec at ;; https://github.com/opencontainers/runtime-spec/blob/v1.0.0-rc2/config.md#platform @@ -125,7 +131,7 @@ (define* (config layer time arch #:key entry-point (environment '())) (container_config . #nil) (os . "linux") (rootfs . ((type . "layers") - (diff_ids . #(,(layer-diff-id layer))))))) + (diff_ids . ,(list->vector layers-diff-ids)))))) (define directive-file ;; Return the file or directory created by a 'evaluate-populate-directive' @@ -136,6 +142,37 @@ (define directive-file (('directory name _ ...) (string-trim name #\/)))) +(define %docker-image-max-layers + 100) + +(define (paths-split-sort paths) + "Split list of PATHS at %DOCKER-IMAGE-MAX-LAYERS and sort by disk usage." + (let* ((paths-length (length paths)) + (port (apply open-pipe* OPEN_READ + (append '("du" "--summarize") paths))) + (output (read-string port))) + (close-port port) + (receive (head tail) + (split-at + (map (match-lambda ((size . path) path)) + (sort (map (lambda (line) + (match (string-split line #\tab) + ((size path) + (cons (string->number size) path)))) + (string-split + (string-trim-right output #\newline) + #\newline)) + (lambda (path1 path2) + (< (match path2 ((size . _) size)) + (match path1 ((size . _) size)))))) + (if (>= paths-length %docker-image-max-layers) + (- %docker-image-max-layers 2) + (1- paths-length))) + (list head tail)))) + +(define (create-empty-tar file) + (invoke "tar" "-cf" file "--files-from" "/dev/null")) + (define* (build-docker-image image paths prefix #:key (repository "guix") @@ -146,11 +183,13 @@ (define* (build-docker-image image paths prefix entry-point (environment '()) compressor - (creation-time (current-time time-utc))) - "Write to IMAGE a Docker image archive containing the given PATHS. PREFIX -must be a store path that is a prefix of any store paths in PATHS. REPOSITORY -is a descriptive name that will show up in \"REPOSITORY\" column of the output -of \"docker images\". + (creation-time (current-time time-utc)) + layered-image? + root-system) + "Write to IMAGE a layerer Docker image archive containing the given PATHS. +PREFIX must be a store path that is a prefix of any store paths in PATHS. +REPOSITORY is a descriptive name that will show up in \"REPOSITORY\" column of +the output of \"docker images\". When DATABASE is true, copy it to /var/guix/db in the image and create /var/guix/gcroots and friends. @@ -172,7 +211,14 @@ (define* (build-docker-image image paths prefix SYSTEM is a GNU triplet (or prefix thereof) of the system the binaries in PATHS are for; it is used to produce metadata in the image. Use COMPRESSOR, a command such as '(\"gzip\" \"-9n\"), to compress IMAGE. Use CREATION-TIME, a -SRFI-19 time-utc object, as the creation time in metadata." +SRFI-19 time-utc object, as the creation time in metadata. + +When LAYERED-IMAGE? is true build layered image, providing a Docker +image with many of the store paths being on their own layer to improve sharing +between images. + +ROOT-SYSTEM is a directory with a provisioned root file system, which will be +added to image as a layer." (define (sanitize path-fragment) (escape-special-chars ;; GNU tar strips the leading slash off of absolute paths before applying @@ -203,6 +249,53 @@ (define* (build-docker-image image paths prefix (if (eq? '() transformations) '() `("--transform" ,(transformations->expression transformations)))) + (define layers-hashes + (match-lambda + (((head ...) (tail ...) id) + (create-empty-tar "image.tar") + (let* ((head-layers + (map + (lambda (file) + (invoke "tar" "cf" "layer.tar" file) + (let* ((file-hash (layer-diff-id "layer.tar")) + (file-name (string-append file-hash "/layer.tar"))) + (mkdir file-hash) + (rename-file "layer.tar" file-name) + (invoke "tar" "-rf" "image.tar" file-name) + (delete-file file-name) + file-hash)) + head)) + (tail-layer + (begin + (create-empty-tar "layer.tar") + (for-each (lambda (file) + (invoke "tar" "-rf" "layer.tar" file)) + tail) + (let* ((file-hash (layer-diff-id "layer.tar")) + (file-name (string-append file-hash "/layer.tar"))) + (mkdir file-hash) + (rename-file "layer.tar" file-name) + (invoke "tar" "-rf" "image.tar" file-name) + (delete-file file-name) + file-hash))) + (customization-layer + (let* ((file-id (string-append id "/layer.tar")) + (file-hash (layer-diff-id file-id)) + (file-name (string-append file-hash "/layer.tar"))) + (mkdir file-hash) + (rename-file file-id file-name) + (invoke "tar" "-rf" "image.tar" file-name) + file-hash)) + (all-layers + (append head-layers (list tail-layer customization-layer)))) + (with-output-to-file "manifest.json" + (lambda () + (scm->json (manifest prefix + (map (cut string-append <> "/layer.tar") + all-layers) + repository)))) + (invoke "tar" "-rf" "image.tar" "manifest.json") + all-layers)))) (let* ((directory "/tmp/docker-image") ;temporary working directory (id (docker-id prefix)) (time (date->string (time-utc->date creation-time) "~4")) @@ -229,26 +322,39 @@ (define* (build-docker-image image paths prefix (with-output-to-file "json" (lambda () (scm->json (image-description id time)))) - ;; Create a directory for the non-store files that need to go into the - ;; archive. - (mkdir "extra") + (if root-system + (let ((directory (getcwd))) + (with-directory-excursion root-system + (apply invoke "tar" + "-cf" (string-append directory "/layer.tar") + `(,@transformation-options + ,@(tar-base-options) + ,@(scandir "." + (lambda (file) + (not (member file '("." ".."))))))))) + (begin + ;; Create a directory for the non-store files that need to go + ;; into the archive. + (mkdir "extra") - (with-directory-excursion "extra" - ;; Create non-store files. - (for-each (cut evaluate-populate-directive <> "./") - extra-files) + (with-directory-excursion "extra" + ;; Create non-store files. + (for-each (cut evaluate-populate-directive <> "./") + extra-files) - (when database - ;; Initialize /var/guix, assuming PREFIX points to a profile. - (install-database-and-gc-roots "." database prefix)) + (when database + ;; Initialize /var/guix, assuming PREFIX points to a + ;; profile. + (install-database-and-gc-roots "." database prefix)) - (apply invoke "tar" "-cf" "../layer.tar" - `(,@transformation-options - ,@(tar-base-options) - ,@paths - ,@(scandir "." - (lambda (file) - (not (member file '("." "..")))))))) + (apply invoke "tar" "-cf" "../layer.tar" + `(,@transformation-options + ,@(tar-base-options) + ,@(if layered-image? '() paths) + ,@(scandir "." + (lambda (file) + (not (member file '("." "..")))))))) + (delete-file-recursively "extra"))) ;; It is possible for "/" to show up in the archive, especially when ;; applying transformations. For example, the transformation @@ -261,24 +367,33 @@ (define* (build-docker-image image paths prefix ;; error messages. (with-error-to-port (%make-void-port "w") (lambda () - (system* "tar" "--delete" "/" "-f" "layer.tar"))) - - (delete-file-recursively "extra")) + (system* "tar" "--delete" "/" "-f" "layer.tar")))) (with-output-to-file "config.json" (lambda () - (scm->json (config (string-append id "/layer.tar") - time arch - #:environment environment - #:entry-point entry-point)))) - (with-output-to-file "manifest.json" - (lambda () - (scm->json (manifest prefix id repository)))) - (with-output-to-file "repositories" - (lambda () - (scm->json (repositories prefix id repository))))) - - (apply invoke "tar" "-cf" image "-C" directory - `(,@(tar-base-options #:compressor compressor) - ".")) + (scm->json + (config (if layered-image? + (layers-hashes (append (paths-split-sort paths) + (list id))) + (list (layer-diff-id (string-append id "/layer.tar")))) + time arch + #:environment environment + #:entry-point entry-point)))) + (if layered-image? + (begin + (invoke "tar" "-rf" "image.tar" "config.json") + (apply invoke `(,@compressor "image.tar")) + (copy-file "image.tar.gz" image)) + (begin + (with-output-to-file "manifest.json" + (lambda () + (scm->json (manifest prefix + (list (string-append id "/layer.tar")) + repository)))) + (with-output-to-file "repositories" + (lambda () + (scm->json (repositories prefix id repository)))) + (apply invoke "tar" "-cf" image + `(,@(tar-base-options #:compressor compressor) + "."))))) (delete-file-recursively directory))) diff --git a/guix/scripts/pack.scm b/guix/scripts/pack.scm index 0dc9979194..3fefd2eac3 100644 --- a/guix/scripts/pack.scm +++ b/guix/scripts/pack.scm @@ -8,6 +8,7 @@ ;;; Copyright © 2020, 2021, 2022, 2023 Maxim Cournoyer ;;; Copyright © 2020 Eric Bavier ;;; Copyright © 2022 Alex Griffin +;;; Copyright © 2023 Oleg Pykhalov ;;; ;;; This file is part of GNU Guix. ;;; @@ -28,6 +29,7 @@ (define-module (guix scripts pack) #:use-module (guix scripts) #:use-module (guix ui) #:use-module (guix gexp) + #:use-module ((guix build utils) #:select (%xz-parallel-args)) #:use-module (guix utils) #:use-module (guix store) #:use-module ((guix status) #:select (with-status-verbosity)) @@ -53,6 +55,8 @@ (define-module (guix scripts pack) #:use-module ((gnu packages compression) #:hide (zip)) #:use-module (gnu packages guile) #:use-module (gnu packages base) + #:use-module (gnu packages shells) + #:autoload (gnu packages package-management) (guix) #:autoload (gnu packages gnupg) (guile-gcrypt) #:autoload (gnu packages guile) (guile2.0-json guile-json) #:use-module (srfi srfi-1) @@ -67,6 +71,7 @@ (define-module (guix scripts pack) debian-archive rpm-archive docker-image + docker-layered-image squashfs-image %formats @@ -597,12 +602,14 @@ (define* (docker-image name profile localstatedir? (symlinks '()) (archiver tar) - (extra-options '())) + (extra-options '()) + layered-image?) "Return a derivation to construct a Docker image of PROFILE. The image is a tarball conforming to the Docker Image Specification, compressed with COMPRESSOR. It can be passed to 'docker load'. If TARGET is true, it must a be a GNU triplet and it is used to derive the architecture metadata in -the image." +the image. If LAYERED-IMAGE? is true, the image will with many of the +store paths being on their own layer to improve sharing between images." (define database (and localstatedir? (file-append (store-database (list profile)) @@ -653,7 +660,13 @@ (define* (docker-image name profile `((directory "/tmp" ,(getuid) ,(getgid) #o1777) ,@(append-map symlink->directives '#$symlinks))) - (setenv "PATH" #+(file-append archiver "/bin")) + (setenv "PATH" + (string-join `(#+(file-append archiver "/bin") + #+@(if layered-image? + (list (file-append coreutils "/bin") + (file-append gzip "/bin")) + '())) + ":")) (build-docker-image #$output (map store-info-item @@ -671,7 +684,8 @@ (define* (docker-image name profile #$entry-point))) #:extra-files directives #:compressor #+(compressor-command compressor) - #:creation-time (make-time time-utc 0 1)))))) + #:creation-time (make-time time-utc 0 1) + #:layered-image? #$layered-image?))))) (gexp->derivation (string-append name ".tar" (compressor-extension compressor)) @@ -679,6 +693,33 @@ (define* (docker-image name profile #:target target #:references-graphs `(("profile" ,profile)))) +(define* (docker-layered-image name profile + #:key target + (profile-name "guix-profile") + (compressor (first %compressors)) + entry-point + localstatedir? + (symlinks '()) + (archiver tar) + (extra-options '()) + (layered-image? #t)) + "Return a derivation to construct a Docker image of PROFILE. The image is a +tarball conforming to the Docker Image Specification, compressed with +COMPRESSOR. It can be passed to 'docker load'. If TARGET is true, it must a +be a GNU triplet and it is used to derive the architecture metadata in the +image. If LAYERED-IMAGE? is true, the image will with many of the store paths +being on their own layer to improve sharing between images." + (docker-image name profile + #:target target + #:profile-name profile-name + #:compressor compressor + #:entry-point entry-point + #:localstatedir? localstatedir? + #:symlinks symlinks + #:archiver archiver + #:extra-options extra-options + #:layered-image? layered-image?)) + ;;; ;;; Debian archive format. @@ -1353,6 +1394,7 @@ (define %formats `((tarball . ,self-contained-tarball) (squashfs . ,squashfs-image) (docker . ,docker-image) + (docker-layered . ,docker-layered-image) (deb . ,debian-archive) (rpm . ,rpm-archive))) @@ -1361,15 +1403,17 @@ (define (show-formats) (display (G_ "The supported formats for 'guix pack' are:")) (newline) (display (G_ " - tarball Self-contained tarball, ready to run on another machine")) + tarball Self-contained tarball, ready to run on another machine")) + (display (G_ " + squashfs Squashfs image suitable for Singularity")) (display (G_ " - squashfs Squashfs image suitable for Singularity")) + docker Tarball ready for 'docker load'")) (display (G_ " - docker Tarball ready for 'docker load'")) + docker-layered Tarball with a layered image ready for 'docker load'")) (display (G_ " - deb Debian archive installable via dpkg/apt")) + deb Debian archive installable via dpkg/apt")) (display (G_ " - rpm RPM archive installable via rpm/yum")) + rpm RPM archive installable via rpm/yum")) (newline)) (define (required-option symbol) diff --git a/guix/scripts/system.scm b/guix/scripts/system.scm index d7163dd3eb..e4bf0347c7 100644 --- a/guix/scripts/system.scm +++ b/guix/scripts/system.scm @@ -11,6 +11,7 @@ ;;; Copyright © 2021 Brice Waegeneire ;;; Copyright © 2021 Simon Tournier ;;; Copyright © 2022 Tobias Geerinckx-Rice +;;; Copyright © 2023 Oleg Pykhalov ;;; ;;; This file is part of GNU Guix. ;;; @@ -727,13 +728,15 @@ (define* (system-derivation-for-action image action #:graphic? graphic? #:disk-image-size image-size #:mappings mappings)) - ((image disk-image vm-image docker-image) + ((image disk-image vm-image docker-image docker-layered-image) (when (eq? action 'disk-image) (warning (G_ "'disk-image' is deprecated: use 'image' instead~%"))) (when (eq? action 'vm-image) (warning (G_ "'vm-image' is deprecated: use 'image' instead~%"))) (when (eq? action 'docker-image) (warning (G_ "'docker-image' is deprecated: use 'image' instead~%"))) + (when (eq? action 'docker-layered-image) + (warning (G_ "'docker-layered-image' is deprecated: use 'image' instead~%"))) (lower-object (system-image image)))))) (define (maybe-suggest-running-guix-pull) @@ -980,6 +983,8 @@ (define (show-help) image build a Guix System image\n")) (display (G_ "\ docker-image build a Docker image\n")) + (display (G_ "\ + docker-layered-image build a Docker layered image\n")) (display (G_ "\ init initialize a root file system to run GNU\n")) (display (G_ "\ @@ -1193,7 +1198,7 @@ (define actions '("build" "container" "vm" "vm-image" "image" "disk-image" "list-generations" "describe" "delete-generations" "roll-back" "switch-generation" "search" "edit" - "docker-image")) + "docker-image" "docker-layered-image")) (define (process-action action args opts) "Process ACTION, a sub-command, with the arguments are listed in ARGS. @@ -1242,6 +1247,8 @@ (define (process-action action args opts) (image (let* ((image-type (case action ((vm-image) qcow2-image-type) ((docker-image) docker-image-type) + ((docker-layered-image) + docker-layered-image-type) (else image-type))) (image-size (assoc-ref opts 'image-size)) (volatile? diff --git a/tests/pack.scm b/tests/pack.scm index ce5a2f8a53..432ab1b2ea 100644 --- a/tests/pack.scm +++ b/tests/pack.scm @@ -2,6 +2,7 @@ ;;; Copyright © 2017, 2018, 2019, 2020, 2021 Ludovic Courtès ;;; Copyright © 2018 Ricardo Wurmus ;;; Copyright © 2021, 2023 Maxim Cournoyer +;;; Copyright © 2023 Oleg Pykhalov ;;; ;;; This file is part of GNU Guix. ;;; @@ -29,6 +30,7 @@ (define-module (test-pack) #:use-module (guix gexp) #:use-module (guix modules) #:use-module (guix utils) + #:use-module ((guix build utils) #:select (%store-directory)) #:use-module (gnu packages) #:use-module ((gnu packages base) #:select (glibc-utf8-locales)) #:use-module (gnu packages bootstrap) @@ -250,6 +252,52 @@ (define rpm-for-tests (mkdir #$output))))))) (built-derivations (list check)))) + (unless store (test-skip 1)) + (test-assertm "docker-layered-image + localstatedir" store + (mlet* %store-monad + ((guile (set-guile-for-build (default-guile))) + (profile -> (profile + (content (packages->manifest (list %bootstrap-guile))) + (hooks '()) + (locales? #f))) + (tarball (docker-layered-image "docker-pack" profile + #:symlinks '(("/bin/Guile" -> "bin/guile")) + #:localstatedir? #t)) + (check (gexp->derivation "check-tarball" + (with-imported-modules '((guix build utils)) + #~(begin + (use-modules (guix build utils) + (ice-9 match)) + + (define bin + (string-append "." #$profile "/bin")) + + (define store + (string-append "." #$(%store-directory))) + + (setenv "PATH" (string-append #$%tar-bootstrap "/bin")) + (mkdir "base") + (with-directory-excursion "base" + (invoke "tar" "xvf" #$tarball)) + + (match (find-files "base" "layer.tar") + ((layers ...) + (for-each (lambda (layer) + (invoke "tar" "xvf" layer) + (invoke "chmod" "--recursive" "u+w" store)) + layers))) + + (when + (and (file-exists? (string-append bin "/guile")) + (file-exists? "var/guix/db/db.sqlite") + (file-is-directory? "tmp") + (string=? (string-append #$%bootstrap-guile "/bin") + (pk 'binlink (readlink bin))) + (string=? (string-append #$profile "/bin/guile") + (pk 'guilelink (readlink "bin/Guile")))) + (mkdir #$output))))))) + (built-derivations (list check)))) + (unless store (test-skip 1)) (test-assertm "squashfs-image + localstatedir" store (mlet* %store-monad -- 2.38.0 From debbugs-submit-bounces@debbugs.gnu.org Wed May 31 04:48:44 2023 Received: (at 62153) by debbugs.gnu.org; 31 May 2023 08:48:44 +0000 Received: from localhost ([127.0.0.1]:34850 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1q4HVb-0007OO-RH for submit@debbugs.gnu.org; Wed, 31 May 2023 04:48:44 -0400 Received: from mail-lf1-f45.google.com ([209.85.167.45]:38124) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1q4HVZ-0007O3-1h for 62153@debbugs.gnu.org; Wed, 31 May 2023 04:48:41 -0400 Received: by mail-lf1-f45.google.com with SMTP id 2adb3069b0e04-4f3ba67864fso993747e87.0 for <62153@debbugs.gnu.org>; Wed, 31 May 2023 01:48:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1685522915; x=1688114915; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Wz5S2n8eVP/ZYokf7n/jlwV46b8g4EfBbM/AUkqAmok=; b=qKoVBrUBHJPyLfQH7IW96kFGKRD2e/3UU2PgMWgTBSeh4SgquJw+EFLo5fr34pjjRy Yq8Wq34mmhAGnF/hoVtYm2r9YdDsqpCbcDTrYw2F2aO+GqpNThyp0s2DYKHAW1gofH73 MAZTXgJx2JZgvRj64h3orEXf+AdH6bTVl9VobPIDJ37VDkjgHWqgxXsqTgJy10ehBrZD Dl+Inrsa8FAjuIrBDe0S3jYflfIrA4cdkX75RdL4lzdlV8Pk6FRAM7WMaZ8E7QQ2oSno a5hzbE9V5l7iRgdenyFZw/PYWgJFeCOk/Ut7ZG6isU8e3NylaDoMP1lz8s+dqMVkSqcn eK8Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685522915; x=1688114915; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Wz5S2n8eVP/ZYokf7n/jlwV46b8g4EfBbM/AUkqAmok=; b=ep8CX72MQFI+xXtdVE3ei2TZG0JkOr6KuyPfgzxF2K3K80sBNHOyhvDktaugVCvA9D JIInz5G2MUKGt2EB5EtaeuoL7qb0gxHu7U4nrX7dVHIVA8fn6OOBTDXQ2XkTifZQE+Gc 3XaccvpIR6z+nf7oW97Kdc/dO2/BvvwbjjtAmcFqXvaCEaVDfSYDbv0dnOkUiUrB6zLF Ide5Uk9oVK87i1PN4R4bU5x0e54pivFxCcN8aknSEHEa2tZhjCJ7TSVTkxOy8CIL7Qqa ksE4W97rHnOPI5guvCxI+DRRQKFR7N9fvLcc5TOQTnFcRcKMhT9Lw3QCMNcQp9pDUFSo Vjag== X-Gm-Message-State: AC+VfDx3BHl2Tug6+tQrdpNY4HlEYytoS6s4JLIJKCMFoQxjW2xF2t0A Eu2RQ7Py23EiRzrloT3kNRMQHeMiRZw= X-Google-Smtp-Source: ACHHUZ66a6BNt5cJwzmfCLbhKl/rGvd6MHCAXuwSevJqtBC4IwOLjFSwf+ckI+WhFN/iAu//WG7MJA== X-Received: by 2002:a19:f712:0:b0:4f3:af9f:3d14 with SMTP id z18-20020a19f712000000b004f3af9f3d14mr536825lfe.5.1685522914884; Wed, 31 May 2023 01:48:34 -0700 (PDT) Received: from guixsd.wugi.info ([88.201.161.72]) by smtp.gmail.com with ESMTPSA id m9-20020ac24ac9000000b004f3886a63d1sm620295lfp.298.2023.05.31.01.48.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 May 2023 01:48:34 -0700 (PDT) From: Oleg Pykhalov To: 62153@debbugs.gnu.org Subject: [PATCH] news: Add entry for the new 'docker-layered' distribution format. Date: Wed, 31 May 2023 11:47:54 +0300 Message-Id: X-Mailer: git-send-email 2.38.0 In-Reply-To: References: MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 62153 Cc: Oleg Pykhalov X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) * etc/news.scm: Add entry. --- etc/news.scm | 58 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 58 insertions(+) diff --git a/etc/news.scm b/etc/news.scm index 314f0ab352..cb2dc34876 100644 --- a/etc/news.scm +++ b/etc/news.scm @@ -18,6 +18,7 @@ ;; Copyright © 2021 Andrew Tropin ;; Copyright © 2021, 2023 Jonathan Brielmaier ;; Copyright © 2022 Thiago Jung Bauermann +;; Copyright © 2023 Oleg Pykhalov ;; ;; Copying and distribution of this file, with or without modification, are ;; permitted in any medium without royalty provided the copyright notice and @@ -26,6 +27,63 @@ (channel-news (version 0) + (entry (commit "dd6c7c816bcb414682e1006d7e83b45e8ac6c575") + (title + (de "Neues Format @samp{docker-layered} für den Befehl @command{guix pack}") + (en "New @samp{docker-layered} format for the @command{guix pack} command") + (ru "Новый @samp{docker-layered} формат для @command{guix pack} команды")) + (body + (de "Sie können jetzt auch mehrschichtige Docker-Abbilder mit dem Befehl +@command{guix pack --format=docker-layered} erzeugen. Damit bekommen Sie ein +Docker-Abbild, bei dem Store-Pfade auf getrennten Schichten („Layer“) +untergebracht sind, die sich mehrere Abbilder teilen können. Das Abbild wird +im Store als gzip-komprimierter Tarball erzeugt. Hier ist ein einfaches +Beispiel, wo ein mehrschichtiges Docker-Abbild für das Paket @code{hello} +angelegt wird: + +@example +guix pack --format=docker-layered --symlink=/usr/bin/hello=bin/hello hello +@end example + +@command{guix system image} kann jetzt geschichtete Docker-Abbilder erzeugen, +indem Sie @code{docker-layered} an die Befehlszeilenoption @option{--image-type} +übergeben. + +Siehe @command{info \"(guix.de) Aufruf von guix pack\"} und +@command{info \"(guix.de) Systemabbilder\"} für weitere Informationen.") + (en "Docker layered images can now be produced via the @command{guix +pack --format=docker-layered} command, providing a Docker image with many of +the store paths being on their own layer to improve sharing between images. +The image is realized into the GNU store as a gzipped tarball. Here is a +simple example that generates a layered Docker image for the @code{hello} +package: + +@example +guix pack --format=docker-layered --symlink=/usr/bin/hello=bin/hello hello +@end example + +The @command{guix system image} can now produce layered Docker image by passing +@code{docker-layered} to @option{--image-type} option. + +See @command{info \"(guix) Invoking guix pack\"} and +@command{info \"(guix) System Images\"} for more information.") + (ru "Появилась команда создания многослойных Docker образов с помощью +@command{guix pack --format=docker-layered}, которая соберет Docker образ с +путями в store расположенными на отдельных слоях, ускоряя таким образом +передачу образов. Образ будет создан в GNU store в качестве gzipped tarball. + +Пример создания Docker layered образ с @code{hello} пакетом: +@example +guix pack --format=docker-layered --symlink=/usr/bin/hello=bin/hello hello +@end example + +@command{guix system image} теперь может создавать layered Docker образ путем +указания в опции @option{--image-type} параметра @code{docker-layered}. + +Смотрите @command{info \"(guix) Invoking guix pack\"} и +@command{info \"(guix) System Images\"} для получения более детальных +сведений."))) + (entry (commit "ba5da5125a81307500982517e2f458d57b024668") (title (en "New @code{arguments} rule for @command{guix style}") -- 2.38.0 From debbugs-submit-bounces@debbugs.gnu.org Wed May 31 08:54:04 2023 Received: (at 62153) by debbugs.gnu.org; 31 May 2023 12:54:05 +0000 Received: from localhost ([127.0.0.1]:35039 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1q4LL2-0007xB-OS for submit@debbugs.gnu.org; Wed, 31 May 2023 08:54:04 -0400 Received: from mail-yb1-f182.google.com ([209.85.219.182]:45092) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1q4LL0-0007wb-Ve for 62153@debbugs.gnu.org; Wed, 31 May 2023 08:54:03 -0400 Received: by mail-yb1-f182.google.com with SMTP id 3f1490d57ef6-ba8151a744fso7697178276.2 for <62153@debbugs.gnu.org>; Wed, 31 May 2023 05:54:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=greghogan-com.20221208.gappssmtp.com; s=20221208; t=1685537637; x=1688129637; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=L3yBMo2iEbnzXiZoWSdoOPiexPz1Keh8ntCtksvC11Q=; b=3Ih+6SHNdmYx+s5nQhoATGmvU/J3vXBzrHsZ65XStKYXcW8enmz6b/87vGmEiPDvpC Pu0NTYdSKx6y/DAz3sk93cDgkGgaJ+79dBpJZSDdnugyhEgXUWAB9fjVP/t5yGqRqkMe bYMc/+CjVvBGAlsjO0djl6qHhvXceb0D6FAi2klHIlGgzaPjMyeLbcPUDaDBJ8VE5Ris 0U+Eyh6woslPPuh7xmPrgQwU4/5Wb/w6Vb0mJkRuzefZ9IaNC/Hw5hFLUAzcdfhtHCL1 2Ya1l2JqB2qpfVAddcRLoRVbl8JKwlnix+F7CfrHEVQMqtj1Am8mtcXQQm5atRhX5i4w AUtQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685537637; x=1688129637; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=L3yBMo2iEbnzXiZoWSdoOPiexPz1Keh8ntCtksvC11Q=; b=fBegkZIXtOGfwrARq36XjBb5PGbaRVuvGfCZwIUZO5jDBZuV0opyChd51eccSUHl9B 0vN7C36wdNjw55Sz/vwnsh/oBYk1xE2sH2T3ozRpd7y2qxyCo4kHoQkCGl7HvoEDtpTY DkUYM13ycYPqCnjZ+AoT+QYqL6sVuZq1X+NJcIYzt6v6/iwk3SJjh/73eGJ0vrpUhuR0 96D/320+Jh9/HtJVPBy6YOQ2ht6TUfdmCE85hXj+77rGoLtTBsYdL5cFTCAyG5/MX6Tm LObv6fuo0iLn7DCHpNh4IwSwur/WLAs1oYqjgLDaQ0Y+tFaNnr+PwHrHceI4+P/h2WVb DCeQ== X-Gm-Message-State: AC+VfDxRHspKb6i2rKbcCcHdhuZz8w04dIWTOwN40VbcF7Az6OCUhsqN V8iT02EPNIDAcaR3e4ZTc3qWKZdnOe8dJKUVd9RTqQ== X-Google-Smtp-Source: ACHHUZ5tzuUZM4RNryHxmYWRHMHWyMAiSJSCR1nfccsQqjOC7yubZF/nkZNAWIEUMgdmaHE6wGViWbLO5MtbPICYM/8= X-Received: by 2002:a25:4046:0:b0:bac:81a2:93ab with SMTP id n67-20020a254046000000b00bac81a293abmr7294951yba.50.1685537637153; Wed, 31 May 2023 05:53:57 -0700 (PDT) MIME-Version: 1.0 References: <20230313003012.14325-1-go.wigust@gmail.com> In-Reply-To: From: Greg Hogan Date: Wed, 31 May 2023 08:53:46 -0400 Message-ID: Subject: Re: [bug#62153] [PATCH] Add Docker layered image for pack and system (v3) To: Oleg Pykhalov Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 62153 Cc: 62153@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) On Wed, May 31, 2023 at 4:46=E2=80=AFAM Oleg Pykhalov = wrote: > > Hi, Guix. > > These patches series is rebased on origin/master. Also, the Python script= is > replaced in favour of calls to GNU Tar and GNU Gzip programs. Passed tes= ts: > make check TESTS=3D"tests/pack.scm" > make check-system TESTS=3D"docker-system" > make check-system TESTS=3D"docker-layered-system" > > > Oleg Pykhalov (2): > guix: docker: Build layered image. > news: Add entry for the new 'docker-layered' distribution format. Why not use layered images for all docker packs? From debbugs-submit-bounces@debbugs.gnu.org Wed May 31 09:15:10 2023 Received: (at 62153) by debbugs.gnu.org; 31 May 2023 13:15:10 +0000 Received: from localhost ([127.0.0.1]:35081 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1q4LfS-00004q-6Z for submit@debbugs.gnu.org; Wed, 31 May 2023 09:15:10 -0400 Received: from mail-lf1-f54.google.com ([209.85.167.54]:40141) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1q4LfM-0008Va-Sn for 62153@debbugs.gnu.org; Wed, 31 May 2023 09:15:08 -0400 Received: by mail-lf1-f54.google.com with SMTP id 2adb3069b0e04-4f3aa8327d9so1039142e87.0 for <62153@debbugs.gnu.org>; Wed, 31 May 2023 06:15:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1685538899; x=1688130899; h=mime-version:user-agent:message-id:in-reply-to:date:references :subject:cc:to:from:from:to:cc:subject:date:message-id:reply-to; bh=yXmSVgnm0PiSjeRYbkjHAgIp3ikdpXj1rvjSpPEqYCE=; b=m1Omk5wcAPYjdJxWGLn9g8mv0HTfsHFbp6B6QC+HSTuJX00ZhpZJPxmyi2uYQn25Ll oG/p8dgPTLBCZYpdBjzMzg72juITAubnkNPaTIak2OLHWHQB+jl7gFEnk8e4b8tTd3ib 3OXQicI3DheqDq40VmlUNC4+FQ7WYssPaRF/zojpLqYoo8o6psTPfDSqpEaIbEnjQ7jM SaElrS+qPVziZwxWD4SFeX1XzKOsAddk+OzsphSOKAcDXeFoR1BLrNpm7erbXNNXvzt9 ph2g2uBcNn5P1BwEYGsCuE1TFLIDYjEIbMRpVp/8v8mKcVABSUK7pMfRf2N01nZfLfnm 9YtA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685538899; x=1688130899; h=mime-version:user-agent:message-id:in-reply-to:date:references :subject:cc:to:from:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=yXmSVgnm0PiSjeRYbkjHAgIp3ikdpXj1rvjSpPEqYCE=; b=MnmpPoxyzsHG8tkufMmi5tYBuEJSJVlDPBIEjUK3y0fEB0I0tePX751Zv7RNEIXbPj L90atop9jxbjsRy77n5y/RDBJ78r3hcRQeg6u6wXV8B+AiRjaTLNdD0gJqhmj3K4ZI6M 1AKGlPNr2lRMeH8AGs2RS3DucQNE92axY9PENfarxfUfM9I/FtcoChH/QozOY/P0zrq/ CaS+aHlz4k/y6+kKuu/sGEIZ72CMoBMtdU0RVEEeu5zVMoP7cmC53mpvziyPP9Lzl77X rU+HN43keEzYlvngy4Gtl9sYBnDZJ1y/Uds1gF0XYkGl2Q78iKqAUPy0iyVfHJ/wf5g0 ov5g== X-Gm-Message-State: AC+VfDwsIH1MI+S+oT8utIWEzZaMai13msV+BYHutD71f0lJfMJoI3Ry ETzuUfl19KXjVTrEhyqVhvw= X-Google-Smtp-Source: ACHHUZ7aXF8IhHR8JoVz2WUVRZ8cSKfSpGLvR9oIl+0H3Zeac2jI1SnpbQDvSGS9GyqIH+1xd7AqaQ== X-Received: by 2002:ac2:514c:0:b0:4f1:3d76:d1a2 with SMTP id q12-20020ac2514c000000b004f13d76d1a2mr1839468lfd.0.1685538898440; Wed, 31 May 2023 06:14:58 -0700 (PDT) Received: from localhost ([88.201.161.72]) by smtp.gmail.com with ESMTPSA id a21-20020a19f815000000b004f3af454556sm716479lff.246.2023.05.31.06.14.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 May 2023 06:14:57 -0700 (PDT) From: Oleg Pykhalov To: Greg Hogan Subject: Re: [bug#62153] [PATCH] Add Docker layered image for pack and system (v3) References: <20230313003012.14325-1-go.wigust@gmail.com> Date: Wed, 31 May 2023 16:14:57 +0300 In-Reply-To: (Greg Hogan's message of "Wed, 31 May 2023 08:53:46 -0400") Message-ID: <87cz2gd4mm.fsf@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (gnu/linux) MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 62153 Cc: 62153@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Greg Hogan writes: > On Wed, May 31, 2023 at 4:46=E2=80=AFAM Oleg Pykhalov wrote: >> >> Hi, Guix. >> >> These patches series is rebased on origin/master. Also, the Python scrip= t is >> replaced in favour of calls to GNU Tar and GNU Gzip programs. Passed te= sts: >> make check TESTS=3D"tests/pack.scm" >> make check-system TESTS=3D"docker-system" >> make check-system TESTS=3D"docker-layered-system" >> >> >> Oleg Pykhalov (2): >> guix: docker: Build layered image. >> news: Add entry for the new 'docker-layered' distribution format. > > Why not use layered images for all docker packs? Do you mean use layered images by default without ability to build all in a single layer? Current layered implementation is slow to build because it needs to calculate a size of each layer, pack, and compress. So if user wants a faster build, a non-layered image is still an option. Regards, Oleg. --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQJIBAEBCgAyFiEEcjhxI46s62NFSFhXFn+OpQAa+pwFAmR3SFEUHGdvLndpZ3Vz dEBnbWFpbC5jb20ACgkQFn+OpQAa+pzUmA/9Hqd7U8yB00GA/wF1F7TU/3AcpaFP uY6LHfUB0ZePuu5r3cQcx2LlCISJ6KLrXOk+d6vTDc+WK4TpTQrH5V5Gt8WAU3L2 Nt58WtDiZnZwX6uuO1Dxk1efF4eJRIIzLqtXluDzJ+jpCj2Q9M43Js8EODRk21rC w+j+e5Xq2xF3R7aBaGRUN8euFI6nyfZgZC2kiuOAc9xBCt6hMmOPEm4mwaqIUABG BSa4Nr/Qv0W/GbUEhWgJOWI4NGljbanWLuRlC5IXd4tHEAU2p7lEGg4uWm1Brm+r 2mvHfu0y8tLwnb8F9SjscjBG3OXieM3YjmYeTeOSiNRY5tqnOTo9o4Kq7wmeezQU 02FYA3+U7vvIJ7WCWbEowZgv4ZEW3UCsDiNdt0+4GR72J10jR/MTdFVOhxV0Heo6 4alOxBm0qoLJh9+3dYwY6wEYlWfAwIPXaLeGX4F4GQ0YdPsMFMCCeSdkz1vN4A6l 1q8A29IAuRQcLxv9G3T9+bO4tsYpN7jtB6c8M/4+na03Tc/ugAOWIY+Ne0wBBAAA k1sqv69JljPC9rE1gaxwUU7FTkY8/pbhGzH0MFZG/C7qUYbz4L5Z7ANzlAnhZGWl isVFrq2VkTC0XZjxISaQEUsRLcAP82HrTVDfV7KIsK3iuczQVWVPJXgu/rgJ9rgi rMJcHluPJHJ5IUc= =mfr+ -----END PGP SIGNATURE----- --=-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Fri Jun 02 13:03:00 2023 Received: (at 62153) by debbugs.gnu.org; 2 Jun 2023 17:03:00 +0000 Received: from localhost ([127.0.0.1]:40781 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1q58B2-0006v2-1J for submit@debbugs.gnu.org; Fri, 02 Jun 2023 13:03:00 -0400 Received: from mail-yw1-f172.google.com ([209.85.128.172]:48203) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1q58Ax-0006ui-Jn for 62153@debbugs.gnu.org; Fri, 02 Jun 2023 13:02:57 -0400 Received: by mail-yw1-f172.google.com with SMTP id 00721157ae682-565cdb77b01so21648057b3.0 for <62153@debbugs.gnu.org>; Fri, 02 Jun 2023 10:02:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=greghogan-com.20221208.gappssmtp.com; s=20221208; t=1685725370; x=1688317370; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=B6D2CgKVv6lon54u3CVC6QDfN54dU1/zm84lryKcrqI=; b=hkt+RHiSRhr/mZtNRr54VMU8cyc7gWi6i3Hf+lpwJWKvR7EQKKo4NQAgXvtvrLVI0T FhKsCbyCtY1SagJ/cbx4ihYKQZGFNHnL3MFH9Yj1H37G6efGcmbVhyhpMy51ORj0yhda bs82/TKUmOFPHbghxaupBQ2Sdyfu4Z18w7AncgYJIShzFKwuTtFEGEcKWytFCrlF1ErK yuwoJR8gLMg8Ozs1aMd9s8R8l2u8ff5wA0f4rfZ5cikKL1fmW/oj1LzFCbrdAVUrc+fW AQ+zSpzgrHoRCSpW/JJbKKUU09XsCQycTfzSSY3uDeeOvYLAjtx2O7q1zlgBFJPSoFTq BpQw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685725370; x=1688317370; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=B6D2CgKVv6lon54u3CVC6QDfN54dU1/zm84lryKcrqI=; b=feKe/lOfYMnHTHXpTgHrtvPj6P6sHmMXzqMhbXlKPQu4kwxR9ae+obs2jWGtiwwacE AS4Rv28dl9i348kfJGSb6CF+Tc4ZODgoUOJNI57NXndwA8Us2ZzHKjlCJ8xe6zPngVDn 3Hpx4MyRfRjtMaLqXVi371GzbVYMs2GXT1raPZmPpAcShVAmANjiL0qSZ+Bg2c+O+cJi g+8XWL/vU/h5STqXAVvqCMA86sUzu6TnlZ4C7M2AJchB1oAgU2w3RFOj9V/gE+CMRXGa rE2vz3gA0s/QU+7WDlg/DT3qFJNo9V7yEQLP3lnQoVqcxSPqifVv8XxFX3cr5yDC9fV+ ZkZQ== X-Gm-Message-State: AC+VfDwnjIfADXpwnK5nMDByVrv81VQtRD0KGZbKN98e0EC8eXqu8j0I gy6IyWqwNnws2Ec8V3HJd9gkNKKg8uDn3ET7NsYSew== X-Google-Smtp-Source: ACHHUZ5eMHUqOuEFZguS7R6LwdVQXU21YDKLidQmrxRRaK0tozZHhT6lVyd7UBksq4X/1ZvW9+GdBBSruWNkLeWoRKM= X-Received: by 2002:a25:d5:0:b0:bac:748a:5759 with SMTP id 204-20020a2500d5000000b00bac748a5759mr3246941yba.37.1685725369815; Fri, 02 Jun 2023 10:02:49 -0700 (PDT) MIME-Version: 1.0 References: <20230313003012.14325-1-go.wigust@gmail.com> <87cz2gd4mm.fsf@gmail.com> In-Reply-To: <87cz2gd4mm.fsf@gmail.com> From: Greg Hogan Date: Fri, 2 Jun 2023 13:02:38 -0400 Message-ID: Subject: Re: [bug#62153] [PATCH] Add Docker layered image for pack and system (v3) To: Oleg Pykhalov Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 62153 Cc: 62153@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) On Wed, May 31, 2023 at 9:14=E2=80=AFAM Oleg Pykhalov = wrote: [...] > Do you mean use layered images by default without ability to build all > in a single layer? Current layered implementation is slow to build > because it needs to calculate a size of each layer, pack, and compress. > So if user wants a faster build, a non-layered image is still an option. > > Regards, > Oleg. I am trying out your patch, and wanted to benchmark the runtime difference between docker and docker-layered packs, but the latter looks to be failing with any compression other than the default gzip. In particular, I was looking to disable compression with '--compression=3Dnone'. From debbugs-submit-bounces@debbugs.gnu.org Sat Jun 03 15:10:28 2023 Received: (at 62153) by debbugs.gnu.org; 3 Jun 2023 19:10:28 +0000 Received: from localhost ([127.0.0.1]:44315 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1q5Wdw-0002sP-BI for submit@debbugs.gnu.org; Sat, 03 Jun 2023 15:10:28 -0400 Received: from mail-lj1-f170.google.com ([209.85.208.170]:38064) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1q5Wdr-0002s3-Fb for 62153@debbugs.gnu.org; Sat, 03 Jun 2023 15:10:27 -0400 Received: by mail-lj1-f170.google.com with SMTP id 38308e7fff4ca-2b1b1dd208dso3267291fa.0 for <62153@debbugs.gnu.org>; Sat, 03 Jun 2023 12:10:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1685819417; x=1688411417; h=mime-version:user-agent:message-id:in-reply-to:date:references :subject:cc:to:from:from:to:cc:subject:date:message-id:reply-to; bh=2dz+7HlpM0CpG71xUvwWd7jmSIkaK4R84zQ2Wkw9A80=; b=UR8qPywuD0uv0EXU+JQnO0ehm1MsfQDZ9pOdqOtUn/qFH1f5jS7vhCbcXnzICl6qiV nbQ3a8lJcEhszkoU+IChn0uOt3Wx71/f7ON+ZJRODAAFkRgcvrur2uoHd6pwIRrk7Gd2 CCRXhdm7YEytEdoltJL1djvhvVzu03/33g+bUJLixhAXFd0UA6UZyq1svCrjq4J0yKOC TEp01iBUkY7dcO/5re4jM8yYQTAKnkGFZfk9hU7DEKdYLrP7YL7rHYLrJJOC3gquQuUc 5PFErMHk8QalG5O/iO72aRRzib9Ha7e4bZTePqkDUVjRnQgToCQVxbOAH7UOGJFwe2Es Z3+w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685819417; x=1688411417; h=mime-version:user-agent:message-id:in-reply-to:date:references :subject:cc:to:from:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=2dz+7HlpM0CpG71xUvwWd7jmSIkaK4R84zQ2Wkw9A80=; b=HRe84FqWgQGHYrytVjUPat+T40/rD3iq0vQtUB19gtYc4owW2R2cV2fAroZ0ZSFCHu iO9V2YYDEAXxFKdEGmRxRjEVm7PiYe2+XSjXalDWGnCq6OZTIA4lN+rphgeb1LsXY9zF yeVnxlOq9txW07BY73VrAUHgYL+e1auFimhE8jICQm76L/Q5H1rJ26qpcZAQwwlaazX+ o3BHy6TsWtnHYr5oec3kI6V/NlsRj1wwmm9fvR8Kmuf1HmbZd1Nzgxn/2VLKlHb1Tw15 b4ob1I6niGVUXCunK7BYSeGgVZriTTWVibytq1RCDvRKfs+FmZNhi6M3tco3EAadB1Bk 8h4Q== X-Gm-Message-State: AC+VfDyESb7Q1Ja+prVFMLB5Q7PJzHtxYnZ94TGbYD+iREZFeLr6nuUd syGGzjEVMx8PtPP27u5RPfTbvravKtk= X-Google-Smtp-Source: ACHHUZ4qeOdboX6LlMaql8rK2BYOAtGIOk2AZ+Hr2XHBdm3jEkEbmEqoYQYy/rVLG6H/oe0wPXgQDA== X-Received: by 2002:ac2:43b1:0:b0:4f6:19bf:1a32 with SMTP id t17-20020ac243b1000000b004f619bf1a32mr692111lfl.4.1685819417109; Sat, 03 Jun 2023 12:10:17 -0700 (PDT) Received: from localhost ([88.201.161.72]) by smtp.gmail.com with ESMTPSA id c21-20020ac244b5000000b004eb2ac90f36sm553957lfm.207.2023.06.03.12.10.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 03 Jun 2023 12:10:16 -0700 (PDT) From: Oleg Pykhalov To: Greg Hogan Subject: Re: bug#62153: [PATCH 0/2] Add Docker layered image for pack and system References: <20230313003012.14325-1-go.wigust@gmail.com> <87cz2gd4mm.fsf@gmail.com> Date: Sat, 03 Jun 2023 22:10:15 +0300 In-Reply-To: (Greg Hogan's message of "Fri, 2 Jun 2023 13:02:38 -0400") Message-ID: <87y1l0icq0.fsf_-_@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (gnu/linux) MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 62153 Cc: 62153@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Greg Hogan writes: > On Wed, May 31, 2023 at 9:14=E2=80=AFAM Oleg Pykhalov wrote: > [...] >> Do you mean use layered images by default without ability to build all >> in a single layer? Current layered implementation is slow to build >> because it needs to calculate a size of each layer, pack, and compress. >> So if user wants a faster build, a non-layered image is still an option. >> >> Regards, >> Oleg. > > I am trying out your patch, and wanted to benchmark the runtime > difference between docker and docker-layered packs, but the latter > looks to be failing with any compression other than the default gzip. > In particular, I was looking to disable compression with > '--compression=3Dnone'. I'll send a fixed v4 revision for '--compression=3Dnone'. Unfortunately, because we cannot append to an existing compressed tarball: tar: Cannot update compressed archives Try 'tar --help' or 'tar --usage' for more information. adding more compression types requires to write a handler for every compressor separately in guix/docker.scm file: =2D-8<---------------cut here---------------start------------->8--- (if layered-image? (begin (invoke "tar" "-rf" "image.tar" "config.json") (if compressor (begin (apply invoke `(,@compressor "image.tar")) (copy-file "image.tar.gz" image)) (copy-file "image.tar" image))) =2D-8<---------------cut here---------------end--------------->8--- I would like to vote that addional compressors could be added later if needed. Regards, Oleg. --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQJIBAEBCgAyFiEEcjhxI46s62NFSFhXFn+OpQAa+pwFAmR7kBgUHGdvLndpZ3Vz dEBnbWFpbC5jb20ACgkQFn+OpQAa+pxoKA//fUia6BxwXHIQNdFmQueeIVN7RUvr gx19n9Qpe3rmkqRwdXqqk9V81daqSYtEwm9/vpvo2HO85yliUxkMUOiHRQLJxOPQ X0Vx1v871DyXkdg4D9Jh1/GE6sjqcT8Jka5u8pFB5LAPVto12tqy+0+lMxP4AcqX X6GxWuEClj6yB+nqtE/00xul6qDF1YaHAeOrMJjpYf4y1dwBYw9WFQwJJvVuXxdE WukW6F9qTqj9qXrG0+EkamWQjWcmKrSkv7+aTskIK/xdEuXor2OSOC2o6Tff7K7p GeePvEdDIGetiV5bNVU4EwewvJH/oF/FFZcPdvBx0DCUJ8AeVMEzwqnWExTW/UXe pTqQH3dwTNhnh1tDhAAEKXhcBuOntmiCp5knArmCCirpzyW+c7rZ/F8JVC5vZJTf PS3x9O+R3ZkueoimoQzSd/JGPSH1eh2O8SMNy42QB1w4j3SLEkv0ShsswCbkdMhk OXy/wjqo9GJRRMRNTYAxeu7OLDOwM5RrhBX3s2g0Nu+6YEa3ISDU7s4qEBbQHNFW lD0dplzXNZudhLab/xTP/ptgqsh7SoaYszsRitz+noT35DuaJt7wwm7Qp+rn55ve PqMd+P94YCrzq8RslsUfpEE+VBJKxJbFnNTZdkaDqAI/HCwWSSe9gzag3zsYpQDx 8blQqUJbYRIXBiE= =xQTj -----END PGP SIGNATURE----- --=-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Sat Jun 03 15:15:47 2023 Received: (at 62153) by debbugs.gnu.org; 3 Jun 2023 19:15:47 +0000 Received: from localhost ([127.0.0.1]:44321 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1q5Wj4-00030u-4F for submit@debbugs.gnu.org; Sat, 03 Jun 2023 15:15:47 -0400 Received: from mail-lj1-f171.google.com ([209.85.208.171]:38383) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1q5Wj0-00030d-60 for 62153@debbugs.gnu.org; Sat, 03 Jun 2023 15:15:44 -0400 Received: by mail-lj1-f171.google.com with SMTP id 38308e7fff4ca-2b1b5d1bbf1so2699631fa.1 for <62153@debbugs.gnu.org>; Sat, 03 Jun 2023 12:15:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1685819736; x=1688411736; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=8bCoMswzXLyVMiWVi6zut9aKRYDecWs52QFLj3N2Bus=; b=rh/VR7q442wJ77UXPvwD45OHcdNoxrgwZaL1oZdRXK1foW1VO3+kZYE1wDtrZO3KL3 IomygK5eW5Pr9HQEg/ywAMu8qKIefv/7e7PDPVdTq7xg1oTP95A6sOSsRNCfB+ZxH3Rr CJqHrKNCV3JikDYFp4TBgjlmwaERZjlhTI4Vc6A6OQW58LB17gUzjFKm37/OWfvMCebt gXR34eSw+x9ksSek6u9eIwaO15W3eIelyPjpgA6jbFueRhcojFfE3R0RJBHypa4ztvFN BiGsk2K8DCRhz0YSzUyJhbxkluy8wyuXN6g2lqlazKTsMHPA5Q9VQ5S92a2AduMBFo69 Ss7g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685819736; x=1688411736; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=8bCoMswzXLyVMiWVi6zut9aKRYDecWs52QFLj3N2Bus=; b=cvXb1Bre1/BM1Dka2QMLIf5kvlXIg1cIrf6Vg4w4af8qqrBsoJKQHWdETup+dOvtBT N4kDHmaSWPJYV9wi0ID2HmY8Fb3EfZkoyABk0m/TmBAz+TT7Yf4/L3mSwE+EAZ1QV5xn wPK0FhD0HBDrfu7jAHG8XD3urA1w3dpYPlsC39xsGbm7KRPhVLCMguYPAsv/0vAclzNZ rIZxN63a159wxJatCFBEFsZBvfRZlXXNg/uPB7p88/ST8dXP0GYuoFsevLEPGJDTshnV /7OiXm5LXRyFAUF5ZolI/8L+apHL/71Ad180LgC/eOyNDEHHF5NhSlY8qfLgcoI4Wkb1 gCSg== X-Gm-Message-State: AC+VfDyeuxlzfoiPl3FWT1k/TmNfLY035SKUv7KRCizdMhm2vuUXr7/l i1oouBRZGi/iMlzGFjfRLM32ik8WrP0= X-Google-Smtp-Source: ACHHUZ7CYw6wInr1JsKnj0Gd6tGurKikqCqVrXHNHKfPbpUlkNwADbTDww7qQS8pEBlboF7BQbG02Q== X-Received: by 2002:ac2:43b1:0:b0:4f6:19bf:1a32 with SMTP id t17-20020ac243b1000000b004f619bf1a32mr694368lfl.4.1685819735531; Sat, 03 Jun 2023 12:15:35 -0700 (PDT) Received: from guixsd.wugi.info ([88.201.161.72]) by smtp.gmail.com with ESMTPSA id u4-20020a056512040400b004f262997496sm555913lfk.76.2023.06.03.12.15.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 03 Jun 2023 12:15:35 -0700 (PDT) From: Oleg Pykhalov To: 62153@debbugs.gnu.org Subject: [PATCH v4 1/2] guix: docker: Build layered image. Date: Sat, 3 Jun 2023 22:14:59 +0300 Message-Id: <457c813653a44117e296deaa49e79fc701b90791.1685819700.git.go.wigust@gmail.com> X-Mailer: git-send-email 2.38.0 In-Reply-To: <87y1l0icq0.fsf_-_@gmail.com> References: <87y1l0icq0.fsf_-_@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 62153 Cc: Oleg Pykhalov , Greg Hogan X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) * doc/guix.texi (Invoking guix pack): Document docker-layered format. (image Reference): Same. (image-type Reference): Document docker-layered-image-type. * gnu/image.scm (validate-image-format)[docker-layered]: New image format. * gnu/system/image.scm (docker-layered-image, docker-layered-image-type): New variables. (system-docker-image)[layered-image?]: New argument. (system-docker-layered-image): New procedure. (image->root-file-system)[docker-layered]: New image format. * gnu/tests/docker.scm (%test-docker-layered-system): New test. * guix/docker.scm (%docker-image-max-layers): New variable. (build-docker-image)[stream-layered-image, root-system]: New arguments. * guix/scripts/pack.scm (stream-layered-image.py): New variable. (docker-image)[layered-image?]: New argument. (docker-layered-image): New procedure. (%formats)[docker-layered]: New format. (show-formats): Document this. * guix/scripts/system.scm (system-derivation-for-action)[docker-layered-image]: New action. (show-help): Document this. (actions)[docker-layered-image]: New action. (process-action): Add this. * tests/pack.scm: Add "docker-layered-image + localstatedir" test. --- doc/guix.texi | 18 +++- gnu/image.scm | 3 +- gnu/system/image.scm | 76 +++++++++++---- gnu/tests/docker.scm | 20 +++- guix/docker.scm | 208 +++++++++++++++++++++++++++++++--------- guix/scripts/pack.scm | 62 ++++++++++-- guix/scripts/system.scm | 11 ++- tests/pack.scm | 48 ++++++++++ 8 files changed, 369 insertions(+), 77 deletions(-) diff --git a/doc/guix.texi b/doc/guix.texi index 7f8d8d66e9..483be6ef16 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -56,7 +56,7 @@ Copyright @copyright{} 2017, 2018, 2019, 2020 Arun Isaac@* Copyright @copyright{} 2017 nee@* Copyright @copyright{} 2018 Rutger Helling@* -Copyright @copyright{} 2018, 2021 Oleg Pykhalov@* +Copyright @copyright{} 2018, 2021, 2023 Oleg Pykhalov@* Copyright @copyright{} 2018 Mike Gerwitz@* Copyright @copyright{} 2018 Pierre-Antoine Rouby@* Copyright @copyright{} 2018, 2019 Gábor Boskovits@* @@ -6984,9 +6984,15 @@ Invoking guix pack guix pack -f docker -S /bin=bin guile guile-readline @end example +or + +@example +guix pack -f docker-layered -S /bin=bin guile guile-readline +@end example + @noindent -The result is a tarball that can be passed to the @command{docker load} -command, followed by @code{docker run}: +The result is a tarball with image or layered image that can be passed +to the @command{docker load} command, followed by @code{docker run}: @example docker load < @var{file} @@ -44347,6 +44353,8 @@ image Reference @item @code{docker}, a Docker image. +@item @code{docker-layered}, a layered Docker image. + @item @code{iso9660}, an ISO-9660 image. @item @code{tarball}, a tar.gz image archive. @@ -44682,6 +44690,10 @@ image-type Reference Build an image based on the @code{docker-image} image. @end defvar +@defvar docker-layered-image-type +Build a layered image based on the @code{docker-layered-image} image. +@end defvar + @defvar raw-with-offset-image-type Build an MBR image with a single partition starting at a @code{1024KiB} offset. This is useful to leave some room to install a bootloader in diff --git a/gnu/image.scm b/gnu/image.scm index 523653dd77..8a6a0d8479 100644 --- a/gnu/image.scm +++ b/gnu/image.scm @@ -1,5 +1,6 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2020, 2022 Mathieu Othacehe +;;; Copyright © 2023 Oleg Pykhalov ;;; ;;; This file is part of GNU Guix. ;;; @@ -152,7 +153,7 @@ (define-syntax-rule (define-set-sanitizer name field set) ;; The supported image formats. (define-set-sanitizer validate-image-format format - (disk-image compressed-qcow2 docker iso9660 tarball wsl2)) + (disk-image compressed-qcow2 docker docker-layered iso9660 tarball wsl2)) ;; The supported partition table types. (define-set-sanitizer validate-partition-table-type partition-table-type diff --git a/gnu/system/image.scm b/gnu/system/image.scm index afef79185f..3a502f19ec 100644 --- a/gnu/system/image.scm +++ b/gnu/system/image.scm @@ -4,6 +4,7 @@ ;;; Copyright © 2022 Pavel Shlyak ;;; Copyright © 2022 Denis 'GNUtoo' Carikli ;;; Copyright © 2022 Alex Griffin +;;; Copyright © 2023 Oleg Pykhalov ;;; ;;; This file is part of GNU Guix. ;;; @@ -78,6 +79,7 @@ (define-module (gnu system image) efi-disk-image iso9660-image docker-image + docker-layered-image tarball-image wsl2-image raw-with-offset-disk-image @@ -89,6 +91,7 @@ (define-module (gnu system image) iso-image-type uncompressed-iso-image-type docker-image-type + docker-layered-image-type tarball-image-type wsl2-image-type raw-with-offset-image-type @@ -167,6 +170,10 @@ (define docker-image (image-without-os (format 'docker))) +(define docker-layered-image + (image-without-os + (format 'docker-layered))) + (define tarball-image (image-without-os (format 'tarball))) @@ -237,6 +244,11 @@ (define docker-image-type (name 'docker) (constructor (cut image-with-os docker-image <>)))) +(define docker-layered-image-type + (image-type + (name 'docker-layered) + (constructor (cut image-with-os docker-layered-image <>)))) + (define tarball-image-type (image-type (name 'tarball) @@ -633,9 +645,12 @@ (define (image-with-label base-image label) (define* (system-docker-image image #:key - (name "docker-image")) + (name "docker-image") + (archiver tar) + layered-image?) "Build a docker image for IMAGE. NAME is the base name to use for the -output file." +output file. If LAYERED-IMAGE? is true, the image will with many of the store +paths being on their own layer to improve sharing between images." (define boot-program ;; Program that runs the boot script of OS, which in turn starts shepherd. (program-file "boot-program" @@ -678,9 +693,11 @@ (define* (system-docker-image image (use-modules (guix docker) (guix build utils) (gnu build image) + (srfi srfi-1) (srfi srfi-19) (guix build store-copy) - (guix store database)) + (guix store database) + (ice-9 receive)) ;; Set the SQL schema location. (sql-schema #$schema) @@ -700,18 +717,31 @@ (define* (system-docker-image image #:register-closures? #$register-closures? #:deduplicate? #f #:system-directory #$os) - (build-docker-image - #$output - (cons* image-root - (map store-info-item - (call-with-input-file #$graph - read-reference-graph))) - #$os - #:entry-point '(#$boot-program #$os) - #:compressor '(#+(file-append gzip "/bin/gzip") "-9n") - #:creation-time (make-time time-utc 0 1) - #:system #$image-target - #:transformations `((,image-root -> "")))))))) + (when #$layered-image? + (setenv "PATH" + (string-join (list #+(file-append archiver "/bin") + #+(file-append coreutils "/bin") + #+(file-append gzip "/bin")) + ":"))) + (apply build-docker-image + (append (list #$output + (append (if #$layered-image? + '() + (list image-root)) + (map store-info-item + (call-with-input-file #$graph + read-reference-graph))) + #$os + #:entry-point '(#$boot-program #$os) + #:compressor + '(#+(file-append gzip "/bin/gzip") "-9n") + #:creation-time (make-time time-utc 0 1) + #:system #$image-target + #:transformations `((,image-root -> ""))) + (if #$layered-image? + (list #:root-system image-root + #:layered-image? #$layered-image?) + '())))))))) (computed-file name builder ;; Allow offloading so that this I/O-intensive process @@ -720,6 +750,18 @@ (define* (system-docker-image image #:options `(#:references-graphs ((,graph ,os)) #:substitutable? ,substitutable?)))) +(define* (system-docker-layered-image image + #:key + (name "docker-image") + (archiver tar) + (layered-image? #t)) + "Build a docker image for IMAGE. NAME is the base name to use for the +output file." + (system-docker-image image + #:name name + #:archiver archiver + #:layered-image? layered-image?)) + ;;; ;;; Tarball image. @@ -811,7 +853,7 @@ (define (image->root-file-system image) "Return the IMAGE root partition file-system type." (case (image-format image) ((iso9660) "iso9660") - ((docker tarball wsl2) "dummy") + ((docker docker-layered tarball wsl2) "dummy") (else (partition-file-system (find-root-partition image))))) @@ -948,6 +990,8 @@ (define* (system-image image) ("bootcfg" ,bootcfg)))) ((memq image-format '(docker)) (system-docker-image image*)) + ((memq image-format '(docker-layered)) + (system-docker-layered-image image*)) ((memq image-format '(tarball)) (system-tarball-image image*)) ((memq image-format '(wsl2)) diff --git a/gnu/tests/docker.scm b/gnu/tests/docker.scm index edc9804414..0cccc02ad2 100644 --- a/gnu/tests/docker.scm +++ b/gnu/tests/docker.scm @@ -1,6 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2019 Danny Milosavljevic ;;; Copyright © 2019-2023 Ludovic Courtès +;;; Copyright © 2023 Oleg Pykhalov ;;; ;;; This file is part of GNU Guix. ;;; @@ -43,7 +44,8 @@ (define-module (gnu tests docker) #:use-module (guix build-system trivial) #:use-module ((guix licenses) #:prefix license:) #:export (%test-docker - %test-docker-system)) + %test-docker-system + %test-docker-layered-system)) (define %docker-os (simple-operating-system @@ -316,3 +318,19 @@ (define %test-docker-system (locale-libcs (list glibc))) #:type docker-image-type))) run-docker-system-test))))) + +(define %test-docker-layered-system + (system-test + (name "docker-layered-system") + (description "Run a system image as produced by @command{guix system +docker-layered-image} inside Docker.") + (value (with-monad %store-monad + (>>= (lower-object + (system-image (os->image + (operating-system + (inherit (simple-operating-system)) + ;; Use locales for a single libc to + ;; reduce space requirements. + (locale-libcs (list glibc))) + #:type docker-layered-image-type))) + run-docker-system-test))))) diff --git a/guix/docker.scm b/guix/docker.scm index 5e6460f43f..b40cfb2374 100644 --- a/guix/docker.scm +++ b/guix/docker.scm @@ -3,6 +3,7 @@ ;;; Copyright © 2017, 2018, 2019, 2021 Ludovic Courtès ;;; Copyright © 2018 Chris Marusich ;;; Copyright © 2021 Maxim Cournoyer +;;; Copyright © 2023 Oleg Pykhalov ;;; ;;; This file is part of GNU Guix. ;;; @@ -28,6 +29,8 @@ (define-module (guix docker) delete-file-recursively with-directory-excursion invoke)) + #:use-module (guix diagnostics) + #:use-module (guix i18n) #:use-module (gnu build install) #:use-module (json) ;guile-json #:use-module (srfi srfi-1) @@ -38,6 +41,9 @@ (define-module (guix docker) #:use-module (rnrs bytevectors) #:use-module (ice-9 ftw) #:use-module (ice-9 match) + #:use-module (ice-9 popen) + #:use-module (ice-9 rdelim) + #:use-module (ice-9 receive) #:export (build-docker-image)) ;; Generate a 256-bit identifier in hexadecimal encoding for the Docker image. @@ -92,12 +98,12 @@ (define (canonicalize-repository-name name) (make-string (- min-length l) padding-character))) (_ normalized-name)))) -(define* (manifest path id #:optional (tag "guix")) +(define* (manifest path layers #:optional (tag "guix")) "Generate a simple image manifest." (let ((tag (canonicalize-repository-name tag))) `#(((Config . "config.json") (RepoTags . #(,(string-append tag ":latest"))) - (Layers . #(,(string-append id "/layer.tar"))))))) + (Layers . ,(list->vector layers)))))) ;; According to the specifications this is required for backwards ;; compatibility. It duplicates information provided by the manifest. @@ -106,8 +112,8 @@ (define* (repositories path id #:optional (tag "guix")) `((,(canonicalize-repository-name tag) . ((latest . ,id))))) ;; See https://github.com/opencontainers/image-spec/blob/master/config.md -(define* (config layer time arch #:key entry-point (environment '())) - "Generate a minimal image configuration for the given LAYER file." +(define* (config layers-diff-ids time arch #:key entry-point (environment '())) + "Generate a minimal image configuration for the given LAYERS files." ;; "architecture" must be values matching "platform.arch" in the ;; runtime-spec at ;; https://github.com/opencontainers/runtime-spec/blob/v1.0.0-rc2/config.md#platform @@ -125,7 +131,7 @@ (define* (config layer time arch #:key entry-point (environment '())) (container_config . #nil) (os . "linux") (rootfs . ((type . "layers") - (diff_ids . #(,(layer-diff-id layer))))))) + (diff_ids . ,(list->vector layers-diff-ids)))))) (define directive-file ;; Return the file or directory created by a 'evaluate-populate-directive' @@ -136,6 +142,37 @@ (define directive-file (('directory name _ ...) (string-trim name #\/)))) +(define %docker-image-max-layers + 100) + +(define (paths-split-sort paths) + "Split list of PATHS at %DOCKER-IMAGE-MAX-LAYERS and sort by disk usage." + (let* ((paths-length (length paths)) + (port (apply open-pipe* OPEN_READ + (append '("du" "--summarize") paths))) + (output (read-string port))) + (close-port port) + (receive (head tail) + (split-at + (map (match-lambda ((size . path) path)) + (sort (map (lambda (line) + (match (string-split line #\tab) + ((size path) + (cons (string->number size) path)))) + (string-split + (string-trim-right output #\newline) + #\newline)) + (lambda (path1 path2) + (< (match path2 ((size . _) size)) + (match path1 ((size . _) size)))))) + (if (>= paths-length %docker-image-max-layers) + (- %docker-image-max-layers 2) + (1- paths-length))) + (list head tail)))) + +(define (create-empty-tar file) + (invoke "tar" "-cf" file "--files-from" "/dev/null")) + (define* (build-docker-image image paths prefix #:key (repository "guix") @@ -146,11 +183,13 @@ (define* (build-docker-image image paths prefix entry-point (environment '()) compressor - (creation-time (current-time time-utc))) - "Write to IMAGE a Docker image archive containing the given PATHS. PREFIX -must be a store path that is a prefix of any store paths in PATHS. REPOSITORY -is a descriptive name that will show up in \"REPOSITORY\" column of the output -of \"docker images\". + (creation-time (current-time time-utc)) + layered-image? + root-system) + "Write to IMAGE a layerer Docker image archive containing the given PATHS. +PREFIX must be a store path that is a prefix of any store paths in PATHS. +REPOSITORY is a descriptive name that will show up in \"REPOSITORY\" column of +the output of \"docker images\". When DATABASE is true, copy it to /var/guix/db in the image and create /var/guix/gcroots and friends. @@ -172,7 +211,14 @@ (define* (build-docker-image image paths prefix SYSTEM is a GNU triplet (or prefix thereof) of the system the binaries in PATHS are for; it is used to produce metadata in the image. Use COMPRESSOR, a command such as '(\"gzip\" \"-9n\"), to compress IMAGE. Use CREATION-TIME, a -SRFI-19 time-utc object, as the creation time in metadata." +SRFI-19 time-utc object, as the creation time in metadata. + +When LAYERED-IMAGE? is true build layered image, providing a Docker +image with many of the store paths being on their own layer to improve sharing +between images. + +ROOT-SYSTEM is a directory with a provisioned root file system, which will be +added to image as a layer." (define (sanitize path-fragment) (escape-special-chars ;; GNU tar strips the leading slash off of absolute paths before applying @@ -203,6 +249,53 @@ (define* (build-docker-image image paths prefix (if (eq? '() transformations) '() `("--transform" ,(transformations->expression transformations)))) + (define layers-hashes + (match-lambda + (((head ...) (tail ...) id) + (create-empty-tar "image.tar") + (let* ((head-layers + (map + (lambda (file) + (invoke "tar" "cf" "layer.tar" file) + (let* ((file-hash (layer-diff-id "layer.tar")) + (file-name (string-append file-hash "/layer.tar"))) + (mkdir file-hash) + (rename-file "layer.tar" file-name) + (invoke "tar" "-rf" "image.tar" file-name) + (delete-file file-name) + file-hash)) + head)) + (tail-layer + (begin + (create-empty-tar "layer.tar") + (for-each (lambda (file) + (invoke "tar" "-rf" "layer.tar" file)) + tail) + (let* ((file-hash (layer-diff-id "layer.tar")) + (file-name (string-append file-hash "/layer.tar"))) + (mkdir file-hash) + (rename-file "layer.tar" file-name) + (invoke "tar" "-rf" "image.tar" file-name) + (delete-file file-name) + file-hash))) + (customization-layer + (let* ((file-id (string-append id "/layer.tar")) + (file-hash (layer-diff-id file-id)) + (file-name (string-append file-hash "/layer.tar"))) + (mkdir file-hash) + (rename-file file-id file-name) + (invoke "tar" "-rf" "image.tar" file-name) + file-hash)) + (all-layers + (append head-layers (list tail-layer customization-layer)))) + (with-output-to-file "manifest.json" + (lambda () + (scm->json (manifest prefix + (map (cut string-append <> "/layer.tar") + all-layers) + repository)))) + (invoke "tar" "-rf" "image.tar" "manifest.json") + all-layers)))) (let* ((directory "/tmp/docker-image") ;temporary working directory (id (docker-id prefix)) (time (date->string (time-utc->date creation-time) "~4")) @@ -229,26 +322,39 @@ (define* (build-docker-image image paths prefix (with-output-to-file "json" (lambda () (scm->json (image-description id time)))) - ;; Create a directory for the non-store files that need to go into the - ;; archive. - (mkdir "extra") + (if root-system + (let ((directory (getcwd))) + (with-directory-excursion root-system + (apply invoke "tar" + "-cf" (string-append directory "/layer.tar") + `(,@transformation-options + ,@(tar-base-options) + ,@(scandir "." + (lambda (file) + (not (member file '("." ".."))))))))) + (begin + ;; Create a directory for the non-store files that need to go + ;; into the archive. + (mkdir "extra") - (with-directory-excursion "extra" - ;; Create non-store files. - (for-each (cut evaluate-populate-directive <> "./") - extra-files) + (with-directory-excursion "extra" + ;; Create non-store files. + (for-each (cut evaluate-populate-directive <> "./") + extra-files) - (when database - ;; Initialize /var/guix, assuming PREFIX points to a profile. - (install-database-and-gc-roots "." database prefix)) + (when database + ;; Initialize /var/guix, assuming PREFIX points to a + ;; profile. + (install-database-and-gc-roots "." database prefix)) - (apply invoke "tar" "-cf" "../layer.tar" - `(,@transformation-options - ,@(tar-base-options) - ,@paths - ,@(scandir "." - (lambda (file) - (not (member file '("." "..")))))))) + (apply invoke "tar" "-cf" "../layer.tar" + `(,@transformation-options + ,@(tar-base-options) + ,@(if layered-image? '() paths) + ,@(scandir "." + (lambda (file) + (not (member file '("." "..")))))))) + (delete-file-recursively "extra"))) ;; It is possible for "/" to show up in the archive, especially when ;; applying transformations. For example, the transformation @@ -261,24 +367,36 @@ (define* (build-docker-image image paths prefix ;; error messages. (with-error-to-port (%make-void-port "w") (lambda () - (system* "tar" "--delete" "/" "-f" "layer.tar"))) - - (delete-file-recursively "extra")) + (system* "tar" "--delete" "/" "-f" "layer.tar")))) (with-output-to-file "config.json" (lambda () - (scm->json (config (string-append id "/layer.tar") - time arch - #:environment environment - #:entry-point entry-point)))) - (with-output-to-file "manifest.json" - (lambda () - (scm->json (manifest prefix id repository)))) - (with-output-to-file "repositories" - (lambda () - (scm->json (repositories prefix id repository))))) - - (apply invoke "tar" "-cf" image "-C" directory - `(,@(tar-base-options #:compressor compressor) - ".")) + (scm->json + (config (if layered-image? + (layers-hashes (append (paths-split-sort paths) + (list id))) + (list (layer-diff-id (string-append id "/layer.tar")))) + time arch + #:environment environment + #:entry-point entry-point)))) + (if layered-image? + (begin + (invoke "tar" "-rf" "image.tar" "config.json") + (if compressor + (begin + (apply invoke `(,@compressor "image.tar")) + (copy-file "image.tar.gz" image)) + (copy-file "image.tar" image))) + (begin + (with-output-to-file "manifest.json" + (lambda () + (scm->json (manifest prefix + (list (string-append id "/layer.tar")) + repository)))) + (with-output-to-file "repositories" + (lambda () + (scm->json (repositories prefix id repository)))) + (apply invoke "tar" "-cf" image + `(,@(tar-base-options #:compressor compressor) + "."))))) (delete-file-recursively directory))) diff --git a/guix/scripts/pack.scm b/guix/scripts/pack.scm index 0dc9979194..3fefd2eac3 100644 --- a/guix/scripts/pack.scm +++ b/guix/scripts/pack.scm @@ -8,6 +8,7 @@ ;;; Copyright © 2020, 2021, 2022, 2023 Maxim Cournoyer ;;; Copyright © 2020 Eric Bavier ;;; Copyright © 2022 Alex Griffin +;;; Copyright © 2023 Oleg Pykhalov ;;; ;;; This file is part of GNU Guix. ;;; @@ -28,6 +29,7 @@ (define-module (guix scripts pack) #:use-module (guix scripts) #:use-module (guix ui) #:use-module (guix gexp) + #:use-module ((guix build utils) #:select (%xz-parallel-args)) #:use-module (guix utils) #:use-module (guix store) #:use-module ((guix status) #:select (with-status-verbosity)) @@ -53,6 +55,8 @@ (define-module (guix scripts pack) #:use-module ((gnu packages compression) #:hide (zip)) #:use-module (gnu packages guile) #:use-module (gnu packages base) + #:use-module (gnu packages shells) + #:autoload (gnu packages package-management) (guix) #:autoload (gnu packages gnupg) (guile-gcrypt) #:autoload (gnu packages guile) (guile2.0-json guile-json) #:use-module (srfi srfi-1) @@ -67,6 +71,7 @@ (define-module (guix scripts pack) debian-archive rpm-archive docker-image + docker-layered-image squashfs-image %formats @@ -597,12 +602,14 @@ (define* (docker-image name profile localstatedir? (symlinks '()) (archiver tar) - (extra-options '())) + (extra-options '()) + layered-image?) "Return a derivation to construct a Docker image of PROFILE. The image is a tarball conforming to the Docker Image Specification, compressed with COMPRESSOR. It can be passed to 'docker load'. If TARGET is true, it must a be a GNU triplet and it is used to derive the architecture metadata in -the image." +the image. If LAYERED-IMAGE? is true, the image will with many of the +store paths being on their own layer to improve sharing between images." (define database (and localstatedir? (file-append (store-database (list profile)) @@ -653,7 +660,13 @@ (define* (docker-image name profile `((directory "/tmp" ,(getuid) ,(getgid) #o1777) ,@(append-map symlink->directives '#$symlinks))) - (setenv "PATH" #+(file-append archiver "/bin")) + (setenv "PATH" + (string-join `(#+(file-append archiver "/bin") + #+@(if layered-image? + (list (file-append coreutils "/bin") + (file-append gzip "/bin")) + '())) + ":")) (build-docker-image #$output (map store-info-item @@ -671,7 +684,8 @@ (define* (docker-image name profile #$entry-point))) #:extra-files directives #:compressor #+(compressor-command compressor) - #:creation-time (make-time time-utc 0 1)))))) + #:creation-time (make-time time-utc 0 1) + #:layered-image? #$layered-image?))))) (gexp->derivation (string-append name ".tar" (compressor-extension compressor)) @@ -679,6 +693,33 @@ (define* (docker-image name profile #:target target #:references-graphs `(("profile" ,profile)))) +(define* (docker-layered-image name profile + #:key target + (profile-name "guix-profile") + (compressor (first %compressors)) + entry-point + localstatedir? + (symlinks '()) + (archiver tar) + (extra-options '()) + (layered-image? #t)) + "Return a derivation to construct a Docker image of PROFILE. The image is a +tarball conforming to the Docker Image Specification, compressed with +COMPRESSOR. It can be passed to 'docker load'. If TARGET is true, it must a +be a GNU triplet and it is used to derive the architecture metadata in the +image. If LAYERED-IMAGE? is true, the image will with many of the store paths +being on their own layer to improve sharing between images." + (docker-image name profile + #:target target + #:profile-name profile-name + #:compressor compressor + #:entry-point entry-point + #:localstatedir? localstatedir? + #:symlinks symlinks + #:archiver archiver + #:extra-options extra-options + #:layered-image? layered-image?)) + ;;; ;;; Debian archive format. @@ -1353,6 +1394,7 @@ (define %formats `((tarball . ,self-contained-tarball) (squashfs . ,squashfs-image) (docker . ,docker-image) + (docker-layered . ,docker-layered-image) (deb . ,debian-archive) (rpm . ,rpm-archive))) @@ -1361,15 +1403,17 @@ (define (show-formats) (display (G_ "The supported formats for 'guix pack' are:")) (newline) (display (G_ " - tarball Self-contained tarball, ready to run on another machine")) + tarball Self-contained tarball, ready to run on another machine")) + (display (G_ " + squashfs Squashfs image suitable for Singularity")) (display (G_ " - squashfs Squashfs image suitable for Singularity")) + docker Tarball ready for 'docker load'")) (display (G_ " - docker Tarball ready for 'docker load'")) + docker-layered Tarball with a layered image ready for 'docker load'")) (display (G_ " - deb Debian archive installable via dpkg/apt")) + deb Debian archive installable via dpkg/apt")) (display (G_ " - rpm RPM archive installable via rpm/yum")) + rpm RPM archive installable via rpm/yum")) (newline)) (define (required-option symbol) diff --git a/guix/scripts/system.scm b/guix/scripts/system.scm index d7163dd3eb..e4bf0347c7 100644 --- a/guix/scripts/system.scm +++ b/guix/scripts/system.scm @@ -11,6 +11,7 @@ ;;; Copyright © 2021 Brice Waegeneire ;;; Copyright © 2021 Simon Tournier ;;; Copyright © 2022 Tobias Geerinckx-Rice +;;; Copyright © 2023 Oleg Pykhalov ;;; ;;; This file is part of GNU Guix. ;;; @@ -727,13 +728,15 @@ (define* (system-derivation-for-action image action #:graphic? graphic? #:disk-image-size image-size #:mappings mappings)) - ((image disk-image vm-image docker-image) + ((image disk-image vm-image docker-image docker-layered-image) (when (eq? action 'disk-image) (warning (G_ "'disk-image' is deprecated: use 'image' instead~%"))) (when (eq? action 'vm-image) (warning (G_ "'vm-image' is deprecated: use 'image' instead~%"))) (when (eq? action 'docker-image) (warning (G_ "'docker-image' is deprecated: use 'image' instead~%"))) + (when (eq? action 'docker-layered-image) + (warning (G_ "'docker-layered-image' is deprecated: use 'image' instead~%"))) (lower-object (system-image image)))))) (define (maybe-suggest-running-guix-pull) @@ -980,6 +983,8 @@ (define (show-help) image build a Guix System image\n")) (display (G_ "\ docker-image build a Docker image\n")) + (display (G_ "\ + docker-layered-image build a Docker layered image\n")) (display (G_ "\ init initialize a root file system to run GNU\n")) (display (G_ "\ @@ -1193,7 +1198,7 @@ (define actions '("build" "container" "vm" "vm-image" "image" "disk-image" "list-generations" "describe" "delete-generations" "roll-back" "switch-generation" "search" "edit" - "docker-image")) + "docker-image" "docker-layered-image")) (define (process-action action args opts) "Process ACTION, a sub-command, with the arguments are listed in ARGS. @@ -1242,6 +1247,8 @@ (define (process-action action args opts) (image (let* ((image-type (case action ((vm-image) qcow2-image-type) ((docker-image) docker-image-type) + ((docker-layered-image) + docker-layered-image-type) (else image-type))) (image-size (assoc-ref opts 'image-size)) (volatile? diff --git a/tests/pack.scm b/tests/pack.scm index ce5a2f8a53..432ab1b2ea 100644 --- a/tests/pack.scm +++ b/tests/pack.scm @@ -2,6 +2,7 @@ ;;; Copyright © 2017, 2018, 2019, 2020, 2021 Ludovic Courtès ;;; Copyright © 2018 Ricardo Wurmus ;;; Copyright © 2021, 2023 Maxim Cournoyer +;;; Copyright © 2023 Oleg Pykhalov ;;; ;;; This file is part of GNU Guix. ;;; @@ -29,6 +30,7 @@ (define-module (test-pack) #:use-module (guix gexp) #:use-module (guix modules) #:use-module (guix utils) + #:use-module ((guix build utils) #:select (%store-directory)) #:use-module (gnu packages) #:use-module ((gnu packages base) #:select (glibc-utf8-locales)) #:use-module (gnu packages bootstrap) @@ -250,6 +252,52 @@ (define rpm-for-tests (mkdir #$output))))))) (built-derivations (list check)))) + (unless store (test-skip 1)) + (test-assertm "docker-layered-image + localstatedir" store + (mlet* %store-monad + ((guile (set-guile-for-build (default-guile))) + (profile -> (profile + (content (packages->manifest (list %bootstrap-guile))) + (hooks '()) + (locales? #f))) + (tarball (docker-layered-image "docker-pack" profile + #:symlinks '(("/bin/Guile" -> "bin/guile")) + #:localstatedir? #t)) + (check (gexp->derivation "check-tarball" + (with-imported-modules '((guix build utils)) + #~(begin + (use-modules (guix build utils) + (ice-9 match)) + + (define bin + (string-append "." #$profile "/bin")) + + (define store + (string-append "." #$(%store-directory))) + + (setenv "PATH" (string-append #$%tar-bootstrap "/bin")) + (mkdir "base") + (with-directory-excursion "base" + (invoke "tar" "xvf" #$tarball)) + + (match (find-files "base" "layer.tar") + ((layers ...) + (for-each (lambda (layer) + (invoke "tar" "xvf" layer) + (invoke "chmod" "--recursive" "u+w" store)) + layers))) + + (when + (and (file-exists? (string-append bin "/guile")) + (file-exists? "var/guix/db/db.sqlite") + (file-is-directory? "tmp") + (string=? (string-append #$%bootstrap-guile "/bin") + (pk 'binlink (readlink bin))) + (string=? (string-append #$profile "/bin/guile") + (pk 'guilelink (readlink "bin/Guile")))) + (mkdir #$output))))))) + (built-derivations (list check)))) + (unless store (test-skip 1)) (test-assertm "squashfs-image + localstatedir" store (mlet* %store-monad base-commit: 66c9b82fed3c59ee07187898592c688c82fed273 -- 2.38.0 From debbugs-submit-bounces@debbugs.gnu.org Sat Jun 03 15:17:40 2023 Received: (at 62153) by debbugs.gnu.org; 3 Jun 2023 19:17:40 +0000 Received: from localhost ([127.0.0.1]:44325 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1q5Wku-00034C-25 for submit@debbugs.gnu.org; Sat, 03 Jun 2023 15:17:40 -0400 Received: from mail-lj1-f171.google.com ([209.85.208.171]:39161) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1q5Wks-00033z-0J for 62153@debbugs.gnu.org; Sat, 03 Jun 2023 15:17:38 -0400 Received: by mail-lj1-f171.google.com with SMTP id 38308e7fff4ca-2b07762d292so3722651fa.0 for <62153@debbugs.gnu.org>; Sat, 03 Jun 2023 12:17:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1685819852; x=1688411852; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=4URUR0lq/s5OhVUjnZV3N6WIgKKyEjRmH7hmQWx4TKA=; b=ie+uK0uXNGMXKCnon9t1GBWF/b1m62rQTFYYhWDKCjIiPNUWLLYl0D5oAIeua7LD07 7RwHmiCHnvq26Q3ZNSUwC47s0Zi8om3sAOJ6m7M0myPox5x9SvmTPKh7ux6FkK46+4zt 1KjBq2dfgE+8li6teIVktwzfspas3DBDlDosYwg6HU20z1R1mNBdqp701yfc3Tp6G1Y+ Ueo3puigyVtU47UQNffCEmdu7fW/IvSSZPzxa7pHI+JpIe1AC7ozkk+F+YiCWEHMiYd0 BcJQC4O30Caedz82RXogv9ySk2iAuvWSt9cDGXrj85BvJ+DdiV5gy2Czk11DHC1XrJXM /Mtw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685819852; x=1688411852; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=4URUR0lq/s5OhVUjnZV3N6WIgKKyEjRmH7hmQWx4TKA=; b=XOpOfg+09KUecm5egdvf9ti475pM4xxtRYTBGi8IprZ3rAr9qBSKAhGWaFeVrLqBy1 hZmxNGBbz+U4rd7uIAOqZ7gJxZ2mwVY9Xqx/j5krlpjKEAabM+ENKn8b6/xJww/NoPIu EAHk7T3S8c90S4J/cwtmdwF8SDnr/Lrsrf4BaZJX9Bd85dLz+Q5xHS50ao1MCkyb9VXu 5JMGUXQyclaGhmJKBCXL0suVlcmEUlIyR9bezZX5hiyygGRVsBB9csdDiLo+yBK8jC5Y RL4sA9VhgR9Mtf/PtPGARTG8K18DV+j7a5JBhtvwgZpOAByT3ZnP1G/qAEjUWQy/cS+u tarw== X-Gm-Message-State: AC+VfDziaxMI7LXt0GaMB9V5bfPiJRyMihqiDdoxaWNVtGXAfR44G0Cb WwXaZy2wqed8rHa9gnGhx/1PXgTA9FE= X-Google-Smtp-Source: ACHHUZ5a35livKk8rXM7W/wMdXkAnt9ANL2OsZm86wbZVDMCrtKxcCSnfu8I/6an0D9X4vbu7O/IYw== X-Received: by 2002:ac2:4951:0:b0:4f6:1c33:b9e3 with SMTP id o17-20020ac24951000000b004f61c33b9e3mr687875lfi.1.1685819851550; Sat, 03 Jun 2023 12:17:31 -0700 (PDT) Received: from guixsd.wugi.info ([88.201.161.72]) by smtp.gmail.com with ESMTPSA id v6-20020ac25606000000b004f39bb71ed2sm553317lfd.138.2023.06.03.12.17.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 03 Jun 2023 12:17:31 -0700 (PDT) From: Oleg Pykhalov To: 62153@debbugs.gnu.org Subject: [PATCH v4] news: Add entry for the new 'docker-layered' distribution format. Date: Sat, 3 Jun 2023 22:16:27 +0300 Message-Id: <270e247ee913f6ede40883919d3f6971fe1b01aa.1685819787.git.go.wigust@gmail.com> X-Mailer: git-send-email 2.38.0 In-Reply-To: <87y1l0icq0.fsf_-_@gmail.com> References: <87y1l0icq0.fsf_-_@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 62153 Cc: Oleg Pykhalov , Greg Hogan X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) * etc/news.scm: Add entry. --- etc/news.scm | 58 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 58 insertions(+) diff --git a/etc/news.scm b/etc/news.scm index 314f0ab352..158a9284b0 100644 --- a/etc/news.scm +++ b/etc/news.scm @@ -18,6 +18,7 @@ ;; Copyright © 2021 Andrew Tropin ;; Copyright © 2021, 2023 Jonathan Brielmaier ;; Copyright © 2022 Thiago Jung Bauermann +;; Copyright © 2023 Oleg Pykhalov ;; ;; Copying and distribution of this file, with or without modification, are ;; permitted in any medium without royalty provided the copyright notice and @@ -26,6 +27,63 @@ (channel-news (version 0) + (entry (commit "457c813653a44117e296deaa49e79fc701b90791") + (title + (de "Neues Format @samp{docker-layered} für den Befehl @command{guix pack}") + (en "New @samp{docker-layered} format for the @command{guix pack} command") + (ru "Новый @samp{docker-layered} формат для @command{guix pack} команды")) + (body + (de "Sie können jetzt auch mehrschichtige Docker-Abbilder mit dem Befehl +@command{guix pack --format=docker-layered} erzeugen. Damit bekommen Sie ein +Docker-Abbild, bei dem Store-Pfade auf getrennten Schichten („Layer“) +untergebracht sind, die sich mehrere Abbilder teilen können. Das Abbild wird +im Store als gzip-komprimierter Tarball erzeugt. Hier ist ein einfaches +Beispiel, wo ein mehrschichtiges Docker-Abbild für das Paket @code{hello} +angelegt wird: + +@example +guix pack --format=docker-layered --symlink=/usr/bin/hello=bin/hello hello +@end example + +@command{guix system image} kann jetzt geschichtete Docker-Abbilder erzeugen, +indem Sie @code{docker-layered} an die Befehlszeilenoption @option{--image-type} +übergeben. + +Siehe @command{info \"(guix.de) Aufruf von guix pack\"} und +@command{info \"(guix.de) Systemabbilder\"} für weitere Informationen.") + (en "Docker layered images can now be produced via the @command{guix +pack --format=docker-layered} command, providing a Docker image with many of +the store paths being on their own layer to improve sharing between images. +The image is realized into the GNU store as a gzipped tarball. Here is a +simple example that generates a layered Docker image for the @code{hello} +package: + +@example +guix pack --format=docker-layered --symlink=/usr/bin/hello=bin/hello hello +@end example + +The @command{guix system image} can now produce layered Docker image by passing +@code{docker-layered} to @option{--image-type} option. + +See @command{info \"(guix) Invoking guix pack\"} and +@command{info \"(guix) System Images\"} for more information.") + (ru "Появилась команда создания многослойных Docker образов с помощью +@command{guix pack --format=docker-layered}, которая соберет Docker образ с +путями в store расположенными на отдельных слоях, ускоряя таким образом +передачу образов. Образ будет создан в GNU store в качестве gzipped tarball. + +Пример создания Docker layered образ с @code{hello} пакетом: +@example +guix pack --format=docker-layered --symlink=/usr/bin/hello=bin/hello hello +@end example + +@command{guix system image} теперь может создавать layered Docker образ путем +указания в опции @option{--image-type} параметра @code{docker-layered}. + +Смотрите @command{info \"(guix) Invoking guix pack\"} и +@command{info \"(guix) System Images\"} для получения более детальных +сведений."))) + (entry (commit "ba5da5125a81307500982517e2f458d57b024668") (title (en "New @code{arguments} rule for @command{guix style}") base-commit: 66c9b82fed3c59ee07187898592c688c82fed273 prerequisite-patch-id: 9c90b67b3c2bb18d7fd17d083b0ab0d1cd5333cd -- 2.38.0 From debbugs-submit-bounces@debbugs.gnu.org Sat Aug 26 23:17:09 2023 Received: (at 62153) by debbugs.gnu.org; 27 Aug 2023 03:17:10 +0000 Received: from localhost ([127.0.0.1]:43723 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qa6Gz-0000dg-5T for submit@debbugs.gnu.org; Sat, 26 Aug 2023 23:17:09 -0400 Received: from mail-lj1-x233.google.com ([2a00:1450:4864:20::233]:40396) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qa6Gs-0000cw-Nq; Sat, 26 Aug 2023 23:17:07 -0400 Received: by mail-lj1-x233.google.com with SMTP id 38308e7fff4ca-2bcb5423dc7so6315371fa.1; Sat, 26 Aug 2023 20:16:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1693106211; x=1693711011; h=mime-version:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=gCr6X5zu05YfQsUceHTN3oMLBVOXaAl8PrVaR7LDphA=; b=Vwyhb8LRJBGsY9rZXZ23mDDufWgOJxUWei3H1tDjtkj9ZsmuxYABIIsY1/ElP5//2P cgZmv3lbcxrb9NAOA0+LMCq6lH20sDakZ1Dp+4gXeX9l/YA8GzcWTZwdA9Bh8oFvWtBL ms5ACYavGCIeL6UMiUzBrKzEBWMebewhWf2VjiYVwY2LMlw2rnPDWu8VAsCoMI2pUNge d+SKP+unRrYKUtQLO87UGedJ+5Mw0zW+6qBx/SU0BESc32C3UrlCVozVyiKNpnm+Ntfq 09ymFbe4JYXBIGV3s7kPTS8OLIyCZvX7zZ/cwSzNrVCtNUcnTm3OTV+jZm1ldUqDyruZ OKuQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1693106211; x=1693711011; h=mime-version:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=gCr6X5zu05YfQsUceHTN3oMLBVOXaAl8PrVaR7LDphA=; b=i/jbY5jdddwumE1Hbofj1kC3VuDpsilAh1jOGZxhPKPb7S0UHvDTXvbK0bcaPGUC9d hAbst5ipJ7IWgauSyxYLJRaxLYXggf3yGad05mP1amUk6qC5Cqpb8FrAggD68qiXxvzm mblOSjHUsEh6Y3XjwJJtcPrlUlnOkAPAfJhDBOfHMo2E7HRROJNE3rz6xwku78w6uGD2 4FOT5l7HsB2lpGymjAmaknVjru85/2ycThxWpjCDhxUCj/7+2vZnchLU2fmJkCq7e6aV IHxL1UJtW+ia6ZH5y7usCaEKb+FPp53nYPZKsUeAMceIYd8BF4CUnSm4gkFLWHi3BXqn WQjg== X-Gm-Message-State: AOJu0Yz8to5wplWcJ97ZWeO9S4bXrGkf1XPjVytv36nACOl9UIJEPHyY 4wb7CkbRvJmt8UV1DlKotDA= X-Google-Smtp-Source: AGHT+IEzzYbVbDfIur1aD9HdRnL9TL2ZhrkTwRHsyMMV7eF9OtVa0JTH3Vkiq76pEl2Lpj0qeR/kUQ== X-Received: by 2002:a2e:3509:0:b0:2b9:7034:9bbe with SMTP id z9-20020a2e3509000000b002b970349bbemr13876844ljz.4.1693106210854; Sat, 26 Aug 2023 20:16:50 -0700 (PDT) Received: from localhost ([88.201.161.72]) by smtp.gmail.com with ESMTPSA id n12-20020a2e878c000000b002bcd3761ba9sm1046610lji.24.2023.08.26.20.16.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 26 Aug 2023 20:16:50 -0700 (PDT) From: Oleg Pykhalov To: guix-devel Subject: Merging guix pack changes for Docker containers packaging Date: Sun, 27 Aug 2023 06:16:49 +0300 Message-ID: <878r9xb2e6.fsf@gmail.com> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 62153 Cc: 64173@debbugs.gnu.org, 62153@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hi Guix, I would like to merge 62153. After 64173 will be merge, merging 62153 is not possible without conflict resolving with Git. 64173 introduces =E2=80=98%docker-format-options=E2=80=99 variable. With t= his variable it's possible in 62153 to replace =E2=80=98--image-type=3Ddocker-layered=E2= =80=99 with =E2=80=98--docker-layers=3DN=E2=80=99 option, where: if =E2=80=98N=E2=80=99 is zero, then use current non layered format if =E2=80=98N=E2=80=99 is bigger than zero, then use layered format Number of layers specification is nice to have, because Docker layers are limited. So if user would like to modify a Docker image by adding more layers on top, then hacks like squashing layers are not required. Also, it will be possible to delete code which builds non layered Docker image without deprecating command line options. Is it possible to partially merge 64173, specifically =E2=80=98%docker-format-options=E2=80=99 variable and it requirements, so i= t can be used in 62153 for =E2=80=98--docker-layers=3DN=E2=80=99 option? [1]: https://issues.guix.gnu.org/issue/62153 [2]: https://issues.guix.gnu.org/64173 Regards, Oleg. --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQJIBAEBCgAyFiEEcjhxI46s62NFSFhXFn+OpQAa+pwFAmTqwCEUHGdvLndpZ3Vz dEBnbWFpbC5jb20ACgkQFn+OpQAa+pwLNg//XqUbjKOGD1st7X1i7XAZKAwkPN/j yOzVa1agcRi2bIwLqoh+/WWLf4afG0tE7b52VQKUbgSMdSfJi2RJoLs7PmTjhhhJ /QKl35ffdDr4C6MgutK/nir/Dh2wYuZ4KtKdmj6VmXkC5uSd+GA4mNriwZGUf5IX Oo2YqvEv9e9wkilRRX7BN/iVboKPlPWHxNT82x/gB2UTdgXxcICqUHa9L/h+4obl 7xrsBiGtIJMWmC0A6bZXscDrWTxlKdqAXZTkGRAh2XJ1EWdZeh6ED3s1n4lZySsE gapcNf8112Th0+TRhSV3FP7OUc5LHK8nvosNCnr3WtLMEOG/TGeTx5POc6YwUVwA oJQQu9Z9k8GCQKD58Uo4Bq7f8uhbq+iVfChESn0+ezNeJagir63CAD9eqegaQI1f e2EVxEC3sjEe264ue4fW3L7zx5SkHTHehv8mVfDKAJJTmNgCXx1t+jjP51T1vmxg n3wzP9hU4Ugg10r+whWC2BuH6k2B+wS/bJ1mkg/qSqHAB9/3wvhVikSSFFBmpI+s FmRkjTDguLW9rfZQW8Q9EJtAqOF74FRt4JXkGZffOClTdLrYNuf958wN6r4uuW00 MTBRKy753fJrU7PlwvPi7SQEe6ziaCY7w+tvYOKSV8DPPShMZBhuYnrzRM4IvuDB 5Xgx6marsPHb7AA= =VfnU -----END PGP SIGNATURE----- --=-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Fri Dec 22 17:10:38 2023 Received: (at 62153) by debbugs.gnu.org; 22 Dec 2023 22:10:38 +0000 Received: from localhost ([127.0.0.1]:48006 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rGnj3-00071R-Lw for submit@debbugs.gnu.org; Fri, 22 Dec 2023 17:10:38 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:44370) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rGnj0-000718-6r for 62153@debbugs.gnu.org; Fri, 22 Dec 2023 17:10:36 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rGnip-0006zS-NT; Fri, 22 Dec 2023 17:10:23 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:Date:References:In-Reply-To:Subject:To: From; bh=WIdghL2EFkaMIkV3uFgNz4YTc2irAtyxalC2c8RxhrI=; b=YuYjRw36Aa+I7sW1cTkQ gNfW9G9ESk2EqZzrFSKgLf7m7Cis9gZcI2qDz3h/oPm5rqvV1+DfgX3iNAdbQImNDlw+289Uxhk+s 5nz6GZtJJDRhRxTUR6b5Njk+/nBGwOgkNgtx2BmDxGDMU6JdK914mfvGJE5aHbyvrzsy6MQJiu7WL oxcvM3AmdiWzCOKXorOQ5YScKizhwFWjtANS3kS7wu4hvnAJh8MXBLXLmloTuoxDXe1MXzdPgxFqE tGIgn0TJtGDWBWX6i8/yiDPh842ZiJsN3nfQKNHZurny+u9AsATVBuiZius7GUeyoZ/EyyhR4oHlg dhMDMoYCY0AcKQ==; From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Oleg Pykhalov Subject: Re: [bug#62153] [PATCH v4 1/2] guix: docker: Build layered image. In-Reply-To: <457c813653a44117e296deaa49e79fc701b90791.1685819700.git.go.wigust@gmail.com> (Oleg Pykhalov's message of "Sat, 3 Jun 2023 22:14:59 +0300") References: <87y1l0icq0.fsf_-_@gmail.com> <457c813653a44117e296deaa49e79fc701b90791.1685819700.git.go.wigust@gmail.com> Date: Fri, 22 Dec 2023 23:10:20 +0100 Message-ID: <878r5l99df.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 62153 Cc: Greg Hogan , 62153@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Oleg Pykhalov skribis: > * doc/guix.texi (Invoking guix pack): Document docker-layered format. > (image Reference): Same. > (image-type Reference): Document docker-layered-image-type. > * gnu/image.scm > (validate-image-format)[docker-layered]: New image format. > * gnu/system/image.scm > (docker-layered-image, docker-layered-image-type): New variables. > (system-docker-image)[layered-image?]: New argument. > (system-docker-layered-image): New procedure. > (image->root-file-system)[docker-layered]: New image format. > * gnu/tests/docker.scm (%test-docker-layered-system): New test. > * guix/docker.scm (%docker-image-max-layers): New variable. > (build-docker-image)[stream-layered-image, root-system]: New arguments. > * guix/scripts/pack.scm (stream-layered-image.py): New variable. > (docker-image)[layered-image?]: New argument. > (docker-layered-image): New procedure. > (%formats)[docker-layered]: New format. > (show-formats): Document this. > * guix/scripts/system.scm > (system-derivation-for-action)[docker-layered-image]: New action. > (show-help): Document this. > (actions)[docker-layered-image]: New action. > (process-action): Add this. > * tests/pack.scm: Add "docker-layered-image + localstatedir" test. [...] > + #:use-module (guix diagnostics) > + #:use-module (guix i18n) (guix docker) shouldn=E2=80=99t need these. > + #:use-module (ice-9 popen) > + #:use-module (ice-9 rdelim) > + #:use-module (ice-9 receive) For consistency, I=E2=80=99d recommend (srfi srfi-71) instead of (ice-9 receive). > +(define %docker-image-max-layers > + 100) I=E2=80=99d add a comment on the second line, like =E2=80=9C;; Maximum numb= er of layers allowed in a Docker image according to =E2=80=A6=E2=80=9D. > +(define (paths-split-sort paths) > + "Split list of PATHS at %DOCKER-IMAGE-MAX-LAYERS and sort by disk usag= e." Nitpick: maybe (define (size-sorted-store-items items) =E2=80=A6)? > + (let* ((paths-length (length paths)) > + (port (apply open-pipe* OPEN_READ > + (append '("du" "--summarize") paths))) > + (output (read-string port))) > + (close-port port) How about: (map (lambda (item) (cons item (file-size item))) items) ? See (guix build store-copy) for the definition of =E2=80=98file-size=E2=80= =99. That way we avoid the dependency on Coreutils and code that =E2=80=9Cparses= =E2=80=9D the output of =E2=80=98du=E2=80=99. > + (define layers-hashes A short comment explaining what the inputs and outputs of this procedure are would be great. > + (match-lambda > + (((head ...) (tail ...) id) > + (create-empty-tar "image.tar") > + (let* ((head-layers > + (map > + (lambda (file) > + (invoke "tar" "cf" "layer.tar" file) > + (let* ((file-hash (layer-diff-id "layer.tar")) > + (file-name (string-append file-hash "/layer.tar= "))) > + (mkdir file-hash) > + (rename-file "layer.tar" file-name) > + (invoke "tar" "-rf" "image.tar" file-name) > + (delete-file file-name) > + file-hash)) > + head)) > + (tail-layer > + (begin > + (create-empty-tar "layer.tar") > + (for-each (lambda (file) > + (invoke "tar" "-rf" "layer.tar" file)) > + tail) > + (let* ((file-hash (layer-diff-id "layer.tar")) > + (file-name (string-append file-hash "/layer.tar"= ))) > + (mkdir file-hash) > + (rename-file "layer.tar" file-name) > + (invoke "tar" "-rf" "image.tar" file-name) > + (delete-file file-name) > + file-hash))) > + (customization-layer > + (let* ((file-id (string-append id "/layer.tar")) > + (file-hash (layer-diff-id file-id)) > + (file-name (string-append file-hash "/layer.tar"))) > + (mkdir file-hash) > + (rename-file file-id file-name) > + (invoke "tar" "-rf" "image.tar" file-name) > + file-hash)) > + (all-layers > + (append head-layers (list tail-layer customization-layer)= ))) Maybe this can be factorized a bit with: (define (seal-layer) ;; Add 'layer.tar' to 'image.tar' under the right name. Return its has= h. (let* ((file-hash (layer-diff-id "layer.tar")) (file-name (string-append file-hash "/layer.tar"))) (mkdir file-hash) (rename-file "layer.tar" file-name) (invoke "tar" "-rf" "image.tar" file-name) (delete-file file-name) file-hash))) ? Apart from this stylistic issues, it looks great to me. Thanks, Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Fri Dec 22 17:11:22 2023 Received: (at 62153) by debbugs.gnu.org; 22 Dec 2023 22:11:22 +0000 Received: from localhost ([127.0.0.1]:48011 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rGnjm-000735-AH for submit@debbugs.gnu.org; Fri, 22 Dec 2023 17:11:22 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:48040) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rGnjk-00072o-Tx; Fri, 22 Dec 2023 17:11:21 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rGnja-00077A-If; Fri, 22 Dec 2023 17:11:10 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:Date:References:In-Reply-To:Subject:To: From; bh=4V949yqYxOTNxiTYPqRVEZr4h/YjcFRIeFuKluGjCYc=; b=fwgyCj70X49QOzA51mGd 10tM7wWb8jtBkp6NVFMs8D9Qmj6dDfb9scn4jO0bCkMfa3sRRvEYyFwykQOIm3jZuom0GWpdUiFA0 y/VFCcNCwUBSmm6jW4KShM0lFiES+Vy8giNCy9+RQZBsKrlPqEv+s7cdg/QZhcFSOIp75ByMCKizt Uxj+O2m62LPOsF4bJmwJvLMRJONBLbNup51BSGKJ63V+u2cAOU3Q9OpoHTX/DwIYOqfWmeLxCK9/C EJ2vB2W4PZa76nD9Fn8jm/jeChp48NA5EMTCYf6YiRIex8DgM4vQyeS2m7Lst7QKZ9jRCG35VRczA vbl8vjA3VrrIdQ==; From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Oleg Pykhalov Subject: Re: [bug#62153] Merging guix pack changes for Docker containers packaging In-Reply-To: <878r9xb2e6.fsf@gmail.com> (Oleg Pykhalov's message of "Sun, 27 Aug 2023 06:16:49 +0300") References: <20230313003012.14325-1-go.wigust@gmail.com> <878r9xb2e6.fsf@gmail.com> Date: Fri, 22 Dec 2023 23:11:07 +0100 Message-ID: <875y0p99c4.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 62153 Cc: guix-devel , 62153@debbugs.gnu.org, 64173@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hi Oleg, Apologies for not replying earlier. I occasionally get reminded of the fact that building single-layer images is a problem, but only now did I take the time to look more closely at the latest version of these patches. Oleg Pykhalov skribis: > I would like to merge 62153. After 64173 will be merge, merging 62153 > is not possible without conflict resolving with Git. > > 64173 introduces =E2=80=98%docker-format-options=E2=80=99 variable. With= this variable > it's possible in 62153 to replace =E2=80=98--image-type=3Ddocker-layered= =E2=80=99 with > =E2=80=98--docker-layers=3DN=E2=80=99 option, where: > > if =E2=80=98N=E2=80=99 is zero, then use current non layered format > if =E2=80=98N=E2=80=99 is bigger than zero, then use layered format OK we should do that. However, the original submitter of #64173 apparently dropped the ball as we were approaching the final version. Would you like to adopt it and submit/push a version that incorporates the latest comments? Alternatively, we could do the opposite: merge the Docker layer patches first, and then rebase the =E2=80=98%docker-format-options=E2=80=99 patch, = after which we could add the =E2=80=98--docker-layers=E2=80=99 option. What=E2=80=99s your preference? > Number of layers specification is nice to have, because Docker layers > are limited. So if user would like to modify a Docker image by adding > more layers on top, then hacks like squashing layers are not required. > Also, it will be possible to delete code which builds non layered Docker > image without deprecating command line options. Agreed. Anyway, apart from the stylistic issues I reported, v4 of this patch set looks great to me. (For clarity I=E2=80=99d have preferred three patches, = one for (guix docker), one for =E2=80=98guix pack=E2=80=99, and one for =E2=80= =98guix system=E2=80=99; but it=E2=80=99s really a detail, let=E2=80=99s not block this patch series any= longer.) Thanks, Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Mon Dec 25 21:16:17 2023 Received: (at 62153) by debbugs.gnu.org; 26 Dec 2023 02:16:18 +0000 Received: from localhost ([127.0.0.1]:55599 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rHwzR-0000XH-K9 for submit@debbugs.gnu.org; Mon, 25 Dec 2023 21:16:17 -0500 Received: from mail-lf1-x12f.google.com ([2a00:1450:4864:20::12f]:38230) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rHwzM-0000MH-VP for 62153@debbugs.gnu.org; Mon, 25 Dec 2023 21:16:16 -0500 Received: by mail-lf1-x12f.google.com with SMTP id 2adb3069b0e04-50e79299da9so148225e87.1 for <62153@debbugs.gnu.org>; Mon, 25 Dec 2023 18:16:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1703556959; x=1704161759; darn=debbugs.gnu.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=a/XY8KinnS4brQMaAbOgzVNpkoeH7VCXUU+XNj2uXRo=; b=hGZViRdtXnCUVctsHBovoBGWD6igWG8zY0aWrKW+Opnc5em9Lg4lI1NmIpag08k7ll PplgoQAhmfcgQgj6jZakPHnaaUMWLGBP0MJr3ynaKxKbem5rzC9BrP4kURv2YxVy9u8E RJID8lpsZbZGezt9f/UZPIQZCklKnJHamLk8XPWw8g50RR9SL7jZC5jscLNaFucN5odx fwVjxMIG8IvsDMDVldUyN5WLHfysvd5LURvXLlgljHFexNZULiF1+dBq+Isd8qOKqMdB WXdSoKF6zMLCpYR6XIp3RNt44sBa436M/MxfFc1sMU33hhaRZKIQTmZFQTXSAK1Mxoze zWmw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1703556959; x=1704161759; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=a/XY8KinnS4brQMaAbOgzVNpkoeH7VCXUU+XNj2uXRo=; b=subidG7R29CkFk+eXnizy/fqCAO83FKBRR9Nj/hOoyAMC4MQ/aLT8doqekSj6fhc9X WdompcozwV2C6Cb4JkspiDKLRwCaacJA2jLOmAS7BSnEdEhi5WJOIuSsCercBxlGv6lx YGMawOMYI8Pbkf8zd1ANyFahJ6qM2QQwPdhO2ykawGiSHFuPL745mxI3NSSmZM50Bo0h 0FtLUJKiFbs9yZqnVSd2GfrgmzBryPBk6XAJRHT3sC4uIrWtbfAWXeXheSrM9d0ObMgS rcfAdfpL7O0//X6RmnP8ADPaAmELBZwmdVjxejv0TWZZwwTj8oX2qhrqZigfASc6dshH pmuA== X-Gm-Message-State: AOJu0YxqJyKystoFgdYQRceu09FgNEHA0W2TlMJdo2dJfQ2v8j7iWFPc gUQdJlVHhH92aXO0HaVG6MfXXy6hniM= X-Google-Smtp-Source: AGHT+IEjsdhPGr5jJUnnm6U3bQEnyoEfdRZvsGtZAq1jPZCUn6zo39BDNJfJQnnD7/jK0nGROXTM7g== X-Received: by 2002:a05:6512:3c98:b0:50e:76a4:f593 with SMTP id h24-20020a0565123c9800b0050e76a4f593mr2828461lfv.5.1703556959189; Mon, 25 Dec 2023 18:15:59 -0800 (PST) Received: from guixsd.wugi.info ([88.201.161.72]) by smtp.gmail.com with ESMTPSA id f3-20020ac25323000000b0050e78f183e1sm669170lfh.82.2023.12.25.18.15.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Dec 2023 18:15:58 -0800 (PST) From: Oleg Pykhalov To: 62153@debbugs.gnu.org Subject: [PATCH v5 0/5] Add Docker layered image for pack and system Date: Tue, 26 Dec 2023 05:15:02 +0300 Message-ID: X-Mailer: git-send-email 2.41.0 MIME-Version: 1.0 X-Debbugs-Cc: Christopher Baines , Josselin Poiret , Ludovic Courtès , Mathieu Othacehe , Ricardo Wurmus , Simon Tournier , Tobias Geerinckx-Rice Content-Transfer-Encoding: 8bit X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 62153 Cc: Oleg Pykhalov , =?UTF-8?q?Ludovic=20Court=C3=A8s?= , Greg Hogan X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) This patch series applies 64173 and for 62153 replaces 'docker-layered-image' format with '--max-layers=N' option for 'guix pack' and 'guix system image'. Graham James Addis (1): guix: pack: Add '--entry-point-argument' option. Oleg Pykhalov (4): tests: docker-system: Increase image size. guix: docker: Build layered images. guix: pack: Build layered images. scripts: system: Build layered images. doc/guix.texi | 40 +++++++- gnu/image.scm | 4 + gnu/system/image.scm | 41 +++++--- gnu/tests/docker.scm | 2 +- guix/docker.scm | 212 +++++++++++++++++++++++++++++++--------- guix/scripts/pack.scm | 80 ++++++++++++--- guix/scripts/system.scm | 28 +++++- tests/pack.scm | 49 ++++++++++ 8 files changed, 379 insertions(+), 77 deletions(-) base-commit: a4a14ab6d79f6f1f926a82dd50db4655232042b7 -- 2.41.0 From debbugs-submit-bounces@debbugs.gnu.org Mon Dec 25 21:20:02 2023 Received: (at 62153) by debbugs.gnu.org; 26 Dec 2023 02:20:02 +0000 Received: from localhost ([127.0.0.1]:55609 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rHx33-0000jz-88 for submit@debbugs.gnu.org; Mon, 25 Dec 2023 21:20:02 -0500 Received: from mail-lj1-x231.google.com ([2a00:1450:4864:20::231]:39939) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rHx30-0000jh-JW for 62153@debbugs.gnu.org; Mon, 25 Dec 2023 21:20:00 -0500 Received: by mail-lj1-x231.google.com with SMTP id 38308e7fff4ca-2ccb6ddf651so5292681fa.0 for <62153@debbugs.gnu.org>; Mon, 25 Dec 2023 18:19:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1703557185; x=1704161985; darn=debbugs.gnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Hv0rEVoBjazJDcZYBpF+gk3yoNxlURtQzepwVar2VuU=; b=mG4n0XE7QK49MGBrg+B/EzRH9VkUYzn6+Fnh6ZV9vD7/m6luUhT6l3X0OKAi1IFBxZ oYlaywrCtLlFWvRlhyBAQ++mXlXlRZ16icongN9XmGAUHzJp9GPy5f24bmgE5s59PCUx JUz9mqb6zV3prvaiofY0rpGT18E9eTpE/Op7Ad7B1kyoJQjh0eCU1qCHnhxXGuDWsEoA LjlHHz6xsB0Rt0oz52BbnSxifPu75q5fe8TKfSAAGrCBrnxjq0sY5GjthyRyxJGjrLmw PpQMMAb9xBOzgUJu87r6nbbeFXjrmFZOu7taFcWJKD88dk35ywmx/FqctOCEE9xvVSkT TeQg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1703557185; x=1704161985; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Hv0rEVoBjazJDcZYBpF+gk3yoNxlURtQzepwVar2VuU=; b=CeN3ve6uxjl9RvkZP3kRHKZYyFEnCRubzlAbdhYplpWWjsQoxLbwNKxCdcGrvipM7Y gO3fOzre+U45VhGhoEuuWH0y9752zauo7xa52/7Z2NrdgRfKZZOxk6ag2/UjQVgIaWSA dCqrHlUrwPq+5cAtzFP6TtelYpy/UsnIm2VvMs1jazw8U4CwYsllKKbVqJwNN2uaLoY7 X2lpb1IYEOjPucTnIU/unzz9lzTjXYiaq81t56MZDX37dkx0c8M2hLXCxteEZkKgWJE1 KJoR+5H2my5kslh3tWHUVwU3E/TkfJw4mZiQZaQkwQl4ugIK1/oHUKmqdE3J1lWtQckh GG2Q== X-Gm-Message-State: AOJu0YzeUZaRlbDIE8wB/QTDYx6lrcgPLsThAFC4ZLRS//LbyLUDCd27 xAiTzp/XgGNfhD2nHr3Sm5uFHBadI0o= X-Google-Smtp-Source: AGHT+IHFFetgLYy48aec+x5VrKfcF3LG25sJhYmRI+fZj2xsKHhXYI9eM+a26dsCpmSmezfECQQhIg== X-Received: by 2002:a05:651c:1187:b0:2cc:6ce9:b927 with SMTP id w7-20020a05651c118700b002cc6ce9b927mr4134558ljo.1.1703557184654; Mon, 25 Dec 2023 18:19:44 -0800 (PST) Received: from guixsd.wugi.info ([88.201.161.72]) by smtp.gmail.com with ESMTPSA id e11-20020a05651c038b00b002cca609ef8bsm1633436ljp.111.2023.12.25.18.19.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Dec 2023 18:19:44 -0800 (PST) From: Oleg Pykhalov To: 62153@debbugs.gnu.org Subject: [PATCH 1/5] guix: pack: Add '--entry-point-argument' option. Date: Tue, 26 Dec 2023 05:18:53 +0300 Message-ID: <9e2ecba897127f31c8c6fedd9ce20b6cdc0be4a2.1703556298.git.go.wigust@gmail.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: References: MIME-Version: 1.0 X-Debbugs-Cc: Christopher Baines , Josselin Poiret , Ludovic Courtès , Mathieu Othacehe , Ricardo Wurmus , Simon Tournier , Tobias Geerinckx-Rice Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 62153 Cc: Oleg Pykhalov , Graham James Addis X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) From: Graham James Addis * guix/scripts/pack.scm: (entry-point-argument-spec-option-parser): New procedure. (docker-image, %default-options, %docker-format-options, show-docker-format-options/detailed, %options, show-docker-format-options, guix-pack): Handle '--entry-point-argument' option. * doc/guix.texi: (Invoking guix pack): Document this Signed-off-by: Oleg Pykhalov Change-Id: I1124feff6af39dcc63c85fd6cc7ad50f398489dc --- doc/guix.texi | 14 +++++++++++- guix/scripts/pack.scm | 50 +++++++++++++++++++++++++++++++++++-------- 2 files changed, 54 insertions(+), 10 deletions(-) diff --git a/doc/guix.texi b/doc/guix.texi index 76b4eae67f..cca250dc31 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -122,6 +122,7 @@ Copyright @copyright{} 2023 Foundation Devices, Inc.@* Copyright @copyright{} 2023 Thomas Ieong@* Copyright @copyright{} 2023 Saku Laesvuori@* +Copyright @copyright{} 2023 Graham James Addis@* Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.3 or @@ -7406,7 +7407,7 @@ Invoking guix pack @env{GUIX_EXECUTION_ENGINE} environment variable accordingly. @end quotation -@cindex entry point, for Docker images +@cindex entry point, for Docker and Singularity images @item --entry-point=@var{command} Use @var{command} as the @dfn{entry point} of the resulting pack, if the pack format supports it---currently @code{docker} and @code{squashfs} (Singularity) @@ -7429,6 +7430,17 @@ Invoking guix pack docker run @var{image-id} @end example +@cindex entry point arguments, for docker images +@item --entry-point-argument=@var{command} +@itemx -A @var{command} +Use @var{command} as an argument to @dfn{entry point} of the resulting pack. +This option is only valid in conjunction with @code{--entry-point} and can +appear multiple times on the command line. + +@example +guix pack -f docker --entry-point=bin/guile --entry-point-argument="--help" guile +@end example + @item --expression=@var{expr} @itemx -e @var{expr} Consider the package @var{expr} evaluates to. diff --git a/guix/scripts/pack.scm b/guix/scripts/pack.scm index 8071840de1..4c0a602eb1 100644 --- a/guix/scripts/pack.scm +++ b/guix/scripts/pack.scm @@ -8,6 +8,7 @@ ;;; Copyright © 2020, 2021, 2022, 2023 Maxim Cournoyer ;;; Copyright © 2020 Eric Bavier ;;; Copyright © 2022 Alex Griffin +;;; Copyright © 2023 Graham James Addis ;;; ;;; This file is part of GNU Guix. ;;; @@ -202,6 +203,16 @@ (define (symlink-spec-option-parser opt name arg result) (leave (G_ "~a: invalid symlink specification~%") arg)))) +(define (entry-point-argument-spec-option-parser opt name arg result) + "A SRFI-37 opion parser for the --entry-point-argument option. The spec +takes multiple occurances. The entries are used in the exec form for the +docker entry-point. The values are used as parameters in conjunction with +the --entry-point option which is used as the first value in the exec form." + (let ((entry-point-argument (assoc-ref result 'entry-point-argument))) + (alist-cons 'entry-point-argument + (append entry-point-argument (list arg)) + (alist-delete 'entry-point-argument result eq?)))) + (define (set-utf8-locale profile) "Configure the environment to use the \"en_US.utf8\" locale provided by the GLIBC-UT8-LOCALES package." @@ -562,10 +573,22 @@ (define* (docker-image name profile `((directory "/tmp" ,(getuid) ,(getgid) #o1777) ,@(append-map symlink->directives '#$symlinks))) + (define (form-entry-point prefix entry-point entry-point-argument) + ;; Construct entry-point parameter for build-docker-image. The + ;; first entry is constructed by prefixing the entry-point with + ;; the supplied index subsequent entries are taken from the + ;; --entry-point-argument options. + (and=> entry-point + (lambda (entry-point) + (cons* (string-append prefix "/" entry-point) + entry-point-argument)))) + (setenv "PATH" #+(file-append archiver "/bin")) (let-keywords '#$extra-options #f - ((image-tag #f)) + ((image-tag #f) + (entry-point-argument #f)) + (build-docker-image #$output (map store-info-item (call-with-input-file "profile" @@ -578,11 +601,10 @@ (define* (docker-image name profile #:database #+database #:system (or #$target %host-type) #:environment environment - #:entry-point - #$(and entry-point - #~(list - (string-append #$profile "/" - #$entry-point))) + #:entry-point (form-entry-point + #$profile + #$entry-point + entry-point-argument) #:extra-files directives #:compressor #+(compressor-command compressor) @@ -1264,6 +1286,7 @@ (define %default-options (debug . 0) (verbosity . 1) (symlinks . ()) + (entry-point-argument . ()) (compressor . ,(first %compressors)))) (define %formats @@ -1299,7 +1322,9 @@ (define (required-option symbol) rest)))) (define %docker-format-options - (list (required-option 'image-tag))) + (list (required-option 'image-tag) + (option '(#\A "entry-point-argument") #t #f + entry-point-argument-spec-option-parser))) (define (show-docker-format-options) (display (G_ " @@ -1308,7 +1333,12 @@ (define (show-docker-format-options) (define (show-docker-format-options/detailed) (display (G_ " --image-tag=NAME - Use the given NAME for the Docker image repository")) + Use the given NAME for the Docker image repository + + -A, --entry-point-argument=COMMAND/PARAMETER + Value(s) to use for the Docker EntryPoint arguments. + Multiple instances are accepted. This is only valid + in conjunction with the --entry-point option")) (newline) (exit 0)) @@ -1619,7 +1649,9 @@ (define-command (guix-pack . args) (extra-options (match pack-format ('docker (list #:image-tag - (assoc-ref opts 'image-tag))) + (assoc-ref opts 'image-tag) + #:entry-point-argument + (assoc-ref opts 'entry-point-argument))) ('deb (list #:control-file (process-file-arg opts 'control-file) -- 2.41.0 From debbugs-submit-bounces@debbugs.gnu.org Mon Dec 25 21:20:08 2023 Received: (at 62153) by debbugs.gnu.org; 26 Dec 2023 02:20:08 +0000 Received: from localhost ([127.0.0.1]:55614 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rHx3A-0000ky-90 for submit@debbugs.gnu.org; Mon, 25 Dec 2023 21:20:08 -0500 Received: from mail-lj1-x233.google.com ([2a00:1450:4864:20::233]:41104) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rHx31-0000jk-Qv for 62153@debbugs.gnu.org; Mon, 25 Dec 2023 21:20:03 -0500 Received: by mail-lj1-x233.google.com with SMTP id 38308e7fff4ca-2ccae571ac4so8870631fa.0 for <62153@debbugs.gnu.org>; Mon, 25 Dec 2023 18:19:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1703557186; x=1704161986; darn=debbugs.gnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=qO1LqsbW/+uFtxIVMwh5hKow66K9/hH0YccsRzB92H0=; b=CksmI1jpD2Z0vSrMymiBr2jrn6V5jbRyBnFjbxksRzPHj0s3hgbn1ckClzDK+ixwa3 O2IGFFoiyFyRX+u1cX1PwYSDE6Zh7trJF4zS4HacbzwH6IY7KnMD83lsMYsJEUG/n5NV jhxShqhJZQQSSQSDwiwRQFXd/vjmBpkvt8V+LKX00GLsYWZCd6/LgyNvW/AdCetyTJLp m+/xCUdqQcSddR/XrhWW3/hi9tpymEPxY6HH6U/G6Km/4DvK6FiwXL8cfXOrFQ4uQnZr rWebPuCN47EXsukxutHiKtuFHB8WSd5UzzYUWYgD0Vp//6RbOn73kzFMjlT008Rt6LZ/ 5Hpg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1703557186; x=1704161986; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=qO1LqsbW/+uFtxIVMwh5hKow66K9/hH0YccsRzB92H0=; b=UbwgFIQRenltG+o0HQ8Dzh1ggBS9n8w3bRaFEHNS2bNLNlnr3QfCHU9wOLv7w3pURh ncsOfw7q4bnbOlMuPJ5Cd7RMuyy/VLXJgJ7deSSFe0aAqH9fZqIOB8P1lah26Oh8kUdO ux9HIxNo7UMZ7oUw6e+fRvAXJT++wMZr3fExRb2NGvpJJ5Mt8mLjubew91lJuiHq/NBl TxbRBU9UGWlBoAb+0P2tsOFfubOZhrb0FKChyy3++cAOnNU0lwWjP1OtzKzlcr5vJ5M/ SVyDTtfkOwmdbn7Z0eIu3swx6f9ivK4MHjij4EVeqS1EMSQN4ILJtIt6v28RskDSZOqi XCVQ== X-Gm-Message-State: AOJu0YyfHvJ8Zk75Kx/QvKapbd5UdWwhDTm6ub7x0Kzp1bBuqmYwHuAj xvINiUqs7mXPDQb4/N8Zv2EIsyolK5M= X-Google-Smtp-Source: AGHT+IFuLXSGL8/unhJZAUoAdDptCTHPwz9anwYutnU9teGNada/S+8GYZJRMLJjdz9MNGADekM7BA== X-Received: by 2002:a05:651c:1688:b0:2cc:6ddf:1504 with SMTP id bd8-20020a05651c168800b002cc6ddf1504mr4906372ljb.1.1703557186115; Mon, 25 Dec 2023 18:19:46 -0800 (PST) Received: from guixsd.wugi.info ([88.201.161.72]) by smtp.gmail.com with ESMTPSA id e11-20020a05651c038b00b002cca609ef8bsm1633436ljp.111.2023.12.25.18.19.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Dec 2023 18:19:45 -0800 (PST) From: Oleg Pykhalov To: 62153@debbugs.gnu.org Subject: [PATCH 2/5] tests: docker-system: Increase image size. Date: Tue, 26 Dec 2023 05:18:54 +0300 Message-ID: <6baac8dbf5f43a2e8458b592723463b205ba57f3.1703556298.git.go.wigust@gmail.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 62153 Cc: Oleg Pykhalov X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) * gnu/tests/docker.scm (run-docker-system-test)[vm]: Increase 'disk-image-size'. Change-Id: If88588d8981efdfdc539460900f1cbb9a663f9cb --- gnu/tests/docker.scm | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/gnu/tests/docker.scm b/gnu/tests/docker.scm index edc9804414..9e9d2e2d07 100644 --- a/gnu/tests/docker.scm +++ b/gnu/tests/docker.scm @@ -212,7 +212,7 @@ (define (run-docker-system-test tarball) (virtual-machine (operating-system os) (volatile? #f) - (disk-image-size (* 5500 (expt 2 20))) + (disk-image-size (* 6000 (expt 2 20))) (memory-size 2048) (port-forwardings '()))) -- 2.41.0 From debbugs-submit-bounces@debbugs.gnu.org Mon Dec 25 21:20:09 2023 Received: (at 62153) by debbugs.gnu.org; 26 Dec 2023 02:20:09 +0000 Received: from localhost ([127.0.0.1]:55616 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rHx3A-0000l0-JC for submit@debbugs.gnu.org; Mon, 25 Dec 2023 21:20:09 -0500 Received: from mail-lf1-x130.google.com ([2a00:1450:4864:20::130]:38233) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rHx33-0000jm-7P for 62153@debbugs.gnu.org; Mon, 25 Dec 2023 21:20:04 -0500 Received: by mail-lf1-x130.google.com with SMTP id 2adb3069b0e04-50e79299da9so148826e87.1 for <62153@debbugs.gnu.org>; Mon, 25 Dec 2023 18:19:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1703557188; x=1704161988; darn=debbugs.gnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=B4CW8S5mwrxE9wU3Tigfd4N1Fl8MimAsHBmgn6Zu100=; b=YWO1WIjaxNOwYz/0PB1myKwtlkcq4rjfgKUQuCy7jbuXSyZE/C6UFC5gtGwwVQFVjR KVJwbW3rl1/8hR1gPm5kIinkZzI856WNWDgU8ED792RQlqnYjc83sKyoD3P8LSG5QmCr Wm6NdAjcxUa+q/djP6qLqyQVzEFPEBZ4IWRevumhkM9FFjmFLC46dgVyV3Qmg9oUS6Bz 2jqIezLM3J8Y/o86cgERmmWuCwfRP9vncWOqz3ek8FzGinrQ7NkJL2A/u4UOYIJGkycX rqz4HF1YMmw8Q9CPO5s1MH5EzTbKI4lWIxXBqb2MvzmbXs6R9VwbRQWbj6b1jofwZoJk Isxw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1703557188; x=1704161988; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=B4CW8S5mwrxE9wU3Tigfd4N1Fl8MimAsHBmgn6Zu100=; b=uo320T4sX4zO+VKPyx0/Ai14oo92cO9XaqQPlY64mXWDnGqDxQq0F7fGgj3vPzB1nP CBWsF2dElBiYh7Zl6z/c7ykq27tQUoq47Kdf8owWlNd+JshDgl1Sm/l1EBKVXB9gYbXq 2QJU6R8J4RTzGp0T45UKl2MPzhEYrZrKOBqJ4MRTRPIEBW93cPiKTcj4i++FFEYehTti caUeKqfX6JxLVGpsxQ1zWhkehCdI/ryzKo9+QeO27vki2QaVDIvDsIEIt1j5mCPRky0h SbDZ4+cwHVdmATTI1hCnXn1IASKwFqrp47hv2NNAQ8i1NgLZQ7DezaezUyEaFfX7O1T3 OgKA== X-Gm-Message-State: AOJu0YyhtaPwtqsQHmf+/2wRWKw3viDFI8a61MwckZP0KRBJLC2xHg9g fKUBn8RhXqBblF6owgCC49B4deZFmhk= X-Google-Smtp-Source: AGHT+IEmtJ9jElPwdlzxbbQk89wdTmtszxClF2esPBA6hju6UpZDj9fvWN+r2Gmem9IEnA/2kFIw3Q== X-Received: by 2002:a2e:9dd2:0:b0:2ca:143:a52f with SMTP id x18-20020a2e9dd2000000b002ca0143a52fmr5071742ljj.2.1703557187638; Mon, 25 Dec 2023 18:19:47 -0800 (PST) Received: from guixsd.wugi.info ([88.201.161.72]) by smtp.gmail.com with ESMTPSA id e11-20020a05651c038b00b002cca609ef8bsm1633436ljp.111.2023.12.25.18.19.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Dec 2023 18:19:46 -0800 (PST) From: Oleg Pykhalov To: 62153@debbugs.gnu.org Subject: [PATCH 3/5] guix: docker: Build layered images. Date: Tue, 26 Dec 2023 05:18:55 +0300 Message-ID: <49f8906ba06af461e17d9badcbbf3967f1a8be3b.1703556298.git.go.wigust@gmail.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: References: MIME-Version: 1.0 X-Debbugs-Cc: Christopher Baines , Josselin Poiret , Ludovic Courtès , Mathieu Othacehe , Ricardo Wurmus , Simon Tournier , Tobias Geerinckx-Rice Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 62153 Cc: Oleg Pykhalov X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) * guix/docker.scm (%docker-image-max-layers): New variable. (size-sorted-store-items, create-empty-tar): New procedures. (config, manifest, build-docker-image): Build layered images. Change-Id: I4c8846bff0a3ceccb77e6bdf95d4942e5c3efe41 --- guix/docker.scm | 212 +++++++++++++++++++++++++++++++++++++----------- 1 file changed, 166 insertions(+), 46 deletions(-) diff --git a/guix/docker.scm b/guix/docker.scm index 5e6460f43f..5deca2afdb 100644 --- a/guix/docker.scm +++ b/guix/docker.scm @@ -3,6 +3,7 @@ ;;; Copyright © 2017, 2018, 2019, 2021 Ludovic Courtès ;;; Copyright © 2018 Chris Marusich ;;; Copyright © 2021 Maxim Cournoyer +;;; Copyright © 2023 Oleg Pykhalov ;;; ;;; This file is part of GNU Guix. ;;; @@ -29,16 +30,27 @@ (define-module (guix docker) with-directory-excursion invoke)) #:use-module (gnu build install) + #:use-module ((guix build store-copy) + #:select (file-size)) #:use-module (json) ;guile-json #:use-module (srfi srfi-1) #:use-module (srfi srfi-19) #:use-module (srfi srfi-26) + #:use-module (srfi srfi-71) #:use-module ((texinfo string-utils) #:select (escape-special-chars)) #:use-module (rnrs bytevectors) #:use-module (ice-9 ftw) #:use-module (ice-9 match) - #:export (build-docker-image)) + #:export (%docker-image-max-layers + build-docker-image)) + +;; The maximum number of layers allowed in a Docker image is typically around +;; 128, although it may vary depending on the Docker daemon. However, we +;; recommend setting the limit to 100 to ensure sufficient room for future +;; extensions. +(define %docker-image-max-layers + #f) ;; Generate a 256-bit identifier in hexadecimal encoding for the Docker image. (define docker-id @@ -92,12 +104,12 @@ (define (canonicalize-repository-name name) (make-string (- min-length l) padding-character))) (_ normalized-name)))) -(define* (manifest path id #:optional (tag "guix")) +(define* (manifest path layers #:optional (tag "guix")) "Generate a simple image manifest." (let ((tag (canonicalize-repository-name tag))) `#(((Config . "config.json") (RepoTags . #(,(string-append tag ":latest"))) - (Layers . #(,(string-append id "/layer.tar"))))))) + (Layers . ,(list->vector layers)))))) ;; According to the specifications this is required for backwards ;; compatibility. It duplicates information provided by the manifest. @@ -106,8 +118,8 @@ (define* (repositories path id #:optional (tag "guix")) `((,(canonicalize-repository-name tag) . ((latest . ,id))))) ;; See https://github.com/opencontainers/image-spec/blob/master/config.md -(define* (config layer time arch #:key entry-point (environment '())) - "Generate a minimal image configuration for the given LAYER file." +(define* (config layers-diff-ids time arch #:key entry-point (environment '())) + "Generate a minimal image configuration for the given LAYERS files." ;; "architecture" must be values matching "platform.arch" in the ;; runtime-spec at ;; https://github.com/opencontainers/runtime-spec/blob/v1.0.0-rc2/config.md#platform @@ -125,7 +137,7 @@ (define* (config layer time arch #:key entry-point (environment '())) (container_config . #nil) (os . "linux") (rootfs . ((type . "layers") - (diff_ids . #(,(layer-diff-id layer))))))) + (diff_ids . ,(list->vector layers-diff-ids)))))) (define directive-file ;; Return the file or directory created by a 'evaluate-populate-directive' @@ -136,6 +148,26 @@ (define directive-file (('directory name _ ...) (string-trim name #\/)))) +(define (size-sorted-store-items items max-layers) + "Split list of ITEMS at %MAX-LAYERS and sort by disk usage." + (let* ((items-length (length items)) + (head tail + (split-at + (map (match-lambda ((size . item) item)) + (sort (map (lambda (item) + (cons (file-size item) item)) + items) + (lambda (item1 item2) + (< (match item2 ((size . _) size)) + (match item1 ((size . _) size)))))) + (if (>= items-length max-layers) + (- max-layers 2) + (1- items-length))))) + (list head tail))) + +(define (create-empty-tar file) + (invoke "tar" "-cf" file "--files-from" "/dev/null")) + (define* (build-docker-image image paths prefix #:key (repository "guix") @@ -146,11 +178,13 @@ (define* (build-docker-image image paths prefix entry-point (environment '()) compressor - (creation-time (current-time time-utc))) - "Write to IMAGE a Docker image archive containing the given PATHS. PREFIX -must be a store path that is a prefix of any store paths in PATHS. REPOSITORY -is a descriptive name that will show up in \"REPOSITORY\" column of the output -of \"docker images\". + (creation-time (current-time time-utc)) + max-layers + root-system) + "Write to IMAGE a layerer Docker image archive containing the given PATHS. +PREFIX must be a store path that is a prefix of any store paths in PATHS. +REPOSITORY is a descriptive name that will show up in \"REPOSITORY\" column of +the output of \"docker images\". When DATABASE is true, copy it to /var/guix/db in the image and create /var/guix/gcroots and friends. @@ -172,7 +206,14 @@ (define* (build-docker-image image paths prefix SYSTEM is a GNU triplet (or prefix thereof) of the system the binaries in PATHS are for; it is used to produce metadata in the image. Use COMPRESSOR, a command such as '(\"gzip\" \"-9n\"), to compress IMAGE. Use CREATION-TIME, a -SRFI-19 time-utc object, as the creation time in metadata." +SRFI-19 time-utc object, as the creation time in metadata. + +When MAX-LAYERS is not false build layered image, providing a Docker +image with many of the store paths being on their own layer to improve sharing +between images. + +ROOT-SYSTEM is a directory with a provisioned root file system, which will be +added to image as a layer." (define (sanitize path-fragment) (escape-special-chars ;; GNU tar strips the leading slash off of absolute paths before applying @@ -203,6 +244,59 @@ (define* (build-docker-image image paths prefix (if (eq? '() transformations) '() `("--transform" ,(transformations->expression transformations)))) + (define (seal-layer) + ;; Add 'layer.tar' to 'image.tar' under the right name. Return its hash. + (let* ((file-hash (layer-diff-id "layer.tar")) + (file-name (string-append file-hash "/layer.tar"))) + (mkdir file-hash) + (rename-file "layer.tar" file-name) + (invoke "tar" "-rf" "image.tar" file-name) + (delete-file file-name) + file-hash)) + (define layers-hashes + ;; Generate a tarball that includes container image layers as tarballs, + ;; along with a manifest.json file describing the layer and config file + ;; locations. + (match-lambda + (((head ...) (tail ...) id) + (create-empty-tar "image.tar") + (let* ((head-layers + (map + (lambda (file) + (invoke "tar" "cf" "layer.tar" file) + (seal-layer)) + head)) + (tail-layer + (begin + (create-empty-tar "layer.tar") + (for-each (lambda (file) + (invoke "tar" "-rf" "layer.tar" file)) + tail) + (let* ((file-hash (layer-diff-id "layer.tar")) + (file-name (string-append file-hash "/layer.tar"))) + (mkdir file-hash) + (rename-file "layer.tar" file-name) + (invoke "tar" "-rf" "image.tar" file-name) + (delete-file file-name) + file-hash))) + (customization-layer + (let* ((file-id (string-append id "/layer.tar")) + (file-hash (layer-diff-id file-id)) + (file-name (string-append file-hash "/layer.tar"))) + (mkdir file-hash) + (rename-file file-id file-name) + (invoke "tar" "-rf" "image.tar" file-name) + file-hash)) + (all-layers + (append head-layers (list tail-layer customization-layer)))) + (with-output-to-file "manifest.json" + (lambda () + (scm->json (manifest prefix + (map (cut string-append <> "/layer.tar") + all-layers) + repository)))) + (invoke "tar" "-rf" "image.tar" "manifest.json") + all-layers)))) (let* ((directory "/tmp/docker-image") ;temporary working directory (id (docker-id prefix)) (time (date->string (time-utc->date creation-time) "~4")) @@ -229,26 +323,39 @@ (define* (build-docker-image image paths prefix (with-output-to-file "json" (lambda () (scm->json (image-description id time)))) - ;; Create a directory for the non-store files that need to go into the - ;; archive. - (mkdir "extra") + (if root-system + (let ((directory (getcwd))) + (with-directory-excursion root-system + (apply invoke "tar" + "-cf" (string-append directory "/layer.tar") + `(,@transformation-options + ,@(tar-base-options) + ,@(scandir "." + (lambda (file) + (not (member file '("." ".."))))))))) + (begin + ;; Create a directory for the non-store files that need to go + ;; into the archive. + (mkdir "extra") - (with-directory-excursion "extra" - ;; Create non-store files. - (for-each (cut evaluate-populate-directive <> "./") - extra-files) + (with-directory-excursion "extra" + ;; Create non-store files. + (for-each (cut evaluate-populate-directive <> "./") + extra-files) - (when database - ;; Initialize /var/guix, assuming PREFIX points to a profile. - (install-database-and-gc-roots "." database prefix)) + (when database + ;; Initialize /var/guix, assuming PREFIX points to a + ;; profile. + (install-database-and-gc-roots "." database prefix)) - (apply invoke "tar" "-cf" "../layer.tar" - `(,@transformation-options - ,@(tar-base-options) - ,@paths - ,@(scandir "." - (lambda (file) - (not (member file '("." "..")))))))) + (apply invoke "tar" "-cf" "../layer.tar" + `(,@transformation-options + ,@(tar-base-options) + ,@(if max-layers '() paths) + ,@(scandir "." + (lambda (file) + (not (member file '("." "..")))))))) + (delete-file-recursively "extra"))) ;; It is possible for "/" to show up in the archive, especially when ;; applying transformations. For example, the transformation @@ -261,24 +368,37 @@ (define* (build-docker-image image paths prefix ;; error messages. (with-error-to-port (%make-void-port "w") (lambda () - (system* "tar" "--delete" "/" "-f" "layer.tar"))) - - (delete-file-recursively "extra")) + (system* "tar" "--delete" "/" "-f" "layer.tar")))) (with-output-to-file "config.json" (lambda () - (scm->json (config (string-append id "/layer.tar") - time arch - #:environment environment - #:entry-point entry-point)))) - (with-output-to-file "manifest.json" - (lambda () - (scm->json (manifest prefix id repository)))) - (with-output-to-file "repositories" - (lambda () - (scm->json (repositories prefix id repository))))) - - (apply invoke "tar" "-cf" image "-C" directory - `(,@(tar-base-options #:compressor compressor) - ".")) + (scm->json + (config (if max-layers + (layers-hashes + (append (size-sorted-store-items paths max-layers) + (list id))) + (list (layer-diff-id (string-append id "/layer.tar")))) + time arch + #:environment environment + #:entry-point entry-point)))) + (if max-layers + (begin + (invoke "tar" "-rf" "image.tar" "config.json") + (if compressor + (begin + (apply invoke `(,@compressor "image.tar")) + (copy-file "image.tar.gz" image)) + (copy-file "image.tar" image))) + (begin + (with-output-to-file "manifest.json" + (lambda () + (scm->json (manifest prefix + (list (string-append id "/layer.tar")) + repository)))) + (with-output-to-file "repositories" + (lambda () + (scm->json (repositories prefix id repository)))) + (apply invoke "tar" "-cf" image + `(,@(tar-base-options #:compressor compressor) + "."))))) (delete-file-recursively directory))) -- 2.41.0 From debbugs-submit-bounces@debbugs.gnu.org Mon Dec 25 21:20:10 2023 Received: (at 62153) by debbugs.gnu.org; 26 Dec 2023 02:20:10 +0000 Received: from localhost ([127.0.0.1]:55618 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rHx3B-0000l8-FY for submit@debbugs.gnu.org; Mon, 25 Dec 2023 21:20:10 -0500 Received: from mail-lf1-x136.google.com ([2a00:1450:4864:20::136]:38077) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rHx34-0000jp-Nf for 62153@debbugs.gnu.org; Mon, 25 Dec 2023 21:20:05 -0500 Received: by mail-lf1-x136.google.com with SMTP id 2adb3069b0e04-50e7a0d07a0so122988e87.0 for <62153@debbugs.gnu.org>; Mon, 25 Dec 2023 18:19:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1703557189; x=1704161989; darn=debbugs.gnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=BhVNkDMX7JP5NGW3OEcnZMSwtVpydjEjv3znzIK2QTY=; b=IAD3+8Wopa8r9AmKhjSvQWV92KOV2fGIw0PoJrm4Jgk7d16QMI84WtOjTHL48hJBVw uGPJReZzFGb8TggXZbFZrwv/SO7sc1SjY2erhEOwDePrasSSFffWngVtb8Xh4s9C5ydW v4LPMv52x3ocvwtDrs169+BukvJvK4jcM2V5lA9QmbDHciOpjO7bLp4sHDR1/0gJWTMr eFakZpRC65hakwv+dBls3N0WaELOJLQviVY3HO9U+RlJKpS7B74kf1FJPOJFVAGc+bZ9 MmnS+OtETtesZ3RM8+cvW3Xjs2vjV6cnJwzHtLWFkC1Jnr97usFoA67N3Ih7u4GRdi4T NT9Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1703557189; x=1704161989; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=BhVNkDMX7JP5NGW3OEcnZMSwtVpydjEjv3znzIK2QTY=; b=tmar6f+NSnk7ZENlbReUxUDLrmtUY5vc5zo/KbuTfLbpWZAQnvTLo3wnhQU8p8Q5UF ZcvNiYRifcoG0Wy88NLKVvIuFiOAsnD6xSOO3qJiCQw4seoNKOqOWiSNKgpxaFbaFMtH rTONWBQTLGKKy9h0diEYla4Iu3MxGCvssBbZviSyFteYG1plMobG7S48VqydxHHOPWCp /c+dZwAevjO7YMd0U97laXDDoQ02cjEv1O+tXXKwma1f7FBeAC6d64nTYRnIDopaiVsR y6+mra7q0DSbFSoGViV3lzvDJI+317rc+G4w6cq2E+kPyB440dzy0ZdSXJSNWYBiEnav Nmcg== X-Gm-Message-State: AOJu0YzOOQ0YeTWFfYN+l9VcZaW8tle8W4lxnubrt8l/1VmgmNBF6cvg BRfYnIXzU8RZ8b7Ww/8tnavAqO9rlHk= X-Google-Smtp-Source: AGHT+IH7eZs1bzPjO9KpFR9WnVJAvA8I3VrH9Vm3u7m+R9ydzyw1a1Fa81eaIPjQWV2ELzKWCpRh8A== X-Received: by 2002:a2e:9dd2:0:b0:2ca:143:a52f with SMTP id x18-20020a2e9dd2000000b002ca0143a52fmr5071748ljj.2.1703557189003; Mon, 25 Dec 2023 18:19:49 -0800 (PST) Received: from guixsd.wugi.info ([88.201.161.72]) by smtp.gmail.com with ESMTPSA id e11-20020a05651c038b00b002cca609ef8bsm1633436ljp.111.2023.12.25.18.19.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Dec 2023 18:19:48 -0800 (PST) From: Oleg Pykhalov To: 62153@debbugs.gnu.org Subject: [PATCH 4/5] guix: pack: Build layered images. Date: Tue, 26 Dec 2023 05:18:56 +0300 Message-ID: <77bae2565cb5506342028896e1a45c757f3bad51.1703556298.git.go.wigust@gmail.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: References: MIME-Version: 1.0 X-Debbugs-Cc: Christopher Baines , Josselin Poiret , Ludovic Courtès , Mathieu Othacehe , Ricardo Wurmus , Simon Tournier , Tobias Geerinckx-Rice Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 62153 Cc: Oleg Pykhalov X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) * guix/scripts/pack.scm (docker-image, guix-pack, %default-options, %docker-format-options, show-docker-format-options/detailed): Handle '--max-layers' option. * doc/guix.texi (Invoking guix pack): Document this. Change-Id: I90660b2421fcdde891f003469fe2e2edaac7da41 --- doc/guix.texi | 26 ++++++++++++++++++++++- guix/scripts/pack.scm | 38 ++++++++++++++++++++++++++------- tests/pack.scm | 49 +++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 104 insertions(+), 9 deletions(-) diff --git a/doc/guix.texi b/doc/guix.texi index cca250dc31..d21048405a 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -56,7 +56,7 @@ Copyright @copyright{} 2017, 2018, 2019, 2020, 2023 Arun Isaac@* Copyright @copyright{} 2017 nee@* Copyright @copyright{} 2018 Rutger Helling@* -Copyright @copyright{} 2018, 2021 Oleg Pykhalov@* +Copyright @copyright{} 2018, 2021, 2023 Oleg Pykhalov@* Copyright @copyright{} 2018 Mike Gerwitz@* Copyright @copyright{} 2018 Pierre-Antoine Rouby@* Copyright @copyright{} 2018, 2019 Gábor Boskovits@* @@ -7441,6 +7441,30 @@ Invoking guix pack guix pack -f docker --entry-point=bin/guile --entry-point-argument="--help" guile @end example +@cindex maximum layers argument, for docker images +@item --max-layers=@code{n} +Specifies the maximum number of Docker image layers allowed when +building an image. + +@example +guix pack -f docker --max-layers=100 guile +@end example + +This option allows you to limit the number of layers in a Docker image. +Docker images are comprised of multiple layers, and each layer adds to +the overall size and complexity of the image. By setting a maximum +number of layers, you can control the following effects: + +@itemize +@item Disk Usage: +Increasing the number of layers can help optimize the disk space +required to store multiple images built with a similar package graph. + +@item Pulling: +When transferring images between different nodes or systems, having more +layers can reduce the time required to pull the image. +@end itemize + @item --expression=@var{expr} @itemx -e @var{expr} Consider the package @var{expr} evaluates to. diff --git a/guix/scripts/pack.scm b/guix/scripts/pack.scm index 4c0a602eb1..22f0dd6061 100644 --- a/guix/scripts/pack.scm +++ b/guix/scripts/pack.scm @@ -9,6 +9,7 @@ ;;; Copyright © 2020 Eric Bavier ;;; Copyright © 2022 Alex Griffin ;;; Copyright © 2023 Graham James Addis +;;; Copyright © 2023 Oleg Pykhalov ;;; ;;; This file is part of GNU Guix. ;;; @@ -48,6 +49,7 @@ (define-module (guix scripts pack) #:use-module (guix scripts build) #:use-module (guix transformations) #:use-module ((guix self) #:select (make-config.scm)) + #:use-module ((guix docker) #:select (%docker-image-max-layers)) #:use-module (gnu compression) #:use-module (gnu packages) #:use-module (gnu packages bootstrap) @@ -517,12 +519,15 @@ (define* (docker-image name profile localstatedir? (symlinks '()) (archiver tar) - (extra-options '())) + (extra-options '()) + max-layers) "Return a derivation to construct a Docker image of PROFILE. The image is a tarball conforming to the Docker Image Specification, compressed with COMPRESSOR. It can be passed to 'docker load'. If TARGET is true, it must a be a GNU triplet and it is used to derive the architecture metadata in -the image. EXTRA-OPTIONS may contain the IMAGE-TAG keyword argument." +the image. EXTRA-OPTIONS may contain the IMAGE-TAG keyword argument. +If MAX-LAYERS is not false, the image will with many of the store paths being +on their own layer to improve sharing between images." (define database (and localstatedir? (file-append (store-database (list profile)) @@ -583,11 +588,17 @@ (define* (docker-image name profile (cons* (string-append prefix "/" entry-point) entry-point-argument)))) - (setenv "PATH" #+(file-append archiver "/bin")) + (setenv "PATH" + (string-join `(#+(file-append archiver "/bin") + #+@(if max-layers + (list (file-append gzip "/bin")) + '())) + ":")) (let-keywords '#$extra-options #f ((image-tag #f) - (entry-point-argument #f)) + (entry-point-argument #f) + (max-layers #f)) (build-docker-image #$output (map store-info-item @@ -609,7 +620,8 @@ (define* (docker-image name profile #:compressor #+(compressor-command compressor) #:creation-time - (make-time time-utc 0 1))))))) + (make-time time-utc 0 1) + #:max-layers max-layers)))))) (gexp->derivation (string-append name ".tar" (compressor-extension compressor)) @@ -1287,6 +1299,7 @@ (define %default-options (verbosity . 1) (symlinks . ()) (entry-point-argument . ()) + (max-layers . ,%docker-image-max-layers) (compressor . ,(first %compressors)))) (define %formats @@ -1324,7 +1337,11 @@ (define (required-option symbol) (define %docker-format-options (list (required-option 'image-tag) (option '(#\A "entry-point-argument") #t #f - entry-point-argument-spec-option-parser))) + entry-point-argument-spec-option-parser) + (option '("max-layers") #t #f + (lambda (opt name arg result) + (alist-cons 'max-layers (string->number* arg) + result))))) (define (show-docker-format-options) (display (G_ " @@ -1338,7 +1355,10 @@ (define (show-docker-format-options/detailed) -A, --entry-point-argument=COMMAND/PARAMETER Value(s) to use for the Docker EntryPoint arguments. Multiple instances are accepted. This is only valid - in conjunction with the --entry-point option")) + in conjunction with the --entry-point option + + --max-layers=N + Number of image layers")) (newline) (exit 0)) @@ -1651,7 +1671,9 @@ (define-command (guix-pack . args) (list #:image-tag (assoc-ref opts 'image-tag) #:entry-point-argument - (assoc-ref opts 'entry-point-argument))) + (assoc-ref opts 'entry-point-argument) + #:max-layers + (assoc-ref opts 'max-layers))) ('deb (list #:control-file (process-file-arg opts 'control-file) diff --git a/tests/pack.scm b/tests/pack.scm index ac78817a70..fda4dc04c6 100644 --- a/tests/pack.scm +++ b/tests/pack.scm @@ -2,6 +2,7 @@ ;;; Copyright © 2017-2021, 2023 Ludovic Courtès ;;; Copyright © 2018 Ricardo Wurmus ;;; Copyright © 2021, 2023 Maxim Cournoyer +;;; Copyright © 2023 Oleg Pykhalov ;;; ;;; This file is part of GNU Guix. ;;; @@ -29,6 +30,7 @@ (define-module (test-pack) #:use-module (guix gexp) #:use-module (guix modules) #:use-module (guix utils) + #:use-module ((guix build utils) #:select (%store-directory)) #:use-module (gnu packages) #:use-module ((gnu packages base) #:select (libc-utf8-locales-for-target)) #:use-module (gnu packages bootstrap) @@ -250,6 +252,53 @@ (define rpm-for-tests (mkdir #$output))))))) (built-derivations (list check)))) + (unless store (test-skip 1)) + (test-assertm "docker-layered-image + localstatedir" + (mlet* %store-monad + ((guile (set-guile-for-build (default-guile))) + (profile -> (profile + (content (packages->manifest (list %bootstrap-guile))) + (hooks '()) + (locales? #f))) + (tarball (docker-image "docker-pack" profile + #:symlinks '(("/bin/Guile" -> "bin/guile")) + #:localstatedir? #t + #:max-layers 100)) + (check (gexp->derivation "check-tarball" + (with-imported-modules '((guix build utils)) + #~(begin + (use-modules (guix build utils) + (ice-9 match)) + + (define bin + (string-append "." #$profile "/bin")) + + (define store + (string-append "." #$(%store-directory))) + + (setenv "PATH" (string-append #$%tar-bootstrap "/bin")) + (mkdir "base") + (with-directory-excursion "base" + (invoke "tar" "xvf" #$tarball)) + + (match (find-files "base" "layer.tar") + ((layers ...) + (for-each (lambda (layer) + (invoke "tar" "xvf" layer) + (invoke "chmod" "--recursive" "u+w" store)) + layers))) + + (when + (and (file-exists? (string-append bin "/guile")) + (file-exists? "var/guix/db/db.sqlite") + (file-is-directory? "tmp") + (string=? (string-append #$%bootstrap-guile "/bin") + (pk 'binlink (readlink bin))) + (string=? (string-append #$profile "/bin/guile") + (pk 'guilelink (readlink "bin/Guile")))) + (mkdir #$output))))))) + (built-derivations (list check)))) + (unless store (test-skip 1)) (test-assertm "squashfs-image + localstatedir" (mlet* %store-monad -- 2.41.0 From debbugs-submit-bounces@debbugs.gnu.org Mon Dec 25 21:20:11 2023 Received: (at 62153) by debbugs.gnu.org; 26 Dec 2023 02:20:11 +0000 Received: from localhost ([127.0.0.1]:55620 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rHx3C-0000lF-AN for submit@debbugs.gnu.org; Mon, 25 Dec 2023 21:20:10 -0500 Received: from mail-lj1-x232.google.com ([2a00:1450:4864:20::232]:39940) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rHx35-0000jr-Kd for 62153@debbugs.gnu.org; Mon, 25 Dec 2023 21:20:06 -0500 Received: by mail-lj1-x232.google.com with SMTP id 38308e7fff4ca-2ccb6ddf651so5292801fa.0 for <62153@debbugs.gnu.org>; Mon, 25 Dec 2023 18:19:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1703557190; x=1704161990; darn=debbugs.gnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=6Ln2kxpz4n9WOq6SK1nelI1pbUl1f4tnE9yQEyDFGgY=; b=YOGYfJItCrRAr3tfB58bHsKHB0WqUJuQxEi8yUN7wRCp21GgrUUXG2pYDNJWXJFcEt IX0mgeQ8bZPx2ybnSs9lRdR+Pmp8bvj7U5geV8ShqEMTGAg+NbIIsukYxmEqL/0GSXr2 EDKWVmhQ1svq1RGxUKFs9k4+NVryYQsBWzyF6y8WaLnsmQ1+xAq98zSLz1WWbgu7yXay qfD/GiYWc5V3gZzX2HDuiFmCZo35ku26fNQgKyvtfWQyfzppI0a80AeAgDcxWP3YiRQQ JjeMduWnu1lya9hp4NL7OQHWZUjghQLnpJlszR7pm3Uz3Ivw1ZIM8PSdMMflG7eMMURa j6fQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1703557190; x=1704161990; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=6Ln2kxpz4n9WOq6SK1nelI1pbUl1f4tnE9yQEyDFGgY=; b=UYnwQeXYNOcyRuziqmwA0cV0lu8dBhu65uI3n4LDdBn9Uoq6oKkRncQ0e9wuXxKA1f eE6ZMo4FPjQxETi6JCOvrCDIGWFT6Nbroqp+EAjQH85iF5qPw6+u3t1qO5XDZzo4iDvm TnuWEcLaUFCgDCh3DchXF1JnFTy2EHaU9chPO8SmZ85eYV3qgi8nQOwqD5kjYvl+4vBM CUgWy6l0GVC55E7iSPQEbs/sGnsm5vSEpDxrXP1IPkeGynCv7jxc2jZNcfqF+Xjidh/K gsTBL8hHNLSGKr1+fVP9QejTVgbbnajwHl7vbgVJ4Bes1RSatJEJwSJXAYxgteXKqDEu FKkw== X-Gm-Message-State: AOJu0YxYxxvok8V1RDuwzFMdJKmRQX9jwLd0tpRSqQ35wbAKFlKwcD9j qUL4WTUHbRrDbUHLg7y6Elmi7F9oCm4= X-Google-Smtp-Source: AGHT+IG/rxMVlkCB9cE/GnGt/Uh6hI6Qx34yP2sHrosJXhsK6g98pVrfRXdpCjFgIhFS0Ophc2iXsw== X-Received: by 2002:a2e:88ce:0:b0:2cc:85a6:25a3 with SMTP id a14-20020a2e88ce000000b002cc85a625a3mr5202517ljk.2.1703557190516; Mon, 25 Dec 2023 18:19:50 -0800 (PST) Received: from guixsd.wugi.info ([88.201.161.72]) by smtp.gmail.com with ESMTPSA id e11-20020a05651c038b00b002cca609ef8bsm1633436ljp.111.2023.12.25.18.19.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Dec 2023 18:19:49 -0800 (PST) From: Oleg Pykhalov To: 62153@debbugs.gnu.org Subject: [PATCH 5/5] scripts: system: Build layered images. Date: Tue, 26 Dec 2023 05:18:57 +0300 Message-ID: <99155dabc366c37acb71f6624aa6e6025b3e571b.1703556298.git.go.wigust@gmail.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: References: MIME-Version: 1.0 X-Debbugs-Cc: Christopher Baines , Josselin Poiret , Ludovic Courtès , Mathieu Othacehe , Ricardo Wurmus , Simon Tournier , Tobias Geerinckx-Rice Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 62153 Cc: Oleg Pykhalov X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) * guix/scripts/system.scm (show-help, %docker-format-options, %options, %default-options, show-docker-format-options, show-docker-format-options/detailed, process-action): Handle '--max-layers' option. * gnu/system/image.scm (system-docker-image): Same. * gnu/image.scm ()[max-layers]: New record field. Change-Id: I2726655aefd6688b976057fd5a38e9972ebfc292 --- gnu/image.scm | 4 ++++ gnu/system/image.scm | 41 ++++++++++++++++++++++++++++------------- guix/scripts/system.scm | 28 ++++++++++++++++++++++++++-- 3 files changed, 58 insertions(+), 15 deletions(-) diff --git a/gnu/image.scm b/gnu/image.scm index 523653dd77..7fb06dec10 100644 --- a/gnu/image.scm +++ b/gnu/image.scm @@ -1,5 +1,6 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2020, 2022 Mathieu Othacehe +;;; Copyright © 2023 Oleg Pykhalov ;;; ;;; This file is part of GNU Guix. ;;; @@ -42,6 +43,7 @@ (define-module (gnu image) image-format image-platform image-size + image-max-layers image-operating-system image-partition-table-type image-partitions @@ -170,6 +172,8 @@ (define-record-type* (size image-size ;size in bytes as integer (default 'guess) (sanitize validate-size)) + (max-layers image-max-layers ;number of layers as integer + (default #false)) (operating-system image-operating-system) ; (partition-table-type image-partition-table-type ; 'mbr or 'gpt (default 'mbr) diff --git a/gnu/system/image.scm b/gnu/system/image.scm index b825892232..2cc1012893 100644 --- a/gnu/system/image.scm +++ b/gnu/system/image.scm @@ -5,6 +5,7 @@ ;;; Copyright © 2022 Denis 'GNUtoo' Carikli ;;; Copyright © 2022 Alex Griffin ;;; Copyright © 2023 Efraim Flashner +;;; Copyright © 2023 Oleg Pykhalov ;;; ;;; This file is part of GNU Guix. ;;; @@ -686,7 +687,8 @@ (define (image-with-label base-image label) (define* (system-docker-image image #:key - (name "docker-image")) + (name "docker-image") + (archiver tar)) "Build a docker image for IMAGE. NAME is the base name to use for the output file." (define boot-program @@ -731,6 +733,7 @@ (define* (system-docker-image image (use-modules (guix docker) (guix build utils) (gnu build image) + (srfi srfi-1) (srfi srfi-19) (guix build store-copy) (guix store database)) @@ -754,18 +757,30 @@ (define* (system-docker-image image #:register-closures? #$register-closures? #:deduplicate? #f #:system-directory #$os) - (build-docker-image - #$output - (cons* image-root - (map store-info-item - (call-with-input-file #$graph - read-reference-graph))) - #$os - #:entry-point '(#$boot-program #$os) - #:compressor '(#+(file-append gzip "/bin/gzip") "-9n") - #:creation-time (make-time time-utc 0 1) - #:system #$image-target - #:transformations `((,image-root -> "")))))))) + (when #$(image-max-layers image) + (setenv "PATH" + (string-join (list #+(file-append archiver "/bin") + #+(file-append gzip "/bin")) + ":"))) + (apply build-docker-image + (append (list #$output + (append (if #$(image-max-layers image) + '() + (list image-root)) + (map store-info-item + (call-with-input-file #$graph + read-reference-graph))) + #$os + #:entry-point '(#$boot-program #$os) + #:compressor + '(#+(file-append gzip "/bin/gzip") "-9n") + #:creation-time (make-time time-utc 0 1) + #:system #$image-target + #:transformations `((,image-root -> ""))) + (if #$(image-max-layers image) + (list #:root-system image-root + #:max-layers #$(image-max-layers image)) + '())))))))) (computed-file name builder ;; Allow offloading so that this I/O-intensive process diff --git a/guix/scripts/system.scm b/guix/scripts/system.scm index f85b663d64..a21ecd4d1e 100644 --- a/guix/scripts/system.scm +++ b/guix/scripts/system.scm @@ -58,6 +58,7 @@ (define-module (guix scripts system) #:use-module (guix scripts system reconfigure) #:use-module (guix build utils) #:use-module (guix progress) + #:use-module ((guix docker) #:select (%docker-image-max-layers)) #:use-module (gnu build image) #:use-module (gnu build install) #:autoload (gnu build file-systems) @@ -1053,6 +1054,8 @@ (define (show-help) (newline) (show-native-build-options-help) (newline) + (show-docker-format-options) + (newline) (display (G_ " -h, --help display this help and exit")) (display (G_ " @@ -1060,6 +1063,12 @@ (define (show-help) (newline) (show-bug-report-information)) +(define %docker-format-options + (list (option '("max-layers") #t #f + (lambda (opt name arg result) + (alist-cons 'max-layers (string->number* arg) + result))))) + (define %options ;; Specifications of the command-line options. (cons* (option '(#\h "help") #f #f @@ -1154,7 +1163,8 @@ (define %options (alist-cons 'list-installed (or arg "") result))) (append %standard-build-options %standard-cross-build-options - %standard-native-build-options))) + %standard-native-build-options + %docker-format-options))) (define %default-options ;; Alist of default option values. @@ -1175,7 +1185,8 @@ (define %default-options (label . #f) (volatile-image-root? . #f) (volatile-vm-root? . #t) - (graph-backend . "graphviz"))) + (graph-backend . "graphviz") + (max-layers . ,%docker-image-max-layers))) (define (verbosity-level opts) "Return the verbosity level based on OPTS, the alist of parsed options." @@ -1183,6 +1194,17 @@ (define (verbosity-level opts) (if (eq? (assoc-ref opts 'action) 'build) 3 1))) +(define (show-docker-format-options) + (display (G_ " + --help-docker-format list options specific to the docker image type."))) + +(define (show-docker-format-options/detailed) + (display (G_ " + --max-layers=N + Number of image layers")) + (newline) + (exit 0)) + ;;; ;;; Entry point. @@ -1245,6 +1267,7 @@ (define (process-action action args opts) ((docker-image) docker-image-type) (else image-type))) (image-size (assoc-ref opts 'image-size)) + (image-max-layers (assoc-ref opts 'max-layers)) (volatile? (assoc-ref opts 'volatile-image-root?)) (shared-network? @@ -1258,6 +1281,7 @@ (define (process-action action args opts) (image-with-label base-image label) base-image)) (size image-size) + (max-layers image-max-layers) (volatile-root? volatile?) (shared-network? shared-network?)))) (os (or (image-operating-system image) -- 2.41.0 From debbugs-submit-bounces@debbugs.gnu.org Mon Dec 25 21:41:10 2023 Received: (at 62153) by debbugs.gnu.org; 26 Dec 2023 02:41:10 +0000 Received: from localhost ([127.0.0.1]:55665 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rHxNW-0003vV-2G for submit@debbugs.gnu.org; Mon, 25 Dec 2023 21:41:10 -0500 Received: from mail-lf1-x129.google.com ([2a00:1450:4864:20::129]:38239) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rHxNT-0003v9-9g; Mon, 25 Dec 2023 21:41:08 -0500 Received: by mail-lf1-x129.google.com with SMTP id 2adb3069b0e04-50e79299da9so152231e87.1; Mon, 25 Dec 2023 18:41:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1703558454; x=1704163254; darn=debbugs.gnu.org; h=mime-version:user-agent:message-id:date:references:in-reply-to :subject:cc:to:from:from:to:cc:subject:date:message-id:reply-to; bh=StihbCwtZIccK/iQz2hr0HlNhJyiwYTRwsM7QPheyAs=; b=ULU/nUIifJJay0fxiGJR+qkzoaAy9twwWfKAn2eo6tYO3IMJ7gYPD6NL3akRAJXx7e x10x0aJsSKt63yK79ZJTIjRG/g6yTL9URf6TjpXFrWPVDO8e2SeXjYFZ2U+t2NVQ/1bp 60cUEa6LL/1mwJpKwvQGM4FPB/00x3BxhMpkiOf8QEiLvKvHuuVIbXoaJo6ssMkMgLhQ bXe/kJAjFL5mnFN99KITwxhtQPcK7MBhf52HOKDHuyN+M0as7dWFRNtFAh/HHtNuywPL BX79ALZM0UJ3VNPXUf+t/xvMubGv4NP+0J3ePcwO7InOg9oBxNLZjoQNlqh7zNyS2atW BomQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1703558454; x=1704163254; h=mime-version:user-agent:message-id:date:references:in-reply-to :subject:cc:to:from:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=StihbCwtZIccK/iQz2hr0HlNhJyiwYTRwsM7QPheyAs=; b=QHIhzVDgz5KbMp9+zGWTCx8rXZlIYU7v/lN41tLTQekGTznj+jDhDPyZ4rMsXHrkqC Gj8bnqWRw4eDFzJYcb1L17s6X8730e5HmF4/W6vCnnSoiVGHzVYZ3OvEcVw8gU1Zfv8P eellwul2HPAOe8O13KStnokpPuekN1IsM88DqXVvt6cmeYUZ/d5Vn/qqOu1bHamcJfcm eVfl7jqTR9RL8KH+Jnmw/8UbkugB1weXM6cd/8jXMwrMYjDM9ur9kMh4WA1NeFgxcuAj WStvrdu+Tcgd01/l7HoG/nvn5mJKdA65/tR9Fdl2Kai7/DBk22eRmn5JzuYfOC0C/sTu nTWA== X-Gm-Message-State: AOJu0YwhJzwsiOPYnO94A8xgXprumOgKZ7u04HPU3LukIwz/3YBkYDqd WjePD+KxaNBrVxjyxcKoq2Q= X-Google-Smtp-Source: AGHT+IHslTkt/P+oQU6CM7JZzMIOtx1bq5VtCsyGLum6EJxG/ogEd17Pjc8SXji60RTd5YJ2fkDOxg== X-Received: by 2002:a2e:8883:0:b0:2cc:9882:4ca2 with SMTP id k3-20020a2e8883000000b002cc98824ca2mr4667261lji.5.1703558454196; Mon, 25 Dec 2023 18:40:54 -0800 (PST) Received: from localhost ([88.201.161.72]) by smtp.gmail.com with ESMTPSA id ce26-20020a2eab1a000000b002ccabbc3c5asm1473979ljb.8.2023.12.25.18.40.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Dec 2023 18:40:53 -0800 (PST) From: Oleg Pykhalov To: Ludovic =?utf-8?Q?Court=C3=A8s?= Subject: Re: [bug#62153] Merging guix pack changes for Docker containers packaging In-Reply-To: <875y0p99c4.fsf@gnu.org> ("Ludovic =?utf-8?Q?Court=C3=A8s=22'?= =?utf-8?Q?s?= message of "Fri, 22 Dec 2023 23:11:07 +0100") References: <20230313003012.14325-1-go.wigust@gmail.com> <878r9xb2e6.fsf@gmail.com> <875y0p99c4.fsf@gnu.org> Date: Tue, 26 Dec 2023 05:40:52 +0300 Message-ID: <87il4l4rez.fsf@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 62153 Cc: 64173@debbugs.gnu.org, 62153@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hi Ludovic, Ludovic Court=C3=A8s writes: > Apologies for not replying earlier. I occasionally get reminded of the > fact that building single-layer images is a problem, but only now did I > take the time to look more closely at the latest version of these > patches. > > Oleg Pykhalov skribis: > >> I would like to merge 62153. After 64173 will be merge, merging 62153 >> is not possible without conflict resolving with Git. >> >> 64173 introduces =E2=80=98%docker-format-options=E2=80=99 variable. Wit= h this variable >> it's possible in 62153 to replace =E2=80=98--image-type=3Ddocker-layered= =E2=80=99 with >> =E2=80=98--docker-layers=3DN=E2=80=99 option, where: >> >> if =E2=80=98N=E2=80=99 is zero, then use current non layered format >> if =E2=80=98N=E2=80=99 is bigger than zero, then use layered format > > OK we should do that. However, the original submitter of #64173 > apparently dropped the ball as we were approaching the final version. > > Would you like to adopt it and submit/push a version that incorporates > the latest comments? > > Alternatively, we could do the opposite: merge the Docker layer patches > first, and then rebase the =E2=80=98%docker-format-options=E2=80=99 patch= , after which > we could add the =E2=80=98--docker-layers=E2=80=99 option. > > What=E2=80=99s your preference? [=E2=80=A6] Patches 64173 and 62153 (v5) have been sent to 62153. If you don't mind, I have changed the option naming to '--max-layers=3DN' instead of '--docker-layers=3DN' to align with the format of '--entry-point-argument' (without specifying Docker as the only image format that utilizes layers). I did not include code to check if 'N' is zero and use the current non-layered format. Instead, I opted for the default value of '#false' as it was easier to implement. Regards, Oleg. --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQJIBAEBCgAyFiEEcjhxI46s62NFSFhXFn+OpQAa+pwFAmWKPTQUHGdvLndpZ3Vz dEBnbWFpbC5jb20ACgkQFn+OpQAa+pzYpBAAuIqdacRLFs0TgY6PKw6yiUARQSRb AKT1b/9dryTHfxL9STKiHdgkSXzCK7SvfXX/SkQrACybakckunGf0BD8ZTCZ47i8 d3vLKOA8iRlewHStIVeuJUzzJgwb+CefTkkXX4WwEVrrfvdFbCqs6iDZPsGZzI90 KmYfRkbNwfwEGXV+efZMKZ982gs+56dCMbJk6rK9TYAXxx4zxQJdQ7zGHyG9J2I2 zAMK/YsfvmaVVzcgtDkiFn0FBZ1hrXtL2EipzEGs3VtUO0l1XdHmei4auTyaxpD5 xlMYEhSXrTlW0JxXzEEQigJMeoKECesj9CzO1C18ByVTnJNT5ABJAcwXKMO7JT7i iJlrlO1vce+u2XdeAlIaNU0225Nn21NZWoAce2O424wrIUvmtp0w835R+Zgp9OTR uan2o5XTYXTRp6F9IRdTiDORBF5C7T7vay+f7TVPBafJDMpz2it87FXp9FlICFPl HqNW56R0780AOl0N+Qg6UOlr8rQ2wlmjeIRCeLiSuQBvKE/eJzaXL1i72b/koHn6 c47tZbWbjTJqJgoDXC7QnJiUlvhlLvbR7c9cH+QKgj4Oi7IejG96prSZXrDli12z IVPERJM9UsPmFSNoqOlgaVChbda3i8gPJjzZro6TxFPlk7JorMlmsUURvMWYhi5z HAhQvJxZLviRjEY= =Dj7B -----END PGP SIGNATURE----- --=-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Wed Dec 27 13:15:14 2023 Received: (at 62153) by debbugs.gnu.org; 27 Dec 2023 18:15:14 +0000 Received: from localhost ([127.0.0.1]:37742 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rIYQz-0004zq-Rw for submit@debbugs.gnu.org; Wed, 27 Dec 2023 13:15:14 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:56364) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rIYQx-0004ar-Gi for 62153@debbugs.gnu.org; Wed, 27 Dec 2023 13:15:12 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rIYQq-0004bz-4o; Wed, 27 Dec 2023 13:15:04 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:Date:References:In-Reply-To:Subject:To: From; bh=0BN65qDMkFOlKWy7bYFrlvgs5yHkvJkmyc5sFLPP1N8=; b=iG50CZsl3jur2Qmv5FoH KIJfRpfdE8zZqeeHL/UHzyzQ6bY49luPcgJApAz/U6t74/idPi05SGbSNvjSghDdkZorOrLfZTCT5 JhdtVzlt4v2URZdkpaEEfnsUk9qnFn2RIwTQLpJg5tVSlRGgZiSc+zBLosBlnLn2rbzjOwZbfIk5f dvx3FyK74y2f9RZnraMPOuEGdbVdDEBnUFOvs8Tltiz7szHSXgsUqgkUcfggqR3t44hqNgjIsJ+a2 uiAseFfxZCT8JsXcmpgpM4sK7IV4AW+0X62Lzf197UOOIwqQGtBHL/xHQCHyEmynN3N8rumvoMM3r RhXgRqJ52Rta0w==; From: Mathieu Othacehe To: Oleg Pykhalov Subject: Re: [bug#62153] [PATCH 1/5] guix: pack: Add '--entry-point-argument' option. In-Reply-To: <9e2ecba897127f31c8c6fedd9ce20b6cdc0be4a2.1703556298.git.go.wigust@gmail.com> (Oleg Pykhalov's message of "Tue, 26 Dec 2023 05:18:53 +0300") References: <9e2ecba897127f31c8c6fedd9ce20b6cdc0be4a2.1703556298.git.go.wigust@gmail.com> Date: Wed, 27 Dec 2023 19:14:57 +0100 Message-ID: <87edf75x7i.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 62153 Cc: Josselin Poiret , Simon Tournier , Graham James Addis , Tobias Geerinckx-Rice , Ricardo Wurmus , Christopher Baines , Ludovic =?utf-8?Q?Court=C3=A8s?= , 62153@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hello Oleg, > + "A SRFI-37 opion parser for the --entry-point-argument option. The spec option -> option > +takes multiple occurances. The entries are used in the exec form for the orrurances -> occurrences, also use two spaces after a dot. > + ;; first entry is constructed by prefixing the entry-point with > + ;; the supplied index subsequent entries are taken from the subsequent -> , subsequent > + -A, --entry-point-argument=COMMAND/PARAMETER > + Value(s) to use for the Docker EntryPoint arguments. EntryPoint -> ENTRYPOINT Thanks, Mathieu From debbugs-submit-bounces@debbugs.gnu.org Wed Dec 27 13:17:05 2023 Received: (at 62153) by debbugs.gnu.org; 27 Dec 2023 18:17:05 +0000 Received: from localhost ([127.0.0.1]:37747 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rIYSn-00070S-Da for submit@debbugs.gnu.org; Wed, 27 Dec 2023 13:17:05 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:39568) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rIYSl-0006zv-SD for 62153@debbugs.gnu.org; Wed, 27 Dec 2023 13:17:04 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rIYSh-0004s1-OR; Wed, 27 Dec 2023 13:16:59 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:Date:References:In-Reply-To:Subject:To: From; bh=TDDqZgwOi1rV9V9txgX+W7eydXwTNuv+qtBVUNOHFj8=; b=d4iYA3KNcbDImd2BDThn +Y9Ab12DvQxaj7UYrWWgaKtikR2zwPak3M/X0SdyJcU5eiDjmvnUc/ULSO/jmyygRTrEGOMkxYsDB Tiy1fu7JLXjwkg8ycUIIU8fAUAJU66Rr/1I2q4PKmZH9kfYTXorblB0m/+4mQmkw9t2rW9vHpdWhL 6VkHpowmPgu6KJzeGhSoqJmd26lboMJZr/mfBqmLGoif+i4ywwaLtHRauvNAHzHTPUpW+eUnghNqM 4bH05f9HuUPPDh8o7dmQrMCVR4HPyKTYtSaGF1sFT4U4ZtOywFdNpWMx9lFWikhz4sVQx+gqVjTdH Ix3wm1lQ82EfkQ==; From: Mathieu Othacehe To: Oleg Pykhalov Subject: Re: [bug#62153] [PATCH 1/5] guix: pack: Add '--entry-point-argument' option. In-Reply-To: <87edf75x7i.fsf@gnu.org> (Mathieu Othacehe's message of "Wed, 27 Dec 2023 19:14:57 +0100") References: <9e2ecba897127f31c8c6fedd9ce20b6cdc0be4a2.1703556298.git.go.wigust@gmail.com> <87edf75x7i.fsf@gnu.org> Date: Wed, 27 Dec 2023 19:16:55 +0100 Message-ID: <877ckz5x48.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 62153 Cc: Josselin Poiret , Simon Tournier , Graham James Addis , Tobias Geerinckx-Rice , Ricardo Wurmus , Christopher Baines , Ludovic =?utf-8?Q?Court=C3=A8s?= , 62153@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) + (and=> entry-point + (lambda (entry-point) + (cons* (string-append prefix "/" entry-point) + entry-point-argument)))) ^ ^ There are also two tabulations here that need to be removed. Thanks, Mathieu From debbugs-submit-bounces@debbugs.gnu.org Wed Dec 27 15:16:06 2023 Received: (at 62153) by debbugs.gnu.org; 27 Dec 2023 20:16:06 +0000 Received: from localhost ([127.0.0.1]:37796 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rIaJx-0005qz-OX for submit@debbugs.gnu.org; Wed, 27 Dec 2023 15:16:06 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:55012) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rIaJw-0005gY-3L for 62153@debbugs.gnu.org; Wed, 27 Dec 2023 15:16:04 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rIaJn-0006Vj-Tb; Wed, 27 Dec 2023 15:15:58 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:Date:References:In-Reply-To:Subject:To: From; bh=X+pN0Ep+Z4rc9ddT9CKniHg8GyXMscJIOKnA8AHA6uA=; b=h0tVxBQ7A9FhKObP5Hq6 BZVYQ0OAYrjmtF+9NGqc3J8HneJP/0xbgcku04TYkMWQH/K8tmjl5CGlqMWqdU8Cvn4GmnE+lV6z1 DK48OyYn3cYO5G7dgN5/CaSP26pduIckyx8t26MS6LXD9jRi+dSpVy2IIMLNj8O0XqLtiP1O+o2+n z8y9QnlUL+Z2WrGq8/eaSng/iIPBU2h6w2j56XQT6UwI3V8tcdbHCZI6817nDdl5auQ5PxO7LEtMk vj19R/4Wb2s+pCdU2CG9eVrqh4nFZfNpbI6kLysMkeG5kBa3B/Sr6S9QyQjS/QhrpbysH0CsHk/PJ +RW+dh6iREv3xA==; From: Mathieu Othacehe To: Oleg Pykhalov Subject: Re: [bug#62153] [PATCH 3/5] guix: docker: Build layered images. In-Reply-To: <49f8906ba06af461e17d9badcbbf3967f1a8be3b.1703556298.git.go.wigust@gmail.com> (Oleg Pykhalov's message of "Tue, 26 Dec 2023 05:18:55 +0300") References: <49f8906ba06af461e17d9badcbbf3967f1a8be3b.1703556298.git.go.wigust@gmail.com> Date: Wed, 27 Dec 2023 21:15:51 +0100 Message-ID: <8734vn5rm0.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 62153 Cc: Josselin Poiret , Simon Tournier , Ludovic =?utf-8?Q?Court=C3=A8s?= , Tobias Geerinckx-Rice , Ricardo Wurmus , Christopher Baines , 62153@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) > +When MAX-LAYERS is not false build layered image, providing a Docker > +image with many of the store paths being on their own layer to improve sharing > +between images. "many of the store paths being on their own layer" is a big vague. It could be rephrased to "store paths splitted in their own layers" or so. Thanks, Mathieu From debbugs-submit-bounces@debbugs.gnu.org Wed Dec 27 15:25:42 2023 Received: (at 62153) by debbugs.gnu.org; 27 Dec 2023 20:25:42 +0000 Received: from localhost ([127.0.0.1]:37802 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rIaTG-0006OU-0R for submit@debbugs.gnu.org; Wed, 27 Dec 2023 15:25:42 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:44760) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rIaTD-0006OG-Jl for 62153@debbugs.gnu.org; Wed, 27 Dec 2023 15:25:40 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rIaT6-0000j1-7Y; Wed, 27 Dec 2023 15:25:33 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:Date:References:In-Reply-To:Subject:To: From; bh=5cVuvXeCzi2m8XPyZcRGoMEkGT2SyvxYbd/czJI0Ljw=; b=HMlojZdCZWuklz43Gpc4 xYAYtqagnWMlt60Oc6vt+F/remuRuh99yXBVX4yEgrhZW7lCViMZhaHefKJlGuJlo+zooREGsvnY1 OAI/S+zJowjxcC0Ol7BAUMGQk2MlSfHkZixXdB8rVPSmwHWR/OpEVmBvpa5bcVBHmCcS7Bghj4KRI 0zU6zgPXaLjx+Z6Je/l9gKYpP2NhvJUAztsua7vt5GfLMRcWJNpSD12I9O3HcPsYKD4pNgzeNJ7FF wSkd4mv4bvexCsTSMhmWF21/14U9wefdXZdHkbfeTlYd5j0XWcYiDtkGRnT8vxj0XQgOahQKbxvCE 0hJEUv9p6bNMOg==; From: Mathieu Othacehe To: Oleg Pykhalov Subject: Re: [bug#62153] [PATCH 4/5] guix: pack: Build layered images. In-Reply-To: <77bae2565cb5506342028896e1a45c757f3bad51.1703556298.git.go.wigust@gmail.com> (Oleg Pykhalov's message of "Tue, 26 Dec 2023 05:18:56 +0300") References: <77bae2565cb5506342028896e1a45c757f3bad51.1703556298.git.go.wigust@gmail.com> Date: Wed, 27 Dec 2023 21:25:02 +0100 Message-ID: <87y1df4cm9.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 62153 Cc: Josselin Poiret , Simon Tournier , Ludovic =?utf-8?Q?Court=C3=A8s?= , Tobias Geerinckx-Rice , Ricardo Wurmus , Christopher Baines , 62153@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) > +If MAX-LAYERS is not false, the image will with many of the store paths being > +on their own layer to improve sharing between images." Same comment as in previous patch, "many" is a bit vague. "the image will be splitted in up to MAX-LAYERS layers" or so could work. > + (check (gexp->derivation "check-tarball" > + (with-imported-modules '((guix build utils)) You could rearrange as: --8<---------------cut here---------------start------------->8--- (check (gexp->derivation "check-tarball" (with-imported-modules '((guix build utils)) --8<---------------cut here---------------end--------------->8--- to reduce the indentation of this block: > + #~(begin > + (use-modules (guix build utils) > + (ice-9 match)) ... > + (string=? (string-append #$%bootstrap-guile "/bin") > + (pk 'binlink (readlink bin))) > + (string=? (string-append #$profile "/bin/guile") > + (pk 'guilelink (readlink "bin/Guile")))) Left over pk's? Thanks, Mathieu From debbugs-submit-bounces@debbugs.gnu.org Wed Dec 27 15:29:20 2023 Received: (at 62153) by debbugs.gnu.org; 27 Dec 2023 20:29:21 +0000 Received: from localhost ([127.0.0.1]:37808 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rIaWm-0006TT-KY for submit@debbugs.gnu.org; Wed, 27 Dec 2023 15:29:20 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:38112) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rIaWk-0006TH-Re for 62153@debbugs.gnu.org; Wed, 27 Dec 2023 15:29:19 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rIaWf-0001WA-5n; Wed, 27 Dec 2023 15:29:13 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:Date:References:In-Reply-To:Subject:To: From; bh=t5pHR3n19v041msXksBY8fNd05ge23kK5qEp75wWAf0=; b=E6OeAR1CpiA1iaWBF1Aj liNyPpC9CWG9OA5jt01D6zuiEGiL7IoCQyJEj1jEjbVBpPXcRZtxPOSa8TIGN9qKlO+kDTIDpaRkM wrZuPK1rxbgbE2QfZmFw+yE4/R8TsTU1pxHyChH9NCczcuFWTI9jWm+buHcmKVgZkSstu8i73+nxm 9NmybDZNcP15aa4NMXM2ASJp3ZQrGGEphdDvCfJJKCXnhbfJSaFUNcmrwnJeBReS19SxXoM96WPa0 pJiLpVDni40AHLFeanowadRtxInu9qmoCzTst2O4yT+qVPG6BlVTUI7LBhqbnYBlk7dZ+o7wPEn7V 4BwpUtaHUW0O+g==; From: Mathieu Othacehe To: Oleg Pykhalov Subject: Re: [bug#62153] [PATCH 5/5] scripts: system: Build layered images. In-Reply-To: <99155dabc366c37acb71f6624aa6e6025b3e571b.1703556298.git.go.wigust@gmail.com> (Oleg Pykhalov's message of "Tue, 26 Dec 2023 05:18:57 +0300") References: <99155dabc366c37acb71f6624aa6e6025b3e571b.1703556298.git.go.wigust@gmail.com> Date: Wed, 27 Dec 2023 21:29:05 +0100 Message-ID: <87tto34cfi.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 62153 Cc: Josselin Poiret , Simon Tournier , Ludovic =?utf-8?Q?Court=C3=A8s?= , Tobias Geerinckx-Rice , Ricardo Wurmus , Christopher Baines , 62153@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Other than the few cosmetic remarks, the series looks great. I tested producing multi-layers pack and images with success. You may want to wait for Ludo's opinion as a reviewer of the v4, but as far as I am concerned, I think that you can directly proceed with the cosmetic issues fixed. Thanks, Mathieu From debbugs-submit-bounces@debbugs.gnu.org Mon Jan 08 11:49:20 2024 Received: (at 62153) by debbugs.gnu.org; 8 Jan 2024 16:49:20 +0000 Received: from localhost ([127.0.0.1]:37392 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rMsoR-00045f-QT for submit@debbugs.gnu.org; Mon, 08 Jan 2024 11:49:20 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:51266) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rMsoQ-00045U-Jx for 62153@debbugs.gnu.org; Mon, 08 Jan 2024 11:49:18 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rMsoE-0001hc-SV; Mon, 08 Jan 2024 11:49:06 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:Date:References:In-Reply-To:Subject:To: From; bh=eo3VYvCSLKY57Md8UnsnVT9KJwA0JrQInOLMvNgGZ2U=; b=pwfCG/FFlhM0LgbPKOCU eP1do93s6yW/xnRthrb6SjbzKgQeN1XJqsEtifwZvA7Gvkyj1KSz7P6aO4c83L6/Eq1gT8FSlLHzc yswKa3+d4ZSpPSCEHpwVlq2od766uxCVgFj4Lywoh9fsjHtOp7Rxd9q4n5NRE0a9faFNEDarlgetZ V/zM0OBVi6um+N6PJTGUyIfBBDDSne0DKMVk2G++lxmTbb5PDQot2MMoUrNu/vfl1gZ26qfeVM3AG hC/Pl72B0v/khIcHoS6YeeFfqmLFgvuaemLY52CPuvdppfp9qia78rNDzRV6tez2DcJFl/lcLCkc8 22nhzcDc1Oajag==; From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Mathieu Othacehe Subject: Re: [bug#62153] [PATCH 5/5] scripts: system: Build layered images. In-Reply-To: <87tto34cfi.fsf@gnu.org> (Mathieu Othacehe's message of "Wed, 27 Dec 2023 21:29:05 +0100") References: <99155dabc366c37acb71f6624aa6e6025b3e571b.1703556298.git.go.wigust@gmail.com> <87tto34cfi.fsf@gnu.org> Date: Mon, 08 Jan 2024 17:49:03 +0100 Message-ID: <87jzoj4vpc.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 62153 Cc: Josselin Poiret , Simon Tournier , Oleg Pykhalov , Tobias Geerinckx-Rice , Ricardo Wurmus , Christopher Baines , 62153@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hello Oleg and all, Mathieu Othacehe skribis: > Other than the few cosmetic remarks, the series looks great. I tested > producing multi-layers pack and images with success. > > You may want to wait for Ludo's opinion as a reviewer of the v4, but as > far as I am concerned, I think that you can directly proceed with the > cosmetic issues fixed. I haven=E2=80=99t been able to test it yet, but I agree with Mathieu=E2=80= =99s suggestions and I agree that you can proceed with this small fixes. As a bonus, consider adding an entry in =E2=80=98etc/news.scm=E2=80=99: it= =E2=80=99s definitely news-worthy! Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Tue Jan 09 07:59:16 2024 Received: (at 62153-done) by debbugs.gnu.org; 9 Jan 2024 12:59:16 +0000 Received: from localhost ([127.0.0.1]:38633 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rNBhM-0000Qh-DW for submit@debbugs.gnu.org; Tue, 09 Jan 2024 07:59:16 -0500 Received: from mail-lj1-x235.google.com ([2a00:1450:4864:20::235]:40300) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rNBhI-0000QQ-L5 for 62153-done@debbugs.gnu.org; Tue, 09 Jan 2024 07:59:15 -0500 Received: by mail-lj1-x235.google.com with SMTP id 38308e7fff4ca-2cd6c0b7dbbso814931fa.1 for <62153-done@debbugs.gnu.org>; Tue, 09 Jan 2024 04:59:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1704805140; x=1705409940; darn=debbugs.gnu.org; h=mime-version:user-agent:message-id:date:references:in-reply-to :subject:cc:to:from:from:to:cc:subject:date:message-id:reply-to; bh=Ci7/Kefp3GXP2YBH9zOmvxTlH5NOjS59i6rcQPHBTC4=; b=JDXhqB/ypT+693L619nPADyfOPxNEP4QY3Lg1qMfHZhBVCgnvQ96K1h7mRMiSDM1pu KoyBB9zgFFQWs/ZH/3t1sdozgYm7kVqi/AIGScaiThB4otSkJuXIFo4sFBKoZ5lFXhF6 cSXmH/+7C1vCobPXaxAiVA6xcbA51E21eZaDGbeqDWAfvstH7s+VrKQwPcCPs4uO6BYB GtsXXTmnsld3RUTJtnqtU/nNaf+bZ5IPCk3GjFOuv1K8RyNVe84NGbhzU9G4rDDSTweO 7QMfyJjumBKUc3YZEcHfleCalhK143ducGJpCkDLMbewijAjCdTOJjOychJdWsMPoNcj ZNhg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704805140; x=1705409940; h=mime-version:user-agent:message-id:date:references:in-reply-to :subject:cc:to:from:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=Ci7/Kefp3GXP2YBH9zOmvxTlH5NOjS59i6rcQPHBTC4=; b=C7q+06HFxqa/r2iEsny4itE940SD2yXXn7e0BGa5RKN4pG1dtJ7qy2jNY+UquAE238 HxRElqf9VmRrrTIpztyBMVg4cb6QRyqZXdqsxUStgaZ4+dKPWu3JNuFndbKQBbQvFuCZ cYcVNZQnT6hE/MmRKek3IXFgutZ0T7uchpiQrkAiOYO9UJKDrhsquzLJymsQoNhkiqM3 UyrFm5wVfNJtMmFlXCL68ch4hGHzUVko+PBQvq+87aIfTwcEAZViRSccZvLn6Z2RuV/s k8YfpmFD2l8WMTpbr1+LZQmjGhldEBhI1oTzI1D+kItiUNX+0aEeoZZzDznRALWYndSG 53bg== X-Gm-Message-State: AOJu0YxH/iCi5sBWPGqKIUvC+idGtFwdi7SxQTqQ6P5WSXo437i5Fo29 5JLa8KgKqF1LWYXTP9PYSUI= X-Google-Smtp-Source: AGHT+IHwuRCxwwSh+2UhbknSM01OyqxjaMFlRvP2eeTsNDvnvJCG6BRGCFZV3i+xn0FG4JMzitI4Og== X-Received: by 2002:a2e:3213:0:b0:2cd:f45:f150 with SMTP id y19-20020a2e3213000000b002cd0f45f150mr4779937ljy.1.1704805139572; Tue, 09 Jan 2024 04:58:59 -0800 (PST) Received: from localhost ([88.201.161.72]) by smtp.gmail.com with ESMTPSA id u7-20020a05651c140700b002ccf414d47csm419350lje.24.2024.01.09.04.58.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 09 Jan 2024 04:58:59 -0800 (PST) From: Oleg Pykhalov To: Ludovic =?utf-8?Q?Court=C3=A8s?= Subject: Re: [bug#62153] [PATCH 5/5] scripts: system: Build layered images. In-Reply-To: <87jzoj4vpc.fsf@gnu.org> ("Ludovic =?utf-8?Q?Court=C3=A8s=22'?= =?utf-8?Q?s?= message of "Mon, 08 Jan 2024 17:49:03 +0100") References: <99155dabc366c37acb71f6624aa6e6025b3e571b.1703556298.git.go.wigust@gmail.com> <87tto34cfi.fsf@gnu.org> <87jzoj4vpc.fsf@gnu.org> Date: Tue, 09 Jan 2024 15:58:58 +0300 Message-ID: <877cki1x4d.fsf@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 62153-done Cc: Josselin Poiret , Simon Tournier , Mathieu Othacehe , Tobias Geerinckx-Rice , Ricardo Wurmus , 62153-done@debbugs.gnu.org, Christopher Baines X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hello Ludovic, Ludovic Court=C3=A8s writes: >> Other than the few cosmetic remarks, the series looks great. I tested >> producing multi-layers pack and images with success. >> >> You may want to wait for Ludo's opinion as a reviewer of the v4, but as >> far as I am concerned, I think that you can directly proceed with the >> cosmetic issues fixed. > > I haven=E2=80=99t been able to test it yet, but I agree with Mathieu=E2= =80=99s > suggestions and I agree that you can proceed with this small fixes. > > As a bonus, consider adding an entry in =E2=80=98etc/news.scm=E2=80=99: i= t=E2=80=99s definitely > news-worthy! All Mathieu's suggestions applied, etc/news.scm from v4 slightly modified accordingly. Everything pushed to master. Thanks everyone for helping adding this feature. Oleg. --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQJIBAEBCgAyFiEEcjhxI46s62NFSFhXFn+OpQAa+pwFAmWdQxIUHGdvLndpZ3Vz dEBnbWFpbC5jb20ACgkQFn+OpQAa+pxzsg/+NCIxAqM1ZRgHUmq4il9DhWwT7dxr ibo6xCyqQtJe+9KVU5r2HU9ARZeA0MJWUKTC7HbGsztrSu5FidzJIDDFhhblGx95 JhlGPGKMKZpBaeGFnQl9yJTo0MjHnFlPlCiOIxXNSrsRe2W4qwLyrFBdUsnkkZ8U O//avuJUNojiAipSIo/rDGYTTQQeoJe3zpwMlGA/YedQVL216QSUDIcUVs8+X2iJ 28CxqfNKGnW/XycPH81i/uZ3z4z6nv2y2FUk+u9mDho2QbAj6y+02siAwLMAMg2N JMfW5s0YtIhXLgSzIBrz+T3objHh/NAqdN05KqtLhcVziYv6HzM/5mh5y8Fw8bwX chpUKV7JqWyHUYbQJRZ1G6A0Xkku1sn/EN0HANa/teG9wXMN/DU/Our4z6db9IHb T5NeegE4LO9VAg8+K8QmQq4toXyAicU5Cked2ar3zu2qDq37DYmk9XgY3yc97SqS AW15wb//sJJki3nER+rQXmEFrkFlWGaNxlLK81oJP3YKYxxiraFOTugVF6NFOMLR uwkK9h0jvKt7+8z+5JjReMggqNOn79sYx2vQTBiwA/iSv2rq8/6s/FWUTkRJ1ua4 nVYK6k67Wyc+V8soGxJSAu13bwih56Ji0geb2l91S4SXehfnTpNP2iV39q+mgtDZ 2w+tcjQp7WxGBxg= =/NHv -----END PGP SIGNATURE----- --=-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Thu Jan 18 09:55:52 2024 Received: (at 62153) by debbugs.gnu.org; 18 Jan 2024 14:55:52 +0000 Received: from localhost ([127.0.0.1]:55927 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rQTo8-0002KN-3a for submit@debbugs.gnu.org; Thu, 18 Jan 2024 09:55:52 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:50760) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rQTo6-0002FV-T3 for 62153@debbugs.gnu.org; Thu, 18 Jan 2024 09:55:51 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rQTnz-0001xZ-4W; Thu, 18 Jan 2024 09:55:43 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:Date:References:In-Reply-To:Subject:To: From; bh=6fjABtUfkZlmiDalv7TdOvTnN3duYyalgFaqihTKZZs=; b=iJ2YTcrbSPXqbg7Aej05 GAxclE+rAehTceqcz88f3A2K1Cd/qxxFb/uqUmzNW9aiq23Y9Qk4KLyXMrbDYMKok8QSN9Tqfv4hL /3cNQbHlDOTpl9lkBBZM7jsBkiVv+28MtZ4uXKjif6uIE71bYPIJ+RKeWIb5v0nZg6pSY9/VdIuFk npG4S58Cej7q1SC57qKi4qJn5kysvsOGWwBpB/WbK1MkIsbX5nF3mJWft3UPuHTTSWuMwFmmFCzy5 KrDDdiRVlJTwjeLgSiP29zE+v2teuE+nymw4y75FmSyeBbX0ZkWo/etuQswtsqWzNdLUtmrF4s5GW KjRAxbGgajI/1w==; From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Oleg Pykhalov Subject: Re: [bug#62153] [PATCH 3/5] guix: docker: Build layered images. In-Reply-To: <49f8906ba06af461e17d9badcbbf3967f1a8be3b.1703556298.git.go.wigust@gmail.com> (Oleg Pykhalov's message of "Tue, 26 Dec 2023 05:18:55 +0300") References: <49f8906ba06af461e17d9badcbbf3967f1a8be3b.1703556298.git.go.wigust@gmail.com> Date: Thu, 18 Jan 2024 15:55:39 +0100 Message-ID: <871qaeeln8.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 62153 Cc: Josselin Poiret , Simon Tournier , Mathieu Othacehe , Tobias Geerinckx-Rice , Ricardo Wurmus , Christopher Baines , 62153@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hi Oleg! Oleg Pykhalov skribis: > +;; The maximum number of layers allowed in a Docker image is typically a= round > +;; 128, although it may vary depending on the Docker daemon. However, we > +;; recommend setting the limit to 100 to ensure sufficient room for futu= re > +;; extensions. > +(define %docker-image-max-layers > + #f) It just occurred to me that the meaning of #f is unclear here; also, the manual does not specify the default value of =E2=80=98--max-layers=E2=80=99. Should we: 1. Set =E2=80=98%docker-image-max-layers=E2=80=99 to an integer, maybe 10= 0, in accordance with the comment above? 2. Clarify in the manual what the default is, and explain that users can pass =E2=80=98--max-layers=3D1=E2=80=99 if they want, well, a sing= le layer. Thoughts? Ludo=E2=80=99. From unknown Fri Jun 20 07:16:26 2025 Received: (at fakecontrol) by fakecontrolmessage; To: internal_control@debbugs.gnu.org From: Debbugs Internal Request Subject: Internal Control Message-Id: bug archived. Date: Fri, 16 Feb 2024 12:24:10 +0000 User-Agent: Fakemail v42.6.9 # This is a fake control message. # # The action: # bug archived. thanks # This fakemail brought to you by your local debbugs # administrator