GNU bug report logs - #60406
[PATCH 0/2] Add DwarFS

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 60406 in the body.
You can then email your comments to 60406 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Pierre Neidhardt <mail <at> ambrevar.xyz>
To: guix-patches <at> gnu.org
Subject: [PATCH 0/2] Add DwarFS
Date: Thu, 29 Dec 2022 15:50:24 +0100

[Message part 1 (text/plain, inline)]

This adds the dwarfs package.

1. For libdwarf, Guix lint complains that the source archive fails to be
extracted, complaining that 3600 is not a valid date.  Another source
archive is fetched from a fallback source instead.

2. For dwarfs, I've disabled the tests because 1 test fails on 'modprobe
fuse'.  If someone knows how to disable a specific test in the
'googletest' framework, please let me know.

Pierre Neidhardt (2):
  gnu: Add libdwarf.
  gnu: Add dwarfs.

 gnu/packages/elf.scm          |  25 ++++++++-
 gnu/packages/file-systems.scm | 100 ++++++++++++++++++++++++++++++++++
 2 files changed, 124 insertions(+), 1 deletion(-)


base-commit: 77d4bff94c6918eb0c0ccd20ee610e6dfc823aec
-- 
2.38.1

[signature.asc (application/pgp-signature, inline)]

Message #8 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Pierre Neidhardt <mail <at> ambrevar.xyz>
To: guix-patches <at> gnu.org
Cc: 60406 <at> debbugs.gnu.org
Subject: [PATCH 1/2] gnu: Add libdwarf.
Date: Thu, 29 Dec 2022 15:55:04 +0100

[Message part 1 (text/plain, inline)]

* gnu/packages/elf.scm (libdwarf): New variable.
---
 gnu/packages/elf.scm | 25 ++++++++++++++++++++++++-
 1 file changed, 24 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/elf.scm b/gnu/packages/elf.scm
index 8cc2ff3637..7f2865f17f 100644
--- a/gnu/packages/elf.scm
+++ b/gnu/packages/elf.scm
@@ -31,7 +31,7 @@ (define-module (gnu packages elf)
   #:use-module (guix packages)
   #:use-module (guix download)
   #:use-module (guix build-system gnu)
-  #:use-module ((guix licenses) #:select (gpl3+ lgpl3+ lgpl2.0+))
+  #:use-module ((guix licenses) #:select (gpl3+ lgpl3+ lgpl2.0+ lgpl2.1 gpl2 bsd-2))
   #:use-module (gnu packages)
   #:use-module (gnu packages autotools)
   #:use-module (gnu packages compression)
@@ -282,3 +282,26 @@ (define-public patchelf
     (properties
      '((release-monitoring-url . "https://github.com/NixOS/patchelf/releases")))
     (license gpl3+)))
+
+(define-public libdwarf
+  (package
+    (name "libdwarf")
+    (version "0.5.0")
+    (source (origin
+              (method url-fetch)
+              (uri (string-append "https://www.prevanders.net/libdwarf-"
+                                  version ".tar.xz"))
+              (sha256
+               (base32
+                "1a8j05z0rxf5ab9p5av2hqz6kxl8hfgar8h1387a0zric0n85yhi"))))
+    (build-system gnu-build-system)
+    (arguments
+     '(#:configure-flags '("--enable-shared")))
+    (native-inputs (list python))
+    (inputs (list elfutils))
+    (home-page "https://www.prevanders.net/dwarf.html")
+    (synopsis "Handle DWARF debugging information")
+    (description "@code{libdwarf} is a library that handles the DWARF
+debugging information format.")
+    ;; See https://www.prevanders.net/dwarflicense.html:
+    (license (list lgpl2.1 gpl2 bsd-2))))
-- 
2.38.1

[signature.asc (application/pgp-signature, inline)]

Message #11 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Pierre Neidhardt <mail <at> ambrevar.xyz>
To: guix-patches <at> gnu.org
Cc: 60406 <at> debbugs.gnu.org
Subject: [PATCH 2/2] gnu: Add dwarfs.
Date: Thu, 29 Dec 2022 15:55:37 +0100

[Message part 1 (text/plain, inline)]

* gnu/packages/file-systems.scm (dwarfs): New variable.
---
 gnu/packages/file-systems.scm | 100 ++++++++++++++++++++++++++++++++++
 1 file changed, 100 insertions(+)

diff --git a/gnu/packages/file-systems.scm b/gnu/packages/file-systems.scm
index 3d45634169..e268c52f06 100644
--- a/gnu/packages/file-systems.scm
+++ b/gnu/packages/file-systems.scm
@@ -46,26 +46,36 @@ (define-module (gnu packages file-systems)
   #:use-module (gnu packages admin)
   #:use-module (gnu packages attr)
   #:use-module (gnu packages autotools)
+  #:use-module (gnu packages backup)
   #:use-module (gnu packages base)
   #:use-module (gnu packages bash)
   #:use-module (gnu packages bison)
+  #:use-module (gnu packages boost)
   #:use-module (gnu packages check)
   #:use-module (gnu packages compression)
+  #:use-module (gnu packages cpp)
   #:use-module (gnu packages crypto)
   #:use-module (gnu packages curl)
   #:use-module (gnu packages cyrus-sasl)
   #:use-module (gnu packages datastructures)
+  #:use-module (gnu packages digest)
   #:use-module (gnu packages documentation)
   #:use-module (gnu packages docbook)
+  #:use-module (gnu packages elf)
   #:use-module (gnu packages flex)
   #:use-module (gnu packages gawk)
   #:use-module (gnu packages glib)
   #:use-module (gnu packages gnupg)
   #:use-module (gnu packages golang)
   #:use-module (gnu packages guile)
+  #:use-module (gnu packages jemalloc)
   #:use-module (gnu packages kerberos)
+  #:use-module (gnu packages libevent)
   #:use-module (gnu packages libffi)
+  #:use-module (gnu packages libunwind)
   #:use-module (gnu packages linux)
+  #:use-module (gnu packages maths)
+  #:use-module (gnu packages man)
   #:use-module (gnu packages nfs)
   #:use-module (gnu packages onc-rpc)
   #:use-module (gnu packages openldap)
@@ -73,6 +83,8 @@ (define-module (gnu packages file-systems)
   #:use-module (gnu packages perl)
   #:use-module (gnu packages photo)
   #:use-module (gnu packages pkg-config)
+  #:use-module (gnu packages popt)
+  #:use-module (gnu packages pretty-print)
   #:use-module (gnu packages python)
   #:use-module (gnu packages python-crypto)
   #:use-module (gnu packages python-web)
@@ -1790,3 +1802,91 @@ (define-public fuse-overlayfs
     (description "This package provides an implementation of overlay+shiftfs
 in FUSE for rootless containers.")
     (license license:gpl3)))
+
+(define-public dwarfs
+  (package
+    (name "dwarfs")
+    (version "0.6.2")
+    (source (origin
+              (method url-fetch)
+              ;; The release archive is needed so that version.h is included.
+              (uri (string-append "https://github.com/mhx/dwarfs/releases/download/v"
+                                  version "/dwarfs-" version ".tar.xz"))
+              (sha256
+               (base32
+                "1kncxf85gsj3anck8ccjmxn2azp5ifqbgkiky2kharmvphkbmfcv"))
+              (snippet
+               #~(begin
+                   (use-modules (guix build utils))
+                   ;; Prefer system libraries instead of submodules.
+                   ;; TODO: Package fbthrift.
+                   ;; TODO: Can we use Guix own folly?  There is no CMake option for it.
+                   ;; TODO: Package parallel-hashmap.
+                   (for-each delete-file-recursively
+                             '(;; "fbthrift"
+                               ;; "folly"
+                               ;; "parallel-hashmap"
+                               "xxHash"
+                               "zstd"))))))
+    (build-system cmake-build-system)
+    (arguments
+     '(#:tests? #f ; TODO: 1 test fails because 'modprobe fuse' needs privileged access.
+       #:configure-flags
+       (list "-DPREFER_SYSTEM_ZSTD=ON"
+             "-DPREFER_SYSTEM_XXHASH=ON"
+             "-DPREFER_SYSTEM_GTEST=ON"
+             "-DWITH_TESTS=ON"
+             ;; Disable man pages since ronn fails to run without hpricot.
+             "-DWITH_MAN_PAGES=OFF")))
+    (native-inputs
+     (list
+      ;; FIXME: Building with ronn fail because hpricot is missing.
+      ;; ronn
+      folly googletest libdwarf libevent pkg-config))
+    (inputs
+     (list
+      boost
+      double-conversion
+      fmt
+      fuse-3
+      gflags
+      jemalloc
+      libarchive
+      libunwind
+      lz4
+      openssl
+      xxhash
+      xz
+      `(,zstd "lib")))
+    (home-page "https://github.com/mhx/dwarfs")
+    (synopsis "Fast high compression read-only file system")
+    (description "DwarFS is a read-only file system with a focus on achieving
+very high compression ratios in particular for very redundant data.
+
+DwarFS also doesn't compromise on speed and for some cases it is on par with
+or performs better than SquashFS.  For the primary use case, DwarFS
+compression is an order of magnitude better than SquashFS compression, it's 6
+times faster to build the file system, it's typically faster to access files
+on DwarFS and it uses less CPU resources.
+
+Distinct features of DwarFS are:
+
+@itemize
+
+@item Clustering of files by similarity using a similarity hash function.  This
+makes it easier to exploit the redundancy across file boundaries.
+
+@item Segmentation analysis across file system blocks in order to reduce the
+size of the uncompressed file system.  This saves memory when using the
+compressed file system and thus potentially allows for higher cache hit rates
+as more data can be kept in the cache.
+
+@item Highly multi-threaded implementation.  Both the file system creation tool
+as well as the FUSE driver are able to make good use of the many cores of your
+system.
+
+@item Optional experimental Python scripting support to provide custom
+filtering and ordering functionality.
+
+@end itemize\n")
+    (license license:gpl3)))
-- 
2.38.1

[signature.asc (application/pgp-signature, inline)]

Message #20 received at 60406 <at> debbugs.gnu.org (full text, mbox):

From: Simon Tournier <zimon.toutoune <at> gmail.com>
To: Pierre Neidhardt <mail <at> ambrevar.xyz>, 60406 <at> debbugs.gnu.org
Subject: Re: [bug#60406] [PATCH 0/2] Add DwarFS
Date: Thu, 12 Jan 2023 14:58:25 +0100

Hi Pierre,

On jeu., 29 déc. 2022 at 15:50, Pierre Neidhardt <mail <at> ambrevar.xyz> wrote:

> 1. For libdwarf, Guix lint complains that the source archive fails to be
> extracted, complaining that 3600 is not a valid date.  Another source
> archive is fetched from a fallback source instead.

What is the issue the Git repository?

    https://github.com/davea42/libdwarf-code


Because that does not sound good: :-)

--8<---------------cut here---------------start------------->8---
$ guix download https://www.prevanders.net/libdwarf-0.5.0.tar.xz

Starting download of /tmp/guix-file.oWkq5d
From https://www.prevanders.net/libdwarf-0.5.0.tar.xz...
Bad Date header: 3600


Starting download of /tmp/guix-file.oWkq5d
From https://web.archive.org/web/20230112145352/https://www.prevanders.net/libdwarf-0.5.0.tar.xz...
following redirection to `https://web.archive.org/web/20221224112643/https://www.prevanders.net/libdwarf-0.5.0.tar.xz'...
 ….5.0.tar.xz  2.4MiB                                                                                              1.0MiB/s 00:02 [##################] 100.0%
/gnu/store/0xcriql6m20wcdzjv4a5fbcg7h0nmq8h-libdwarf-0.5.0.tar.xz
1a8j05z0rxf5ab9p5av2hqz6kxl8hfgar8h1387a0zric0n85yhi
--8<---------------cut here---------------end--------------->8---


Cheers,
simon

Message #23 received at 60406 <at> debbugs.gnu.org (full text, mbox):

From: Pierre Neidhardt <mail <at> ambrevar.xyz>
To: Simon Tournier <zimon.toutoune <at> gmail.com>, 60406 <at> debbugs.gnu.org
Subject: Re: [bug#60406] [PATCH 0/2] Add DwarFS
Date: Fri, 13 Jan 2023 13:09:51 +0100

[Message part 1 (text/plain, inline)]

Oh right, I can just use the GitHub repo.
Thank you Simon! 
Problem 1 solved then.

I'll push with a note about the tests.

[signature.asc (application/pgp-signature, inline)]

Message #26 received at 60406 <at> debbugs.gnu.org (full text, mbox):

From: Pierre Neidhardt <mail <at> ambrevar.xyz>
To: Simon Tournier <zimon.toutoune <at> gmail.com>, 60406 <at> debbugs.gnu.org
Subject: Re: [bug#60406] [PATCH 0/2] Add DwarFS
Date: Mon, 23 Jan 2023 14:20:03 +0100

[Message part 1 (text/plain, inline)]

Pushed with

04ade01beb95da45d63ca1f58126508d0491d99d

and

3a1b18aa4540e6f96ded0a98dd907a8033262582

Thanks!
Pierre

[signature.asc (application/pgp-signature, inline)]

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.