GNU bug report logs - #43637
[PATCH] gnu: packages: Add grokmirror.

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 43637 in the body.
You can then email your comments to 43637 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Kyle Meyer <kyle <at> kyleam.com>
To: guix-patches <at> gnu.org
Subject: [PATCH] gnu: packages: Add grokmirror.
Date: Sat, 26 Sep 2020 16:48:19 -0400

* gnu/packages/version-control.scm (grokmirror): New variable.
---
 gnu/packages/version-control.scm | 39 +++++++++++++++++++++++++++++++-
 1 file changed, 38 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/version-control.scm b/gnu/packages/version-control.scm
index 2ee8247071..68971c60f9 100644
--- a/gnu/packages/version-control.scm
+++ b/gnu/packages/version-control.scm
@@ -7,7 +7,7 @@
 ;;; Copyright © 2014, 2015, 2016 Mark H Weaver <mhw <at> netris.org>
 ;;; Copyright © 2014, 2016, 2019 Eric Bavier <bavier <at> member.fsf.org>
 ;;; Copyright © 2015, 2016, 2017, 2018, 2019, 2020 Efraim Flashner <efraim <at> flashner.co.il>
-;;; Copyright © 2015, 2018 Kyle Meyer <kyle <at> kyleam.com>
+;;; Copyright © 2015, 2018, 2020 Kyle Meyer <kyle <at> kyleam.com>
 ;;; Copyright © 2015, 2017, 2018, 2020 Ricardo Wurmus <rekado <at> elephly.net>
 ;;; Copyright © 2016, 2017 Leo Famulari <leo <at> famulari.name>
 ;;; Copyright © 2016, 2017, 2018 Nikita <nikita <at> n0.is>
@@ -2269,6 +2269,43 @@ (define-public myrepos
 Mercurial, Bazaar, Darcs, CVS, Fossil, and Veracity.")
     (license license:gpl2+)))
 
+(define-public grokmirror
+  (package
+    (name "grokmirror")
+    (version "2.0.0")
+    (source
+     (origin
+       (method git-fetch)
+       (uri (git-reference
+             (url (string-append "https://git.kernel.org/pub/scm/"
+                                 "utils/grokmirror/grokmirror.git"))
+             (commit (string-append "v" version))))
+       (file-name (string-append name "-" version "-checkout"))
+       (sha256
+        (base32 "1cs43vf87x8x5k5ncgiwiclc92a1dvxpg2z6lh6psaiip808gylp"))))
+    (build-system python-build-system)
+    (arguments
+     `(#:tests? #f                      ; no test suite
+       #:phases
+       (modify-phases %standard-phases
+         (add-after 'install 'install-manpages
+           (lambda* (#:key outputs #:allow-other-keys)
+             (let* ((man (string-append (assoc-ref outputs "out")
+                                        "/man/man1/")))
+               (mkdir-p man)
+               (for-each (lambda (file) (install-file file man))
+                         (find-files "." "\\.1$")))
+             #t)))))
+    (propagated-inputs
+     `(("python-requests" ,python-requests)))
+    (home-page
+     "https://git.kernel.org/pub/scm/utils/grokmirror/grokmirror.git")
+    (synopsis "Framework to smartly mirror git repositories")
+    (description "Grokmirror enables replicating large git repository
+collections efficiently.  Mirrors decide to clone and update repositories
+based on a manifest file published by servers.")
+    (license license:gpl3+)))
+
 (define-public git-annex-remote-rclone
   (package
     (name "git-annex-remote-rclone")

base-commit: 3876ed32f25e160c47ba046d987ad9097b2bf0cb
-- 
2.28.0.849.g665beee440

Message #8 received at 43637 <at> debbugs.gnu.org (full text, mbox):

From: Kyle Meyer <kyle <at> kyleam.com>
To: 43637 <at> debbugs.gnu.org
Cc: zimoun <zimon.toutoune <at> gmail.com>
Subject: Re: [PATCH] gnu: packages: Add grokmirror.
Date: Sat, 26 Sep 2020 16:58:20 -0400

A recent message on bug-guix [1] mentioned cloning the archives at
<https://yhetil.org>.  public-inbox archives, including those at
yhetil.org, publish a grokmirror-compatible manifest.  So, if you're
interested in mirroring the Guix-related archives at yhetil.org for any
reason [2], you can use grokmirror to do so in a way that's nicer than
periodically hitting yhetil.org with a git-fetch call for each
Guix-related inbox.

You can clone (and then later update) Guix-related inboxes with

    $ grok-pull -vc /path/to/config

and a config file that looks something like this:

--8<---------------cut here---------------start------------->8---
[core]
toplevel = /path/to/inboxes
manifest = ${toplevel}/manifest.js.gz
log = ${toplevel}/grok.log
loglevel = info
objstore = ${toplevel}/objstore

[pull]
pull_threads = 2
include = /guix-*

[remote]
site = https://yhetil.org
manifest = ${site}/manifest.js.gz
--8<---------------cut here---------------end--------------->8---

[1] https://issues.guix.gnu.org/43528#4
    CAJ3okZ0g25+-wsL2OepE+xMR5QWShmS73hSHsjEmEgYBiOWwAg <at> mail.gmail.com

[2] to have a fast local copy, to publish your own mirror, to feed the
    archives into some other tool...

Message #13 received at 43637-done <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Kyle Meyer <kyle <at> kyleam.com>
Cc: 43637-done <at> debbugs.gnu.org
Subject: Re: [bug#43637] [PATCH] gnu: packages: Add grokmirror.
Date: Sun, 27 Sep 2020 22:51:48 +0200

Hi,

Kyle Meyer <kyle <at> kyleam.com> skribis:

> * gnu/packages/version-control.scm (grokmirror): New variable.

Applied, thanks!

> A recent message on bug-guix [1] mentioned cloning the archives at
> <https://yhetil.org>.  public-inbox archives, including those at
> yhetil.org, publish a grokmirror-compatible manifest.  So, if you're
> interested in mirroring the Guix-related archives at yhetil.org for any
> reason [2], you can use grokmirror to do so in a way that's nicer than
> periodically hitting yhetil.org with a git-fetch call for each
> Guix-related inbox.

So yhetil.org stores mailing list archives as a Git repo?

Sounds interesting.

Thanks,
Ludo’.

Message #16 received at 43637-done <at> debbugs.gnu.org (full text, mbox):

From: Kyle Meyer <kyle <at> kyleam.com>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 43637-done <at> debbugs.gnu.org
Subject: Re: [bug#43637] [PATCH] gnu: packages: Add grokmirror.
Date: Sun, 27 Sep 2020 17:14:52 -0400

Ludovic Courtès writes:

> Hi,
>
> Kyle Meyer <kyle <at> kyleam.com> skribis:
>
>> * gnu/packages/version-control.scm (grokmirror): New variable.
>
> Applied, thanks!

Thanks for reviewing/applying!

>> A recent message on bug-guix [1] mentioned cloning the archives at
>> <https://yhetil.org>.  public-inbox archives, including those at
>> yhetil.org, publish a grokmirror-compatible manifest.  So, if you're
>> interested in mirroring the Guix-related archives at yhetil.org for any
>> reason [2], you can use grokmirror to do so in a way that's nicer than
>> periodically hitting yhetil.org with a git-fetch call for each
>> Guix-related inbox.
>
> So yhetil.org stores mailing list archives as a Git repo?
>
> Sounds interesting.

Yep, messages are stored in a Git repo or, for archives greater than
around one gigabyte, a collection of repos [*].  (A freshly gc'd
guix-devel weighs in at ~140M.)

  [*] documentation on public-inbox's v2 format:
      https://public-inbox.org/public-inbox-v2-format.html

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.