GNU bug report logs - #39841
[WIP PATCH] synapse

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 39841 in the body.
You can then email your comments to 39841 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Alex ter Weele <alex.ter.weele <at> gmail.com>
To: "guix-patches" <guix-patches <at> gnu.org>
Subject: [WIP PATCH] synapse
Date: Sat, 29 Feb 2020 11:53:03 -0600

[Message part 1 (text/plain, inline)]

Hello,

These are some patches to add Synapse
<https://github.com/matrix-org/synapse>.

I would appreciate some help getting the tests to run for some of the
Python dependencies, they're marked with TODOs.

Synapse appears to have some optional dependencies. I haven't packaged
all of them. The remaining ones are also marked with TODOs.

But, it seems to run! Here's what I had to do to start it:

guix environment guix -- make -j4 && ./pre-inst-env guix environment synapse

cd ~/src/synapse/

python -m synapse.app.homeserver --server-name test.local --config-path /tmp/homeserver.yaml --generate-config --report-stats=no

exit

guix environment guix -- make -j4 && ./pre-inst-env guix environment --ad-hoc synapse -- synctl --no-daemonize start /tmp/homeserver.yaml

[0001-gnu-python-typing-extensions-Update-to-3.7.4.1.patch (text/x-patch, attachment)]

[0002-gnu-Add-python-frozendict.patch (text/x-patch, attachment)]

[0003-gnu-Add-python-unpaddedbase64.patch (text/x-patch, attachment)]

[0004-gnu-Add-python-canonicaljson.patch (text/x-patch, attachment)]

[0005-gnu-Add-python-signedjson.patch (text/x-patch, attachment)]

[0006-gnu-Add-python-daemonize.patch (text/x-patch, attachment)]

[0007-gnu-Add-python-pymacaroons.patch (text/x-patch, attachment)]

[0008-gnu-Add-python-prometheus-client.patch (text/x-patch, attachment)]

[0009-gnu-Add-python-ldap3.patch (text/x-patch, attachment)]

[0010-gnu-Add-python-txacme.patch (text/x-patch, attachment)]

[0011-gnu-Add-python-matrix-synapse-ldap3.patch (text/x-patch, attachment)]

[0012-gnu-Add-synapse.patch (text/x-patch, attachment)]

Message #8 received at 39841 <at> debbugs.gnu.org (full text, mbox):

From: Alex ter Weele <alex.ter.weele <at> gmail.com>
To: 39841 <at> debbugs.gnu.org
Subject: Re: [bug#39841] [WIP PATCH] synapse
Date: Sat, 29 Feb 2020 13:12:44 -0600

[Message part 1 (text/plain, inline)]

This patch series is the same as the last but cleaned up according to
guix lint.

[0001-gnu-Add-emacs-punpun-theme.patch (text/x-patch, attachment)]

[0001-gnu-Add-fennel.patch (text/x-patch, attachment)]

[0001-gnu-python-typing-extensions-Update-to-3.7.4.1.patch (text/x-patch, attachment)]

[0002-gnu-Add-python-frozendict.patch (text/x-patch, attachment)]

[0003-gnu-Add-python-unpaddedbase64.patch (text/x-patch, attachment)]

[0004-gnu-Add-python-canonicaljson.patch (text/x-patch, attachment)]

[0005-gnu-Add-python-signedjson.patch (text/x-patch, attachment)]

[0006-gnu-Add-python-daemonize.patch (text/x-patch, attachment)]

[0007-gnu-Add-python-pymacaroons.patch (text/x-patch, attachment)]

[0008-gnu-Add-python-prometheus-client.patch (text/x-patch, attachment)]

[0009-gnu-Add-python-ldap3.patch (text/x-patch, attachment)]

[0010-gnu-Add-python-txacme.patch (text/x-patch, attachment)]

[0011-gnu-Add-python-matrix-synapse-ldap3.patch (text/x-patch, attachment)]

[0012-gnu-Add-synapse.patch (text/x-patch, attachment)]

Message #11 received at 39841 <at> debbugs.gnu.org (full text, mbox):

From: Alex ter Weele <alex.ter.weele <at> gmail.com>
To: 39841 <at> debbugs.gnu.org
Subject: Re: [bug#39841] [WIP PATCH] synapse
Date: Sat, 29 Feb 2020 13:14:19 -0600

0001-gnu-Add-emacs-punpun-theme.patch and 0001-gnu-Add-fennel.patch were
included by mistake, sorry about that.

Message #14 received at 39841 <at> debbugs.gnu.org (full text, mbox):

From: Marius Bakke <mbakke <at> fastmail.com>
To: Alex ter Weele <alex.ter.weele <at> gmail.com>, 39841 <at> debbugs.gnu.org
Subject: Re: [bug#39841] [WIP PATCH] synapse
Date: Sun, 29 Mar 2020 22:03:10 +0200

[Message part 1 (text/plain, inline)]

Alex ter Weele <alex.ter.weele <at> gmail.com> writes:

> Hello,
>
> These are some patches to add Synapse
> <https://github.com/matrix-org/synapse>.

Cool!  Now Guix users can run their own Matrix home servers!

Can you rebase this series on the current master branch?  Let's try to
merge the patches that are not marked with TODO, to make reviewing the
remainder easier.

I've applied the first patch already!  :-)

Short feedback on the series: inputs should probably be
propagated-inputs in all of these.  Please also run 'guix lint' on each
package to catch cosmetic issues.

Thanks in advance, and sorry for the delay!

[signature.asc (application/pgp-signature, inline)]

Message #17 received at 39841 <at> debbugs.gnu.org (full text, mbox):

From: Alex ter Weele <alex.ter.weele <at> gmail.com>
To: Marius Bakke <mbakke <at> fastmail.com>
Cc: 39841 <at> debbugs.gnu.org
Subject: Re: [bug#39841] [WIP PATCH] synapse
Date: Sun, 29 Mar 2020 20:28:45 -0400

Marius Bakke <mbakke <at> fastmail.com> writes:

> Alex ter Weele <alex.ter.weele <at> gmail.com> writes:
>
>> Hello,
>>
>> These are some patches to add Synapse
>> <https://github.com/matrix-org/synapse>.
>
> Cool!  Now Guix users can run their own Matrix home servers!
>
> Can you rebase this series on the current master branch?  Let's try to
> merge the patches that are not marked with TODO, to make reviewing the
> remainder easier.
>
> I've applied the first patch already!  :-)
>
> Short feedback on the series: inputs should probably be
> propagated-inputs in all of these.  Please also run 'guix lint' on each
> package to catch cosmetic issues.
>
> Thanks in advance, and sorry for the delay!

Here is  a new patch series atop current master. It should be
guix-linted and use propagated-inputs.

I also want to mention: I indiscriminately put everything in
python-xyz.scm, some of these packages may be a better fit for other
python-*.scm files.

Thanks for the review!

Message #20 received at 39841 <at> debbugs.gnu.org (full text, mbox):

From: Alex ter Weele <alex.ter.weele <at> gmail.com>
To: Marius Bakke <mbakke <at> fastmail.com>
Cc: 39841 <at> debbugs.gnu.org
Subject: Re: [bug#39841] [WIP PATCH] synapse
Date: Sun, 29 Mar 2020 20:35:47 -0400

[Message part 1 (text/plain, inline)]

Alex ter Weele <alex.ter.weele <at> gmail.com> writes:

> Marius Bakke <mbakke <at> fastmail.com> writes:
>
>> Alex ter Weele <alex.ter.weele <at> gmail.com> writes:
>>
>>> Hello,
>>>
>>> These are some patches to add Synapse
>>> <https://github.com/matrix-org/synapse>.
>>
>> Cool!  Now Guix users can run their own Matrix home servers!
>>
>> Can you rebase this series on the current master branch?  Let's try to
>> merge the patches that are not marked with TODO, to make reviewing the
>> remainder easier.
>>
>> I've applied the first patch already!  :-)
>>
>> Short feedback on the series: inputs should probably be
>> propagated-inputs in all of these.  Please also run 'guix lint' on each
>> package to catch cosmetic issues.
>>
>> Thanks in advance, and sorry for the delay!
>
> Here is  a new patch series atop current master. It should be
> guix-linted and use propagated-inputs.
>
> I also want to mention: I indiscriminately put everything in
> python-xyz.scm, some of these packages may be a better fit for other
> python-*.scm files.
>
> Thanks for the review!

I forgot to attach the patch series 🤦

[0001-gnu-Add-python-frozendict.patch (text/x-patch, attachment)]

[0002-gnu-Add-python-unpaddedbase64.patch (text/x-patch, attachment)]

[0003-gnu-Add-python-canonicaljson.patch (text/x-patch, attachment)]

[0004-gnu-Add-python-signedjson.patch (text/x-patch, attachment)]

[0005-gnu-Add-python-daemonize.patch (text/x-patch, attachment)]

[0006-gnu-Add-python-pymacaroons.patch (text/x-patch, attachment)]

[0007-gnu-Add-python-prometheus-client.patch (text/x-patch, attachment)]

[0008-gnu-Add-python-ldap3.patch (text/x-patch, attachment)]

[0009-gnu-Add-python-boltons.patch (text/x-patch, attachment)]

[0010-gnu-Add-python-eliot.patch (text/x-patch, attachment)]

[0011-gnu-Add-python-pem.patch (text/x-patch, attachment)]

[0012-gnu-Add-python-txsni.patch (text/x-patch, attachment)]

[0013-gnu-Add-python-txacme.patch (text/x-patch, attachment)]

[0014-gnu-Add-python-pysaml2.patch (text/x-patch, attachment)]

[0015-gnu-Add-python-matrix-synapse-ldap3.patch (text/x-patch, attachment)]

[0016-gnu-Add-synapse.patch (text/x-patch, attachment)]

Message #23 received at 39841 <at> debbugs.gnu.org (full text, mbox):

From: Alex ter Weele <alex.ter.weele <at> gmail.com>
To: Marius Bakke <mbakke <at> fastmail.com>
Cc: 39841 <at> debbugs.gnu.org
Subject: Re: [bug#39841] [WIP PATCH] synapse
Date: Sun, 29 Mar 2020 20:42:08 -0400

[Message part 1 (text/plain, inline)]

Alex ter Weele <alex.ter.weele <at> gmail.com> writes:

> Marius Bakke <mbakke <at> fastmail.com> writes:
>
>> Alex ter Weele <alex.ter.weele <at> gmail.com> writes:
>>
>>> Hello,
>>>
>>> These are some patches to add Synapse
>>> <https://github.com/matrix-org/synapse>.
>>
>> Cool!  Now Guix users can run their own Matrix home servers!
>>
>> Can you rebase this series on the current master branch?  Let's try
>> to
>> merge the patches that are not marked with TODO, to make reviewing
>> the
>> remainder easier.
>>
>> I've applied the first patch already!  :-)
>>
>> Short feedback on the series: inputs should probably be
>> propagated-inputs in all of these.  Please also run 'guix lint' on
>> each
>> package to catch cosmetic issues.
>>
>> Thanks in advance, and sorry for the delay!
>
> Here is a new patch series atop current master. It should be
> guix-linted and use propagated-inputs.
>
> I also want to mention: I indiscriminately put everything in
> python-xyz.scm, some of these packages may be a better fit for other
> python-*.scm files.
>
> Thanks for the review!

I forgot to attach the patch series, whoops

[0001-gnu-Add-python-frozendict.patch (text/x-patch, attachment)]

[0002-gnu-Add-python-unpaddedbase64.patch (text/x-patch, attachment)]

[0003-gnu-Add-python-canonicaljson.patch (text/x-patch, attachment)]

[0004-gnu-Add-python-signedjson.patch (text/x-patch, attachment)]

[0005-gnu-Add-python-daemonize.patch (text/x-patch, attachment)]

[0006-gnu-Add-python-pymacaroons.patch (text/x-patch, attachment)]

[0007-gnu-Add-python-prometheus-client.patch (text/x-patch, attachment)]

[0008-gnu-Add-python-ldap3.patch (text/x-patch, attachment)]

[0009-gnu-Add-python-boltons.patch (text/x-patch, attachment)]

[0010-gnu-Add-python-eliot.patch (text/x-patch, attachment)]

[0011-gnu-Add-python-pem.patch (text/x-patch, attachment)]

[0012-gnu-Add-python-txsni.patch (text/x-patch, attachment)]

[0013-gnu-Add-python-txacme.patch (text/x-patch, attachment)]

[0014-gnu-Add-python-pysaml2.patch (text/x-patch, attachment)]

[0015-gnu-Add-python-matrix-synapse-ldap3.patch (text/x-patch, attachment)]

[0016-gnu-Add-synapse.patch (text/x-patch, attachment)]

Message #26 received at 39841 <at> debbugs.gnu.org (full text, mbox):

From: Alex ter Weele <alex.ter.weele <at> gmail.com>
To: Alex ter Weele <alex.ter.weele <at> gmail.com>
Cc: 39841 <at> debbugs.gnu.org
Subject: Re: [bug#39841] [WIP PATCH] synapse
Date: Sat, 28 Mar 2020 17:22:02 -0500

[Message part 1 (text/plain, inline)]

This new patch series should apply against master. There are a few more
packages and I have guix linted.

I'd appreciate help from people famaliar with Python testing to get
tests working. They should be marked with TODOs.

[0001-gnu-python-typing-extensions-Update-to-3.7.4.1.patch (text/x-patch, attachment)]

[0002-gnu-Add-python-frozendict.patch (text/x-patch, attachment)]

[0003-gnu-Add-python-unpaddedbase64.patch (text/x-patch, attachment)]

[0004-gnu-Add-python-canonicaljson.patch (text/x-patch, attachment)]

[0005-gnu-Add-python-signedjson.patch (text/x-patch, attachment)]

[0006-gnu-Add-python-daemonize.patch (text/x-patch, attachment)]

[0007-gnu-Add-python-pymacaroons.patch (text/x-patch, attachment)]

[0008-gnu-Add-python-prometheus-client.patch (text/x-patch, attachment)]

[0009-gnu-Add-python-ldap3.patch (text/x-patch, attachment)]

[0010-gnu-Add-python-boltons.patch (text/x-patch, attachment)]

[0011-gnu-Add-python-eliot.patch (text/x-patch, attachment)]

[0012-gnu-Add-python-pem.patch (text/x-patch, attachment)]

[0013-gnu-Add-python-txsni.patch (text/x-patch, attachment)]

[0014-gnu-Add-python-txacme.patch (text/x-patch, attachment)]

[0015-gnu-Add-python-pysaml2.patch (text/x-patch, attachment)]

[0016-gnu-Add-python-matrix-synapse-ldap3.patch (text/x-patch, attachment)]

[0017-gnu-Add-synapse.patch (text/x-patch, attachment)]

Message #31 received at 39841-done <at> debbugs.gnu.org (full text, mbox):

From: Marius Bakke <mbakke <at> fastmail.com>
To: Alex ter Weele <alex.ter.weele <at> gmail.com>
Cc: 39841-done <at> debbugs.gnu.org
Subject: Re: [bug#39841] [WIP PATCH] synapse
Date: Thu, 02 Apr 2020 20:49:40 +0200

[Message part 1 (text/plain, inline)]

Phew...  It took a while, but I've merged the whole series.  \o/

PS: Some of the later packages did not pass 'guix lint'.  There were
periods in synopses, lack of period or capitalizations in descriptions
and other minor issues.

I've also modified some of the descriptions to be more ... descriptive,
and use full sentences, instead of just a copy-paste of the synopsis.

I added a 'Co-authored-by:' line for the packages that were heavily
modified to save you some of the blame.  ;-)

A brief feedback on the patches:

Alex ter Weele <alex.ter.weele <at> gmail.com> writes:

> +(define-public python-signedjson
> +  (package
> +    (name "python-signedjson")
> +    (version "1.1")
> +    (source
> +     (origin
> +       (method url-fetch)
> +       (uri (pypi-uri "signedjson" version))
> +       (sha256
> +        (base32 "13wfi50wzz38j6xsz7hmx26vf0zvk0fj7516wavzpsyfk9xy3g5v"))))
> +    (build-system python-build-system)
> +    (arguments
> +     ;; TODO "AttributeError: type object 'Callable' has no attribute
> +     ;; '_abc_registry'"
> +     `(#:tests? #f))

This was because of the inclusion of 'python-typing', which is part of
the Python standard library and apparently caused some conflict.
Removing it from propagated-inputs made all tests pass.

> +(define-public python-prometheus-client
> +  (package
> +    (name "python-prometheus-client")
> +    (version "0.7.1")
> +    (source
> +     (origin
> +       (method url-fetch)
> +       (uri (pypi-uri "prometheus_client" version))
> +       (sha256
> +        (base32 "1ni2yv4ixwz32nz39ckia76lvggi7m19y5f702w5qczbnfi29kbi"))))
> +    (build-system python-build-system)
> +    (arguments
> +     ;; TODO "ModuleNotFoundError: No module named 'tests'"
> +     `(#:tests? #f))

This is because the PyPI tarball contains no tests.  I adjusted the
comment to reflect that.

> +    (propagated-inputs
> +     `(("python-twisted" ,python-twisted)))
> +    (native-inputs
> +     `(("python-tox" ,python-tox)))

...and also removed Tox, as it is pretty useless in Guix.  Whenever you
come across a package that uses Tox, you should read what tox.ini does
and do that directly in the check phase instead.

(Running Tox does not work because it wants to go on the internet and
create a virtualenv and does not play well with native Guix packages)

> +(define-public python-ldap3
> +  (package
> +    (name "python-ldap3")
> +    (version "2.6.1")
> +    (source
> +     (origin
> +       (method url-fetch)
> +       (uri (pypi-uri "ldap3" version))
> +       (sha256
> +        (base32 "0ag5xqlki6pjk3f50b8ar8vynx2fmkna7rfampv3kdgwg8z6gjr7"))))
> +    (build-system python-build-system)
> +    (arguments
> +     ;; TODO "ImportError: Failed to import test module: ordDict"
> +     `(#:tests? #f))

There were two problems here: the pyasn1 version was too old, and
ultimately running the full suite requires an LDAP server, so I adjusted
the comment to reflect that (and also updated pyasn1 to prevent runtime
errors).

> +(define-public python-eliot
> +  (package
> +    (name "python-eliot")
> +    (version "1.12.0")
> +    (source
> +     (origin
> +       (method url-fetch)
> +       (uri (pypi-uri "eliot" version))
> +       (sha256
> +        (base32 "0wabv7hk63l12881f4zw02mmj06583qsx2im0yywdjlj8f56vqdn"))))
> +    (build-system python-build-system)
> +    (arguments
> +     `(#:phases
> +       (modify-phases %standard-phases
> +         (add-after 'unpack 'remove-journald-support
> +           (lambda _
> +             (invoke "rm"
> +                     "eliot/tests/test_journald.py"
> +                     "eliot/journald.py")
> +             #t))
> +         (add-after 'remove-journald-support 'remove-eliot-prettyprint-tests
> +           ;; remove command-line tool's tests. TODO eliot-prettyprint should
> +           ;; be installed and these tests should pass.
> +           (lambda _
> +             (invoke "rm" "eliot/tests/test_prettyprint.py")
> +             #t)))))

I changed the (invoke "rm" ...) stanzas to (delete-file ...) to avoid
the fork. 

> +(define-public python-pem
> +  (package
> +    (name "python-pem")
> +    (version "20.1.0")
> +    (source
> +     (origin
> +       (method url-fetch)
> +       (uri (pypi-uri "pem" version))
> +       (sha256
> +        (base32
> +         "1xh88ss279fprxnzd10dczmqwjhppbyvljm33zrg2mgybwd66qr7"))))
> +    (build-system python-build-system)
> +    (native-inputs
> +     `(("python-certifi" ,python-certifi)
> +       ("python-coverage" ,python-coverage)
> +       ("python-pretend" ,python-pretend)
> +       ("python-pyopenssl" ,python-pyopenssl)
> +       ("python-pytest" ,python-pytest)
> +       ("python-sphinx" ,python-sphinx)
> +       ("python-twisted" ,python-twisted)))
> +    (home-page "https://pem.readthedocs.io/")
> +    (synopsis "Easy PEM file parsing in Python.")
> +    (description "Easy PEM file parsing in Python.")

I'm curious why all those inputs are just native, but did not
investigate.  I did fix the trailing dot in the synopsis, as well as
expanded a bit on the description.

> +(define-public python-txsni
> +  (package
> +    (name "python-txsni")
> +    (version "0.1.9")
> +    (source
> +     (origin
> +       (method url-fetch)
> +       (uri (pypi-uri "TxSNI" version))
> +       (sha256
> +        (base32
> +         "1c72nbhx4z0v5djb89yzrcqlmjdlyryj7qgwl9ivnn816126z5kj"))))
> +    (build-system python-build-system)
> +    (arguments
> +     ;; TODO confusing test failure
> +     `(#:tests? #f))

So this turned out the be because the latest TxSNI release is
incompatible with newer OpenSSL.  I ended up pulling the package
straight from git, and made myself the package author.

> +    (propagated-inputs
> +     `(("python-pyopenssl" ,python-pyopenssl)
> +       ("python-service-identity" ,python-service-identity)
> +       ("python-twisted" ,python-twisted)))
> +    (home-page "https://github.com/glyph/txsni")
> +    (synopsis "easy-to-use SNI endpoint for twisted")
> +    (description
> +     "easy-to-use SNI endpoint for twisted")

...I also rewrote synopsis and description to be more Guix-y.

> +(define-public python-pysaml2
> +  (package
> +    (name "python-pysaml2")
> +    (version "5.0.0")
> +    (source
> +     (origin
> +       (method url-fetch)
> +       (uri (pypi-uri "pysaml2" version))
> +       (sha256
> +        (base32
> +         "1h8cmxh9cvxhrdfmkh92wg6zpxmhi2fixq1cy4hxismmaar7bsny"))))
> +    (build-system python-build-system)
> +    (propagated-inputs
> +     `(("python-cryptography" ,python-cryptography)
> +       ("python-dateutil" ,python-dateutil)
> +       ("python-defusedxml" ,python-defusedxml)
> +       ("python-pyopenssl" ,python-pyopenssl)
> +       ("python-pytz" ,python-pytz)
> +       ("python-requests" ,python-requests)
> +       ("python-six" ,python-six)))
> +    (home-page "https://idpy.org")
> +    (synopsis
> +     "Python implementation of SAML Version 2 Standard")
> +    (description
> +     "Python implementation of SAML Version 2 Standard")

Here I took the description from the README which was very informative.

> * gnu/packages/matrix.scm: New file. (python-matrix-synapse-ldap3) New
>   variable.
> * gnu/local.mk (GNU_SYSTEM_MODULES): Add matrix.scm.
> ---
>  gnu/local.mk            |  1 +
>  gnu/packages/matrix.scm | 33 +++++++++++++++++++++++++++++++++
>  2 files changed, 34 insertions(+)
>  create mode 100644 gnu/packages/matrix.scm
>
> diff --git a/gnu/local.mk b/gnu/local.mk
> index 7d8145b10d..8c50da9c10 100644
> --- a/gnu/local.mk
> +++ b/gnu/local.mk
> @@ -343,6 +343,7 @@ GNU_SYSTEM_MODULES =				\
>    %D%/packages/marst.scm			\
>    %D%/packages/mate.scm             \
>    %D%/packages/maths.scm			\
> +  %D%/packages/matrix.scm			\
>    %D%/packages/maven.scm			\
>    %D%/packages/mc.scm				\
>    %D%/packages/mcrypt.scm			\
> diff --git a/gnu/packages/matrix.scm b/gnu/packages/matrix.scm
> new file mode 100644
> index 0000000000..f399d2aabc
> --- /dev/null
> +++ b/gnu/packages/matrix.scm
> @@ -0,0 +1,33 @@
> +(define-module (gnu packages matrix)
> +  #:use-module ((guix licenses) #:prefix license:)
> +  #:use-module (gnu packages python-crypto)
> +  #:use-module (gnu packages python-xyz)
> +  #:use-module (guix build-system python)
> +  #:use-module (guix download)
> +  #:use-module (guix packages))

You forgot to add the license notice at the top of this file.  Fixed!

[signature.asc (application/pgp-signature, inline)]

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.