Package: guix-patches;
Reported by: Lars-Dominik Braun <lars <at> 6xq.net>
Date: Sun, 15 Nov 2020 12:40:01 UTC
Severity: normal
Tags: patch
Done: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
Bug is archived. No further changes may be made.
View this message in rfc822 format
From: help-debbugs <at> gnu.org (GNU bug Tracking System) To: Maxim Cournoyer <maxim.cournoyer <at> gmail.com> Cc: tracker <at> debbugs.gnu.org Subject: bug#44656: closed ([PATCH] Upgrade pypy3) Date: Thu, 05 Aug 2021 16:56:01 +0000
[Message part 1 (text/plain, inline)]
Your message dated Thu, 05 Aug 2021 12:54:47 -0400 with message-id <877dgz96i0.fsf_-_ <at> gmail.com> and subject line Re: bug#44656: [PATCH] Upgrade pypy3 has caused the debbugs.gnu.org bug report #44656, regarding [PATCH] Upgrade pypy3 to be marked as done. (If you believe you have received this mail in error, please contact help-debbugs <at> gnu.org.) -- 44656: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=44656 GNU Bug Tracking System Contact help-debbugs <at> gnu.org with problems
[Message part 2 (message/rfc822, inline)]
From: Lars-Dominik Braun <lars <at> 6xq.net> To: guix-patches <at> gnu.org Subject: [PATCH] Upgrade pypy3 Date: Sun, 15 Nov 2020 13:39:11 +0100[Message part 3 (text/plain, inline)]Hi, the first patch in this series upgrades pypy3 to 7.3.2 and the second adjusts several aspects of the package. Lars[0001-gnu-pypy3-Update-to-7.3.2.patch (text/x-diff, attachment)][0002-gnu-pypy3-Various-package-fixes.patch (text/x-diff, attachment)]
[Message part 6 (message/rfc822, inline)]
From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com> To: Lars-Dominik Braun <lars <at> 6xq.net> Cc: 44656-done <at> debbugs.gnu.org Subject: Re: bug#44656: [PATCH] Upgrade pypy3 Date: Thu, 05 Aug 2021 12:54:47 -0400Hello again, Maxim Cournoyer <maxim.cournoyer <at> gmail.com> writes: > Hi Lars, > > Lars-Dominik Braun <lars <at> 6xq.net> writes: > >> Hi, >> >> the first patch in this series upgrades pypy3 to 7.3.2 and the second >> adjusts several aspects of the package. >> >> Lars >> >> >>>>From caf5e4bf9acd10f91de2e6a9c60185475144a1b9 Mon Sep 17 00:00:00 2001 >> From: Lars-Dominik Braun <lars <at> 6xq.net> >> Date: Sun, 15 Nov 2020 10:54:26 +0100 >> Subject: [PATCH 2/2] gnu: pypy3: Various package fixes. >> >> * gnu/packages/python.scm (pypy3) [patches]: Add new patch. >> [inputs]: Remove bash-minimal and add nss-certs. >> [native-inputs]: Remove nss-certs here. > > In Guix we leave the choice of the TLS certs to the users; meaning we > don't hard-code their location in packages and instead have the users > explicitly install them (and in the case of OpenSSL, set the necessary > environment variables), the same we do for icons and other 'choosable' > things; while convenient the above goes against this tradition. > >> [arguments]: Use gdbm compat library, add 2to3 binary. >> [native-search-paths]: Add search path. >> * gnu/packages/patches/pypy3-7.3.1-ssl-paths.patch: New file. >> * gnu/local.mk: Add it. >> --- >> gnu/local.mk | 1 + >> .../patches/pypy3-7.3.1-ssl-paths.patch | 41 +++++++ >> gnu/packages/python.scm | 109 ++++++++++++------ >> 3 files changed, 116 insertions(+), 35 deletions(-) >> create mode 100644 gnu/packages/patches/pypy3-7.3.1-ssl-paths.patch >> >> diff --git a/gnu/local.mk b/gnu/local.mk >> index 91a3295e75..b644391a11 100644 >> --- a/gnu/local.mk >> +++ b/gnu/local.mk >> @@ -1536,6 +1536,7 @@ dist_patch_DATA = \ >> %D%/packages/patches/python-unittest2-remove-argparse.patch \ >> %D%/packages/patches/python-waitress-fix-tests.patch \ >> %D%/packages/patches/pypy3-7.3.1-fix-tests.patch \ >> + %D%/packages/patches/pypy3-7.3.1-ssl-paths.patch \ >> %D%/packages/patches/qemu-build-info-manual.patch \ >> %D%/packages/patches/qemu-glibc-2.27.patch \ >> %D%/packages/patches/qrcodegen-cpp-make-install.patch \ >> diff --git a/gnu/packages/patches/pypy3-7.3.1-ssl-paths.patch b/gnu/packages/patches/pypy3-7.3.1-ssl-paths.patch >> new file mode 100644 >> index 0000000000..d21133b4ae >> --- /dev/null >> +++ b/gnu/packages/patches/pypy3-7.3.1-ssl-paths.patch >> @@ -0,0 +1,41 @@ >> +Fix default certificate search path, still allowing the user to override it >> +with environment variables. >> + >> +--- a/lib_pypy/_cffi_ssl/_stdssl/__init__.py >> ++++ b/lib_pypy/_cffi_ssl/_stdssl/__init__.py >> +@@ -1679,20 +1679,9 @@ def get_default_verify_paths(): >> + https://golang.org/src/crypto/x509/root_linux.go (for the files) >> + ''' >> + certFiles = [ >> +- "/etc/ssl/certs/ca-certificates.crt", # Debian/Ubuntu/Gentoo etc. >> +- "/etc/pki/tls/certs/ca-bundle.crt", # Fedora/RHEL 6 >> +- "/etc/ssl/ca-bundle.pem", # OpenSUSE >> +- "/etc/pki/tls/cacert.pem", # OpenELEC >> +- "/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem", # CentOS/RHEL 7 >> +- "/etc/ssl/cert.pem", # Alpine Linux >> + ] >> + certDirectories = [ >> +- "/etc/ssl/certs", # SLES10/SLES11 >> +- "/system/etc/security/cacerts", # Android >> +- "/usr/local/share/certs", # FreeBSD >> +- "/etc/pki/tls/certs", # Fedora/RHEL >> +- "/etc/openssl/certs", # NetBSD >> +- "/var/ssl/certs", # AIX >> ++ "@GUIX_CERT_PATH@", >> + ] >> + >> + # optimization: reuse the values from a local varaible >> +@@ -1707,9 +1696,10 @@ def get_default_verify_paths(): >> + ofile = _cstr_decode_fs(lib.X509_get_default_cert_file()) >> + odir = _cstr_decode_fs(lib.X509_get_default_cert_dir()) >> + >> +- if os.path.exists(ofile) and os.path.exists(odir): >> +- get_default_verify_paths.retval = (ofile_env, ofile, odir_env, odir) >> +- return get_default_verify_paths.retval >> ++ if not os.path.exists(ofile): >> ++ ofile = None >> ++ if not os.path.exists(odir): >> ++ odir = None >> + >> + # OpenSSL didn't supply the goods. Try some other options >> + for f in certFiles: > > As mentioned above, the choice of certs should not be hard coded the in > the package definition. The correct behavior of honoring SSL_CERT_DIR > and SSL_CERT_FILE environment variables, and falling back to the system > provided location is sane, so the above patch is unwelcome, IMHO. > >> diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm >> index 8ef8ae2e1d..c0bd3335e3 100644 >> --- a/gnu/packages/python.scm >> +++ b/gnu/packages/python.scm >> @@ -696,7 +696,8 @@ ease from the desktop to a microcontroller or embedded system.") >> (sha256 >> (base32 >> "03f1fdw6yk2mypa9pbmgk26r8y1hhmw801l6g36zry9zsvz7aqgx")) >> - (patches (search-patches "pypy3-7.3.1-fix-tests.patch")))) >> + (patches (search-patches "pypy3-7.3.1-fix-tests.patch" >> + "pypy3-7.3.1-ssl-paths.patch")))) >> (build-system gnu-build-system) >> (native-inputs >> `(("python-2" ,python-2) >> @@ -704,13 +705,13 @@ ease from the desktop to a microcontroller or embedded system.") >> ("tar" ,tar) ; Required for package.py >> ("python2-pycparser" ,python2-pycparser) >> ("python2-hypothesis" ,python2-hypothesis) >> - ("nss-certs" ,nss-certs) ; For ssl tests >> ("gzip" ,gzip))) >> (inputs >> `(("libffi" ,libffi) >> ("zlib" ,zlib) >> ("ncurses" ,ncurses) >> ("openssl" ,openssl) >> + ("nss-certs" ,nss-certs) ; For ssl module > > This change is not needed without the TLS patching above. > >> ("expat" ,expat) >> ("bzip2" ,bzip2) >> ("sqlite" ,sqlite) >> @@ -718,10 +719,9 @@ ease from the desktop to a microcontroller or embedded system.") >> ("tcl" ,tcl) >> ("tk" ,tk) >> ("glibc" ,glibc) >> - ("bash-minimal" ,bash-minimal) ; Used as /bin/sh >> ("xz" ,xz))) ; liblzma >> (arguments >> - `(#:tests? #f ;FIXME: Disabled for now, there are many tests failing. >> + `(#:tests? #f ;FIXME: Disabled for now, there are many tests failing. >> #:modules ((ice-9 ftw) (ice-9 match) >> (guix build utils) (guix build gnu-build-system)) >> #:phases (modify-phases %standard-phases >> @@ -750,6 +750,11 @@ ease from the desktop to a microcontroller or embedded system.") >> (substitute* '("lib_pypy/_curses_build.py") >> ;; Find curses >> (("/usr/local") (assoc-ref inputs "ncurses"))) >> + (substitute* '("lib_pypy/_dbm.py") >> + ;; Use gdbm compat library, so we don’t need to pull in bdb >> + (("ctypes.util.find_library\\('db'\\)") >> + (string-append "'" (assoc-ref inputs "gdbm") >> + "/lib/libgdbm_compat.so'"))) > > OK. > >> (substitute* '("lib_pypy/_sqlite3_build.py") >> ;; Always use search paths >> (("sys\\.platform\\.startswith\\('freebsd'\\)") "True") >> @@ -761,11 +766,18 @@ ease from the desktop to a microcontroller or embedded system.") >> "/lib/libsqlite3.so.0'"))) >> (substitute* '("lib-python/3/subprocess.py") >> ;; Fix shell path >> - (("/bin/sh") >> - (string-append (assoc-ref inputs "bash-minimal") "/bin/sh"))) >> + (("/bin/sh") (which "sh"))) > > OK. > >> (substitute* '("lib-python/3/distutils/unixccompiler.py") >> ;; gcc-toolchain does not provide symlink cc -> gcc >> (("\"cc\"") "\"gcc\"")) >> + (substitute* '("lib_pypy/_cffi_ssl/_stdssl/__init__.py") >> + ;; Add nss-certs to default certificate search path, >> + ;; otherwise every packages has to specify nss-certs and >> + ;; openssl as input to set the proper env variables. >> + ;; Depends on -ssl-paths.patch. >> + (("@GUIX_CERT_PATH@") >> + (string-append (assoc-ref inputs "nss-certs") >> + "/etc/ssl/certs"))) > > Not every package; the required SSL environment variables it should be > set in the environment by the user or by via a native search path, and > the certs manually provided (installed) by the user. > >> #t)) >> (add-after >> 'unpack 'set-source-file-times-to-1980 >> @@ -785,7 +797,8 @@ ease from the desktop to a microcontroller or embedded system.") >> (string-append "--make-jobs=" >> (number->string (parallel-job-count))) >> "-Ojit" >> - "targetpypystandalone")) >> + "targetpypystandalone" >> + "--allworkingmodules")) >> ;; Build c modules and package everything, so tests work. >> (with-directory-excursion "pypy/tool/release" >> (unsetenv "PYTHONPATH") ; Do not use the system’s python libs: >> @@ -793,7 +806,12 @@ ease from the desktop to a microcontroller or embedded system.") >> ; attribute 'IntFlag' >> (invoke "python2" "package.py" >> "--archive-name" "pypy-dist" >> - "--builddir" (getcwd))))) >> + "--builddir" (getcwd)) >> + ;; install pip and setuptools into the dist directory. >> + ;; XXX: Breaks virtualenv, because it does not set +w >> + ;; on files copied from the store. >> + ;(invoke "pypy-dist/bin/pypy3" "-m" "ensurepip") >> + ))) > > Since the above attempt to bundle pip failed and is new, I'd just leave > it out. > >> (replace 'check >> (lambda* (#:key tests? #:allow-other-keys) >> (if tests? >> @@ -811,32 +829,54 @@ ease from the desktop to a microcontroller or embedded system.") >> #t)) >> (replace 'install >> (lambda* (#:key inputs outputs #:allow-other-keys) >> - (with-directory-excursion "pypy/tool/release" >> - ;; Delete test data. >> - (for-each >> - (lambda (x) >> - (delete-file-recursively (string-append >> - "pypy-dist/lib-python/3/" x))) >> - '("tkinter/test" >> - "test" >> - "sqlite3/test" >> - "lib2to3/tests" >> - "idlelib/idle_test" >> - "distutils/tests" >> - "ctypes/test" >> - "unittest/test")) >> - ;; Patch shebang referencing python2 >> - (substitute* '("pypy-dist/lib-python/3/cgi.py" >> - "pypy-dist/lib-python/3/encodings/rot_13.py") >> - (("#!.+/bin/python") >> - (string-append "#!" (assoc-ref outputs "out") "/bin/pypy3"))) >> - (with-fluids ((%default-port-encoding "ISO-8859-1")) >> - (substitute* '("pypy-dist/lib_pypy/_md5.py" >> - "pypy-dist/lib_pypy/_sha1.py") >> - (("#!.+/bin/python") >> - (string-append "#!" (assoc-ref outputs "out") "/bin/pypy3")))) >> - (copy-recursively "pypy-dist" (assoc-ref outputs "out"))) >> - #t))))) >> + (let* ((out (assoc-ref outputs "out")) >> + (bin-pypy3 (string-append out "/bin/pypy3")) >> + (shebang-match-python "#!.+/bin/python") >> + (shebang-pypy3 (string-append "#!" bin-pypy3)) >> + (dist-dir "pypy/tool/release/pypy-dist")) >> + (with-directory-excursion dist-dir >> + ;; Delete test data. >> + (for-each >> + (lambda (x) >> + (delete-file-recursively (string-append >> + "lib-python/3/" x))) >> + '("tkinter/test" >> + "test" >> + "sqlite3/test" >> + "lib2to3/tests" >> + "idlelib/idle_test" >> + "distutils/tests" >> + "ctypes/test" >> + "unittest/test")) >> + ;; Patch shebang referencing python2 >> + (substitute* '("lib-python/3/cgi.py" >> + "lib-python/3/encodings/rot_13.py") >> + ((shebang-match-python) shebang-pypy3)) >> + (with-fluids ((%default-port-encoding "ISO-8859-1")) >> + (substitute* '("lib_pypy/_md5.py" >> + "lib_pypy/_sha1.py") >> + ((shebang-match-python) shebang-pypy3)))) >> + (copy-recursively dist-dir out) >> + ;; Make sure pypy3 is callable as python/python3, so we >> + ;; don’t have to patch every single package. >> + (symlink bin-pypy3 (string-append out "/bin/python")) >> + (symlink bin-pypy3 (string-append out "/bin/python3")) >> + ;; 2to3 is missing from pypy3, create it. >> + (let ((2to3 (string-append out "/bin/2to3"))) >> + (call-with-output-file 2to3 >> + (lambda (port) >> + (format port "#!~a~%" (string-append out "/bin/pypy3")) >> + (format port " >> +import sys >> +from lib2to3.main import main >> + >> +sys.exit(main('lib2to3.fixes'))"))) >> + (chmod 2to3 #o755)) >> + #t)))))) > > I'm unconvinced about the above symlinks; it seems preferable for users > to be able to unambiguously run both pypy3 and python3 (cpython) in the > same profile without conflicts. The pypy3 package on Debian contains > the following, for example: > > $ apt-file show pypy3 > pypy3: /usr/bin/pypy3 > pypy3: /usr/bin/pypy3clean > pypy3: /usr/bin/pypy3compile > pypy3: /usr/lib/libpypy3-c.so > pypy3: /usr/lib/pypy3/bin/libpypy3-c.so > pypy3: /usr/lib/pypy3/bin/pypy3-c > pypy3: /usr/lib/pypy3/include/pypy_decl.h > pypy3: /usr/lib/pypy3/include/pypy_macros.h > pypy3: /usr/lib/pypy3/include/pypy_marshal_decl.h > pypy3: /usr/lib/pypy3/include/pypy_structmember_decl.h > pypy3: /usr/lib/pypy3/lib_pypy/_audioop_cffi.pypy36-pp73-x86_64-linux-gnu.so > pypy3: /usr/lib/pypy3/lib_pypy/_blake2/_blake2b_cffi.pypy36-pp73-x86_64-linux-gnu.so > pypy3: /usr/lib/pypy3/lib_pypy/_blake2/_blake2s_cffi.pypy36-pp73-x86_64-linux-gnu.so > pypy3: /usr/lib/pypy3/lib_pypy/_curses_cffi.pypy36-pp73-x86_64-linux-gnu.so > pypy3: /usr/lib/pypy3/lib_pypy/_decimal_cffi.pypy36-pp73-x86_64-linux-gnu.so > pypy3: /usr/lib/pypy3/lib_pypy/_gdbm_cffi.pypy36-pp73-x86_64-linux-gnu.so > pypy3: /usr/lib/pypy3/lib_pypy/_lzma_cffi.pypy36-pp73-x86_64-linux-gnu.so > pypy3: /usr/lib/pypy3/lib_pypy/_pwdgrp_cffi.pypy36-pp73-x86_64-linux-gnu.so > pypy3: /usr/lib/pypy3/lib_pypy/_pypy_openssl.pypy36-pp73-x86_64-linux-gnu.so > pypy3: /usr/lib/pypy3/lib_pypy/_resource_cffi.pypy36-pp73-x86_64-linux-gnu.so > pypy3: /usr/lib/pypy3/lib_pypy/_sha3/_sha3_cffi.pypy36-pp73-x86_64-linux-gnu.so > pypy3: /usr/lib/pypy3/lib_pypy/_sqlite3_cffi.pypy36-pp73-x86_64-linux-gnu.so > pypy3: /usr/lib/pypy3/lib_pypy/_syslog_cffi.pypy36-pp73-x86_64-linux-gnu.so > pypy3: /usr/share/doc/pypy3/TODO.Debian > pypy3: /usr/share/doc/pypy3/changelog.Debian.gz > pypy3: /usr/share/doc/pypy3/copyright > pypy3: /usr/share/lintian/overrides/pypy3 > pypy3: /usr/share/man/man1/pypy3.1.gz > >> + (native-search-paths >> + (list (search-path-specification >> + (variable "PYTHONPATH") >> + (files '("lib/pypy3.6/site-packages"))))) > > About this search path, how it is supposed to work? The version is > wrong (3.6) and the package doesn't include this directory, and we do > not have a pypy build system or other pypy packages that would make use > of it. Am I missing something? > > Thank you! > > Maxim I've upgraded pypy to version 7.3.5 and pushed the non-controversial changes from your 'small fixes' commit above as 7e05fda6e8dd72b63009181072a282ad15c25c4f. We can fix the search path after the next core-updates merge, where we'll be able to uniformize how it works with Python (GUIX_PYTHONPATH). Thanks, Closing. Maxim
GNU bug tracking system
Copyright (C) 1999 Darren O. Benham,
1997,2003 nCipher Corporation Ltd,
1994-97 Ian Jackson.