Package: guix-patches;
Reported by: David Elsing <david.elsing <at> posteo.net>
Date: Wed, 6 Mar 2024 19:39:01 UTC
Severity: normal
Tags: patch
Done: Ludovic Courtès <ludo <at> gnu.org>
Bug is archived. No further changes may be made.
View this message in rfc822 format
From: David Elsing <david.elsing <at> posteo.net> To: 69591 <at> debbugs.gnu.org Cc: David Elsing <david.elsing <at> posteo.net> Subject: [bug#69591] [PATCH 00/31] Unbundle and update python-pytorch Date: Wed, 6 Mar 2024 19:37:23 +0000
This patch series unbundles all dependencies of python-pytorch (updated to version 2.2.1) and removes all* autogenerated files, completing the work by Ludo' [1]. I also applied the unbundling to python-pytorch-for-r-torch (version 2.0.1) and made sure that r-torch still builds. Originally, I wanted to package PyTorch with ROCm, but decided that it would be better to first update the package and complete the unbundling. This was more work than I initially thought though, so this patch series is quite long. Caffe2 is heavily integrated in the PyTorch code, so unbundling it does not make sense. For QNNPACK, the situation is strange: There is an old version [2] and an internal fork in aten/src/ATen/native/quantized/cpu/qnnpack, which PyTorch both depends on (as well as XNNPACK). The internal fork does not depend on other parts of the code currently, so I unbundled it as well as a package variant of qnnpack. ideep [3] has a separate branch for PyTorch (the master branch does not seem maintained), which only contains header files without build instructions. Therefore, I made a simple package with the header files. miniz also requires a package variant for PyTorch. Several scripts for autogenerated files depend on the compiled PyTorch library. However, it was possible to patch the scripts with empty values and later regenerate the files with the initial PyTorch version. Therefore, I introduced the 'codegen2, 'build2 and 'install2 build phases running after the initial library is installed. In 'codegen2, the installed library is used to regenerate the files properly, which leads to a recompilation of only 12 files in the 'build2 phase. The core tests can be run as well, but I disabled them as they take a long time after an already long build. I also wanted to try and build python-pytorch on i686-linux, as it doesn't require emulation on an x86_64 CPU, but it depends on Rust through python-cryptography, for which there are no substitutes available currently. Cheers, David * with the exception of test/cpp/api/init_baseline.h and test/cpp/api/optim_baseline.h, which are just lists of previously calculated numbers; or any other generated file I might have missed. [1] https://hpc.guix.info/blog/2021/09/whats-in-a-package/ [2] https://github.com/pytorch/QNNPACK [3] https://github.com/intel/ideep/tree/pytorch David Elsing (31): gnu: asmjit: Update to commit 3ca5c18. gnu: Add python-typing-extensions-4.10. gnu: Add python-optree. gnu: Add python-pytest-flakefinder. gnu: Add python-pytest-shard. gnu: Add python-expecttest. gnu: Add python-pytest-rerunfailures-13. gnu: Add miniz. gnu: Add miniz-for-pytorch. gnu: Add libnop. gnu: Remove flatbuffers-next-shared. gnu: python-flatbuffers-next: Update to 23.5.26. gnu: pthreadpool: Update to commit 178e3e0. gnu: cpuinfo: Update to commit aa4b216. gnu: clog: Add "-DUSE_SYSTEM_LIBS=ON" configure flag. gnu: nnpack: Update to commit 70a77f4. gnu: oneapi-dnnl: Update to 3.3.5. gnu: Add tensorpipe. gnu: Add fbgemm. gnu: Add qnnpack. gnu: Add foxi. gnu: Add ideep-pytorch. gnu: xnnpack: Update to commit 51a9875. gnu: Remove xnnpack-for-torch2. gnu: Add qnnpack-pytorch. gnu: python-pytorch: Update to 2.2.1 and unbundle dependencies. gnu: python-torchvision: Update to 0.17.1. gnu: Add ideep-pytorch-for-r-torch. gnu: Add oneapi-dnnl-for-r-torch. gnu: Add qnnpack-pytorch-for-r-torch. gnu: python-pytorch-for-r-torch: Adjust to new python-pytorch. gnu/local.mk | 9 +- gnu/packages/compression.scm | 49 + gnu/packages/cpp.scm | 6 +- gnu/packages/machine-learning.scm | 874 +++- gnu/packages/parallel.scm | 30 +- .../patches/clog-fix-shared-build.patch | 39 +- .../patches/cpuinfo-system-libraries.patch | 50 - .../patches/fbgemm-use-system-libraries.patch | 62 + gnu/packages/patches/foxi-fix-build.patch | 55 + gnu/packages/patches/miniz-for-pytorch.patch | 47 + .../patches/python-optree-fix-32-bit.patch | 122 + ...ython-pytorch-1.9.0-system-libraries.patch | 139 - .../patches/python-pytorch-fix-codegen.patch | 167 + ...thon-pytorch-for-r-torch-fix-codegen.patch | 167 + ...pytorch-for-r-torch-system-libraries.patch | 390 ++ .../patches/python-pytorch-runpath.patch | 19 +- .../python-pytorch-system-libraries.patch | 324 +- .../python-pytorch-without-kineto.patch | 60 + .../python-pytorch2-system-libraries.patch | 156 - .../xnnpack-for-torch2-system-libraries.patch | 2660 ----------- .../patches/xnnpack-remove-broken-tests.patch | 335 ++ .../patches/xnnpack-system-libraries.patch | 4133 ++++++++++------- gnu/packages/python-build.scm | 12 + gnu/packages/python-check.scm | 88 + gnu/packages/python-xyz.scm | 29 + gnu/packages/serialization.scm | 53 +- 26 files changed, 5111 insertions(+), 4964 deletions(-) delete mode 100644 gnu/packages/patches/cpuinfo-system-libraries.patch create mode 100644 gnu/packages/patches/fbgemm-use-system-libraries.patch create mode 100644 gnu/packages/patches/foxi-fix-build.patch create mode 100644 gnu/packages/patches/miniz-for-pytorch.patch create mode 100644 gnu/packages/patches/python-optree-fix-32-bit.patch delete mode 100644 gnu/packages/patches/python-pytorch-1.9.0-system-libraries.patch create mode 100644 gnu/packages/patches/python-pytorch-fix-codegen.patch create mode 100644 gnu/packages/patches/python-pytorch-for-r-torch-fix-codegen.patch create mode 100644 gnu/packages/patches/python-pytorch-for-r-torch-system-libraries.patch create mode 100644 gnu/packages/patches/python-pytorch-without-kineto.patch delete mode 100644 gnu/packages/patches/python-pytorch2-system-libraries.patch delete mode 100644 gnu/packages/patches/xnnpack-for-torch2-system-libraries.patch create mode 100644 gnu/packages/patches/xnnpack-remove-broken-tests.patch -- 2.41.0
GNU bug tracking system
Copyright (C) 1999 Darren O. Benham,
1997,2003 nCipher Corporation Ltd,
1994-97 Ian Jackson.