GNU bug report logs - #30537
glibc 2.26 refuses to run on CentOS 6.8

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 30537 in the body.
You can then email your comments to 30537 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Ricardo Wurmus <ricardo.wurmus <at> mdc-berlin.de>
To: <bug-guix <at> gnu.org>
Cc: guix-devel <at> gnu.org
Subject: glibc 2.26 refuses to run on CentOS 6.8
Date: Mon, 19 Feb 2018 19:46:02 +0100

[Message part 1 (text/plain, inline)]

Hi Guix,

I have a bad day.  After the upgrade to glibc 2.26 none of the
Guix-installed software runs on the HPC cluster running CentOS 6.8.

The glibc 2.26 expects a minimum kernel version of 3.x on x86_64, but
CentOS 6.8 only comes with a heavily patched 2.6.32.

The NixOS developers patch glibc to make sure that all software still
runs on Linux 2.6.32:

    https://raw.githubusercontent.com/NixOS/nixpkgs/master/pkgs/development/libraries/glibc/allow-kernel-2.6.32.patch

Can we please also apply this?  Without this Guix on HPC is pretty much
dead at the MDC.

[allow-kernel-2.6.32.patch (text/x-patch, inline)]

diff --git a/sysdeps/unix/sysv/linux/configure b/sysdeps/unix/sysv/linux/configure
index cace758c01..38fe7fe0b0 100644
--- a/sysdeps/unix/sysv/linux/configure
+++ b/sysdeps/unix/sysv/linux/configure
@@ -69,7 +69,7 @@ fi
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for kernel header at least $minimum_kernel" >&5
 $as_echo_n "checking for kernel header at least $minimum_kernel... " >&6; }
 decnum=`echo "$minimum_kernel.0.0.0" | sed 's/\([0-9]*\)\.\([0-9]*\)\.\([0-9]*\).*/(\1 * 65536 + \2 * 256 + \3)/'`;
-abinum=`echo "$minimum_kernel.0.0.0" | sed 's/\([0-9]*\)\.\([0-9]*\)\.\([0-9]*\).*/\1,\2,\3/'`;
+abinum=`echo "2.6.32.0.0.0" | sed 's/\([0-9]*\)\.\([0-9]*\)\.\([0-9]*\).*/\1,\2,\3/'`;
 cat confdefs.h - <<_ACEOF >conftest.$ac_ext
 /* end confdefs.h.  */
 #include <linux/version.h>
diff --git a/sysdeps/unix/sysv/linux/configure.ac b/sysdeps/unix/sysv/linux/configure.ac
index 13abda0a51..6abc12eaed 100644
--- a/sysdeps/unix/sysv/linux/configure.ac
+++ b/sysdeps/unix/sysv/linux/configure.ac
@@ -50,7 +50,7 @@ fi
 AC_MSG_CHECKING(for kernel header at least $minimum_kernel)
 changequote(,)dnl
 decnum=`echo "$minimum_kernel.0.0.0" | sed 's/\([0-9]*\)\.\([0-9]*\)\.\([0-9]*\).*/(\1 * 65536 + \2 * 256 + \3)/'`;
-abinum=`echo "$minimum_kernel.0.0.0" | sed 's/\([0-9]*\)\.\([0-9]*\)\.\([0-9]*\).*/\1,\2,\3/'`;
+abinum=`echo "2.6.32.0.0.0" | sed 's/\([0-9]*\)\.\([0-9]*\)\.\([0-9]*\).*/\1,\2,\3/'`;
 changequote([,])dnl
 AC_TRY_COMPILE([#include <linux/version.h>
 #if LINUX_VERSION_CODE < $decnum
diff --git a/sysdeps/unix/sysv/linux/dl-osinfo.h b/sysdeps/unix/sysv/linux/dl-osinfo.h
index 823cd8224d..482caaeeec 100644
--- a/sysdeps/unix/sysv/linux/dl-osinfo.h
+++ b/sysdeps/unix/sysv/linux/dl-osinfo.h
@@ -39,7 +39,7 @@
 	  GLRO(dl_osversion) = version;					      \
 									      \
 	/* Now we can test with the required version.  */		      \
-	if (__LINUX_KERNEL_VERSION > 0 && version < __LINUX_KERNEL_VERSION)   \
+	if (__LINUX_KERNEL_VERSION > 0 && version < __LINUX_KERNEL_VERSION && version != 0x020620)   \
 	  /* Not sufficent.  */						      \
 	  FATAL ("FATAL: kernel too old\n");				      \
       }									      \

[Message part 3 (text/plain, inline)]

--
Ricardo

Message #8 received at 30537 <at> debbugs.gnu.org (full text, mbox):

From: Ricardo Wurmus <ricardo.wurmus <at> mdc-berlin.de>
To: <guix-devel <at> gnu.org>
Cc: 30537 <at> debbugs.gnu.org
Subject: Re: glibc 2.26 refuses to run on CentOS 6.8
Date: Mon, 19 Feb 2018 20:41:07 +0100

[Message part 1 (text/plain, inline)]

Ricardo Wurmus <ricardo.wurmus <at> mdc-berlin.de> writes:

> Ricardo Wurmus <ricardo.wurmus <at> mdc-berlin.de> writes:
>
>> Ricardo Wurmus <ricardo.wurmus <at> mdc-berlin.de> writes:
>>
>>> The NixOS developers patch glibc to make sure that all software still
>>> runs on Linux 2.6.32:
>>>
>>>     https://raw.githubusercontent.com/NixOS/nixpkgs/master/pkgs/development/libraries/glibc/allow-kernel-2.6.32.patch
>>>
>>> Can we please also apply this?
>>
>> We could also apply this conditionally to x86_64 only, because that’s
>> probably the only architecture that’s used for HPC systems running
>> CentOS.
>
> Here’s a patch to graft the glibc to apply the patch to allow the 2.6.32
> kernel.  I’m going to apply this at work now.

That patch had a couple of problems.  Here’s a new version.

The main differences are:

* fix an undefined variable (“base” –> “glibc”)
* use package/inherit for glibc-final and
  glibc-final-with-bootstrap-bash; don’t override the “replacement”
  field with “#f”

-- 
Ricardo

[0001-WIP-graft-glibc-to-allow-execution-on-Linux-2.6.32.patch (text/x-patch, inline)]

From 3cf46dce4129cb52574de52b4221f6c4dde566fe Mon Sep 17 00:00:00 2001
From: Ricardo Wurmus <rekado <at> elephly.net>
Date: Mon, 19 Feb 2018 20:04:06 +0100
Subject: [PATCH] WIP graft glibc to allow execution on Linux 2.6.32.

* gnu/packages/patches/glibc-allow-kernel-2.6.32.patch: New file.
* gnu/local.mk (dist_patch_DATA): Add it.
* gnu/packages/base.scm (glibc/linux)[replacement]: New field.
(glibc-2.26-patched): New variable.
(glibc-2.25, glibc-2.24, glibc-2.23, glibc-2.22): Disable replacement.
* gnu/packages/commencement.scm (glibc-final-with-bootstrap-bash,
glibc-final): Use package/inherit to enable grafted glibc.
---
 gnu/local.mk                                       |  3 +-
 gnu/packages/base.scm                              | 15 +++++++++
 gnu/packages/commencement.scm                      |  5 +--
 .../patches/glibc-allow-kernel-2.6.32.patch        | 39 ++++++++++++++++++++++
 4 files changed, 59 insertions(+), 3 deletions(-)
 create mode 100644 gnu/packages/patches/glibc-allow-kernel-2.6.32.patch

diff --git a/gnu/local.mk b/gnu/local.mk
index f1834e368..c7bd4c6a9 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -7,7 +7,7 @@
 # Copyright © 2016, 2017 Kei Kebreau <kkebreau <at> posteo.net>
 # Copyright © 2016, 2017 Rene Saavedra <rennes <at> openmailbox.org>
 # Copyright © 2016 Adonay "adfeno" Felipe Nogueira <https://libreplanet.org/wiki/User:Adfeno> <adfeno <at> openmailbox.org>
-# Copyright © 2016, 2017 Ricardo Wurmus <rekado <at> elephly.net>
+# Copyright © 2016, 2017, 2018 Ricardo Wurmus <rekado <at> elephly.net>
 # Copyright © 2016 Ben Woodcroft <donttrustben <at> gmail.com>
 # Copyright © 2016, 2017, 2018 Alex Vong <alexvong1995 <at> gmail.com>
 # Copyright © 2016, 2017 Efraim Flashner <efraim <at> flashner.co.il>
@@ -705,6 +705,7 @@ dist_patch_DATA =						\
   %D%/packages/patches/glibc-CVE-2017-1000366-pt1.patch		\
   %D%/packages/patches/glibc-CVE-2017-1000366-pt2.patch		\
   %D%/packages/patches/glibc-CVE-2017-1000366-pt3.patch		\
+  %D%/packages/patches/glibc-allow-kernel-2.6.32.patch		\
   %D%/packages/patches/glibc-bootstrap-system.patch		\
   %D%/packages/patches/glibc-ldd-x86_64.patch			\
   %D%/packages/patches/glibc-locales.patch			\
diff --git a/gnu/packages/base.scm b/gnu/packages/base.scm
index b2c1d232f..111bbbcec 100644
--- a/gnu/packages/base.scm
+++ b/gnu/packages/base.scm
@@ -12,6 +12,7 @@
 ;;; Copyright © 2017 Mathieu Othacehe <m.othacehe <at> gmail.com>
 ;;; Copyright © 2017 Marius Bakke <mbakke <at> fastmail.com>
 ;;; Copyright © 2017 Eric Bavier <bavier <at> member.fsf.org>
+;;; Copyright © 2018 Ricardo Wurmus <rekado <at> elephly.net>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -537,6 +538,7 @@ store.")
    ;; Note: Always use a dot after the minor version since various places rely
    ;; on "version-major+minor" to determine where locales are found.
    (version "2.26.105-g0890d5379c")
+   (replacement glibc-2.26-patched)
    (source (origin
             (method url-fetch)
             (uri (string-append "https://alpha.gnu.org/gnu/guix/mirror/"
@@ -839,10 +841,20 @@ GLIBC/HURD for a Hurd host"
 ;; Below are old libc versions, which we use mostly to build locale data in
 ;; the old format (which the new libc cannot cope with.)
 
+(define glibc-2.26-patched
+  (package
+    (inherit glibc)
+    (replacement #f)
+    (source (origin
+              (inherit (package-source glibc))
+              (patches (cons (search-patch "glibc-allow-kernel-2.6.32.patch")
+                             (origin-patches (package-source glibc))))))))
+
 (define-public glibc-2.25
   (package
     (inherit glibc)
     (version "2.25")
+    (replacement #f)
     (source (origin
               (inherit (package-source glibc))
               (uri (string-append "mirror://gnu/glibc/glibc-"
@@ -862,6 +874,7 @@ GLIBC/HURD for a Hurd host"
   (package
     (inherit glibc)
     (version "2.24")
+    (replacement #f)
     (source (origin
               (inherit (package-source glibc))
               (uri (string-append "mirror://gnu/glibc/glibc-"
@@ -882,6 +895,7 @@ GLIBC/HURD for a Hurd host"
   (package
     (inherit glibc)
     (version "2.23")
+    (replacement #f)
     (source (origin
               (inherit (package-source glibc))
               (uri (string-append "mirror://gnu/glibc/glibc-"
@@ -905,6 +919,7 @@ GLIBC/HURD for a Hurd host"
   (package
     (inherit glibc)
     (version "2.22")
+    (replacement #f)
     (source (origin
               (inherit (package-source glibc))
               (uri (string-append "mirror://gnu/glibc/glibc-"
diff --git a/gnu/packages/commencement.scm b/gnu/packages/commencement.scm
index 7286e954c..db43691fd 100644
--- a/gnu/packages/commencement.scm
+++ b/gnu/packages/commencement.scm
@@ -4,6 +4,7 @@
 ;;; Copyright © 2012 Nikita Karetnikov <nikita <at> karetnikov.org>
 ;;; Copyright © 2014, 2015, 2017 Mark H Weaver <mhw <at> netris.org>
 ;;; Copyright © 2017, 2018 Efraim Flashner <efraim <at> flashner.co.il>
+;;; Copyright © 2018 Ricardo Wurmus <rekado <at> elephly.net>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -486,7 +487,7 @@ the bootstrap environment."
   ;; built just below; the only difference is that this one uses the
   ;; bootstrap Bash.
   (package-with-bootstrap-guile
-   (package (inherit glibc)
+   (package/inherit glibc
      (name "glibc-intermediate")
      (arguments
       `(#:guile ,%bootstrap-guile
@@ -664,7 +665,7 @@ exec ~a/bin/~a-~a -B~a/lib -Wl,-dynamic-linker -Wl,~a/~a \"$@\"~%"
 
 (define glibc-final
   ;; The final glibc, which embeds the statically-linked Bash built above.
-  (package (inherit glibc-final-with-bootstrap-bash)
+  (package/inherit glibc-final-with-bootstrap-bash
     (name "glibc")
     (inputs `(("static-bash" ,static-bash-for-glibc)
               ,@(alist-delete
diff --git a/gnu/packages/patches/glibc-allow-kernel-2.6.32.patch b/gnu/packages/patches/glibc-allow-kernel-2.6.32.patch
new file mode 100644
index 000000000..ce18b874c
--- /dev/null
+++ b/gnu/packages/patches/glibc-allow-kernel-2.6.32.patch
@@ -0,0 +1,39 @@
+diff --git a/sysdeps/unix/sysv/linux/configure b/sysdeps/unix/sysv/linux/configure
+index cace758c01..38fe7fe0b0 100644
+--- a/sysdeps/unix/sysv/linux/configure
++++ b/sysdeps/unix/sysv/linux/configure
+@@ -69,7 +69,7 @@ fi
+ { $as_echo "$as_me:${as_lineno-$LINENO}: checking for kernel header at least $minimum_kernel" >&5
+ $as_echo_n "checking for kernel header at least $minimum_kernel... " >&6; }
+ decnum=`echo "$minimum_kernel.0.0.0" | sed 's/\([0-9]*\)\.\([0-9]*\)\.\([0-9]*\).*/(\1 * 65536 + \2 * 256 + \3)/'`;
+-abinum=`echo "$minimum_kernel.0.0.0" | sed 's/\([0-9]*\)\.\([0-9]*\)\.\([0-9]*\).*/\1,\2,\3/'`;
++abinum=`echo "2.6.32.0.0.0" | sed 's/\([0-9]*\)\.\([0-9]*\)\.\([0-9]*\).*/\1,\2,\3/'`;
+ cat confdefs.h - <<_ACEOF >conftest.$ac_ext
+ /* end confdefs.h.  */
+ #include <linux/version.h>
+diff --git a/sysdeps/unix/sysv/linux/configure.ac b/sysdeps/unix/sysv/linux/configure.ac
+index 13abda0a51..6abc12eaed 100644
+--- a/sysdeps/unix/sysv/linux/configure.ac
++++ b/sysdeps/unix/sysv/linux/configure.ac
+@@ -50,7 +50,7 @@ fi
+ AC_MSG_CHECKING(for kernel header at least $minimum_kernel)
+ changequote(,)dnl
+ decnum=`echo "$minimum_kernel.0.0.0" | sed 's/\([0-9]*\)\.\([0-9]*\)\.\([0-9]*\).*/(\1 * 65536 + \2 * 256 + \3)/'`;
+-abinum=`echo "$minimum_kernel.0.0.0" | sed 's/\([0-9]*\)\.\([0-9]*\)\.\([0-9]*\).*/\1,\2,\3/'`;
++abinum=`echo "2.6.32.0.0.0" | sed 's/\([0-9]*\)\.\([0-9]*\)\.\([0-9]*\).*/\1,\2,\3/'`;
+ changequote([,])dnl
+ AC_TRY_COMPILE([#include <linux/version.h>
+ #if LINUX_VERSION_CODE < $decnum
+diff --git a/sysdeps/unix/sysv/linux/dl-osinfo.h b/sysdeps/unix/sysv/linux/dl-osinfo.h
+index 823cd8224d..482caaeeec 100644
+--- a/sysdeps/unix/sysv/linux/dl-osinfo.h
++++ b/sysdeps/unix/sysv/linux/dl-osinfo.h
+@@ -39,7 +39,7 @@
+ 	  GLRO(dl_osversion) = version;					      \
+ 									      \
+ 	/* Now we can test with the required version.  */		      \
+-	if (__LINUX_KERNEL_VERSION > 0 && version < __LINUX_KERNEL_VERSION)   \
++	if (__LINUX_KERNEL_VERSION > 0 && version < __LINUX_KERNEL_VERSION && version != 0x020620)   \
+ 	  /* Not sufficent.  */						      \
+ 	  FATAL ("FATAL: kernel too old\n");				      \
+       }									      \
-- 
2.16.2

Message #11 received at 30537 <at> debbugs.gnu.org (full text, mbox):

From: Jan Nieuwenhuizen <janneke <at> gnu.org>
To: Ricardo Wurmus <ricardo.wurmus <at> mdc-berlin.de>
Cc: guix-devel <at> gnu.org, 30537 <at> debbugs.gnu.org
Subject: Re: glibc 2.26 refuses to run on CentOS 6.8
Date: Mon, 19 Feb 2018 21:28:25 +0100

Ricardo Wurmus writes:

>> Here’s a patch to graft the glibc to apply the patch to allow the 2.6.32
>> kernel.  I’m going to apply this at work now.
>
> That patch had a couple of problems.  Here’s a new version.

Sad to hear of your troubles, thanks a lot for posting this.  We
discovered at Verum that a guix pack would not run on CentOS and
took another road in the end.  Good to know there's still a way
to work around this.

Thanks,
janneke

-- 
Jan Nieuwenhuizen <janneke <at> gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

Message #14 received at 30537 <at> debbugs.gnu.org (full text, mbox):

From: Ricardo Wurmus <rekado <at> elephly.net>
To: Danny Milosavljevic <dannym <at> scratchpost.org>
Cc: guix-devel <at> gnu.org, Ricardo Wurmus <ricardo.wurmus <at> mdc-berlin.de>,
 30537 <at> debbugs.gnu.org
Subject: Re: glibc 2.26 refuses to run on CentOS 6.8
Date: Mon, 19 Feb 2018 22:22:28 +0100

Hi Danny,

> Can you try just passing --enable-kernel=2.6.32 to "configure" of glibc instead?
>
> It should set the minimal version without any weird patching.

Does this work even though the official minimum kernel version for glibc
2.26 is 3.2.0?

> But newer glibc has moved a lot of kernel definitions into glibc, might cause a
> problem if glibc just assumes it's all there but in fact it's not there at
> runtime (like the recent Haskell problem etc).

The Red Hat kernels are a bit special in that they are not just old
kernels, but heavily patched to work with newer software.  The Nix
people wrote that they have confirmed that 2.6.32 works up to
glibc-2.26-131.

There are additional notes on how that was done:

    # HOWTO: check glibc sources for changes in kernel requirements
    git log -p glibc-2.25.. sysdeps/unix/sysv/linux/x86_64/kernel-features.h sysdeps/unix/sysv/linux/kernel-features.h
    # get kernel sources (update the URL)
    mkdir tmp && cd tmp
    curl http://vault.centos.org/6.9/os/Source/SPackages/kernel-2.6.32-696.el6.src.rpm | rpm2cpio - | cpio -idmv
    tar xf linux-*.bz2
    # check syscall presence, for example
    less linux-*?/arch/x86/kernel/syscall_table_32.S

If there was a way to test for kernel features instead of looking at the
kernel version I’d do that instead of looking for a way to relax the
lower kernel version bound.

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Message #17 received at 30537 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: Ricardo Wurmus <rekado <at> elephly.net>
Cc: guix-devel <at> gnu.org, Ricardo Wurmus <ricardo.wurmus <at> mdc-berlin.de>,
 30537 <at> debbugs.gnu.org
Subject: Re: glibc 2.26 refuses to run on CentOS 6.8
Date: Mon, 19 Feb 2018 23:46:03 +0100

Hi Ricardo,

> Does this work even though the official minimum kernel version for glibc
> 2.26 is 3.2.0?

I think so, BUT the patchset looks pretty similar to what would happen
if you specified the configure flag except for one spot.
So maybe Nix found out some ill effects.

The most worrying part in glibc is

#if __LINUX_KERNEL_VERSION < 0x040300
# undef __ASSUME_ACCEPT4_SYSCALL
# undef __ASSUME_SENDMSG_SYSCALL
# undef __ASSUME_RECVMSG_SYSCALL
# undef __ASSUME_CONNECT_SYSCALL
# undef __ASSUME_RECVFROM_SYSCALL
# undef __ASSUME_SENDTO_SYSCALL
#endif

So that would have to be watched out for.

__ASSUME_CONNECT_SYSCALL is some scary stuff.  Getting it wrong could break all
networking in the system.

> The Red Hat kernels are a bit special in that they are not just old
> kernels, but heavily patched to work with newer software.  The Nix
> people wrote that they have confirmed that 2.6.32 works up to
> glibc-2.26-131.

Oh, I didn't know that.  If it's tested that way, let's use it that way
for the time being.

> If there was a way to test for kernel features instead of looking at the
> kernel version I’d do that instead of looking for a way to relax the
> lower kernel version bound.

Yeah...

Message #20 received at 30537 <at> debbugs.gnu.org (full text, mbox):

From: Leo Famulari <leo <at> famulari.name>
To: Ricardo Wurmus <ricardo.wurmus <at> mdc-berlin.de>
Cc: 30537 <at> debbugs.gnu.org
Subject: Re: glibc 2.26 refuses to run on CentOS 6.8
Date: Mon, 19 Feb 2018 20:22:29 -0500

[Message part 1 (text/plain, inline)]

On Mon, Feb 19, 2018 at 07:46:02PM +0100, Ricardo Wurmus wrote:
> The NixOS developers patch glibc to make sure that all software still
> runs on Linux 2.6.32:
> 
>     https://raw.githubusercontent.com/NixOS/nixpkgs/master/pkgs/development/libraries/glibc/allow-kernel-2.6.32.patch
> 
> Can we please also apply this?  Without this Guix on HPC is pretty much
> dead at the MDC.

My questions are, how does the glibc team choose the minimum kernel
version? What could go wrong if we apply this patch and somebody uses
Guix on a pre-3.2.0 kernel?

Perhaps they simply chose to not support glibc on any kernel that is not
supported upstream; 3.2.x is the oldest supported release series. But,
we should have some idea of their reasoning, in my opinion.

[signature.asc (application/pgp-signature, inline)]

Message #23 received at 30537 <at> debbugs.gnu.org (full text, mbox):

From: Efraim Flashner <efraim <at> flashner.co.il>
To: Ricardo Wurmus <ricardo.wurmus <at> mdc-berlin.de>
Cc: guix-devel <at> gnu.org, 30537 <at> debbugs.gnu.org
Subject: Re: bug#30537: glibc 2.26 refuses to run on CentOS 6.8
Date: Tue, 20 Feb 2018 11:39:10 +0200

[Message part 1 (text/plain, inline)]

On Mon, Feb 19, 2018 at 07:46:02PM +0100, Ricardo Wurmus wrote:
> Hi Guix,
> 
> I have a bad day.  After the upgrade to glibc 2.26 none of the
> Guix-installed software runs on the HPC cluster running CentOS 6.8.
> 
> The glibc 2.26 expects a minimum kernel version of 3.x on x86_64, but
> CentOS 6.8 only comes with a heavily patched 2.6.32.
> 
> The NixOS developers patch glibc to make sure that all software still
> runs on Linux 2.6.32:
> 
>     https://raw.githubusercontent.com/NixOS/nixpkgs/master/pkgs/development/libraries/glibc/allow-kernel-2.6.32.patch
> 
> Can we please also apply this?  Without this Guix on HPC is pretty much
> dead at the MDC.
> 

We should also update the docs with the (new) minimum specs as needed.

-- 
Efraim Flashner   <efraim <at> flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[signature.asc (application/pgp-signature, inline)]

Message #26 received at 30537 <at> debbugs.gnu.org (full text, mbox):

From: Leo Famulari <leo <at> famulari.name>
To: Ricardo Wurmus <ricardo.wurmus <at> mdc-berlin.de>
Cc: 30537 <at> debbugs.gnu.org
Subject: Re: glibc 2.26 refuses to run on CentOS 6.8
Date: Tue, 20 Feb 2018 06:52:21 -0500

[Message part 1 (text/plain, inline)]

On Mon, Feb 19, 2018 at 08:22:29PM -0500, Leo Famulari wrote:
> My questions are, how does the glibc team choose the minimum kernel
> version? What could go wrong if we apply this patch and somebody uses
> Guix on a pre-3.2.0 kernel?
> 
> Perhaps they simply chose to not support glibc on any kernel that is not
> supported upstream; 3.2.x is the oldest supported release series. But,
> we should have some idea of their reasoning, in my opinion.

As far as I can tell, the discussion started on the libc-alpha mailing
list in January 2016:

https://sourceware.org/ml/libc-alpha/2016-01/msg00885.html

... and continued into February:

https://sourceware.org/ml/libc-alpha/2016-02/msg00002.html

These messages can provide some useful context.

[signature.asc (application/pgp-signature, inline)]

Message #29 received at 30537 <at> debbugs.gnu.org (full text, mbox):

From: Ricardo Wurmus <ricardo.wurmus <at> mdc-berlin.de>
To: Leo Famulari <leo <at> famulari.name>
Cc: 30537 <at> debbugs.gnu.org
Subject: Re: glibc 2.26 refuses to run on CentOS 6.8
Date: Tue, 20 Feb 2018 13:34:27 +0100

Leo Famulari <leo <at> famulari.name> writes:

> On Mon, Feb 19, 2018 at 08:22:29PM -0500, Leo Famulari wrote:
>> My questions are, how does the glibc team choose the minimum kernel
>> version? What could go wrong if we apply this patch and somebody uses
>> Guix on a pre-3.2.0 kernel?
>>
>> Perhaps they simply chose to not support glibc on any kernel that is not
>> supported upstream; 3.2.x is the oldest supported release series. But,
>> we should have some idea of their reasoning, in my opinion.
>
> As far as I can tell, the discussion started on the libc-alpha mailing
> list in January 2016:
>
> https://sourceware.org/ml/libc-alpha/2016-01/msg00885.html
>
> ... and continued into February:
>
> https://sourceware.org/ml/libc-alpha/2016-02/msg00002.html
>
> These messages can provide some useful context.

The only reason for moving the lower bound to Linux 3.2 is that 2.6 has
reached EOL.  This allows the glibc developers to assume certain
kernel features and simplify their code.

The RHEL kernels are special, though, in that they are continuously
patched beyond recognition, backporting features.  A RHEL 2.6.32 kernel
is very different from a stock 2.6.32 kernel.

The patch explicitly permits a *single* extra kernel version (0x020620 =
2.6.32) at runtime, not *all* older kernels, so it isn’t as bad as it
may seem.

For future updates to the glibc we would have to re-evaluate if the
current RHEL 6.x kernel still supports all features the glibc expects,
and decide once more if we can justify patching glibc to allow that one
particular kernel version.

--
Ricardo

Message #32 received at 30537 <at> debbugs.gnu.org (full text, mbox):

From: Leo Famulari <leo <at> famulari.name>
To: Ricardo Wurmus <ricardo.wurmus <at> mdc-berlin.de>
Cc: 30537 <at> debbugs.gnu.org
Subject: Re: glibc 2.26 refuses to run on CentOS 6.8
Date: Tue, 20 Feb 2018 07:51:36 -0500

[Message part 1 (text/plain, inline)]

On Tue, Feb 20, 2018 at 01:34:27PM +0100, Ricardo Wurmus wrote:
> The only reason for moving the lower bound to Linux 3.2 is that 2.6 has
> reached EOL.  This allows the glibc developers to assume certain
> kernel features and simplify their code.
> 
> The RHEL kernels are special, though, in that they are continuously
> patched beyond recognition, backporting features.  A RHEL 2.6.32 kernel
> is very different from a stock 2.6.32 kernel.
> 
> The patch explicitly permits a *single* extra kernel version (0x020620 =
> 2.6.32) at runtime, not *all* older kernels, so it isn’t as bad as it
> may seem.

Okay, thanks for the clarification.

> For future updates to the glibc we would have to re-evaluate if the
> current RHEL 6.x kernel still supports all features the glibc expects,
> and decide once more if we can justify patching glibc to allow that one
> particular kernel version.

Yes... and this will probably continue for many years. But I do think we
should do something to work around the issue now, and reevaluate our
solution when the pressure is off.

It would be nice if the graft only applied to x86_64-linux, but I've
never considered if that is easy to do or not.

[signature.asc (application/pgp-signature, inline)]

Message #35 received at 30537 <at> debbugs.gnu.org (full text, mbox):

From: Ricardo Wurmus <ricardo.wurmus <at> mdc-berlin.de>
To: Leo Famulari <leo <at> famulari.name>
Cc: 30537 <at> debbugs.gnu.org
Subject: Re: glibc 2.26 refuses to run on CentOS 6.8
Date: Tue, 20 Feb 2018 15:33:56 +0100

Leo Famulari <leo <at> famulari.name> writes:

>> For future updates to the glibc we would have to re-evaluate if the
>> current RHEL 6.x kernel still supports all features the glibc expects,
>> and decide once more if we can justify patching glibc to allow that one
>> particular kernel version.
>
> Yes... and this will probably continue for many years. But I do think we
> should do something to work around the issue now, and reevaluate our
> solution when the pressure is off.
>
> It would be nice if the graft only applied to x86_64-linux, but I've
> never considered if that is easy to do or not.

I don’t know if we can graft a package only for a single architecture.
At least at the time of ungrafting we could apply the patch only on
x86_64 (and only rebuild the world for that architecture).

FWIW: I’ve applied this patch to the installation at the MDC and it
works fine.  The biggest downside here is the slowness of grafts over
NFS (I still need to update the protocol to NFS 4.1) and the many lines
of output that are amplified by communicating with a remote daemon.

Other than that I’m happy that this crisis could be temporarily averted.

I’d like to have this in master, though, so that people can run “guix
pull” again and actually get working software.

--
Ricardo

Message #38 received at 30537 <at> debbugs.gnu.org (full text, mbox):

From: Ricardo Wurmus <ricardo.wurmus <at> mdc-berlin.de>
Cc: 30537 <at> debbugs.gnu.org
Subject: Re: glibc 2.26 refuses to run on CentOS 6.8
Date: Tue, 20 Feb 2018 18:55:48 +0100

Ricardo Wurmus <ricardo.wurmus <at> mdc-berlin.de> writes:

> Leo Famulari <leo <at> famulari.name> writes:
>
>>> For future updates to the glibc we would have to re-evaluate if the
>>> current RHEL 6.x kernel still supports all features the glibc expects,
>>> and decide once more if we can justify patching glibc to allow that one
>>> particular kernel version.
>>
>> Yes... and this will probably continue for many years. But I do think we
>> should do something to work around the issue now, and reevaluate our
>> solution when the pressure is off.
>>
>> It would be nice if the graft only applied to x86_64-linux, but I've
>> never considered if that is easy to do or not.
>
> I don’t know if we can graft a package only for a single architecture.
> At least at the time of ungrafting we could apply the patch only on
> x86_64 (and only rebuild the world for that architecture).
>
> FWIW: I’ve applied this patch to the installation at the MDC and it
> works fine.  The biggest downside here is the slowness of grafts over
> NFS (I still need to update the protocol to NFS 4.1) and the many lines
> of output that are amplified by communicating with a remote daemon.
>
> Other than that I’m happy that this crisis could be temporarily averted.
>
> I’d like to have this in master, though, so that people can run “guix
> pull” again and actually get working software.

I should note that now every Guix command first prints this:

  GC Warning: pthread_getattr_np or pthread_attr_getstack failed for main thread

I suspect that this is not good.  Guix was built with the grafted glibc.

--
Ricardo

Message #41 received at 30537 <at> debbugs.gnu.org (full text, mbox):

From: Mark H Weaver <mhw <at> netris.org>
To: Ricardo Wurmus <ricardo.wurmus <at> mdc-berlin.de>
Cc: guix-devel <at> gnu.org, 30537 <at> debbugs.gnu.org
Subject: Re: bug#30537: glibc 2.26 refuses to run on CentOS 6.8
Date: Wed, 21 Feb 2018 18:12:31 -0500

Hi Ricardo,

Ricardo Wurmus <ricardo.wurmus <at> mdc-berlin.de> writes:

> Ricardo Wurmus <ricardo.wurmus <at> mdc-berlin.de> writes:
>
>> Here’s a patch to graft the glibc to apply the patch to allow the 2.6.32
>> kernel.  I’m going to apply this at work now.
>
> That patch had a couple of problems.  Here’s a new version.

[...]

> diff --git a/gnu/packages/base.scm b/gnu/packages/base.scm
> index b2c1d232f..111bbbcec 100644
> --- a/gnu/packages/base.scm
> +++ b/gnu/packages/base.scm
> @@ -12,6 +12,7 @@
>  ;;; Copyright © 2017 Mathieu Othacehe <m.othacehe <at> gmail.com>
>  ;;; Copyright © 2017 Marius Bakke <mbakke <at> fastmail.com>
>  ;;; Copyright © 2017 Eric Bavier <bavier <at> member.fsf.org>
> +;;; Copyright © 2018 Ricardo Wurmus <rekado <at> elephly.net>
>  ;;;
>  ;;; This file is part of GNU Guix.
>  ;;;
> @@ -537,6 +538,7 @@ store.")
>     ;; Note: Always use a dot after the minor version since various places rely
>     ;; on "version-major+minor" to determine where locales are found.
>     (version "2.26.105-g0890d5379c")
> +   (replacement glibc-2.26-patched)
>     (source (origin
>              (method url-fetch)
>              (uri (string-append "https://alpha.gnu.org/gnu/guix/mirror/"
> @@ -839,10 +841,20 @@ GLIBC/HURD for a Hurd host"
>  ;; Below are old libc versions, which we use mostly to build locale data in
>  ;; the old format (which the new libc cannot cope with.)
>  
> +(define glibc-2.26-patched
> +  (package
> +    (inherit glibc)
> +    (replacement #f)
> +    (source (origin
> +              (inherit (package-source glibc))
> +              (patches (cons (search-patch "glibc-allow-kernel-2.6.32.patch")
> +                             (origin-patches (package-source glibc))))))))
> +
>  (define-public glibc-2.25
>    (package
>      (inherit glibc)
>      (version "2.25")
> +    (replacement #f)
>      (source (origin
>                (inherit (package-source glibc))
>                (uri (string-append "mirror://gnu/glibc/glibc-"
> @@ -862,6 +874,7 @@ GLIBC/HURD for a Hurd host"
>    (package
>      (inherit glibc)
>      (version "2.24")
> +    (replacement #f)
>      (source (origin
>                (inherit (package-source glibc))
>                (uri (string-append "mirror://gnu/glibc/glibc-"
> @@ -882,6 +895,7 @@ GLIBC/HURD for a Hurd host"
>    (package
>      (inherit glibc)
>      (version "2.23")
> +    (replacement #f)
>      (source (origin
>                (inherit (package-source glibc))
>                (uri (string-append "mirror://gnu/glibc/glibc-"
> @@ -905,6 +919,7 @@ GLIBC/HURD for a Hurd host"
>    (package
>      (inherit glibc)
>      (version "2.22")
> +    (replacement #f)
>      (source (origin
>                (inherit (package-source glibc))
>                (uri (string-append "mirror://gnu/glibc/glibc-"

These (replacement #f) fields should not be needed.  'replacement' is
now an 'innate' field of the package record type, which means that it is
not inherited.

> diff --git a/gnu/packages/commencement.scm b/gnu/packages/commencement.scm
> index 7286e954c..db43691fd 100644
> --- a/gnu/packages/commencement.scm
> +++ b/gnu/packages/commencement.scm
> @@ -4,6 +4,7 @@
>  ;;; Copyright © 2012 Nikita Karetnikov <nikita <at> karetnikov.org>
>  ;;; Copyright © 2014, 2015, 2017 Mark H Weaver <mhw <at> netris.org>
>  ;;; Copyright © 2017, 2018 Efraim Flashner <efraim <at> flashner.co.il>
> +;;; Copyright © 2018 Ricardo Wurmus <rekado <at> elephly.net>
>  ;;;
>  ;;; This file is part of GNU Guix.
>  ;;;
> @@ -486,7 +487,7 @@ the bootstrap environment."
>    ;; built just below; the only difference is that this one uses the
>    ;; bootstrap Bash.
>    (package-with-bootstrap-guile
> -   (package (inherit glibc)
> +   (package/inherit glibc
>       (name "glibc-intermediate")
>       (arguments
>        `(#:guile ,%bootstrap-guile
> @@ -664,7 +665,7 @@ exec ~a/bin/~a-~a -B~a/lib -Wl,-dynamic-linker -Wl,~a/~a \"$@\"~%"
>  
>  (define glibc-final
>    ;; The final glibc, which embeds the statically-linked Bash built above.
> -  (package (inherit glibc-final-with-bootstrap-bash)
> +  (package/inherit glibc-final-with-bootstrap-bash
>      (name "glibc")
>      (inputs `(("static-bash" ,static-bash-for-glibc)
>                ,@(alist-delete

We seem to be oscillating on the question of whether to graft these
early GLIBCs.  In June 2017, I switched to using 'package/inherit' here
in commit 13f7f2fd2b208c29361ef2290f55911879a6adf2, and in October those
changes were reverted in commit 848f550f2c105326dc3be4033c8aaf35ec21cde4
by Efraim, although I'm not sure why.

It'll be painful to have *everything* grafted until the next
core-updates cycle, but I suppose it's necessary.

      Thanks,
        Mark

Message #44 received at 30537 <at> debbugs.gnu.org (full text, mbox):

From: Efraim Flashner <efraim <at> flashner.co.il>
To: Mark H Weaver <mhw <at> netris.org>
Cc: guix-devel <at> gnu.org, Ricardo Wurmus <ricardo.wurmus <at> mdc-berlin.de>,
 30537 <at> debbugs.gnu.org
Subject: Re: bug#30537: glibc 2.26 refuses to run on CentOS 6.8
Date: Thu, 22 Feb 2018 22:30:54 +0200

[Message part 1 (text/plain, inline)]

On Wed, Feb 21, 2018 at 06:12:31PM -0500, Mark H Weaver wrote:
> Hi Ricardo,
> 
...
> 
> We seem to be oscillating on the question of whether to graft these
> early GLIBCs.  In June 2017, I switched to using 'package/inherit' here
> in commit 13f7f2fd2b208c29361ef2290f55911879a6adf2, and in October those
> changes were reverted in commit 848f550f2c105326dc3be4033c8aaf35ec21cde4
> by Efraim, although I'm not sure why.
> 

Probably a misunderstanding on my part, I assumed it was switched to
package/inherit as part of the grafting, and then should be switched
back.

> It'll be painful to have *everything* grafted until the next
> core-updates cycle, but I suppose it's necessary.
> 
>       Thanks,
>         Mark
> 

We could also have an 'ungraft everything' branch before we get to the
next round of core-updates.

-- 
Efraim Flashner   <efraim <at> flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[signature.asc (application/pgp-signature, inline)]

Message #47 received at 30537 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Mark H Weaver <mhw <at> netris.org>
Cc: guix-devel <at> gnu.org, Ricardo Wurmus <ricardo.wurmus <at> mdc-berlin.de>,
 30537 <at> debbugs.gnu.org
Subject: Grafts vs. early bootstrapping packages
Date: Fri, 23 Feb 2018 23:01:25 +0100

Hello,

Mark H Weaver <mhw <at> netris.org> skribis:

>> Ricardo Wurmus <ricardo.wurmus <at> mdc-berlin.de> writes:

[...]

>>  (define glibc-final
>>    ;; The final glibc, which embeds the statically-linked Bash built above.
>> -  (package (inherit glibc-final-with-bootstrap-bash)
>> +  (package/inherit glibc-final-with-bootstrap-bash
>>      (name "glibc")
>>      (inputs `(("static-bash" ,static-bash-for-glibc)
>>                ,@(alist-delete
>
> We seem to be oscillating on the question of whether to graft these
> early GLIBCs.  In June 2017, I switched to using 'package/inherit' here
> in commit 13f7f2fd2b208c29361ef2290f55911879a6adf2, and in October those
> changes were reverted in commit 848f550f2c105326dc3be4033c8aaf35ec21cde4
> by Efraim, although I'm not sure why.

I doesn’t make sense to graft “glibc-intermediate” because it’s only
used in ‘static-bash-for-glibc’, which statically links against it.  The
situation is similar with the “-boot0” packages: they are not referenced
by the packages we use.

So I think 848f550f2c105326dc3be4033c8aaf35ec21cde4 was a good idea.
f00b85ff8d34df0a1879e593d4a85629b8586af7 does something similar.

Ludo’.

Message #50 received at 30537 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Ricardo Wurmus <ricardo.wurmus <at> mdc-berlin.de>
Cc: guix-devel <at> gnu.org, 30537 <at> debbugs.gnu.org
Subject: Re: glibc 2.26 refuses to run on CentOS 6.8
Date: Fri, 23 Feb 2018 23:26:12 +0100

Hello,

Ricardo Wurmus <ricardo.wurmus <at> mdc-berlin.de> skribis:

> I have a bad day.  After the upgrade to glibc 2.26 none of the
> Guix-installed software runs on the HPC cluster running CentOS 6.8.

Bah.  :-(

> The glibc 2.26 expects a minimum kernel version of 3.x on x86_64, but
> CentOS 6.8 only comes with a heavily patched 2.6.32.

It’s annoying, but we can surely apply the patch you sent (though rather
by passing ‘--enable-kernel’ if possible, as Danny suggested.)


personality(2) has a knob to change the kernel version reported by
uname(2) to 2.6.  Here it’s a case where we’d need the reverse:
reporting 3.2 instead of 2.6.  That doesn’t seem to exist.

Looking for other hacks (or kludges), I found the kernel module below at
<https://www.linuxquestions.org/questions/linux-general-1/uname-hack-modules-in-kernel-2-6-a-172167/>,
which could be adjusted to report a different kernel version:

--8<---------------cut here---------------start------------->8---
#include <linux/kernel.h>
#include <linux/module.h>
#include <linux/utsname.h>

#ifndef UNAME_DUMB_STEPPING
#define UNAME_DUMB_STEPPING '5';
#endif

MODULE_AUTHOR("The one who invented the uname hack");
MODULE_DESCRIPTION("Changes the uname output");
MODULE_LICENSE("GPL");

static int uname_hack_init() {
	save = system_utsname.machine[1];
	system_utsname.machine[1] = UNAME_DUMB_STEPPING;
	return 0;
}

static void uname_hack_cleanup() {
	system_utsname.machine[1] = save;
}

module_init(uname_hack_init);
module_exit(uname_hack_cleanup);
--8<---------------cut here---------------end--------------->8---

Another option would be to ptrace processes, handle the first ‘uname’
call, and then PTRACE_DETACH.  Ugly.

Ludo’.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.