GNU bug report logs - #50151
Coreutils, aarch64 and chroot

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 50151 in the body.
You can then email your comments to 50151 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Frans de Boer <frans <at> fransdb.nl>
To: bug-coreutils <at> gnu.org
Subject: Coreutils, aarch64 and chroot
Date: Sat, 21 Aug 2021 12:33:29 +0200

LS,

I have an issue with chroot in the next context:

qemu-aarch64 -cpu cortex-a53 -L /newroot /newroot/usr/sbin/chroot /newroot \
  /usr/bin/env -i \
    HOME=/root \
    TERM="$TERM" \
    PS1='(arm chroot) \u:\w\$ ' \
    PATH=/usr/bin:/usr/sbin \
    /usr/bin/bash --login +h

qemu-aarch64 is running on top of x86_64 linux and chroot, env and bash 
are compiled as aarch64 binaries.
By themselves they all run under qemu-aarch64. But when using the above 
command, it always fails with 'Exec format error'. The same if running:

qemu-aarch64 -L /newroot /newroot/usr/sbin/chroot /newroot

It should run /bin/bash, but ends with the same error message.

Running 'qemu-aarch64 -L /newroot /newroot/usr/bin/bash -c /usr/bin/env 
--help' does show the env help text. So, I guess chroot is to blame?

Regards, Frans.


-- 
A: Yes, just like that                            A: Ja, net zo
Q: Oh, Just like reading a book backwards         Q: Oh, net als een boek achterstevoren lezen
A: Because it upsets the natural flow of a story  A: Omdat het de natuurlijke gang uit het verhaal haalt
Q: Why is top-posting annoying?                   Q: Waarom is Top-posting zo irritant?

Message #8 received at 50151 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Frans de Boer <frans <at> fransdb.nl>
Cc: 50151 <at> debbugs.gnu.org
Subject: Re: bug#50151: Coreutils, aarch64 and chroot
Date: Sat, 21 Aug 2021 09:43:33 -0700

On 8/21/21 3:33 AM, Frans de Boer wrote:
> So, I guess chroot is to blame?

Seems unlikely that it'd be a coreutils bug. Possibly you simply have 
the wrong type of executable installed as /newroot/usr/sbin/chroot.

If it is a coreutils bug, please give complete steps to let others 
reproduce the bug.

Message #11 received at 50151 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Frans de Boer <frans <at> fransdb.nl>
Cc: 50151 <at> debbugs.gnu.org
Subject: Re: bug#50151: Coreutils, aarch64 and chroot
Date: Tue, 24 Aug 2021 01:39:16 -0700

On 8/24/21 1:31 AM, Frans de Boer wrote:

> If it is the wrong executable, would it ever startup and continue and 
> (try to) load bash or env?

No.

> It even displays the help message!

Not from what you've shown so far. You reported the output of env 
--help, not the output of chroot --help.

> I can supply more, but what exactly would help you forward?

A complete set of instructions for an outsider to reproduce the problem 
from scratch.  Assume the outsider is running Fedora 34 x86-64 (since 
that's what I'm running :-).

However, I think it'll be a better use of our time for you to debug this 
one yourself. It doesn't sound like a Coreutils problem; it sounds like 
a problem in your virtual machine setup, and you're the best expert on 
that setup.

Message #14 received at 50151 <at> debbugs.gnu.org (full text, mbox):

From: Frans de Boer <frans <at> fransdb.nl>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: 50151 <at> debbugs.gnu.org
Subject: Re: bug#50151: Coreutils, aarch64 and chroot
Date: Tue, 24 Aug 2021 10:31:40 +0200

On 8/21/21 6:43 PM, Paul Eggert wrote:
> On 8/21/21 3:33 AM, Frans de Boer wrote:
>> So, I guess chroot is to blame?
>
> Seems unlikely that it'd be a coreutils bug. Possibly you simply have 
> the wrong type of executable installed as /newroot/usr/sbin/chroot.
>
> If it is a coreutils bug, please give complete steps to let others 
> reproduce the bug.
If it is the wrong executable, would it ever startup and continue and 
(try to) load bash or env?
It even displays the help message!

As I said before, on its own it does function but, as opposed to bash, 
it does NOT load other programs. All programs are compiled the same with 
aarch64-cross-linux tools.
No problem when operating under qemu-x86_64 or qemu-i386 - of course 
both compiled with their own cross compiler tools. I am currently trying 
this with other architectures (powerpc64)

I can supply more, but what exactly would help you forward?

--- Frans

Message #17 received at 50151 <at> debbugs.gnu.org (full text, mbox):

From: Assaf Gordon <assafgordon <at> gmail.com>
To: Frans de Boer <frans <at> fransdb.nl>
Cc: Paul Eggert <eggert <at> cs.ucla.edu>, 50151 <at> debbugs.gnu.org
Subject: Re: bug#50151: Coreutils, aarch64 and chroot
Date: Wed, 25 Aug 2021 02:16:24 -0600

Hello,

On 2021-08-24 2:39 a.m., Paul Eggert wrote:
> However, I think it'll be a better use of our time for you to debug this 
> one yourself. It doesn't sound like a Coreutils problem; it sounds like 
> a problem in your virtual machine setup, and you're the best expert on 
> that setup.

Few suggestions to check, that might help you and us to troubleshoot:

1. ensure the binaries are indeed for aarch64:

   file /newroot/usr/sbin/chroot
   file /newroot/usr/bin/env
   file /newroot/usr/bin/bash

it should say something like
  "ELF 64-bit LSB pie executable, ARM aarch64"
for all of them.


2. ensure each binary works by itself:

 qemu-aarch64 -L /newroot /newroot/usr/sbin/chroot --version
 qemu-aarch64 -L /newroot /newroot/usr/bin/env --version
 qemu-aarch64 -L /newroot /newroot/usr/bin/bash --version

(the actual version doesn't matter here, the main thing is that
the qemu user-mode emulator was able to run the binaries.)

On 2021-08-21 4:33 a.m., Frans de Boer wrote:
> 
> Running 'qemu-aarch64 -L /newroot /newroot/usr/bin/bash -c 
> /usr/bin/env> --help' does show the env help text. So, I guess chroot
> is to blame?
Note that the above command runs your *host's* /usr/bin/env
because chroot is not used - the binary under qemu
 (/newroot/usr/bin/bash) sees your host's file system.

Observe with:

  qemu-aarch64 -L /newroot /newroot/usr/bin/bash -c /bin/uname -m
  qemu-aarch64 -L /newroot /newroot/usr/bin/env /bin/uname -m

I'm guessing you will see "x86_64", not "aarch64".

3. What you should try is:

  qemu-aarch64 -L /newroot \
     /newroot/usr/bin/bash -c /newroot/usr/bin/env --version
and:
  qemu-aarch64 -L /newroot \
     /newroot/usr/bin/env /newroot/usr/bin/bash --version

In both cases, one aarch64 binary will try to execute another aach64 
binary. Do these work for you, or are you seeing an error?



4. Use qemu's "-strace" to see the syscalls, hopefully
that will help pinpoint the cause:

  qemu-aarch64 -strace -L /newroot \
      /newroot/usr/sbin/chroot /newroot /usr/bin/env --version 2&1 \
          | tee log.txt

If the command results in an error, the "log.txt" file will show
more details about what failed.
If you're not familiar with 'strace' output, post it here as an email 
attachment.


Hope this helps,
 - assaf

P.S.

On 2021-08-24 2:39 a.m., Paul Eggert wrote:
> A complete set of instructions for an outsider to reproduce the
> problem from scratch.  Assume the outsider is running Fedora 34
> x86-64 (since that's what I'm running :-).
I'm not familiar with Fedora, but on Debian/x86_64 the following works:

   apt-get qemu-user
   apt-get install crossbuild-essential-arm64 libc6-arm64-cross

   cd coreutils
   ./configure --host=aarch64-linux-gnu
   make

then:

    $ qemu-aarch64 -L /usr/aarch64-linux-gnu/ ./src/uname -m
    aarch64

Somewhat related:

    $ qemu-aarch64 -L /usr/aarch64-linux-gnu/ ./src/env ./src/uname -m
    /lib/ld-linux-aarch64.so.1: No such file or directory

This fails because once "inside" qemu, the aarch64 searches for
"/lib/ld-linux-aarch64.so.1" but the file is in
"/usr/aarch64-linux-gnu/lib/ld-linux-aarch64.so.1".
One possible work-around is to build static binaries.

I don't want to assume that is the culprit for Frans, so we'll wait for 
the logs...

Message #20 received at 50151 <at> debbugs.gnu.org (full text, mbox):

From: Frans de Boer <frans <at> fransdb.nl>
To: Assaf Gordon <assafgordon <at> gmail.com>
Cc: Paul Eggert <eggert <at> cs.ucla.edu>, 50151 <at> debbugs.gnu.org
Subject: Re: bug#50151: Coreutils, aarch64 and chroot
Date: Wed, 25 Aug 2021 20:54:16 +0200

[Message part 1 (text/plain, inline)]

On 8/25/21 10:16 AM, Assaf Gordon wrote:
> Hello,
>
> On 2021-08-24 2:39 a.m., Paul Eggert wrote:
>> However, I think it'll be a better use of our time for you to debug 
>> this one yourself. It doesn't sound like a Coreutils problem; it 
>> sounds like a problem in your virtual machine setup, and you're the 
>> best expert on that setup.
>
> Few suggestions to check, that might help you and us to troubleshoot:
>
> 1. ensure the binaries are indeed for aarch64:
>
>    file /newroot/usr/sbin/chroot
>    file /newroot/usr/bin/env
>    file /newroot/usr/bin/bash
>
> it should say something like
>   "ELF 64-bit LSB pie executable, ARM aarch64"
> for all of them.
>
>
> 2. ensure each binary works by itself:
>
>  qemu-aarch64 -L /newroot /newroot/usr/sbin/chroot --version
>  qemu-aarch64 -L /newroot /newroot/usr/bin/env --version
>  qemu-aarch64 -L /newroot /newroot/usr/bin/bash --version
>
> (the actual version doesn't matter here, the main thing is that
> the qemu user-mode emulator was able to run the binaries.)
>
> On 2021-08-21 4:33 a.m., Frans de Boer wrote:
>>
>> Running 'qemu-aarch64 -L /newroot /newroot/usr/bin/bash -c 
>> /usr/bin/env> --help' does show the env help text. So, I guess chroot
>> is to blame?
> Note that the above command runs your *host's* /usr/bin/env
> because chroot is not used - the binary under qemu
>  (/newroot/usr/bin/bash) sees your host's file system.
>
> Observe with:
>
>   qemu-aarch64 -L /newroot /newroot/usr/bin/bash -c /bin/uname -m
>   qemu-aarch64 -L /newroot /newroot/usr/bin/env /bin/uname -m
>
> I'm guessing you will see "x86_64", not "aarch64".
>
> 3. What you should try is:
>
>   qemu-aarch64 -L /newroot \
>      /newroot/usr/bin/bash -c /newroot/usr/bin/env --version
> and:
>   qemu-aarch64 -L /newroot \
>      /newroot/usr/bin/env /newroot/usr/bin/bash --version
>
> In both cases, one aarch64 binary will try to execute another aach64 
> binary. Do these work for you, or are you seeing an error?
>
>
>
> 4. Use qemu's "-strace" to see the syscalls, hopefully
> that will help pinpoint the cause:
>
>   qemu-aarch64 -strace -L /newroot \
>       /newroot/usr/sbin/chroot /newroot /usr/bin/env --version 2&1 \
>           | tee log.txt
>
> If the command results in an error, the "log.txt" file will show
> more details about what failed.
> If you're not familiar with 'strace' output, post it here as an email 
> attachment.
>
>
> Hope this helps,
>  - assaf
>
> P.S.
>
> On 2021-08-24 2:39 a.m., Paul Eggert wrote:
>> A complete set of instructions for an outsider to reproduce the
>> problem from scratch.  Assume the outsider is running Fedora 34
>> x86-64 (since that's what I'm running :-).
> I'm not familiar with Fedora, but on Debian/x86_64 the following works:
>
>    apt-get qemu-user
>    apt-get install crossbuild-essential-arm64 libc6-arm64-cross
>
>    cd coreutils
>    ./configure --host=aarch64-linux-gnu
>    make
>
> then:
>
>     $ qemu-aarch64 -L /usr/aarch64-linux-gnu/ ./src/uname -m
>     aarch64
>
> Somewhat related:
>
>     $ qemu-aarch64 -L /usr/aarch64-linux-gnu/ ./src/env ./src/uname -m
>     /lib/ld-linux-aarch64.so.1: No such file or directory
>
> This fails because once "inside" qemu, the aarch64 searches for
> "/lib/ld-linux-aarch64.so.1" but the file is in
> "/usr/aarch64-linux-gnu/lib/ld-linux-aarch64.so.1".
> One possible work-around is to build static binaries.
>
> I don't want to assume that is the culprit for Frans, so we'll wait 
> for the logs...
>
>
@assaf: your suggestions no. 1 and 2, had the predicted results. Thus, 
suggestion no. 3 failed because of suggestion no.2. I followed then 
suggestion 4 and attached the strace output to this message. It seems 
that chroot is working as expected, only env seems to fail with an error.

I hope you have some clue of what is going wrong.

--- Frans.

-- 
A: Yes, just like that                            A: Ja, net zo
Q: Oh, Just like reading a book backwards         Q: Oh, net als een boek achterstevoren lezen
A: Because it upsets the natural flow of a story  A: Omdat het de natuurlijke gang uit het verhaal haalt
Q: Why is top-posting annoying?                   Q: Waarom is Top-posting zo irritant?

[strace.txt (text/plain, attachment)]

Message #23 received at 50151 <at> debbugs.gnu.org (full text, mbox):

From: Assaf Gordon <assafgordon <at> gmail.com>
To: Frans de Boer <frans <at> fransdb.nl>
Cc: Paul Eggert <eggert <at> cs.ucla.edu>, 50151 <at> debbugs.gnu.org
Subject: Re: bug#50151: Coreutils, aarch64 and chroot
Date: Wed, 25 Aug 2021 17:25:44 -0600

tag 50151 notabug
close 50151
stop

On 2021-08-25 12:54 p.m., Frans de Boer wrote:
> On 8/25/21 10:16 AM, Assaf Gordon wrote:
>>   qemu-aarch64 -strace -L /newroot \
>>       /newroot/usr/sbin/chroot /newroot /usr/bin/env --version 2&1 \
>>           | tee log.txt
>>
> @assaf: your suggestions no. 1 and 2, had the predicted results. Thus, 
> suggestion no. 3 failed because of suggestion no.2. I followed then 
> suggestion 4 and attached the strace output to this message. It seems 
> that chroot is working as expected, only env seems to fail with an error.

Not exactly:
The 'chroot' system-call *seems* to succeed,
followed by a failed "execve(2)" system call to execute another binary.
That "execve" system fails - so it is not 'env' per-se,
it is any program that will try to execute another aarch64 binary.

Learning that, searching for "qemu-user", "chroot" and "architecture"
leads to several web pages detailing similar errors (and few suggested
solutions):

https://wiki.gentoo.org/wiki/Crossdev_qemu-static-user-chroot

https://newbedev.com/how-can-i-chroot-into-a-filesystem-with-a-different-architechture

https://ownyourbits.com/2018/06/13/transparently-running-binaries-from-any-architecture-in-linux-with-qemu-and-binfmt_misc/

> I hope you have some clue of what is going wrong.

With the above information, we can conclude this is not a bug
in coreutils - it is a limitation of the linux+qemu-user setup.

So I'm closing this item and marking it as "not a bug",
but discussion can continue by replying to this thread.

regards,
 - assaf

Message #30 received at 50151 <at> debbugs.gnu.org (full text, mbox):

From: Frans de Boer <frans <at> fransdb.nl>
To: Assaf Gordon <assafgordon <at> gmail.com>
Cc: Paul Eggert <eggert <at> cs.ucla.edu>, 50151 <at> debbugs.gnu.org
Subject: Re: bug#50151: Coreutils, aarch64 and chroot
Date: Thu, 26 Aug 2021 10:00:26 +0200

On 8/26/21 01:25, Assaf Gordon wrote:
> tag 50151 notabug
> close 50151
> stop
>
> On 2021-08-25 12:54 p.m., Frans de Boer wrote:
>> On 8/25/21 10:16 AM, Assaf Gordon wrote:
>>>   qemu-aarch64 -strace -L /newroot \
>>>       /newroot/usr/sbin/chroot /newroot /usr/bin/env --version 2&1 \
>>>           | tee log.txt
>>>
>> @assaf: your suggestions no. 1 and 2, had the predicted results. 
>> Thus, suggestion no. 3 failed because of suggestion no.2. I followed 
>> then suggestion 4 and attached the strace output to this message. It 
>> seems that chroot is working as expected, only env seems to fail with 
>> an error.
>
> Not exactly:
> The 'chroot' system-call *seems* to succeed,
> followed by a failed "execve(2)" system call to execute another binary.
> That "execve" system fails - so it is not 'env' per-se,
> it is any program that will try to execute another aarch64 binary.
>
> Learning that, searching for "qemu-user", "chroot" and "architecture"
> leads to several web pages detailing similar errors (and few suggested
> solutions):
>
> https://wiki.gentoo.org/wiki/Crossdev_qemu-static-user-chroot
>
> https://newbedev.com/how-can-i-chroot-into-a-filesystem-with-a-different-architechture 
>
>
> https://ownyourbits.com/2018/06/13/transparently-running-binaries-from-any-architecture-in-linux-with-qemu-and-binfmt_misc/ 
>
>
>> I hope you have some clue of what is going wrong.
>
> With the above information, we can conclude this is not a bug
> in coreutils - it is a limitation of the linux+qemu-user setup.
>
> So I'm closing this item and marking it as "not a bug",
> but discussion can continue by replying to this thread.
>
> regards,
>  - assaf
>
Okay, thanks anyhow for your effort. I will explore the suggested 
solutions next.
I was not sure, but I tried the same also with x86_64 cross compiled 
programs using qemu-i386, which worked. So, using your conclusion too, 
it looks like qemu is not behaving as expected.

Regards, Frans.

Message #33 received at 50151 <at> debbugs.gnu.org (full text, mbox):

From: Andreas Schwab <schwab <at> linux-m68k.org>
To: Frans de Boer <frans <at> fransdb.nl>
Cc: Assaf Gordon <assafgordon <at> gmail.com>, 50151 <at> debbugs.gnu.org,
 Paul Eggert <eggert <at> cs.ucla.edu>
Subject: Re: bug#50151: Coreutils, aarch64 and chroot
Date: Thu, 26 Aug 2021 10:19:09 +0200

On Aug 26 2021, Frans de Boer wrote:

> I was not sure, but I tried the same also with x86_64 cross compiled
> programs using qemu-i386, which worked.

Of course, they can be natively executed.

Andreas.

-- 
Andreas Schwab, schwab <at> linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Message #36 received at 50151 <at> debbugs.gnu.org (full text, mbox):

From: Frans de Boer <frans <at> fransdb.nl>
To: Andreas Schwab <schwab <at> linux-m68k.org>
Cc: Assaf Gordon <assafgordon <at> gmail.com>, 50151 <at> debbugs.gnu.org,
 Paul Eggert <eggert <at> cs.ucla.edu>
Subject: Re: bug#50151: Coreutils, aarch64 and chroot
Date: Thu, 26 Aug 2021 11:14:10 +0200

On 8/26/21 10:19, Andreas Schwab wrote:
> On Aug 26 2021, Frans de Boer wrote:
>
>> I was not sure, but I tried the same also with x86_64 cross compiled
>> programs using qemu-i386, which worked.
> Of course, they can be natively executed.
>
> Andreas.
>
I understand now better how things are connected. Until some time ago I 
always used qemu-system-xxx, but have failed to understand that the 
qemu-xxx user emulation is only about the cpu. So, starting programs 
with a different coding will fail. The binfmt is the way to solve this 
issue. Now only find a decent explanation how to setup the 
/etc/binfmt.d/register for the aarch64, powerpc and maybe other 
architectures.
At kernel.org there is an explanation, but fairly limited and thus not 
very helpful at this point in time.

I keep on digging.

Frans

Message #39 received at 50151 <at> debbugs.gnu.org (full text, mbox):

From: Frans de Boer <frans <at> fransdb.nl>
To: Andreas Schwab <schwab <at> linux-m68k.org>
Cc: Assaf Gordon <assafgordon <at> gmail.com>, 50151 <at> debbugs.gnu.org,
 Paul Eggert <eggert <at> cs.ucla.edu>
Subject: Re: bug#50151: Coreutils, aarch64 and chroot
Date: Thu, 26 Aug 2021 11:38:36 +0200

On 8/26/21 11:14, Frans de Boer wrote:
> On 8/26/21 10:19, Andreas Schwab wrote:
>> On Aug 26 2021, Frans de Boer wrote:
>>
>>> I was not sure, but I tried the same also with x86_64 cross compiled
>>> programs using qemu-i386, which worked.
>> Of course, they can be natively executed.
>>
>> Andreas.
>>
> I understand now better how things are connected. Until some time ago 
> I always used qemu-system-xxx, but have failed to understand that the 
> qemu-xxx user emulation is only about the cpu. So, starting programs 
> with a different coding will fail. The binfmt is the way to solve this 
> issue. Now only find a decent explanation how to setup the 
> /etc/binfmt.d/register for the aarch64, powerpc and maybe other 
> architectures.
> At kernel.org there is an explanation, but fairly limited and thus not 
> very helpful at this point in time.
>
> I keep on digging.
>
> Frans
Ok, found the magic number list and (mostly) how to read it. Time to 
experiment...

Frans.

-- 
A: Yes, just like that                            A: Ja, net zo
Q: Oh, Just like reading a book backwards         Q: Oh, net als een boek achterstevoren lezen
A: Because it upsets the natural flow of a story  A: Omdat het de natuurlijke gang uit het verhaal haalt
Q: Why is top-posting annoying?                   Q: Waarom is Top-posting zo irritant?

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.