Subject: bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

From: George Valkov <[email-masked]>
Date: Thu, 16 Feb 2023 00:40:49 +0200

> On 2023-02-15, at 9:26 PM, Paul Eggert <[email-masked]> wrote:
> 
> On 2023-02-15 06:05, George Valkov wrote:
>> gcc d.c && ./a.out
>> src 3  dst 4
>> c 14053376  p 20344832  h 20344832  d 6291456
>> total bytes copied 14053376 / 27551296
> 
> Thanks, this is due to a known incompatibility in macOS lseek that coreutils is supposed to work around. See <https://www.gnu.org/software/gnulib/manual/html_node/lseek.html>, which says, "On some platforms, lseek (fd, offset, SEEK_DATA) returns a value greater than offset even when offset addresses data: macOS 12".

Hi Paul! There are two possible solutions here:
1. Do not use lseek(SEEK_DATA) on macOS.
2. Update the cached list of valid sectors which is used by lseek. See §§§ below. There could be some API to achieve this. A workaround is to fclonefileat to a temporary file and then remove it. Internally fclonefileat updates that cache. We might be able to find how it’s done.

manpage
If whence is SEEK_DATA, the offset is set to the start of the next non-hole file region greater than or equal to the supplied offset.

I believe lseek operates correctly, however the underlaying cache it uses is not current after certain conditions occur.

> I guess that somehow, the way you're building coreutils defeats the workaround. If so, we'll need to change how coreutils is built in your environment, or fix coreutils 'configure' so that the workaround isn't defeated in your environment.

Here are the build commands I use:
git clone https://github.com/coreutils/coreutils.git

cd
 coreutils
git submodule foreach git pull origin master
./bootstrap
./configure
make -j 16

Here is a tarball of coreutils after building:
https://httpstorm.com/share/.openwrt/test/2023-02-06_coreutils-9.1/coreutils-cf80f988eeb97cc3f8c65ae58e735d36f865277b.tgz

> Although in <https://bugs.gnu.org/61386#128> Pádraig was dubious about this guess, his reasoning that the bug is likely specific to APFS rather than an API mismatch could be wrong, as I think HFS doesn't support SEEK_DATA at all or reports trivial answers, so coreutils is not likely to run into the problem on HFS even if the bug is an API issue.

> Here are some things we can do to test this guess.
> 
> 1. Please try the attached program e.c in place of your d.c program. e.c is like d.c, except it attempts to use the coreutils workaround. What symptoms do you observe? If e.c works then it's almost surely a problem in how coreutils is built (compiler options or whatnot), not in the coreutils workaround. If e.c does not work it's likely that the Gnulib workaround does not suffice on your macOS platform, in which case we need to improve the workaround by hacking further on e.c and porting the result back to Gnulib. (There are other possibilities.)

It produces the same corrupted copy as the original sample:
gcc e.c && ./a.out 
__APPLE__ 1  __MACH__ 1  SEEK_DATA 4
src 3  dst 4
c 14053376  p 20344832  h 20344832  d 6291456
total bytes copied 14053376 / 27551296

In hex
c d67000  p 1367000  h 1367000  d 600000
total bytes copied d67000 / 1a46640

419494dd4527ebb22c7bf2b388316a4beb5c73d2  cc1
7b447132f56f3b549ef62a4780945e82d26f7d2c  cc1-sparse

The after I use clone to update the cached list of valid sectors for cc1
./coreutils-2023-02-15/src/cp cc1 cc1-clone

It returns a bunch of errors, but produces a valid copy. There is no point in using this workaround on macOS, it does not offer any improvements. I would recommend making sure that lseek sees an updated list of valid sectors.
gcc e.c && ./a.out 
__APPLE__ 1  __MACH__ 1  SEEK_DATA 4
src 3  dst 4
c 1000  p 1000  h 1000  d 0
c 1a44640  p 1a46640  h 1a46640  d 2000
lseek failed  ENXIO 6  EBADF 9  EINVAL 22  EOVERFLOW 84  ESPIPE 29
  p 1a46640
  d 1a46640  6 Device not configured
  h ffffffffffffffff  6 Device not configured
  a 1a46640  6 Device not configured
  b 1a46640  6 Device not configured
total bytes copied 1a45640 / 1a46640

419494dd4527ebb22c7bf2b388316a4beb5c73d2  cc1
419494dd4527ebb22c7bf2b388316a4beb5c73d2  cc1-clone
419494dd4527ebb22c7bf2b388316a4beb5c73d2  cc1-sparse

> 2. Please verify that coreutils cp is using the Gnulib workaround. In the src directory, the shell command "nm -o *.o | grep lseek" should output only lines containing "rpl_lseek"; there shouldn't be any lines saying just "lseek". Also, please run the command "objdump -d lib/libcoreutils_a-lseek.o" and verify that the replacement lseek is actually doing something nontrivial (you should get maybe three dozen lines of assembly language; if it's much less then this is the problem).

nm -o *.o | grep lseek
cat.o:                  U _rpl_lseek
copy.o:                  U _rpl_lseek
dd.o:                  U _rpl_lseek
head.o:                  U _rpl_lseek
selinux.o: no symbols
shred.o:                  U _rpl_lseek
shuf.o:                  U _rpl_lseek
split.o:                  U _rpl_lseek
tac.o:                  U _rpl_lseek
tail.o:                  U _rpl_lseek
tail.o: 0000000000002d80 t _xlseek
truncate.o:                  U _rpl_lseek
wc.o:                  U _rpl_lseek

objdump -d lib/libcoreutils_a-lseek.o

lib/libcoreutils_a-lseek.o:	file format mach-o 64-bit x86-64

Disassembly of section __TEXT,__text:

0000000000000000 <_rpl_lseek>:
       0: 55                           	pushq	%rbp
       1: 48 89 e5                     	movq	%rsp, %rbp
       4: 41 57                        	pushq	%r15
       6: 41 56                        	pushq	%r14
       8: 53                           	pushq	%rbx
       9: 50                           	pushq	%rax
       a: 49 89 f7                     	movq	%rsi, %r15
       d: 41 89 fe                     	movl	%edi, %r14d
      10: 83 fa 04                     	cmpl	$4, %edx
      13: 75 23                        	jne	0x38 <_rpl_lseek+0x38>
      15: 44 89 f7                     	movl	%r14d, %edi
      18: 4c 89 fe                     	movq	%r15, %rsi
      1b: ba 03 00 00 00               	movl	$3, %edx
      20: e8 00 00 00 00               	callq	0x25 <_rpl_lseek+0x25>
      25: 48 89 c3                     	movq	%rax, %rbx
      28: 48 85 c0                     	testq	%rax, %rax
      2b: 78 20                        	js	0x4d <_rpl_lseek+0x4d>
      2d: 31 d2                        	xorl	%edx, %edx
      2f: 4c 39 fb                     	cmpq	%r15, %rbx
      32: 0f 94 c2                     	sete	%dl
      35: c1 e2 02                     	shll	$2, %edx
      38: 44 89 f7                     	movl	%r14d, %edi
      3b: 4c 89 fe                     	movq	%r15, %rsi
      3e: 48 83 c4 08                  	addq	$8, %rsp
      42: 5b                           	popq	%rbx
      43: 41 5e                        	popq	%r14
      45: 41 5f                        	popq	%r15
      47: 5d                           	popq	%rbp
      48: e9 00 00 00 00               	jmp	0x4d <_rpl_lseek+0x4d>
      4d: e8 00 00 00 00               	callq	0x52 <_rpl_lseek+0x52>
      52: 83 38 06                     	cmpl	$6, (%rax)
      55: 49 0f 44 df                  	cmoveq	%r15, %rbx
      59: 48 89 d8                     	movq	%rbx, %rax
      5c: 48 83 c4 08                  	addq	$8, %rsp
      60: 5b                           	popq	%rbx
      61: 41 5e                        	popq	%r14
      63: 41 5f                        	popq	%r15
      65: 5d                           	popq	%rbp
      66: c3                           	retq

> 3. Please confirm that _DARWIN_C_SOURCE is defined to 1 in lib/config.h.

Yes
Lib/config.h : line 3640
/* Enable general extensions on macOS.  */
#ifndef _DARWIN_C_SOURCE
# define _DARWIN_C_SOURCE 1
#endif

> 4. What is the output of the following commands, in the coreutils build directory?
> 
> rm lib/libcoreutils_a-lseek.o
> make V=1 lib/libcoreutils_a-lseek.o
> gcc -E -Ilib lib/lseek.c<e.c>

rm lib/libcoreutils_a-lseek.o
make V=1 lib/libcoreutils_a-lseek.o
gcc  -I. -I./lib  -Ilib -I./lib -Isrc -I./src -I/usr/local/include   -Wno-cast-qual -Wno-conversion -Wno-float-equal -Wno-sign-compare -Wno-undef -Wno-unused-function -Wno-unused-parameter -Wno-float-conversion -Wimplicit-fallthrough -Wno-pedantic -Wno-sign-conversion -Wno-type-limits -g -O2 -MT lib/libcoreutils_a-lseek.o -MD -MP -MF lib/.deps/libcoreutils_a-lseek.Tpo -c -o lib/libcoreutils_a-lseek.o `test -f 'lib/lseek.c' || echo './'`lib/lseek.c
mv -f lib/.deps/libcoreutils_a-lseek.Tpo lib/.deps/libcoreutils_a-lseek.Po
gcc -E -Ilib lib/lseek.c < e.c > epp.txt
https://httpstorm.com/share/.openwrt/test/2023-02-06_coreutils-9.1/epp.txt

§§§
I updated d.c to print in hex and use O_TRUNC on dst. I wrote another sample to read the entire source and write non-blank sectors creating a sparse file.

Copy cc1 to B
gcc f.c && ./a.out 
src 3  dst 4
00001000 skip
total bytes copied 1a45640 / 1a46640 1a46640

Copy B to cc1-sparse
src 3  dst 4
c 1000  p 1000  h 1000  d 0
c 1a44640  p 1a46640  h 1a46640  d 2000
total bytes copied 1a45640 / 1a46640

c6daa8af9a75ab24e03252e63c6a855979f04095  cc1
c6daa8af9a75ab24e03252e63c6a855979f04095  B
c6daa8af9a75ab24e03252e63c6a855979f04095  cc1-sparse

I also tried playing with smaller values for BLOCK_SIZE down to 1. The output from d.c will always be the same because cc1 has only one sector that is a hole 0x1000-0x2000. The resulting sparse file B is then copied correctly by d.c, which suggests that lseek(SEEK_DATA) operates correctly when the file-system is in a good state, however it is likely that it returns cached data that could become invalid under certain conditions. If I unmount the sparse disk image and mount it again or use fclonefileat on cc1, the cache is updated and lseek returns good results. It would be interesting to trace the creation of cc1 during gcc/initial/compile.

Copy cc1 to cc1-sparse before the cached list of valid sectors is updated
src 3  dst 4
c d67000  p 1367000  h 1367000  d 600000
total bytes copied d67000 / 1a46640
7b447132f56f3b549ef62a4780945e82d26f7d2c  cc1-sparse

Update the cached list of valid sectors for cc1
./coreutils-2023-02-15/src/cp cc1 cc1-clone

Copy cc1 to cc1-sparse after the cached list of valid sectors is updated
gcc d.c && ./a.out 
src 3  dst 4
c 1000  p 1000  h 1000  d 0
c 1a44640  p 1a46640  h 1a46640  d 2000
total bytes copied 1a45640 / 1a46640
c6daa8af9a75ab24e03252e63c6a855979f04095  cc1-sparse

For a normal file, the list of valid sectors is up to date
gcc d.c && ./a.out 
src 3  dst 4
c 229  p 229  h 229  d 0
total bytes copied 229 / 229

The sector size on the internal NVME disk is 4 KB (0x1000).

Georgi Valkov
httpstorm.com
nano RTOS