Subject: bug#6131: [PATCH]: fiemap support for efficient sparse file copy

From: Sunil Mushran <[email-masked]>
Date: Thu, 27 May 2010 12:03:09 -0700

On 05/27/2010 11:43 AM, Jim Meyering wrote:
> Sunil Mushran wrote:
>    
>> Jim Meyering wrote:
>>      
>>>> Hi Jeff,
>>>>
>>>> I've included below the state of my local changes.
>>>> Unfortunately, with that 5-patch series, there is always a test failure
>>>> on F13/ext4.  Maybe someone who knows more about extents can provide an
>>>> explanation?
>>>>
>>>> Here's a small example to demonstrate:
>>>>
>>>> Create a file with many extents:
>>>>
>>>>       perl -e 'BEGIN { $n = 19 * 1024; *F = *STDOUT }' \
>>>>         -e 'for (1..100) { sysseek (*F, $n, 1)' \
>>>>         -e '&&   syswrite (*F, "."x$n) or die "$!"}'>   j1
>>>>
>>>> Using the patched "cp", repeat the following 10 or 20 times:
>>>>
>>>>       ./cp --sparse=always j1 j2; sync
>>>>       filefrag -v j1 | awk '/^ / {print $1,$2}'>   ff1 || fail=1
>>>>       filefrag -v j2 | awk '/^ / {print $1,$2}'>   ff2 || fail=1
>>>>       diff -u ff1 ff2 || fail=1
>>>>
>>>> Usually there is no diff output, but occasionally it'll print this:
>>>> [hmm... today it consistently prints these differences every other time.]
>>>>          
>> The reason it does not work is because the sparse file created by cp
>> may not be sparse (or sparse enough). And that is because cp reads
>> is chunks of st_blocksize and skips the write only if the entire chunk
>> is zero. The perl script creates the file in 19K chunks (alternate writes
>> and holes).
>>      
> Thanks for replying.
>
> However, your description of how GNU cp works suggests that you're
> looking at the pre-FIEMAP semantics.  Please refer to the patches here
>
>      http://thread.gmane.org/gmane.comp.gnu.coreutils.bugs/20534/focus=20630
>
> that make it use FIEMAP.
>
>    
>> So on a 4K fs, the file created by the script will have 4 blocks as holes
>> (avg). But when cp makes it, it could fill out those holes because the read
>> granularity could be coarser. For example, ocfs2 fills out st_blocksize
>> with the fs cluster size (allocation size) which could be larger than the
>> block size.
>>
>> My suggestion is to not use filefrag but to use md5sum to compare the two
>> files.
>>      
> That would be pointless.
> The goal of the test is to determine that the FIEMAP copy
> did indeed preserve the extents.  If I do as you suggest, even if
> cp mistakenly filled in all holes, the test would still pass.
>    

I wouldn't call it pointless. More efficient read is still a big win.
My problem with the filefrag compare is that it is expecting a
behavior that no filesystem guarantees.

But having said that, I am intrigued as to why it is not working
as expected.

Do you have a strace for me to look at? strace of the fiemap
enabled cp. Also, output of "stat -f j1; stat j1 j2;filefrag -v j1; 
filefrag -v j2;"

Thanks
Sunil