GNU bug report logs - #6131
[PATCH]: fiemap support for efficient sparse file copy

Previous Next

Full log

Message #95 received at submit <at> debbugs.gnu.org (full text, mbox):

From: "jeff.liu" <jeff.liu <at> oracle.com>
To: Jim Meyering <jim <at> meyering.net>, Tao Ma <tao.ma <at> oracle.com>
Cc: Sunil Mushran <sunil.mushran <at> oracle.com>, bug-coreutils <at> gnu.org,
	Joel Becker <Joel.Becker <at> oracle.com>, Chris Mason <chris.mason <at> oracle.com>
Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy
Date: Mon, 31 May 2010 10:51:51 +0800

Jim Meyering wrote:
> Tao Ma wrote:
>> Hi Jim,
>> On 05/29/2010 12:44 AM, Jim Meyering wrote:
>>> Tao Ma wrote:
>>>
>>>> Hi Jim
>>>>
>>>> On 05/27/2010 06:30 PM, Jim Meyering wrote:
>>>>> jeff.liu wrote:
>>>>>> This is the revised version, it fixed the fiemap-start offset calculation
>>>>>> approach to remove it out
>>>>>> of the 'for (i = 0; i<   fiemap->fm_mapped_extents; i++)' loop.
>>>>> Hi Jeff,
>>>>>
>>>>> I've included below the state of my local changes.
>>>>> Unfortunately, with that 5-patch series, there is always a test failure
>>>>> on F13/ext4.  Maybe someone who knows more about extents can provide an
>>>>> explanation?
>>>> Just want to clarify why ocfs2 didn't work here. I guess the reason
>>>> also works for ext4 since both ext4 and ocfs2 use block group to
>>>> organize their blocks in the volume.
>>> Hi Tao,
>>>
>>> Thank you for the explanation.
>>> I'm glad to hear that there is no underlying problem.
>>>
>>>> I checked the perl test script to create sparse src file, it will
>>>> create contiguous bytes(around 20-24k) at an interval of around 40k.So
>>>> in general, these 20-24k should be contiguous. But that does exist
>>>> some scenario that they could be separately into 2 extents. Consider
>>>> one block group is used to allocate blocks to this file, when the
>>>> block group only has 10K left while you are requiring 20K, it will use
>>>> the left 10K in this group and allocate 10K from another block
>>>> group. That would become 2 extents since they can't be contiguous.
>>>> So I guess the right step is to check the holes by using filefrag if
>>>> you prefer this tool and want to make sure cp doesn't copy holes(I get
>>> Do you know of a tool other than filefrag that I could use?
>> nope.
>>> It looks like a small script could filter filefrag -v output, detect
>>> split extents and rewrite to make the output match what's expected.
>>> Probably not worth it, though, since this is already a very fragile test.
> 
> I went ahead and did it, after all.
> Here's the script, filefrag-extent-compare.
> With it, this test should pass when run on any of those four
> file system types.
> 
> 
> eval '(exit $?0)' && eval 'exec perl -wS "$0" ${1+"$@"}'
>   & eval 'exec perl -wS "$0" $argv:q'
>     if 0;
> # Determine whether two files have the same extents by comparing
> # the logical block numbers and lengths from filefrag -v for each.
> 
> # Invoke like this:
> # This helper function, f, extracts logical block number and lengths.
> # f() { awk '/^ *[0-9]/ {printf "%d %d ",$2,NF<5?$NF:$5} END {print ""}'; }
> # { filefrag -v j1 | f; filefrag -v j2 | f; } | ./filefrag-extent-compare
> 
> use warnings;
> use strict;
> (my $ME = $0) =~ s|.*/||;
> 
> my @line = <>;
> my $n_lines = @line;
> $n_lines == 2
>   or die "$ME: expected exactly two input lines; got $n_lines\n";
> 
> my @A = split ' ', $line[0];
> my @B = split ' ', $line[1];
> @A % 2 || @B % 2
>   and die "$ME: unexpected input: odd number of numbers; expected even\n";
> 
> my @a;
> my @b;
> foreach my $i (0..@A/2-1) { $a[$i] = { L_BLK => $A[2*$i], LEN => $A[2*$i+1] } };
> foreach my $i (0..@B/2-1) { $b[$i] = { L_BLK => $B[2*$i], LEN => $B[2*$i+1] } };
> 
> my $i = 0;
> my $j = 0;
> while (1)
>   {
>     !defined $a[$i] && !defined $b[$j]
>       and exit 0;
>     defined $a[$i] && defined $b[$j]
>       or die "\@a and \@b have different lengths, even after adjustment\n";
>     ($a[$i]->{L_BLK} == $b[$j]->{L_BLK}
>      && $a[$i]->{LEN} == $b[$j]->{LEN})
>       and next;
>     ($a[$i]->{LEN} < $b[$j]->{LEN}
>      && exists $a[$i+1] && $a[$i]->{LEN} + $a[$i+1]->{LEN} == $b[$j]->{LEN})
>       and ++$i, next;
>     exists $b[$j+1] && $a[$i]->{LEN} == $b[$i]->{LEN} + $b[$i+1]->{LEN}
>       and ++$j, next;
>     die "differing extent:\n"
>       . "  [$i]=$a[$i]->{L_BLK} $a[$i]->{LEN}\n"
>       . "  [$j]=$b[$j]->{L_BLK} $b[$j]->{LEN}\n"
>   }
> continue
>   {
>     ++$i;
>     ++$j;
>   }
> 
> 
> 
Thanks Tao and Jim for helping this out!

-Jeff

-- 
With Windows 7, Microsoft is asserting legal control over your computer and is using this power to
abuse computer users.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.