GNU bug report logs - #9500
cp: use posix_fallocate where supported

Previous Next

Full log

Message #26 received at 9500 <at> debbugs.gnu.org (full text, mbox):

From: Pádraig Brady <P <at> draigBrady.com>
To: 9500 <at> debbugs.gnu.org
Cc: Kelly Anderson <kelly <at> silka.with-linux.com>,
	Goswin von Brederlow <goswin-v-b <at> web.de>
Subject: Re: bug#9500: [PATCH]: use posix_fallocate where supported
Date: Fri, 25 Nov 2011 10:13:39 +0000

On 11/25/2011 09:58 AM, Pádraig Brady wrote:
> -------- Original Message --------
> Subject: bug#9500: [PATCH]: use posix_fallocate where supported
> Date: Fri, 25 Nov 2011 10:35:34 +0100
> From: Goswin von Brederlow <goswin-v-b <at> web.de>
> To: Padraig Brady <p <at> draigBrady.com>
> 
> Hi,
> 
> On 09/14/2011 03:46 PM, Pádraig Brady wrote:
> 
>> I thought a little about this today.
>>
>> fallocate() is a feature to quickly allocate space in a file system.
>> It's useful for 3 things as far as I can see:
>>
>>   1. Improved file layout for subsequent access
>>   2. Immediate indication of ENOSPC
>>   3. Efficient writing of NUL portions
>>
>> Note 1. is somewhat moot with newer file systems that do "delayed allocation".
>> So what do we need to consider when using fallocate on the destination file?
>> Considering just cp for the moment, its inputs impacting this are the options:
> 
> Not every filesystem does delayed allocation and delayed allocation only
> works for file considerably smaller than the amount of cache.

Right.

> Also a note on sparse files:
> 
> If you have fiemap then you can fallocate the used chunks in the
> destination. But there might be zero filled blocks that would become
> unallocated with current cp. Those would remain allocated (but not zero
> filles, which might be a difference) in the destination. Same if the
> file is fully allocated but has empty blocks. So a bit more has to be
> done.
> 
> So here is what I would do for sparse files:
> 
> - get fiemap of file, if it fails the fiemap is just one chunk for the
>   whole file
> 
> - For each chunk in the fiemap:
>   1 read block from source
>   2 if (block == ZERO) goto 7
>   3 fallocate(offset, end of chunk)
>   4 write block
>   5 read block from source
>   6 if (block != ZERO) goto 4
>   7 fruncate(offset)
>   8 read block from source
>   9 if (block == ZERO) goto 8
>  10 goto 3

I didn't fully understand the above,
but I was going to do essentially that I think.
The adjustments to the fiemap copy code
should be fairly obvious I think.

> Failure of fallocate should probably be ignored to preserve current
> behaviour and because if empty blocks are found less space will be used
> and might actually fit.

> The above would result in the same allocation for the file as current cp
> creates for sparse files.
> 
> If you want to copy files while preserving the allocation of the source
> then I think cp needs a new --sparse=source mode.

I think --spare=auto (the default) should cover that case if needed.
However as previously stated, this might be outside cp's remit.

> In that mode setp 7
> would change from ftruncate() to write(ZERO) to preserve the empty
> blocks and you would only fallocate once per chunk for the full chunk.
> 
> Hope that helps.
> 
> MfG
>         Goswin
> 
> PS: Feel free to forward to the mailinglist, gmane doesn't show its
> address or I would cc.

Hmm, one of your points above got me thinking.
Might fallocate() fail to allocate an extent with ENOSPC,
but there could be fragmented space available to write()?
That would scupper benefit (2) above :(
I'll ask linux-fsdevel <at> vger.kernel.org

cheers,
Pádraig.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.