GNU bug report logs - #26311
dd: add support for POSIX_FADV_NOREUSE etc.

Previous Next

To reply to this bug, email your comments to 26311 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: bug-coreutils <at> gnu.org
Subject: dd support for POSIX_FADV_NOREUSE etc.
Date: Thu, 30 Mar 2017 08:42:53 -0700

The 'dd' command has some complicated logic for iflag=nocache and 
oflag=nocache, which I don't entirely understand but expect is in there 
for good reason. I suggest that 'dd' also have simpler logic for direct 
usage of posix_fadvise options. For example, 'dd iflag=noreuse' could 
cause it to call fadvise (STDIN_FILENO, FADVISE_NOREUSE). I suppose 
'sequential' should be the default.

Dumb question: why does dd iflag=nocache check the return value of 
posix_fadvise? The fadvise function ignores the return value and says 
why; do these reasons not apply to 'dd'?

Message #8 received at 26311 <at> debbugs.gnu.org (full text, mbox):

From: Pádraig Brady <P <at> draigBrady.com>
To: Paul Eggert <eggert <at> cs.ucla.edu>, 26311 <at> debbugs.gnu.org
Subject: Re: bug#26311: dd support for POSIX_FADV_NOREUSE etc.
Date: Thu, 30 Mar 2017 09:30:44 -0700

On 30/03/17 08:42, Paul Eggert wrote:
> The 'dd' command has some complicated logic for iflag=nocache and 
> oflag=nocache, which I don't entirely understand but expect is in there 
> for good reason. I suggest that 'dd' also have simpler logic for direct 
> usage of posix_fadvise options. For example, 'dd iflag=noreuse' could 
> cause it to call fadvise (STDIN_FILENO, FADVISE_NOREUSE). I suppose 
> 'sequential' should be the default.

Yes maybe, though the low level meanings of these flags are
a bit confusing and I'm not sure how consistent they are over kernel versions,
hence why a higher level "nocache" item was chosen initially.
I have some notes on these settings at:
https://github.com/coreutils/coreutils/blob/c7bcea1b/src/sort.c#L894-L939

> Dumb question: why does dd iflag=nocache check the return value of 
> posix_fadvise? The fadvise function ignores the return value and says 
> why; do these reasons not apply to 'dd'?

Well for dd it's a request, which for other utils it's a performance advisement.
As stated in the info docs, for dd it's only significant in the special case
of dropping cache for the whole file:

  $ : | dd iflag=nocache count=0 status=none
  dd: failed to discard cache for: 'standard input': Illegal seek

cheers,
Pádraig

Message #11 received at 26311 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Pádraig Brady <P <at> draigBrady.com>, 26311 <at> debbugs.gnu.org
Subject: Re: bug#26311: dd support for POSIX_FADV_NOREUSE etc.
Date: Thu, 30 Mar 2017 10:55:20 -0700

On 03/30/2017 09:30 AM, Pádraig Brady wrote:
> Yes maybe, though the low level meanings of these flags are
> a bit confusing and I'm not sure how consistent they are over kernel versions

Nor I. Still, 'dd' to some extent is supposed to be a low-level program 
that lets one directly access syscall flags, so following that design 
philosophy we should just pass the flags through and let users 
experiment with them.

>
>> Dumb question: why does dd iflag=nocache check the return value of
>> posix_fadvise? The fadvise function ignores the return value and says
>> why; do these reasons not apply to 'dd'?
> Well for dd it's a request, which for other utils it's a performance advisement.
> As stated in the info docs, for dd it's only significant in the special case
> of dropping cache for the whole file:
>
>    $ : | dd iflag=nocache count=0 status=none
>    dd: failed to discard cache for: 'standard input': Illegal seek

Yes, and that was what I was asking about. iflag=nocache is just advice 
which the kernel is not obliged to follow even when the syscall succeeds 
(and conversely, it's advice which the kernel might decided to follow 
even when the syscall fails!). So why bother to report the syscall 
failure in this particular case?

It's no big deal. More of a philosophical question, really.

Message #14 received at 26311 <at> debbugs.gnu.org (full text, mbox):

From: Pádraig Brady <P <at> draigBrady.com>
To: Paul Eggert <eggert <at> cs.ucla.edu>, 26311 <at> debbugs.gnu.org
Subject: Re: bug#26311: dd support for POSIX_FADV_NOREUSE etc.
Date: Thu, 30 Mar 2017 20:02:17 -0700

On 30/03/17 10:55, Paul Eggert wrote:
> On 03/30/2017 09:30 AM, Pádraig Brady wrote:
>> Yes maybe, though the low level meanings of these flags are
>> a bit confusing and I'm not sure how consistent they are over kernel versions
> 
> Nor I. Still, 'dd' to some extent is supposed to be a low-level program 
> that lets one directly access syscall flags, so following that design 
> philosophy we should just pass the flags through and let users 
> experiment with them.
> 
>>
>>> Dumb question: why does dd iflag=nocache check the return value of
>>> posix_fadvise? The fadvise function ignores the return value and says
>>> why; do these reasons not apply to 'dd'?
>> Well for dd it's a request, which for other utils it's a performance advisement.
>> As stated in the info docs, for dd it's only significant in the special case
>> of dropping cache for the whole file:
>>
>>    $ : | dd iflag=nocache count=0 status=none
>>    dd: failed to discard cache for: 'standard input': Illegal seek
> 
> Yes, and that was what I was asking about. iflag=nocache is just advice 
> which the kernel is not obliged to follow even when the syscall succeeds 
> (and conversely, it's advice which the kernel might decided to follow 
> even when the syscall fails!). So why bother to report the syscall 
> failure in this particular case?
> 
> It's no big deal. More of a philosophical question, really.

Yes I'm not adamant on this behaviour, but my thinking
was with count=0 iflag=nocache, you are _only_ asking
for cache invalidation, so dd has more leeway to error
in this case (even though the only effective error reported
is EPIPE).  With other dd options, you could be combining
the nocaching directive with other operations and so it
would be ok for the nocache advice failure to be ignored.

cheers,
Pádraig

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.