GNU bug report logs - #12489
inconsistent output from 'dd' w/rt stats and suppression.

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 12489 in the body.
You can then email your comments to 12489 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Linda Walsh <coreutils <at> tlinx.org>
To: bug-coreutils <at> gnu.org
Subject: inconsistent output from 'dd' w/rt stats and suppression.
Date: Sat, 22 Sep 2012 12:09:55 -0700

If I have 'dd' running, and want to see stats, the manpage says I can 
send it
a USR1 signal and it will print "I/O statistics to standard error and 
then resume copying".  I see something like:
          18335302+0  records  in  18335302+0 records out 9387674624 bytes
             (9.4 GB) copied, 34.6279 seconds, 271 MB/s

The same I/O statistics are displayed at the end of a 'dd' session:

18335302+0 records  in
18335302+0 records out
9387674624 bytes (9.4 GB) copied, 34.6279 seconds, 271 MB/s

The man page says I can use the
"status=noxfer" to suppress transfer statistics".

But when I use it the number of records transfered into 'dd'
and out of 'dd' are still displayed.  Suppressed is the
sum of the bytes transferred in and out, and the time+rate.

So... it seems status=noxfer isn't fully suppressing the statistics
but more accurately only suppresses the summary, time and rate
stats and not the # records transfered in and out.

Either way... shouldn't it just suppress the whole message,
or what's the point?  What switch would one use to suppress
all I/O statistics?

Thanks,
Linda

Message #8 received at 12489 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Linda Walsh <coreutils <at> tlinx.org>
Cc: 12489 <at> debbugs.gnu.org
Subject: Re: bug#12489: inconsistent output from 'dd' w/rt stats and
	suppression.
Date: Sat, 22 Sep 2012 12:32:37 -0700

On 09/22/2012 12:09 PM, Linda Walsh wrote:
> shouldn't it just suppress the whole message,
> or what's the point?

I think the main point of status=noxfer is to get
GNU dd to behave more like traditional dd does.
Admittedly this should be documented better.

Message #11 received at 12489 <at> debbugs.gnu.org (full text, mbox):

From: Linda Walsh <coreutils <at> tlinx.org>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: 12489 <at> debbugs.gnu.org
Subject: Re: bug#12489: inconsistent output from 'dd' w/rt stats and
	suppression.
Date: Sat, 22 Sep 2012 16:17:58 -0700

[Message part 1 (text/plain, inline)]

Paul Eggert wrote:
> On 09/22/2012 12:09 PM, Linda Walsh wrote:
>   
>> shouldn't it just suppress the whole message,
>> or what's the point?
>>     
>
> I think the main point of status=noxfer is to get
> GNU dd to behave more like traditional dd does.
> Admittedly this should be documented better.
>   

I don't know about a traditional 'dd'.

I just need a switch that turns off the superfluous output, which I thought
was the point of 'noxfer', I don't have another 'dd' to compare it to.

This seems to be a trend in the documentation... with 'rm', there is no
mention of "." being special and not working as a target as it does in
'cp' or 'rsync' or 'tar'  -- one would have to go find the latest edition of
posix, or find what the posix version of the moment is that a given util is
supposed to be following...

That right there should be a hint that POSIX isn't following standards 
-- since
the documentation hasn't been altered to reflect the newer changes.

My version of 'rm' still has "--force"  -- which says to ignore 
nonexistent files never prompt.  It also used to ignore write protected 
files and give
no error messages.  giving -f was a way to shut it up about anything it
couldn't remove.

It no longer does that.

Despite what some people think there have been multiple changes recently 
enough
in the gnu utils that the documentation hasn't caught up. 

So is this superfluous output with 'dd' required by POSIX  is that why 
it has to be displayed now?  Otherwise, I think if the user wants to 
turn off transfer
statistics -- they mean all of them the blocks read in, blocks written 
out, and
the summary and speed line.

Maybe add a 'status=nosum' for just the statistics on Input and output 
but with no summary or speed?

[Message part 2 (text/html, inline)]

Message #14 received at 12489 <at> debbugs.gnu.org (full text, mbox):

From: g.esp <at> free.fr
To: Linda Walsh <coreutils <at> tlinx.org>
Cc: 12489 <at> debbugs.gnu.org
Subject: Re: bug#12489: inconsistent output from 'dd' w/rt stats and
	suppression.
Date: Sun, 23 Sep 2012 10:33:19 +0200 (CEST)

----- Mail original -----
> De: "Linda Walsh" <coreutils <at> tlinx.org>
> À: 12489 <at> debbugs.gnu.org
> Envoyé: Samedi 22 Septembre 2012 21:09:55
> Objet: bug#12489: inconsistent output from 'dd' w/rt stats and suppression.
> 
> If I have 'dd' running, and want to see stats, the manpage says I can
> send it
> a USR1 signal and it will print "I/O statistics to standard error and
> then resume copying".  I see something like:
>            18335302+0  records  in  18335302+0 records out 9387674624
>            bytes
>               (9.4 GB) copied, 34.6279 seconds, 271 MB/s
> 
> The same I/O statistics are displayed at the end of a 'dd' session:
> 
> 18335302+0 records  in
> 18335302+0 records out
> 9387674624 bytes (9.4 GB) copied, 34.6279 seconds, 271 MB/s
> 
> The man page says I can use the
> "status=noxfer" to suppress transfer statistics".
> 
> But when I use it the number of records transfered into 'dd'
> and out of 'dd' are still displayed.  Suppressed is the
> sum of the bytes transferred in and out, and the time+rate.
> 
> So... it seems status=noxfer isn't fully suppressing the statistics
> but more accurately only suppresses the summary, time and rate
> stats and not the # records transfered in and out.
> 
> Either way... shouldn't it just suppress the whole message,
> or what's the point?  What switch would one use to suppress
> all I/O statistics?
> 
> Thanks,
> Linda
> 
> 
dd if=/dev/zero of=/tmp/foo count=1024 2>/dev/null || echo "dd fail"

Gilles

Message #17 received at 12489 <at> debbugs.gnu.org (full text, mbox):

From: Pádraig Brady <P <at> draigBrady.com>
To: Linda Walsh <coreutils <at> tlinx.org>
Cc: 12489 <at> debbugs.gnu.org
Subject: Re: bug#12489: inconsistent output from 'dd' w/rt stats and
	suppression.
Date: Sun, 23 Sep 2012 12:12:52 +0100

On 09/22/2012 08:09 PM, Linda Walsh wrote:
> If I have 'dd' running, and want to see stats, the manpage says I can send it
> a USR1 signal and it will print "I/O statistics to standard error and then resume copying". I see something like:
> 18335302+0 records in 18335302+0 records out 9387674624 bytes
> (9.4 GB) copied, 34.6279 seconds, 271 MB/s
>
> The same I/O statistics are displayed at the end of a 'dd' session:
>
> 18335302+0 records in
> 18335302+0 records out
> 9387674624 bytes (9.4 GB) copied, 34.6279 seconds, 271 MB/s
>
> The man page says I can use the
> "status=noxfer" to suppress transfer statistics".
>
> But when I use it the number of records transfered into 'dd'
> and out of 'dd' are still displayed. Suppressed is the
> sum of the bytes transferred in and out, and the time+rate.
>
> So... it seems status=noxfer isn't fully suppressing the statistics
> but more accurately only suppresses the summary, time and rate
> stats and not the # records transfered in and out.
>
> Either way... shouldn't it just suppress the whole message,
> or what's the point? What switch would one use to suppress
> all I/O statistics?

I think there was general consensus that the status=noinfo should be applied
http://lists.gnu.org/archive/html/bug-coreutils/2010-02/msg00159.html

I'll look at that this evening.

cheers,
Pádraig.

Message #20 received at 12489 <at> debbugs.gnu.org (full text, mbox):

From: Jim Meyering <jim <at> meyering.net>
To: Pádraig Brady <P <at> draigBrady.com>
Cc: 12489 <at> debbugs.gnu.org,
	Pozsár Balázs <pozsy <at> uhulinux.hu>,
	Linda Walsh <coreutils <at> tlinx.org>
Subject: Re: bug#12489: inconsistent output from 'dd' w/rt stats and
	suppression.
Date: Sun, 23 Sep 2012 13:35:33 +0200

Pádraig Brady wrote:
> On 09/22/2012 08:09 PM, Linda Walsh wrote:
...
> I think there was general consensus that the status=noinfo should be applied
> http://lists.gnu.org/archive/html/bug-coreutils/2010-02/msg00159.html

That does ring a bell.  Oh, wow.  It's 2.5 years old.

> I'll look at that this evening.

Thank you, Pádraig.
Sorry it took so long, Pozsár.
And thanks to Linda for the prod.

Message #23 received at 12489 <at> debbugs.gnu.org (full text, mbox):

From: Linda Walsh <coreutils <at> tlinx.org>
To: g.esp <at> free.fr
Cc: 12489 <at> debbugs.gnu.org
Subject: Re: bug#12489: inconsistent output from 'dd' w/rt stats and
	suppression.
Date: Sun, 23 Sep 2012 14:02:40 -0700

[Message part 1 (text/plain, inline)]

It is being called from a program to copy text to a file as root.

It's not running shell.

I wanted to leave stderr for ERROR's and turn off statistics...

Maybe Unix needs to add a STDSTAT on FD3?

1/2 :-)  (i.e. there are a few instances where a stat line separate from
and error line would be handy).

Like running FLAC -- first they came out with a --silent to silence
statistic... but then that was too much.

So then they add a --totally-silent.  And guess what! 
They still issue warnings about things that don't prevent correct
encoding.  If you have MS-metadata in your WAV's, flac will complain whether
you tell it to ignore it or you don't.  If you don't ignore it and try to
explicitly include it as data in 'passthrough', it warns you that this 
option is
experimental.  If you don't use that option, it tells you it found such 
data and
to use passthrough!!!

What part of "--totally silent", especially after they already broke 
--silent,
did they not understand?  ARG!!!  So
back to dd...

It's a unix util.   Generally, quiet is beautiful.  It means everything went
well.  That's generally been the unix motto (apart from deliberate silencing
switches, which are intended to ignore error conditions).  So why not make
no xfer really not put out ANY xfer stats (as records in/out are xfer 
statistics)?

Then add another option (or vice versa -- add another to shut both up, 
whatever), like nosummary, or noIO to skip the records I/O...
Could even (in copious spare time, I'm sure, through in a format facility
like 'DATE'... for the output format (like BASH does with HISTTIMEFORMAT):
"%i records in, %I partial\n"%o records out, %O partial\n%b total bytes 
(%{HB}) in %{H:+%H hours, }%{M:+%M minutes, }%{S:+%S seconds}, %{XPS}" 
(speed/rate)


But I shouldn't have to turn off all errors (close STDERR) just to shut up
the chit-chat (silence is beautiful!)




g.esp <at> free.fr wrote:
> dd if=/dev/zero of=/tmp/foo count=1024 2>/dev/null || echo "dd fail"
> Gilles
>   

[Message part 2 (text/html, inline)]

Message #28 received at 12489-done <at> debbugs.gnu.org (full text, mbox):

From: Pádraig Brady <P <at> draigBrady.com>
To: pozsy <at> uhulinux.hu
Cc: 12489-done <at> debbugs.gnu.org, Linda Walsh <coreutils <at> tlinx.org>
Subject: Re: bug#12489: inconsistent output from 'dd' w/rt stats and
	suppression.
Date: Mon, 24 Sep 2012 03:47:25 +0100

[Message part 1 (text/plain, inline)]

On 09/23/2012 12:12 PM, Pádraig Brady wrote:
> I think there was general consensus that the status=noinfo should be applied
> http://lists.gnu.org/archive/html/bug-coreutils/2010-02/msg00159.html
>
> I'll look at that this evening.

I changed things a bit in the attached patch.
I used 'status=none' rather than 'status=noinfo',
to make it obvious all stderr messages were being suppressed.
Also, I kept the bitmask nature of the existing code
to make it more consistent and extendible.

cheers,
Pádraig.

[dd-status-none.diff (text/plain, attachment)]

Message #31 received at 12489 <at> debbugs.gnu.org (full text, mbox):

From: Jim Meyering <jim <at> meyering.net>
To: 12489 <at> debbugs.gnu.org
Cc: P <at> draigBrady.com
Subject: Re: bug#12489: inconsistent output from 'dd' w/rt stats and
	suppression.
Date: Mon, 24 Sep 2012 09:05:25 +0200

Pádraig Brady wrote:
> On 09/23/2012 12:12 PM, Pádraig Brady wrote:
>> I think there was general consensus that the status=noinfo should be applied
>> http://lists.gnu.org/archive/html/bug-coreutils/2010-02/msg00159.html
>>
>> I'll look at that this evening.
>
> I changed things a bit in the attached patch.
> I used 'status=none' rather than 'status=noinfo',
> to make it obvious all stderr messages were being suppressed.
> Also, I kept the bitmask nature of the existing code
> to make it more consistent and extendible.

Thanks.  That looks fine.

> Subject: [PATCH] dd: add support for status=none to suppress all info to
>  stderr

"all info to stderr" might be interpreted as including diagnostics.
Maybe something like this instead?
  dd: new option, status=none to suppress output statistics
or
  dd: new option, status=none to suppress all informational output

> * src/dd.c (STATUS_NONE): A new bitmask combining all STATUS_
> options, thus used to suppress all informational output.
> (struct symbol_value statuses): Expose the "none" option,
> corresponding to the STATUS_NONE bitmask above.
> (print_stats): Return early if STATUS_NONE is specified.
> Also move the call to gethrxtime() down so that it's only
> called when needed.
> (usage): Describe the new options.
> * doc/coreutils.texi (dd invocation): Likewise.
> * NEWS: Mention the new feature.
> * tests/dd/misc.sh: Ensure the new option works.

Message #34 received at 12489-done <at> debbugs.gnu.org (full text, mbox):

From: Linda Walsh <coreutils <at> tlinx.org>
To: Pádraig Brady <P <at> draigBrady.com>
Cc: pozsy <at> uhulinux.hu, 12489-done <at> debbugs.gnu.org
Subject: Re: bug#12489: inconsistent output from 'dd' w/rt stats and
	suppression.
Date: Mon, 24 Sep 2012 13:15:15 -0700

Thanks, um... (update docs? -- manpages?)


Pádraig Brady wrote:
> On 09/23/2012 12:12 PM, Pádraig Brady wrote:
>> I think there was general consensus that the status=noinfo should be 
>> applied
>> http://lists.gnu.org/archive/html/bug-coreutils/2010-02/msg00159.html
>>
>> I'll look at that this evening.
>
> I changed things a bit in the attached patch.
> I used 'status=none' rather than 'status=noinfo',
> to make it obvious all stderr messages were being suppressed.
> Also, I kept the bitmask nature of the existing code
> to make it more consistent and extendible.
>
> cheers,
> Pádraig.

Message #37 received at 12489-done <at> debbugs.gnu.org (full text, mbox):

From: Eric Blake <eblake <at> redhat.com>
To: Linda Walsh <coreutils <at> tlinx.org>
Cc: 12489-done <at> debbugs.gnu.org, pozsy <at> uhulinux.hu,
	Pádraig Brady <P <at> draigBrady.com>
Subject: Re: bug#12489: inconsistent output from 'dd' w/rt stats and
	suppression.
Date: Mon, 24 Sep 2012 14:32:19 -0600

[Message part 1 (text/plain, inline)]

[we tend to avoid top-posting on this list]

On 09/24/2012 02:15 PM, Linda Walsh wrote:
> Thanks, um... (update docs? -- manpages?)

Yep, those were included.  Docs via:

+++ b/doc/coreutils.texi
@@ -8115,10 +8115,24 @@ of everything until the end of the file.
 if @samp{iflag=count_bytes} is specified, @var{n} is interpreted
 as a byte count rather than a block count.

-@item status=noxfer
+@item status=@var{which}
 @opindex status
-Do not print the overall transfer rate and volume statistics
-that normally make up the third status line when @command{dd} exits.
+Transfer information is normally output to stderr upon
+receipt of the @samp{INFO} signal or when @command{dd} exits.
+Specifying @var{which} will identify which information to suppress.
+
+@table @samp
+
+@item noxfer
+@opindex noxfer
+Do not print the transfer rate and volume statistics
+that normally make up the last status line.
+
+@item none
+@opindex none
+Do not print any informational messages to stderr.
+
+@end table

 @item conv=@var{conversion}[,@var{conversion}]@dots{}
 @opindex conv



and manpage (which is generated from --help output) via:

@@ -536,7 +540,8 @@ Copy a file, converting and formatting according to
the operands.\n\
   oflag=FLAGS     write as per the comma separated symbol list\n\
   seek=N          skip N obs-sized blocks at start of output\n\
   skip=N          skip N ibs-sized blocks at start of input\n\
-  status=noxfer   suppress transfer statistics\n\
+  status=WHICH    WHICH info to suppress outputting to stderr;\n\
+                  'noxfer' suppresses transfer stats, 'none' suppresses
all\n\
 "), stdout);
       fputs (_("\
 \n\



-- 
Eric Blake   eblake <at> redhat.com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org

[signature.asc (application/pgp-signature, attachment)]

Message #40 received at 12489-done <at> debbugs.gnu.org (full text, mbox):

From: Linda Walsh <coreutils <at> tlinx.org>
To: Eric Blake <eblake <at> redhat.com>
Cc: 12489-done <at> debbugs.gnu.org, pozsy <at> uhulinux.hu, Pádraig Brady <P <at> draigBrady.com>
Subject: Re: bug#12489: inconsistent output from 'dd' w/rt stats and
	suppression.
Date: Mon, 24 Sep 2012 13:43:11 -0700

Thanks, missed it....

Eric Blake wrote:
> Yep, those were included.  Docs via:
>   

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.