GNU bug report logs - #30174
shred bug - 1st byte written is wrong sometimes

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 30174 in the body.
You can then email your comments to 30174 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: devzero <at> web.de
To: bug-coreutils <at> gnu.org
Subject: shred bug - 1st byte written is wrong sometimes
Date: Fri, 19 Jan 2018 19:19:13 +0100

Hi,

i`m testing wear-levelling of an SLC USB Stick (cheap one i want to
use them for long-term data-logging) and found shred to be a useful
and fast utility to repeatedly overwrite a file's region (the 
datalogger i'm building will use rrdtool)

As i already did some other testing before, shred made me believe
my usb stick would already be "worn out", because i was seeing 1st 
byte flip occasionally when taking a look into the written data with 
xxd.

I investigated and as you can see below, when repeatedly writing to 
the same file with shred, we can see with strace that shred is doing 
wrong.

Looks like a bug to me.

I tested this on recent CentOS7 (shred 8.22) and older Ubuntu based
system with shred 8.25. I also compiled coreutils 8.29 and tested
with the recent shred release - they all show this behaviour.

regards
Roland


shred -n 10000000 -s 100 file &

# cat file |xxd
0000000: 7fff ffff ffff ffff ffff ffff ffff ffff ................ <- (!)
0000010: ffff ffff ffff ffff ffff ffff ffff ffff ................
0000020: ffff ffff ffff ffff ffff ffff ffff ffff ................
0000030: ffff ffff ffff ffff ffff ffff ffff ffff ................
0000040: ffff ffff ffff ffff ffff ffff ffff ffff ................
0000050: ffff ffff ffff ffff ffff ffff ffff ffff ................
0000060: ffff ffff ....
# cat file |xxd
0000000: 2492 4924 9249 2492 4924 9249 2492 4924 $.I$.I$.I$.I$.I$
0000010: 9249 2492 4924 9249 2492 4924 9249 2492 .I$.I$.I$.I$.I$.
0000020: 4924 9249 2492 4924 9249 2492 4924 9249 I$.I$.I$.I$.I$.I
0000030: 2492 4924 9249 2492 4924 9249 2492 4924 $.I$.I$.I$.I$.I$
0000040: 9249 2492 4924 9249 2492 4924 9249 2492 .I$.I$.I$.I$.I$.
0000050: 4924 9249 2492 4924 9249 2492 4924 9249 I$.I$.I$.I$.I$.I
0000060: 2492 4924 $.I$
# cat file |xxd
0000000: aaaa aaaa aaaa aaaa aaaa aaaa aaaa aaaa ................
0000010: aaaa aaaa aaaa aaaa aaaa aaaa aaaa aaaa ................
0000020: aaaa aaaa aaaa aaaa aaaa aaaa aaaa aaaa ................
0000030: aaaa aaaa aaaa aaaa aaaa aaaa aaaa aaaa ................
0000040: aaaa aaaa aaaa aaaa aaaa aaaa aaaa aaaa ................
0000050: aaaa aaaa aaaa aaaa aaaa aaaa aaaa aaaa ................
0000060: aaaa aaaa ....
# cat file |xxd
0000000: b6db 6db6 db6d b6db 6db6 db6d b6db 6db6 ..m..m..m..m..m.
0000010: db6d b6db 6db6 db6d b6db 6db6 db6d b6db .m..m..m..m..m..
0000020: 6db6 db6d b6db 6db6 db6d b6db 6db6 db6d m..m..m..m..m..m
0000030: b6db 6db6 db6d b6db 6db6 db6d b6db 6db6 ..m..m..m..m..m.
0000040: db6d b6db 6db6 db6d b6db 6db6 db6d b6db .m..m..m..m..m..
0000050: 6db6 db6d b6db 6db6 db6d b6db 6db6 db6d m..m..m..m..m..m
0000060: b6db 6db6 ..m.
# cat file |xxd
0000000: 6eee eeee eeee eeee eeee eeee eeee eeee n............... <- (!)
0000010: eeee eeee eeee eeee eeee eeee eeee eeee ................
0000020: eeee eeee eeee eeee eeee eeee eeee eeee ................
0000030: eeee eeee eeee eeee eeee eeee eeee eeee ................
0000040: eeee eeee eeee eeee eeee eeee eeee eeee ................
0000050: eeee eeee eeee eeee eeee eeee eeee eeee ................
0000060: eeee eeee ....
# cat file |xxd
0000000: a222 2222 2222 2222 2222 2222 2222 2222 .""""""""""""""" <- (!)
0000010: 2222 2222 2222 2222 2222 2222 2222 2222 """"""""""""""""
0000020: 2222 2222 2222 2222 2222 2222 2222 2222 """"""""""""""""
0000030: 2222 2222 2222 2222 2222 2222 2222 2222 """"""""""""""""
0000040: 2222 2222 2222 2222 2222 2222 2222 2222 """"""""""""""""
0000050: 2222 2222 2222 2222 2222 2222 2222 2222 """"""""""""""""
0000060: 2222 2222

# dd if=/dev/urandom of=/tmp/file bs=1k count=1 ; strace -f -xx -e write=all shred -n 10000000 -s 100 /tmp/file 2>&1 |grep -A1 "write(3" |head -n 50
1+0 Datensätze ein
1+0 Datensätze aus
1024 Bytes (1,0 kB) kopiert, 0,000307051 s, 3,3 MB/s
write(3, "\x39\x75\x95\xd8\xfa\x3b\xf4\xdf\x11\x09\x66\xae\x2a\x14\x4f\x24\x74\xa9\x96\x69\x3e\xe7\xb6\x75\x22\x39\x76\xee\xa6\xf8\x22\xb7"..., 100) = -1 EINVAL (Invalid argument)
fcntl(3, F_GETFL) = 0xc001 (flags O_WRONLY|O_DIRECT|O_LARGEFILE)
--
write(3, "\x39\x75\x95\xd8\xfa\x3b\xf4\xdf\x11\x09\x66\xae\x2a\x14\x4f\x24\x74\xa9\x96\x69\x3e\xe7\xb6\x75\x22\x39\x76\xee\xa6\xf8\x22\xb7"..., 100) = 100
| 00000 39 75 95 d8 fa 3b f4 df 11 09 66 ae 2a 14 4f 24 9u...;....f.*.O$ |
--
write(3, "\x55\x55\x55\x55\x55\x55\x55\x55\x55\x55\x55\x55\x55\x55\x55\x55\x55\x55\x55\x55\x55\x55\x55\x55\x55\x55\x55\x55\x55\x55\x55\x55"..., 100) = 100
| 00000 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 UUUUUUUUUUUUUUUU |
--
write(3, "\xcc\xcc\xcc\xcc\xcc\xcc\xcc\xcc\xcc\xcc\xcc\xcc\xcc\xcc\xcc\xcc\xcc\xcc\xcc\xcc\xcc\xcc\xcc\xcc\xcc\xcc\xcc\xcc\xcc\xcc\xcc\xcc"..., 100) = 100
| 00000 cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc ................ |
--
write(3, "\xaa\xaa\xaa\xaa\xaa\xaa\xaa\xaa\xaa\xaa\xaa\xaa\xaa\xaa\xaa\xaa\xaa\xaa\xaa\xaa\xaa\xaa\xaa\xaa\xaa\xaa\xaa\xaa\xaa\xaa\xaa\xaa"..., 100) = 100
| 00000 aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................ |
--
write(3, "\x91\x11\x11\x11\x11\x11\x11\x11\x11\x11\x11\x11\x11\x11\x11\x11\x11\x11\x11\x11\x11\x11\x11\x11\x11\x11\x11\x11\x11\x11\x11\x11"..., 100) = 100 <-(!)
| 00000 91 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 ................ | <-(!)
--
write(3, "\x80\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00"..., 100) = 100
| 00000 80 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ |
--
write(3, "\xa2\x22\x22\x22\x22\x22\x22\x22\x22\x22\x22\x22\x22\x22\x22\x22\x22\x22\x22\x22\x22\x22\x22\x22\x22\x22\x22\x22\x22\x22\x22\x22"..., 100) = 100  <-(!)
| 00000 a2 22 22 22 22 22 22 22 22 22 22 22 22 22 22 22 .""""""""""""""" | <-(!)
--
write(3, "\x92\x49\x24\x92\x49\x24\x92\x49\x24\x92\x49\x24\x92\x49\x24\x92\x49\x24\x92\x49\x24\x92\x49\x24\x92\x49\x24\x92\x49\x24\x92\x49"..., 100) = 100
| 00000 92 49 24 92 49 24 92 49 24 92 49 24 92 49 24 92 .I$.I$.I$.I$.I$. |
--
write(3, "\xbb\xbb\xbb\xbb\xbb\xbb\xbb\xbb\xbb\xbb\xbb\xbb\xbb\xbb\xbb\xbb\xbb\xbb\xbb\xbb\xbb\xbb\xbb\xbb\xbb\xbb\xbb\xbb\xbb\xbb\xbb\xbb"..., 100) = 100
| 00000 bb bb bb bb bb bb bb bb bb bb bb bb bb bb bb bb ................ |
--
write(3, "\x11\x11\x11\x11\x11\x11\x11\x11\x11\x11\x11\x11\x11\x11\x11\x11\x11\x11\x11\x11\x11\x11\x11\x11\x11\x11\x11\x11\x11\x11\x11\x11"..., 100) = 100
| 00000 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 ................ |
--
write(3, "\x87\xc8\x72\xf5\xeb\x3f\x8b\x9c\xa5\xbd\xf1\xc5\xdd\x74\x10\x19\x6a\xc1\xa5\x5c\xcc\x1f\x99\xba\xe7\x92\x55\x92\x55\xfb\x94\x95"..., 100) = 100
| 00000 87 c8 72 f5 eb 3f 8b 9c a5 bd f1 c5 dd 74 10 19 ..r..?.......t.. |
--
write(3, "\x92\x49\x24\x92\x49\x24\x92\x49\x24\x92\x49\x24\x92\x49\x24\x92\x49\x24\x92\x49\x24\x92\x49\x24\x92\x49\x24\x92\x49\x24\x92\x49"..., 100) = 100
| 00000 92 49 24 92 49 24 92 49 24 92 49 24 92 49 24 92 .I$.I$.I$.I$.I$. |
--
write(3, "\x66\x66\x66\x66\x66\x66\x66\x66\x66\x66\x66\x66\x66\x66\x66\x66\x66\x66\x66\x66\x66\x66\x66\x66\x66\x66\x66\x66\x66\x66\x66\x66"..., 100) = 100
| 00000 66 66 66 66 66 66 66 66 66 66 66 66 66 66 66 66 ffffffffffffffff |
--
write(3, "\x3b\xbb\xbb\xbb\xbb\xbb\xbb\xbb\xbb\xbb\xbb\xbb\xbb\xbb\xbb\xbb\xbb\xbb\xbb\xbb\xbb\xbb\xbb\xbb\xbb\xbb\xbb\xbb\xbb\xbb\xbb\xbb"..., 100) = 100 <-(!)
| 00000 3b bb bb bb bb bb bb bb bb bb bb bb bb bb bb bb ;............... | <-(!)
--
write(3, "\xb3\x33\x33\x33\x33\x33\x33\x33\x33\x33\x33\x33\x33\x33\x33\x33\x33\x33\x33\x33\x33\x33\x33\x33\x33\x33\x33\x33\x33\x33\x33\x33"..., 100) = 100 <-(!)
| 00000 b3 33 33 33 33 33 33 33 33 33 33 33 33 33 33 33 .333333333333333 | <-(!)
--
write(3, "\xaa\xaa\xaa\xaa\xaa\xaa\xaa\xaa\xaa\xaa\xaa\xaa\xaa\xaa\xaa\xaa\xaa\xaa\xaa\xaa\xaa\xaa\xaa\xaa\xaa\xaa\xaa\xaa\xaa\xaa\xaa\xaa"..., 100) = 100
| 00000 aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................ |

Message #8 received at 30174 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: devzero <at> web.de, 30174 <at> debbugs.gnu.org
Subject: Re: bug#30174: shred bug - 1st byte written is wrong sometimes
Date: Fri, 19 Jan 2018 16:21:10 -0800

On 01/19/2018 10:19 AM, devzero <at> web.de wrote:
> shred is doing
> wrong.
>
> Looks like a bug to me.

I don't see why it is a bug. It does appear to be intended behavior. 
Look in the source code, at src/shred.c, and look for "Invert the first 
bit of every sector."

Message #11 received at 30174 <at> debbugs.gnu.org (full text, mbox):

From: Pádraig Brady <P <at> draigBrady.com>
To: devzero <at> web.de, 30174 <at> debbugs.gnu.org
Subject: Re: bug#30174: shred bug - 1st byte written is wrong sometimes
Date: Fri, 19 Jan 2018 19:33:59 -0800

tag 30174 notabug
close 30174
stop

On 19/01/18 10:19, devzero <at> web.de wrote:
> Hi,
> 
> i`m testing wear-levelling of an SLC USB Stick (cheap one i want to
> use them for long-term data-logging) and found shred to be a useful
> and fast utility to repeatedly overwrite a file's region (the 
> datalogger i'm building will use rrdtool)
> 
> As i already did some other testing before, shred made me believe
> my usb stick would already be "worn out", because i was seeing 1st 
> byte flip occasionally when taking a look into the written data with 
> xxd.
> 
> I investigated and as you can see below, when repeatedly writing to 
> the same file with shred, we can see with strace that shred is doing 
> wrong.
> 
> Looks like a bug to me.
> 
> I tested this on recent CentOS7 (shred 8.22) and older Ubuntu based
> system with shred 8.25. I also compiled coreutils 8.29 and tested
> with the recent shred release - they all show this behaviour.
> 
> regards
> Roland
> 
> 
> shred -n 10000000 -s 100 file &
> 
> # cat file |xxd
> 0000000: 7fff ffff ffff ffff ffff ffff ffff ffff ................ <- (!)

This is intentional.

From the source:

  /* The following patterns have the first bit per 512 byte block flipped */
  8, 0x1000, 0x1249, 0x1492, 0x16DB, 0x1924, 0x1B6D, 0x1DB6, 0x1FFF,
  14, 0x1111, 0x1222, 0x1333, 0x1444, 0x1555, 0x1666, 0x1777,
  0x1888, 0x1999, 0x1AAA, 0x1BBB, 0x1CCC, 0x1DDD, 0x1EEE,

There was no mention why unfortunately.

cheers,
Pádraig

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.