Subject: bug#12656: cp since 8.11 corrupts files

From: "Mike Gerth" <[email-masked]>
Date: Mon, 15 Oct 2012 17:05:56 +0200

Hello,

I noticed an error in cp (8.11 upwards till 8.19) when copying many files 
from an fragmented ext3fs (to an empty partition).

ERROR:
        The copied file has exact length. 
        The SourceFile has in the middle a 4K-block filled with ZERO, the 
DestFile this 4k-block and the previous 4k-block also filled with ZERO.

I can reproduce this:
        Conditions:
                *I have a strong fragmented 110GB ext3-partition with >300 
Files
                *I have to copy about 100 files (ca. 15GB) to another 
HDD-Partition or ramdisk
                *cp must be 8.11 or higher (compiled under SLES11SP2 
kernel 3.0) (everything is ok with 6.11. 8.05..8.10)
                *Command: 
                        cp819    $FILES $DEST #here the corruption comes
                        *$FILES is a list of about 100 DatabaseFiles. The 
last one is the file that becomes corrupted
 
I can _NOT_ reproduce this:
                *with cp <8.11
                *Command:
                        for F in $FILES; do cp -dp $SRC$F $DEST$F ;done 
#no corruption

Reproduce:
        *tested with SLES11SP1, SLES11SP2, Ubuntu 12.04
                (cp less 8.11 is good, cp 8.11 upwards corrupts the file)
        *when I bulk copy the about 100 files, the corruption always 
happen with the same file and same position
        *if I copy less files or the single file there is no corruption 
        *if I copy all files there are more corruptions

Additional info:
# ./fragments /mnt/Source/
Summary statistics for /mnt/Source/ including subdirectories:

 blocks    files  fragmented  percent  fragm./file
---------------------------------------------------
      1        1           0     0.00         1.00
  <= 12        0           0     0.00         0.00
<=  524        0           0     0.00         0.00
<= 1036        0           0     0.00         0.00
<= 4108       11          11   100.00        54.00
 > 4108      307         307   100.00       159.71
---------------------------------------------------
    all      319         318    99.69       155.57
    > 1      318         318   100.00       156.05
Fragmentation index:  0.428 percent.
frag: 49625  n: 318  n_blocks: 11513587  (n_frag: 318)

#cmp Julius.nsf /mnt/ramdisk/Julius.nsf
Julius.nsf /mnt/ramdisk/Julius.nsf differ: char 18395137, line 345000
18395137=4491*4K + 1

=======================================================================================

#dd if=JuliusGOOD.nsf bs=4096 skip=4491 |hexdump -C |more
00000000  16 00 00 ff 00 10 00 00  00 00 00 00 00 00 02 00 
|................|
00000010  00 00 00 00 00 00 00 00  10 0f 00 00 00 00 00 00 
|................|
00000020  af 2f 06 00 1d 00 16 0a  00 00 00 00 40 09 88 00 
|./..........@...|
....
00000a30  00 00 00 6e 64 65 00 00  00 00 00 00 00 00 00 00 
|...nde..........|
00000a40  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
|................|
*
00000c00  01 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
|................|
00000c10  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
|................|
*
00000e00  01 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
|................|
00000e10  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
|................|
*
00002000  18 00 00 ff 00 10 00 00  00 00 00 00 00 00 02 00 
|................|

---------------

#dd if=/mnt/ramdisk/JuliusBAD.nsf bs=4096 skip=4491 |hexdump -C |more
00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
|................|
*
00002000  18 00 00 ff 00 10 00 00  00 00 00 00 00 00 02 00 
|................|

=======================================================================================
# filefrag JuliusGood.nsf -v
Filesystem type is: ef53
Filesystem cylinder groups is approximately 881
File size of JuliusGood.nsf is 175935488 (42953 blocks, blocksize 4096)
 ext logical physical expected length flags
   0       0  2008262               2 merged
   1       2  2008281  2008263      7 merged
   2       9  2008303  2008287      1 merged
   3      10  2008307  2008303      2 merged
   4      12  2008310  2008308      2 merged
   5      14  2008319  2008311      1 merged
   6      15  2008321  2008319      4 merged
   7      19  2031582  2008324      2 merged
   8      21  2033663  2031583     41 merged
   9      62  2033706  2033703      1 merged
  10      63  2033704  2033706      2 merged
  11      65  2033707  2033705     22 merged
  12      87  2033736  2033728     23 merged
  13     110  7961027  2033758      1 merged
  14     111  7961587  7961027      1 merged
  15     112  7943051  7961587      2 merged
  16     114  7932925  7943052      1 merged
  17     115  7939414  7932925      1 merged
  18     116  2053842  7939414      2 merged
  19     119  7961063  2053843      1 merged
  20     120  7961059  7961063      2 merged
  21     122  2033759  7961060    160 merged
  22     282  7943053  2033918      1 merged
  23     283  7961035  7943053      1 merged
  24     284  7961030  7961035      1 merged
  25     285  2033919  7961030    108 merged
  26     393  7961054  2034026      1 merged
  27     394  7961039  7961054      1 merged
  28     395  2034027  7961039    104 merged
  29     499  2034143  2034130      3 merged
  30     502  2034159  2034145      8 merged
  31     510  7961047  2034166      1 merged
  32     511  7961033  7961047      1 merged
  33     512  2034167  7961033    494 merged
  34    1006  2034666  2034660      1 merged
  35    1007  2034670  2034666     29 merged
  36    1036  2034701  2034698    478 merged
  37    1514  2035182  2035178    510 merged
  38    2024  2035697  2035691     24 merged
  39    2048  2038674  2035720      1 merged
  40    2049  2035721  2038674     11 merged
  41    2060  2035733  2035731    385 merged
  42    2445  2036120  2036117      2 merged
  43    2447  2036128  2036121      1 merged
  44    2448  2036136  2036128      1 merged
  45    2449  2036518  2036136      1 merged
  46    2450  2037030  2036518    634 merged
  47    3084  2037665  2037663    117 merged
  48    3201  2037784  2037781    886 merged
  49    4087  7961058  2038669      1 merged
  50    4088  7961049  7961058      1 merged
  51    4089  2038670  7961049      4 merged
  52    4093  2040616  2038673      1 merged
  53    4094  2038675  2040616     14 merged
  54    4108  2038690  2038688    291 merged
  55    4399  7961036  2038980      1 merged
  56    4400  7961050  7961036      1 merged
  57    4401  2038981  7961050     10 merged
  58    4411  2038993  2038990      9 merged
  59    4431  2038991  2039001      2 merged
  60    4433  7961064  2038992      9 merged
  61    4453  2039002  7961072     10 merged
  62    4463  2053899  2039011      1 merged
  63    4464  7932921  2053899      1 merged
  64    4465  7936120  7932921      1 merged
  65    4466  2039012  7936120     18 merged
  66    4484  7943055  2039029      1 merged
  67    4485  7961056  7943055      1 merged
  68    4486  7961041  7961056      1 merged
  69    4487  7961045  7961041      2 merged
  70    4489  7961053  7961046      1 merged
  71    4490  7961031  7961053      1 merged
  72    4491  7961034  7961031      1 merged
  73    4493  7961044  7961034      1 merged
  74    4494  2039030  7961044    285 merged
  75    4779  2054871  2039314      1 merged
  76    4780  2053844  2054871      1 merged
....

Copied corrupted File:
# filefrag /mnt/DEST/JuliusBad.nsf -v
Filesystem type is: ef53
Filesystem cylinder groups is approximately 881
File size of /mnt/DEST/gschouten.nsf is 175935488 (42953 blocks, blocksize 
4096)
 ext logical physical expected length flags
   0       0  3146242               6 merged
   1       6  3155036  3146247      6 merged
   2      12  3155043  3155041    106 merged
   3     119  3155149             917 merged
   4    1036  3156068  3156065   1024 merged
   5    2060  3157093  3157091   1024 merged
   6    3084  3158118  3158116   1024 merged
   7    4108  3159143  3159141    312 merged
   8    4431  3159455              11 merged
   9    4453  3159466              38 merged
  10    4493  3159504             639 merged
  11    5132  3160144  3160142    417 merged
  12    5550  3160561             441 merged
......

###This is the fragmented,good SourceFile:
debugfs 1.41.9 (22-Aug-2009)
debugfs:  stat JuliusGood.nsf
Inode: 102   Type: regular    Mode:  0644   Flags: 0x0
Generation: 1499014015    Version: 0x00000000
User:  1001   Group:  1001   Size: 175935488
File ACL: 0    Directory ACL: 0
Links: 1   Blockcount: 343576
Fragment:  Address: 0    Number: 0    Size: 0
ctime: 0x505e346a -- Sat Sep 22 23:58:02 2012
atime: 0x506041bf -- Mon Sep 24 13:19:27 2012
mtime: 0x505e346a -- Sat Sep 22 23:58:02 2012
Size of extra inode fields: 4
BLOCKS:
(0-1):2008262-2008263, (2-8):2008281-2008287, (9):2008303, 
(10-11):2008307-2008308, (IND):2008309, (12-13):2008310-2008311, 
(14):2008319, (15-18):2008321-2008324, (19-20):2031582
-2031583, (21-61):2033663-2033703, (62):2033706, (63-64):2033704-2033705, 
(65-86):2033707-2033728, (87-109):2033736-2033758, (110):7961027, 
(111):7961587, (112-113):7943051-79430
52, (114):7932925, (115):7939414, (116-117):2053842-2053843, 
(119):7961063, (120-121):7961059-7961060, (122-281):2033759-2033918, 
(282):7943053, (283):7961035, (284):7961030, (28
5-392):2033919-2034026, (393):7961054, (394):7961039, 
(395-498):2034027-2034130, (499-501):2034143-2034145, 
(502-509):2034159-2034166, (510):7961047, (511):7961033, (512-1005):20
34167-2034660, (1006):2034666, (1007-1035):2034670-2034698, 
(DIND):2034699, (IND):2034700, (1036-1513):2034701-2035178, 
(1514-2023):2035182-2035691, (2024-2047):2035697-2035720,
(2048):2038674, (2049-2059):2035721-2035731, (IND):2035732, 
(2060-2444):2035733-2036117, (2445-2446):2036120-2036121, (2447):2036128, 
(2448):2036136, (2449):2036518, (2450-3083):
2037030-2037663, (IND):2037664, (3084-3200):2037665-2037781, 
(3201-4086):2037784-2038669, (4087):7961058, (4088):7961049, 
(4089-4092):2038670-2038673, (4093):2040616, (4094-4107)
:2038675-2038688, (IND):2038689, (4108-4398):2038690-2038980, 
(4399):7961036, (4400):7961050, (4401-4410):2038981-2038990, 
(4411-4419):2038993-2039001, (4431-4432):2038991-203899
2, (4433-4441):7961064-7961072, (4453-4462):2039002-2039011, 
(4463):2053899, (4464):7932921, (4465):7936120, 
(4466-4483):2039012-2039029, (4484):7943055, (4485):7961056, (4486):7
961041, (4487-4488):7961045-7961046, (4489):7961053, (4490):7961031, 
(4491):7961034, (4493):7961044, (4494-4778):2039030-2039314, 
(4779):2054871, (4780):2053844, (4781-4800):2039

###Here comes the corrupted File:
debugfs 1.41.9 (22-Aug-2009)
debugfs:  stat JuliusBad.nsf
Inode: 114   Type: regular    Mode:  0644   Flags: 0x0
Generation: 3620008692    Version: 0x00000000
User:     0   Group:     0   Size: 175935488
File ACL: 0    Directory ACL: 0
Links: 1   Blockcount: 343568
Fragment:  Address: 0    Number: 0    Size: 0
ctime: 0x507831f2 -- Fri Oct 12 17:06:26 2012
atime: 0x507831f0 -- Fri Oct 12 17:06:24 2012
mtime: 0x507831f2 -- Fri Oct 12 17:06:26 2012
Size of extra inode fields: 4
BLOCKS:
(0-5):3146242-3146247, (6-11):3155036-3155041, (IND):3155042, 
(12-117):3155043-3155148, (119-1035):3155149-3156065, (DIND):3156066, 
(IND):3156067, (1036-2059):3156068-3157091, (IND):315709
2, (2060-3083):3157093-3158116, (IND):3158117, 
(3084-4107):3158118-3159141, (IND):3159142, (4108-4419):3159143-3159454, 
(4431-4441):3159455-3159465, (4453-4490):3159466-3159503, (4493-5131
):3159504-3160142,

I am 95% sure that in previous Tests if have also seen a corrupted file 
with several 100k filled with ZERO.

Surely it would be most simple to reproduce the error if I could provide 
the whole partition as is.
But since the Files are Database files I can not provide it. Are there any 
tests that I can do?

Best regards and thanks in advance for your help,
Mike

Mit freundlichen Grüßen
Mike Gerth

Dipl. Inf. Mike Gerth
+49 30 399 76 136
IT-Systemadministrator
AVM Audiovisuelles Marketing und Computersysteme GmbH 
Alt-Moabit 95, 10559 Berlin
HRB 23075 AG Charlottenburg
Geschäftsführer: Johannes Nill