From unknown Fri Aug 15 19:27:42 2025 X-Loop: help-debbugs@gnu.org Subject: bug#8200: cp -lr uses a lot of CPU time. Resent-From: Rogier Wolff Original-Sender: debbugs-submit-bounces@debbugs.gnu.org Resent-To: owner@debbugs.gnu.org Resent-CC: bug-coreutils@gnu.org Resent-Date: Tue, 08 Mar 2011 05:23:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 8200 X-GNU-PR-Package: coreutils X-GNU-PR-Keywords: To: 8200@debbugs.gnu.org X-Debbugs-Original-To: bug-coreutils@gnu.org Received: via spool by submit@debbugs.gnu.org id=B.1299561725854 (code B ref -1); Tue, 08 Mar 2011 05:23:02 +0000 Received: (at submit) by debbugs.gnu.org; 8 Mar 2011 05:22:05 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1PwpMy-0000Di-C7 for submit@debbugs.gnu.org; Tue, 08 Mar 2011 00:22:05 -0500 Received: from eggs.gnu.org ([140.186.70.92]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1PwpAC-0008NS-RR for submit@debbugs.gnu.org; Tue, 08 Mar 2011 00:08:53 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1PwpA6-0003VD-U0 for submit@debbugs.gnu.org; Tue, 08 Mar 2011 00:08:47 -0500 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-0.9 required=5.0 tests=BAYES_00,RDNS_DYNAMIC autolearn=no version=3.3.1 Received: from lists.gnu.org ([199.232.76.165]:47308) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1PwpA6-0003V0-RU for submit@debbugs.gnu.org; Tue, 08 Mar 2011 00:08:46 -0500 Received: from [140.186.70.92] (port=48144 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1PwpA5-0000DX-L8 for bug-coreutils@gnu.org; Tue, 08 Mar 2011 00:08:46 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1PwpA4-0003UB-30 for bug-coreutils@gnu.org; Tue, 08 Mar 2011 00:08:45 -0500 Received: from cust-95-128-94-82.breedbanddelft.nl ([95.128.94.82]:55022 helo=abra2.bitwizard.nl) by eggs.gnu.org with smtp (Exim 4.71) (envelope-from ) id 1PwpA3-0003TV-OH for bug-coreutils@gnu.org; Tue, 08 Mar 2011 00:08:44 -0500 Received: (qmail 13733 invoked by uid 1000); 8 Mar 2011 06:08:40 +0100 Date: Tue, 8 Mar 2011 06:08:40 +0100 From: Rogier Wolff Message-ID: <20110308050839.GA9120@bitwizard.nl> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Organization: BitWizard.nl User-Agent: Mutt/1.5.13 (2006-08-11) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 2) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 2) X-Received-From: 199.232.76.165 X-Spam-Score: -6.6 (------) X-Mailman-Approved-At: Tue, 08 Mar 2011 00:22:02 -0500 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -6.6 (------) Hi, In my backupscripts I need a "cp -lr" every day. I make backups of directories that hold up to millions of files. When cp -lr sourc dest runs for a while, it becomes CPU limited. Virtual memory is only about 2Mb. "resident" is under 1M. Top reports: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 26721 root 20 0 2456 720 468 R 58.0 0.1 65:32.60 cp 2855 root 20 0 2560 936 624 R 40.8 0.1 30:30.52 cp and I doubt they are half way through. I wrote an application I call "cplr" which does the obvious in the obvious manner, and it doesn't have this problem. I've run "strace", and determined that it is not doing systemcalls that take much CPU time. Most system calls return in microseconds. The time spent /between/ system calls runs up into the hundreds of milliseconds. You might say: well that's way less than a second. Sure. But if you need to do that tens of thousands of times it becomes quite significant.... So my question is: Why does cp -lr take such rediculous amounts of CPU time? Or another way: BUG REPORT: cp -lr takes unneccessary amounts of CPU time. Roger. -- ** R.E.Wolff@BitWizard.nl ** http://www.BitWizard.nl/ ** +31-15-2600998 ** ** Delftechpark 26 2628 XH Delft, The Netherlands. KVK: 27239233 ** *-- BitWizard writes Linux device drivers for any device you may have! --* Q: It doesn't work. A: Look buddy, doesn't work is an ambiguous statement. Does it sit on the couch all day? Is it unemployed? Please be specific! Define 'it' and what it isn't doing. --------- Adapted from lxrbot FAQ From unknown Fri Aug 15 19:27:42 2025 X-Loop: help-debbugs@gnu.org Subject: bug#8200: cp -lr uses a lot of CPU time. Resent-From: Jim Meyering Original-Sender: debbugs-submit-bounces@debbugs.gnu.org Resent-To: owner@debbugs.gnu.org Resent-CC: bug-coreutils@gnu.org Resent-Date: Tue, 08 Mar 2011 15:06:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 8200 X-GNU-PR-Package: coreutils X-GNU-PR-Keywords: To: Rogier Wolff Cc: 8200@debbugs.gnu.org Received: via spool by 8200-submit@debbugs.gnu.org id=B8200.129959671424286 (code B ref 8200); Tue, 08 Mar 2011 15:06:02 +0000 Received: (at 8200) by debbugs.gnu.org; 8 Mar 2011 15:05:14 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1PwyTJ-0006Je-Pg for submit@debbugs.gnu.org; Tue, 08 Mar 2011 10:05:14 -0500 Received: from mx.meyering.net ([82.230.74.64]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1PwyTG-0006JR-Td for 8200@debbugs.gnu.org; Tue, 08 Mar 2011 10:05:11 -0500 Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000) id D0FF660294; Tue, 8 Mar 2011 16:05:04 +0100 (CET) From: Jim Meyering In-Reply-To: <20110308050839.GA9120@bitwizard.nl> (Rogier Wolff's message of "Tue, 8 Mar 2011 06:08:40 +0100") References: <20110308050839.GA9120@bitwizard.nl> Date: Tue, 08 Mar 2011 16:05:04 +0100 Message-ID: <87zkp5ps9r.fsf@rho.meyering.net> Lines: 89 MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: -5.8 (-----) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.8 (-----) Rogier Wolff wrote: > In my backupscripts I need a "cp -lr" every day. I make backups of > directories that hold up to millions of files. > > When > > cp -lr sourc dest > > runs for a while, it becomes CPU limited. Virtual memory is only about > 2Mb. "resident" is under 1M. Thank you for the bug report. That sounds like there is a serious problem, somewhere. If you give us enough information, we'll find the cause. For starters, what version of cp did you use? Run cp --version > Top reports: > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 26721 root 20 0 2456 720 468 R 58.0 0.1 65:32.60 cp > 2855 root 20 0 2560 936 624 R 40.8 0.1 30:30.52 cp > > and I doubt they are half way through. > > I wrote an application I call "cplr" which does the obvious in the > obvious manner, and it doesn't have this problem. > > I've run "strace", and determined that it is not doing systemcalls > that take much CPU time. Most system calls return in microseconds. Please give us a sample listing of the syscalls that strace shows you when you trace one of those long-running cp commands. A few hundred lines worth would be good. > The time spent /between/ system calls runs up into the hundreds of > milliseconds. You might say: well that's way less than a > second. Sure. But if you need to do that tens of thousands of times it > becomes quite significant.... > > > So my question is: Why does cp -lr take such rediculous amounts of CPU > time? > > Or another way: BUG REPORT: cp -lr takes unneccessary amounts of CPU > time. What type of file system are you using, and is it nearly full? Run this from e.g, the source directory: df -hT . Ideally, you'd attach to one of those processes with gdb and step through the code enough to tell us where it's spending its time, presumably in coreutils-8.10/src/copy.c. Just running "gdb -p 26721" (where 26721 is the PID of one of your running cp processes) and typing "backtrace" at the prompt may give us a good clue. Next best, you would give us access to your system or a copy of your hierarchy. But we don't even ask for that, because that's rarely feasible. Next best: you would give us the output of these two commands: [if you can do this, please respond privately, not to the list] find YOUR_SRC_DIR -ls | xz -e > src.find.xz find YOUR_DST_DIR -ls | xz -e > dst.find.xz [if you don't have xz, install it or use bzip2 -9 instead of "xz -e"; xz is better] With that, we'd get an idea of hard link counts and max/average number of entries per directory, name length, etc. However, most people don't want to share file names like that. If you can, please put those two compressed files somewhere like an upload site and reply with links to them. Otherwise, please give us some statistics describing your two hierarchies by running these commands: These give counts of files and directories for each of your source and destination directories: find YOUR_SRC_DIR -type f |wc -l find YOUR_SRC_DIR -type d |wc -l find YOUR_DST_DIR -type f |wc -l find YOUR_DST_DIR -type d |wc -l Print the total number of links for each of those directories: find YOUR_SRC_DIR -type f -printf '%n\n'|awk '{s += $1} END {printf "%F\n", s}' find YOUR_DST_DIR -type f -printf '%n\n'|awk '{s += $1} END {printf "%F\n", s}' Jim From unknown Fri Aug 15 19:27:42 2025 X-Loop: help-debbugs@gnu.org Subject: bug#8200: cp -lr uses a lot of CPU time. Resent-From: Rogier Wolff Original-Sender: debbugs-submit-bounces@debbugs.gnu.org Resent-To: owner@debbugs.gnu.org Resent-CC: bug-coreutils@gnu.org Resent-Date: Tue, 08 Mar 2011 16:36:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 8200 X-GNU-PR-Package: coreutils X-GNU-PR-Keywords: To: Jim Meyering Cc: 8200@debbugs.gnu.org, Rogier Wolff Received: via spool by 8200-submit@debbugs.gnu.org id=B8200.129960214732482 (code B ref 8200); Tue, 08 Mar 2011 16:36:01 +0000 Received: (at 8200) by debbugs.gnu.org; 8 Mar 2011 16:35:47 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Pwzsv-0008Rq-ML for submit@debbugs.gnu.org; Tue, 08 Mar 2011 11:35:46 -0500 Received: from cust-95-128-94-82.breedbanddelft.nl ([95.128.94.82] helo=abra2.bitwizard.nl) by debbugs.gnu.org with smtp (Exim 4.69) (envelope-from ) id 1Pwzsf-0008RJ-Gd for 8200@debbugs.gnu.org; Tue, 08 Mar 2011 11:35:44 -0500 Received: (qmail 28467 invoked by uid 1000); 8 Mar 2011 17:35:23 +0100 Date: Tue, 8 Mar 2011 17:35:22 +0100 From: Rogier Wolff Message-ID: <20110308163522.GB3125@bitwizard.nl> References: <20110308050839.GA9120@bitwizard.nl> <87zkp5ps9r.fsf@rho.meyering.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <87zkp5ps9r.fsf@rho.meyering.net> Organization: BitWizard.nl User-Agent: Mutt/1.5.13 (2006-08-11) X-Spam-Score: -4.5 (----) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -3.9 (---) Hi Jim & others, Aaargh... It seems the bug has been fixed... Feel free to ignore my explanation below. On Tue, Mar 08, 2011 at 04:05:04PM +0100, Jim Meyering wrote: > For starters, what version of cp did you use? > Run cp --version -> cp (GNU coreutils) 8.5 > > Top reports: > > > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > > 26721 root 20 0 2456 720 468 R 58.0 0.1 65:32.60 cp > > 2855 root 20 0 2560 936 624 R 40.8 0.1 30:30.52 cp > > > > and I doubt they are half way through. > > > > I wrote an application I call "cplr" which does the obvious in the > > obvious manner, and it doesn't have this problem. > > > > I've run "strace", and determined that it is not doing systemcalls > > that take much CPU time. Most system calls return in microseconds. > > Please give us a sample listing of the syscalls that strace > shows you when you trace one of those long-running cp commands. > A few hundred lines worth would be good. I ran: time strace -tttTp 11453 | & head -1000 | awk '{print ($1-t)*1000 , $0 ; t=$1;}' to get the output of 1000 of the process' system calls. Previously I had omitted the "*1000" which made things harder to read, and I hadn't noticed that the mkdir calls were the calls that took a long time..... My own "cplr" program I started one time without any arguments and it said: => Usage: cplr srcdir dstdir => Copy srcdir to dstdir by making hardlinks => (Like cp -lR, but without consuming lots of memory) So apparently the problem we ran into when I made that was that cp was consuming much too much memory.... This apparently has been fixed in the meantime. Here is a typical section of the strace output. This is from my own cplr program, as the "cp" has scrolled out of my screen and I've stopped the cp -lr, as the problem has been fixed. 0.0741482 1299598743.435264 link("current/linux-2.6.0-test2-clean/fs/file_table.c", "test2/linux-2.6.0-test2-clean/fs/file_table.c") = 0 <0.000047> 0.133991 1299598743.435398 link("current/linux-2.6.0-test2-clean/fs/read_write.c", "test2/linux-2.6.0-test2-clean/fs/read_write.c") = 0 <0.000036> 0.122786 1299598743.435521 link("current/linux-2.6.0-test2-clean/fs/xattr_acl.c", "test2/linux-2.6.0-test2-clean/fs/xattr_acl.c") = 0 <0.000041> 0.13113 1299598743.435652 link("current/linux-2.6.0-test2-clean/fs/jffs2", "test2/linux-2.6.0-test2-clean/fs/jffs2") = -1 EPERM (Operation not permitted) <0.000046> 0.740051 1299598743.436392 lstat64("current/linux-2.6.0-test2-clean/fs/jffs2", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 <0.000015> 0.119925 1299598743.436512 open("current/linux-2.6.0-test2-clean/fs/jffs2/", O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY|O_CLOEXEC) = 6 <0.000022> 0.0889301 1299598743.436601 mkdir("test2/linux-2.6.0-test2-clean/fs/jffs2/", 0777) = 0 <0.031938> 32.057 1299598743.468658 getdents(6, /* 36 entries */, 32768) = 776 <0.000317> > What type of file system are you using, and is it nearly full? > Run this from e.g, the source directory: df -hT . Filesystem Type Size Used Avail Use% Mounted on /dev/md3 ext3 2.7T 2.4T 190G 93% /backup > Ideally, you'd attach to one of those processes with gdb and step > through the code enough to tell us where it's spending its time, > presumably in coreutils-8.10/src/copy.c. Just running "gdb -p > 26721" (where 26721 is the PID of one of your running cp processes) > and typing "backtrace" at the prompt may give us a good clue. It's spending time in mkdir. It's visible from the strace output. For a sample directory, download the linux source code, unpack it some 300 times into different subdirs (after unpacking, rename the resulting tree to linux-1 linux-2, etc.) (My count comes to 325 copies, but many are 2.4, so a lot smaller than current kernels). > Next best, you would give us access to your system or a copy of your hierarchy. > But we don't even ask for that, because that's rarely feasible. > Next best: you would give us the output of these two commands: > [if you can do this, please respond privately, not to the list] > > find YOUR_SRC_DIR -ls | xz -e > src.find.xz > find YOUR_DST_DIR -ls | xz -e > dst.find.xz > > [if you don't have xz, install it or use bzip2 -9 instead of "xz -e"; > xz is better] > > With that, we'd get an idea of hard link counts and max/average > number of entries per directory, name length, etc. > > However, most people don't want to share file names like that. > If you can, please put those two compressed files somewhere like > an upload site and reply with links to them. > Otherwise, please give us some statistics describing your > two hierarchies by running these commands: > > These give counts of files and directories for each of your source > and destination directories: dest dir is created by the cp -lr. so it starts out empty, and ends up with the same number as the source dir. :-) > find YOUR_SRC_DIR -type f |wc -l About 4.7 million. > find YOUR_SRC_DIR -type d |wc -l About 325000. > Print the total number of links for each of those directories: You say links for directories, but your command counts the links on the files... > find YOUR_SRC_DIR -type f -printf '%n\n'|awk '{s += $1} END {printf "%F\n", s}' 539 million. So about 100 links to each file on average. Rogier. -- ** R.E.Wolff@BitWizard.nl ** http://www.BitWizard.nl/ ** +31-15-2600998 ** ** Delftechpark 26 2628 XH Delft, The Netherlands. KVK: 27239233 ** *-- BitWizard writes Linux device drivers for any device you may have! --* Q: It doesn't work. A: Look buddy, doesn't work is an ambiguous statement. Does it sit on the couch all day? Is it unemployed? Please be specific! Define 'it' and what it isn't doing. --------- Adapted from lxrbot FAQ From unknown Fri Aug 15 19:27:42 2025 MIME-Version: 1.0 X-Mailer: MIME-tools 5.427 (Entity 5.427) X-Loop: help-debbugs@gnu.org From: help-debbugs@gnu.org (GNU bug Tracking System) To: Rogier Wolff Subject: bug#8200: closed (Re: bug#8200: cp -lr uses a lot of CPU time.) Message-ID: References: <87vczqvprd.fsf@rho.meyering.net> <20110308050839.GA9120@bitwizard.nl> X-Gnu-PR-Message: they-closed 8200 X-Gnu-PR-Package: coreutils Reply-To: 8200@debbugs.gnu.org Date: Thu, 10 Mar 2011 17:40:03 +0000 Content-Type: multipart/mixed; boundary="----------=_1299778803-20062-1" This is a multi-part message in MIME format... ------------=_1299778803-20062-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Your bug report #8200: cp -lr uses a lot of CPU time.=20 which was filed against the coreutils package, has been closed. The explanation is attached below, along with your original report. If you require more details, please reply to 8200@debbugs.gnu.org. --=20 8200: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D8200 GNU Bug Tracking System Contact help-debbugs@gnu.org with problems ------------=_1299778803-20062-1 Content-Type: message/rfc822 Content-Disposition: inline Content-Transfer-Encoding: 7bit Received: (at 8200-done) by debbugs.gnu.org; 10 Mar 2011 17:39:43 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Pxjpu-0005D1-MP for submit@debbugs.gnu.org; Thu, 10 Mar 2011 12:39:42 -0500 Received: from mx.meyering.net ([82.230.74.64]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Pxjps-0005Cn-Gc for 8200-done@debbugs.gnu.org; Thu, 10 Mar 2011 12:39:41 -0500 Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000) id 5D363601D7; Thu, 10 Mar 2011 18:39:34 +0100 (CET) From: Jim Meyering To: Rogier Wolff Subject: Re: bug#8200: cp -lr uses a lot of CPU time. In-Reply-To: <20110308163522.GB3125@bitwizard.nl> (Rogier Wolff's message of "Tue, 8 Mar 2011 17:35:22 +0100") References: <20110308050839.GA9120@bitwizard.nl> <87zkp5ps9r.fsf@rho.meyering.net> <20110308163522.GB3125@bitwizard.nl> Date: Thu, 10 Mar 2011 18:39:34 +0100 Message-ID: <87vczqvprd.fsf@rho.meyering.net> Lines: 21 MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: -5.8 (-----) X-Debbugs-Envelope-To: 8200-done Cc: 8200-done@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.8 (-----) Rogier Wolff wrote: > Aaargh... It seems the bug has been fixed... Feel free to ignore my > explanation below. Thanks. I've marked this ticket as closed. > On Tue, Mar 08, 2011 at 04:05:04PM +0100, Jim Meyering wrote: >> For starters, what version of cp did you use? >> Run cp --version > > -> cp (GNU coreutils) 8.5 In v7.0-63-g3ece035, I made this change: http://git.sv.gnu.org/cgit/coreutils.git/commit/?id=3ece0355d52e41a1 cp: use far less memory in some cases cp --link was "remembering" many name,dev,inode triples unnecessarily. ... which would make a big difference in your case. ------------=_1299778803-20062-1 Content-Type: message/rfc822 Content-Disposition: inline Content-Transfer-Encoding: 7bit Received: (at submit) by debbugs.gnu.org; 8 Mar 2011 05:22:05 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1PwpMy-0000Di-C7 for submit@debbugs.gnu.org; Tue, 08 Mar 2011 00:22:05 -0500 Received: from eggs.gnu.org ([140.186.70.92]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1PwpAC-0008NS-RR for submit@debbugs.gnu.org; Tue, 08 Mar 2011 00:08:53 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1PwpA6-0003VD-U0 for submit@debbugs.gnu.org; Tue, 08 Mar 2011 00:08:47 -0500 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-0.9 required=5.0 tests=BAYES_00,RDNS_DYNAMIC autolearn=no version=3.3.1 Received: from lists.gnu.org ([199.232.76.165]:47308) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1PwpA6-0003V0-RU for submit@debbugs.gnu.org; Tue, 08 Mar 2011 00:08:46 -0500 Received: from [140.186.70.92] (port=48144 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1PwpA5-0000DX-L8 for bug-coreutils@gnu.org; Tue, 08 Mar 2011 00:08:46 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1PwpA4-0003UB-30 for bug-coreutils@gnu.org; Tue, 08 Mar 2011 00:08:45 -0500 Received: from cust-95-128-94-82.breedbanddelft.nl ([95.128.94.82]:55022 helo=abra2.bitwizard.nl) by eggs.gnu.org with smtp (Exim 4.71) (envelope-from ) id 1PwpA3-0003TV-OH for bug-coreutils@gnu.org; Tue, 08 Mar 2011 00:08:44 -0500 Received: (qmail 13733 invoked by uid 1000); 8 Mar 2011 06:08:40 +0100 Date: Tue, 8 Mar 2011 06:08:40 +0100 From: Rogier Wolff To: bug-coreutils@gnu.org Subject: cp -lr uses a lot of CPU time. Message-ID: <20110308050839.GA9120@bitwizard.nl> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Organization: BitWizard.nl User-Agent: Mutt/1.5.13 (2006-08-11) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 2) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 2) X-Received-From: 199.232.76.165 X-Spam-Score: -6.6 (------) X-Debbugs-Envelope-To: submit X-Mailman-Approved-At: Tue, 08 Mar 2011 00:22:02 -0500 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -6.6 (------) Hi, In my backupscripts I need a "cp -lr" every day. I make backups of directories that hold up to millions of files. When cp -lr sourc dest runs for a while, it becomes CPU limited. Virtual memory is only about 2Mb. "resident" is under 1M. Top reports: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 26721 root 20 0 2456 720 468 R 58.0 0.1 65:32.60 cp 2855 root 20 0 2560 936 624 R 40.8 0.1 30:30.52 cp and I doubt they are half way through. I wrote an application I call "cplr" which does the obvious in the obvious manner, and it doesn't have this problem. I've run "strace", and determined that it is not doing systemcalls that take much CPU time. Most system calls return in microseconds. The time spent /between/ system calls runs up into the hundreds of milliseconds. You might say: well that's way less than a second. Sure. But if you need to do that tens of thousands of times it becomes quite significant.... So my question is: Why does cp -lr take such rediculous amounts of CPU time? Or another way: BUG REPORT: cp -lr takes unneccessary amounts of CPU time. Roger. -- ** R.E.Wolff@BitWizard.nl ** http://www.BitWizard.nl/ ** +31-15-2600998 ** ** Delftechpark 26 2628 XH Delft, The Netherlands. KVK: 27239233 ** *-- BitWizard writes Linux device drivers for any device you may have! --* Q: It doesn't work. A: Look buddy, doesn't work is an ambiguous statement. Does it sit on the couch all day? Is it unemployed? Please be specific! Define 'it' and what it isn't doing. --------- Adapted from lxrbot FAQ ------------=_1299778803-20062-1--