From debbugs-submit-bounces@debbugs.gnu.org Fri Apr 09 20:32:58 2010 Received: (at submit) by debbugs.gnu.org; 10 Apr 2010 00:32:58 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1O0Od7-0003XS-Cm for submit@debbugs.gnu.org; Fri, 09 Apr 2010 20:32:58 -0400 Received: from mail.gnu.org ([199.232.76.166] helo=mx10.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1O0OZZ-0003Vr-Sq for submit@debbugs.gnu.org; Fri, 09 Apr 2010 20:29:18 -0400 Received: from lists.gnu.org ([199.232.76.165]:37821) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1O0OZW-0003s9-60 for submit@debbugs.gnu.org; Fri, 09 Apr 2010 20:29:14 -0400 Received: from mailman by lists.gnu.org with tmda-scanned (Exim 4.43) id 1O0OZV-0007d7-OP for bug-coreutils@gnu.org; Fri, 09 Apr 2010 20:29:13 -0400 Received: from [140.186.70.92] (port=54111 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1O0OZU-0007cG-Db for bug-coreutils@gnu.org; Fri, 09 Apr 2010 20:29:13 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.0 (2010-01-18) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00 autolearn=unavailable version=3.3.0 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1O0OZS-0001aU-CN for bug-coreutils@gnu.org; Fri, 09 Apr 2010 20:29:12 -0400 Received: from www.viadmin.org ([195.145.128.101]:56782) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1O0OZR-0001a3-Uk for bug-coreutils@gnu.org; Fri, 09 Apr 2010 20:29:10 -0400 Received: from localhost (localhost.localdomain [127.0.0.1]) by www.viadmin.org (Postfix) with ESMTP id 0F8F25670E; Sat, 10 Apr 2010 02:29:06 +0200 (CEST) X-Virus-Scanned: Debian amavisd-new at www.viadmin.org Received: from www.viadmin.org ([127.0.0.1]) by localhost (www.viadmin.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id EekAbHtfyPXF; Sat, 10 Apr 2010 02:28:57 +0200 (CEST) Received: by www.viadmin.org (Postfix, from userid 1005) id 70FB7565C1; Sat, 10 Apr 2010 02:28:57 +0200 (CEST) Date: Sat, 10 Apr 2010 02:28:57 +0200 From: Heinrich Langos To: Andreas Schwab Subject: Re: [dd] conv=sparse option Message-ID: <20100410002857.GB17172@www.viadmin.org> References: <20071229224147.GA5914@implementation> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Archive: encrypt User-Agent: Mutt/1.5.18 (2008-05-17) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -4.0 (----) X-Debbugs-Envelope-To: submit X-Mailman-Approved-At: Fri, 09 Apr 2010 20:32:56 -0400 Cc: Samuel Thibault , bug-coreutils@gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.3 (-----) Hello Andreas, Samuel and list, sorry to pick up such an old thread, but I stumbled upon it while looking for an efficient way to "re-sparse" files that contain a lot of zero blocks but 1) had already been expanded or 2) are being expanded due to pipes. On Sun, Dec 30, 2007 at 10:19:54AM +0100, Andreas Schwab wrote: > Samuel Thibault writes: > > > Some time ago, I wrote a conv=sparse option for dd, attached is the > > patch. > > How is it different from cp --sparse=always? I'd say in enough ways to make such an option highly desirable. a) "dd" will maintain an existing of=target file including the inode number, thus respecting existing hard links. "cp" will depending on the other options given (e.g. "-a") maintain or break existing hard links to an existing target file. b) "dd" could read a stream from a device or stdin and write it directly to a sparse file. no need to "dd" from e.g. a block device to a file and afterwards do a "cp --sparse=always file sparse-file". this will save a lot of disk space, io operations and time. example transcript for a) : 1 hlangos@jukebox:~/sparse$ ls -lis 2 total 1984 3 114692 496 -rw-r--r-- 4 hlangos hlangos 500000 2010-04-10 01:55 non-sparse 4 114692 496 -rw-r--r-- 4 hlangos hlangos 500000 2010-04-10 01:55 non-sparse2 5 114692 496 -rw-r--r-- 4 hlangos hlangos 500000 2010-04-10 01:55 non-sparse3 6 114692 496 -rw-r--r-- 4 hlangos hlangos 500000 2010-04-10 01:55 non-sparse4 7 114690 0 -rw-r--r-- 1 hlangos hlangos 500000 2010-04-10 01:56 sparse 8 hlangos@jukebox:~/sparse$ cp sparse non-sparse 9 hlangos@jukebox:~/sparse$ ls -lis 10 total 0 11 114692 0 -rw-r--r-- 4 hlangos hlangos 500000 2010-04-10 01:56 non-sparse 12 114692 0 -rw-r--r-- 4 hlangos hlangos 500000 2010-04-10 01:56 non-sparse2 13 114692 0 -rw-r--r-- 4 hlangos hlangos 500000 2010-04-10 01:56 non-sparse3 14 114692 0 -rw-r--r-- 4 hlangos hlangos 500000 2010-04-10 01:56 non-sparse4 15 114690 0 -rw-r--r-- 1 hlangos hlangos 500000 2010-04-10 01:56 sparse 16 hlangos@jukebox:~/sparse$ dd if=/dev/zero bs=1 count=500000 of=non-sparse 17 500000+0 records in 18 500000+0 records out 19 500000 bytes (500 kB) copied, 3.96621 s, 126 kB/s 20 hlangos@jukebox:~/sparse$ ls -lis 21 total 1984 22 114692 496 -rw-r--r-- 4 hlangos hlangos 500000 2010-04-10 01:57 non-sparse 23 114692 496 -rw-r--r-- 4 hlangos hlangos 500000 2010-04-10 01:57 non-sparse2 24 114692 496 -rw-r--r-- 4 hlangos hlangos 500000 2010-04-10 01:57 non-sparse3 25 114692 496 -rw-r--r-- 4 hlangos hlangos 500000 2010-04-10 01:57 non-sparse4 26 114690 0 -rw-r--r-- 1 hlangos hlangos 500000 2010-04-10 01:56 sparse 27 hlangos@jukebox:~/sparse$ cp -a sparse non-sparse 28 hlangos@jukebox:~/sparse$ ls -lis 29 total 1488 30 114691 0 -rw-r--r-- 1 hlangos hlangos 500000 2010-04-10 01:56 non-sparse 31 114692 496 -rw-r--r-- 3 hlangos hlangos 500000 2010-04-10 01:57 non-sparse2 32 114692 496 -rw-r--r-- 3 hlangos hlangos 500000 2010-04-10 01:57 non-sparse3 33 114692 496 -rw-r--r-- 3 hlangos hlangos 500000 2010-04-10 01:57 non-sparse4 34 114690 0 -rw-r--r-- 1 hlangos hlangos 500000 2010-04-10 01:56 sparse 35 hlangos@jukebox:~/sparse$ As you see in line 30, a new "non-sparse" file has been created with a different inode number while the link count of the other "non-sparse*" files has be reduced. I'd very much like to see the patch make it into "dd", though I think it might be better to integrate that function as "oflag=sparse" instead of "conv=sparse". After all you don't convert data but change the way the output is done. cheers -henrik From debbugs-submit-bounces@debbugs.gnu.org Sat Apr 10 11:33:28 2010 Received: (at 5918) by debbugs.gnu.org; 10 Apr 2010 15:33:28 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1O0cgZ-0001RD-Q4 for submit@debbugs.gnu.org; Sat, 10 Apr 2010 11:33:27 -0400 Received: from mail1.slb.deg.dub.stisp.net ([84.203.253.98]) by debbugs.gnu.org with smtp (Exim 4.69) (envelope-from ) id 1O0cgX-0001R7-Ng for 5918@debbugs.gnu.org; Sat, 10 Apr 2010 11:33:26 -0400 Received: (qmail 44248 invoked from network); 10 Apr 2010 15:33:22 -0000 Received: from unknown (HELO ?192.168.2.25?) (84.203.137.218) by mail1.slb.deg.dub.stisp.net with SMTP; 10 Apr 2010 15:33:22 -0000 Message-ID: <4BC09A33.1060802@draigBrady.com> Date: Sat, 10 Apr 2010 16:33:07 +0100 From: =?ISO-8859-1?Q?P=E1draig_Brady?= User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.8) Gecko/20100227 Thunderbird/3.0.3 MIME-Version: 1.0 To: Heinrich Langos Subject: Re: bug#5918: [dd] conv=sparse option References: <20071229224147.GA5914@implementation> <20100410002857.GB17172@www.viadmin.org> In-Reply-To: <20100410002857.GB17172@www.viadmin.org> X-Enigmail-Version: 1.0.1 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit X-Spam-Score: -3.8 (---) X-Debbugs-Envelope-To: 5918 Cc: Andreas Schwab , 5918@debbugs.gnu.org, samuel.thibault@ens-lyon.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -3.6 (---) On 10/04/10 01:28, Heinrich Langos wrote: > Hello Andreas, Samuel and list, > > sorry to pick up such an old thread, but I stumbled upon it while > looking for an efficient way to "re-sparse" files that contain a > lot of zero blocks but > 1) had already been expanded > or > 2) are being expanded due to pipes. > > On Sun, Dec 30, 2007 at 10:19:54AM +0100, Andreas Schwab wrote: >> Samuel Thibault writes: >> >>> Some time ago, I wrote a conv=sparse option for dd, attached is the >>> patch. >> >> How is it different from cp --sparse=always? > > I'd say in enough ways to make such an option highly desirable. > > a) "dd" will maintain an existing of=target file including the inode > number, thus respecting existing hard links. "cp" will depending > on the other options given (e.g. "-a") maintain or break existing > hard links to an existing target file. I don't think that's possible as holes can only be created at the end of a file. Well I think NTFS supports punching holes in the "middle" but it's not common. > > b) "dd" could read a stream from a device or stdin and write it directly > to a sparse file. no need to "dd" from e.g. a block device to a file and > afterwards do a "cp --sparse=always file sparse-file". this will save a > lot of disk space, io operations and time. This seems to work: cp --sparse=always /dev/stdin file cheers, P�draig. From debbugs-submit-bounces@debbugs.gnu.org Sat Apr 10 13:17:06 2010 Received: (at 5918) by debbugs.gnu.org; 10 Apr 2010 17:17:06 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1O0eIr-0002lN-Nu for submit@debbugs.gnu.org; Sat, 10 Apr 2010 13:17:06 -0400 Received: from solo.fdn.fr ([80.67.169.19]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1O0dp3-0002YV-8H for 5918@debbugs.gnu.org; Sat, 10 Apr 2010 12:46:18 -0400 Received: from const.ipv6 (youpi.is-a-geek.org [80.67.176.89]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (Client did not present a certificate) by solo.fdn.fr (Postfix) with ESMTP id F008A4435B; Sat, 10 Apr 2010 18:46:13 +0200 (CEST) Received: from samy by const.ipv6 with local (Exim 4.71) (envelope-from ) id 1O0doz-0004Ed-5g; Sat, 10 Apr 2010 18:46:13 +0200 Date: Sat, 10 Apr 2010 18:46:13 +0200 From: Samuel Thibault To: =?iso-8859-1?Q?P=E1draig?= Brady Subject: Re: bug#5918: [dd] conv=sparse option Message-ID: <20100410164613.GV4963@const.famille.thibault.fr> References: <20071229224147.GA5914@implementation> <20100410002857.GB17172@www.viadmin.org> <4BC09A33.1060802@draigBrady.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline In-Reply-To: <4BC09A33.1060802@draigBrady.com> User-Agent: Mutt/1.5.12-2006-07-14 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -1.1 (-) X-Debbugs-Envelope-To: 5918 X-Mailman-Approved-At: Sat, 10 Apr 2010 13:17:04 -0400 Cc: Andreas Schwab , 5918@debbugs.gnu.org, Heinrich Langos X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -1.9 (-) P=E1draig Brady, le Sat 10 Apr 2010 16:33:07 +0100, a =E9crit : > On 10/04/10 01:28, Heinrich Langos wrote: > > a) "dd" will maintain an existing of=3Dtarget file including the inod= e=20 > > number, thus respecting existing hard links. "cp" will depending=20 > > on the other options given (e.g. "-a") maintain or break existing=20 > > hard links to an existing target file. >=20 > I don't think that's possible as holes can only be created at the end o= f a file. > Well I think NTFS supports punching holes in the "middle" but it's not = common. I believe there's demand for supporting punching holes in the middle of files and it will eventually show up in Linux. For instance, the combination of IDE TRIM support and virtualization can allow virtualized guest to take as less disk space as possible in file-backed virtual disks. Samuel From debbugs-submit-bounces@debbugs.gnu.org Sun Apr 11 10:01:50 2010 Received: (at 5918) by debbugs.gnu.org; 11 Apr 2010 14:01:50 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1O0xjR-0003i2-Sx for submit@debbugs.gnu.org; Sun, 11 Apr 2010 10:01:50 -0400 Received: from smtp3-g21.free.fr ([212.27.42.3]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1O0xjP-0003hx-6Y for 5918@debbugs.gnu.org; Sun, 11 Apr 2010 10:01:48 -0400 Received: from smtp3-g21.free.fr (localhost [127.0.0.1]) by smtp3-g21.free.fr (Postfix) with ESMTP id 1E94F818102 for <5918@debbugs.gnu.org>; Sun, 11 Apr 2010 16:01:38 +0200 (CEST) Received: from mx.meyering.net (mx.meyering.net [82.230.74.64]) by smtp3-g21.free.fr (Postfix) with ESMTP id 402BC81811B for <5918@debbugs.gnu.org>; Sun, 11 Apr 2010 16:01:36 +0200 (CEST) Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000) id 16D6AD54; Sun, 11 Apr 2010 16:01:36 +0200 (CEST) From: Jim Meyering To: =?utf-8?Q?P=C3=A1draig?= Brady Subject: Re: bug#5918: [dd] conv=sparse option In-Reply-To: <4BC09A33.1060802@draigBrady.com> (=?utf-8?Q?=22P=C3=A1draig?= Brady"'s message of "Sat, 10 Apr 2010 16:33:07 +0100") References: <20071229224147.GA5914@implementation> <20100410002857.GB17172@www.viadmin.org> <4BC09A33.1060802@draigBrady.com> Date: Sun, 11 Apr 2010 16:01:35 +0200 Message-ID: <87vdbyrsu8.fsf@meyering.net> Lines: 35 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.9 (--) X-Debbugs-Envelope-To: 5918 Cc: Andreas Schwab , 5918@debbugs.gnu.org, samuel.thibault@ens-lyon.org, Heinrich Langos X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -2.9 (--) P=C3=A1draig Brady wrote: > On 10/04/10 01:28, Heinrich Langos wrote: >> Hello Andreas, Samuel and list, >> >> sorry to pick up such an old thread, but I stumbled upon it while >> looking for an efficient way to "re-sparse" files that contain a >> lot of zero blocks but >> 1) had already been expanded >> or >> 2) are being expanded due to pipes. >> >> On Sun, Dec 30, 2007 at 10:19:54AM +0100, Andreas Schwab wrote: >>> Samuel Thibault writes: >>> >>>> Some time ago, I wrote a conv=3Dsparse option for dd, attached is the >>>> patch. >>> >>> How is it different from cp --sparse=3Dalways? >> >> I'd say in enough ways to make such an option highly desirable. >> >> a) "dd" will maintain an existing of=3Dtarget file including the inode >> number, thus respecting existing hard links. "cp" will depending >> on the other options given (e.g. "-a") maintain or break existing >> hard links to an existing target file. > > I don't think that's possible as holes can only be created at the end of = a file. > Well I think NTFS supports punching holes in the "middle" but it's not co= mmon. I would like at least cp to be able to copy sparse files efficiently, and considering the FIEMAP patches that Jeff Liu is working on, we don't have long to wait. BTW, I'm pretty sure it is possible to punch a hole in the middle of a file with XFS. Maybe with other CoW file systems, too? From debbugs-submit-bounces@debbugs.gnu.org Tue Apr 13 06:31:12 2010 Received: (at submit) by debbugs.gnu.org; 13 Apr 2010 10:31:12 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1O1dOi-0001dv-GU for submit@debbugs.gnu.org; Tue, 13 Apr 2010 06:31:12 -0400 Received: from mail.gnu.org ([199.232.76.166] helo=mx10.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1O1dOf-0001dq-4i for submit@debbugs.gnu.org; Tue, 13 Apr 2010 06:31:09 -0400 Received: from lists.gnu.org ([199.232.76.165]:44901) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1O1dOZ-00061t-Jb for submit@debbugs.gnu.org; Tue, 13 Apr 2010 06:31:03 -0400 Received: from mailman by lists.gnu.org with tmda-scanned (Exim 4.43) id 1O1dOZ-0005l8-7K for bug-coreutils@gnu.org; Tue, 13 Apr 2010 06:31:03 -0400 Received: from [140.186.70.92] (port=43145 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1O1dOX-0005l0-By for bug-coreutils@gnu.org; Tue, 13 Apr 2010 06:31:02 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.0 (2010-01-18) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00 autolearn=unavailable version=3.3.0 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1O1dOU-0008HJ-NK for bug-coreutils@gnu.org; Tue, 13 Apr 2010 06:31:01 -0400 Received: from www.viadmin.org ([195.145.128.101]:57764) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1O1dOU-0008Gg-IB for bug-coreutils@gnu.org; Tue, 13 Apr 2010 06:30:58 -0400 Received: from localhost (localhost.localdomain [127.0.0.1]) by www.viadmin.org (Postfix) with ESMTP id 0212B2E8C0 for ; Tue, 13 Apr 2010 12:30:55 +0200 (CEST) X-Virus-Scanned: Debian amavisd-new at www.viadmin.org Received: from www.viadmin.org ([127.0.0.1]) by localhost (www.viadmin.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id i-DuVSyKSyCt for ; Tue, 13 Apr 2010 12:30:47 +0200 (CEST) Received: by www.viadmin.org (Postfix, from userid 1005) id B2FA12E7CB; Tue, 13 Apr 2010 12:30:47 +0200 (CEST) Date: Tue, 13 Apr 2010 12:30:47 +0200 From: "H. Langos" To: bug-coreutils@gnu.org Subject: Re: bug#5918: [dd] conv=sparse option Message-ID: <20100413103047.GC17172@www.viadmin.org> References: <20071229224147.GA5914@implementation> <20100410002857.GB17172@www.viadmin.org> <4BC09A33.1060802@draigBrady.com> <20100410164613.GV4963@const.famille.thibault.fr> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline In-Reply-To: <20100410164613.GV4963@const.famille.thibault.fr> X-Archive: encrypt User-Agent: Mutt/1.5.18 (2008-05-17) Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -4.6 (----) X-Debbugs-Envelope-To: submit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.7 (-----) On Sat, Apr 10, 2010 at 06:46:13PM +0200, Samuel Thibault wrote: > P=E1draig Brady, le Sat 10 Apr 2010 16:33:07 +0100, a =E9crit : > > On 10/04/10 01:28, Heinrich Langos wrote: > > > a) "dd" will maintain an existing of=3Dtarget file including the in= ode=20 > > > number, thus respecting existing hard links. "cp" will depending= =20 > > > on the other options given (e.g. "-a") maintain or break existin= g=20 > > > hard links to an existing target file. > >=20 > > I don't think that's possible as holes can only be created at the end= of a file. > > Well I think NTFS supports punching holes in the "middle" but it's no= t common. =20 I was not advocating support for punching hole in existing files (though this is what I want to do in the end). I was only interested in an option that would create sparse output files = by seeking forward in the output file whenever zero bytes in the input strea= m ocurr. This is completely independent of the filesystem underneath. It should even work if the FS doesn't support holes at all, as long as the=20 interface is POSIX compliant. > I believe there's demand for supporting punching holes in the middle > of files and it will eventually show up in Linux. For instance, the > combination of IDE TRIM support and virtualization can allow virtualize= d > guest to take as less disk space as possible in file-backed virtual > disks. True. "Thin Provisioning", that is the usage of disk images that start ou= t by allocating only the used disk blocks and growing on demand, suffers fr= om the inability of guest systems to "give back" unused blocks whenever a bl= ock is released. There are ways around it, but none of them work very well wi= th=20 a life guest. The question there is wether this is a task for "dd" or for a specialized tool. -henrik From debbugs-submit-bounces@debbugs.gnu.org Tue Apr 13 08:15:10 2010 Received: (at submit) by debbugs.gnu.org; 13 Apr 2010 12:15:10 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1O1f1J-0002q6-Vu for submit@debbugs.gnu.org; Tue, 13 Apr 2010 08:15:10 -0400 Received: from mx10.gnu.org ([199.232.76.166]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1O1f1I-0002q1-Jg for submit@debbugs.gnu.org; Tue, 13 Apr 2010 08:15:09 -0400 Received: from lists.gnu.org ([199.232.76.165]:40254) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1O1f1D-0000Ym-3v for submit@debbugs.gnu.org; Tue, 13 Apr 2010 08:15:03 -0400 Received: from mailman by lists.gnu.org with tmda-scanned (Exim 4.43) id 1O1f1C-0000X8-JL for bug-coreutils@gnu.org; Tue, 13 Apr 2010 08:15:02 -0400 Received: from [140.186.70.92] (port=43845 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1O1f1A-0000Wv-Gb for bug-coreutils@gnu.org; Tue, 13 Apr 2010 08:15:01 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.0 (2010-01-18) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00 autolearn=unavailable version=3.3.0 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1O1f18-0008A4-OH for bug-coreutils@gnu.org; Tue, 13 Apr 2010 08:15:00 -0400 Received: from www.viadmin.org ([195.145.128.101]:56406) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1O1f18-00089k-HY for bug-coreutils@gnu.org; Tue, 13 Apr 2010 08:14:58 -0400 Received: from localhost (localhost.localdomain [127.0.0.1]) by www.viadmin.org (Postfix) with ESMTP id 702BE2E7C9 for ; Tue, 13 Apr 2010 14:14:56 +0200 (CEST) X-Virus-Scanned: Debian amavisd-new at www.viadmin.org Received: from www.viadmin.org ([127.0.0.1]) by localhost (www.viadmin.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id fhttm0E70aD2 for ; Tue, 13 Apr 2010 14:14:48 +0200 (CEST) Received: by www.viadmin.org (Postfix, from userid 1005) id AA37E15352; Tue, 13 Apr 2010 14:14:48 +0200 (CEST) Date: Tue, 13 Apr 2010 14:14:48 +0200 From: "H. Langos" To: bug-coreutils@gnu.org Subject: Re: bug#5918: [dd] conv=sparse option Message-ID: <20100413121448.GD17172@www.viadmin.org> References: <20071229224147.GA5914@implementation> <20100410002857.GB17172@www.viadmin.org> <4BC09A33.1060802@draigBrady.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline In-Reply-To: <4BC09A33.1060802@draigBrady.com> X-Archive: encrypt User-Agent: Mutt/1.5.18 (2008-05-17) Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -5.9 (-----) X-Debbugs-Envelope-To: submit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -6.1 (------) On Sat, Apr 10, 2010 at 04:33:07PM +0100, P=E1draig Brady wrote: > On 10/04/10 01:28, Heinrich Langos wrote: > > Hello Andreas, Samuel and list, > >=20 > > sorry to pick up such an old thread, but I stumbled upon it while > > looking for an efficient way to "re-sparse" files that contain a=20 > > lot of zero blocks but=20 > > 1) had already been expanded=20 > > or=20 > > 2) are being expanded due to pipes. > >=20 > > On Sun, Dec 30, 2007 at 10:19:54AM +0100, Andreas Schwab wrote: > >> Samuel Thibault writes: > >> > >>> Some time ago, I wrote a conv=3Dsparse option for dd, attached is t= he > >>> patch. > >> > >> How is it different from cp --sparse=3Dalways? > >=20 > > I'd say in enough ways to make such an option highly desirable. > >=20 > > a) "dd" will maintain an existing of=3Dtarget file including the inod= e=20 > > number, thus respecting existing hard links. "cp" will depending=20 > > on the other options given (e.g. "-a") maintain or break existing=20 > > hard links to an existing target file. >=20 > I don't think that's possible as holes can only be created at the end o= f a file. > Well I think NTFS supports punching holes in the "middle" but it's not = common. >=20 > >=20 > > b) "dd" could read a stream from a device or stdin and write it direc= tly=20 > > to a sparse file. no need to "dd" from e.g. a block device to a fi= le and=20 > > afterwards do a "cp --sparse=3Dalways file sparse-file". this will= save a=20 > > lot of disk space, io operations and time. >=20 > This seems to work: > cp --sparse=3Dalways /dev/stdin file Yeap. That worked! > hlangos@pc-hlangos:~/zaurus$ ls -lisa foo > 958477 4 -rw-r--r-- 1 hlangos hlangos 3072 2010-04-13 12:12 foo > hlangos@pc-hlangos:~/zaurus$ dd if=3D/dev/zero bs=3D1k count=3D100 | c= p --sparse=3Dalways /dev/stdin foo > 100+0 records in > 100+0 records out > 102400 bytes (102 kB) copied, 0.0802346 s, 1.3 MB/s > hlangos@pc-hlangos:~/zaurus$ ls -lisa foo > 958477 0 -rw-r--r-- 1 hlangos hlangos 102400 2010-04-13 14:06 foo It doesn't change the target file's inode (and also maintains the existin= g hard links). Cheers -henrik From debbugs-submit-bounces@debbugs.gnu.org Mon Nov 22 11:15:17 2010 Received: (at 5918) by debbugs.gnu.org; 22 Nov 2010 16:15:18 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1PKZ2z-00076h-MI for submit@debbugs.gnu.org; Mon, 22 Nov 2010 11:15:17 -0500 Received: from mail1.slb.deg.dub.stisp.net ([84.203.253.98]) by debbugs.gnu.org with smtp (Exim 4.69) (envelope-from ) id 1PKZ2x-00076b-1C for 5918@debbugs.gnu.org; Mon, 22 Nov 2010 11:15:15 -0500 Received: (qmail 59030 invoked from network); 22 Nov 2010 16:20:29 -0000 Received: from unknown (HELO ?192.168.2.25?) (84.203.137.218) by mail1.slb.deg.dub.stisp.net with SMTP; 22 Nov 2010 16:20:29 -0000 Message-ID: <4CEA980D.4090103@draigBrady.com> Date: Mon, 22 Nov 2010 16:19:25 +0000 From: =?UTF-8?B?UMOhZHJhaWcgQnJhZHk=?= User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.8) Gecko/20100227 Thunderbird/3.0.3 MIME-Version: 1.0 To: Jim Meyering Subject: Re: bug#5918: [dd] conv=sparse option References: <20071229224147.GA5914@implementation> <20100410002857.GB17172@www.viadmin.org> <4BC09A33.1060802@draigBrady.com> <87vdbyrsu8.fsf@meyering.net> In-Reply-To: <87vdbyrsu8.fsf@meyering.net> X-Enigmail-Version: 1.0.1 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Spam-Score: -2.7 (--) X-Debbugs-Envelope-To: 5918 Cc: Andreas Schwab , 5918@debbugs.gnu.org, samuel.thibault@ens-lyon.org, Heinrich Langos X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -2.7 (--) On 11/04/10 15:01, Jim Meyering wrote: > Pádraig Brady wrote: >> On 10/04/10 01:28, Heinrich Langos wrote: >>> Hello Andreas, Samuel and list, >>> >>> sorry to pick up such an old thread, but I stumbled upon it while >>> looking for an efficient way to "re-sparse" files that contain a >>> lot of zero blocks but >>> 1) had already been expanded >>> or >>> 2) are being expanded due to pipes. >>> >>> On Sun, Dec 30, 2007 at 10:19:54AM +0100, Andreas Schwab wrote: >>>> Samuel Thibault writes: >>>> >>>>> Some time ago, I wrote a conv=sparse option for dd, attached is the >>>>> patch. >>>> >>>> How is it different from cp --sparse=always? >>> >>> I'd say in enough ways to make such an option highly desirable. >>> >>> a) "dd" will maintain an existing of=target file including the inode >>> number, thus respecting existing hard links. "cp" will depending >>> on the other options given (e.g. "-a") maintain or break existing >>> hard links to an existing target file. >> >> I don't think that's possible as holes can only be created at the end of a file. >> Well I think NTFS supports punching holes in the "middle" but it's not common. > > I would like at least cp to be able to copy sparse files efficiently, > and considering the FIEMAP patches that Jeff Liu is working on, we > don't have long to wait. > > BTW, I'm pretty sure it is possible to punch a hole in the middle of > a file with XFS. Maybe with other CoW file systems, too? They're just now adding FALLOC_FL_PUNCH_HOLE to fallocate() in the Linux kernel. It's supported by xfs and ocfs2, and other filesystems will for now, return EOPNOTSUPP fallocate(..,FALLOC_FL_PUNCH_HOLE) will return EOPNOTSUPP on older kernels even for xfs and ocfs2. cheers, Pádraig. From debbugs-submit-bounces@debbugs.gnu.org Wed Oct 10 12:44:25 2018 Received: (at 5918) by debbugs.gnu.org; 10 Oct 2018 16:44:25 +0000 Received: from localhost ([127.0.0.1]:43775 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gAHaz-0005zp-IZ for submit@debbugs.gnu.org; Wed, 10 Oct 2018 12:44:25 -0400 Received: from mail-pf1-f175.google.com ([209.85.210.175]:39339) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gAHax-0005zV-M7; Wed, 10 Oct 2018 12:44:24 -0400 Received: by mail-pf1-f175.google.com with SMTP id c25-v6so2917070pfe.6; Wed, 10 Oct 2018 09:44:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-language:content-transfer-encoding; bh=veyIZ5WmYCNfx5q6gLG4YCstLp3wA57Pb+Dw8HsMPVY=; b=j0hgRorZdLRYv1/tWy6y048bD/qzkns3n649tcFHtRIsp+xKyRLLu6KOw144NU/Uoi 0VMDEbcrqv8eC5LYTsDgKNyIrPqz35nNIrqk6VHvcxAIYGqFu4BaMSZMA5zog90kEhEt Eminf2P2ReXpzHn7QagFbXT/qJsSCIQk5o7SFh5ktWPHiATEVGBjadgeHRwR00egJwpa vIq03yVqHKiKz/rVs/eMfvG1s03ayPc/nEqAHuxzlc+dTNN1GFFPzfA1xCm/meWM0U7a 2bS0L9JZPZADCl0ytyC/11Cc54rSUkh1MR1Dujn94wyap6JY4znNB66TDP4WOrD1Qv4f NYdw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=veyIZ5WmYCNfx5q6gLG4YCstLp3wA57Pb+Dw8HsMPVY=; b=YnjpJglSUYmzNAX5WCVUXlCGoKCV7hFOWLmU8JwQ/EauAM38u7IqG6+vZtPp6cjuVY je0zOS6azs+qIO0D3oyhu3knxG42zlpfYsW0JRhO9GXRDfaZsBMby6muM05OUn086d28 4eW5kPtxoTMCs8wVyMwRay14F/iPr07GhFkgkmyEfXN/rI/3O+7nszbJ+k9CmtAL1LM3 IZAiHrHj+niHY8WbbD5Xw8GcNrkBIKUaEpnjOqvIodnJl/ofVqsnUYZd4JfpTMvRrr53 6jRivtTXXhOIUhVbPRUXcsqz4sBEshALL+HHxPTpuEIrpk8zRbqF/hWiIZPyAtaltHiI DUxA== X-Gm-Message-State: ABuFfoikNvGGBwoFoQ50oOe6GYarG1oFfUqke2PrVOLBnstv+ZOxTDHK goTIpGNdIJ8zjG+QNg1/QjGXZ2vejfk= X-Google-Smtp-Source: ACcGV60Z8mpsrd5bPnkq4GAdd2ED7pZM+LCbBAdlW4R+sno3cbxI4I7PjqYlgJ25zdp0EIQ616om3A== X-Received: by 2002:a65:6409:: with SMTP id a9-v6mr30643386pgv.204.1539189857162; Wed, 10 Oct 2018 09:44:17 -0700 (PDT) Received: from tomato.housegordon.com (moose.housegordon.com. [184.68.105.38]) by smtp.googlemail.com with ESMTPSA id f22-v6sm35772534pff.29.2018.10.10.09.44.15 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 10 Oct 2018 09:44:15 -0700 (PDT) Subject: Re: bug#5918: [dd] conv=sparse option To: 5918@debbugs.gnu.org References: <20071229224147.GA5914@implementation> <20100410002857.GB17172@www.viadmin.org> <4BC09A33.1060802@draigBrady.com> <87vdbyrsu8.fsf@meyering.net> <4CEA980D.4090103@draigBrady.com> From: Assaf Gordon Message-ID: Date: Wed, 10 Oct 2018 10:44:14 -0600 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.9.1 MIME-Version: 1.0 In-Reply-To: <4CEA980D.4090103@draigBrady.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 5918 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) tags 5918 fixed close 5918 stop Hello, Coreutils version 8.16 (released 2012) gained "dd conv=sparse" option, see https://git.savannah.gnu.org/cgit/coreutils.git/commit/?id=4e776faa8482ae630d2ea9bc767298e664f07ba9 closing this bug. regards, - assaf From unknown Mon Jun 23 04:14:40 2025 Received: (at fakecontrol) by fakecontrolmessage; To: internal_control@debbugs.gnu.org From: Debbugs Internal Request Subject: Internal Control Message-Id: bug archived. Date: Thu, 08 Nov 2018 12:24:05 +0000 User-Agent: Fakemail v42.6.9 # This is a fake control message. # # The action: # bug archived. thanks # This fakemail brought to you by your local debbugs # administrator