From debbugs-submit-bounces@debbugs.gnu.org Wed Dec 15 15:41:00 2010 Received: (at submit) by debbugs.gnu.org; 15 Dec 2010 20:41:00 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1PSy9j-0000Pv-Qo for submit@debbugs.gnu.org; Wed, 15 Dec 2010 15:41:00 -0500 Received: from eggs.gnu.org ([140.186.70.92]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1PSy3j-0000GD-Cg for submit@debbugs.gnu.org; Wed, 15 Dec 2010 15:34:48 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1PSy9n-0006ev-BH for submit@debbugs.gnu.org; Wed, 15 Dec 2010 15:41:04 -0500 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,T_RP_MATCHES_RCVD autolearn=unavailable version=3.3.1 Received: from lists.gnu.org ([199.232.76.165]:51672) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1PSy9n-0006er-9Y for submit@debbugs.gnu.org; Wed, 15 Dec 2010 15:41:03 -0500 Received: from [140.186.70.92] (port=58401 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1PSy9m-0002mN-Cu for bug-coreutils@gnu.org; Wed, 15 Dec 2010 15:41:03 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1PSy9l-0006ed-C1 for bug-coreutils@gnu.org; Wed, 15 Dec 2010 15:41:02 -0500 Received: from sigmagames.com ([72.9.230.138]:48913) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1PSy9l-0006eL-9W for bug-coreutils@gnu.org; Wed, 15 Dec 2010 15:41:01 -0500 Received: by sigmagames.com (Postfix, from userid 600) id DF6CC22128; Wed, 15 Dec 2010 16:01:33 -0500 (EST) Date: Wed, 15 Dec 2010 16:01:33 -0500 To: bug-coreutils@gnu.org Subject: sparse files and commands User-Agent: Heirloom mailx 12.4 7/29/08 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-Id: <20101215210133.DF6CC22128@sigmagames.com> From: support@sigmagames.com X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 2) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 2) X-Spam-Score: -6.6 (------) X-Debbugs-Envelope-To: submit X-Mailman-Approved-At: Wed, 15 Dec 2010 15:40:58 -0500 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -6.6 (------) Hi, Just an idea: Am working with sparse text (ascii) files and need command line options. For instance, cat could have --sparse=text meaning 2 things: 1) it is a sparse file and so skip sparse recs. 2) it is text and so if(byte==0) skip it, it is sure to work. maybe do this for all commands. Thanks Charles Carter, President Sigma Software, Inc. From debbugs-submit-bounces@debbugs.gnu.org Wed Dec 15 16:09:53 2010 Received: (at 7649) by debbugs.gnu.org; 15 Dec 2010 21:09:53 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1PSybh-00016U-Dw for submit@debbugs.gnu.org; Wed, 15 Dec 2010 16:09:53 -0500 Received: from mx1.redhat.com ([209.132.183.28]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1PSybd-00016H-Cx for 7649@debbugs.gnu.org; Wed, 15 Dec 2010 16:09:50 -0500 Received: from int-mx02.intmail.prod.int.phx2.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) by mx1.redhat.com (8.13.8/8.13.8) with ESMTP id oBFLG5Vj015137 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Wed, 15 Dec 2010 16:16:06 -0500 Received: from [10.3.113.37] (ovpn-113-37.phx2.redhat.com [10.3.113.37]) by int-mx02.intmail.prod.int.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id oBFLG5CZ029187; Wed, 15 Dec 2010 16:16:05 -0500 Message-ID: <4D093014.4040506@redhat.com> Date: Wed, 15 Dec 2010 14:16:04 -0700 From: Eric Blake Organization: Red Hat User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.13) Gecko/20101209 Fedora/3.1.7-0.35.b3pre.fc14 Lightning/1.0b3pre Mnenhy/0.8.3 Thunderbird/3.1.7 MIME-Version: 1.0 To: support@sigmagames.com Subject: Re: bug#7649: sparse files and commands References: <20101215210133.DF6CC22128@sigmagames.com> In-Reply-To: <20101215210133.DF6CC22128@sigmagames.com> X-Enigmail-Version: 1.1.2 OpenPGP: url=http://people.redhat.com/eblake/eblake.gpg Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="------------enig33C0EF4892B28B4E931FFA5C" X-Scanned-By: MIMEDefang 2.67 on 10.5.11.12 X-Spam-Score: -10.2 (----------) X-Debbugs-Envelope-To: 7649 Cc: 7649@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -10.2 (----------) This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig33C0EF4892B28B4E931FFA5C Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable On 12/15/2010 02:01 PM, support@sigmagames.com wrote: > Hi,=20 > Just an idea: > Am working with sparse text (ascii) files and need command line options= =2E > For instance, cat could have --sparse=3Dtext > meaning 2 things: > 1) it is a sparse file and so skip sparse recs. > 2) it is text and so if(byte=3D=3D0) skip it, it is sure to work. > maybe do this for all commands. Rather than modify lots of commands to add a new option, why not use existing features of existing commands to do what you need? tr -d '\0' is a great way to remove NUL bytes (including sparse blocks) from a file. Then, for everywhere you want to use a sparse file but ignore the NUL bytes, you instead just pass a pipe with tr doing the work for you. cmp <(tr -d '\0' < file) otherfile Meanwhile, I can certainly see a case for teaching tr(1) to optimize deletion of just the NUL byte in order to be more efficient on sparse files. In fact, we are already in the middle of optimizing cp based on sparseness, and have plans to optimize other tools like cmp and tar to reuse the framework developed for cp to detect sparse blocks where the system supports such information, so tr is an easy tool to add to that list of potential beneficiaries. That way, what you want to do works now (albeit not with optimal speed) and might improve in the future (if we teach tr to special case NUL elision); and we don't have to spend time retrofitting a new option onto scores of programs (where you would have to wait for the new coreutils to propagate to the machines you use). --=20 Eric Blake eblake@redhat.com +1-801-349-2682 Libvirt virtualization library http://libvirt.org --------------enig33C0EF4892B28B4E931FFA5C Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Public key at http://people.redhat.com/eblake/eblake.gpg Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/ iQEcBAEBCAAGBQJNCTAUAAoJEKeha0olJ0NqzeQH/3j787SUZDAeoJ2XQM27gHiZ 8g3T+zgzD02wPuQ0IsWRauDTKrhTX77IUoFfSH7Vy3b5V2hAmh6g153FXPvtSukx hT17/OY+V7qF9LBN9noGHVh2ox3/OfhnsMl+I/erqHj7QsJUC9nxA+/clEIQLzF3 3mexWCbpjeVeXhLk3C51kM9ZnIV7nsky4ROoC6mQu8NwvgEFwNcZjyNk1mrwzw9i HLQwF6XfN1cPq/BVAw0g/zZIf4KT1iDPC4560uPRLTGxt/qX/mA+Xgl0S1WymTvN dyUksvLHPg/RFf/Cix1jlns6sXRcLl6HXSPqMMs5MGeD71svCb2Dqi/Vhtim1bk= =aZw5 -----END PGP SIGNATURE----- --------------enig33C0EF4892B28B4E931FFA5C-- From debbugs-submit-bounces@debbugs.gnu.org Wed Dec 15 17:06:02 2010 Received: (at 7649) by debbugs.gnu.org; 15 Dec 2010 22:06:02 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1PSzU2-0002Jt-B3 for submit@debbugs.gnu.org; Wed, 15 Dec 2010 17:06:02 -0500 Received: from mx1.redhat.com ([209.132.183.28]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1PSzU0-0002Ja-Cm for 7649@debbugs.gnu.org; Wed, 15 Dec 2010 17:06:01 -0500 Received: from int-mx10.intmail.prod.int.phx2.redhat.com (int-mx10.intmail.prod.int.phx2.redhat.com [10.5.11.23]) by mx1.redhat.com (8.13.8/8.13.8) with ESMTP id oBFMCGgs026879 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Wed, 15 Dec 2010 17:12:16 -0500 Received: from [10.3.113.37] (ovpn-113-37.phx2.redhat.com [10.3.113.37]) by int-mx10.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id oBFMCFiI030782; Wed, 15 Dec 2010 17:12:16 -0500 Message-ID: <4D093D3F.2030402@redhat.com> Date: Wed, 15 Dec 2010 15:12:15 -0700 From: Eric Blake Organization: Red Hat User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.13) Gecko/20101209 Fedora/3.1.7-0.35.b3pre.fc14 Lightning/1.0b3pre Mnenhy/0.8.3 Thunderbird/3.1.7 MIME-Version: 1.0 To: support@sigmagames.com, 7649@debbugs.gnu.org Subject: Re: bug#7649: sparse files and commands References: <20101215210133.DF6CC22128@sigmagames.com> <4D093014.4040506@redhat.com> <20101215222031.C262222128@sigmagames.com> In-Reply-To: <20101215222031.C262222128@sigmagames.com> X-Enigmail-Version: 1.1.2 OpenPGP: url=http://people.redhat.com/eblake/eblake.gpg Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="------------enig18D0FEAC863D7C1653503007" X-Scanned-By: MIMEDefang 2.68 on 10.5.11.23 X-Spam-Score: -10.2 (----------) X-Debbugs-Envelope-To: 7649 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -10.2 (----------) This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig18D0FEAC863D7C1653503007 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable [please keep the list in the loop, and don't top-post on technical lists]= On 12/15/2010 03:20 PM, support@sigmagames.com wrote: > THANKS, I see what you mean. > Also food for thought, IF OS supports it, skip file blocks that are all= null in a sparse file. > This way less IO and less processing. That's what I was alluding to - we are working on adding fiemap support here: http://git.savannah.gnu.org/cgit/coreutils.git/log/?h=3Dfiemap-copy as well as a proposal for a hole iterator on OSs that support it (Linux supports it for ext4 and btrfs via fiemap ioctl; Solaris supports it via lseek(,SEEK_HOLE) http://lists.gnu.org/archive/html/bug-coreutils/2010-07/msg00117.html Note that an explicit block of all 0s cannot be efficiently skipped; this only works for sparse files (where the block is completely represented in the metadata of the file, and is implicitly all 0s). Also, there has been movement to get Linux to add support for punching holes into an existing file (right now, the only way to make a file more sparse is to create a new file with the same contents while leaving holes where the source had all-0 blocks, but that's obviously not as efficient as modifying a file in-place to add a hole). Once hole iteration is implemented (and right now, cp is our guinea pig), then we can port that effort to a number of other programs (tr, cmp, tar, ...) to make for more efficient I/O on files where we can behave differently if we know that a block is sparse. --=20 Eric Blake eblake@redhat.com +1-801-349-2682 Libvirt virtualization library http://libvirt.org --------------enig18D0FEAC863D7C1653503007 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Public key at http://people.redhat.com/eblake/eblake.gpg Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/ iQEcBAEBCAAGBQJNCT0/AAoJEKeha0olJ0NqzgUH/1OO8BqcbmntjwYxsuZqiJtV 7aGs/bvHIYRPVVy4j9SkvBcf4zI3ALyjP7eqPJUgY5GvI68fLR2dalRUOwWAnFRR 1BEVboaWN8Yq/JqW9xPlxedev0CYfnapRln8oSsvjQOYmdlINrjIoEkBW+jd2rPX jrZ1AZGEtNjzzqzPV3gLfqXRAw32v8jf2MM1zzgcJAF8g6MW4mp+Rdk/kF2Nt8FK N4rbITxCNEgMEoCRIb0YR9LYnu3Tc7WMpyY6iaDbpDlZ9nP24Qht2JRtYNgoVAJ7 hEBCCWbGGMpLaq+zpPjlYLL5xBvteCsCJyY7pTcZqbwq5OFfzXwH5fXUJMLSxRU= =TFdh -----END PGP SIGNATURE----- --------------enig18D0FEAC863D7C1653503007-- From debbugs-submit-bounces@debbugs.gnu.org Tue Oct 30 03:55:49 2018 Received: (at 7649) by debbugs.gnu.org; 30 Oct 2018 07:55:50 +0000 Received: from localhost ([127.0.0.1]:53203 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gHOsP-0005ez-NK for submit@debbugs.gnu.org; Tue, 30 Oct 2018 03:55:49 -0400 Received: from mail-io1-f54.google.com ([209.85.166.54]:35186) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gHOsN-0005ef-Nd; Tue, 30 Oct 2018 03:55:47 -0400 Received: by mail-io1-f54.google.com with SMTP id 79-v6so6681147iou.2; Tue, 30 Oct 2018 00:55:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=to:from:subject:message-id:date:user-agent:mime-version :content-language:content-transfer-encoding; bh=OF/MJ08n87hKwBrALEBKYfQ6pWb/SGi1OXsA3lCe8Tw=; b=mE5Xg83Aa+FGX823ivNACCaeyVbV8aUAyyldJ0H6uOe6f8/S7flpkvdTy5EGPizcPD OX7ulpwDCuRi6seqFLnm+0Adcb+849uQllM2ueo4bgO2EBXyPWPr7mSAgNDcZ+vzoMKj Or0QeNNeJBgZGIv3F7RBgXlU9FI+ZTE67CSIwbr6jHwdF7nFWMZB7gBjS+MniHSK/z2j kaN0Pykkw59QIj8xqHTtK7aU6K3b4V1mT0rhHNQQxOXS6i6u0sq9QRk+g1frEsPrsXW8 o2z9y+dYKCpwN06DyHe5dWALeK7h9js06EqknqVXDIZswOwN5yUzgd/rQf2+EDs5atk8 MvEA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:to:from:subject:message-id:date:user-agent :mime-version:content-language:content-transfer-encoding; bh=OF/MJ08n87hKwBrALEBKYfQ6pWb/SGi1OXsA3lCe8Tw=; b=jDzM+cUE55nYO9h1ytgrJMj8G/tWkzNb4NwXzF3szRxy/Rke95tz+YOUNRbMNr/ElC W/VJkUwu2UXqmppltN+RVlGqXvnvneUB2ly/zlRRfAi9Ly1xZlWnokL1TBjISvbCDSqu btJsFtJFNHu68u9/GN2wRaFNQjLOiwBUCWZxGfoeYmTdEpf1ix+g1CGAyxLEioQxg6a7 Ii+NwbQgAM/TKDOhq+YzZvj6Kk1NyUYE8joHmDuWn651h/n/9aRTyeNf9RE4FBVmAJt2 fPkvhpJ0uypY0MVDiotnXQNxpatQee2bKBNJJYt0kkrjZIYsnY+PM/gjZN6EwjybWfQY Ao+g== X-Gm-Message-State: AGRZ1gLB/Iz4xvn4IFtavLbHzekm77YkqasPd0UhJquabvUr7CTumRhe SL7eNQsGt5vZvHJ2UFCB/2B52H3lKbg= X-Google-Smtp-Source: AJdET5exFCrtJeXyfyrITX5piqT3MkYDjMivwFAqRRZlvvr/4uIa7x97Fd+Y5URyWvn+/RcA9PRk8g== X-Received: by 2002:a6b:f409:: with SMTP id i9-v6mr10636106iog.278.1540886141518; Tue, 30 Oct 2018 00:55:41 -0700 (PDT) Received: from tomato.housegordon.com (moose.housegordon.com. [184.68.105.38]) by smtp.googlemail.com with ESMTPSA id q205-v6sm8486611itc.2.2018.10.30.00.55.39 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 30 Oct 2018 00:55:40 -0700 (PDT) To: 7649@debbugs.gnu.org From: Assaf Gordon Subject: sparse files and commands Message-ID: <8ed3ffeb-685c-b6c2-be5b-bad5deb1b9af@gmail.com> Date: Tue, 30 Oct 2018 01:55:38 -0600 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.2.1 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 7649 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) close 7649 stop (triaging old bugs) With no follow-ups in 7 years, I'm closing this bug. -assaf From unknown Sun Jun 22 22:45:58 2025 Received: (at fakecontrol) by fakecontrolmessage; To: internal_control@debbugs.gnu.org From: Debbugs Internal Request Subject: Internal Control Message-Id: bug archived. Date: Tue, 27 Nov 2018 12:24:08 +0000 User-Agent: Fakemail v42.6.9 # This is a fake control message. # # The action: # bug archived. thanks # This fakemail brought to you by your local debbugs # administrator