From debbugs-submit-bounces@debbugs.gnu.org Sat Mar 26 16:29:49 2022 Received: (at submit) by debbugs.gnu.org; 26 Mar 2022 20:29:49 +0000 Received: from localhost ([127.0.0.1]:54742 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nYD2i-0003k8-Od for submit@debbugs.gnu.org; Sat, 26 Mar 2022 16:29:48 -0400 Received: from lists.gnu.org ([209.51.188.17]:44836) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nYD2h-0003k0-Bb for submit@debbugs.gnu.org; Sat, 26 Mar 2022 16:29:47 -0400 Received: from eggs.gnu.org ([209.51.188.92]:44292) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nYD2h-0001zX-3d for bug-coreutils@gnu.org; Sat, 26 Mar 2022 16:29:47 -0400 Received: from freefriends.org ([96.88.95.60]:38296) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nYD2f-0007L3-0D for bug-coreutils@gnu.org; Sat, 26 Mar 2022 16:29:46 -0400 X-Envelope-From: karl@freefriends.org X-Envelope-To: Received: from freefriends.org (freefriends.org [96.88.95.60]) by freefriends.org (8.14.7/8.14.7) with ESMTP id 22QKTgQd018159 (version=TLSv1/SSLv3 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Sat, 26 Mar 2022 14:29:43 -0600 Received: (from apache@localhost) by freefriends.org (8.14.7/8.14.7/Submit) id 22QKTg6b018158; Sat, 26 Mar 2022 14:29:42 -0600 Date: Sat, 26 Mar 2022 14:29:42 -0600 Message-Id: <202203262029.22QKTg6b018158@freefriends.org> From: Karl Berry To: bug-coreutils@gnu.org Subject: dd conv options doc Received-SPF: pass client-ip=96.88.95.60; envelope-from=karl@freefriends.org; helo=freefriends.org X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-Spam-Score: -1.4 (-) X-Debbugs-Envelope-To: submit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.4 (--) The dd Texinfo doc says, for the conv= option (https://gnu.org/s/coreutils/manual/html_node/dd-invocation.html) 'fdatasync' Synchronize output data just before finishing. This forces a physical write of output data. 'fsync' Synchronize output data and metadata just before finishing. This forces a physical write of output data and metadata. Weirdly, these descriptions are inducing quite a bit of FUD in me. Why would I ever want the writes to be incomplete after running dd? Seems like that is dd's whole purpose. Well, I suppose it is too late to make such a radical change as forcing a final sync. In which case I suggest adding another sentence along the lines of "If these options are not specified, the data will be physically written when the system schedules the syncs, ordinarily every few seconds" (correct?). "You can also manually sync the output filesystem yourself afterwards (xref sync)." Otherwise it feels uncertain when or whether the data will be physically written, or how to look into it further. As for "metadata", what does dd have to do with metadata? My wild guess is that this is referring to filesystem metadata, not anything about dd specifically. Whatever the case, I suggest adding a word or two to the doc to give a clue. Further, why would I want data to be synced and not metadata? Seems like fdatasync and fsync should both do both; or at least document that normally they'd be used together. Or, if there is a real-life case where a user would want one and not the other, how about documenting that? My imagination is failing me, but presumably these seemingly-undesirable options were invented for a reason. BTW, I came across these options on a random page discussing dumping a .iso to a USB drive; the example was dd if=foo.iso of=/dev/sde conv=fdatasync .. seems now like fsync should also have been given, for certainty. --thanks, karl. From debbugs-submit-bounces@debbugs.gnu.org Mon Apr 04 16:24:33 2022 Received: (at 54586) by debbugs.gnu.org; 4 Apr 2022 20:24:33 +0000 Received: from localhost ([127.0.0.1]:53159 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nbTFY-0008ID-Rg for submit@debbugs.gnu.org; Mon, 04 Apr 2022 16:24:33 -0400 Received: from havoc.proulx.com ([96.88.95.61]:57548) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nbTFX-0008Hz-II for 54586@debbugs.gnu.org; Mon, 04 Apr 2022 16:24:32 -0400 Received: from joseki.proulx.com (localhost [127.0.0.1]) by havoc.proulx.com (Postfix) with ESMTP id E60AC88; Mon, 4 Apr 2022 14:24:25 -0600 (MDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=proulx.com; s=dkim2048; t=1649103865; bh=LzLPQq2U2SsPgYxhR9+HK5BInSN1sQUGNGtwjA8OMOs=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=CFWySiQ0/2heHA24HMVYKIosrUkdB/ilSl3AK6njZOpT6Rrv6FFf75CYzy+X3CJ0R EV/IOlz34Swf6Q3vnpe0E5y16xAuL6UzxPp/UODa1O7AaCc/+8qk7X7w3bcxN1zCHe QpmaterYyRQP3ZBFIUGYiX4ixvM1vYu+eICbXO4uExDoqkTTGyOrau4KH3OepPFy9a koolCnWbwqEp1s/KjY/7qN0VeENsv2Ir3nCDDLe1JuFynv6b3h72blBjI4t7Eus4dz e2ZvaxT9QDn4PBQLBJxjiRA/N9lbyGek93vEimCO9zvqo5Bc7plkoUFVxtzSFiiwSy LmkTs4XhAt98A== Received: from hysteria.proulx.com (hysteria.proulx.com [192.168.230.119]) by joseki.proulx.com (Postfix) with ESMTP id B9ED496702; Mon, 4 Apr 2022 14:24:25 -0600 (MDT) Received: by hysteria.proulx.com (Postfix, from userid 1000) id 997352DCA5; Mon, 4 Apr 2022 14:24:25 -0600 (MDT) Date: Mon, 4 Apr 2022 14:24:25 -0600 From: Bob Proulx To: Karl Berry Subject: Re: bug#54586: dd conv options doc Message-ID: <20220403180849190043957@bob.proulx.com> References: <202203262029.22QKTg6b018158@freefriends.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <202203262029.22QKTg6b018158@freefriends.org> X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 54586 Cc: 54586@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Karl Berry wrote: > 'fdatasync' > Synchronize output data just before finishing. This forces a > physical write of output data. > > 'fsync' > Synchronize output data and metadata just before finishing. > This forces a physical write of output data and metadata. > > Weirdly, these descriptions are inducing quite a bit of FUD in me. > > Why would I ever want the writes to be incomplete after running dd? > Seems like that is dd's whole purpose. Yes. FUD. The writes are not incomplete. It is no different than any other write. echo "Hello, World!" > file1 Is that write complete? It's no different. If one is incomplete then so is the other. Note that the documentation does not say "incomplete" but says "physical write". As in, chiseled into stone. The dd utility exists with a plethora of low level options not typically available in other utilities. Other utilities such as cp for example. That is one of the distinguishing features making dd useful in a very large number of cases when otherwise we would use cp, rsync, or one of the others. Very low level control of option flags. But just because options exist does not mean they should always be used. Most of the time they should not be used. > Well, I suppose it is too late to make such a radical change as forcing > a final sync. Please, no. Opposing this is the motivation for me writing this response. Things are wastefully slow already due to the number of fsync() calls now coded into everywhere all over the place. Other programs. Not referring to the coreutils here. Let's not make the problem worse by adding them where they are not desired. And that is why it is an option to dd and not on by default. In those specific cases where it is useful then it can be specified as an option. dd is exposing the interface for when it is useful. As a practical matter I think with GNU dd's extensions that I never ever use conv=fsync or conv=fdatasync but instead would always in those same cases use oflag=direct,sync. Such as when writing a removable storage device like a USB drive, that I subsequently will want to remove. There is no benefit to caching the data since it will be invalidated immediately. Not using buffer cache avoids flushing some other data that would be useful to keep in file system buffer cache. When the write is done then the removable media can be removed. This avoids needing to run sync explicitly. Which sync's *everything*. > In which case I suggest adding another sentence along the lines of > "If these options are not specified, the data will be physically > written when the system schedules the syncs, ordinarily every few > seconds" (correct?). Yes. However the behavior might vary slightly between the different kernels such as Linux kernel, BSD kernel, or even HP-UX kernel. Therefore the documentation of it is kernel specific. Even if all of the kernels operated similarly. > "You can also manually sync the output filesystem yourself > afterwards (xref sync)." Otherwise it feels uncertain when or > whether the data will be physically written, or how to look into it > further. Generally this is a task that the operating system should be handling. The programmer taking explicit control defeating the cache is almost always going to be less efficient at it than the operating system. However as you later mention writing an image to a removable storage device like a USB thumbdrive needs to have the data flushed through before removing the device. GNU dd is good for this as I will describe below but otherwise yes a "sync" (either the standalone or the oflag) would be needed to ensure that the data has been flushed through. > As for "metadata", what does dd have to do with metadata? My wild guess > is that this is referring to filesystem metadata, not anything about dd > specifically. Whatever the case, I suggest adding a word or two to the > doc to give a clue. It's not dd's fault. The OS created it first! It's a property given meaning by the OS. The OS defines the option flags. The dd utility is simply a thin layer giving access to the OS file option flags. > Further, why would I want data to be synced and not metadata? Seems like > fdatasync and fsync should both do both; or at least document that > normally they'd be used together. Or, if there is a real-life case where > a user would want one and not the other, how about documenting that? My > imagination is failing me, but presumably these seemingly-undesirable > options were invented for a reason. The fdatasync() man page provides the information. The aim of fdatasync() is to reduce disk activity for applications that do not require all metadata to be synchronized with the disk. In short fdatasync() is less heavy than fsync(). > BTW, I came across these options on a random page discussing dumping a > .iso to a USB drive; the example was > dd if=foo.iso of=/dev/sde conv=fdatasync > .. seems now like fsync should also have been given, for certainty. For completely portable use one can only write the data and then call sync afterward and then remove the removable storage after the sync completes. I don't know of any better fully portable way. It's silent if there are no errors. Depending upon the speed of the destination it might be tens of minutes before it completes. dd if=someimage.img of=/dev/sdX obs=16M sync Where /dev/sdX is the device path name of the destination. Always be very careful to ensure the correct destination name. Do not overwrite the wrong target destination. Doing so could destroy your system. For writing images to USB with GNU dd and the Linux kernel I prefer This following combination. It's the most friendly with very good user feedback. pv someimage.img | dd of=/dev/sdX obs=16M oflag=direct,sync Then use of pv wil provide a nice progress notification. Check it out! 4.31GiB 0:08:13 [8.94MiB/s] [==============================================>] 100% The main points being to use a output buffer size large enough to be efficient but small enough such that regular notification of progress is reported to the user. If it is too large then the progress reporting will be too "chunky". Ideally it will be a multiple of the internal flash NAND write block size. Which we can't know and can only take a guess. To keep this entirely within GNU dd there is the new status=progress option. $ dd if=someimage.img of=/dev/sdX obs=16M oflag=direct status=progress 426349056 bytes (426 MB, 407 MiB) copied, 3 s, 142 MB/s ... Honestly though it isn't anywhere near as nice as the progress report from pv and I always use pv+dd for this task. Give it a try! :-) Bob From debbugs-submit-bounces@debbugs.gnu.org Thu Jul 07 00:47:47 2022 Received: (at 54586-done) by debbugs.gnu.org; 7 Jul 2022 04:47:47 +0000 Received: from localhost ([127.0.0.1]:55487 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1o9JQZ-0003p4-2e for submit@debbugs.gnu.org; Thu, 07 Jul 2022 00:47:47 -0400 Received: from zimbra.cs.ucla.edu ([131.179.128.68]:54492) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1o9JQW-0003o8-NF for 54586-done@debbugs.gnu.org; Thu, 07 Jul 2022 00:47:45 -0400 Received: from localhost (localhost [127.0.0.1]) by zimbra.cs.ucla.edu (Postfix) with ESMTP id 6F39B1600E5; Wed, 6 Jul 2022 21:47:38 -0700 (PDT) Received: from zimbra.cs.ucla.edu ([127.0.0.1]) by localhost (zimbra.cs.ucla.edu [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id 8UAxvr3NeoXC; Wed, 6 Jul 2022 21:47:37 -0700 (PDT) Received: from localhost (localhost [127.0.0.1]) by zimbra.cs.ucla.edu (Postfix) with ESMTP id 992821600ED; Wed, 6 Jul 2022 21:47:37 -0700 (PDT) X-Virus-Scanned: amavisd-new at zimbra.cs.ucla.edu Received: from zimbra.cs.ucla.edu ([127.0.0.1]) by localhost (zimbra.cs.ucla.edu [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id J6YQ0x89OU58; Wed, 6 Jul 2022 21:47:37 -0700 (PDT) Received: from [192.168.0.205] (ip72-206-2-24.fv.ks.cox.net [72.206.2.24]) by zimbra.cs.ucla.edu (Postfix) with ESMTPSA id 4BAE01600E5; Wed, 6 Jul 2022 21:47:37 -0700 (PDT) Content-Type: multipart/mixed; boundary="------------erDkKrkWdJtlrQKf0h00JtEM" Message-ID: Date: Wed, 6 Jul 2022 23:47:36 -0500 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.9.1 Subject: Re: bug#54586: dd conv options doc Content-Language: en-US To: Karl Berry References: <202203262029.22QKTg6b018158@freefriends.org> From: Paul Eggert In-Reply-To: <202203262029.22QKTg6b018158@freefriends.org> X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 54586-done Cc: 54586-done@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) This is a multi-part message in MIME format. --------------erDkKrkWdJtlrQKf0h00JtEM Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit On 3/26/22 15:29, Karl Berry wrote: > why would I want data to be synced and not metadata? Performance, in apps that don't care about the metadata. Admittedly for dd the use case is rare; it's mostly present so that dd exports all the open flags to the user. I installed the attached to try to document this better. --------------erDkKrkWdJtlrQKf0h00JtEM Content-Type: text/x-patch; charset=UTF-8; name="0001-dd-doc-improvement-Bug-54586.patch" Content-Disposition: attachment; filename="0001-dd-doc-improvement-Bug-54586.patch" Content-Transfer-Encoding: base64 RnJvbSAxZWZjZTU2NjM1NTQ2MTlkYjM0ZDI3MjJiZTdkNmU1YTE0NDA0MDY1IE1vbiBTZXAg MTcgMDA6MDA6MDAgMjAwMQpGcm9tOiBQYXVsIEVnZ2VydCA8ZWdnZXJ0QGNzLnVjbGEuZWR1 PgpEYXRlOiBXZWQsIDYgSnVsIDIwMjIgMjM6NDI6MTkgLTA1MDAKU3ViamVjdDogW1BBVENI XSBkZDogZG9jIGltcHJvdmVtZW50IChCdWcjNTQ1ODYpCgoqIGRvYy9jb3JldXRpbHMudGV4 aSAoZGQgaW52b2NhdGlvbik6IEV4cGxhaW4KZmRhdGFzeW5jIGFuZCBmc3luYyBiZXR0ZXIu Ci0tLQogZG9jL2NvcmV1dGlscy50ZXhpIHwgMTMgKysrKysrKysrKystLQogMSBmaWxlIGNo YW5nZWQsIDExIGluc2VydGlvbnMoKyksIDIgZGVsZXRpb25zKC0pCgpkaWZmIC0tZ2l0IGEv ZG9jL2NvcmV1dGlscy50ZXhpIGIvZG9jL2NvcmV1dGlscy50ZXhpCmluZGV4IDdiY2EzN2I3 MS4uZTBjODdkMWFkIDEwMDY0NAotLS0gYS9kb2MvY29yZXV0aWxzLnRleGkKKysrIGIvZG9j L2NvcmV1dGlscy50ZXhpCkBAIC05NDY2LDcgKzk0NjYsMTMgQEAgQ29udGludWUgYWZ0ZXIg cmVhZCBlcnJvcnMuCiBAY2luZGV4IHN5bmNocm9uaXplZCBkYXRhIHdyaXRlcywgYmVmb3Jl IGZpbmlzaGluZwogU3luY2hyb25pemUgb3V0cHV0IGRhdGEganVzdCBiZWZvcmUgZmluaXNo aW5nLAogZXZlbiBpZiB0aGVyZSB3ZXJlIHdyaXRlIGVycm9ycy4KLVRoaXMgZm9yY2VzIGEg cGh5c2ljYWwgd3JpdGUgb2Ygb3V0cHV0IGRhdGEuCitUaGlzIGZvcmNlcyBhIHBoeXNpY2Fs IHdyaXRlIG9mIG91dHB1dCBkYXRhLAorc28gdGhhdCBldmVuIGlmIHBvd2VyIGlzIGxvc3Qg dGhlIG91dHB1dCBkYXRhIHdpbGwgYmUgcHJlc2VydmVkLgorSWYgbmVpdGhlciB0aGlzIG5v ciBAc2FtcHtmc3luY30gYXJlIHNwZWNpZmllZCwgb3V0cHV0IGlzIHRyZWF0ZWQgYXMKK3Vz dWFsIHdpdGggZmlsZSBzeXN0ZW1zLCBpLmUuLCBvdXRwdXQgZGF0YSBhbmQgbWV0YWRhdGEg bWF5IGJlIGNhY2hlZAoraW4gcHJpbWFyeSBtZW1vcnkgZm9yIHNvbWUgdGltZSBiZWZvcmUg dGhlIG9wZXJhdGluZyBzeXN0ZW0gcGh5c2ljYWxseQord3JpdGVzIGl0LCBhbmQgdGh1cyBv dXRwdXQgZGF0YSBhbmQgbWV0YWRhdGEgbWF5IGJlIGxvc3QgaWYgcG93ZXIgaXMgbG9zdC4K K0B4cmVme3N5bmMgaW52b2NhdGlvbn0uCiBUaGlzIGNvbnZlcnNpb24gaXMgYSBHTlUgZXh0 ZW5zaW9uIHRvIFBPU0lYLgogCiBAaXRlbSBmc3luYwpAQCAtOTQ3NCw3ICs5NDgwLDEwIEBA IFRoaXMgY29udmVyc2lvbiBpcyBhIEdOVSBleHRlbnNpb24gdG8gUE9TSVguCiBAY2luZGV4 IHN5bmNocm9uaXplZCBkYXRhIGFuZCBtZXRhZGF0YSB3cml0ZXMsIGJlZm9yZSBmaW5pc2hp bmcKIFN5bmNocm9uaXplIG91dHB1dCBkYXRhIGFuZCBtZXRhZGF0YSBqdXN0IGJlZm9yZSBm aW5pc2hpbmcsCiBldmVuIGlmIHRoZXJlIHdlcmUgd3JpdGUgZXJyb3JzLgotVGhpcyBmb3Jj ZXMgYSBwaHlzaWNhbCB3cml0ZSBvZiBvdXRwdXQgZGF0YSBhbmQgbWV0YWRhdGEuCitUaGlz IGFjdHMgbGlrZSBAc2FtcHtmZGF0YXN5bmN9IGV4Y2VwdCBpdCBhbHNvIHByZXNlcnZlcyBv dXRwdXQgbWV0YWRhdGEsCitzdWNoIGFzIHRoZSBsYXN0LW1vZGlmaWVkIHRpbWUgb2YgdGhl IG91dHB1dCBmaWxlOyBmb3IgdGhpcyByZWFzb24gaXQKK21heSBiZSBhIGJpdCBzbG93ZXIg dGhhbiBAc2FtcHtmZGF0YXN5bmN9IGFsdGhvdWdoIHRoZSBwZXJmb3JtYW5jZQorZGlmZmVy ZW5jZSBpcyB0eXBpY2FsbHkgaW5zaWduaWZpY2FudCBmb3IgQGNvbW1hbmR7ZGR9LgogVGhp cyBjb252ZXJzaW9uIGlzIGEgR05VIGV4dGVuc2lvbiB0byBQT1NJWC4KIAogQGVuZCB0YWJs ZQotLSAKMi4zNi4xCgo= --------------erDkKrkWdJtlrQKf0h00JtEM-- From unknown Sat Aug 16 21:59:39 2025 Received: (at fakecontrol) by fakecontrolmessage; To: internal_control@debbugs.gnu.org From: Debbugs Internal Request Subject: Internal Control Message-Id: bug archived. Date: Thu, 04 Aug 2022 11:24:06 +0000 User-Agent: Fakemail v42.6.9 # This is a fake control message. # # The action: # bug archived. thanks # This fakemail brought to you by your local debbugs # administrator