From debbugs-submit-bounces@debbugs.gnu.org Sat Mar 02 17:11:06 2019 Received: (at submit) by debbugs.gnu.org; 2 Mar 2019 22:11:06 +0000 Received: from localhost ([127.0.0.1]:57930 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1h0CqX-0005LJ-MO for submit@debbugs.gnu.org; Sat, 02 Mar 2019 17:11:05 -0500 Received: from eggs.gnu.org ([209.51.188.92]:48446) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1h0CqW-0005Kq-GZ for submit@debbugs.gnu.org; Sat, 02 Mar 2019 17:11:04 -0500 Received: from lists.gnu.org ([209.51.188.17]:58533) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1h0CqR-0003nN-B8 for submit@debbugs.gnu.org; Sat, 02 Mar 2019 17:10:59 -0500 Received: from eggs.gnu.org ([209.51.188.92]:34583) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1h0CqQ-0000PO-G3 for bug-coreutils@gnu.org; Sat, 02 Mar 2019 17:10:59 -0500 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: * X-Spam-Status: No, score=1.1 required=5.0 tests=BAYES_50, FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM autolearn=disabled version=3.3.2 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1h0CqP-0003jr-RS for bug-coreutils@gnu.org; Sat, 02 Mar 2019 17:10:58 -0500 Received: from mail-wm1-x32d.google.com ([2a00:1450:4864:20::32d]:52404) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1h0CqP-0003dw-Ea for bug-coreutils@gnu.org; Sat, 02 Mar 2019 17:10:57 -0500 Received: by mail-wm1-x32d.google.com with SMTP id m1so1283170wml.2 for ; Sat, 02 Mar 2019 14:10:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:subject:from:to:date:mime-version :content-transfer-encoding; bh=khUB+1fz/L8XLaL2C4A143mNv/TrW9c97b4vvzMcGqw=; b=L65lILmXgpMi98QOVu9FBv+ErHzebKJJ6MuTNq+Fdz82iwv41PLP1h5GDbI/v2DCer qMUecCx+/XIBOz1FAXcNr3d1JsljFIVWIpwuELvAkgsVNssEjOYJzfbE/fXI8VL1hlvj Ia6D3EqFU52Vt5/OaGwWmz+zjRnUwgPESU3C2jb1lb9+J6YsYlUzd1aewqRpqJUF42pV xYc9+9vKJDgLBs5ZeERmM3KavWN/6hquuZHIMz5D7s2/hiMiZSgj+fB5O5jrcbphRuIL XwzGPkpKkOhAJzG0/CgKgsj1YbMtjIl+NVT8ViFVuM0mE7hjGIib6ORa0CqlCyu1vpy7 xXSQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:subject:from:to:date:mime-version :content-transfer-encoding; bh=khUB+1fz/L8XLaL2C4A143mNv/TrW9c97b4vvzMcGqw=; b=PWm92n0vzt81VbgP6/XxC1IHpJIwmx8fAer/MncPHZTk9xOg54B1Vz87pXPU/GvZja QBrMlrsmMyJXzWBrDGkD9i61I6iZ5beUXw+9Cqm2e30wHpk/pTHkjokEE44IRrE18YCM 7FREGsWekH0m31RvCKFmd08AiMhBdshFJmVeX7sHqghvk0T+MJ1PggG9nRlhxD7xsiiC aXDpwCP43WZL7G9nqNTN4j/HMatvk4RGOtjG5sM+idol+5iQuEb8UoPDLILI7rpb9Yqz /1z3OIrn4AergwcGWnnVGUpRAiZNM80fG4Zf6fd+b9EJiT6Lvs/1g9BVz2eDQ4MC0R0b npOA== X-Gm-Message-State: AHQUAuaVX8NptSqE8QYLBgQjZVH9Hj6PmlpTq0ZVRh33fJntf3MR2xwo 43JbfUizlIfzOOnhsygmuiPa77DA2Ew= X-Google-Smtp-Source: AHgI3IaIqBX6KoKuMfgLiJx1d5oObMo5IdRy/7mA26aTqg/QnxxTsoOEGArRNHEcxFf2IBXYba8DoQ== X-Received: by 2002:a1c:2545:: with SMTP id l66mr6414913wml.96.1551564653398; Sat, 02 Mar 2019 14:10:53 -0800 (PST) Received: from big-box ([2a02:8071:219c:7a00:482f:d185:e772:ec0c]) by smtp.googlemail.com with ESMTPSA id v2sm2364678wme.29.2019.03.02.14.10.52 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Sat, 02 Mar 2019 14:10:52 -0800 (PST) Message-ID: <7f982026973b2c3d6e7429583834b04032ad59f2.camel@gmail.com> Subject: Files vanishing when moving to different FS From: Christoph Michelbach To: bug-coreutils@gnu.org Date: Sat, 02 Mar 2019 23:10:51 +0100 Content-Type: text/plain; charset="UTF-8" X-Mailer: Evolution 3.28.5-0ubuntu0.18.04.1 Mime-Version: 1.0 Content-Transfer-Encoding: 7bit X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:4864:20::32d X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6.x X-Spam-Score: 1.2 (+) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: To reproduce this bug, you need two different file systems. Adapt the paths to fit your system. Set the experimental file structure up like this: mkdir exp cd exp mkdir a cd a touch a{1..100000} cd .. mkdir b cd b touch b{1..10000} mkdir /t/ae # /t has to be on a different file system Content analysis details: (1.2 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- 1.0 SPF_SOFTFAIL SPF: sender does not match SPF record (softfail) 0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider (michelbach94[at]gmail.com) 0.2 FREEMAIL_ENVFROM_END_DIGIT Envelope-from freemail username ends in digit (michelbach94[at]gmail.com) X-Debbugs-Envelope-To: submit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 0.2 (/) To reproduce this bug, you need two different file systems. Adapt the paths to fit your system. Set the experimental file structure up like this: mkdir exp cd exp mkdir a cd a touch a{1..100000} cd .. mkdir b cd b touch b{1..10000} mkdir /t/ae # /t has to be on a different file system Then have two terminals open in the exp directory created above. In one, execute this command: mv a /t/ae In the other, execute this one while the one in the first terminal still is running (hence the large number of files so you have time to do this): mv b/* a You will end up with 100 000 files in /t/ae. The 10 000 files beginning with the letter b will be gone. -- Christoph Michelbach From debbugs-submit-bounces@debbugs.gnu.org Mon Mar 04 13:33:28 2019 Received: (at 34713) by debbugs.gnu.org; 4 Mar 2019 18:33:28 +0000 Received: from localhost ([127.0.0.1]:60162 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1h0sP2-0002xb-8S for submit@debbugs.gnu.org; Mon, 04 Mar 2019 13:33:28 -0500 Received: from havoc.proulx.com ([96.88.95.61]:47650) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1h0sP0-0002xF-82; Mon, 04 Mar 2019 13:33:26 -0500 Received: from joseki.proulx.com (localhost [127.0.0.1]) by havoc.proulx.com (Postfix) with ESMTP id 618BA442; Mon, 4 Mar 2019 11:33:19 -0700 (MST) Received: from hysteria.proulx.com (hysteria.proulx.com [192.168.230.119]) by joseki.proulx.com (Postfix) with ESMTP id 128CD2123E; Mon, 4 Mar 2019 11:33:14 -0700 (MST) Received: by hysteria.proulx.com (Postfix, from userid 1000) id DA97E2DC7C; Mon, 4 Mar 2019 11:33:13 -0700 (MST) Date: Mon, 4 Mar 2019 11:33:13 -0700 From: Bob Proulx To: Christoph Michelbach Subject: Re: bug#34713: Files vanishing when moving to different FS Message-ID: <20190304110659352723727@bob.proulx.com> References: <7f982026973b2c3d6e7429583834b04032ad59f2.camel@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <7f982026973b2c3d6e7429583834b04032ad59f2.camel@gmail.com> User-Agent: Mutt/1.10.1 (2018-07-13) X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 34713 Cc: 34713@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) tags 34713 notabug close 34713 thanks Hello Christoph, Christoph Michelbach wrote: > To reproduce this bug, you need two different file systems. Adapt the > paths to fit your system. Thank you for making this bug report. However what you are experiencing is due to the race condition created by the non-atomic nature of copying files from one file system to another, removing files, and renaming files. This is not a bug in mv but is an intrinsic behavior. > Set the experimental file structure up like this: > > mkdir exp > cd exp > mkdir a > cd a > touch a{1..100000} > cd .. > mkdir b > cd b > touch b{1..10000} > mkdir /t/ae # /t has to be on a different file system Thank you for the very nice test case. > Then have two terminals open in the exp directory created above. This is a clue to the nature of the problem being a race condition. It describes simultaneous parallel processes. > In one, execute this command: > > mv a /t/ae Because /t is on a different file system mv cannot simply rename the files but must perform the action in two steps. It copies the file from source to destination. It removes source file. This is documented in the mv with: 'mv' can move any type of file from one file system to another. Prior to version '4.0' of the fileutils, 'mv' could move only regular files between file systems. For example, now 'mv' can move an entire directory hierarchy including special device files from one partition to another. It first uses some of the same code that's used by 'cp -a' to copy the requested directories and files, then (assuming the copy succeeded) it removes the originals. If the copy fails, then the part that was copied to the destination partition is removed. If you were to copy three directories from one partition to another and the copy of the first directory succeeded, but the second didn't, the first would be left on the destination partition and the second and third would be left on the original partition. The mv a /t/ae action is similar to cp -a a t/ae && rm -r a when the action is successful. Similar because there are two steps happening. A first step with the copy and a second step with the removal and there is a time skew between those actions. > In the other, execute this one while the one in the first terminal > still is running (hence the large number of files so you have time to > do this): > > mv b/* a This is the second part of the race condition. It it moving files into the a directory at the same time that files are being copied out of the directory and the directory itself being removed. > You will end up with 100 000 files in /t/ae. The 10 000 files beginning > with the letter b will be gone. Look at the two actions explicitly: Process 1: cp -a a /t/ae rm -rf a Process 2: mv b/* a Now it is more obvious that as soon as the first process copy finishes that it will remove the source location, that is having files moved into it by the second process, that the directory will be deleted by the first process. Does that make it easier to understand what is happening? The copy and remove two actions do not occur when both the source and destination are on the same file system. In that case the file can be renamed atomically without doing a copy. But when the action is across two file systems this is not possible and it is simulated (or perhaps emulated) by the copy and remove two step action. Whenever tasks are moving files into and out of the same directory at the same time this is always something to be aware of regardless because they may be an overlap of actions in that directory. In this particular example the problem can be avoided by renaming "a" first and then transfering the files to the other file system. Because it was removed then the second process can create it without collision. Something like this pseudo-code. However to safely use temporary file names will require more code than this. This is simply for illustration purposes. Process 1: mv a tmpdirname cp -a tmpdirname /t/ae rm -rf tmpdirname Process 2: mkdir a mv b/* a I hope this helps. Since this is not a bug in mv I am going to close the ticket in our bug database. But please we would like to hear back from you in this ticket for any further discussion. Bob From debbugs-submit-bounces@debbugs.gnu.org Mon Mar 04 14:57:56 2019 Received: (at 34713) by debbugs.gnu.org; 4 Mar 2019 19:57:56 +0000 Received: from localhost ([127.0.0.1]:60240 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1h0til-0006z1-Ux for submit@debbugs.gnu.org; Mon, 04 Mar 2019 14:57:56 -0500 Received: from mail-wr1-f44.google.com ([209.85.221.44]:34160) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1h0tik-0006yn-39 for 34713@debbugs.gnu.org; Mon, 04 Mar 2019 14:57:54 -0500 Received: by mail-wr1-f44.google.com with SMTP id f14so6936077wrg.1 for <34713@debbugs.gnu.org>; Mon, 04 Mar 2019 11:57:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:to:cc:references:from:message-id:date:user-agent :mime-version:in-reply-to:content-transfer-encoding:content-language; bh=Cw4OIEbzkbx3lyShhWlMPyMjRHGQTDbeHTwFGzJl+4w=; b=vcIIqc9G5WrKPkyN+8xYPvVX/1MyQ8unKnJ13MfCkYKcOJ1C7uCjSxO6Gw5fcwl7T6 yqqvRc9Yqutq2l44jFSSeBhySYZHHvQEWOJIP1/3xiwR5h9BM2jBbsryjal32W86Gq0v XVv7+j7FSOudTft1PufFLkL3js7cOcp/BptkBFdCZIMA0YaciTtUpK3XfF5qvC/2d8QL dXBaUWso8NE4XZBTZRzi0yLEZp0ZkkZIMcoHiZNe3msvtk54gnG5Ln9qUirSDNFtpb5s T26h5ADZx6fxnE3nelETQuSBuRzh23Zmjs6FnlAkT1fR9dISTH5JW0+Qfd2SxmQ3yI/l rlLA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding :content-language; bh=Cw4OIEbzkbx3lyShhWlMPyMjRHGQTDbeHTwFGzJl+4w=; b=RuLR5hxRqyQQn5ysqGzyXMvRmXysDjWoh1sgVsnju0zv9A5GZbsEJRn1V72knufF/4 +vf7uuShd6f/6fEiG4hjt4vr7chKrYDqLDLPy9odiXQmP4RTAQL+7cI+BMKIAl04t+rw Rdye1rh7dagnJQBzk8EdqyjP6MYAHBIXOLCiT6HBsuSEUBHR4wHCyLdWDuyK9/rJSpgK 7KrsOBII/Fsv62aUfISepIsrLJTWkcyyq/AC0/7O4PUSMi2Xye3vI+ewob82jTOmMmAO UbJX8vamhFekY+IWXbwhEVETwpgF9K9HeVhexuGDYtvFBfu8VUsN1xgNUc6BKRKHhCXo cC0A== X-Gm-Message-State: APjAAAWzm7Z7SKlZAvP7CMq+BJ03cmaoWr0XsEn0RAMQEH8wXCgXbmf+ yc/DDeReuwU30pyT5Teh5QSPhUOH57w= X-Google-Smtp-Source: APXvYqyXf9GBG1bUetU0qBtna5XDSiPg44Gs69J9NAov0Im1nL49jx63dl1HgmEZT5vNlwTRlnxMtg== X-Received: by 2002:a5d:510b:: with SMTP id s11mr14424777wrt.131.1551729467666; Mon, 04 Mar 2019 11:57:47 -0800 (PST) Received: from ?IPv6:2a02:8071:219c:7a00:58ae:96f5:cbae:a570? ([2a02:8071:219c:7a00:58ae:96f5:cbae:a570]) by smtp.gmail.com with ESMTPSA id z10sm4731776wml.39.2019.03.04.11.57.46 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 04 Mar 2019 11:57:46 -0800 (PST) Subject: Re: bug#34713: Files vanishing when moving to different FS To: Bob Proulx References: <7f982026973b2c3d6e7429583834b04032ad59f2.camel@gmail.com> <20190304110659352723727@bob.proulx.com> From: Christoph Michelbach Message-ID: <37a0b65b-d779-baaf-adbb-8c24963bed26@gmail.com> Date: Mon, 4 Mar 2019 20:57:45 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.5.1 MIME-Version: 1.0 In-Reply-To: <20190304110659352723727@bob.proulx.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-US X-Spam-Score: 0.2 (/) X-Debbugs-Envelope-To: 34713 Cc: 34713@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.8 (/) Hello, Bob Thank you for your explanation. I understand what's going on and even looked at the part of the source code that causes the problem prior to filing the bug report. Even if you don't consider it a bug, it's still unexpected behavior that could be avoided. I think that mv's behavior would be more logical if it only removed files that it copied prior to removal (simple solution to the problem without destroying user data) or also move files that appeared in the destination location during the copying process (emulating moving a directory within a file system better and not destroying user data). In the first case, it should still try to remove directories after removing all files in them but adhere to the rule that non-empty directories cannot be deleted instead of simply emptying them of their content prior to deletion. In the second case, it should attempt to delete the directory and if this fails because it is non-empty, copy the newly appeared files over to the destination. I think that the second solution is better for the users as it emulates the behavior of a move operation within a file system. However, I see that the first solution is easier to implement. The current behavior of mv destroys user data which I think is highly unexpected behavior. Christoph From unknown Fri Aug 15 15:36:45 2025 Received: (at fakecontrol) by fakecontrolmessage; To: internal_control@debbugs.gnu.org From: Debbugs Internal Request Subject: Internal Control Message-Id: bug archived. Date: Tue, 02 Apr 2019 11:24:06 +0000 User-Agent: Fakemail v42.6.9 # This is a fake control message. # # The action: # bug archived. thanks # This fakemail brought to you by your local debbugs # administrator