GNU bug report logs - #23206
25.0.92; dired-shell-stuff-it: wait until all parallel jobs finish

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 23206 in the body.
You can then email your comments to 23206 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Tino Calancha <f92capac <at> gmail.com>
To: bug-gnu-emacs <at> gnu.org
Subject: 25.0.92; dired-shell-stuff-it: wait until all parallel jobs finish
Date: Sun, 3 Apr 2016 22:02:17 +0900 (JST)

[Message part 1 (text/plain, inline)]

Good (or morning afternoon evening night)

When `dired-do-shell-command' runs COMMAND over FILES in
parallel, the `shell-command' process may finish before all the
files have being processed.

emacs -Q ~

* / & du SPC -s RET
; If the number of lines in #<buffer *Async Shell Command*> is less
; than (1+ (length (dired-get-marked-files))) then NOT all files
; has being processed.

Attached patch fix this issue adding 'wait' at the end of COMMAND
when FILES are procesed in parallel.

In GNU Emacs 25.0.92.1 (x86_64-pc-linux-gnu, GTK+ Version 2.24.30)
 of 2016-04-01 built on calancha-pc
Repository revision: a3daa34336da158555d96d670077eedb9eaeeb9c

[dired-aux.patch (text/plain, attachment)]

Message #8 received at 23206 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Tino Calancha <f92capac <at> gmail.com>
Cc: 23206 <at> debbugs.gnu.org
Subject: Re: bug#23206: 25.0.92;
 dired-shell-stuff-it: wait until all parallel jobs finish
Date: Sun, 03 Apr 2016 18:03:52 +0300

> Date: Sun, 3 Apr 2016 22:02:17 +0900 (JST)
> From: Tino Calancha <f92capac <at> gmail.com>
> 
> When `dired-do-shell-command' runs COMMAND over FILES in
> parallel, the `shell-command' process may finish before all the
> files have being processed.
> 
> emacs -Q ~
> 
> * / & du SPC -s RET
> ; If the number of lines in #<buffer *Async Shell Command*> is less
> ; than (1+ (length (dired-get-marked-files))) then NOT all files
> ; has being processed.
> 
> Attached patch fix this issue adding 'wait' at the end of COMMAND
> when FILES are procesed in parallel.

Thanks.  Can you elaborate on why such a wait is needed?

Message #11 received at 23206 <at> debbugs.gnu.org (full text, mbox):

From: Tino Calancha <f92capac <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: Tino Calancha <f92capac <at> gmail.com>, 23206 <at> debbugs.gnu.org
Subject: Re: bug#23206: 25.0.92; dired-shell-stuff-it: wait until all parallel
 jobs finish
Date: Mon, 4 Apr 2016 01:38:52 +0900 (JST)

> Thanks.  Can you elaborate on why such a wait is needed?

Let's assume COMMAND is "du -s"
and LIST contains the dirs foo bar baz:
foo is ~ Gb  with depth > 1
and bar and baz are few kB dirs with depth 1.

The effective shell command to run is:
du -s foo& du -s bar& du -s baz&

This creates a new process PROC in the system.
PROC returns with the return code of the last command in the list
(du -s baz):  this is a shell feature.
The output from the remaining jobs is still comming to stdout,
but is not associated to PROC, which already succeded.

With `dired-do-shell-command', we get all output only if foo
is the last dir in LIST.

POSIX shells provide the builtin 'wait' to change this behaviour:
it forces PROC to wait until all background jobs end.

You can run from terminal following scripts:
I)
#!/bin/sh
echo "id $$"
du -s foo& du -s bar& du -s baz&

II)
#!/bin/sh
echo "id $$"
du -s foo& du -s bar& du -s baz&
wait

Then list the process id of the scripts while you still have
output comming:
ps -f -p ID

I) No process in the system with ID.
II) There is a process with ID.


Well, maybe it can be solve in many other ways.  I just found
very simple: just adding 4 letters fix it; also very natural,
because we are building a shell command, so its logical to use the
shell sintaxis.

Message #14 received at 23206 <at> debbugs.gnu.org (full text, mbox):

From: Tino Calancha <f92capac <at> gmail.com>
To: 23206 <at> debbugs.gnu.org
Cc: Tino Calancha <f92capac <at> gmail.com>
Subject: Re: bug#23206: 25.0.92; dired-shell-stuff-it: wait until all parallel
 jobs finish
Date: Mon, 4 Apr 2016 15:20:13 +0900 (JST)

I must clarify the tittle of this thread (the word wait).
Of course, the process is still asynchronous! :-)

The command wait just avoid `shell-command-sentinel' be triggered
too early.  Once all jobs in the background finish, the shell
returns with the return code of the last command in the list; then
`shell-command-sentinel' do its stuff.

The command 'wait' is specified by posix.
I have verified that it is present in following shells:

bash: 4.3, 3.2
dash: 0.5.8-2.2
posh: 0.12.6
bsd-csh: 1.65 (20110502-2.1)
tcsh: 6.18, 6.14
ksh: 93u+ 2012-08-01,  20100202-1
zsh: 5.2, 4.2

Message #17 received at 23206 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Tino Calancha <f92capac <at> gmail.com>
Cc: 23206 <at> debbugs.gnu.org
Subject: Re: bug#23206: 25.0.92; dired-shell-stuff-it: wait until all parallel
 jobs finish
Date: Mon, 04 Apr 2016 20:25:25 +0300

> Date: Mon, 4 Apr 2016 01:38:52 +0900 (JST)
> From: Tino Calancha <f92capac <at> gmail.com>
> cc: Tino Calancha <f92capac <at> gmail.com>, 23206 <at> debbugs.gnu.org
> 
> Let's assume COMMAND is "du -s"
> and LIST contains the dirs foo bar baz:
> foo is ~ Gb  with depth > 1
> and bar and baz are few kB dirs with depth 1.
> 
> The effective shell command to run is:
> du -s foo& du -s bar& du -s baz&
> 
> This creates a new process PROC in the system.
> PROC returns with the return code of the last command in the list
> (du -s baz):  this is a shell feature.
> The output from the remaining jobs is still comming to stdout,
> but is not associated to PROC, which already succeded.
> 
> With `dired-do-shell-command', we get all output only if foo
> is the last dir in LIST.
> 
> POSIX shells provide the builtin 'wait' to change this behaviour:
> it forces PROC to wait until all background jobs end.

Thanks.  This means the fix should only affect systems with a Posix
shell.  The stock MS-Windows shells don't have a 'wait' command (and
don't need this fix in the first place, as the problem doesn't exist
on MS-Windows, AFAICS).

Can you modify the patch to do that?

Thanks.

Message #20 received at 23206 <at> debbugs.gnu.org (full text, mbox):

From: Tino Calancha <f92capac <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: Tino Calancha <f92capac <at> gmail.com>, 23206 <at> debbugs.gnu.org
Subject: Re: bug#23206: 25.0.92; dired-shell-stuff-it: wait until all parallel
 jobs finish
Date: Tue, 5 Apr 2016 14:24:41 +0900 (JST)

[Message part 1 (text/plain, inline)]

>This means the fix should only affect systems with a Posix
>shell.
>The stock MS-Windows shells don't have a 'wait' command (and
>don't need this fix in the first place, as the problem doesn't exist
>on MS-Windows, AFAICS).
I see.  Thank you.
I rewrote the patch so that it adds 'wait' when 'system-type' is
not in '(ms-dos windows-nt).

[0001-Wait-until-all-parallel-shell-commands-finish.patch (text/plain, attachment)]

Message #23 received at 23206 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Tino Calancha <f92capac <at> gmail.com>
Cc: 23206 <at> debbugs.gnu.org
Subject: Re: bug#23206: 25.0.92; dired-shell-stuff-it: wait until all parallel
 jobs finish
Date: Tue, 05 Apr 2016 17:59:22 +0300

> Date: Tue, 5 Apr 2016 14:24:41 +0900 (JST)
> From: Tino Calancha <f92capac <at> gmail.com>
> cc: Tino Calancha <f92capac <at> gmail.com>, 23206 <at> debbugs.gnu.org
> 
> >This means the fix should only affect systems with a Posix
> >shell.
> >The stock MS-Windows shells don't have a 'wait' command (and
> >don't need this fix in the first place, as the problem doesn't exist
> >on MS-Windows, AFAICS).
> I see.  Thank you.
> I rewrote the patch so that it adds 'wait' when 'system-type' is
> not in '(ms-dos windows-nt).

Thanks.

I have one more nit:

> * lisp/dired-aux.el (dired-shell-stuff-it):
> Drop all trailing ';' and '&' in command;
> change indentation with \t to ?\s;

These are unrelated changes, so please don't mix them with the changes
that really fix the problem at hand.  In particular, removing all the
trailing ';' and '&' characters might cause bugs, because the code you
are modifying doesn't really understand what the shell command is
about, so it might inadvertently delete characters that are there for
a reason.  It is up to the user to type a correct command, and Emacs
shouldn't try second-guessing those commands.  So let's only remove a
single instance of these characters at the end, as the original code
did.

Thanks.

Message #26 received at 23206 <at> debbugs.gnu.org (full text, mbox):

From: Tino Calancha <f92capac <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: Tino Calancha <f92capac <at> gmail.com>, 23206 <at> debbugs.gnu.org
Subject: Re: bug#23206: 25.0.92; dired-shell-stuff-it: wait until all parallel
 jobs finish
Date: Wed, 6 Apr 2016 00:58:30 +0900 (JST)

[Message part 1 (text/plain, inline)]

>These are unrelated changes, so please don't mix them with the changes
>that really fix the problem at hand.
I am sorry tha's my fault.  I should always do atomic changes.

>It is up to the user to type a correct command, and Emacs
>shouldn't try second-guessing those commands.  So let's only remove a
>single instance of these characters at the end, as the original code
>did.
I was 'baby sitting' the user (and my self). Yeah, too much care, 
Emacs users can protect themselves very well ;-)

Attached atomic commits without the extra care.

[0001-Wait-until-all-parallel-shell-commands-finish.patch (text/plain, attachment)]

[0002-dired-shell-stuff-it-Consistent-indentation.patch (text/plain, attachment)]

Message #31 received at 23206-done <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Tino Calancha <f92capac <at> gmail.com>
Cc: 23206-done <at> debbugs.gnu.org
Subject: Re: bug#23206: 25.0.92; dired-shell-stuff-it: wait until all parallel
 jobs finish
Date: Fri, 08 Apr 2016 17:06:44 +0300

> Date: Wed, 6 Apr 2016 00:58:30 +0900 (JST)
> From: Tino Calancha <f92capac <at> gmail.com>
> cc: Tino Calancha <f92capac <at> gmail.com>, 23206 <at> debbugs.gnu.org
> 
> Attached atomic commits without the extra care.

Thanks, pushed to the master branch.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.