From unknown Sat Jun 14 19:39:26 2025 X-Loop: help-debbugs@gnu.org Subject: bug#56745: Cannot forward process output to both stdout and stderr (in batch mode) Resent-From: Paul Pogonyshev Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Sun, 24 Jul 2022 16:28:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 56745 X-GNU-PR-Package: emacs X-GNU-PR-Keywords: To: 56745@debbugs.gnu.org X-Debbugs-Original-To: bug-gnu-emacs@gnu.org Received: via spool by submit@debbugs.gnu.org id=B.16586800738104 (code B ref -1); Sun, 24 Jul 2022 16:28:02 +0000 Received: (at submit) by debbugs.gnu.org; 24 Jul 2022 16:27:53 +0000 Received: from localhost ([127.0.0.1]:48553 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1oFeSO-00026d-NL for submit@debbugs.gnu.org; Sun, 24 Jul 2022 12:27:53 -0400 Received: from lists.gnu.org ([209.51.188.17]:44320) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1oFeSN-00026W-2Y for submit@debbugs.gnu.org; Sun, 24 Jul 2022 12:27:51 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:57252) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1oFeSM-0006vs-U9 for bug-gnu-emacs@gnu.org; Sun, 24 Jul 2022 12:27:50 -0400 Received: from mail-yw1-x1134.google.com ([2607:f8b0:4864:20::1134]:36581) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1oFeSK-0000wI-Ug for bug-gnu-emacs@gnu.org; Sun, 24 Jul 2022 12:27:50 -0400 Received: by mail-yw1-x1134.google.com with SMTP id 00721157ae682-31e7ca45091so88856707b3.3 for ; Sun, 24 Jul 2022 09:27:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:from:date:message-id:subject:to; bh=uIf9yFGR7bkPmUoE/qH5a/ttdet8UhRJxsRgMbIpdbE=; b=Qu7GJ4bvTxJ85QVic3rDNFHJT79qEo2XGAgAunWrdLn8QibYIoC5O+D4NPAzPLadBL 8cYUQUEkdi4xUWCasvh7CM5wrwVb/3B+s22oJsMcRgRxSDYkPPDl/ILPyudif5TCu0q4 TeWYkGY7iERT6JoMFjHOvYLuFLZayTHhtKLOYzqNqkIeaGbxIWMNshFnSnTjqFASp4nf J0bS7J/rS2SEupFKEGtrKJRRxtl/GH9Pz7HQOTyCaCiWdcwfxKr/ZBSmMfnXvEHOlaH2 5GhG7x6pL9Mytp4ZMSndkxpfO7NHcm0Ez/jMpeptl7kwyqsdTddqj3B+o9HYXGmp45KN SpKA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=uIf9yFGR7bkPmUoE/qH5a/ttdet8UhRJxsRgMbIpdbE=; b=iDwb4yzf45S/qgvI7t1s8w6mPgCD9QfLmymnGxHsixcfDlnkX7NiRc7lu6e2Y7VZG0 dW4h1WpYl5L1HcfhS+UW5oLcbF26YACph4iaxMPCx+F7zDXirstlUd0asmu147qi/0MF bqWRsbKHv2cgYl7iiRHiNlEq9O2O4SlJLkkbUHNztaDNtpFFt44hpItX9DAtdvylDimy 4T9uSPzrtJBqnQIhRpFjVanFQ/kefIJJs00gXuxSa8Fl6bkJQzeIs2dmh12k7dbGyORX vi47FUGqu1F5ixQgnfGTWqcrbeloxVDCGRxB5fCkFQNwtFYQho7oGZ4xpQ2hMWsTtA8I kprg== X-Gm-Message-State: AJIora9DH+FVJ5ztVt/mT+KC5HApudlXSW9ALSAuc9Oxw39fErzxykgn APnVioFuQ37Us2p+9V/KsmEkNb6psyEiD08gGwS/BQlXcA== X-Google-Smtp-Source: AGRyM1s1PMmn4LV16PV1Z2oTlyaWVoXp7qbNKDzAJlpDrut4SVoDz3is9BuXiJpyMEVU/9OspPjZa/asbk1qER59e94= X-Received: by 2002:a81:1c0a:0:b0:31e:6ab0:7f73 with SMTP id c10-20020a811c0a000000b0031e6ab07f73mr7030747ywc.353.1658680064348; Sun, 24 Jul 2022 09:27:44 -0700 (PDT) MIME-Version: 1.0 From: Paul Pogonyshev Date: Sun, 24 Jul 2022 18:27:32 +0200 Message-ID: Content-Type: multipart/mixed; boundary="00000000000008878405e48f8c48" Received-SPF: pass client-ip=2607:f8b0:4864:20::1134; envelope-from=pogonyshev@gmail.com; helo=mail-yw1-x1134.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-Spam-Score: -1.3 (-) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.3 (--) --00000000000008878405e48f8c48 Content-Type: multipart/alternative; boundary="00000000000008877105e48f8c46" --00000000000008877105e48f8c46 Content-Type: text/plain; charset="UTF-8" Especially in batch mode it is often useful to run an external process, letting it write to stdout and stderr as if it was the main executed process. As far as I see (and I have spent quite a lot of time investigating this), with Emacs it is not possible. I found two options for stdout+stderr, but with neither allowing me to achieve quite what I want. 1. You can run a process with stdout and stderr combined, then use e.g. process filter function to e.g. `princ' received output. The disadvantage is that stdout and stderr get bunched together. For example, if someone runs your script with `> /dev/null', he will also miss stderr output of the child process. Perhaps even more importantly, robust automated processing of output becomes impossible, as it can contain "garbage", i.e. output meant for stderr. 2. You can split stdout and stderr e.g. by using `make-pipe-process' for stderr and then e.g. use `princ' in the filter for the "real" process (only stdout output) and `message' for stderr process. The problem is that here output comes completely unsynchronized and (at least usually) in larger chunks. If the child process intermixes stdout and stderr output, it comes out not in the original order. You also lose interactivity if the child process e.g. runs for minutes and prints progress messages regularly during that time. In comparison, with Python (or pretty much any other "real" language) this is trivial, see attached `output-forwarder.py'. I don't see a way to replicate these three lines in Elisp. It would be nice if e.g. `make-process' grew an argument `:forward' (possible values `t', `stderr' and `stdout'; the latter to forward only one stream). It would be fine if this was incompatible with `:buffer' and `:stderr', i.e. if this couldn't be specified together with those options. I wouldn't mind any other interface either, as long as it worked. Attachments: * `mixed-output.py': a simple Python scripts that writes to both stdout and stderr with small delays; * `output-forwarder.py': a simple Python scripts that runs whatever command is specified on the command line and forwards it output cleanly; usage, e.g.: `$ ./output-forwarder.py ./mixed-output.py' or `$ ./output-forwarder.py ./mixed-output.py >/dev/null'. Paul --00000000000008877105e48f8c46 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Especially in batch mode it is often useful to run an exte= rnal process, letting it write to stdout and stderr as if it was the main e= xecuted process. As far as I see (and I have spent quite a lot of time inve= stigating this), with Emacs it is not possible.

I found = two options for stdout+stderr, but with neither allowing me to achieve quit= e what I want.

1. You can run a process with stdou= t and stderr combined, then use e.g. process filter function to e.g. `princ= ' received output. The disadvantage is that stdout and stderr get bunch= ed together. For example, if someone runs your script with `> /dev/null&= #39;, he will also miss stderr output of the child process. Perhaps even mo= re importantly, robust automated processing of output becomes impossible, a= s it can contain "garbage", i.e. output meant for stderr.

2. You can split stdout and stderr e.g. by using `make-pi= pe-process' for stderr and then e.g. use `princ' in the filter for = the "real" process (only stdout output) and `message' for std= err process. The problem is that here output comes completely unsynchronize= d and (at least usually) in larger chunks. If the child process intermixes = stdout and stderr output, it comes out not in the original order. You also = lose interactivity if the child process e.g. runs for minutes and prints pr= ogress messages regularly during that time.

In com= parison, with Python (or pretty much any other "real" language) t= his is trivial, see attached `output-forwarder.py'. I don't see a w= ay to replicate these three lines in Elisp.

It wou= ld be nice if e.g. `make-process' grew an argument `:forward' (poss= ible values `t', `stderr' and `stdout'; the latter to forward o= nly one stream). It would be fine if this was incompatible with `:buffer= 9; and `:stderr', i.e. if this couldn't be specified together with = those options. I wouldn't mind any other interface either, as long as i= t worked.

Attachments:
* `mixed-output.p= y': a simple Python scripts that writes to both stdout and stderr with = small delays;
* `output-forwarder.py': a simple Python script= s that runs whatever command is specified on the command line and forwards = it output cleanly; usage, e.g.: `$ ./output-forwarder.py ./mixed-output.py&= #39; or `$ ./output-forwarder.py ./mixed-output.py >/dev/null'.

Paul
--00000000000008877105e48f8c46-- --00000000000008878405e48f8c48 Content-Type: text/x-python; charset="US-ASCII"; name="mixed-output.py" Content-Disposition: attachment; filename="mixed-output.py" Content-Transfer-Encoding: base64 Content-ID: X-Attachment-Id: f_l5zj59ym0 IyEgL3Vzci9iaW4vZW52IHB5dGhvbgoKaW1wb3J0IHN5cwppbXBvcnQgdGltZQpmb3IgayBpbiBy YW5nZSAoMyk6CiAgICBzeXMuc3Rkb3V0LndyaXRlICgiW1NURE9VVF1cbiIpCiAgICB0aW1lLnNs ZWVwICgwLjIpCiAgICBzeXMuc3RkZXJyLndyaXRlICgiW3N0ZGVycl1cbiIpCiAgICB0aW1lLnNs ZWVwICgwLjIpCg== --00000000000008878405e48f8c48 Content-Type: text/x-python; charset="US-ASCII"; name="output-forwarder.py" Content-Disposition: attachment; filename="output-forwarder.py" Content-Transfer-Encoding: base64 Content-ID: X-Attachment-Id: f_l5zj5jc31 IyEgL3Vzci9iaW4vZW52IHB5dGhvbgoKaW1wb3J0IHN1YnByb2Nlc3MKaW1wb3J0IHN5cwoKc3Vi cHJvY2Vzcy5ydW4gKHN5cy5hcmd2WzE6XSkK --00000000000008878405e48f8c48--