From unknown Wed Jun 18 23:14:23 2025 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Mailer: MIME-tools 5.509 (Entity 5.509) Content-Type: text/plain; charset=utf-8 From: bug#56342 <56342@debbugs.gnu.org> To: bug#56342 <56342@debbugs.gnu.org> Subject: Status: TRAMP (sh) issues way too many commands, thus being very slow over high-ping networks Reply-To: bug#56342 <56342@debbugs.gnu.org> Date: Thu, 19 Jun 2025 06:14:23 +0000 retitle 56342 TRAMP (sh) issues way too many commands, thus being very slow= over high-ping networks reassign 56342 emacs submitter 56342 Paul Pogonyshev severity 56342 wishlist thanks From debbugs-submit-bounces@debbugs.gnu.org Fri Jul 01 13:14:26 2022 Received: (at submit) by debbugs.gnu.org; 1 Jul 2022 17:14:26 +0000 Received: from localhost ([127.0.0.1]:39115 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1o7KDq-0000ip-Bj for submit@debbugs.gnu.org; Fri, 01 Jul 2022 13:14:26 -0400 Received: from lists.gnu.org ([209.51.188.17]:45610) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1o7KDn-0000ig-HV for submit@debbugs.gnu.org; Fri, 01 Jul 2022 13:14:25 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:33020) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1o7KDn-0001Oc-DI for bug-gnu-emacs@gnu.org; Fri, 01 Jul 2022 13:14:23 -0400 Received: from mail-yb1-xb34.google.com ([2607:f8b0:4864:20::b34]:45853) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1o7KDl-0001tv-K0 for bug-gnu-emacs@gnu.org; Fri, 01 Jul 2022 13:14:23 -0400 Received: by mail-yb1-xb34.google.com with SMTP id x184so5075711ybg.12 for ; Fri, 01 Jul 2022 10:14:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:from:date:message-id:subject:to; bh=lb+xDgaB7/ccMFM10Q4OCSSHsn62OXul5w1QRxtC7dU=; b=JAFPZpQGyORSVcqwFMKpLCDmMEwITqTPWMV1NHUGCecaZjY+5yoFYRtRKe92kQIEpt ngVB8vim6UEuzPpDP8PBK+RU2UrzMhsela2QmfcpCp8TPCLnoj28RUsw1wucyST5sMpB b+9teCdfWS/E+YtlB4JDgWfYS6apHWrO02Q/4RWyM+FM4RquhwTb/hNR6f0NWUtKRyf0 baIiox+ALHKxRWlxHAZqa7ik105lA1mpGALfEeIJ2DzwIM7+oyDNh2w8mNJyr+npvk/L xc6EZzZ/GxxtEsR9LW2XOZeAlNh3BkRJGercYAYnataXmigIegWB3UQEbe8zEFpaWk2u K3zA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=lb+xDgaB7/ccMFM10Q4OCSSHsn62OXul5w1QRxtC7dU=; b=7q4sqNzXuURvaGAWgeqnG1NRm+aOxPGNVmdNtDOqaZk8CDaV3a8ulikh55cbt+hrvi 937Ng4DNeF3QKwIk3QLBxe6S1FT0aJrvCGNlFmfKlqk2GUflkUh7izFoUXa3H65dQHEX LqvREsNt0bmCo3yFJYsQmXLyPPPsYf9pEaNBRIwsEfFRKaGrq3hql/hWfZxSa3O9LB8M cOVmW5Rnt9/sJ7UmNDVJ0qAYgPRShWE2sQ3PoQC1CN5tJldRZ+1MHjf83gDDeLVQsWp7 d6q4ux7cit5hWUmRYvITFjqu2wsiVxKwWVxYAhZ/S+i3kxzP8Acnju8ftXHXcPNhhr3L q/iA== X-Gm-Message-State: AJIora/FZ2cX103aIz/9nDqWBmXHUb8CuPzcg6Rq8L730W5rYz0HZrJS nnnLlJWIImCvyHdU0MizhRVedHCetLO6V38E+Tb+EudtLoUp X-Google-Smtp-Source: AGRyM1utDB5FcAFf57MV01Qz/TXNtsQqlID5LeIVnIfzDSgIqRZb+C4d/aqoFofS4WI+f0Hn4TjkzIfFwCvFdkV5nK8= X-Received: by 2002:a25:d917:0:b0:66c:f9a1:8304 with SMTP id q23-20020a25d917000000b0066cf9a18304mr16610250ybg.170.1656695659884; Fri, 01 Jul 2022 10:14:19 -0700 (PDT) MIME-Version: 1.0 From: Paul Pogonyshev Date: Fri, 1 Jul 2022 19:14:08 +0200 Message-ID: Subject: TRAMP (sh) issues way too many commands, thus being very slow over high-ping networks To: bug-gnu-emacs@gnu.org Content-Type: multipart/alternative; boundary="0000000000004ef69f05e2c184de" Received-SPF: pass client-ip=2607:f8b0:4864:20::b34; envelope-from=pogonyshev@gmail.com; helo=mail-yb1-xb34.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-Spam-Score: -1.3 (-) X-Debbugs-Envelope-To: submit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.3 (--) --0000000000004ef69f05e2c184de Content-Type: text/plain; charset="UTF-8" Emacs version: GNU Emacs 28.1.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version 2.24.33, cairo version 1.16.0) of 2022-05-24 I have an internet connection with a relatively large ping, so TRAMP for me works quite slowly. I notice this when editing and saving files (also when Emacs stores backup copies), with Magit and otherwise. I tried to understand why it is _so_ slow for me. I have found out variable `tramp-verbose' and used it to figure out which commands TRAMP executes on the remote machine. For testing, I used a real library's (Logview's) buffer refreshing command that boils down to this: (with-temp-buffer (insert-file-contents the-file nil from-somewhere-in-the-middle nil) ; do something about the data ) In other words, it inserts into a buffer not the whole file, but a part of it (the reason is irrelevant here). Well, this results in _eleven_ commands from TRAMP! Here they are: 1) check if connection is alive (`echo are you awake'); 2) test if the file exists; 3) creating a temporary file for the chunk to be inserted; I guess it tries until it finds an unused filename, e.g. here it seems to be done after `test -e /tmp/tramp.OD3cCu', which doesn't exist; 4) 'touch' on the temporary file, presumably to create it; 5) 'chmod' on the temporary, presumably so that other users cannot read it; 6) copying the requested chunk from the full file into the temporary (using `dd'); 7) finding the real name of the temporary with `readlink'; 8) finding attributes of the temporary with `stat'; 9) gzipping the temporary for transmition from the remote to the local machine; 10) testing if the temporary is a directory (WTF?); 11) removing the temporary. I guess it should be obvious that this is a bit too much for one `insert-file-contents' call. Suggested improvements: * TRAMP should issue just one `stat' command to find out most of the things about a file: whether it exists, if it is a directory, its real name when dereferencing links and whatever stats it is used to find now; from `$ stat --help' this seems to be possible. In other words, TRAMP shouldn't use simple commands like `test -e': any ping, even nominal, will negate any gains from using a tad faster command. Instead, if it needs to find anything about a file, it should ask the remote about as many things as possible in one go: it is very likely that the additional information will be needed soon and even if not, this is basically free compared to ping anyway. * TRAMP code should prefer the approach "try do something and handle resulting errors" where possible. For example, don't check if the file exists, try to read it right away and handle failures properly. Code like `(when (file-exists-p ...) do-something)' adds an unnecessary command call and creates a racing condition anyway. Also, error-free requests should be more frequent, so they should be the main optimization goal. I'm not sure if it is applicable to TRAMP itself and doesn't come from a higher level, though. --0000000000004ef69f05e2c184de Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Emacs version:=C2=A0GNU Emacs 28.1.50 (build 1, x86_6= 4-pc-linux-gnu, GTK+ Version 2.24.33, cairo version 1.16.0) of 2022-05-24

I have an internet connection with a relatively lar= ge ping, so TRAMP for me works quite slowly. I notice this when editing and= saving files (also when Emacs stores backup copies), with Magit and otherw= ise.

I tried to understand why it is _so_ slow for= me. I have found out variable `tramp-verbose' and used it to figure ou= t which commands TRAMP executes on the remote machine. For testing, I used = a real library's (Logview's) buffer refreshing command=C2=A0 that b= oils down to this:

=C2=A0 =C2=A0 (with-temp-buffer=
=C2=A0 =C2=A0 =C2=A0 (insert-file-contents the-file nil from-som= ewhere-in-the-middle nil)
=C2=A0 =C2=A0 =C2=A0 ; do something= about the data
=C2=A0 =C2=A0 =C2=A0 )

I= n other words, it inserts into a buffer not the whole file, but a part of i= t (the reason is irrelevant here).

Well, this resu= lts in _eleven_ commands from TRAMP! Here they are:

1) check if connection is alive (`echo are you awake');
2) = test if the file exists;
3) creating a temporary file for the chu= nk to be inserted; I guess it tries until it finds an unused filename, e.g.= here it seems to be done after `test -e /tmp/tramp.OD3cCu', which does= n't exist;
4) 'touch' on the temporary file, presumab= ly to create it;
5) 'chmod' on the temporary, presumably = so that other users cannot read it;
6) copying the requested chun= k from the full file into the temporary (using `dd');
7) find= ing the real name of the temporary with `readlink';
8) findin= g attributes of the temporary with `stat';
9) gzipping the te= mporary for transmition from the remote to the local machine;
10)= testing if the temporary is a directory (WTF?);
11) removing the= temporary.

I guess it should be obvious that this= is a bit too much for one `insert-file-contents' call.

<= /div>
Suggested improvements:

* TRAMP should i= ssue just one `stat' command to find out most of the things about a fil= e: whether it exists, if it is a directory, its real name when dereferencin= g links and whatever stats it is used to find now; from `$ stat --help'= this seems to be possible. In other words, TRAMP shouldn't use simple = commands like `test -e': any ping, even nominal, will negate any gains = from using a tad faster command. Instead, if it needs to find anything abou= t a file, it should ask the remote about as many things as possible in one = go: it is very likely that the additional information will be needed soon a= nd even if not, this is basically free compared to ping anyway.
<= br>
* TRAMP code should prefer the approach "try do somethin= g and handle resulting errors" where possible. For example, don't = check if the file exists, try to read it right away and handle failures pro= perly. Code like `(when (file-exists-p ...) do-something)' adds an unne= cessary command call and creates a racing condition anyway. Also, error-fre= e requests should be more frequent, so they should be the main optimization= goal. I'm not sure if it is applicable to TRAMP itself and doesn't= come from a higher level, though.
--0000000000004ef69f05e2c184de-- From debbugs-submit-bounces@debbugs.gnu.org Sat Jul 02 11:58:54 2022 Received: (at 56342) by debbugs.gnu.org; 2 Jul 2022 15:58:54 +0000 Received: from localhost ([127.0.0.1]:42616 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1o7fWD-0000l1-T3 for submit@debbugs.gnu.org; Sat, 02 Jul 2022 11:58:54 -0400 Received: from mout.gmx.net ([212.227.17.21]:55131) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1o7fVx-0000kX-Vi for 56342@debbugs.gnu.org; Sat, 02 Jul 2022 11:58:48 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.net; s=badeba3b8450; t=1656777506; bh=MtM60nc1KMdSxWxSNJRbSiLze047/ykc1JWPMZZodhA=; h=X-UI-Sender-Class:From:To:Cc:Subject:References:Date:In-Reply-To; b=ZGgGx+frSJ7N9//b9DqHx0MhOWQpZ8v1uXYlqZ/2zOJWjwWgnQWMhPFKM9xzQeLQF X7QG/lM6uye1j2+eJkIGMjQ96pKW5+S5SdYh4oHJ+vrvrrzZFHOThkF7ZaBn4lD83y +el+OKdfXBkL5wUJCaFzTpwWJO2+8wEFxwTMK2vA= X-UI-Sender-Class: 01bb95c1-4bf8-414a-932a-4f6e2808ef9c Received: from gandalf.gmx.de ([79.140.124.34]) by mail.gmx.net (mrgmx105 [212.227.17.168]) with ESMTPSA (Nemesis) id 1Mdvqg-1naGEm2Jx0-00b0Sv; Sat, 02 Jul 2022 17:58:26 +0200 From: Michael Albinus To: Paul Pogonyshev Subject: Re: bug#56342: TRAMP (sh) issues way too many commands, thus being very slow over high-ping networks References: Date: Sat, 02 Jul 2022 17:58:25 +0200 In-Reply-To: (Paul Pogonyshev's message of "Fri, 1 Jul 2022 19:14:08 +0200") Message-ID: <8735fjh5ge.fsf@gmx.de> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/29.0.50 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Provags-ID: V03:K1:vYFVPzj1cDr+YNIPnrd0wNeQwua1NszhmPQTBbcOcERKFm9lCpZ wVslu824VsW726KEL9kzzXrs8ILrUdEwqRDje1wf1kBihRij4c6RvtmlOYOypg5dWm7K+fR DXTuYFdA4u2mX5SwIlWvsZC00awDW9QiPESIfyPu27PgM2TU6hDawX7GZ5d3km0B1VsAuMq rtO+vj33ubDWyiuSlCudQ== X-Spam-Flag: NO X-UI-Out-Filterresults: notjunk:1;V03:K0:CGXOlt0FqJo=:n7XoCTZWglEetoSHYmeQTM nb/Ls3/SbB1E1kXwA21ddZqWoCVdWAm8zyoEsY5WMTF711xBifq6liE2EEVqhlKi/Y07QWyOw qgBW8nD7hoNcAdZzCmE0L5xvyC+Ne7cUiu0ME+5a0xwmRh2MLt76cvQfcgybRfuAu18VItRHR M3+/xrgfKjeDxZl8O8WFJ7nE4XLOU/pMYtCb/aNg/ga1OjLI2bT0qidrFNi+/5My1GLiTPfPw 7KSkzqmCkm7waWpA6VVqAKozTXJ33SMeaVvuCeEFdIX73PUwbtKgY818maP8NBei3bmOnxskt nttMzQ43QmKqpP5S/IGf5XnpJZYmtEAUiTMfNpQd2kN/+j51g7oYz74SQEiofr5fuleElYjKh lar+9G905WvG/je1sktbp/oJcTX5l8tkcWfGgZ4SkOyVZeN9zs4kPkPNGWrG28zmZ7IjWnP9j RdVPhkoOBlHUdzeGb9YNf88NRg8Uiq4AUVMzHlnN0wUakwrU2qdgeexF8Nz7TpGuRStpcHk2F g12nUFawN9530vn08fsgxme5waLOnp4silRt/Ut0q9qy9zQ8bONQf4NQjHZgrzJ3DOJOUq/4e loeVp7/T7BOG/dWXTC+j8c5y7ak245GJ7Cs0h/+5uSf0ZcACPrtK6CL/H0n5iePqC5YrODhM0 jlfBmo5GgJgwmm0IgetD4i1kwiALd3yypOStHgtepH34WLn4leKtl+LQfqyu9yheSkvlPFqfa vJ4Y4Ai8/v8UT6bjigfEF9X1+l7XpwrPGxRP7oEPAIfiAxL3CteJV2gWvCxQwqD+7Jfg0P0Eo A8yrD0bwyAjEAXue42RZvOv3Bt1LrIYjz7FiT7q22G/qKBz+uuDvccy2j99AdcpPX1hjwRe6o nqYvsncCI6+pskkgh3H8OM0w3suS+IXjYh8i/4AbsP6JJ8v0Hn1gsi9uHfU1yLgAoFr9vmyj+ WioKL7arQTpvBZgD5nr00nAsWelGve8BqcGwOJ3y4gtZ88gXNmHRFkJwtOG7iSJib5gUhqjZg ZLPUkArmBVPQu/s7f8Qnw/RVMqqvssTn+0TP4inqsvZBZwh8tAe8mf1xyJ7RnejMVJIXUkPkY vJJ8pyEKwdHIhIfOSTWpGi7w8odwDOt3raBkQPhsXpusNxwjK7gUKZrHw== X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 56342 Cc: 56342@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) Paul Pogonyshev writes: Hi Paul, > 1) check if connection is alive (`echo are you awake'); > 2) test if the file exists; > 3) creating a temporary file for the chunk to be inserted; I guess it > tries until it finds an unused filename, e.g. here it seems to be done > after `test -e /tmp/tramp.OD3cCu', which doesn't exist; > 4) 'touch' on the temporary file, presumably to create it; > 5) 'chmod' on the temporary, presumably so that other users cannot > read it; > 6) copying the requested chunk from the full file into the temporary > (using `dd'); > 7) finding the real name of the temporary with `readlink'; > 8) finding attributes of the temporary with `stat'; > 9) gzipping the temporary for transmition from the remote to the local > machine; > 10) testing if the temporary is a directory (WTF?); > 11) removing the temporary. > > I guess it should be obvious that this is a bit too much for one > `insert-file-contents' call. In general, I agree. However, some of the commands are caused by primitive file operations, like file-exists-p. Tramp cannot know what will be the next call, and it doesn't have all the opportunities to optimize, compared with the overall picture you see in the eleven steps. > Suggested improvements: > > * TRAMP should issue just one `stat' command to find out most of the > things about a file: whether it exists, if it is a directory, its real > name when dereferencing links and whatever stats it is used to find > now; from `$ stat --help' this seems to be possible. In other words, > TRAMP shouldn't use simple commands like `test -e': any ping, even > nominal, will negate any gains from using a tad faster command. > Instead, if it needs to find anything about a file, it should ask the > remote about as many things as possible in one go: it is very likely > that the additional information will be needed soon and even if not, > this is basically free compared to ping anyway. Not all remote hosts carry a stat command, and not all existing stat's are GNU compatible. But yes, if possible, Tramp shall gather as much information in one run, and cache the results for further use. I will see what could be done. Will come back with a proposal next days (note that this will be for Emacs 29, ie git master). > * TRAMP code should prefer the approach "try do something and handle > resulting errors" where possible. For example, don't check if the file > exists, try to read it right away and handle failures properly. Code > like `(when (file-exists-p ...) do-something)' adds an unnecessary > command call and creates a racing condition anyway. Also, error-free > requests should be more frequent, so they should be the main > optimization goal. I'm not sure if it is applicable to TRAMP itself > and doesn't come from a higher level, though. Indeed, this is not Tramp's responsibility. Tramp is a stupid library. If there is a call for file-exists-p, it must return the answer. It doesn't know what will be the next request. So I'm rather pesimistic that Tramp can improve here. Best regards, Michael. From debbugs-submit-bounces@debbugs.gnu.org Sat Jul 02 11:59:20 2022 Received: (at control) by debbugs.gnu.org; 2 Jul 2022 15:59:20 +0000 Received: from localhost ([127.0.0.1]:42620 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1o7fWi-0000m5-A0 for submit@debbugs.gnu.org; Sat, 02 Jul 2022 11:59:20 -0400 Received: from mout.gmx.net ([212.227.15.18]:56227) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1o7fWe-0000ls-Rl for control@debbugs.gnu.org; Sat, 02 Jul 2022 11:59:18 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.net; s=badeba3b8450; t=1656777550; bh=H+olClqt9cZzEb1MWHjSdj16jYpS3GluRkoNjw4SMzY=; h=X-UI-Sender-Class:Date:To:From:Subject; b=h8NwlG0cGQxFGIDM7e2PwouQomP9aJRSf4e6PfNTDgq1ziE5pFS6eZUgKZ3N1wIff N6D7Icp0ZcGT5t47Y3C8V2cIFQntpeocFY3u0dAUEUTtvIIhfunY/58QtkH/U7cSVU RYX3agJ4uii+rRUz2ToPGBVCsCTbDNOPnxgDQ5dY= X-UI-Sender-Class: 01bb95c1-4bf8-414a-932a-4f6e2808ef9c Received: from gandalf.gmx.de ([79.140.124.34]) by mail.gmx.net (mrgmx005 [212.227.17.190]) with ESMTPSA (Nemesis) id 1N7zBR-1nUweb2RRD-014yx0 for ; Sat, 02 Jul 2022 17:59:10 +0200 Date: Sat, 02 Jul 2022 17:59:09 +0200 Message-Id: <871qv3h5f6.fsf@gmx.de> To: control@debbugs.gnu.org From: Michael Albinus Subject: control message for bug #56342 X-Provags-ID: V03:K1:XXO7rm60tOqdcraQFXydRjVBZldW26QIIvf6UN58H+sbfalNkOQ u2b3m+hlqJCv2FxoQ1L8dMRTlaldgRjfHRuxWqyWp6id/1trViL3Dm1du+FNSzJe2ZrXEHL bi8pEkbDnYOyR0UyQpWmWPgSb6j7+E1TYafdwPI2hnkmMxuFNPRvR9pVc6iEYE87VS3X4vC O+vNYGhf/1fdbYrxsW0yw== X-Spam-Flag: NO X-UI-Out-Filterresults: notjunk:1;V03:K0:r3Q+6ozTWh8=:KB0IOcHZ7wthiGne01jQwG cRg5aHsyL6aLE4ZTq4rzgAJ6gUy1P5ww12dQ71l5kCxwvkIbeF47vUyAuJAAVDZ4FB6im9p6b woASeaSuHLVSwLr/+aGQGlp7/kLJvaRiGYgLGMZjLe7XA0qkttwRMg45Oc17oM+297Tyqyyq4 8Z2laYCXDBqlgF2jqhC/HXpEyRaw8nNZVMWMEm/xLdL59qrLm9mJZXX1D/czAd80hynyX4ipq xfvYpiq1M2OcQa6jppUjUeNqydVoOcBAvt6bg2MLIceZFdnf2ExQk06VfjtGsyl1cGjkgEJ6v CtgP02sXK7QJ7u9BTDH9gBSYnyzbElk17/RnBBVQkSnmmPTdOToPnH4YmWwu7QdpY+xfTmH5Q Z3udkKvCFVHqFF3huDgDybsl4FxkiKNQtDIQBoMXHuev3ulb+WhRwPLlB50O52uoqDPkYeGRi PdbR3ZW+pSQ9UUqYuWofa2tp821PmvYpRQrT2+GD9sMAqFdHzkc/dukoelMp/WtrcxlTlCUI9 m6Z+m4pvK1Bx91Y3dYvMThA0NjE75AnMCNHwcgj9DWvfH8NrOYSwn149QAJQVpLYY8XWzW0XH vcaCE9yxNJGBJuERUs7ImbfEpGd7PQSh65BC5LLUxhuZATVPMqdE54RbQ9J4SQYHnbPU88lru Gt/aTMu7byht3FDH33xDg57mY0IDAvepE55TuURVZ3DPOnNQv89axdrRKsJn0Guuw6zc16t0N QIessUc7ZR7wIcuZx18RQ5+/IJ2EwEEN7Fg72KnMvPEV5piSatjzig359lpwzd47lb6z7mQ0k mNZQlPVkJkSL1wgBztRJ43mQHyGdmTtKpwvFIH7fPUjN/wgPkAjzlg7yHm9UIOrmtkTZF9aLr 49seJ74NkB6amnrBLRl/eVSuIFayixK3j97jyLSSbfeKmUXnhWM//d/W60tMRVMQQEpNNLDQa 4CEVLZA1RK7FWagpM+j/8DNy+9/5scogk1BohecQYLSqFAfhNbOTWqkULMsFj7awlGtO8tdBu Ce1C8MEhZ56LSAvslmRhOr2jvNC93walPGrg7ie28Un1DBGl9bnQdF0aO2i1UIbgmnviG2itc kG33jurBaDOnrnPsv++ZPGwqV8zOVIja6Dgm3AECbStb4ry/ZCSEs0YuQ== X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: control X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) severity 56342 wishlist quit From debbugs-submit-bounces@debbugs.gnu.org Sat Jul 02 14:14:56 2022 Received: (at 56342) by debbugs.gnu.org; 2 Jul 2022 18:14:56 +0000 Received: from localhost ([127.0.0.1]:42831 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1o7hdw-0000Gv-1K for submit@debbugs.gnu.org; Sat, 02 Jul 2022 14:14:56 -0400 Received: from mail-yw1-f176.google.com ([209.85.128.176]:35345) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1o7hds-0000Gg-8f for 56342@debbugs.gnu.org; Sat, 02 Jul 2022 14:14:54 -0400 Received: by mail-yw1-f176.google.com with SMTP id 00721157ae682-31c782f7d96so19207707b3.2 for <56342@debbugs.gnu.org>; Sat, 02 Jul 2022 11:14:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=AokJDcy7hKPJp8AABywEI8gLvXO/gAn1sDPkj8v0nvw=; b=VpyjD+nmKe/0UJF2lt5CYkrt0FbhbREJDpswwsiZaIjcRpjuhJ6nivlsYoa0ZE3A2U YVwee4XzOeDrDLnXsXutcw5DzMPxkreh9tgMLQt0OqCkW9WprtU2CEeeH6rZRVqm1XJd fi92TuLTUDNayqizT+jVx4gUAOCnjXqSVro6IJeXYjgCdtERh5uWzr9d5aGDMCLeoyEk aSRpPi3VtHsndOBfT5GyMTHX4RWbcKcamFGLCAe6KGQo+e1I4iN4EANIHWZmraJG82sg sF9+rZ5DtXVVSn9PJ9dPGYhU9MpCgMvMxKnPNDkuH6ntXmx/3BvwnwpmYQYrDQv2gbpg 5jfw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=AokJDcy7hKPJp8AABywEI8gLvXO/gAn1sDPkj8v0nvw=; b=EEOEpQ/NgrKqa0ytHXsFHYbeDM/w75z/goQhBNnPDK9XakMrbmIiasG0j4gQmfdb4X kHfqhoSSYHoWYkYpSA/PPGcNDA2IRt1Vlaf4TkYMs9nI8BFQGUN60DGvUZgNkfC3Iv4T GbqDCn75eR4/eEgveZPFnHiNJETm4GwwzyIKaEfhBBeLvOLxpjMoi84zlb6ee7ehEnHS MYvAb52yk3WlAFXTX+gtunk+R74kDIaf36Ff5YOonyjJwVkQvp1BFCbRbQFvMccIIkl0 iOetKJ2Vjh5nuIjJyl9GY+J1nkjsxkVI8ZnK97who6IHNWeoqd9AYH3ctGseNnOvBhCO ug1Q== X-Gm-Message-State: AJIora+Ja5bdNlBoZI8+8eFmRVdNR2eSyItMIl9HNivZU6LUSzFDo0Ah 9600s7U//gFqPVDGufE6rPU8EE4b6eBCgSZW1JHPnXm40+pM X-Google-Smtp-Source: AGRyM1ttpVTMFnoUhAR8kCduK8sNpw2zwPlijRyhK5gNMiLkUdDZR2UQ7726+cptSLpTWbpmeo9hS7X0gxrnSK0EFZ0= X-Received: by 2002:a81:6cd2:0:b0:31c:95d:552 with SMTP id h201-20020a816cd2000000b0031c095d0552mr23512852ywc.28.1656785686496; Sat, 02 Jul 2022 11:14:46 -0700 (PDT) MIME-Version: 1.0 References: <8735fjh5ge.fsf@gmx.de> In-Reply-To: <8735fjh5ge.fsf@gmx.de> From: Paul Pogonyshev Date: Sat, 2 Jul 2022 20:14:35 +0200 Message-ID: Subject: Re: bug#56342: TRAMP (sh) issues way too many commands, thus being very slow over high-ping networks To: Michael Albinus Content-Type: multipart/alternative; boundary="00000000000050098e05e2d67ade" X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 56342 Cc: 56342@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) --00000000000050098e05e2d67ade Content-Type: text/plain; charset="UTF-8" Some more thoughts. Why does it even need `echo are you awake'? It's a network connection, it can still fail even if it worked fine 1 ms before when you checked. So, why not just let the first command fail if the connection is dead and restart the connection if it fails in such a way as to suspect that it is dead (i.e. no output)? Maybe limit this to read commands. A way to let higher-level code avoid certain `file-exists-p' calls: add a dynamic variable that tells TRAMP to skip certain commands if the result is not available from a cache. Something similar to `process-file-side-effects'. Calling code could then do sth. like this: (when (let ((tramp-may-skip-if-not-cached `((file-exists-p unknown ,file)))) (file-exists-p file)) ; TRAMP will return t or nil if it knows or 'unknown if not cached; for local files there is no effect ...) Suggested semantics: list of (FUNCTION INSTANT-RESULT-IF-NOT-CACHED ARGUMENT...). Any element of the list with unknown function name etc. would be simply ignored. Code that doesn't let-bind this variable will behave as before. Code that cares can be optimized. Paul On Sat, 2 Jul 2022 at 17:58, Michael Albinus wrote: > Paul Pogonyshev writes: > > Hi Paul, > > > 1) check if connection is alive (`echo are you awake'); > > 2) test if the file exists; > > 3) creating a temporary file for the chunk to be inserted; I guess it > > tries until it finds an unused filename, e.g. here it seems to be done > > after `test -e /tmp/tramp.OD3cCu', which doesn't exist; > > 4) 'touch' on the temporary file, presumably to create it; > > 5) 'chmod' on the temporary, presumably so that other users cannot > > read it; > > 6) copying the requested chunk from the full file into the temporary > > (using `dd'); > > 7) finding the real name of the temporary with `readlink'; > > 8) finding attributes of the temporary with `stat'; > > 9) gzipping the temporary for transmition from the remote to the local > > machine; > > 10) testing if the temporary is a directory (WTF?); > > 11) removing the temporary. > > > > I guess it should be obvious that this is a bit too much for one > > `insert-file-contents' call. > > In general, I agree. However, some of the commands are caused by > primitive file operations, like file-exists-p. Tramp cannot know what > will be the next call, and it doesn't have all the opportunities to > optimize, compared with the overall picture you see in the eleven steps. > > > Suggested improvements: > > > > * TRAMP should issue just one `stat' command to find out most of the > > things about a file: whether it exists, if it is a directory, its real > > name when dereferencing links and whatever stats it is used to find > > now; from `$ stat --help' this seems to be possible. In other words, > > TRAMP shouldn't use simple commands like `test -e': any ping, even > > nominal, will negate any gains from using a tad faster command. > > Instead, if it needs to find anything about a file, it should ask the > > remote about as many things as possible in one go: it is very likely > > that the additional information will be needed soon and even if not, > > this is basically free compared to ping anyway. > > Not all remote hosts carry a stat command, and not all existing stat's > are GNU compatible. But yes, if possible, Tramp shall gather as much > information in one run, and cache the results for further use. > > I will see what could be done. Will come back with a proposal next days > (note that this will be for Emacs 29, ie git master). > > > * TRAMP code should prefer the approach "try do something and handle > > resulting errors" where possible. For example, don't check if the file > > exists, try to read it right away and handle failures properly. Code > > like `(when (file-exists-p ...) do-something)' adds an unnecessary > > command call and creates a racing condition anyway. Also, error-free > > requests should be more frequent, so they should be the main > > optimization goal. I'm not sure if it is applicable to TRAMP itself > > and doesn't come from a higher level, though. > > Indeed, this is not Tramp's responsibility. Tramp is a stupid > library. If there is a call for file-exists-p, it must return the > answer. It doesn't know what will be the next request. So I'm rather > pesimistic that Tramp can improve here. > > Best regards, Michael. > --00000000000050098e05e2d67ade Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Some more thoughts. Why does it even need `echo are you aw= ake'? It's a network connection, it can still fail even if it worke= d fine 1 ms before when you checked. So, why not just let the first command= fail if the connection is dead and restart the connection if it fails in s= uch a way as to suspect that it is dead (i.e. no output)? Maybe limit this = to read commands.

A way to let higher-level code avoid c= ertain `file-exists-p' calls: add a dynamic variable that tells TRAMP t= o skip certain commands if the result is not available from a cache. Someth= ing similar to `process-file-side-effects'. Calling code could then do = sth. like this:

=C2=A0 =C2=A0 (when (let ((tramp-m= ay-skip-if-not-cached `((file-exists-p unknown ,file))))
=C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 (file-exists-p file))=C2=A0 ; TRAMP will= return t or nil if it knows or 'unknown if not cached; for local files= there is no effect
=C2=A0 =C2=A0 =C2=A0 ...)

Suggested semantics: list of (FUNCTION INSTANT-RESULT-IF-NOT-CACHED A= RGUMENT...). Any element of the list with unknown function name etc. would = be simply ignored.

Code that doesn't let-bind = this variable will behave as before. Code that cares can be optimized.

Paul

On Sat, 2 Jul 2022 at 17:58, Michael Albinus= <michael.al= binus@gmx.de> wrote:
Paul Pogonyshev <pogonyshev@gmail.com> writes:

Hi Paul,

> 1) check if connection is alive (`echo are you awake');
> 2) test if the file exists;
> 3) creating a temporary file for the chunk to be inserted; I guess it<= br> > tries until it finds an unused filename, e.g. here it seems to be done=
> after `test -e /tmp/tramp.OD3cCu', which doesn't exist;
> 4) 'touch' on the temporary file, presumably to create it;
> 5) 'chmod' on the temporary, presumably so that other users ca= nnot
> read it;
> 6) copying the requested chunk from the full file into the temporary > (using `dd');
> 7) finding the real name of the temporary with `readlink';
> 8) finding attributes of the temporary with `stat';
> 9) gzipping the temporary for transmition from the remote to the local=
> machine;
> 10) testing if the temporary is a directory (WTF?);
> 11) removing the temporary.
>
> I guess it should be obvious that this is a bit too much for one
> `insert-file-contents' call.

In general, I agree. However, some of the commands are caused by
primitive file operations, like file-exists-p. Tramp cannot know what
will be the next call, and it doesn't have all the opportunities to
optimize, compared with the overall picture you see in the eleven steps.
> Suggested improvements:
>
> * TRAMP should issue just one `stat' command to find out most of t= he
> things about a file: whether it exists, if it is a directory, its real=
> name when dereferencing links and whatever stats it is used to find > now; from `$ stat --help' this seems to be possible. In other word= s,
> TRAMP shouldn't use simple commands like `test -e': any ping, = even
> nominal, will negate any gains from using a tad faster command.
> Instead, if it needs to find anything about a file, it should ask the<= br> > remote about as many things as possible in one go: it is very likely > that the additional information will be needed soon and even if not, > this is basically free compared to ping anyway.

Not all remote hosts carry a stat command, and not all existing stat's<= br> are GNU compatible. But yes, if possible, Tramp shall gather as much
information in one run, and cache the results for further use.

I will see what could be done. Will come back with a proposal next days
(note that this will be for Emacs 29, ie git master).

> * TRAMP code should prefer the approach "try do something and han= dle
> resulting errors" where possible. For example, don't check if= the file
> exists, try to read it right away and handle failures properly. Code > like `(when (file-exists-p ...) do-something)' adds an unnecessary=
> command call and creates a racing condition anyway. Also, error-free > requests should be more frequent, so they should be the main
> optimization goal. I'm not sure if it is applicable to TRAMP itsel= f
> and doesn't come from a higher level, though.

Indeed, this is not Tramp's responsibility. Tramp is a stupid
library. If there is a call for file-exists-p, it must return the
answer. It doesn't know what will be the next request. So I'm rathe= r
pesimistic that Tramp can improve here.

Best regards, Michael.
--00000000000050098e05e2d67ade-- From debbugs-submit-bounces@debbugs.gnu.org Sun Jul 03 08:16:36 2022 Received: (at 56342) by debbugs.gnu.org; 3 Jul 2022 12:16:36 +0000 Received: from localhost ([127.0.0.1]:43487 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1o7yWh-0002nI-To for submit@debbugs.gnu.org; Sun, 03 Jul 2022 08:16:36 -0400 Received: from mout.gmx.net ([212.227.15.18]:49201) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1o7yWe-0002gE-Rw for 56342@debbugs.gnu.org; Sun, 03 Jul 2022 08:16:33 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.net; s=badeba3b8450; t=1656850586; bh=Bq8goL2n1LU3p25fZjzxCenjliNxFIJnopPzgnaFLa8=; h=X-UI-Sender-Class:From:To:Cc:Subject:References:Date:In-Reply-To; b=IbSOV6fBaXy9MwUhayzqavELHxVoTUFWjS65D14Mt3pmFCgFnO9i3rNxH40kCmAyr N19wAc6ceEXYOkG5N+AuQzGQUOL3VgcFtxI8/TkXgLRL8pgAaogVyeYPwxBDXHrnj1 yYmpR/vZY01h/7ZK0DzJzyyrnbu9aJFvXlctMEtg= X-UI-Sender-Class: 01bb95c1-4bf8-414a-932a-4f6e2808ef9c Received: from gandalf.gmx.de ([213.220.148.177]) by mail.gmx.net (mrgmx005 [212.227.17.190]) with ESMTPSA (Nemesis) id 1Mk0Ne-1njfGb3mnY-00kPpU; Sun, 03 Jul 2022 14:16:26 +0200 From: Michael Albinus To: Paul Pogonyshev Subject: Re: bug#56342: TRAMP (sh) issues way too many commands, thus being very slow over high-ping networks References: <8735fjh5ge.fsf@gmx.de> Date: Sun, 03 Jul 2022 14:16:24 +0200 In-Reply-To: (Paul Pogonyshev's message of "Sat, 2 Jul 2022 20:14:35 +0200") Message-ID: <87v8sefl2f.fsf@gmx.de> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/29.0.50 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Provags-ID: V03:K1:fA846orykYZJmc49cCPmkSQuNSGii/X/NhHmuSWc1Ww8X/qRXPT LL1o1avDHZ0vLoT7y5istjnWj91Cy6C8yDqXX1qaParzH7pT+iu1EX3JOp0hvHsWLbf7nK8 Dwyj8eYLiCSnvQ51KkCHSB2dYUz5D9ky9JZR4Q2d10CoHt5QT7L4SInBC6juLF/981uAuJs VpH8GUm9jvqHPzljwoozg== X-Spam-Flag: NO X-UI-Out-Filterresults: notjunk:1;V03:K0:ovNbRz7LHho=:ALNJHY0u8kzTLOoEr9GL3y qepwazmOimzJMrTuOo7UhpK8DHMkoUCOBabDOUjCkLQx9uFDZBQi5zKhAYumKZS6y3K+FPN06 bCj+5Ulyf7+qns16KEpExki4MP+iE4KfcP/RJ+8fYpu0QZjVNHQ+AGWX9UxBxZuqdKG487APg KvQZ4kNu4U5TZxNG74I99iAy2R3gUARVLFheohj9uTpmayRvDq4T1Pa0DdIIcZQQvRkKIGzpg 7mR0wbNDDF2+OHL4qjAuWLdA2Ltvgy6xxaNPd2VXGAXTDDqyTl4vtI3imoiFrnQHck8XPENPW OCBHGDSXNqMmTYql+9xl2pyEnjFi2LNJrvM7UMMxQSrN7Y3I0YUs6fOC6jGmfUj3QkPfFO5ji viW2t2EgpPrCvQdc4c8c/TdcImxxR9kJxB1Twd0dS7tY/17BhZoTybce6zH88sE5x38bWH+Na vkVgD6ZB+eM9Oa3JxmBkwQk+FGrD+kYrbdEVkf5IfJ+tf0P/5vqtc+AitYlB6puALye+O9Kvi 8WfqnEPxoEmR8FeRSckDKlvtnoP6qv2D+V2aiP/NcFncWcroh8b5CvlURx8s7SPVwz5tSXgq6 AkBOUdtA7Wv6Ui8DoKVfPfgf686m90rHdWDs5QFQJksZtA1d2TY9thCMB1QjS/w5QDCpEY3rM hob6xqadZiANZEJ+YB3gFhegkja+FoXUZufhiMNSP51r+KF9YWziUYhsZiG8b+kSEQoTP20Yd dHVR0+9csLH5BOnZDNpyEBMVkEbbHQLm3tP79fyXvL9Ewtx2CQ/plG5FVmSfE6RQ8oJpeSlkD UYZFY0JOcxKpsqKe1iUOdNn4xggm5qykzKaXQwBottoACLrriQ1P6MYrIelXRbLEeFW+sHZk8 OjJXwzgKQt1iz3Qx0W371YXTd34Zjm0ObY3tpW7TcCJtvwfnLAzsCNaPTZoYl0YwfoH/KzgsH 3RC5KVKyI2YSJatcc4BtcTstLKavmZ+Vyhfh5RE2N0iX27p18DFNtLEcDiyLG7E/LokHLaQoH S5evzV+tXDKZdRi0KnRWb4zBZ8NPs2vqmhfq4oOn3Y1w5XNxfi3xdFbQwFanNi4JDXPI+ENgP m/Y8PDicwGN0nUSIAf8kVQjuA8onKuTZQSijuFyvs/J5focZ87HRQBKow== X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 56342 Cc: 56342@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) Paul Pogonyshev writes: Hi Paul, > Some more thoughts. Why does it even need `echo are you awake'? It's a > network connection, it can still fail even if it worked fine 1 ms > before when you checked. So, why not just let the first command fail > if the connection is dead and restart the connection if it fails in > such a way as to suspect that it is dead (i.e. no output)? Maybe limit > this to read commands. This is a sanity check. It avoids to hang in a blocked connection, because this special command is surrounded by a timeout of 10 sec. Other commands w/o this protection could hang forever. See also the comment in tramp-maybe-open-connection. > A way to let higher-level code avoid certain `file-exists-p' calls: > add a dynamic variable that tells TRAMP to skip certain commands if > the result is not available from a cache. Something similar to > `process-file-side-effects'. Calling code could then do sth. like > this: > > (when (let ((tramp-may-skip-if-not-cached `((file-exists-p unknown > ,file)))) > (file-exists-p file)) ; TRAMP will return t or nil if it > knows or 'unknown if not cached; for local files there is no effect > ...) > > Suggested semantics: list of (FUNCTION INSTANT-RESULT-IF-NOT-CACHED > ARGUMENT...). Any element of the list with unknown function name etc. > would be simply ignored. > > Code that doesn't let-bind this variable will behave as before. Code > that cares can be optimized. If a caller can live w/o a valid result of file-exists-p, it shouldn't call it. Everything else is too sophisticated and good for trouble, I believe. In general, packages shall not care what's the implementation of a given function like file-exists-p. If they care, they could still use file-remote-p in order to distinguish. > Paul Best regards, Michael. From debbugs-submit-bounces@debbugs.gnu.org Sun Jul 03 10:01:04 2022 Received: (at 56342) by debbugs.gnu.org; 3 Jul 2022 14:01:04 +0000 Received: from localhost ([127.0.0.1]:45325 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1o809o-0002bZ-9Y for submit@debbugs.gnu.org; Sun, 03 Jul 2022 10:01:04 -0400 Received: from mail-yb1-f171.google.com ([209.85.219.171]:46781) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1o809i-0002at-Gl for 56342@debbugs.gnu.org; Sun, 03 Jul 2022 10:01:02 -0400 Received: by mail-yb1-f171.google.com with SMTP id l11so12357139ybu.13 for <56342@debbugs.gnu.org>; Sun, 03 Jul 2022 07:00:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=8MQccKLGauc625dU0r+abtvxNPTltMfSoLHCk2kna38=; b=UgPa7HV/u+FHbRp0etIy+Q3z/61JybvBH9xCHhNDbJYCk/+cVCPQtV6NUfyag79WrQ s1JBM+48Fh57OAuSXPEEifRxpT9r4n57FLVJwbWWwlaWoVNX7WRLjACn18SQy/IJimx4 WeDWSnxsVykTrbVQ1K8nDFTorx69/dvC2K3X6EUkTy8vK4LKpvGJcpu9knw2bSS8xs92 v+crZLmaOQReP27BPsdtj7zDBu5zJPm3UlByWT/jCl0LPQ4Pk4U7Xq3gcyMfscPZ/Oup fHXSSDEdEkUgzlkKFPp1K4BncbCDKA/s7AVv4lwVuOYlTNEAStDJ9p10ITc7g1IviBjF SuVA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=8MQccKLGauc625dU0r+abtvxNPTltMfSoLHCk2kna38=; b=kI+G+g6u7uNYWt7dj3S6FNttHMyFIotduW8X78otrdN6R/GEs0VAy7HfBNWqjEiatc 5f1LBnLhi4nFV30w25KPBWuPbPRYf63zgEqrRp7DyFLbGh1W6pYrT+Z0Pmk/Ne0aU1bh pcG78YNDmFxPIGsqB43YETBSusRpjpql6DSWQc1KzXeAWoMp6m/BZ33WvKsnxTS+oS3O aXzUlwlS4F868HkPse+5u3UWCqBoNdxKhmQXZo5lqz7SiNzTzh4p4kFEzVLoxddRsdeu 4aqVyReykkaQxxuM5BEl6VeIk0oey7B/xZwumMTUkiVWXdXti7VvUlGfI7YjL/BetJ1e u81w== X-Gm-Message-State: AJIora8WKMNraJM/w0KJoq9JtjI1vktQcNw7hX47HcIV2MwDWQf3md48 EBHiJMp6qBQNDNBhHJFsANAePYCgexSOI6fiPqqHNHmbSdpK X-Google-Smtp-Source: AGRyM1uw/Ncn67MUAS3Z6Wo48buIOUELe8K+tSZZTiHBZnhc8l9Cjhe0gW5RErIIKuwQnMrtcRKGSFv3lS2GlOgahlo= X-Received: by 2002:a25:870e:0:b0:66c:b173:f047 with SMTP id a14-20020a25870e000000b0066cb173f047mr25644243ybl.635.1656856852723; Sun, 03 Jul 2022 07:00:52 -0700 (PDT) MIME-Version: 1.0 References: <8735fjh5ge.fsf@gmx.de> <87v8sefl2f.fsf@gmx.de> In-Reply-To: <87v8sefl2f.fsf@gmx.de> From: Paul Pogonyshev Date: Sun, 3 Jul 2022 16:00:41 +0200 Message-ID: Subject: Re: bug#56342: TRAMP (sh) issues way too many commands, thus being very slow over high-ping networks To: Michael Albinus Content-Type: multipart/alternative; boundary="000000000000267a0705e2e70c21" X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 56342 Cc: 56342@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) --000000000000267a0705e2e70c21 Content-Type: text/plain; charset="UTF-8" > This is a sanity check. It avoids to hang in a blocked connection, > because this special command is surrounded by a timeout of 10 sec. Other > commands w/o this protection could hang forever. See also the comment in > tramp-maybe-open-connection. Maybe you could then single out commands that are supposed to return result nearly-instantly anyway, e.g. `stat' or `test' (though the latter I have suggested to replace with `stat' before). If they fail to produce the result in 10 seconds, the connection can be assumed dead, just as with `echo are you awake'. Commands that involve transmitting (potentially) large amount of data this or that way can work as now. So, you would usually avoid one extra command (which with a high ping means sth. like 0.2 s, already pretty noticeable in UI) and achieve the same result. Sanity checks are good. but not if they visibly slow down normal operation. > If a caller can live w/o a valid result of file-exists-p, it shouldn't > call it. Everything else is too sophisticated and good for trouble, I believe. It might be easier to convince the rest of Emacs developers to use an extra variable to possibly skip `file-exists-p' on only remote files than to remove the call altogether. Though in either case it is required that `file-missing' error is handled properly which is, incidentally, easier to test if the call to `file-exists-p' is dropped. A target here would be `insert-file-contents' and whatever it calls, and this is a part of Emacs, not some external library. Yet another idea: removing temporary should be done asynchronously (I haven't checked, maybe it's already the case, but likely not). The caller doesn't really care about call result and even if it has succeeded. Again, not sure if this is easy to achieve interface-wise, maybe it's from a higher level. But I suspect it's something like `with-temp-file' and can be optimized: one-time optimization are worth it even if that costs readability. Paul On Sun, 3 Jul 2022 at 14:16, Michael Albinus wrote: > Paul Pogonyshev writes: > > Hi Paul, > > > Some more thoughts. Why does it even need `echo are you awake'? It's a > > network connection, it can still fail even if it worked fine 1 ms > > before when you checked. So, why not just let the first command fail > > if the connection is dead and restart the connection if it fails in > > such a way as to suspect that it is dead (i.e. no output)? Maybe limit > > this to read commands. > > This is a sanity check. It avoids to hang in a blocked connection, > because this special command is surrounded by a timeout of 10 sec. Other > commands w/o this protection could hang forever. See also the comment in > tramp-maybe-open-connection. > > > A way to let higher-level code avoid certain `file-exists-p' calls: > > add a dynamic variable that tells TRAMP to skip certain commands if > > the result is not available from a cache. Something similar to > > `process-file-side-effects'. Calling code could then do sth. like > > this: > > > > (when (let ((tramp-may-skip-if-not-cached `((file-exists-p unknown > > ,file)))) > > (file-exists-p file)) ; TRAMP will return t or nil if it > > knows or 'unknown if not cached; for local files there is no effect > > ...) > > > > Suggested semantics: list of (FUNCTION INSTANT-RESULT-IF-NOT-CACHED > > ARGUMENT...). Any element of the list with unknown function name etc. > > would be simply ignored. > > > > Code that doesn't let-bind this variable will behave as before. Code > > that cares can be optimized. > > If a caller can live w/o a valid result of file-exists-p, it shouldn't > call it. Everything else is too sophisticated and good for trouble, I > believe. > > In general, packages shall not care what's the implementation of a given > function like file-exists-p. If they care, they could still use > file-remote-p in order to distinguish. > > > Paul > > Best regards, Michael. > --000000000000267a0705e2e70c21 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
> This is a sanity check. It avoids to hang in a blocke= d connection,
>=C2=A0because this special command is surrounded by a = timeout of 10 sec. Other
>=C2=A0commands w/o this protection could ha= ng forever. See also the comment in
>=C2=A0tramp-maybe-open-connectio= n.
Maybe you could then single out commands that are supposed to r= eturn result nearly-instantly anyway, e.g. `stat' or `test' (though= the latter I have suggested to replace with `stat' before). If they fa= il to produce the result in 10 seconds, the connection can be assumed dead,= just as with `echo are you awake'. Commands that involve transmitting = (potentially) large amount of data this or that way can work as now. So, yo= u would usually avoid one extra command (which with a high ping means sth. = like 0.2 s, already pretty noticeable in UI) and achieve the same result.

Sanity checks are good. but not if they visibly slo= w down normal operation.

> If a caller can live= w/o a valid result of file-exists-p, it shouldn't
> call it. E= verything else is too sophisticated and good for trouble, I believe.

It might be easier to convince the rest of Emacs developer= s to use an extra variable to possibly skip `file-exists-p' on only rem= ote files than to remove the call altogether. Though in either case it is r= equired that `file-missing' error is handled properly which is, inciden= tally, easier to test if the call to `file-exists-p' is dropped. A targ= et here would be `insert-file-contents' and whatever it calls, and this= is a part of Emacs, not some external library.

Ye= t another idea: removing temporary should be done asynchronously (I haven&#= 39;t checked, maybe it's already the case, but likely not). The caller = doesn't really care about call result and even if it has succeeded. Aga= in, not sure if this is easy to achieve interface-wise, maybe it's from= a higher level. But I suspect it's something like `with-temp-file'= and can be optimized: one-time optimization are worth it even if that cost= s readability.

Paul


=
On Sun, 3 = Jul 2022 at 14:16, Michael Albinus <michael.albinus@gmx.de> wrote:
Paul Pogonyshev <pogonyshev@gmail.com> writes:

Hi Paul,

> Some more thoughts. Why does it even need `echo are you awake'? It= 's a
> network connection, it can still fail even if it worked fine 1 ms
> before when you checked. So, why not just let the first command fail > if the connection is dead and restart the connection if it fails in > such a way as to suspect that it is dead (i.e. no output)? Maybe limit=
> this to read commands.

This is a sanity check. It avoids to hang in a blocked connection,
because this special command is surrounded by a timeout of 10 sec. Other commands w/o this protection could hang forever. See also the comment in tramp-maybe-open-connection.

> A way to let higher-level code avoid certain `file-exists-p' calls= :
> add a dynamic variable that tells TRAMP to skip certain commands if > the result is not available from a cache. Something similar to
> `process-file-side-effects'. Calling code could then do sth. like<= br> > this:
>
>=C2=A0 =C2=A0 =C2=A0(when (let ((tramp-may-skip-if-not-cached `((file-e= xists-p unknown
> ,file))))
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(file-exists-p file))= =C2=A0 ; TRAMP will return t or nil if it
> knows or 'unknown if not cached; for local files there is no effec= t
>=C2=A0 =C2=A0 =C2=A0 =C2=A0...)
>
> Suggested semantics: list of (FUNCTION INSTANT-RESULT-IF-NOT-CACHED > ARGUMENT...). Any element of the list with unknown function name etc.<= br> > would be simply ignored.
>
> Code that doesn't let-bind this variable will behave as before. Co= de
> that cares can be optimized.

If a caller can live w/o a valid result of file-exists-p, it shouldn't<= br> call it. Everything else is too sophisticated and good for trouble, I belie= ve.

In general, packages shall not care what's the implementation of a give= n
function like file-exists-p. If they care, they could still use
file-remote-p in order to distinguish.

> Paul

Best regards, Michael.
--000000000000267a0705e2e70c21-- From debbugs-submit-bounces@debbugs.gnu.org Sun Jul 03 14:47:22 2022 Received: (at 56342) by debbugs.gnu.org; 3 Jul 2022 18:47:22 +0000 Received: from localhost ([127.0.0.1]:45510 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1o84cs-0001oX-0n for submit@debbugs.gnu.org; Sun, 03 Jul 2022 14:47:22 -0400 Received: from mout.gmx.net ([212.227.15.18]:36119) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1o84cp-0001oJ-Gk for 56342@debbugs.gnu.org; Sun, 03 Jul 2022 14:47:20 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.net; s=badeba3b8450; t=1656874032; bh=avHaiaUxfjUxgDetkRUCbin0jIBb51PTeasHARgeyTQ=; h=X-UI-Sender-Class:From:To:Cc:Subject:References:Date:In-Reply-To; b=S+JDwa4ssNjRTuhCM/1jgkTeTXOx87xGU8wN1tltfayAb/GaWnBiHAgwjqeef0+4E ljgaZcLqUQfr2JxXuY3PHjuNPbenVf7EYbXpC6Ct9id5PgkZ1ywPSOyIIT7IlcxRyw pJAQIvoRBTz1Rg0dNZ+2ro8JLFiAxU3+wjrJP+yM= X-UI-Sender-Class: 01bb95c1-4bf8-414a-932a-4f6e2808ef9c Received: from gandalf.gmx.de ([213.220.148.177]) by mail.gmx.net (mrgmx005 [212.227.17.190]) with ESMTPSA (Nemesis) id 1N6KUd-1nWBi50iU5-016dK6; Sun, 03 Jul 2022 20:47:12 +0200 From: Michael Albinus To: Paul Pogonyshev Subject: Re: bug#56342: TRAMP (sh) issues way too many commands, thus being very slow over high-ping networks References: <8735fjh5ge.fsf@gmx.de> <87v8sefl2f.fsf@gmx.de> Date: Sun, 03 Jul 2022 20:47:10 +0200 In-Reply-To: (Paul Pogonyshev's message of "Sun, 3 Jul 2022 16:00:41 +0200") Message-ID: <874jzyc9u9.fsf@gmx.de> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/29.0.50 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Provags-ID: V03:K1:AXFntJXEX1BgdvCrzNfIfkOe393XcCFTQneHiG9OsRjupM0OQvb TSr/L3wKL4esZTb4vV8swdG/tnoGpe1Zmr3/cxXFUXXHrpTxAPKaieDIhXiFw/GqTK0Wm16 yO8v6QS4b0szZbhd+pquri23pCr9QQBTFUNQFSXvG4JuEvKdJF75hGXkYWIWSHcXJ4c7RHd USvQmw4JBkPT8sjfMahWg== X-Spam-Flag: NO X-UI-Out-Filterresults: notjunk:1;V03:K0:hN7A+WAmyrs=:Gf29JUJT7XqHPRFolhI0yW PP5SKT26g5NUGLy3un6yPfsNL350DIxGSB3E5Fcfr+sv/3JgJ1x/oyUJZgt7FGxjnzGiY5TgX ORVfXPn+kWr2odiCTsknZwTEJ63A3G2DJrQjPSVsDKw4MS6SkD7LzMCcgRNf9AAn3bpJqqCOQ 6EKlXiPS4BfGyGL/YIsVRnWyG5OabPPQT34B9hl+BRCaRFLyKncZepu1kREgV8AAsAX5Tzz4R VWMcc4iGFe0QkA3QJm3krz9bT0DUDc5qINbUqzjP0IS5+tEtZ1njLAzwPxo2b6Q+sJzUQ11om mMda0lfQE2/oxFsbiGIox6CDHv/WSw4UfirsmN/+eXfRDcKJ7VNGMFqZNQ4aisGM5nyWheXsM AGzL/Io0tSJECSvcFND2sACKi4bsKZVO6SptJGom3emuk2s2jzGajhnaOq0bg11lgjJlwPDyQ eL8out5lpHK1Z1mMhdhTzOhkrP/tYpCakhqxP+CpVL7FOAmXwAA6i29odxQ3zQdkvqZG4Rb2k 1lYf9OZCKHmzCepUgPVpj4y1/SrvMXbHgCoscQfSeLbinsyRQAvh4ZXelBdxjMwtShZJl3Izh aADwEcpiDPpw4fzxNUdynG/IVHAF6LpJ5+WTiW0rkQE0JT0DOQig2DxlUohIOnmSVeqSn5oTD zAH7HQG21FDRq23MboOOnjX9iJbRR143T/3+OomGRLElYipVoYpkdrEW+Bce2iuBPdTOeRp1U z68Ca/RcC0X6zR1j9H7FHEDV9F/wgAoo9XvuNNdVn7RQoBqCkbTRThnaR+BxADa7kI+I7aQPj Vkwryq4E5AMKHJLnOBtSg6uSDT+JNM2T2JQEcfi7EvQoB47NrE8UfgaSONnzxDAQFszAxAFr0 cY3jVMTm/2emEyYaL3WJ0qHvOHhBAcUiEI44yFCIjqSoybtZBW96RZFZT9gvmD46ETOQkk7j9 Q2Nn3DwLMlFaNNG9DmcuON6jHsV6vpahU1cqILcJCX7vEEaGV6diY/0rG3adGXvVmW0mMD62G C/l7lXFs6pE45hzpmkvQs9IgxCxW8JZFPBGoea6SBquV9SPvOw++rSP4exA4vGY0mikYxguel qhXVYIBJyDWHmf4FK/5FCCMcyMGH133iWdQlwwur72YgLt0xjL8VjdbEw== X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 56342 Cc: 56342@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) Paul Pogonyshev writes: Hi Paul, > Maybe you could then single out commands that are supposed to return > result nearly-instantly anyway, e.g. `stat' or `test' (though the > latter I have suggested to replace with `stat' before). If they fail > to produce the result in 10 seconds, the connection can be assumed > dead, just as with `echo are you awake'. Commands that involve > transmitting (potentially) large amount of data this or that way can > work as now. So, you would usually avoid one extra command (which with > a high ping means sth. like 0.2 s, already pretty noticeable in UI) > and achieve the same result. > > Sanity checks are good. but not if they visibly slow down normal > operation. I will see whether I can do something along these lines. > It might be easier to convince the rest of Emacs developers to use an > extra variable to possibly skip `file-exists-p' on only remote files > than to remove the call altogether. Though in either case it is > required that `file-missing' error is handled properly which is, > incidentally, easier to test if the call to `file-exists-p' is > dropped. A target here would be `insert-file-contents' and whatever it > calls, and this is a part of Emacs, not some external library. Tramp has an own implementation of insert-file-contents, called tramp-handle-insert-file-contents. And indeed, file-exists-p is called in order to raise the file-missing error if needed. I have no idea how we could generate this otherwise. Perhaps changing the order: First try to insert the file contents, and if this errs out, a call with file-exists-p in order to raise the error. I'll play with this. OTOH, with longer cache expiration time (see below), file-exists-p doesn't cost a roundtrip. > Yet another idea: removing temporary should be done asynchronously (I > haven't checked, maybe it's already the case, but likely not). The > caller doesn't really care about call result and even if it has > succeeded. Again, not sure if this is easy to achieve interface-wise, > maybe it's from a higher level. But I suspect it's something like > `with-temp-file' and can be optimized: one-time optimization are worth > it even if that costs readability. Doing it asynchronously would require a second connection to the remote host. Performance would rather degrade. One thing you could do w/o code change is to change the value of remote-file-name-inhibit-cache. Its default value is 10, meaning caches expire already after 10 seconds. With your slow connection, a higher value should help already. > Paul Best regards, Michael. From debbugs-submit-bounces@debbugs.gnu.org Sun Jul 03 15:53:14 2022 Received: (at 56342) by debbugs.gnu.org; 3 Jul 2022 19:53:15 +0000 Received: from localhost ([127.0.0.1]:45521 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1o85ec-0003Qy-BE for submit@debbugs.gnu.org; Sun, 03 Jul 2022 15:53:14 -0400 Received: from mail-yb1-f175.google.com ([209.85.219.175]:40624) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1o85ea-0003Ql-S9 for 56342@debbugs.gnu.org; Sun, 03 Jul 2022 15:53:13 -0400 Received: by mail-yb1-f175.google.com with SMTP id o2so8160367yba.7 for <56342@debbugs.gnu.org>; Sun, 03 Jul 2022 12:53:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=Y6nTzgBZF6uyYLCi3FOGFGUC5Pr8bjYuApmIWbAbww8=; b=OIoVpAfy4ZXc+2nmPaj7dKMAvXsYJzgeGuUmyAC6uCuE1QO4F+hm4VOsGNEyT0f3Pt WntbOExra7rBz+qP0fzNKeIQZIxT9ceIv3Ig18zOzBa2NRlm/TvFZsi8rcisJ2C5KF69 YJG+82MBKdwCkR1oxAZAKz4tm2hY8n7fndRzZb6cyuM/+i0ZUvQTdCJX6rTphPgoSOL0 etzQ+hwQUIiODi5DD9W5B79EaN7zMAuuShcbuKesowC3TCZIuYAW2eXqp24jT7rNCaM3 KEMkSgB4LAuxhVhd9YFr8afuhEKWugMiBacGW6AzgiRIRiJNmOo4R7g0uuvkbA+434o9 CKvA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=Y6nTzgBZF6uyYLCi3FOGFGUC5Pr8bjYuApmIWbAbww8=; b=lgNsrIL2slheyDyWmt9HxfWcyCax/ZC0eqrcvDYjnLKzciypfDLSXkwqqdHMKn+RNz HvChPcyVIvRNH8KI2zrHhbKd1Ux+G0lIs6qrNYp3S1Z79YIcCih4LvwdVDaygVgbBRQ5 Li51sMDiGDzJuDYXEM+q60wcgU3jr7gaTqgWIhtWFZ+gmHWO9RaZw/bNrK2d768gdaHI 75FKSwKwFsPDB0J4SHw8Ksx3+Lj/bIPc3rsz3ou3nGNlmUHDK/Ocwu7BMDJjTW2Z9cW5 9RF4xmlZgsyUrZc/opbiz6bp4yrCwGTmTlA+grP2iCRa7OMNXzqxgYDBlneOUa7f+C15 ZOAQ== X-Gm-Message-State: AJIora+ZCdaEK4HZVM+1fg5+emjuSbM/VjU5+/mHOxqKTEHAOE2adYEA /AzHOwXgBghOg8k+wz5f40YL8BT7JEjg2NluIw== X-Google-Smtp-Source: AGRyM1sZlDcIci6yR9h2PtXpySS3lMgI+4YA2JrY0h4X8C667yrvwSp/Hsz0DZdhIHojNdAfuSWe8a1lbtopvVx5QfU= X-Received: by 2002:a25:6ed5:0:b0:669:8b84:bb57 with SMTP id j204-20020a256ed5000000b006698b84bb57mr26798090ybc.227.1656877986971; Sun, 03 Jul 2022 12:53:06 -0700 (PDT) MIME-Version: 1.0 References: <8735fjh5ge.fsf@gmx.de> <87v8sefl2f.fsf@gmx.de> <874jzyc9u9.fsf@gmx.de> In-Reply-To: <874jzyc9u9.fsf@gmx.de> From: Paul Pogonyshev Date: Sun, 3 Jul 2022 21:52:55 +0200 Message-ID: Subject: Re: bug#56342: TRAMP (sh) issues way too many commands, thus being very slow over high-ping networks To: Michael Albinus Content-Type: multipart/alternative; boundary="000000000000d9855f05e2ebf70a" X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 56342 Cc: 56342@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) --000000000000d9855f05e2ebf70a Content-Type: text/plain; charset="UTF-8" > I will see whether I can do something along these lines. Thank you. I'm just spewing out ideas here, you know the code better and what will and will not work. > Perhaps changing the order: First try > to insert the file contents, and if this errs out, a call with > file-exists-p in order to raise the error. I'll play with this. Yes, this would be a better way from my point of view. I don't care how slowly (unless it is like 50 times slower) it fails, that is an unlikely case. The most likely one, when the file exists, should be the benchmark. > Doing it asynchronously would require a second connection to the remote > host. Performance would rather degrade. Maybe not really asynchronously, just let it return early, not waiting for the result? I'm not sure how TRAMP receives responses, but can it just keep executing commands sequentially, as now, but give control back to the caller in case of commands where the result doesn't really matter (cleanup, e.g. deleting a temporary file). Of course, this means that if an "important" command is issued right away, it has to wait for the response to the previous inessential command. But when such an inessential command is the last in a batch, this waiting would be effectively merged with Emacs' idling in the normal UI command loop, making things more responsive for the user. > One thing you could do w/o code change is to change the value of > remote-file-name-inhibit-cache. Thank you for the hint, will try. Paul On Sun, 3 Jul 2022 at 20:47, Michael Albinus wrote: > Paul Pogonyshev writes: > > Hi Paul, > > > Maybe you could then single out commands that are supposed to return > > result nearly-instantly anyway, e.g. `stat' or `test' (though the > > latter I have suggested to replace with `stat' before). If they fail > > to produce the result in 10 seconds, the connection can be assumed > > dead, just as with `echo are you awake'. Commands that involve > > transmitting (potentially) large amount of data this or that way can > > work as now. So, you would usually avoid one extra command (which with > > a high ping means sth. like 0.2 s, already pretty noticeable in UI) > > and achieve the same result. > > > > Sanity checks are good. but not if they visibly slow down normal > > operation. > > I will see whether I can do something along these lines. > > > It might be easier to convince the rest of Emacs developers to use an > > extra variable to possibly skip `file-exists-p' on only remote files > > than to remove the call altogether. Though in either case it is > > required that `file-missing' error is handled properly which is, > > incidentally, easier to test if the call to `file-exists-p' is > > dropped. A target here would be `insert-file-contents' and whatever it > > calls, and this is a part of Emacs, not some external library. > > Tramp has an own implementation of insert-file-contents, called > tramp-handle-insert-file-contents. And indeed, file-exists-p is called > in order to raise the file-missing error if needed. I have no idea how > we could generate this otherwise. Perhaps changing the order: First try > to insert the file contents, and if this errs out, a call with > file-exists-p in order to raise the error. I'll play with this. > > OTOH, with longer cache expiration time (see below), file-exists-p > doesn't cost a roundtrip. > > > Yet another idea: removing temporary should be done asynchronously (I > > haven't checked, maybe it's already the case, but likely not). The > > caller doesn't really care about call result and even if it has > > succeeded. Again, not sure if this is easy to achieve interface-wise, > > maybe it's from a higher level. But I suspect it's something like > > `with-temp-file' and can be optimized: one-time optimization are worth > > it even if that costs readability. > > Doing it asynchronously would require a second connection to the remote > host. Performance would rather degrade. > > One thing you could do w/o code change is to change the value of > remote-file-name-inhibit-cache. Its default value is 10, meaning caches > expire already after 10 seconds. With your slow connection, a higher > value should help already. > > > Paul > > Best regards, Michael. > --000000000000d9855f05e2ebf70a Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
> I will see whether I can do something along these lin= es.

Thank you. I'm just spewing out ideas here, = you know the code better and what will and will not work.

> Perhaps changing the order: First try
> to insert the = file contents, and if this errs out, a call with
> file-exists-p in o= rder to raise the error. I'll play with this.

Yes, t= his would be a better way from my point of view. I don't care how slowl= y (unless it is like 50 times slower) it fails, that is an unlikely case. T= he most likely one, when the file exists, should be the benchmark.

> Doing it asynchronously would require a second connec= tion to the remote
> host. Performance would rather degrade.

Maybe not really asynchronously, just let it return early,= not waiting for the result? I'm not sure how TRAMP receives responses,= but can it just keep executing commands sequentially, as now, but give con= trol back to the caller in case of commands where the result doesn't re= ally matter (cleanup, e.g. deleting a temporary file). Of course, this mean= s that if an "important" command is issued right away, it has to = wait for the response to the previous inessential command. But when such an= inessential command is the last in a batch, this waiting would be effectiv= ely merged with Emacs' idling in the normal UI command loop, making thi= ngs more responsive for the user.

> One thing you could do w/o co= de change is to change the value of
> remote-file-name-inhibit-cache.=

Thank you for the hint, will try.

<= /div>
Paul


On Sun, 3 Jul 2022 at 20:47, Michael= Albinus <michael.albinus@gmx.= de> wrote:
pogonyshev@gmail.com> writes:

Hi Paul,

> Maybe you could then single out commands that are supposed to return > result nearly-instantly anyway, e.g. `stat' or `test' (though = the
> latter I have suggested to replace with `stat' before). If they fa= il
> to produce the result in 10 seconds, the connection can be assumed
> dead, just as with `echo are you awake'. Commands that involve
> transmitting (potentially) large amount of data this or that way can > work as now. So, you would usually avoid one extra command (which with=
> a high ping means sth. like 0.2 s, already pretty noticeable in UI) > and achieve the same result.
>
> Sanity checks are good. but not if they visibly slow down normal
> operation.

I will see whether I can do something along these lines.

> It might be easier to convince the rest of Emacs developers to use an<= br> > extra variable to possibly skip `file-exists-p' on only remote fil= es
> than to remove the call altogether. Though in either case it is
> required that `file-missing' error is handled properly which is, > incidentally, easier to test if the call to `file-exists-p' is
> dropped. A target here would be `insert-file-contents' and whateve= r it
> calls, and this is a part of Emacs, not some external library.

Tramp has an own implementation of insert-file-contents, called
tramp-handle-insert-file-contents. And indeed, file-exists-p is called
in order to raise the file-missing error if needed. I have no idea how
we could generate this otherwise. Perhaps changing the order: First try
to insert the file contents, and if this errs out, a call with
file-exists-p in order to raise the error. I'll play with this.

OTOH, with longer cache expiration time (see below), file-exists-p
doesn't cost a roundtrip.

> Yet another idea: removing temporary should be done asynchronously (I<= br> > haven't checked, maybe it's already the case, but likely not).= The
> caller doesn't really care about call result and even if it has > succeeded. Again, not sure if this is easy to achieve interface-wise,<= br> > maybe it's from a higher level. But I suspect it's something l= ike
> `with-temp-file' and can be optimized: one-time optimization are w= orth
> it even if that costs readability.

Doing it asynchronously would require a second connection to the remote
host. Performance would rather degrade.

One thing you could do w/o code change is to change the value of
remote-file-name-inhibit-cache. Its default value is 10, meaning caches
expire already after 10 seconds. With your slow connection, a higher
value should help already.

> Paul

Best regards, Michael.
--000000000000d9855f05e2ebf70a-- From debbugs-submit-bounces@debbugs.gnu.org Mon Jul 04 06:33:56 2022 Received: (at 56342) by debbugs.gnu.org; 4 Jul 2022 10:33:56 +0000 Received: from localhost ([127.0.0.1]:46179 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1o8JOu-0000hp-2l for submit@debbugs.gnu.org; Mon, 04 Jul 2022 06:33:56 -0400 Received: from mout.gmx.net ([212.227.17.21]:44073) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1o8JOn-0000hV-No for 56342@debbugs.gnu.org; Mon, 04 Jul 2022 06:33:55 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.net; s=badeba3b8450; t=1656930823; bh=oHvJohFq3CH1KTNEH1WX4jkHgUcsxvAqLcrXPi6Amkc=; h=X-UI-Sender-Class:From:To:Cc:Subject:References:Date:In-Reply-To; b=M8Ch6KF4agNgTvscoHx4C0miGP/xHA2pYmPmBRF+9RKJxu5R++AVYoxktgVGqHiJm COuIMBudUYfmlG2xclaR3cnAwY2YMiZs+S62eC0Jc7nUbkrDoo/oKzA6jlumSOB74u /BE4pnGMQJfO2t2SNT2Gvq3wsy0Tv0vZPfj64pHk= X-UI-Sender-Class: 01bb95c1-4bf8-414a-932a-4f6e2808ef9c Received: from gandalf.gmx.de ([79.140.119.4]) by mail.gmx.net (mrgmx104 [212.227.17.168]) with ESMTPSA (Nemesis) id 1Mn2aN-1nidhB2C2B-00k8qV; Mon, 04 Jul 2022 12:33:43 +0200 From: Michael Albinus To: Paul Pogonyshev Subject: Re: bug#56342: TRAMP (sh) issues way too many commands, thus being very slow over high-ping networks References: <8735fjh5ge.fsf@gmx.de> Date: Mon, 04 Jul 2022 12:33:39 +0200 In-Reply-To: <8735fjh5ge.fsf@gmx.de> (Michael Albinus's message of "Sat, 02 Jul 2022 17:58:25 +0200") Message-ID: <87wnctb20s.fsf@gmx.de> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/29.0.50 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Provags-ID: V03:K1:6kO8ar5uvbON1/o84b/sOfWDqYIFvce4kN/1TBOaygJSzJSlphm YbpLeZUoNoyyLvba23pn7iiQq3MZe57OZZdl8nl2GbwmoWiBLNNbX6aTDDwdD/GAhgCXMFH HQC4g389TK9v2+kVaK6sITNOMZUWiPMP5yMWd50X8XewYNf/+IeJUyC9QQbNV5teID4RetU 8PAx7N7xKQk/SiL6Mh6cw== X-Spam-Flag: NO X-UI-Out-Filterresults: notjunk:1;V03:K0:HzkWuYWpokc=:3vx0tzUnXWz2y5jSdA/8sH xBoE2IfIQvqPfHvHi5tXqeGN/z1Kh1Z7uub1a46izbFNxNxMzCyVRTc/WAIZdbARfmGZwTCvm I0GLTfvY0dezTgEMyAOCpkoOxTp/T58u3O/QhG9ABV2c5sCblSyCpXutg75620yUIdSF/F0PN jXzGJaMpHyvDkYmZz4k1JR4X5O5g3d5RBKcEUG6G+PlXYAPnByh3nX9uVSFLkpiAyai4Cbst/ BoOvfBrpj2Vq/2jG7m7LsDEy3zXV5RHwiUO2e9NAlQhGbHRDNxQTetvu24jmizdziD/Z884WW UmcMoJHMOdbIwOqymgHokMvRvzf+Mml0+Nk5cjwZxJ2ySkLPFuP4lrRdJLPVglqmxKgIPLmFr 8VKAlVyYyTUwi9fRZeiqvsDAASOb1rT0bocBizb6i6chnE86noB9Sgc4tx9CmPC+toHhmIDin LZYL4mZcT1qOHN4Z5rZKFVQ7sKvprI5d/fmDnzOOVrTiOJQw9gGGduTEt/+pzA8hS4bqCWQZM SOnxJvewF24YpIjo2pCuaogyAZirqrYBgC48vGAt8WNxC7pAF2NR0R1iBKkruyclyDGsN90dy fk+BRs2NAJvUtkFQ2eSQYgJROGxIujjJwNzL9EKa2U+S4hj+YoTqgds+RB+m1xpCXJ2MFWDBY umq7PCLpcftDVQB9aAoQlPe25B9uDvDkIn/q9+Fkds/rw1QqDV23C+3dnp+/r7Y1PYtxD4OdV bFEmH3DiGUUmXWgPYevxPbR37bIetJYEYUXdqoRar46oL/YZ7wD5bO0TrOC9RUN/toOf4KkwC ih7Um6BrH1Nj14Gbuvo7yy/uGSW0uX+3I5Z9puBNpAbj+7q/LuLXM8cKkJGPwHyFJzGuTlYsP w94QC3DTN5ZRyF/3tkDU/oML2mZKSQ0kr4M3z4iSjGGQAlDSWEE40Qa9r7PooslepvqOgJKkr l6Nb200yrSjafYebzIVQV7guv+MeSxdlLT65vH0rS+DfDQXFaxvyal8lMWGb/NT3u9xJcnPIH 1vXvmf+M4XolAT2fKuQe+OGp1DmdE3XwYRQm6mE46Xn/pj/Vi+zngpHOtBNlHQQNgeY2Bu3dZ qvOIaQMS4LgThYLp8x2PT+bl6zrvldlRKsRq+HHloWT/o6Id65O2mlCxQ== X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 56342 Cc: 56342@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) Michael Albinus writes: Hi Paul, >> Suggested improvements: >> >> * TRAMP should issue just one `stat' command to find out most of the >> things about a file: whether it exists, if it is a directory, its real >> name when dereferencing links and whatever stats it is used to find >> now; from `$ stat --help' this seems to be possible. In other words, >> TRAMP shouldn't use simple commands like `test -e': any ping, even >> nominal, will negate any gains from using a tad faster command. >> Instead, if it needs to find anything about a file, it should ask the >> remote about as many things as possible in one go: it is very likely >> that the additional information will be needed soon and even if not, >> this is basically free compared to ping anyway. > > Not all remote hosts carry a stat command, and not all existing stat's > are GNU compatible. But yes, if possible, Tramp shall gather as much > information in one run, and cache the results for further use. FTR, stat is not sufficient to detect the real file name. Example: # touch /tmp/a # ln -s /tmp/a /tmp/b # ln -s /tmp/b /tmp/c # stat -c %N /tmp/c It returns "'/tmp/c' -> '/tmp/b'". However, we need "/tmp/a". So we must still use "readlink --canonicalize". Best regards, Michael. From debbugs-submit-bounces@debbugs.gnu.org Mon Jul 04 07:19:54 2022 Received: (at 56342) by debbugs.gnu.org; 4 Jul 2022 11:19:54 +0000 Received: from localhost ([127.0.0.1]:46308 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1o8K7O-0006Qg-BP for submit@debbugs.gnu.org; Mon, 04 Jul 2022 07:19:54 -0400 Received: from mout.gmx.net ([212.227.17.21]:54269) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1o8K7N-0006QS-14 for 56342@debbugs.gnu.org; Mon, 04 Jul 2022 07:19:53 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.net; s=badeba3b8450; t=1656933586; bh=h8vZtzxJxknb48sioZgkMRt+/8FaBbm7di2v8kCQUYQ=; h=X-UI-Sender-Class:From:To:Cc:Subject:References:Date:In-Reply-To; b=CRSaYIjC/RAMxWLztBvzs1bLh23U2SSBW0ap5a+6q3uMvPq0ci54UqSD9zHnOVheJ WwMQVLBuamaap+/O1OHUeT2Ej4co0qIcMa3LADZywzIvENBEb8svokkdnxwnlHRZIF uxxHz+4/VSojPyaf7Iwgui/5szWL37irlrzPD0a8= X-UI-Sender-Class: 01bb95c1-4bf8-414a-932a-4f6e2808ef9c Received: from gandalf.gmx.de ([79.140.119.4]) by mail.gmx.net (mrgmx105 [212.227.17.168]) with ESMTPSA (Nemesis) id 1M26r3-1oAn5n0e1h-002VEG; Mon, 04 Jul 2022 13:19:46 +0200 From: Michael Albinus To: Paul Pogonyshev Subject: Re: bug#56342: TRAMP (sh) issues way too many commands, thus being very slow over high-ping networks References: <8735fjh5ge.fsf@gmx.de> <87v8sefl2f.fsf@gmx.de> <874jzyc9u9.fsf@gmx.de> Date: Mon, 04 Jul 2022 13:19:45 +0200 In-Reply-To: (Paul Pogonyshev's message of "Sun, 3 Jul 2022 21:52:55 +0200") Message-ID: <87sfnhazvy.fsf@gmx.de> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/29.0.50 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Provags-ID: V03:K1:a048jtjG92FRil+jbLcLKcGHynb52eVKmf+Plm7GSvhccMZwJL8 JGid2wv1fc2wCkzVxWwQSccDB3UW+dmTrbIYrQKexSRiBgrL+03r5UUEFG+n0VyBJHHtZYt kQGm0yf2TrZlG1cl5HSDVsLtpi9+84at3g2ydHSmj9PZSLRwaZGHrluEXLEkPXoHyDibPzx 1AKVK4Oy53XQx6UiL1dJw== X-Spam-Flag: NO X-UI-Out-Filterresults: notjunk:1;V03:K0:h1Vx4CkIH2A=:PrGW5p5mfT8yVfqGZW6UCr 17pVHaGG93PWmI2+KX89wAwlcXzJYYgsRA4ZWhrAW1WJWbXjdesfLZNPWd61lWl3/GRtXbUEa FCI1+RnKcG4J3zTnBih4R+gMiwCXTbG/pAHlhtq8XmOhaj5+/iv7TpUTPuNffq5naON3dCbEs g2Jfyt6mufgUulLDQMUKQHYxggd22wbcpiQFTfcUCnYZxPCXd9H6VT6njRh/xAXmB1y3erOZj cKtvuwlxW4p3ZiSsRmXdQ4jMUxiumaviAsspSEifeiSO3WesR1kYFSLdAF87EtFT7wZxAVwTG 39FIIZzgOl4zpd5NQvc4vEx/PavtamqlFJT330rDukoE3wZLvPIsLvm2HV+FWUAoD9h4bmpHt IVA9Iu3ZryFUsBg9+Hqo62imgzpEQhpOJRn+Qi+tJZ8x3Eoye3XVUdZ0v6iAe6KRIBePDEDLd Pvd5TjKeduQRhUyKoaDhVzzsuJrh13I+u/peXDJcKAHRmV3r9KkAZw6tJhziXSnlXNd/PB9e6 KaHAtDMszsl9ClZeeBxXBGvqmMvynwzSPeWVku2mINHkBzw2FHLWjMrhnTXd2qTDjSLMbhjmm EP/nHWAdAYgnbi7QGn+ynG4/QMVKdgYPo+MV0r4KFDpXyZbEUm+jYAr1cfMfx4Z4griimFr7Z El43jCOb3f9WfuXUK5qMK9trDC6alw28DrYN5F8crrCImHVXy50OHaXmM2ux7OLxPaJcOB2Nu /XIOar/TmxbqbWbfIovuwnonMcKNCQTPGt2Skz9yQXzXffqK2JBX4sgE0e3NMQyJxGs6G2gHf f++mzr5oOHecIh5azwrHKu1iUBR9+1eGqrQJcYnqA02sqfIj01+D9HEnUf19TfNXxw5JhPdFU n5CmE62EV1rTy6/VJTYAFVqgHXg0QSZhtjrwbOyr3D3sxilermj84E/BzjvQDllKdUO0Tzuga aqURjB32uAopA8VCN5DeopYKP9VQvuXwrpeLe6SXDXf/Pg2ND2KwOhFRnVNQyOxdFQ8Kk7tJo U+r+4TPGpTl5PRf3KsOeCJaxgrfD6iB78pUFhHF7DSGKKtr7ZDANdKCceX4u053pJHVjwt4gg vponUMXAYXgdT6Ds8jIOZ3DIYW8faVfVz3dGLdLg3DhL1Yu0nuJcWg+Bg== X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 56342 Cc: 56342@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) Paul Pogonyshev writes: Hi Paul, >> Doing it asynchronously would require a second connection to the remote >> host. Performance would rather degrade. > > Maybe not really asynchronously, just let it return early, not waiting > for the result? I'm not sure how TRAMP receives responses, but can it > just keep executing commands sequentially, as now, but give control > back to the caller in case of commands where the result doesn't really > matter (cleanup, e.g. deleting a temporary file). Of course, this > means that if an "important" command is issued right away, it has to > wait for the response to the previous inessential command. But when > such an inessential command is the last in a batch, this waiting would > be effectively merged with Emacs' idling in the normal UI command > loop, making things more responsive for the user. Tramps communication with the remote host is like a REPL engine. It sends shell commands to the remote hosts, reads the result, and waits for the shell prompt. If it doesn't wait for the final shell prompt, it is likely that the result or the shell prompt will be seen when reading the output of the next command. This confuses. So no, I don't see a chance to implement this kind of "asynchronity". > Paul Best regards, Michael. From debbugs-submit-bounces@debbugs.gnu.org Mon Jul 04 10:42:47 2022 Received: (at 56342) by debbugs.gnu.org; 4 Jul 2022 14:42:47 +0000 Received: from localhost ([127.0.0.1]:48379 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1o8NHj-00060U-2V for submit@debbugs.gnu.org; Mon, 04 Jul 2022 10:42:47 -0400 Received: from mail-yw1-f177.google.com ([209.85.128.177]:33289) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1o8NHg-000605-1A for 56342@debbugs.gnu.org; Mon, 04 Jul 2022 10:42:45 -0400 Received: by mail-yw1-f177.google.com with SMTP id 00721157ae682-31c89111f23so33827777b3.0 for <56342@debbugs.gnu.org>; Mon, 04 Jul 2022 07:42:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=jr/B1g9LGAvqvTYCa0EDnel71tzPyxGDbO2Mt+h1vXg=; b=aVLdYbFOrw/Mvt6pMTOFwKu2t/KdMW9zzbpfuhz50F8S/GRnAz46IkZEpKDQs+NMSt tW7UzegHDhwOJowEXPLlw1jvVQ9YgWpQY8V7vLtkpal7BqoBP4AHq+MMVn2p4h8bWMGf coTferDfS4ZcBk80jza+PQjGGu0Ialwl5WX1fdDJyRzd99oBRy8xNoUZB59ZQdCbEK2U 617kv/iIh68LNH7GBZlmlt0jCcOton0Xcau8cbIq9/a0kU3qUZS3dwYf0u/Y/HLddwlK rDI87HggTP3XFYwRTz2FRhL3YQiZlSQ0vB1fW5Y2MjmsQcGCNiGx2BWS8NWxytdLv1Uq wdqQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=jr/B1g9LGAvqvTYCa0EDnel71tzPyxGDbO2Mt+h1vXg=; b=UEK8ov/YQH35vhgO9AHdHfTCSEw/WdwqmWZbGeZ5bDcfQ7DTwFw3NLmHEfg+petPwc +lM5mAC2u7yI3FffNlcs+zz0y5bAnoqUMxSaQJUwiS/hHo3PX+jwyZ+KugJIJ5gB7ELG bj0ofpMWlc60Ouk1K5k9AoHHRjwjQPrRjRNY+BiJrILwLfdIQI/Ohts/PimBCCWSGOCE QECb/woHIacUaxtIhup20V+38uSv0Xop0rfiJUvGYWgHuMacCICDxNMSTcGa4NMfyIwN +qVF6fX66hkcvFMTpT38R5Mxwl6SmjtL9QMENmvH09nmJodQztxZ6wHVsBnfQkHS6qse LetQ== X-Gm-Message-State: AJIora+QLFWRly+uz55ZmlpClm+++iXwvwJ6H6JyG0ieLe3qGoU/HMM6 kLqJcqQk7tnFYSg472iuXRjWE2wdKbeptHCEKC97jeSxmA== X-Google-Smtp-Source: AGRyM1sLDVjumx9IIZh7xBV0DKu+JO5nEBX5w6SveNf1SekWHkYzHsnwA4+yMFFZ7rg+bwdgLtCD4YD+ikYyK0Z7580= X-Received: by 2002:a81:106:0:b0:2d0:e682:8a7a with SMTP id 6-20020a810106000000b002d0e6828a7amr34241796ywb.257.1656945758306; Mon, 04 Jul 2022 07:42:38 -0700 (PDT) MIME-Version: 1.0 References: <8735fjh5ge.fsf@gmx.de> <87v8sefl2f.fsf@gmx.de> <874jzyc9u9.fsf@gmx.de> <87sfnhazvy.fsf@gmx.de> In-Reply-To: <87sfnhazvy.fsf@gmx.de> From: Paul Pogonyshev Date: Mon, 4 Jul 2022 16:42:26 +0200 Message-ID: Subject: Re: bug#56342: TRAMP (sh) issues way too many commands, thus being very slow over high-ping networks To: Michael Albinus Content-Type: multipart/alternative; boundary="00000000000056054c05e2fbbf5c" X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 56342 Cc: 56342@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) --00000000000056054c05e2fbbf5c Content-Type: text/plain; charset="UTF-8" > It returns "'/tmp/c' -> '/tmp/b'". However, we need "/tmp/a". So we must > still use "readlink --canonicalize". According to a quick search, it is possible to merge output of several shell commands together. This seems to work even with dumb `sh', it's not a Bash extension: $ sh -c '{ stat xxx && readlink xxx; }' I guess TRAMP could just sth. similar, as I understand it doesn't have to be strictly one executable call, just one command given to the remote shell. > Tramps communication with the remote host is like a REPL engine. It > sends shell commands to the remote hosts, reads the result, and waits > for the shell prompt. If it doesn't wait for the final shell prompt, it > is likely that the result or the shell prompt will be seen when reading > the output of the next command. This confuses. So no, I don't see a > chance to implement this kind of "asynchronity". I see parameter `nooutput' to `tramp-send-command'. Couldn't that be used? Even if not, I could imagine sth. like this: (defvar pending-commands nil) (defvar reading-output nil) (defun send-command (x output-inessential) (if output-inessential (setf pending-commands (nconc pending-commands (list x))) (while reading-output ; make sure the connection is free for the next essential command (read-next-output-chunk) (when (and (not reading-output) pending-commands) (do-send-command (pop pending-commands)))) (do-send-command x) (read-output-now))) (defun do-send-command (x) (really-do-send-it x) (setf reading-output t)) (defun read-output-now () (while reading-output (read-next-output-chunk)) (extract-received-output-from-process-buffer)) (defun emacs-idling () ; hooked up using `run-with-idle-timer' or something like that (cond (reading-output (read-next-output-chunk)) (pending-commands (do-send-command (pop pending-commands))))) (defun read-next-output-chunk () (when reading-output (do-read-output-chunk) ; this should be non-blocking (when (end-of-command-output) (setf reading-output nil)))) Paul On Mon, 4 Jul 2022 at 13:19, Michael Albinus wrote: > Paul Pogonyshev writes: > > Hi Paul, > > >> Doing it asynchronously would require a second connection to the remote > >> host. Performance would rather degrade. > > > > Maybe not really asynchronously, just let it return early, not waiting > > for the result? I'm not sure how TRAMP receives responses, but can it > > just keep executing commands sequentially, as now, but give control > > back to the caller in case of commands where the result doesn't really > > matter (cleanup, e.g. deleting a temporary file). Of course, this > > means that if an "important" command is issued right away, it has to > > wait for the response to the previous inessential command. But when > > such an inessential command is the last in a batch, this waiting would > > be effectively merged with Emacs' idling in the normal UI command > > loop, making things more responsive for the user. > > Tramps communication with the remote host is like a REPL engine. It > sends shell commands to the remote hosts, reads the result, and waits > for the shell prompt. If it doesn't wait for the final shell prompt, it > is likely that the result or the shell prompt will be seen when reading > the output of the next command. This confuses. So no, I don't see a > chance to implement this kind of "asynchronity". > > > Paul > > Best regards, Michael. > --00000000000056054c05e2fbbf5c Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
> It returns "'/tmp/c' -> '/tmp/b&#= 39;". However, we need "/tmp/a". So we must
> still us= e "readlink --canonicalize".

According to = a quick search, it is possible to merge output of several shell commands to= gether. This seems to work even with dumb `sh', it's not a Bash ext= ension:

=C2=A0 =C2=A0 $ sh -c '{ stat xxx &= ;& readlink xxx; }'

I guess TRAMP coul= d just sth. similar, as I understand it doesn't have to be strictly one= executable call, just one command given to the remote shell.
> Tramps communication with the remote host is like a REPL e= ngine. It
>=C2=A0sends shell commands to the remote hosts, reads the = result, and waits
>=C2=A0for the shell prompt. If it doesn't wait= for the final shell prompt, it
>=C2=A0is likely that the result or t= he shell prompt will be seen when reading
>=C2=A0the output of the ne= xt command. This confuses. So no, I don't see a
>=C2=A0chance to = implement this kind of "asynchronity".

I see parameter `nooutput' to `tramp-send-command'. Couldn't= that be used?

Even if not, I could imagine sth. l= ike this:

=C2=A0 =C2=A0 (defvar pending-commands n= il)
=C2=A0 =C2=A0 (defvar reading-output nil)

=C2=A0 =C2=A0 (defu= n send-command (x output-inessential)
=C2=A0 =C2=A0 =C2=A0 (if output-in= essential
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 (setf pending-commands (nco= nc pending-commands (list x)))
=C2=A0 =C2=A0 =C2=A0 =C2=A0 (while readin= g-output =C2=A0; make sure the connection is free for the next essential co= mmand
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 (read-next-output-chunk)
=C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 (when (and (not reading-output) pending-com= mands)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 (do-send-command (pop p= ending-commands))))
=C2=A0 =C2=A0 =C2=A0 =C2=A0 (do-send-command x)
= =C2=A0 =C2=A0 =C2=A0 =C2=A0 (read-output-now)))

=C2=A0 =C2=A0 (defun= do-send-command (x)
=C2=A0 =C2=A0 =C2=A0 (really-do-send-it x)
=C2= =A0 =C2=A0 =C2=A0 (setf reading-output t))

=C2=A0 =C2=A0 (defun read= -output-now ()
=C2=A0 =C2=A0 =C2=A0 (while reading-output
=C2=A0 =C2= =A0 =C2=A0 =C2=A0 (read-next-output-chunk))
=C2=A0 =C2=A0 =C2=A0 (extrac= t-received-output-from-process-buffer))

=C2=A0 =C2=A0 (defun emacs-i= dling () =C2=A0; hooked up using `run-with-idle-timer' or something lik= e that
=C2=A0 =C2=A0 =C2=A0 (cond (reading-output
=C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(read-next-output-chunk))
=C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 (pending-commands
=C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0(do-send-command (pop pending-commands)))))

= =C2=A0 =C2=A0 (defun read-next-output-chunk ()
=C2=A0 =C2=A0 =C2=A0 (whe= n reading-output
=C2=A0 =C2=A0 =C2=A0 =C2=A0 (do-read-output-chunk) =C2= =A0; this should be non-blocking
=C2=A0 =C2=A0 =C2=A0 =C2=A0 (when (end-= of-command-output)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 (setf reading-outp= ut nil))))

Paul

On Mon, 4 Jul 2022 at 13:19, = Michael Albinus <michael.albinus@gmx.de> wrote:
Paul Pogonyshev <pogonyshev@gmail.com> writes:

Hi Paul,

>> Doing it asynchronously would require a second connection to the r= emote
>> host. Performance would rather degrade.
>
> Maybe not really asynchronously, just let it return early, not waiting=
> for the result? I'm not sure how TRAMP receives responses, but can= it
> just keep executing commands sequentially, as now, but give control > back to the caller in case of commands where the result doesn't re= ally
> matter (cleanup, e.g. deleting a temporary file). Of course, this
> means that if an "important" command is issued right away, i= t has to
> wait for the response to the previous inessential command. But when > such an inessential command is the last in a batch, this waiting would=
> be effectively merged with Emacs' idling in the normal UI command<= br> > loop, making things more responsive for the user.

Tramps communication with the remote host is like a REPL engine. It
sends shell commands to the remote hosts, reads the result, and waits
for the shell prompt. If it doesn't wait for the final shell prompt, it=
is likely that the result or the shell prompt will be seen when reading
the output of the next command. This confuses. So no, I don't see a
chance to implement this kind of "asynchronity".

> Paul

Best regards, Michael.
--00000000000056054c05e2fbbf5c-- From debbugs-submit-bounces@debbugs.gnu.org Mon Jul 04 12:30:26 2022 Received: (at 56342) by debbugs.gnu.org; 4 Jul 2022 16:30:26 +0000 Received: from localhost ([127.0.0.1]:48553 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1o8Oxt-0000rm-Iu for submit@debbugs.gnu.org; Mon, 04 Jul 2022 12:30:26 -0400 Received: from mout.gmx.net ([212.227.17.22]:48375) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1o8Oxp-0000gC-35 for 56342@debbugs.gnu.org; Mon, 04 Jul 2022 12:30:23 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.net; s=badeba3b8450; t=1656952214; bh=8saIDYDSHn71BE2qAjBSOZH16WAO72ZcpW6fT26qWPE=; h=X-UI-Sender-Class:From:To:Cc:Subject:References:Date:In-Reply-To; b=G2e2fBNBOr0lGb0xoY6JT0jE6Z9MRX3PR1zXGZb2IT5KcKG6sTqeENOUfVKotEkok vPwEx7nKf2X1OHOucjLGpXMXXhiAeXZ/V2QmOswH0b2h+QPgiqEwPq3gSUyzQlj17+ ehTkv6MytyRQeBknfCQY2pP0loi2f6W/cld9U3iA= X-UI-Sender-Class: 01bb95c1-4bf8-414a-932a-4f6e2808ef9c Received: from gandalf.gmx.de ([79.140.119.4]) by mail.gmx.net (mrgmx105 [212.227.17.168]) with ESMTPSA (Nemesis) id 1MiJZO-1neGrw1bOw-00fOCq; Mon, 04 Jul 2022 18:30:14 +0200 From: Michael Albinus To: Paul Pogonyshev Subject: Re: bug#56342: TRAMP (sh) issues way too many commands, thus being very slow over high-ping networks References: <8735fjh5ge.fsf@gmx.de> <87v8sefl2f.fsf@gmx.de> <874jzyc9u9.fsf@gmx.de> <87sfnhazvy.fsf@gmx.de> Date: Mon, 04 Jul 2022 18:30:12 +0200 In-Reply-To: (Paul Pogonyshev's message of "Mon, 4 Jul 2022 16:42:26 +0200") Message-ID: <87k08sc02z.fsf@gmx.de> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/29.0.50 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Provags-ID: V03:K1:dSQ9HtSdG2WGRcqeFpEQTABfATtY5LGBM3X23ZKTyrkUL8fyCQd N0hARi61W+PRuDIqy+/W5PxNTaoiNvDTJIjhhPWbhdp1QFrTrOLPcSz1qiDKUnivhF8fGfQ oORRhuemBVly3jHRR4EPP6JWbRhq2GpNalAx3S3Iz2/LgGGU8FyT0za6+XDmbG5V8IdbljY QQKSTayQrrkGaE5NGy3KQ== X-Spam-Flag: NO X-UI-Out-Filterresults: notjunk:1;V03:K0:6cvVaUrf+jI=:omIe9eaw0N6lh8fXQZqZg+ /M1x2LBCt64QygNxzEMQ2kxj8QhZ8maE7vLrphok2cixRyfGSULt8Dw6HwQFta3sKry5vJg84 VObZTJ53z0SwoLDiogI+doAySbqRpJtJ2ZDaoyRjMvKY6PnkakaAJAG7ZVuAxMu2orRRRvMjh igmek8LSIXKFJ44mC6c4ydzYfpY73zFz7hVdBmhTtPLlkCtgcrxbpZ5r99FHU3LZjvJG9ZB+C 9+dfZ7T57onHilV1QNLa/dp/L9FqhbIAEFU+/WZAF90SRbifcVqV9tY/TZ+yl3d+SlqSXKlyn DmYLtE4Q2flJXpIq7n5uJI9LwYlpjWdCY+wLEyS4YXeaUi6HSj+AC2NMUm/jXzJ9PpNJcdS0R vDLTMibHdR3MugzAhUSGUjSjomrTg3HI9C5zVr/Pt11bAfmM4SgDys4rCWhYoIQQc++B55LDC bTmvpPd9iaJ93e3B2qY3aPUX14UTeSYa+CGFsl0Ea4OfIJh7/wFFNeinHZ3Zn4eNlBDNMAtnK oKxd5CICv2RLfhQ6SM9jk503rau06xDQ/G9UBzuFTTsyY2SN5NKA0WBrUm2Rz1EJuQZ7K1idL 4JY53hMxPWVpwmP/grQx92lupKv6wUG801E9j3zz1t7y/xsPTo6brmY+8ov20EIRS9M0sdexY Z8J0Gqf16WM7TCiIWtY/DvzzfrB0xdrFyZHHRIfUkR1R/fmRyqnNgq0bzcMVUk8FGbcMIz9y2 8zGsreWA5XaHl0U0X1gA90CzBrLFwO8KcxWYH+wo+NdvUfM+u5HB0I7tSxS8xNl9BSiHvx7Om 729w+Yaa0DNZeOfllzhPaqDjDxu5MlOH6KRtz5UlfL2Z8ePGqEA6IKGnP7CEUa1hRLgOwvRLv vqbZmE+y2JXD3Bab413LtlVDjyifva28IrwCad00i36yyEmcuteLNByPxwQQXZ5arXYS6eRft Ck3O7vM71M8pu43s5PWcYPXpKylTySujsXJq0jKr1lcylURDi098XJNiQ5GjetIVg38gsu77Z u4qs5dpXIW1vt0dVBf1ifPRdcMia6UOW3MXQzawfcPdgj1QQ6NVIdtU7DpMJrr0NQDHJub+w1 1lxKA3kG7CCc4fJmz8iM5CnAmvqBfUxzCjcfHovNEGeUi7th9niEGDlmA== X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 56342 Cc: 56342@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) Paul Pogonyshev writes: Hi Paul, >> It returns "'/tmp/c' -> '/tmp/b'". However, we need "/tmp/a". So we must >> still use "readlink --canonicalize". > > According to a quick search, it is possible to merge output of several > shell commands together. This seems to work even with dumb `sh', it's > not a Bash extension: > > $ sh -c '{ stat xxx && readlink xxx; }' > > I guess TRAMP could just sth. similar, as I understand it doesn't have > to be strictly one executable call, just one command given to the > remote shell. I'll check, perhaps it could be used in this special case. But Tramp uses already combination of shell commands, and it uses even private shell functions when applicable. Don't expect too much in general. >> Tramps communication with the remote host is like a REPL engine. It >> sends shell commands to the remote hosts, reads the result, and waits >> for the shell prompt. If it doesn't wait for the final shell prompt, it >> is likely that the result or the shell prompt will be seen when reading >> the output of the next command. This confuses. So no, I don't see a >> chance to implement this kind of "asynchronity". > > I see parameter `nooutput' to `tramp-send-command'. Couldn't that be > used? No. This argument tells tramp-send-command not to call the final accept-process-output. But this is only because on the caller side, accept-process-output will be called with other arguments but the default ones. > Even if not, I could imagine sth. like this: > > (defvar pending-commands nil) > (defvar reading-output nil) > > (defun send-command (x output-inessential) > (if output-inessential > (setf pending-commands (nconc pending-commands (list x))) > (while reading-output ; make sure the connection is free for > the next essential command > (read-next-output-chunk) > (when (and (not reading-output) pending-commands) > (do-send-command (pop pending-commands)))) > (do-send-command x) > (read-output-now))) > > (defun do-send-command (x) > (really-do-send-it x) > (setf reading-output t)) > > (defun read-output-now () > (while reading-output > (read-next-output-chunk)) > (extract-received-output-from-process-buffer)) > > (defun emacs-idling () ; hooked up using `run-with-idle-timer' or > something like that > (cond (reading-output > (read-next-output-chunk)) > (pending-commands > (do-send-command (pop pending-commands))))) > > (defun read-next-output-chunk () > (when reading-output > (do-read-output-chunk) ; this should be non-blocking > (when (end-of-command-output) > (setf reading-output nil)))) Something like this could be done, yes. Perhaps not with an (idle) timer, because they are known to interrupt Tramp's REPL engine at any point, with (sometimes) desastrous results. In general, one shall try to avoid Tramp actions in timers, process sentinels and process filters. So we could use this for deleting temporary files and alike, but there won't be too many performance boosters I fear. I will play with this idea, perhaps it helps. But for sure it will be an opt-in feature only. > Paul Best regards, Michael. From debbugs-submit-bounces@debbugs.gnu.org Tue Jul 26 04:01:01 2022 Received: (at 56342) by debbugs.gnu.org; 26 Jul 2022 08:01:01 +0000 Received: from localhost ([127.0.0.1]:52149 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1oGFUy-0007na-QU for submit@debbugs.gnu.org; Tue, 26 Jul 2022 04:01:01 -0400 Received: from mail-yw1-f174.google.com ([209.85.128.174]:42818) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1oGFUw-0007nM-GD for 56342@debbugs.gnu.org; Tue, 26 Jul 2022 04:00:59 -0400 Received: by mail-yw1-f174.google.com with SMTP id 00721157ae682-2ef5380669cso133936907b3.9 for <56342@debbugs.gnu.org>; Tue, 26 Jul 2022 01:00:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=xVieKpmRoq5vdeEhQ7eKL8ARrdBsLUXHYEtEFB7VEaI=; b=ZyZfdQl4RCtaOFEdGNIzTkXKKlz6C7ED8Z/mhdTjzOCnZVkxx9QamU+loiIEcup1Ra Sa9g9Qu3h2xumBGhjlSrOFcF5cpGZckuiYgG26FZ+vy5rTibfGPf0N3xOgk1sNGavltK 8TtKxCl4C8BBh1a5eYv7TFPV2XWTweqmcah03dUoD3jv2NzB7Y22Zh8n+in74tKsWRbF ucDzE2NCCRbJdMfk+OggXtj3AFH9uIp9gZv2Pooi2dISsbeNLypOTH+vivuHfcOarxGQ RmiH+LmVDYOGOmDZCFEoxDATUc3Vn/BXpIi+ToKCqb+kKNWZxumlKPYgL/ndP2pyl9Wz Cv7g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=xVieKpmRoq5vdeEhQ7eKL8ARrdBsLUXHYEtEFB7VEaI=; b=CVvf0SjVp9VTMvL60MiTg/QBYvA3brKVaEsealtdv+XysoKZCVQlMh1S9k+FFDUP5G EdRGbWOPET2JtrCOCVJUB4Wo5zsX2OgPa0HKSwtMlwS7+5AbcqXqxto7kvQmw0XlwICy jT0WHE1x/faXyFX1Jl/AlSVtimwRLtzD6lZE4VjqSGQTiT72C6uFdOCsB4oXaB3RejCe VwPRDmmIoLDnMWoXwCSGFbrDneoHq4I+VtVvDTy4+TzCBmgfxyoRzkv4enJWh8qJaSuk r3FbKu5q0r8039cdNVVHxub40Hya2mwqjkv4F+xUfoa8CTV4gP+6MUBqGaM2no5CKXL9 PIIg== X-Gm-Message-State: AJIora+VORXQtmN+qrHzNhKMyjDeip0iqTM3y3ttdg2IntXIVY28k9YI YiiSoIsadKgcmU5vojq/xz5D878Jtkd4yPGMFg== X-Google-Smtp-Source: AGRyM1sBdFFx5CvD5NGbFppHeqCxt23d6ljLX2D7OFvwjyBosi07TKUGWWrcXbjcDXqpx+UVk/1H3C4Hv31Xioe5qtg= X-Received: by 2002:a0d:df04:0:b0:31e:573a:37f4 with SMTP id i4-20020a0ddf04000000b0031e573a37f4mr13649498ywe.76.1658822451858; Tue, 26 Jul 2022 01:00:51 -0700 (PDT) MIME-Version: 1.0 References: <8735fjh5ge.fsf@gmx.de> <87v8sefl2f.fsf@gmx.de> <874jzyc9u9.fsf@gmx.de> <87sfnhazvy.fsf@gmx.de> <87k08sc02z.fsf@gmx.de> In-Reply-To: <87k08sc02z.fsf@gmx.de> From: Paul Pogonyshev Date: Tue, 26 Jul 2022 10:00:40 +0200 Message-ID: Subject: Re: bug#56342: TRAMP (sh) issues way too many commands, thus being very slow over high-ping networks To: Michael Albinus Content-Type: multipart/alternative; boundary="000000000000fd113c05e4b0b222" X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 56342 Cc: 56342@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) --000000000000fd113c05e4b0b222 Content-Type: text/plain; charset="UTF-8" Hello, any news on this? Paul On Mon, 4 Jul 2022 at 18:30, Michael Albinus wrote: > Paul Pogonyshev writes: > > Hi Paul, > > >> It returns "'/tmp/c' -> '/tmp/b'". However, we need "/tmp/a". So we must > >> still use "readlink --canonicalize". > > > > According to a quick search, it is possible to merge output of several > > shell commands together. This seems to work even with dumb `sh', it's > > not a Bash extension: > > > > $ sh -c '{ stat xxx && readlink xxx; }' > > > > I guess TRAMP could just sth. similar, as I understand it doesn't have > > to be strictly one executable call, just one command given to the > > remote shell. > > I'll check, perhaps it could be used in this special case. > > But Tramp uses already combination of shell commands, and it uses even > private shell functions when applicable. Don't expect too much in > general. > > >> Tramps communication with the remote host is like a REPL engine. It > >> sends shell commands to the remote hosts, reads the result, and waits > >> for the shell prompt. If it doesn't wait for the final shell prompt, it > >> is likely that the result or the shell prompt will be seen when reading > >> the output of the next command. This confuses. So no, I don't see a > >> chance to implement this kind of "asynchronity". > > > > I see parameter `nooutput' to `tramp-send-command'. Couldn't that be > > used? > > No. This argument tells tramp-send-command not to call the final > accept-process-output. But this is only because on the caller side, > accept-process-output will be called with other arguments but the > default ones. > > > Even if not, I could imagine sth. like this: > > > > (defvar pending-commands nil) > > (defvar reading-output nil) > > > > (defun send-command (x output-inessential) > > (if output-inessential > > (setf pending-commands (nconc pending-commands (list x))) > > (while reading-output ; make sure the connection is free for > > the next essential command > > (read-next-output-chunk) > > (when (and (not reading-output) pending-commands) > > (do-send-command (pop pending-commands)))) > > (do-send-command x) > > (read-output-now))) > > > > (defun do-send-command (x) > > (really-do-send-it x) > > (setf reading-output t)) > > > > (defun read-output-now () > > (while reading-output > > (read-next-output-chunk)) > > (extract-received-output-from-process-buffer)) > > > > (defun emacs-idling () ; hooked up using `run-with-idle-timer' or > > something like that > > (cond (reading-output > > (read-next-output-chunk)) > > (pending-commands > > (do-send-command (pop pending-commands))))) > > > > (defun read-next-output-chunk () > > (when reading-output > > (do-read-output-chunk) ; this should be non-blocking > > (when (end-of-command-output) > > (setf reading-output nil)))) > > Something like this could be done, yes. Perhaps not with an (idle) > timer, because they are known to interrupt Tramp's REPL engine at any > point, with (sometimes) desastrous results. In general, one shall try to > avoid Tramp actions in timers, process sentinels and process filters. > > So we could use this for deleting temporary files and alike, but there > won't be too many performance boosters I fear. > > I will play with this idea, perhaps it helps. But for sure it will be an > opt-in feature only. > > > Paul > > Best regards, Michael. > --000000000000fd113c05e4b0b222 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hello,

any news on this?

=
Paul

On Mon, 4 Jul 2022 at 18:30, Michael Albinus <michael.albinus@gmx.de> wrote:<= br>
Paul Pogonyshev = <pogonyshev@gm= ail.com> writes:

Hi Paul,

>> It returns "'/tmp/c' -> '/tmp/b'". Ho= wever, we need "/tmp/a". So we must
>> still use "readlink --canonicalize".
>
> According to a quick search, it is possible to merge output of several=
> shell commands together. This seems to work even with dumb `sh', i= t's
> not a Bash extension:
>
>=C2=A0 =C2=A0 =C2=A0$ sh -c '{ stat xxx && readlink xxx; }&= #39;
>
> I guess TRAMP could just sth. similar, as I understand it doesn't = have
> to be strictly one executable call, just one command given to the
> remote shell.

I'll check, perhaps it could be used in this special case.

But Tramp uses already combination of shell commands, and it uses even
private shell functions when applicable. Don't expect too much in
general.

>> Tramps communication with the remote host is like a REPL engine. I= t
>> sends shell commands to the remote hosts, reads the result, and wa= its
>> for the shell prompt. If it doesn't wait for the final shell p= rompt, it
>> is likely that the result or the shell prompt will be seen when re= ading
>> the output of the next command. This confuses. So no, I don't = see a
>> chance to implement this kind of "asynchronity".
>
> I see parameter `nooutput' to `tramp-send-command'. Couldn'= ;t that be
> used?

No. This argument tells tramp-send-command not to call the final
accept-process-output. But this is only because on the caller side,
accept-process-output will be called with other arguments but the
default ones.

> Even if not, I could imagine sth. like this:
>
>=C2=A0 =C2=A0 =C2=A0(defvar pending-commands nil)
>=C2=A0 =C2=A0 =C2=A0(defvar reading-output nil)
>
>=C2=A0 =C2=A0 =C2=A0(defun send-command (x output-inessential)
>=C2=A0 =C2=A0 =C2=A0 =C2=A0(if output-inessential
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(setf pending-commands (nconc = pending-commands (list x)))
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(while reading-output=C2=A0 ; make su= re the connection is free for
> the next essential command
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(read-next-output-chunk)
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(when (and (not reading-output= ) pending-commands)
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(do-send-command (pop p= ending-commands))))
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(do-send-command x)
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(read-output-now)))
>
>=C2=A0 =C2=A0 =C2=A0(defun do-send-command (x)
>=C2=A0 =C2=A0 =C2=A0 =C2=A0(really-do-send-it x)
>=C2=A0 =C2=A0 =C2=A0 =C2=A0(setf reading-output t))
>
>=C2=A0 =C2=A0 =C2=A0(defun read-output-now ()
>=C2=A0 =C2=A0 =C2=A0 =C2=A0(while reading-output
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(read-next-output-chunk))
>=C2=A0 =C2=A0 =C2=A0 =C2=A0(extract-received-output-from-process-buffer= ))
>
>=C2=A0 =C2=A0 =C2=A0(defun emacs-idling ()=C2=A0 ; hooked up using `run= -with-idle-timer' or
> something like that
>=C2=A0 =C2=A0 =C2=A0 =C2=A0(cond (reading-output
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 (read-next-output-chun= k))
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(pending-commands
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 (do-send-command (pop = pending-commands)))))
>
>=C2=A0 =C2=A0 =C2=A0(defun read-next-output-chunk ()
>=C2=A0 =C2=A0 =C2=A0 =C2=A0(when reading-output
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(do-read-output-chunk)=C2=A0 ; this s= hould be non-blocking
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(when (end-of-command-output)
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(setf reading-output nil))))
Something like this could be done, yes. Perhaps not with an (idle)
timer, because they are known to interrupt Tramp's REPL engine at any point, with (sometimes) desastrous results. In general, one shall try to avoid Tramp actions in timers, process sentinels and process filters.

So we could use this for deleting temporary files and alike, but there
won't be too many performance boosters I fear.

I will play with this idea, perhaps it helps. But for sure it will be an opt-in feature only.

> Paul

Best regards, Michael.
--000000000000fd113c05e4b0b222-- From debbugs-submit-bounces@debbugs.gnu.org Tue Jul 26 10:19:01 2022 Received: (at 56342) by debbugs.gnu.org; 26 Jul 2022 14:19:01 +0000 Received: from localhost ([127.0.0.1]:54060 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1oGLOj-00039G-Ok for submit@debbugs.gnu.org; Tue, 26 Jul 2022 10:19:00 -0400 Received: from mout.gmx.net ([212.227.17.21]:59449) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1oGLOe-00038p-6k for 56342@debbugs.gnu.org; Tue, 26 Jul 2022 10:18:56 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.net; s=badeba3b8450; t=1658845125; bh=vw0nyrkUJWaevXlWIPv2SjChRgkb+IQ+TS9MsWeNm9c=; h=X-UI-Sender-Class:From:To:Cc:Subject:In-Reply-To:References:Date; b=SLi0A14Gwz3+EAGVJJYBuAJWoqBD1RGvvVRuQ+rvOcz8eXFuR4Yyf3M7A1ZIjlQLW IXkFkf+2roGlSfUU4KyXve4Kr1hj5c2X31HZaoBiGtx6Ua5YRnbtkQX7uTq62Rf+PR UoVWRBFPjbPdoD31IwG4O58SgYN7xzqDNT7kXaq8= X-UI-Sender-Class: 01bb95c1-4bf8-414a-932a-4f6e2808ef9c Received: from gandalf.gmx.de ([213.220.148.6]) by mail.gmx.net (mrgmx105 [212.227.17.168]) with ESMTPSA (Nemesis) id 1MAwXh-1oMdrg2Qqv-00BLID; Tue, 26 Jul 2022 16:18:45 +0200 From: Michael Albinus To: Paul Pogonyshev Subject: Re: bug#56342: TRAMP (sh) issues way too many commands, thus being very slow over high-ping networks In-Reply-To: (Paul Pogonyshev's message of "Tue, 26 Jul 2022 10:00:40 +0200") References: <8735fjh5ge.fsf@gmx.de> <87v8sefl2f.fsf@gmx.de> <874jzyc9u9.fsf@gmx.de> <87sfnhazvy.fsf@gmx.de> <87k08sc02z.fsf@gmx.de> Date: Tue, 26 Jul 2022 16:18:44 +0200 Message-ID: <87r128gdob.fsf@gmx.de> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/29.0.50 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Provags-ID: V03:K1:XYWJswaw6voPJ9DEr768pQvPkfysTyA5VSVYJvTUt6V0A1CDGjO HCc0ZduvAouHd4shAh1YynwzuJe18XXbR4Oi1uk8JHpehLwz7yKAlmoXbkt6G256zxvHJCv EwG1W98npae4sviTuawnYVbEQwse6bE/aaZU+0VmwInhriusIc3+SwWy/ZAetsXoGFbjxwe HqQI+OfFCfzgXSrth3OrA== X-Spam-Flag: NO X-UI-Out-Filterresults: notjunk:1;V03:K0:STE4sFae9fU=:FYbWLl5TnPpU334HOcGAvm Bc7p5Sk8FE2VSG5pn5urIq0/QxG8xBiBbxpntRT7s1p9jP72V3XnZcKuUV+tTOD6LRa+TBjLD LU6e8RKFGqskK807hTpZIEpqi3gXr4HA0FlQ5hNe4LE2Z3Wi7LGwaoWizAaZj5UF4M4jvhIxe U97yBR9h97bVwhqVQsrigb1uSpn8P/TQ5iE7Zl2/0bq3fLArujOiDzk/uh8Ir36E3QMYd8NaJ wC4VlrGt4/E2ABBPQNtejVrA92W2ebAlp2kk1mx/gNeQ1twr3a2bcuR0x6rWKSZ2U9y5hWd36 emM92znPZ5cU7DvEYb0vgH54PTmzi06zF2D7i1X9YDc5c8fGdpXxe30MJdffI6w0fTE7Sikkv Vzv079F17wPUAYvNDZ0evYn8GnbkLWqGp9OtZS6g8ek5pKf06pc3Y8TG3HAvMw0SNve9mIyxL jfkdbtIgwfzt4mY/QahFfkRXeEsVWuQDllEMS5EqmO6+88PJdQbsB5MMakZC0dK9XWgLctMWn zuK7rYtOMQvO1oZwhx6tqol2D+ANfiEI34TuYkB+pJKo+Tw5U1Ysawa58IG6NYt0LlYlcDyvc EHnniH1i8vCgBiWNO1eCisNVoX3P19cumfMS9jwnl2NuXdwSBtot2ELbFgQBDQLvXls7tTYVg chwmomXsYfHo0P5XA9Dx9pwOA7Dhb8rubK0LwqJ6o771Z2IKJoFtmttZTa4gXbhcu722eHz6E I9paBKgCuF+nqcyD/NT5BR3n6m/E+koGPBE3wLKyPqBkwOmY1w5vwqX7IcMWNa2KksnIxdIt3 N02HRIK+2j0QSvdqS9XkE32KwH4wSrF287ALX92DWj+CAvsLAE0Zzr38VcEgGrjgSFWzAonXh WjkgMNrbwCfCCpswxpLKOxmmAVBGRBw4BmwzfR62R+fBUjYyK+JNtU0+LwO64Lz5qfke5pBI5 NTxSNFNzgtjtUHg9N+03tSVOxujCkgBwVhkwaMUL1T1+Wwj7K5C2ACipmDLVJ1j+GHXkwcp11 NAjFnCVhSQrz0xJDFJxsew57OHvJbYFOzAKh3++4Zm/fjBbee6yc5JfnpcZeowbwTf0Xs8IGB Bl7fxhXsdDKClOT51YkZs5T2Gbl0pui9bLSo1CWQfQrm2H0sll4nZZOeg== X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 56342 Cc: 56342@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) Paul Pogonyshev writes: > Hello, Hi Paul, > any news on this? I've worked last weeks almost full time on this. With limited success. Combining stat and readlink into one remote command have even slowed down the machinery. Likely, because it isn't needed always, but it always consumes more time to compute both on the remote side, and to read all the results. Moving the check of the existence of a file to the end, and let the commands fail in case the file doesn't exist, isn't possible. Often, Tramp needs to know in advance the size of a given file, for example to decide whether to encode/decode the file or to invoke scp, or to know whether the file shall be compressed prior transfer. Checking the file size is much the same as checking the file existence. Avoiding the "are you awake" command is still on my todo list, but it isn't trivial. There is a complex machinery to be invoked in case the command fails, in order to reestablish the connection in the background. It still might be possible to change this, but it isn't trivial. Bu there are also some results. With commit 9ed5c39aad09571314097be91cb28e7504614421 I've refactored major Tramp parts in order to have a better chance to apply performance changes. This includes already some changes to combine several commands (avoiding superfluous roundtrips) and alike. There's also commit dfa16cadc18930fad76fa6113750eaa27d367e72 fixing a regression which was introduced with the former patch. In my test environment, performance improvement of these changes is not overwhelming. But I sit in a LAN, with good connections. Perhaps you give it a try to see, whether your use cases behave better. I'll continue to work on this, but don't expect wonders. > Paul Best regards, Michael. From debbugs-submit-bounces@debbugs.gnu.org Tue Jul 26 12:17:21 2022 Received: (at 56342) by debbugs.gnu.org; 26 Jul 2022 16:17:21 +0000 Received: from localhost ([127.0.0.1]:54265 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1oGNFI-0006vA-Eq for submit@debbugs.gnu.org; Tue, 26 Jul 2022 12:17:21 -0400 Received: from mail-yw1-f173.google.com ([209.85.128.173]:37388) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1oGNFG-0006uw-KC for 56342@debbugs.gnu.org; Tue, 26 Jul 2022 12:17:19 -0400 Received: by mail-yw1-f173.google.com with SMTP id 00721157ae682-31e623a4ff4so148213827b3.4 for <56342@debbugs.gnu.org>; Tue, 26 Jul 2022 09:17:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=/r5CgC8aZteyQrDEsi6qi4Y/pEmHhZCYKcvPhHtGS8A=; b=VQS53uKm0UQkaDMASdiaaEYYFG2dmOKuGQRLWxPAs6JU4ZNgQnw+rHpMG7lX/k/cY/ qNFl80PLbPqouhYXDT5vXtL+LMbyi0rN1v7upb3PuBVHTJtg9tBRfTqtdbjQJOJLG+M/ 0FxTrKEMbdQkzZMQE9ITLMe7IQyQmc8VBAoyweQ17gZuCc5FRgFU56zLPo4bLYt8Tmoc 9qt/4hNnlnkwE/Kl+YhM4OUqD54YUo7Vf16VXM4svxxdx3qyVWNaF3aTBjdyyJrB14Z4 l0kzPcZVEJsVCMUJQ8VA454uEH2r4bPojiqM1CqiuWstInN+L0wja9t0rtvwAHKqup5u BvEQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=/r5CgC8aZteyQrDEsi6qi4Y/pEmHhZCYKcvPhHtGS8A=; b=O9L6eA66w/6w4aYJp/94HTmEyaq/YDF5ScMMCxOnnkecdsj5fbtSyl5evSeirGz309 k+lJOa9uhG7By+1TnV+72R2a7D/G97MJ5WdMqkxBL+N3FmKk565CB/3cJ6APi7akHuHE 7s/C01wrbbyfJy91M68RBxW4PpFgLpP53OnAHxJo4lbMqmboPKcOA6YEkrxRr3ZQ5tdk cbz4nAvNAHmzSLPhaJqnxJrPz6VJIP5f5Mc3aaZpvWSlh7MrU79dzb6jPEdSPujWOsaQ 0OrVmua/SKDs6olwq9DNA/ChBsfVqrO9B9w804zDY72wyhqxGxWTD5VqPIoHQavvREOy rx/Q== X-Gm-Message-State: AJIora9iMqAjVYDw1CaDfRECXl7a6djBTk87ZIJPqoMx0QOU1UU7rfjI 6F+SzP/OUo78fKj7AOW3NLujR7Tys+TC8rmXug== X-Google-Smtp-Source: AGRyM1uw6MZ/5hL6b1X7Kmdu7tmY5wvROD1LSdIR9qXHvmnZbbPrGg6sKJT7kPlKz5WSeVDCqDHqcuwk6Lhstypu5LQ= X-Received: by 2002:a81:1c0a:0:b0:31e:6ab0:7f73 with SMTP id c10-20020a811c0a000000b0031e6ab07f73mr15336238ywc.353.1658852232691; Tue, 26 Jul 2022 09:17:12 -0700 (PDT) MIME-Version: 1.0 References: <8735fjh5ge.fsf@gmx.de> <87v8sefl2f.fsf@gmx.de> <874jzyc9u9.fsf@gmx.de> <87sfnhazvy.fsf@gmx.de> <87k08sc02z.fsf@gmx.de> <87r128gdob.fsf@gmx.de> In-Reply-To: <87r128gdob.fsf@gmx.de> From: Paul Pogonyshev Date: Tue, 26 Jul 2022 18:17:01 +0200 Message-ID: Subject: Re: bug#56342: TRAMP (sh) issues way too many commands, thus being very slow over high-ping networks To: Michael Albinus Content-Type: multipart/alternative; boundary="0000000000001084de05e4b7a234" X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 56342 Cc: 56342@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) --0000000000001084de05e4b7a234 Content-Type: text/plain; charset="UTF-8" > I've worked last weeks almost full time on this. Thank you! > Combining stat and readlink into one remote command have even slowed > down the machinery. Likely, because it isn't needed always, but it > always consumes more time to compute both on the remote side, and to > read all the results. Well, that's hard to believe. I mean, it is understandable that `readlink' (whatever Elisp call results in that shell command) is not often needed, but how long can that take on a modern computer, 1 ms? It might be that we evaluate performance changes differently. I care more about worst cases (since they affect me directly, yeah) and not so much if Tramp becomes 1 ms slower "per user-visible operation" on average. > Perhaps you give it a try to see, whether your use cases behave better. I tested it a bit now. Initially mentioned usecase (`insert-file-contents', but only a file part, not the whole file) appears to be considerably faster after the changes (hard to say, feels like 30-50% faster). The connection log says that number of commands is 10 now, with the originally mentioned list heavily changed. However, there must still be space for improvement, e.g. `M-x occur tramp-send RET' suggests there are outright duplicated commands now: 18:03:41.617738 tramp-send-command (6) # \readlink --canonicalize-missing /tmp/tramp.H3AZYb 2>/dev/null; echo tramp_exit_status $? 18:03:41.650294 tramp-send-command (6) # tramp_stat_file_attributes /tmp/tramp.H3AZYb 2>/dev/null; echo tramp_exit_status $? 18:03:41.699627 tramp-send-command (6) # base64 -d -i >/tmp/tramp.H3AZYb <<'9190079abe64738d52b0f040fd94461c' 2>/dev/null; echo tramp_exit_status $? 18:03:41.731669 tramp-send-command (6) # chown 1000:1001 /tmp/tramp.H3AZYb 18:03:41.760145 tramp-send-command (6) # dd bs=1 skip=8166969 if=[...] of=/tmp/tramp.H3AZYb 18:03:41.843785 tramp-send-command (6) # \readlink --canonicalize-missing /tmp/tramp.H3AZYb 2>/dev/null; echo tramp_exit_status $? 18:03:41.872072 tramp-send-command (6) # tramp_stat_file_attributes /tmp/tramp.H3AZYb 2>/dev/null; echo tramp_exit_status $? 18:03:41.902388 tramp-send-command (6) # (env GZIP= gzip /dev/null; echo tramp_exit_status $? 18:03:41.952283 tramp-send-command (6) # rm -f /tmp/tramp.H3AZYb 2>/dev/null; echo tramp_exit_status $? On the other hand, refreshing a remote-tracking Magit (Emacs Git interface) buffer doesn't feel any faster than before. Logs also suggest that number of commands that tramp send stays as before (43). Some commands are certainly excessive, e.g. it repeats `test -d [...] 2>/dev/null; echo tramp_exit_status $?' 4 times for the same directory (intermixed with other commands), suggesting that Tramp doesn't cache the result. Three times it issues `test -r ...' for the same directory, the source checkout root. Most other commands cannot be generically skipped by Tramp though, this would rather depend on Magit. Paul On Tue, 26 Jul 2022 at 16:18, Michael Albinus wrote: > Paul Pogonyshev writes: > > > Hello, > > Hi Paul, > > > any news on this? > > I've worked last weeks almost full time on this. With limited success. > > Combining stat and readlink into one remote command have even slowed > down the machinery. Likely, because it isn't needed always, but it > always consumes more time to compute both on the remote side, and to > read all the results. > > Moving the check of the existence of a file to the end, and let the > commands fail in case the file doesn't exist, isn't possible. Often, > Tramp needs to know in advance the size of a given file, for example to > decide whether to encode/decode the file or to invoke scp, or to know > whether the file shall be compressed prior transfer. Checking the file > size is much the same as checking the file existence. > > Avoiding the "are you awake" command is still on my todo list, but it > isn't trivial. There is a complex machinery to be invoked in case the > command fails, in order to reestablish the connection in the > background. It still might be possible to change this, but it isn't > trivial. > > Bu there are also some results. With commit > 9ed5c39aad09571314097be91cb28e7504614421 I've refactored major Tramp > parts in order to have a better chance to apply performance > changes. This includes already some changes to combine several commands > (avoiding superfluous roundtrips) and alike. There's also commit > dfa16cadc18930fad76fa6113750eaa27d367e72 fixing a regression which was > introduced with the former patch. > > In my test environment, performance improvement of these changes is not > overwhelming. But I sit in a LAN, with good connections. Perhaps you > give it a try to see, whether your use cases behave better. > > I'll continue to work on this, but don't expect wonders. > > > Paul > > Best regards, Michael. > --0000000000001084de05e4b7a234 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
> I've worked last weeks almost full time on this.<= br>

Thank you!

> Combining s= tat and readlink into one remote command have even slowed
>=C2=A0down= the machinery. Likely, because it isn't needed always, but it
>= =C2=A0always consumes more time to compute both on the remote side, and to<= br>>=C2=A0read all the results.

Well, that&= #39;s hard to believe. I mean, it is understandable that `readlink' (wh= atever Elisp call results in that shell command) is not often needed, but h= ow long can that take on a modern computer, 1 ms? It might be that we evalu= ate performance changes differently. I care more about worst cases (since t= hey affect me directly, yeah) and not so much if Tramp becomes 1 ms slower = "per user-visible operation" on average.

> Perhaps you
give it a try to see, whether your use cases behave be= tter.

I tested it a bit now. Initially mention= ed usecase (`insert-file-contents', but only a file part, not the whole= file) appears to be considerably faster after the changes (hard to say, fe= els like 30-50% faster). The connection log says that number of commands is= 10 now, with the originally mentioned list heavily changed. However, there= must still be space for improvement, e.g. `M-x occur tramp-send RET' s= uggests there are outright duplicated commands now:

=C2=A0 =C2=A0 18:03:41.617738 tramp-send-command (6) # \readlink --canoni= calize-missing /tmp/tramp.H3AZYb 2>/dev/null; echo tramp_exit_status $?<= br>=C2=A0 =C2=A0 18:03:41.650294 tramp-send-command (6) # tramp_stat_file_a= ttributes /tmp/tramp.H3AZYb 2>/dev/null; echo tramp_exit_status $?
= =C2=A0 =C2=A0 18:03:41.699627 tramp-send-command (6) # base64 -d -i >/tm= p/tramp.H3AZYb <<'9190079abe64738d52b0f040fd94461c' 2>/dev= /null; echo tramp_exit_status $?
=C2=A0 =C2=A0 18:03:41.731669 tramp-sen= d-command (6) # chown 1000:1001 /tmp/tramp.H3AZYb
=C2=A0 =C2=A0 18:03:41= .760145 tramp-send-command (6) # dd bs=3D1 skip=3D8166969 if=3D[...] of=3D/= tmp/tramp.H3AZYb
=C2=A0 =C2=A0 18:03:41.843785 tramp-send-command (6) # = \readlink --canonicalize-missing /tmp/tramp.H3AZYb 2>/dev/null; echo tra= mp_exit_status $?
=C2=A0 =C2=A0 18:03:41.872072 tramp-send-command (6) #= tramp_stat_file_attributes /tmp/tramp.H3AZYb 2>/dev/null; echo tramp_ex= it_status $?
=C2=A0 =C2=A0 18:03:41.902388 tramp-send-command (6) # (env= GZIP=3D gzip </tmp/tramp.H3AZYb | base64) 2>/dev/null; echo tramp_ex= it_status $?
=C2=A0 =C2=A0 18:03:41.952283 tramp-send-command (6) # rm -= f /tmp/tramp.H3AZYb 2>/dev/null; echo tramp_exit_status $?

On the other hand, refreshing a remote-tracking Magit (Emac= s Git interface)=C2=A0buffer=C2=A0doesn't feel any faster than before. = Logs also suggest that number of commands that tramp send stays as before (= 43). Some commands are certainly excessive, e.g. it repeats `test -d [...] = 2>/dev/null; echo tramp_exit_status $?' 4 times for the same directo= ry (intermixed with other commands), suggesting that Tramp doesn't cach= e the result. Three times it issues `test -r ...' for the same director= y, the source checkout root. Most other commands cannot be generically skip= ped by Tramp though, this would rather depend on Magit.

Paul

On Tue, 26 Jul 2022 at 16:18, Michael Albinus <michael.albinus@gmx.de> wrote:
Paul Pogonyshev &l= t;pogonyshev@gmai= l.com> writes:

> Hello,

Hi Paul,

> any news on this?

I've worked last weeks almost full time on this. With limited success.<= br>
Combining stat and readlink into one remote command have even slowed
down the machinery. Likely, because it isn't needed always, but it
always consumes more time to compute both on the remote side, and to
read all the results.

Moving the check of the existence of a file to the end, and let the
commands fail in case the file doesn't exist, isn't possible. Often= ,
Tramp needs to know in advance the size of a given file, for example to
decide whether to encode/decode the file or to invoke scp, or to know
whether the file shall be compressed prior transfer. Checking the file
size is much the same as checking the file existence.

Avoiding the "are you awake" command is still on my todo list, bu= t it
isn't trivial. There is a complex machinery to be invoked in case the command fails, in order to reestablish the connection in the
background. It still might be possible to change this, but it isn't
trivial.

Bu there are also some results. With commit
9ed5c39aad09571314097be91cb28e7504614421 I've refactored major Tramp parts in order to have a better chance to apply performance
changes. This includes already some changes to combine several commands
(avoiding superfluous roundtrips) and alike. There's also commit
dfa16cadc18930fad76fa6113750eaa27d367e72 fixing a regression which was
introduced with the former patch.

In my test environment, performance improvement of these changes is not
overwhelming. But I sit in a LAN, with good connections. Perhaps you
give it a try to see, whether your use cases behave better.

I'll continue to work on this, but don't expect wonders.

> Paul

Best regards, Michael.
--0000000000001084de05e4b7a234-- From debbugs-submit-bounces@debbugs.gnu.org Tue Jul 26 13:51:52 2022 Received: (at 56342) by debbugs.gnu.org; 26 Jul 2022 17:51:53 +0000 Received: from localhost ([127.0.0.1]:54334 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1oGOil-0001TR-PR for submit@debbugs.gnu.org; Tue, 26 Jul 2022 13:51:52 -0400 Received: from mout.gmx.net ([212.227.17.21]:34533) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1oGOii-0001T8-Gb for 56342@debbugs.gnu.org; Tue, 26 Jul 2022 13:51:49 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.net; s=badeba3b8450; t=1658857900; bh=gCSIGaikhw2iMZ1Q4LRIJdtM0b/ZyIXMxfo0oXyf4vc=; h=X-UI-Sender-Class:From:To:Cc:Subject:In-Reply-To:References:Date; b=YQRJsvEImDPNEJjwm+jOdbpcCwNn6CQbAA+mvSNLO58cfnpKEaodkmtyhAG0+K8sv G/aFtl4c/YkzzB67LUq8BxW+Xbt06xXR1WrvgJNg/a+9WpyFbEKrFYjumH453Y2xbS AO146VlE7IP7ws1bx1hRvnPpcbLa8wYr8MlPV0P4= X-UI-Sender-Class: 01bb95c1-4bf8-414a-932a-4f6e2808ef9c Received: from gandalf.gmx.de ([213.220.148.6]) by mail.gmx.net (mrgmx105 [212.227.17.168]) with ESMTPSA (Nemesis) id 1Mt757-1nSGw62pjd-00tRuF; Tue, 26 Jul 2022 19:51:39 +0200 From: Michael Albinus To: Paul Pogonyshev Subject: Re: bug#56342: TRAMP (sh) issues way too many commands, thus being very slow over high-ping networks In-Reply-To: (Paul Pogonyshev's message of "Tue, 26 Jul 2022 18:17:01 +0200") References: <8735fjh5ge.fsf@gmx.de> <87v8sefl2f.fsf@gmx.de> <874jzyc9u9.fsf@gmx.de> <87sfnhazvy.fsf@gmx.de> <87k08sc02z.fsf@gmx.de> <87r128gdob.fsf@gmx.de> Date: Tue, 26 Jul 2022 19:51:38 +0200 Message-ID: <87mtcvhidx.fsf@gmx.de> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/29.0.50 (gnu/linux) MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" X-Provags-ID: V03:K1:eTiYIBM/JzkiN9Zh02iTjvrXEL9+H0RDbf81EhNTOTwLw8SoLLL Wxh+VRUcXUnptQH38eIOkSl8fKASXuK/Pyds88+WQosMlPsXpHCCPn40v4cZ1cx62pyJtYx g4g4BvSbka80RVcy/9do5QF35r56/WBtpFWxBehaYo+2rL0NKRdfcDbHdDwraIHw1ghJH16 SsF6xGR6ltiof4zGuPC5A== X-Spam-Flag: NO X-UI-Out-Filterresults: notjunk:1;V03:K0:IZcu8Uy6Ctw=:CPCoManlUb8Dutcvn0SjyS NIlu1XF+xDFP71UAoc9VD3DeBDY8ZMToy4OUrggS1LQNS/zt5apbwF2J+gzB0j0sLgR6dRmLr Oa903u7JLldqShKouaVlPY6caPUD38MHazQkMC4Sr0iDiccYS65/Ixd1lz0dC6V0nFUtYfmmL dsv2TslRH3dZNvuiKmFEbk4zMAPKoY5rZxxM/A76K9o+orfJYESGVl5zVlwsxEpEint7AmSXg GJU3rSNKAV1xy5oJwozV63V/r5stOHkrZqi0HUx0P3Gng1gamBKSbYXOd000Un9rgyY69vuTr 92GsRpAH0VspRIwXm6gXrdagJhvH+02dnk0fkLq/1OM1Dvw9J+I/5JzXQ8X09khoAoKYOG+Ji H6nelqVRWAfhn71Q2dzdzyDnsE+3ctb3bDhvT057ZBpXd0RauIojQ3pj9PDFUj736u/AvH0Tc TW1bSdE0B/Nop+j0LBrsoqb4gdUEJl6lAwhhYPsMHdaDmRs1bJI78OfV5QQZSoiP0G4uNd5jr iz6BFciOf5FH7JJmE01HqiEBC8JFlUPx6XXZnftYOtt0UFh96sKUyE3LpzfVa3DBrXKRB/FEp px1GFgY3acvmsFl/RZnw5WRS+wJDclJxiouengvhUVxLiC2y2sF3u6CcTrKtYmbzVLnVdI+lR LfNxHuOyMcA4vhVSWGDPZ2hpLgWHHTp9WrpTKbVn34H8dd8d7Xywe+fEvAzxMXm4h1AHEXnQL MLyt5nuU1/Kf/hydD1LjrrjYd//fvxTzg7/4qRlLxydO7xIXaCMDN1ASY9O8HrJw1BNNIuYTz hi/c/+psIcXV1X4ibDoqysEXN1kcrdu1cokfGe5hVvjNS30ZJ3vsg8s6OBHN7Ai62E/0zZcw/ b1BAZCOdDo/wuqQpnQyQg85PBtXzitBi7sFWkhjftU65SOzMk1jBU4G1GP9JmKi3prWsSKLj2 21SzvRNafP0FnYEccgSyVEbdaUA5eTTBdtSNPKE57+q42EodADZT6Yyrg9Tmke/MgZiUzSuvC V0f0l7KHZWkblKf9vSLFuCwmydYdw1PYYIwH4B68QBe5ppUnS1RiZnhbdAhCAr75vmDOOS0FL zo8F2UiqyrLaAo+pxYPtvndMZ+J2USg9OVtJcaURXTVxYJ+JSEiNViHmg== X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 56342 Cc: 56342@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) --=-=-= Content-Type: text/plain Paul Pogonyshev writes: Hi Paul, >> Combining stat and readlink into one remote command have even slowed >> down the machinery. Likely, because it isn't needed always, but it >> always consumes more time to compute both on the remote side, and to >> read all the results. > > Well, that's hard to believe. I mean, it is understandable that > `readlink' (whatever Elisp call results in that shell command) is not > often needed, but how long can that take on a modern computer, 1 ms? > It might be that we evaluate performance changes differently. I care > more about worst cases (since they affect me directly, yeah) and not > so much if Tramp becomes 1 ms slower "per user-visible operation" on > average. As said, I live in a pretty good connected LAN. remote commands do not result in heavy performance degradations, compared with the rest of Elisp computing. > I tested it a bit now. Initially mentioned usecase > (`insert-file-contents', but only a file part, not the whole file) > appears to be considerably faster after the changes (hard to say, > feels like 30-50% faster). Oh, good. In my environment it was something about 5%. > The connection log says that number of > commands is 10 now, with the originally mentioned list heavily > changed. However, there must still be space for improvement, e.g. `M-x > occur tramp-send RET' suggests there are outright duplicated commands > now: Pretty sure there's still place for improvements! > On the other hand, refreshing a remote-tracking Magit (Emacs Git > interface) buffer doesn't feel any faster than before. Logs also > suggest that number of commands that tramp send stays as before (43). > Some commands are certainly excessive, e.g. it repeats `test -d [...] > 2>/dev/null; echo tramp_exit_status $?' 4 times for the same directory > (intermixed with other commands), suggesting that Tramp doesn't cache > the result. Three times it issues `test -r ...' for the same > directory, the source checkout root. Most other commands cannot be > generically skipped by Tramp though, this would rather depend on > Magit. Caches are problematic. They are limited to 10 seconds by default, did you try to increase that number? VC related commands are also problematic, Emacs tests different VC's for availability. It is always a good idea to reduce the number of VC backends. If you completely trust in magit, you shall deactivate Emacs' vc.el. Something like (setq vc-handled-backends nil) should help. And directories are also problematic for caches. As soon as something changes there (creation or deletion of a file, for example), the cached values of the directory must be flushed. I propose we try step by step. That is, we investigate typical code snippets, and see what Tramp does. Case by case. The Tramp debug buffer is a little bit noisy. So you could do (progn (trace-function-background 'tramp-send-command) (trace-function-background 'tramp-file-name-handler)) By this, only the calls of tramp-send-command (go remote) and tramp-file-name-handler (any invocation of Tramp) are traced. The latter is important to know, what has triggered a given remote command. Starting with your initial example (with-temp-buffer (insert-file-contents "/ssh:detlef:.emacs" nil 100 nil)) I get the *trace-output* buffer as appended. Let's study this first, and see, what could be done in order to improve this case. Obviously, most of the commands are about the temporary file /tmp/tramp.qgnQ01, which is superfluous. I'll check whether handling temporary files could be handled better. When we're happy with this example, let's go to the next example. > Paul Best regards, Michael. PS: Next 2 or 3 days I'm kind of absent: my granddaughter will visit us. Sorry, but you know: priorities :-) --=-=-= Content-Type: text/plain Content-Disposition: attachment ====================================================================== 1 -> (tramp-file-name-handler expand-file-name "/ssh:detlef:.emacs" nil) 1 <- tramp-file-name-handler: "/ssh:detlef:/home/albinus/.emacs" ====================================================================== 1 -> (tramp-file-name-handler insert-file-contents "/ssh:detlef:/home/albinus/.emacs" nil 2 4 nil) | 2 -> (tramp-file-name-handler expand-file-name "/ssh:detlef:/home/albinus/.emacs" nil) | 2 <- tramp-file-name-handler: "/ssh:detlef:/home/albinus/.emacs" | 2 -> (tramp-file-name-handler expand-file-name "tramp." "/ssh:detlef:/tmp") | 2 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp." | 2 -> (tramp-file-name-handler expand-file-name "/ssh:detlef:/tmp/tramp." "/tmp/") | 2 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp." | 2 -> (tramp-file-name-handler expand-file-name "/ssh:detlef:/tmp/tramp.qgnQ01" nil) | 2 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01" | 2 -> (tramp-file-name-handler write-region "" nil #1="/ssh:detlef:/tmp/tramp.qgnQ01" nil silent #1# excl) | | 3 -> (tramp-file-name-handler expand-file-name "/ssh:detlef:/tmp/tramp.qgnQ01" nil) | | 3 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01" | | 3 -> (tramp-file-name-handler file-truename "/ssh:detlef:/tmp/tramp.qgnQ01") | | | 4 -> (tramp-file-name-handler file-remote-p "/ssh:detlef:/tmp/tramp.qgnQ01" localname nil) | | | | 5 -> (tramp-file-name-handler expand-file-name "/ssh:detlef:/tmp/tramp.qgnQ01" nil) | | | | 5 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01" | | | 4 <- tramp-file-name-handler: "/tmp/tramp.qgnQ01" | | | 4 -> (tramp-file-name-handler expand-file-name "/ssh:detlef:/tmp/tramp.qgnQ01" nil) | | | 4 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01" | | | 4 -> (tramp-file-name-handler file-remote-p "/ssh:detlef:/tmp/tramp.qgnQ01" localname nil) | | | | 5 -> (tramp-file-name-handler expand-file-name "/ssh:detlef:/tmp/tramp.qgnQ01" nil) | | | | 5 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01" | | | 4 <- tramp-file-name-handler: "/tmp/tramp.qgnQ01" | | | 4 -> (tramp-file-name-handler file-remote-p "/ssh:detlef:/tmp/tramp.qgnQ01" nil nil) | | | | 5 -> (tramp-file-name-handler expand-file-name "/ssh:detlef:/tmp/tramp.qgnQ01" nil) | | | | 5 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01" | | | 4 <- tramp-file-name-handler: "/ssh:detlef:" | | | 4 -> (tramp-send-command (tramp-file-name "ssh" nil nil "detlef" nil "/tmp/tramp.qgnQ01" nil) "\\readlink --canonicalize-missing /tmp/tramp.qgnQ01 2>/dev/null; echo tramp_exit_status $?") | | | | 5 -> (tramp-send-command (tramp-file-name "ssh" nil nil "detlef" nil "/tmp/tramp.qgnQ01" nil) "echo are you awake" t t) | | | | 5 <- tramp-send-command: nil | | | 4 <- tramp-send-command: 39 | | | 4 -> (tramp-file-name-handler expand-file-name "/ssh:detlef:/tmp/tramp.qgnQ01" nil) | | | 4 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01" | | | 4 -> (tramp-file-name-handler file-symlink-p "/ssh:detlef:/tmp/tramp.qgnQ01") | | | | 5 -> (tramp-file-name-handler expand-file-name "/ssh:detlef:/tmp/tramp.qgnQ01" nil) | | | | 5 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01" | | | | 5 -> (tramp-file-name-handler file-attributes "/ssh:detlef:/tmp/tramp.qgnQ01") | | | | | 6 -> (tramp-file-name-handler expand-file-name "/ssh:detlef:/tmp/tramp.qgnQ01" nil) | | | | | 6 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01" | | | | | 6 -> (tramp-send-command (tramp-file-name "ssh" nil nil "detlef" nil "/tmp/tramp.qgnQ01" nil) "tramp_stat_file_attributes /tmp/tramp.qgnQ01 2>/dev/null; echo tramp_exit_status $?") | | | | | 6 <- tramp-send-command: 25 | | | | 5 <- tramp-file-name-handler: nil | | | 4 <- tramp-file-name-handler: nil | | 3 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01" | | 3 -> (tramp-file-name-handler expand-file-name "/ssh:detlef:/tmp/tramp.qgnQ01" nil) | | 3 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01" | | 3 -> (tramp-file-name-handler file-exists-p "/ssh:detlef:/tmp/tramp.qgnQ01") | | 3 <- tramp-file-name-handler: nil | | 3 -> (tramp-file-name-handler file-locked-p "/ssh:detlef:/tmp/tramp.qgnQ01") | | | 4 -> (tramp-file-name-handler make-lock-file-name "/ssh:detlef:/tmp/tramp.qgnQ01") | | | 4 <- tramp-file-name-handler: nil | | 3 <- tramp-file-name-handler: nil | | 3 -> (tramp-file-name-handler expand-file-name "/ssh:detlef:/tmp/tramp.qgnQ01" nil) | | 3 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01" | | 3 -> (tramp-file-name-handler file-attributes "/ssh:detlef:/tmp/tramp.qgnQ01" integer) | | | 4 -> (tramp-file-name-handler expand-file-name "/ssh:detlef:/tmp/tramp.qgnQ01" nil) | | | 4 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01" | | 3 <- tramp-file-name-handler: nil | | 3 -> (tramp-file-name-handler expand-file-name "/ssh:detlef:/tmp/tramp.qgnQ01" nil) | | 3 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01" | | 3 -> (tramp-file-name-handler file-attributes "/ssh:detlef:/tmp/tramp.qgnQ01" integer) | | | 4 -> (tramp-file-name-handler expand-file-name "/ssh:detlef:/tmp/tramp.qgnQ01" nil) | | | 4 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01" | | 3 <- tramp-file-name-handler: nil | | 3 -> (tramp-file-name-handler expand-file-name "/ssh:detlef:/tmp/tramp.qgnQ01" nil) | | 3 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01" | | 3 -> (tramp-file-name-handler file-acl "/ssh:detlef:/tmp/tramp.qgnQ01") | | 3 <- tramp-file-name-handler: nil | | 3 -> (tramp-file-name-handler expand-file-name "/ssh:detlef:/tmp/tramp.qgnQ01" nil) | | 3 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01" | | 3 -> (tramp-file-name-handler file-selinux-context "/ssh:detlef:/tmp/tramp.qgnQ01") | | 3 <- tramp-file-name-handler: (nil nil nil nil) | | 3 -> (tramp-file-name-handler file-name-nondirectory "/ssh:detlef:/tmp/tramp.qgnQ01") | | 3 <- tramp-file-name-handler: "tramp.qgnQ01" | | 3 -> (tramp-file-name-handler file-remote-p "/ssh:detlef:/tmp/tramp.qgnQ01" nil nil) | | | 4 -> (tramp-file-name-handler expand-file-name "/ssh:detlef:/tmp/tramp.qgnQ01" nil) | | | 4 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01" | | 3 <- tramp-file-name-handler: "/ssh:detlef:" | | 3 -> (tramp-file-name-handler lock-file "/ssh:detlef:/tmp/tramp.qgnQ01") | | | 4 -> (tramp-file-name-handler file-locked-p "/ssh:detlef:/tmp/tramp.qgnQ01") | | | | 5 -> (tramp-file-name-handler make-lock-file-name "/ssh:detlef:/tmp/tramp.qgnQ01") | | | | 5 <- tramp-file-name-handler: nil | | | 4 <- tramp-file-name-handler: nil | | | 4 -> (tramp-file-name-handler make-lock-file-name "/ssh:detlef:/tmp/tramp.qgnQ01") | | | 4 <- tramp-file-name-handler: nil | | | 4 -> (tramp-file-name-handler make-lock-file-name "/ssh:detlef:/tmp/tramp.qgnQ01") | | | 4 <- tramp-file-name-handler: nil | | 3 <- tramp-file-name-handler: nil | | 3 -> (tramp-file-name-handler expand-file-name "/ssh:detlef:/tmp/tramp.qgnQ01" nil) | | 3 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01" | | 3 -> (tramp-file-name-handler file-modes "/ssh:detlef:/tmp/tramp.qgnQ01" nofollow) | | | 4 -> (tramp-file-name-handler expand-file-name "/ssh:detlef:/tmp/tramp.qgnQ01" nil) | | | 4 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01" | | | 4 -> (tramp-file-name-handler file-attributes "/ssh:detlef:/tmp/tramp.qgnQ01") | | | | 5 -> (tramp-file-name-handler expand-file-name "/ssh:detlef:/tmp/tramp.qgnQ01" nil) | | | | 5 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01" | | | 4 <- tramp-file-name-handler: nil | | 3 <- tramp-file-name-handler: nil | | 3 -> (tramp-file-name-handler file-name-nondirectory "/ssh:detlef:/tmp/tramp.qgnQ01") | | 3 <- tramp-file-name-handler: "tramp.qgnQ01" | | 3 -> (tramp-send-command (tramp-file-name "ssh" nil nil "detlef" nil "/tmp/tramp.qgnQ01" nil) "base64 -d -i >/tmp/tramp.qgnQ01 <<'b03f0306a9a1e716a81252ccc4134b92' 2>/dev/null; echo tramp_exit_status $?\nb03f0306a9a1e716a81252ccc4134b92") | | 3 <- tramp-send-command: 21 | | 3 -> (tramp-file-name-handler file-name-directory "/ssh:detlef:/tmp/tramp.qgnQ01") | | 3 <- tramp-file-name-handler: "/ssh:detlef:/tmp/" | | 3 -> (tramp-file-name-handler expand-file-name "/ssh:detlef:/tmp/" nil) | | 3 <- tramp-file-name-handler: "/ssh:detlef:/tmp/" | | 3 -> (tramp-file-name-handler directory-file-name "/ssh:detlef:/tmp/") | | 3 <- tramp-file-name-handler: "/ssh:detlef:/tmp" | | 3 -> (tramp-file-name-handler file-modes "/ssh:detlef:/tmp" nil) | | | 4 -> (tramp-file-name-handler expand-file-name "/ssh:detlef:/tmp" nil) | | | 4 <- tramp-file-name-handler: "/ssh:detlef:/tmp" | | | 4 -> (tramp-file-name-handler file-attributes "/ssh:detlef:/tmp") | | | | 5 -> (tramp-file-name-handler expand-file-name "/ssh:detlef:/tmp" nil) | | | | 5 <- tramp-file-name-handler: "/ssh:detlef:/tmp" | | | | 5 -> (tramp-send-command (tramp-file-name "ssh" nil nil "detlef" nil "/tmp" nil) "tramp_stat_file_attributes /tmp 2>/dev/null; echo tramp_exit_status $?") | | | | 5 <- tramp-send-command: 129 | | | 4 <- tramp-file-name-handler: (t 30 0 0 (25305 58381 0 0) (25312 7412 0 0) (25312 7412 0 0) 135168 "drwxrwxrwt" t 3801089 (-1 . 2)) | | 3 <- tramp-file-name-handler: 1023 | | 3 -> (tramp-file-name-handler tramp-set-file-uid-gid "/ssh:detlef:/tmp/tramp.qgnQ01" 1000 1000) | | | 4 -> (tramp-send-command (tramp-file-name "ssh" nil nil "detlef" nil "/tmp/tramp.qgnQ01" nil) "chown 1000:1000 /tmp/tramp.qgnQ01") | | | 4 <- tramp-send-command: 1 | | 3 <- tramp-file-name-handler: 1 | | 3 -> (tramp-file-name-handler expand-file-name "/ssh:detlef:/tmp/tramp.qgnQ01" "~/") | | 3 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01" | | 3 -> (tramp-file-name-handler set-file-acl "/ssh:detlef:/tmp/tramp.qgnQ01" nil) | | | 4 -> (tramp-file-name-handler expand-file-name "/ssh:detlef:/tmp/tramp.qgnQ01" nil) | | | 4 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01" | | 3 <- tramp-file-name-handler: nil | | 3 -> (tramp-file-name-handler expand-file-name "/ssh:detlef:/tmp/tramp.qgnQ01" "~/") | | 3 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01" | | 3 -> (tramp-file-name-handler set-file-selinux-context "/ssh:detlef:/tmp/tramp.qgnQ01" (nil nil nil nil)) | | 3 <- tramp-file-name-handler: nil | | 3 -> (tramp-file-name-handler unlock-file "/ssh:detlef:/tmp/tramp.qgnQ01") | | | 4 -> (tramp-file-name-handler make-lock-file-name "/ssh:detlef:/tmp/tramp.qgnQ01") | | | 4 <- tramp-file-name-handler: nil | | 3 <- tramp-file-name-handler: nil | 2 <- tramp-file-name-handler: nil | 2 -> (tramp-send-command (tramp-file-name "ssh" nil nil "detlef" nil "/home/albinus/.emacs" nil) "dd bs=1 skip=2 if=/home/albinus/.emacs count=2 of=/tmp/tramp.qgnQ01") | 2 <- tramp-send-command: 72 | 2 -> (tramp-file-name-handler file-local-copy "/ssh:detlef:/tmp/tramp.qgnQ01") | | 3 -> (tramp-file-name-handler file-truename "/ssh:detlef:/tmp/tramp.qgnQ01") | | | 4 -> (tramp-file-name-handler file-remote-p "/ssh:detlef:/tmp/tramp.qgnQ01" localname nil) | | | | 5 -> (tramp-file-name-handler expand-file-name "/ssh:detlef:/tmp/tramp.qgnQ01" nil) | | | | 5 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01" | | | 4 <- tramp-file-name-handler: "/tmp/tramp.qgnQ01" | | | 4 -> (tramp-file-name-handler expand-file-name "/ssh:detlef:/tmp/tramp.qgnQ01" nil) | | | 4 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01" | | | 4 -> (tramp-file-name-handler file-remote-p "/ssh:detlef:/tmp/tramp.qgnQ01" localname nil) | | | | 5 -> (tramp-file-name-handler expand-file-name "/ssh:detlef:/tmp/tramp.qgnQ01" nil) | | | | 5 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01" | | | 4 <- tramp-file-name-handler: "/tmp/tramp.qgnQ01" | | | 4 -> (tramp-file-name-handler file-remote-p "/ssh:detlef:/tmp/tramp.qgnQ01" nil nil) | | | | 5 -> (tramp-file-name-handler expand-file-name "/ssh:detlef:/tmp/tramp.qgnQ01" nil) | | | | 5 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01" | | | 4 <- tramp-file-name-handler: "/ssh:detlef:" | | | 4 -> (tramp-send-command (tramp-file-name "ssh" nil nil "detlef" nil "/tmp/tramp.qgnQ01" nil) "\\readlink --canonicalize-missing /tmp/tramp.qgnQ01 2>/dev/null; echo tramp_exit_status $?") | | | 4 <- tramp-send-command: 39 | | | 4 -> (tramp-file-name-handler expand-file-name "/ssh:detlef:/tmp/tramp.qgnQ01" nil) | | | 4 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01" | | | 4 -> (tramp-file-name-handler file-symlink-p "/ssh:detlef:/tmp/tramp.qgnQ01") | | | | 5 -> (tramp-file-name-handler expand-file-name "/ssh:detlef:/tmp/tramp.qgnQ01" nil) | | | | 5 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01" | | | | 5 -> (tramp-file-name-handler file-attributes "/ssh:detlef:/tmp/tramp.qgnQ01") | | | | | 6 -> (tramp-file-name-handler expand-file-name "/ssh:detlef:/tmp/tramp.qgnQ01" nil) | | | | | 6 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01" | | | | | 6 -> (tramp-send-command (tramp-file-name "ssh" nil nil "detlef" nil "/tmp/tramp.qgnQ01" nil) "tramp_stat_file_attributes /tmp/tramp.qgnQ01 2>/dev/null; echo tramp_exit_status $?") | | | | | 6 <- tramp-send-command: 148 | | | | 5 <- tramp-file-name-handler: (nil 1 1000 1000 (25312 7412 0 0) (25312 7412 0 0) (25312 7412 0 0) 2 "-rw-rw-r--" nil 3821258 (-1 . 2)) | | | 4 <- tramp-file-name-handler: nil | | 3 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01" | | 3 -> (tramp-file-name-handler file-name-nondirectory "/ssh:detlef:/tmp/tramp.qgnQ01") | | 3 <- tramp-file-name-handler: "tramp.qgnQ01" | | 3 -> (tramp-file-name-handler expand-file-name "/ssh:detlef:/tmp/tramp.qgnQ01" nil) | | 3 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01" | | 3 -> (tramp-file-name-handler file-attributes "/ssh:detlef:/tmp/tramp.qgnQ01") | | | 4 -> (tramp-file-name-handler expand-file-name "/ssh:detlef:/tmp/tramp.qgnQ01" nil) | | | 4 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01" | | 3 <- tramp-file-name-handler: (nil 1 1000 1000 (25312 7412 0 0) (25312 7412 0 0) (25312 7412 0 0) 2 "-rw-rw-r--" nil 3821258 (-1 . 2)) | | 3 -> (tramp-send-command (tramp-file-name "ssh" nil nil "detlef" nil "/tmp/tramp.qgnQ01" nil) "base64 /dev/null; echo tramp_exit_status $?") | | 3 <- tramp-send-command: 26 | | 3 -> (tramp-file-name-handler expand-file-name "/ssh:detlef:/tmp/tramp.qgnQ01" nil) | | 3 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01" | | 3 -> (tramp-file-name-handler file-modes "/ssh:detlef:/tmp/tramp.qgnQ01" nil) | | | 4 -> (tramp-file-name-handler expand-file-name "/ssh:detlef:/tmp/tramp.qgnQ01" nil) | | | 4 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01" | | | 4 -> (tramp-file-name-handler file-attributes "/ssh:detlef:/tmp/tramp.qgnQ01") | | | | 5 -> (tramp-file-name-handler expand-file-name "/ssh:detlef:/tmp/tramp.qgnQ01" nil) | | | | 5 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01" | | | 4 <- tramp-file-name-handler: (nil 1 1000 1000 (25312 7412 0 0) (25312 7412 0 0) (25312 7412 0 0) 2 "-rw-rw-r--" nil 3821258 (-1 . 2)) | | 3 <- tramp-file-name-handler: 436 | 2 <- tramp-file-name-handler: "/tmp/tramp.6ce5x9.qgnQ01" | 2 -> (tramp-file-name-handler expand-file-name "/ssh:detlef:/tmp/tramp.qgnQ01" nil) | 2 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01" | 2 -> (tramp-file-name-handler file-directory-p "/ssh:detlef:/tmp/tramp.qgnQ01") | 2 <- tramp-file-name-handler: nil | 2 -> (tramp-file-name-handler expand-file-name "/ssh:detlef:/tmp/tramp.qgnQ01" nil) | 2 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01" | 2 -> (tramp-file-name-handler delete-file "/ssh:detlef:/tmp/tramp.qgnQ01" nil) | | 3 -> (tramp-file-name-handler expand-file-name "/ssh:detlef:/tmp/tramp.qgnQ01" nil) | | 3 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01" | | 3 -> (tramp-send-command (tramp-file-name "ssh" nil nil "detlef" nil "/tmp/tramp.qgnQ01" nil) "rm -f /tmp/tramp.qgnQ01 2>/dev/null; echo tramp_exit_status $?") | | 3 <- tramp-send-command: 21 | 2 <- tramp-file-name-handler: t 1 <- tramp-file-name-handler: ("/ssh:detlef:/home/albinus/.emacs" 2) --=-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Mon Aug 01 16:21:05 2022 Received: (at 56342) by debbugs.gnu.org; 1 Aug 2022 20:21:05 +0000 Received: from localhost ([127.0.0.1]:41708 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1oIbuQ-0002c9-Pe for submit@debbugs.gnu.org; Mon, 01 Aug 2022 16:21:05 -0400 Received: from mail-oa1-f47.google.com ([209.85.160.47]:38711) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1oIbuM-0002bG-DS for 56342@debbugs.gnu.org; Mon, 01 Aug 2022 16:21:01 -0400 Received: by mail-oa1-f47.google.com with SMTP id 586e51a60fabf-10ee900cce0so3040109fac.5 for <56342@debbugs.gnu.org>; Mon, 01 Aug 2022 13:20:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=c2pwCdT3ROKkjRiqywkTD1ufWj6tVX6cmM2APwTRvgI=; b=aINzs4KtinTIAChkWviRHLJeWLzejQ1BitTQJKh1VtitsEYyahTGq1Z7bFREPPHlkM KOgXs2Uf+y8BmhoiiE3Is9P2+0dhl8kJ10DIUAWt7m2S1p0iWdgNEt7QtRoVnc5wM+eg nZGhNpxX25h5mrBFCmCE3obim/AlXgUgfH00mw72nuiQp+Zw5MO4IMjrKSOnmu3yZC7S IImKWqJF23hNIDqiuTzr03qthZyHLTG+NzK6cyzYWhxaNkBFPLeR4E3GCamGMg5d0CM2 5UnNwJ/sssM1kTHDbp47NotAkeqbynnFZCwsW42Uw6dghVaj9Go6Qsi5LmDqhFFHhd5T UcHA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=c2pwCdT3ROKkjRiqywkTD1ufWj6tVX6cmM2APwTRvgI=; b=VtC16oiQilKE6iqDKsjwRRogaUCboGb2mLJ72RPhI3FaP3tQXrdRWQ8itrfOHsmJZp e03Agvqdu3oBJ4nuG6ILdyPiPz4qrFJXTV0NNxf5IK8BwASP1/RwgaonKdwxPZ+y1+12 g9ClvUygCxZL5cLjwmMLjSewQ64ssdT83perxoupMiDB84OKyXRQaW3pnmTRJd3I5Rz/ KpczXlvh0QVft11WRK+QaR7neNuKYFGgPp1Ldz+wimIRsWac/lSsJTS78uIfRB1nuMs8 NgnjRdTvzAgjPnYc2QUVDXhoju6ut/wdMzJKzBq3p5m+gCJK4SRp4xTlCSZRw4TQMpQQ JlJA== X-Gm-Message-State: AJIora/UOuiJBNdU8cCvMgpxqr310ofkzDyI4Dg6GvKWavIy2qb+5gL8 gaCrLIdQYSPxVFn/fktTD8TNEsffXe6dOBCGdH5nBn6eDA== X-Google-Smtp-Source: AGRyM1setK2PZbCZm8msb2xkFoWb7u961sZmnA461rbPA9jR6xAVRR5lzmdslHvtDJm0vDSPPSos+Je/GlwWW2wR3/U= X-Received: by 2002:a05:6870:d78a:b0:10e:45a6:a400 with SMTP id bd10-20020a056870d78a00b0010e45a6a400mr8034013oab.27.1659385252458; Mon, 01 Aug 2022 13:20:52 -0700 (PDT) MIME-Version: 1.0 References: <8735fjh5ge.fsf@gmx.de> <87v8sefl2f.fsf@gmx.de> <874jzyc9u9.fsf@gmx.de> <87sfnhazvy.fsf@gmx.de> <87k08sc02z.fsf@gmx.de> <87r128gdob.fsf@gmx.de> <87mtcvhidx.fsf@gmx.de> In-Reply-To: <87mtcvhidx.fsf@gmx.de> From: Paul Pogonyshev Date: Mon, 1 Aug 2022 22:20:41 +0200 Message-ID: Subject: Re: bug#56342: TRAMP (sh) issues way too many commands, thus being very slow over high-ping networks To: Michael Albinus Content-Type: multipart/alternative; boundary="00000000000084b39605e533bc46" X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 56342 Cc: 56342@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) --00000000000084b39605e533bc46 Content-Type: text/plain; charset="UTF-8" > As said, I live in a pretty good connected LAN. remote commands do not > result in heavy performance degradations, compared with the rest of > Elisp computing. Understood, but I propose to adopt a different benchmark: number of remote commands. As soon as you are not in a LAN, even not when using a particularly slow network, this becomes an order or two of magnitude more important than everything else. E.g. with slightly slower commands or particularly inefficient Elisp you can make it 2-10 ms slower for everyone. But with unoptimized command flow (i.e. more remote commands) it can be 100-500 ms slower, though not for everyone, but people using this over non-LAN. I think in such cases extremes are more important than the average. > > On the other hand, refreshing a remote-tracking Magit (Emacs Git > > interface) buffer doesn't feel any faster than before. Logs also > > suggest that number of commands that tramp send stays as before (43). > > Some commands are certainly excessive, e.g. it repeats `test -d [...] > > 2>/dev/null; echo tramp_exit_status $?' 4 times for the same directory > > (intermixed with other commands), suggesting that Tramp doesn't cache > > the result. Three times it issues `test -r ...' for the same > > directory, the source checkout root. Most other commands cannot be > > generically skipped by Tramp though, this would rather depend on > > Magit. > > Caches are problematic. They are limited to 10 seconds by default, did > you try to increase that number? Yes, `remote-file-name-inhibit-cache' is now 600 here. > And directories are also problematic for caches. As soon as something > changes there (creation or deletion of a file, for example), the cached > values of the directory must be flushed. Yeah, I suppose Tramp has no generic way to know that Magit issues reading commands. Can we devise a generic interface that would tell Tramp: "Commands within this block will not modify file system", e.g. with let-binding some variable? In general, it feels like Tramp flushes its caches too often or maybe doesn't even cache certain things at all. I.e. it's not about those 10 seconds (following your advice I have even increased that). It's that while serving one user-level command here (i.e. within 3 seconds at most), it can issue the same remote command several times, thus not reusing previous results. E.g. in the traces you have attached this can be seen. The following two commands repeat twice: | | | 4 -> (tramp-send-command (tramp-file-name "ssh" nil nil "detlef" nil "/tmp/tramp.qgnQ01" nil) "\\readlink --canonicalize-missing /tmp/tramp.qgnQ01 2>/dev/null; echo tramp_exit_status $?") ... | | | 4 -> (tramp-send-command (tramp-file-name "ssh" nil nil "detlef" nil "/tmp/tramp.qgnQ01" nil) "\\readlink --canonicalize-missing /tmp/tramp.qgnQ01 2>/dev/null; echo tramp_exit_status $?") and | | | | | 6 -> (tramp-send-command (tramp-file-name "ssh" nil nil "detlef" nil "/tmp/tramp.qgnQ01" nil) "tramp_stat_file_attributes /tmp/tramp.qgnQ01 2>/dev/null; echo tramp_exit_status $?") ... | | | | | 6 -> (tramp-send-command (tramp-file-name "ssh" nil nil "detlef" nil "/tmp/tramp.qgnQ01" nil) "tramp_stat_file_attributes /tmp/tramp.qgnQ01 2>/dev/null; echo tramp_exit_status $?") Paul On Tue, 26 Jul 2022 at 19:51, Michael Albinus wrote: > Paul Pogonyshev writes: > > Hi Paul, > > >> Combining stat and readlink into one remote command have even slowed > >> down the machinery. Likely, because it isn't needed always, but it > >> always consumes more time to compute both on the remote side, and to > >> read all the results. > > > > Well, that's hard to believe. I mean, it is understandable that > > `readlink' (whatever Elisp call results in that shell command) is not > > often needed, but how long can that take on a modern computer, 1 ms? > > It might be that we evaluate performance changes differently. I care > > more about worst cases (since they affect me directly, yeah) and not > > so much if Tramp becomes 1 ms slower "per user-visible operation" on > > average. > > As said, I live in a pretty good connected LAN. remote commands do not > result in heavy performance degradations, compared with the rest of > Elisp computing. > > > I tested it a bit now. Initially mentioned usecase > > (`insert-file-contents', but only a file part, not the whole file) > > appears to be considerably faster after the changes (hard to say, > > feels like 30-50% faster). > > Oh, good. In my environment it was something about 5%. > > > The connection log says that number of > > commands is 10 now, with the originally mentioned list heavily > > changed. However, there must still be space for improvement, e.g. `M-x > > occur tramp-send RET' suggests there are outright duplicated commands > > now: > > Pretty sure there's still place for improvements! > > > On the other hand, refreshing a remote-tracking Magit (Emacs Git > > interface) buffer doesn't feel any faster than before. Logs also > > suggest that number of commands that tramp send stays as before (43). > > Some commands are certainly excessive, e.g. it repeats `test -d [...] > > 2>/dev/null; echo tramp_exit_status $?' 4 times for the same directory > > (intermixed with other commands), suggesting that Tramp doesn't cache > > the result. Three times it issues `test -r ...' for the same > > directory, the source checkout root. Most other commands cannot be > > generically skipped by Tramp though, this would rather depend on > > Magit. > > Caches are problematic. They are limited to 10 seconds by default, did > you try to increase that number? VC related commands are also > problematic, Emacs tests different VC's for availability. It is always a > good idea to reduce the number of VC backends. If you completely trust > in magit, you shall deactivate Emacs' vc.el. Something like (setq > vc-handled-backends nil) should help. > > And directories are also problematic for caches. As soon as something > changes there (creation or deletion of a file, for example), the cached > values of the directory must be flushed. > > I propose we try step by step. That is, we investigate typical code > snippets, and see what Tramp does. Case by case. > > The Tramp debug buffer is a little bit noisy. So you could do > > (progn > (trace-function-background 'tramp-send-command) > (trace-function-background 'tramp-file-name-handler)) > > By this, only the calls of tramp-send-command (go remote) and > tramp-file-name-handler (any invocation of Tramp) are traced. The latter > is important to know, what has triggered a given remote command. > > Starting with your initial example > > (with-temp-buffer > (insert-file-contents "/ssh:detlef:.emacs" nil 100 nil)) > > I get the *trace-output* buffer as appended. Let's study this first, and > see, what could be done in order to improve this case. Obviously, most > of the commands are about the temporary file /tmp/tramp.qgnQ01, which is > superfluous. I'll check whether handling temporary files could be > handled better. > > When we're happy with this example, let's go to the next example. > > > Paul > > Best regards, Michael. > > PS: Next 2 or 3 days I'm kind of absent: my granddaughter will visit > us. Sorry, but you know: priorities :-) > > ====================================================================== > 1 -> (tramp-file-name-handler expand-file-name "/ssh:detlef:.emacs" nil) > 1 <- tramp-file-name-handler: "/ssh:detlef:/home/albinus/.emacs" > ====================================================================== > 1 -> (tramp-file-name-handler insert-file-contents > "/ssh:detlef:/home/albinus/.emacs" nil 2 4 nil) > | 2 -> (tramp-file-name-handler expand-file-name > "/ssh:detlef:/home/albinus/.emacs" nil) > | 2 <- tramp-file-name-handler: "/ssh:detlef:/home/albinus/.emacs" > | 2 -> (tramp-file-name-handler expand-file-name "tramp." > "/ssh:detlef:/tmp") > | 2 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp." > | 2 -> (tramp-file-name-handler expand-file-name "/ssh:detlef:/tmp/tramp." > "/tmp/") > | 2 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp." > | 2 -> (tramp-file-name-handler expand-file-name > "/ssh:detlef:/tmp/tramp.qgnQ01" nil) > | 2 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01" > | 2 -> (tramp-file-name-handler write-region "" nil > #1="/ssh:detlef:/tmp/tramp.qgnQ01" nil silent #1# excl) > | | 3 -> (tramp-file-name-handler expand-file-name > "/ssh:detlef:/tmp/tramp.qgnQ01" nil) > | | 3 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01" > | | 3 -> (tramp-file-name-handler file-truename > "/ssh:detlef:/tmp/tramp.qgnQ01") > | | | 4 -> (tramp-file-name-handler file-remote-p > "/ssh:detlef:/tmp/tramp.qgnQ01" localname nil) > | | | | 5 -> (tramp-file-name-handler expand-file-name > "/ssh:detlef:/tmp/tramp.qgnQ01" nil) > | | | | 5 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01" > | | | 4 <- tramp-file-name-handler: "/tmp/tramp.qgnQ01" > | | | 4 -> (tramp-file-name-handler expand-file-name > "/ssh:detlef:/tmp/tramp.qgnQ01" nil) > | | | 4 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01" > | | | 4 -> (tramp-file-name-handler file-remote-p > "/ssh:detlef:/tmp/tramp.qgnQ01" localname nil) > | | | | 5 -> (tramp-file-name-handler expand-file-name > "/ssh:detlef:/tmp/tramp.qgnQ01" nil) > | | | | 5 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01" > | | | 4 <- tramp-file-name-handler: "/tmp/tramp.qgnQ01" > | | | 4 -> (tramp-file-name-handler file-remote-p > "/ssh:detlef:/tmp/tramp.qgnQ01" nil nil) > | | | | 5 -> (tramp-file-name-handler expand-file-name > "/ssh:detlef:/tmp/tramp.qgnQ01" nil) > | | | | 5 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01" > | | | 4 <- tramp-file-name-handler: "/ssh:detlef:" > | | | 4 -> (tramp-send-command (tramp-file-name "ssh" nil nil "detlef" nil > "/tmp/tramp.qgnQ01" nil) "\\readlink --canonicalize-missing > /tmp/tramp.qgnQ01 2>/dev/null; echo tramp_exit_status $?") > | | | | 5 -> (tramp-send-command (tramp-file-name "ssh" nil nil "detlef" > nil "/tmp/tramp.qgnQ01" nil) "echo are you awake" t t) > | | | | 5 <- tramp-send-command: nil > | | | 4 <- tramp-send-command: 39 > | | | 4 -> (tramp-file-name-handler expand-file-name > "/ssh:detlef:/tmp/tramp.qgnQ01" nil) > | | | 4 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01" > | | | 4 -> (tramp-file-name-handler file-symlink-p > "/ssh:detlef:/tmp/tramp.qgnQ01") > | | | | 5 -> (tramp-file-name-handler expand-file-name > "/ssh:detlef:/tmp/tramp.qgnQ01" nil) > | | | | 5 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01" > | | | | 5 -> (tramp-file-name-handler file-attributes > "/ssh:detlef:/tmp/tramp.qgnQ01") > | | | | | 6 -> (tramp-file-name-handler expand-file-name > "/ssh:detlef:/tmp/tramp.qgnQ01" nil) > | | | | | 6 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01" > | | | | | 6 -> (tramp-send-command (tramp-file-name "ssh" nil nil "detlef" > nil "/tmp/tramp.qgnQ01" nil) "tramp_stat_file_attributes /tmp/tramp.qgnQ01 > 2>/dev/null; echo tramp_exit_status $?") > | | | | | 6 <- tramp-send-command: 25 > | | | | 5 <- tramp-file-name-handler: nil > | | | 4 <- tramp-file-name-handler: nil > | | 3 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01" > | | 3 -> (tramp-file-name-handler expand-file-name > "/ssh:detlef:/tmp/tramp.qgnQ01" nil) > | | 3 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01" > | | 3 -> (tramp-file-name-handler file-exists-p > "/ssh:detlef:/tmp/tramp.qgnQ01") > | | 3 <- tramp-file-name-handler: nil > | | 3 -> (tramp-file-name-handler file-locked-p > "/ssh:detlef:/tmp/tramp.qgnQ01") > | | | 4 -> (tramp-file-name-handler make-lock-file-name > "/ssh:detlef:/tmp/tramp.qgnQ01") > | | | 4 <- tramp-file-name-handler: nil > | | 3 <- tramp-file-name-handler: nil > | | 3 -> (tramp-file-name-handler expand-file-name > "/ssh:detlef:/tmp/tramp.qgnQ01" nil) > | | 3 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01" > | | 3 -> (tramp-file-name-handler file-attributes > "/ssh:detlef:/tmp/tramp.qgnQ01" integer) > | | | 4 -> (tramp-file-name-handler expand-file-name > "/ssh:detlef:/tmp/tramp.qgnQ01" nil) > | | | 4 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01" > | | 3 <- tramp-file-name-handler: nil > | | 3 -> (tramp-file-name-handler expand-file-name > "/ssh:detlef:/tmp/tramp.qgnQ01" nil) > | | 3 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01" > | | 3 -> (tramp-file-name-handler file-attributes > "/ssh:detlef:/tmp/tramp.qgnQ01" integer) > | | | 4 -> (tramp-file-name-handler expand-file-name > "/ssh:detlef:/tmp/tramp.qgnQ01" nil) > | | | 4 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01" > | | 3 <- tramp-file-name-handler: nil > | | 3 -> (tramp-file-name-handler expand-file-name > "/ssh:detlef:/tmp/tramp.qgnQ01" nil) > | | 3 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01" > | | 3 -> (tramp-file-name-handler file-acl "/ssh:detlef:/tmp/tramp.qgnQ01") > | | 3 <- tramp-file-name-handler: nil > | | 3 -> (tramp-file-name-handler expand-file-name > "/ssh:detlef:/tmp/tramp.qgnQ01" nil) > | | 3 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01" > | | 3 -> (tramp-file-name-handler file-selinux-context > "/ssh:detlef:/tmp/tramp.qgnQ01") > | | 3 <- tramp-file-name-handler: (nil nil nil nil) > | | 3 -> (tramp-file-name-handler file-name-nondirectory > "/ssh:detlef:/tmp/tramp.qgnQ01") > | | 3 <- tramp-file-name-handler: "tramp.qgnQ01" > | | 3 -> (tramp-file-name-handler file-remote-p > "/ssh:detlef:/tmp/tramp.qgnQ01" nil nil) > | | | 4 -> (tramp-file-name-handler expand-file-name > "/ssh:detlef:/tmp/tramp.qgnQ01" nil) > | | | 4 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01" > | | 3 <- tramp-file-name-handler: "/ssh:detlef:" > | | 3 -> (tramp-file-name-handler lock-file > "/ssh:detlef:/tmp/tramp.qgnQ01") > | | | 4 -> (tramp-file-name-handler file-locked-p > "/ssh:detlef:/tmp/tramp.qgnQ01") > | | | | 5 -> (tramp-file-name-handler make-lock-file-name > "/ssh:detlef:/tmp/tramp.qgnQ01") > | | | | 5 <- tramp-file-name-handler: nil > | | | 4 <- tramp-file-name-handler: nil > | | | 4 -> (tramp-file-name-handler make-lock-file-name > "/ssh:detlef:/tmp/tramp.qgnQ01") > | | | 4 <- tramp-file-name-handler: nil > | | | 4 -> (tramp-file-name-handler make-lock-file-name > "/ssh:detlef:/tmp/tramp.qgnQ01") > | | | 4 <- tramp-file-name-handler: nil > | | 3 <- tramp-file-name-handler: nil > | | 3 -> (tramp-file-name-handler expand-file-name > "/ssh:detlef:/tmp/tramp.qgnQ01" nil) > | | 3 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01" > | | 3 -> (tramp-file-name-handler file-modes > "/ssh:detlef:/tmp/tramp.qgnQ01" nofollow) > | | | 4 -> (tramp-file-name-handler expand-file-name > "/ssh:detlef:/tmp/tramp.qgnQ01" nil) > | | | 4 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01" > | | | 4 -> (tramp-file-name-handler file-attributes > "/ssh:detlef:/tmp/tramp.qgnQ01") > | | | | 5 -> (tramp-file-name-handler expand-file-name > "/ssh:detlef:/tmp/tramp.qgnQ01" nil) > | | | | 5 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01" > | | | 4 <- tramp-file-name-handler: nil > | | 3 <- tramp-file-name-handler: nil > | | 3 -> (tramp-file-name-handler file-name-nondirectory > "/ssh:detlef:/tmp/tramp.qgnQ01") > | | 3 <- tramp-file-name-handler: "tramp.qgnQ01" > | | 3 -> (tramp-send-command (tramp-file-name "ssh" nil nil "detlef" nil > "/tmp/tramp.qgnQ01" nil) "base64 -d -i >/tmp/tramp.qgnQ01 > <<'b03f0306a9a1e716a81252ccc4134b92' 2>/dev/null; echo tramp_exit_status > $?\nb03f0306a9a1e716a81252ccc4134b92") > | | 3 <- tramp-send-command: 21 > | | 3 -> (tramp-file-name-handler file-name-directory > "/ssh:detlef:/tmp/tramp.qgnQ01") > | | 3 <- tramp-file-name-handler: "/ssh:detlef:/tmp/" > | | 3 -> (tramp-file-name-handler expand-file-name "/ssh:detlef:/tmp/" nil) > | | 3 <- tramp-file-name-handler: "/ssh:detlef:/tmp/" > | | 3 -> (tramp-file-name-handler directory-file-name "/ssh:detlef:/tmp/") > | | 3 <- tramp-file-name-handler: "/ssh:detlef:/tmp" > | | 3 -> (tramp-file-name-handler file-modes "/ssh:detlef:/tmp" nil) > | | | 4 -> (tramp-file-name-handler expand-file-name "/ssh:detlef:/tmp" > nil) > | | | 4 <- tramp-file-name-handler: "/ssh:detlef:/tmp" > | | | 4 -> (tramp-file-name-handler file-attributes "/ssh:detlef:/tmp") > | | | | 5 -> (tramp-file-name-handler expand-file-name "/ssh:detlef:/tmp" > nil) > | | | | 5 <- tramp-file-name-handler: "/ssh:detlef:/tmp" > | | | | 5 -> (tramp-send-command (tramp-file-name "ssh" nil nil "detlef" > nil "/tmp" nil) "tramp_stat_file_attributes /tmp 2>/dev/null; echo > tramp_exit_status $?") > | | | | 5 <- tramp-send-command: 129 > | | | 4 <- tramp-file-name-handler: (t 30 0 0 (25305 58381 0 0) (25312 > 7412 0 0) (25312 7412 0 0) 135168 "drwxrwxrwt" t 3801089 (-1 . 2)) > | | 3 <- tramp-file-name-handler: 1023 > | | 3 -> (tramp-file-name-handler tramp-set-file-uid-gid > "/ssh:detlef:/tmp/tramp.qgnQ01" 1000 1000) > | | | 4 -> (tramp-send-command (tramp-file-name "ssh" nil nil "detlef" nil > "/tmp/tramp.qgnQ01" nil) "chown 1000:1000 /tmp/tramp.qgnQ01") > | | | 4 <- tramp-send-command: 1 > | | 3 <- tramp-file-name-handler: 1 > | | 3 -> (tramp-file-name-handler expand-file-name > "/ssh:detlef:/tmp/tramp.qgnQ01" "~/") > | | 3 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01" > | | 3 -> (tramp-file-name-handler set-file-acl > "/ssh:detlef:/tmp/tramp.qgnQ01" nil) > | | | 4 -> (tramp-file-name-handler expand-file-name > "/ssh:detlef:/tmp/tramp.qgnQ01" nil) > | | | 4 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01" > | | 3 <- tramp-file-name-handler: nil > | | 3 -> (tramp-file-name-handler expand-file-name > "/ssh:detlef:/tmp/tramp.qgnQ01" "~/") > | | 3 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01" > | | 3 -> (tramp-file-name-handler set-file-selinux-context > "/ssh:detlef:/tmp/tramp.qgnQ01" (nil nil nil nil)) > | | 3 <- tramp-file-name-handler: nil > | | 3 -> (tramp-file-name-handler unlock-file > "/ssh:detlef:/tmp/tramp.qgnQ01") > | | | 4 -> (tramp-file-name-handler make-lock-file-name > "/ssh:detlef:/tmp/tramp.qgnQ01") > | | | 4 <- tramp-file-name-handler: nil > | | 3 <- tramp-file-name-handler: nil > | 2 <- tramp-file-name-handler: nil > | 2 -> (tramp-send-command (tramp-file-name "ssh" nil nil "detlef" nil > "/home/albinus/.emacs" nil) "dd bs=1 skip=2 if=/home/albinus/.emacs count=2 > of=/tmp/tramp.qgnQ01") > | 2 <- tramp-send-command: 72 > | 2 -> (tramp-file-name-handler file-local-copy > "/ssh:detlef:/tmp/tramp.qgnQ01") > | | 3 -> (tramp-file-name-handler file-truename > "/ssh:detlef:/tmp/tramp.qgnQ01") > | | | 4 -> (tramp-file-name-handler file-remote-p > "/ssh:detlef:/tmp/tramp.qgnQ01" localname nil) > | | | | 5 -> (tramp-file-name-handler expand-file-name > "/ssh:detlef:/tmp/tramp.qgnQ01" nil) > | | | | 5 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01" > | | | 4 <- tramp-file-name-handler: "/tmp/tramp.qgnQ01" > | | | 4 -> (tramp-file-name-handler expand-file-name > "/ssh:detlef:/tmp/tramp.qgnQ01" nil) > | | | 4 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01" > | | | 4 -> (tramp-file-name-handler file-remote-p > "/ssh:detlef:/tmp/tramp.qgnQ01" localname nil) > | | | | 5 -> (tramp-file-name-handler expand-file-name > "/ssh:detlef:/tmp/tramp.qgnQ01" nil) > | | | | 5 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01" > | | | 4 <- tramp-file-name-handler: "/tmp/tramp.qgnQ01" > | | | 4 -> (tramp-file-name-handler file-remote-p > "/ssh:detlef:/tmp/tramp.qgnQ01" nil nil) > | | | | 5 -> (tramp-file-name-handler expand-file-name > "/ssh:detlef:/tmp/tramp.qgnQ01" nil) > | | | | 5 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01" > | | | 4 <- tramp-file-name-handler: "/ssh:detlef:" > | | | 4 -> (tramp-send-command (tramp-file-name "ssh" nil nil "detlef" nil > "/tmp/tramp.qgnQ01" nil) "\\readlink --canonicalize-missing > /tmp/tramp.qgnQ01 2>/dev/null; echo tramp_exit_status $?") > | | | 4 <- tramp-send-command: 39 > | | | 4 -> (tramp-file-name-handler expand-file-name > "/ssh:detlef:/tmp/tramp.qgnQ01" nil) > | | | 4 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01" > | | | 4 -> (tramp-file-name-handler file-symlink-p > "/ssh:detlef:/tmp/tramp.qgnQ01") > | | | | 5 -> (tramp-file-name-handler expand-file-name > "/ssh:detlef:/tmp/tramp.qgnQ01" nil) > | | | | 5 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01" > | | | | 5 -> (tramp-file-name-handler file-attributes > "/ssh:detlef:/tmp/tramp.qgnQ01") > | | | | | 6 -> (tramp-file-name-handler expand-file-name > "/ssh:detlef:/tmp/tramp.qgnQ01" nil) > | | | | | 6 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01" > | | | | | 6 -> (tramp-send-command (tramp-file-name "ssh" nil nil "detlef" > nil "/tmp/tramp.qgnQ01" nil) "tramp_stat_file_attributes /tmp/tramp.qgnQ01 > 2>/dev/null; echo tramp_exit_status $?") > | | | | | 6 <- tramp-send-command: 148 > | | | | 5 <- tramp-file-name-handler: (nil 1 1000 1000 (25312 7412 0 0) > (25312 7412 0 0) (25312 7412 0 0) 2 "-rw-rw-r--" nil 3821258 (-1 . 2)) > | | | 4 <- tramp-file-name-handler: nil > | | 3 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01" > | | 3 -> (tramp-file-name-handler file-name-nondirectory > "/ssh:detlef:/tmp/tramp.qgnQ01") > | | 3 <- tramp-file-name-handler: "tramp.qgnQ01" > | | 3 -> (tramp-file-name-handler expand-file-name > "/ssh:detlef:/tmp/tramp.qgnQ01" nil) > | | 3 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01" > | | 3 -> (tramp-file-name-handler file-attributes > "/ssh:detlef:/tmp/tramp.qgnQ01") > | | | 4 -> (tramp-file-name-handler expand-file-name > "/ssh:detlef:/tmp/tramp.qgnQ01" nil) > | | | 4 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01" > | | 3 <- tramp-file-name-handler: (nil 1 1000 1000 (25312 7412 0 0) (25312 > 7412 0 0) (25312 7412 0 0) 2 "-rw-rw-r--" nil 3821258 (-1 . 2)) > | | 3 -> (tramp-send-command (tramp-file-name "ssh" nil nil "detlef" nil > "/tmp/tramp.qgnQ01" nil) "base64 /dev/null; echo > tramp_exit_status $?") > | | 3 <- tramp-send-command: 26 > | | 3 -> (tramp-file-name-handler expand-file-name > "/ssh:detlef:/tmp/tramp.qgnQ01" nil) > | | 3 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01" > | | 3 -> (tramp-file-name-handler file-modes > "/ssh:detlef:/tmp/tramp.qgnQ01" nil) > | | | 4 -> (tramp-file-name-handler expand-file-name > "/ssh:detlef:/tmp/tramp.qgnQ01" nil) > | | | 4 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01" > | | | 4 -> (tramp-file-name-handler file-attributes > "/ssh:detlef:/tmp/tramp.qgnQ01") > | | | | 5 -> (tramp-file-name-handler expand-file-name > "/ssh:detlef:/tmp/tramp.qgnQ01" nil) > | | | | 5 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01" > | | | 4 <- tramp-file-name-handler: (nil 1 1000 1000 (25312 7412 0 0) > (25312 7412 0 0) (25312 7412 0 0) 2 "-rw-rw-r--" nil 3821258 (-1 . 2)) > | | 3 <- tramp-file-name-handler: 436 > | 2 <- tramp-file-name-handler: "/tmp/tramp.6ce5x9.qgnQ01" > | 2 -> (tramp-file-name-handler expand-file-name > "/ssh:detlef:/tmp/tramp.qgnQ01" nil) > | 2 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01" > | 2 -> (tramp-file-name-handler file-directory-p > "/ssh:detlef:/tmp/tramp.qgnQ01") > | 2 <- tramp-file-name-handler: nil > | 2 -> (tramp-file-name-handler expand-file-name > "/ssh:detlef:/tmp/tramp.qgnQ01" nil) > | 2 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01" > | 2 -> (tramp-file-name-handler delete-file > "/ssh:detlef:/tmp/tramp.qgnQ01" nil) > | | 3 -> (tramp-file-name-handler expand-file-name > "/ssh:detlef:/tmp/tramp.qgnQ01" nil) > | | 3 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01" > | | 3 -> (tramp-send-command (tramp-file-name "ssh" nil nil "detlef" nil > "/tmp/tramp.qgnQ01" nil) "rm -f /tmp/tramp.qgnQ01 2>/dev/null; echo > tramp_exit_status $?") > | | 3 <- tramp-send-command: 21 > | 2 <- tramp-file-name-handler: t > 1 <- tramp-file-name-handler: ("/ssh:detlef:/home/albinus/.emacs" 2) > --00000000000084b39605e533bc46 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
> As said, I live in a pretty good connected LAN. remot= e commands do not
> result in heavy performance degradations, compare= d with the rest of
> Elisp computing.

Understood, but I propos= e to adopt a different benchmark: number of
remote commands.=C2=A0 As so= on as you are not in a LAN, even not when using
a particularly slow netw= ork, this becomes an order or two of magnitude
more important than every= thing else.=C2=A0 E.g. with slightly slower
commands or particularly ine= fficient Elisp you can make it 2-10 ms
slower for everyone.=C2=A0 But wi= th unoptimized command flow (i.e. more
remote commands) it can be 100-50= 0 ms slower, though not for everyone,
but people using this over non-LAN= .=C2=A0 I think in such cases extremes
are more important than the avera= ge.

> > On the other hand, refreshing a remote= -tracking Magit (Emacs Git
> > interface) buffer doesn't feel = any faster than before. Logs also
> > suggest that number of comma= nds that tramp send stays as before (43).
> > Some commands are ce= rtainly excessive, e.g. it repeats `test -d [...]
> > 2>/dev/nu= ll; echo tramp_exit_status $?' 4 times for the same directory
> &= gt; (intermixed with other commands), suggesting that Tramp doesn't cac= he
> > the result. Three times it issues `test -r ...' for the= same
> > directory, the source checkout root. Most other commands= cannot be
> > generically skipped by Tramp though, this would rat= her depend on
> > Magit.
>
> Caches are problematic. = They are limited to 10 seconds by default, did
> you try to increase = that number?

Yes, `remote-file-name-inhibit-cache' is now 600 he= re.

> And directories are also problematic for caches. As soon as= something
> changes there (creation or deletion of a file, for examp= le), the cached
> values of the directory must be flushed.

Yea= h, I suppose Tramp has no generic way to know that Magit issues
reading = commands.=C2=A0 Can we devise a generic interface that would tell
Tramp:= "Commands within this block will not modify file system",
e.g= . with let-binding some variable?

In general, it feels like Tramp fl= ushes its caches too often or maybe
doesn't even cache certain thing= s at all.=C2=A0 I.e. it's not about those
10 seconds (following your= advice I have even increased that).=C2=A0 It's
that while serving o= ne user-level command here (i.e. within 3 seconds
at most), it can issue= the same remote command several times, thus
not reusing previous result= s.

E.g. in the traces you have attached this can be seen.=C2=A0 The = following
two commands repeat twice:

| | | 4 -> (tramp-send-co= mmand (tramp-file-name "ssh" nil nil "detlef" nil "= ;/tmp/tramp.qgnQ01" nil) "\\readlink --canonicalize-missing /tmp/= tramp.qgnQ01 2>/dev/null; echo tramp_exit_status $?")
...
| |= | 4 -> (tramp-send-command (tramp-file-name "ssh" nil nil &qu= ot;detlef" nil "/tmp/tramp.qgnQ01" nil) "\\readlink --c= anonicalize-missing /tmp/tramp.qgnQ01 2>/dev/null; echo tramp_exit_statu= s $?")

and

| | | | | 6 -> (tramp-send-command (tramp-= file-name "ssh" nil nil "detlef" nil "/tmp/tramp.q= gnQ01" nil) "tramp_stat_file_attributes /tmp/tramp.qgnQ01 2>/d= ev/null; echo tramp_exit_status $?")
...
| | | | | 6 -> (tram= p-send-command (tramp-file-name "ssh" nil nil "detlef" = nil "/tmp/tramp.qgnQ01" nil) "tramp_stat_file_attributes /tm= p/tramp.qgnQ01 2>/dev/null; echo tramp_exit_status $?")

Paul=

On Tue, 26 Jul 2022 at 19:51, Michael Albinus <michael.albinus@gmx.de> wrote:
<= blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-l= eft:1px solid rgb(204,204,204);padding-left:1ex">Paul Pogonyshev <pogonyshev@gmail.com> writes:

Hi Paul,

>> Combining stat and readlink into one remote command have even slow= ed
>> down the machinery. Likely, because it isn't needed always, bu= t it
>> always consumes more time to compute both on the remote side, and = to
>> read all the results.
>
> Well, that's hard to believe. I mean, it is understandable that > `readlink' (whatever Elisp call results in that shell command) is = not
> often needed, but how long can that take on a modern computer, 1 ms? > It might be that we evaluate performance changes differently. I care > more about worst cases (since they affect me directly, yeah) and not > so much if Tramp becomes 1 ms slower "per user-visible operation&= quot; on
> average.

As said, I live in a pretty good connected LAN. remote commands do not
result in heavy performance degradations, compared with the rest of
Elisp computing.

> I tested it a bit now. Initially mentioned usecase
> (`insert-file-contents', but only a file part, not the whole file)=
> appears to be considerably faster after the changes (hard to say,
> feels like 30-50% faster).

Oh, good. In my environment it was something about 5%.

> The connection log says that number of
> commands is 10 now, with the originally mentioned list heavily
> changed. However, there must still be space for improvement, e.g. `M-x=
> occur tramp-send RET' suggests there are outright duplicated comma= nds
> now:

Pretty sure there's still place for improvements!

> On the other hand, refreshing a remote-tracking Magit (Emacs Git
> interface) buffer doesn't feel any faster than before. Logs also > suggest that number of commands that tramp send stays as before (43).<= br> > Some commands are certainly excessive, e.g. it repeats `test -d [...]<= br> > 2>/dev/null; echo tramp_exit_status $?' 4 times for the same di= rectory
> (intermixed with other commands), suggesting that Tramp doesn't ca= che
> the result. Three times it issues `test -r ...' for the same
> directory, the source checkout root. Most other commands cannot be
> generically skipped by Tramp though, this would rather depend on
> Magit.

Caches are problematic. They are limited to 10 seconds by default, did
you try to increase that number? VC related commands are also
problematic, Emacs tests different VC's for availability. It is always = a
good idea to reduce the number of VC backends. If you completely trust
in magit, you shall deactivate Emacs' vc.el. Something like (setq
vc-handled-backends nil) should help.

And directories are also problematic for caches. As soon as something
changes there (creation or deletion of a file, for example), the cached
values of the directory must be flushed.

I propose we try step by step. That is, we investigate typical code
snippets, and see what Tramp does. Case by case.

The Tramp debug buffer is a little bit noisy. So you could do

(progn
=C2=A0 (trace-function-background 'tramp-send-command)
=C2=A0 (trace-function-background 'tramp-file-name-handler))

By this, only the calls of tramp-send-command (go remote) and
tramp-file-name-handler (any invocation of Tramp) are traced. The latter is important to know, what has triggered a given remote command.

Starting with your initial example

(with-temp-buffer
=C2=A0 (insert-file-contents "/ssh:detlef:.emacs" nil 100 nil))
I get the *trace-output* buffer as appended. Let's study this first, an= d
see, what could be done in order to improve this case. Obviously, most
of the commands are about the temporary file /tmp/tramp.qgnQ01, which is superfluous. I'll check whether handling temporary files could be
handled better.

When we're happy with this example, let's go to the next example.
> Paul

Best regards, Michael.

PS: Next 2 or 3 days I'm kind of absent: my granddaughter will visit us. Sorry, but you know: priorities :-)

=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
1 -> (tramp-file-name-handler expand-file-name "/ssh:detlef:.emacs&= quot; nil)
1 <- tramp-file-name-handler: "/ssh:detlef:/home/albinus/.emacs&quo= t;
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
1 -> (tramp-file-name-handler insert-file-contents "/ssh:detlef:/ho= me/albinus/.emacs" nil 2 4 nil)
| 2 -> (tramp-file-name-handler expand-file-name "/ssh:detlef:/home= /albinus/.emacs" nil)
| 2 <- tramp-file-name-handler: "/ssh:detlef:/home/albinus/.emacs&q= uot;
| 2 -> (tramp-file-name-handler expand-file-name "tramp." &quo= t;/ssh:detlef:/tmp")
| 2 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp."
| 2 -> (tramp-file-name-handler expand-file-name "/ssh:detlef:/tmp/= tramp." "/tmp/")
| 2 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp."
| 2 -> (tramp-file-name-handler expand-file-name "/ssh:detlef:/tmp/= tramp.qgnQ01" nil)
| 2 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01"= ;
| 2 -> (tramp-file-name-handler write-region "" nil #1=3D"= ;/ssh:detlef:/tmp/tramp.qgnQ01" nil silent #1# excl)
| | 3 -> (tramp-file-name-handler expand-file-name "/ssh:detlef:/tm= p/tramp.qgnQ01" nil)
| | 3 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01&qu= ot;
| | 3 -> (tramp-file-name-handler file-truename "/ssh:detlef:/tmp/t= ramp.qgnQ01")
| | | 4 -> (tramp-file-name-handler file-remote-p "/ssh:detlef:/tmp= /tramp.qgnQ01" localname nil)
| | | | 5 -> (tramp-file-name-handler expand-file-name "/ssh:detlef= :/tmp/tramp.qgnQ01" nil)
| | | | 5 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ0= 1"
| | | 4 <- tramp-file-name-handler: "/tmp/tramp.qgnQ01"
| | | 4 -> (tramp-file-name-handler expand-file-name "/ssh:detlef:/= tmp/tramp.qgnQ01" nil)
| | | 4 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01&= quot;
| | | 4 -> (tramp-file-name-handler file-remote-p "/ssh:detlef:/tmp= /tramp.qgnQ01" localname nil)
| | | | 5 -> (tramp-file-name-handler expand-file-name "/ssh:detlef= :/tmp/tramp.qgnQ01" nil)
| | | | 5 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ0= 1"
| | | 4 <- tramp-file-name-handler: "/tmp/tramp.qgnQ01"
| | | 4 -> (tramp-file-name-handler file-remote-p "/ssh:detlef:/tmp= /tramp.qgnQ01" nil nil)
| | | | 5 -> (tramp-file-name-handler expand-file-name "/ssh:detlef= :/tmp/tramp.qgnQ01" nil)
| | | | 5 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ0= 1"
| | | 4 <- tramp-file-name-handler: "/ssh:detlef:"
| | | 4 -> (tramp-send-command (tramp-file-name "ssh" nil nil = "detlef" nil "/tmp/tramp.qgnQ01" nil) "\\readlink = --canonicalize-missing /tmp/tramp.qgnQ01 2>/dev/null; echo tramp_exit_st= atus $?")
| | | | 5 -> (tramp-send-command (tramp-file-name "ssh" nil ni= l "detlef" nil "/tmp/tramp.qgnQ01" nil) "echo are = you awake" t t)
| | | | 5 <- tramp-send-command: nil
| | | 4 <- tramp-send-command: 39
| | | 4 -> (tramp-file-name-handler expand-file-name "/ssh:detlef:/= tmp/tramp.qgnQ01" nil)
| | | 4 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01&= quot;
| | | 4 -> (tramp-file-name-handler file-symlink-p "/ssh:detlef:/tm= p/tramp.qgnQ01")
| | | | 5 -> (tramp-file-name-handler expand-file-name "/ssh:detlef= :/tmp/tramp.qgnQ01" nil)
| | | | 5 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ0= 1"
| | | | 5 -> (tramp-file-name-handler file-attributes "/ssh:detlef:= /tmp/tramp.qgnQ01")
| | | | | 6 -> (tramp-file-name-handler expand-file-name "/ssh:detl= ef:/tmp/tramp.qgnQ01" nil)
| | | | | 6 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgn= Q01"
| | | | | 6 -> (tramp-send-command (tramp-file-name "ssh" nil = nil "detlef" nil "/tmp/tramp.qgnQ01" nil) "tramp_s= tat_file_attributes /tmp/tramp.qgnQ01 2>/dev/null; echo tramp_exit_statu= s $?")
| | | | | 6 <- tramp-send-command: 25
| | | | 5 <- tramp-file-name-handler: nil
| | | 4 <- tramp-file-name-handler: nil
| | 3 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01&qu= ot;
| | 3 -> (tramp-file-name-handler expand-file-name "/ssh:detlef:/tm= p/tramp.qgnQ01" nil)
| | 3 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01&qu= ot;
| | 3 -> (tramp-file-name-handler file-exists-p "/ssh:detlef:/tmp/t= ramp.qgnQ01")
| | 3 <- tramp-file-name-handler: nil
| | 3 -> (tramp-file-name-handler file-locked-p "/ssh:detlef:/tmp/t= ramp.qgnQ01")
| | | 4 -> (tramp-file-name-handler make-lock-file-name "/ssh:detle= f:/tmp/tramp.qgnQ01")
| | | 4 <- tramp-file-name-handler: nil
| | 3 <- tramp-file-name-handler: nil
| | 3 -> (tramp-file-name-handler expand-file-name "/ssh:detlef:/tm= p/tramp.qgnQ01" nil)
| | 3 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01&qu= ot;
| | 3 -> (tramp-file-name-handler file-attributes "/ssh:detlef:/tmp= /tramp.qgnQ01" integer)
| | | 4 -> (tramp-file-name-handler expand-file-name "/ssh:detlef:/= tmp/tramp.qgnQ01" nil)
| | | 4 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01&= quot;
| | 3 <- tramp-file-name-handler: nil
| | 3 -> (tramp-file-name-handler expand-file-name "/ssh:detlef:/tm= p/tramp.qgnQ01" nil)
| | 3 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01&qu= ot;
| | 3 -> (tramp-file-name-handler file-attributes "/ssh:detlef:/tmp= /tramp.qgnQ01" integer)
| | | 4 -> (tramp-file-name-handler expand-file-name "/ssh:detlef:/= tmp/tramp.qgnQ01" nil)
| | | 4 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01&= quot;
| | 3 <- tramp-file-name-handler: nil
| | 3 -> (tramp-file-name-handler expand-file-name "/ssh:detlef:/tm= p/tramp.qgnQ01" nil)
| | 3 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01&qu= ot;
| | 3 -> (tramp-file-name-handler file-acl "/ssh:detlef:/tmp/tramp.= qgnQ01")
| | 3 <- tramp-file-name-handler: nil
| | 3 -> (tramp-file-name-handler expand-file-name "/ssh:detlef:/tm= p/tramp.qgnQ01" nil)
| | 3 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01&qu= ot;
| | 3 -> (tramp-file-name-handler file-selinux-context "/ssh:detlef= :/tmp/tramp.qgnQ01")
| | 3 <- tramp-file-name-handler: (nil nil nil nil)
| | 3 -> (tramp-file-name-handler file-name-nondirectory "/ssh:detl= ef:/tmp/tramp.qgnQ01")
| | 3 <- tramp-file-name-handler: "tramp.qgnQ01"
| | 3 -> (tramp-file-name-handler file-remote-p "/ssh:detlef:/tmp/t= ramp.qgnQ01" nil nil)
| | | 4 -> (tramp-file-name-handler expand-file-name "/ssh:detlef:/= tmp/tramp.qgnQ01" nil)
| | | 4 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01&= quot;
| | 3 <- tramp-file-name-handler: "/ssh:detlef:"
| | 3 -> (tramp-file-name-handler lock-file "/ssh:detlef:/tmp/tramp= .qgnQ01")
| | | 4 -> (tramp-file-name-handler file-locked-p "/ssh:detlef:/tmp= /tramp.qgnQ01")
| | | | 5 -> (tramp-file-name-handler make-lock-file-name "/ssh:det= lef:/tmp/tramp.qgnQ01")
| | | | 5 <- tramp-file-name-handler: nil
| | | 4 <- tramp-file-name-handler: nil
| | | 4 -> (tramp-file-name-handler make-lock-file-name "/ssh:detle= f:/tmp/tramp.qgnQ01")
| | | 4 <- tramp-file-name-handler: nil
| | | 4 -> (tramp-file-name-handler make-lock-file-name "/ssh:detle= f:/tmp/tramp.qgnQ01")
| | | 4 <- tramp-file-name-handler: nil
| | 3 <- tramp-file-name-handler: nil
| | 3 -> (tramp-file-name-handler expand-file-name "/ssh:detlef:/tm= p/tramp.qgnQ01" nil)
| | 3 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01&qu= ot;
| | 3 -> (tramp-file-name-handler file-modes "/ssh:detlef:/tmp/tram= p.qgnQ01" nofollow)
| | | 4 -> (tramp-file-name-handler expand-file-name "/ssh:detlef:/= tmp/tramp.qgnQ01" nil)
| | | 4 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01&= quot;
| | | 4 -> (tramp-file-name-handler file-attributes "/ssh:detlef:/t= mp/tramp.qgnQ01")
| | | | 5 -> (tramp-file-name-handler expand-file-name "/ssh:detlef= :/tmp/tramp.qgnQ01" nil)
| | | | 5 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ0= 1"
| | | 4 <- tramp-file-name-handler: nil
| | 3 <- tramp-file-name-handler: nil
| | 3 -> (tramp-file-name-handler file-name-nondirectory "/ssh:detl= ef:/tmp/tramp.qgnQ01")
| | 3 <- tramp-file-name-handler: "tramp.qgnQ01"
| | 3 -> (tramp-send-command (tramp-file-name "ssh" nil nil &q= uot;detlef" nil "/tmp/tramp.qgnQ01" nil) "base64 -d -i = >/tmp/tramp.qgnQ01 <<'b03f0306a9a1e716a81252ccc4134b92' 2&= gt;/dev/null; echo tramp_exit_status $?\nb03f0306a9a1e716a81252ccc4134b92&q= uot;)
| | 3 <- tramp-send-command: 21
| | 3 -> (tramp-file-name-handler file-name-directory "/ssh:detlef:= /tmp/tramp.qgnQ01")
| | 3 <- tramp-file-name-handler: "/ssh:detlef:/tmp/"
| | 3 -> (tramp-file-name-handler expand-file-name "/ssh:detlef:/tm= p/" nil)
| | 3 <- tramp-file-name-handler: "/ssh:detlef:/tmp/"
| | 3 -> (tramp-file-name-handler directory-file-name "/ssh:detlef:= /tmp/")
| | 3 <- tramp-file-name-handler: "/ssh:detlef:/tmp"
| | 3 -> (tramp-file-name-handler file-modes "/ssh:detlef:/tmp"= ; nil)
| | | 4 -> (tramp-file-name-handler expand-file-name "/ssh:detlef:/= tmp" nil)
| | | 4 <- tramp-file-name-handler: "/ssh:detlef:/tmp"
| | | 4 -> (tramp-file-name-handler file-attributes "/ssh:detlef:/t= mp")
| | | | 5 -> (tramp-file-name-handler expand-file-name "/ssh:detlef= :/tmp" nil)
| | | | 5 <- tramp-file-name-handler: "/ssh:detlef:/tmp"
| | | | 5 -> (tramp-send-command (tramp-file-name "ssh" nil ni= l "detlef" nil "/tmp" nil) "tramp_stat_file_attrib= utes /tmp 2>/dev/null; echo tramp_exit_status $?")
| | | | 5 <- tramp-send-command: 129
| | | 4 <- tramp-file-name-handler: (t 30 0 0 (25305 58381 0 0) (25312 7= 412 0 0) (25312 7412 0 0) 135168 "drwxrwxrwt" t 3801089 (-1 . 2))=
| | 3 <- tramp-file-name-handler: 1023
| | 3 -> (tramp-file-name-handler tramp-set-file-uid-gid "/ssh:detl= ef:/tmp/tramp.qgnQ01" 1000 1000)
| | | 4 -> (tramp-send-command (tramp-file-name "ssh" nil nil = "detlef" nil "/tmp/tramp.qgnQ01" nil) "chown 1000:= 1000 /tmp/tramp.qgnQ01")
| | | 4 <- tramp-send-command: 1
| | 3 <- tramp-file-name-handler: 1
| | 3 -> (tramp-file-name-handler expand-file-name "/ssh:detlef:/tm= p/tramp.qgnQ01" "~/")
| | 3 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01&qu= ot;
| | 3 -> (tramp-file-name-handler set-file-acl "/ssh:detlef:/tmp/tr= amp.qgnQ01" nil)
| | | 4 -> (tramp-file-name-handler expand-file-name "/ssh:detlef:/= tmp/tramp.qgnQ01" nil)
| | | 4 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01&= quot;
| | 3 <- tramp-file-name-handler: nil
| | 3 -> (tramp-file-name-handler expand-file-name "/ssh:detlef:/tm= p/tramp.qgnQ01" "~/")
| | 3 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01&qu= ot;
| | 3 -> (tramp-file-name-handler set-file-selinux-context "/ssh:de= tlef:/tmp/tramp.qgnQ01" (nil nil nil nil))
| | 3 <- tramp-file-name-handler: nil
| | 3 -> (tramp-file-name-handler unlock-file "/ssh:detlef:/tmp/tra= mp.qgnQ01")
| | | 4 -> (tramp-file-name-handler make-lock-file-name "/ssh:detle= f:/tmp/tramp.qgnQ01")
| | | 4 <- tramp-file-name-handler: nil
| | 3 <- tramp-file-name-handler: nil
| 2 <- tramp-file-name-handler: nil
| 2 -> (tramp-send-command (tramp-file-name "ssh" nil nil &quo= t;detlef" nil "/home/albinus/.emacs" nil) "dd bs=3D1 sk= ip=3D2 if=3D/home/albinus/.emacs count=3D2 of=3D/tmp/tramp.qgnQ01") | 2 <- tramp-send-command: 72
| 2 -> (tramp-file-name-handler file-local-copy "/ssh:detlef:/tmp/t= ramp.qgnQ01")
| | 3 -> (tramp-file-name-handler file-truename "/ssh:detlef:/tmp/t= ramp.qgnQ01")
| | | 4 -> (tramp-file-name-handler file-remote-p "/ssh:detlef:/tmp= /tramp.qgnQ01" localname nil)
| | | | 5 -> (tramp-file-name-handler expand-file-name "/ssh:detlef= :/tmp/tramp.qgnQ01" nil)
| | | | 5 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ0= 1"
| | | 4 <- tramp-file-name-handler: "/tmp/tramp.qgnQ01"
| | | 4 -> (tramp-file-name-handler expand-file-name "/ssh:detlef:/= tmp/tramp.qgnQ01" nil)
| | | 4 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01&= quot;
| | | 4 -> (tramp-file-name-handler file-remote-p "/ssh:detlef:/tmp= /tramp.qgnQ01" localname nil)
| | | | 5 -> (tramp-file-name-handler expand-file-name "/ssh:detlef= :/tmp/tramp.qgnQ01" nil)
| | | | 5 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ0= 1"
| | | 4 <- tramp-file-name-handler: "/tmp/tramp.qgnQ01"
| | | 4 -> (tramp-file-name-handler file-remote-p "/ssh:detlef:/tmp= /tramp.qgnQ01" nil nil)
| | | | 5 -> (tramp-file-name-handler expand-file-name "/ssh:detlef= :/tmp/tramp.qgnQ01" nil)
| | | | 5 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ0= 1"
| | | 4 <- tramp-file-name-handler: "/ssh:detlef:"
| | | 4 -> (tramp-send-command (tramp-file-name "ssh" nil nil = "detlef" nil "/tmp/tramp.qgnQ01" nil) "\\readlink = --canonicalize-missing /tmp/tramp.qgnQ01 2>/dev/null; echo tramp_exit_st= atus $?")
| | | 4 <- tramp-send-command: 39
| | | 4 -> (tramp-file-name-handler expand-file-name "/ssh:detlef:/= tmp/tramp.qgnQ01" nil)
| | | 4 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01&= quot;
| | | 4 -> (tramp-file-name-handler file-symlink-p "/ssh:detlef:/tm= p/tramp.qgnQ01")
| | | | 5 -> (tramp-file-name-handler expand-file-name "/ssh:detlef= :/tmp/tramp.qgnQ01" nil)
| | | | 5 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ0= 1"
| | | | 5 -> (tramp-file-name-handler file-attributes "/ssh:detlef:= /tmp/tramp.qgnQ01")
| | | | | 6 -> (tramp-file-name-handler expand-file-name "/ssh:detl= ef:/tmp/tramp.qgnQ01" nil)
| | | | | 6 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgn= Q01"
| | | | | 6 -> (tramp-send-command (tramp-file-name "ssh" nil = nil "detlef" nil "/tmp/tramp.qgnQ01" nil) "tramp_s= tat_file_attributes /tmp/tramp.qgnQ01 2>/dev/null; echo tramp_exit_statu= s $?")
| | | | | 6 <- tramp-send-command: 148
| | | | 5 <- tramp-file-name-handler: (nil 1 1000 1000 (25312 7412 0 0) = (25312 7412 0 0) (25312 7412 0 0) 2 "-rw-rw-r--" nil 3821258 (-1 = . 2))
| | | 4 <- tramp-file-name-handler: nil
| | 3 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01&qu= ot;
| | 3 -> (tramp-file-name-handler file-name-nondirectory "/ssh:detl= ef:/tmp/tramp.qgnQ01")
| | 3 <- tramp-file-name-handler: "tramp.qgnQ01"
| | 3 -> (tramp-file-name-handler expand-file-name "/ssh:detlef:/tm= p/tramp.qgnQ01" nil)
| | 3 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01&qu= ot;
| | 3 -> (tramp-file-name-handler file-attributes "/ssh:detlef:/tmp= /tramp.qgnQ01")
| | | 4 -> (tramp-file-name-handler expand-file-name "/ssh:detlef:/= tmp/tramp.qgnQ01" nil)
| | | 4 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01&= quot;
| | 3 <- tramp-file-name-handler: (nil 1 1000 1000 (25312 7412 0 0) (253= 12 7412 0 0) (25312 7412 0 0) 2 "-rw-rw-r--" nil 3821258 (-1 . 2)= )
| | 3 -> (tramp-send-command (tramp-file-name "ssh" nil nil &q= uot;detlef" nil "/tmp/tramp.qgnQ01" nil) "base64 </t= mp/tramp.qgnQ01 2>/dev/null; echo tramp_exit_status $?")
| | 3 <- tramp-send-command: 26
| | 3 -> (tramp-file-name-handler expand-file-name "/ssh:detlef:/tm= p/tramp.qgnQ01" nil)
| | 3 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01&qu= ot;
| | 3 -> (tramp-file-name-handler file-modes "/ssh:detlef:/tmp/tram= p.qgnQ01" nil)
| | | 4 -> (tramp-file-name-handler expand-file-name "/ssh:detlef:/= tmp/tramp.qgnQ01" nil)
| | | 4 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01&= quot;
| | | 4 -> (tramp-file-name-handler file-attributes "/ssh:detlef:/t= mp/tramp.qgnQ01")
| | | | 5 -> (tramp-file-name-handler expand-file-name "/ssh:detlef= :/tmp/tramp.qgnQ01" nil)
| | | | 5 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ0= 1"
| | | 4 <- tramp-file-name-handler: (nil 1 1000 1000 (25312 7412 0 0) (2= 5312 7412 0 0) (25312 7412 0 0) 2 "-rw-rw-r--" nil 3821258 (-1 . = 2))
| | 3 <- tramp-file-name-handler: 436
| 2 <- tramp-file-name-handler: "/tmp/tramp.6ce5x9.qgnQ01"
| 2 -> (tramp-file-name-handler expand-file-name "/ssh:detlef:/tmp/= tramp.qgnQ01" nil)
| 2 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01"= ;
| 2 -> (tramp-file-name-handler file-directory-p "/ssh:detlef:/tmp/= tramp.qgnQ01")
| 2 <- tramp-file-name-handler: nil
| 2 -> (tramp-file-name-handler expand-file-name "/ssh:detlef:/tmp/= tramp.qgnQ01" nil)
| 2 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01"= ;
| 2 -> (tramp-file-name-handler delete-file "/ssh:detlef:/tmp/tramp= .qgnQ01" nil)
| | 3 -> (tramp-file-name-handler expand-file-name "/ssh:detlef:/tm= p/tramp.qgnQ01" nil)
| | 3 <- tramp-file-name-handler: "/ssh:detlef:/tmp/tramp.qgnQ01&qu= ot;
| | 3 -> (tramp-send-command (tramp-file-name "ssh" nil nil &q= uot;detlef" nil "/tmp/tramp.qgnQ01" nil) "rm -f /tmp/tr= amp.qgnQ01 2>/dev/null; echo tramp_exit_status $?")
| | 3 <- tramp-send-command: 21
| 2 <- tramp-file-name-handler: t
1 <- tramp-file-name-handler: ("/ssh:detlef:/home/albinus/.emacs&qu= ot; 2)
--00000000000084b39605e533bc46-- From debbugs-submit-bounces@debbugs.gnu.org Tue Aug 02 10:23:29 2022 Received: (at 56342) by debbugs.gnu.org; 2 Aug 2022 14:23:29 +0000 Received: from localhost ([127.0.0.1]:44651 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1oIsnx-0007x6-Bl for submit@debbugs.gnu.org; Tue, 02 Aug 2022 10:23:29 -0400 Received: from mout.gmx.net ([212.227.15.15]:47123) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1oIsnv-0007ws-0y for 56342@debbugs.gnu.org; Tue, 02 Aug 2022 10:23:27 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.net; s=badeba3b8450; t=1659450200; bh=rbPqwGVaYBK4Y3cb8m6tvuEkIdqoohWpw5o4IN2mmKw=; h=X-UI-Sender-Class:From:To:Cc:Subject:In-Reply-To:References:Date; b=afLnSMPxu6Lcn7nu1Z9u5H/vzN/cSu0og3l8LUWh4Go0TRzLCMzK1Jmyg8dG/me/w UcyaL7dYWG39pDH3eo6iBzJQj1KGZTd7EWlGQm8e0jmqrg4aCgBga0mX+pSc+J5x0a cH2jyuDQ9QSgO/fsxqfvnAgZIkm9wnAyv+LXWd24= X-UI-Sender-Class: 01bb95c1-4bf8-414a-932a-4f6e2808ef9c Received: from gandalf.gmx.de ([213.220.149.238]) by mail.gmx.net (mrgmx004 [212.227.17.190]) with ESMTPSA (Nemesis) id 1MmULr-1naiIQ3z7J-00iUcd; Tue, 02 Aug 2022 16:23:20 +0200 From: Michael Albinus To: Paul Pogonyshev Subject: Re: bug#56342: TRAMP (sh) issues way too many commands, thus being very slow over high-ping networks In-Reply-To: (Paul Pogonyshev's message of "Mon, 1 Aug 2022 22:20:41 +0200") References: <8735fjh5ge.fsf@gmx.de> <87v8sefl2f.fsf@gmx.de> <874jzyc9u9.fsf@gmx.de> <87sfnhazvy.fsf@gmx.de> <87k08sc02z.fsf@gmx.de> <87r128gdob.fsf@gmx.de> <87mtcvhidx.fsf@gmx.de> Date: Tue, 02 Aug 2022 16:23:18 +0200 Message-ID: <87o7x22089.fsf@gmx.de> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/29.0.50 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Provags-ID: V03:K1:RtC6lBH9r/qi7V/eCsLw7BaUb2k0aIsBFr5MfrA9XMiEg/SdJqN djZw3QHMY0OFJtsvesbx3WAn/kX8465NliiHIfEoZjoToTB42hG1m7h7J/k08u6Wmqytb/p yjzllDxv7J4MOplBFq7QGzaas52kR7EsDIYdy1hJmAx1MBDNOxNVU8goyvG9vaMpV3CqEVR +uCsQz7G4YyX5aYwg4ekg== X-Spam-Flag: NO X-UI-Out-Filterresults: notjunk:1;V03:K0:5Qm1KTtVKF0=:i2ivwQlGD7cC9ZLGH0+CAp 69Abq7W9myMbVFU1T2jcreQryiBRbryLs8hGVbS/lb5Qc4Q8WLTjNLPzrTPqXs4s07s0xvDuj z7CLtxDKGu4TA5bPn1+18aKppvMjrcqTrlt5uUMjh+CgiRIDxxQZJ7SWH33FH9/HBIrEnjcZN 3w/JEkXcRayff2qGRUP+l/mpkSP4iaMgYxk9pPPuXQsINR68X4wQPpKl0K1x/FZbalCria5GA LlP/dEhjTpBZJoa3JiYIGg8HtrDVxfyRClZdUHQJIm81vnZY9gg5Y8R/FWRCFOFtghQIQOjVG EF8Rqw9eldqU8KJ7Em0f0chJRXJ7hO5z4GEimHuNTNAc2K1S9BSBprIj6mFDRb/IgNLWzhmaW D97hSJxEpHkW2Doh7t5LAqHjNNNmRPiUCGA5KE83rkFsFenRYtyOlJHqj+a6wfblJOUeDrLvR pL3tDbN5+35xx9/pAX52X4Gf649Le4KDjuDmcCQ9VbmBS6vzm1SkBUs8u3pJcmur2sg6LwHqk CucOUv22O/4RbJrhyHpWJ77bKvl34FuYvWX6sCyUQGZcgAY06ng8WEWIi0+PgWxLISAq24U/6 8qbj0YWfjJ6iQe8BwYrXC7dLDEXia2/6i5XK0ojUHDTAR/hXF5YSaLs9mEc8TI3S9cKvyjbJ/ gIEjKv8RVaVoeZMg7X8hVp3I+ZNjYGV+DFHeHNhJMpBgw2KLp0vlfM4OfszANwVBiuqWtYAIk /eBO2G9zO8bYnz6rFkObP2WRs2/ooXwV3/zXX1KAyyD04DHjWXQXrMrcZTMozKwHH2iQi+XIY BEInGtXbQHiVDsq+Kw3b3LvUxy6NQ+nL/siSpDcUGxpvscahZIZdn+YHqes95BJQRDpbWYvWF k3wAjR7Q4fOzcCulbRsHzTLViwI8BFH46VCMRS0+y+wCX0ho1LCsyyAtNJXgEFG2qxTnde8RH gQskJ0yrGZ6nvOojey7pOWuFurDXK1BypnEF1Jbpe+X8rzw+NaDDhHVSmqI1/VNdQBAEeOrii HqeR3YMKlmFZk/CTTF1BxwLYn1g8OaZ0jfrCuHVa5hln3mng22Pj4fAit7XuMmgdOTeUMsfUi qfyO01AN5dLiVxol6uQrRQYCJ60347sIGm53lFfybZ1uItTXCdAJ3hX7g== Content-Transfer-Encoding: quoted-printable X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 56342 Cc: 56342@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) Paul Pogonyshev writes: Hi Paul, > Understood, but I propose to adopt a different benchmark: number of > remote commands. As soon as you are not in a LAN, even not when using > a particularly slow network, this becomes an order or two of magnitude > more important than everything else. E.g. with slightly slower > commands or particularly inefficient Elisp you can make it 2-10 ms > slower for everyone. But with unoptimized command flow (i.e. more > remote commands) it can be 100-500 ms slower, though not for everyone, > but people using this over non-LAN. I think in such cases extremes > are more important than the average. I agree, sending something over the wire is always slower than computing clever caches locally. >> And directories are also problematic for caches. As soon as something >> changes there (creation or deletion of a file, for example), the cached >> values of the directory must be flushed. > > Yeah, I suppose Tramp has no generic way to know that Magit issues > reading commands. Can we devise a generic interface that would tell > Tramp: "Commands within this block will not modify file system", > e.g. with let-binding some variable? They will modify the file system, although sometimes only the timestamps. Until now we have only something similar for synchronous processes: process-file-side-effects. I have no idea whether package authors are aware of this, and let-bind it to nil in case of. In the magit sources, I haven't found this variable. > In general, it feels like Tramp flushes its caches too often or maybe > doesn't even cache certain things at all. I.e. it's not about those > 10 seconds (following your advice I have even increased that). It's > that while serving one user-level command here (i.e. within 3 seconds > at most), it can issue the same remote command several times, thus > not reusing previous results. Well, that's right. If, for example, the file modes are changed, Tramp flushes all caches for that file, although some of them ("file-exists-p", "file-directory-p" etc) aren't affected. > E.g. in the traces you have attached this can be seen. The following > two commands repeat twice: I haven't investigated this special case yet, but last days I'm working on exact this problem. Flushing caches shouldn't be a sledge hammer, but fine grained, selecting the needed properties to be flushed. Let's see where we land. > Paul Best regards, Michael. From debbugs-submit-bounces@debbugs.gnu.org Sun May 19 20:05:16 2024 Received: (at 56342) by debbugs.gnu.org; 20 May 2024 00:05:16 +0000 Received: from localhost ([127.0.0.1]:38752 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1s8qWh-0000Kj-IK for submit@debbugs.gnu.org; Sun, 19 May 2024 20:05:16 -0400 Received: from fhigh8-smtp.messagingengine.com ([103.168.172.159]:57787) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1s8qWf-0000Ka-8l for 56342@debbugs.gnu.org; Sun, 19 May 2024 20:05:14 -0400 Received: from compute6.internal (compute6.nyi.internal [10.202.2.47]) by mailfhigh.nyi.internal (Postfix) with ESMTP id BC15511400A5; Sun, 19 May 2024 20:05:02 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute6.internal (MEProxy); Sun, 19 May 2024 20:05:02 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gutov.dev; h=cc :cc:content-transfer-encoding:content-type:content-type:date :date:from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:subject:subject:to:to; s=fm2; t=1716163502; x=1716249902; bh=iA1tdACcQ0COos/fqJIOylaP8Oc4urgNG/bVnoyHQlw=; b= NZvTrZvnQHn67XqVVmyl7ZE7Y+rhXbO1N/2AikMFvDha+p/k9vtbIf8HXXFeQm8i G2U5T/UhcpmjBk7Ku4ZK9QKwFTI5hAZ0yece7wufz8YkKHi4UH9TYsxFjCRQGsaU Njw5/E5ZQYhdaAws7m3VDn7IYbGUAaxcKcqV/cx/o9WuLaJTLazVyjAnOnmfFF6P Bfbzg11S9PGDQSLZexKZnuqeVrDnbbKk/+Npicu7Ry9FZwDEnAqfrMzjCv8W9/55 85vvSvWDHB11wsTdOJ9O2A8DzKye9Js/EHkvotBIlf+eg6jnntoXtUSNgbe3PFR8 zXQtnRPfXXFRoygq5jonfQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:feedback-id:feedback-id :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:subject:subject:to:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm1; t=1716163502; x= 1716249902; bh=iA1tdACcQ0COos/fqJIOylaP8Oc4urgNG/bVnoyHQlw=; b=C UH/vzTdCBgwSy8nVYoKfg20qw+qrwD0amGN643THVNNLnoSN7OSZ1zkr9mrkUJ3W l/FIrfqoAv5DlSK555x4PLlWn5xt1iHYU3aXl+O0IxYZREdLjmILKHYrK2b2PjHu IXYdY0DhW59o1cIe7RAoilgg0JgFv4CBosQpjouJ2oZ8FXSPlDiQKepINdUM0BBx AzQxmlWT2McdNcSwzUnXvDM8KjD2B5QUmOAfHU6qMPrlR8W3yG5yns6WnmjwgPGx I1tLKxsE4wJiBNBwITIt8XokexafIo2xHukoP6e/n982vDsNpOdJ/SC/8In5ZWsg 7Vgts1TH5rmoOdA4W5P9Q== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrvdehledgvdelucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepkfffgggfuffvvehfhfgjtgfgsehtjeertddtvdejnecuhfhrohhmpeffmhhi thhrhicuifhuthhovhcuoegumhhithhrhiesghhuthhovhdruggvvheqnecuggftrfgrth htvghrnhepteduleejgeehtefgheegjeekueehvdevieekueeftddvtdevfefhvdevgedu jeehnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepug hmihhtrhihsehguhhtohhvrdguvghv X-ME-Proxy: Feedback-ID: i0e71465a:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Sun, 19 May 2024 20:05:01 -0400 (EDT) Message-ID: <901c1ee9-d942-4b8e-9c73-45f6c1388abf@gutov.dev> Date: Mon, 20 May 2024 03:04:59 +0300 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: bug#56342: TRAMP (sh) issues way too many commands, thus being very slow over high-ping networks To: Michael Albinus , Paul Pogonyshev References: <8735fjh5ge.fsf@gmx.de> <87v8sefl2f.fsf@gmx.de> <874jzyc9u9.fsf@gmx.de> <87sfnhazvy.fsf@gmx.de> <87k08sc02z.fsf@gmx.de> <87r128gdob.fsf@gmx.de> <87mtcvhidx.fsf@gmx.de> <87o7x22089.fsf@gmx.de> Content-Language: en-US From: Dmitry Gutov In-Reply-To: <87o7x22089.fsf@gmx.de> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 56342 Cc: 56342@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Hi Michael and Paul, I've found this report to be relevant, since the original scenario looks fairly similar to the pain point I'm seeing (which is the synchronized delay during 'save-buffer'. C-x C-s is something I hit pretty regularly, so I think it's one of the scenarios to optimize first. Almost 2 years have passed since the last message in this thread, so maybe you'll want to give an update, if you have particular changes in mind. I'd like to offer some profiler output with the latest Tramp and maybe a few suggestions. On 02/08/2022 17:23, Michael Albinus wrote: >> E.g. in the traces you have attached this can be seen. The following >> two commands repeat twice: > I haven't investigated this special case yet, but last days I'm working > on exact this problem. Flushing caches shouldn't be a sledge hammer, but > fine grained, selecting the needed properties to be flushed. Let's see > where we land. I was wondering whether the cache in question was useful for the case where we regularly change the contents and update the modification time. But perhaps it is, if going by the profiler output below. Anyway, the scenario is: 1. (setq remote-file-name-inhibit-locks t) ; big QoL improvement 2. visit some file on a remote host, via /ssh: 3. insert char (e.g. by pressing RET) 4. save buffer 5. repeat steps 3 and 4 10-15 times quickly, without pause The profiler report: 6341 77% - command-execute 6341 77% - call-interactively 6341 77% - apply 6341 77% - call-interactively@ido-cr+-record-current-command 6341 77% - apply 6341 77% - # 6150 75% - funcall-interactively 6134 75% - save-buffer 6132 75% - basic-save-buffer 3592 43% - basic-save-buffer-1 3592 43% - basic-save-buffer-2 3260 39% - write-region 3260 39% - tramp-file-name-handler 3260 39% - apply 3260 39% - tramp-sh-file-name-handler 3260 39% - apply 3260 39% - tramp-sh-handle-write-region 1771 21% - tramp-barf-unless-okay 1771 21% + tramp-send-command-and-check 1299 15% - file-attributes 1299 15% - tramp-file-name-handler 1299 15% - apply 1299 15% - tramp-sh-file-name-handler 1299 15% - apply 1299 15% - tramp-sh-handle-file-attributes 1299 15% + tramp-do-file-attributes-with-stat 171 2% + file-truename 7 0% + # 4 0% + expand-file-name 4 0% write-region 4 0% + tramp-message 332 4% + file-writable-p 2536 31% - verify-visited-file-modtime 2536 31% - tramp-file-name-handler 2536 31% - apply 2536 31% - tramp-sh-file-name-handler 2536 31% - apply 2536 31% - tramp-sh-handle-verify-visited-file-modtime 2536 31% - file-attributes 2536 31% - tramp-file-name-handler 2536 31% - apply 2536 31% - tramp-sh-file-name-handler 2536 31% - apply 2536 31% - tramp-sh-handle-file-attributes 2528 30% + tramp-do-file-attributes-with-stat 4 0% + tramp-get-file-property What I see here is a few things: 1. First we look at the file in verify-visited-file-modtime and then in write-region. It might be more economical to combine that in some script sent to the remote host to be executed. But that seems hard (even aside the coding required - it'd need a new file handler, I guess, one that would basically do the whole job of basic-save-buffer). 2. There are tramp-sh-handle-file-attributes calls in different branches that a fair amount of time gets spent in. Could those be cached, to only issue one call? Could a binding for 'process-file-side-effects' help? 3. The file-truename call. Curiously, it's only called by tramp-sh-handle-write-region. Does the handler really need to know the canonical file name? It seems like it'd be okay if it wrote to one of the symlinked names instead. Ah, looking at tramp-skeleton-write-region, this seems to have something to do with locking. Could we optimize this at least for the case when remote-file-name-inhibit-locks is set to t? Here's also some filtered output from the Tramp debug log from *one* save-buffer call: 02:39:44.488432 tramp-send-command (6) # tramp_stat_file_attributes /home/d/dgutov/git-test/b/c/caca 2>/dev/null; echo tramp_exit_status $? 02:39:44.688011 tramp-send-command (6) # test -e /home/d/dgutov/git-test/b/c/ 2>/dev/null; echo tramp_exit_status $? 02:39:44.890427 tramp-send-command (6) # test -w /home/d/dgutov/git-test/b/c/ 2>/dev/null; echo tramp_exit_status $? 02:39:45.097458 tramp-send-command (6) # (if test -h "/srv/data/home/d/dgutov/git-test/b/c/caca"; then echo t; else echo nil; fi) && \readlink --canonicalize-missing /srv/data/home/d/dgutov/git-test/b/c/caca 2>/dev/null; echo tramp_exit_status $? 02:39:45.303095 tramp-send-command (6) # tramp_stat_file_attributes /home/d/dgutov/git-test/b/c/caca 2>/dev/null; echo tramp_exit_status $? 02:39:45.508024 tramp-send-command (6) # base64 -d -i >/home/d/dgutov/git-test/b/c/caca <<'45aa0e36081f9f2cbcfda421e708711e' 2>/dev/null; echo tramp_exit_status $? 02:39:45.708813 tramp-send-command (6) # tramp_stat_file_attributes /home/d/dgutov/git-test/b/c/caca 2>/dev/null; echo tramp_exit_status $? It seems to stat the file (with tramp_stat_file_attributes) 3 times. One of these is from verify-visited-file-modtime, and two others - from other branches of execution. And another verify-visited-file-modtime call (with another tramp-do-file-attributes-with-stat) happens whenever you type the first character after saving the buffer. Which also contributes to perceived latency. Could these be cached by adding some new process-file-side-effects bindings? Other things we see above is several separate 'test' calls. Not sure if those could be combined (by having a command return all the data together, and then have it cached?), but that'd be nice.