GNU bug report logs - #64735
29.0.92; find invocations are ~15x slower because of ignores

Previous Next

Full log

View this message in rfc822 format

From: Ihor Radchenko <yantar92 <at> posteo.net>
To: Dmitry Gutov <dmitry <at> gutov.dev>
Cc: Spencer Baugh <sbaugh <at> janestreet.com>, 64735 <at> debbugs.gnu.org
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Date: Thu, 20 Jul 2023 15:42:17 +0000

Dmitry Gutov <dmitry <at> gutov.dev> writes:

>>> ... Last I checked, Lisp-native file
>>> listing was simply slower than 'find'.
>> 
>> Could it be changed?
>> In my tests, I was able to improve performance of the built-in
>> `directory-files-recursively' simply by disabling
>> `file-name-handler-alist' around its call.
>
> Then it won't work with Tramp, right? I think it's pretty nifty that 
> project-find-regexp and dired-do-find-regexp work over Tramp.

Sure. It might also be optimized. Without trying to convince find devs
to do something about regexp handling.

And things are not as horrible as 15x slowdown in find.

>> See https://yhetil.org/emacs-devel/87cz0p2xlc.fsf <at> localhost/
>> (the thread also continues off-list, and it looks like there is a lot of
>> room for improvement in this area)
>
> Does it get close enough to the performance of 'find' this way?

Comparable:

(ignore (let ((gc-cons-threshold most-positive-fixnum)) (benchmark-progn (directory-files-recursively "/home/yantar92/.data" ""))))
;; Elapsed time: 0.633713s
(ignore (let ((gc-cons-threshold most-positive-fixnum)) (benchmark-progn (let ((file-name-handler-alist)) (directory-files-recursively "/home/yantar92/.data" "")))))
;; Elapsed time: 0.324341s
;; time find /home/yantar92/.data >/dev/null
;; real	0m0.129s
;; user	0m0.017s
;; sys	0m0.111s

> Also note that processing all matches in Lisp, with many ignores 
> entries, will incur the proportional overhead in Lisp. Which might be 
> relatively slow as well.

Not significant.
I tried to unwrap recursion in `directory-files-recursively' and tried
to play around with regexp matching of the file list itself - no
significant impact compared to `file-name-handler-alist'.

I am pretty sure that Emacs's native file routines can be optimized to
the level of find.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.