Subject: bug#75893: texlive: kpathsea doesn't use ls-R database

From: Nicolas Goaziou <[email-masked]>
Date: Wed, 29 Jan 2025 19:11:20 +0100

Hello,

vicvbcun <[email-masked]> writes:

> Consider the following example latex document:
>
> --8<---------------cut here---------------start------------->8---
> \documentclass{article}
> 	\usepackage{mathtools}
>
> \begin{document}
> 	hello world
> \end{document}
> --8<---------------cut here---------------end--------------->8---
>
> Compiling it with LuaLaTeX under strace in a shell with 
> texlive-scheme-basic, texlive-collection-luatex and 
> texlive-collection-latexextra, it seems like most of the time is spent 
> recursively searching for input files:
>
> --8<---------------cut here---------------start------------->8---
> % time     seconds  usecs/call     calls    errors syscall
> ------ ----------- ----------- --------- --------- ----------------
>   27.70    0.080138           2     30174           getdents64
>   21.99    0.063605           4     15455       259 openat
>   17.44    0.050460           3     16179        32 newfstatat
>   14.37    0.041583           3     10440     10296 access
>    8.42    0.024348           1     15196           close
>    7.76    0.022456           1     15201           fstat
>    0.79    0.002278           1      1868           write
> --8<---------------cut here---------------end--------------->8---
>
> and similarly for pdflatex.
>
> As an extreme example, consider
>
> --8<---------------cut here---------------start------------->8---
> \documentclass{tudapub}
>
> \begin{document}
> 	hello world
> \end{document}
> --8<---------------cut here---------------end--------------->8---
>
> compiled with
>
> --8<---------------cut here---------------start------------->8---
> texlive-scheme-basic
> texlive-collection-luatex
> texlive-collection-latexextra
> texlive-roboto texlive-urcls
> texlive-xcharter
> texlive-tuda-ci
> --8<---------------cut here---------------end--------------->8---
>
>
> This takes over 14 seconds (compared to about 2.7 seconds for lualatex 
> from Arch Linux) and from strace:
>
> --8<---------------cut here---------------start------------->8---
> % time     seconds  usecs/call     calls    errors syscall
> ------ ----------- ----------- --------- --------- ----------------
>   32.60    5.926537           3   1801518           getdents64
>   26.46    4.809462           5    900841       284 openat
>   20.90    3.799744           4    896057    895349 access
>   10.19    1.851520           2    900557           close
>    9.49    1.724891           1    900575           fstat
>    0.28    0.050743           2     17680       229 newfstatat
>    0.04    0.007077           1      6073           read
> --8<---------------cut here---------------end--------------->8---

Thank you for the report. I confirm the issue, unfortunately.

> The cause for this seems to be kpathsea doesn't treat the ls-R database 
> as authoritative.  It is opened but kpathsea falls back to recursive 
> searching.

AFAIU, this should not happen. According to "The TeX Live Guide 2024":

  If a file is not found in the database, by default Kpathsea goes ahead
  and searches the disk. If a particular path element begins with ‘!!’,
  however, only the database will be searched for that element, never
  the disk.

IOW, even if the "!!" prefix is not there, Kpathsea should first look
for files in ls-R, and then on the disk. As you point out, it doesn’t
happen like this, and I don’t know why.

> In the package definition for texlive-libkpathsea, texmf.cnf is modified 
> such that the TEXMF variable is set without !! in front of 
> $TEXMFSYSCONFIG, $TEXMFSYSVAR and $TEXMFDIST. 
> If I override $TEXMF via --cnf-line like
>
> --8<---------------cut here---------------start------------->8---
> lualatex \
> 	--cnf-line='TEXMF =
> 	{$TEXMFCONFIG,$TEXMFVAR,$TEXMFHOME,!!$TEXMFSYSCONFIG,!!$TEXMFSYSVAR,!!$TEXMFDIST}' \
> 	example.ltx
> --8<---------------cut here---------------end--------------->8---
>
> compilation time for the extreme example above falls to about 2.5 
> seconds, without excessive searching.

At least it proves our ls-R file is valid, at the expected location.

> The comment above the substitution says that the !! construct wouldn't 
> work for texlive-build-system or when building profiles.  I don't know 
> if it would be possible to work around this but perhaps it could be 
> possible to work around this if installed in profile (or environment)?

I don’t understand what you want to install in a profile. The ls-R file
is already built during profile generation. See "guix/profiles.scm".

Maybe we could keep "!!" prefix and create a ls-R file each time
`texlive-build-system' builds a package and every time
`texlive-updmap.cfg' is an input used to build documentation. In this
case I'm not sure about what should be done for packages propagating TeX
Live libraries without actually using them.

In any case, this would require some experimentation. And it still is
a workaround for a problem we don’t understand yet.

Regards,
-- 
Nicolas Goaziou