GNU bug report logs - #50960
[PATCH 00/10] Add 'guix shell' to subsume 'guix environment'

Previous Next

Full log

View this message in rfc822 format

From: Ludovic Courtès <ludo <at> gnu.org>
To: Maxime Devos <maximedevos <at> telenet.be>
Cc: 50960 <at> debbugs.gnu.org
Subject: [bug#50960] [PATCH 10/10] shell: Maintain a profile cache.
Date: Fri, 08 Oct 2021 09:37:55 +0200

Hi Maxime,

Maxime Devos <maximedevos <at> telenet.be> skribis:

> Ludovic Courtès schreef op ma 04-10-2021 om 10:19 [+0200]:
>> > A documented flag to always consider the cache stale seems good, though I think
>> > at least the dependencies made with the common macros and procedures 'include',
>> > 'load', 'include-from-path', 'load-from-path', 'use-modules' and non-recursive
>> > 'local-file' could be tracked, though this could be left as a TODO for later
>> > I suppose.
>> 
>> Tracking those uses reliably is impossible: there could be same-named
>> bindings that do other things, there could be custom macros, there could
>> be “dynamic arguments” (whose value is not known statically), etc.  You
>> have to expand + evaluate the code to get better results, and even then,
>> there might be different paths in the code so you can’t be sure you got
>> it right.
>
> I think there's a miscommunication here.  From what I'm reading, what you have
> in mind is that, to determine the dependency information, "guix shell" would
> open "guix.scm", read it with the procedure 'read' and sniff for 'load',
> 'include-from-path', 'load-from-path', 'use-modules' and 'local-file' form
> -- something like 'module-file-dependencies' and 'extract-dependencies', but
> more general.

Yes, that’s roughly what I thought you had in mind, sorry!

> However, my idea is to replace these macros, such that, when "guix shell"
> loads "guix.scm" or "manifest.scm", these macros inform "guix shell"
> that "guix.scm" or "manifest.scm" depend on certain files referred to
> by 'load', 'include-from-path', etc. forms, using a mechanism like the
> 'notice-dependency' defined in <https://issues.guix.gnu.org/50384>.
>
> Then, when "guix shell" puts the resulting profile in the cache,
> it includes the generated list of files.  And when "guix shell" finds
> an entry in the cache, it will check if the files in this list (and guix.scm
> or manifest.scm of course) have been modified.  If some are (or the forcing
> flag is set), guix.scm needs to be loaded and a new profile needs to be
> generated.  If they are all unchanged, guix.scm will _not_ be read: the cached
> profile can be reused.
>
> It's not 100% reliable (e.g. the list of packages in the manifest could
> depend on the phase of the moon if (current-time) is used) (is that what
> you mean by ‘different paths’ and ‘dynamic arguments’?), but it should
> cover the vast majority of cases.
>
> I don't know a non-artifical situation where ‘custom macros’ are a problem
> -- do you know an example in the wild where this dependency tracking scheme
> would fail to work?

I think we have to look at the cost and at the benefits.  The cost:
either we spread ‘notice-dependency’ calls all over the place, or we
interpose on ‘open-file’ altogether, assuming that even works.  The
benefit: we’d have an 80% solution.  There would still be edge cases not
handled correctly; a realistic example is a ‘guix.scm’ whose execution
differs based on the presence of a file or environment variable.  People
would have to know about this pitfall and pass the right flag.

[...]

>> In such situations, I err on the side of not even trying.  The added
>> complexity for a flaky result doesn’t pay off to me.  I prefer to be
>> upfront, document limitations, and let users handle them as they see
>> fit.
>
> About complexity: there's some extra code, sure, but it doesn't seem complex
> to me.  To track dependencies in <https://issues.guix.gnu.org/50384>,
> I only needed to add 'notice-dependency' and some other code to (guix build compile),
> and some code to build-aux/compile-all.scm to save/load the dependency information
> to/from the builddir and to also check the mtime of dependencies.
>
> Does it still seem flaky to you, after my explanation on how the dependency
> information would be determined?
>
> Being upfront, documenting limitations, and having a ‘force rebuild flag’
> (is that what you mean by ‘letting users handle them as they see fit’?)
> (possibly also documenting 'notice-dependency') is not incompatible with
> the automatic dependency tracking.
>
> More abstractly, this seems more like a ‘Perfect is the enemy of the good’ situation.
> While I would prefer 'perfect' above 'good', and the automatic dependency tracking
> certainly isn't 'perfect', it does seem 'good' to me, and perfection appears to
> be impossible and there's the ‘force rebuild flag’ as an escape hatch, so I believe
> 'good-but-not-perfect' to be acceptable here, as long as it is documented in the manual
> that there are some limitation on what dependencies are automatically tracked.

Yeah, I can relate to that sentiment, even if I’m all too often on the
“perfectionist” side of things.  ;-)

I guess I could live with an 80% dependency tracking solution.  However,
my criteria for it would be that (1) it should be orthogonal (e.g., not
requiring explicit calls to ‘notice-dependency’ in many places), and (2)
it should be self-contained and reasonably small.  Perhaps having
‘load*’ or similar instrument ‘open-file’ or similar could help achieve
that?

I would rather not block ‘guix shell’ on that though.

WDYT?

Thanks for taking the time to clarify what you had in mind!

Ludo’.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.