Subject: Re: bug#61817: 30.0.50; Project.el finds incorrect project roots in git worktrees

From: Dmitry Gutov <[email-masked]>
Date: Wed, 1 Mar 2023 01:51:09 +0200

On 28/02/2023 02:34, Arthur Miller wrote:
> Dmitry Gutov <[email-masked]> writes:
> 
>> On 27/02/2023 16:15, Arthur Miller wrote:
>>
>>>> We could change project-try-vc to follow the link to the parent repo, but how is
>>>> the rest of it going to work?
>>>>
>>>> If the project root is the parent repo, which set of files would (project-files
>>>> pr) return? And how could that be implemented?
>>> I don't know how it works now, so I can't really tell you, but you have called
>>> git root "actual git root", so perhaps something that differentiate between
>>> directory hierarchy root (worktree), and real git root (project). Perhaps
>>> project-vc-root, if you don't already use that name, or something similar?
>>
>> Do you mean you want a separate helper to get to the "worktree parent"
>> repository?
> 
> I wouldn't call it "worktree parent", because it relatively git specific; if you
> prefer to be project neutral; but something like that yes.

It's a Git-based relation.

Depending on how we might define a "project parent" in the future 
(probably as "project of root's parent directory", though), and how the 
worktrees reside on disk relative to the parent, these notions won't 
necessarily match.

>> repository? We could add something like that to VC, I guess. But your own code
>> seems to be working well too.
> 
> I didn't meant to ask for someone to write it for me :), I can use my own
> code, but as said I had somewhat different expectation from project.el when I
> did this request. You can just close it, it is ok; I am sorry for the trouble.

Sure, closing. I just didn't want to shut the thread down before all 
questions are answered.

>> I haven't had a use for "worktree parent" myself, though. And until we get a
>> good idea for how people will use a feature, it's a little difficult to choose a
>> place for it.
> 
> If you have a branch "main", work in a branch X, and would like to test/fix
> something unrelated to X, wouldn't you create a new workree Y from clean "main"?
> It is a bit special case, which I personally use, so to automate this, "worktree"
> as root is not enough, I need the actual root. For other purposes I need the
> worktree. So as said, both are needed. Of course, worktree will be used more
> frequently for natural reasons.

That sounds like a fine approach, though not the only possible one. I 
usually have a limited number of worktrees: some constant, and some for 
temporary branches. I suppose it depends whether you need to create some 
configuration files specific to the local system when deploying and 
testing a new repository checkout.

>>> I don't mean it is not useful, we need both,
>>> operations that work as projectile/project, on directory trees, but also those
>>> that actually understand a possible project structure. To me projectile and now
>>> project.el seems more like they are an extension to Dired, not in some negative
>>> connotation, but at least conceptually, then really project handling. I am not
>>> trying to be negative, I am a bit oversimplistic and exaggerating, becuase I am
>>> trying to illustrate the difference how I think about "projects".
>>
>> It a way -- yes, because we describe a project as a set of files. With a
>> well-defined root. But it doesn't absolutely have to be in the shape of a single
>> directory tree, though it usually is.
>>
>> Whenever one chooses a different shape, though, they should consider how that
>> will affect the common uses.
> 
> It is not about the shape; you can put a worktree as a subdirectory in a main
> repository, it does not need to be out of source tree; project.el will still not
> find the real git root. You can test it yourself in Emacs source:

Indeed, it won't, because nobody told it it should traverse up to 
"worktree parent", always or under certain condition.

That structure would also be quite weird, though: root in dir A, but all 
project files residing in some sub-sub-directory D of A. Doesn't that 
sound weird?

> In Emacs repo create new worktree: git worktree add new-test; change to newly
> created folder and test your tools, they will work only on new-test.

Sure. Just keep in mind that worktrees are not always a subdirectory of 
the "worktree parent". I keep them side-by-side, for example.

>>> Version control has become an integral part of all projects, at least in
>>> software industry, just as building projects, boilerplate generating projects
>>> etc. What I am saying is that project-neutral is OK, we don't want just a
>>> different name for Git commands, but any project library nowadays should be able
>>> to work with vc systems, build systems etc. At least for software management
>>> projects.
>>
>> You seem to be assigning a lot of meaning to the fact that the root doesn't
>> point to "worktree root", but to the current repository root (modulo the search
>> in the parent directory when the current repo is detemined to be a submodule).
>>
>> But would it be a better choice?
> 
> I don't think it is either one; definitely not an exclusive choice. I tried to
> say that we need both.

Then the question would be how to provide both of those pieces of 
information to callers. If we add the constraint that it has to be in 
some abstract way (independent from Git), then the answer is probably 
"no way".

But the feature with "parent" projects will likely satisfy some uses and 
scenarios.

> But the root :) of this discussion, is that I missunderstood project.el as an
> attempt to create a useful middleware for general working with projects, but I
> guess you and other authors are more seeing it as an end-user application. I
> appologize, I should have probably looked more into the code and asked. I just
> tried to use some functions :).

I do believe that it's a "useful middleware". And a trait of such 
middlewares is they usually try to present some abstraction over the 
implementation.

>> When I use worktrees, it's to carry on development in a certain branch. Meaning,
>> I need to be able to pull/push/commit in that branch, all within the same
>> worktree.
> 
> Yes, in some cases that is a workflow. But there are cases when its prefered to
> checkout other branch, while work on something else; and as said above, in that
> case is prefered to make a new worktree from either "clean" main or some other branch.

Sure.

>> I'm all for adding new Version Control related features, but they should be
>> based off realistic, specific user scenarios.
> 
> Automation. I don't want to open terminal (or magit, does not matter), switch
> branches and do that stuff manually (if I don't have to).

If, perhaps, we would add a command doing something like the above to 
project.el, or we envision somebody doing it in some popular extension, 
*and* it would be possible to do with several project backends (maybe 
real and potential ones), then that would sound like a feature for 
project.el.

If a caller, OTOH, needs to know that they're working with Git, and with 
a particular project backend, to do what they wanted, then that would be 
a poor fit for the API. But we could try to fit in features in some 
other way. Adding them to the VC package, perhaps.

>>> As said I thought project.el was more general to work with projects on a deeper
>>> level, sort-of EDE replacement or complement, and I wanted to use it as library,
>>> but I understand now it is not.
>>
>> The idea was for a more simple/transparent and extensible EDE alternative. Or an
>> extension point to plug Projectile in.
> 
> Ede is extensible; it is just a bit arcane and there is not too much
> documentation, unfortunately. I would prefer if Ede was re-used instead of the
> duplicated effort, but I do understand if you want to replace it.

I honestly don't understand EDE enough to have a solid opinion. But it 
did seem to have some pronounced pieces that didn't make sense for 
projects I regularly interact with.

>> It can still grow some additions (such as build tool support), but for now we
>> have what we have.
> 
> In my opinion, Emacs lacks
> 
> 1) a good and easy to use boiler-plate generator framework. Ede is meant for that,
> but it is arcane to use and is meant to be used with existing projects. Also, Emacs
> will automatically create a new folder if asked to find-file, when folder does
> not exists. But, if a foder exists, it creates new file in that folder, which is not
> preferred behavior.
> 
> In the end, as an end-user I have to manually create folders or run scripts
> before I can ask Ede to fill-in with some code generator.
> 
> There is no reason to not automate project generation. We do have all the tools
> in Emacs already but I haven't seen any good generator. Maybe there is some, but
> I am not aware of one.
> 
> I am playing with a little framework based on org-capture; but it is nothing
> polished I wish to share with anyone. With boilerplate I mean init git, generate
> readme, lsp database, makefiles and that sort of stuff. I think a combo between
> Ede and project.el would be nice.

That sounds like a decent feature, though one that is often taken up by 
command line tools. It not quite clear what Emacs integration would add 
to that. Besides, perhaps being able to write templates in Org, or 
something.

But the beauty of it, it doesn't have to be strongly bound to 
project.el. It can *use* project-current/project-root, but be a package 
of its own, with separate naming/features/hooks and language/framework 
detection.

> 2) better integration with tools for common workflow(s). Adding
> features, fixes, tests etc, is mostly about creating a new patch, which with
> addition of worktrees in Git can be super nicely automated. For example: add new
> library, or add new executable, or add new fix, it all boiles down to add new
> branch which leads to add new worktree, add a possible target in Makefile,
> create possibly a test file, probably some other thing. It would be nice to be
> able to say: M-x add-new-shared-library or M-x add-new-executable, or M-x
> add-new-fix and Emacs will do at least initial boilerplate.
> 
> If you plan to add build tool support to project.el, it would be welcome :).

Hmm, as I see it, this is split into two different parts:

- Do a Git-specific thing (create a worktree), once per body of work.
- Do a build tool specific thing, probably several.

The thing about the latter, you have described pretty well the general 
direction of what features a hypothetical buildtool.el might contain. 
Alas, I have little experience with languages where you regularly do 
things like that (adding new shared libraries, new executables, other 
build artefacts), and it seems to be an open question whether there is 
point in adding a language agnostic API on top of that, or if people 
should just create c++-buildtool.el, java-buildtool.el, etc, separately 
without trying to unify the UI.

If somebody else wanted to work on this, though, probably a fair amount 
of people on this list would welcome that, and help with 
advice/review/some later contributions.