Subject: bug#71345: Feature: unleash font-lock's secret weapon; handle Qfontified = non-nil

From: JD Smith <[email-masked]>
Date: Thu, 6 Jun 2024 23:27:33 -0400

Brief followup.  I took Stefan's concept of multiple jit-lock backends taking turns with the help of additional "marker" properties and, as a test of the idea, implemented a bespoke version, as follows:

    • I have two "backends": normal font-lock (syntactic + keyword), and my own bar drawing.

    • Rather than "already-done" I used the complementary "pending" approach.  Since I have only two backends, and one is always run, only one needs to indicate it is "pending".

    • Merely for convenience, to implement the bar-drawing backend, I wrapped the `font-lock-fontify-region-function' with a function [1] that can run (or, if indicated, inhibit) font-lock, after which it always draws indent bars.  Basically I stuffed two backends into one slot.

    • To determine when font-lock is needed for a region, I had to advise `jit-lock-context' and provide a `jit-lock-after-change-extend-region-function', both of which simply serve to mark text requested for font-locking with an `indent-bars-font-lock-pending' property.

    • The presence of that property on a region is then the sign that font-lock must be run [2].

This is clearly a non-generalizable approach, but in my testing it works very well to reduce unnecessary refontification to a minimum.  It also actually simplified and shortened the code somewhat.  I'm now confident that a multi-jit-lock capability that allows for expressing dependencies like the above would work well.

A few relevant things I learned:

    • Backends need a way to opt in or out of contextual fontification.

    • Since several backends may operate during one redisplay cycle (e.g. imagine font-lock updated by an after-change + scope altered by a PCH), backends must mark their pendency defensively, and assume nothing about other backend's status.

    • I "hard-coded" the dependency between my backends by always drawing the bars after possibly running font-lock.  A more general solution of expressing such dependencies would be preferable.

One idea for the latter: backends can indicate which text-properties they intend to alter, and, when a backend needs to be run on a region, all others which share properties with it and appear _after_ it on the `jit-lock-functions' list will also be run.  So I'd express this as:

jit-lock-functions = (font-lock-fontify-region my-backend-fontify-region)
jit-lock-function-properties = ((font-lock-fontify-region face) 
                                              (my-backend-fontify-region face my-display-alias))

Potential flaws in my approach:

    • Advising `jit-lock-context' is not ideal, but that function currently provides no hooks for backends.

    • This basically hard-codes two backends together underneath font-lock.  Whether and how it would work with other jit-lock backends remains to be seen. 

    • Maintaining my own properties to determine when to run each backend is not so bad, but having this handled automatically would be simpler.

    • Technically I don't need to refontify on context-refontification, but because font-lock and I both use 'face, I do.  I wish both of us could switch to aliases (I do use a custom 'display alias where needed).

So, to summarize, Stefan's idea of re-using fontified=nil without any changes to redisplay for multiple (in)dependent backends, managed by jit-lock using other text-properties, seems very tractable and likely to be quite useful, if the API can be clearly expressed and documented.  

Very happy to assist with this, for example by "porting" my mode to use in-development versions.

[1] https://github.com/jdtsmith/indent-bars/blob/ea0cd3de21cac3c7bbfd0a0cd8fc1cfa58469290/indent-bars.el#L1190-L1205
[2] https://github.com/jdtsmith/indent-bars/blob/ea0cd3de21cac3c7bbfd0a0cd8fc1cfa58469290/indent-bars-ts.el#L394-L402