X-Git-Url: https://sipb.mit.edu/gitweb.cgi/ikiwiki.git/blobdiff_plain/8bb94bb197714fcac1ac48f9b330bef4d17dd800..b6b7dc3a43266515822c31b177bfd26e8d59f359:/doc/todo/dependency_types.mdwn diff --git a/doc/todo/dependency_types.mdwn b/doc/todo/dependency_types.mdwn index db7d06914..4bc38e9c0 100644 --- a/doc/todo/dependency_types.mdwn +++ b/doc/todo/dependency_types.mdwn @@ -1,5 +1,24 @@ -Moved this relevant discussion to here from -[[tracking_bugs_with_dependencies]]: --[[Joey]] +Ikiwiki currently only has one type of dependency between pages +(plus wikilinks special cased in on the side). This has resulted in various +problems, and it's seemed for a long time to me that ikiwiki needs to get +smarter about what types of dependencies are supported. + +### unnecessary work + +The current single dependency type causes the depending page to be rebuilt +whenever a matching dependency is added, removed, or *modified*. But a +great many things don't care about the modification case, and often cause +unnecessary page rebuilds: + +* map only cares if the pages are added or removed. Content change does + not matter (unless show=title is used). +* brokenlinks, orphans, pagecount, ditto (generally) +* inline in archive mode cares about page title, author changing, but + not content. (Ditto for meta with show=title.) +* Causes extra work when solving the [[bugs/transitive_dependencies]] + problem. + +### two types of dependencies needed for [[tracking_bugs_with_dependencies]] >> it seems that there are two types of dependency, and ikiwiki >> currently only handles one of them. The first type is "Rebuild this @@ -81,3 +100,166 @@ Moved this relevant discussion to here from >>>>> For old-style pagespecs without backlinks, the dependency set for a pagespec is the same as the set of pages the pagespec matches. >>>>> Unfortunately, with existential matching, the set of pages that each >>>>> pagespec depends upon can quickly become "*", which is not very useful. -- [[Will]] + +### proposal + +I propose the following. --[[Joey]] + +* Add a second type of dependency, call it an "presence dependency". +* `add_depends` defaults to adding a regular ("full") dependency, as + before. (So nothing breaks.) +* `add_depends($page, $spec, presence => 0)` adds an presence dependency. +* `refresh` only looks at added/removed pages when resolving presence + dependencies. + +This seems straightforwardly doable. I'd like [[Will]]'s feedback on it, if +possible. The type types of dependencies I am proposing are not identical +to the two types he talks about above, but I hope are close enough that +they can be used. + +This doesn't deal with the stuff that only depend on the metadata of a +page, as collected in the scan pass, changing. But it does leave a window +open for adding such a dependency type later. + +---- + +I implemented the above in a branch. +[[!template id=gitbranch branch=origin/dependency-types author="[[joey]]"]] + +Then I found some problems: + +* Something simple like pagecount, that seems like it could use a + presence dependency, can have a pagespec that uses metadata, like + `author()` or `copyright()`. +* pagestats, orphans and brokenlinks cannot use presence dependencies + because they need to update when links change. + +Now I'm thinking about having a special dependency look at page +metadata, and fire if the metadata changes. And it seems links should +either be included in that, or there should be a way to make a dependency +that fires when a page's links change. (And what about backlinks?) + +It's easy to see when a page's links change, since there is `%oldlinks`. +To see when metadata is changed is harder, since it's stored in the +pagestate by the meta plugin. Also, there are many different types of +metadata, that would need to be matched with the pagespecs somehow. + +Quick alternative: Make add_depends look at the pagespec. Ie, if it +is a simple page name, or a glob, we know a presence dependency +can be valid. If's more complex, convert the dependency from +presence to full. + +There is a lot to dislike about this method. Its parsing of the pagespec, +as currently implemented, does not let plugins add new types of pagespecs +that only care about presence. Its pagespec parsing is also subject to +false negatives (though these should be somewhat rare, and no false +positives). Still, it does work, and it makes things like simple maps and +pagecounts much more efficient. + +---- + +#### Will's first pass feedback. + +If the API is going to be updated, then it would be good to make it forward compatible. +I'd like for the API to be extendible to what is useful for complex pagespecs, even if we +that is a little redundant at the moment. + +My attempt to play with this is in my git repo. [[!template id=gitbranch branch=origin/depends-spec author="[[will]]"]] +That branch is a little out of date, but if you just look at the changes in IkiWiki.pm you'll see the concept I was looking at. +I added an "add_depends_spec()" function that adds a dependency on the pagespec passed to it. If the set of matched pages +changes, then the dependent page is rebuilt. At the moment the implementation uses the same hack used by map and inline - +just add all the pages that currently exist as traditional content dependencies. + +Getting back to commenting on your proposal: + +Just talking about the definition of a "presence dependency" for the moment, and ignoring implementation. Is a +"presence dependency" supposed to cause an update when a page disappears? I assume so. Is a presence dependency +supposed to cause an update when a pages existence hasn't changed, but it no longer matches the pagespec. +(e.g. you use `created_before(test_page)` in a pagespec, and there was a page, `new_page`, that was created +after `test_page`. `new_page` will not match the spec. Now we'll delete and then re-create `test_page`. Now +`new_page` will match the spec, and yet `new_page` itself hasn't changed. Nor has its 'presence' - it was present +before and it is present now. Should this cause a re-build of any page that has a 'presence' dependency on the spec? + +I think that is another version of the problem you encountered with meta-data. + +In the longer term I was thinking we'd have to introduce a concept of 'internal pagespec dependencies'. Note that I'm +defining 'internal' pagespec dependencies differently to the pagespec dependencies I defined above. Perhaps an example: +If you had a pagespec that was `created_before(test_page)`, then you could list all pages created before `test_page` +with a `map` directive. The map directive would add a pagespec dependency on `created_before(test_page)`. +Internally, there would be a second page-spec parsing function that discovers which pages a given pagespec +depends on. As well as the function `match_created_before()`, we'd have to add a new function `depend_created_before()`. +This new function would return a list of pages, which when any of them change, the output of `match_created_before()` +would change. In this example, it would just return `test_page`. + +These lists of dependent pages could just be concatenated for every `match_...()` function in a pagespec - you can ignore +the boolean formula aspects of the pagespec for this. If a content dependency were added on these pages, then I think +the correct rebuilds would occur. + +In all, this is a surprisingly difficult problem to solve perfectly. Consider the following case: + +PageA.mdwn: + +> [ShavesSelf] + +PageB.mdwn + +> Doesn't shave self. + +ShavedByBob.mdwn: + +> [!include pages="!link(ShavesSelf)"] + +Does ShavedByBob.mdwn include itself? + +(Yeah - in IkiWiki currently links are included by include, but the idea holds. I had a good example a while back, but I can't think of it right now.) + +sigh. + +-- [[Will]] + +---- + +### Link dependencies + +* `add_depends($page, $spec, links => 1, presence => 1)` + adds a links + presence dependency. +* Use backlinks change code to detect changes to link dependencies too. +* So, brokenlinks can fire whenever any links in any of the + pages it's tracking change, or when pages are added or + removed. +* To determine if a pagespec is valid to be used with a links dependency, + use the same set that are valid for presence dependencies. But also + allow `backlinks()` to be used in it, since that matches pages + that the page links to, which is just what link dependencies are + triggered on. + +[[done]] +---- + +### the removal problem + +So far I have not addressed fixing the removal problem (which Will +discusses above). + +Summary of problem: A has a dependency on a pagespec such as +"bugs/* and !link(done)". B currently matches. Then B is updated, +in a way that makes A's dependency not match it (ie, it links to done). +Now A is not updated, because ikiwiki does not realize that it +depended on B before. + +This was worked around to fix [[bugs/inline_page_not_updated_on_removal]] +by inline and map adding explicit dependencies on each page that appears +on them. Then a change to B triggers the explicit dep. While this works, +it's 1) ugly 2) probably not implemented by all plugins that could +be affected by this problem (ie, linkmap) and 3) is most of the reason why +we grew the complication of `depends_simple`. + +One way to fix this is to include with each dependency, a list of pages +that currently match it. If the list changes, the dependency is triggered. + +Should be doable, but seems to involve a more work than +currently. Consider that a dependency on "bugs/*" currently +is triggered by just checking until *one* page is found to match it. +But to store the list, *every* page would have to be tried against it. +Unless the list can somehow be intelligently updated, looking at only the +changed pages.