major optimisation idea

diff --git a/doc/todo/optimisation_via_git_log.mdwn b/doc/todo/optimisation_via_git_log.mdwn
new file mode 100644 (file)
index 0000000..ad22792
--- /dev/null
+++ b/doc/todo/optimisation_via_git_log.mdwn
@@ -0,0 +1,25 @@
+Currently each time ikiwiki runs a refresh, it has to traverse the entire
+directory tree, looking at each file, to determine what has changed.
+
+Profiling shows this can take many seconds on large sites, and tends to be
+where around half the time is spent (other half being loadindex/saveindex).
+
+This could be sped up by remembering the HEAD ref that was rendered last,
+and using git log to get a list of files changed since then. Only render
+those files.
+
+Looking at the current implementation, the list of all files in the tree is
+also used when eg, iterating to find pages that need to be changed because
+one of their dependencies has changed. That would need to be modified,
+probably to use info from loadindex instead (ie, values %pagesources), with
+added/removed files added/removed from it.
+
+The actual git log parsing would be easy to add; there is already code to
+parse git-log in raw form.
+
+The main complication would probably be files in the tree that are not
+checked into git would not be rendered. The underlay dirs would still need
+to be scanned as now, as would the transient directory. Since some sites
+may depend on files being put into the tree and not committed, this
+optimisation would probably need to be something that can be optionally
+enabled.