Item8307: Speed up DirectedGraphWebMapPlugin

DirectedGraphWebMapPlugin renders every topic in the web for each graph that it produces. This is slow and does not scale well.

Some remedies are needed:

1. Re-enable the primitive caching that is currently commented out, so that one topic may produce several graphs, without (necessarily) needing to render every topic for each graph.
2. Provide different algorithms for building the web-map data, such as using "search" to find references to topics.

-- MichaelTempest - 11 Oct 2009

I enabled the primitive caching, and it makes a huge difference when one topic has multiple topic maps with similar settings.

However, using the "search" API to build a web-map is slowing than rendering every topic and searching for links. I think this is because the standard "search" uses grep, and there is a separate search (i.e. a separate grep) for every topic in the web, and each grep has to search every topic. The time for a grep-based approach is thus roughly proportional to the square of the number of topics, whereas rendering every topic is uses time roughly linear with the number of topics. A "search" based algorithm could be faster if the search itself is roughly constant and less than the average time to render the topic text.

KinoSearchContrib might provide the necessary performance and scalability. It is not as flexible as the grep-based search, so I must still check if it can do what DirectedGraphWebMapPlugin needs.

-- MichaelTempest - 12 Oct 2009

Re: DirectedGraphWebMapPlugin, just a quick note to say I made a little tip at IncludeTopicHierarchyIntoDotGraphs to show how to create topic hierarchies without it (Via DBCachePlugin).

Really appreciate the work so far, thanks!

-- PaulHarvey - 18 Oct 2009

I studied DirectedGraphPlugin and learnt a few things from it. DirectedGraphWebMapPlugin now caches the maps and graphs on disk in the plugin work_area. Subsequent views of pages are substantially quicker.

-- MichaelTempest - 15 Oct 2010