Subject: Practical Advice on Clustering


I'm looking into adding document clustering capabilities to Solr,  
using Mahout [1][2].  I already have search-results clustering, thanks  
to Carrot2.  What I'm looking for is practical advice on deploying a  
system that is going to cluster potentially large corpora (but not  
huge, and let's assume one machine for now, but it shouldn't matter)

Here are some thoughts I have:

In Solr, I expect to send a request to go off and build the clusters  
for some non-trivial set of documents in the index.  The actual  
building needs to happen in a background thread, so as to not hold up  
the caller.  My thinking is the request will come in and spawn off a  
job that goes and calculates a similarity matrix for all the documents  
in the set (need to store the term vectors in Lucene) and then goes  
and runs the clustering job (user configurable, based on the  
implementations we have: k-means, mean-shift, fuzzy, whatever) and  
stores the results into Solr's data directory somehow (so that it can  
be replicated, but not a big concern of mine at the moment)

Then, at any time, the application can ask Solr for the clusters  
(whatever that means) and it will return them (docids, fields,  
whatever the app asks for).  If the background task isn't done yet,  
the results set will be empty, or it will return a percentage  
completion or something useful.

Obviously, my first step is to get it working, but...

Is it practical to return a partially done set of results?  i.e. the  
best clusters so far, with perhaps a percentage to completion value or  
perhaps a list of the comparisons that haven't been done yet?

What if something happens?  How can I make Mahout fault-tolerant, such  
that, conceivably I could pick up the job again from where it went  
down, or at least be able to get the clusters so far.  How do people  
approach this to date (w/ or w/o Mahout)  What needs to be done in  
Mahout to make this possible?  I suspect Hadoop has some support for it.

Anything else I don't know?  Does what I'm thinking about make sense?

Thanks for any insight,
Grant

[1] http://wiki.apache.org/solr/ClusteringComponent
[2] https://issues.apache.org/jira/browse/SOLR-769