Solr Digest, December 2010

Just the other day, we posted the Lucene & Solr highlights in our Lucene & Solr: 2010 in Review post, and now it’s time to really conclude 2010 in Solr world with December Solr Digest. Although one might expect festive period to take its toll on the Solr development velocity, it wasn’t like that at all. Open source never sleeps.  Here are the most interesting highlights:

Interesting features in development

• In our July’s Digest we mentioned LanguageIdentifierUpdateProcessor feature which is being proposed under JIRA issue SOLR-1979. Some artifacts in the form of patches are starting to appear attached to that issue, so if you’re interested in this feature, take a look.
• Solr’s spatial capabilities are being further refined. With issue SOLR-2268, Solr will get “support for Point in Polygon searches”. This should enable features like “for a given point, return all documents which contain a polygon inside of which that point lays” and “for a given polygon, return all documents which have a point contained inside of that polygon“. Of course, negated versions of such feature will be supported. The work is in early stages, one patch is attached, but it can be used only as a general pointer about how this thing will be implemented, nothing else.
• Support for “ColognePhonetic” encoder was added to PhoneticFilterFactory. Since “ColognePhonetic” will be added to Commons Codec 1.4.1, the patch provided in SOLR-2276 will wait until that version gets released.

Interesting new features

• Solr is getting JOIN functionality – sort of. As part of SOLR-2272 , Solr got a working patch that provides SQL JOIN-like functionality. Of course, this is not exactly the JOIN you might know from SQL, but it is probably the closest thing to it which can be implemented in Solr. It is likely that this feature will be integrated into Lucene as well – it makes no sense to have it strictly in Solr. It is also likely that this feature will be expanded in the future; currently it has only one algorithm and supports many-to-many type of JOIN.
• As part of SolrCloud, new feature SolrCloud distributed indexing will be added to Solr some day. SOLR-2293 will be likely JIRA home for this feature. Before SolrCloud, anyone using distributed indexing had to create a custom logic which handled distribution of documents over various shards in the cluster. With SolrCloud, this will be transparent to the clients. Also, SolrCloud will include some out-of-the-box distribution algorithms, while, of course, plugging in custom algorithms will be easy to accomplish. However, don’t hold your breath waiting for this feature. At the moment, there exists only JIRA issue (and some guidelines in the Wiki) related to this feature.

Miscellaneous

• Solr’s Jetty is now upgraded to the latest 6.1.26 version. The change was committed to 3_x branch and trunk but, of course, it doesn’t mean you have to upgrade your Jetty too. It means just that stuff (various jars and few xmls) under /solr/example got upgraded. Details of this change can be found in SOLR-2265.
• Did you experience any problems with DataImportHandler and its multi-threaded option? If so, you are not the only one. More details about the nature of the problem can be found in SOLR-2186, while it appears that SOLR-2233 already contains a patch which might help in case of JDBC data source. That patch contains a  few other DataImportHandler multithreading fixes. Nothing related to this issue has been committed to trunk or 3_x branch yet.
• Many problems people encounter with Solr are related to OutOfMemory error. There were many interesting discussions on the ML in December, but we consider two of them related to that topic which you might find interesting OutOfMemory GC: GC overhead limit exceeded – Why isn’t WeakHashMap getting collected? and Memory use during merges (OOM)
• If you found “bf” parameter somewhat limited – it doesn’t accept complex nested expressions with lots of whitespaces – you might find patch from  SOLR-2014 useful. It appears that this is a common problem: another similar bug report (although this actually isn’t a bug) was also opened recently – SOLR-2267. However, consider that “bf” parameter will likely be deprecated in the future, since “bq” could be used to achieve everything “bf” can and more.
• In our November’s Digest, we mentioned SOLR-2154 and Solr’s problem with multi-valued spatial fields. What we missed was SOLR-2155, which actually provides a patch for such problems. Although it isn’t committed, the patch appears to be functional, so give it a try.

And that would be all for Solr in 2010.  We’ll be back with the new Solr Digest in a month.  Follow @sematext for other interesting search news.