I gave a talk on Thursday about our usage of Solr at Trovit. Covered an overview of our architecture, different of our 0ut 0f the box and custom features and some of the future lines we have in mind.

“Munching and Crunching: Lucene Index post-processing” was definitelly my favourite talk. Andrzej Bialecki covered topics I have never even thought about. Among other things there was a pretty complete explanation about index splitting, pruning and multi-tiered search.
People tends to think all data processing must be done during indexing time. Andrzej showed us that many good stuff can be done once the index is already built.

Yonik explained in an hour the main features that are coming with new Solr releases, “Solr 1.5 and  Beyond”. Extended DisMax query parser, quick introduction to SolrCloud, Spatial Search, Realtime Time and Field Collapsing where covered.

Grant Ingersoll spoke about Lucene / Solr relevance: “Practical Relevance: Tips and Tricks for Understanding and Improving Search Quality”.
It was very interesting to hear about the most commonly used techincques to do relevance testing:
A/B test, log analysis, empirical tests, asking or using related projects as Open Relevance or TREC.

Mark Miller talked about SolrCloud. It promises to make life so much easier to Solr distributed installations admins.

There were really good topics in the MeetUp as well. “How We Scaled Solr to 3+ Billion Documents” by Jason Rutherglen was the one I was expecting the most. I always like to hear about big Solr deployments and Hadoop usage related to Lucene and Solr indexing. This one I think is the biggest I know.

So, these days have been really useful. Many new ideas, many stuff to test.

After the quick talks, the following topics were discussed:

• What guarantees do I have with Cloud Computing?
• What legal issues are there with your data?
• Are standards important? If so, wich ones?
• What is the benefit for a company with only a few dozens of servers?
• Best platfrom to starting a cloud hosting company?
• Is cloud computing green? If so, what?

In the end people were divided in groups depending on in wich topic wanted to go deeper. I attended to “How to develope applications that are going to run in the cloud”. There I could have an interesting quick chat about application scalability and how to dump mysql databases to HDFS using the Cloudera’s tool Sqoop.

Was good to share use cases and experiences in person with people who I had just spoken with in forums.
I spend the first two days in the hackathon doing some research and test of different ASF projects. Put special interest in Pig.

There were really interesting chats. I found specially great Mahout project. I had discovered it in ApacheCon 2008 in New Orleans, I almost just heard about it in there but paid more atention this time and looks full of possibilities. It is used for machine learning and runs under Hadoop.
Was also good to get some info about Servlet 3.0 and learn about servlets doFilter function and some other stuff.
HBase is another project I was interested in. Looks good to be used as a “data warehouse” but seems really difficult (at least at first impression) to deal with the stored data.

Meetups were so good too. There was a presentation about the new Lucene contrib TrieRangeQuery. It is still not available in the official release but you can use it graving a nightly build. In the next few days I will try to write with more detail about this and other presented projects.

After checking the example with detail, what comes now to my mind is if it would be possible to create a Lucene index using Pig and MapReduce jobs retrieving data from a distributed HBase data store system… I am wandering if there would be Lucene analyzers problems (or any other), for example.

I have read that Pig is not specially fast accessing to data. However, in indexing cases, probably this would be more than compensated with the MapReduce jobs.

How fast would it be? I still have lots of research and tests to do…

• Store huge amounts of log data in the HDFS.
• MapReduce is used to create Lucene indexs from the stored data using Solr.
• Once built, indexes are compressed in Hadoop nodes.
• These index are merged using Solr webapps, deployed in Tomcat servers witch are stored in Hadoop nodes too (that is for me the most impressive part). These Solr instances allow fast search request aswell.

Probably this kind of arquitecture could be used to sort scalability problems in other fields not just log deal with. Search engines, for example. Maybe there the amount of data to deal with would be less but probably much more features would be needed.