Comments on: Solr and Hadoop integration against scalability problems

By: Marc Sturlese

Marc Sturlese — Fri, 02 Sep 2011 13:27:32 +0000

Well Katta+Hadoop is more about deploying indexs. Solr+Hadoop is for index generation. There are patches to integrate Solr index deployment with Katta (https://issues.apache.org/jira/browse/SOLR-1395) as well as there are patches to build Solr indexes using hadoop (https://issues.apache.org/jira/browse/SOLR-1301)
I’m really interested in the Lucene/Solr search implementation on HBase:
https://issues.apache.org/jira/browse/HBASE-3529

By: sinoantony

sinoantony — Tue, 05 Jul 2011 08:55:35 +0000

Are there any comparison available between Katta+Hadoop and Solr+Hadoop for big data search solution? .. Thanks in advance for any advice..

By: Marc Sturlese

Marc Sturlese — Sun, 11 Jul 2010 21:00:51 +0000

Hi Yun,
As you mentioned, HDFS is not good place to build the Lucene/Solr index as it requires lots of operation.so. If you look for open source examples of MapReduce indexing you’ll see that when building an index, it’s built on the local file system and then uploaded to HDFS. When all the process is done, index/shards have to be downloaded to local file system again.
MapReduce and HDFS are very good to build huge index in a fast way. But if you need real time updates I think it’s better to execute the updates straight to the index/shards placed on the local file system.

By: yun chen

yun chen — Thu, 08 Jul 2010 07:23:08 +0000

Thanks for your post.
I just have a question:
as i know HDFS is not suitable for frequent operation.so how do you solve the update of your index for searching in situation of real time.
Expect your answer!

By: Marc Sturlese

Marc Sturlese — Sat, 27 Feb 2010 18:59:32 +0000

Thankfully I have learned a bit more about Hadoop and Solr since I writed the post.
What you say makes sense. However, I think there is no need to have all the set of index in all nodes.
I imagine the scenario as:
An index shard is built in each datanode and is stored in HDFS. This datanode also contains a Solr instance in the local file system. The shard has to be copied to the local file system, this way Solr can serach into it.
Having a Solr instance in another server you could search across all the shards in the different datanodes using Solr distributed search.
What do you think?

By: Nick Adelman

Nick Adelman — Fri, 26 Feb 2010 03:59:01 +0000

I know this is an old post as of today, but I just wanted to add a comment regarding the tomcat instance. Reading the article, and having just done some POC work with Hadoop, I think that the tomcat instances are simply running on the same machines that are nodes in the Hadoop cluster. I don’t know the details, but I think this would allow each Solr instance to leverage data locality in the cluster by working with the index(s) that exist on that particular node. If replication was configured such that all nodes in the hadoop cluster received a copy of every index, then all of the solr instances would be leveraging the same set of indexes. However, I’m sure it is much more complicated than that, especially since it is not typical to have data replicated to all nodes in a hadoop cluster of any significant size.