http://www.marcsturlese.com/2009/02/06/solr-and-hadoop-integration-against-scalability-problems/ Life, code and stuff Sat, 27 Feb 2010 18:59:32 +0100 http://wordpress.org/?v=2.8.4 hourly 1 http://www.marcsturlese.com/2009/02/06/solr-and-hadoop-integration-against-scalability-problems/comment-page-1/#comment-228 Marc Sturlese Sat, 27 Feb 2010 18:59:32 +0000 http://www.marcsturlese.com/?p=62#comment-228 Thankfully I have learned a bit more about Hadoop and Solr since I writed the post. What you say makes sense. However, I think there is no need to have all the set of index in all nodes. I imagine the scenario as: An index shard is built in each datanode and is stored in HDFS. This datanode also contains a Solr instance in the local file system. The shard has to be copied to the local file system, this way Solr can serach into it. Having a Solr instance in another server you could search across all the shards in the different datanodes using Solr distributed search. What do you think? Thankfully I have learned a bit more about Hadoop and Solr since I writed the post.
What you say makes sense. However, I think there is no need to have all the set of index in all nodes.
I imagine the scenario as:
An index shard is built in each datanode and is stored in HDFS. This datanode also contains a Solr instance in the local file system. The shard has to be copied to the local file system, this way Solr can serach into it.
Having a Solr instance in another server you could search across all the shards in the different datanodes using Solr distributed search.
What do you think?

]]> http://www.marcsturlese.com/2009/02/06/solr-and-hadoop-integration-against-scalability-problems/comment-page-1/#comment-210 Nick Adelman Fri, 26 Feb 2010 03:59:01 +0000 http://www.marcsturlese.com/?p=62#comment-210 I know this is an old post as of today, but I just wanted to add a comment regarding the tomcat instance. Reading the article, and having just done some POC work with Hadoop, I think that the tomcat instances are simply running on the same machines that are nodes in the Hadoop cluster. I don't know the details, but I think this would allow each Solr instance to leverage data locality in the cluster by working with the index(s) that exist on that particular node. If replication was configured such that all nodes in the hadoop cluster received a copy of every index, then all of the solr instances would be leveraging the same set of indexes. However, I'm sure it is much more complicated than that, especially since it is not typical to have data replicated to all nodes in a hadoop cluster of any significant size. I know this is an old post as of today, but I just wanted to add a comment regarding the tomcat instance. Reading the article, and having just done some POC work with Hadoop, I think that the tomcat instances are simply running on the same machines that are nodes in the Hadoop cluster. I don’t know the details, but I think this would allow each Solr instance to leverage data locality in the cluster by working with the index(s) that exist on that particular node. If replication was configured such that all nodes in the hadoop cluster received a copy of every index, then all of the solr instances would be leveraging the same set of indexes. However, I’m sure it is much more complicated than that, especially since it is not typical to have data replicated to all nodes in a hadoop cluster of any significant size.

]]>