Comments for Marc Sturlese

Comment on Solr and Hadoop integration against scalability problems by Marc Sturlese

Marc Sturlese — Fri, 02 Sep 2011 13:27:32 +0000

Well Katta+Hadoop is more about deploying indexs. Solr+Hadoop is for index generation. There are patches to integrate Solr index deployment with Katta (https://issues.apache.org/jira/browse/SOLR-1395) as well as there are patches to build Solr indexes using hadoop (https://issues.apache.org/jira/browse/SOLR-1301)
I’m really interested in the Lucene/Solr search implementation on HBase:
https://issues.apache.org/jira/browse/HBASE-3529

Comment on Using LZO Compression with Hadoop and Snow Leopard by Edgar Henigan

Edgar Henigan — Tue, 30 Aug 2011 19:48:44 +0000

I wish more people would write sites like this that are actually interesting to read. With all the fluff floating around on the internet, it is a great change of pace to read a site like yours instead.

Comment on Solr and Hadoop integration against scalability problems by sinoantony

sinoantony — Tue, 05 Jul 2011 08:55:35 +0000

Are there any comparison available between Katta+Hadoop and Solr+Hadoop for big data search solution? .. Thanks in advance for any advice..

Comment on Lucene FieldCache.StringIndex and multiValued fields by Marc Sturlese

Marc Sturlese — Thu, 06 Jan 2011 11:02:23 +0000

Hey Amit,
I’m not sure, but maybe since Solr 1.4 you can do something similar using the UnInvertedField class, used to facet on multiValued fields.
http://bit.ly/ig3aFC

Comment on Lucene FieldCache.StringIndex and multiValued fields by Amit Nithian

Amit Nithian — Thu, 06 Jan 2011 05:45:29 +0000

This is a good post. I have run into a similar issue before where getting all multiple values for a multi-valued field was difficult. I ended up extending the FieldCacheImpl of Solr 1.2 and had this working.. of course the API changed and so porting this to 1.5 is a bit harder but am working on it. I have some applications where I need access to all the values to make some decisions so this feature is necessary.

Comment on Index scalability using Pig by Marc Sturlese

Marc Sturlese — Wed, 21 Jul 2010 11:25:52 +0000

Hi Pablo,
I just tested the example in the post I pointed at and it worked great. But in the end I thought that trying to build a Lucene index using Pig wasn’t worth it. I mean, I thing it’s much more easy if you write your own MapReduce job to do that.
There are some examples out there.
I succeeded building Lucene index retrieving data from HBase using MapReduce jobs.

Comment on Index scalability using Pig by Pablo Mendes

Pablo Mendes — Mon, 19 Jul 2010 09:47:01 +0000

Hi Marc,
Did you give this a shot? I’m in a research setting working with tons of data. This means that I try many different indexing/weighting strategies that take a long time every time. I’m already using Pig to precompute some stats for me, so I thought: why not have it build the index already.
The main advantage for me is that pig gives me a quick way to manipulate my input data before I give it to the index, all of that over a cluster.
So I thought that somebody must have thought of this before me. Did you get anywhere with your idea, or did you drop it for some reason?
Cheers,
Pablo

Comment on Solr and Hadoop integration against scalability problems by Marc Sturlese

Marc Sturlese — Sun, 11 Jul 2010 21:00:51 +0000

Hi Yun,
As you mentioned, HDFS is not good place to build the Lucene/Solr index as it requires lots of operation.so. If you look for open source examples of MapReduce indexing you’ll see that when building an index, it’s built on the local file system and then uploaded to HDFS. When all the process is done, index/shards have to be downloaded to local file system again.
MapReduce and HDFS are very good to build huge index in a fast way. But if you need real time updates I think it’s better to execute the updates straight to the index/shards placed on the local file system.

Comment on Solr and Hadoop integration against scalability problems by yun chen

yun chen — Thu, 08 Jul 2010 07:23:08 +0000

Thanks for your post.
I just have a question:
as i know HDFS is not suitable for frequent operation.so how do you solve the update of your index for searching in situation of real time.
Expect your answer!

Comment on Lucene 2.4.1 available from today by Marc Sturlese

Marc Sturlese — Mon, 14 Jun 2010 22:09:40 +0000

Hey Rajani,
I supose you are working with Solr and not directly with Lucene. For a correct usage of Solr sharding I would recomend you keep the same schema for all the shards.
There isn’t a way to merge fields from different documents in a single doc on the Solr distributed search merging process.
I think you should consider reorganizing the data in your index. Shards are meant to scale data horizontally.