Comments for Marc Sturlese http://www.marcsturlese.com Life, code and stuff Wed, 21 Jul 2010 11:25:52 +0000 hourly 1 http://wordpress.org/?v=3.0 Comment on Index scalability using Pig by Marc Sturlese http://www.marcsturlese.com/2009/03/02/index-scalability-using-pig/comment-page-1/#comment-373 Marc Sturlese Wed, 21 Jul 2010 11:25:52 +0000 http://www.marcsturlese.com/?p=70#comment-373 Hi Pablo, I just tested the example in the post I pointed at and it worked great. But in the end I thought that trying to build a Lucene index using Pig wasn't worth it. I mean, I thing it's much more easy if you write your own MapReduce job to do that. There are some examples out there. I succeeded building Lucene index retrieving data from HBase using MapReduce jobs. Hi Pablo,
I just tested the example in the post I pointed at and it worked great. But in the end I thought that trying to build a Lucene index using Pig wasn’t worth it. I mean, I thing it’s much more easy if you write your own MapReduce job to do that.
There are some examples out there.
I succeeded building Lucene index retrieving data from HBase using MapReduce jobs.

]]> Comment on Index scalability using Pig by Pablo Mendes http://www.marcsturlese.com/2009/03/02/index-scalability-using-pig/comment-page-1/#comment-372 Pablo Mendes Mon, 19 Jul 2010 09:47:01 +0000 http://www.marcsturlese.com/?p=70#comment-372 Hi Marc, Did you give this a shot? I'm in a research setting working with tons of data. This means that I try many different indexing/weighting strategies that take a long time every time. I'm already using Pig to precompute some stats for me, so I thought: why not have it build the index already. :) The main advantage for me is that pig gives me a quick way to manipulate my input data before I give it to the index, all of that over a cluster. So I thought that somebody must have thought of this before me. Did you get anywhere with your idea, or did you drop it for some reason? Cheers, Pablo Hi Marc,
Did you give this a shot? I’m in a research setting working with tons of data. This means that I try many different indexing/weighting strategies that take a long time every time. I’m already using Pig to precompute some stats for me, so I thought: why not have it build the index already. :)
The main advantage for me is that pig gives me a quick way to manipulate my input data before I give it to the index, all of that over a cluster.
So I thought that somebody must have thought of this before me. Did you get anywhere with your idea, or did you drop it for some reason?
Cheers,
Pablo

]]> Comment on Solr and Hadoop integration against scalability problems by Marc Sturlese http://www.marcsturlese.com/2009/02/06/solr-and-hadoop-integration-against-scalability-problems/comment-page-1/#comment-370 Marc Sturlese Sun, 11 Jul 2010 21:00:51 +0000 http://www.marcsturlese.com/?p=62#comment-370 Hi Yun, As you mentioned, HDFS is not good place to build the Lucene/Solr index as it requires lots of operation.so. If you look for open source examples of MapReduce indexing you'll see that when building an index, it's built on the local file system and then uploaded to HDFS. When all the process is done, index/shards have to be downloaded to local file system again. MapReduce and HDFS are very good to build huge index in a fast way. But if you need real time updates I think it's better to execute the updates straight to the index/shards placed on the local file system. Hi Yun,
As you mentioned, HDFS is not good place to build the Lucene/Solr index as it requires lots of operation.so. If you look for open source examples of MapReduce indexing you’ll see that when building an index, it’s built on the local file system and then uploaded to HDFS. When all the process is done, index/shards have to be downloaded to local file system again.
MapReduce and HDFS are very good to build huge index in a fast way. But if you need real time updates I think it’s better to execute the updates straight to the index/shards placed on the local file system.

]]> Comment on Solr and Hadoop integration against scalability problems by yun chen http://www.marcsturlese.com/2009/02/06/solr-and-hadoop-integration-against-scalability-problems/comment-page-1/#comment-368 yun chen Thu, 08 Jul 2010 07:23:08 +0000 http://www.marcsturlese.com/?p=62#comment-368 Thanks for your post. I just have a question: as i know HDFS is not suitable for frequent operation.so how do you solve the update of your index for searching in situation of real time. Expect your answer! Thanks for your post.
I just have a question:
as i know HDFS is not suitable for frequent operation.so how do you solve the update of your index for searching in situation of real time.
Expect your answer!

]]> Comment on Lucene 2.4.1 available from today by Marc Sturlese http://www.marcsturlese.com/2009/03/09/lucene-241-available-from-today/comment-page-1/#comment-365 Marc Sturlese Mon, 14 Jun 2010 22:09:40 +0000 http://www.marcsturlese.com/?p=76#comment-365 Hey Rajani, I supose you are working with Solr and not directly with Lucene. For a correct usage of Solr sharding I would recomend you keep the same schema for all the shards. There isn't a way to merge fields from different documents in a single doc on the Solr distributed search merging process. I think you should consider reorganizing the data in your index. Shards are meant to scale data horizontally. Hey Rajani,
I supose you are working with Solr and not directly with Lucene. For a correct usage of Solr sharding I would recomend you keep the same schema for all the shards.
There isn’t a way to merge fields from different documents in a single doc on the Solr distributed search merging process.
I think you should consider reorganizing the data in your index. Shards are meant to scale data horizontally.

]]> Comment on Lucene 2.4.1 available from today by Rajani http://www.marcsturlese.com/2009/03/09/lucene-241-available-from-today/comment-page-1/#comment-364 Rajani Mon, 14 Jun 2010 07:32:26 +0000 http://www.marcsturlese.com/?p=76#comment-364 Hello Marc.. I want know a way where i can merge attributes across cores into single DOC not just merging cores; it will not merge attributes in same single doc.. I split my data in different cores wid different schemas in hierarchy.. Now i want to perform search across all cores that i am doing using Shards concepts.. But i have a problem that is regarding Logical operator.. I cannot perform AND operation across cores.. The result will fail..(as data in cores different) Is there any way of dynamic merging of attributes..So that i can fix this logical operator problem.. Hello Marc..
I want know a way where i can merge attributes across cores into single DOC not just merging cores; it will not merge attributes in same single doc..

I split my data in different cores wid different schemas in hierarchy..
Now i want to perform search across all cores that i am doing using Shards concepts..
But i have a problem that is regarding Logical operator..
I cannot perform AND operation across cores..
The result will fail..(as data in cores different)
Is there any way of dynamic merging of attributes..So that i can fix this logical operator problem..

]]> Comment on Solr and Hadoop integration against scalability problems by Marc Sturlese http://www.marcsturlese.com/2009/02/06/solr-and-hadoop-integration-against-scalability-problems/comment-page-1/#comment-228 Marc Sturlese Sat, 27 Feb 2010 18:59:32 +0000 http://www.marcsturlese.com/?p=62#comment-228 Thankfully I have learned a bit more about Hadoop and Solr since I writed the post. What you say makes sense. However, I think there is no need to have all the set of index in all nodes. I imagine the scenario as: An index shard is built in each datanode and is stored in HDFS. This datanode also contains a Solr instance in the local file system. The shard has to be copied to the local file system, this way Solr can serach into it. Having a Solr instance in another server you could search across all the shards in the different datanodes using Solr distributed search. What do you think? Thankfully I have learned a bit more about Hadoop and Solr since I writed the post.
What you say makes sense. However, I think there is no need to have all the set of index in all nodes.
I imagine the scenario as:
An index shard is built in each datanode and is stored in HDFS. This datanode also contains a Solr instance in the local file system. The shard has to be copied to the local file system, this way Solr can serach into it.
Having a Solr instance in another server you could search across all the shards in the different datanodes using Solr distributed search.
What do you think?

]]> Comment on Solr and Hadoop integration against scalability problems by Nick Adelman http://www.marcsturlese.com/2009/02/06/solr-and-hadoop-integration-against-scalability-problems/comment-page-1/#comment-210 Nick Adelman Fri, 26 Feb 2010 03:59:01 +0000 http://www.marcsturlese.com/?p=62#comment-210 I know this is an old post as of today, but I just wanted to add a comment regarding the tomcat instance. Reading the article, and having just done some POC work with Hadoop, I think that the tomcat instances are simply running on the same machines that are nodes in the Hadoop cluster. I don't know the details, but I think this would allow each Solr instance to leverage data locality in the cluster by working with the index(s) that exist on that particular node. If replication was configured such that all nodes in the hadoop cluster received a copy of every index, then all of the solr instances would be leveraging the same set of indexes. However, I'm sure it is much more complicated than that, especially since it is not typical to have data replicated to all nodes in a hadoop cluster of any significant size. I know this is an old post as of today, but I just wanted to add a comment regarding the tomcat instance. Reading the article, and having just done some POC work with Hadoop, I think that the tomcat instances are simply running on the same machines that are nodes in the Hadoop cluster. I don’t know the details, but I think this would allow each Solr instance to leverage data locality in the cluster by working with the index(s) that exist on that particular node. If replication was configured such that all nodes in the hadoop cluster received a copy of every index, then all of the solr instances would be leveraging the same set of indexes. However, I’m sure it is much more complicated than that, especially since it is not typical to have data replicated to all nodes in a hadoop cluster of any significant size.

]]> Comment on JAD Java Decomplier by ivmai http://www.marcsturlese.com/2009/04/23/jad-java-decomplier/comment-page-1/#comment-30 ivmai Tue, 15 Dec 2009 13:51:25 +0000 http://www.marcsturlese.com/?p=104#comment-30 Unfortunately, Jad hasn't been upgraded for a long time (the latest release is jad1.5.8g) while the java compilers continue to evolve... To achieve better results (including for Java 1.5+ classes) in decompilation, process the classes with JadRetro (jadretro.sf.net) tool before decompiling them by Jad. Unfortunately, Jad hasn’t been upgraded for a long time (the latest release is jad1.5.8g) while the java compilers continue to evolve… To achieve better results (including for Java 1.5+ classes) in decompilation, process the classes with JadRetro (jadretro.sf.net) tool before decompiling them by Jad.

]]> Comment on Analyzing java heaps with jmap and jhat by andre parodi http://www.marcsturlese.com/2009/05/09/analyzing-java-heaps-with-jmap-and-jhat/comment-page-1/#comment-29 andre parodi Thu, 10 Dec 2009 11:46:16 +0000 http://www.marcsturlese.com/?p=115#comment-29 if you are getting OOM when using jhat try eclipse mat. if you are getting OOM when using jhat try eclipse mat.

]]>