http://www.marcsturlese.com/2009/03/02/index-scalability-using-pig/ Life, code and stuff Wed, 21 Jul 2010 11:25:52 +0000 hourly 1 http://wordpress.org/?v=3.0 http://www.marcsturlese.com/2009/03/02/index-scalability-using-pig/comment-page-1/#comment-373 Marc Sturlese Wed, 21 Jul 2010 11:25:52 +0000 http://www.marcsturlese.com/?p=70#comment-373 Hi Pablo, I just tested the example in the post I pointed at and it worked great. But in the end I thought that trying to build a Lucene index using Pig wasn't worth it. I mean, I thing it's much more easy if you write your own MapReduce job to do that. There are some examples out there. I succeeded building Lucene index retrieving data from HBase using MapReduce jobs. Hi Pablo,
I just tested the example in the post I pointed at and it worked great. But in the end I thought that trying to build a Lucene index using Pig wasn’t worth it. I mean, I thing it’s much more easy if you write your own MapReduce job to do that.
There are some examples out there.
I succeeded building Lucene index retrieving data from HBase using MapReduce jobs.

]]> http://www.marcsturlese.com/2009/03/02/index-scalability-using-pig/comment-page-1/#comment-372 Pablo Mendes Mon, 19 Jul 2010 09:47:01 +0000 http://www.marcsturlese.com/?p=70#comment-372 Hi Marc, Did you give this a shot? I'm in a research setting working with tons of data. This means that I try many different indexing/weighting strategies that take a long time every time. I'm already using Pig to precompute some stats for me, so I thought: why not have it build the index already. :) The main advantage for me is that pig gives me a quick way to manipulate my input data before I give it to the index, all of that over a cluster. So I thought that somebody must have thought of this before me. Did you get anywhere with your idea, or did you drop it for some reason? Cheers, Pablo Hi Marc,
Did you give this a shot? I’m in a research setting working with tons of data. This means that I try many different indexing/weighting strategies that take a long time every time. I’m already using Pig to precompute some stats for me, so I thought: why not have it build the index already.
The main advantage for me is that pig gives me a quick way to manipulate my input data before I give it to the index, all of that over a cluster.
So I thought that somebody must have thought of this before me. Did you get anywhere with your idea, or did you drop it for some reason?
Cheers,
Pablo

]]>