Marc Sturlese » Hadoop http://www.marcsturlese.com Life, code and stuff Sun, 27 Jun 2010 14:45:42 +0000 en hourly 1 http://wordpress.org/?v=3.0 Index scalability using Pig http://www.marcsturlese.com/2009/03/02/index-scalability-using-pig/ http://www.marcsturlese.com/2009/03/02/index-scalability-using-pig/#comments Mon, 02 Mar 2009 22:37:41 +0000 Marc Sturlese http://www.marcsturlese.com/?p=70 Here is a really interesting example of how to build an inverted index using Pig. As I have seen in Hadoop, to create a Lucene index you must start from a text file and use MapReduce jobs to build it. Pig however, allows you to retrieve data not just from a text file but from SQL databases, HBase or other data sources.

After checking the example with detail, what comes now to my mind is if it would be possible to create a Lucene index using Pig and MapReduce jobs retrieving data from a distributed HBase data store system… I am wandering if there would be Lucene analyzers problems (or any other), for example.

I have read that Pig is not specially fast accessing to data. However, in indexing cases, probably this would be more than compensated with the MapReduce jobs.

How fast would it be? I still have lots of research and tests to do…

]]> http://www.marcsturlese.com/2009/03/02/index-scalability-using-pig/feed/ 2 Solr and Hadoop integration against scalability problems http://www.marcsturlese.com/2009/02/06/solr-and-hadoop-integration-against-scalability-problems/ http://www.marcsturlese.com/2009/02/06/solr-and-hadoop-integration-against-scalability-problems/#comments Thu, 05 Feb 2009 23:01:27 +0000 Marc Sturlese http://www.marcsturlese.com/?p=62 Recently I read an article explaining how Rackspace solved their huge log data deal with problem. They have implemented the best Hadoop and Solr integration I have seen until now, it really looks amazing.
I don’t know hadoop with detail but to run Solr instances from a Tomcat server stored in HDFS (Hadoop’s distributed file system) sounds like pretty good job!
All the process is well described in the article, I just want to mention the basic steps they followed:

  • Store huge amounts of log data in the HDFS.
  • MapReduce is used to create Lucene indexs from the stored data using Solr.
  • Once built, indexes are compressed in Hadoop nodes.
  • These index are merged using Solr webapps, deployed in Tomcat servers witch are stored in Hadoop nodes too (that is for me the most impressive part). These Solr instances allow fast search request aswell.

Probably this kind of arquitecture could be used to sort scalability problems in other fields not just log deal with. Search engines, for example. Maybe there the amount of data to deal with would be less but probably much more features would be needed.

Hadoop open source

]]> http://www.marcsturlese.com/2009/02/06/solr-and-hadoop-integration-against-scalability-problems/feed/ 4