First install the LZO libraries for Mac OS X from the tarball:

tar -xzf lzo-2.04.tar.gz
env CFLAGS=”-arch x86_64″ ./configure –build=x86_64-darwin –enable-shared –disable-asm –prefix=/path_to_lzo-2.04/
make; make install

The problem comes now. You have to compile the LZO compression for Hadoop:

env JAVA_HOME=/System/Library/Frameworks/JavaVM.framework/Versions/1.6/Home/ C_INCLUDE_PATH=/path_to_lzo-2.04/include LIBRARY_PATH=/path_to_lzo-2.04/lib CFLAGS=”-arch x86_64″ ant clean compile-native test tar

And this error happens:

This is because the soft link that points to the java headers on Snow Leopard is broken and the headers are nowhere. To get the proper ones you have to install the Mac OS X development tools which contains them. Then you just have to remove the broken soft link and create the correct one:

ln -s /System/Library/Frameworks/JavaVM.framework/Versions/1.6/Home/inclue -> /Developer/SDKs/MacOSX10.6.sdk/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Headers

Once compiled you’ll have the jar file and the compiled LZO libraries (which you’ll have to add to the JAVA_LIBRARY_PATH on the hadoop-env.sh path) in ~./kevinweil-hadoop-lzo-0e70051/build/native/Mac_OS_X-x86_64-64

Hadoop native libraries are needed too. The default compiled ones doesn’t work on Mac OS X so you will have to compile them too. If the headers are properly set, no problems should happen:

env JAVA_HOME=/System/Library/Frameworks/JavaVM.framework/Versions/1.6/Home CFLAGS=”-arch x86_64″ LDFLAGS=-L/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Libraries ANT_OPTS=-d64 ant clean compile-native

The compiled native libraries (you’ll have to add them to the JAVA_LIBRARY_PATH too) will be created in ~./hadoop-0.20.2+737/build/native/Mac_OS_X-x86_64-64/lib

Now just have to set properly the libraries paths and configure Hadoop to use the LZO compression as explained on the mentioned sites. Then, you’re done!

Just renaming the syslog-ng.conf-dist file to syslog-ng.conf will fix this issue but errors will keep coming:

Here we just have to add an @ at the beginning of the line zero just before the version number.
So, change version 3.0 to @version: 3.0
(Note that ‘:’ have to be added aswell)

Once this is done, syslog-ng won’t start yet. Now will give us a warning (but won’t start due to other errors that will mention later).

To make the warn disapear, as the message says, we have to change the line:
options { long_hostnames(off); sync(0); };
for
options { long_hostnames(off); flush_lines(0); };

We are not done yet, run the daemon again and:

To fix this, I created a new destination as the one by default seems to be used by the default system log in the OS.
Comment the line:
destination syslog { file(“/var/log/syslog”); };
and add:
destination d_syslog { file(“/var/log/syslog.log”); };

After changing the destination we have to do some more changes, otherwise the conf file will have inconsistencies and the execution will end up with more errors.
We need to apply the syslog filter to the new destination we have created. We do that changing:
log { source(src); filter(f_syslog); destination(syslog); };
for
log { source(src); filter(f_syslog); destination(d_syslog); };

What I did at this point was to also modify the syslog-ng filter, by default the f_syslog filter is:
filter f_syslog { not facility(authpriv, mail); };
I changed it to make it filter the messages from the localO, where I have the log4j configured to send the logs to:
filter f_syslog { facility(local0); };
Note that this step is not a must to have the syslog-ng working, it’s just a custom configuration.

Last error I got said that it was not possible to use the tty12 (wich is used in the default conf):
destination console_all { file(“/dev/tty12″); };
So I just changed it to ‘console’:
destination console_all { file(“/dev/console”); };

Now we can properly start the log system typing:
syslog-ng

FieldCache.StringIndex loads an int[] (order) and a String[] (lookup). The String[] contains all the terms on a field. The int[] array contains for each document an index to the lookup array.
It was curious to see that loading this structure for some multiValued fields on the index was working all rite. However, for some others was giving me back a RuntimeException I haven’t seen before, saying there were more terms than documents in the field ‘x’.

FieldCache is a structure meant to be used on single token (per document) fields. All trouble starts because in my tests I am not respecting that.
FieldCache can not hanlde more than one value per field. When loading FieldCache.StringIndex it does a test to ensure there’s no more than a term per field (it checks if the number of unique terms is greater than the number of docs). In my tests I am creating false negatives of these checks and seeing unexpected behavior.

So, let’s say I have an index with 100 docs and a multiValued field. The multiValued field has 2 values per document. If none of the field values is the same in the whole index I will get the exception. That’s due to the check done by the StringIndex.If I just have two different values and all the documents have these two values, no exception is thrown (false negative of the check). We can see that when the number of unique terms exceeds the number of docs the exception is thrown. That explains why when loading a FieldCache.StringIndex on a field with more than just one term can end up with a nasty exception or act as nothing is wrong.

There have been some fixes in the latter Lucene versions  (trunk, 3x, 3.0, 2.9 branches). The behavior now it that once the number of terms  > total documents, the array will not grow anymore so at least no RunTimeExceptions is going to happen.

More info can be found in the jira for the issue LUCENE-2142

I gave a talk on Thursday about our usage of Solr at Trovit. Covered an overview of our architecture, different of our 0ut 0f the box and custom features and some of the future lines we have in mind.

“Munching and Crunching: Lucene Index post-processing” was definitelly my favourite talk. Andrzej Bialecki covered topics I have never even thought about. Among other things there was a pretty complete explanation about index splitting, pruning and multi-tiered search.
People tends to think all data processing must be done during indexing time. Andrzej showed us that many good stuff can be done once the index is already built.

Yonik explained in an hour the main features that are coming with new Solr releases, “Solr 1.5 and  Beyond”. Extended DisMax query parser, quick introduction to SolrCloud, Spatial Search, Realtime Time and Field Collapsing where covered.

Grant Ingersoll spoke about Lucene / Solr relevance: “Practical Relevance: Tips and Tricks for Understanding and Improving Search Quality”.
It was very interesting to hear about the most commonly used techincques to do relevance testing:
A/B test, log analysis, empirical tests, asking or using related projects as Open Relevance or TREC.

Mark Miller talked about SolrCloud. It promises to make life so much easier to Solr distributed installations admins.

There were really good topics in the MeetUp as well. “How We Scaled Solr to 3+ Billion Documents” by Jason Rutherglen was the one I was expecting the most. I always like to hear about big Solr deployments and Hadoop usage related to Lucene and Solr indexing. This one I think is the biggest I know.

So, these days have been really useful. Many new ideas, many stuff to test.

Hello Lucene users,

On behalf of the Lucene development community I would like to announce the release of Lucene Java versions 3.0.1 and 2.9.2:

Both releases fix bugs in the previous versions:

- 2.9.2 is a bugfix release for the Lucene Java 2.x series, based on Java 1.4
- 3.0.1 has the same bug fix level but is for the Lucene Java 3.x series, based on Java 5.

New users of Lucene are advised to use version 3.0.1 for new developments, because it has a clean, type-safe API.

Important improvements in these releases include:

- An increased maximum number of unique terms in each index segment.
- Fixed experimental CustomScoreQuery to respect per-segment search. This introduced an API change!
- Important fixes to IndexWriter: a commit() thread-safety issue, lost document deletes in near real-time indexing.
- Bugfixes for Contrib’s Analyzers package.
- Restoration of some public methods that were lost during deprecation removal.
- The new Attribute-based TokenStream API now works correctly with different class loaders.

Both releases are fully compatible with the corresponding previous versions. We strongly recommend upgrading to 2.9.2 if you are using 2.9.1 or 2.9.0; and to 3.0.1 if you are using 3.0.0.

See core changes at
http://lucene.apache.org/java/3_0_1/changes/Changes.html
http://lucene.apache.org/java/2_9_2/changes/Changes.html

and contrib changes at
http://lucene.apache.org/java/3_0_1/changes/Contrib-Changes.html
http://lucene.apache.org/java/2_9_2/changes/Contrib-Changes.html

Binary and source distributions are available at
http://www.apache.org/dyn/closer.cgi/lucene/java/

Lucene artifacts are also available in the Maven2 repository at
http://repo1.maven.org/maven2/org/apache/lucene/

This week I have discovered via twitter a really interesting open source search project, ElasticSearch for the cloud. ElasticSearch has been createded by Shay Banon. It’s a RESTful search engine built on top of Lucene and very well prepared for high scalability. It includes shard merging, replication and much more features.

Lately I have been working a lot with search scalability and what I liked the most for the moment of ElasticSearch is that it allows 4 different types of distributed requests.

The most simple (Query and fetch) is just one request per relevant shard. Once all the requests are done, results are merged and… that’s it!
In this type of search, all fields of a document are returned to the merger for all the returned documents.

In another search type (Query then fetch, this one is not that simple), a first request is done across all shards. Here you don’t ask for the document content at the moment. Once the results are merged, you only need to ask for the whole document data of the most relevant documents, the ones you want to show.
If you have to search across lots of shards that’s definitely the way to go (the merger will just receive the fields of the important documents, wich means less data is sent across the network).

Both options present a typical problem in distributed search. The relevance is calculated relative to the shard, it’s not absolute across all of them.
To solve this, in ElasticSearch, both search options can be supplemented with an initial request. This one queries for the necessary term frequencies information to allow an “absolute relevance”.
This is not for free, you are paying with an extra trip (even it can be cached). It’s good if you can avoid that. A good way to do that is at indexing time, when you decide in wich shard a document must be added. Choosing it randomly will more or less ensure you that term frequencies won’t differ so much among shards.

Still have not had the chance to dig into the source but already have downloaded it from the git repository.
Anyone that want to share experiences with ElasticSearch is more than welcome.

After the quick talks, the following topics were discussed:

• What guarantees do I have with Cloud Computing?
• What legal issues are there with your data?
• Are standards important? If so, wich ones?
• What is the benefit for a company with only a few dozens of servers?
• Best platfrom to starting a cloud hosting company?
• Is cloud computing green? If so, what?

In the end people were divided in groups depending on in wich topic wanted to go deeper. I attended to “How to develope applications that are going to run in the cloud”. There I could have an interesting quick chat about application scalability and how to dump mysql databases to HDFS using the Cloudera’s tool Sqoop.

You just need to create a .jmx file wich will contain all the information needed to do the requests. Host name, port number, protocol, method, url path, url variables… You can actually tell JMeter to read the url variables from an external .dat file. It will allow you to give different values to the variables for each request.
The .jmx can be written manually but it’s much easier to create it via the JMeter’s GUI.

You will have to tell JMeter the number of threads that must be executing requests and the number of requests per thread. It allows you to leave the threads making requests indefinitely.
Once a test is launched you can see in real time the number of samples that have been executed and the Deviation, Throughput, Average and Median of the requests done by the threads (think of a thread as a user doing a request via browser).

This is just how to do a basic test plan but the application is really more complete than this and has much more interesting features.

Jmap allows you to create a dump of the java memory heap at any moment in the life of your running application. It will contain all the live objects and classes at that moment. To create the heap dump it’s as easy as:

jmap -dump:file=my_stack.bin 4365

Where my_stack.bin is the name of the file where you want the dump and 4365 is the pid of the java application process.

If you are running a servlet application under a java server and it ends with a:

java.lang.OutOfMemoryError: Java heap space

You can trigger a dump of the java heap at the OutOfMemory moment specifying these parameters to the server:

-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/home/sturlese/stack_test/

This will create a .hprof file (named with the pid’s process) containing the dump in the specified path.
HeapDumpPath param is not compulsory. If we don’t specify it the dump will be created in the folder where Tomcat launches the webapps.

Now we have the dump of the java heap. To analyze it we will use jhat. Once we launch jhat specifying the dump to analyze it will start an HTTP server (in the port 7000 by default) and will let you surf along all the classes and objects. You will be able to check how many instances of each class where alive in the moment the heap was created. To launch jhat:

jhat my_stack.bin

It’s easy to get an OutOfMememory exception when opening the java heap. The dump file can be very memory consuming if you app was in the moment it was taken. If you experience the problem you should give to the JVM as much memory as you can:

jhat -J-mx2000m my_stack.bin

Now is the moment to point your web browser to http://localhost:7000 and start analyzing the heap!