FieldCache.StringIndex loads an int[] (order) and a String[] (lookup). The String[] contains all the terms on a field. The int[] array contains for each document an index to the lookup array.
It was curious to see that loading this structure for some multiValued fields on the index was working all rite. However, for some others was giving me back a RuntimeException I haven’t seen before, saying there were more terms than documents in the field ‘x’.

FieldCache is a structure meant to be used on single token (per document) fields. All trouble starts because in my tests I am not respecting that.
FieldCache can not hanlde more than one value per field. When loading FieldCache.StringIndex it does a test to ensure there’s no more than a term per field (it checks if the number of unique terms is greater than the number of docs). In my tests I am creating false negatives of these checks and seeing unexpected behavior.

So, let’s say I have an index with 100 docs and a multiValued field. The multiValued field has 2 values per document. If none of the field values is the same in the whole index I will get the exception. That’s due to the check done by the StringIndex.If I just have two different values and all the documents have these two values, no exception is thrown (false negative of the check). We can see that when the number of unique terms exceeds the number of docs the exception is thrown. That explains why when loading a FieldCache.StringIndex on a field with more than just one term can end up with a nasty exception or act as nothing is wrong.

There have been some fixes in the latter Lucene versions  (trunk, 3x, 3.0, 2.9 branches). The behavior now it that once the number of terms  > total documents, the array will not grow anymore so at least no RunTimeExceptions is going to happen.

More info can be found in the jira for the issue LUCENE-2142

I gave a talk on Thursday about our usage of Solr at Trovit. Covered an overview of our architecture, different of our 0ut 0f the box and custom features and some of the future lines we have in mind.

“Munching and Crunching: Lucene Index post-processing” was definitelly my favourite talk. Andrzej Bialecki covered topics I have never even thought about. Among other things there was a pretty complete explanation about index splitting, pruning and multi-tiered search.
People tends to think all data processing must be done during indexing time. Andrzej showed us that many good stuff can be done once the index is already built.

Yonik explained in an hour the main features that are coming with new Solr releases, “Solr 1.5 and  Beyond”. Extended DisMax query parser, quick introduction to SolrCloud, Spatial Search, Realtime Time and Field Collapsing where covered.

Grant Ingersoll spoke about Lucene / Solr relevance: “Practical Relevance: Tips and Tricks for Understanding and Improving Search Quality”.
It was very interesting to hear about the most commonly used techincques to do relevance testing:
A/B test, log analysis, empirical tests, asking or using related projects as Open Relevance or TREC.

Mark Miller talked about SolrCloud. It promises to make life so much easier to Solr distributed installations admins.

There were really good topics in the MeetUp as well. “How We Scaled Solr to 3+ Billion Documents” by Jason Rutherglen was the one I was expecting the most. I always like to hear about big Solr deployments and Hadoop usage related to Lucene and Solr indexing. This one I think is the biggest I know.

So, these days have been really useful. Many new ideas, many stuff to test.

Hello Lucene users,

On behalf of the Lucene development community I would like to announce the release of Lucene Java versions 3.0.1 and 2.9.2:

Both releases fix bugs in the previous versions:

- 2.9.2 is a bugfix release for the Lucene Java 2.x series, based on Java 1.4
- 3.0.1 has the same bug fix level but is for the Lucene Java 3.x series, based on Java 5.

New users of Lucene are advised to use version 3.0.1 for new developments, because it has a clean, type-safe API.

Important improvements in these releases include:

- An increased maximum number of unique terms in each index segment.
- Fixed experimental CustomScoreQuery to respect per-segment search. This introduced an API change!
- Important fixes to IndexWriter: a commit() thread-safety issue, lost document deletes in near real-time indexing.
- Bugfixes for Contrib’s Analyzers package.
- Restoration of some public methods that were lost during deprecation removal.
- The new Attribute-based TokenStream API now works correctly with different class loaders.

Both releases are fully compatible with the corresponding previous versions. We strongly recommend upgrading to 2.9.2 if you are using 2.9.1 or 2.9.0; and to 3.0.1 if you are using 3.0.0.

See core changes at
http://lucene.apache.org/java/3_0_1/changes/Changes.html
http://lucene.apache.org/java/2_9_2/changes/Changes.html

and contrib changes at
http://lucene.apache.org/java/3_0_1/changes/Contrib-Changes.html
http://lucene.apache.org/java/2_9_2/changes/Contrib-Changes.html

Binary and source distributions are available at
http://www.apache.org/dyn/closer.cgi/lucene/java/

Lucene artifacts are also available in the Maven2 repository at
http://repo1.maven.org/maven2/org/apache/lucene/

This week I have discovered via twitter a really interesting open source search project, ElasticSearch for the cloud. ElasticSearch has been createded by Shay Banon. It’s a RESTful search engine built on top of Lucene and very well prepared for high scalability. It includes shard merging, replication and much more features.

Lately I have been working a lot with search scalability and what I liked the most for the moment of ElasticSearch is that it allows 4 different types of distributed requests.

The most simple (Query and fetch) is just one request per relevant shard. Once all the requests are done, results are merged and… that’s it!
In this type of search, all fields of a document are returned to the merger for all the returned documents.

In another search type (Query then fetch, this one is not that simple), a first request is done across all shards. Here you don’t ask for the document content at the moment. Once the results are merged, you only need to ask for the whole document data of the most relevant documents, the ones you want to show.
If you have to search across lots of shards that’s definitely the way to go (the merger will just receive the fields of the important documents, wich means less data is sent across the network).

Both options present a typical problem in distributed search. The relevance is calculated relative to the shard, it’s not absolute across all of them.
To solve this, in ElasticSearch, both search options can be supplemented with an initial request. This one queries for the necessary term frequencies information to allow an “absolute relevance”.
This is not for free, you are paying with an extra trip (even it can be cached). It’s good if you can avoid that. A good way to do that is at indexing time, when you decide in wich shard a document must be added. Choosing it randomly will more or less ensure you that term frequencies won’t differ so much among shards.

Still have not had the chance to dig into the source but already have downloaded it from the git repository.
Anyone that want to share experiences with ElasticSearch is more than welcome.

TrieRangeQuery mainly sort out some RangeQuery problems:

• Tipical RangeQuery can end in TooManyClausesException if our ranges are so large.

• Tipical RangeQuery or even ConstantScoreRangeQuery are slow if have to classify using large ranges or the index is huge.

To explain it in an easy way, what TrieRangeQuery do is to search the data values skipping the less relevant “digits” in function of a precision parameter.

Let’s say for example we need to classify thousands of numbers of 6 figures. This could be a slow process using ConstantScoreRangeQuery in a huge index, not with TrieRangeQuery. Ranges will be divided recurively in function of  a precision parámeter (set at index time). Numbers from the middle of the range will be classified using the minimum precision value while numbers from extrems will use a higher precision. This will make the query run extremely much faster.

Depending on the level of presicionStep parameter given at index time we will be able to search with more or less precision.  The more precision marging we choose the more the lucene document will occuppy. It is due to we will have to index the field more times with the different precisions.

We need to index data in a special way to be able to search it using Lucene TrieRangeQuery. We must index our fields using TrieUtils. We can index numbers directly. It supports java signed int, long, float, double. There’s no loss of precision for doubles or floats. There’s no round for their creation, instead a long/int representation is used for cents.
Indexing numbers with TrieUtils will make us forget about maual padding.
We can index Dates aswell (from java timestamps data type).

As seen, Lucene TrieRangeQuery is totally a step forward for Lucene queries scalability.

Was good to share use cases and experiences in person with people who I had just spoken with in forums.
I spend the first two days in the hackathon doing some research and test of different ASF projects. Put special interest in Pig.

There were really interesting chats. I found specially great Mahout project. I had discovered it in ApacheCon 2008 in New Orleans, I almost just heard about it in there but paid more atention this time and looks full of possibilities. It is used for machine learning and runs under Hadoop.
Was also good to get some info about Servlet 3.0 and learn about servlets doFilter function and some other stuff.
HBase is another project I was interested in. Looks good to be used as a “data warehouse” but seems really difficult (at least at first impression) to deal with the stored data.

Meetups were so good too. There was a presentation about the new Lucene contrib TrieRangeQuery. It is still not available in the official release but you can use it graving a nightly build. In the next few days I will try to write with more detail about this and other presented projects.

We will be able to see more new features in the Lucene 2.9 release (available in developers version).

Here I mention all the improvements of Lucene 2.4.1, wich I read from the official lucene’s site:

• Fixed silent data-loss case whereby binary fields are truncated to 0 bytes during merging if the segments being merged are non-congruent (same field name maps to different field numbers).
• Don’t throw incorrect IllegalStateException from IndexWriter.close() if you’ve hit an OOM when autoCommit is true.
• If IndexReader.flush() is called twice when there were pending deletions, it could lead to later false AssertionError during IndexReader.open.
• Fix false AlreadyClosedException from IndexReader.open (masking an actual IOException) that takes String or File path.
• Multiple-valued NOT_ANALYZED fields can double-count token offsets.
• Ensure IndexReader.reopen() does not result in incorrectly closing the shared FSDirectory. This bug would only happen if you use IndexReader.open with a File or String argument.
• Fix possible overflow bugs during binary searches.
• Fix CachingWrapperFilter to not throw exception if both bits() and getDocIdSet() methods are called.
• Fix int overflow bug during segment merging.
  
• Fix int overflow bug when flushing segment.
• Fix deadlock in IndexWriter.addIndexes(IndexReader[]).
• NearSpansOrdered returns payloads from first possible match rather than the correct, shortest match; Payloads could be returned even if the max slop was exceeded; The wrong payload could be returned in certain situations.
• Add Analyzer.close() to free internal ThreadLocal resources.
• Fix IndexWriter.addIndexes(IndexReader[]) to properly rollback IndexWriter’s internal state on hitting an exception.

After checking the example with detail, what comes now to my mind is if it would be possible to create a Lucene index using Pig and MapReduce jobs retrieving data from a distributed HBase data store system… I am wandering if there would be Lucene analyzers problems (or any other), for example.

I have read that Pig is not specially fast accessing to data. However, in indexing cases, probably this would be more than compensated with the MapReduce jobs.

How fast would it be? I still have lots of research and tests to do…

• Store huge amounts of log data in the HDFS.
• MapReduce is used to create Lucene indexs from the stored data using Solr.
• Once built, indexes are compressed in Hadoop nodes.
• These index are merged using Solr webapps, deployed in Tomcat servers witch are stored in Hadoop nodes too (that is for me the most impressive part). These Solr instances allow fast search request aswell.

Probably this kind of arquitecture could be used to sort scalability problems in other fields not just log deal with. Search engines, for example. Maybe there the amount of data to deal with would be less but probably much more features would be needed.

First of all I downloaded the Luke’s source code . Opened the lib folder and replaced the 2.4 for the ones of the lucene developers release. Once done I compiled the surce with ant and… that’s it, I could check my indexed data using my compiled version Luke!