iLuminate Aims To Enable Exploratory Analytics

iLuminate 4.0 has just been released, and its parent firm, Illuminate, is ramping a campaign to join the “new analytic DBMS” party. With a very different architecture based on storing and indexing values in an inverted-list like model, the “correlation database” or CDBMS has been winning some traction with a powerful value proposition: shorter “time to analytics” with minimal design time, lightning fast performance, and lower cost. It will catch the eye of its avowed target market: the “business middle class” defined by character, not size. Illuminate believes mid-sized companies and departments within larger ones are an under-penetrated market, because enterprise data warehouse (EDW) projects look too big and scary. DB and BI tech-savvy analysts and especially non-savvy ones alike often have tools and marts but no larger model. But like it or not, they can be successful without it, and they will pay for the privilege.

The new analytic DBMSs build on modern, innovative structures to eliminate the basic problem of RDBMS optimization for analytics: focus on one thing and the others suffer. RDBMSs were built and optimized for decades to perform well on transaction-related tasks and manage concurrency. As IT took on “decision support,” reporting, ad hoc analysis and today’s panoply of business intelligence paradigms, DBMS designers and DBAs struggled to fit their square, transaction-optimized pegs into the new round, analytics-demanding holes. In an effort to optimize all possible requirements, they now spend large percentages of their time on logical and physical design, partitioning strategies, indexing, re-indexing, redesigning, aggregate building, cube building, etc. Even the analytics-targeted products require substantial care and feeding from the start, and as they scale it becomes harder. Ask a DBA how much time she spends on maintenance and how much on adding new value.

iLuminate means to change that. The core idea is increasingly heard from BI tools vendors as well as analytic DBMS players: analysis by business people is search-oriented, iterative, and ad hoc. One question leads to another, which leads to another, and as inspiration strikes, analysts must be able to veer off in unexpected directions to pursue it. Tools are evolving to include search-based paradigms and advanced visualization to support this model; the underlying databases are too. iLuminate and its associated tool iCorrelate allow rapid preliminary analysis of data to find significant directions for relating and exploring correlations across dimensions. Data is stored in “value pools” which can dramatically speed retrieval for many common analytic approaches. Loading can proceed without extensive prior design, the time from data loading to usage is impressive, and compression rivals or beats columnar systems. If you want some examples, more product detail, and an engaging hands-on discussion, read Jason Brooks’ eWeek review . Not all questions are so simple, of course – complex correlated subqueries, recursive joins and other tough problems are all worth testing – and you should. Quite sizable tests can be run on hardware that costs less than $10k.

I’ve known Joe Foley, CTO of Illuminate, for a decade, and the firm’s predecessor, Synera, which first built the underlying technology, was a client in the 90s. That effort eventually shut down, and Joe has taken his new firm through a robust cycle of funding and infrastructure building that has served Illuminate well.  With the launch of release 4.0 and its 64-bit, multi-threaded architecture, iLuminate is ready to take on all comers in POCs – and has built its marketing and selling effort around challenging prospects to engage in one. And although Foley is one of the benchmark skeptics himself, he’ll tell anyone who asks about the tests iLuminate has run and why he beats all comers on them. Go ahead – make his day. Joe happily chronicles many of the challenges in his entertaining blog, Queries From Hell. Illuminate is also happy to talk about its dozens of customers (mostly near its base in Barcelona, Spain, but beginning to spring up in North America as well.) In the next few quarters, I hope that we’ll see some installations that begin to stretch the product’s scalability, which still needs to be demonstrated in the field.

Illuminate is taking an intriguing approach in its route to market, and IT Market Strategy believes it will take another substantial step forward in revving its engine in the second half of 2009. I’ll talk about the business model and developments in another post.