The open source community is remarkable in many ways. For me, one of the most significant aspects of it is exactly that: it IS a community. It’s composed of people who communicate and share in deep and productive ways. One of the most interesting manifestations of that spirit I’ve run across is the Eigenbase project, an extensible platform being used by some very creative folks for the creation and continuing development of databases for data warehousing (the LucidDB DBMS) and stream processing (the SQLstream continuous query engine). I haven’t posted about either of those yet but will, and I’m watching their continuing evolution with great interest.
Organizationally, Eigenbase is a 501(c)(3) tax-exempt public charity. It has had contributions of various kinds – servers, IP, etc. from numerous companies and individuals since its founding in 2004. John Sichi, president and chairman, worked for Quadbase Systems and Broadbase Software, and was the original developer of Fennel and Farrago, two Eigenbase framework sub-projects. John contributes to a number of other open source projects inside and outside of Eigenbase, including JGraphT and LucidDB. Julian Hyde, treasurer, also sports a database pedigree from Oracle, Broadbase and Pentaho; he founded Mondrian and other projects. Julian is Chief Architect for SQLstream and Chief Architect, OLAP for Pentaho. He and John were together at Broadbase when John came up with some of the key Eigenbase constructs. Other key players include director Damian Black, whose career includes stints at Hewlett-Packard Research Laboratories in Europe where he worked on combining logic programming, object-oriented and relational database systems. Damian also took on business roles, which helped prep him for his role today as CEO of SQLstream.
The notion that “one size fits all” DBMSs are an artifact of an old model, and that it’s time to move on, is one that has gained a great deal of academic momentum, and the ADBMS vendors are aggressively pitching the idea that they have identified at least one use case that proves it. Mike Stonebraker has a few other ones in his thinking, and is commercially involved with two in addition to Vertica. But Eigenbase takes a rather different tack: it’s factored the data management tasks in such a way that different use cases can be supported by grafting different “features” onto it, as illustrated in this figure.
The premise is that as ideas for specialized use cases emerge, the developer can construct the extensions desired. In the case of LucidDB, it’s the column store bits (pun intended) that support the ADBMS case; in SQLstream, it’s SQL-based support for event processing. The platform includes a methodology for the assembly of these innovations, and a community of programmers work together to advance the model.
Platform capabilities include a source code repository, a release management process that uses Sourceforge, a wiki for documentation, a defect tracking system, mailing lists, standards enforcement and a great deal more. These things are not without precedent, but they form a coherent whole that attracts some very talented people. “The factoring is why Eigenbase is a vibrant community,” John told me recently over a breakfast in Half Moon Bay. “Lucid DB is beginning to get to a wider audience: people are kicking the tires,” Julian added. Lucid continues despite the closure of Lucidera, and development is now moving from core stuff to the integration side.
Quite a crew. There are others, whose brief bios on the Eigenbase site read like a who’s who, but these are the folks I’ve had the pleasure of meeting in several settings. Their commitment – to quality, and to the open source model – is extraordinary. If you’re interested in these technology categories, open source and/or collaborative processes you should get to know them. Follow the links and say hello. As they continue their work, don’t be surprised if we see some commercialization, evangelism as the dozens of implementations gain their own footholds, and the emergence of yet another contender.
Merv –
I agree that the Open Source community is remarkable, but is SQLstream a true open source product? Or is it a proprietary extension to the Eigenbase platform?
A stream processing paradigm is a natural fit to working with real-time (or near-time and eventually consistent) event systems that are en vogue.
Good question. Open is as open does, and different users/contributors have taken different approaches to licensing and commercialization. I have not dug in to that issue yet with them. But perhaps we’ll see a response here before long.
I completely agree with you about the importance of stream processing. IBM’s entry obviously made a lot of ears stand up (and the hairs on the back of some people’s necks)….
Eigenbase is an exceptional organization; the quality and volume of code built by members and released as open source is exceptional.
Eclipse of “Data Systems” should be the motto – a data management system for “everything” and “nothing specifically” all at the same time. It is very impressive the level which the companies have invested, to date, in adhering to standards, building exceptionally high quality code, and working together in an open, collaborative fashion.
Nicholas, thanks. I’ve added your blog to my BI section – enjoy your on-the-ground updates.
Merv – In case anyone else was curious (as I was): SQLStream is not OSS, according to Julian Hyde.
From Twitter:
“The #SQLStream product as a whole is not open source, but we develop the major ‘standard SQL’ portion as OS in Eigenbase.”
http://twitter.com/julianhyde/status/3713522075
Michael,
I reject your attempt to categorize this issue as black vs. white; and I sense your righteous indignation that a proprietary company should be associated with an open-source effort.
Merv did not say, or even imply, that SQLstream was open source, and unless you expect open source software to be developed by teenagers and trust-fund kids, one should expect there to be a commercial angle behind serious open source projects these days.
I view software development as like mountaineering: it only gets interesting when you are 90% of the way up. SQLstream has made very significant contributions to Eigenbase over more than four years, and continues to put core SQL infrastructure into Eigenbase. This is not altruism; it makes sense for our code to be tested, scrutinized and enhanced by as wide an audience as possible.
Eigenbase is a pragmatic way for SQLstream to collaborate with like-minded companies. It has been successful for SQLstream, but has also created an open source platform that other companies and projects can build innovative data-processing technologies upon.
If the participants in Eigenbase were non-commercial entities, it would be unremarkable. What is remarkable is that Eigenbase has provided a model, and a means, for self-interested commercial entities like SQLstream to collaborate for the greater good.
Julian Hyde
Chief Architect and Founder, SQLstream