January 2018 Hadoop Tracker

Last month’s update was obsolete before it published. This often happens because of multiple moving parts and my extended gestation period. I needed to correct entries for both AWS and Hortonworks. The new Tracker is correct as far as I know as of January 2, 2018. Enjoy.


December 2017 Tracker – Where’s Hadoop?

The leading 2017 story of Hadoop distributions is that nobody seems to want to be accused of being in the business of providing them. Some former champions are expanding their shiny new positioning: Cloudera is selling Enterprise Data Hubs and Analytic DBs; Hortonworks offers DataPlanes and Next-Gen Data Platforms; MapR touts the Converged Data Platform. In the cloud world, Amazon’s EMR is at least designed to “run and scale Apache Hadoop, Spark, HBase, Presto, Hive, and other Big Data Frameworks” while on Google’s Cloud Platform page the word Hadoop appears once inside the description of Cloud Dataproc.


Strata Standards Stories: Different Stores For Different Chores

Has HDFS joined MapReduce in the emerging “legacy Hadoop project” category, continuing the swap-out of components that formerly answered the question “what is Hadoop?” Stores for data were certainly a focus at Strata/Hadoop World in NY, O’Reilly’s well-run, well-attended, and always impactful fall event. The limitations of HDFS, including its append-only nature, have become inconvenient enough to push the community to “invent” something DBMS vendors like Oracle did decades ago: a bypass. After some pre-event leaks about its arrival, Cloudera chose its Strata keynote to announce Kudu, a new columnstore written in C++, bypassing HDFS entirely. Kudu will use an Apache license and will be submitted to the Apache process at some undetermined future time.


Hadoop Projects Supported By Only One Distribution

The Apache Software Foundation has succeeded admirably in becoming a place where new software ideas are developed: today over 350 projects are underway. The challenges for the Hadoop user are twofold: trying to decide which projects might be useful in big data-related cases, and determining which are supported by commercial distributors. In Now, What is Hadoop? And What’s Supported? I list 10 supported by only one: Atlas, Calcite, Crunch, Drill, Falcon, Kite, LLAMA, Lucene, Phoenix and Presto. Let’s look at them a little more.


Hadoop Summit Recap Part Two – SELECT FROM hdfs WHERE bigdatavendor USING SQL

Probably the most widespread, and commercially imminent, theme at the Summit was “SQL on Hadoop.” Since last year, many offerings have been touted, debated, and some have even shipped. In this post, I offer a brief look at where things stood at the Summit and how we got there. To net it out: offerings today range from the not-even-submitted to GA – if you’re interested, a bit of familiarity will help. Even more useful: patience.


Hadoop Distributions And Kids’ Soccer

The big players are moving in for a piece of the big data action.  IBM, EMC, and NetApp have stepped up their messaging, in part to prevent startup upstarts like Cloudera from cornering the Apache Hadoop distribution market. They are all elbowing one another to get closest to “pure Apache” while still “adding value.” Numerous other startups have emerged, with greater or lesser reliance on, and extensions or substitutions for, the core Apache distribution. Yahoo! has found a funding partner and spun its team out, forming a new firm called Hortonworks, whose claim to fame begins with an impressive roster responsible for most of the code in the core Hadoop projects. Think of the Doctor Seuss children’s book featuring that famous elephant, and you’ll understand the name.

While we’re talking about kids – ever watch young kids play soccer? Everyone surrounds the ball. It takes years to learn their position on the field and play accordingly. There are emerging alphas, a few stragglers on the sidelines hoping for a chance to play, community participants – and a clear need for governance. Tech markets can be like that, and with 1600 attendees packing late June’s Hadoop Summit event, all of those scenarios were playing out. Leaders, new entrants, and the big silents, like the absent Oracle and Microsoft.


At Oracle, Closed May be the New Open. Whither MySQL?

I hope I can be forgiven the cute headline. It speaks to a series of events that were heard in Oracle Open World messaging, where the word “open” appeared much less frequently than in years past. Oracle is fortifying its borders, opening new fronts in its market battles, and slowly closing itself off from some former partners and community relationships. It’s Fortress Oracle time. Its overall posture has hardened, and the implications for any but the largest MySQL customers are worrisome.

Many actions support this interpretation. The “fork you” message to Red Hat at OOW was an obvious indicator, tightening the OS play that accompanies the hardware ownership now rounding out Oracle’s full-stack story. Now, a few weeks later, Oracle’s move to drop low-end MySQL support, abandoning/conceding low-end customers to others, seems indicative both of Oracle’s willingness to move away from “open,” and to minimize investment in low-end customers. Mark Hurd is the new owner of support, and his reputation for cost-cutting should not be ignored in considering this; moreover, Windows is the majority platform for MySQL, and Oracle doesn’t want to invest there either. Read more of this post