The True Story of the Grid Engine Dream

In Sun,  they called me Mr. Grid Engine. I was the product manager of Sun Grid Engine for a decade, until January 2010. This blog is a story of my journey. Where are  we today?

Defying many predictions, Grid Engine, formerly CODINE, is the Distributed Resource Management software most used in High Performance (HPC), and it refuses to yield to the cloud business model. One day HPC will be delivered as a cloud. Not yet

If you want a metaphor, the Volkswagen Beetle created the cult car that made successful the entire company. Similarly, Grid Engine can be viewed as a not perfect, but fascinating product. It can become the launching pad for something much bigger, with a much wider adoption.   

Grid Engine is, yes - not a typo error - a cult product, an enviable positioning, which defies logic. Apple, Java, Facebook are cult products. Grid Engine has the potential grow from this reputation well beyond it is today.

Commercial supported distributions are Univa Grid Engine and Oracle Grid Engine. Both claim to be the successor of Sun Grid Engine. The Open Source  distributions are Son of Grid Engine, the latest release being 8.0.0e on April 19, 2012, and Open Grid Scheduler .

Here is a quote from Inc. magazine latest article 8 Core beliefs of Extraordinary Bosses

·         Average bosses see business as a conflict between companies, departments and groups. They… demonize competitors as "enemies," and treat customers as "territory" to be conquered. 

·         Extraordinary bosses see business as a symbiosis where the most diverse firm is most likely to survive and thrive. They naturally create teams that adapt easily to new markets and can quickly form partnerships with other companies, customers ... and even competitors.

 

  Grid Engine  started with an extraordinary entrepreneur, Dr Wolfgang Gentzsch, who founded Genias  Software in 1991 later re-named Gridware in 1999.

 Wolfgang  says there is only one Grid Engine community, which forms an ecosystem, which he calls a symbiosis of diversity. It all originated in Genias  It implied a huge physical and mental effort, going through Sun acquisition of Gridware in 2000 and later - when Oracle took over Sun in 2011 the creation of the GE ecosystem.

After Wolfgang left Sun, - many fine people in Sun had to leave at that time - it was frustrating to see how our efforts to have two Sun Grid Engine products (one available by subscription and one available as free Open Source) failed because of management veto. On one hand we were under pressure to be profitable as a unit, on the other hand, our customers appeared to have no reason to pay even one cent for a subscription or license.

Oracle still has IP control of Grid Engine. Both Univa and Oracle decided to make no more contributions to the open source. While in Oracle open source policies are clear, Univa, a champion of open source for many years, has surprised the community. This has created an agitated  thread on Grid Engine discussion group.

Quoting from Inc. again:

Extraordinary bosses see change as an inevitable part of life. While they don't value change for its own sake, they know that success is only possible if employees and organization embrace new ideas and new ways of doing business.

The paradox is the companies who make real big money  do things as if not interested in money. Quora has a new thread, titled  What is Mark Zuckerberg's true attitude towards money? Here is a quote from highest ranking answer:

 Mark's main motivations were pretty clearly based around materially changing the world and building technology that was used by everyone on the planet.... My impression back then was that if he had to choose, he'd rather be the most important/influential person in the world rather than the richest. And I think that's visible in how he directed the company to focus on user growth and product impact rather than revenue or business considerations. Even today, while Facebook makes a ton of money, it could probably make magnitudes more if that were its primary goal.

The Zuckerbergs

What people usually say? 

   "Well I am not Zuckerberg." or 

   "I am not Steve Jobs." or

   "We can never make Grid Engine a business of this magnitude."

   "Yes we can!"

This my answer.

Oracle is part of the Grid Engine Ecosystem, They are one of the most powerful high tech companies in the world. In September 2008, Larry Ellison dismissed the concept of Cloud. In 2012, Cloud Computing is one of the main initiatives in Oracle. 

 

Univa web site points out that Oracle does not have a Grid Engine roadmap. This can change any moment, as Big Data becomes the buzz of the decade. Since Oracle takeover, Grid Engine is part of the Ops Center, a business unit whose culture is not in sync with Grid Engine. This may change

Rackspace announced at the OpenStack Design Summit and Conference that it’s ready to run its public cloud service on OpenStack, an open source software they own and made accessible to anyone.  55 companies worldwide, including IBM support implementations. Oracle may get some inspiration here for Grid Engine

The Grid Engine ecosystem has extremely gifted contributors. In addition to Univa team (Gary Tyreman, Bill Bryce, Fritz Ferstl), we have a superb team from open source including Chris Dagdigian from BioTeam (the creator of the legendary http://gridengine.info domain), Daniel Templeton, now with Cloudera. 

We have Rayson Ho, Ron Chen, Chansup Byun, Stephen Dennis,  John Tseng , Dave Love, Chi Chan, Reuti and many more from either Open Grid Scheduler/Grid Engine or the Son of Grid Engine.

Sweden's Gridcore Computing has joined Open Grid Scheduler, They pre-announced that an update version of Grid Engine 2011.11 will be released at the Gompute User Meeting on May 12 2012 in Gothenburg, Sweden. They add  Dev Subramanian, a top notch Grid Engine architect to the open source Grid Engine resources.

Wolfgang Gentzsch tops the list to restore the soul of Grid Engine.  He has made it before. He might do it again.

I believe in miracles. When Steve Jobs returned to Apple, they had a month or so to dismantle and liquidate the company. But...