Labels

Breakthrough in HPC. Microsoft acquires GreenButton

Here my two tweets on this subject

I interacted with Scott as Sun Grid Engine product manager in Sun Micro. He and his team added features and modified the code  "como Pedro por su casa"  meaning  as if they owned the place, making significant contribution to Grid Engine. We were supposed to support them - and we did - but we ended learning more from him than any other user I dealt with.
Scott Houston fortune cookie
Scott Houston does not come out of the blue. It was his extraordinary engineering talent to make movies like the Lord of the Rings possible. He was the soul of  NZSC (New Zealand Supercomputer Center) while CTO at Weta Digital.  When asked whether he is a "power hungry executive", he replied:  "A power processing hungry."

Weta's founders are a bunch of New Zealand filmmakers including Peter Jackson, Richard Taylor and Jamie Selkirk. It later split into two specialized halves - Weta Digital (digital effects) and Weta Workshop (physical effects).

Here is a description of the work culminating in the Avatar movie production
The lead visual effects company was Weta Digital in Wellington, New Zealand, at one point employing 900 people to work on the film. Because of the huge amount of data which needed to be stored, cataloged and available for everybody involved, even on the other side of the world, a new cloud computing and Digital Asset Management (DAM) system named Gaia was created by Microsoft especially for Avatar, which allowed the crews to keep track of and coordinate all stages in the digital processing.
To render Avatar, Weta used a 10,000 sq ft (930 m2) server farm making use of 4,000 Hewlett-Packard servers with 35,000 processor cores running Ubuntu Linux and the Grid Engine cluster manager.
[Note; They used the open source version of Grid Grid Engine]
The render farm occupies the 193rd to 197th spots in the TOP500 list of the world's most powerful supercomputers. A new texturing and paint software system, called Mari, was developed by The Foundry in cooperation with Weta. Creating the Na'vi characters and the virtual world of Pandora required over a petabyte of digital storage,and each minute of the final footage for Avatar occupies 17.28 gigabytes of storage. To help finish preparing the special effects sequences on time, a number of other companies were brought on board, including Industrial Light & Magic, which worked alongside Weta Digital to create the battle sequences. ILM was responsible for the visual effects for many of the film's specialized vehicles and devised a new way to make CGI explosions.
When we read the above text, as many other similar HPC implementations, the first thought to my mind is COMPLEXITY. The second word that comes to my mind is COST. This not something an indie producer and amateur movie-maker can afford.

Avatar grossed - stay put - nearly three billion dollars worldwide so far.

What GreenButton did, was to remove this COMPLEXITY and COST. As we read in Microsoft blog:
GreenButton is a leading provider of integrated on-demand solutions that allow customers to manage compute-intensive workloads in the cloud. These solutions are known in the industry for their ease of use. Using GreenButton’s solutions, applications can be cloud-enabled quickly without recoding existing software – and without a PhD in computer science.
 With this acquisition, we are looking forward to democratizing the use Big Compute through the power of the cloud, so that organizations from all walks of life can use their data to transform their business and the world. 
Among the innovations of Green Button, here is a smashing one:
"If you need the results faster, slide it up the scale, and you get more computing cores to do the job," added Houston. With Cloud Fabric, IT organizations can make the economics of cloud computing work for their internal timetables. "If you don't need the job until tomorrow or a couple of days from now, slide it down," explained Houston.
 Microsoft acquisition of GreenButton proves that
  • HPC future is in the cloud
  • The success of HPC is making it easy to use, not adding more features only PhDs can use
  • The adoption of HPC for-all  by the legendary Microsoft is an historical benchmark
  • It validates the award winning  Uber Cloud Project from the UberCloud HPC Experiment
  • Clouds and Big Data are intertwined and inseparable
  • It brings New Zealand respect and admiration. Kudos to Scott Houston and team!
  • It validates the choice made by Microsoft in naming Satya Nadella as its CEO

Labels:

Comments

Post a Comment

Popular Posts