Wednesday, July 31, 2013

Why Bosco went to XSEDE'13

Somewhere on the web, there is one  questionnaire Find your computacy level . It asks you questions like how to make bold letters in Word and what is a file.

However there is no computacy test for XSEDE, which mean Extreme Science and Engineering Discovery Environment. It may come to a surprise that most scientists in the world do not know how to operate supercomputers. 
Edward O Wilson
Edward  Wilson said in WSJ a few month ago  that a Great Scientist ≠ Good at Math. Discoveries emerge from ideas, not number-crunching.

But after the idea is generated, there is a lot of number crunching to be done in science

The crowd of people at XSEDE are wise because they are not arrogant, probably not to upset Professor Wilson from Harvard. They know so much more than other mortals, yet they are among most sensitive and insightful humans on this planet.

Our Bosco people attending XSEDE'13  (Derek Weitzel, Lauren Michael, Mats Rynge, Miha Ahronovitz)  felt like the  unknown Beatles at Ed Sullivan show in1964:

Derek, Miha, Lauren. Mats Rynge is somewhere outside the picture
We went to the poster session we placed this poster  on the wall. Click on the picture to enlarge

Standing in front of poster we see XSEDE'13 people looked at it and understood. Compared to anything else presented there, our Bosco is disarmingly simple.

If you have a laptop, we said,  - must be a Mac for now. - all you do is download Bosco using a single script command. The script stops and ask you what is the name of the cluster (or supercomputer) you want to connect and password (all knew we meant this is an ssh password), Then you use your laptop to send work both to XSEDE resources  (HPC) and Open Science Grid (HTC resources).

This it was too simple. Indeed Bosco is not perfect for any jobs, We do not not support message passing and we excel in embarrassingly parallel jobs. But what fascinated our very knowledgeable visitors was the simplicity we offer, the user experience.

As most work lately is in big data data, why not make XSEDE available to any doctor in San Diego and later in USA to do a personalized genomic comparison for each patient, compared it on one one with hundred of millions of genomes in the data base and have the results back faster than doing a urine test.

We were at XSEDE thanks to the team (Frank W├╝rthwein, Lothar Bauderick, Igor Sfiligoi, Brian Bockelman et al)  who did this amazing feat.
Using Bosco, the Compact Muon Solenoid (CMS) was able to process 125TB of collision data. Bosco enabled the seamless integration of the XSEDE resource into the CMS framework.
 Frank gave us a nice testimonial:
 Bosco and glideinWMS provided the crucial "glue" to make job submissions to Gordon possible to process 125TB of CMS data. This accelerated availability of this data for doing science from 6 months to 1 month.
 This work is not for the light of heart, but using Bosco created a reason for us to be at XSEDE'13
 We are here to be part of future solutions that will help XSEDE be used by non-power user scientists and students. We can contribute by lowering the ability threshold, increasing user desire, and creating triggers to actively engage new users. Ask us how! 

Monday, July 15, 2013

Teaching Microsoft how HTCondor can be magic on their workstations

Following on my previous post,  Microsoft Reorg, Keith Richards, and the rest of us

I inserted below the slides prepared by Guy Tel-Zur on how to install HTCondor 8.0 on Windows. Yes, Windows

If you want to know what this is good for, see Guy's  other presentation HTCondor flocking between two clouds where you can create a pool of HTCondor jobs on Amazon EC2 and move them (flocking them, like shepherds do with lambs) to Rackspace and the other way around..  I call this  "A spark of genius coming out of sheer simplicity." This uses Windows Server 2008

Talking about One-Microsoft, maybe Mr. Ballmer sees the hint on how to unify Windows computers

Sunday, July 14, 2013

Microsoft Reorg, Keith Richards, and the rest of us

Nicole Hemsoth comments in HPCwire about the "Text of an internal email from Microsoft chief executive officer Steve Ballmer to employees regarding changes in the senior leadership team at the company."

I don't know how an "internal memo" is splashed over Microsoft web News pages.. Maybe this is intended for the outside world, because Mr. Ballmer must show a skeptical  Wall Street (see WSJ Steve Ballmer's Epitaph) that he is delivering something to keep Microsoft growing.

Here is my comment to Nicole's post on LinkedIn:
I notice the creation of processes" that span groups to ensure we succeed against our goals". This - together with the slogan "One Strategy, One Microsoft ", reminds me of a communist manifesto. Mr. Ballmer good intention has no room for "creative chaos"
If anyone has a bright idea, it will go against of a wall of managers tasked to say: this is not fit the the Big Boss dogma of One Strategy, One Microsoft.
One Microsoft? This is what Mr. Ballmer says; "We are rallying behind a single strategy as one company — not a collection of divisional strategies. Although we will deliver multiple devices and services to execute and monetize the strategy, the single core strategy will drive us to set shared goals for everything we do." 

Nicole said:  "We’ve been working  -we at HPCwire - on getting someone with an HPC slant from Microsoft to chat with us but right now it’s all about the cloud.

Cloud is great, but this is not cloud. This is One Microsoft cloud slogan.

Lady Gaga is not Lady Gaga, in reality she is Stefani Joanne Angelina Germanotta. She uses this  name to create music we  love.

Lady Gaga, ageless

If you want to pass an interview with Microsoft. that's they want to hear; One Microsoft. You can not be anything else.
Keith Richards,  69  forever young
Keith Richards from Rolling Stones is probably the quintessential non-Microsoft individual. In his autobiography, Life. he writes
Like  Rastafarians - the religious  group of black Jamaican who consider themselves Jewish - when the grid and the irons and the bars closed on societies everywhere, and they go tighter and tighter, they loosened from them. They were not going to work for Babylon 

But how many people are Lady Gaga, Keith Richards, Steve Jobs or Zuckerberg?  How many? Most people dream of joining Microsoft and stop thinking, for big money.

LinkedIn has a job advertised  Director / VP, Business Development SAP Ventures  By the time I write this post, there are 378 applications. Three hundred seventy eight individuals who day-dream to start ventures as employees of another company, with other people  visions, with benefits and health insurance. It does not matter which one of the 378 applicants is selected. It does not matter the criteria of selection (they can run as well a lottery).  But I can bet SAP Venture that their business will not be better or worst. It will be the same. Most probably SAP executives - who are themselves high level employees - will recruit a mirror image of them and a  yes man, who could be replaced if things go South.

If I were in Microsoft, with all their cash, I would start a Microsoft  Y  Combinator, or even better, I will make a partnership with them. Paul Graham, who will never apply for a job in Microsoft, invented the anti-thesis of one-anything.
Paul Graham
 Paul developed a new model of startup funding. He says: Twice a year we invest a small amount of money ($14-20k + an $80k note) in a large number of startups (most recently 52). The startups move to Silicon Valley for 3 months, during which we work intensively with them to get the company into the best possible shape and refine their pitch to investors"

As I work with the best minds in High Throughout Computing, here are my personal ideas on what a Microsoft  startup incubator might do.
In  HPC, Microsoft never had a real presence. But I am surprised, as kings of workstations, that they did not make some simple subtle changes in Windows to allow the creation of clusters for High Throughput Computing, as developed by the Center of High Throughput Computing and CERN and many other. It was the HTC technology that enabled the discovery of the Higgs particle . Most probably the Nobel Prize this year will reflect this success.
Mac desktops and laptops are much easier part of extremely powerful clusters, as they are Unix based, HTCondor has a deep port for Windows (The best there is for cluster software  - in my opinion - yet Windows has intrinsic limitations that can not translate to distributed computers.
Imagine bringing what Microsoft does best , UX (user experience) to each Windows laptop which can create clusters easily from thousands of machines costing a few hundred dollars.
Microsoft lacks cluster computing skill set. Now with all the processes and oneness they preach, an HTC voice will be lost in the jungle.
Nicole will not give up:   Once the reorg dust settles, we’ll definitely follow up, she says.

But according to WSJ on July 12, 2013, there is no hope for HPC.
"The first thing I expect is a more collaborative and unified Microsoft," said Patrick Moorhead, who heads market-research firm Moor Insights & Strategy. "What this translates to is a much more unified experience between the phone, tablet, PC, and Xbox."
Analysts and former Microsoft executives noted that the change will require more discussion and negotiation among managers, neither of which are necessarily conducive to speedy action.
 Yes. This is One-Microsoft today:

See the follow up post:

Blog Archive

About Me

My photo

AI and ML for Conversational Economy