Thursday, September 27, 2012

ANL's OSG BOSCO Job Submission sw. Because we still need grids, in addition to clouds

So what is  the difference between Grids and Clouds, and why we still need (badly) grids 


Grids assume a fixed size of resources and the management software (LSF, PBS, GE, and so on)  assumes there is more user demand, especially at peaks, - than the resources available. As a result there are a lot of policies based on priorities, urgency, set from the top down. Users expect different levels of SLA's depending on how many want to access the grid simultaneously. For low level users, the service levels decay substantially.

A cloud is a  grid with elasticity and billing. The  elasticity means the service levels can be maintained, as resources are added or withdrawn automatically as needed. The billing means one can can always get what s/he pays for and does not depend on top-down policies. This creates an entire new business model

Another key difference from Grids and Clouds, is that in clouds the utilization of virtual resources is only 17% for small # of instances and it can drop to 2% for very large # of instances .The fact that one rents a virtual node, it does not mean that it can not  sit idle for days and weeks. This is a bombshell slide titled "Under-utilization is rampant" from Cloudyn regarding Clouds.

There is no way to submit high throughput  100,000 jobs to a cloud (AWS, RackSpace) made up from virtual nodes. We still need grids.

In grids,- manged by DRM (Distributed Management Software)  such as Grid Engine (Univa, Oracle Grid Engine, Son of Grid Engine,) ,  LSF (now with IBM), Condor  or PBS Professional, - the software knows exactly what is going on on every single node, physical or virtual and this makes possible the maximum utilization closer to 100%

Sure we can use clouds, but the clouds must have configured clusters (Grids), identified as a resource within a cloud.  Normally the grids are stand alone, and multiple grids are accessed at the time.

A new Job Submission software, called Bosco. Why we need it?

  My ex colleague in Sun, Dan Fraser is a Principal Solutions Architect with Argonne National Lab and a Production Coordinator with Open Science Grid (OSG)  

In Sun Dan created the first ever web portal to access a grid the easy way.

Those familiar with DRM software, know it is not easy to use , unless you are an experienced system administrator. And faced with a myriad of grids managed with different DRM brands, why not use a single tool to access them all ? In the cloud, outside the cloud, where grids access is permitted

Dr. Dan Fraser,
Argonne National Labs
A Campus Infrastructure is a resource sharing capability that enables faculty, students, and researchers to submit jobs to multiple computational resources simultaneously.... A campus infrastructure is not limited to resources on a campus but incorporates the ability to use resources directly from other campuses.

Bosco Job Submission   software   is probably the ideal product to implement the Campus grid concept. Imagine writing only one script, and then the software does the rest: it talks with GE or LSF or Condor or PBS, without bothering the user to change anything. This is  huge. Don't be deluded of the appearance that a Job Submission software is a legacy product. OSG  proves this tool is needed more than ever,

When one says; "Hmm, how I didn't think of this before?", this is a sign we bumped into a great idea

Bosco v1.0 is a beta release

Q. Why OSG decided to there a need for BOSCO?

We are trying to make it as easy as possible for a researcher (who in most cases is not a system administrator) to access more resources than the single cluster they may be currently using. Using this desktop tool, they can gradually expand to use other resources on their campus, or even utilize resources on a collaborators campus. Bosco provides a stepping stone to enable the automatic distribution and management of jobs across multiple resources. The beauty of this model is that once they start accessing multiple resources in this manner, they can even use Bosco to access resources from the OSG itself.

Q  Are there any similar BOSCO software Job Submission schedulers?

Most batch scheduler systems have software that schedules jobs between clusters of the same flavor. In other words PBS users can schedule and distribute jobs across different PBS systems if it is configured properly. Grid Engine has this feature, as does Condor, and LSF. Bosco allows users to schedule jobs across different kinds of batch schedulers at the same time. The only other capabilities I am aware of that do something similar are CREAM (an EGI product) and Globus-GRAM2/5, and these products are akin to what is used in the Open Science Grid.

The biggest difference between these products and Bosco is the security model. Both CREAM and GRAM use the Grid Security Interface requiring users to obtain and manage grid certificates and software. For new grid users however, this has proven to be an unfamiliar and daunting requirement. Bosco is designed to utilize whatever security  the Campus uses. This is both familiar and readily understandable to Bosco users.

Q. Do they track downloads?

Currently they are not tracking downloads.

Q. Is Bosco in use only for Open Science Grid researchers? Any external users?

Bosco was designed for researchers that may not yet be using the Open Science Grid. They can start by using resources within their campus, or perhaps accessing resources on another campus. Once users use this model, it is very straightforward to start using the national cyberinfrastructure such as OSG.

Q. How many open grid researchers work High Throughput?

The Open Science Grid is 100% dedicated to high throughput computing.

Q. Who is responsible for project Bosco?

Dan Fraser  is the Project Lead. In addition to Argonne, he also has a joint appointment with the University of Chicago. Brooklin Gore is the Project Manager for Bosco. He works at the University of Wisconsin, Madison.

Q. Is support free for outside users? Or only community support?

Currently, as for many open source projects, support is free for outside users as well as community users.

Bosco will have a dedicated product URL. Until then you can download the open source, play with it,  and perhaps  make money with it.
Post a Comment

Blog Archive

About Me

My photo

AI and ML for Conversational Economy