The New Joyent

Summary 

Joyent Container Services will  transform the company. It  will run Docker containers along side Joyent containers as equal citizens . The Docker containers will run natively in Joyent cloud architecture will add huge value for Docker users. 

Joyent will  minimize and eventually eliminate the gap between containers in development and containers in production. The Docker revolution is a blessing

Last, but not least,  Joyent will open up and make products people will love and use, and not just admire  like a picture in a Museum of Modern Art.  

Joyent : Cloud and Containers DNA

This is part 2  of this story.  But I covered Joyent many times in this blog.

Four years ago, in August 2011, I wrote the article Net Promoter Score For four Cloud IaaS providers. Joyent had highest score. Joyent was viewed as the coolest IaaS cloud company

Fig. 1: August 11, 2011: Joyent by far was the winner 

I met with Bryan Cantrill, Joyent's CTO  last week. I know Bryan from Sun, where his picture was on many posters all over Menlo Park campus, now occupied by Facebook. This was how Sun motivated and gave full credit to its' engineering champions. They had a movie star status. I liked Joyent because they had some of Sun's best engineers and a superior culture I was familiar with.

Brian is very technical, very precise and uses metaphors. Each of his presentations resembles an artistic performance

Miha: I read your blog post called Predictoria 2015. By the way, I could not find the word in any dictionary...

Bryan: I invented it (smile). I like a restaurant called Osteria in Palo Alto This is a place where fifteen years ago  my colleagues in kernel development at Sun and I we would get together to form predictions for the coming year. So Osteria inspired  Predictoria.

There is a good chance that  Predictoria will  become a new word on the Valley jargon and Bryan will be credited for it.

Docker Containers

Predicteria 2015 first bullet is:

2015 is the year of the container. ... Thanks to Docker, the world is finally figuring out that OS-based virtualization is actually highly disruptive (better performance and lower cost!), and I think that this realization will become mainstream in 2015. I don’t think that Docker will necessarily be the only substrate,

The second bullet is even more interesting

The impedance mismatch between containers in development and containers in production will be a wellspring of innovation. Currently, containers have a ton of developer enthusiasm — but limited production deployments, in part because of production concerns around security, persistence and network virtualization. But it’s a sure bet that there will be (many) players tackling the problems in interesting ways.

Miha: This impedance mismatch ,can you explain how did you discover it?

Bryan: When I speak to a developer event, I ask people the audience: "How many of you have used Docker containers?"  About one third to half of the s audience raise the hand. Then I ask:  "How many of you have used Docker containers in production?" and I see just a few hands  still up
__________________________________________________________

I did a little research on my own. I am not surprised that Docker has "tons of developer enthusiasm - but limited production deployments". Everyone talks docker, docker, docker. Everyone looks at the picture below and thinks. "This is Docker", easy

Fig. 2: Containers share OS versus VMs which need Hypervisors.

The devil is in the details

Docker Blues

Stack Overflow is a question and answer site for professional and enthusiast programmers. It is a natural site for Docker developers

Sample question 1

I've watched a ton of youtube videos and read all of the docker docs. However I still do not get a core concept that is stopping me from understanding docker. I am using windows and have boot2docker installed. I've downloaded images from docker hub and run basic commands. BUT How do I take an existing application sitting on my local machine (lets just say it has one file 'index.php', for simplicity). How do I take that and put it into a docker image and run it?

Answers 1 to question 1

Your index.php is not really an application. The application is your Apache or nginx or even PHP's own server.

Because Docker uses features not available in the Windows core, you are running it inside an actual virtual machine. The only purpose for that would be training or preparing images for your real server environment.

 Answers 2 to question 1

You'll need to build a docker image first, using a dockerFile, you'd probably setup apache on it, tell the docker File to copy your index.php file into your apache and expose a port.

After I read this, I wonder if ever I want to dockerize anything on Windows. This is complicated. This is complex. This is not for a mainstream Data Center staff to handle.

Sample question 2

After reading the introduction of the phusion/baseimage I feel like creating containers from the Ubuntu image or any other official distro image and running a single application process inside the container is wrong. The main reasons in short:

• No proper init process (that handles zombie and orphaned processes)
• No syslog service

Based on this facts, most of the official docker images available on docker hub seem to do things wrong. ...

Now the question arises which is the appropriate way to run an service inside docker container. Is it wrong to run only a single application process inside a docker container and not provide basic Linux system services like syslog? Does it depend on the type of service running inside the container?

Answer to question 2

Check this discussion for a good read on this issue. Basically the (semi) official line from Solomon Hykes and docker is that docker containers should be as close to single processes micro servers as possible. There may be many such servers on a single 'real' server. If a processes fails you should just launch a new docker container rather than try to setup initialization etc inside the containers. So if you are looking for the canonical best practices the answer is yeah no basic linux services. It also makes sense when you think in terms of many docker containers running on a single node, you really want them all to run their own versions of these services?

That being said the state of logging in the docker service is famously broken. Even Solomon Hykes the creator of docker admits its a work in progress. In addition you normally need a little more flexibility for a real world deployment.

Bottom line, the easy to use docker containers for production do not exist yet, unless one hires the top top developers,

The  Docker super-star, Solomon Hykes is French.  He is a seductive dreamer:

 ...users nowadays expect apps to behave like the Internet: unattached to a machine, readily available, working the same no matter how you interface with them. This means software needs to be decoupled from the underlying machine -- "the app needs to be everywhere and nowhere,"

This is an elegant thought, but the Internet today is not safe. Walter Isaacson, the author of Steve Jobs biography, wrote in the article from Time Magazine, January 19, 2015 entitled  "Time to Build a More Secure Internet" .  The Internet started from network built by Pentagon Advanced Research Projects Agency (ARPA), hence the network name, ARPANET. The idea was to have a network able to survive a nuclear attack. But the engineers who actually devised the traffic rules  "anti-authoritarian to the core" were graduate students - many of them avoiding the draft during the Vietnam war, did not care about about the military uses of of the Net. Now Net's architecture makes it difficult" to control or even trace the packets that dart through its nodes" As a result, 40 million Americans had personal information stolen in cybercriminal attacks and the 2013 loss to the U.S. economy was a staggering  $100 billion.

Just watch a viral video from 2010, The un-ignorable Internet to see what ARPANET looks like today. I imagine a docker container that has all packages of an application plus the intelligence built in on how run it. If this is the ideal place for docker as a platform ?

Solomon Hykes  explains Docker can not produce the set of tools for every  developer, because, in respect for human people, each developer is different and does things differently according to his beliefs and personal style. This is a superior thinking that will make Docker a winner for all developers in the world. Hence Docker in its' pure higher level abstraction can not be a production container, as is.

The corollary of this is that Docker in it's pure form is by definition insecure. As of today Security is a task for companies who have real enterprise customers. Joyent is one of those companies

Joyent and Bell Telephone Company

Although I know Bryan for many years, this is the first time we had a  longer in depth conversation about his work. 

I am reading a new book on Nicola Tesla by W. Bernard Carlson . Tesla talks about two ways to create inventions. The Edison style is

purely experimental of which undoubtedly Edison is the greatest and most successful exponent. The moment you construct a device to carry into practice a crude idea you will find yourself inevitably engrossed with the details and defects of the apparatus. As you go on improving and reconstructing, your force of concentration diminishes and you lose sight of the great underlying principle. You obtain results , but at the sacrifice of quality

The Tesla style described by Tesla himself

My method is different. I do not rush into constructive work. When I get an idea, I start right away to build it up in my mind. I change the structure, I make improvements, I experiment, I run the device in my mind. It is absolutely the same to me whether I operate my turbine in thought or test it actually in my shop . It makes no difference, the results are the same. In this way , you see, I can rapidly develop and perfect an invention, without touching anything. When I have gone so far that I have put into the device every possible improvement I can think of, that I can see no fault anywhere, I then construct this final product of my brain. Every time my device works as I conceive it should and my experiment comes out exactly as I plan it

The inventor - and every reader of this blog has at least  a tiny inventor DNA  inside - knows on one side, "because nature does not readily yield up her secrets, one could say that an inventor “negotiates” with nature."

 On the other side," it took the Bell Telephone Company decades to convince Americans that every home should have a telephone. Bell and his successor companies had to invent not only the telephone but also a marketing strategy that reflected the interests of users. In this sense, inventors “negotiate” with society."

I see Bryan Cantrill and Joyent  as a company are more like Tesla style innovators.  They had a consistence to provide cloud rock-solid  Docker fits in their cloud and data center products, like  a glove

I see Joyent's story similar to Bell Telephone Company a century ago. It may take a decade to develop a marketing strategy, that reflected the interest of the users. This is happening right now. I am not surprised visionaries like Peter Thiel invested in Joyent. I am not surprised Telefonica invested too,  Because Joyent anticipated business success is now more credible than ever before.

How Joyent  will  adopt Docker Containers?

I ask the Casey Bisson, Joyent's  new product manager this question

Joyent containers (Solaris/SmartOS zones) and Docker containers are very similar. 

Joyent’s container technology originated a decade ago with Solaris zones, designed to provide multiple, isolated, yet complete operating system environments. This allowed more efficient use of hardware with few or no changes to application design.

Docker is a container technology that, ideally, isolates applications even further into micro-services. This is a significant innovation that could result in some exciting advances in application design and deployment practices. 

Joyent’s Docker service will  launch in beta in early 2015 combines these technologies to gain the best advantages of both.

Casey hints that a beta Joyent’s Docker service  will available before the end of April, that Bryan Cantrill mentioned as a release date.

Micro-services. What are they?

Fig. 3: Slide 9 screen shot Jerome Petazzoni slide deck (Docker Inc.)

The slide is self-explanatory, But why we need Micro-services?

Fig, 4: Why Micro-services are useful? Jerome Petazzoni,  slide 10, Docker Inc

I like  the Jeff Bezos "two-pizza rule". This is one of those ideas that it is so obvious that makes people say: "Gee, how that I  did not notice this before?"

However it is worth noting: Joyent containers can run microservices independently of Docker. Joyent’s SmartDataCenter 7  (open source at https://github.com/joyent/sdc), released in mid 2013, is built on the micro-services pattern.

According Casey Bisson, 

The Docker container service that we’re building now will run Docker containers along side Joyent containers as equal citizens in our cloud.

Both Joyent and Docker containers can run microservices,  but Docker has done much to popularize microservices architectures. The patterns for containerized development and deployment that are emerging now are really exciting, and Docker deserves credit for promoting that.

He expresses

We’re excited to see the growth in interest in containerization, and we think supporting Docker containers natively in our cloud architecture will add huge value for Docker users. The benefits include  bare metal performance, ZFS, and DTrace, as well as the years of experience we have in running containers in production.

This is very cool 

The company introduction video 2013

After I wrote this blog, I found on You Tube this 2013 Joyent high level video: It says:

A big idea can  change the world and the faster it can be shared, the faster it can go to work 

Docker will  perfect the Joyent marketing strategy to "negotiate with society"   in a way everyone can see the light. Joyent may become ubiquitous if the people understand exactly it does. As part of the negotiations with the society, the 2015 video must show why Joyent Containers solution in production is 10x better than the competition today. Why 10x? Because this is the criteria to be perceived as a zero-to-one product and company.

Here is the original video

Joyent's roots from Sun

 Miha: How is Joyent different as a participant in Docker revolution?

Bryan: In Joyent we brought in the container technology from Sun. We at Sun were blessed. We saw containers long before Docker. Sun was making machines so big, selling well to customers to run simultaneously multiple application on one machine. By the end of  1990's  the application codes became so complex, so we had no option but to virtualize  the machine.

Miha: How did you do it?

Bryan: We did not come from a cloud direction. We  came from a need for workload consolidation on  a server that had to be secure all the way down to the bedrock

Note from Wikipedia ; "Solaris Container was any type of environment constrained by Solaris Resource Management, even when this didn't include the use of a Zone. Over time, common usage changed this to mean a Zone combined with Resource Management."

Miha:  "Secure all way down to the bedrock"  How we measure that?

Bryan: When Zones came around, we did a contest, by offering  a $10,000 winner prize, to anyone who managed to break a Zone.  The winner was Casper Dick, Sun long standing top security expert, He managed to find some crazy detail that he was able to exploit. But since then no one reported a breach of security. Solaris Zones were bedrock solid. That is the path  Joyent containers are coming from. We have built Joyent based on that foundation.

Miha: Now Docker is coming.

Bryan: In the past  half year, yes Docker was coming and it was embraced enthusiastically by all developers, who said: "Docker is great abstraction." because -  in addition to packaged software -  a  Docker image contains also the dependencies. A Docker has the intelligence built in
Miha's Note: See Using Linux Containers to Support Portable Application Deployment 

Bryan:  The reason I came to Joyent, is because I believed  in the cloud and I believed in the OS reservations. So Docker is seductive in its elegant abstraction, but it is not coming from having the rock solid container for usage in enterprises.We can run very well Docker on Joyent Public Cloud and on  Joyent SmartDataCenter, These environments are rock solid, but we want more.

Miha:  More?

Bryan: Until now Linux was a moving target, but it slowed down a lot. So we intend to run Linux application directly via Joyent Container Services on SmartOS. We will end this game of trying to convince customers to port their Linux apps to SmartOS

To complement what Bryan means here is a quote from Bill Fine, Joyent VP of Product and Marketing:

Holistic simplicity alone, however, isn’t enough to be able to claim that Joyent is the best place to run Docker. We also needed to provide an infrastructure container run-time environment superior to the good enough, “batteries included” option provided by Docker. So, we set out to seamlessly integrate Docker into Joyent SmartDataCenter, the container infrastructure run-time environment used today by Joyent customers to power their mission critical applications across thousands of infrastructure containers in the Joyent Public Cloud, and the foundation for container-based, private clouds at some of the world’s most recognizable companies.

Now, through our work (in the (link: https://github.com/joyent/sdc-docker text: open)) to extend the Docker Engine to SmartDataCenter and enhance SmartDataCenter with Linux Branded Zones, we are nearing our goal of providing “Dockerized” Linux applications a run-time environment that is secure, delivers both bare metal performance and cloud economics, provides integrated virtual networking and virtual storage, and is available via a public cloud service or private cloud software download.

According to Bryan, this will happen at the end of April, 2015. "There is no question that we will get there" he says

Miha: Lets say you deliver the Joyent Container Services as expected, successfully. What does it mean to the user? The user will say: "I am going to use Joyent, because of.what?"

Bryan: There are a bunch of reasons. I am giving you the one that I think will be most compelling to people. When you blow up the hardware virtualization layer - there is no hardware virtualization layer in docker - that means you have now a global visibility across the entire data center or cloud. This realizes the promise of cloud computing. You create a  docker host, a machine, and then you create the containers on that machine. Because this is a hardware virtualization machine, when you run out capacity, you create a second docker host that has the same global visibility as the first one.

As I wrote in my blog post of Dec 15, 2014 Joyent is betting that developers that build applications using Docker containers would much rather attain higher levels of performance using bare-metal servers rather than relying on hypervisors to access virtual machines.

Miha:  Assume I  create my docker machines, one, two, three, as needed.  How is this different from the big players AWS , Google and so on?

Bryan: Because in order to give you, the user a global visibility, it looks to as you are a gigantic virtual computer. You have to administrate the virtual computer and the containers. In Joyent, you only worry about the containers. We are the equivalent of the hardware hypervisor, we offer stack simplicity, where simplicity itself is the power

Miha: And now (end of April 2015) one can run Linux application on Joyent,

Bryan:This is huge as we do not have to run after people to convince them to port from Linux. We run Linux apps without any degradation in performance on Smart OS

_____________________________________________________________

What kind of a production container Joyent provides?

Bryan jotted down this on the white board

Fig. 5: Three bullets for Joyent containers

• Bedrock solid security, 
• Power to operate by using optimized SDN (Software Defined Networking)
• Revolutionary performance: you can do more work with less infrastructure costs. You can run the same data center created in AWS or Google to to 70% less costs

I will cover these three bullets more in detail after the launch of Joyent Container Service in May 2015

User Experience

This is NOT interface design, although it includes it. This is about what people sentiment is when using the product

• Ease of use, aka UX (user experience) for mainstream sysadmins and engineers or even plain end users (the Know-nots)  in addition to expert docker developers (the Knows)
• Joyent's production containers can  create a habit as defined by Nir Eyal

• The way we reach user hearts and brains is via Habits What is a Habit? ("automatic behaviors triggered by situational cue; things we do with little or no conscious thought"). A habit-forming company ... links its services to the users' daily routines and emotions"

I looked up Casey Bisson,  - the new  Joyent PM - profile and I read:

I am an engineer who enjoys making products that are easy and fun to use, social, and sticky. 

Joyent will open up and make a product people will love and use, and not just admire  like an art object in Museum of Modern Art - which we are not allowed to touch.
__________________________________________________________