So the cloud is here, but are you moving with the times or are you behind in your thinking? It’s a question people will never admit to, but the reality is becoming very apparent that SAN and NAS do not scale to large clouds such as Amazon or AT&T. So how do the big guns do cloud?
So lets take a service such as Amazon who have one huge infrastructure which spans global data centers with one huge flexible namespace which can grow with no complexities and minimal management costs. Amazons new offering which is Amazon Glacier released back in September, which is now 1 penny per GB per month! How do you get costs down to that price point and still turn a profit, and to give you some perspective on how big the Amazon infrastructure is this year alone 260 Billion objects were added! Imagine trying to manage that with traditional thinking such as silos of storage in SAN and NAS storage? Amazon’s pricing has dropped 12 times this year alone and they just under cut the market every time.
So lets look at their thinking, one thing that all the big cloud names have in common is that they do not use file systems for their cloud, this includes Facebook, Twitter, E bay, Amazon, You Tube. So why do they not use this, the answer is simple cost and scale. These infrastructures are huge and when they set about creating their clouds they wanted massive scale 10’s of petabytes if not hundreds with minimal growth disruption and management over head.
So lets take a different angle for one moment, it was us the end-user which created this costing point, because technology is so readily available now, for example if you have a credit card you can get a server and some storage from Amazon in a matter of minutes, but they are by no means the only ones doing this. The consumer market is so diverse now that if the price point is not right we just move on it’s as simple as that. How many times have you checked the price of something on the internet while shopping in a store?
So back to the point lets use Amazon, they use an object based API which incorporates, security, encryption, security, billing and a policy engine talking to commodity x86 servers and commodity storage. Hardware fails we accept that as everyone does, but the key is the software at the top layer. Object based storage does not work like traditional file systems and it spans one single namespace meaning you can geographically disperse your data centers and have one giant object store which according to policies set it replicates and protects your data. The simplest way of explaining this is “Drop box”, we all love and use drop box and it may surprise you to know that it too uses Amazons philosophy as above. Policies are the key in this object based world, lets take your free 10GB subscription with Drop Box, as that is a “Free” service it is very unlikely that a copy of your data is made, it is replicated or encrypted and they do not guarantee it will always be there. But what if I pay the fee per month? Well then you would have a paid policy which would replicate to another data center, encrypt your data and more importantly bill you on usage metrics such as bandwidth and space used.
Now this is the key component here Object storage is subject to policies, an object contains meta data and the content, the intelligent API’s look at the meta data and decide what to do with this data according to policies set. This is key to understanding the management of the cloud. Let’s take E Bay how many photos do you think get uploaded to E Bay every single minute? Well I imagine this number is huge, but how do you manage how long those photos stay there? Before policies E Bay were having to run jobs to delete these photos every night, but there came a point where they were doing this constantly, so with policies in the API they simply set one up to delete these after 3 months has passed. It is as simple as that, all that management has gone and is automated.
The technology that Amazon and E Bay use is EMC Atmos. It is the intelligent API with commodity hardware underneath defining it as a purpose-built cloud platform giving you up to 1.3PB per floor tile. Atmos allows you to easily scale your cloud over geographic distances as it acts as one great big storage pool with one namespace, the API abstraction layer takes care of all the storage calls so developers who are paving the way in browser-based applications which are WAN friendly do not have to care what goes on below the software layer. Atmos takes care of all this, so lets imagine you have 5 data centers globally all connected and your objects are behaving according to your policies and automatically billing your end users based on your policies set (security, replication etc), which you don’t have to back up, Isn’t that the way to do things? Just imagine trying to back up Amazons cloud……………….no thanks.
As the intelligence in an Object store and resilience is also built-in you can lose multiple drives or nodes and your service does not go down, People such as Amazon and E Bay accept that hardware eventually fails, so they just stock pile this and when drives fail they replace them eventually, as it is not critical. Has E Bay ever gone down? The answer is no and there is good reason for this if E Bay went down it would cost E Bay $3,900 per second!
So EMC Atmos is arming the cloud, and the service providers are monetizing this platform in to services that me and you consume every single day. SAN and NAS ways of thinking are fast becoming limiting in the way they can scale in comparison to Object stores and this is now in my personal view why service providers are switching on to this change, traditional service providers are offering things such as “back up to the cloud” etc, and what they need to be doing is appealing to the developers who have written so many of their programs for Amazon S3, as their applications could run on Atmos as it understands S3. This would enable them to keep with the curve in this changing marketplace. And the best part is that this Atmos API is yours, you can edit, modify it, do what ever you like with it to make it work for your company and give a portal to your end users, and bill them accordingly.
So to sign off is Amazon trail blazing the way ahead? No, they have just done this before anyone thought of it, and they are now so large that they can just dynamically grow, and everyone thinks that cloud is slow, but look at Amazon using commodity hardware and servers, the sheer scale means the amount of compute and storage available is all there for the taking!