To The Cloud: Working with Amazon Web Services

“To the Cloud” was initially coined by some commercials from Microsoft back in 2010.  From the beginning, cloud offerings didn’t really excite me.  Having toothed myself in the corporate world on Open Systems, I’ve never been a fan of the bleeding edge.  Back in 2010, cloud offerings still kind of seemed on the fringe, to me at least.  Naturally, I did do a bit of reading on the subject, and I arrived at the same conclusion as many “experts”.  That corporations might have issues with putting their data out in the cloud.  There are a whole slew of arguments against implementing cloud based solutions at the enterprise level.  There are just as many arguments for it.  This post isn’t about that argument, so I’ll skip that discussion. It’s been beaten to death anyway.

XCentium want’s to specialize on cloud based solutions, so as a part of my job, I get to do some playing around with Amazon AWS, Windows Azure, Rackspace, and a few other cloud offerings.  This post will be focusing on Amazon AWS, and more specifically one of their service offerings known as “Cloud Formation”.  We need to lay some groundwork first though.

Amazon AWS is one of the major players, if not THE major player, in the Cloud space.  AWS stands for Amazon Web Services.  AWS is a complete platform for computing.  By that I mean it provides all the functionality one would need for any sort of computing task. Some of the components of AWS are:

Each of these services are typically self contained, but can be configured to interact to form any possible computing platform.

One of XCentium’s clients needed an enterprise level system designed.  The system itself would be handling highly sensitive financial information, so it needed to have some really secure portions of it. Additionally, this system would be experiencing periods of intense activity during some parts of the year and periods of lull during other parts of the year. Finally, they needed a way to sort of genericize the entire system, package it, and spin it up for different applications, potentially from different industries.  After some discussion it was decided to try to come up with something in AWS, since the elastic computing part of it was key.  AWS seemed like it had enough of a security framework to do the job, and though none of us were terribly familiar with their Cloud Formation service, it also seemed like it would fit the bill by allowing us to spin up an entire environment from a single job, if we could figure out the language.

Let me digress here for just a few sentences and exclaim just how freaking HOOKED I am on Amazon AWS.  Over my IT career I’ve served in just about every possible technical position imaginable:  Tech Support, PC Technician, Network Admin, DBA, Systems Admin (both Open Systems and Windows), Software developer, Webmaster, Content Manager, Mainframe Operator, Project Manager, Manager, Director and even CTO.  This has given me both a very broad and, in most cases, detailed view of the IT industry.  System Administration is at the core of any system. It is needed by small businesses with a small IT footprint and enterprises with a massive IT footprint.  Amazon AWS has taken Systems Admin to a new level.  After having spent a couple years working with AWS, I cannot think of a single solution that couldn’t be implemented.  It does have some drawbacks in certain places, but they are not anything that cannot be overcome or that the AWS developers themselves haven’t provided a solution for, though perhaps in a different shape.  Additionally, they LISTEN to their customers.  The AWS team is constantly rolling out new service offerings and upgrades to existing offerings.  My work with AWS over the last couple years has completely erased any doubt I initially had about Cloud based offerings.

Okay, back to the ranch.  We needed to design a highly secure system that could scale up or down based on load requirements, and we needed to be able to package it.  The first step was to actually design the system.  Amazon has a great Visio stencil (and other icons) here. So I got to work designing a system.  I cannot post the system design docs due to NDA, but I can tell you it was ugly, or beautiful, depending on your view of the IT world.  There were at least 10 subnets with individual virtual routers between them all, each with specific static routes and specific ingress and egress ports and gateways set up.  Between the routers, paths, load balancers, autoscaling, gateways and links to external systems, the thing looked like a can of worms.

The question that then came up was how to go about getting this massive configuration built into the Cloud Formation script.  Cloud Formation uses JSON as it’s data file format. The name used by Cloud Formation for an entire system is called a “stack” The options were to try to build the script out from scratch utilizing Amazon’s very good Cloud Formation API and Documentation.  Amazon, at the time, also had a Cloud Formation Template Generator that supposedly would take an existing configuration and generate the stack based what components one selects while running the wizard.  This seemed like a good place to start, so I built out a skeleton framework of the system I had designed and ran the template generator against it.  It only captured a few of the components (still in development mode), but it did a good enough job to get me started.  The basic JSON file generated was about 500 lines line long.

So I started hacking the JSON file.  It wasn’t very long before I realized manually editing the JSON file was the wrong approach.  I scrounges around the new for some JSON visualizers or Visual Studio addins.  Nothing I found was optimal. I did settle on the JSON Editor Online for a while. Even then, once my JSON file got to about 100 elements, it became too bulky to do much with.

I eventually ended up writing a Visual Studio desktop application (well, it wasn’t little) that I could represent the different components of my system with and assign the various properties needed by each component. This worked pretty well, as I was able to visually set things up in my Windows App and could get an 80,000 ft view of the system as well as drill down into the different subnets, load balancers, auto-scalers and routers to see the nitty-gritty details.  When I first started, I had dreams of making a generic application that could be used to drag-drop visual components onto a form and eventually have it generate the full JSON script like that.  Unfortunately, that would be such a huge task that I doubt my bosses would sport for the 6 months or so it would take to do it.  I’m not sure there is enough call for Cloud Formation template generators out there to warrant the work.  Additionally, Amazon themselves may have matured their current template generator to the point that it would generate a full stack script from a given environment.

To sum things up, I’m hooked on AWS. Their service offering and their commitment to that service offering has cleared any initial doubts I might have had about Cloud computing.  Amazon does a really great job of supporting their platform. That’s probably because they have to use it for their internal systems as well. That means they have a customers view of their platform.  I’m sure that goes a long way.

Categories: Cloud