Category Archives: IAAS

Farming your CloudStack cloud

A couple of years ago, I blogged about my prototype of StackMate, a tool and a service that interprets AWS CloudFormation-style templates and creates CloudStack resources. The idea was to provide an application management solution. I didn’t develop the idea beyond a working prototype. Terraform from Hashicorp is a similar idea, but with the ability to add extensions (providers)  to drive resource creation in different clouds and service providers. Fortunately Terraform is solid and widely used. Even better, Sander van Harmelen (@_svanharmelen_) has written a well-documented CloudStack provider.

Terraform templates have a different (but json-style) syntax than AWS Cloudformation, which among other things lets you add comments. Like StackMate, it figures out the order of resource creation by creating a dependency graph. You can also add explicity “depends_on” relationships. I played around with Terraform and created a couple of templates here:

https://github.com/chiradeep/terraform-cloudstack-examples

One template creates a VPC and 2 subnets and 2 VMS. The other template creates 2 isolated networks and a couple of VMs (one with nics on both networks).

Pull requests accepted.

While there are awesome services and products out there that can do similar things (RightScale, Scalr, Citrix Lifecycle Management), it is great to see something open sourced and community-driven.

How to manage a million firewalls – part 1

In my last post I argued that security groups eliminate the need for network security devices in certain parts of the datacenter. The trick that enables this is the network firewall in the hypervisor. Each hypervisor hosts dozens or hundreds of VMs — and provides a firewall per VM. The figure below shows a typical setup, with Xen as the hypervisor. Ingress network traffic flows through the hardware into the control domain (“dom0”) where it is switched in software (so called virtual switch or vswitch) to the appropriate VM.

sg_groups_pptx7

The vswitch provides filtering functions that can block or allow certain types of traffic into the VM. Traffic between the VMs on the same hypervisor goes through the vswitch as well. The vswitch used in this design is the Linux Bridge; the firewall function is provided by netfilter ( “iptables”).

Security groups drop all traffic by default and only allow those configured by the rules. Suppose the red VMs in the figure (“Guest 1” and “Guest 4”) are in a security group “management”. We want to allow access to them from the subnet 192.168.1.0/24 on port 22 (ssh). The iptables rules might look like this:

iptables -A FORWARD -p tcp --dport 22 --src 192.168.1.0/24 -j ACCEPT 
iptables -A FORWARD -j DROP

Line 1 reads: for packets forwarded across the bridge (vswitch) that are destined for port 22, and are from source 192.168.1.0/24, allow (ACCEPT) them. Line 2 reads: DROP everything. The rules form a chain: packets traverse the chain until they match. (this is highly simplified: we want to match on the particular bridge ports that are connected to the VMs in question as well).

Now, let’s say we want to allow members of the ‘management’ group access their members over ssh as well. Let’s say there are 2 VMs in the group, with IPs of ‘A’ and ‘B’.  We calculate the membership and for each VM’s firewall, we write additional rules:

#for VM A
iptables -I FORWARD -p tcp --dport 22 --source B -j ACCEPT
#for VM B
iptables -I FORWARD -p tcp --dport 22 --source A -j ACCEPT

As we add more VMs to this security group, we have to add more such rules to each VM’s firewall. (A VM’s firewall is the chain of iptables rules that are specific to the VM).  If there are ‘N’ VMs in the security group, then each VM has N-1 iptables rules for just this one security group rule. Remember that a packet has to traverse the iptables rules until it matches or gets dropped at the end. Naturally each rule adds latency to a packet (at least to the connection-initiating ones).  After a certain number (few hundreds) of rules, the latency tends to go up hockey-stick fashion. In a large cloud, each VM could be in several security groups and each security group could have rules that interact with other security groups — easily leading to several hundred rules.

Aha, you might say, why not just summarize the N-1 source IPs and write a single rule like:

iptables -I FORWARD -p tcp --dport 22 --source <summary cidr> -j ACCEPT

Unfortunately, this isn’t possible since it is never guaranteed that the N-1 IPs will be in a single CIDR block. Fortunately this is a solved problem: we can use ipsets. We can add the N-1 IPs to a single named set (“ipset”). Then:

ipset -A mgmt <IP1>
ipset -A mgmt <IP2>
...
iptables -I FORWARD -p tcp --dport 22 -m set match-set mgmt src -j ACCEPT

IPSets matching is usually very fast and fixes the ‘scale up’ problem. In practice, I’ve seen it handle tens of thousands of IPs without significantly affecting latency or CPU load.

The second (perhaps more challenging) problem is that when the membership of a group changes, or a rule is added / deleted, a large number of VM firewalls have to be updated. Since we want to build a very large cloud, this usually means thousands or tens of thousands of hypervisors have to be updated with these changes. Let’s say in the single group/single rule example above, there are 500 VMs in the security groups. Adding a VM to the group means that 501 VM firewalls have to be updated. Adding a rule to the security group means that 500 VM firewalls have to be updated. In the worst case, the VMs are on 500 different hosts — making this a very big distributed systems problem.

If we consider a typical datacenter of 40,000 hypervisor hosts, with each hypervisor hosting an average of 25 VMs, this becomes the million firewall problem.

Part 2 will examine how this is solved in CloudStack’s Basic Zone.

CloudStack Basic Networking : frictionless infrastructure

Continuing on my series exploring CloudStack’s Basic Zone:

Back to Basics

Basic Networking deep dive

The origin of the term ‘Basic’ lies in the elimination of switch and router configuration (primarily VLANs) that trips up many private cloud implementations. When the cloud operator creates a Basic Zone, she is asked to add Pods to the availability zone. Pods are containers for hypervisor hosts. sg_groups_pptx6

The figure above shows a section of a largish Basic Zone. The cloud operator has chosen to map each Rack to one Pod in CloudStack. Two Pods (Rack 1 and Rack 24) are shown with a sample of hypervisor hosts. VMs in three security groups are shown. As described in the previous post, the Pod subnets are defined by the cloud operator when she configures the Pods in CloudStack. The cloud user cannot chose the Pod (or subnet) when deploying a VM.

The firewalls shown in each host reflect the fact that the security group rules are enforced in the hypervisor firewall and not on any centralized or in-line appliance. CloudStack orchestrates the configuration of these firewalls (essentially iptables rules) every time a VM state changes or a security group is reconfigured using the user API.

Each Rack can have multiple uplinks to the L3 core. In fact this is the way data centers are architected for cloud and big data workloads. In a modern datacenter, the racks form the leafs and the L3 core consist of multiple spine routers. Each host has multiple network paths to every other host — at equal cost. CloudStack’s Basic Zone takes advantage of this any-to-any east-to-west bandwidth availability by not constraining the placement of VMs by networking location (although such a facility [placement groups] is available in CloudStack).

networking_in_the_cloud_age_lisa2013_pptx

The cloud operator can still use VLANs for the rack-local links. For example, access VLAN 100 can be used in each  rack to connect to the hypervisors (the “guest network”), while the untagged interface (the “management network”) can be used to connect to the management interface of each hypervisor.

CloudStack automatically instantiates a virtual DHCP appliance (“virtual router”) in every Pod that serves DHCP and DNS to the VMs in the pod. The same appliance also serves as the userdata server and password change service. No guest traffic flows through the appliance. All traffic between VMs goes entirely over the physical infrastructure (leaf and spine routers). No network virtualization overhead is incurred. Broadcast storms, STP configurations, VLANs — all the traditional bugbears of a datacenter network are virtually eliminated.

When the physical layer of the datacenter network is architected right, Basic Zone provides tremendous scale and ease-of-use:

  1. Location-independent high bandwidth between any pair of VMs
  2. Elimination of expensive bandwidth sucking, latency-inducing security appliances
  3. Easy security configuration by end-users
  4. Elimination of VLAN-configuration friction
  5. Proven scale : tens of thousands of hypervisors
  6. Egress firewalls provide security for the legacy / non-cloud portions of the datacenter.
  7. The ideal architecture for your micro-services based applications, without the network virtualization overhead

CloudStack Basic Networking : deeper dive

In my last post I sang the praise of the simplicity of Basic Networking. There’s a few more details which even seasoned users of CloudStack may not be aware of:

  1. Security group rules are stateful. This means active connections enabled by the rules are tracked so that traffic can flow bidirectionally. Although UDP and ICMP are connectionless protocols, their “connection” is defined by the tuple. Stateful connection also has the somewhat surprising property that if you remove a rule, the existing connections enabled by rule continue to exist, until closed by either end of the connection. This is identical to AWS security groups behavior.
  2. Security group rules can allow access to VMs from other accounts: Suppose you have a shared monitoring service across accounts. The VMs in the monitoring service can belong to the cloud operator. Other tenants can allow access to them:
    • > authorize securitygroupingress securitygroupname=web account=operator usersecuritygrouplist=nagios,cacti protocol=tcp startport=12489 ...
  3. There is always a default security group: Just like EC2-classic, if you don’t place a VM in a security group, it gets placed in the default security group. Each account has its own default security group.
  4. Security group rules work between availability zones:  Security groups in an account are common across a region (multiple availability zones). Therefore, if the availability zones are routable (without NAT) to each other then the security groups work just as well between zones. This is similar to AWS EC2-classic security groups.
  5. Subnets are shared between accounts / VMs in a security group may not share a subnet. Although tenants cannot create or choose subnets in Basic networking, their VMs are placed in subnets (“Pods”) predefined by the cloud operator. The table below shows a sample of VMs belonging to two accounts spread between two subnets.
    • sg_groups_pptx4
  6. BUM traffic is silently dropped. Broadcast and multicast traffic is dropped at the VM egress to avoid attacks on other tenants in the same subnet. VMs cannot spoof their mac address either: unicast traffic with the wrong source mac is dropped as well.
  7. Anti-spoofing protection. VMs cannot spoof their mac address. VMs cannot send ARP responses for IP addresses they do not own. VMs cannot spoof DHCP server responses either. ARP is allowed only when the source MAC matches the VM’s assigned MAC. DHCP and DNS queries to the pod-local DHCP server are always allowed. If you run Wireshark/tcpdump within the VM you cannot see your neighbors traffic even though your NIC is set to promiscuous mode.
  8. Multiple IP addresses per VM: Once the VM is started you can request an additional IP for the VM (use the addIptoNic API).
  9. Live migration of the VM works as expected: When the operator migrates a VM, the security group rules move with the VM. Existing connections may get dropped during the migration.
  10. High Availability: As with any CloudStack installation, High Availability (aka Fast Restart) works as expected. When the VM moves to a different host, the rules move along with the VM.
  11. Effortless scaling: The largest CloudStack clouds (tens of thousands of nodes) use Basic networking. Just add more management servers.
  12. Available LBaaS: You can use a Citrix Netscaler to provide load balancing as well as Global Server Load Balancing (GSLB)
  13. Available Static NAT: You can use a Citrix Netscaler to provide Static NAT from a “public” IP to the VM IP.

There are limitations however when you use Basic Zone:

  1. Security groups function is only available on Citrix XenServer and KVM
  2. You can’t mix Advanced Networks and Basic Networks in the same availability zone, unlike AWS EC2
  3. You can’t add/remove security groups to a VM after it has been created. This is the same as EC2-classic
  4. No VPN functions are available.

The best way to deploy your Basic Zone is to engineer your physical network according to the same principles as web-scale operators. Read on

Back to Basics: CloudStack Basic Networking

The first choice to make when creating a zone in Apache CloudStack is the network type: basic or advanced. The blurb for “Advanced” promises “sophisticated network topologies”, while Basic promises “AWS-style networking”. Those who cut their teeth on the AWS cloud in 2008 may fondly remember what AWS now calls “EC2 classic platform“.

Platform Introduced In Description
EC2-Classic The original release of Amazon EC2 Your instances run in a single, flat network that you share with other customers.
EC2-VPC The original release of Amazon VPC Your instances run in a virtual private cloud (VPC) that’s logically isolated to your AWS account.

With a few differences, these map to CloudStack’s “Basic” and “Advanced” networking.

The fundamental network isolation technique in Basic zone is security groups. By default all network access to a VM is denied. When you launch an instance (VM), you deploy it in one or more security groups.

Using cloudmonkey:

> deploy virtualmachine securitygroupnames=management,web displayname=web0001 templateid=7464f3a6-ec56-4893-ac51-d120a71049dd serviceofferingid=48f813b7-2061-4270-93b2-c873a0fac336 zoneid=c78c2018-7181-4c7b-ab08-57204bc2eed3

Of course you have to create the security groups first:

> create securitygroup name=web
> create securitygroup name=management

Security groups are containers for firewall rules.

> authorize securitygroupingress  securitygroupname=web protocol=tcp startport=80 endport=80 cidrlist=0.0.0.0/0
> authorize securitygroupingress   securitygroupname=management protocol=tcp startport=22 endport=22 cidrlist=192.168.1.0/24

In a basic zone, all network access to a VM is denied by default. These two rules allow access to our VM on the HTTP port (80) from anywhere and on the SSH port (22) only from computers in the 192.168.1.0/24 subnet.

Let’s start another web VM with these security groups

> deploy virtualmachine securitygroupnames=management,web displayname=web0002 ...

We can log in to web0002 over ssh when our ssh client is in the 192.168.1.0/24 subnet. But when we try to login to web0001 from web0002 over ssh, we get denied, since neither of the ingress rules we wrote above allow that. We can fix that:

> authorize securitygroupingress   securitygroupname=management protocol=tcp startport=22 endport=22 securitygroupname=management

As long as the ssh client is on a VM in the management security group ssh access is allowed to any other VM in the management security group.

Let’s create some more:

Create appserver group and a db group

> create securitygroup name=appserver
> create securitygroup name=db

Let’s add these rules: Allow web VMs access to the app servers on port 8080. Allow app servers access to the DB VMs on the MySQL port (3306)

> authorize securitygroupingress   securitygroupname=appserver protocol=tcp startport=8080 endport=8080 securitygroupname=web
> authorize securitygroupingress   securitygroupname=db protocol=tcp startport=3306 endport=3306 securitygroupname=appserver

Deploy some virtual machines (instances) in these groups….

> deploy virtualmachine securitygroupnames=management,appserver displayname=app0001 ...
> deploy virtualmachine securitygroupnames=management,appserver displayname=app0002 ...
> deploy virtualmachine securitygroupnames=management,db displayname=db0001 ...

The network security architecture now looks like

sg_groups_pptx3

Pretty complicated, with just a handful of rules. The beauty of it is that it captures the intent accurately. After all, as a network admin you want to exactly say “allow app VMs access to the DB VMs on tcp port 3306”.

In a traditional network, you’d create subnets and insert security devices between them. On the security device you would have entered complicated ACLs. The ACLs may have to be changed any time you created / destroyed VMs. In a Basic Zone, once you define the groups and rules, everything is taken care of automatically. You can even edit the rules after the VMs are running. Let’s allow ICMP pings to all the VMs from our management subnet:

> authorize securitygroupingress securitygroupname=management protocol=icmp icmptype=-1 icmpcode=-1

To do the inverse:

> revoke securitygroupingress securitygroupname=management protocol=icmp ...

A significant difference from EC2-classic is that CloudStack allows you to create egress rules as well:

> authorize securitygroupegress securitygroupname=management protocol=icmp ...

This controls traffic out of the VMs. The egress rules only take effect the first time an egress rule is added to the security group. Once the first rule is added, egress is by default ‘deny all’ and only the specific egress rules allow traffic out of the VM.

Security group rules are stateful. This means that you don’t have to define a corresponding egress rule for every ingress rule. For example, when someone from the internet connects on port 80 to a web VM, response traffic (out of the web VM) associated with that connection is automatically allowed. Stateful connection tracking also apply to stateless protocols such as UDP and ICMP.

While ‘Basic’ Zone might seem well, basic, it offers powerful network isolation techniques that directly map your intent. The simple interface actually masks a sophisticated implementation which I hope to describe in a future post. I hope I have convinced you that ‘Basic’ is indeed sophisticated!

Read on for a deeper dive.

99 problems in my private cloud and networking is most of them

The state of private cloud is dire according to a number of pundits. Twitter’s de-facto cloud prognosticator warns: Do not build private clouds. Matt Asay declares private cloud to be a failure for a number of reasons, including the failure to change the way enterprises do business:

Private cloud lets enterprises pretend to be innovative, embracing pseudo-cloud computing even as they dress up antiquated IT in fancy nomenclature. But when Gartner surveyed enterprises that had deployed private clouds, only 5% claimed success

But he also lays blame on the most-hyped infrastructure technology of the past few years, OpenStack:

An increasing number of contributing companies are trying to steer OpenStack in highly divergent directions, making it hard for the newbie to figure out how to successfully use OpenStack. No wonder, then, that Joyent’s Bryan Cantrill hinted that the widespread failure of private clouds may be “due to OpenStack complexities.”

A large part of these complexities appear to be networking related:

No wonder most touted OpenStack successes have bespoke network architectures:

  • @WalmartLabs says they have 100k cores running, but

SDN is going to be our next step. Network is one area we need to put a lot of effort into. When you grow horizontally, you add compute, and the network is kind of the bottleneck for everything. That’s an area where you want more redundancy

  • Paypal runs a large (8500 servers) cloud, but uses VMWare’s NVP for networking
  • CERN runs a large OpenStack cloud but uses a custom network driver

In a different article, Matt Asay even cites industry insiders to state that OpenStack’s “dirty little secret” is that it doesn’t scale, largely due to broken networking.

In fact, as I’ve heard from a range of companies, a dirty secret of OpenStack is that it starts to fall over and can’t scale past 30 nodes if you are running plain vanilla main trunk OpenStack software

Frustrated cloud operators might look at the newest darling on the block to solve their complexities: Docker. At least it has a single voice and the much vaunted BDFL. Things should be better right? Well, not yet. Hopes are high, but both networking and storage are pretty much “roll your own”. There’s exotic options like Kubernetes, which pretty much only work in public clouds, SDN-like solutions (this, this, this, and more) and patchworks of proxies. Like the network operator needs yet another SDN solution rammed down her throat.

There is a common strand here: tone-deafness. Are folks thinking about how network operators really work? This lack of empathy sticks out like a sore thumb. If the solutions offered a genuine improvement to the state of networking then operators might take a chance at using something new. Network operators hoping to emulate web-scale operators such as AWS, Google and Facebook face a daunting task as well: private cloud solutions often add gratuitous complexity and take away none.

My favorite cloud software Apache CloudStack is not immune to these problems. The out-of-the-box network configuration is often a suboptimal choice for private clouds. Scalable solutions such as Basic Networking are ignored because, well, who wants something “basic”? In future posts, I hope to outline how private cloud operators can take architect their CloudStack networks for a better, scalable experience.

How dual-speed IT impacts private cloud architecture

An intriguing insight / hypothesis from Gartner is that IT can be more successful when it clearly demarcates ‘agile’ IT and ‘traditional IT”. According to Lydia Leong:

Traditional IT is focused on “doing IT right”, with a strong emphasis on efficiency and safety, approval-based governance and price-for-performance. Agile IT is focused on “doing IT fast”, supporting prototyping and iterative development, rapid delivery, continuous and process-based governance, and value to the business (being business-centric and close to the customer)

The idea is that “agile” IT is better served with cloud : either IAAS or PaaS while traditional IT could stick to their knitting and do business as usual. At some point, agile IT figures out how to do ‘cloud’ right and helps the other gang adopt the cloud. Of course, there’s dissent: Simon Wardley argues for trimodal IT, with the middle group mediating the extremes.

Lydia goes on to argue that:

Bimodal IT also implies that hybrid IT is really simply the peaceful coexistence of non-cloud and cloud application components — not the idea that it’s one set of management tools that sit on top of all environments.

Non-cloud application components are (my guess here) the domain of traditional IT, cloud application components are the domain of agile IT. The dichotomy also argues for 2 types of infrastructure: cloud and non-cloud.

A somewhat unrelated insight comes from Geoffrey Moore, that there’s 2 kinds of IT systems: Systems of Record (“Enterprise IT 1.0”) and Systems of Engagement (“the next stage of IT”). Systems of Record are:

global information systems that capture every dimension of our commercial landscape, from financial transactions to human resources to order processing to inventory management to customer relationship management to supply chain management to product lifecycle management, and on and on

Systems of engagement by contrast:

the focus instead will be on empowering the middle of the enterprise to communicate and collaborate across business boundaries, global time zones, and language and culture barriers, using next-generation IT applications and infrastructure adapted from the consumer space.

Systems of Record are the cost of doing business. They need to be highly optimized, low risk, rock solid and rely on a processes such as six sigma to deliver the quality and efficiency demanded by business. It is unlikely that these will be moved into the cloud in the near future.

The hypothesis (mine) here is that the systems of record are hosted on traditional IT / non-cloud infrastructure and private/public cloud hosts the systems of engagement.

Obviously the newer systems of engagement whether deployed on private clouds or public clouds may need access to the data held by the system of record.

If you have a private cloud for agile/systems of engagement, then the interaction looks like this:

Slide1If you use a public cloud for your systems of engagement, then it looks like:

Slide2

Yet another way to look at it might be the “pets vs. cattle” schema.

systems-of-record2_pets

Public clouds make this interconnection “easy” by providing required infrastructure. For example, AWS provides VPN Gateway and AWS Direct Connect. These facilities allow applications hosted on instances in the AWS cloud access resources that are “on-prem” (and vice-versa).

Theoretically the interconnect should be dead simple in the private cloud case. After all both parts of the infrastructure are hosted on the same local network infrastructure; presumably a single administrative domain. Complications can arise from:

  1. Business needs
  2. Artifacts of the private cloud implementation.

First the business needs: integrating systems of record and systems of engagement often involves crossing security boundaries: the former is guarded like Fort Knox; the latter has more fluid requirements. So, the solution might involve for example, inserting security devices in the path.

Slide2

The challenge is that the system on the right is extremely fluid: the network is constantly being reconfigured. Each change in the right might require changes in the security devices. The required level of network automation (to automate the security policy) is an unseen cost of implementing this architecture.

Private cloud networking brings its own complexities: it is often the most challenging part of implementing a private cloud. While the private cloud software stack might provide a solution that works within the cloud, it won’t provide a solution for the security policy automation problem mentioned above.

Bimodal IT is an interesting idea but can lead to ‘gaps’ between the modes, including the infrastructure domain. In a future post I hope to convince you that Apache CloudStack has some tricks up it’s sleeve to solve some of these problems.

How did they build that — EC2 Enhanced Networking

Among the flurry of new features introduced by AWS in 2013, is a performance enhancement known as ‘Enhanced Networking‘. According to the blurb: ” enhanced networking on your instance results in higher performance (packets per second), lower latency, and lower jitter’. The requirements are that you install an Intel 10GbE driver (ixgbef) in your instance and enable a feature called SR-IOV.

The AWS cloud is built around virtualization technology — specifically your instances are virtual machines running on top of a version of the open source Xen Hypervisor.
The hypervisor is what guarantees the isolation between my instance and your instance when they both run on the same set of CPUs.

The hypervisor intercepts all I/O from the virtual machine so that the virtual machine is abstracted from the hardware — this provides security as well as portability since the VM doesn’t need to care about the drivers for the I/O hardware. The VM sees a NIC that is software defined and as a result the hypervisor can inspect all traffic to and from the VM. -This allows AWS to control the networking traffic between the VM and the rest of the infrastructure. This is used to deliver features such as security groups and ACL.

The downside of processing all network traffic to/from the VM is that the host CPU cycles are consumed processing this traffic. This is quite a significant overhead compared to a bare-metal instance. The hypervisor needs to apply stateful firewall rules on every packet, switch the packet and encapsulate it. Some estimates put this overhead as high as 70% of the CPU available to the hypervisor (at 10 Gb/s). Software processing also introduces problems of noisy neighbors — variable jitter and high latency at 10Gbps are common.

Slide2

Fortunately, SR-IOV (Single Root IO Virtualization) provides a direct path for the VM to access the underlying hardware NIC. Bypassing the hypervisor leads to line-rate performance. Enhanced Networking takes advantage of this: in order to benefit from this, your AMI needs to have SR-IOV drivers installed in it.

Slide3

Great — but now that the hypervisor is out of the path, how does AWS provide software-defined features such as security groups and ACL? The current generation of SR-IOV NICs (AWS uses the Intel 82599 ) do not have stateful firewalls or the ability to have process large number of ACL. Furthermore, we know that AWS must be using some kind of encapsulation / tunnelling so that VPC are possible. The Intel 82599 does not provide encapsulation support.

The solution then would be to do the extra processing elsewhere — either off the host or in the host, using a co-processor. This schematic shows processing happening at the TOR switch. The drawback is that even intra-host traffic has to be tromboned via the TOR. Furthermore the switch now becomes a pretty big bottleneck and a failure in the switch could lead to several hosts losing network connectivity.Slide4

 

Using a co-processor would be the best solution. Tilera is one such processor that comes to mind. Since the Tilera provides general purpose processing cores, the encap/decap/filtering/stateful firewall processing could be done in software instead of ASICs or FPGAs.

Slide5

 

The software/hardware solution could allow AWS to introduce further innovations in its networking portfolio, including end-to-end encryption, IDS and IPS.

Disclaimer: I have no knowledge of AWS internals. This is just an exploration of “how did they build it?”.

Update: a confirmation of sorts on Werner Vogel’s blog: http://www.allthingsdistributed.com/2016/03/10-lessons-from-10-years-of-aws.html

Is AWS S3 the CDO of the Cloud?

The answer: not really, but the question needs examination.

One of the causes of the financial crisis of 2008 was the flawed ratings of complex financial instruments by supposed experts (ratings bodies such as S&P). Instruments such as CDOs comingled mortgages of varying risk levels, yet managed to get the best (AAA) ratings. The math (simplified) was that the likelihood of all the component mortgages failing at the same time was very low, and hence the CDO (more accurately the AAA rated portion of the CDO) itself was safe. For example if the probability of a single mortgage defaulting is 5%, then the probability of 5 mortgages defaulting at the same time is 1 in 3 million. The painful after effects of the bad assumptions underlying the math is still being felt globally, 5 years later.

If there is a AAA equivalent in the Cloud, it is AWS S3 which promises “11 9s of durability” (99.999999999%) or:

“if you store 10,000 objects with Amazon S3, you can on average expect to incur a loss of a single object once every 10,000,000 years”.

However, AWS does not give us the mathematical reasoning behind it. We have some clues:

“Amazon S3 redundantly stores your objects on multiple devices across multiple facilities in an Amazon S3 Region. The service is designed to sustain concurrent device failures by quickly detecting and repairing any lost redundancy”.

Let’s assume S3 stores 3 copies of each object in 3 datacenters that do not share the same failure domains. For example the datacenters each can have different sources of power, be located in different earthquake fault zones and flood zones. So, by design the possibility of 3 simultaneous failures is presumably low. Intuitively, if the probability of losing one object is 10-4, then the possibility of losing all 3 is (10-4)3 or 10-12. This is an oversimplification, there are other factors in the calculation such as the time to recover a failed copy. This graph (courtesy of Mozy) shows that the probability of losing an object is far lower with 3 copies (blue) than with 2 copies (yellow).

Image

The fatal flaw with the CDO ratings was that the failures of the component mortgages did chillingly correlate when the housing bubble burst : something that the ratings agency had not considered in their models. Perhaps the engineers and modelers at Amazon are far better at prediction science. But remember that the Fukushima nuclear disaster was also a result of a failure to account for an earthquake larger than 8.6. What we do know that at least in the US Standard region, AWS stores the object bi-coastally, so indeed the odds of a natural disaster simultaneously wiping out everything must indeed be quite low. There’s still manmade disasters (bugs, malicious attacks) of course. But given the uncertainty of such events (known unknowns), it is likely that the 10e-11 figure ignores such eventualities. Other AWS regions (EU, Japan, etc) do not store the objects with such wide geographic separation, but there are no caveats on the 10e-11 figure for those regions, so I’m guessing that the geographical separation in the U.S. Standard region does not figure in the 10e-11 calculation.

Interestingly none of S3’s competitors make similar claims. I can’t find Google Storage or Azure Storage making similar claims (or any claims). This riposte from the Google Storage team says

“We don’t believe that quoting a number without hard data to back it up is meaningful to our customers … we can’t share the kind of architectural information necessary to back up a durability number”.

Windows Azure storage SLA quotes an “availability” number of 99.9%. This availability number is same as that of Amazon S3.

Waitaminnit. What happened to the 11 9’s we saw above?

The difference between availability and durability is that while the object (at least 1 copy) might exist in AWS S3, it may not be available via the S3 APIs. So there’s a difference of 8 nines between the availability and durability of an S3 object.

Given the actual track record of AWS S3, it is perhaps time to revise the durability  estimates of this amazing service. If I were an enterprise or government organization considering moving my applications to the AWS cloud, I would certainly want to examine the math and modeling behind the figures. During congressional testimony, the ratings agencies defended their work by saying that it was impossible to anticipate the housing downturn: but clearly lots of people had anticipated it and made a killing by shorting those gold-plated CDOs. The suckers who trusted the ratings agencies blindly failed to do their own due diligence.

I do believe that storing your data in AWS S3 is less risky than your own data center and probably cheaper than storing it across your own multiple data centers. But it behooves one to use a backup: preferably with another provider such as Google Storage. As chief cloud booster Adrian Cockroft of Netflix says:

The SDN behemoth hiding in plain sight

Hint: it is Amazon Web Services (AWS). Let’s see it in action:

Create a VPC with 2 tiers: one public (10.0.0.0/24)  and one private (10.0.1.0/24). These are connected via a router. Spin up 2 instances, one in each tier (10.0.0.33 and 10.0.1.168).

[ec2-user@ip-10-0-0-33 ~]$ ping 10.0.1.168 -c 2
    PING 10.0.1.168 (10.0.1.168) 56(84) bytes of data.
    64 bytes from 10.0.1.168: icmp_seq=1 ttl=64 time=1.35 ms
    64 bytes from 10.0.1.168: icmp_seq=2 ttl=64 time=0.412 ms

The sharp-eyed might have noticed an oddity: the hop count (ttl) does not decrement despite the presence of a routing hop between the 2 subnets. So, clearly it isn’t a commercial router or any standard networking stack. AWS calls it an ‘implied router‘. What is likely happening is that the VPC is realized by creation of overlays. When the ethernet frame (ping packet) exits 10.0.0.33, the virtual switch on the hypervisor sends the ethernet frame directly to the hypervisor that is running 10.0.1.168 inside the overlay. The vswitches do not bother to decrement the ttl since that will cause an expensive recomputation of checksums in the ip header. Note that AWS introduced this feature in 2009 — well before open vswitch even had its first release.

One could also argue that security groups and elastic ips at the scale of AWS’s datacenters also bear the hallmarks of Software Defined Networking : clearly it required them to innovate beyond standard vendor gear to provide such to-date-unmatched L3 services. And these date back to the early days of AWS (2007 and 2008).

It doesn’t stop there. Elastic Load Balancing (ELB) from AWS orchestrates virtual load balancers across availability zones — providing L7 software defined networking. And that’s from 2009.

Last month’s ONS 2013 conference saw Google and Microsoft (among others) presenting facts and figures about the use of Software Defined Networking in their data centers. Given the far-and-away leadership of AWS in the cloud computing infrastructure space, it is interesting (to me) that AWS is seldom mentioned in the same breath as the pixie dust du jour “SDN”.

In CloudStack’s native overlay controller, implied routers has been on the wish list for some time. CloudStack also has a scalable implementation of security groups (although scaling to 100s of thousands of hypervisors might take another round of engineering). CloudStack also uses virtual appliances to provide L4-L7 services such as site-to-site IPSec VPN and load balancing. While the virtual networking feature-set in CloudStack is close to that of AWS, clearly the AWS implementation is likely an order of magnitude bigger.