How HP Labs nearly invented the cloud

On the heels of HP’s news of not-quite abandoning the Cloud, there is coverage of how AWS stole a march on Sun’s plans to provide compute-on-demand. The timeline for AWS starts late 2003 when an internal team in Amazon hatched a plan that among other things could offer virtual servers as a retail offering. Sun’s offering involved bare metal and running jobs, not virtual machines.

In a paper published in 2004 a group of researchers at HP Labs proposed what they called “SoftUDC” – a software-based utility data center. The project involved:

  • API access  to virtual resources
  • Virtualization using the Xen Hypervisor
  • Network virtualization using UDP overlays almost identical to VxLAN
  • Virtual Volumes accessible over the network from any virtual machine (like EBS)
  • “Gatekeeper” software in the hypervisor that provides the software and network virtualization
  • Multi-tier networking using subnetting and edge appliances (“VPC”)
  • Automated OS and application upgrades using the “cattle” technique (just replace instead of upgrade).
  • Control at the edge: firewalls, encryption and QoS guarantees provided at the hypervisor

Many of these ideas now seem “obvious”, but remember this was 2004. Many of these ideas were even implemented. For example, VNET is the name of the network virtualization stack / protocol. This was implemented as a driver in Xen dom0 that would take Ethernet frames exiting the hypervisor and encapsulate them in UDP frames.

Does this mean HP could have been the dominant IAAS player instead of AWS if it only had acted on its Labs innovation? Of course not. But, lets say in 2008 when AWS was a clear danger, it could’ve dug a little deeper inside its own technological inventory to produce a viable competitor early on.  Instead we got OpenStack.

Many of AWS’s core components are based on similar concepts: the Xen hypervisor, network virtualization, virtual volumes, security groups, and so on. No doubt they came up with these concepts on their own — more importantly they implemented them and had a strategy for building a business around it.

Who knows what innovations are cooking today in various big companies, only to get discarded as unviable ideas. This can be framed as the Innovator’s Dilemma as well.

Advertisements

How to manage a million firewalls – part 1

In my last post I argued that security groups eliminate the need for network security devices in certain parts of the datacenter. The trick that enables this is the network firewall in the hypervisor. Each hypervisor hosts dozens or hundreds of VMs — and provides a firewall per VM. The figure below shows a typical setup, with Xen as the hypervisor. Ingress network traffic flows through the hardware into the control domain (“dom0”) where it is switched in software (so called virtual switch or vswitch) to the appropriate VM.

sg_groups_pptx7

The vswitch provides filtering functions that can block or allow certain types of traffic into the VM. Traffic between the VMs on the same hypervisor goes through the vswitch as well. The vswitch used in this design is the Linux Bridge; the firewall function is provided by netfilter ( “iptables”).

Security groups drop all traffic by default and only allow those configured by the rules. Suppose the red VMs in the figure (“Guest 1” and “Guest 4”) are in a security group “management”. We want to allow access to them from the subnet 192.168.1.0/24 on port 22 (ssh). The iptables rules might look like this:

iptables -A FORWARD -p tcp --dport 22 --src 192.168.1.0/24 -j ACCEPT 
iptables -A FORWARD -j DROP

Line 1 reads: for packets forwarded across the bridge (vswitch) that are destined for port 22, and are from source 192.168.1.0/24, allow (ACCEPT) them. Line 2 reads: DROP everything. The rules form a chain: packets traverse the chain until they match. (this is highly simplified: we want to match on the particular bridge ports that are connected to the VMs in question as well).

Now, let’s say we want to allow members of the ‘management’ group access their members over ssh as well. Let’s say there are 2 VMs in the group, with IPs of ‘A’ and ‘B’.  We calculate the membership and for each VM’s firewall, we write additional rules:

#for VM A
iptables -I FORWARD -p tcp --dport 22 --source B -j ACCEPT
#for VM B
iptables -I FORWARD -p tcp --dport 22 --source A -j ACCEPT

As we add more VMs to this security group, we have to add more such rules to each VM’s firewall. (A VM’s firewall is the chain of iptables rules that are specific to the VM).  If there are ‘N’ VMs in the security group, then each VM has N-1 iptables rules for just this one security group rule. Remember that a packet has to traverse the iptables rules until it matches or gets dropped at the end. Naturally each rule adds latency to a packet (at least to the connection-initiating ones).  After a certain number (few hundreds) of rules, the latency tends to go up hockey-stick fashion. In a large cloud, each VM could be in several security groups and each security group could have rules that interact with other security groups — easily leading to several hundred rules.

Aha, you might say, why not just summarize the N-1 source IPs and write a single rule like:

iptables -I FORWARD -p tcp --dport 22 --source <summary cidr> -j ACCEPT

Unfortunately, this isn’t possible since it is never guaranteed that the N-1 IPs will be in a single CIDR block. Fortunately this is a solved problem: we can use ipsets. We can add the N-1 IPs to a single named set (“ipset”). Then:

ipset -A mgmt <IP1>
ipset -A mgmt <IP2>
...
iptables -I FORWARD -p tcp --dport 22 -m set match-set mgmt src -j ACCEPT

IPSets matching is usually very fast and fixes the ‘scale up’ problem. In practice, I’ve seen it handle tens of thousands of IPs without significantly affecting latency or CPU load.

The second (perhaps more challenging) problem is that when the membership of a group changes, or a rule is added / deleted, a large number of VM firewalls have to be updated. Since we want to build a very large cloud, this usually means thousands or tens of thousands of hypervisors have to be updated with these changes. Let’s say in the single group/single rule example above, there are 500 VMs in the security groups. Adding a VM to the group means that 501 VM firewalls have to be updated. Adding a rule to the security group means that 500 VM firewalls have to be updated. In the worst case, the VMs are on 500 different hosts — making this a very big distributed systems problem.

If we consider a typical datacenter of 40,000 hypervisor hosts, with each hypervisor hosting an average of 25 VMs, this becomes the million firewall problem.

Part 2 will examine how this is solved in CloudStack’s Basic Zone.

Back to Basics: CloudStack Basic Networking

The first choice to make when creating a zone in Apache CloudStack is the network type: basic or advanced. The blurb for “Advanced” promises “sophisticated network topologies”, while Basic promises “AWS-style networking”. Those who cut their teeth on the AWS cloud in 2008 may fondly remember what AWS now calls “EC2 classic platform“.

Platform Introduced In Description
EC2-Classic The original release of Amazon EC2 Your instances run in a single, flat network that you share with other customers.
EC2-VPC The original release of Amazon VPC Your instances run in a virtual private cloud (VPC) that’s logically isolated to your AWS account.

With a few differences, these map to CloudStack’s “Basic” and “Advanced” networking.

The fundamental network isolation technique in Basic zone is security groups. By default all network access to a VM is denied. When you launch an instance (VM), you deploy it in one or more security groups.

Using cloudmonkey:

> deploy virtualmachine securitygroupnames=management,web displayname=web0001 templateid=7464f3a6-ec56-4893-ac51-d120a71049dd serviceofferingid=48f813b7-2061-4270-93b2-c873a0fac336 zoneid=c78c2018-7181-4c7b-ab08-57204bc2eed3

Of course you have to create the security groups first:

> create securitygroup name=web
> create securitygroup name=management

Security groups are containers for firewall rules.

> authorize securitygroupingress  securitygroupname=web protocol=tcp startport=80 endport=80 cidrlist=0.0.0.0/0
> authorize securitygroupingress   securitygroupname=management protocol=tcp startport=22 endport=22 cidrlist=192.168.1.0/24

In a basic zone, all network access to a VM is denied by default. These two rules allow access to our VM on the HTTP port (80) from anywhere and on the SSH port (22) only from computers in the 192.168.1.0/24 subnet.

Let’s start another web VM with these security groups

> deploy virtualmachine securitygroupnames=management,web displayname=web0002 ...

We can log in to web0002 over ssh when our ssh client is in the 192.168.1.0/24 subnet. But when we try to login to web0001 from web0002 over ssh, we get denied, since neither of the ingress rules we wrote above allow that. We can fix that:

> authorize securitygroupingress   securitygroupname=management protocol=tcp startport=22 endport=22 securitygroupname=management

As long as the ssh client is on a VM in the management security group ssh access is allowed to any other VM in the management security group.

Let’s create some more:

Create appserver group and a db group

> create securitygroup name=appserver
> create securitygroup name=db

Let’s add these rules: Allow web VMs access to the app servers on port 8080. Allow app servers access to the DB VMs on the MySQL port (3306)

> authorize securitygroupingress   securitygroupname=appserver protocol=tcp startport=8080 endport=8080 securitygroupname=web
> authorize securitygroupingress   securitygroupname=db protocol=tcp startport=3306 endport=3306 securitygroupname=appserver

Deploy some virtual machines (instances) in these groups….

> deploy virtualmachine securitygroupnames=management,appserver displayname=app0001 ...
> deploy virtualmachine securitygroupnames=management,appserver displayname=app0002 ...
> deploy virtualmachine securitygroupnames=management,db displayname=db0001 ...

The network security architecture now looks like

sg_groups_pptx3

Pretty complicated, with just a handful of rules. The beauty of it is that it captures the intent accurately. After all, as a network admin you want to exactly say “allow app VMs access to the DB VMs on tcp port 3306”.

In a traditional network, you’d create subnets and insert security devices between them. On the security device you would have entered complicated ACLs. The ACLs may have to be changed any time you created / destroyed VMs. In a Basic Zone, once you define the groups and rules, everything is taken care of automatically. You can even edit the rules after the VMs are running. Let’s allow ICMP pings to all the VMs from our management subnet:

> authorize securitygroupingress securitygroupname=management protocol=icmp icmptype=-1 icmpcode=-1

To do the inverse:

> revoke securitygroupingress securitygroupname=management protocol=icmp ...

A significant difference from EC2-classic is that CloudStack allows you to create egress rules as well:

> authorize securitygroupegress securitygroupname=management protocol=icmp ...

This controls traffic out of the VMs. The egress rules only take effect the first time an egress rule is added to the security group. Once the first rule is added, egress is by default ‘deny all’ and only the specific egress rules allow traffic out of the VM.

Security group rules are stateful. This means that you don’t have to define a corresponding egress rule for every ingress rule. For example, when someone from the internet connects on port 80 to a web VM, response traffic (out of the web VM) associated with that connection is automatically allowed. Stateful connection tracking also apply to stateless protocols such as UDP and ICMP.

While ‘Basic’ Zone might seem well, basic, it offers powerful network isolation techniques that directly map your intent. The simple interface actually masks a sophisticated implementation which I hope to describe in a future post. I hope I have convinced you that ‘Basic’ is indeed sophisticated!

Read on for a deeper dive.

The SDN behemoth hiding in plain sight

Hint: it is Amazon Web Services (AWS). Let’s see it in action:

Create a VPC with 2 tiers: one public (10.0.0.0/24)  and one private (10.0.1.0/24). These are connected via a router. Spin up 2 instances, one in each tier (10.0.0.33 and 10.0.1.168).

[ec2-user@ip-10-0-0-33 ~]$ ping 10.0.1.168 -c 2
    PING 10.0.1.168 (10.0.1.168) 56(84) bytes of data.
    64 bytes from 10.0.1.168: icmp_seq=1 ttl=64 time=1.35 ms
    64 bytes from 10.0.1.168: icmp_seq=2 ttl=64 time=0.412 ms

The sharp-eyed might have noticed an oddity: the hop count (ttl) does not decrement despite the presence of a routing hop between the 2 subnets. So, clearly it isn’t a commercial router or any standard networking stack. AWS calls it an ‘implied router‘. What is likely happening is that the VPC is realized by creation of overlays. When the ethernet frame (ping packet) exits 10.0.0.33, the virtual switch on the hypervisor sends the ethernet frame directly to the hypervisor that is running 10.0.1.168 inside the overlay. The vswitches do not bother to decrement the ttl since that will cause an expensive recomputation of checksums in the ip header. Note that AWS introduced this feature in 2009 — well before open vswitch even had its first release.

One could also argue that security groups and elastic ips at the scale of AWS’s datacenters also bear the hallmarks of Software Defined Networking : clearly it required them to innovate beyond standard vendor gear to provide such to-date-unmatched L3 services. And these date back to the early days of AWS (2007 and 2008).

It doesn’t stop there. Elastic Load Balancing (ELB) from AWS orchestrates virtual load balancers across availability zones — providing L7 software defined networking. And that’s from 2009.

Last month’s ONS 2013 conference saw Google and Microsoft (among others) presenting facts and figures about the use of Software Defined Networking in their data centers. Given the far-and-away leadership of AWS in the cloud computing infrastructure space, it is interesting (to me) that AWS is seldom mentioned in the same breath as the pixie dust du jour “SDN”.

In CloudStack’s native overlay controller, implied routers has been on the wish list for some time. CloudStack also has a scalable implementation of security groups (although scaling to 100s of thousands of hypervisors might take another round of engineering). CloudStack also uses virtual appliances to provide L4-L7 services such as site-to-site IPSec VPN and load balancing. While the virtual networking feature-set in CloudStack is close to that of AWS, clearly the AWS implementation is likely an order of magnitude bigger.