Comparing Amazon VPC connectivity options

In August 2009 Amazon announced its Virtual Private Cloud (VPC) service, essentially giving enterprise customers worried about security and control in the cloud a solution to that concern. Since then the Amazon VPC has matured as more and more services have become available from within the VPC.

Amazon Virtual Private Cloud allows IT administrators to provision a private, isolated section of the Amazon Web Services (AWS) Cloud where they can launch AWS resources in a virtual network that they define. They can have complete control over the virtual networking environment, including selection of IP address ranges, configuration of routing tables, subnets and network gateways.

Furthermore customers can connect their existing data centers and branch offices to the Amazon VPC and access the AWS cloud as if it is an extension of the corporate network. This connectivity between the corporate offices and the Amazon VPC can be accomplished in several ways.

In this short blog, we will explore the options available for connecting the enterprise network to the Amazon VPC whilst we compare and contrast the advantages, disadvantages and associated costs.

Amazon Direct Connect

AWS Direct Connect is an AWS service that allows you establish a dedicated network connection between your WAN network and the Amazon Web Service global network. If your corporate network has presence in one of these locations, Direct Connect facilitates dedicated 1G or 10G connectivity between your network equipment at that location and Amazon's routers.

Pricing information can be found here.

If connecting in London Telecity, a single 1G port will cost at least $223 per month for the port connection-hours. Additionally you pay $0.03 per GB for data transfers outbound from the VPC to the corporate network. Furthermore, if your corporate offices and datacenters are already reachable from the Direct Connect peering location across the enterprise WAN, only minimal configuration will be required to route traffic between the VPC and those offices.

Advantages

• Reduces bandwidth costs for traffic-heavy applications.
• Provides consistent network performance compared to other options.
• Can be used for accessing AWS services outside the VPC.

Disadvantages

• Requires existing network presence in a very limited set of locations.
• Requires more complex network hardware and configuration, for example 802.1q VLANs, BGP ..etc.
• If the traffic loads are not heavy enough, this is an expensive option.
• Not very elastic, the options are 1G or 10G ports, there is nothing in between. 

Varnish and Autoscaling... a love story

While working on a cool project at Cloudreach, I stumbled upon Varnish, and fell in love with it instantly. The first thing I tried to do was to combine Varnish with the awesomeness provided by AWS Elastic Load Balancer (ELB), in a combination which looks like:





While the frontend ELB works out of the box with Varnish (no surprises here), the backend ELB doesn’t work as expected with Varnish. The problem lies on the fact that Varnish is resolving the name assigned to the ELB, and it’s caching the IP addresses until the VCL get’s reloaded. Because of the dynamic nature of the ELB, the IPs linked to the cname can change at any time, resulting in Varnish routing traffic to an IP which is not linked to the correct ELB anymore.

The problem is discussed here and here but after Googling around I couldn't find any solution which didn’t involve doing:

ELB -> VARNISH -> NGINX (or HAproxy) ->  ELB -> AUTOSCALING GROUP

Going through so many layers seemed too much, taking into consideration that Varnish can be used to load balance requests and perform health checks on the backend nodes without the need for an Internal ELB. The more I thought about it, the more I realised how simple it would be to implement a solution..... so I did it. Using Varnish to perform the load-balancing, removes the overhead of going through an internal ELB, and it will require reloading the backend nodes only when an autoscaling activity takes place.


The solution I've implemented uses varnishadmin command line tool, boto, and some bash scripting to glue all together.

First of all we need to get the backend nodes configured in Varnish and store them on a file:

varnishadm -T $HOSTPORT -S $SECRET backend.list > varnish_ips

Then, we will have to query the autoscaling group, and update the backends if any instance has been added/terminated. The following Python code does most of the job:


Let’s break it down:

• get_autoscaling_ips gets the IPs associated with instances added to a specific autoscaling group.
• get_varnish_ips loads the backend IPs in a Python array
• update_vlc_file compares the two list of IPs. If there is any difference (you might want to reconsider this aspect) in the two lists of IPs, it creates a new VCL file containing the IPs retrieved from the autoscaling group.

In order to decouple the VCL section which is used to define request handling and document caching policies (unlikely to change according to the autoscaling group)  from the section which is used to configure the backends, the Python script outputs the new VCL in the following format:

include /etc/varnish/healthcheck.vcl;

node definitions

director definitions

include /etc/varnish/use.vcl

The node definition and the director definition is dynamically generated by the script, while healthcheck.vcl is a static file where the healthchek conditions are defined (what a surprise:) and use.vcl is another static Varnish config file, which makes use of the director definition.

Once the new VCL is generated, it’s just a matter of reloading it, running:

varnishadm -T $HOSTPORT -S $SECRET vcl.load $NAME $FILE
varnishadm -T $HOSTPORT -S $SECRET vcl.use $NAME

Something I noticed when creating the script, is that backend.list returns the list of the configured backends, regardless if the VCL which defines them is in use or not. This behaviour makes the all exercise of comparing VCL backends with autoscaling IPs useless, so we need to remove all the previous VCL configs running:

varnishadm -T $HOSTPORT -S $SECRET vcl.discard $OLD_VCL

The three scripts can be glued together on a bash script which runs as a cron job on each Varnish server. The code above has not been used in production yet, so please do test thoroughly before usage. II’m always curious to hear of any feedback, so get in touch if you have any comments on this.

As usual, please reach out to us if you need any help or advice using AWS!

Nicola Salvo

System Developer

Auto Scaling and Chef Nodes Deregistration

Creating an infrastructure which scales dynamically according to the traffic volumes hitting a web site, is a reality using Amazon Web Services (AWS). However, there are a number of key design points, which differentiate a professional solution from an ad hoc collection of services.

One of these design principles for auto scaling an application is to strive keep it stateless.

Sometimes this is not possible, for example when OpsCode Chef is used to configure and deploy code to instances created via an auto scaling process. In order to use Chef on a new instance, it needs to first register itself first with a central Chef server and resolve credentials and ssh access so that code and commands can be executed.
The problem with registering autoscaling nodes, is that when those are terminated for one reason or the other, Chef Server is left in a “dirty” status, with references to nodes which are no longer active.

Fortunately, the AWS team have enhanced their platform with lots of additional services complementing EC2, two of which are useful to solve the problem described above.

When an action is triggered in an auto scaling group, this can generate a notification using the Amazon Simple Notification Service (SNS). SNS works across all the services offered by AWS, and an auto scaling group can be configured in order to generate an outbound notification for both scale up and scale down events.
SNS receives and publishes events using “topics”.  A topic is a communication channel to send messages and subscribe to notifications. It provides an access point for publishers and subscribers to communicate with each other.

An SNS topic can be created very easily using the web UI, or using the command line:

sns-create-topic MyTopic

Once a topic for autoscaling events is created, the auto scaling group needs to know where to publish notifications. This can be achieved in one of two ways; either using the command line interface:

as-put-notification-configuration MyGroup --topic-arn arn:placeholder:MyTopic --notification-types autoscaling: autoscaling:EC2_INSTANCE_TERMINATE topic-ARN MyTopic

or using CloudFormation:

"NotificationConfiguration" : {
             "TopicARN" : { "Ref" : "MyTopic" },
             "NotificationTypes" : [ "autoscaling:EC2_INSTANCE_TERMINATE"]
       },


Notifications by themselves are not very useful if something doesn’t consume them and act upon them. A naive approach would be to use email or SMS alerts and manually remove instances using the Chef user interface. Fortunately we can do better than this, and instruct machines (or scripts) to perform this (boring!) task on our behalf.

The method of consuming an SNS notification programmatically is to use Amazon Simple Queue Service (SQS) as an endpoint to SNS. The SNS topic can be configured in order to act as a producer for an SQS queue.

Once SNS and SQS are linked together, it’s just a matter of writing a consumer which is able to read EC2_INSTANCE_TERMINATE messages, and remove the node from Chef.

Given that Chef uses Ruby as language to define “recipes”, I choose to use FOG in order to leverage the SQS RESTful API.

Before retrieving the messages in the queue, authentication needs to take place.

sqs = Fog::AWS::SQS.new(
        :aws_access_key_id => access_key_id,
        :aws_secret_access_key => secret_access_key,
        :region => "eu-west-1"
       )


If the authentication is successful, the method receive_message will return a list containing  the number of messages specified as a parameter.

response = sqs.receive_message(QUEUE_URL, { 'Attributes' => [], 'MaxNumberOfMessages' => 10 })

From this point on, is a matter of parsing the JSON message, and if the event is a termination notification deleting the instance.

messages = response.body['Message']
unless messages.empty?
   messages.each do |m|
   body = JSON.parse(m['Body'])
   message = JSON.parse(body["Message"])
   if message["Event"].include? AUTOSCALING_NOTIFICATION["Terminate"]


In order to keep the code simple, the node is removed by invoking knife from the command line, so that there is no need to handle the authentication:

instance_id = message["EC2InstanceId"]
delete_node  = "knife node delete #{instance_id} -y"
delete_client= "knife client delete #{instance_id} -y"
output = `#{delete_node}`
result=$?.success?
if result == true
   result = `#{delete_client}`
end

        
Once a message is processed successfully, it can be removed from the queue:

sqs.delete_message(QUEUE_URL, m['ReceiptHandle'])

In order to work in an enterprise production environment, the script needs few additions, but this should be enough to get you started!

Alternatively, come and talk to the experts at Cloudreach.


Nicola Salvo
System Engineer
Cloudreach Limited

A small step in the cloud, a giant leap for the datacentre

Yesterday Amazon Web Services announced the ability for their Virtual Private Cloud (VPC) product to send and received traffic directly from the internet as well as that routed via the private site-to-site connection to your on-premise router. Although this may seem like a small step forward, it is in truth a transforming feature in the maturity of cloud infrastructure. To date the features inherent to virtualisation that underpin AWS and other clouds have rewarded adopters with an abstraction of hardware from their running machines. The benefit include mobility (in the case of a hardware problem) and fully atomic backups that are very useful when rolling out major patch updates or introducing significant change on a core service. The main limiting factor for anyone wishing to deploy complex enterprise environments into AWS has been the flexibly of the networking elements. Those who require network segmentation, outbound network ACLs and inter-network connectivity were limited by the fact that unless you were inside VPC you had limited control of the subnet in which your machines run. The downside of running in VPC was that all traffic then had to route over your site-to-site connection to your router. With this announcement you are now free to enjoy an enhanced VPC that allows secure access from the internet and a high-throughput secure connection to your datacentre.

From a Cloudreach point of view this means the near-death of our much loved LAN-to-cloud VPN service that we use in production and disaster recovery deployments. Although this OpenVPN based service is in the main part superseded, it still has a place in environments where terminating infrastructure does not support BGP peering within IPSEC tunnels. As a company we very much welcome this extension of the AWS feature set as it moves the platform on and keeps it well out of reach of its nearest competitors who were already scrabbling to keep up with the existing feature set. In our mind it opens up the possibility of the truly virtual datacentre with features that match or exceed the functionality of the legacy best practise Cisco/VMWare/NetApp solutions, without the hassle of running and maintaining complex kit.

For those of us who are used to seeing our servers and Network Attached Storage as lines on a web-page, we can get excited about the future where as well as our servers we will be able to add virtual network appliances into our subnets which exhibit equivalent functionality to traditional equipment. I say watch out Cisco …  the world is going virtual, and a vendor will emerge with equivalent functionality to your GSR that is designed as a software only product to run within cloud infrastructure. The Cisco Nexus project is a great idea, but you still need a bloody great bit of Cisco kit to control your virtual appliance. Something seems wrong about that! Our current excitement is for companies like Zeus who provide a world-class Layer 7 switch extending their offering to offer Layer 3-6 functionality in this virtual world.

The functionality released into AWS means that almost all deployments from now on will run within VPC containers and many of these will use the advanced networking without the site-to-site secure link back to your datacentre, just to take benefit of fixed IP addresses.  Up until now the default gateway of any of your instances had to be the AWS routing infrastructure, and not your own server acting as a router, which made deploying things like client VPNs for mobile workers a little tricky. You can now create multiple subnets within a VPC deployment and control the inbound and outbound network traffic that transits between them. It just makes you think, why bother doing this stuff yourself?

In the history of Amazon Web Services we see this announcement as even more fundamental than the Dec 2009 update that your virtual servers could now be persistent and stopped and started freely. This small step in the AWS cloud is truly a giant leap for the datacentre as we know it.

James Monico
Technical Director

Keep Calm and Carry On - Microsoft Windows License Activation in AWS

The AWS Cloud has now become a familiar pat of our cloud existence but some of you may have come across a few problems that are not immediately obvious from start of use. One particularly funny (as in ‘strange’, not ‘haha’) problem you can find in Amazon is the Genuine Advantage Program from Microsoft. This tells you that your instance’s licence is not valid.

First impressions on seeing this are usually, “Wasn't this supposed to be managed by Amazon?” and obviously “Why are they asking me to activate my Windows license?” The fact is that Amazon actually does take care of this problem.

Namely:

• On first start up Amazon Ec2WindowsActivate service registers the copy of Windows and sets the activation server link.
• Later the instance can reconfirm its license by connecting these servers.

At the time of writing these are:

us-east-1:

ec2-174-129-233-152.compute-1.amazonaws.com.

ec2-174-129-233-141.compute-1.amazonaws.com.

us-west-1:

ec2-204-236-129-123.us-west-1.compute.amazonaws.com.

ec2-204-236-129-122.us-west-1.compute.amazonaws.com.

eu-west-1:

ec2-79-125-16-172.eu-west-1.compute.amazonaws.com.

ec2-79-125-16-108.eu-west-1.compute.amazonaws.com.

ap-southeast-1:

ec2-175-41-130-16.ap-southeast-1.compute.amazonaws.com

ec2-175-41-130-20.ap-southeast-1.compute.amazonaws.com

All now seems pretty logical - but then... “Why am I getting the black wallpaper, and why doesnt Windowsre-activate?” The key point is that DNS names can only  be resolved by the internal Amazon DNS. So if you change your Windows DNS servers to some others (your Active Directory ones for example) your server won't be able to resolve it.

So far there are two solutions;

• You can restore Amazon DNS in the machine configuration, or   
• You run manually (or scripted) 'slmgr.vbs /skms IP ' and 'slmgr.vbs /ato'  commands in CLI after resolving the DNS request against the Amazon DNS and getting the actual internal IP you instance should point to.

Hope that helps you all Cloud users.

Emilio Garcia

Senior Engineer

Cloudreach

Amazon SES: A Quick First Look

Amazon Web Services seem to pull out new cards from their collective sleeves almost every single day. This is of course a source of great joy at Cloudreach HQ - in addition to enabling us to provide better value-added services to our clients, the geeks among us like it because it’s yet another toy to have a play with.

Today’s toy is Amazon Simple Email Service (Beta), a highly scalable and cheap bulk email sending service. This came to us at quite an interesting time as we had just built a Postfix email server for one of our clients to send bulk emails from, and having to battle ISPs who were suspicious of our client’s motives.

Amazon SES takes away a large amount of the complexity involved in sending bulk emails; you do not need to build your own email server to be able to send bulk emails.

As SES is still in Beta, you cannot access it from the console. You can however use a bunch of perl scripts they provide you with to get started. The rest of this blogpost describes how this may be done:

1. Sign up for Amazon SES. You can do this by visiting http://aws.amazon.com/ses
2. You will now have access to the Amazon SES developer sandbox using the perl scripts provided. The perl scripts use the Access Key and Secret Access Key for authentication. Copy them into a file as follows, and make sure the file’s permissions are set to 600 (Linux).AWSAccessKeyId=xxxxxxxxxxxxxxx
   AWSSecretKey=xxxxxxxxxxxxxxxxxxx
   
   See the Getting Started guide here from AWS for details.
3. However, to get the perl scripts working on Ubuntu, you will need to install the following modules:libxml-libxml-perl
   libcrypt-ssleay-perl
   perl-doc
   libxml2-dev
   
4. You will then need to verify your email address and any other email addresses you wish to send emails to (because you’re still sandboxed, you cannot send emails to email addresses you haven’t verified). The developer’s guide has details on how to do this.
5. To send emails to all of your verified email addresses, you can use this simple Python wrapper script I’ve knocked together. You can also download it from our public Amazon S3 bucket by clicking here.

To learn how to execute this file, type python send_email.py -h.

Now that you’re done playing with the SES sandbox, you might want to request Amazon Web Services to grant you production access by filling this form up. AWS indicate most requests will be approved within 24 hours; however, we can’t comment on that just yet because we got our hands on this just a couple of hours ago!!

Siddhu Warrier

www.cloudreach.co.uk

Cloud Computing - The New Rock and Roll

So a new year has begun and its out with the old and in with the new. Well for the most part. It was during my post-Christmas clean-up yesterday that I found myself furtively slipping X-Factor’s latest offering into the lonely depths of my bottom CD drawer. Another act, another fad. Yet, they are called today’s Rockstars. Surely, such a term should be saved for the Bowies and REMs of this world, demonstrating resilience and evolution through the years, sticking it out when the going was tough and long before the recognition, hype and groupies set it? Well at least there is a rightful new chart-topper in the world of Business Intelligence and whats more, this one is here to stay. 

A recent study by Cisco IBSG(1), estimated that 12% of enterprise workload will run in the cloud by 2013 and that the cost-savings from using cloud services will benefit the UK economy alone to the tune of 30 Billion Euros (2) over the next five years. Yet, there was no Simon Cowell back in the 60’s when the earliest hint of utility computing fell on the deaf ears of a technologically primitive world. Only in the 90’s amidst the clang of Blur and Oasis was it that a major record label of the IT world, Amazon, signed up to the idea and put cloud computing to the wider audience with the launch of Amazon Web Services (AWS) in 2002. The rest is history and roll on five years to find even Google and IBM amongst the fans in the front row with their offerings of Google Apps(3) and IBM Smart Business Development and Test(4). 

The fact is, cloud computing has been around for a while and only now with years of sound iterative development and millions of trusted users, has the Hype finally set in. With the immediate forecast being a year-on-year growth rate of 43% (5), cloud computing’s fan-base is set to increase relentlessly. And boy is it going to be some show.

(1)
http://www.cisco.com/web/about/ac79/docs/wp/sp/Service_Providers_as_Cloud_Providers_IBSG.pdf

(2)
http://www.pcr-online.biz/news/35321/Cloud-to-benefit-UK-economy-by-30-billion-Euros

(3)
http://www.google.com/apps/intl/en/business/index.html

(4)
http://www-935.ibm.com/services/us/igs/cloud-development/

(5)
http://www.newinnovationsguide.com/Cloud.html


Fabien Flight
Senior Engineer - Cloudreach