IDG News Service >

Enterprise cloud put to the test

o Tom Henderson and Brendan Allen
05.04.2010 kl 19:44 |

The potential benefits of public clouds are obvious to most IT execs, but so are the pitfalls -- outages, security concerns, compliance issues, and questions about performance, management, service-level agreements and billing. At this point, it's fair to say that most IT execs are wary of entrusting sensitive data or important applications to the public cloud.


The potential benefits of public clouds are obvious to most IT execs, but so are the pitfalls -- outages, security concerns, compliance issues, and questions about performance, management, service-level agreements and billing. At this point, it's fair to say that most IT execs are wary of entrusting sensitive data or important applications to the public cloud.

How we tested these cloud computing products

Archive of Network World tests

But a technology as hyped as cloud computing can't be ignored either. IT execs are exploring the public cloud in pilot programs, they're moving to deploy cloud principles in their own data centers, or they are eyeing an alternative that goes by a variety of names -- enterprise cloud, virtual private cloud or managed private cloud.

We're using the term enterprise cloud to mean an extension of data center resources into the cloud with the same security, audit, and management/administrative components that are best practices within the enterprise. Common use cases would be a company that wanted to add systems resources without a capital outlay during a busy time of the year or for a special, resource-intensive project or application.

In this first-of-its-kind test, we invited cloud vendors to provide us with 20 CPUs that would be used for five instances of Windows 2008 Server and five instances of Red Hat Enterprise Linux -- two CPUs per instance. We also asked for a 40GB internal or SAN/iSCSI disk connection, and 1Mbps of bandwidth from our test site to the cloud provider. And we required a secure VPN connection.

Rackspace, Terremark and BlueLock accepted our invitation. Amazon did, then did not and refused to communicate further. The services we tested were comparable in many respects. Rackspace Managed Private Cloud scored points for cost transparency, a solid administrative portal and good overall performance. Rackspace was the slowest in many portions of the tasks we needed them to complete, although, to be fair, we were making requests that were outside of their traditional sales channels. Terremark Enterprise Cloud delivered speed and the best administrative portal, and also offered the lowest cost. The BlueLock Virtual Cloud offered strong processes and good administrative support, but was the most expensive.

Over the course of conducting this test, we learned several things. First, a customer can expect to have an enterprise cloud deployed and up and running within a week after the selection process is complete. Second, all of the vendors delivered strong security and comparable performance, albeit with vastly contrasting management components.

And, we found that enterprise cloud services can be expensive. We also discovered that each vendor seemed "squishy" on overall pricing. Our recommendation is to not assume that the enterprise cloud route is automatically cheaper than buying and provisioning your own servers. Do a thorough cost analysis and make sure to pin down your vendor when it comes to specific items like bandwidth.

Seeding the clouds

We contacted each vendor, described our requirements and waited for the proposals.

Each vendor has a different process to arrive at a quote for the resources we asked for, which amounted to a small subset to the wide array of possible offerings in each vendor's menu. While each vendor had a different list of options, there were many commonalities. Ordering virtual private cloud or enterprise cloud services meant getting dedicated machines with gear we wanted and a connectivity method that would link our network operations center at n|Frame in Indianapolis to the vendor's resources through VPN connectivity, which should be used as a demarcation point for both security and cost purposes.

BlueLock's hardware choices were among the narrowest, but they won points for having a thorough and deliberate quotation and subsequent provisioning process. They use forms made of Excel worksheets to exchange information, but the interactivity of information exchanged was thorough and well thought-through. By contrast, Rackspace offered the most flexibility in many ways.

Terremark's rapid speed of delivery (three days) earned the product high marks as it delivered quickly and to spec -- all things we like in a cloud vendor. But the other vendors weren't far behind -- BlueLock delivered in five days and Rackspace in six.


BlueLock has an openly published security process, which initially intrigued us, and we were reminded of an almost military provisioning process. We e-mailed them with our desired configuration, and Bluelock responded with a detailed proposal. Bluelock creates the offering from a source document build list. Once we said "go", Bluelock created the entire private cloud, operating systems deployment, initial security, IP routing, and so on. We didn't create the virtual machines and BlueLock provisioned the VMware instances (VMware 3.5 at this writing; 4.0 soon). We received dedicated hardware running on HP blades, which are their only hardware platform.

For connectivity via VPN and firewalling, BlueLock provided a CheckPoint SSL VPN whose administrative interface doesn't work with very many browser platforms; we tried various setups but only were able to get it to work in Windows XP and Internet Explorer (and Firefox 3.5 with Java installed). Windows 7 with IE8 or Firefox, Mac OS X 10.5/10.6.x with Safari, Firefox, did not work at all. Once inside CheckPoint, it works well and it's an enterprise class workhorse firewall and VPN. Bluelock was also able to pass our not-a-Cisco VPN test, by connecting to our Vyatta router/VPN appliance quickly.

The management interface to our 10 operating systems instances could have been better. There is no Web interface for accessing VMs (you can only connect to instances directly after connected through the SSL VPN or through IPSec site-to-site VPN, we tried both). Cloud administration was stiff. Bluelocks's own Vital Signs portal is a Web-based shell program that in turn calls other administrative applications. Vital Signs displays choices including a Vital Signs Diagram (which wasn't useful, as it shows a user count, and our agreement did not concern users, so it displayed - one user), and Event Monitoring Portal (the FOSS tool, Nagios), a Trend Portal (the FOSS tool Cacti), a non-working Reports screen, a Ticket and Support System (trouble ticket submission and process control), a portal user account maintenance facility, and FAQs.

Nagios is an open source network monitoring tool that we used to monitor network services such as http or mysql servers, along with whether the host is alive (ping test). We could also set alarms or notifications if a Nagio-tested service failed. The Cacti trend portal showed us virtual machine and firewall information. Cacti does a great job of showing time series sample graphs of CPU usage, network activity, memory usage and disk usage. We found Bluelock's Vital Signs Ticket and Support System to be frustrating, as it gave us only summarized information and no transaction or billing history. The Vital Signs portal isn't well connected, in terms of applications integration, as pieces can't be related together as objects in easy ways. While most of the discrete applications are useful, they're very disjointed.

We logged on to check BlueLock's administrative interface, then dove into forming our test suite, which consisted of installing LAMP/WAMP onto each OS instance that had been created. We checked Bluelock's performance with an Apache benchmark. It turned out that all of the vendors performed within a narrow window.

We tested storage expansion, which was simply a matter of submitting a new support ticket. And Bluelock configured the IPSec tunnel correctly -- except for our public IP, none of the resources could be seen, and the CheckPoint firewall and tunnel manager kept it that way.

BlueLock had a very fast connection to our NOC -- uploads at 7.26Mbps and downloads at 8.8Mbps. But it's also located only a few miles away from our n|Frame NOC resources (our subscribed bandwidth was 1Mbps burstable to 10Mbps).

Overall, BlueLock's negotiation process is good, and its security components were well-managed. The BlueLock administrative method had applications that feel like separate products. Nothing is really connected together, most portals launch in another browser window, some even require a separate login/password combo. Administration is unnecessarily confusing using these tools. And since BlueLock controls changes to the operating systems deployed, the time between ticket submission and a change could be considerable. We wanted to occasionally use our root account just to get things done.


Terremark's negotiation process is less formal than BlueLock's, although all of our private cloud metrics were met fully by Terremark. Terremark's hardware offerings are just slightly more expansive than those from BlueLock, as Terremark uses HP 580 and 585 servers. Terremark also offered us a variety of bundles that were pre-defined hardware/software asset combinations.

The build-time was shorter -- they were the first online and were ready-to-go quickly, although part of the speed came from the fact that Terremark didn't provision our instances of RedHat, and only offered Windows 2008 (not R2) server instances, with no maintenance, although it can be procured.

We told them the specs, they replied with a few questions, and in a couple of days, the components were built and we connected our NOC and the Terremark NOC. Terremark used virtual machines, like BlueLock, as the substrate for our requested network, and the connections to our Vyatta router/VPN appliance integrated quickly with their Cisco components.

Administrative interface

In the interest of time, Terremark had us provision our own virtual machines, which was a simple task. We were allocated the desired number of CPUs, RAM, disk and network for us to divide into the "shape" of the cloud we wanted. The Terremark-developed DigitalOps administrative app interface was used to deploy our Windows and Linux instances from one-click templates. Terremark supplied the Windows licenses (ostensibly from a volume license) and supplied Red Hat operating systems -- but we registered licenses supplied to us by Red Hat. Rollout, therefore, was drama-free and just 10 clicks for 10 instances. Terremark can optionally install everything for you at additional cost. We had the option of rolling out other types of server licenses operating systems from ISO images as well.

DigitalOps has a user interface that's separated into two main tabs, Environment and My Account. Under Environment there are three tabs: Resources, Devices and Network. The Resources tab displays information about processor, memory and storage usage. The main Resources page has a summary of each for the past 24 hours and is very easy to understand. We could get more detailed information by using the sub-tabs about each individual component (processor, memory, storage) if desired. The Devices tab lists all the virtual machines that we created, and the virtual machines can be sorted into groups and rows. We could create virtual machines from pre-built templates or create a blank server using our own ISO, as mentioned.

We could also use a VPN Connect button that allowed us to link to an SSL VPN (which is required to actually connect to the consoles of the virtual machines created). The final tab in the Environment section is network. Here we could view the IP networks assigned to us, internal, external and public IP addresses. We could also setup firewall and port-forwarding rules, although, they are very basic and we couldn't customize it very much.

Site to Site VPNs were a separate package deal, but possible to do using the IPSec protocol. Terremark only supports certain hardware or software VPNs, but they will do a "best effort" to try to get things working, if you have something different. We had something different, the aforementioned Vyatta appliance and we got the VPN working with minimal trouble. Once everything was setup, we ran some brief upload tests between our NOC and their servers. During an ISO transfer using scp, we maxed out around 120KBps (average). Normal FTP was about the same around 125KBps. The connection was limited to 1Mbit (not burstable) which is about 128KB, so it was pretty much maxing out the connection.

Terremark supplied an older VMware console plugin (which oddly doesn't work in Windows 7 under IE 8 or Firefox 3.6 but did work in Firefox 3.5.7) but none of the other competitors offered any option to connect to the virtual machines via their respective Web interfaces -- and Terremark did. This wasn't as much of an issue with the Windows virtual machines (meaning console virtual machine access) as the Windows Server virtual machines had Remote Desktop turned on to give us access. We had a few small quibbles with the templates used to generate the RHEL virtual machines, as the template did not create a user besides root (therefore, we couldn't SSH in, as root SSH is disabled by default).

The Terremark committed bandwidth pricing is complicated and is based on a "95th percentile" scheme, where they take the top 5% of your traffic for the month, drop that from calculations and use the final 95% of the bandwidth you used to figure out a price. You must purchase a Committed Bandwidth package. Ours was the 5Mbit package, which is $25 per Mbit, so $125 in total.

If you stay within the committed 5Mbit, you will only pay $125 a month. The extra charge comes in when you use more than your allocated bandwidth. Say you go over by 1Mbit for a total of 6Mbit, then you will have to pay 2x the Mbit fee (so $25 per Mbit would be $50 per Mbit for overage). Our total for the month would then be $175. Fortunately, Terremark allowed us to cap the bandwidth at 5Mbps for the VPN connection, which is all we used. According to the billing invoice our Committed Bandwidth was in the 5M to 50Mbps Tier but that does not apply to the VPN. The VPN bandwidth is a flat-rate per month based on connection speed and is not included in regular bandwidth calculations. They have the following tiers, 1Mbit = $200, 3Mbit = $550, 6Mbit = $1085, 10Mbit = $1285.

Overall, we liked Terremark's management app, and its speed to delivery. Provisioning was simple -- even though we did all of the virtual machines from the pool allocation allotted to us, and integration with our non-standard router was painless. We don't mind pain for gain, but it wasn't necessary with Terremark.


We were a little frustrated by Rackspace. Rackspace's process was slow, and may be faster for others as our negotiation and installation were done somewhat outside of their normal sales processes. The upside is that Rackspace's costs were more transparent and once rolling, their performance was very good. Rackspace provisioned us on Dell hardware, but emphasizes that most other top tier brands/models are available. We got the feeling that they're used to dealing on longer negotiation cycles with more diverse hardware needs, and deployment cycles associated with very large organizations.

Once the hardware and virtual machines had been provisioned, our site-to-site VPN took a while to integrate as well -- and much longer than the competition in our not-using-Cisco test. Once the VPN worked, it was smooth sailing, although IIS was installed on every Windows Server 2008 machine (we used Apache for testing), so we had to uninstall everything (IIS stuff) first. Some of these seeming disconnects could have been the result of our abnormal provisioning. The Red Hat Enterprise Linux virtual machines were correctly setup. As with BlueLock, Rackspace's virtual private cloud was fully provisioned on top of VMware ESX 3.5 by Rackspace so we didn't have to create the machines ourselves. There is a spot in the administrative Web interface to create new virtual machines (through a request), but this is limited to Windows Server 2003 and RedHat RHEL 3, 4 and 5. It's possible to have what you like (such as virtual machines), but you must submit a ticket for that with incumbent additional cost.

We liked the Rackspace administrative portal. The portal had six main divisions, Support, Products, Services, Network, Account (management) and Community. It's integrated, like Terremark's, and offers a tabular method of drilling down to support tickets, viewing each server resource utilization, viewing time series of performance characteristics, and administering our account. We found the Community tab interesting, as it took us to a private user forum. The forum is designed not be used for trouble tickets, rather communication among Rackspace clients for items like application integration, performance tweaks, and so on. This type of community-based communications was missing in BlueLock's and Terremark's offerings. It's like an internal user group.

Rackspace's communications with our n|Frame NOC was very fast, despite the long distance (Indianapolis to Austin) and we were happily surprised at the speed. Our ability to control VMs was also good, and we could manipulate our VMs readily although we couldn't actually connect to the console of the VM from an external (to the VPN) connection. It's also possible to review antivirus and URL monitors, but we didn't 'purchase' these. Interestingly, we could use the portal to buy SSL certificates (five types from VeriSign or two types from Thwate) -- very convenient, we thought.

We provisioned the Rackspace virtual machines for testing with our benchmark and connectivity tests. There were no mysteries, and Rackspace's Dell hardware performed well. We had no difficulties administering changes with Rackspace although gaps in their response were as mentioned, likely to have been the product of not being an actual customer.

We liked Rackspace and were it not for its slowness, we'd have liked the product much better, even though we know we were exceptions to their normal sales/fulfillment process. Rackspace's portal is useful, although with fewer choices than Terremark's and with a bit less functionality. As we seemed to have hurried them, we didn't get the full customer experience we were hoping for. Nonetheless, they were in the mid-range of pricing, and performed very well.


We asked each competitor to keep track of costs for us. Each competitor was a bit cagey and all three wanted to emphasize that costs are variable and tiered. They did, however, eventually get us pricing that reflected our utilization figures after we tested each private cloud with a performance analyzer to gauge CPU, bandwidth, VPN, storage, and other costs.

We also attempted to compare the three service providers with a do-it-yourself option -- in other words, buying hardware and software and deploying the apps on your own. With the comparison lies strong caveats. If one uses a DIY-type solution, there are hidden expenses involved that we didn't include in our estimate. These include support staffing, and leasehold costs, although we did include a collocation cost for power and space, at $45 per rack one unit per month prorated over the cost of the Dell hardware we chose in our DIY cost simulation. We also didn't include applications or application support, although these aren't covered by our competitors, either. Nor is the cost of negotiations, procurement, shipping, or building the hardware components included.

Our final caveat is that pricing appears to be a moving target, and a heavily guarded sales secret. The phrase "it depends" applies heavily here, as it became clear early in our review that we'd have to keep a taut line on our specification to allow even a close oranges-to-oranges comparison. And for those using virtual private clouds for availability, N+1 or 2N availability requires off-premises extensions of equipment, making DIY impractical.

Henderson is principal researcher and Allen is a researcher for ExtremeLabs in Indianapolis. They can be reached at

NW Lab Alliance

Henderson is also a member of the Network World Lab Alliance, a cooperative of the premier reviewers in the network industry each bringing to bear years of practical experience on every review. For more Lab Alliance information, including what it takes to become a member, go to

Read more about data center in Network World's Data Center section.

Keywords: Internet  
Latest news from IDG News Service

Copyright 2009 IDG Magazines Norge AS. All rights reserved

Postboks 9090 Grønland - 0133 OSLO / Telefon 22053000

Ansvarlig redaktør Henning Meese / Utviklingsansvarlig Ulf Helland / Salgsdirektør Tore Harald Pettersen