Essential Technology: A GCS Blog

Being called "a worst case scenario of cloud computing" on Twitter demanded I pick this book up. It is actually a terrifyingly good read that demonstrates Suarez's technical knowledge as well as his ability to craft a great thriller. In his fantasy world thousands of corporate networks are penetrated with an advanced botnet. Oh wait, that's the real world too. In Daemon the botnet is controlled by a deceased game developer with enough money to ensure his ambitions persist beyond the grave. These ambitions include murder, mayhem, extortion, and more. Good times!

A few thoughts:

1) Though much of the technology is still in early stages of adoption, it exists today. If you can hunt a deer over the internet, you obviously can kill a man.

2) Security breaches of the sort required to perpetrate a more realistic version of this attack occur constantly.

3) The book incorporates interesting socio-economic themes as well. Suarez is obviously under the impression that private industry exercises near absolute control over our government. I, for one, welcome our corporate overlords.

4) The disaffection of corporate IT employees from the business was another key theme. Similar to Heinlein's "The Roads Must Roll" in which a technical class realizes the power they can exert over those dependent on their abilities. A massive, unionized strike flexes their muscle. If you're an IT executive this should be yet another thing to keep you awake at night.

5) For anyone not using offline storage such as tape, read this book. Offline storage is a critical last defense against many attacks. Unfortunately a large percentage of IT execs don't value it to the extend it demands. In much the same way accounting standards dictate separation of tasks, offline backup tasks should be split from normal IT tasks and, where possible, from your IT staff. Let offline storage be your panic room.

Check this book out soon. The next installment of the story is released in January. I hope there are a dozen in this series.

I was invited to be a guest speaker at the Texas Organization for Rural Community Hospitals' weekly webinar. This week's topic was an overview of IT Security for IT Executives and managers. My presentation covered basic risk assessment, common types of attacks and prevention strategies. I discussed several up and coming technologies such as application whitelisting, layer 7 firewalls, and network access protection. I also discussed various security standards such as SAS 70, PCI, and HIPAA.

As promised to the attendees, here is the slide deck:IT Security Overview.ppt (6.67 mb)

Cloud computing is already disrupting staffing, procurement and support practices in IT departments around the world. GCS' Early Happy Hour event will provide insight into the coming changes to your business practices. We'll profile solutions from vendors such as Amazon, Microsoft, Rackspace, Terremark and more. We'll look at private clouds and public clouds while breaking down the differences between I-a-a-S, P-a-a-S, and S-a-a-S.

We will present a high level strategy discussion on the topic of cloud computing. We hope that our attendees will share their cloud experiences and opinions with us in an open format.

The Early Happy Hour will be Thursday, December 3rd from 3:00PM to 5:00PM at Sullivan's Ring Side. GCS will provide refreshments including adult beverages.

Click here to RSVP. An RSVP is required. If you have any questions please email joe@gcsaustin.com.

http://www.gcsaustin.com/seminar/

When considering a hosted or cloud provider, ask the 10 simple questions below to further your analysis.

Data Center: Are your servers stored in a data center? Please describe your power, data and cooling redundancies?

Compliance: Are you compliant with PCI and SAS 70 standards?

Longevity: What happens in the event your business fails? How do we recover our data? How do we use it, once recovered?

Backup Systems: How do you backup the data? How often is it stored offsite? How is it backed up onsite?

Architecture: Do you utilize virtualization with shared storage?

Reliability: Do you offer a Service Level Agreement? How much credit do we receive when you are down? At what amount of downtime do I receive the credit?

Performance: How does our user count compare to your largest client and to your total user count?

Bandwidth: Approximately how much bandwidth per user is required at our office?

Ownership: Do you own the equipment and licenses on which you're hosted?

Support: What are your support departments hours of operation? How is after hours support provided?

This is a quick start but should start separating the real providers from the pretenders.

A great, in depth article on the DirectAccess feature in Windows 7 was posted recently on Informit.com. They nailed down the architecture well.

So, can DirectAccess kill your VPN? Yes and no. It is not intended to work on non-company owned PCs. Why would you want it to? You'll still need VPN for that. I've seen many organizations utilize VPN for vendor access. DirectAccess is not a replacement for vendor access. DirectAccess also requires IPV6. Oh. Though many organizations are unintentionally running IPV6 already, this will present some pause to many IT Managers.

A total lack of faith in existing anti-virus products has forced us to watch Application Whitelisting technology for a while. A comprehensive review of several top products was recently posted over on Computerworld.com. It was nice to see fellow Austinites, CoreTrace, score so well. The article correctly tagged one of CoreTrace's best features, buffer overflow protection.

Though we have a close eye on Application Whitelisting, we have yet to deploy it for a single client. Frankly I'm terrified of user adoption issues. For this technology to succeed we will have to convince the users that the trading flexibility for security is worth it, or work with organizations that can afford to ignore user complaints. Winning hearts and minds is a losers game. Today I'm more convinced than ever that Application Whitelisting will be a hard sell. For those organizations that can stomach it, it could dramatically reduce time spent on virus and spyware issues.

The first part of this comparison generated some heated interest in the blogosphere. It earned a response on Dell's Equallogic Blog as well as a couple of well composed comments from both HP and Dell. Dylan Locsin from Dell thought that a few of my points were inaccurate and he even insinuates that I may be misrepresenting myself. I'll respond to his points here:

1) Clustering: His point that my calling Equallogic "clustered" is inaccurate is completely correct. I should have call it "distributed." Equallogic does not offer true clustering across SANs. They don't offer a comparable feature to the Lefthand's Network RAID functionality which allows striping across SANs. Equallogic cannot pool IO resources as it scales.

2) Groups: Dylan interpreted my argument with Equallogic that "Data cannot span more than these two SANs" as a replication argument. It wasn't. With HP Lefthand SANs I can add performance and capacity with each node up to 16 nodes (and even beyond). This avoids silos of performance and storage. I did fail to point out that I was considering the PS4000 series which is the most commonly encountered Equallogic SAN considered device by my clients. The PS6000 series improves this situation.

Dylan wondered why I didn't offer a disclaimer that I sold HP Lefthand. If he scrolls down a bit he'll notice the disclaimer. I also prominently place the Lefthand SANs on my website. GCS does sell Dell servers, desktops and notebooks. We support MD3000i and Equallogic SANs from Dell. I think Equallogic is a fine product. However I firmly believe the HP Lefthand SAN offers the best value for my clients in the SMB market. That said, Dylan, I'm not opposed to also offering Equallogic to my clients that bleed blue. There are many of those, especially in Austin. I don't mean to insinuate that Equallogic is bad product. I've seen clients succeed with both options.

By far our clients and prospects have two SANs in mind, the Dell Equallogic and the HP Lefthand. While both are impressive SANs that are very similar, there are some critical differences.

Clustered Storage vs Cluster-able Storage: Both products offer clustered storage but only Lefthand can tout that out-of-the-box. Each Lefthand includes two chassis by default where-as Equallogic offers dual-controllers and an extremely reliable single chassis configuration. What is the actual difference in reliability? I have not found any real-world tests but if I can get two for the same price as one, I'll go that route. Advantage: Lefthand

Licensing: Both Dell and HP have listened to their clients anger at complicated licensing systems. Both the HP Lefthand and Dell Equallogic offer an all-inclusive licensing method. Unfortunately unless you buy the HP Lefthand Starter SAN Solution, you may need to be the Multi-Site/DR license in the future for real-time failover and failback. Advantage: Equallogic

Density: The HP Lefthand is essentially a ProLiant DL320s server. The Dell Equallogic is custom designed chassis. Because of this difference the Equallogic gets more spindles per U. This is certainly a consideration when you're ordering racks of SANs. It is much less of a concern when order a SAN or two. Advantage: Equallogic

Groups: EqualLogic PS series allows only two SANs in a group. Data cannot span more than these two SANs. With HP's SAN/IQ Network RAID you can span multiple SANs in a variety of configurations. This provides better performance and reliability. Advantage: Lefthand

Site to Site Replication: Out of the box both SANs offer site to site replication. Only the HP Lefthand supports synchronous data replication with automated failover and failback. The Lefthand supports multiple sites in all configurations. Advantage: Lefthand

Obviously a lot of features are excluded as they are quite similar between both products. The reason GCS chose to emphasize the HP Lefthand SAN was a significant price and value advantage. However it is not quite so apparent as when compared to other SAN vendors on the market.

Here is a helpful comparison chart for HP and Dell's iSCSI SAN lines:

Calculating usable disk space on a SAN is dependent on a number of variables. Make sure you're buying the right model with GCS' HP Lefthand SAN Usable Disk Space Calculator. The calculator displays usable disk space across a variety of RAID and Network RAID configurations for the HP Lefthand P4000 SANs.

Most of our clients choose between an Entry Level or Mid-Level SAN, as described in Part 1 of this Buyer's Guide. In this post I'll identify the features that differentiate those two classes of SANs.

Clustering: Clustered SANs operate like clustered servers. If one fails the other takes over immediately. This is only available in the Mid-Level products such as HP Lefthand, Dell Equallogic, etc. Only HP's Lefthand offers true clustering out of the box. The minimum order for a Lefthand SAN is two completely separate units.

Thin Provisioning: Mid-level SANs allow  you to oversubscribe storage by allocating storage to a volume without reserving that storage. If you create a 100GB volume but only use 40GB, the remaining 60GB is free to be allocated to another volume. This feature is essential in maximizing the storage efficiency of a SAN.

Offsite Replication: Replication between SANs is the foundation for an excellent DR solution. Replicate all data and VMs to another site. Many of the mid-level SANs offer this solution but in some it is a licensed add-on. HP's Lefthand includes scheduled replication at no additional cost but real time replication and automated failover is an additional license fee.

Snapshots: Snapshotting technology is an on-array backup method that utilizes a relatively small amount of disk space. This is possible to restore entire volumes quickly without relying on external storage.

De-duplification: Long an enterprise only feature, integrated de-duplification is making its way into mid-level SANs. This can dramatically increase the efficiency of storage but can have a significant performance cost.

This brief guide will outline the different SAN classes available to the SMB buyer.

Entry Level: If you're buying a SAN, you need a few basic features. iSCSI support and vendor certification (Microsoft, VMware, Citrix) will meet minimal needs. These will enable your virtual environment to utilize high availability features in any of those vendor's environments. HP's MSA 2012i G2 and Dell's MD3000i both meet these criteria. Expect to spend as much as $15k on these devices. The HP will let you mix and match SAS and SATA drives in a single chassis for best use of your SAN dollars. Neither of these solutions offer thin provisioning or clustered storage. What's that mean? Less efficient per GB and much greater risk of failure. If you're concerned about putting all your VMs in one basket, and you should be, then look to the Mid Level, below.

Mid Level: This is what you want, if you can afford it. Two major features enter play here: thin provisioning and clustered storage. I'll touch on both of these now with more to come soon. Thin provisioning allows for oversubscription of storage. Don't worry about it, just do it. Clustered storage is like clustered servers. Two, or more, boxes configured for failover. The HP Lefthand provides the lowest entry cost to true clustered storage. Dell's Equallogic, Compellent, NetApp, Xiotech and others each offer some unique features. Expect to spend at least $30k on this device.

Enterprise: Forget about it. You can't afford it and wouldn't fit in your server closet if you could. Vendors such as HP, EMC, IBM, NetApp and others live here.

Virtual Storage Appliance: For those organizations that may already have a large investment in internal storage in servers or direct attached storage, a VSA may be the best bet. This software solution aggregates storage across your servers into an iSCSI SAN with similiar feature benefits to a full system. You will be able to support High Availabilty and VMotion/Live Migration with this solution. Obviously since you are only buying the software the entry cost is much lower than a hardware solution. HP's Lefthand offers a VSA for VMWare (Xen and Hyper-V are coming). StorMagic has an interesting option that currently supports only VMWare as well, but Hyper-V support is coming.

A not-too-brief market overview should whet your appetite. Look for more info on the features and key differences between vendors to come soon.

We've long been fans of Sonicwall firewalls at GCS. Advanced features, easy web-based configuration, low failure rates and low cost make it a very compelling option for many clients. With the rollout of the new product lines Sonicwall offers the Enhanced Firmware features (most notably WAN Failover and Load Balancing) as a standard item on every product. My small office clients can now get WAN failover in a device for less than $400. This is a real cloud enabler. For a few grand you get WAN failover plus a High Availability configuration on your firewalls.

Today Avaya released a KB article describing how to configure Sonicwalls to prioritize voice traffic between sites. With the rise in popularity of MPLS and managed routers, we are seeing far more Ethernet handoffs than T1 handoffs these days. The Sonicwall NSA 240 is a great device to terminate that MPLS circuit and appropriately manage the traffic.

Live Migration is the killer app in Hyper-V R2, which is due out in mere weeks. Microsoft finally can match VMWare feature by feature in many environments - or can they? If you don't want backups, Live Migration works fine. What? Again, please? If you're using Data Protection Manager 2007, you will not be able to backup VMs using Clustered Shared Volumes. Clustered Shared Volumes are required to utilize Live Migration.

Fortunately DPM 2010 released to Beta on 9/29. Not only does it add support for VMs using Clustered Shared Volumes but it also enables mobile laptop backups. The mobile laptop backups work over a VPN and are designed for the user off the LAN. DPM to DPM replication offers a poor man's disaster recovery solution.

Data Protection Manager is by far the best backup solution for Hyper-V virtualized environments. It includes brick level backup of Exchange, a SQL agent, and a Sharepoint agent. It integrates to Shadowcopy for backups of the VMs. All DPM agents are included in the System Center Server Management Suite, though you have to buy a seperate DPM server license.

Hold your breath. Live Migration is coming, just not quite as fast as the marketing indicates.

GCS' Early Happy Hour was filled to capacity by IT Executives from all over Central Texas. For many it was their first look at Windows 7. Many attendees agreed that the upgrade to Windows 7 is unavoidable. Following the demo, some attendees even looked forward to it! Our discussion highlighted the improved UI, Branch Caching, Windows XP Mode and AppLocker.

In addition to Windows 7 the attendees got a look at Windows Server 2008 R2. We spent quite a bit of time talking about Hyper-V, System Center Virtual Machine Manager R2 and the new Live Migration features. Marquis covered Clustered Shared Volumes in depth, as they pertain to Live Migration. We also covered the new Active Directory Recycle Bin and Branch Caching.

We wrapped up the event with a look at VMWare's VSphere 4. The new Fault Tolerance features garnered the most interest. VSphere's Fault Tolerance allows for failover between hosts without downtime. They have set the standard here. Unfortunately most of the new features in VSphere 4 are not included in the Essentials and Essentials Plus package which is the only version that is price competitive to Hyper-V.

We'll be hosting our next even soon, focused on Cloud Computing in the SMB environment.

We're finally seeing the ProLiant G5s and virtualization-enabled PowerEdges become widely available in the used server market. Many clients are asking for our recommendations. In a virtualized environment with VMware's High Availability or Microsoft's Failover Clustering, is used hardware an acceptable option?

First let me say that I've long been opposed to used hardware in traditional environments. The hardware represents a small portion of the overall cost of the implementation. The installation and configuration, whether performed in-house or by a consultant, is wasted when the hardware has to replaced. I would rather see that service cost spread across four of five years of use. Why save a few thousand dollars to sacrifice a lot of ROI? Cash is the only valid reason. The difference between new and used gear is rarely enough to determine the fate of a project.

Now we have a highly available, virtualized system. We have architected enough capacity to easily run in the event of a single server failure. Are we more comfortable pushing the typical refresh cycles of servers? YES! If we're only refreshing from a fear of hardware failure, then used equipment should be an option considered. In two recent projects we've been able to go from 20ish servers to two servers without buying a single new server! We're reusing equipment from the organization in these cases, but that compares well to procuring used hardware.

Make sure when ordering that your hardware is identical to take advantage of the high availability features. SANs are still new enough to have limited availability on the used market. If you can knock $20k or $30k out of the upfront cost by using used servers, why not?

You've taken the plunge (or not) and are ready for SIP. Now, how do you configure SIP trunks on an Avaya IP Office?

1. Buy licenses from an Avaya reseller (I can think of one). Each SIP trunk is licensed.
2. Order SIP service from an Avaya-supported provider. 
3. Your SIP provider will request a public IP address that has port 5060 forwarded to the IP Office.
4. Reference the Application Notes with step-by-step instructions from Avaya for your provider here. Any App Note is a decent guide for any provider.

Not all SIP providers have Applications Notes at this time. However, if your provider is listed your Avaya reseller (*ahem* GCS) will be able to configure SIP service. If you're just getting started with SIP it may helpful to read GCS' SIP Primer.

SIP is a mature technology that will likley displace PRIs as the standard for dial tone in the SMB market. The IP Office offers both SIP trunking and endpoints.

HP's Proliant G6 maxes out at 144GB of RAM. 16 core processors are due out in 2012. How many VMs can you cram in a quad, 16-core box with a 144GB of RAM? Enough. Arthur Cole describes trends toward both mainframe-style aggregration and grids of smaller systems. Which is right your for you? One factor in determing the best strategy is:

Additional servers reduce the overhead for virtulization failover. Much like RAID configurations, with 2 servers you must allocate 50% overhead for failover capacity on each server. Three servers only need 33% overhead and so on.

The co-founder and CEO of Oracle carries a lot of weight in many tech circles. An event last night at Silicon Valley's renowned Churchill Club saw Ellison again downplay the importance of "cloud computing." This is not the first time Ellison has been so bold. His point is consistent - cloud computing encompasses a wide variety of solutions that are already in use.

It feels like cloud computing has accelerated in deployment so quickly for this very reason. Solutions that were previously hosted, managed, Saas, etc are now grouped together under a "cloud" moniker. The real opportunity, that has preceded "cloud" systems, is the shift from fixed costs to variable costs in IT budgets. For many organizations this improves their flexibility tremendously.

Come join us on October 1st at 3:00PM for demonstrations and discussions of these exciting new products. What will users like most about Windows 7? What three features separate HyperV R2 from VSphere 4? Can your environment use both hypervisors effectively? Our event will be held at Sullivan's Ring Side. Click here to RSVP.

This seminar is focused exclusively on IT personnel, in a management or strategic role. Engage directly with technical experts experienced in the installation and management of these products.

I hope you can make it. Your RSVP is appreciated.

When should I recommend Cloud solutions to my clients? I don't know. Today I'm going to start figuring it out. I'm going to compare an on-premise build with a cloud based solution for a hypothetical 50 employee organization. Allow me to skip a lot of details and say it will provide roughly equivalent features and security. Anything required in both solutions was a wash and therefore ignored. What you're left with is mostly a lot hardware (servers, SANs, firewalls, switches for iSCSI, etc), deployment services and hardware-level support.

NOTE: All pricing is retail and rounded. Individual proposals may vary. A greenfield is assumed - no data migration. Everything that was common to both solutions was excluded so this may represent a small portion of the overall project. 

Cloud Offering:

In the cloud we'll procure 20Ghz of processor capacity, 50GB RAM, 3 TBs storage with 10 Mbps of bandwidth to host 15-18 Virtual Machines. Onsite backup for 1.5 TBs. This will run about $10,000 per month.

On-Premise Offering:

On-premise will provide provide about 60Ghz (over 24 cores), 96GB RAM, 3TB direct attached storage for backup, 3TB iSCSI SAN. We'll plan on hosting 25-30 VMs on this platform. You'll get this for about $160,000 including 200 hours for installation and configuration. See my build here. Note we dropped 25% of capacity for high-availability.

To even the playing field we have to factor in datacenter costs for the On-Premise Offering. I see a rack, 10Mbps of bandwidth, and 40A of power. Say $3000 per month?

Conclusion:

Let's compare 3 years of costs. In the cloud you're out $360,000. At home, including the data center costs, you're looking at $268,000. The $92,000 delta has to cover the hardware support and maintenance for 3 years. There is some difference in capacity and reliability. I hope it helps your evaluation to say that the cloud offering, in this case, is 35% more expensive.

Click image for single page PDF

If you're curious about SIP, you're not alone. SIP, Session Initiation Protocol, is a method for terminating dial tone or connecting handsets to a phone switch. It enables vendor competition on proprietary phone systems for the high-margin handsets sales. It also allows dial tone to be passed over internet connections at a higher capacity than a traditional PRI. Most major phone system manufacturers and software based systems support SIP trunking and endpoints. SIP is already in use as a means to transmit voice on many carrier backbones.

Here are several important SIP facts:

• SIP is run on a typical network connection. Like any VoIP you must have QoS end-to-end for consistently high quality service.
• Many providers offer SIP to anywhere by pushing it across the public internet. Be wary. Look for vendors who are bringing in their own managed internet connection.
• On 1 T1 you can have either 23 lines (PRI) or 45+ SIP trunks. If you have ethernet options your phone capacity can scale very easily.
• SIP supports much of the functionailty of the traditional PRI such as Direct Inward Dial and Caller ID.
• SIP handsets may not be a good deal. Most major VoIP manufacturers charge a per-handset license fee equal to about the cost savings. However in response to price competition vendors such as Avaya have released lower priced handsets such as the 1600 series.
• SIP handsets lack functionality available on the proprietary handsets
• SIP is implemented differently by many manufacturers. Ensure your manufacturer is supported by your provider. Not suprisingly Cisco and Avaya users have many options.

SIP is a mature product with significant cost advantages and flexibility. It deserves consideration as a trunking option for all new installs.

In today's post on the CIO.com blog, Michael Hugos raises several interesting points about transitioning IT asset costs from fixed costs to variable costs. He boldly predicts that doing so will herald a rapid economic expansion. Though I'm not willing to go so far as he is, I am a big proponent of emphasizing variable costs. The justification is simple dollars and cents.

Even in small organizations IT asset acquisition is well into the tens or hundreds of thousands of dollars per year.  Dollar for dollar it is hard to beat rolling your own virtualized environment.  Do-it-yourself, if you're able, is the cheapest method. There are several mitigating factors to this:

1) The competition for those budgeted dollars is diverse. Instead of spending $100,000 this year on assets, what if you spent $35,000 on cloud-based offers and moved $65k into a marketing program, new sales reps or other projects that increase top line revenues.

2) The opportunity cost of building and maintaining that system interferes with other IT projects. By migrating routine and time-consuming tasks into optimized environments that are designed for those tasks IT departments can focus on projects that require highly specialized knowledge.

3) New cloud offerings from Terremark, Amazon and 3Tera (among others) are radically altering the scale at which IT exists in small and mid-sized businesses. The cost savings that come from purchasing a small piece of a big pie are significant when compared to your home baked pie.

Many organizations would do well to investigate methods that take IT assets off the books and put those capital dollars into more revenue generating tasks. In any economy that is just good sense.

I needed to find a home for seven wayward servers. The prospective owner had decided to stop owning hardware just as the project to deploy these severs was underway. My job seemed easy enough - replicate a basic Microsoft stack (Exchange, MOSS, SQL, Dynamics GP) for a fixed monthly fee in a land far, far away. I've been closely following the development of hosted/managed/cloud systems for years, have built numerous virtualized environments and I have a very health monthly data center bill. Still the last few days has opened my eyes to the breadth of the cloud's offers. In the end this dynamic market, with a vast pool of options, won over a skeptic like me.

I want to put that last statement into the proper context. I own a value-added-reseller business that has generated millions of dollars in revenue selling hardware and licenses. Last year almost 40% of our revenue was equipment resold from vendors such as HP, Dell, Cisco, Sonicwall and more. I suspect that in the next few years that number will dwindle substantially.

After extensive deliberation I have come up with one singular reason to embrace the 'cloud' that applies to IT departments as well as VARs. We all have better things to do. The time spent procuring, installing and supporting hardware and software detracts from time spent integrating, training and customizing. What if a typical IT project directed a bigger part of the budget at these services? Businesses will derive more value from the technology. The employees will use more of the functionality. The system will work better with the other technology around it. It will more closely match business processes.

Far from competing against the services offered by most IT departments and VARs, cloud computing offers the opportunity to expand these organizations. Those that successfully transition their skill sets can expect to see revenues and budgets increase as businesses receive more consistent performance from existing systems and lower costs to add new systems. This transition will not occur overnight, but in next decade we will see fewer and fewer servers in offices, more servers in the clouds. Think hard about investments in on-premise equipment. There are compelling options out there.