March 31, 2010, 10:51 AM — This interview is part of ITworld's regular "Lessons Learned" series where IT professionals share something they learned during a recent project or one of their favorite "Aha" moments.
Anyone who's been around the block a time or two knows not to believe everything you read in print, and that false assumptions can get you into trouble sometimes. That's why in the world of IT, testing before implementing in production is direly important. Seasoned IT pros know just how much trouble a product's next version can be on a production network if not vetted properly. Bill Fife, director of technology at Wholesale Electric Supply, in Houston, was recently reminded of the nebulous nature of vendor documentation during what should have been a routine enterprise firewall implementation. Thank goodness he tested first, Fife says. In this interview with contributing writer Beth Schultz, Fife reflects on …
SOUND REASONING: About five years ago, we bought an EAL4+ compliant firewall, the only one on the market without a published security vulnerability against it from any of the reviewing organizations. 'Well, that's definitely a step in the right direction,' I thought.
A CHANGE – OR TWO – IN OWNERSHIP: Shortly thereafter, the firewall vendor was acquired, but the new owner largely kept the enterprise firewall product intact, moved it along, maintained it -- the later version still being EAL4+ compliant. [Three years later,] this company gets absorbed by another. The latest company is no longer supporting or developing the original firewall I had purchased because the later product is largely a better, more secure and advanced unit.
VENDOR LOYALTY: We knew we were going to stay with the latest company because I had no desire for Cisco's ASA firewalls, which seem to have frequent firmware updates. Additionally, the latest enterprise firewall product is still EAL4+ compliant. We don't necessarily require that certification, but if you're going to use a product to protect corporate assets, then you should make it valuable -- make sure it is a highly regarded and tested unit. So I bought a couple of firewall units from the company thinking (maybe assuming) that they'd have similar interfaces and capabilities given the product information and abilities of the acquired company products.
BUMPY TRANSITION: Well, of course, the company changed the interface definitions. Then it changed where you 'NAT' the traffic going from the outside to the inside. And the licensing -- well it's not bad, but things could be a little simpler.
And then when we had the primary firewall up -- just for testing, it's nowhere close to production use yet -- we were configuring the secondary one, and our region interfaces (for example, Internal, External, DMZ, etc.) just kept moving around because there was a bad port on the card.
FULL SUPPORT? Then we had a DNS problem. For almost every DNS record, you need a TTL -- time to live -- value. In this firewall, the company says it has full DNS support, but there is NO place to set a record's TTL value. This is a problem for us because of our hub-and-spoke VPN configuration. We configure each office with two Internet circuits in an active-active setup. If the circuit providing VPN functionality goes down, those private network DNS records are no longer valid. The timely fail-over process to avoid those painful browser "Page cannot be displayed" messages requires a short TTL value that we have set to five minutes. So if someone at that affected location tries to access one of our sites when the VPN is down and the record has expired, a new DNS query is performed returning a different, valid record. This allows internal users to access our sites without any intervention or a work interrupting reboot.
Well, on the new firewall, there's no place to put a TTL value in the GUI setup. I think, 'OK, fine, we'll go in and edit the files on the firewall directly and plug in our own values.' So that's what we did – we went in, edited the file, it all looks nice. The problem is, if you go back into the DNS region within the GUI and make any change whatsoever it rewrites the TTL values for all the records. It completely ignores what you had manually entered and rewrites with default values of two days.
WORKAROUND: Now there are some bad hacks to get around this problematic issue. If all of your clients contact Windows Active Directory servers then through group policy you can change it to ignore TTL values on all DNS records and set a new max TTL value. But that's ridiculous. We don't need to change other people's TTL records and there's no reason we should even need to change our own.
After we fought with the firewall for about four hours and finally figured out exactly what it was doing, we called tech support. The response was, 'Yeah, it's a fairly well-known issue and it's been submitted to engineering. But we just don't know if or when it's going to be fixed.' Well, this is a little bit of an issue to me – 'You say you offer this, but then you don't.'
MAKING THE BEST OF THE SITUATION: So what we've accepted as a practice is that we're going to have to go in and through the firewall editor on the firewall, create our own zone as well as name files, edit solely in the command line editor then save them. Just in case an outside consultant or myself, through some stupid rage, goes in and makes any change within the GUI we'll have backup copies so we could easily correct mistakes.
LESSON LEARNED: What I've learned is if you're evaluating new hardware or an appliance, even if it's from the same company whose products you're already using, try to get a test version in house, look at it, and read the documentation. Make sure it will do exactly what you want and need it to do -- BEFORE you buy it. I had believed the company when it said there was full DNS support and that full functionality instead of a subset would be available.
So we had purchased the firewalls, and had I not been paying attention in testing, they'd be implemented by now and we'd be staring at a big problem.
What do you know now that you wish you'd known then? Share your tales here or contact Beth Schultz, at email@example.com.