March 12, 2001, 11:09 AM — Steve Etzell saw for himself how quickly a minor unauthorized change can foul up a Web site. Etzell, director of Web technology at Select Comfort Corp. in Minneapolis, was on vacation when he got a call telling him the bed maker and retailer's Web site performance had gone "into the tank." The reason: A developer had let a business group user "twist his arm" into dynamically generating user-specific price quotes on a Web page that showed an entire category of Select Comfort's products. The site had previously sent users to a cached page that showed the same prices to everyone.
That change "seemed fairly innocuous," Etzell recalls. But the page "is accessed potentially 100,000 times per day . . . and you'll bring the server to its knees" by forcing it to dynamically create the page for each visitor, he adds.
"About an hour later, they realized what they had done and turned the caching back on" for the category page, says Etzell. The long-term answer was to remove prices from the categories page and instead put them on Web pages that describe specific products. Since those are accessed far less often than the categories page, the site can deliver customized pricing without taking a huge performance hit.
Balancing Act
It's that kind of unplanned, untested change that Web site managers hate and users love. Managing change on the Web is a "balancing act" between the need to keep your very public Web site up and running and the need to update it often enough to keep it attractive to visitors, says Etzell.
The more important your site, the more reliable it needs to be. The more transactions you do, the more it needs the kind of rock-solid stability once associated only with mainframes in a data center. Keeping embarrassing and costly outages to a minimum requires IT managers to create standard change-management policies, automate them as much as possible and outsource them if they must. Repeatable, consistent procedures, performed either by skilled support staff or automated tools, are the best way to cope with the pressures of a public-facing Web site.
The Web environment is unique because users demand changes within hours, not weeks. Changes to content aren't done by database administrators who first check the validity of the data and its effect on site performance, but by marketing managers. There's no single mainframe vendor to release updates or patches on a regular schedule, but rather a half-dozen or more suppliers that find and fix flaws in their products on their own schedules.
Then there's security, which can require major changes to sites as hackers discover new ways to bring them down. "There's a lot more changes going on in these Web-facing systems, with most of those relating to security," says Jason Lochhead, co-founder and chief technology officer at Data Return Corp., a Dallas-based managed hosting company. "You didn't have to worry so much on legacy systems because they're isolated from public traffic." Microsoft Corp. acknowledged in late January, for example, that its defenses had been inadequate after it was hit by denial-of-service attacks two days in a row. In response, Microsoft planned changes to its network architecture, including a backup set of domain name servers (DNS).
Even routine, planned changes can crash a site if they're done incorrectly. Just days before the hackers hit, Microsoft suffered a 22-hour outage that left many of its Web sites unavailable. The company blamed the problem on a faulty configuration change to the routers on its DNS network.
When Don Ursem compares the reliability of his Web site with that of the telephone system, he isn't kidding. Ursem is vice president of network operations at VocalPoint Inc., a San Francisco-based application service provider that lets consumers access Web sites via phone by converting HTML into voice responses. VocalPoint sells the service to telephone companies and in vertical markets such as health care. For the end user, "it's a telephone application," not a computer application, and "you expect your telephone to work all of the time," says Ursem.
But that's easier said than done. First, there's the volume: VocalPoint leases two T3 data lines, each of which can handle 644 simultaneous incoming calls and needs 135 servers to process them. Then there's growth: As VocalPoint adds T3 lines, Ursem expects that he'll be managing about 650 servers across three sites by June.
VocalPoint rolls out a new release of its voice Web-browsing software every three months and is converting about 30 Windows NT servers to Linux to support a new text-to-speech engine.
