Managing e-mail for maximum uptime
"We'd like e-mail to be like the telephone," says Brian Glass, manager of network administration and infrastructure at Del Webb Corp., neatly encapsulating every network manager's idea of nirvana. But he's just spent an hour explaining how his company's generally solid GroupWise e-mail software from Novell Inc. falls short of the ideal.
"We're not really interested in managing it," Glass explains, having earlier confided his concerns that an upcoming switch to Microsoft Corp.'s Exchange 2000 will only complicate the management of e-mail at his firm, a Phoenix-based real estate developer.
Software vendors, market analysts and users like Glass say the need for e-mail management tools and techniques has risen in recent years, along with the volume of e-mail and its increasing centrality to business processes. The number of e-mail messages and the size of file attachments have grown exponentially within the past two years, these observers say, clogging e-mail gateways and filling up network storage.
Some network managers believe their role is to keep e-mail conduits free and clear while providing a handful of critical e-mail-related services. Many e-mail management tools, such as the AppManager suite from NetIQ Corp. in San Jose, therefore function as a sort of early-warning system against threats to the channel's availability, such as bandwidth-hogging viruses or stressed-out server hardware.
A smaller category of tools addresses content management, helping to ensure, for example, that e-mail isn't used in a way that could subject a company to sexual harassment suits and other legal challenges. Homegrown software and policies and the e-mail systems' built-in features are also typically part of the management mix.
David Druker, an analyst at San Francisco-based Ferris Research Inc., says he's noticed a shift in emphasis toward treating e-mail like an always-on utility maintained by IT departments that function like internal service providers. "Virtually everyone talks about how the management is allowing them to meet service-level agreements," Druker says.
The demand has given rise to more than 30 performance-monitoring companies. Many are tiny one-product operations, but others, like NetIQ and rivals BMC Software Inc. in Houston and Candle Corp. in Santa Monica, Calif., are major players in the larger e-commerce infrastructure market. Druker also points out that those performance-monitoring firms don't include other e-mail management applications such as antivirus, directory management and synchronization, or migration tools.
Managing e-mail can be tricky because there's no one-stop solution. "Nobody buys just one of these products," says Druker, who interviewed users for a report, "The Messaging Management Marketplace," which is being published by Ferris Research this month. "Some of these management products are a management challenge themselves."
Druker advises corporate IT to ask vendors for specific features and then make sure they listen. "A lot of the people who seem to have things working the best seem to have the ear of the vendors," he says.
Keeping Things Legal
At Robert W. Baird & Co., a Milwaukee-based investment banking and wealth management firm, three central Microsoft Exchange servers provide e-mail to 3,000 desktops in 81 U.S. and European offices. While availability is important to Senior Vice President Brian Brylow, he says his paramount concern is maintaining the company's e-mail paper trail.
"An SEC regulation requires us to save a copy of every message sent out of the firm and sent into the firm," says Brylow, who's often asked to produce the information on a moment's notice. In the past, the solution was "journaling" out of each user's personal folder files, or PSTs. But each PST holds up to 17,000 files, and Exchange doesn't provide an organized way of searching through them. All Brylow's people could do was sort message headers alphabetically and visually scan them for relevant messages.
Since late last year, Brylow has turned to EmailXtender from OTG Software Inc. in Bethesda, Md., which brings organization and searchability to the company's e-mail data stores. "Searches that would have taken days or weeks are 5- to 10-minute affairs now," Brylow says. EmailXtender has freed his staff to monitor the e-mail system instead of spending their days researching regulatory requests.
There are also intangible benefits to having a faster, more reliable way to comply with regulations. "I had a very time-sensitive request to have a large volume of e-mail reviewed," Brylow says. He might have turned it down in the past, but "using the EmailXtender product, I satisfied the request in 30 minutes. This is putting my entire history of e-mail at my fingertips," he says.
Brylow's content management tool kit doesn't stop there. He uses MIMEsweeper from Baltimore Technologies PLC in Dublin to monitor the content of Simple Mail Transfer Protocol (SMTP) messages. A second package, Assentor from SRA International Inc. in Fairfax, Va., monitors Exchange messages to prevent Robert W. Baird's financial advisers from violating federal regulations by, for example, promising customers a guaranteed rate of return on certain investments. Brylow handles e-mail security with internally developed tools that he won't discuss.
Another monitoring tool Brylow uses is ProVision from Platinum Technology International Inc. (since acquired by Islandia, N.Y.-based Computer Associates International Inc. and discontinued), a networkwide tool with some Exchange features. ProVision performs uptime monitoring by pinging the Exchange servers on a regular basis. Custom-written Perl and shell script utilities monitor the flow of SMTP and Exchange messages.
"What we're looking for is that mail is moving internally among servers," Brylow says. "We wrote the tools because we had some problems in the past with an ISP connection going down and mail not going out at 2 a.m. I now have someone 24/7 getting hit on their pager if it's not received on the Internet side or [if] a reply on the Internet side doesn't come back internally. In a moment's notice, I know I've got an outage, and I'm reacting to it. That's how we provide the firm 24/7 uptime."
Brylow also ensures performance by using several generic network-monitoring tools, including Hewlett-Packard Co.'s OpenView and CiscoWorks from San Jose-based Cisco Systems Inc. He says he's considering more comprehensive monitoring suites, including BMC Software's Patrol and products from NetIQ. However, Brylow says he sees such tools as more important to someone trying to squeeze availability out of hardware that's running near capacity.
"We tend to oversize servers as opposed to running really tight," he says. "They're really not breaking a sweat."
Electronic Weigh Station
Del Webb is a medium-size company with a large appetite for e-mail. It maintains six dedicated GroupWise 5.53 servers plus about 70 shared machines to provide e-mail for 3,300 of its 4,700 employees in 13 offices. Besides being the internal lifeline, GroupWise is a temporary communication channel for contractors and has processed half a million sales leads forwarded from the company's Web site.
The company has used GroupWise for seven years and finds it reliable and easy to manage, says Dirk Ellsworth, senior director of information services. With 10 network administrators at Del Webb's headquarters, Ellsworth says, he couldn't afford to run high-maintenance e-mail. "We probably have half a person who's dedicated to e-mail management," he says.
As a firm that regularly circulates large documents such as master plans and engineering blueprints, Del Webb is especially vulnerable to e-mail attachments that can bog down the entire network, causing gateways to crash. "We strongly encourage people not to attach 20MB worth of attachments," says Glass. "The gateways generally fail when you send 20MB." He acknowledges that the attachments policy hasn't been effective, though, so to stay aware of gateway uptime, the company uses Novell's ManageWise to automatically send alerts to network administrators, who then must reboot the offending gateway. Additional network monitoring is provided by seven data center administrators using HP's OpenView.
Guinevere antivirus software from Industrial Economics Inc. in Cambridge, Mass., recommended by Novell, is a less closely managed uptime and security tool, since GroupWise isn't as popular a target for hackers as Exchange, according to Glass. But Guinevere frequently rejects messages that are virus-free, says Glass, so the company runs it only on servers rather than distributing copies for each desktop. "We're just not finding anything [antivirus] that's really good for GroupWise," he says. GroupWise's built-in security is generally adequate, though Ellsworth says he regrets its lack of encryption for Internet mail, a shortcoming he hasn't tried to address with third-party software.
Moving users to new locations or PCs is easy in GroupWise, requiring the migration of just the post office to the new client system, Ellsworth says. Upgrading GroupWise itself is a different story. Ironically, Del Webb normally uses Novell's own ZENWorks for Desktops suite to distribute and install application software, but he can't use it for Novell's own e-mail application.
"We tried using ZENWorks for GroupWise, with disastrous results on this last version," Glass says. "We got a barrage of errors at the workstation."
Because of apprehension about Novell's financial prospects and commitment to GroupWise -- the recent outsourcing of support to less-knowledgeable HP technicians was cause for concern -- Del Webb has decided to replace GroupWise with Exchange 2000, starting this spring. Ellsworth says the expected benefit is better integration -- without custom programming -- with back-office systems such as SaleLogix and Lotus cc:Mail, a legacy program GroupWise doesn't support well.
Despite assurances from Microsoft that Exchange won't add to the management burden, Ellsworth and Glass are skeptical. Ellsworth says he wonders if the new version improves on past versions' remote access tools, which, in his experience, have been "just a nightmare."
Prudential's E-Mail Insurance Policy
E-mail management at The Prudential Insurance Company of America in Newark, N.J., tends to center on size issues. The company's Lotus Notes system consists of 250 servers running Domino 5.06 and Notes 5.04a clients on the desktops of 65,000 mostly U.S.-based employees. Four thousand Notes databases hold mission-critical policy documents and workflow applications.
"Managing the size of users' mailboxes -- that's a huge problem," says Michael Boatright, vice president of data center technology and IT operations at Prudential. Prudential employees tend to keep all of their messages because of regulatory requirements. "All of the other Notes companies that we talk to tend to face this problem. [Users] tend to keep their file directory in mail," he says. The practice strains storage systems and puts a premium on efficient backup, which is done with CA's ArcServe software and managed by another Prudential department.
The company struggles to satisfy requests to restore backed-up messages that often come from users who have accidentally deleted them. Restoring the Notes databases with ArcServe isn't always smooth. "There are some cases where we'd like other tools," says Michael Mandelbaum, Prudential's vice president of information systems.
Creating and managing large Notes distribution lists is another challenge. Prudential has developed an application called Notes.com in a Notes programming language. "It can select users in a department, building, grade level, job function, expense code, etc.," Mandelbaum says. "It does real-time queries against a DB2 database and returns a list of e-mail addresses."
Notes.com also screens for large messages and attachments that could tie up mail servers, especially when the attachments are addressed to a large number of employees. Warning messages pop up on users' screens and the operations department is notified. "We have put size warning messages on messages that exceed 200K [and] attachments greater than 500K," says Mandelbaum. "We have [also] created a monthly summary that gets sent to the owners of large mail files, giving them links to large and old messages and statistics about their mail files."
"The Notes.com tool has caught many people trying to send very large jobs," Boatright says, including one user who tried to send a 500KB file containing a logo to 65,000 people.
Attachments greater than 5MB are completely blocked. "It notifies the person who sent it that it's being held for review," says Doug Conway, a Prudential project manager. Further management is provided by what Mandelbaum calls an attachment warehouse: 10 4GB databases spread out over several servers programmed to purge the files after 14 days. IntelliWatch Pinnacle software from Candle is the company's monitoring tool, providing alerts when disk space runs low or messages aren't getting through.
Internet mail presents unique problems in content and traffic management. Approximately 30,000 outbound and 40,000 inbound messages flow through the company daily, according to Conway. Prudential employs several commercial "spam blockers" to fend off unsolicited mail but won't identify the software for security reasons.
McAfee Anti-Virus from Santa Clara, Calif.-based Network Associates Inc. provides virus protection. Alerts are sent via Notes mail, and the company's support staff is notified when the software tries to remove a virus from an employee's PC.
Like many companies, Prudential also employs policies and procedures to manage e-mail. Employees are sent twice-yearly reminders about inappropriate personal use of Notes, though the policy is now being reviewed, says Mandelbaum.
The chief benefit of the company's e-mail management efforts, Mandelbaum says, is "99.99% uptime. Without some of these things, I think we would have chaos, to say the least. These tools have been invaluable in keeping our environment healthy."