From: www.itworld.com

Best Practice:
Automating a SAN change management process

January 18, 2005 —

Strategy in Practice

Business requirements

Business value

Rules for success

Five classic mistakes

Questions you must ask

Are you a candidate?

Last words

This Best Practices is part of a collection of advice provided by information technology professionals on how they have solved various challenges, and addressed IT priorities within their organizations.

Company:

State Street Global Advisors (SSgA)

The world's largest institutional asset-management firm with $1.3 trillion in assets, based in Boston.

Challenge:

Automating a storage area network (SAN) change management process to effectively change and scale SANs at the pace of business.

SsgA has nearly doubled in size in the last three years. During that period, SSgA's storage network environment grew in size and complexity. The company's IT organization recognized the challenges and importance of providing reliability for SANs in order to support the business, deliver greater value to shareholders and bring greater return on clients' investments.

"We make extremely large block trades and stability in our storage is essential," said Robert Shinn, Principal at Boston-based SSgA. "We can't have applications down any longer than people can hold their breath."

Limitations in preventing and troubleshooting problems in SANs preclude IT organizations (including SSgA) from attaining high application availability - a priority for CIOs. Sixty to 80 percent of problems in SANs - including the most challenging to find and fix - result directly from SAN changes. Basic day-to-day actions cause SAN errors: Someone from the cabling group disconnects the wrong cable without your awareness. A member of the storage team makes a typing error when adding LUN masking for a new volume, but you only learn of the mistake after downtime occurs. A storage administrator is required to shut down a switch, but doesn't fully understand its impact on the SAN.

Organizations may have no choice but to manage changes and events in storage networks with dark-age methods - error-prone and time-consuming manual techniques and cumbersome spreadsheets. To validate the impact of a zone change, for example, the storage administrator must compare and correlate data from the HBA details spreadsheet, the zoning database and the storage masking spreadsheet. As a result, half of IT time has been spent managing changes and change-induced problems.

Adding more people is not an answer given the sheer complexity of managing and scaling SANs. A 100-server SAN requires maintenance of more than 20,000 actual access paths and 100,000 potential logical paths. To be successful, a storage manager must understand how a single change in such a SAN can affect several thousand paths. From this example, it becomes clear that humans cannot possibly understand the impact of changes throughout the maze of access paths, their relationships and inter-dependencies.

Solution:

Onaro, Inc.'s SANscreen Predictive Change Management software

Today, Onaro's SANscreen Predictive Change Management software is installed and running to protect many large SANs at Fortune 1000 companies. These companies have documented several events when application downtime was prevented, troubleshooting problems was accelerated, and coordination between groups responsible for changes was dramatically improved.

Specifically, SSgA applies SANscreen to predict the impact of SAN changes and troubleshoot problems. By doing so, SSgA effectively manages changes and growth in storage networks, attaining high application availability and reducing the risks associated with day-to-day SAN operations. Users are able to:

• Identify and repair errors faster.
• Predict the impact of planned changes before implementation.
• Monitor the implementation to ensure correct execution.
• Gain better control of their storage networks through automated, best practice change processes that coordinate activity among dispersed groups responsible for making changes.

SANscreen identifies and analyzes errors and vulnerabilities before, during and after change implementation. First, the software establishes a baseline of the existing SAN environment, identifying discrepancies between findings and business rules. On an ongoing basis, SANscreen continuously monitors and validates the health of SANs throughout changes, growth, migrations and consolidations. For storage administrators and groups responsible for changes, the software gives them back most of their weekends and nights, when most changes are implemented. "We've gone from putting out fires to enabling our storage staff to focus on more strategic activities that can help the business," Shinn said.

At an organizational level, SANscreen reduces the risks, time and resources inherent in managing and scaling SANs and improves coordination among groups responsible for changes. Gartner, Inc. storage analyst and Vice President Ray Paquet views Change Management for SANs as must-have technology that belongs at the top of storage management budgets.

Primary components of a SAN Change Management solution include:

• Path management to understand all physical and logical relationships between an application and its data.
• Continuous monitoring and troubleshooting to pinpoint problems and identify violations that stray from policies and best practices.
• Deterministic root-cause analysis to accelerate problem resolution.
• Automated planning and simulation of end-to-end access paths to predict the impact of changes before implementation.
• Capture of change history for audit, regulatory compliance and a step-by-step understanding of change events.

By automating the SAN change management process, SSgA says it has gained an independent "check-and-balances" layer that guards the functioning of SRM and provisioning tools and continuously validates all planned and unplanned changes.

SAN Change Management ensures a stable foundation that empowers all other storage management investments, notes Steve Duplessie, founder and senior analyst at Enterprise Strategy Group. "You can't utilize your storage products if your infrastructure is fraught with holes and errors." Duplessie maintains that companies "have been putting the cart before the horse" by investing first in SRM over the SAN change management foundation.

Although SANscreen relies on advanced patent-pending technology, it implements quickly to produce immediate results. Organizations install SANscreen in 24 hours. Within that time frame they gain a complete assessment of their SAN that shows discrepancies with business rules - such as violations in unauthorized paths, redundancy, and host ports access, and vulnerabilities in LUN sharing, zoning and clustering. These results are often used for immediate SAN stabilization and cleanup. SANscreen is a completely non-invasive solution - agent-less, read-only and operates out of band.

According to Gartner, improving IT change management processes is one of the best investments an enterprise can make. "Companies that don't properly manage IT changes lose time, money and efficiency and subject the entire business to undo risk."

Business requirements at SSgA:

SANscreen has met business requirements that the company set for a change management solution:

• Enable SAN growth without increasing operational resources.
• Reduce storage deployment time for business applications.
• Reduce over-deployment of SAN capital assets.
• Increase SAN stability by anticipating, reducing and quickly resolving problems and disruptions (For example, the software will evaluate a plan to add a switch or alter LUN masking, apply simulation to determine the impact of that change, identify steps required in the proper sequence to avoid problems, and validate that the change was made correctly.)

Business value for SSgA:

A detailed ROI analysis showed a fast payback in operational efficiency alone (about seven months). ROI did not include benefits derived from improved application availability. The company determined that automation of its SAN change management process would yield the following benefits:

• A $543,000 savings annually in SAN change process operations.
• A $295,000 savings annually in SAN problem resolution.
• A 55 percent reduction in storage deployment time.
• Preventing 35 percent of problems from ever occurring.
• An 80 percent reduction in SAN-induced business disruptions.
• SAN growth without an increase in operational resources.
• An increase in IT infrastructure stability.

"A large amount of outages are caused by changes that aren't deployed properly," Shinn said. "SANscreen avoids many problems and outages in the first place, reduces our time to react to others, deploys new storage faster, and helps us meet ever-changing business needs."

Onaro customers generally gain a return on their investment in about six months.

Rules for success:

• SAN Change Management must be able to simulate changes to determine the impact of any change on a SAN prior to implementation, analyzing the change impact on application availability, performance and security.
• SAN Change Management must perform deterministic root-cause analysis to troubleshoot problems quickly.
• SAN Change Management must continuously validate changes by correlating functions from various devices with business rules, detecting any violations or vulnerabilities in near real time.

The SAN Change technology must be completely non-invasive. For example, SANscreen is agent-less, read-only and operates out of band. These features overcome resistance from accepting a new technology in a complex environment by eliminating risk of implementation. "We've been able to attain stability and predictability without increasing overhead," Shinn said.

• The technology must produce immediate results. Although SANscreen relies on advanced patent-pending technology, it implements quickly to produce immediate benefit. Organizations install SANscreen in 24 hours. Within that time frame they gain a complete assessment of their SAN that shows discrepancies with business rules - such as violations in unauthorized paths, redundancy, and host ports access, and vulnerabilities in LUN sharing, zoning and clustering. These results are often used for immediate SAN stabilization and cleanup.
• SAN Change Management must improve coordination among dispersed groups responsible for implementing changes. "We've improved our cross-organization change management processes and dramatically reduced phone calls to our technology people in the middle of the night," Shinn reported. "We now have a standard change process and vocabulary that understood across all the different groups, that's much more effective at avoiding and pinpointing problems. And we've freed up resource for things that are of great value to the business."
• SAN Change Management must capture complete change history, for audit, regulatory compliance and step-by-step understanding of change events. Users can then easily roll back to the last steady state and identify where problems surfaced and what caused them. For example, SANscreen's user interface provides green check marks and red alerts to mark change steps performed correctly and incorrectly. "Not only can we see what has change, but we can see anything that impacted that change," Shinn said. "Before (SANscreen) it was virtually impossible and enormously costly to get through one change cycle before having to make another change."

Five classic mistakes:

1. Assume that the generic SRM reporting applications will solve your Change Management problems.
2. Make a decision on a product without having it installed, up and running and showing value in your environment.
3. Fail to take into account the cost and risk of deploying the product. For example, how much will it cost to deploy and maintain agents on all hosts in the data center?
4. Assume that purchasing from a major storage vendor major is virtually risk-free. (Always calculate the risk to your environment when not having an optimal Change Management solution.)
5. Assume you will always be able to have the best storage people available to extend your team to support change and growth. (There comes a point when throwing more bodies at a problem won't help - when the complexity of the SAN outstretches human ability to view and understand how changes are impacted by a maze of end-to-end access paths, their relationships and interdependencies.)

Questions you must ask:

• Is the set of required functionality for SAN Change management fully supported by the product?
• Does the vendor's roadmap for the product extend from SAN Change Management to performing Change Management across the entire data center?
• Does the vendor provide a customer-customizable ROI model that demonstrates a solid return based on savings in operational expenses alone (not including downtime)?

Are you a candidate?

You are if your...

When SANs reach a certain size that manual methods are no longer effective at managing changes and growth and being able to make changes at the speed of business. When organizations want to provide a sound SAN infrastructure needed to support a migration or consolidation of equipment. Generally, a SAN should have at least 250 switch ports.

Last words:

In the next three years, solving change management problems across the data center will become a CIO priority. Given that SANs are a complex, strategic piece of the IT infrastructure, SAN Change Management will continue to grow in importance, according to both Gartner and Forrester.

This Best Practice was provided by:

Onaro, Inc.

For more information, contact: Steve Feldman, Marketing Director, steve.feldman@onaro.com and 617-338-0068, x 21.

The ideas expressed in this article are solely those of the vendor and its client, and do not necessarily reflect the opinions of ITworld.com.

storage.itworld.com