October 16, 2009, 9:45 AM — Outsourcing Virginia’s IT operations to Northrop Grumman was supposed to put the state’s network beyond politics, overhaul and standardize it and improve operations, but the effort has failed in almost every respect.
The $2 billion, 10-year contract with Northrop Grumman calls for a network overhaul -- or transformation as the state calls it -- is months behind scheduled despite repeated deadline extensions.
Network services to agencies have been cut off or have been poor enough to affect the services agencies deliver to the public.Members of boards are overstepping their authority to negotiate with Northrop Grumman and the chain of command is so fragmented that it’s difficult to get any one entity in control of the problems.
This is all according to a report issued this week by the states Joint Legislative Audit and Review Commission (JLARC), which points a finger at Northrop Grumman, but also at boards, agencies and individuals for problems like these:
• In May, Virginia State Police in Newport News lost Internet access for 78 hours, affecting the ability to perform its duties.
• Also in May Environmental Quality in Roanoke lost its network connection for 31 hours.
• In June the Department of Motor Vehicles in Bland lost the network for 31 hours, affecting customer service.
• One state correctional facility lost incoming phone service at 4 a.m. and was assigned a priority level that gave Northrop Grumman 18 hours to fix it. The priority was assigned based on the number of employees at the facility – 30 to 40 – not on the need dictated by the fact that the facility houses 1,000 inmates.
The state has tried extending deadlines, withholding payments and levying penalties in order to resolve the problems, but they persist, according to the JLARC report. As things stand now, the state could get out of the contract, which runs through 2015, but it could cost as much as $399 million to do so, and the state would still have to pay for someone else to run the network or develop state resources to do so.
The report cites benefits of the contract with Northrop Grumman. These include replacement of networking equipment – 45,200 of 57,500 PCs - to standardize the infrastructure across the state that was scheduled to be completed in last July. Replacing aging PCs actually enabled applications that older PCs couldn’t support, one large agency says.
Internet traffic is funneled through a gateway rather than hundreds of individual agency connections. A program to monitor for outages and performance and to alert the help desk to the most severe problems is underway.
A transition to a single e-mail infrastructure is underway, with 26,200 of 63,500 accounts having been migrated. The new e-mail system rides on the state network rather than the Internet and is screened for viruses and spam. The help desk has been centralized, and more online user self-help sites are available.
Even so, state agencies pan many of the services. The report says 44% of agencies find the disaster recovery and backup services are inadequate. Network services were ranked poor by 41% of agencies.
The help desk was rated poor by 48% of agencies with complaints including calls being routed to technicians with the wrong expertise and with complaints being assigned lower priority to trigger longer resolution times.
In a letter responding to the interim report, Northrop Grumman’s vice president and general manager of the firm’s Civil Systems Division Tom Shelman says the project is the first of its kind and that no other project on this scale exists. “Virginia is breaking new ground and should be proud of that fact in spite of the challenges we all acknowledge exist. Other states will follow the lead and draw on the Virginia experience.”
The work to be done includes 59 projects over 72 agencies at more than 2,000 sites and encompasses replacing PCs, servers, mainframes, e-mail, network, security, help desk and telecom. “Projects are interdependent and delays with one project can have cascading effects on other projects,” the report says.
The work was to be done by July of this year, and a corrective action plan was filed in August promising to finish by June 2010, but it leaves out four agencies. As of September, 32 of 59 projects were completed. The report says the delays happened because “agency needs have not been fully addressed or fully understood.”
