September 16, 2013, 3:52 PM — A Microsoft MVP -- Most Valued Professional -- and Windows expert has sent company CEO Steve Ballmer a letter asking him to look into the worrisome trend of sub-standard patches that crippled computers, forced IT personnel to scramble to undo snafus and damaged Microsoft's hard-earned reputation.
Susan Bradley, one of the moderators of the Patchmanagement.org email list -- called a "listserv" -- who also frequently offers free advice on Microsoft's support forums and writes a weekly column on patching for the "Windows Secrets" newsletter, posted her Ballmer letter to the list last Wednesday.
"On behalf of everyone in this community, may I respectfully request that you assign someone in a management position to investigate what is going on with quality control with patch testing lately?" Bradley asked Ballmer.
"This month in particular leaves me deeply disturbed that issues that should have been found before these updates were released are being found by us -- your customers -- after they are released and we are having to deal with the aftermath," Bradley continued. "Bottom line, sir, this is unacceptable to all of us in the patching community, and quite frankly, it should be just as unacceptable to you."
Bradley cited issues with many of the Sept. 10 updates, including one that emptied the Outlook 2013 folder pane and four others that repeatedly demanded customers install them even after they had been deployed.
Microsoft's patch problem goes further back: In August, the Redmond, Wash. company yanked an Exchange security update, admitting it had not properly tested the patches. And in April, Microsoft urged Windows 7 users to uninstall an update that crippled PCs with the infamous "Blue Screen of Death;" it re-released the update two weeks later.
While Bradley hasn't received a reply from Ballmer, she gave a tip of the hat to one of the two Microsoft managers who weighed in on the mailing list.
"The fact that Gray Knowlton from the Office team joined the Patchmanagement.org listserv is a huge start in the right direction towards better communications," Bradley said in an email reply to questions today. "Kudos to Gray for that."
Knowlton, a principal group program manager for Office, gave Microsoft's most detailed account yet for the September screw-ups in a Friday message to the listserv.
"Both of these errors are anomalies in our release operation," said Knowlton. "The XML config[uration] entries had to be hand-authored due to some product code changes. We rarely do this; they are typically machine-generated. In [the blank folder pane in Outlook 2013], a late change to the list of things we intended to ship resulted in a specific configuration not executing as expected."
Knowlton argued that the quality level for Office updates is "very high" considering the volume of updates issued and the number of customers who apply them. He also promised that the quality of patches would improve -- a message Microsoft has used before -- saying, "We are as concerned as any of our customers about these issues and we will come back in October better than we were before September."
Another Microsoft manager, however, sounded peeved that Bradley had emailed the CEO.
"We are following up with the people who published those updates. And no, it's not because Mr. Ballmer intervened," wrote Ben Herila, who identified himself as the program manager for WSUS (Windows Server Update Services), the widely used enterprise patch management service Microsoft runs. "Rather, it's because Susan so kindly let us (the WSUS product team) know about her problem."
Dustin Childs, a group manager of Microsoft's Trustworthy Computing group, also alluded to doing something -- he did not specify what -- to put a stop to the mistakes. "The quality of security updates is critical to our customers, and it is a high priority for us, too," Childs said. "We are actively looking at where improvements can be made with the goal of reducing implementation issues, and we will remain transparent with our customers about security threats, protections and update issue resolution."
It may take a lot more than words to calm the roiled waters.
"Not only are the end users suffering by these bad patches, the IT administrators are suffering even more because they have to hear all of the complaints from the end users and they have to spend time troubleshooting the issues and get things fixed," wrote John Hallis on the same mailing list thread. "You would think a company that has received billions of dollars from us would actually listen to what we are telling them about patching issues and get right on it."
And Bradley saw the problem as endemic at Microsoft.
"I think that releasing 80 non-security updates on an already busy patch month is releasing way too much code at one time," she said via in an email to Computerworld today. "You are going to get stuff missed."
Like other patch and security professionals, she cited the advantage baked into the cloud when compared to on-premise software. "Cloud gets a build to build deployment and thus when Exchange 2013 got its first security update, their cloud servers were fine, [but] on-premise servers barfed," she said, referring to the August update gaffe involving Exchange.
But she also blamed overstretch for the slide in quality.
"My rant wasn't just about the quality of security updates -- but the quality of patching as a whole," Bradley said. "Documentation is lacking, quality of updates -- especially in certain categories of updates -- is clearly lacking.
"I'm not paranoid enough to believe that this is Microsoft's way to showcase how it will be better in the cloud where they patch and deal with these issues. I'm not naive enough to believe that even once we all are in the cloud that we will suffer no patching issues.
"I feel that they are just managing a lot of different kinds of problems and patching [and] along with the faster cadence, there are just a lot more moving parts to keep track of these days ... and things are slipping through the cracks."
Microsoft's next regularly-scheduled security updates are to ship Oct. 8.
Gregg Keizer covers Microsoft, security issues, Apple, Web browsers and general technology breaking news for Computerworld. Follow Gregg on Twitter at @gkeizer, on Google+ or subscribe to Gregg's RSS feed. His email address is email@example.com.
Read more about malware and vulnerabilities in Computerworld's Malware and Vulnerabilities Topic Center.