CIA's 'Vengeful Librarians' read mood of foreign streets using Twitter, Facebook

It's not 'Burn Notice' but the intelligence it gathers is often more accurate than espionage

It must be traumatic for the majority of CIA employees who operate desks rather than networks of covert assets in foreign countries to shift their self image from the suavely violent Michael Weston, to what the AP reports are the "affectionately" nicknamed "vengeful librarians."

The AP story is an exclusive and the revelation that CIA analysts look at as many as 5 million tweets, blogs, Facebook updates and other social-network postings per day is a little disturbing.

CIA analysts have been doing the same thing with newspapers, magazines, radio and TV broadcasts, the text of speeches and every other source of public information they could get from or about foreign countries ever since the agency was founded.

The Vengeful Librarians' group is relatively new – formed at the recommendation of the 9/11 Commission, which noticed that the CIA didn't predict attacks anything like the thoroughly planned, highly coordinated four-plane hijacking and Kamikaze mission al-Queda had been working on for years.

Of course, there are a lot of things the CIA could have predicted, but didn't: the Korean War, much of anything useful about the Vietnam war, the fall of the Berlin Wall and end of Communism in the Soviet Union or, more recently, the persistence, extent and sources of the insurgency (not to mention the imaginary WMDs) in Iraq following the U.S. invasion.

Major errors are inevitable in international intelligence gathering and analysis, many experts argue.

The CIA, and most other intelligence agencies, is divided into Operational groups that gather intelligence through local assets (spies and traitors), and departments focused on analyzing that data to figure out what it means and what else the United States needs to know.

It's not hard to figure out which parts are still missing from the schematics of a new Russian fighter jet, or to ask Operations to see if someone can't pick those up and fax a copy over.

It's a lot harder to gauge the mood and likely behavior of populations in countries in which the CIA has not been able to build a significant network of local assets – expensive, dangerous, difficult work that never produces a complete picture even of the specific areas on which the Agency wants to focus.

"The word on the street" comes in at 100Mbit/sec.

When it comes to what overwrought TV reporters often call "the mood of the street," however – the attitudes and likely behavior of the populations in countries in which the CIA may or may not have networks of spies with accurate local knowledge – there are few better sources than actually hearing voices from the street.

Hearing those voices and figuring out what they mean is the mission of the CIA's Open Source Center – a large department with the agency's analysis wing is responsible for gathering intelligence from open sources of information like newspapers and TV and, now, Facebook, Twitter and every other social-networking site they can log or crack their way into.

Just by listening to what locals were complaining about – and how – the center was able to predict the uprising in Egypt and that social media would break out of the control of the Mubarak regime and become a critical asset to the dissidents, according to Open Sources Center director Doug Naquin, as quoted in the AP.

Open Source Center analysts are not typical spies; they're not even typical CIA analysts.

They need the deep knowledge of foreign cultures and advanced language skills of typical analysts; those who speak several languages and grew up speaking more than two "make a powerful open source officer," AP quotes Naquin as saying.

For spies, they're quite nice

Open Source Center people are even polite – especially for spies. I came across their site a couple of months ago accidentally, and initially thought it was just some sort of clearinghouse for government reports. Even after I figured it out, and realized there was no access without a login granted by some authority at CIA, the site still looked like a good source of non-classified information.

So I wrote to ask for press access.

I got a very polite note back in a few days saying the OSC was charged with providing information to government and military employees and contractors and, unfortunately, its charter didn't permit access from the press or public. If I could demonstrate some connection with state or federal government, though, they'd be glad to reconsider my request.

It was the most polite refusal I've ever gotten from a clandestine intelligence service.

As far as I know.

Skillsets a little different from the average spy's

Rather than specialties in politics, military engineering, military sciences or other specialties that help other departments figure out how good that Russian jet is going to be, Open Source researchers often have backgrounds in library science or other humanities that are heavier on textual and contextual research than on nuts, bolts and calculations.

Even the Open-Source group didn't start focusing on social networks heavily until after the Green Revolution in Iran – months of often violent street protests over an election many claimed was stolen by Iranian President Mahmoud Ahmadinejad.

Farsi – the most common language in Iran – was also the third most common in social media blogs across the whole World Wide Web, Naquin said.

That helped many in the Agency realize that just being able to understand the gabble of foreign mobs could be a rich source of intelligence (despite the often low-seeming intelligence of the gabble on U.S. based social nets).

The value of social networks as an accurate gauge of national moods and intentions isn't proven yet, Naquin said.

Hearing the voice of the street, even when the streets are on fire

The Open Source Center did accurately estimate the impact of a speech about Middle Eastern issues President Obama gave several weeks after the U.S. raid that killed Osama bin Laden. Arabic and Turkic tweets portrayed Obama as favoring Israel; those in Hebrew charged he was pro-Arab, Naquin told AP.

In that case, at least, the Center was ahead of the media, which came to the same conclusion several days later based on local opinion polls.

The Center continues to measure its results against polling data to identify where its accuracy is particularly good and what its weaknesses may be.

People in developing countries who use social networks heavily tend to be more urban and more educated than average for their regions, Naquin said.

That bias is much greater in most countries than results would be from intercepted cell-phone calls from those same countries, because Internet access reliable enough to develop locals into social networkers is still more rare than cell-phone networks.

Social nets do provide something the news media, networks of spies and even cell-phone intercepts can't, however: up-to-the-minute updates on fast-moving crises that limit traditional sources of information.

In April and May of 2010, the AP quotes an anonymous CIA operative who sometimes works undercover in the area as saying, the riots that tore through Bangkok left U.S. decision makers out of the loop.

Most reporters in the area were trapped in hotels or other locations by government crackdowns or the risk of being attacked by mobs; embassy staff were trapped in place as well.

"Within an hour, it was all surging out on Twitter and Facebook," the CIA source said.

The Center took in all the reports it could get, cross-referenced them with local news and other sources to figure out which were providing reliable information, and stuck with those throughout the crisis.

About two thirds of the reports the Embassy sent or approved from the area came from the Open Source Center and its largely-unknowing network of Twitter and Facebook users.

Is 'Open Source' intelligence insidious? Or educational?

There are lots of negative reactions to all this in the same venues the Center monitors in other countries.

As much as I think agencies of the U.S. government should limit their surveillance of Americans and be restricted by judicial oversight and limits from the First and Fourth Amendments, I can't honestly see a huge problem with intelligence agencies listening in on Twitter and Facebook conversations.

Social networks, by definition, are public. What you say on them you may intend for a circle of your friends, but it's been years since anyone has believed a word said or a picture posted on either service would stay within a particular circle.

Graduating college students have known for 10 years to edit the party pictures out of their social-network profiles and limit how often their friends tag them.

Managing the public profile of companies and individuals online has become a huge business and a routine part of any public relations campaign.

It is not a secret that anyone in the country could read the potentially embarrassing tweet you just posted. It would not be a surprise to see that embarrassing video turn up on Tosh.0 or Web Soup, let alone in the briefing books of intelligence operatives in foreign countries.

Actually listening to what relatively normal people (i.e. not only the ones in power) have to say about U.S. policies toward their country, the offenses of their own leaders or the tendency of those leaders to turn into some other power's lapdog through an inability to offend can only give the CIA and (hopefully) other branches of government a better idea of who they're dealing with in foreign policy.

The expectation that U.S. troops would be greeted in Baghdad by crowds throwing flowers instead of grenades sounds ludicrous after a decade of insurgency.

Listening a little closer to the voice of the street – or at least the voice of the Baghdad Internets – might have given all the decision makers a little clearer idea of what they were getting us all into.

The question is – both for the Bush administration and now Obama's – would they listen?

Read more of Kevin Fogarty's CoreIT blog and follow the latest IT news at ITworld. Follow Kevin on Twitter at @KevinFogarty. For the latest IT news, analysis and how-tos, follow ITworld on Twitter and Facebook.

ITWorld DealPost: The best in tech deals and discounts.
Shop Tech Products at Amazon