Washington News Review: Exclusive: Inside Darpa’s Secret Afghan Spy Machine

By Noah Shachtman
July 21, 2011 |

The Pentagon’s top researchers have rushed a classified and controversial intelligence program into Afghanistan. Known as “Nexus 7,” and previously undisclosed as a war-zone surveillance effort, it ties together everything from spy radars to fruit prices in order to glean clues about Afghan instability.

The program has been pushed hard by the leadership of the Defense Advanced Research Projects Agency. They see Nexus 7 as both a breakthrough data-analysis tool and an opportunity to move beyond its traditional, long-range research role and into a more active wartime mission.

But those efforts are drawing fire from some frontline intel operators who see Nexus 7 as little more than a glorified grad-school project, wasting tens of millions on duplicative technology that has nothing to do with stopping the Taliban.

“There are no models and there are no algorithms,” says one person familiar with the program, echoing numerous others who spoke on condition of anonymity because they are not authorized to discuss the program publicly. Just “200 lines of buggy Python code to do what imagery analysts do every day.”

During a decade of war, American forces have gathered exabytes of information on its enemies in Afghanistan. Nexus 7 aims to tap that data to find out more about the U.S.’ alleged friends: the people of Afghanistan, and how they interact with their government and with one another.

Not that you’d be able to figure that out, examining the one public reference to Nexus 7. Tucked away in the Pentagon’s gargantuan budget (.pdf), it makes the program sound like an obscure computer-science project, using “cluster analysis” to find “social networks.” There’s no reference to its operational utility.

On the military’s classified network, however, Darpa technologists pitch Nexus 7 as far-reaching and revolutionary, culling “hundreds of existing data sources from multiple Agencies and Services” to produce “population-centric, cultural intelligence.”

They boast of Nexus 7’s ties to special operations and to America’s most secretive surveillance groups, and its sophisticated tools to “perform automated cross-correlation and analysis of massive, sparse datasets — recomputing stability indicators within minutes of new data updates.”

In practice, that means Nexus 7 culls the vast U.S. spy apparatus to figure out which communities in Afghanistan are falling apart and which are stabilizing; which are loyal to the government in Kabul and which are falling under the influence of the militants.

A small Nexus 7 team is currently working in Afghanistan with military-intelligence officers, while a much larger group in Virginia with a “large-scale processing capacity” handles the bulk of the data crunching, Darpa spokesman Eric Mazzacone confirmed in e-mails with Danger Room. “Data in the hands of some of the best computer scientists working side by side with operators provides useful insights in ways that might not have otherwise been realized.”

That sometimes means turning traditional intelligence work on its head. Instead of using all those eyes in the sky and reports from the ground to hunt for the proverbial needle in the haystack –- the lone insurgent in a large group of people –- Nexus 7 sometimes examines the makeup of the entire haystack. Of everyone.

“Let’s take that God’s-eye view,” says one person familiar with the program. “Instead of tracking a car, why not track all cars?”

The most senior officers in the military have all been briefed on the program, as has incoming CIA director David Petraeus. And whether it succeeds or fails, the project raises questions about the role of the government’s most-celebrated technologists and the direction of the war effort in Afghanistan.

Should the United States even bother with a “population-centric” counterinsurgency there, or just target militants? Should Darpa focus on those wartime efforts, or stay focused on the long-term research that has helped the agency reshape the world again and again?

But the most eye-opening aspect of Nexus 7 might not be the questions it raises, or its grand ambition, or its secret nature, or the controversy it has generated. It’s the fact that a program this weighty started with a quirky contest to find a bunch of red balloons.

Nexus 7 has many intellectual godfathers. One is David Kilcullen, the retired Australian Lieutenant Colonel who became a rock star in national security circles as a guru of counterinsurgency, which he described as a “competition … to win the hearts, minds and acquiescence of the population.” (.pdf) The reputation only grew, after he became a top advisor to Gen. David Petraeus in Iraq.

As his career moved on, Kilcullen became increasingly focused on numbers. In a traditional battle, it’s easy to measure territory gained on enemies killed. But how do you gauge something as squishy as a town’s loyalty? How do you tell if those allegiances are getting stronger or weaker over time?

The military spent hundreds of millions of dollars on everything from war-zone polling to computer modeling to figure that out. None of it seemed to work. But Kilcullen had a few ideas.

How do you gauge something as squishy as a town’s loyalty? The military spent hundreds of millions to figure that out. None of it seemed to work.

The best troops could do, Kilcullen concluded, was to collect indirect metrics: the price of goods in the local market, or the hassles of getting the products there. They’re decent surrogates for “general popular confidence and security,” he wrote. “In particular, exotic vegetables — those grown outside a particular district that have to be transported further at greater risk in order to be sold in that district — can be a useful telltale marker.” (.pdf)

A second Nexus 7 godfather is Maj. Gen. Michael Flynn. Until recently, Flynn was the head of U.S. intelligence in Afghanistan.

But he didn’t always think too highly of the apparatus he ran. In report publicly released in December 2009 , Flynn excoriated his fellow intelligence professionals for being “only marginally relevant to the overall strategy.”

They were so focused on old-school metrics like body counts, he complained, they hadn’t bothered to learn the first thing about Afghanistan’s people. Rudimentary questions about Afghanistan’s social and cultural fabric had gone largely unasked and unanswered.

But Flynn also offered the intelligence community a way out. The U.S. military had in its databases a “vast and underappreciated body of information,” he wrote. Tapped right, that information could form “a map for leveraging popular support and marginalizing the insurgency.”

Assembling data on how a society functions is a passion of Alex “Sandy” Pentland, the bushy-bearded MIT Media Lab professor and evangelist for a new type of information-gathering dubbed “reality mining.”

With GPS-enabled cellphones and advanced surveillance cameras now everywhere, it’s possible to track almost everyone in a given area at once. That’s not creepy, in Pentland’s world. It’s wonderful. Because all that information can tell you exactly where a town is working and where it’s broken, where the traffic piles up and where it flows free.

Public actions, when added up, could serve as one of Kilcullen’s indirect metrics.

“People leave behind digital traces. If you aggregate those traces in the right way, you can see where incomes are dropping, where people feel scared, where the babies are dying,” he tells Danger Room. “With technology comes transparency. And with transparency comes an ability to see when things work, and when they go off the rails.”

At first, Pentland used reality mining to grok workplace interactions at Bank of America and find cabs in New York City. But over time, more of his attention turned to places like Afghanistan. Maybe war zones could use some reality mining, too. He began pitching military officials on his notion of “computational counterinsurgency.”

He found a willing audience at Darpa.

Pentland became something of a hero to the agency in December 2009, just a few days before Flynn released his controversial paper. Darpa had launched an attention-grabbing contest, to find 10 red balloons placed around the country.

On the surface, the challenge seemed simple enough. But tracking down all of the 8-foot-wide spheres was considered by some U.S. intelligence professionals to be “impossible by conventional intelligence-gathering methods.”

Under the supervision of Pentland (pictured, left) and his graduate student Galen Pickard (center), a team from MIT recruited 4,400 people online for the balloon hunt. Together, they tracked down all 10 in just eight hours and 52 minutes.

Regina Dugan, the new director at Darpa, had bet big on crowdsourcing — the idea that smart ideas and big tasks could be accomplished by opening them up to large groups. Thanks to Pentland’s team, Dugan’s bet appeared to be a good one. He gave Dugan her first major, public win.

Dugan was looking to do more than turn heads with a quirky contest, however. To her, the Defense Department’s leading research shop had been stuck at quirky for too long — pursuing pet projects that didn’t seem to have much battlefield relevance.

‘Darpa is not the place of dreamlike musings.’ For an agency known for its shape-shifting robots, that was a revolutionary statement.

Darpa, of course, had a long history of big, risky ideas that paid off indirectly and way, way down the road: stealth technology, GPS, the internet itself. No other government agency was tasked with thinking that wild and that far into the future.

Darpa, in effect, supplied the seed corn for American ingenuity.

Under Dugan’s predecessor, Tony Tether, Darpa program managers were encouraged to chase their interests in fields like artificial intelligence and quantum computing; the military implications would be figured out later.

To Dugan, that was unacceptable during wartime. Darpa had to be visionary, of course. But the agency had drifted too far into the clouds.

“There is a time and a place for daydreaming. But it is not at Darpa,” she told a Congressional panel last March (.pdf). “Darpa is not the place of dreamlike musings or fantasies, not a place for self-indulging in wishes and hopes. Darpa is a place of doing.”

For an agency that spent millions of dollars on shape-shifting robots and mind-controlled limbs, it was something of a revolutionary statement.

A trip to Afghanistan, a few months after the contest, only reinforced that view, Dugan added. The officers there “did not believe Darpa was in the fight with them.”

Maybe Pentland and his MIT crew could help. Military intelligence specialists, still reeling from Flynn’s scathing report, were looking for new ways to refocus on the people and society of Afghanistan. Over the years, the military had refined their tools into a lethally effective machine for hunting individuals.

But tracking people in the aggregate — that was still beyond it. Pentland’s “reality mining” concept seemed to have a chance of fixing that. By plumbing the depths of the military’s intelligence databases — and correlating that with people’s movements — they could assemble some indicators of whether a particular region was recovering from war, or going to hell.

“State of the art in ISR [intelligence surveillance and reconnaissance] has produced massive amounts of data that is difficult to process using conventional approaches,” as Mazzacone, the Darpa spokesman, puts it. “The DoD [Department of Defense] invests billions of dollars in ISR. Having a better means of analyzing that sensor data has the potential to save lives, increase mission success and save money.”

In the spring of 2010, plans quickly took shape for a 90-day pilot project to see if Afghanistan could be reality mined. Galen Pickard, the red-balloon alumnus and Pentland disciple, would help write code to scour various military datasets, and fuse the information together.

David Kilcullen, who had advocated society-centric metrics for so long, would put together a team counterinsurgency experts and social scientists to figure out what the data all meant. Employees from Kilcullen’s new consultancy — Caerus Associates, named after the personification of opportunity in Greek myth — started schooling the geeks in counterinsurgency and the lay of the Afghan land.

A few days before Kilcullen’s wedding to Pentagon policy official Janine Davidson (a wedding attended by Petraeus, among others) that pilot project launched.

Hopes soared. Then, just as quickly, they came hurtling back down to the ground.

In Nexus 7, the geeks saw a chance to use their skills to do something a lot more important than find balloons. Kilcullen’s crew hoped to find those slippery counterinsurgency metrics that had eluded the military for so long. Maybe they’d even be able to prove empirically whether all that stuff they preached about winning hearts and minds was really true.

“It’s a big opportunity to test COIN [counterinsurgency] theory with as much data as you ever wanted,” one source familiar with the program says.

Step one was to dive into SIGACTS, the military database that contained accounts of nearly every firefight American troops fought. (The information later formed the bedrock of WikiLeaks’ “war logs.”)

Drizzled between the gun battles were occasional accounts of villages stabilized and town elders met. But, written as random notes, the accounts were hard to insert into a database. There was nothing consistent, nothing you could plot as a trend over time.

“These were intelligence reports, not measurable data,” the source says. “The population-centric information wasn’t to be found there.”

‘One assumed there was some secret mound of data to be exploited. But it’s just not true.’

So the team widened their search, without much luck. The most reliable data they could find was weekly fruit prices from Jalalabad, a city in northeastern Afghanistan. At least those could be measured over time.

“One assumed there was some secret mound of data to be exploited. But it’s just not true,” the source adds.

At the end of the summer, the 90 days for the pilot project were up. It was time to demonstrate to Dugan — and the rest of the military — what Nexus 7 could do. In a conference room on the seventh floor of Darpa’s Arlington, Virginia, headquarters, the team made a series of presentations, to show what they could do.

On the surface, there wasn’t much to it: just a graph of violence in the Jalalabad region, and a plot of those fruit prices. When the level of violence was stable — reliably low, or reliably high — so were those prices. Fruit sellers knew what to expect. But when there were sudden swings in the number of attacks, the prices shot up.

Therefore, the Nexus 7 team said, you could use the fruit as an indirect indicator of instability.

The reaction was less than rapturous.

“Right from the start, I’m like: Oh. My. God,” one of the people who attended a Nexus 7 presentation tells Danger Room. “A high school kid could do that.”

Afterward, Dugan presented the pilot as a triumph — a “big breakthrough” that impressed a bevy of four-star generals.

Privately, she was underwhelmed. Dugan was looking for projects that could save troops’ lives, and maybe even bend the direction of the war. By that standard, fruit-price swings seemed pretty inconsequential.

But the presenters maintained an aura of confidence. Oh, this is just a test. Give us more data sources, they said, and we’ll make better connections. We’ve got the hardware: a cloud computing platform that would soak up all kinds of classified and open source intelligence data. We’ve got the software: these social science PhDs and counterinsurgency veterans, who can figure out how to apply that data to rebuild Afghanistan.

“They led us to believe they had access to all this data, and they could share with us. They said they had a working intel center in Kabul, and were briefing Petraeus all the time,” one meeting attendee says.

He — and many of the others — were skeptical. But they figured they’d take a risk, and give the Darpa team some data to play with. Maybe they’d come up with something cool. After all, “these guys are geniuses; they invented the internet, right?”

A team from the National Security Agency — the super-secret eavesdropping service — offered to give so-called “minimized summaries” of phone-call records to the Nexus 7 team in return for data about Afghanistan’s many redevelopment projects.

The summaries wouldn’t tell the Nexus 7 team who was calling whom, or what the contents of the conversations were. (Darpa wasn’t authorized to handle that kind of raw intelligence data, anyway.) But it could give a sense of the ebb and flow of communications — another input to be reality mined.

Despite the misgivings, the program was authorized. On Sept. 16, a $6.1 million contract was signed, effective immediately. Nexus 7 was heading to Afghanistan.

This wasn’t the first time Darpa had gone to war.

In 1961, with the situation in Vietnam already slipping, the Advanced Research Projects Agency (“Arpa,” the “D” came later) launched its wide-ranging Project Agile. As Danger Room co-founder Sharon Weinberger recounts, the array of research efforts produced everything from the M-16 rifle to now-infamous defoliant Agent Orange to “a jet belt designed to propel individual soldiers on the battlefield.”

In the weeks after 9/11, Darpa launched a series of efforts designed to help intelligence analysts comb disparate databases for terror threats. The best-known of those efforts was called “Total Information Awareness,” or TIA. And it aimed to collect as much information from as many people as possible — e-mails, credit card statements, even veterinarian bills — in order to find a signature of terrorist behavior.

Congress freaked out over TIA’s potential to be an all-seeing eye. Darpa was forced to drop it and several related efforts from its public portfolio.

After a single, incomplete test, Nexus 7 was shipping out for war.

But Darpa-backed gadgets did become staples of the war efforts in Iraq and Afghanistan. The agency’s handheld Phraselator translation gadgets gave troops a rudimentary way to communicate in a foreign tongue. Its Boomerang gunshot detector uncovered enemy snipers. Its Command Post of the Future (CPOF) allowed officers to collaboratively plan attacks, and instantly spot every friendly vehicle on the battlefield. CPOF became the de facto standard for mission planners everywhere. Most recently, Darpa pushed out to the field a wide-area, high-def laser-ranging system that collected 3-D map data 10 times quicker than its predecessors.

All of these efforts were the product of years of testing and evaluation. CPOF got its start in 1999, four years before it was first taken to Iraq. Phraselator inventor Lee Morin was winning military awards for his translation tech in 1994. The laser-ranging system was five years in the making.

Nexus 7 was on an entirely different timetable. Dugan was determined to have Darpa make a difference in the war effort — not after years of development, but now. After a single 90-day test, Nexus 7 was shipping out for deployment.

The program’s geeks and social scientists loved it, of course. It was a way to turbocharge their research, and make an immediate impact. Why bother holing Nexus 7 up at a stateside test bed, one person familiar with Nexus 7 asks, “when you can give it to a company in Afghanistan and get 1,000 times the number of observations? It’s not like these are weapons. If it doesn’t work, the worst that happens is it doesn’t work.”

That devil-may-care attitude didn’t extend to Kabul, where Nexus 7 almost immediately set off a series of bureaucratic knife fights in the command center of the U.S.-led coalition in Afghanistan.

Darpa is breathlessly credited with inventing the internet. Dugan calls the agency her “elite army of futuristic technogeeks.” But it’s not like Darpa program managers are writing code or planting electrodes in telekinetic monkeys in the basement of the agency’s HQ. Darpa acts less like an army and more like a bank — of ideas, and of cash.

A program manager ordinarily puts out a “Broad Agency Announcement,” or BAA, outlining her new science or technology goals. Academics and defense contractors submit research proposals to meet those targets. The winners get Darpa cash, and carry out the needed experiments in their university or corporate labs. The telekinetic monkeys, for instance, are at Duke University.

Dugan rarely gets involved with an individual project or a particular researcher. Quite the opposite. Some program managers, who used to spend hours briefing the director on their projects, now say their annual audiences with Dugan last no more than 60 seconds.

Nexus 7 was the outlier. No BAA was ever issued for this program, which is supposed to cost $30 million in the next fiscal year. About 20 employees of Caerus Associates, Potomac Fusion, along with Data Tactics Corporation and other firms work directly in Darpa’s headquarters, not in some outside facility.

In theory, these contractors report to Nexus 7 program manager Randy Garrett; in practice, Dugan is the one holding the reins.

“She spent an inordinate amount of time on it,” says one Darpa staffer. “In all-hands meetings, she’d tout it as one of the three or four projects that’s changing the world.”

‘There’s no such thing as Nexus 7 data. There’s no Nexus 7 analytics. No computing. No cloud; just five dudes with laptops.’

In the fall, Dugan dispatched red-balloon alum Galen Pickard (pictured) to Kabul, joining fellow graduate student Chris White. Neither was familiar with working in a military setting, and it showed.

They’d plant themselves in the Joint Intelligence Operations Center (JIOC) in Kabul without introducing themselves to the officers in charge. This is a facility packed with top-secret information; strangers weren’t exactly welcome. They’d bring their personal, unclassified laptops into this secure location, where every piece of hardware was supposed to be vetted and approved.

This was, at minimum, a serious violation of information-handling rules — the kind of transgression that allowed Private First Class Bradley Manning to smuggle hundreds of thousands of documents to WikiLeaks.

The grad students also were less than collegial with others trying to steer the vast U.S. intelligence enterprise toward learning more about Afghan society. Nexus 7 was one of several attempts to do so. But when officers and contractors working on these other projects would try to share information with the grad students, they wouldn’t say a word about what they were doing.

The pros couldn’t decide if the amateurs were being needlessly squirrelly — or completely clueless.

“This guy doesn’t know anything,” one complained, after a meeting with White. “You couldn’t pin him down on anything. Couldn’t define a goal. Couldn’t say what cooperation means, what data sharing means. He artfully weaseled his way out of everything.”

Friendly observers chalked the reticence up to a kind of post-traumatic stress — trauma, as in the beating Darpa took during the “Total Information Awareness” days. Agency insiders knew that big data-mining programs could be spun in a very unkind way. “There’s an irrational sensitivity built up like an aurora around this program,” one Darpa-watcher says.

Nerves were even further frayed when Dugan came to Kabul. In a meeting with Petraeus, she talked about how good her new intelligence-crunching efforts were — and how screwed-up his spy shop was. Petraeus, who had spent the better part of the last decade in Iraq and Afghanistan, didn’t appreciate a war-zone dilettante telling him how to do his job.

But Petraeus wasn’t one to pass up a new, potentially useful intelligence tool — especially not one backed by Kilcullen, his old confidant. White and Pickard were given access to the log files from overhead surveillance radars.

This “GMTI” information (short for Ground Moving Target Indicator) was useful in showing where vehicles moved over time. The military had used the data for years to track the travels of potential foes (.pdf). For the Nexus 7 team, it was ore to be reality mined.

Instead of tracking a single vehicle, they looked at all of them, in aggregate. They watched out for “low-pressure centers” that seemed to suck cars in; maybe it was an indicator of a thriving local market. They saw what roads the locals avoided (a possible indication of Taliban checkpoints nearby) and what paths they used instead.

The data was inconsistent — the drones and other aircraft carrying the radars didn’t consistently fly over the same places at regular intervals. The exact lessons that could be extracted from the GMTIs weren’t always clear. But a fuzzy picture, the Nexus 7 team figured, was better than no picture at all.

Perhaps it was professional jealousy. Perhaps it was a lack of understanding of the program — the Cro-Magnons not getting what the Homo Sapiens were up to. Perhaps it was the entitled way that the Nexus 7 grad students seemed to carry themselves. Perhaps it was the security slip-ups. Perhaps it was simply intelligence bureaucrats protecting their turf.

Or maybe the Nexus 7 work was just that bad. For whatever reason, intelligence specialists in Afghanistan were openly complaining about the effort by February. Tracking GMTI movements over time — that was old hat.

“But when Darpa briefs the [PowerPoint] slides, though, it suddenly has something to do with ‘networks of networks’ and stability, governance, blah, blah, blah,” says one person in Kabul familiar with the project. “There are no models and there are no algorithms. It’s just GMTI and slides.”

‘There are no models and there are no algorithms. It’s just radar tracks and slides.’

The view was far from universal. Gen. James “Hoss” Cartwright, the Vice Chairman of the Joint Chiefs of Staff and the most tech-savvy of America’s senior officers, was impressed with Nexus 7, warts and all. And he liked that Darpa wasn’t content to let the war drag on without them.

“One of the strengths Darpa brings to operations is an ability to meld huge pools of data in new ways and use it to map terrain in great detail, track patterns of life, and improve our understanding of the warfighting environment,” Cartwright said in an e-mail to Danger Room.

Both in Washington and in Kabul, the Nexus 7 team went looking for ways to make that big data pool even bigger. They asked for financial tallies, reconstruction reports, even phone records.

As the winter grew on, they got more aggressive, saying that military agencies and spy services “owed” Darpa information. More often than not, they were turned away, supposedly because the Nexus 7 team didn’t have the proper clearances or the officially accredited systems to hold the data.

There were also heated debates about whether Darpa is even authorized to be involved with this processing of raw intelligence. Executive Order 12333 is quite explicit that phone taps, for example, are to be left up to the NSA, which is “the Functional Manager for signals intelligence … control[ling] signals intelligence collection and processing activities, including … the direct support of military commanders.”

Eventually, some intelligence personnel would complain to Petraeus’ brain trust about the requests.

“I told them: There’s no such thing as Nexus 7 data; we already have all that,” a source familiar with program says.

For now, Nexus 7 is continuing to roll along. On its website, Kilcullen’s firm touts its “mash-up [of] high-capacity data processing with cutting edge social-science analytical methodologies to enable enhanced remote observation and extended situational awareness, monitoring and evaluation, and decision-making capability.”

(Caerus Associates declined to comment for this story. But, full disclosure: Kilcullen’s company employs several friends of this blog. His wife was my predecessor as a non-resident fellow at the Brookings Institution’s 21st Century Defense Initiative.)

In her public speeches, Dugan (pictured) proudly discusses her “90-day Skunkworks activity that brought together some of the country’s best computer and social scientists, counterinsurgency experts, economists and analysts; advanced training tools; and organically developed capabilities that harness crowd sourcing and social networking technologies.”

Troops are starting to withdraw from Afghanistan. Nexus 7 is being positioned to take their place.

“I have watched young men and women, some as young as 27, go toe to toe with four-star generals,” she adds. “Because it mattered, and because it had become deeply personal to them. They decided they could make a difference, so they got in the fight. It is their way of serving [their] country.”

If there’s been any blowback from the controversies surrounding Nexus 7 — or surrounding Dugan’s questionable business dealings with her family firm — they don’t appear to have affected her standing with the President. The two recently appeared together in Pittsburgh, to promote Darpa’s manufacturing initiatives.

Meanwhile, the Nexus 7 team seems to be positioning itself for the long term — even as the war’s strategy moves away from counterinsurgency, and toward a campaign of taking out individual militants. Troops are starting, ever-so-slowly, to withdraw from Afghanistan.

So, Nexus 7’s backers argue that a bit of reality mining might be able to take their place. Nexus 7 relies “principally [on] remote collection,” according to its secret website. “Consequentially, analysis can be performed in areas without Coalition presence. ‘Observer effects’ are minimized and stability indicators are scalable into geographic areas where we historically or currently have no physical presence.”

“The visionaries have begun to get it,” Pentland tells me. “If you get transparency, you don’t need boots on the ground.”

Photos: USAF, Army, Darpa, Flickr/Todd Huffman, Wikimedia, Facebook, Flickr/ExpertInfantry, Flickr/CSUFNewsPhotos