This week's deadly attacks in France by Islamist gunmen showed the limits of spy and anti-terrorist agencies, which often have information about perpetrators in advance but are only able to assemble all the clues after the bloodletting has taken place.
From the Sept. 11 attacks in the United States in 2001 through a series of outrages in Europe and other parts of the world, U.S. and European security and intelligence officials say a key problem has been making connections from a mass of data.
"Whenever something goes bad, one of the first things you do is check all the data bases," said retired Gen. Michael Hayden, a former director of both the CIA and the U.S. National Security Agency. "Invariably, you do have something. It's inevitable."
Bruce Riedel, a former top CIA analyst, said: "The problem for the French intelligence and security services is that there are so many French citizens who have gone to Syria or Iraq or elsewhere to join the jihad and then come home that they can not monitor all of them 24 hours a day."
"If they have not broken any laws, intelligence services in the democratic world cannot arrest you or follow you constantly just because you are a fanatic jihadist," Riedel said.
"Intelligence is not going to predict when a fanatic goes from being a radical thinker to a violent terrorist in most cases," Riedel said.
French and U.S. spy agencies several years ago classified Said and Cherif Kouachi, the brothers believed to have attacked the satirical weekly Charlie Hebdo in Paris, as very high priority terrorism suspects, European and U.S. officials said.
Their names were entered into TIDE, a classified database of 1.2 million individuals the United States considers terrorism suspects, and the smaller "no fly" list barring them from boarding flights to or in America, two U.S. officials said.
They had been designated high-priority targets for surveillance after Cherif was implicated in a group recruiting French fighters for an Al Qaeda affiliate in Iraq, and Said went to train with al Qaeda in Yemen in 2011, officials said.
But the U.S. and European officials, who spoke on condition of anonymity, said French authorities scaled back monitoring of the Kouachis when they kept a low profile during the last few years.
LONG TERM PLANS?
The officials said that after Said returned from Yemen, the Kouachis appear to have consciously avoided contact with others they knew to be under surveillance. The officials said this suggested in hindsight that they might have been planning to mount an attack for years.
The U.S. and European security officials said law enforcement and spy agencies have to prioritize which suspects to monitor closely because of the huge number of potential suspects, and such surveillance requires considerable manpower.
Agencies have to assign as many as 30 personnel each day to watch a single suspect and follow any suspicious contacts he or she might meet.
The challenge has been complicated by thousands of foreigners who have gone to join Syria-based militant Islamic groups such as Islamic State and Nusrah, many of whom are now returning to their homeland with battlefield experience.
Investigations following actual or attempted attacks by militants regularly show that spy agencies had advance information that would have indicated the suspects posed an imminent threat if the bits of data had been properly connected.
After the Sept. 11 attacks, investigations established that the CIA and FBI both had early clues to the identities of some of the men who hijacked the planes and flew them into the World Trade center and the Pentagon.
But the information, including that they had militant connections, was not properly shared.
Parliamentary investigations showed that British agencies had collected intelligence on two of the four men who bombed London's underground system in July 2005 in an earlier counter-terrorism investigation. But before the bombings, spy agencies never deemed the suspects high-priority targets for monitoring.
There were similar circumstances surrounding two other men who plotted against the United States - an Army psychiatrist who killed 13 in a mass shootings at Ft. Hood army base in 2009 and a Nigerian who narrowly failed to bring down an airliner headed for Detroit with a bomb in his underwear.
U.S. agencies already had some intelligence tying each man to the same al Qaeda affiliate in Yemen that investigators have linked to Said Kouachi. But the clues were not assembled into a pattern until after the attacks were launched.
