In the biggest official files leak in history nearly 400,000 Iraq war logs reveal the massive scale of civilian deaths and new torture allegations following an investigation by Channel 4's Dispatches.

Channel 4 News has accessed the data in the classified documents via The Bureau of Investigative Journalism and WikiLeaks but has been unable to independently verify their authenticity.

Warning: You may find some of the details in this report and the accompanying video disturbing.

Please wait while this video loads. If it doesn't load after a few seconds you may need to have Adobe Flash installed.

In total 391,832 individual logs - written by American troops in combat - tell the story of the Iraq war during the period 2004 to 2009.

The documents were leaked by whistleblowers' website WikiLeaks and obtained by Channel 4 News via The Bureau of Investigative Journalism (TBIJ) ahead of an exclusive report in Iraq's Secret War Files on Channel 4's Dispatches on Monday at 8pm.

They catalogue every aspect of the conflict; from checkpoint "escalations of force" and apparent surrender killings to the rising profile of al-Qaeda in Iraq.

The files contain precise locations and the identities of civilians, suspected insurgents and the troops involved. Their names have been redacted for their safety.

More: iraqwarlogs.com

Statement from US Pentagon:
We deplore WikiLeaks for inducing individuals to break the law, leak classified documents and then cavalierly share that secret information with the world, including our enemies.

We know terrorist organizations have been mining the leaked Afghan documents for information to use against us and this Iraq leak is more than four times as large. By disclosing such sensitive information, WikiLeaks continues to put at risk the lives of our troops, their coalition partners and those Iraqis and Afghans working with us.

The only responsible course of action for Wikileaks at this point is to return the stolen material and expunge it from their websites as soon as possible.

Civilian cost of war

Retired general Tommy Franks, a former commander of US Central Command who led the invasions of both Afghanistan and Iraq, said in 2002: "We don't do body counts".

In fact his troops on the ground in both conflicts were required to record deaths and the leaked Iraq war files reveal that the civilian toll could be 15,000 higher for the six year period than previous estimates suggest.

Secret war files: Afghanistan to Iraq - special report

In all 109,032 deaths are logged in the leaked Iraq "Sigacts" (significant activity reports), via the military acronym "KIA" for "killed in action". Of these deaths, the logs show 66,089 were civilians - just under two-thirds - in the six year period.

Secret war files: Iraq

The logs show that US troops were killing significantly more civilians than insurgents at checkpoints. Under the military's special rules of engagement, known as "escalation of force" (EOF), a vehicle approaching a military checkpoint is required to slow down, stop and be searched.

A cold analytical eye on this clearly, I think, indicates that far too many Iraqi civilians are being killed for no reason. Toby Dodge

In EOF incidents between 2004 and 2009 the data shows that four times as many civilians (681) were killed at checkpoints as insurgents. In one log we can see that a doctor driving a pregnant woman to hospital, at speed, was shot. Analysis by TBIJ shows 13 coalition troops were killed in these incidents.

Toby Dodge, a political scientist and former adviser to General Petraeus and Tony Blair, told Channel 4's Dispatches: "A cold analytical eye on this clearly, I think, indicates that far too many Iraqi civilians are being killed for no reason."

Read more - Afghanistan: secret war files raise questions over civilian deaths

Air attacks, Hellfire missiles and 'surrender killings'

The logs reveal the changing nature of the Iraq war following the appointment of General David Petraeus (now commander of the International Security Assistance Force in Afghanistan) as commanding general of the multi-national force in Iraq in January 2007. Four years after the invasion, with the number of civilian deaths spiralling and al-Qaeda activity on the rise, Iraq was close to a state of anarchy with an escalating daily tally of truck bombings and suicide blasts.

General Petraeus launched a new strategy and called it the "surge". Its aim was to target the insurgents behind these attacks.

In his opening statement before a Senate hearing on January 23 2007, Petraeus said: "The security of the population, especially in Baghdad, and in partnership with the Iraqi Security Forces, will be the focus of the military effort."

But the key aspect of this new phase was a massive increase in the use of air power, in particular Hellfire missile attacks. Analysis of the logs by TBIJ shows 80 per cent of Hellfire missiles were fired after the start of the Petraeus surge.

Iraq secret files: the war in pictures

Chart shows rate of civilian deaths and the rise in Hellfire attacks by US forces. (Ciaran Hughes)

Michael Clarke, from defence analysts Rusi, told Dispatches: "The use of air power in counter-insurgency is as a monitor.

"You use it to see what's going on and to listen to, collect intelligence. But using it either for bombing or attacking, or strafing a building on the ground, or somewhere where you think insurgents are hiding, is very dangerous indeed."

In one case, the files show, a group of children herding goats and collecting tree roots for firewood were mistaken for IED planters and fired on with a Hellfire missile from a US Apache helicopter. A 13-year-old boy was killed in front of his friends.

The documents also show American forces killed people who appeared to be trying to surrender. One log describes suspected insurgents "wanting to surrender."

A helicopter crew seeks advice via radio before being cleared to shoot by a military lawyer - who says the men cannot surrender to an aircraft. Moments later they are killed with a Hellfire missile. However, elsewhere in the files there are cases where enemy fighters were allowed to give themselves up to aircraft.

The detailed military records confirm a huge overall increase in Hellfire launches - a leap from 91 in 2006 to 911 in 2007. But they record just 103 civilian deaths due to air attacks in the whole six year period.

Iraq files: Apache helicopter launches Hellfire missile. (Getty)
History of the Hellfire
The Hellfire missile can be launched from air, sea, and ground platforms.

It is the primary 100lb class air-to-ground precision weapon for the armed forces of the United States and was designed primarily to defeat tanks. It can lock onto targets before or after launch. (Source: Lockheed Martin)

It was first used by US troops during Operation Just Cause in Panama in 1989 but came into full force during the Gulf War of 1991. Each Hellfire missile costs $60,000-70,000.

In Iraq they were primarily launched from Apache helicopters. Fixed-wing platforms for the weapon include the unmanned Predator and Reaper drones, deployed in Afghanistan.
Abu Ghraib: videograb from Al-Arabiya television channel rebroadcasting from US news program 60 minutes showing a US soldier pointing at nude hooded Iraqi prisoners. (Getty)

Abuse and torture

Even after the shock abuse scandal of Abu Ghraib in 2004, where photos of US troops physically and sexually humiliating Iraqi prisoners came to light, it appears there was no major change in US military guidance on the treatment of detainees.

There are 300 files which contain allegations of abuse by coalition forces.

One of the files records claims by a prisoner that a US marine "kicked him hard in the stomach" before forcing him to dig with his hands for a suspected IED (improvised explosive device). In another log a detainee alleged that American forces punched him, threw urine on him and applied electric shocks to his body.

Iraqi on Iraqi torture is also documented in the logs but apparently "overlooked" by forces deployed to bring about democracy in a country terrorised by Saddam Hussein.

The analysis found in excess of 1300 cases of alleged abuse by Iraqi forces at police stations and army bases - which coalition forces witnessed. The TBIJ also identified significant military orders, issued after Abu Ghraib, instructing US troops on how to deal with abuse.

These guidelines appear to recommend that unless coalition troops are involved in detainee abuse no further investigation is necessary "unless directed by higher headquarters".

The rise of al-Qaeda

The files appear to expose startling figures about al-Qaeda, the terrorist network behind the 9/11 attacks on the US which underpinned both invasions of Afghanistan and Iraq.

According to analysis by TBIJ and Dispatches, in the earliest logs there is very little evidence of the terror group's activities. In fact in the whole of 2004 there are just seven brief mentions of "AQI" (Al-Qaeda in Iraq).

Yet four years later in 2008, after Saddam Hussein's demise and the Petraeus "surge", the logs reference al-Qaeda on 8,208 occasions.

In 2004 the number of killings attributed to AQI is zero. In 2008, the logs show they were linked to the deaths of 45 coalition soldiers.

The reports also contain details which show the Americans believed al-Qaeda cells were recruiting children and the mentally disabled as suicide bombers. And one entry from June 2006 states that al-Qaeda insurgents "plan to deploy a chemical weapon using mortars as a delivery system".

After the "surge" the Sigact data shows that US troops recorded 300 suspected AQI members killed in 2007, a further 800 in 2008.

Dispatches: Iraq's Secret War Files, Monday 8pm, Channel 4 News.

Pentagon crest. (Getty)
Full Pentagon statement:
We deplore WikiLeaks for inducing individuals to break the law, leak classified documents and then cavalierly share that secret information with the world, including our enemies.

We know terrorist organizations have been mining the leaked Afghan documents for information to use against us and this Iraq leak is more than four times as large. By disclosing such sensitive information, WikiLeaks continues to put at risk the lives of our troops, their coalition partners and those Iraqis and Afghans working with us.

The only responsible course of action for Wikileaks at this point is to return the stolen material and expunge it from their websites as soon as possible.We strongly condemn the unauthorized disclosure of classified information and will not comment on these leaked documents other than to note that "significant activities" reports are initial, raw observations by tactical units.

They are essentially snapshots of events, both tragic and mundane, and do not tell the whole story. That said, the period covered by these reports has been well-chronicled in news stories, books and films and the release of these field reports does not bring new understanding to Iraq's past.

However, it does expose secret information that could make our troops even more vulnerable to attack in the future. Just as with the leaked Afghan documents, we know our enemies will mine this information looking for insights into how we operate, cultivate sources, and react in combat situations, even the capability of our equipment.

This security breach could very well get our troops and those they are fighting with killed.

Please wait while this video loads. If it doesn't load after a few seconds you may need to have Adobe Flash installed.

Files dwarf Afghan WikiLeaks

One of the staggering aspects of this secret files leak is its sheer scale. The documents contain 38 million words across 391,832 separate entries - all written by US combat troops in Iraq between 2004 and 2009.

The scope of the files is so vast, The Bureau of Investigative Journalism and Channel 4's Dispatches created a purpose-built database in order to search and correlate the military codes, operational terms and abbreviations.

The previous leak on this scale came earlier in 2010 with the WikiLeaks files on the Afghanistan war, also covering a six year period. That breach was said to be the biggest of its kind since the Pentagon Papers on US-Vietnam relations in 1971.

These Iraq files are more than four times the size of the Afghan leaks and at 66,089 compared with 4,232, contain 16 times as many dead civilians - each one logged and filed by a member of the United States military.

WikiLeaks war files: Afghanistan's hidden war

More on this story