What are they so anxious to hide?

Hiding the truth

It was perhaps inevitable that after such a transparent and successful process that we should compare this experience with our FOI requests to the British authorities. The most obvious comparison was that the FBI found its copy of the Burgess recording, but the British authorities told us they couldn't find theirs. We couldn't make a wider comparison because we had only made the one request to the FBI, but many more to the British authorities. That was because, apart from our search for the voice of Burgess, we had a question about him that only the Foreign Office could answer. How did the Foreign Office come to recruit a Russian spy whose many imperfections were so well known to those who had worked with him, including senior figures at the Foreign Office?

Initially we were met by what we wrote off as Foreign Office bureaucratic inefficiency. It supplied multiple copies of the same pages, but with different sections blacked out. One document was supplied to us with a certain organisation's name blacked out, but the same document had appeared in full, including the name, in The Times back in 1956.

Then it got a bit more serious; we discovered that box after box of surviving Foreign Office material on Burgess and Maclean had still not been released 30 years after the 30-year rule would have allowed it. The bigger picture became clear when Ian Cobain of The Guardian revealed that The National Archives building in Kew wasn't Britain's only archive centre. There was another one, a secret archive of Foreign Office files, behind barbed wire at Hanslope Park in Buckinghamshire, where no fewer than 1.2 million files in so-called “special collections” are kept.

This discovery came about after years of challenges by researchers looking for files on the torture of Mau Mau suspects in colonial Kenya. They were repeatedly told all relevant documents were at the Kew archives. Once the existence of these so-called “migrated archives” was revealed, a former British diplomat, Anthony Cary, was asked to investigate how this deception had come about. His report brought about government promises of reform. Cary discovered a fascinating set of criteria which the Foreign Office drew up in 1961 to judge which colonial documents to destroy or hide back in the UK.

These were papers that

might embarrass Her Majesty's Government or other governments

These criteria raise a host of issues including the concept that democratically elected ministers in a successor government should not be able to see files because they might use them “unethically”.

But the bottom line is that many working journalists will have recognised these criteria, particularly potential embarrassment, as what they have always suspected were the real reasons that they were refused access to documents on a range of subjects in the past rather than the stated reasons they were given at the time, such as overriding national security factors.

After 45 years in journalism this helped influence me towards the conclusion that the British state cannot be trusted to tell the truth. Reactions to that conclusion may vary from “that seems a bit harsh” to “has it taken you 45 years to work that out?”

Challenging the D-Notice

I suppose I had been a sceptic since my time back in the early 1990s representing ITN on the so-called D-Notice Committee, which had representatives from the Ministry of Defence and other government departments, and from the news media. It seemed to me that the first secretary of the committee that I encountered – the secretary is always a former military officer paid by the MoD but allegedly independent – was keen to be seen by his paymasters as suppressing attempts by both former civil servants and journalists to throw even the slightest light on how Britain's secret services and special forces worked whether national security was threatened or not. But the other media representatives on the committee at the time – apart from the excellent Richard Ayre from BBC News – seemed rather seduced by the access to the inner portals of Whitehall and the free sherry served from cut-glass decanters before the meetings.

I began challenging how the system handled specific incidents when the “advice” to the media was “don't publish”. It seemed to me that with the end of the Cold War the authorities were searching for a new rationale for the system. I can even remember the need for police inquiries into organised crime being cited as a new one.

The official history of the D-Notice system, Secrecy and the Media by a later secretary, Nicholas Wilkinson, published in 2009, recorded my attempts to get it changed. The idea of a review had been floated. The book revealed that when the idea of a review was floated there was resistance by the then chairman of the Joint Intelligence Committee (and later BBC governor) Dame Pauline Neville-Jones, who felt “the D-Notice system serves us well”. It also quoted one more sympathetic internal MoD analysis that “the Purvis view was more genuinely representative of the media at the operational level” and that “a new breed moving in at the top of the ranks might not in future be so co-operative”. The response internally was that this new breed would in “due course become editors themselves, when they would realise their responsibilities and might welcome advice”. Nicholas Wilkinson also revealed that the prime minister of the time, John Major, had asked to see my correspondence with the MoD, something I wasn't aware of.

Eventually a review was held and there were some partial reforms – the D-Notice system became the DA-Notice system (Defence Advisory) to emphasise that it was a voluntary, not compulsory, code. Eventually and perhaps most significantly, the creation of a website, www.dnotice.org.uk, meant that working journalists could see what these previously mysterious D-Notices did and didn't say without having to search for the only in-house printed copy somewhere in their managing editor's files.

For nearly two decades the system ticked over with relatively few controversial incidents – until last year.

The story of Edward Snowden, The Guardian and the DA-Notice system is one of the more overlooked elements of the NSA saga. Instead, the focus has been on the rather bizarre tale of how GCHQ technicians stood in a basement at The Guardian checking that staff destroyed computer hard drives containing copies of Snowden's documents even though everybody knew there were other copies elsewhere. But the minutes of a DA-Notice Committee meeting posted on its website and an interview the editor of The Guardian, Alan Rusbridger, later gave at a conference at the London School of Economics have shed some light on what was going on behind the scenes.

It is now clear that in June last year The Guardian deliberately chose not do what the system would like it to do in such circumstances – contact the committee secretary for “advice” before it published the first Snowden revelation about the UK. This was a story headlined “GCHQ intercepted foreign politicians’ communications at G20 summits”. According to the committee minutes, the secretary, Air Vice-Marshall Andrew Vallance, reported that “this failure to seek advice was a key source of concern and considerable efforts had been made to address it”.

Rusbridger was anxious

Rusbridger's explanation is: “I was anxious that by ringing the DA-Notice Committee it would trigger awareness in advance that we were about to publish something.” He believed that if the government had known about the story in advance it could have sought an injunction, something the committee disputes.

But perhaps more significantly Vallance told his committee that apart from that first story “The Guardian had been engaged”. The minutes say that “towards the end of July The Guardian had begun to seek and accept DA advice not to publish certain highly sensitive details, and since then the dialogue with the secretariat had been reasonable and improving”.

We can conclude that The Guardian observed DA-Notice 5 (a) that stories about “specific covert operations, sources and methods of the Security Service, SIS and GCHQ, Defence Intelligence Units, Special Forces and those involved with them, the application of those methods, including the interception of communications, and their targets” should “not be published without first seeking advice”. Rusbridger has confirmed that he consulted the committee on subsequent Snowden stories and has even said that “we've found them to be helpful”.

Observers might take all this as good news for the system. But during the episode it became clear the current PM didn't know about the reforms two decades ago. David Cameron, angered by The Guardian's revelations, called for “social responsibility” from the press because “I don't want to have to use injunctions or D-Notices or other tougher measures”. Simon Bucks of Sky News, who leads for the media side on the committee, responded that the Snowden episode “highlighted once again a lack of complete understanding among some editors and others – including the prime minister – about the role of the DA system”.

Then came news that the MoD was planning to review the system. Bucks reacted: “Any suggestion that the current system be abolished would potentially be a precursor of a coercive system, which I believe the entire British media would oppose.” Little has been heard of this review since and another MoD initiative to examine “an enhanced administrative relationship” between the D-Notice secretariat and the MoD's own press operation has come to nothing. Perhaps, once the PM knew how the system is meant to work, and it seems on this occasion eventually did work, his advisers went off the idea of putting the MoD spin-doctors in charge.

Perhaps the Foreign Office's “migrated archives” will one day find their way to Kew. Hulbert and I will be sitting patiently in the excellent tearoom chatting with the helpful staff on the off chance that a “special collection” about Burgess arrives. We know we got lucky with our one-off pitch outside the system to the FBI so count us as firmly still on the side of the sceptics.