It’s January 1, the day each year when our minds turn to newly released Cabinet records from @naagovau. But while the media focuses on the records that have been made open, I’ll be spending the day looking at those that were closed. What weren’t you allowed to see in 2020?

This will be a *slow* thread, as I gradually pull the data together and document things. But this year I’ll be sharing all the data and code through the #GLAMWorkbench, so stay tuned...

This’ll be the sixth consecutive year in which I’ve harvested all NAA files with an access status of ‘closed’ on or about 1 January. For some background and past analyses, see my @insidestorymag article from 2018: https://insidestory.org.au/withheld-pending-advice/

The code that I’m using to harvest all the ‘closed’ files is here: https://glam-workbench.github.io/recordsearch/#harvest-files-with-the-access-status-of-closed It scrapes the data from RecordSearch, the NAA’s online database. It’s one of a number of handy RecordSearch tools in the #GLAMWorkbench.

Ok, so the harvest is done and, as of today, there are 11,140 files in the NAA’s RecordSearch database with an access status of ‘closed’. This is down from 11,867 a year ago.

A quick recap for new players…

All Commonwealth government records are supposed to be opened to the public after 20 years (30 years for Cabinet documents). However, there are exemptions laid out in Section 33 of the Archives Act. http://www8.austlii.edu.au/cgi-bin/viewdoc/au/legis/cth/consol_act/aa198398/s33.html

Before being released to the public, records go through a process known as ‘access examination’ to check them against the exemption categories (these are things like national security and privacy). There’s more info about the process here: https://www.naa.gov.au/help-your-research/using-collection/access-records-under-archives-act

After examination, records are assigned an an access status, either:

* Open - Yay! Most records are available for public access.
* Open with exception – Some have sensitive material removed, but are otherwise open.
* Closed – A small number are completely closed to the public.

It’s these ‘Closed' files that I’m looking at – files that have been through the examination process and have been withheld from the public.

(Actually, as we’ll see, some of the closed files are actually part way through the examination process…)

The reasons why files have been closed are recorded in RecordSearch. Here’s the number of closed files in today’s dataset citing each reason.

Aside – multiple reasons can be cited by an individual file, so the number of cited reasons will be greater than the number of closed files.

The reasons starting with ’33’ refer to specific parts of Section 33 of the Archives Act. They’re also listed on the NAA website: https://www.naa.gov.au/help-your-research/using-collection/access-records-under-archives-act/why-we-refuse-access

Of the most common – 33(1)(g) relates to individual privacy, while 33(1)(a) is the national security catch-all.

‘Withheld pending adv’ is not really a reason, more of a work-in-progress marker. During access examination, some files are referred back to the relevant Govt agency for advice, and are marked as ‘closed’ while the NAA waits for a response. More about that later...

Now let’s have a look at how the ‘closed’ files are distributed across series. There are 686 individual series represented in the dataset. (From memory there’s about 60,000 series in RecordSearch.)

What series have the most ‘closed’ files?

* K60 are repatriation files from DVA
* A1838 is a general correspondence series from DFAT

Can you guess why?

To find out, let’s add the reasons into that last chart.

From the above chart you can see that most of the K60 (DVA repatriation) files are closed under section 33(1)(g) of the Archives Act, which relates to privacy. Fair enough.

On the other hand, most of the files from A1838 (DFAT) are ‘Withheld pending adv’. The delays and backlogs in getting access to files from A1838 are well known. I wrote them in my Inside Story article: https://insidestory.org.au/withheld-pending-advice/

You might also notice A6122. That’s a series of ASIO surveillance files. As you can see most are closed on grounds of national security (33(1)(a)). And yeah, there’s some irony in the fact that public access to some ASIO surveillance files is closed due to privacy concerns…

While we’re at it, here’s the series that most frequently cite the national security exemption (33(1)(a)).

* A6122 – ASIO surveillance files
* AWM54 – WW2 records from Defence
* A1838 – DFAT correspondence files
* A1209 – PMs Dept correspondence

The next series on that list citing 33(1)(a) is interesting. The whole of C5326 is closed (42 files). Why? It's all about how to blow up bridges. http://recordsearch.naa.gov.au/scripts/AutoSearch.asp?Number=C5326

So that’s the overview… I’m going to have a bit of a break, clean up some code, and get stuff online.

Later today (or tomorrow) I’ll look at changes across all 6 annual harvests. I’ll also compare 2020 to 2019 to see what changed in the last 12 months.

Here’s a taste…

Just inserting a reminder here that if you ever want to make persistent links to things in RecordSearch (avoiding nasty session timeout errors), you can use this simple tool: https://recordsearch-links.glitch.me/ 

One more thing I intended to add to the overview relates to the ages of these files. There's no automatic re-assessment or time limit on 'closed' files. They stay closed until someone asks for them to be re-examined. So some are very old. Here’s the ages of files charted.

The age here is calculated by subtracting the contents end date from today’s date.

If you look at the proportions, 25% of the closed files are more than 63 years old.

If we look just at the national security exemptions (33(1)(a)), we see that the mean age of closed files is 61 years, and 25% of the files are more than 71 years old. Are 70 year-old files still a risk to our security?

Thanks for all the interest in this! I'm currently examining changes in ‘closed’ files between 2019 and 2020, and I think I’ve finally got the numbers adding up! More details tomorrow...

For example, the 2019 harvest included 519 more files with the reason ‘Closed period’ than the 2020 harvest. 538 ‘closed period’ files dropped out of the harvest in 2020, and 19 were added – a difference of 519. But where did those 538 files go? Looks like…