Data Processing and Archiving

Principles for data processing

The ESS Data Archive's overall guiding principle is to produce harmonised and standardised data files that balance two aims:

An important principle is that data processing should be done in close collaboration with the National Coordinators and their teams. During the data processing stage each National Team has full access to the catalogue (“virtual workspace” containing all data and all programmes) where the processing of their data is actually taking place. All decisions about the editing of data are based on advice from the National Coordinators, who have first-hand knowledge of their national data, and the National Coordinators also have to approve the final drafts of the data files.

The ESS Data Protocol is the key specification document available for the National Teams. The Data Protocol helps to achieve cross-national uniformity in data delivery, as it gives specifications for coding of data, the production of and the delivery of data files and other electronic deliverables. It contains sections on the procedures for collaboration between the National Teams and the ESS Data Archive, electronic deliverables, principles of variable definitions, standards and classifications, etc. In addition to this, its largest part is a detailed coding plan which defines the variable names, answer categories, whether numeric or alphanumeric codes are used, and detailed routing instructions consistent with the instructions given in the source questionnaire. The routing instructions are supplemented by flowcharts.

Stages in data processing

In total, 16 data programmes are applied from the stage when the files are deposited to the ESS Archive Intranet by the National Coordinators until the final draft files are ready for control and approval. Some of the programmes do automatic checks of the data files, while others produce output to be controlled manually. All programmes, files and outputs are available for the National Coordinators, securing full transparency during the processing.

Each programme represents a distinct step in the processing; however, we can identify two main stages, each of them completed by a data processing report to the National Coordinators and their feedback. The first stage consists of what we describe as data ingest and data checks. The second stage covers data edits, edit controls and the final data approval. For a more detailed account of the two stages of data processing, we recommend the ESS Data Archive Team's in-depth article available from the sidebar as further reading.