Each year, an evaluation of the academic impact of the ESS is produced by Core Scientific Team (CST) member, Brina Malnar (University of Ljubljana). In-depth analysis of academic publications listed on Google Scholar and the ESS Bibliography informs this annual report.
In order to evaluate the academic impact of our data, this monitoring provides:
- Empirical evidence on the scope, geographical and disciplinary patterns of ESS academic usage and policy references;
- Support for informed decision-making by ESS bodies concerning questionnaire content.
- Summary bibliographic reports, statistics and tailor-made lists of citations that help demonstrate ESS academic relevance to European and national funders and users.
- Bibliographic repository for other work packages to be used for methodological testing, updating of ESS online bibliography, as well as to support ESS communication actions and produce relevant outreach materials.
The latest research - published in December 2020 - found that there have been 4,913 academic publications that include substantial use of our data (2003-19). This includes 2,527 peer-reviewed journal articles and 743 books/chapters. The report also includes analysis of the research topics covered, the contribution of ESS data to theory development and the usage of survey questions.
Appendix 1 to the main report - ESS International Bibliography 2003-19 - includes a list of all 4,913 substantive and methodological publications, based on Google Scholar indexing.
Appendix 2 - ESS Item usage report - details the amount of times data collected from each survey question has been analysed for published articles.
Information included in a previous report - published in December 2017 - was used to produce an Infographic highlighting the geographic spread of first authors, the topics most commonly covered and the subject areas of the journals that include the most ESS publications.
Analysis of our Online Bibliography was undertaken by Professor Birger Larsen (Aalborg University), leading to the publication of a report in August 2018: Assessment of the registration of bibliographical data in the ESS Bibliography database.
Funded by the ESS-SUSTAIN project (Horizon 2020, Grant Agreement 676166), Technopolis Group (UK) were tasked with undertaking an ESS ERIC Impact Study in June 2016.
Published in September 2017, the report assessed the academic, non-academic and teaching impact that has been achieved through the ESS. It also considered how these were achieved, what constitutes best practice and made recommendations to ensure the project’s long-term future.
In undertaking the study, Technopolis examined academic resources, sought out examples of non-academic use of our data, interviewed stakeholders and carried out an online survey of data users.
The report found that the ESS continues to be highly regarded amongst international values and attitudes surveys, offering extremely high quality data that is a critical academic resource for researchers across the social sciences.
It also found that ESS data is notable as a teaching resource, particularly within materials developed for use by students who are learning about survey methodology and quantitative data analysis.
The final report included bibliometric analysis undertaken by the Centre for Science and Technology Studies (CWTS), which found that the ESS online Bibliography underestimates the actual number of journal articles using our data by around 20%.
CWTS also discovered that most academic output using ESS data tends to perform highly on bibliometric indicators across a broad range of topics.
The impact study also found evidence of non-academic impact. Across member countries, many cases where ESS data was used in many different policy and practice contexts was found. This has helped to influence policy decisions and inform public and political debates.
The Technopolis report found that in some current member/observer countries, there is widespread use of the ESS as a teaching resource and in others the data is widely used in parliaments, ministries or government agencies.
Examples of where ESS data has been used by Governments or their agencies include the Austrian Ministry for Social Affairs, Estonia’s Ministry of Social Affairs, the French governmental think tank, France Stratégie, the German Federal Government, the Irish police force, Poland’s Central Statistical Office, the Portuguese Centre for Judicial Studies and the Institute of Macroeconomic Analysis and Development of the Republic of Slovenia.
ESS data has also helped inform international organisations such as Eurofound, United Nations Economic Commission for Europe (UNECE), European Commission’s DG EMPL (Employment, Social Affairs and Inclusion), OECD as well as think tanks including the Centre for European Policy Studies and European Policy Centre.
In making recommendations, the Technopolis report stressed the importance of continued funding and consistent involvement of countries over time. The addition of other European countries not already involved in the ESS would further help to ensure the long-term sustainability of the project.
A supplementary report was published in December 2018.
Comparative impact study of the ESS ERIC (September 2017)
Report annex: Impact case studies (September 2017)
Supplementary report to ESS ERIC (December 2018)