ESS Impact

Bibliographic Monitoring

Each year, an evaluation of the academic impact of the ESS is produced by Core Scientific Team (CST) member, Brina Malnar (University of Ljubljana). In-depth analysis of academic publications listed on Google Scholar and the ESS Bibliography informs this annual report. In order to evaluate the academic impact of our data, this monitoring provides:

• Longitudinal empirical evidence on the scope, geographical and disciplinary patterns of ESS academic usage, theory development and policy references;
• Empirical support for informed decision-making of the ESS bodies (CST, SAB and QDTs) concerning the questionnaire content (item and modules selection and revision), targeting training and communication strategies and similar;
• Summary bibliographic reports, a full list of citations with a possibility to produce tailor-made sub-lists according to various criteria (Appendix 1) and item usage statistics (Appendix 2). These documents help demonstrate ESS academic relevance to European and national funders and users (NCs, GA);
• Bibliographic repository for other work packages to be used for methodological testing, updating of ESS online bibliography, as well as to support ESS communication actions and produce relevant outreach materials;
• The source of annual refreshments to the new ESS online bibliography which renders bibliographic variables available to the general audience and enables users to browse ESS publications according to a variety of criteria. For data users, it is essential to know which other analyses have been already performed with the data set and bibliographic information is an important source of contextual metadata about a study (Fear 2013; Kern 2015).

The latest research - published in May 2023 - found that 5,966 English-language academic publications include substantial primary analysis of our data (2003-21). This includes 3,279 articles published in peer-reviewed journals and 839 books/chapters. The report also includes analysis of the disciplinary profile and country affiliation of authors, research topics and theoretical approaches, use of questionnaire items, and policy relevance of articles.

Appendix 1 to the main report - ESS International Bibliography 2003-21 - includes a list of all 5,253 substantive and 713 methodological publications, based on Google Scholar indexing. Year of publication, author, title, publication and DOI information is also included (APA citation).

Appendix 2 - ESS Item usage report 2003-21 - details the amount of times data collected from each survey question has been analysed for published articles.

Analysis of our Online Bibliography was undertaken by Professor Birger Larsen (Aalborg University), leading to the publication of a report in August 2018: Assessment of the registration of bibliographical data in the ESS Bibliography database.

Impact Study: 2022

In February 2022, ESS ERIC published a new report: SUSTAIN-2: Impact study of the European Social Survey. The report was authored by Technopolis Group (UK).

Funded by the ESS-SUSTAIN-2 project (Horizon 2020, grant agreement number 676166), the study identifies academic, non-academic and teaching impact of ESS data.

This study was commissioned by the ESS ERIC and conducted by Technopolis, with bibliometric analysis from the Centre for Science and Technology Studies (CWTS) at the University of Leiden.

The report is a follow-up to the original impact study of the ESS, which was published in 2017.

Registered users

To access our data, users have to complete a short registration form.

Analysis of data from this registration form by Technopolis Group (UK) found that, as of June 2021, there had been 182,778 registrations- almost double the figure at the start of the original Impact Study (June 2016).

The number of registered users has grown consistently by 14-15% in each of the last five years and, in the year up to June 2021, 74% registered users had downloaded our data.

Of the 50,000 non-student registered users who had accessed the data up until 2020, 7,712 had logged in and downloaded data within the calendar year.

Teaching impact

The report found that our data is increasingly used as a teaching tool, across participating countries and beyond and in an increasing number of institutions for both methodological and thematic courses.

Technopolis Group (UK) report that the actual number of students using our data is likely higher than our data suggests, as many lecturers download data for their students.

In 2016, there were 18 institutions with more than 500 confirmed users (including four with over 1,000). By the start of 2021, this increased to 42 (including 15 institutions with over 1,000 confirmed users).

Academic impact

Technopolis Group (UK) report that the overall number of academic publications including significant analysis of our data has increased by at least 150% since the first Impact Study.

Including various different publication types and non-English language publications, University of Ljubljana data suggests that there are over 7,500 publications in existence (the first Impact Study reported 2,704).

Centre for Science and Technology Studies (CWTS) was able to identify 2,448 academic publications listed on Web of Science, which compares to around 1,000 in the previous study.

The citation impact of these publications is well above average, being about 70% more highly cited than average, with 21% of all ESS publications belonging to the top 10%. The journals in which work is being published have a citation impact of 40% above the world average.

Our data is mostly used by ESS member countries in terms of publication, producing almost three quarters these.

Non-academic impact

Technopolis Group (UK) found many examples of non-academic impact, of many different types and across different domains, including:

• General intelligence and insight for NGOs or government ministries, agencies or advisory bodies
• Agenda setting by using our data to highlight a particular problem or challenge, triggering various types of policy action
• Influence on public debate or highlighting certain issues to the general public through presentation of our data or findings in the news media
• Monitoring: using our data as indicators to track certain aspects of societal progress, to help assess whether certain policies are achieving the desired outcomes

Impact Study: 2017

Funded by the ESS-SUSTAIN project (Horizon 2020, Grant Agreement 676166), Technopolis Group (UK) were tasked with undertaking an ESS ERIC Impact Study in June 2016.

Published in September 2017, the report assessed the academic, non-academic and teaching impact that has been achieved through the ESS. It also considered how these were achieved, what constitutes best practice and made recommendations to ensure the project’s long-term future.

In undertaking the study, Technopolis examined academic resources, sought out examples of non-academic use of our data, interviewed stakeholders and carried out an online survey of data users.

The report found that the ESS continues to be highly regarded amongst international values and attitudes surveys, offering extremely high quality data that is a critical academic resource for researchers across the social sciences.

It also found that ESS data is notable as a teaching resource, particularly within materials developed for use by students who are learning about survey methodology and quantitative data analysis.

The final report included bibliometric analysis undertaken by the Centre for Science and Technology Studies (CWTS), which found that the ESS online Bibliography underestimates the actual number of journal articles using our data by around 20%.

CWTS also discovered that most academic output using ESS data tends to perform highly on bibliometric indicators across a broad range of topics.

The impact study also found evidence of non-academic impact. Across member countries, many cases where ESS data was used in many different policy and practice contexts was found. This has helped to influence policy decisions and inform public and political debates.

The Technopolis report found that in some current member/observer countries, there is widespread use of the ESS as a teaching resource and in others the data is widely used in parliaments, ministries or government agencies.

Examples of where ESS data has been used by Governments or their agencies include the Austrian Ministry for Social Affairs, Estonia’s Ministry of Social Affairs, the French governmental think tank, France Stratégie, the German Federal Government, the Irish police force, Poland’s Central Statistical Office, the Portuguese Centre for Judicial Studies and the Institute of Macroeconomic Analysis and Development of the Republic of Slovenia.

ESS data has also helped inform international organisations such as Eurofound, United Nations Economic Commission for Europe (UNECE), European Commission’s DG EMPL (Employment, Social Affairs and Inclusion), OECD as well as think tanks including the Centre for European Policy Studies and European Policy Centre.

In making recommendations, the Technopolis report stressed the importance of continued funding and consistent involvement of countries over time. The addition of other European countries not already involved in the ESS would further help to ensure the long-term sustainability of the project.

A supplementary report was published in December 2018.