European flag
>
>
How we gather evidence

How we gather evidence

On this page

When the Scientific Advice Mechanism receives a request for advice from the European Commission, the first step in our process is usually for SAPEA to assemble an interdisciplinary working group of Europe’s top independent experts.

This is the group that will write the evidence review report — a detailed overview of the current scientific knowledge on the topic — which the Group of Chief Scientific Advisors will use to inform their policy recommendations. The working group works independently of the Advisors and of other outside influences.

This page contains a summary of how we gather evidence. For a full and detailed description of all our processes, you can read our ‘Quality assurance guidelines and Procedures on science advice for policy and society’.

Assembling the working group

Establishing criteria

Members of the working group are chosen for their scientific excellence and the relevance of their expertise for the topic we are working on. These are always the main criteria. They are not paid for their work.

We also ensure that each working group has the full range of interdisciplinary expertise necessary to cover the scope of the topic we have been asked to work on. Depending on the topic, this can include experts from natural sciences, engineering and technology, medical, health, social sciences, agricultural sciences, and the humanities.

As well as this, we have a strategy of diversity and inclusiveness.

Issuing a call for nominations

The main way that we identify suitable experts is to send out a call, via our academy networks, to more than 110 individual academies of sciences across Europe.

We also sometimes identify possible experts through desk research (using bibliographic sources and others), and by approaching selected organisations which have expertise in the topic.

We always want the best independent experts. This is why the responsibility for nominating experts lies with our academies and academy networks. But each expert is nominated purely on the basis of his or her expertise, not academy membership. This means that experts nominated by academies do not have to be Fellows (members) of those academies. Instead, the role of the academy or network is to assess the scientific excellence of the person they are nominating.

If you are a researcher and you are interested in joining the working group for a forthcoming topic, please approach either the academy in your country or the academy network in charge of the topic.

Selecting working group members

Once we have collected all the nominations, we form a selection committee consisting of:

  • members of the SAPEA board
  • the chair of the working group
  • an independent expert on the topic

This selection committee works through the list of nominations, applying the criteria, and proposes a list of experts to form the working group.

The SAPEA board makes the final decision.

Checking for conflicts of interests

We assess whether any of the experts have a conflict of interests which could jeopardise their independence, and take appropriate action where needed.

Experts’ declarations of interests are published on this website at the same time as the evidence review report is published, and stay here for six months.

Reviewing the evidence

Our working group meets several times to draft their evidence review report, usually with the support of a science writer.

Each report contains:

  • an executive summary
  • a description of the background and request for advice
  • several chapters describing and summarising the evidence
  • a chapter on conclusions and evidence-based policy options
  • references
  • a list of the names and affiliations of the working group members and how they were selected
  • a description of the  involvement of the Advisors and the SAM secretariat
  • a description of literature reviews, the expert workshop and the peer review
  • acknowledgements of further people involved in the preparation of the report, such as SAPEA staff

Methodology

The working group members work to achieve a consensus. They are guided by the scoping paper. They write the report based on their discussion and the results of literature reviews.

In general, they describe, summarise, evaluate and document the evidence in a systematic way. Where judgements about scientific evidence are being made, the experts indicate the uncertainties associated with findings.

Depending on the topic and the questions posed, both the methods and the scale will vary. But whatever methods are used, these are described in the evidence review report.

When the report reaches its final draft stage, it is the responsibility of the chair(s) of the working group to ensure that sources are cited and referenced accurately.

Gathering information

Information (scientific knowledge, evidence, context, and so on) is gathered using processes such as:

  • reviews of the scientific literature
  • systematic mapping of the policy ecosystem
  • expert hearings, workshops or other forms of expert elicitation
  • fact-finding and fact-checking
  • other forms of expert elicitation, such as structured interviews, desk research, interviews with stakeholders, and the involvement of further contributors

Protecting from outside influence

The working group holds its meetings behind closed doors. No interim drafts are published, and the names of the working group members are also not published.

The Advisors and staff from the SAM secretariat may attend working group meetings as observers, by agreement of the working group members themselves.

SAPEA staff ensures that the proceedings of the working group are well documented so that there is a clear record of how the group reached its decisions.

Expert workshop

The first complete draft of the evidence review report, or its key findings, are discussed in an expert workshop.

The experts invited to this workshop have a scientific background, with knowledge of the applied or policy context in the field. As well as gender and geographical balance, we aim for an institutional balance (for instance, including non-governmental organisations, business and industry, trade unions, civil society organisations and so on).

The aims of the expert workshop are:

  • to discuss and review the evidence, especially to tackle blind spots or biases of the members of the working group
  • to check if the scope and scale of the evidence and the way it is presented correspond to the current state of discussions among stakeholders and the public
  • to ensure that the policy options proposed are practical within the relevant timescales
  • to perform a critical appraisal (benchmarking) of the evidence

Double-blind peer review

The evidence review report is subject to peer review, to cover:

  • scientific and technical quality
  • completeness, including the full range of information and opinions
  • impartiality and objectivity
  • whether the report addresses the questions of the scoping paper (where appropriate)

We use a double-blind peer review process, which means that the identities of the reviewers and the working group members are concealed from each other throughout the review process. The names and affiliations of the reviewers are included in the final report.

After peer review, the working group revises the report (if needed) to take into account the feedback received.

The working group responds to all peer review comments, but need not agree with them, and records how the comments are addressed. The final report includes an annex containing this information.

Plagiarism check

We use plagiarism-check software to check that the text is correctly referenced.

Endorsement

The SAPEA board receives the final draft of the evidence review report. Each member of the Board endorses the report on behalf of their Academy Network.

In the exceptional case where a Network does not endorse the final report, the reasons for this decision are included in the final report.

Other evidence-gathering processes

By SAPEA

Occasionally, SAPEA may use different processes to gather evidence, instead of following the steps described above. For instance, we may convene expert workshops, or provide a list of experts for the Advisors to consult directly.

These alternative processes are used, for instance, when the deadline to deliver advice is very urgent.

When we use an alternative process, it may not follow all of the steps described above. Instead, each process has its own  methodology and quality assurance processes, which are decided by the SAPEA board in advance.

By the SAM secretariat and the Advisors

In addition to SAPEA’s work, the Advisors may also ask the SAM secretariat to gather any other sources of evidence that they judge appropriate, such as:

  • data and reports provided by the European Commission’s Joint Research Centre
  • contributions from other directorates-general of the European Commission, and EU agencies
  • evidence from EU-funded research projects, such as those funded under Horizon Europe
  • consultations with other experts, for instance by organising expert hearings or workshops
  • discussions with science advisors and other evidence providers in EU member states and beyond

The Advisors may also choose to organise visits (for instance, to laboratories) or to meet relevant stakeholders.

Uncle SAM

Staff login