Characteristics of the evaluated literature
The search strategies in the four bibliographic databases identified 5293 records, which were reduced to 3621 unique records after de-duplication. In addition, 547 further records were excluded as their type was beyond the scope of this systematic review, thus leaving 3074 records for the first stage of screening. During screening of titles and abstracts, 87 records met the eligibility criteria and were carried over to the second stage of full paper screening. Twenty-three studies10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32 were ultimately included in the systematic review after the conclusion of the full paper screening. The selection process, including the reasons for exclusion during full paper screening, is illustrated in a PRISMA flowchart in Fig. 1. A list of the excluded studies along with the reason for exclusion can be found in Supplementary Table 1.
*Records of books (n = 268), conference proceedings (n = 179), and theses (n = 100) were removed before screening as they are beyond the focus of this systematic review. The screening process is reported in a PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) flow diagram.
Details of the included studies are presented in Table 1. Out of the 23 studies, 1812,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29 included wind power as an intervention in tackling climate change or reaching the net zero target (decarbonisation), while one focused on hydrogen32 and one on transportation30. Three studies focused on more than one of the themes of this systematic review. One study10 looked at both wind power and transportation, one11 at both wind power and hydrogen energy and one31 at both hydrogen energy and transportation.
Fifteen of the studies focused their analysis on one country and the remaining eight on multiple countries (Table 1). The vast majority of the studies (22 out of 23) used a type of modelling for their analysis. Most studies (14 out of 23) employed an econometric methodology in their analysis; three used traditional modelling approaches, and two used energy system modelling. Economic modelling, dynamic risk transmission modelling, and multiple criteria decision-making modelling were used by each of the remaining studies, while there was also one case study. Further details on the methods applied can be found in Table 1.
Only one study28 collected primary data for their analysis using a bespoke online portal. The rest of the studies used a mix of sources for their modelling data coming from: publicly available sources, such as the World Development Indicators (WDI) from the World Bank databank33 and International Energy Agency (IEA) reports34; national authorities/companies, such as the National Energy Council of Indonesia (DEN) and the Korea Electric Power Corporation (KOSIS); data from peer-reviewed papers; and, data from commercial companies such as the PB energy outlook reports35 (Table 1).
Methodological variation and policy recommendation themes
The focus of the interventions and the measured outcomes varied. Two studies22,28 did not specify the geographical areas that their analysis focused on. Fourteen studies employed econometric-empirical modelling, and ten10,14,16,18,19,20,21,22,23,25 of them used CO2 emissions as their dependent variable, which was the focus of their analysis. Amongst these studies, the independent variables used in the modelling varied substantially (Table 2). Only two independent variables, namely the use of renewable energies and economic growth was used in all studies. Therefore, a direct comparison of the studies’ findings and PRs is not feasible.
Only two studies11,30 defined PRs as primary outcomes. The PRs made by the studies often expanded beyond the scope of this systematic review, only the PRs relevant to climate change mitigation and/or reaching net zero targets were extracted, in line with the eligibility criteria. The full PRs can be found in Supplementary Table 2. In a thematic analysis, four themes of PRs were identified: economic measures, energy measures, public-facing measures and innovation-technology measures. The results of the thematic analysis, along with specific practical examples of PR, are illustrated in Supplementary Figure 1. The majority of the studies presented their PRs in a separate section right after the discussion section, either following or preceding the conclusions section.
Dual appraisal reveals high risk of bias in evidence and communication
Two quality appraisal tools were used in this systematic review, the Collaboration for Environmental Evidence Critical Appraisal Tool (CEECAT)36 and the Evidence Communication Rules for Policy (ECR-P) critical appraisal tool37, each focusing on different aspects of the studies. The results of the CEECAT assessment are illustrated in Fig. 2. Out of the seven domains (criteria) of the tool, only five were used, as criteria three and four are designed for study types that were not identified nor included in our systematic review (observational and experimental). The highest bias was identified in criterion 1 of the tool, concerning risk arising from confounding biases. Fourteen studies12,13,14,16,18,19,20,21,22,23,24,25,28,32 were found to be of overall high RoB, eight10,11,17,26,27,29,30,31 exhibiting medium RoB, and only one study15 was assessed to be of low RoB. The results for each individual study for each criterion are illustrated in Fig. 3.
Collaboration for Environmental Evidence critical appraisal tool (CEECAT) addresses the risk of bias (RoB) in the five different domains illustrated in the five top bars of the figure. An overall RoB rating is also attributed to each paper. Here, the three ratings of high RoB, medium RoB and low RoB are illustrated by percentage for the entirety of the 23 studies that are included in the systematic review.
The Collaboration for Environmental Evidence critical appraisal tool (CEECAT) addresses the risk of bias (RoB) in five different domains; D1: risk of confounding biases, D2: risk of selection biases, D3: risk of detection biases, D4: risk of outcome reporting biases, and D5: risk of outcome assessment biases. (+) indicates low RoB and (−) indicates medium RoB and (x) indicates high RoB, RoB is examined and presented for each individual study: St.1, Calvillo and Turner30; St.2, Cheng and Yao24; St.3, Cheng et al.25; St.4, Gilmore et al.11; St.5, Govindarajan and Ganesh28; St.6, Handayani et al.27; St.7, Horobet et al.22; St.8, Hossain et al.17; St.9, Ifaei et al.29; St.10, Jahanger et al.14; St.11, Logan et al.31; St.12, Obobisa23; St.13, Qadeer et al.32; St.14, Raihan and Tuspekova21; St.15, Raihan and Tuspekova18; St.16, Raihan et al.20; St.17, Raihan et al.19; St.18, Raihan et al.16; St.19, Song and Chen12; St.20, Sun and Dong26; St.21, Sun et al.10; St.22, Zhao, C. et al.15; St.23, Zhao, L. et al.13.
The domain of confounding biases was identified with the highest potential for risk across the studies (11 out of 23). Nine out of these 11 studies used econometric methods as described in the previous section. The omission or addition of variables in the models that seemingly had very similar research objectives raises significant questions on the existence of confounding parameters and their potential effects on the modelling results. The inconsistent configuration of the models (Table 2) and the lack of supporting justification are reflected in the CEECAT assessment results.
Five domains were examined within the ECR-P critical appraisal tool. ECR-P was specifically designed for assessing the communication and the quality of scientific-based PRs as well as their evidence base. This systematic review is the first in which this tool has been used, following extensive piloting and validation. The summary of the assessment is illustrated in Fig. 4, while the assessment for individual studies is presented in Fig. 5.
The Evidence Communication Rules for Policy (ECR-P) critical appraisal tool addresses the risk of bias (RoB) in the five different domains illustrated in the five top bars of the figure. An overall RoB rating is also attributed to each paper. Here, the three ratings of high RoB, some concerns and low RoB are illustrated by percentage for the entirety of the 23 studies that are included in the systematic review.
The Evidence Communication Rules for Policy (ECR-P) critical appraisal tool addresses the risk of bias (RoB) in five different domains; D1: Inform not persuade, D2: Offer balance, not false balance, D3: Disclose uncertainties, D4: State evidence quality, D5: pre-empt misunderstandings. (x) indicates high RoB, (+) indicates low RoB and (−) indicates some concerns. RoB is examined and presented for each individual study: St.1, Calvillo and Turner30; St.2, Cheng and Yao24; St.3, Cheng et al.25; St.4, Gilmore et al.11; St.5, Govindarajan and Ganesh28; St.6, Handayani et al.27; St.7, Horobet et al.22; St.8, Hossain et al.17; St.9, Ifaei et al.29; St.10, Jahanger et al.14; St.11, Logan et al.31; St.12, Obobisa23; St.13, Qadeer et al.32; St.14, Raihan and Tuspekova21; St.15, Raihan and Tuspekova18; St.16, Raihan et al.20; St.17, Raihan et al.19; St.18, Raihan et al.16; St.19, Song and Chen12; St.20, Sun and Dong26; St.21, Sun et al.10; St.22, Zhao, C. et al.15; St.23, Zhao, L. et al.13.
ECR-P assessment is executed for each domain at two different levels: the study level and the PRs level. Therefore, in the first instance, results are presented separately for each of these two levels. The results of each level are then combined for each domain, and an overall assessment is provided across all domains. In order to achieve the highest transparency and to draw conclusions around the drivers of quality, the assessment results for each lever per domain are also provided. Supplementary Figs. 2 and 3 present the summary assessment for study level and PRs level, respectively, while Supplementary Figs. 4 and 5 present the individual study assessment results. The results for each domain of the ECR-P critical appraisal tool are presented in the following sections.
Advocacy and the use of emotive language
In the domain inform, not persuade, 87% of the studies (20 out of 24) were rated as high risk of bias. This is the second-worst-rated domain in the tool. It should be noted that only 11 out of the 20 studies were assessed for high risk in both study level and PRs level (Supplementary Figs. 4 and 5). At the study level, the areas that drove the high-risk results were concerned with the studies not reporting the limitations of their analysis and lacking a clear connection between their findings and conclusions.
Similarly, in the PRs level, these two areas were also identified as problematic since studies did not report the limitations of their PRs, and did not clearly connect their PRs to the findings of their research. From the studies that did report PRs’ limitations, the majority (10 out of 11) related the reported study limitations to both findings and PRs. Only one study31 addressed specifically the PRs’ limitations and recognised that they are contingent on factors that were not part of their analysis.
Almost half of the studies (48%) provided some PRs relevant to their findings but also put forward PRs that were not scientifically based (based on the scientific findings of their study). For example, it was observed that a lot of studies had PRs on promoting environmental consciousness as a way of increasing the use of green energies that would lead to reducing carbon emissions15,16,18,19,20,21. Although, these PRs might ‘make sense’ in the context of tackling climate change, they were not connected to the studies’ scientific analysis and findings.
In the PRs level, a third area was identified as problematic. Six studies were found to have used emotive language in communicating their PRs, indicating advocacy rather than neutral scientific reporting. Examples of emotive language included: “Climate-minded policymakers should implement unprecedented reforms and wean their citizens off fossil fuels…”23, “ …the use of obsolete, polluting technologies must be forbidden.”16, “Alternatives that are more ecologically friendly should be used in place of obsolete and incompetent technology”32.
On the other hand, the studies overall did very well in clearly reporting their aims and objectives for both research outcomes and PRs, proposing ways to tackle the reported study limitations in future research, avoiding the use of emotive language in their findings and conclusions, and using accessible language in their PRs.
Balance and the consideration of policy complexity
In the domain offer balance, but not false balance, 35% and 39% of the studies were rated as low RoB and some concerns, respectively (Fig. 5). On the study level, only two studies22,28 were rated as high RoB. Both these studies included a lot of different geographical areas (163 countries and 45 cities), but didn’t specify them or report how and why they were chosen for their analysis. For this domain and level, the tool asked whether a reporting guideline had been used. To the best of our knowledge, a discipline-specific reporting guideline does not exist for this field, therefore, the use of a guideline was not assessed.
The PRs level in this domain focuses on two areas: whether the authors acknowledge the inherent complexity of PRs and thus consider their potential multiple implications, and whether they have an overview of the current policies that are in place (or a lack thereof) for the issues they are researching. Indeed, most studies offered multiple PRs and in different policy areas, but only one study32 considered the potential negative implications of their PRs. Nineteen studies exhibited knowledge of current policies, but only nine of them considered the implications of not changing the status quo (Supplementary Fig. 5). High risk of bias was attributed to the four studies that did not mention the current policy status.
Disclosing and addressing uncertainty
The domain disclose uncertainties was rated with the highest risk of bias across all studies. Indeed, 96% were rated as of high RoB and 4% as exhibiting some concerns (Fig. 5). The study level focuses on reporting uncertainties in study findings and proposing ways to reduce them in the future. Most studies (20 of 23) did indeed report uncertainties, but only one30 suggested a way to alleviate them.
In the PRs level, studies overwhelmingly did not address uncertainties. Only one study discussed uncertainties, but only briefly and partially, stating ‘Therefore, we see this analysis as necessary first step for further research on the full implications of the EV rollout in the energy system and the wider economy’30. It should be noted that the study did not propose a way forward in addressing uncertainties in the future in any specific way.
An important distinction has to be made between the concepts of limitations and uncertainties. Limitations are found in methodology, data input and data analysis. Ultimately, these affect and ‘limit’ the study’s outputs. Uncertainties largely relate to the difference between the research findings and the ‘true values’ and help readers understand the degree of confidence in the study findings.
Evidence quality and the lack of reflection
Most of the studies (70%) raised only some concerns about bias in the domain state evidence quality. There is a clear mismatch between the rating of the study level and the PRs level (Supplementary Figs. 2–5). In the study level, 17 studies appear to have used high-quality data inputs, but only a small subsection of them actively discussed their quality. Only three studies went into detail about how data collection processes ensured data quality28, how data production was relevant for their analysis31, or acknowledged poor quality for a subset of their data11. On the other hand, six studies10,13,15,17,26,30 reported that they had used data of questionable quality, mainly coming from commercial companies’ reports (see Table 1), with no consideration given to their quality or to the obvious potential for competing interests. This led to these six studies having a high RoB rating at the study level. No specific metrics for data quality were used by any studies.
On the PRs level, the focus was on whether the studies considered the quality of the study findings that formulated the evidence base for the PRs. Only one study actively discussed this issue, stating: ‘Our study, like any other research, has limitations. They are caused by the data used and the availability of data, the time period under consideration, and the variables included in the models. All of these constraints can be addressed in future research, as well as the impact of specific policies on electricity-generated pollution’22. Not considering evidence quality for PRs led to nineteen studies being rated as of high RoB in the PRs level of this domain.
Misunderstandings and clarity of communication
The last domain examined was pre-empt misunderstandings. This domain focused on preventing misunderstandings, to help inoculate against misinformation those who were using the evidence. This was the domain that was rated best across all studies for both study level and PRs level. Only one study32 was found to raise some bias concerns, while the remaining 22 were rated as of low RoB. On the study level, the quality of the communication was assessed as to whether studies tried to pre-emptively address potential misunderstandings about the study findings and conclusions and ultimately, if any ambiguity was identified. In the PRs level, in addition to assessing the clarity of the PRs themselves, the targeted policy makers also needed to be defined. Both areas were rated as low RoB across all studies.
ECR-P overall quality reveals the prevalence of a high risk of bias
All studies were rated as having an overall high RoB within the ECR-P critical appraisal tool. Studies did better in some domains than others, but in the synthesis of the domains, all studies were rated with high RoB in at least one of them, thus resulting in an overall high RoB rating. These results indicate poor quality of PRs’ formulation and communication. The flow of the ratings through the five domains is illustrated in Fig. 6. The two worst-rated domains were domain 1: inform not persuade, and domain 3: disclose uncertainties.
The Evidence Communication Rules for Policy (ECR-P) critical appraisal tool addresses the risk of bias (RoB) in the five different domains that are illustrated at the bottom of the figure, followed by an overall RoB rating. Rating is either high RoB, some concerns or low RoB.
The critical appraisal process revealed that there were specific questions of the ECR-P tool that all studies felt short in addressing. The question that had the most negative responses (either “no” or “probably no”) was 3.2 ‘Were uncertainties of the policy recommendations reported?’, with 22 negative responses; followed by the conditional question 3.1.1 (If Y/PY to 3.1) ‘Did the study propose ways to reduce uncertainties in the future?’, with 19 negative responses; and question 4.2 ‘Was the quality of the study findings, that formulated the evidence base for the policy recommendations, considered?’, also with 19 negative responses.
The domain that was rated best across all studies was domain 5: Pre-empt Misunderstandings. The three studies that were best rated were by Calvillo and Turner30, Horobet et al.22, and Song and Chen12. The study that was rated best across all levels and domains was one by Calvillo and Turner30, rated with low RoB in three domains (1, 2 and 5), one domain with some concerns (3) and one with high RoB (4). This study focused on the rollout of electric vehicles in the UK. Interestingly, it was one of the two studies which had PRs as their main research outcome (the other being the study by Gilmore et al.11), and it was the only study that addressed uncertainties in their PRs. Calvillo and Turner30 were only rated poorly in domain 4. Regarding the study level, although they used secondary evidence as inputs for their energy system modelling from commercial companies (including Scottish Power and Bloomberg New Energy Finance), they did not consider the quality of their data sources. The issue was propagated in the PRs section. On the other hand, Song and Chen12 were also rated poorly in only one domain, but here it was domain 3, as the study did not address uncertainties in their findings or their PRs. This was the only case study included in the systematic review. Horobet et al.22 was the only other study (apart from Calvillo and Turner30) that was rated with low RoB in three separate domains (1, 4 and 5). Calvillo and Turner30 and Horobet et al.22 were the only two studies that were rated low RoB in the PRs level in domain 1, providing high-quality communication across the areas addressed in this domain (Supplementary Table 3).
There was sufficient agreement between the overall ratings of the two critical appraisal tools. The ratings were the same for 14 out of the 23 studies and for two additional studies when comparing the overall rating from CEECAT and the study-level rating from ECR-P. As expected, the agreement was much stronger for the studies that were rated poorly in the CEECAT. The direction of the rating was different (low versus high risk of bias) for only one paper. Agreement between the tools is presented in Supplementary Table 4.
