Ungsinn has gone through considerable changes during the period of 2015-2016. The website has been converted from a knowledge database to an electronic scientific journal. At the same time, the criteria for classification of interventions have been revised. In articles published after November 2015, the interventions are classified according to the new criteria described below.
Basis for the criteria
The term, “evidence-based interventions” (interventions with documented effect) has been traditionally used to define interventions that have been documented with the so-called “gold standard” in research design. The gold standard has been and is still associated with RCT studies. The drawback in using the “gold standard” as the only criterion for evidence, is that results from studies with a weaker research design are not recognized. Interventions with positive results from studies using a less rigorous design are thereby evaluated in the same way as those without any documentation.
To exclusively acknowledge RCT as the only valid design for collecting evidence may result in intervention developers losing the motivation to perform evaluations. RCT studies require resources and are sometimes impossible to carry out due to practical or ethical considerations. At Ungsinn, we therefore argue that it is appropriate to have a more flexible system for the assessment of an intervention’s evidence. Less comprehensive evaluations may suggest indications of the effects of an intervention. The results are not as valid as those of an RCT study, but provide more knowledge about the potential effects of the intervention than if no study were performed at all.
Aside from the research design, other aspects of the quality of research methodology have substantial significance for the reliability and validity of research results as well. An RCT study may have such considerable methodological weaknesses that it is impossible to conclude whether the intervention is effective or not. On the other hand, non-randomized (quasi-experimental) studies of high quality can also provide strong grounds for claiming that the intervention is effective. By solely placing weight on the research design in evaluating a study, rather than on other methodological aspects as well, it is possible to make an incorrect assessment. Therefore, it is imperative to look at not only the research design but also the quality of other methodological aspects of the study. .
Classification of evidence levels
Ungsinn thus has five levels for evidence. The levels build on each other and contain requirements for the description of the intervention, the theoretical rationale, the number of studies conducted, the research design, methodological quality, effect sizes, follow-up studies and the implementation strategy. For the lowest evidence levels, where efficacy or effectiveness studies have not yet been conducted, research and evaluation requirements are not relevant.
The evidence levels, ranging from the simplest to the most comprehensive, are as follows:
Level 1: Well described interventions
Level 2: Theoretically-based interventions
Level 3: Interventions with some documentation of effect
Level 4: Interventions with satisfactory documentation of effect
Level 5: Interventions with strong documentation of effect
The requirements for the different evidence levels are described in detail below.
Level 1: Well described interventions
The intervention is thoroughly described and may be effective. The target group and objectives of the intervention are clearly stated, in addition to how the intervention is organized and which methods are employed.
| Evidence level 1: Well described interventions | |
| Dimension | Requirements |
| Description of the intervention | There is a satisfactory description of the intervention with objectives, target groups, intervention arenas, design and methods. |
| Theoretical rationale | The theoretical rationale for the intervention is not described or poorly described. |
| Total number of effect studies | No effect evaluations have been carried out for the intervention. |
| Research design | Not relevant. |
| Quality of research methodology | Not relevant. |
| Effects on the primary outcomes | Not relevant. |
| Follow-up studies | Not relevant. |
| Implementation requirements | Not relevant. |
Level 2: Theoretically-based interventions
In the description of the intervention, it is clearly stated how the methods employed are considered to lead to fulfillment of the objectives.
| Evidence level 2: Theoretically-based interventions | |
| Dimension | Requirements |
| Description of the intervention | There is a satisfactory description of the intervention and its objectives, target groups, intervention arenas, methods and progress. |
| Theoretical rationale | The intervention has a general explanatory model for human behavior or functioning combined with reflections on the correlation between the intervention and the outcomes for the target group. |
| Number of effect studies | No effect studies have been performed for the intervention. |
| Research design | Not relevant. |
| Quality of research methodology | Not relevant. |
| Effects on the main outcome variables | Not relevant. |
| Follow-up studies | Not relevant. |
| Implementation requirements | Not relevant. |
Level 3: Interventions with some documentation of effect
There is an indication that the intervention may be effective. Simpler effect studies in Nordic countries or international systematic reviews of the intervention have been carried out. The studies may have methodological weaknesses that create some uncertainty as to whether the results are valid, or there may be uncertainty as to whether the results from foreign studies are transferrable to Norwegian conditions.
| Evidence level 3: Interventions with some documentation of effect | |
| Dimension | Requirements |
| Description of the intervention | There is a satisfactory description of the intervention and its objectives, target groups, intervention arenas, design and methods. |
| Theoretical rationale | The intervention has a well-grounded theory for the anticipated effect mechanisms. The theory is directly related to relevant outcome variables that may be tested empirically. |
| Number of effect studies | There is either one Nordic study, or an international systematic review on the effects of the intervention, or there is a positive evaluation of the intervention’s evidence in international knowledge databases. |
| Research design | The studies are performed with pre-/post-design, norm or reference studies. Qualitative studies and case investigations or single-case studies, in which the effect of the intervention is examined, may be included. |
| Quality of research methodology | The study is carried out with satisfactory methodological quality according to the selected design. |
| Effects on main outcome variables | Effects are found for the main outcome goal(s), secondary outcome goal(s) or there is reporting of experienced effect from relevant qualitative studies. |
| Follow-up studies | Not relevant |
| Implementation requirements | Not relevant |
Level 4: Interventions with satisfactory documentation of effect
At least one Nordic efficacy study has been conducted with an adequate research design (RCT or quasi-experimental with control group). The methodological quality must be at least satisfactory for most aspects of the study, and effects are detected on the main outcome variables. In other words, there is satisfactory documentation that the intervention is effective when delivered in general practice.
| Level 4: Interventions with satisfactory documentation of effect | |
| Dimension | Requirements |
| Description of the intervention | There is a detailed manual or corresponding description of the intervention and its objectives, target groups, intervention arenas, design and methods. |
| Theoretical rationale | The intervention has a well-grounded theory for the anticipated effect mechanisms. It is empirically-based or directly related to relevant outcome variables. |
| Number of effect studies | There is at least one Nordic study. |
| Research design | RCT studies have been performed under ideal conditions or studies with a quasi-experimental design have been carried out. |
| Quality of research methodology | The study is performed with satisfactory quality regarding statistical analyses, use of measurement instruments along with evaluation of sources of bias, fidelity and external validity. Methodological quality must be scored at a minimum of level 2 for most of the methodological evaluation points of Ungsinn. |
| Effects on the main outcome variables | Effect sizes on the main outcome goals are provided or can be calculated. Effects are found on the main or secondary outcome variables. |
| Follow-up studies | Not relevant |
| Implementation requirements | Not relevant |
Level 5: Interventions with strong documentation of effect
There are at least two Nordic effectiveness studies (alternatively, one Nordic and one systematic review) for the intervention. The studies are conducted using a solid design (RCT or quasi-experiment under natural conditions), and the ratings of methodological quality are good. There should be effects on the important outcomes, also over time. There should be systems in place to ensure the fidelity and successful implementation of the intervention.
| Evidence level 5: Interventions with strong documentation of effect | |
| Dimension | Requirements |
| Description of the intervention | There is a detailed manual or corresponding description of the intervention and its objectives, target groups, intervention arenas, design and methods. |
| Theoretical rationale | The intervention has a well-grounded theory for the anticipated effect mechanisms. It is empirically-based and directly related to relevant outcome variables. |
| Number of effect studies | There are at least two studies, of which at least one is Nordic. If only one Nordic study exists, there should also be an international systematic review of the effects of the intervention, or a positive evaluation of the intervention’s evidence in an international knowledge database. At least one study must have been performed by an independent research group (i.e., independent of the intervention developer or other research group that has evaluated the intervention). |
| Research design | RCT effectiveness studies (natural conditions) or good quasi-experimental studies have been carried out under natural conditions (including longitudinal cohort studies). |
| Quality of research methodology | The study is performed with good quality as pertains to statistical analyses, use of measurements and evaluation of sources of error, fidelity and external validity. Methodological quality must be scored at a minimum of level 3 in the results section of the Ungsinn article. |
| Effects on main outcome variables | Effect sizes on the main outcome goal are provided or may be calculated. There are substantial enough effects on the main outcome goal that the differences between groups (cohorts) are assessed as having practical significance. |
| Follow-up studies | Follow-up measurements of the intervention’s effects at least 6 months after the end of the intervention show satisfactory results. |
| Implementation requirements | The intervention is ready for implementation and dissemination. There are quality assurance systems that enhance implementation quality and make it probable that the research results will be sustained in the course of general practice. |
Preferable supplemental documentation for Level 5: Interventions with strong documentation of effect
For the best-documented interventions, in addition to the requirements described above it is preferable to have more than one year of follow-up measurements, cost-benefit analyses, implementation studies, meta-analyses, supplemental qualitative studies and mediator analyses.
Interventions without effect or with negative effect
If studies are published documenting that an intervention has no effect, or even a negative effect, the intervention will receive a specific negative classification in Ungsinn. However, it is equally important that such documentation be reliable and valid in the same way as when positive effects are documented. If an intervention receives a negative classification, the studies that provide the basis must meet methodological requirements for evidence level 5 or 4 (Interventions with strong documentation of effect or interventions with satisfactory documentation of effect).
Unique research design:
There are many possible designs for evaluations of the effectiveness of interventions. This may include, for example, interrupted time series design, multiple baseline design, or N=1 design. Experience at Ungsinn to date is that such designs are not employed. If future searches discover these types of studies, they will be judged based on an evaluation of the design’s suitability to determine the effects of the intervention.
The text above is taken and translated from Martinussen, M., Reedtz, C., Eng, H., Neumer, S. P., Patras, J., & Mørch, W.T. (2016). Ungsinn – kriterier og prosedyrer for vurdering og klassifisering av tiltak. [Ungsinn – Criteria and procedures for evaluation and classification of interventions]. Tromsø: UiT The Arctic University of Norway