Other Pages in this Section
STIRS Framework with Enduring Understandings* and Keywords**
STIRS Framework Components:
- Evidence what it is and how it is used- Definition and uses of evidence across the disciplines
- Research Methods-Obtaining and ensuring the quality of the evidence
- Evidence-based Problem Solving- Using evidence to define and solve problems
- Evidence-Based Decision Making- Using evidence to define options and make decisions
NOTE: To view keyword definitions, hover your cursor over the underlined terms or see the full keyword glossary.
Definition and uses of evidence across the disciplines
1. Reductionist approaches to use of evidence
Evidence is built on information and forms the basis for supporting a conclusion. Evidence is defined, obtained, and used in a wide range of disciplines in the sciences, social sciences, health, humanities and the fine arts. Methods for obtaining and using evidence may be divided into one factor at a time or reductionist approaches and integrative approaches or systems thinking. Reductionist approaches aim to simplify by creating study and control groups that are as similar as possible except for the factor under investigation. Reductionist approaches begin with hypothesis generation which may result from inductive or deductive logic. Reductionist approaches aim at explanation or establishing the existence of cause and effect relationships such as whether an intervention has efficacy.
2. Integrative approaches to the use of evidence
Integrative approaches often build upon reductionist approaches. They draw from multiple disciplines incorporating multiple influences or determinants of outcomes; look for interactions between factors; and use evidence-based approaches to understand and propose strategies for addressing complex problems.; Integrative approaches aim to model the multiple influences or determinants of a single outcome rather than test a hypothesis. In doing this they view outcomes as the results of complex interacting systems. Reflecting on complex systems and questions before problems are posed and solutions considered is a key skill for integrative approaches.
3. Theories and paradigm shifts
Both reductionist and integrative approaches are grounded in theories that attempt to explain fundamental relationships in the material and social worlds. Theories need to enable hypothesis testing which aims to refute the theory. Phenomena that cannot be explained by current theories often lay the groundwork for new theories that challenge the existing paradigm and lead to infrequent but immensely important paradigm shifts.
4. Uses and display of evidence
Evidence can be used to achieve a range of important goals including problem description or question framing, generation of hypotheses, demonstration of etiology and efficacy, measurement of harms and benefits in applied settings i.e. net-effectiveness, evaluation of interventions, and prediction of future outcomes. Disciplines may focus on a limited number of these goals. Evidence-based problem solving aims to define one or more options with net-effectiveness for addressing a defined problem. Evidence-based decision making aims to provide methods for deciding between available options. Data-based reasoning requires an understanding of the relationships between the ways that information is presented and interpreted. Understanding how the visual display of data can effectively summarize large quantities of evidence as well as misrepresent evidence is fundamental to evidence-based problem solving and to integrating evidence into decision making.
5. Roles of statistical reasoning
Statistical significance testing uses data from study sample(s) to draw conclusions or inferences about larger populations. Statistical inference may be expressed as P-values or derived from 95% confidence intervals. Inherent in statistical significance testing are Type I and Type II errors. The strength of the relationship can be expressed in a number of ways including differences, proportions, and rates. The strength of the relationship or estimation needs to be distinguished from statistically significant. Adjustment for potential confounding variables is often needed before the investigator can draw conclusions about the existence of an association between a particular independent variable with the outcome or dependent variable. Multiple variable adjustment using multiple regression procedures allows the investigator to simultaneously take into account a large number of potential confounding variables.
6. Roles of analytical , intuitive, and logical reasoning
Analytical reasoning requires understanding the structure of relationships and drawing conclusions based on that structure. Analytical reasoning includes historical and interpretive methods as used in the humanities and fine arts. Logic models display the expected sequence of events and underlying assumptions which must be fulfilled to successfully achieve an intended outcome. Analytical frameworks plus logic models provide a coherent approach to evaluating potential interventions and drawing interpretations. Reasoning by analogy, determining how additional evidence affects an argument, applying principles or rules, and identifying flaws in arguments are all key skills in analytical reasoning.
Intuitive thinking utilizes information from multiple sources and senses and may produce outcomes without the user being able to identify the process used. Intuitive thinking based on experience is widely used and its strengths and limitation need to be understood. Intuitive thinking may also produce unique outcomes not based on experience which may contribute insights in the sciences, social sciences, health, humanities, and fine arts. It is important to appreciate the unique role of intuitive thinking in the humanities and fine arts. Disciplines have discipline-specific approaches to combining analytical, logical, and intuitive reasoning. An introduction to discipline-specific approaches and methods can provide an understanding of the thought processes and skills required for pursuing expertise in a discipline and may help students determine the fit of the discipline with their own talents and interests.
Where the evidence comes from and how its quality can be ensured
1. Research methods
Qualitative and quantitative evidence provide complementary methods for collection and use of evidence. Unique methods exist for obtaining and using evidence in the humanities and fine arts including reflection which is often critical for interpreting and creatively responding to problems. Qualitative evidence may provide the basis for generating hypotheses, exploring mechanism underlying observed outcomes, better understanding the interactions between influences which produce an outcome, exploring the factors which produce change as the basis for prediction, as well as understanding the basis for differing interpretations etc.
Quantitative study designs may be described as experimental or observational. In experimental designs the investigator intervenes to change the conditions and compares the outcomes in the intervention or study group to outcomes in the control or comparison group without the intervention. In observational studies the investigator observes the occurrence of events without intervening. Prior to beginning a quantitative investigation a study hypothesis needs to be defined. A study population defined by inclusion and exclusion criteria is also required. A sample size with adequate statistical power also needs to be estimated prior to beginning the investigation.
2. Assignment to groups and assessment of outcomes
Assignment to study and control groups in research studies may utilize randomization that is, assignment using a chance process. Assignment to groups may be observed without the intervention of the investigator. Differences between the groups which affect outcomes i.e. potential confounding variables may occur by chance or due to the assignment process. Pairing may also be used to prevent confounding variables and allow comparisons between individual(s) or situations.
Assessment of outcome needs to address the question being investigated. It needs to measure what it intends to measure which requires accurate, precise, and complete data collection. When surrogate or substitute measures are used they need to be strongly correlated with primary outcome of interest. Assignment and assessment are ideally masked so that neither the participants nor the investigators are aware of the group assignment when making the assessment of outcome.
3. Analysis of results
Estimation or measurement of the strength of a relationship or the size of a difference; inference or statistical significance; and adjustment for potential confounding variables are all expected components of the analysis of results in quantitative research. To accomplish these analyses, data must be categorized or placed into categories which may be classified as continuous, ordinal, or nominal data. The type(s) of categories used determine the amount of data which is incorporated into the analysis and the type of methods which are needed to analyze the data. Analysis should be conducted based on one primary outcome. If multiple outcomes representing multiple hypotheses are included in the analysis, the impact of these multiple comparisons needs to be taken into account when determining whether the results are statistically significant.
4. Interpretation-Criteria for cause and effect
Interpretation addresses conclusions for the situation or individuals included in the investigation such as whether a cause and effect or contributory cause exists. Observational studies are potentially capable of establishing the first requirement of contributory cause: (a) there is an association between the independent and the dependent variable at the individual level and the second requirement: (b) the "cause" precedes the "effect". Experimental interventions are often required to definitively establish the third requirement, namely (c) altering the "cause" alters the "effect". Contributory cause needs to be distinguished from necessary cause in which the presence of a "cause" is required to bring about an "effect" and sufficient "cause" in which the presence of a "cause" is all that is needed to bring about an "effect.
Often supportive or ancillary criteria need to be utilized to draw interpretations about cause and effect or efficacy. These supportive criteria include the strength of the relationship, dose-response relationship, consistency, and the biological plausibility of findings. Interpretation also includes examination of the impact of interventions on subgroups within a population. Subgroup analysis may include examination of the impact of interventions based on age, gender, race, socioeconomic status etc. An investigation should examine a limited number of subgroups defined prior to collecting the data.
Applying research to new situations or to a new population or context requires making assumptions. This process is called extrapolation or generalizability. Extrapolations to similar situations require the fewest assumptions and can be made using data from the investigation. Extrapolations that extend the use of the intervention to new situations or to new populations need to explicitly state the assumptions being made in performing the extrapolation. Prediction of individual and future outcomes is a form of extrapolation which is extremely difficult requiring large numbers of assumptions. A broad concept of extrapolation includes reflection or "looking back" as well as "drawing out" or carefully extending beyond the evidence.
6. Ethical principles for research
Research needs to be conducted based on ethical research principles including the principles of respect for persons, beneficence, and justice. Planning and implementation of research aims to maximize the benefits and minimize the harms to the participants. This requires prior review of research proposals by an objective external body, high quality research designs, informed consent for human interventional research as well as an expanding set of safeguards to ensure ethical implementation. Ethical standards for animal and laboratory research also need to be established and maintained.
Using evidence to define and solve problems
1. Approaches to evidence-based problem solving utilize both reductionist and integrative methods as well as evidence derived using qualitative and quantitative methods. Evidence-based problem solving often includes five steps: a) Problem identification and characterization; b) investigation of its etiology or causation and/or efficacy of potential interventions; c) development of evidence-based recommendations for potential interventions; d) examination of options for implementation of the interventions; and e) evaluation of the results of the intervention.
2. Problem framing and description- Framing the problem to solve requires evidence. Discipline- specific evidence is often needed including unique methods applicable to the humanities and fine arts. The time course of the problem, the burden of the problem, and often the financial costs are central to describing many problems. Describing the problem may provide a framework for integrating available qualitative and quantitative evidence to define what is known. It may also assist in developing a strategy for producing additional needed evidence. Evidence that describes the problem may be used to generate hypotheses.
3. Etiology/ efficacy and evidence-based recommendations
Evidence is needed to address the benefits andharms of potential interventions based on high quality study designs that address the definitive and/ or ancillary criteria for cause and effect. The quality of the evidence plus the magnitude of the impact, including potential harms as well as the potential benefits, need to be incorporated into evidence-based recommendations. Evidence-based recommendations need to include a category for insufficient evidence indicating that recommendations cannot be made either for or against adoption of an intervention. Evidence-based recommendations should indicate the process of developing the recommendations including the process and timing for updating the recommendations. Evidence-based recommendations are not limited to cause and effect questions and may relate to conclusions based on interpretation and evaluation of alternative interpretations.
4. Implementation and evaluation
Evidence-based recommendations need to be implemented and the outcomes evaluated. Implementation often requires considering when, who, and how to implement an intervention. That is, at what stage in the development of the problem, aimed at which groups or populations, and using what type(s) of methods. Evaluation needs to address the observed benefits and harms of the implemented intervention(s) as well as the potential population impact or reach of the intervention including its potential for sustained implementation in practice. Comparing net-effectiveness between potential interventions and including costs in the evaluation are desirable features of evaluation.
5. Methods of evidence-based problem solving-data synthesis and translational research
Data synthesis integrates existing research to derive new understandings that go beyond the conclusions from one particular investigation. Systematic reviews aim to address a wide range of relevant questions identifying the issues which are resolved and those requiring additional investigation. Meta-analysis is a quantitative method for combining investigations which address the same basic study question. Translational research provides a framework for connecting the roles of different types of research from basic innovations, to efficacy, to net-effectiveness, to population impact.
Using evidence to define options and make decisions
1. Heuristics and decision rules Heuristics or rules of thumb often govern human decision making due to humans' limited ability to process large quantities of data. Heuristics are an essential part of everyday decision making but are prone to a range of analytical and logical limitations. Decision rules, such as maximizing expected utility and satisficing, provide an objective basis for combining harms and benefits and selecting between options as part of evidence-based decision making. Unique methods are used in the humanities and fine arts to describe and challenge conventional approaches to decision making including understanding a text or historical event differently, which adds to the understanding.
2. Comparing benefits and harms
Evidence-based decision making is often conducted based on maximizing expected utility. This requires taking into account the probabilities of desirable and undesirable outcome(s), utilities or the importance placed on the outcome(s), and the timing of the outcome(s). Additional elements that need to be considered in decision making include attitudes toward risk including risk-taking and risk-avoiding attitudes and the reference point being used. The process required to implement the intervention may greatly impact decision making. At times financial costs may be a factor when comparing benefits and harms. Decision analysis and cost effectiveness analysis can be useful tools to structure the decision making process.
3. Principles of testing- Testing is used in a wide range of disciplines as the basis for making decision at the level of the individual, population, or system. It is important to recognize that testing is rarely perfect and results in both false negative and false positives. Defining false negative and false positive results requires a definitive or gold standard test. Applying testing requires estimation of the pretest probability of the condition. The information provided by the test may be measured using the sensitivity and specificity of the test. The probability of the condition after the results of a test are known is called the predictive value of a positive test and the predictive value of the negative test. Combining tests to make decisions is prone to error unless the tests used have different false positive and false negatives.
4. Prediction and prediction rules- Predictions of future events or the outcomes at the individual, population or systems level is extremely difficult. Prediction rules incorporate the factors observed to be associated with an outcome in the past, the strength of the association, and the interactions between these factors. Prediction rules may take the form of complicated models describing a system. Prediction rules may be used as the basis for individual, group, and systems level decisions. Prediction requires accurate and complete information on the factors affecting outcome based on past observation of these factors. Prediction also assumes that past relationship will continue into the future.
5. Systems analysis
A systems analysis approach models the factors which influence outcomes, the strength of these factors and their interactions as well as their changes over time. Systems analysis can be incorporated into evidence-based decision making. Systems approaches help in recognizing bottlenecks that impede positive outcomes and leverage points that provide opportunities to catalyze improved outcomes.
6. Ethical principles for decision making
Evidence-based decision making needs to recognize and utilize ethical principles. These principles may address the goals of decision making which may range from maximizing overall group benefit to distributional justice or minimizing harm to vulnerable groups of individuals. Ethics issues also bear on the process of decision making including balancing the rights and responsibilities of individuals, stakeholders, and society as a whole.