Peer Review

Scientific Thinking and Integrative Reasoning Skills (STIRS): Essential Outcomes for Medical Education and for Liberal Education

The Scientific Foundations for Future Physicians report recognized that evidence-based medicine is central to medical education and the practice of medicine. The report states: “It is essential not only to read the medical and scientific literature of one’s discipline, but to examine it critically to achieve lifelong learning. These activities require knowledge and skills in critical analysis, statistical inference, and experimental design” (Association of American Medical Colleges and Howard Hughes Medical Institute 2009).

There is strong evidence that most medical students and residents are not equipped with the tools to critically read the evidence and apply it to solve clinical or research problems. For instance, investigators from Yale surveyed Connecticut internal medicine residents state-wide and found, in an investigation published in JAMA, that only about 40 percent were able to correctly answer key study design and statistical questions (Committee to Evaluate the USMLE 2008)

If students are to gain competence in the ability to understand study designs and analyze findings, these skills need to be introduced and reinforced through practical application as part of the undergraduate curriculum as well as medical school and residency. The National Board of Medical Examiners has recognized this need and is piloting questions that utilize abstracts of articles and simulated journal advertisements to test understanding of study design and statistics as well as evidence-based problem solving (Windish, Huot, and Green 2007). Significantly, a basic approach to study design and analysis has been adopted by the MR5, the fifth comprehensive review of the MCAT, with the development of a framework for Scientific Inquiry and Reasoning Skills (SIRS) and the integration of this framework across the physical and biological sciences as well as the behavioral and social sciences components of the new MCAT exam.

The SIRS framework comprises the following skills:

  1. Knowledge of Scientific Concepts and Principles
    Recognizing and using fundamental scientific concepts and principles, retrieving information, identifying the relationships between closely-related concepts, and using mathematical equations
  2. Scientific Reasoning and Evidence-Based Problem Solving
    Using scientific principles, models, and theories to describe, explain, and make predictions about natural phenomena; working with theories and models to solve problems; and evaluating arguments about causes and consequences
  3. Reasoning about the Design and Execution of Research
    Evaluating features of research studies to determine if conclusions of the research are warranted, and making predictions based on the features of a research design
  4. Data-Based and Statistical Reasoning
    Using quantitative and qualitative data to describe and explain phenomena in the natural world; explain the relationships between variables; test hypotheses; solve problems; interpret, draw conclusions, and make comparisons; interpret and make predictions from qualitative or quantitative data presented in tables, figures, and graphs
    (Association of American Medical Colleges 2011):

The MR5 identified and AAMC has endorsed a more detailed list of capacities that students should gain in order to demonstrate each of the SIRS skills as part of the new MCAT exam. See Figure 1.

Figure 1. Capacities that students should gain to demonstrate SIRS Skills

Skill 1—Knowledge of scientific concepts and principles: Students should be able to

  • state or recognize correct science concepts and principles;
  • recognize relationships among closely-related scientific concepts and principles;
  • recognize relationships among different representations of concepts and principles;
  • use mathematical representations of concepts and principles; and recognize graphical or schematic representations of concepts and principles.

Skill 2—Scientific reasoning and evidence-based problem solving: Students should be able to

  • work with scientific models and theories to solve problems;
  • make claims and propose research questions or hypotheses based on scientific theories and models;
  • identify assumptions and logical inconsistencies in arguments to evaluate a conclusion;
  • propose and/or evaluate alternative explanations or predictions; work with the stated conclusions from a research study to solve problems;
  • problems within a given context; and
  • use and evaluate proposed solutions.

Skill 3—Reasoning about design and execution of research: Students should be able to

  • identify foundational aspects of research design (e.g., experimental vs. nonexperimental design, independent and dependent variables);
  • identify the appropriate research designs needed to address specific research questions or hypotheses;
  • critique different aspects of a research design (e.g., identify sources of potential bias, confounds, adequacy of sample);
  • evaluate research designs to determine if conclusions based on the research study are appropriate;
  • recognize ethical issues inherent in research investigations; and
  • make predictions about expected results based on the features of a research design

Skill 4—Data-based and statistical reasoning: Students should be able to

  • use data to describe phenomena in the natural world and/or to describe results of a research study;
  • use descriptive statistics to summarize data (e.g., mean, median, and standard deviation) and relate it to the population (e.g., standard error);
  • interpret data or patterns in data to draw conclusions or evaluate the conclusions made at the end of a research study (e.g., notice whether conclusions logically follow based on the data presented);
  • interpret data, or patterns in data, to make predictions;
  • use statistics to answer research questions and evaluate the strength of the evidence provided in support of given hypotheses; and
  • interpret data patterns presented in tables, figures, and graphs (e.g., histograms, scatterplots) to interpret results, make comparisons, and draw conclusions.
The AAMC MCAT 2015 Preview emphasizes that SIRS is still in the development phase and cannot yet be formulated as a comprehensive list of performance expectations. Consequently, these lists are illustrative, not exhaustive. (Association of American Medical Colleges 2011).

 

Cross Linking SIRS with Content Exams

SIRS will not form the basis for a separate component of the MCAT—rather its four Skills will be cross linked with the Foundational Concepts and content categories from each of the three content area components of the new MCAT exam. The content areas are

  • Biological and Biochemical Foundations of Living Systems
  • Chemical and Physical Foundations of Biological Systems
  • Psychological, Social and Biological Foundations of Behavior

Each of these three sections will be organized around three dimensions:

  • Foundational Concepts—the “big ideas” in the sciences that provide the foundation for learning in medical school;
  • Content Categories—the topics and subtopics that are needed to understand the foundational concepts; and
  • Scientific Inquiry and Reasoning Skills—the inquiry and reasoning skills that are required to solve scientific problems.

For instance, a question about laboratory studies of radiation exposure might require a student to examine the chemical and physical impacts of radiation on living systems and draw conclusions based on laboratory as well as population data. Such questions would link SIRS Skill 2 (Scientific Reasoning and Evidence-based Problem Solving) and SIRS Skill 4 (Data-based and Statistical Reasoning), with Chemical and Physical Foundations Concept 4 (Complex living organisms transport materials, sense their environment, process signals, and respond to changes using processes understood in terms of physical principles.) and Concept 5 (The principles that govern chemical interactions and reactions form the basis for a broader understanding of the molecular dynamics of living systems.)

In another case, SIRS skills could be tested with a set of questions focused on the relationship between neural-tube defects such as spina bifida and folic acid. For instance, a passage might test principles of folic acid biochemistry using Biological and Biochemical Foundational Concept 1 (biomolecules have unique properties that determine how they contribute to the structure and function of cells and how they participate in the processes necessary to maintain life.) and principles of neurological development using Biological and Biochemical Foundational Concept 2 (highly organized assemblies of molecules, cells, and organs interact to carry out the functions of living organisms). It might also incorporate Psychological, Sociological, and Biological Foundations of Behavior Foundational Concept 9 (social and cultural differences influence well-being) and Psychological, Sociological, and Biological Foundations of Behavior Foundational Concept 10 (social stratification affects access to resources and well-being) to ask students to examine the advantages and disadvantages of options for individual folic acid supplementation of women of childbearing age as well as supplementation of the food supply. SIRS Skill 3 (Reasoning About the Design and Execution of Research), as well as SIRS Skill 4 (Data-Based and Statistical Reasoning), can be cross-linked to issues of neural-tube defects and folic acid by asking students to evaluate the randomized controlled trials as well as observational studies used to demonstrate the effectiveness and safety of increased folic acid intake in reducing neural tube defects.

In examples like those above, SIRS should be viewed as the glue binding the diverse components of the MCAT exam and the competencies needed to prepare for medical school. These skills form a solid foundation on which to build a comprehensive sequenced curriculum in evidence-based medicine in medical school and beyond.

Scientific Thinking and Integrative Reasoning Skills (STIRS): A New Project at AAC&U

Integrating Scientific Inquiry and Research Skills into the MCAT and other preprofessional examinations is both a challenge and an opportunity because higher education remains without a comprehensive approach to teaching such basic skills, including research design and analysis, as part of an undergraduate education.

This issue of Peer Review marks the launch of a new project at AAC&U that incorporates these essential skills into a broader effort to encourage evidence-based thinking as a core outcome of general education. The Scientific Thinking and Integrative Reasoning Skills (STIRS) project aims to help reinvigorate the science and social science components of general education for all students, not just those preparing for the health professions. The STIRS project builds on existing AAC&U efforts, especially Liberal Education and America’s Promise (LEAP), to emphasize the importance of integrating evidence-based thinking into a wide range of disciplines including, but not limited to, the sciences and social sciences.

The STIRS project will expand the basic framework developed as part of the MR5 process to include analytical and logical reasoning and decision-making skills such as those used currently in the LSATs, as well as other standardized examinations, such as the GMAT and GREs. AAC&U will incorporate the STIRS outcomes into its ongoing efforts to help campuses create more intentional pathways between learning goals, curricular designs, and demonstrated student learning. In particular, AAC&U will use STIRS to highlight the integrative power of general education and to help students connect the learning that occurs in general education courses with their majors and with applications to real-world problems and projects.

STIRS will build on significant efforts at AAC&U and across the academy to define and advocate a set of Essential Learning Outcomes that inform curricular designs, pedagogical approaches, and resource allocation across undergraduate education. STIRS will focus important attention on efforts to ensure that all students can

  • Use scientific reasoning to gather and evaluate evidence
  • Understand how scientific and social science studies are designed and executed
  • Understand the implications of design choices
  • Use statistical reasoning to evaluate data and use data to communicate effectively
  • Make decisions based on analysis of evidence, logic, and ethics.

The draft STIRS curricular framework includes the following skills:

  • Scientific Reasoning and Evidence-Based Problem Solving
  • Study Design, Execution, and Implications
  • Data-Based and Statistical Thinking
  • Analytical and Logical Reasoning and Evidence-Based Decision Making

The STIRS framework is further illuminated through the development of “Enduring Understandings” designed to identify the key components that students need to take away from the curriculum, incorporate into their majors, and utilize as part of their continuing education for many years to come. Excerpts from a draft version of “Enduring Understandings” for each of the components of the framework are being developed and are illustrated below in figure 2.

Figure 2. Enduring Understandings of the SIRS Project

The following are examples of the types of enduring understanding that are being developed for each of the components of the AAC&U STIRS project:

Scientific Reasoning and Evidence-Based Problem Solving

Scientific reasoning may be divided into reductionist science and integrative science or systems thinking. Social and behavioral sciences as well as the biological and physical sciences may be structured as either reductionist or integrative sciences with integrative sciences often building on reductionist science. Reductionist science aims to create study and control groups that are as similar as possible except for the factor under investigation. Reductionist science is grounded in disciplinary approaches often determined by the levels of organization of the material and social world. Subatomic and atomic physics, inorganic and organic chemistry, zoology and botany, and population sciences such as epidemiology all may take a reductionist approach focusing on one relationship or association at a time.

Study Design, Execution, and Implications

Study designs may be divided into experimental and observational. In experimental designs the investigator intervenes to change the conditions and compare the outcomes in the intervention or study group(s) compared to outcomes in the control group(s) without the intervention. In observational studies the investigator observes the occurrence of events without intervening. The investigator may begin by identifying an independent variable and observing the subsequent occurrence or lack of occurrence of the outcome measured by the dependent variable. Alternatively, the investigator may first identify the occurrence or lack of occurrence of the outcome measured by the dependent variable and look back in time to identify the occurrence or absence of occurrence of the factor measured by the independent variable. Observational studies are potentially capable of establishing the first two criteria of contributory cause or efficacy: (a) that there is an association between the independent and the dependent variable at the individual level and (b) that the “cause” precedes the “effect.” Experimental interventions are often required to definitively establish the third criterion, namely that altering the “cause” alters the “effect.”

Data-Based and Statistical Reasoning

Statistical analysis aims to draw conclusions about large groups or populations based on investigations of smaller groups called samples. Samples are designed to be representative of the larger population of interest. The type of statistical analysis that is conducted depends on the goals of the analysis. Potential goals of statistical analysis include estimation of the magnitude or strength of the relationship, statistical significance or inference, adjustment or taking into account potential confounding variables, and prediction or prognostication. The specific statistical methods used also depends on the type of data used to measure the independent and dependent variables, the distribution of the data, and the frequency of measurement of the dependent or outcome variable.

Analytical and Logical Reasoning and Evidence-Based Decision Making

Analytical reasoning requires understanding of the structure of relationships and drawing logical conclusions based on that structure. Logical reasoning includes the analysis and critical evaluation of statements or arguments and the drawing of well supported conclusions. Reasoning by analogy, determining how additional evidence affects an argument, applying principles or rules, and identifying flaws in arguments are all key skills in critical analysis.

Excerpted from the draft of enduring understandings, developed for components of the AAC&U STIRS project.

STIRS project participants will

  • Develop and sustain a community of educators committed to teaching STIRS skills to all students as part of an integrative approach to general education.
  • Encourage, review, and reward the development of curricular modules that assist in teaching and assessing components of the STIRS framework through peer review recognition and the designation of “STIRS Scholars”
  • Utilize existing AAC&U VALUE Rubrics as an instrument to assess institutional achievement of the STIRS framework.
    For more information, or to get involved in the STIRS project, contact Kevin Hovland at hovland@aacu.org.

Conclusions

The changes in the MCAT set the stage for broader reform in general education, as well as in science and social science majors, that takes an integrative approach to evidence-based thinking. These skills have long been valued as part of liberal education but have not yet been integrated as core skills for all undergraduates. The AAC&U STIRS project aims to take the next steps by developing a curriculum framework encouraging a wide range of faculty to submit curricular materials that may be used to implement the framework. Peer review of the curricular materials and recognition of excellence by designation of “STIRS Scholars” will move scientific thinking and integrative reasoning skills to the center of conversations about the essential learning outcomes associated with general education and liberal learning. Consequently, higher education will be better able to equip students with key skills they need to understand—and perhaps begin to solve—complex, real-world problems.

Note

To receive a complete set of draft enduring understandings and provide to feedback, please contact Richard Riegelman at riegel@gwu.edu

References

Association of American Medical Colleges. 2011.Preview Guide for MCAT2015. Washington, DC: Association of American Medical Colleges.

Association of American Medical Colleges and Howard Hughes Medical Institute. 2009. Scientific Foundations for Future Physicians. Washington, DC: Association of American Medical Colleges.

Committee to Evaluate the USMLE. 2008. USMLE, The Comprehensive Review. http://www.usmle.org/pdfs/cru/CEUP-Summary-Report-June2008.pdf (Accessed October 10, 2012).

Windish D. M , S. J. Huot, M. L. Green. 2007. “Medicine Residents’ Understanding of the Biostatistics and Results in the Medical Literature.” Journal of the American Medical Association 298 (9): 1010–1022.


Richard K. Riegelman is a professor and the founding dean of the School of Public Health and Health Services at The George Washington University; Kevin Hovland is the senior director for global learning and curricular change at AAC&U.

Previous Issues