As part of the Evaluating Healthcare Simulation tools, the Simulation Effectiveness Tool – Modified (SET-M) was revised and modified from the Simulation Effectiveness Tool (2005) by CAE Healthcare (formerly known as METI), as part of the Program of Nursing Curriculum Integration. The SET-M is designed for evaluation of clinical simulation scenarios. Leighton, Ravert, Mudra, and Macintosh (2015) updated the SET to incorporate simulation standards of best practices and updated terminology. The researchers determined the tool required updating to capture the desired outcomes – learner’s perceptions of how well their learning needs in the simulation environment were being met. Several translated versions of the SET-M are available.
Downloadable Files for SET-M:
- The Simulation Effectiveness Tool – Modified
- The Simulation Effectiveness Tool – Modified with Subscales with Creative Commons License
- Sample SET-M Practice. Practice using this with your team to identify areas in need of improvement and create an improvement plan, then prioritize it! Consider how you might use these results to provide coaching or conduct an evaluation session with a simulationist.
- SET-M (5 Point Version)
- SET-M (French-Canadian Version)
- SET-M (Spanish Version)
- SET-M (Turkish Version)
- SET-M (Brazilian Portuguese Version)
Permission to Use FREELY: Permission to Use FREELY: This work is licensed under Creative Commons, which enables you to use the tool under certain conditions. A separate IRB letter is not required for research purposes: Simulation Effectiveness Tool-Modified © 2015 by Kim Leighton, Patricia Ravert, Vickie Mudra, Chris Macintosh is licensed under Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International. View this Creative Commons license.
Learn More About the SET-M
How the Simulation Effectiveness Tool – Modified (SET-M) was Developed: These reports, standards, and guidelines were used to modify language in the original SET:
- INACSL Standards of Best Practice: SimulationSM, (INACSL Standards Committe, 2013, 2016)
- Quality and Safety Education for Nurses (QSEN) competencies (Cronewett et al., 2007)
- Essentials of Baccalaureate Education for Professional Nursing Practice (American Association of Colleges of Nursing, 2008)
The goal was to update this survey’s statements to include active verbs, current terminology, and address missing or under-weighted topics. For example, when the original tool was designed, many simulationists were prebriefing, but it was not yet the standard. In addition, we have grown in our understanding of the importance of debriefing; therefore, expanded that section. The SET-M grew to 19 items, scored on a 3-point Likert scale. Detail about specific item changes can be found in the manuscript referenced below.
Reliability and Validity of the SET-M for Nursing: The study was conducted at two universities, in a total of 13 sites.
Sample: 1288 undergraduate nursing students in the medical-surgical semester participated in the study. Sample for Analysis:
Subscale Reliability:
Factor analysis increased the number of subscales to four:
1. Prebriefing
2. Learning (original scale)
3. Confidence (original scale)
4. Debriefing
Reliability and Validity of the SET-M for Medicine: The study was conducted at one medical school. Third and fourth semester medical students participated in the study. Each participant completed an already scheduled simulation experience using one of three methods: high-fidelity simulation, Harvey task trainer, or standardized patient. The factor analysis for this study found one single scale, with no subscales. Of note, data was not collected for the original prebriefing or debriefing sections at the decision of the school’s faculty.
Reliability of SET-M in Neonatal Resuscitation:
- Palmer, E., Labant, A. L., Edwards, T. F., & Boothby, J. (2019). A collaborative partnership for improving newborn safety: Using simulation for neonatal resuscitation training.The Journal of Continuing Education: Thorofare, 50 (7), 319-324. Doi: 10.3928/00220124-20190612-07
Using the SET-M: The SET-M is designed for evaluation of simulation scenarios. The tool has been tested in the nursing and medicine educational environments and with high-fidelity mannequins, Harvey simulator, and standardized patients. The concepts are broad; feedback has indicated that the tool is appropriate for the clinical simulation learning environments as well.
The SET-M can be administered following every simulated clinical experience or selected ones. Many schools have learners complete the SET-M after every scenario in every course. This allows for early identification of concerns or trends in the data; however, it can lead to survey fatigue among participants causing them to just mark every response the same, regardless of their real thoughts–they just want to get it over with!
Consider the stability of your learners and your facilitators. If the same facilitator has used the same scenario for more than two semesters, and the SET-M scores are acceptable–is it important to continue to collect the data? On the other hand, scores may have been acceptable for two semesters and then the facilitator changes–you should consider going back to data collection to facilitate early identification of concerns. Regardless of your decision, ensure it is consistent and defensible for accreditation and institutional effectiveness purposes.
Evaluation: The SET-M is useful for evaluating the learner’s perception of how effective the simulation was toward meeting their learning needs. Results can be used in three ways:
- Design and implementation of the simulated clinical experience: are there areas that can or should be improved?
- What percentage of the learners are marking the statement lower–if one person marks it low but the rest mark it high, then the impetus for change is lessened. However, if over 25% of the participants mark a statement low, then facilitators must reflect on the experience and make changes.
Evaluation of the Facilitator: When evaluating the performance of the facilitator, it is important that they are meeting the learning needs of the participants. At the end of each semester, individual items, subscales, and the overall scores should be evaluated to determine the facilitator’s effectiveness. This can be done by each course facilitated and overall. Over time, results should be trended. Decisions can then be made as to whether the facilitator is performing to expectations, requires development or remediation.
Evaluation of the Simulation Operations Personnel: Similarly, when evaluating the performance of the simulation operations personnel, it is important that they are meeting the learning needs of the participants. At the end of each semester, individual items, subscales, and the overall scores should be evaluated to determine the operations personnel’s effectiveness. This can be done by each course and overall semester. Over time, results should be trended. Decisions can then be made as to whether the operator is performing to expectations, requires development or remediation.
Scoring: There is no established method to score the SET-M. We suggest that you focus on the lowest scoring items and subscales first, and prioritize the most important changes that should occur, as well as the ones that can be made quickly and easily. Use these low scoring items as a needs assessment for creating a facilitator/simulation operations personnel development plan. An example is provided if you decide to consider use of the tool.
Update Note for Using Set-M for Virtual Simulation: As virtual simulation is a type of simulation, one can reasonably expect that simulation evaluation instruments can be used to measure the same concepts in that environment. Instead of “verbalizing” the SET-M revision to “Communicating” will better allow for learner responses whether the debriefing occurs synchronously immediately following virtual experiences, asynchronously at an established time following one or more experiences, or if debriefing is a written reflection done according to a set of questions.
References:
For additional detail regarding the development and psychometric analysis of the SET-M:
- Leighton, K., Ravert, P., Mudra, V., & Macintosh, C. (2015). Updating the Simulation Effectiveness Tool: Item modifications and reevaluation of psychometric properties. Nursing Education Perspectives, 36(5), 317-323. doi: 10.5480/1 5-1671
- Elfrink Cordi, V. L., Leighton, K., Ryan-Wenger, N., Doyle, T. J., & Ravert, P. (2012). History and development of the Simulation Effectiveness Tool (SET). Clinical Simulation in Nursing, 8(6), e199-e210. doi:10.1016/j.ecns.2011.12.001
- Suggested Webpage Citation: Leighton, K, Ravert, P., Mudra, V., & Macintosh, C. (2018). Simulation Effectiveness Tool – Modified. Retrieved from https://staging.healthysimulation.com/tools/evaluating-healthcare-simulation/simulation-effectiveness-tool-modified-set-m/
- If you need an official ‘Permission to Use’ letter, or have questions about this tool, please contact Dr. Kim Leighton.
SET-M 5 Point Version
The original Simulation Effectiveness Tool (SET) had a 5-point scale (strongly disagree, disagree, undecided, agree, strongly agree), which was changed to 3-points (do not agree, somewhat agree, strongly agree) in 2012 following a study of reliability using that scale (Elfrink et al., 2012). The literature was, and continues to be, inconclusive regarding the number of points that lead to the most reliability. In their manuscript, the original authors stated “We found no significant differences in the reliability of the 5- point versus the 3-point scale; however, future research may want to focus on examining the reliability and utility of a ver- sion of the SET that has a 5-point, Likert-type scale (p. e9).” It was this recommendation that led to the approval of Eric Hall’s work on establishing reliability with a 5-point scale.
Elfrink Cordi, V. L., Leighton, K., Ryan-Wenger, N., Doyle, T. J., & Ravert, P. (2012). History and development of the Simulation Effectiveness (evaluation) Tool (SET). Clinical Simulation in Nursing, 8(6), e1-12. doi:10.1016/j.ecns.2011.12.001
The Vanderbilt University School of Nursing simulation center had adopted the SET-M in 2020 to ensure we established an evaluation process for all our simulations. In 2022, a task force was established to ensure streamlining the simulation process; from scheduling, procurement, scenario development, and faculty education. The group had suggested altering the SET-M to a 5-point scale. As faculty and learners used the space, more felt they wanted a neutral option. Since 2022, our space internally adopted a 5-point scale variant of the SET-M.
Our sample size from January 2023-June 2025 was 330 participants. The data was assessed for reliability using Cronbach’s a. The tool and subscales were found to have excellent reliability for the 5-point scale.
Overall tool: 0.9698
Subscales:
- Confidence = 0.9609
- Debriefing = 0.9778
- Prebriefing = 0.9512
- Learning = 0.9319
Many who had experiences in our lab, especially using newer technologies like virtual and augmented reality simulation, had mixed feelings afterward. Both learners and faculty echoed the sentiment of how certain scenarios or technologies can be useful, but perhaps not best for their personal learning or teaching preferences. Additionally, certain modalities, while useful for one zone of simulation (assessment, skill-based objectives), might not be as useful for a higher zone simulation scenario, and vice versa. We appreciated this feedback; educators and learners everywhere are still learning as we look at the adoption of rapidly growing simulation technology, including artificial intelligence, augmented reality, and hybrid modality simulation. This feedback, and the common recurrence of the “neutral choice” in uses of newer simulation technology, informs us to adapt, refine, or even abandon experiential learning strategies to meet the educational needs of our learners.
We thank Dr. Kim Leighton and her team in allowing us to continue its use and publish novel simulation concepts with the data collected via our adapted SET-M.
Reference: McCall, W. T., & Hall, E. S. (2025). Implementation and evaluation of a mass casualty incident simulation for emergency nurse practitioner students. Clinical Simulation in Nursing, 107, 101808. https://doi.org/10.1016/j.ecns.2025.101808
Contributing Authors:
- Eric S. Hall, MSN, RN, NI-BC, CSSWB (eric.s.hall@vanderbilt.edu), Informatics Specialist, Vanderbilt University School of Nursing
- JoEllen Holt, DNP, RN, CHSE, CSSBB, Assistant Professor; Director, Simulation and Skills Lab
Download the SET-M 5 Point Version
SET-M (French Canadian Version)
The translation to Canadian French (CF) was done by two independent translators. The first translation (English to CF) was done by an expert in simulation pedagogy and reverse translation was done (CF to English) by an expert in program evaluation. Both translators are bilingual.
Download the French-Canadian Version of the SET-M.
The translated version is undergoing psychometric analysis but made available for use.
- Mélanie Radermaker, M.Sc.Inf., CHSE
- Sandra Duarte:
SET-M (Spanish Version)
How the SET-M (Spanish Version) was translated: A Spanish translation of the Simulation Effectiveness Tool – Modified (SET-M) was performed by Olvera-Cortés, H. E., Fernando, D., Argueta-Muñoz, Hershberger, Arena del, Silvia, L., Hernández-Gutiérrez, Guitiérrez-Barreto, and S. E. (2022) using the RAND corporation guidelines for this purpose, performing the translation and the reverse translation. This process was similar to the one used for the Turkish translation.
Download the Spanish Version of the SET-M.
Method: The data were collected from seventh semester students who participated in five different telesimulation-based learning experiences in the Medicine School of the National Autonomous University of Mexico. Descriptive and inferential analysis were performed to know the psychometric characteristics of the instrument. For this purpose, a discrimination analysis was made of each one of the dimensions using the Student’s t-test; Chonbach alpha was used to determine the internal consistency of the test, and to identify the internal structure of the test an exploratory factorial analysis was used.
Results: A non probabilistic sample of 2,479 students were analyzed. From the total population, 68% were women and 32% were man. The internal validity of the instrument is shown in Table 1, the principal components were obtained with Varimax rotation. This components were determine as Prebriefing, Learning and Confidence; and Debriefing.
Validity Analysis of the SET-M (Spanish Version): The measurement tool consisted of a 3-factor construct, and this construct explained 64% of the total variance of the measurement tool. The 3 subscales that occurred as a result of the factor analysis were identified as first subscale being prebriefing, second – learning and confidence, and third – debriefing. In Table 2 it is shown the principal component analysis. There were; 1 and 2 items in the first subscale, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, and 14 items in the second subscale, and 15, 16, 17, 18, and 19 items in the third subscale.
Flow diagram regarding factor loading between relevant items and factors (subscales) obtained after factor analysis is shown in Figure 1.
References:
- Olvera-Cortés, H. E., Fernando, D., Argueta-Muñoz, Hershberger, Arena del, Silvia, L., Hernández-Gutiérrez, Guitiérrez-Barreto, S. E. (2022). Evidencias de validez de la versión en español del Simulation Effectiveness Tool – Modified (SET-M) aplicado en telesimulación (Validity evidence of the Spanish version of the Simulation Effectiveness Tool – Modified (SET-M) applied in telesimulation). Educación Médica, 23(2). https://doi.org/10.1016/j.edumed.2022.100730
- Suggested Webpage Citation: Olvera Cortés, H. E., et al. (2022). Translation of the Simulation Effectiveness Tool – Modified to Spanish. Retrieved from https://staging.healthysimulation.com/tools/evaluating-healthcare-simulation/simulation-effectiveness-tool-modified-set-m/
- If you need an official ‘Permission to Use’ letter, or have questions about this tool, please contact Dr. Hugo Olvera.
Download the Spanish Version of the SET-M.
SET-M (Turkish Version)
How the SET-M (Turkish Version) was translated: The Simulation Effectiveness Tool – Modified (SET-M) was translated into Turkish based on the standardized guidelines including forward translation, back translation, cultural adaptation, and pilot testing.
Download the Simulation Effectiveness Tool – Modified (Turkish Version)
Language Validity of the Tool: For the language validity of the SET-M, a measurement tool was translated from English to Turkish by researchers and two translation experts who had good knowledge of both the languages. Reverse translation was done by two different translation experts who had good knowledge of both the languages. Later, along with the researchers and translation experts, the Turkish and English items were reviewed, and final editing was conducted.
Content Validity of the Tool: SET-M in Turkish, with translation and reverse translations completed, was presented to 10 experts who were working on clinical simulation in nursing for content validity. In the analysis of experts’ opinions, content validity index was used, in which the criteria were as follows: 1 – not relevant, 2 – somewhat relevant, 3 – quite relevant, 4 – highly relevant. According to experts’ opinions, the content validity index of the items was 0.95. The Turkish form of the measurement tool was revised following the experts’ opinions, and a pilot study was executed with 10 nursing students who were not included in the sample size. The measurement tool was not edited after the pilot study.
Reliability and Validity of the SET-M (Turkish Version)
Method: The data were collected from 235 students who participated in the simulation-based learning experience in the Faculty of Nursing of two public universities in Istanbul between January and June 2019. In the data analysis, descriptive statistics, exploratory factor analysis with varimax rotation, confirmatory factor analysis, item-total correlation, test-retest correlation, interclass correlation, Pearson correlation, Cronbach’s alpha coefficient, and ceiling-floor effect analysis were conducted.
Validity Analysis of the SET-M (Turkish Version): EFA results, which are conducted to determine the subscales of the measurement tools, showed that the measurement tool consisted of 4-factor construct, and this construct explained 62.2% of the total variance of the measurement tool. It was determined that EFA of this study explained the biggest part of the total variance with the first factor explaining 21.03%, the second factor 15.97%, the third factor 14.59%, and the fourth factor 10.60%.
The 4 subscales that occurred as a result of the factor analysis were identified as first subscale being prebriefing, second – learning, third – confidence, and fourth – debriefing. There were:
- 1 and 2 items in the first subscale,
- 3, 4, 5, 6, and 7 items in the second subscale,
- 8, 9, 10, 11, 12, 13, and 14 items in the third subscale, and
- 15, 16, 17, 18, and 19 items in the fourth subscale.
Table 2. Fit indexes calculated as a result of confirmatory factor analysis of the measurement:
Flow diagram regarding factor loading between relevant items and factors (subscales) obtained after factor analysis is shown in Figure 1:
Reliability Analysis of the SET-M (Turkish Version)
- Authors identified that the item-total correlation of the Turkish version of the SET-M ranged between r=0.47 and r=0.69.
- Cronbach’s alpha internal consistency coefficient of the subscales was examined, the coefficients were detected as 0.81 for prebriefing, 0.80 for learning, 0.83 for confidence, and 0.86 for debriefing.
- Stability reliability of the Turkish version of the SET-M was evaluated with the test-retest correlation and interclass correlation. It was determined that the test-retest correlations were positive, strong, and statistically significant for the total measurement tool (r=0.92) and the subscales (r=0.84–0.90; p<0.001). Interclass correlations were identified as 0.92 for total measurement tool, 0.90 for prebriefing subscale, 0.85 for learning subscale, 0.89 for confidence subscale, and 0.84 for debriefing subscale (p<0.001).
References:
- Sahin, G., Buzlu, S., Kuguoglu, S., & Yilmaz, S. (2020). Reliability and validity of the Turkish Version of the Simulation Effectiveness Tool – Modified. Florence Nightingale Journal of Nursing, 28(3), 250-257. doi: 10.5152/FNJN.2020.19157.
- Webpage suggested citation: Sahin, G., Buzlu, S., Kuguoglu, S., & Yilmaz, S. (2020). Translation of the Simulation Effectiveness Tool – Modified to Turkish. Retrieved from https://staging.healthysimulation.com/tools/evaluating-healthcare-simulation/simulation-effectiveness-tool-modified-set-m/
- If you need an official ‘Permission to Use’ letter, or have questions about this tool, please contact Gizem Sahin.
Download the Simulation Effectiveness Tool – Modified (Turkish Version)
SET-M (Brazilian Portuguese Version)
The translation of the SET-M to the Brazilian Portuguese Version was conducted by Professor Ellen Cristina Bergamasco, DNP and Professor Dina de Almeida Lopes Monteiro da Cruz, DNP. Their psychometric study of the SET-M adaptation and validation was performed in three stages: 1. Translation, adaptation, and evaluation of content validity; 2. Verification of construct validity (factor analysis and estimates of reliability and stability; and 3. Verification of the criterion validity. The sample included 214 nursing undergraduate students and 221 graduate nursing students. Cronbach’s alpha for the four subscales are: Prebriefing (0.792), Learning (0.823), Confidence (0.874), and Debriefing (0.758).
The article is published open access in Brazilian Portuguese, English, and Spanish and can be found with the following doi link:
Bergamasco EC, Cruz DALM. Simulation effectiveness tool modified (SET-M): adaptation and validation for Brazil. Rev. Latino-Am. Enfermagem. 2021;29:e3437. https://doi.org/10.1590/1518-8345.4282.3437
Contact Info:
Download the Simulation Effectiveness Tool – Modified (Brazilian Portuguese Version)
Return to the Evaluating Healthcare Simulation tools webpage.