bHatim Bukhari, MS1*, Pamela Andreatta, PhD1
, Brian Goldiez,
PhD1 , and Luis Rabelo, PhD1
This article describes a framework that has been developed to monetize the real value of simulation-based training in health care. A significant consideration has been given to the incorporation of the intangible and qualitative benefits, not only the tangible and quantitative benefits of simulation-based training in health care. The framework builds from three works: the value measurement methodology (VMM) used by several departments of the US Government, a methodology documented in several books by Dr Jack Phillips to monetize various training approaches, and a traditional return on investment methodology put forth by Frost and Sullivan, and Immersion Medical. All 3 source materials were adapted to create an integrated methodology that can be readily implemented. This article presents details on each of these methods and how they can be integrated and presents a framework that integrates the previous methods. In addition to that, it describes the concept and the application of the developed framework. As a test of the applicability of the framework, a real case study has been used to demonstrate the application of the framework. This case study provides real data related to the correlation between the pediatric patient cardiopulmonary arrest (CPA) survival rates and a simulation-based mock codes at the University of Michigan tertiary care academic medical center. It is important to point out that the proposed framework offers the capability to consider a wide range of benefits and values, but on the other hand, there are several limitations that has been discussed and need to be taken in consideration.
return on investment, simulation-based training, health care, intangible benefits, qualitative benefits
Simulation is used in areas that are difficult to see, expensive to build, dangerous to operate, and so forth. Simulations can also be expensive to develop and maintain, and their ability to meet the requirements set forth in creating them is often difficult to ascertain in advance. Nevertheless, decision makers who must make investment decisions need some means to know that an investment will be fruitful compared with various alternatives that might be available. Most of the work to find the value of simulation to military training has been organized around cost avoidance. Little has been published in the open literature about a rigorous methodology that takes into consideration the different intangible factors during the life cycle of a simulator and the context of the organization.
The literature review showed 3 components that must be considered in determining the value of simulation: quantitative benefits, qualitative benefits, and costs and contributors to
value. Putting these 3 factors together in an integrated fashion gives decision makers a view of the value simulation offers.1,2 The 3 components of qualitative and quantitative benefits and costs are discussed below.4
The costs associated with the deployment of the simulation for health care include the costs of development, acquisition costs, the costs of maintenance, and the costs of operations. Labor to operate, teach, and maintain the simulator is included as appropriate.
University of Central Florida, Orlando, USA *The author is currently affiliated to University of Jeddah, Jeddah, Saudi Arabia
Received 21 September 2016; revised 8 November 2016; accepted 2 December 2016
Hatim Bukhari, Department of Industrial Engineering and Management
Systems, University of Central Florida, 1157 Hackberry Dr, Orlando,
FL 32825, USA.
Creative Commons Non Commercial CC-BY-NC: This article is distributed under the terms of the Creative Commons Attribution-NonCommercial 3.0 License (http://www.creativecommons.org/licenses/by-nc/3.0/) which permits non-commercial use, reproduction and distribution of the work without further permission provided the original work is attributed as specified on the SAGE and Open Access pages (https://us.sagepub.com/en-us/nam/open-access-at-sage).
The quantitative benefits generally are easier to recognize and measure. It embraces time savings, reduction in errors, faster time to competence, equipment breakage costs, reduction in alternative training costs, and procedures performed (quantity, time savings, etc.).
These factors were included in an ROI Excel-based tool developed by Frost and Sullivan for Immersion Medical.3 This tool includes major categories of benefits, costs, and other required information. The Frost and Sullivan tool represents the only comprehensive automated mean the authors have seen in the literature for computing ROI for computerbased simulation. Previously developed tools for computing ROI for other purposes have been based on cost avoidance with respect to using actual equipment for training or process improvement
Qualitative benefits are the benefits that are hard to measure and transfer into monetary value. Examples of qualitative benefits include the improvement of patient safety, quality of care, employee satisfaction, the reputation of the organization, and others
The work of Phillips and Phillips4 was used as a baseline to consider qualitative aspects related to value and ROI. Phillips recognizes that organizations need an algorithm that facilitates the steps to gather data to make informed investment decisions that can improve organizational training and performance. Phillips introduces an ROI methodology based on Dr Kirkpatrick's 4 levels of evaluation for measuring performance in Training and Human Performance Technology (HPT) programs: Reaction (level 1), Learning (level 2), Behavior (level 3), and Results (level 4).
Phillips expands on Kirkpatrick's 4 levels by creating the “Fifth Level” of ROI methodology for training and HPT programs. His modified approach evaluates the business value to the organization based on a particular investment or project to determine a framework for gathering program data to support and improve established training and performance programs. The use of simulation-based training fits well into Phillips's paradigm.
Phillips's ROI framework includes techniques used to evaluate the effectiveness of training programs. His approach estimates training impact by obtaining information directly from program participants and then having senior management make adjustments to the estimates. Adjustments are “essential because there are many factors that will influence performance data after training.5 ” Phillips reports that the effectiveness of his techniques lay in the assumption that participants who receive training are capable of estimating or determining how much improvement is related to the actual training program. By carefully crafting a questionnaire, information can be extracted to compute the qualitative
aspects of ROI. If the questions are orthogonal, a small number of respondents are needed to achieve statistical power.
This article proposes a generic framework that can be used to evaluate the ROI of different types and forms of simulationbased training in health care. The framework will provide a means to include the qualitative part of the benefits in ROI determination. Therefore, a methodology for measuring the nonfinancial benefits is needed and proposed. The framework is an integration of the value measurement methodology (VMM), Frost and Sullivan model, and Dr Jack Phillips approach in measuring the ROI in health care.
In July 2001, the Social Security Administration and the
General Services Administration along with a team of Booz
Allen analysts and thought leaders affiliated with Harvard
University's Kennedy School of Government developed an
effective methodology to measure the value of electronic services (i.e., quantitative and
qualitative values) that would
abide by current federal regulations and under the Office of
Management and Budget guidance, which is applicable to
the federal government. Later in 2002, the “How-To-Guide
and VMM Highlights” document was released. The methodology allows for a decision framework (US
Council, 2002) 6
to be personalized and adapted to the specific requirements of a project. Through the applications
VMM process, the value of alternatives to a program is articulated and the risk lowered for the
As stated above, the VMM help strategists in the Government to consider both tangible and intangible values when making investment decisions and monitoring benefits. Value is derived from the benefits generated directly to users, society, and other stakeholders
One of the important applications of the VMM methodology in the determination of ROI for simulation-based medical training is considering the value categories of the VMM to identify the tangible and intangible benefits and costs simulation-based medical training. The VMM value categories the authors determined for health care simulation are the following:
For example, Paige et al7 studied the impact of simulation-based interdisciplinary operative team training. All of
Figure 1. Methodology framework.
Note. ROI = return on investment
the participants completed a questionnaire after the training and the majority of them reported that the training would change their practices in the operating room. In addition to that, the training promoted team communication skills, crisis-related teamwork, and improved recognizing operating room errors. All these are forms of operational values that can be captured using the VMM as the operational value is one of the value categories of the methodology. The study was concluded by this qualitative outputs. These outputs require transformation to monetary value and that is when Phillips ROI methodology is applied to quantify the monetary value of this operational improvement. Then it can be considered in the calculation of ROI.
The framework uses the major categories of VMM value structure, which include direct, social, operational, strategic, and financial values, to identify tangible and intangible values and qualitative and quantitative benefits of the medical training simulator. This is accomplished by considering benefits that are preidentified by other studies such as the Frost & Sullivan3 ROI study for simulation based training (SBT) medical training and could be improved by several tools including questionnaires and/or interviews of experts. The next step is to isolate the effects of training using Phillips methodology to monetize the marginal improvement and benefit from using a specific training simulator. The isolation of effects method will vary based on the nature of the selected factors. Finally, the ROI is computed using the outcomes of the previous steps. Figure 1 shows the methodology framework.
Figure 2 summarizes the cost and value structure of the framework. It also depicts the different parameters that should be considered for ROI evaluation. Section (a) of the
figure shows the different types of cost and has 2 categories: project and operational costs. Section (b) shows the VMM categories that help in identifying the key parameters and measures. These measures are classified into qualitative and quantitative. Section (C) shows the transformation of qualitative and quantitative measures into monetary value, tangible values, which is done using Dr Phillips methodology. It also shows that part of the qualitative measures cannot be transformed into monetary value with high credibility, and as a result, will be considered as an intangible value and will not be considered for the calculation part of the ROI but it will help decision makers to make informed decisions.
Key measures and factors identification. The assessment of the ROI of a medical training simulator starts with the identification of the key impact measurement of the simulator that should be considered in the ROI analysis. For example, in the ROI analysis of a central venous catheter simulationbased training program for the medical intensive care unit at Northwestern Memorial Hospital, Pastrana et al1 identified medical care costs, length of stay, and number of complications as key impact factors as important measurements for inclusion in analysis. The measures that should be influenced by the simulator depend on the objectives pursued in acquiring the simulator. The identification process is facilitated by considering the major categories of VMM value structure, which include direct, social, operational, strategic, and financial values.
There are several ways and strategies to identify the measures including questionnaires and/or interviews of experts and executives, especially those who are involved in decision making regarding the simulator acquisition. In addition, considering the preidentified measures for the common types of projects is an effective method to begin with. For example, measures identified by other studies like Frost and Sullivan
Figure 2. Summary of cost and value structures and other factors contributing to ROI.
Note. ROI = return on investment.
ROI study for simulation medical training and Dr Phillips in his book Measuring ROI in Healthcare: Tools and Techniques to Measure the Impact and ROI in Healthcare Improvement Projects and Programs.
Converting qualitative and quantitative measures to tangible values. The outcome of identification process of key measurements is categorized into qualitative and quantitative measures. Quantitative data are easy to transfer to monetary value and considered as a tangible value. Qualitative data inclusion within the tangible values depends on the level of credibility of the converted data. Values lose credibility if the process used for the conversion is too subjective or inaccurate. The determination of lost value follows the guidelines described by Phillips. Qualitative data have 2 scenarios. The first is when the data can be converted to monetary values with high credibility. In this scenario, the data should be converted and included as a tangible value. The second scenario, if the data cannot be converted to monetary value with high credibility, then, it is considered as an intangible measure. Therefore, tangible measures include qualitative data and quantitative data when converted to monetary values with high credibility.
Steps to convert measures to monetary values. The following steps have to be applied for each measure to convert it to monetary value:
Isolating and evaluating the impact of the simulator. The next important step is to isolate the effects of the simulator. There are several approaches to isolate the impact of the training simulator. In general, there is no one single best approach to isolate or evaluate the impact of the simulator. Therefore, impact evaluation approach could vary based on the nature of the specific measure being considered. An analytical approach could be applied by using control groups, trend line analysis, or forecasting methods.
The other approach is the estimation. One way to use estimation to identify the human performance improvement developed as a result of the training is Dr Phillips strategy to estimate the improvement. The initial estimate should be
done through a questionnaire that trainers
and trainees take
prior to and after the training to estimate their performance
improvement due to the training and confidence levels in the
estimations. Decision makers then review and adjust the estimations and identify the important
factors and parameters
that need to be considered in the calculations of the ROI.
Simulator costs calculation. The other major factor to calculate the ROI is the cost and expenditures of the simulator. The costs can be classified into 2 major categories: project phase costs and operational phase costs. Project costs include initial analysis and assessment costs, the costs of development of the project, acquisition costs, and implementation costs. Operational phase costs include maintenance costs, support, overhead costs, labor, and materials supplies. Both categories should be included in the total costs for credible calculation.
ROI evaluation. The ROI evaluation has to be calculated according to the conditions set by the agency and/or health care business environment. The literature recommends the following formula:
ROI (%) Net Benefit of Simulator/ Total Cost of Simulator = x100
Estimated costs and benefits are among the inputs of the
simulator's ROI calculation process. The presence of uncertainty in investment project always
the presence of
risk on investment, for example, negative ROI in certain scenario.9
Therefore, it is essential to consider the different possibilities of costs and benefits to evaluate
risks and to work
on the mitigation. Prioritizing the risks based on impact is an
important step in risk management. In the case of ROI, identifying the key measures that have a
impact on the ROI
is essential for risk management.
Sensitivity analysis studies how the uncertainty in a model outputs can be apportioned to diverse sources of uncertainty in inputs of the model.10 Therefore, performing sensitivity analysis is helpful and could be performed to identify the factors and measures that have the major influence on the ROI. Because the framework involves estimation, which always has a level of uncertainty, we always consider the confidence level in the calculation. The other way to consider uncertainties is using ranges instead of exact values.
Monte Carlo simulation is an effective tool that uses computers and probabilities to generate scenarios when variables are expressed in ranges. For each scenario, a randomly selected value for each unknown variable is used for the calculation. This process is repeated for many times to help decision makes to project the possible scenarios and make informed decisions.
The case study in this section will demonstrate the application of the framework. This case study provides real data
Table 1. The Details and Totals of the Cost of the Training
|Item||Per unit cost||Units||Subtotal|
|Physician routine||125||240||30, 000|
|Faculty educator start-up||80||80||6400|
|Faculty educator routine||80||240||19,200|
|Simulation technician start-up||60||80||4800|
|Simulation technician routine||60||240||14400|
|Total program cost||391 600|
from Andreatta et al11 related to the correlation between the
pediatric patient cardiopulmonary arrest (CPA) survival rates
and a simulation-based mock codes at the University of
Michigan tertiary care academic medical center. The objective of this study was to assess the
effectiveness of the training on patient outcomes at residents’ confidence in
performing resuscitation. This study was conducted over a
period of 48 months, in which mock codes were called on an
increasing rate and the clinicians responsible for pediatric
resuscitation are required to respond just as they would on
and the actual CPA event. Events where recorded and performance feedback was given by clinical
faculty to the participating clinician’s residents, nurses, allied health, and
attending physicians. The CPA survival rate for the hospital
before and during the study was examined. The results of this
study showed that the survival rate was increased by approximately 50% correlating with the
increasing number of mock
The application of the framework will start with developing the cost structure of the training, then the identification of the key parameters to be included in the ROI assessment using the different VMM categories. Next, the conversion of qualitative and quantitative data to tangible values will be executed. After that, the effects of the training will be isolated to evaluate the ROI of the training compared to its cost.
Table 1 includes the details and totals of the cost of the training. These include start-up costs for developing the scenarios, programming the mannequins, coordinating the delivery, and designing the assessment/evaluation strategies. There are also costs associated with the routine occurrences of the
program, which include hourly rates for those who contributed to the start-up as well as the participants who were active during the mock code (average rate for the team is used). The costs for simulator, ancillary equipment, materials, and supplies are included, along with the maintenance agreements for the period of time the program took place. Facility charges are per hour for the code time only.
The framework recommends considering the 5 major categories of value from the VMM:
direct, social, operational, strategic, and financial value. Apart from the direct impact on
social value, strategic value of patient safety, and financial
value of the increased neonatal/pediatric CPA survival rate,
this case study will consider the intangible factor of turnover
rate of physicians, which has a strategic and financial impact
on health care organizations.
Physician turnover is a very costly problem for health care organizations. According to Fibuch and Ahmed,12 the negative impact of physician turnover should be a big concern for health care organizations as it has an impact on the profitability and the quality of care. In addition to the hiring and training costs, negative impacts such as productivity losses, noteworthy loss of organizational history, knowledge and expertise, disturbance of the morale of the remaining employees, and potential adverse publicity for the organization are expected. Therefore, the study highlighted the importance of incorporating employee retention strategy and considered the opportunities for advancement and learning new skills among the important factors of employee retention strategy. Considering the advancement opportunity provided by simulation-based training justify incorporating the cost saving of employee retention in ROI analysis.
Computation of turnover costs and understanding its implications in health care are conceptually challenging because of 3 reasons. First, health care is simultaneously driven by market forces and controlled by regulation and as a result, accounting concepts cannot be applied directly to health care without major adjustment. For example, revenue does not equal reimbursement nor does cost equal charges. Second, the mathematical computation of costs is complex and varies with the type of employee and employer as turnover costs for physicians is far more complex than it is for maintenance staff. Third, due to the difficulty of attributing revenues and costs, the net effect of the turnover is almost noncalculable.13 Therefore, in this case study, we will consider the out-of-pocket costs that have been mentioned in Waldman et al.
Waldman et al13 used several databases at an academic medical center as a foundation for measuring costs of employee selection, hiring, and training, as well as qualitative and quantitative yardsticks used to measure employee productivity. The study by Waldman et al13 drew accounting records and data for specific organizational units within the academic medical center and categorized the costs of turnover by phase of recruiting process: hiring, training, working, and termination. Estimates have been made in few instanced. Even though the study has estimated the average turnover costs for about 6 categories of employee, in this case study, only the turnover cost of physicians and nurses will be considered.
Note that the costs to train individuals involve mandatory courses, orientation classes, and reimbursed time when not generating charges.
Despite the fact that we could not find actual data on the
change of turnover rates for our particular case study, several
studies have shown a connection between lower turnover and
making investments in the training and development of
human resources in health care.
The following assumptions will be used in ROI calculations: over the 48 months of the simulation-based training, at least 1 physician and 1 nurse every year has preferred not to leave the organization because of the training opportunity. The annual cost saving of retention will be = $170°000 + $27°500 = $197°500.
The cost saving over the 48 months of implementation = 4 x$197°500 = $790°000.
ROI (%) = $790 000 -391 600 / 391 600 x 100 = 101.7.
Clinical personnel, whether they are training or maintaining their abilities, need a safe way to practice decision-making and applied skills as individuals and in teams. Mechanisms for assuring opportunities to practice and rehearse using simulation-based
methods have significant benefits for patient safety, not the least
is because actual patients are not involved in the processes. Even
though real world has stress and tension, simulation-based practice provides an environment that
offers the opportunity to focus
on building and acquiring skills where learning is facilitated with
less stress and tension on the practitioner, especially in high-risk
clinical contexts where performance providing care for real
patients could negatively affect the learning process. Therefore,
simulation-based training offers an essential solution for providing clinical personnel the
opportunity to learn, practice, and
maintain their abilities without stress or risk to real patients’
lives.15 Still, administrative decision makers that must determine
if the investment in facilitating simulation-based environments is
sufficiently beneficial with a convincing ROI compared with
various alternatives that might be available.
The integrated framework presented herein enables the determination of ROI with the consideration for both tangible and intangible values and benefits resulting from simulationbased training, including a demonstration of the application of the framework to a specific case study. The application of the framework for this specific case study considered only a single aspect of the value categories of the VMM, with demonstrated %101 ROI for this one aspect alone; a convincing ROI to help the decision-making process. If other aspects were considered in the evaluation, the ROI would be further developed to accommodate both tangible and intangible outcomes and provide a more comprehensive analysis. A limitation of this study is that these data were not available for the analyses conducted for the case study; however, the framework provides a foundation for the types of data that would be beneficial for future studies evaluating the ROI of institutionally supported simulation-facilitated environments
It is important to point out that the proposed framework offers the capability to consider a wide range of benefits and values that fall under any value category of the VMM including direct, social, operational, strategic, and financial values which enable more comprehensive evaluation for the ROI of the program. However, this would depend on the availability of the information that can help in converting these figures and information into monetary values to incorporate them into the ROI calculations. One example of an obvious added value is the legal obligations and consequences that are avoided as a result of improved clinical outcomes, such as survival rates for the considered case study. Another major value credited to extensive training, which could be achieved using simulation-based methods, is minimizing the medical errors. According to Makary and Daniel,16 medical error is the third cause of death in the United States, causing about 400°000 preventable deaths during the year 2013 alone. Understanding the real ROI and value of medical training, including highly effective simulation-facilitated methods, provides a foundation for fostering investment in best practices that have a positive impact on patient safety and quality of care. These are major objectives that impact the whole of health care systems globally
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
The author(s) received no financial support for the research, authorship, and/or publication of this article