Compliance and Surgical Team Perceptions of WHO Surgical Safety Checklist; Systematic Review

World Health Organization (WHO) Surgical Safety

In 2009, the WHO launched the surgical safety checklist (SSC) as part of their “Safe Surgery Saves Lives” campaign to improve surgical care adherence, consistency, and communication. The 19-part WHO-SSC was developed under Professor Gawande's team, who hypothesized that a surgical safety tool, analogous in nature to ones used by aviation pilots, would enhance communication and teamwork, providing more consistent performance of surgical teams in patient safety/care measures.⁷

Consequently, the WHO conducted a landmark multinational, multihospital pilot study accessing 7800 patients across hospitals in 8 economically diverse countries (Canada, India, Jordan, New Zealand, Philippines, Tanzania, England, and the United States) and found that SSC decreased mortality and complications by 48% and 37%, respectively.^8,9 The checklist required the surgical procedures be interrupted at certain times (phases), to allow important the dissemination of information to all participating members in the form of a standardized visual checklist. These phases being: sign in (before induction of anesthesia); time out (before skin incision); and sign out (immediately after skin closure).⁷ The briefing required the undivided attention of all participants and effective leadership from the facilitating member. Importantly, the WHO study reported that implementation of SSC reduced health care–associated costs through a simple, easy-to-use tool.⁹

Rationale

Following the release of this landmark study, the SSC has gradually been introduced in nearly 6000 hospitals worldwide.¹⁰ The WHO estimated that at least 500,000 deaths per year could be prevented through worldwide implementation of this checklist.^10,11 The Royal Australasian College of Surgeons recognized its importance and adopted an amended version for use in Australia and New Zealand.¹² However, the exact mechanism by which SSC improves patient outcomes is poorly understood, with later studies unable to consistently reproduce the marked reduction in mortality and morbidity rates reported in the primary study by WHO. The latest and largest of such studies is the multicenter Canadian study by Urbach et al¹¹ that assessed 215,000 procedures across 101 hospitals, found that morbidity and mortality only decreased by 0.05% postimplementation. Raising questions as to whether these disparities are due to an underlying poor compliance rate; limited training; or the consequence of cultural, hierarchal, or staff priorities influencing outcomes. Moreover, are these factors also pertinent in SSCs implemented across Australia and New Zealand?¹²

This review aims to assess the implementation of the WHO-adapted SSC within the operating room setting. The primary objective is to compare compliance rates of SSC across specialties and hospitals. The secondary objective is to explore surgical team perceptions of SSC pre- and postimplementation and identify potential barriers to its effective compliance.

Search strategy

The preferred reporting items for systematic reviews and meta-analyses (PRISMA) was used as a model to guide selection of articles (Fig. 1).

View Figure

• Download as Powerpoint
• Download Figure

An extensive database search of Medline (Ovid; 1946–present); PubMed (1966–present); and Embase was carried out for all publications up to December 2014. The search used a combination of specific key words (“mesh”, “surgical safety checklist” along with “compliance” or “adherence” or “implementation”) plus a concurrent search with the following terms: “surgical safety checklist”' along with “perceptions” or “attitudes” or “questionnaire” or “survey.” The search strategy was adjusted to the dictionary of other databases as appropriate, but the key terms were kept the same. The search was updated weekly to include any new published material.

Study selection and data collection process

The study selection process was carried out by two researchers (LW, YH). The initial search (LW) was performed based on the aforementioned inclusion and exclusion criteria to establish a preliminary shortlist of applicable studies. Subsequently, all identified titles were independently screened (YH) to exclude or include individual studies. Contentious articles had their abstracts and methods independently reviewed, with any disagreements resolved through consensus.

All major study designs (i.e., cohort, retrospective chart review, questionnaire-based, and observational studies) were examined. After removal of duplicates, the data were selected based on title and abstract to be included in the systematic review. In addition, the citations of the included articles were manually searched for other relevant articles (citation snowballing) to encompass all pertinent studies.

Study quality (risk of bias)

All studies are prone to bias and confounding factors due to methodological decisions. Our search identified a myriad of studies varying in design, strength, and quality. Consequently, pre-established quality assessment tools were deemed necessary to critically appraise each of the study types. The well-recognized strengthening the reporting of observational studies in epidemiology (STROBE)¹³ tool was used for observational studies, while the consolidated criteria for reporting qualitative research checklist (COREQ)¹⁴ was used to evaluate questionnaire studies. Domains selected to determine quality of study were: “conflict of interest” and appropriateness of: study design, participant size, data collection, data analysis, and conclusive reporting.

Accordingly, a 4-point quality assessment scale from 0 to 3 (0 = inappropriate; 1 = appropriate to some extent; 2 = largely appropriate; 3 = appropriate to purpose) adding to a maximum score of 15 was created. A study was considered of good quality if all domains were “largely appropriate” or better, with no conflict of interest reported, resulting in a quality score of >10. Studies were considered to have a high risk of bias if 3 or more domains were assessed as “appropriate to some extent” or if any one of the domains was determined to be “inappropriate.” This quality assessment was performed independently by authors, with disagreements of >3 points resolved through consensus.

Data items

Currently, no protocols exist in terms of data items to be used to access SSC compliance or attitudes. Thus, each study examined a myriad of quantifiable measures. Nonetheless, this review determined 5 key categories of data extraction in each of the studies:

• 1.

  Study details: design, method of participant recruitment, inclusion and exclusion criteria, and consent;

• 2.

  Risk of bias: conflict of interest; study design; participant selection; data collection; blinding, concealment, and analysis; and other sources of bias;

• 3.

  Participants: description, geographical location, setting, number, role of staff in hospital;

• 4.

  Limitations: varied methodology, small sample size, narrow questions; and

• 5.

  Results: compliance measure, improvement in compliance post-training, and quantifiable change in staff perceptions.

Summary measures

For the purpose of this review, compliance studies (Table 1) were tabulated separately from surgical team perceptions (Table 2). In quantitative compliance studies, a numerical measure (%) of compliance was identified and applied to the entire checklist, its individual phases, or other shared endpoints.

Table 1 SSC compliance rates in each study

View Fullscreen

Table 2 Surgical team perceptions of the SSC

View Fullscreen

Questionnaire-based studies examining perceptions and attitudes of surgical teams were evaluated with a set of common predefined categories based on the reviewed articles. Quantifiable measures of these categories were graphically presented. P values were shown where applicable.

Compliance and adherence studies

Prospective observational, retrospective chart review or self-reporting study designs were used in the 13 studies that measured compliance of SSC implementation. Ten studies used direct independent observation in evaluating team member completion of SSC^18,19–25; two of these had trained medical student observers while the remaining studies involved nurses and other trained medical staff. Retrospective chart reviews were used to determine rate of compliance in 2 studies,^26,27 1 adding data from a concurrent observational study.²⁷ In the remaining study by Kasatpibal et al,²⁸ registered nurses present in the operating theatre were asked to self-report compliance with individual points of the SSC. However, all studies provided quantifiable measure (%) to assess compliance. In terms of compliance by specialty, three studies focused on pediatrics, otolaryngology, and traumatology respectively, while the remaining studies measured compliance across specialties.

The studies were often multicentered across mainly developed countries. Three were undertaken in the United States,^20,22,23 2 in New Zealand,^21,24 and 2 in the United Kingdom^18,25; the remaining originated from France, Spain, Sweden, Thailand, Switzerland, and Canada.^{16,19,26–28,31}

Even though these studies used similar methodologies, they measured a myriad of parameters, resulting in a wide heterogeneity of results. To fulfill the aims of this review, 5 common parameters were applied to demonstrate the significant findings from each these study (Fig. 2):

View Figure

• Download as Powerpoint
• Download Figure

• 1.

  Checklist initiation rate for each phase (if given);

• 2.

  Completeness of the checklist;

• 3.

  Verification of participants/ team members;

• 4.

  Verification of patient identity and procedure; and

• 5.

  Other pertinent aspects of patient history and procedure including anticipated critical events, operative site, and prophylactic antibiotic administration.

The results indicated that in 8 out of the 13 studies,¹⁶ checklist rates of initiation were >90%,^{16,19,21–26} indicating a significant rate of compliance (>90%), based on assessment of previous studies and standards executed by the WHO in SSC administration.^29,30 Vogt et al²¹ noted an initiation rate of 100%.²¹ Two studies reported initiation rates of approximately 80%.^18,27 However, Cullati et al³¹ recorded a 99% “quasi-systematic implementation” rate, but only 72% actual compliance rate. The remaining study did not record initiation rates.

Time out adherence rates were assessed in 5 studies, with another 5 assessing compliance of each individual phase of the checklist. All relevant studies concluded that compliance rates were generally higher for sign in and time out phases compared with sign out^{18,21,24–27} (Fig. 3). Vogt et al ²¹ found sign out to have been completed in only 2% of cases; only 8.8% of cases completed sign out in the work of Pickering et al.¹⁸ Other studies noted higher measures of sign out adherence: 47% for Fourcade et al²⁶, 43% for Saturno et al,²⁷ 22% for Hannam et al,²⁴ and 19% for Cullati et al.³¹

View Figure

• Download as Powerpoint
• Download Figure

Checklist completion rates were assessed by 9 studies and found to be considerably less than corresponding compliance rates.^{16,18,19,22–24,26,27,31} Completion rates ranged from 28.4% in the study by Saturno et al²⁷ to 85% for Sparks et al,²² not accounting for accuracy of completion. In the study by Sparks et al,²² a high completion rate (85%) was negated by a low accuracy rate of 54%, associated with direct observation. A consistent difference of >25% between initiation and compliance rates was a recurrent phenomenon across studies by Cullati et al,³¹ Fourcade et al,²⁶ Levy et al ²³ and Saturno et al.²⁷ Interestingly, 2 studies demonstrated a marked improved adherence rate of >90%, following educational intervention and training, illustrating the importance of SSC.^22,25

Across the studies, verification or introduction of team members at the outset of the checklist was marginally compliant (>80%) in 4 out of the 7 studies that assessed this parameter. Levy et al²³ found that team identification occurred in only 10% of cases, citing role confusion between team members and lack of training as probable causes. Observers in the study by Spence et al¹⁶ suggested that the informal approach to SSC completion contributed to verification of team members occurring in only 20% of cases. While the cultural impact on communication styles was implicated in the study by Kasatpibal et al ²⁸ Thai study where team member introduction occurred in 79% of cases.²⁸ Pickering¹⁸ and Saturno²⁷ recorded team identification rates of 77.4% and 95.3%, respectively.

Implementation of patient identity or procedure verification via checklist implementation was reported >70% in 8 studies that assessed this domain.^{3,16,20,23,25–28} Furthermore, 4 out of 7 studies demonstrated >95% adherence to this aspect of the checklist.^23,26–28 Such results are expected, given that patient and procedure identification were essential aspects of OR safety protocols prior to SSC introduction.

Assessment of other aspects of the SCC was also explored in 4 studies. Three studies revealed that attention to pertinent critical aspects of the checklist, including allergies, blood loss, and antibiotic prophylaxis, had a poor adherence when compared with the common domains discussed previously.^19,23,27 However, a high incidence (92%) of “antibiotic prophylaxis given in 60 minutes” was recounted in the study by Rydenfält et al.¹⁹

Staff perception

The search strategy resulted in a total of 13 articles that looked at staff perceptions and attitudes toward SSC. Studies used 2 main methodologies to assess team member attitudes: focused interview of a random sample of operating theatre staff (2 studies)^32,33 and a surgical team questionnaire pre- and post-SSC intervention (8 studies).^9,34–40 The studies varied in participant size with 37 to 40 participants in focus group studies across the OR disciplines,^32,33 while the surgical team survey questionnaire studies had participant sizes as large as 1748. The surgical questionnaires were based on resources from previous studies or adapted from the surgical attitudes questionnaire (SAQ). Typically, all 3 disciplines—surgery, anesthesiology and nursing—were represented in the studies encompassing views of all OR stakeholders in the SSC.

The studies spanned across 17 different, countries including both developed and developing nations; the multinational study by Haynes et al⁹ incorporated 8 different nations. SSC attitudes were assessed within the context of a single surgical specialty in 3 studies,^34,35,38 while the remaining 10 studies were not specialty specific. The surgical specialties reviewed were obstetrics; ear, nose, and throat; trauma; cardiothoracic; and general.

All studies reviewed revealed heterogeneity in study structure and focus, exploring a myriad of staff perceptions.^32,33,35,36 To compare these studies, 4 common aspects encompassing the key issues that WHO-SSC developers claimed the checklist addressed were adopted¹⁰:

• 1.

  Team member communication and teamwork;

• 2.

  Teams member understanding of their own and others roles and responsibilities;

• 3.

  Team perceptions of whether the SSC improves patient or operating room safety; and

• 4.

  Team awareness of procedures to prevent errors.

Within this review, we considered self-perceived benefit of >70% in any of these categories as being significant enough to signify a positive impact.⁴¹

The SSC was reported to have a positive impact on teamwork within the OR in 9 out of 13 studies (Fig. 4).^{9,32–36,40,42,43} Improvements noted were increased “team feeling,”³⁶ “strengthened teamwork and efficiency,”³³ and identified enhanced team communication evidenced by increased discussion of critical events.⁴⁰ Studies undertaken 1-year postimplementation reported 47 to 57% of medical personnel found SSC improved communication.^35,37 A general improvement in communication brought on by the SSC was reflected in all studies, irrespective of specialty, profession and country. However, in the study by Böhmer et al³⁸ the 40% improvement in staff cooperation and communication found three months into SSC introduction was no longer evident 24 months postimplementation.

View Figure

• Download as Powerpoint
• Download Figure

A team member's understanding of their own role and the roles of others is key to effective implementation of SSC.⁶ Establishing each team member's identity and responsibilities during SSC initiation enhanced team integrity, functionality, and sense of worth.^32,40 Yet, in the study by Nilsson et al,³⁶ only 14% of participants thought that “introduction of team members” was important in SSC.³⁶ Additionally, of the 5 studies that examined this category, participants in 3 studies suggested that SSC did not substantially improve team member identification.^9,32,35,39 Six studies conducted cross-sectional analyses across OR staff disciplines (surgeons, anesthetists, and nurses).^{34,35,37,39,40,42} Nurses valued the gains made by SSC introduction far more than other staff members, but also were the most sensitive to the barriers inhibiting its completion, such as poor teamwork.⁴⁰ Anesthetists, on the other hand, were least positively disposed toward checklist completion when compared with surgeons and nurses,^37,40 potentially because it is completed during a period of high workload for anesthetists.⁴⁰ Interestingly, Papaconstantinou et al³⁷ observed that US surgeons rated their participation in SSC implementation higher than that of anesthetists and nurses. On the other hand, Kearn et al ³⁵ demonstrated that nonmedical staff had a significantly more positive outlook toward SSC than medical staff using the checklist.

The overall contribution of SSC to patient or OR safety was examined in 7 studies (Fig. 5). All studies reported significant positive responses from their respective participants.^{9,32,33,36,37,39} When OR staff members were asked about the use of SSC in cases where they themselves were to have an operation, positive responses ranged from 93%³⁶ to more than 80% to this question, indicating the importance of patient safety to team members across these studies.^33,36

View Figure

• Download as Powerpoint
• Download Figure

The degree to which the SSC builds staff awareness of the procedure to prevent errors was measured by 7 studies (Fig. 6).^{9,32,34,36,37,39,43} While studies noted differing, yet positive responses about the ability of the SSC to prevent errors, all studies reported only minor improvement in team awareness. Specifically, team members reported that the SSC provided brief pertinent information about patient's history and risk, as well as the required procedure, increasing overall situational awareness.^32,37,39 However, studies by Nilsson et al³⁶ and Helmiö et al³⁴ reported that the SSC did not provide any new information or “significant change in knowledge of patient's history, medication or allergies,” it just ensured common “mishaps” were not overlooked.^34,36

View Figure

• Download as Powerpoint
• Download Figure

Significantly, studies in which thorough training prior to SSC introduction was undertaken confirmed a dramatic improvement in attitudes toward the SSC across all professions, with participants in all studies advocating its use.^32,34

Surgical team perceptions

Encouraging better teamwork and communication in the operating theatre is a key mechanism through which safety checklists advance surgical safety.^7,8 While this is an argument often cited by checklist developers and implementers, it has not been conclusively reviewed to date.⁴⁶ The methodology of the available studies concerning these aspects of the SSC was largely limited to questionnaire-based staff surveys, which examined a myriad of team member opinions. This diversity in questioning made effective integration of results to reach meaningful conclusions quite challenging. Nonetheless, the important multifold findings were evident.

While self-perceptions of teamwork and communication improved unquestionably, a clear reduction in communication errors following SSC implementation was also evident. However, there was substantial variability in these improvements across the literature.^35,36,42 Takala et al ³⁹ reported more than 95% (P < 0.05) improvement in communication 6 weeks postimplementation, while Böhmer et al ³⁸ recorded no significant improvements 24 months postimplementation. These mixed results, indicate that team-member positive perceptions of the SSC may change over time as complacency grows and pragmatic barriers such as hierarchal differences, staff shortages, or prioritization of other duties become more evident.^19,26 Assessment of the prevalence of these barriers for each individual hospital followed by the implementation of relevant education is essential prior to implementation.⁶

Although the checklist is generally well received by OR staff, a lack of rigor in its application is evident in the literature, leading to a false sense of security and the possibility of compromised safety.⁴⁷ Furthermore, if OR staff place differing importance to SSC adherence, SSC completion might actually antagonize team relationships/interactions and widen pre-existing power differentials.¹⁵ Importantly in some studies, SSC implementation did not mitigate the professional hierarchy, but can actually accentuate the power differential due to its perceived “staged” nature.^38,40,43,48

OR staff perception studies suggest that OR nursing personnel perceive maximum benefit from SSC introduction, while surgeons perceive the least positive impact, with anesthetists falling in-between.^32,34,37,40 Education prior to implementation needs to consider the results of these studies in relation to lead roles within the OR. Given that most hospitals delegate the responsibility for SSC completion to nursing staff,^32,40,43 other OR staff need to understand the value of diligent completion of the SSC and support nurses in this role. Therefore, it is quintessential to involve all OR staff in the implementation processes, to mitigate inherent interdisciplinary differences in attitudes toward the importance of the SSC.^34,39

The argument that SSC implementation takes undue time was reported to be unfounded. Conversely, studies suggest that the SSC saved time by mitigating delays caused by miscommunication and confusion.⁴⁰ More specifically, preoperative briefings actually reduced delays in the OR by one-third, while also involving all team members in a holistic safety protocol.³⁹

Another recurring theme across studies was the importance of effective leadership in cultivating enthusiasm to improve compliance rates.⁴⁰ Generally, stakeholders valued the checklist stating it “provides information I would not otherwise have.”³⁶ Spence et al¹⁶ observed a low rate of actual verbal verification between team members, with observers stating that the SSC was completed informally, with the “nurse sitting of to the side with the checklist” and no formal identification of team members, procedure, antibiotics given, or anesthetic review occurring. Moreover, formal vocalized completion of the SSC, as recommended by the WHO, was reported to heighten belief in patient safety, teamwork, and error prevention, compared with informal, noncommunicative completion, where opposing opinions could not be voiced.^16,36,37

Prospective research implications and review limitation

The heterogeneity of methodology, study design, response rate, and study quality for both compliance and staff perception studies limited the ability to comprehensively analyze all data.²⁹ These articles assessed numerous endpoints along with their primary aim necessitating the development of shared key endpoints from these studies as uniform benchmarks for cross examination.⁴¹ Nonetheless, it was quite difficult to separate the various effects being reported and to identify the impact of each end-point specifically.

In studies examining SSC compliance, it is prudent to acknowledge that numerous local and regional factors influenced these results, illustrating the wide distribution of values across categories. While the original study by Haynes et al⁹ suggested that “mortality was strongly associated with checklist compliance and completion” and compliance rates varied depending on effective implementation, no analysis of this relationship has occurred. Similarly, none of the staff attitude studies assessed good compliance/implementation in respect to improvement in communication, teamwork, or self-perceived reduction in errors. Thus, it is difficult to establish causal links between compliance and end outcomes, as no reproducible standardized tool to measure compliance against mortality/morbidity rates has been developed to date.²⁹ Further research is required to evaluate the impact of SSC using measures of compliance, clinical outcomes, and staff perceptions.

Another limitation is the lack of reliability of recorded SSC compliance due to the Hawthorne effect.²² In this effect, participants perform better in an observational setting than in their default pre-existing setting. This effect was alluded to in a number of studies, with SSC adherence declining marginally when observers were not present.^9,22,27 Perhaps a better way of assessing SSC compliance involves routine or random recording of procedures by OR staff as observers, thus eliminating the Hawthorne effect while also avoiding obtrusive presence of unwanted observers in operating rooms.²⁷

In the staff perception studies, 11 studies used ad-hoc developed questionnaires, 8 of which had not been validated by any means,^{35,37–40,42,43,49} with some studies evaluating only 2 quantifiable endpoints.^33,38 However, validated, reliable tools measuring clearly defined outcomes such as communication, teamwork, and patient safety in a surgical setting are now available. Follow-up studies using recognized tools such as the SAQ⁶ or observational teamwork assessment for surgery instruments'⁴⁶ are warranted.

An additional problem encountered with 6 studies was that they assessed SSC within months of its introduction, without providing sufficient time for the SSC to be fully incorporated into practice and initial implementation issues to be resolved. In the 2 longitudinal staff survey studies, that assessed attitudes 1 to 2 years postimplementation, the results reflected both initial and sustained impacts which were found to be more modest.^36,37A similar argument can be made for compliance studies where no positive effect of the SSC was evident 2 years postimplementation.³⁸ Considering that the SSC introduction is a relatively new safety measure, comprehensive studies have yet to be conducted that explore its long-term success.²⁹ Further research is needed to assess long-term attitudes of OR staff following further training and attention to local logistical factors. Such studies would need to be designed as multicenter blinded observational studies of significant size.

Lastly, our review was conducted across PubMed, MEDLINE, and Embase databases only; hence, some studies might have been overlooked during the initial search. However, since only 5 articles were added as a result of citation snowballing, it is likely that most major studies were included. Nonetheless, due to resource limitations, 2 articles related to this review could not be obtained.^5,50 These articles could have revealed other findings that were not considered in this review.

Study quality

The varied study designs along with the above-determined limitations necessitated the need for a quality assessment tool within this review.²⁹ While all studies were evaluated against the same quality assessment tools (STROBE¹³ and COREQ).¹⁴ Nonetheless, not all assessment criteria were applicable to each of the studies. The differing quality score of the assessed studies can potentially affect the impact of their result (Fig. 7a and 7b). Therefore, false low quality scores could have been assigned to otherwise appropriate studies.

View Figure

• Download as Powerpoint
• Download Figure