Validation of the ACS-NSQIP Liver Module: A Single-Center Experience

Introduction: Large centralized databases are being increasingly utilized to assess surgical outcomes and determine reimbursement. The accuracy of liver resection data within the American College of Surgeons National Surgical Quality Improvement Program (ACS-NSQIP) is assessed. Methods: From August 2007 to May 2013, a retrospective review of liver resections captured by single-institution “standard” ACS-NSQIP data and a prospectively-maintained database was performed. Results: Of the 93 liver resections captured by ACS-NSQIP, 12 were incorrectly classified (false positive rate, 12.9%). Of the 81 true liver resections reported (139 actually performed), ACS-NSQIP demonstrated high fidelity with the prospective database with respect to preoperative characteristics, median length of surgical stay (6 days, interquartile range: 4-7 days vs. 6 days, IQR: 4-7 days), and 30-day mortality (1.2% vs. 1.2%). The “standard” ACS-NSQIP underreported the overall complication rate (29.6% vs. 43.2%; P < 0.001) and failed to capture liver-specific complications including biliary leak, liver failure, pleural effusion, postoperative ascites, and small bowel obstruction (overall sensitivity, 68.6%). Conclusions: Overall reporting of liver resection data within ACSNSQIP is generally robust. While the ACS-NSQIP Procedure-Targeted program for liver resection instituted in 2012 would capture the majority of liver-specific complications, further refinement may improve the accuracy of this database.


Introduction
The American College of Surgeons National Surgical Quality Improvement Program (ACS-NSQIP) was first developed by the Department of Veterans Affairs (VA) in the early 1990s as a means of measuring the system's overall quality of surgical care [1]. Today, ACS-NSQIP gathers data on over 150 clinical and surgical variables across over 550 institutions internationally [1,2].
ACS-NSQIP data has been used to improve general surgical care by means of reduced complications, mortality, and cost [3][4][5]. Surgical subspecialties, e.g., vascular and colorectal surgery, have also benefitted from the multi-institutional input ACS-NSQIP provides [6][7][8]. Recent studies have used ACS-NSQIP data to assess outcomes after hepatobiliary surgeries such as liver resection, including morbidity and mortality in the elderly [9]; surgical site infections and other postoperative complications [10][11][12]; as well as length of surgical stay [5].
Moreover, with the implementation of the Affordable Care Act (ACA) it is anticipated that public reporting of surgical outcomes may become increasingly linked to the Centers for Medicare and Medicaid Services (CMS) reimbursements [6,13,14]. A group of hospitals in ACS-NSQIP are already participating in this by voluntarily publicly reporting a small number of risk-adjusted outcomes [14], and for this reason the validity of ACS-NSQIP and other national databases should be assessed.
Along with its rigorous reviewer training process, ACS-NSQIP performs its own quality-assurance auditing [15]. Nevertheless, a few studies have shown concern over the ability of ACS-NSQIP to effectively assess outcomes regarding specific surgical procedures, namely within the field of hepatobiliary and pancreas surgery [10,16]. As an alternative to the "standard" database, in July 2011 a "Procedure-Targeted" option was implemented to address such issues [17]. Although hepatobiliary-specific modules, including that for liver resection, were implemented in July 2012, their procedurespecific outcomes data may be limited since many variables are not captured by ACS-NSQIP. The main objective of our study was to perform a retrospective review assessing the accuracy of liver resection data within the "standard" ACS-NSQIP database.

Patient population and data source
This study is a retrospective review, which relied on two databases: ACS-NSQIP and chart review. Chart review data was obtained from a prospectively maintained institutional database. This study was approved by the University of California, Irvine Institutional Review Board.
All adult patients who underwent elective or emergent liver resection for any diagnosis at our institution over a 5-and-a-half year period from August 2007 to May 2013 were included. Liver resection was determined using the following Current Procedure Terminology (CPT) codes: partial hepatectomy (47120), trisegmentectomy (47122), total left hepatectomy (47125), and total right hepatectomy (47130).
Preoperative variables were gathered from the medical record. Data included basic demographics, body mass index (BMI), and presence of comorbidities. Perioperative variables were gathered from surgery, anesthesia, and pathology reports, including operative time, tumor size and number of segments resected. Discharge diagnosis was identified using the appropriate diagnostic codes. To simplify the diagnosis variable we classified each diagnostic code into one of seven groups (Table 1).
The ACS-NSQIP was retrospectively reviewed for all reported cases of liver resection using the same CPT codes for single institution

Outcome definitions
Our primary outcomes included overall morbidity, length of surgical stay, and 30-day mortality rate. Overall complication rate was defined as a patient having at least one complication of ANY type within 30 days of operation. Although ACS-NSQIP tracks perioperative morbidity events and mortality that occur within 30 days of operation, the database did not begin gathering all-cause 30day readmission until January 2011. Therefore, complete readmission data was only available in chart review data. We further stratified 30day readmission to include only unplanned readmission related to the index case, as previously described [18].
Secondary outcomes reported were specific postoperative complications gathered by both ACS-NSQIP and our database, which included: superficial and deep incisional surgical site infection (SSI), organ space SSI, wound dehiscence, pneumonia, unplanned intubation, pulmonary embolism, greater than 48 hours of ventilatory assistance (failure to wean), progressive renal insufficiency, acute renal failure, urinary tract infection, stroke or cerebrovascular accident, coma of more than 24 hours, cardiac arrest, myocardial infarction, postoperative transfusion, deep vein thrombosis requiring treatment, sepsis, and septic shock. Superficial surgical site infections, deep surgical site infections, and wound disruptions were grouped together into one variable termed wound infection. Sepsis and septic shock were grouped together as a single sepsis variable [16].
The "standard" ACS-NSQIP database does not capture data on liver-specific complication variables. However, additional postoperative events captured by our institutional database and included in analysis of overall complication rate included biliary leak, liver failure, pleural effusion, postoperative ascites, and small bowel obstruction. Biliary leak was defined according to the International Study Group of Liver Surgery (ISGLS) guidelines as persistent drainage of bilious fluid (total bilirubin greater than three times serum bilirubin concentration) after postoperative day three [19]. However, our study included only those events requiring percutaneous drainage, ERCP stenting, or other surgical intervention. Post-hepatectomy liver failure was also defined according to ISGLS guidelines as an increasing INR and increasing serum bilirubin concentration on or after postoperative day five [20]. Pleural effusion was reported if responsible for hypoxemia and/or dyspnea requiring thoracentesis [21]. Postoperative ascites was defined as intraabdominal fluid requiring procedural intervention, e.g., paracentesis. Small bowel obstruction was only considered if requiring nasogastric decompression or surgical intervention.

Statistical analysis
Chart review was considered the gold standard [16,22]. The level of agreement between ACS-NSQIP and chart review with respect to demographic, preoperative comorbidity, and operative characteristics was assessed with percent agreement and Cohen's kappa statistic [23], except difference in continuous variables (i.e., age, body mass index) were compared using the Wilcoxon signed rank sum test. With regards to outcomes, to measure agreement in postoperative complications, sensitivity and specificity were calculated for each individual variable as well as overall complication rate. In addition, McNemar's test for correlated proportions was employed to assess difference in overall complication rate [24]. We compared differences in length of surgical stay and 30-day mortality with Wilcoxon signed rank sum test. All continuous variables are presented as the median and interquartile range (IQR). When applicable, a significance level of 0.05 was used. Data analysis was performed using IBM SPSS Statistics for Windows, Version 22.0 (IBM Corp., Armonk, NY).

Patient cases, demographics, and operative characteristics
Our prospectively maintained institutional database identified 139 liver resections, while query of the ACS-NSQIP database returned 93 liver resection patients at our institution. Among these 93 ACS-NSQIP patients, 81 patients were identified as true liver resections when compared to chart review, and 12 patients were incorrectly classified (i.e., wedge biopsies, ablations, adrenal resection), giving a false positive rate of 12.9%. Further direct comparisons were made between the 81 patients correctly captured by ACS-NSQIP.
When examining demographic and operative characteristics, ACS-NSQIP demonstrated near complete agreement with the chart review, and there were no statistically significant differences in any variable. Demographic and operative characteristics of both ACS-NSQIP data and chart review are illustrated in detail in Table 1.

Surgical outcomes
ACS-NSQIP accurately captured one death within 30 days of operation (30-day mortality, 1.2%). Median length of surgical stay did not differ between ACS-NSQIP and chart review (6 days, IQR: 4-7 days vs. 6 days, IQR: 4-7 days, respectively). There were 24 patients reported by ACS-NSQIP as having any complication (overall 30-day complication rate, 29.6%), compared with 35 patients in our chart review (overall 30-day complication rate, 43.2%). The 11 cases of patients with any complication who were not captured were due to the following: two organ space infections, two postoperative transfusions, one wound infection, and six liver-specific complications. The overall sensitivity of ACS-NSQIP for reporting patients with ANY complication was 68.6%, while the specificity was 100.0%. An exact McNemar's test determined these overall complication rates to be significantly different (p < 0.001).
Postoperative transfusion was the most commonly reported complication in this patient group. Both chart review and ACS-NSQIP captured 18 cases (22.2%). This variable was the most commonly misidentified. ACS-NSQIP falsely documented three cases while failing to capture three cases, giving ACS-NSQIP recognition of postoperative transfusion sensitivity and specificity of 83.3% and 95.2%, respectively. There were also inaccuracies in reporting wound infection and organ space infection. Seven cases of wound infection and five cases of organ space infection were identified by chart review. ACS-NSQIP failed to capture three cases of wound infection and two cases of organ space infection (sensitivity, 57.1% and 60.0%, respectively), while four cases of organ space infection were falsely reported (specificity, 94.7%). One case of renal failure was identified by chart review but was misclassified as renal insufficiency.  Thirty-day readmission was not introduced into the ACS-NSQIP database until 2011, thus the data for our study dates is incomplete. However, our prospectively maintained database monitored 30day readmission, and we identified 15 readmissions (18.5%) during this period. Eleven (13.6%) readmissions were unplanned related readmissions, with causes including: three episodes each of wound infection (3.7%) and pleural effusion (3.7%); two episodes each of biliary leak (2.5%) and small bowel obstruction (2.5%); and one organ space infection (1.2%).
Liver-specific complications are not captured in the "standard" ACS-NSQIP database. However, our database identified a total of 18 liver-specific complications, including six cases of biliary leak (7.4%), four cases each of pleural effusion and liver failure (4.9%), three cases of small bowel obstruction (3.7%), and one case of postoperative ascites (1.2%). Of the liver failure events, two were ISGLS grade A, one was grade B, and one was grade C. The patient with grade C liver failure experienced multiple postoperative events and was the one case of mortality in this study group. One biliary leak was incorrectly captured by ACS-NSQIP as organ-space SSI.
A summary of postoperative complication incidence, sensitivity, and specificityis presented in Table 2.

Discussion
As ACS-NSQIP use continues to expand, its limitations must be determined and addressed. We examined, through a retrospective chart review, the accuracy of liver resection data within the "standard" ACS-NSQIP database. Our results demonstrate that overall reporting is generally robust, but this database lacks liver-specific postoperative complications. The new procedure-targeted program is a positive shift in direction for ACS-NSQIP but captures only a subset of these hepatobiliary-specific events. This underscores not only the value of the program but also the possible need for a more comprehensive module.
This study is, to our knowledge, the first to compare single institution ACS-NSQIP data to that of chart review in liver resection patients. In this analysis, we found an overall complication rate of 43.2% in chart review as opposed to 29.6% by ACS-NSQIP. Our singleinstitution ACS-NSQIP complication rate was relatively consistent with recent literature on liver resections, which report morbidity rates from 27% to 56% [25,26]. A recent meta-analysis showed among studies of liver resection that definitions of overall morbidity are inconsistent and often undefined in the literature [27]. Moreover, recent data suggests that complication rates are steadily increasing due to an increased willingness to perform liver resection on higher-risk patients, a greater percentage of patients with malignancy undergoing resection, and more complex operations [26,28]. Our findings suggest that with regard to liver resection, overall complication rate has most likely been underreported by the ACS-NSQIP database.
ACS-NSQIP performs regular audits of its clinical data abstractors and has demonstrated an inter-rater reliability within 1.56% overall [2]. Although overall morbidity showed 100% specificity, our results illustrate slight discrepancies in several postoperative variables as well as larger discrepancies in postoperative transfusion, wound infection, and organ-space SSI.
Issues with capturing postoperative transfusion have been cited previously [16,25]. Changes to the ACS-NSQIP definition of bleeding complications in 2010 may have contributed to the discordance in our study group. This variable includes but does not solely identify postoperative hemorrhage, which has been shown to be a risk factor for further postsurgical morbidity and has been linked to poor longterm outcomes in patients undergoing liver resection for malignancy [29][30][31]. Distinctions must be made with this variable in order to parse out the incidence of postoperative hemorrhage versus other reasons for postoperative transfusion captured by ACS-NSQIP.
Surgical site infection (SSI) significantly increases length of stay and rate of hospital readmission, thus increasing cost of care [12,32]. SSIs accounted for a clinically relevant 36.4% of unplanned related readmissions but demonstrated discordance in classification by ACS-NSQIP. Rates of SSI are being used to measure surgical quality [33]; a study of 2007 ACS-NSQIP colon resection data demonstrated that institutions needed to meet a threshold case volume of >94 to achieve an acceptable level of reliability [34]. With this in mind, the discordance for capturing SSI in our study may be attributable to our study size being below a threshold case level. Further research may indicate a different threshold for SSI data reliability for liver resection.
Regarding issues with incorrect classification of liver resection cases, our false positive rate of 12.9% more than likely reflects input error by the clinical data abstractor. Studies have shown administrative data collection to be less accurate [22,35], but both methods of review in our present study are performed by clinical reviewers and should make them comparable [16]. Suggestions for further improvement in ACS-NSQIP fidelity could be to develop more rigorous training modules for clinical nurse reviewers and enhanced communication with surgeons in their institution. At our institution we have begun reviewing all ACS-NSQIP reported complications in our Morbidity and Mortality committee prior to submission.  Through our comparison we can attribute the majority of our higher overall morbidity rate to the inclusion of liver-specific complications. The three most common liver-specific complications we encountered were biliary leak (7.4%), liver failure (4.9%), and pleural effusion (4.9%). Liver failure is considered the most important complication after liver resection and is often reported in institutional studies [10,26,27,[36][37][38][39]. One of the four cases of liver failure was classified as grade C, and ultimately was the one case of 30-day mortality. Although one episode of biliary leak was captured by ACS-NSQIP as an organ-space SSI, the addition of a biliary leak variable further enhances distinctions within the dataset [10]. Incidence of biliary leak after liver resection has been shown to range between 3% and 12% [26,37,39,40] and is increased with the greater the complexity of resection [38]. Further diagnostic testing and intervention required often delay abdominal drain removal and hospital discharge, making it an especially long-term issue for patients having undergone resection for malignant disease [19,[41][42][43].
Pleural effusion is not captured by ACS-NSQIP, nor has it been included in the procedure-targeted option for liver resection. A multivariate analysis of 555 liver resection patients showed a pleural effusion rate of 4.7% requiring non-pharmacological intervention, citing independent risk factors such as right hepatectomy and prolonged surgery time [21]. In addition, early small bowel obstruction is related to any major open abdominal surgery, with conservative reports of incidence ranging between 0.7% to 9.5% within 30 days of operation [44]. Reoperation is required in up to 3% of patients [45]. Chart review captured three cases of small bowel obstruction requiring non-pharmacological intervention, two of which were the cause of readmission, which indicates its more severe impact on patient outcomes as well as hospital costs. Our chart review captured a single case of postoperative ascites, a notable liverspecific complication [37,46], which plays a role in heralding possible underlying portal vein thrombosis [47]. Our study results suggest that it may be useful to include pleural effusion, small bowel obstruction, and postoperative ascites in ACS-NSQIP postoperative variables for liver resection.
Readmission is a complicated and not infrequent event in hepatobiliary patients, and ACS-NSQIP instituted this variable in data collection in 2011 [2,22,25,48]. Sellers et al. demonstrated that among surgical sub-specialties in the ACS-NSQIP database, the highest readmission rate was reported for hepatobiliary surgeries (16.7%) [22]. A recent study at a major academic center observed 18.7% of liver resection patients required hospital readmission [49]. Our all-cause readmission rate (18.5%) is consistent with the literature, although we could not directly compare our data to ACS-NSQIP for this study cohort. Moreover, we observed an unplanned related readmission rate of 13.6%, which is similar to 2011 national data in ACS-NSQIP for liver resection (10.5%) [18]. Major postoperative complications play a key role in early rehospitalization [48,50]. Interestingly, in our study population liver-specific complications caused the majority (63.6%) of unplanned related readmissions. In addition to biliary leak, which is already captured in the procedure-targeted module of ACS-NSQIP, pleural effusion and small bowel obstruction contributed to five readmissions (45.5%), further underscoring the need for a more comprehensive inclusion of hepatobiliary-specific postoperative variables.
ACS-NSQIP has recognized that different operations can have specific risks and unique outcomes, and in July 2011 the first phase of a "Procedure-Targeted" program was implemented to address this issue [1,17]. In addition to capturing "standard" variables, procedurespecific variables were included for 34 high-risk, high-volume procedures. A procedure-targeted module for liver resection was initiated in July 2012, based on an earlier proposal by Pitt et al. [36]. The model includes several additional preoperative and intraoperative variables, as well as two liver-specific outcomes variables, posthepatectomy liver failure and biliary leak.
We believe this procedure-targeted program is a step in the right direction, but its scope may still be limited. Our study validates the merits of this program and suggests the adoption of additional of liver-specific complications may be warranted. Although liver failure and biliary leak are the two most common events, the addition of postoperative variables such as pleural effusion, small bowel obstruction, and ascites will further enhance distinctions of postoperative liver resection data within ACS-NSQIP. Previous studies using ACS-NSQIP data to assess liver resection outcomes lack these liver-specific variables [9,12], but their addition may provide a more accurate estimate of how complicated the postoperative course after liver resection truly is. In addition, with a comprehensive group of outcomes for liver resection ACS-NSQIP can set a standard for the definition of overall morbidity, synchronizing further research on this procedure. This study does have limitations. First, the sample size of our study is small due to utilizing single-institution data allowing small outcome rates to be more easily subjected to minor instances of missing or incorrectly captured data. Second, this study was performed at a highvolume academic medical center in which the ICD-9 coding may be muddled by the complexity of patient cases and therefore possibly less applicable to community institutions.
In conclusion, the use of large, multi-institutional databases such as ACS-NSQIP is crucial to assessing surgical outcomes and as a means of determining reimbursement. Accuracy both overall and within surgical subspecialties must be maintained, especially for complex hepatobiliary patients. We compared chart review of a prospectively maintained database to single-institution ACS-NSQIP data and observed that overall reporting of liver resection data within ACS-NSQIP is generally robust. While recent efforts to improve HPB procedure-specific tracking within ACS-NSQIP are underway, the database may benefit from further refinement with a more comprehensive set of liver-specific postoperative complication variables.