Advertisement

Measuring variation in the quality of systemic anti-cancer therapy delivery across hospitals: A national population-based evaluation

Open AccessPublished:November 29, 2022DOI:https://doi.org/10.1016/j.ejca.2022.10.017

      Highlights

      • Quality of SACT delivery was evaluated in adjuvant and metastatic colorectal cancer patients.
      • Variation in toxicity rates across 106 English NHS hospitals was used to explore this.
      • Adjusted toxicity rates varied from 11% to 49% (adjuvant) and from 25% to 67% (metastatic).
      • Twelve hospitals had toxicity rates >2SD above the national average, and 11 >2SD below.
      • Ongoing reporting of this performance indicator can be used for quality improvement.

      Abstract

      Aim

      To date, there has been little systematic assessment of the quality of care associated with systemic anti-cancer therapy (SACT) delivery across national healthcare systems. We evaluated hospital-level toxicity rates during SACT treatment as a means of identifying variation in care quality.

      Methods

      All colorectal cancer (CRC) patients receiving SACT within 106 English National Health Service (NHS) hospitals between 2016 and 2019 were included.
      Severe acute toxicity rates were derived from hospital administrative data using a validated coding framework. Variation in hospital-level toxicity rates was assessed separately in the adjuvant and metastatic settings. Toxicity rates were adjusted for age, sex, comorbidity, performance status, tumour site, and TNM staging.

      Results

      Eight thousand one hundred and seventy three patients received SACT in the adjuvant setting, and 7,683 patients in the metastatic setting. Adjusted severe acute toxicity rates varied between hospitals from 11% to 49% for the adjuvant cohort, and from 25% to 67% for the metastatic cohort.
      Compared to the national mean toxicity rate in the adjuvant cohort, six hospitals were more than two standard deviations (2SD) above, and four hospitals were more than 2SD below. In the metastatic cohort, six hospitals were more than 2SD above, and seven hospitals were more than 2SD below the national mean toxicity rate.
      Overall, 12 hospitals (12%) had toxicity rates more than 2SD above the national mean, and 11 (10%) had rates more than 2SD below.

      Conclusion

      There is substantial variation in hospital-level severe acute toxicity rates in both the adjuvant and metastatic settings, despite risk-adjustment. Ongoing reporting of this performance indicator can be used to focus further investigation of toxicity rates and stimulate quality improvement initiatives to improve care.

      Keywords

      Abbreviations:

      CAPOX (Capecitabine and oxaliplatin), CRC (Colorectal cancer), CTCAE (Common Terminology Criteria for Adverse Events), FOLFOX (5-FU and oxaliplatin), 5-FU (5-fluorouracil), HES (Hospital Episode Statistics), ICD-10 (International Classification of Diseases, 10th revision), NBOCA (National Bowel Cancer Audit), NHS (National Health Service), ONS (Office for National Statistics), OPCS-4 (Office of Population Censuses and Surveys Classification of Surgical Operations and Procedures, 4th revision), RCT (Randomised controlled trial), SACT (Systemic Anti-Cancer Therapy), 2SD (2 standard deviations), 3SD (3 standard deviations), UK (United Kingdom)

      1. Introduction

      The delivery of systemic anti-cancer therapy (SACT) is a complex care process which includes appropriate patient selection and optimisation, tailoring treatment doses, and the monitoring and management of toxicities [
      • Aggarwal A.
      • Nossiter J.
      • Parry M.
      • et al.
      Public reporting of outcomes in radiation oncology: the national prostate cancer audit.
      ]. Whilst randomised controlled trials (RCTs) have established the efficacy of SACT, there has been little or no systematic assessment of the quality of SACT delivery within routine care. Much of the available literature on the quality of SACT delivery focuses on access to treatments rather than on outcome measures [
      • Shen S.
      • Krzyzanowska M.K.
      A decade of research on the quality of systemic cancer therapy in routine care: what aspects of quality are we measuring?.
      ,
      • Mellett C.
      • O'Donovan A.
      • Hayes C.
      The development of outcome key performance indicators for systemic anti-cancer therapy using a modified Delphi method.
      ,
      • Donabedian A.
      The quality of care: how can it be assessed?.
      ].
      The only performance indicator currently reported and monitored at hospital level is 30-day mortality after the final SACT treatment, which is more a proxy for the appropriate selection of patients for SACT treatment than a measure of quality of care [
      • Wallington M.
      • Saxon E.B.
      • Bomb M.
      • et al.
      30-day mortality after systemic anticancer treatment for breast and lung cancer in England: a population-based, observational study.
      ]. Several studies have suggested that the rate of unplanned hospital admissions during SACT could be used as a potential measure of quality [

      National Institute for Health and Care Excellence. Capecitabine and oxaliplatin in the adjuvant treatment of stage III (Dukes' C) colon cancer [TA 100]. 2006. Available: https://www.nice.org.uk/guidance/ta100 [accessed 31.10.18].

      ]. A study in breast cancer patients showed a hospitalisation rate of 43% in patients during SACT with about three quarters of the admissions confirmed as SACT-related events [
      • Krzyzanowska M.K.
      • Enright K.
      • Moineddin R.
      • et al.
      Can chemotherapy-related acute care visits be accurately identified in administrative data?.
      ].
      We have previously validated an indicator of severe acute toxicity (at least Grade 3 according to the Common Terminology Criteria for Adverse Events (CTCAE)) derived from hospital administrative data in colorectal cancer (CRC) [
      • Boyle J.M.
      • Cowling T.E.
      • Kuryba A.
      • et al.
      Development and validation of a coding framework to identify severe acute toxicity from systemic anti-cancer therapy using hospital administrative data.
      ]. The indicator uses a coding framework to identify specific diagnostic codes during the timeframe of chemotherapy administration which are likely to represent SACT-related severe acute toxicity. As part of this work, we found variations in toxicity rates across different SACT regimens in line with those seen in RCTs. In addition, the rates of toxicity were associated with anticipated risk factors, for example, higher rates in those with comorbidities.
      In the current study, we evaluate the use of this measure as a national-level performance indicator to assess hospital variation in severe acute toxicity rates for CRC patients receiving SACT. The indicator will be used to identify potentially outlying hospital performance and benchmark best practice to support quality improvement processes in SACT delivery.

      2. Methods

      2.1 Data sources

      In this national population-based evaluation, we used National Bowel Cancer Audit (NBOCA) data [

      National bowel cancer audit. Available: https://www.nboca.org.uk/ [accessed 14.01.22].

      ], Hospital Episode Statistics (HES) data [,
      • Herbert A.
      • Wijlaars L.
      • Zylbersztejn A.
      • Cromwell D.
      • Hardelid P.
      Data resource profile: hospital episode statistics admitted patient care (HES APC).
      ], Systemic Anti-Cancer Therapy (SACT) data [
      • Bright C.J.
      • Lawton S.
      • Benson S.
      • et al.
      Data resource profile: the systemic anti-cancer therapy (SACT) dataset.
      ], and Office for National Statistics (ONS) mortality data [

      Office for National Statistics. Deaths. Available: https://www.ons.gov.uk/peoplepopulationandcommunity/birthsdeathsandmarriages/deaths [accessed 17.01.22].

      ] linked at patient level for CRC patients diagnosed and treated in the English National Health Service (NHS). SACT and HES data were available up until 31 March 2020.
      NBOCA is a prospective mandatory database for all newly diagnosed CRC patients in the English NHS. Data items in NBOCA were used to determine sex, age, Eastern Cooperative Oncology Group performance status [
      • Oken M.M.
      • Creech R.H.
      • Tormey D.C.
      • et al.
      Toxicity and response criteria of the Eastern cooperative oncology group.
      ], tumour site, staging (TNM), date of diagnosis, and date of surgery.
      HES is a routinely collected administrative dataset of all admissions to English NHS hospitals [
      • Herbert A.
      • Wijlaars L.
      • Zylbersztejn A.
      • Cromwell D.
      • Hardelid P.
      Data resource profile: hospital episode statistics admitted patient care (HES APC).
      ,
      • Ayanian J.Z.
      • Zaslavsky A.M.
      • Fuchs C.S.
      • et al.
      Use of adjuvant chemotherapy and radiation therapy for colorectal cancer in a population-based cohort.
      ]. Diagnoses are coded using the International Classification of Diseases, 10th revision (ICD-10) [

      NHS Digital TRUD. NHS classifications ICD-10. Available: https://isd.digital.nhs.uk/trud3/user/guest/group/0/pack/28 [accessed 22.03.22].

      ] and procedures are coded using the Office of Population Censuses and Surveys Classification of Surgical Operations and Procedures, 4th revision (OPCS-4) [
      • The Health and Social Care Information Centre
      Chemotherapy regimens clinical coding standards and guidance OPCS-4 April 2017.
      ]. Data from HES were used to identify admissions for severe acute toxicity (see below), to determine the number of comorbidities according to the RCS Charlson comorbidity score [
      • Armitage J.N.
      • van der Meulen J.H.
      Identifying co-morbidity in surgical patients using administrative data with the Royal College of Surgeons Charlson Score.
      ] and as one of the sources of SACT information (see below).
      HES was also used to determine socioeconomic status. This was derived from the Index of Multiple Deprivation (IMD) which ranks 32,482 geographical areas of England according to their level of deprivation across seven domains [
      • Camma C.
      • Giunta M.
      • Fiorica F.
      • et al.
      Preoperative radiotherapy for resectable rectal cancer: a meta-analysis.
      ]. Patients are allocated to an IMD quintile (IMDQ) based on the national ranking of the area corresponding to their postcode.
      The SACT dataset captures detailed drug-level information for SACT administered in any inpatient, day case, outpatient, or community setting, including individual administration dates [
      • Bright C.J.
      • Lawton S.
      • Benson S.
      • et al.
      Data resource profile: the systemic anti-cancer therapy (SACT) dataset.
      ]. The SACT dataset was also used to determine the NHS hospital trust (the organisational unit of NHS hospitals in England that can be located on one or more sites) that delivered SACT. Data submission is mandatory for all chemotherapy providers within the English NHS, excluding a small proportion of privately treated patients. One hundred and six hospitals were identified as delivering SACT. These hospitals needed to have treated at least ten patients during the inclusion period (see below) to be included in further analyses.

      2.2 Study population

      We defined two distinct cohorts of patients aged 18 years and above with a primary diagnosis of CRC (ICD-10: C18, C19 and C20). These included patients identified in the NBOCA database and undergoing treatments at an English NHS hospital during the inclusion period (1 April 2016 to 31 March 2019). These cohorts define the ‘denominator’ of the performance indicator.
      The first cohort included patients with pathological stage III CRC who had received adjuvant SACT according to SACT or HES within the 4-month period after major resection, as per previous work [
      • Boyle J.M.
      • Kuryba A.
      • Braun M.S.
      • et al.
      Validity of chemotherapy information derived from routinely collected healthcare data: a national cohort study of colon cancer patients.
      ]. Patients in the stage III cohort were restricted to those receiving capecitabine and oxaliplatin (CAPOX), 5-fluorouracil (5-FU) and oxaliplatin (FOLFOX), or single agent fluoropyrimidine (capecitabine or 5-FU alone), as per national guidelines [

      National Institute for Health and Care Excellence. Colorectal cancer. NICE guideline [NG151]. 2020. Available: https://www.nice.org.uk/guidance/ng151 [accessed 17.01.22].

      ].
      The second cohort included patients who were diagnosed with stage IV CRC and had commenced SACT within the 4-month period after diagnosis, according to SACT or HES. Of the 6,810 patients (89%) with SACT records and therefore drug-level information, approximately 46% of patients received an oxaliplatin-based regimen, 26% an irinotecan-based regimen, 15% single agent fluoropyrimidine, 8% irinotecan with a targeted therapy (e.g. bevacizumab, cetuximab), 5% fluoropyrimidine with a targeted therapy, less than 1% a targeted therapy alone, and less than 1% other agents (e.g. raltitrexed). For stage IV patients, SACT was restricted to treatments given continuously (gaps of no more than 8 weeks between cycles) for a maximum of 12 months.
      The pooled reporting of stage III and IV patients was deemed inappropriate given the heterogeneity of the groups in terms of disease burden, underlying fitness, differences in chemotherapy regimens used, and reported differences in overall toxicity rates reported by our prior research [
      • Boyle J.M.
      • Cowling T.E.
      • Kuryba A.
      • et al.
      Development and validation of a coding framework to identify severe acute toxicity from systemic anti-cancer therapy using hospital administrative data.
      ].

      2.3 Definition of the performance indicator

      The coding framework defined patients who had experienced severe acute toxicity (the ‘numerator’ of the performance indicator) according to the presence of pre-defined ICD-10 diagnostic codes in HES indicative of a SACT-related toxicity (supplementary Table 1) [
      • Boyle J.M.
      • Cowling T.E.
      • Kuryba A.
      • et al.
      Development and validation of a coding framework to identify severe acute toxicity from systemic anti-cancer therapy using hospital administrative data.
      ]. Severe toxicity was defined as those patients with a selected ICD-10 diagnostic code who required an overnight hospital admission between the administration of the first cycle of SACT and up until 8 weeks after the administration of the last cycle of SACT.
      For the small proportion of patients undergoing a surgical procedure during this timeframe, the date of surgery was used as the cut-off for identifying toxicities to ensure that post-operative complications were not captured.
      The indicator captures all admissions to any English NHS hospital, regardless of whether or not the hospital provides chemotherapy. Toxicities were attributed to the hospital providing the chemotherapy. Planned and unplanned admissions were included to ensure that we captured all possible SACT-related admissions (direct hospital admission, acute oncology referral, or self-referral).

      2.4 Statistical power

      We calculated the statistical power to detect an important difference (defined as a 50% increase in the toxicity rate compared to the overall national rate) in the rate of severe acute toxicity in a typical hospital and the overall national rate in England, for each cohort. We calculated the median number of patients per year receiving SACT at each hospital over the 3-year study period and used this to calculate the statistical power for reporting periods of 1-, 3- and 5-years. The definition of an important difference is arbitrary but was chosen because it has been used in previous work and represents a substantial absolute increase in the rate of toxicity [
      • Walker K.
      • Neuburger J.
      • Groene O.
      • Cromwell D.A.
      • van der Meulen J.
      Public reporting of surgeon outcomes: low numbers of procedures lead to false complacency.
      ]. A 5% significance level was used for testing differences between the hospital-level and national rates because it corresponds to the commonly used 95%-control limits of funnel plots (see below).

      2.5 Fairness

      We determined to what extent we could adjust for the case-mix factors that are likely to affect the risk of severe acute toxicity based on a combination of review of existing literature and expert clinical input. Only factors outside the control of the healthcare provider were adjusted for. We assessed the data completeness of these factors. Missing values for case-mix factors were imputed with multiple imputation [
      • White I.R.
      • Royston P.
      • Wood A.M.
      Multiple imputation using chained equations: issues and guidance for practice.
      ]. We carried out indirect risk-adjustment, using multivariable logistic regression modelling to obtain expected numbers of severe acute toxicity events per hospital [
      • Spiegelhalter D.J.
      Funnel plots for comparing institutional performance.
      ]. The association between patient and clinical characteristics and toxicity were assessed separately for each cohort using Wald tests to calculate p values with the significance level set at 0.05.
      Calibration of the risk-adjustment model across deciles of predicted risk was assessed using the Hosmer–Lemeshow test, constructing an F-statistic to carry out the test in multiply imputed data [
      • Spiegelhalter D.J.
      Funnel plots for comparing institutional performance.
      ]. The C-statistic was used to assess model discrimination, combining estimates across imputed datasets using Rubin's rules [
      • White I.R.
      • Royston P.
      • Wood A.M.
      Multiple imputation using chained equations: issues and guidance for practice.
      ,
      • Marshall A.
      • Altman D.G.
      • Holder R.L.
      • Royston P.
      Combining estimates of interest in prognostic modelling studies after multiple imputation: current practice and guidelines.
      ].

      2.6 Variation between hospitals

      Within our cohort of 106 hospitals, we used funnel plots to identify outlying hospitals defined as those with results more than two standard deviations (2SD) (corresponding to 95%-control, or inner, funnel limits), or three standard deviations (3SD) (corresponding to 99.8%-control, or outer, funnel limits) below or above the overall national rate. This is equivalent to carrying out statistical tests comparing a specific hospital's result with the overall national rate using a two-sided 5% or 0.02% significance level, respectively [
      • Spiegelhalter D.J.
      Funnel plots for comparing institutional performance.
      ,
      • Keating N.L.
      • Cleveland J.L.F.
      • Wright A.A.
      • et al.
      Evaluation of reliability and correlations of quality measures in cancer care.
      ,
      • Grote H.
      • Toma K.
      • Crosby L.
      • et al.
      Outliers from national audits: their analysis and use by the care quality commission in quality assurance and regulation of healthcare services in England.
      ]. Fully-adjusted funnel plots were generated for each cohort.

      3. Results

      3.1 Study cohorts

      Between 1 April 2016 and 31 March 2019, 8,173 patients received adjuvant SACT for stage III CRC. Of these 8,173 patients, 2,074 (25%) had a severe acute toxicity identified according to the indicator. In addition, 7,683 patients received SACT within 4 months of a diagnosis of stage IV CRC. Of these 7,683 patients, 3,625 (47%) had a severe acute toxicity identified. Table 1 summarises the different types of toxicity identified from the coding framework, according to organ system.
      Table 1Distribution of types of toxicities following SACT administration for stage III and IV cohorts [
      • Gollins S.
      • Moran B.
      • Adams R.
      • et al.
      Association of Coloproctology of Great Britain & Ireland (ACPGBI): guidelines for the management of cancer of the colon, rectum and anus (2017) – multidisciplinary management.
      ,

      National Cancer Institute. Common terminology criteria for adverse events (CTCAE). Available: https://ctep.cancer.gov/protocoldevelopment/electronic_applications/docs/CTCAE_v5_Quick_Reference_5x7.pdf [accessed 22.03.22].

      ].
      Toxicity typeStage III (n = 8173)

      (%)
      Stage IV (n = 7683)

      (%)
      Gastrointestinal13.223.1
      Infection10.424.2
      Cardiovascular6.214.1
      Metabolic & endocrine5.210.4
      Constitutional5.010.0
      Renal4.99.1
      Haematology4.112.0
      Pain3.76.5
      Neurological2.63.9
      Neutropenic sepsis2.47.6
      Respiratory1.21.6
      Line complications1.23.5
      Bleeding1.13.0
      Dermatology & rheumatology0.82.3

      3.2 Statistical power

      For the stage III cohort, 97 out of 106 English NHS hospitals had treated more than ten patients over the 3-year inclusion period and were included in further analyses. The annual volumes of patients who received adjuvant SACT in each hospital varied considerably with a median value of 24 (range 5–132, interquartile range 15–33). Similarly, 98 hospitals were included for the stage IV cohort, with a median annual volume of 22 (range 5–142, interquartile range 13–32).
      The statistical power to detect an increase of 50% compared to the overall national mean rate for different reporting periods (1-, 3-, and 5-year) are presented in supplementary Table 2. These power calculations demonstrate that a 3-year reporting period achieves approximately 70% power in the stage III and 99% power in the stage IV cohort to detect a 50% increase compared to the overall national rate. A 1-year reporting period could have been chosen for the stage IV cohort but, for consistency, the same reporting period was used for both.

      3.3 Fairness

      For risk-adjustment, the following case-mix factors were identified within the literature and from clinical expertise: age, sex, number of comorbidities, performance status, tumour site, staging, and socioeconomic status [
      • Wallington M.
      • Saxon E.B.
      • Bomb M.
      • et al.
      30-day mortality after systemic anticancer treatment for breast and lung cancer in England: a population-based, observational study.
      ,
      • Wilson B.E.
      • Jacob S.
      • Yap M.L.
      • et al.
      Estimates of global chemotherapy demands and corresponding physician workforce requirements for 2018 and 2040: a population-based study.
      ]. All of these case-mix factors are typically accessible from routinely collected datasets. We found that their completeness rate is high in the English NHS (Table 2a, Table 2b).
      Table 2aPatient and tumour characteristics and associated severe acute toxicity for patients with stage III disease.
      Stage III (adjuvant) cohort (n = 8173)
      Number (%)Severe acute toxicity (%)Unadjusted odds ratioAdjusted odds ratio
      Adjustment for all other variables in the table.
      (95% CI)
      p value
      Wald test from multivariable model.
      Patient characteristics
      Age (years)0.010
       <602369 (29)566 (24)1.01.0
       60–692720 (33)692 (25)1.091.09 (0.96–1.24)
       70–792631 (32)721 (27)1.201.14 (1.00–1.31)
       ≥80453 (6)95 (21)0.850.78 (0.61–1.01)
      Sex<0.001
       Male4647 (57)1072 (23)1.01.0
       Female3526 (43)1002 (28)1.321.35 (1.22–1.49)
      Socioeconomic status (IMDQ)0.969
       1 (most deprived)1159 (14)293 (25)1.01.0
       21460 (18)374 (26)1.021.02 (0.85–1.22)
       31691 (21)427 (25)1.001.02 (0.85–1.21)
       41958 (24)489 (25)0.981.01 (0.85–1.20)
       5 (least deprived)1896 (23)489 (26)1.031.05 (0.89–1.25)
      Missing9 (0.1)
      RCS Charlson score<0.001
       04985 (61)1181 (24)1.01.0
       12377 (29)640 (27)1.191.19 (1.07–1.34)
       ≥2811 (10)253 (31)1.461.48 (1.25–1.75)
      Performance status0.546
       04925 (67)1217 (25)1.01.0
       11946 (27)526 (27)1.121.06 (0.94–1.20)
       ≥2472 (6)130 (28)1.151.08 (0.87–1.36)
      Missing830 (10)
      Tumour characteristics
      Site0.004
       Colon6147 (75)1540 (25)1.01.0
       Rectosigmoid524 (6)139 (27)1.081.17 (0.95–1.44)
       Rectum1502 (18)395 (26)1.071.25 (1.09–1.44)
      Pathological T-stage<0.001
       T1215 (3)50 (23)1.01.0
       T2760 (9)168 (22)0.940.91 (0.63–1.30)
       T34565 (56)1083 (24)1.031.00 (0.72–1.39)
       T42631 (32)773 (29)1.371.34 (0.96–1.88)
      Missing2 (<0.1)
      Pathological N-stage0.002
       N15338 (65)1280 (24)1.01.0
       N22835 (35)794 (28)1.231.18 (1.06–1.31)
      a Adjustment for all other variables in the table.
      b Wald test from multivariable model.
      Table 2bPatient and tumour characteristics and associated severe acute toxicity for patients with stage IV disease.
      Stage IV (metastatic) cohort (n = 7683)
      Number (%)Severe acute toxicity (%)Unadjusted odds ratioAdjusted odds ratio
      Adjustment for all other variables in the table.
      (95% CI)
      p value
      Wald test from multivariable model.
      Patient characteristics
      Age (years)0.026
       <602630 (34)1249 (47)1.01.0
       60–692331 (30)1073 (46)0.940.92 (0.82–1.03)
       70–792189 (28)1071 (49)1.060.96 (0.85–1.09)
       ≥80533 (7)232 (44)0.850.74 (0.61–0.90)
      Sex0.017
       Male4640 (60)2111 (46)1.01.0
       Female3043 (40)1514 (50)1.191.12 (1.02–1.23)
      Socioeconomic status (IMDQ)
       1 (most deprived)1206 (16)615 (51)1.01.00.069
       21399 (18)683 (49)0.920.93 (0.80–1.09)
       31602 (21)727 (45)0.800.82 (0.71–0.96)
       41732 (23)803 (46)0.830.86 (0.74–0.96)
       5 (least deprived)1732 (23)793 (46)0.810.83 (0.72–0.97)
      Missing12 (0.2)
      RCS Charlson score
       04674 (62)2146 (46)1.01.00.021
       12076 (28)998 (48)1.091.07 (0.97–1.19)
       ≥2759 (10)395 (52)1.291.24 (1.06–1.45)
      Missing174 (2)
      Performance status
       03730 (54)1616 (43)1.01.0<0.001
       12255 (33)1135 (50)1.321.31 (1.18–1.46)
       ≥2902 (13)500 (55)1.651.61 (1.38–1.87)
      Missing796 (10)
      Tumour characteristics
      Site<0.001
       Colon4929 (64)2505 (51)1.01.0
       Rectosigmoid510 (7)235 (46)0.830.83 (0.69–1.00)
       Rectum2244 (29)885 (39)0.630.65 (0.58–0.72)
      Pre-treatment T-stage0.004
       T114 (0.2)6 (43)1.01.0
       T2369 (6)157 (43)1.040.96 (0.34–2.74)
       T33619 (54)1566 (43)1.070.97 (0.34–2.72)
       T42699 (40)1372 (51)1.431.17 (0.42–3.31)
      Missing982 (13)
      Pre-treatment N-stage0.001
       N01186 (18)516 (44)1.01.0
       N12935 (44)1319 (45)1.071.09 (0.96–1.25)
       N22610 (39)1274 (49)1.261.30 (1.12–1.50)
      Missing952 (12)
      a Adjustment for all other variables in the table.
      b Wald test from multivariable model.
      Table 2a, Table 2b summarise for the stage III and stage IV cohorts the results of the logistic regression models that capture the associations between the case-mix factors and risk of severe acute toxicity. In both cohorts, severe acute toxicity was increased for female sex, those with more than 2 comorbidities, and advanced T- and N-stage disease. In the stage III cohort, rectal cancer was associated with increased toxicity whereas in the stage IV cohort it was associated with reduced toxicity. In the stage IV cohort, rectal cancer patients were significantly younger and fitter (according to performance status) than those patients with colon cancer. In the stage IV cohort, poor performance status was associated with increased toxicity but this association was not statistically significant in the stage III cohort.
      Following this analysis, we included age, sex, RCS Charlson comorbidity score, performance status, tumour site, T-stage, and N-stage in the logistic regression models used to adjust for case-mix factors when assessing between-hospital variation in rates of severe acute toxicity. Due to the debate around its appropriateness in case-mix adjustment, and the fact it was not associated with increased toxicity, socioeconomic status was included in the model as a sensitivity analysis.
      There was no evidence of a lack of calibration for either the stage III cohort (p = 0.711) or the stage IV cohort (p = 0.952), according to the Hosmer–Lemeshow test. The C-statistic of discrimination was 0.58 (95% CI: 0.57 to 0.59) for the stage III cohort and 0.64 (95% CI: 0.62 to 0.66) for the stage IV cohort.

      3.4 Variation between hospitals

      The unadjusted rates of severe acute toxicity after adjuvant SACT in stage III CRC patients varied considerably between the 97 included hospitals, ranging from 11% to 47% with ten hospitals outside the 95%-funnel limits, including one outside the 99.8%-funnel limits (Fig. 1A). Adjusting for case-mix factors had little effect on the variation in severe acute toxicity rates. Adjusted severe acute toxicity rates ranged from 11% to 49% with the same outlying hospitals (Fig. 1B). This corresponded to one hospital being 3SD above, five hospitals being 2SD above, and four hospitals being 2SD below, the national mean toxicity rate. A sensitivity analysis including socioeconomic status in the risk-adjustment did not change the outlying hospitals (results not presented).
      Fig. 1
      Fig. 1Funnel plot showing a) unadjusted and b) adjusted rates of severe acute toxicity by English NHS hospital for patients receiving SACT for stage III colorectal cancer.
      The unadjusted rates of severe acute toxicity after SACT for stage IV CRC also varied considerably between the included 98 English NHS hospitals, ranging from 26% to 65% with 12 hospitals outside the 95%-funnel limits (Fig. 2A). Adjusting for case-mix factors had little effect on the variation in severe acute toxicity rates (25%–67%) and outlying hospitals, with 13 hospitals outside the 95%-funnel limits (Figure 2B). This corresponded to six hospitals being 2SD above, and seven hospitals being 2SD below, the national mean toxicity rate. A sensitivity analysis including socioeconomic status in the risk-adjustment generated two new outlying hospitals (results not presented).
      Fig. 2
      Fig. 2Funnel plots showing a) unadjusted and b) adjusted rates of severe acute toxicity by English NHS hospital for patients receiving SACT for stage IV colorectal cancer.
      Across both cohorts, 22 different hospitals were identified as having rates of severe acute toxicity more than 2SD from the national mean toxicity rate (only 1 hospital had rates more than 2SD for both cohorts). The Pearson correlation coefficient comparing the adjuvant and metastatic rates of toxicity for each hospital was 0.2 (p = 0.090).

      4. Discussion

      This study demonstrates how diagnostic coding in hospital administrative data can be used to derive a hospital-level performance indicator of SACT toxicity across hospitals treating CRC patients. We used a previously validated coding framework based on a pre-defined set of specific ICD-10 codes and found considerable variation in severe acute toxicity rates between hospitals in both the adjuvant and metastatic setting. We found that hospital rates of severe acute toxicity requiring an overnight hospital admission (equivalent to at least grade 3 CTCAE) varied between 11% and 49% in stage III patients receiving adjuvant chemotherapy, and were even higher for stage IV patients with rates varying between 25% and 67%. We identified 22 potentially outlying hospitals, even after adjustment for important case-mix factors.
      This hospital performance indicator will be used as part of a publicly reported outlier program in the United Kingdom (UK) from 2022. Within the national outlier process, hospitals are grouped as ‘alerts’ (greater or less than 2SD above the national mean) or ‘alarms’ (greater or less than 3SD above the national mean) [

      Outlier Management for National Clinical Audits. Healthcare quality improvement partnership. Available: https://www.hqip.org.uk/wp-content/uploads/2021/11/Appendix-10-HQIP-Outlier-guidance-v4.pdf [accessed 22.03.22].

      ]. ‘Alarm’ hospitals are contacted to acknowledge the potential outlier status and start by corroborating data completeness and quality [
      • Shahian D.M.
      • Normand S.L.
      What is a performance outlier?.
      ]. Once the data is verified, hospitals are expected to formulate a formal response and action plan to understand which factors might be driving unwarranted variation (Fig. 3) [
      • Enright K.A.
      • Taback N.
      • Powis M.L.
      • et al.
      Setting quality improvement priorities for women receiving systemic therapy for early-stage breast cancer by using population-level administrative data.
      ,
      • Grote H.
      • Toma K.
      • Crosby L.
      • et al.
      Outliers from national audits: their analysis and use by the care quality commission in quality assurance and regulation of healthcare services in England.
      ,
      • Kamal A.H.
      • Power S.
      • Patierno S.R.
      Addressing issues of cancer disparities, equity, and inclusion through systemized quality improvement.
      ]. ‘Alert’ hospitals are monitored and become potential outliers if they are identified as such in consecutive years. The outlier process is entirely publicly reported. Further work to establish best practice is required and might involve, for example, evaluating survival outcomes in outlying hospitals to evaluate the impact of differential toxicity rates.
      Fig. 3
      Fig. 3Quality improvement conceptual framework highlighting potential areas within the SACT care pathway that may represent sources of variation in care.
      These results show that this performance indicator can be used to further explore unwarranted variation in toxicity rates, as well as triggering and guiding initiatives to improve the quality of SACT delivery on a national scale. As the performance indicator is derived from administrative diagnostic coding, the risk of information bias or clinical data manipulation is reduced. The linked datasets included patient and tumour characteristics that allowed the development of a case-mix adjustment model with good calibration and adequate discriminatory power.
      The use of ICD-10 codes in the coding framework makes it internationally applicable as it can be applied in different health systems that use ICD-10 codes within their hospital administrative data. In addition, whilst this study has focused on CRC, the coding framework can be applied across different tumour types and regimens, including targeted therapies and immunotherapy.

      4.1 Differences in hospital-level severe acute SACT toxicity

      The higher rates of toxicity in patients with advanced disease has been previously observed [
      • Du X.L.
      • Osborne C.
      • Goodwin J.S.
      Population-based assessment of hospitalizations for toxicity from chemotherapy in older women with breast cancer.
      ]. This is likely due to differences in baseline characteristics, for example, poorer performance status in those with stage IV disease, and the wider range of SACT drugs used within this cohort. Fig. 3 also considers the points along the SACT care pathway that may contribute to the between-hospital variation in toxicity rates.
      Outlying rates of severe acute toxicity may reflect differences in the assessment and selection of patients regarding fitness or appropriateness of treatment. For example, patients with a high risk of severe acute toxicity might be identified through comprehensive risk-stratification and discussion within the multidisciplinary team, and there may be differential access to specialist geriatrician and prehabilitation services [
      • Silver J.K.
      • Baima J.
      Cancer prehabilitation: an opportunity to decrease treatment-related morbidity, increase cancer treatment options, and improve physical and psychological health outcomes.
      ,
      • Kalsi T.
      • Babic-Illman G.
      • Ross P.J.
      • et al.
      The impact of comprehensive geriatric assessment interventions on tolerance to chemotherapy in older people.
      ,
      • Wildiers H.
      • Heeren P.
      • Puts M.
      • et al.
      International Society of Geriatric Oncology consensus on geriatric assessment in older patients with cancer.
      ]. In addition, patient choice is an important and complex aspect of the decision-making process to consider, with both patient-related factors (e.g. social, cognitive, and psychological issues) and clinician-related factors (e.g. communication and clinician biases) influencing whether SACT is given, as well as what choice of SACT. Patients should be appropriately counselled and consented for SACT treatments by a sufficiently experienced clinician, including the potential short- and long-term side-effects.
      Furthermore, unwarranted variation in rates of toxicity may represent under- or over-treatment of patients, as well as inappropriate regimen use and dosing, inadequate or outdated protocols, insufficient monitoring, or failure to recognise and address early signs of toxicity. Despite clear national guidelines in the UK regarding which SACT drugs are approved for which indication [

      National Institute for Health and Care Excellence. Capecitabine and oxaliplatin in the adjuvant treatment of stage III (Dukes' C) colon cancer [TA 100]. 2006. Available: https://www.nice.org.uk/guidance/ta100 [accessed 31.10.18].

      ,

      National Institute for Health and Care Excellence. Cetuximab and panitumumab for previously untreated metastatic colorectal cancer. [TA439]. 2017. Available: https://www.nice.org.uk/guidance/ta439/chapter/1-Recommendations [accessed 06.10.22].

      ,

      National Institute for Health and Care Excellence. Trifluridine-tipiracil for previously treated metastatic colorectal cancer. [TA405]. 2016. Available: https://www.nice.org.uk/guidance/ta405 [accessed].

      ,

      National Institute for Health and Care Excellence. Aflibercept in combination with irinotecan and fluorouracil-based therapy for treating metastatic colorectal cancer that has progressed following prior oxaliplatin-based chemotherapy. [TA307]. 2014. Available: https://www.nice.org.uk/guidance/ta307 [accessed 06.10.22].

      ,

      National Institute for Health and Care Excellence. Guidance on the use of capecitabine and tegafur with uracil for metastatic colorectal cancer. [TA61]. 2003. Available: https://www.nice.org.uk/guidance/ta61 [accessed 06.10.22].

      ,

      National Institute for Health and Care Excellence. Cetuximab, bevacizumab and panitumumab for the treatment of metastatic colorectal cancer after first-line chemotherapy: cetuximab (monotherapy or combination chemotherapy), bevacizumab (in combination with non-oxaliplatin chemotherapy) and panitumumab (monotherapy) for the treatment of metastatic colorectal cancer after first-line chemotherapy. [TA242]. 2012. Available: https://www.nice.org.uk/guidance/ta242/chapter/1-Guidance [accessed 06.10.22].

      ], we recognise that different providers may have variability in prescribing practices beyond this. For example, even current national guidelines do not offer guidance between the use of FOLFOX, CAPOX, or single agent fluoropyrimidine [

      National Institute for Health and Care Excellence. Colorectal cancer. NICE guideline [NG151]. 2020. Available: https://www.nice.org.uk/guidance/ng151 [accessed 17.01.22].

      ].
      In addition, the IDEA collaborative results were published during the study period and surveys have highlighted ongoing variation in clinical practice with regards to the choice and duration of combination adjuvant chemotherapy with shifts towards 3 months of treatment for low-risk disease, particularly in the UK [
      • Iveson T.
      • Hanna C.
      • Iveson P.
      • et al.
      The early impact of the IDEA collaboration results: how the results changed prescribing practice.
      ,
      • Hanna C.
      • Boyd K.
      • Jones R.
      P-335 Self-reported prescribing practices in the setting of adjuvant treatment for colorectal cancer.
      ]. For the metastatic cohort, there may be variation in which providers participate in clinical trials. However, adjusting for choice or duration of regimen which are under the control of the provider was deemed inappropriate given that this could represent poor clinical practice.
      Finally, the available infrastructure and clinical pathways within hospitals may play a role, for example, variability in access to acute oncology services, emergency services, and the availability of specialist on-site advice out of hours, as well as more generalised disparities in clinical expertise, availability, and training [
      • Enright K.A.
      • Krzyzanowska M.K.
      Benefits and pitfalls of using administrative data to study hospitalization patterns in patients with cancer treated with chemotherapy.
      ]. For example, acute oncology services have been shown to improve outcomes, although substantial variation remains in whether hospitals have access to these services [
      • Navani V.
      How has acute oncology improved care for patients?.
      ]. Acute oncology services may increase planned admissions. Although the vast majority (91%) of admissions for toxicity included in this study were unplanned, this would be an important area to further investigate for potential outliers.
      Only one hospital was identified as a low outlier for both cohorts of patients. When comparing both cohorts, there was evidence of a weak association between the rates of toxicity in the adjuvant and metastatic cohorts within each hospital. However, we would not necessarily expect these to align as the two cohorts are very different as evidenced by their mean toxicity rates. First, this may reflect the reduced fitness of patients in the stage IV cohort, and more complex care pathways they undertake, which mean that potential poor treatment, patient selection, and deficient supportive care, are more likely to be exposed. Second, there are differences in care pathways between the two cohorts which might account for differences in toxicity rates within the same provider. For example, the administration and monitoring of adjuvant chemotherapy may be nurse-led in contrast to chemotherapy given for metastatic disease. Third, patient-clinician communication, decision-making, and patient choice are likely to be more complex and nuanced within the metastatic setting which may also influence toxicity rates.

      4.2 Strengths and limitations

      The strengths of this study include ensuring that the performance indicator meets a pre-defined set of essential criteria (‘validity’, ‘statistical power’, and ‘fairness’ as detailed in the Methods) [
      • Geary R.
      • Knight H.
      • Carroll F.
      • et al.
      A step-wise approach to developing indicators to compare the performance of maternity units using hospital administrative data.
      ]. The validity of the indicator was demonstrated in an earlier study comparing toxicity across SACT regimens [
      • Boyle J.M.
      • Cowling T.E.
      • Kuryba A.
      • et al.
      Development and validation of a coding framework to identify severe acute toxicity from systemic anti-cancer therapy using hospital administrative data.
      ]. Although within this study we have reported the overall incidence of severe toxicity after SACT at each hospital, we are also able to detail specific individual toxicities according using our indicator (e.g. neutropenic sepsis, diarrhoea, and line complications – see Table 1). This is hugely important for providing detailed feedback to hospitals to facilitate quality improvement.
      In addition, routinely available national clinical cancer data linked to SACT data and hospital administrative data provided over 95% case ascertainment across all English NHS hospitals with good recording of comorbidities, performance status, staging, and detailed SACT information. This also allowed the capture of all hospital admissions, regardless of whether the hospital provided chemotherapy, and we assigned the toxicity to the hospital delivering the chemotherapy [
      • Neuss M.
      • Rocque G.
      • Zuckerman D.
      • et al.
      Establishing a core set of performance measures to improve value in cancer care: ASCO consensus conference recommendation report.
      ].
      This, and the good calibration of the risk-adjustment model, meant that toxicity rates between hospitals could be adjusted for important case-mix factors known to influence toxicity, enhancing the fairness of hospital-specific reporting [
      • Hurria A.
      • Togawa K.
      • Mohile S.G.
      • et al.
      Predicting chemotherapy toxicity in older adults with cancer: a prospective multicenter study.
      ]. There is a debate surrounding the complexities of the inclusion of deprivation in risk-adjustment, with varying practice between different reporting programmes [
      • Tran L.D.
      Social risk adjustment in health care performance measures.
      ]. A sensitivity analysis showed minor changes in results for the stage IV cohort which need to be considered. Finally, an overview of potentially actionable areas has been identified as a starting point for targeted local quality improvement (Fig. 3).
      The first limitation of this study is the reliance on ICD-10 diagnostic codes in hospital administrative data. However, these diagnostic codes in HES have been shown to be accurate compared to clinical notes, thereby supporting its use for healthcare performance assessment and research [
      • Burns E.M.
      • Rigby E.
      • Mamidanna R.
      • et al.
      Systematic review of discharge coding accuracy.
      ]. In addition, we used two independent data sources to capture information about the use of SACT. Our previous validation work has shown excellent agreement between the two data sources (SACT and HES data) [
      • Boyle J.M.
      • Kuryba A.
      • Braun M.S.
      • et al.
      Validity of chemotherapy information derived from routinely collected healthcare data: a national cohort study of colon cancer patients.
      ]. Second, there is a possibility that some of the variation between hospitals is due to chance alone. However, we would only expect five hospitals to lie more than two standard deviations from the national mean by chance.
      Third, the coding framework is best suited for studying differences between groups of patients (e.g. those treated in different hospitals or receiving different SACT regimens) rather than estimating absolute rates of toxicity which may be over-estimated. However, to limit the likelihood of overestimation we restricted the indicator to only those diagnoses likely to reflect SACT toxicity and to the precise time period of chemotherapy administration (excluding any post-operative period in a small proportion of patients). In addition, previous studies have suggested the vast majority of hospital admissions during SACT treatment are SACT-related [
      • Krzyzanowska M.K.
      • Enright K.
      • Moineddin R.
      • et al.
      Can chemotherapy-related acute care visits be accurately identified in administrative data?.
      ].
      Finally, other studies have demonstrated that mental health status, nutritional status, and laboratory values (e.g. blood tests) were also important predictors for SACT toxicity in older patients [
      • Hurria A.
      • Togawa K.
      • Mohile S.G.
      • et al.
      Predicting chemotherapy toxicity in older adults with cancer: a prospective multicenter study.
      ,
      • Extermann M.
      • Boler I.
      • Reich R.R.
      • et al.
      Predicting the risk of chemotherapy toxicity in older patients: the chemotherapy risk assessment scale for high-age patients (CRASH) score.
      ]. However, this information is not routinely included in hospital administrative data and so could not be included as part of the risk-adjustment. As a result, a certain level of ‘residual confounding’ will need to be accepted, irrespective of which patient groups are being compared.

      4.3 Implications

      There are several implications of using this performance indicator for a national assessment of the delivery of SACT. First, our study shows that the performance indicator can be used to compare SACT toxicity between hospitals [
      • Grote H.
      • Toma K.
      • Crosby L.
      • et al.
      Outliers from national audits: their analysis and use by the care quality commission in quality assurance and regulation of healthcare services in England.
      ]. In the English NHS, similar hospital-level reporting is already routinely available for patients who had surgery or radiotherapy for prostate, bowel, and oesophageal cancer as part of a programme of national clinical audits [

      National bowel cancer audit. Available: https://www.nboca.org.uk/ [accessed 14.01.22].

      ,
      • Gollins S.
      • Moran B.
      • Adams R.
      • et al.
      Association of Coloproctology of Great Britain & Ireland (ACPGBI): guidelines for the management of cancer of the colon, rectum and anus (2017) – multidisciplinary management.
      ,
      • Leong K.
      • Hartley J.
      • Karandikar S.
      Association of Coloproctology of Great Britain & Ireland (ACPGBI): guidelines for the management of cancer of the colon, rectum and anus (2017) – follow up, lifestyle and survivorship.
      ]. Outcome reporting programmes are also being established internationally [
      • Rechel B.
      • McKee M.
      • Haas M.
      • et al.
      Public reporting on quality, waiting times and patient experience in 11 high-income countries.
      ].
      Second, the performance indicator allows ongoing monitoring of severe acute toxicity events within hospitals which will inform continuous local quality improvement processes. Evidence has shown that quality improvement initiatives are more likely to produce positive effects if continuous monitoring and feedback is undertaken [
      • Grote H.
      • Toma K.
      • Crosby L.
      • et al.
      Outliers from national audits: their analysis and use by the care quality commission in quality assurance and regulation of healthcare services in England.
      ,
      • Wilson B.E.
      • Jacob S.
      • Yap M.L.
      • et al.
      Estimates of global chemotherapy demands and corresponding physician workforce requirements for 2018 and 2040: a population-based study.
      ]. As per previous work, the indicator allows specific individual toxicities to be described in detail which can further inform quality improvement processes [
      • Boyle J.M.
      • Cowling T.E.
      • Kuryba A.
      • et al.
      Development and validation of a coding framework to identify severe acute toxicity from systemic anti-cancer therapy using hospital administrative data.
      ].
      Third, an improved understanding of the risks of SACT will inform the counselling of patients and strengthen the process of ‘shared decision making’ in day-to-day practice, particularly for novel SACT drugs.
      In addition to supporting direct patient care, public reporting of severe acute toxicity rates can also provide transparency around best practice through benchmarking to guide patients in making informed choices about the hospital in which they will receive their SACT treatment [
      • Marshall M.N.
      • Shekelle P.G.
      • Leatherman S.
      • Brook R.H.
      The public release of performance data: what do we expect to gain? A review of the evidence.
      ,
      • Spinks T.E.
      • Walters R.
      • Feeley T.W.
      • et al.
      Improving cancer care through public reporting of meaningful quality measures.
      ]. This avoids the reliance on surrogate markers of care quality (e.g. presence of robotic surgery), and further stimulates quality improvement through competitive mechanisms or regulation by reducing information asymmetry regarding care quality [
      • Aggarwal A.
      • Lewis D.
      • Mason M.
      • et al.
      Effect of patient choice and hospital competition on service configuration and technology adoption within cancer surgery: a national, population-based study.
      ,
      • Berwick D.M.
      • James B.
      • Coye M.J.
      Connections between quality measurement and improvement.
      ,
      • Hibbard J.H.
      • Stockard J.
      • Tusler M.
      Does publicizing hospital performance stimulate quality improvement efforts?.
      ]. This transparency can also guide investments, with outcomes considered as part of pay-for-performance schemes in order to support greater value in care delivery [
      • Scott A.
      • Liu M.
      • Yong J.
      Financial incentives to encourage value-based health care.
      ].
      Finally, the coding framework developed to identify severe acute toxicity was designed to be broad in order to make it applicable to all types of SACT, including traditional cytotoxic chemotherapy, immunotherapy, and targeted agents [
      • Boyle J.M.
      • Cowling T.E.
      • Kuryba A.
      • et al.
      Development and validation of a coding framework to identify severe acute toxicity from systemic anti-cancer therapy using hospital administrative data.
      ]. This means that the performance indicator can be used in a wide range of clinical settings and expanded across most cancer types, following appropriate validation.

      5. Conclusion

      We have evaluated the use of a national performance indicator derived from linked clinical and hospital administrative datasets to assess hospital variation in severe acute toxicity rates for hospitals providing SACT for CRC patients in order to stimulate and support quality improvement. This approach can be applied across different cancer types and in many different countries where similar regional or national clinical and administrative hospital datasets are available.

      Funding

      This study was undertaken alongside the National Bowel Cancer Audit. The audit is commissioned by the Healthcare Quality Improvement Partnership (HQIP) as part of the National Clinical Audit and Patient Outcomes Programme, and funded by NHS England and the Welsh Government (https://www.hqip.org.uk/national-programmes). AA was supported by a National Institute for Health Research (NIHR) Advanced Fellowship [NIHR300599] and TEC by a fellowship from the Medical Research Council [MR/S020470/1]. Neither HQIP or any of the funders had any involvement in the study design; in the collection, analysis and interpretation of data; in the writing of the report; or in the decision to submit the article for publication. The researchers had full independence from HQIP.

      Author contributions

      Jemma M. Boyle – conceptualization, formal analysis, investigation, data curation, writing – original draft, writing – review and editing, visualisation.
      Angela Kuryba – writing – review and editing.
      Thomas E. Cowling – formal analysis, writing – review and editing, visualisation.
      Jan van der Meulen – writing – review and editing, supervision.
      Nicola S. Fearnhead – writing – review and editing.
      Kate Walker – conceptualisation, methodology, writing – review and editing, supervision.
      Michael S. Braun – conceptualisation, methodology, writing – review and editing, supervision.
      Ajay Aggarwal – conceptualisation, methodology, writing – review and editing, supervision.
      Chris Booth – writing – review and editing.

      Conflict of interest statement

      The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

      Acknowledgements

      This study was based on data collected by the National Bowel Cancer Audit linked to the Systemic Anti-cancer Therapy database (https://www.chemodataset.nhs.uk/home) made available by the National Cancer Registration and Analysis Service and Hospital Episode Statistics Admitted Patient Care made available by NHS Digital (https://digital.nhs.uk/data-and-information/data-tools-and-services/data-services/hospital-episode-statistics).
      This work uses data provided by patients and collected by the NHS as part of their care and support.

      Appendix A. Supplementary data

      The following is the Supplementary data to this article:

      References

        • Aggarwal A.
        • Nossiter J.
        • Parry M.
        • et al.
        Public reporting of outcomes in radiation oncology: the national prostate cancer audit.
        Lancet Oncol. 2021; 22: e207-e215
        • Shen S.
        • Krzyzanowska M.K.
        A decade of research on the quality of systemic cancer therapy in routine care: what aspects of quality are we measuring?.
        J Oncol Pract. 2015; 11: 55-61
        • Mellett C.
        • O'Donovan A.
        • Hayes C.
        The development of outcome key performance indicators for systemic anti-cancer therapy using a modified Delphi method.
        Eur J Cancer Care. 2020; 29e13240
        • Donabedian A.
        The quality of care: how can it be assessed?.
        JAMA. 1988; 260: 1743-1748
        • Wallington M.
        • Saxon E.B.
        • Bomb M.
        • et al.
        30-day mortality after systemic anticancer treatment for breast and lung cancer in England: a population-based, observational study.
        Lancet Oncol. 2016; 17: 1203-1216
        • Enright K.A.
        • Taback N.
        • Powis M.L.
        • et al.
        Setting quality improvement priorities for women receiving systemic therapy for early-stage breast cancer by using population-level administrative data.
        J Clin Oncol. 2017; 35: 3207-3214
        • Krzyzanowska M.K.
        • Enright K.
        • Moineddin R.
        • et al.
        Can chemotherapy-related acute care visits be accurately identified in administrative data?.
        J Oncol Pract. 2018; 14: e51-e58
        • Boyle J.M.
        • Cowling T.E.
        • Kuryba A.
        • et al.
        Development and validation of a coding framework to identify severe acute toxicity from systemic anti-cancer therapy using hospital administrative data.
        Cancer Epidemiol. 2022; 77102096
      1. National bowel cancer audit. Available: https://www.nboca.org.uk/ [accessed 14.01.22].

      2. NHS Digital. Available: https://digital.nhs.uk/data-and-information/data-tools-and-services/data-services/hospital-episode-statistics [accessed 17.01.22].

        • Herbert A.
        • Wijlaars L.
        • Zylbersztejn A.
        • Cromwell D.
        • Hardelid P.
        Data resource profile: hospital episode statistics admitted patient care (HES APC).
        Int J Epidemiol. 2017; 46 (1093–93i)
        • Bright C.J.
        • Lawton S.
        • Benson S.
        • et al.
        Data resource profile: the systemic anti-cancer therapy (SACT) dataset.
        Int J Epidemiol. 2019; 49 (15–15l)
      3. Office for National Statistics. Deaths. Available: https://www.ons.gov.uk/peoplepopulationandcommunity/birthsdeathsandmarriages/deaths [accessed 17.01.22].

        • Oken M.M.
        • Creech R.H.
        • Tormey D.C.
        • et al.
        Toxicity and response criteria of the Eastern cooperative oncology group.
        Am J Clin Oncol. 1982; 5: 649-655
        • Ayanian J.Z.
        • Zaslavsky A.M.
        • Fuchs C.S.
        • et al.
        Use of adjuvant chemotherapy and radiation therapy for colorectal cancer in a population-based cohort.
        J Clin Oncol. 2003; 21: 1293-1300
      4. NHS Digital TRUD. NHS classifications ICD-10. Available: https://isd.digital.nhs.uk/trud3/user/guest/group/0/pack/28 [accessed 22.03.22].

        • The Health and Social Care Information Centre
        Chemotherapy regimens clinical coding standards and guidance OPCS-4 April 2017.
        2017 (Available:)
        • Armitage J.N.
        • van der Meulen J.H.
        Identifying co-morbidity in surgical patients using administrative data with the Royal College of Surgeons Charlson Score.
        Br J Surg. 2010; 97: 772-781
        • Camma C.
        • Giunta M.
        • Fiorica F.
        • et al.
        Preoperative radiotherapy for resectable rectal cancer: a meta-analysis.
        JAMA. 2000; 284: 1008-1015
        • Boyle J.M.
        • Kuryba A.
        • Braun M.S.
        • et al.
        Validity of chemotherapy information derived from routinely collected healthcare data: a national cohort study of colon cancer patients.
        Cancer Epidemiol. 2021; 73101971
      5. National Institute for Health and Care Excellence. Colorectal cancer. NICE guideline [NG151]. 2020. Available: https://www.nice.org.uk/guidance/ng151 [accessed 17.01.22].

        • Walker K.
        • Neuburger J.
        • Groene O.
        • Cromwell D.A.
        • van der Meulen J.
        Public reporting of surgeon outcomes: low numbers of procedures lead to false complacency.
        Lancet. 2013; 382: 1674-1677
        • White I.R.
        • Royston P.
        • Wood A.M.
        Multiple imputation using chained equations: issues and guidance for practice.
        Stat Med. 2011; 30: 377-399
        • Spiegelhalter D.J.
        Funnel plots for comparing institutional performance.
        Stat Med. 2005; 24: 1185-1202
        • Marshall A.
        • Altman D.G.
        • Holder R.L.
        • Royston P.
        Combining estimates of interest in prognostic modelling studies after multiple imputation: current practice and guidelines.
        BMC Med Res Methodol. 2009; 9: 57
        • Keating N.L.
        • Cleveland J.L.F.
        • Wright A.A.
        • et al.
        Evaluation of reliability and correlations of quality measures in cancer care.
        JAMA Netw Open. 2021; 4 (e212474-e74)
        • Grote H.
        • Toma K.
        • Crosby L.
        • et al.
        Outliers from national audits: their analysis and use by the care quality commission in quality assurance and regulation of healthcare services in England.
        Clin Med. 2021; 21: e511
        • Wilson B.E.
        • Jacob S.
        • Yap M.L.
        • et al.
        Estimates of global chemotherapy demands and corresponding physician workforce requirements for 2018 and 2040: a population-based study.
        Lancet Oncol. 2019; 20: 769-780
      6. Outlier Management for National Clinical Audits. Healthcare quality improvement partnership. Available: https://www.hqip.org.uk/wp-content/uploads/2021/11/Appendix-10-HQIP-Outlier-guidance-v4.pdf [accessed 22.03.22].

        • Shahian D.M.
        • Normand S.L.
        What is a performance outlier?.
        BMJ Qual Saf. 2015; 24: 95-99
        • Kamal A.H.
        • Power S.
        • Patierno S.R.
        Addressing issues of cancer disparities, equity, and inclusion through systemized quality improvement.
        JCO Oncol Pract. 2021; 17: 461-462
        • Du X.L.
        • Osborne C.
        • Goodwin J.S.
        Population-based assessment of hospitalizations for toxicity from chemotherapy in older women with breast cancer.
        J Clin Oncol. 2002; 20: 4636-4642
        • Silver J.K.
        • Baima J.
        Cancer prehabilitation: an opportunity to decrease treatment-related morbidity, increase cancer treatment options, and improve physical and psychological health outcomes.
        Am J Phys Med Rehabil. 2013; 92: 715-727
        • Kalsi T.
        • Babic-Illman G.
        • Ross P.J.
        • et al.
        The impact of comprehensive geriatric assessment interventions on tolerance to chemotherapy in older people.
        Br J Cancer. 2015; 112: 1435-1444
        • Wildiers H.
        • Heeren P.
        • Puts M.
        • et al.
        International Society of Geriatric Oncology consensus on geriatric assessment in older patients with cancer.
        J Clin Oncol. 2014; 32: 2595-2603
      7. National Institute for Health and Care Excellence. Capecitabine and oxaliplatin in the adjuvant treatment of stage III (Dukes' C) colon cancer [TA 100]. 2006. Available: https://www.nice.org.uk/guidance/ta100 [accessed 31.10.18].

      8. National Institute for Health and Care Excellence. Cetuximab and panitumumab for previously untreated metastatic colorectal cancer. [TA439]. 2017. Available: https://www.nice.org.uk/guidance/ta439/chapter/1-Recommendations [accessed 06.10.22].

      9. National Institute for Health and Care Excellence. Trifluridine-tipiracil for previously treated metastatic colorectal cancer. [TA405]. 2016. Available: https://www.nice.org.uk/guidance/ta405 [accessed].

      10. National Institute for Health and Care Excellence. Aflibercept in combination with irinotecan and fluorouracil-based therapy for treating metastatic colorectal cancer that has progressed following prior oxaliplatin-based chemotherapy. [TA307]. 2014. Available: https://www.nice.org.uk/guidance/ta307 [accessed 06.10.22].

      11. National Institute for Health and Care Excellence. Guidance on the use of capecitabine and tegafur with uracil for metastatic colorectal cancer. [TA61]. 2003. Available: https://www.nice.org.uk/guidance/ta61 [accessed 06.10.22].

      12. National Institute for Health and Care Excellence. Cetuximab, bevacizumab and panitumumab for the treatment of metastatic colorectal cancer after first-line chemotherapy: cetuximab (monotherapy or combination chemotherapy), bevacizumab (in combination with non-oxaliplatin chemotherapy) and panitumumab (monotherapy) for the treatment of metastatic colorectal cancer after first-line chemotherapy. [TA242]. 2012. Available: https://www.nice.org.uk/guidance/ta242/chapter/1-Guidance [accessed 06.10.22].

        • Iveson T.
        • Hanna C.
        • Iveson P.
        • et al.
        The early impact of the IDEA collaboration results: how the results changed prescribing practice.
        JNCI Cancer Spectr. 2021; 5
        • Hanna C.
        • Boyd K.
        • Jones R.
        P-335 Self-reported prescribing practices in the setting of adjuvant treatment for colorectal cancer.
        Ann Oncol. 2020; 31: S198
        • Enright K.A.
        • Krzyzanowska M.K.
        Benefits and pitfalls of using administrative data to study hospitalization patterns in patients with cancer treated with chemotherapy.
        J Oncol Pract. 2016; 12: 140-141
        • Navani V.
        How has acute oncology improved care for patients?.
        Curr Oncol. 2014; 21: 147-149
        • Geary R.
        • Knight H.
        • Carroll F.
        • et al.
        A step-wise approach to developing indicators to compare the performance of maternity units using hospital administrative data.
        BJOG An Int J Obstet Gynaecol. 2018; 125: 857-865
        • Neuss M.
        • Rocque G.
        • Zuckerman D.
        • et al.
        Establishing a core set of performance measures to improve value in cancer care: ASCO consensus conference recommendation report.
        J Oncol Pract. 2016; 13: 135-140
        • Hurria A.
        • Togawa K.
        • Mohile S.G.
        • et al.
        Predicting chemotherapy toxicity in older adults with cancer: a prospective multicenter study.
        J Clin Oncol. 2011; 29: 3457-3465
        • Tran L.D.
        Social risk adjustment in health care performance measures.
        JAMA Netw Open. 2020; 3 (e208020-e20)
        • Burns E.M.
        • Rigby E.
        • Mamidanna R.
        • et al.
        Systematic review of discharge coding accuracy.
        J Public Health. 2012; 34: 138-148
        • Extermann M.
        • Boler I.
        • Reich R.R.
        • et al.
        Predicting the risk of chemotherapy toxicity in older patients: the chemotherapy risk assessment scale for high-age patients (CRASH) score.
        Cancer. 2012; 118: 3377-3386
        • Gollins S.
        • Moran B.
        • Adams R.
        • et al.
        Association of Coloproctology of Great Britain & Ireland (ACPGBI): guidelines for the management of cancer of the colon, rectum and anus (2017) – multidisciplinary management.
        Colorectal Dis. 2017; 19: 37-66
        • Leong K.
        • Hartley J.
        • Karandikar S.
        Association of Coloproctology of Great Britain & Ireland (ACPGBI): guidelines for the management of cancer of the colon, rectum and anus (2017) – follow up, lifestyle and survivorship.
        Colorectal Dis. 2017; 19: 67-70
        • Rechel B.
        • McKee M.
        • Haas M.
        • et al.
        Public reporting on quality, waiting times and patient experience in 11 high-income countries.
        Health Policy. 2016; 120: 377-383
        • Marshall M.N.
        • Shekelle P.G.
        • Leatherman S.
        • Brook R.H.
        The public release of performance data: what do we expect to gain? A review of the evidence.
        JAMA. 2000; 283: 1866-1874
        • Spinks T.E.
        • Walters R.
        • Feeley T.W.
        • et al.
        Improving cancer care through public reporting of meaningful quality measures.
        Health Aff. 2011; 30: 664-672
        • Aggarwal A.
        • Lewis D.
        • Mason M.
        • et al.
        Effect of patient choice and hospital competition on service configuration and technology adoption within cancer surgery: a national, population-based study.
        Lancet Oncol. 2017; 18: 1445-1453
        • Berwick D.M.
        • James B.
        • Coye M.J.
        Connections between quality measurement and improvement.
        Medical care. 2003; 41: I30-I38
        • Hibbard J.H.
        • Stockard J.
        • Tusler M.
        Does publicizing hospital performance stimulate quality improvement efforts?.
        Health Aff. 2003; 22: 84-94
        • Scott A.
        • Liu M.
        • Yong J.
        Financial incentives to encourage value-based health care.
        Med Care Res Rev. 2018; 75: 3-32
      13. National Cancer Institute. Common terminology criteria for adverse events (CTCAE). Available: https://ctep.cancer.gov/protocoldevelopment/electronic_applications/docs/CTCAE_v5_Quick_Reference_5x7.pdf [accessed 22.03.22].