If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
Department of Health Services Research and Policy, London School of Hygiene and Tropical Medicine, London, UKClinical Effectiveness Unit, Royal College of Surgeons of England, London, UK
Department of Health Services Research and Policy, London School of Hygiene and Tropical Medicine, London, UKClinical Effectiveness Unit, Royal College of Surgeons of England, London, UK
Department of Health Services Research and Policy, London School of Hygiene and Tropical Medicine, London, UKClinical Effectiveness Unit, Royal College of Surgeons of England, London, UK
Department of Health Services Research and Policy, London School of Hygiene and Tropical Medicine, London, UKClinical Effectiveness Unit, Royal College of Surgeons of England, London, UK
Department of Health Services Research and Policy, London School of Hygiene and Tropical Medicine, London, UKDepartment of Oncology, Guy's and St. Thomas' NHS Foundation Trust, London, UK
Quality of SACT delivery was evaluated in adjuvant and metastatic colorectal cancer patients.
•
Variation in toxicity rates across 106 English NHS hospitals was used to explore this.
•
Adjusted toxicity rates varied from 11% to 49% (adjuvant) and from 25% to 67% (metastatic).
•
Twelve hospitals had toxicity rates >2SD above the national average, and 11 >2SD below.
•
Ongoing reporting of this performance indicator can be used for quality improvement.
Abstract
Aim
To date, there has been little systematic assessment of the quality of care associated with systemic anti-cancer therapy (SACT) delivery across national healthcare systems. We evaluated hospital-level toxicity rates during SACT treatment as a means of identifying variation in care quality.
Methods
All colorectal cancer (CRC) patients receiving SACT within 106 English National Health Service (NHS) hospitals between 2016 and 2019 were included.
Severe acute toxicity rates were derived from hospital administrative data using a validated coding framework. Variation in hospital-level toxicity rates was assessed separately in the adjuvant and metastatic settings. Toxicity rates were adjusted for age, sex, comorbidity, performance status, tumour site, and TNM staging.
Results
Eight thousand one hundred and seventy three patients received SACT in the adjuvant setting, and 7,683 patients in the metastatic setting. Adjusted severe acute toxicity rates varied between hospitals from 11% to 49% for the adjuvant cohort, and from 25% to 67% for the metastatic cohort.
Compared to the national mean toxicity rate in the adjuvant cohort, six hospitals were more than two standard deviations (2SD) above, and four hospitals were more than 2SD below. In the metastatic cohort, six hospitals were more than 2SD above, and seven hospitals were more than 2SD below the national mean toxicity rate.
Overall, 12 hospitals (12%) had toxicity rates more than 2SD above the national mean, and 11 (10%) had rates more than 2SD below.
Conclusion
There is substantial variation in hospital-level severe acute toxicity rates in both the adjuvant and metastatic settings, despite risk-adjustment. Ongoing reporting of this performance indicator can be used to focus further investigation of toxicity rates and stimulate quality improvement initiatives to improve care.
The delivery of systemic anti-cancer therapy (SACT) is a complex care process which includes appropriate patient selection and optimisation, tailoring treatment doses, and the monitoring and management of toxicities [
]. Whilst randomised controlled trials (RCTs) have established the efficacy of SACT, there has been little or no systematic assessment of the quality of SACT delivery within routine care. Much of the available literature on the quality of SACT delivery focuses on access to treatments rather than on outcome measures [
The only performance indicator currently reported and monitored at hospital level is 30-day mortality after the final SACT treatment, which is more a proxy for the appropriate selection of patients for SACT treatment than a measure of quality of care [
National Institute for Health and Care Excellence. Capecitabine and oxaliplatin in the adjuvant treatment of stage III (Dukes' C) colon cancer [TA 100]. 2006. Available: https://www.nice.org.uk/guidance/ta100 [accessed 31.10.18].
]. A study in breast cancer patients showed a hospitalisation rate of 43% in patients during SACT with about three quarters of the admissions confirmed as SACT-related events [
We have previously validated an indicator of severe acute toxicity (at least Grade 3 according to the Common Terminology Criteria for Adverse Events (CTCAE)) derived from hospital administrative data in colorectal cancer (CRC) [
Development and validation of a coding framework to identify severe acute toxicity from systemic anti-cancer therapy using hospital administrative data.
]. The indicator uses a coding framework to identify specific diagnostic codes during the timeframe of chemotherapy administration which are likely to represent SACT-related severe acute toxicity. As part of this work, we found variations in toxicity rates across different SACT regimens in line with those seen in RCTs. In addition, the rates of toxicity were associated with anticipated risk factors, for example, higher rates in those with comorbidities.
In the current study, we evaluate the use of this measure as a national-level performance indicator to assess hospital variation in severe acute toxicity rates for CRC patients receiving SACT. The indicator will be used to identify potentially outlying hospital performance and benchmark best practice to support quality improvement processes in SACT delivery.
2. Methods
2.1 Data sources
In this national population-based evaluation, we used National Bowel Cancer Audit (NBOCA) data [
] linked at patient level for CRC patients diagnosed and treated in the English National Health Service (NHS). SACT and HES data were available up until 31 March 2020.
NBOCA is a prospective mandatory database for all newly diagnosed CRC patients in the English NHS. Data items in NBOCA were used to determine sex, age, Eastern Cooperative Oncology Group performance status [
] and procedures are coded using the Office of Population Censuses and Surveys Classification of Surgical Operations and Procedures, 4th revision (OPCS-4) [
]. Data from HES were used to identify admissions for severe acute toxicity (see below), to determine the number of comorbidities according to the RCS Charlson comorbidity score [
] and as one of the sources of SACT information (see below).
HES was also used to determine socioeconomic status. This was derived from the Index of Multiple Deprivation (IMD) which ranks 32,482 geographical areas of England according to their level of deprivation across seven domains [
]. Patients are allocated to an IMD quintile (IMDQ) based on the national ranking of the area corresponding to their postcode.
The SACT dataset captures detailed drug-level information for SACT administered in any inpatient, day case, outpatient, or community setting, including individual administration dates [
]. The SACT dataset was also used to determine the NHS hospital trust (the organisational unit of NHS hospitals in England that can be located on one or more sites) that delivered SACT. Data submission is mandatory for all chemotherapy providers within the English NHS, excluding a small proportion of privately treated patients. One hundred and six hospitals were identified as delivering SACT. These hospitals needed to have treated at least ten patients during the inclusion period (see below) to be included in further analyses.
2.2 Study population
We defined two distinct cohorts of patients aged 18 years and above with a primary diagnosis of CRC (ICD-10: C18, C19 and C20). These included patients identified in the NBOCA database and undergoing treatments at an English NHS hospital during the inclusion period (1 April 2016 to 31 March 2019). These cohorts define the ‘denominator’ of the performance indicator.
The first cohort included patients with pathological stage III CRC who had received adjuvant SACT according to SACT or HES within the 4-month period after major resection, as per previous work [
]. Patients in the stage III cohort were restricted to those receiving capecitabine and oxaliplatin (CAPOX), 5-fluorouracil (5-FU) and oxaliplatin (FOLFOX), or single agent fluoropyrimidine (capecitabine or 5-FU alone), as per national guidelines [
National Institute for Health and Care Excellence. Colorectal cancer. NICE guideline [NG151]. 2020. Available: https://www.nice.org.uk/guidance/ng151 [accessed 17.01.22].
The second cohort included patients who were diagnosed with stage IV CRC and had commenced SACT within the 4-month period after diagnosis, according to SACT or HES. Of the 6,810 patients (89%) with SACT records and therefore drug-level information, approximately 46% of patients received an oxaliplatin-based regimen, 26% an irinotecan-based regimen, 15% single agent fluoropyrimidine, 8% irinotecan with a targeted therapy (e.g. bevacizumab, cetuximab), 5% fluoropyrimidine with a targeted therapy, less than 1% a targeted therapy alone, and less than 1% other agents (e.g. raltitrexed). For stage IV patients, SACT was restricted to treatments given continuously (gaps of no more than 8 weeks between cycles) for a maximum of 12 months.
The pooled reporting of stage III and IV patients was deemed inappropriate given the heterogeneity of the groups in terms of disease burden, underlying fitness, differences in chemotherapy regimens used, and reported differences in overall toxicity rates reported by our prior research [
Development and validation of a coding framework to identify severe acute toxicity from systemic anti-cancer therapy using hospital administrative data.
The coding framework defined patients who had experienced severe acute toxicity (the ‘numerator’ of the performance indicator) according to the presence of pre-defined ICD-10 diagnostic codes in HES indicative of a SACT-related toxicity (supplementary Table 1) [
Development and validation of a coding framework to identify severe acute toxicity from systemic anti-cancer therapy using hospital administrative data.
]. Severe toxicity was defined as those patients with a selected ICD-10 diagnostic code who required an overnight hospital admission between the administration of the first cycle of SACT and up until 8 weeks after the administration of the last cycle of SACT.
For the small proportion of patients undergoing a surgical procedure during this timeframe, the date of surgery was used as the cut-off for identifying toxicities to ensure that post-operative complications were not captured.
The indicator captures all admissions to any English NHS hospital, regardless of whether or not the hospital provides chemotherapy. Toxicities were attributed to the hospital providing the chemotherapy. Planned and unplanned admissions were included to ensure that we captured all possible SACT-related admissions (direct hospital admission, acute oncology referral, or self-referral).
2.4 Statistical power
We calculated the statistical power to detect an important difference (defined as a 50% increase in the toxicity rate compared to the overall national rate) in the rate of severe acute toxicity in a typical hospital and the overall national rate in England, for each cohort. We calculated the median number of patients per year receiving SACT at each hospital over the 3-year study period and used this to calculate the statistical power for reporting periods of 1-, 3- and 5-years. The definition of an important difference is arbitrary but was chosen because it has been used in previous work and represents a substantial absolute increase in the rate of toxicity [
]. A 5% significance level was used for testing differences between the hospital-level and national rates because it corresponds to the commonly used 95%-control limits of funnel plots (see below).
2.5 Fairness
We determined to what extent we could adjust for the case-mix factors that are likely to affect the risk of severe acute toxicity based on a combination of review of existing literature and expert clinical input. Only factors outside the control of the healthcare provider were adjusted for. We assessed the data completeness of these factors. Missing values for case-mix factors were imputed with multiple imputation [
]. We carried out indirect risk-adjustment, using multivariable logistic regression modelling to obtain expected numbers of severe acute toxicity events per hospital [
]. The association between patient and clinical characteristics and toxicity were assessed separately for each cohort using Wald tests to calculate p values with the significance level set at 0.05.
Calibration of the risk-adjustment model across deciles of predicted risk was assessed using the Hosmer–Lemeshow test, constructing an F-statistic to carry out the test in multiply imputed data [
Within our cohort of 106 hospitals, we used funnel plots to identify outlying hospitals defined as those with results more than two standard deviations (2SD) (corresponding to 95%-control, or inner, funnel limits), or three standard deviations (3SD) (corresponding to 99.8%-control, or outer, funnel limits) below or above the overall national rate. This is equivalent to carrying out statistical tests comparing a specific hospital's result with the overall national rate using a two-sided 5% or 0.02% significance level, respectively [
Outliers from national audits: their analysis and use by the care quality commission in quality assurance and regulation of healthcare services in England.
]. Fully-adjusted funnel plots were generated for each cohort.
3. Results
3.1 Study cohorts
Between 1 April 2016 and 31 March 2019, 8,173 patients received adjuvant SACT for stage III CRC. Of these 8,173 patients, 2,074 (25%) had a severe acute toxicity identified according to the indicator. In addition, 7,683 patients received SACT within 4 months of a diagnosis of stage IV CRC. Of these 7,683 patients, 3,625 (47%) had a severe acute toxicity identified. Table 1 summarises the different types of toxicity identified from the coding framework, according to organ system.
Table 1Distribution of types of toxicities following SACT administration for stage III and IV cohorts [
Association of Coloproctology of Great Britain & Ireland (ACPGBI): guidelines for the management of cancer of the colon, rectum and anus (2017) – multidisciplinary management.
For the stage III cohort, 97 out of 106 English NHS hospitals had treated more than ten patients over the 3-year inclusion period and were included in further analyses. The annual volumes of patients who received adjuvant SACT in each hospital varied considerably with a median value of 24 (range 5–132, interquartile range 15–33). Similarly, 98 hospitals were included for the stage IV cohort, with a median annual volume of 22 (range 5–142, interquartile range 13–32).
The statistical power to detect an increase of 50% compared to the overall national mean rate for different reporting periods (1-, 3-, and 5-year) are presented in supplementary Table 2. These power calculations demonstrate that a 3-year reporting period achieves approximately 70% power in the stage III and 99% power in the stage IV cohort to detect a 50% increase compared to the overall national rate. A 1-year reporting period could have been chosen for the stage IV cohort but, for consistency, the same reporting period was used for both.
3.3 Fairness
For risk-adjustment, the following case-mix factors were identified within the literature and from clinical expertise: age, sex, number of comorbidities, performance status, tumour site, staging, and socioeconomic status [
]. All of these case-mix factors are typically accessible from routinely collected datasets. We found that their completeness rate is high in the English NHS (Table 2a, Table 2b).
Table 2aPatient and tumour characteristics and associated severe acute toxicity for patients with stage III disease.
Table 2a, Table 2b summarise for the stage III and stage IV cohorts the results of the logistic regression models that capture the associations between the case-mix factors and risk of severe acute toxicity. In both cohorts, severe acute toxicity was increased for female sex, those with more than 2 comorbidities, and advanced T- and N-stage disease. In the stage III cohort, rectal cancer was associated with increased toxicity whereas in the stage IV cohort it was associated with reduced toxicity. In the stage IV cohort, rectal cancer patients were significantly younger and fitter (according to performance status) than those patients with colon cancer. In the stage IV cohort, poor performance status was associated with increased toxicity but this association was not statistically significant in the stage III cohort.
Following this analysis, we included age, sex, RCS Charlson comorbidity score, performance status, tumour site, T-stage, and N-stage in the logistic regression models used to adjust for case-mix factors when assessing between-hospital variation in rates of severe acute toxicity. Due to the debate around its appropriateness in case-mix adjustment, and the fact it was not associated with increased toxicity, socioeconomic status was included in the model as a sensitivity analysis.
There was no evidence of a lack of calibration for either the stage III cohort (p = 0.711) or the stage IV cohort (p = 0.952), according to the Hosmer–Lemeshow test. The C-statistic of discrimination was 0.58 (95% CI: 0.57 to 0.59) for the stage III cohort and 0.64 (95% CI: 0.62 to 0.66) for the stage IV cohort.
3.4 Variation between hospitals
The unadjusted rates of severe acute toxicity after adjuvant SACT in stage III CRC patients varied considerably between the 97 included hospitals, ranging from 11% to 47% with ten hospitals outside the 95%-funnel limits, including one outside the 99.8%-funnel limits (Fig. 1A). Adjusting for case-mix factors had little effect on the variation in severe acute toxicity rates. Adjusted severe acute toxicity rates ranged from 11% to 49% with the same outlying hospitals (Fig. 1B). This corresponded to one hospital being 3SD above, five hospitals being 2SD above, and four hospitals being 2SD below, the national mean toxicity rate. A sensitivity analysis including socioeconomic status in the risk-adjustment did not change the outlying hospitals (results not presented).
Fig. 1Funnel plot showing a) unadjusted and b) adjusted rates of severe acute toxicity by English NHS hospital for patients receiving SACT for stage III colorectal cancer.
The unadjusted rates of severe acute toxicity after SACT for stage IV CRC also varied considerably between the included 98 English NHS hospitals, ranging from 26% to 65% with 12 hospitals outside the 95%-funnel limits (Fig. 2A). Adjusting for case-mix factors had little effect on the variation in severe acute toxicity rates (25%–67%) and outlying hospitals, with 13 hospitals outside the 95%-funnel limits (Figure 2B). This corresponded to six hospitals being 2SD above, and seven hospitals being 2SD below, the national mean toxicity rate. A sensitivity analysis including socioeconomic status in the risk-adjustment generated two new outlying hospitals (results not presented).
Fig. 2Funnel plots showing a) unadjusted and b) adjusted rates of severe acute toxicity by English NHS hospital for patients receiving SACT for stage IV colorectal cancer.
Across both cohorts, 22 different hospitals were identified as having rates of severe acute toxicity more than 2SD from the national mean toxicity rate (only 1 hospital had rates more than 2SD for both cohorts). The Pearson correlation coefficient comparing the adjuvant and metastatic rates of toxicity for each hospital was 0.2 (p = 0.090).
4. Discussion
This study demonstrates how diagnostic coding in hospital administrative data can be used to derive a hospital-level performance indicator of SACT toxicity across hospitals treating CRC patients. We used a previously validated coding framework based on a pre-defined set of specific ICD-10 codes and found considerable variation in severe acute toxicity rates between hospitals in both the adjuvant and metastatic setting. We found that hospital rates of severe acute toxicity requiring an overnight hospital admission (equivalent to at least grade 3 CTCAE) varied between 11% and 49% in stage III patients receiving adjuvant chemotherapy, and were even higher for stage IV patients with rates varying between 25% and 67%. We identified 22 potentially outlying hospitals, even after adjustment for important case-mix factors.
This hospital performance indicator will be used as part of a publicly reported outlier program in the United Kingdom (UK) from 2022. Within the national outlier process, hospitals are grouped as ‘alerts’ (greater or less than 2SD above the national mean) or ‘alarms’ (greater or less than 3SD above the national mean) [
]. Once the data is verified, hospitals are expected to formulate a formal response and action plan to understand which factors might be driving unwarranted variation (Fig. 3) [
Setting quality improvement priorities for women receiving systemic therapy for early-stage breast cancer by using population-level administrative data.
Outliers from national audits: their analysis and use by the care quality commission in quality assurance and regulation of healthcare services in England.
]. ‘Alert’ hospitals are monitored and become potential outliers if they are identified as such in consecutive years. The outlier process is entirely publicly reported. Further work to establish best practice is required and might involve, for example, evaluating survival outcomes in outlying hospitals to evaluate the impact of differential toxicity rates.
Fig. 3Quality improvement conceptual framework highlighting potential areas within the SACT care pathway that may represent sources of variation in care.
These results show that this performance indicator can be used to further explore unwarranted variation in toxicity rates, as well as triggering and guiding initiatives to improve the quality of SACT delivery on a national scale. As the performance indicator is derived from administrative diagnostic coding, the risk of information bias or clinical data manipulation is reduced. The linked datasets included patient and tumour characteristics that allowed the development of a case-mix adjustment model with good calibration and adequate discriminatory power.
The use of ICD-10 codes in the coding framework makes it internationally applicable as it can be applied in different health systems that use ICD-10 codes within their hospital administrative data. In addition, whilst this study has focused on CRC, the coding framework can be applied across different tumour types and regimens, including targeted therapies and immunotherapy.
4.1 Differences in hospital-level severe acute SACT toxicity
The higher rates of toxicity in patients with advanced disease has been previously observed [
]. This is likely due to differences in baseline characteristics, for example, poorer performance status in those with stage IV disease, and the wider range of SACT drugs used within this cohort. Fig. 3 also considers the points along the SACT care pathway that may contribute to the between-hospital variation in toxicity rates.
Outlying rates of severe acute toxicity may reflect differences in the assessment and selection of patients regarding fitness or appropriateness of treatment. For example, patients with a high risk of severe acute toxicity might be identified through comprehensive risk-stratification and discussion within the multidisciplinary team, and there may be differential access to specialist geriatrician and prehabilitation services [
Cancer prehabilitation: an opportunity to decrease treatment-related morbidity, increase cancer treatment options, and improve physical and psychological health outcomes.
]. In addition, patient choice is an important and complex aspect of the decision-making process to consider, with both patient-related factors (e.g. social, cognitive, and psychological issues) and clinician-related factors (e.g. communication and clinician biases) influencing whether SACT is given, as well as what choice of SACT. Patients should be appropriately counselled and consented for SACT treatments by a sufficiently experienced clinician, including the potential short- and long-term side-effects.
Furthermore, unwarranted variation in rates of toxicity may represent under- or over-treatment of patients, as well as inappropriate regimen use and dosing, inadequate or outdated protocols, insufficient monitoring, or failure to recognise and address early signs of toxicity. Despite clear national guidelines in the UK regarding which SACT drugs are approved for which indication [
National Institute for Health and Care Excellence. Capecitabine and oxaliplatin in the adjuvant treatment of stage III (Dukes' C) colon cancer [TA 100]. 2006. Available: https://www.nice.org.uk/guidance/ta100 [accessed 31.10.18].
National Institute for Health and Care Excellence. Trifluridine-tipiracil for previously treated metastatic colorectal cancer. [TA405]. 2016. Available: https://www.nice.org.uk/guidance/ta405 [accessed].
National Institute for Health and Care Excellence. Aflibercept in combination with irinotecan and fluorouracil-based therapy for treating metastatic colorectal cancer that has progressed following prior oxaliplatin-based chemotherapy. [TA307]. 2014. Available: https://www.nice.org.uk/guidance/ta307 [accessed 06.10.22].
National Institute for Health and Care Excellence. Guidance on the use of capecitabine and tegafur with uracil for metastatic colorectal cancer. [TA61]. 2003. Available: https://www.nice.org.uk/guidance/ta61 [accessed 06.10.22].
National Institute for Health and Care Excellence. Cetuximab, bevacizumab and panitumumab for the treatment of metastatic colorectal cancer after first-line chemotherapy: cetuximab (monotherapy or combination chemotherapy), bevacizumab (in combination with non-oxaliplatin chemotherapy) and panitumumab (monotherapy) for the treatment of metastatic colorectal cancer after first-line chemotherapy. [TA242]. 2012. Available: https://www.nice.org.uk/guidance/ta242/chapter/1-Guidance [accessed 06.10.22].
], we recognise that different providers may have variability in prescribing practices beyond this. For example, even current national guidelines do not offer guidance between the use of FOLFOX, CAPOX, or single agent fluoropyrimidine [
National Institute for Health and Care Excellence. Colorectal cancer. NICE guideline [NG151]. 2020. Available: https://www.nice.org.uk/guidance/ng151 [accessed 17.01.22].
In addition, the IDEA collaborative results were published during the study period and surveys have highlighted ongoing variation in clinical practice with regards to the choice and duration of combination adjuvant chemotherapy with shifts towards 3 months of treatment for low-risk disease, particularly in the UK [
]. For the metastatic cohort, there may be variation in which providers participate in clinical trials. However, adjusting for choice or duration of regimen which are under the control of the provider was deemed inappropriate given that this could represent poor clinical practice.
Finally, the available infrastructure and clinical pathways within hospitals may play a role, for example, variability in access to acute oncology services, emergency services, and the availability of specialist on-site advice out of hours, as well as more generalised disparities in clinical expertise, availability, and training [
]. For example, acute oncology services have been shown to improve outcomes, although substantial variation remains in whether hospitals have access to these services [
]. Acute oncology services may increase planned admissions. Although the vast majority (91%) of admissions for toxicity included in this study were unplanned, this would be an important area to further investigate for potential outliers.
Only one hospital was identified as a low outlier for both cohorts of patients. When comparing both cohorts, there was evidence of a weak association between the rates of toxicity in the adjuvant and metastatic cohorts within each hospital. However, we would not necessarily expect these to align as the two cohorts are very different as evidenced by their mean toxicity rates. First, this may reflect the reduced fitness of patients in the stage IV cohort, and more complex care pathways they undertake, which mean that potential poor treatment, patient selection, and deficient supportive care, are more likely to be exposed. Second, there are differences in care pathways between the two cohorts which might account for differences in toxicity rates within the same provider. For example, the administration and monitoring of adjuvant chemotherapy may be nurse-led in contrast to chemotherapy given for metastatic disease. Third, patient-clinician communication, decision-making, and patient choice are likely to be more complex and nuanced within the metastatic setting which may also influence toxicity rates.
4.2 Strengths and limitations
The strengths of this study include ensuring that the performance indicator meets a pre-defined set of essential criteria (‘validity’, ‘statistical power’, and ‘fairness’ as detailed in the Methods) [
Development and validation of a coding framework to identify severe acute toxicity from systemic anti-cancer therapy using hospital administrative data.
]. Although within this study we have reported the overall incidence of severe toxicity after SACT at each hospital, we are also able to detail specific individual toxicities according using our indicator (e.g. neutropenic sepsis, diarrhoea, and line complications – see Table 1). This is hugely important for providing detailed feedback to hospitals to facilitate quality improvement.
In addition, routinely available national clinical cancer data linked to SACT data and hospital administrative data provided over 95% case ascertainment across all English NHS hospitals with good recording of comorbidities, performance status, staging, and detailed SACT information. This also allowed the capture of all hospital admissions, regardless of whether the hospital provided chemotherapy, and we assigned the toxicity to the hospital delivering the chemotherapy [
This, and the good calibration of the risk-adjustment model, meant that toxicity rates between hospitals could be adjusted for important case-mix factors known to influence toxicity, enhancing the fairness of hospital-specific reporting [
]. There is a debate surrounding the complexities of the inclusion of deprivation in risk-adjustment, with varying practice between different reporting programmes [
]. A sensitivity analysis showed minor changes in results for the stage IV cohort which need to be considered. Finally, an overview of potentially actionable areas has been identified as a starting point for targeted local quality improvement (Fig. 3).
The first limitation of this study is the reliance on ICD-10 diagnostic codes in hospital administrative data. However, these diagnostic codes in HES have been shown to be accurate compared to clinical notes, thereby supporting its use for healthcare performance assessment and research [
]. In addition, we used two independent data sources to capture information about the use of SACT. Our previous validation work has shown excellent agreement between the two data sources (SACT and HES data) [
]. Second, there is a possibility that some of the variation between hospitals is due to chance alone. However, we would only expect five hospitals to lie more than two standard deviations from the national mean by chance.
Third, the coding framework is best suited for studying differences between groups of patients (e.g. those treated in different hospitals or receiving different SACT regimens) rather than estimating absolute rates of toxicity which may be over-estimated. However, to limit the likelihood of overestimation we restricted the indicator to only those diagnoses likely to reflect SACT toxicity and to the precise time period of chemotherapy administration (excluding any post-operative period in a small proportion of patients). In addition, previous studies have suggested the vast majority of hospital admissions during SACT treatment are SACT-related [
Finally, other studies have demonstrated that mental health status, nutritional status, and laboratory values (e.g. blood tests) were also important predictors for SACT toxicity in older patients [
]. However, this information is not routinely included in hospital administrative data and so could not be included as part of the risk-adjustment. As a result, a certain level of ‘residual confounding’ will need to be accepted, irrespective of which patient groups are being compared.
4.3 Implications
There are several implications of using this performance indicator for a national assessment of the delivery of SACT. First, our study shows that the performance indicator can be used to compare SACT toxicity between hospitals [
Outliers from national audits: their analysis and use by the care quality commission in quality assurance and regulation of healthcare services in England.
]. In the English NHS, similar hospital-level reporting is already routinely available for patients who had surgery or radiotherapy for prostate, bowel, and oesophageal cancer as part of a programme of national clinical audits [
Association of Coloproctology of Great Britain & Ireland (ACPGBI): guidelines for the management of cancer of the colon, rectum and anus (2017) – multidisciplinary management.
Association of Coloproctology of Great Britain & Ireland (ACPGBI): guidelines for the management of cancer of the colon, rectum and anus (2017) – follow up, lifestyle and survivorship.
Second, the performance indicator allows ongoing monitoring of severe acute toxicity events within hospitals which will inform continuous local quality improvement processes. Evidence has shown that quality improvement initiatives are more likely to produce positive effects if continuous monitoring and feedback is undertaken [
Outliers from national audits: their analysis and use by the care quality commission in quality assurance and regulation of healthcare services in England.
]. As per previous work, the indicator allows specific individual toxicities to be described in detail which can further inform quality improvement processes [
Development and validation of a coding framework to identify severe acute toxicity from systemic anti-cancer therapy using hospital administrative data.
Third, an improved understanding of the risks of SACT will inform the counselling of patients and strengthen the process of ‘shared decision making’ in day-to-day practice, particularly for novel SACT drugs.
In addition to supporting direct patient care, public reporting of severe acute toxicity rates can also provide transparency around best practice through benchmarking to guide patients in making informed choices about the hospital in which they will receive their SACT treatment [
]. This avoids the reliance on surrogate markers of care quality (e.g. presence of robotic surgery), and further stimulates quality improvement through competitive mechanisms or regulation by reducing information asymmetry regarding care quality [
Effect of patient choice and hospital competition on service configuration and technology adoption within cancer surgery: a national, population-based study.
]. This transparency can also guide investments, with outcomes considered as part of pay-for-performance schemes in order to support greater value in care delivery [
Finally, the coding framework developed to identify severe acute toxicity was designed to be broad in order to make it applicable to all types of SACT, including traditional cytotoxic chemotherapy, immunotherapy, and targeted agents [
Development and validation of a coding framework to identify severe acute toxicity from systemic anti-cancer therapy using hospital administrative data.
]. This means that the performance indicator can be used in a wide range of clinical settings and expanded across most cancer types, following appropriate validation.
5. Conclusion
We have evaluated the use of a national performance indicator derived from linked clinical and hospital administrative datasets to assess hospital variation in severe acute toxicity rates for hospitals providing SACT for CRC patients in order to stimulate and support quality improvement. This approach can be applied across different cancer types and in many different countries where similar regional or national clinical and administrative hospital datasets are available.
Funding
This study was undertaken alongside the National Bowel Cancer Audit. The audit is commissioned by the Healthcare Quality Improvement Partnership (HQIP) as part of the National Clinical Audit and Patient Outcomes Programme, and funded by NHS England and the Welsh Government (https://www.hqip.org.uk/national-programmes). AA was supported by a National Institute for Health Research (NIHR) Advanced Fellowship [NIHR300599] and TEC by a fellowship from the Medical Research Council [MR/S020470/1]. Neither HQIP or any of the funders had any involvement in the study design; in the collection, analysis and interpretation of data; in the writing of the report; or in the decision to submit the article for publication. The researchers had full independence from HQIP.
Author contributions
Jemma M. Boyle – conceptualization, formal analysis, investigation, data curation, writing – original draft, writing – review and editing, visualisation.
Angela Kuryba – writing – review and editing.
Thomas E. Cowling – formal analysis, writing – review and editing, visualisation.
Jan van der Meulen – writing – review and editing, supervision.
Nicola S. Fearnhead – writing – review and editing.
Kate Walker – conceptualisation, methodology, writing – review and editing, supervision.
Michael S. Braun – conceptualisation, methodology, writing – review and editing, supervision.
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Setting quality improvement priorities for women receiving systemic therapy for early-stage breast cancer by using population-level administrative data.
Development and validation of a coding framework to identify severe acute toxicity from systemic anti-cancer therapy using hospital administrative data.
National Institute for Health and Care Excellence. Colorectal cancer. NICE guideline [NG151]. 2020. Available: https://www.nice.org.uk/guidance/ng151 [accessed 17.01.22].
Outliers from national audits: their analysis and use by the care quality commission in quality assurance and regulation of healthcare services in England.
Cancer prehabilitation: an opportunity to decrease treatment-related morbidity, increase cancer treatment options, and improve physical and psychological health outcomes.
National Institute for Health and Care Excellence. Capecitabine and oxaliplatin in the adjuvant treatment of stage III (Dukes' C) colon cancer [TA 100]. 2006. Available: https://www.nice.org.uk/guidance/ta100 [accessed 31.10.18].
National Institute for Health and Care Excellence. Trifluridine-tipiracil for previously treated metastatic colorectal cancer. [TA405]. 2016. Available: https://www.nice.org.uk/guidance/ta405 [accessed].
National Institute for Health and Care Excellence. Aflibercept in combination with irinotecan and fluorouracil-based therapy for treating metastatic colorectal cancer that has progressed following prior oxaliplatin-based chemotherapy. [TA307]. 2014. Available: https://www.nice.org.uk/guidance/ta307 [accessed 06.10.22].
National Institute for Health and Care Excellence. Guidance on the use of capecitabine and tegafur with uracil for metastatic colorectal cancer. [TA61]. 2003. Available: https://www.nice.org.uk/guidance/ta61 [accessed 06.10.22].
National Institute for Health and Care Excellence. Cetuximab, bevacizumab and panitumumab for the treatment of metastatic colorectal cancer after first-line chemotherapy: cetuximab (monotherapy or combination chemotherapy), bevacizumab (in combination with non-oxaliplatin chemotherapy) and panitumumab (monotherapy) for the treatment of metastatic colorectal cancer after first-line chemotherapy. [TA242]. 2012. Available: https://www.nice.org.uk/guidance/ta242/chapter/1-Guidance [accessed 06.10.22].
Association of Coloproctology of Great Britain & Ireland (ACPGBI): guidelines for the management of cancer of the colon, rectum and anus (2017) – multidisciplinary management.
Association of Coloproctology of Great Britain & Ireland (ACPGBI): guidelines for the management of cancer of the colon, rectum and anus (2017) – follow up, lifestyle and survivorship.
Effect of patient choice and hospital competition on service configuration and technology adoption within cancer surgery: a national, population-based study.
01320-X&id=gr1.jpg){kind=link}
01320-X&id=gr2.jpg){kind=link}
01320-X&id=gr3.jpg){kind=link}