Advertisement

Content validity of the EORTC quality of life questionnaire QLQ-C30 for use in cancer

Open AccessPublished:November 24, 2022DOI:https://doi.org/10.1016/j.ejca.2022.10.026

      Highlights

      • European Organisation for Research and Treatment of Cancer Quality of Life Questionnaire Core 30 is widely used to measure quality of life in oncology.
      • Content validity is evidence that an instrument measures the concept of interest.
      • Measures should demonstrate content validity directly from patients.
      • This study demonstrates content validity across a broad range of cancers and stages.
      • The European Organisation for Research and Treatment of Cancer measurement model remains valid today for use in research.

      Abstract

      Aim

      The European Organisation for Research and Treatment of Cancer Quality of Life Questionnaire Core 30 (QLQ-C30) is among the most widely used patient-reported outcome measures in cancer research and practice. It was developed prior to guidance that content should be established directly from patients to confirm it measures concepts of interest and is appropriate and comprehensive for the intended population. This study evaluated the content validity of the QLQ-C30 for use with cancer patients.

      Methods

      Adults undergoing cancer treatment in Europe and the USA participated in open-ended concept elicitation interviews regarding their functional health, symptoms, side-effects and impacts on health-related quality of life. Thematic analysis was conducted, and similarities across cancer types, disease stages and countries or languages were explored.

      Results

      Interviews with 113 patients with cancer (85 European, 28 USA) including breast, lung, prostate, colorectal and other cancers were conducted between 2016 and 2020. Conceptual saturation was achieved. The most frequently reported concepts were included in the QLQ-C30 conceptual framework. QLQ-C30 items were widely understood across language versions and were relevant to patients across cancer types and disease stages. While several new concepts were elicited such as difficulty climbing steps or stairs, weight loss, skin problems and numbness, many were not widely experienced and/or could be considered sub-concepts of existing concepts.

      Conclusions

      The QLQ-C30 demonstrates good evidence of content validity for the assessment of functional health, symptom burden and health-related quality of life in patients with localised-to-advanced cancer.

      Keywords

      1. Introduction

      The European Organisation for Research and Treatment of Cancer (EORTC) Quality of Life Questionnaire Core 30 (QLQ-C30) is a patient-reported outcome measure (PROM) designed to assess the functional health, symptom burden and health-related quality of life (HRQoL) of patients with localised-to-advanced cancer. The original version was published in 1993 [
      • Aaronson N.K.
      • Ahmedzai S.
      • Bergman B.
      • et al.
      The European Organisation for Research and Treatment of Cancer QLQ-C30: a quality-of-life instrument for use in international clinical trials in oncology.
      ]. The current version (QLQ-C30 v3.0) contains 30 items and has been used since 1997 [
      • Bjordal K.
      • De Graeff A.
      • Fayers P.
      • et al.
      A 12 country field study of the EORTC QLQ-C30 (version 3.0) and the head and neck cancer specific module (EORTC QLQ-H&N35) in head and neck patients.
      ], translated and validated in over 120 languages and used in more than 5000 studies worldwide. The QLQ-C30 (https://qol.eortc.org/) includes 15 scales: five functional scales assessing physical functioning, role functioning, emotional functioning, cognitive functioning and social functioning; nine multi- and single-item scales assessing fatigue, nausea and vomiting, pain, dyspnoea, insomnia, appetite loss, constipation, diarrhoea, financial difficulties and a global health status/QoL scale. The EORTC measurement model supplements the QLQ-C30 with disease- or treatment-specific modules (e.g. for breast cancer) [
      • Bjelic-Radisic V.
      • Cardoso F.
      • Cameron D.
      • et al.
      An international update of the EORTC questionnaire for assessing quality of life in breast cancer patients: EORTC QLQ-BR45.
      ,
      • Aaronson N.
      • Bullinger M.
      • Ahmedzai S.
      A modular approach to quality-of-life assessment in cancer clinical trials.
      ].
      Despite wide acceptance of the QLQ-C30, evidence of content validity specifically for use in patients with cancer has not been demonstrated (though all recently developed disease-specific modules debriefed in patient interviews have included the QLQ-C30 concepts). Demonstrating a PROM's content validity involves generating qualitative evidence from patients that the questionnaire framework and items are appropriate and comprehensive relative to its intended measurement concept, population and context of use [
      • Smith A.B.
      • Cocks K.
      Content validity and legacy patient-reported outcome measures in cancer.
      ]. During development, the QLQ-C30's conceptual framework and items were based on oncologist and researcher consensus [
      • Aaronson N.
      • Bullinger M.
      • Ahmedzai S.
      A modular approach to quality-of-life assessment in cancer clinical trials.
      ,
      • Aaronson N.
      • Ahmedzai S.
      • Bullinger M.
      • et al.
      The EORTC core quality of life questionnaire: interim results of an international field study.
      ], before undergoing validity and reliability testing with patients [
      • Osoba D.
      • Aaronson N.
      • Zee B.
      • Sprangers M.
      • Te Velde A.
      Modification of the EORTC QLQ-C30 (version 2.0) based on content validity and reliability testing in large samples of patients with cancer.
      ]. However, since initial development, more detailed guidelines and standards have been established for demonstrating PROMs' content validity, focussing on patient input to inform measurement concepts and acceptability of item and instruction wording [
      • Haynes S.N.
      • Richard D.
      • Kubany E.S.
      Content validity in psychological assessment: a functional approach to concepts and methods.
      ,
      US Food and Drug Administration
      Guidance for Industry: patient-reported outcome measures: use in medical product development to support labeling claims.
      ,
      • Patrick D.L.
      • Burke L.B.
      • Gwaltney C.J.
      • et al.
      Content validity—establishing and reporting the evidence in newly developed patient-reported outcomes (PRO) instruments for medical product evaluation: ISPOR PRO Good Research Practices Task Force report: part 2—assessing respondent understanding.
      ,
      • Wheelwright Sally
      • Bjordal Kristen
      • Bottomley Andrew
      • et al.
      EORTC Quality of Life Group
      ].
      To provide evidence of the QLQ-C30's suitability for use with contemporary cancer patients, qualitative exploration of the functional health, symptom burden and HRQoL impact of cancer is required. The extent to which this experience is consistent between patients from populations that differ in terms of demographic and clinical characteristics, and geographic locations must also be understood. Further, evidence of patients' understanding and relevance of QLQ-C30 items in these populations is needed. This research aimed to explore the extent to which the QLQ-C30 is an appropriate and comprehensive measure of functional health, symptom burden and HRQoL for use with cancer patients across disease stages. Due to the extensive use of the QLQ-C30 Version 3, the objective was not only to provide current evidence for the content validity in order to support its ongoing use but also to highlight any gaps that need to be considered when including the QLQ-C30 in a study due to the expanded portfolio of novel cancer treatments. If the study had found significant problems with the understanding or relevance of the items in the QLQ-C30, recommendations would have been made to identify alternative wording to improve reliability. The qualitative analysis was conducted by a team outside the EORTC Quality of Life Group (QLG), who specialise in PROM development and validation.

      2. Material and methods

      This qualitative interview study was conducted with patients undergoing treatment across a range of cancer types and stages in Europe and the USA. The objectives of this research were: 1. to identify the functional health, symptom burden, and HRQoL concepts that impacted by cancer and treatment; 2. to use these findings to determine how appropriate the QLQ-C30 is for the target population; and 3. to evaluate from the patient perspective, the extent to which the domains and items of the QLQ-C30 are understood and perceived as relevant.
      National or local ethical approval was sought as required prior to the conduct of any study activities with patients. All patients provided written informed consent before participating. The study collaborators met biannually at the EORTC Quality of Life Group meetings during the project to discuss progress and emerging results. The Adelphi Values team responsible for conduct of some of the interviews and the full qualitative analysis were not, and are not, members of the EORTC Quality of Life Group (QLG).

      2.1 Patient sample

      Eligible patients were at least 18 years of age with a clinician-confirmed diagnosis of cancer and were receiving cancer treatment for cancer (including surgical treatment within the previous three months). Patients had to be literate and fluent in the local language and willing to participate in an interview. Complete eligibility criteria are provided in Supplementary File 1.
      A purposive sampling approach with recruitment quotas ensured a diverse sample in terms of demographic and clinical characteristics and geographic location. A minimum of 20 patients with each of the four most common types of cancer (breast, lung, prostate and colorectal) and 40 patients with other cancers were targeted for recruitment. Patients were also sampled according to age, disease severity and sex. Patients were recruited from multiple locations in the US, Germany (two sites) and England (two sites), and one site each in Italy, the Netherlands and Poland. Given the relatively small sample size per site, the local investigators aimed to recruit a range of patients across these groups, rather than aiming to meet specific quotas per site.
      In Europe, patients were identified by the local investigator at each recruiting site and approached by a research team member to determine whether they were interested in participating. In the US, patients were identified by a third-party recruitment agency working with selected clinicians/oncologists. The clinicians/oncologists approached potentially relevant patients to determine whether they were interested in participating.
      No record of refusal to participate was recorded, and no patients withdrew from the study after agreeing to participate.
      Adequacy of sample size was assessed through concept saturation [
      • Guest G.
      • Bunce A.
      • Johnson L.
      How many interviews are enough? An experiment with data saturation and variability.
      ]. Once all interviews had been conducted, the interviews were grouped into six sets of 18 or 19 transcripts. The concepts elicited in each group were compared with the next group and saturation was considered achieved if no new concepts relevant to the research question were elicited in the final set of transcripts. All concepts were elicited in the first or second set of interviews.

      2.2 Patient interviews

      A semi-structured interview guide was developed in English, with expert clinical input, and translated into the local language where appropriate. The guide was developed in a way that reduced the potential for confirmation bias (the tendency to elicit or interpret information consistent with existing hypotheses). The interview guide is provided in Supplementary File 2.
      In Europe, interviews were conducted face-to-face by the local investigator or research team member. Interviewers underwent a comprehensive study briefing and were advised to cover as much of the interview guide as possible. In the US, interviews were conducted via telephone [
      • Mazar I.
      • Lamoureux R.
      • Ojo O.
      • et al.
      Telephone versus face-to-face interviews for patient-reported outcome instrument development.
      ] by research team members (JW, EE) who are experienced qualitative researchers based in the UK. All interviews were conducted in the local language and were intended to last approximately 60 min.
      The interview started with open-ended concept elicitation to promote spontaneous discussion about disease- and treatment-related symptoms and how those symptoms impacted patients' lives. Subsequently, there was targeted (probed) discussion about the 15 QLQ-C30 domains to ensure all concepts included in the instrument were explored in detail. Finally, a structured cognitive debriefing of the paper version of the QLQ-C30 explored patients’ understanding of each item, the relevance of the item to their illness experience and appropriateness of the recall period, instructions and response options. All interviews were audio-recorded.

      2.3 Qualitative analysis

      Content validity of the QLQ-C30 was assessed through qualitative analysis of the interview data. Audio files were transcribed verbatim, de-identified and translated to English where relevant. Transcript analysis was first conducted by the local investigator at each study site using a thematic framework approach, which is used to classify and organise data according to key themes, concepts and emergent categories [
      • Gale N.K.
      • Heath G.
      • Cameron E.
      • Rashid S.
      • Redwood S.
      Using the framework method for the analysis of qualitative data in multi-disciplinary health research.
      ]. The framework consisted of an excel spreadsheet listing the existing QLQ-C30 item concepts and items. Analysis was based on the existing QLQ-C30 conceptual framework, which grouped items into concepts and was used as a basis for structuring the interview guide. To map issues identified during the interviews onto the concepts and items in the framework, interview quotes relating to each concept or item were placed in the relevant section. Symptoms were grouped at concept level, which related to the existing QLQ-C30 conceptual framework and then sub-concepts were added based on information elicited from the interviews. New emerging concepts not included in the QLQ-C30 were added to the framework. Research team members (JW, EE, CF) collated the locally completed frameworks. To ensure the framework was consistently and appropriately interpreted, central quality control was conducted by JW, EE and CF whereby an initial three transcripts were cross-checked with the framework for the accuracy of completion. Further transcripts were checked if the initial check identified inaccuracies or inconsistencies. Updates were made as required in consultation with the local site. Finally, the number of patients who reported each issue was counted.
      When analysing the concept elicitation data, a concept was considered elicited if the patient stated they personally experienced it. In the conceptual mapping exercise, concepts elicited by at least 10 patients were reviewed to determine if they are assessed in the QLQ-C30 or could be considered a sub-concept of a QLQ-C30 concept.
      An item or instruction was considered understood if the patient supplied explicit verbal evidence of understanding, including rewording the item or instruction in their own words, supplying an appropriate example or coherently answering a follow-up probe.
      An item was considered relevant if the patients indicated they would select ‘A little’, ‘Quite a bit’ or ‘Very much’ from the response scale, or explicitly stated they had experienced the concept assessed by the item since their cancer diagnosis. Sub-group analyses were conducted by cancer site and disease stage to determine if each item concept was relevant across these sub-groups. An item was deemed relevant to a sub-group if one or more patients from that group reported the item was relevant to their experience.
      Recall period was considered sufficiently understood if the patient reported they were thinking back to the appropriate timeframe. For most items, this was during the past week, but for the physical functioning domain items, current ability is reported (e.g. ‘Do you have any trouble taking a long walk?’). Adequacy of the response scale was assessed by asking patients how easy or difficult it was to select a response using the response scale. Patients were also asked for their opinion on the length of the QLQ-C30.

      3. Results

      3.1 Patient sample

      113 interviews were conducted, and interview length ranged from 20 to 90 min. The sample had approximately equal proportions of patients with a primary cancer diagnosis of breast (n = 19, 17%), lung (n = 19, 17%), prostate (n = 19, 17%) and colorectal (n = 15, 13%) cancer. The sample included a proportion of patients with haematological cancer (n = 12, 11%), skin cancer (n = 8, 7%) and other cancer types/sites (n = 21, 19%) such as head and neck, and ovarian.
      Patients' ages ranged from 23 to 89 years (mean 63.5 years), and there were a similar number of males (n = 62, 55%) and females (n = 51, 45%) in the sample. Most patients had been diagnosed with cancer for one year or less (n = 79, 70%) and were approximately equally spread across cancer stage: metastatic (n = 43, 38%), locally advanced (n = 37, 33%) and localised (n = 28, 25%). Most patients had an Eastern Cooperative Oncology Group status of 0 or 1 (n = 92, 81%) and over half of the patients had a chronic comorbid condition in addition to cancer (n = 71, 63%). Information relating to treatments was collected from clinicians, with a focus on clinical treatment types rather than supportive treatments. Further clinical characteristics of the patient sample can be found in Table 1. Patients' current active treatment was mainly systemic therapy (single agent or combination therapy; n = 71, 63%). Details of the sample's current and past treatments can be found in Table 2.
      Table 1Patient demographic and clinical characteristics (N = 113).
      DescriptionPoland (n = 5)Italy (n = 10)Netherlands (n = 16)Germany (n = 18)US (n = 28)UK (n = 36)Total (n = 113)
      Age, average (range)63.2 (53–72)62.9 (23–80)60.9 (30–80)61.4 (24–79)62.1 (42–89)66.8 (45–84)63.5 (23–89)
      Sex, n (%)
       Male4 (80)6 (60)9 (56)11 (61)7 (25)25 (69)62 (55)
       Female1 (20)4 (40)7 (44)7 (39)21 (75)11 (31)51 (45)
      Primary cancer diagnosis, n (%)
       Breast1 (20)2 (20)3 (19)1 (6)4 (14)8 (22)19 (17)
       Lung003 (19)5 (28)5 (18)6 (17)19 (17)
       Prostate3 (60)1 (10)3 (19)05 (18)7 (19)19 (17)
       Colorectal1 (20)1 (10)2 (13)05 (18)6 (17)15 (13)
      Haematological
      Multiple myeloma0004 (22)2 (7)06 (5)
      (Non) Hodgkin lymphoma01 (10)1 (6)02 (7)04 (4)
      Hodgkin lymphoma001 (6)1 (6)002 (2)
      Skin
      Melanoma02 (20)0004 (11)6 (5)
      Other Skin00002 (7)02 (2)
      Head and Neck01 (10)03 (17)004 (4)
       Ovarian001 (6)02 (7)03 (3)
       Oesophageal000003 (8)3 (3)
       Cervical001 (6)01 (4)02 (2)
       Pancreatic000002 (6)2 (2)
       Bone01 (10)01 (6)002 (2)
      Other cancer
      Salivary gland (adenoid cystic carcinoma)01 (10)00001 (1)
      Cancer of unknown primary (CUP)0001 (6)001 (1)
      Leiomyosarcoma of the scrotum0001 (6)001 (1)
      Adrenal carcinoma0001 (6)001 (1)
      Cardia (stomach) carcinoma001 (6)0001 (1)
      Time since cancer diagnosis, n (%)
       0–6 months4 (80)2 (20)3 (19)6 (33)15 (54)18 (50)48 (43)
       7 months to 1 year1 (20)4 (40)8 (50)1 (6)4 (14)13 (36)31 (27)
       2–5 years02 (20)2 (13)8 (44)4 (14)2 (6)18 (16)
       6+ years02 (20)3 (19)3 (17)5 (18)3 (8)16 (14)
      Current disease stage, n (%)
       Metastatic1 (20)4 (40)11 (69)9 (50)7 (25)11 (31)43 (38)
       Locally advanced3 (60)5 (50)3 (19)2 (11)10 (36)14 (39)37 (33)
       Localised1 (20)1 (10)2 (13)3 (17)10 (36)11 (31)28 (25)
       Remission00001 (4)01 (1)
       Missing Data/Not applicable0004 (22)004 (4)
      ECOG status, n (%)
       01 (20)1 (10)7 (44)010 (36)20 (56)39 (35)
       12 (40)5 (50)3 (19)16 (89)14 (50)13 (36)53 (47)
       21 (20)2 (20)02 (11)3 (11)3 (8)11 (10)
       31 (20)2 (20)6 (38)01 (4)010 (9)
       40000000
      Table 2Patient current and past treatments (N = 113).
      DescriptionPolandItalyNetherlandsGermanyUSUKTotal
      (n = 5)(n = 10)(n = 16)(n = 18)(n = 28)(n = 36)(n = 113)
      Current active treatment, n (%)
       Surgery3 (60)4 (40)003 (11)010 (9)
       Radiation2 (40)003 (17)3 (11)4 (11)12 (11)
       Systemic therapy–single agent03 (30)4 (25)5 (28)18 (64)19 (53)49 (43)
       Systemic therapy–combination1 (20)3 (30)7 (44)5 (28)6 (21)13 (36)35 (31)
       Other treatment
      Immunotherapy (unspecified)001 (6)0001 (1)
      Immunotherapy (Pembrolizumab)000003 (8)3 (3)
      Palbociclib and Fulvestrant00001 (4)01 (1)
      Targeted therapy (Dabrafenib/Trametinib)000001 (3)1 (1)
      Treatment received in the past, n (%)
       Surgery4 (80)7 (70)7 (44)9 (50)14 (50)17 (47)58 (51)
       Systemic therapy–combination05 (50)9 (56)10 (56)4 (14)5 (14)33 (29)
       Radiation1 (20)4 (40)3 (19)5 (28)11 (39)5 (14)29 (26)
       Systemic therapy–single agent01 (10)2 (13)5 (28)5 (18)6 (17)19 (17)
       Hormonal00001 (4)7 (19)8 (7)
       Other treatment
      Ibandrovate and Vexemetasane000001 (3)1 (1)
      Stem cell transplant00002 (7)02 (2)

      3.2 Patient interview findings

      All concepts reported by 10 or more patients were reported within the first two sets of transcripts. Data are supportive of concept saturation having been achieved across the total sample.

      3.2.1 Core concepts associated with HRQoL of cancer patients and QLQ-C30 concept mapping

      Findings from 112 interviews (the concept elicitation recording for one patient became corrupted) are presented. Patients discussed their experience of cancer, including symptoms, treatment side-effects and impacts on HRQoL. Concepts discussed by at least three patients are illustrated in Fig. 1. A further 26 concepts were spontaneously reported by one or two patients each.
      Fig. 1
      Fig. 1Concepts reported by ≥ 3 patients in concept elicitation interviews (N = 112).
      Concept mapping to the QLQ-C30 showed that the 13 concepts most frequently elicited are covered by the QLQ-C30: feeling tired (n = 99), pain (n = 94), impact on sleep (n = 86), need for rest (n = 73), impact on hobbies and leisure activities (n = 78), impact on activities around the house (n = 56), nausea (n = 62), ability to walk long distances (n = 79), impact on appetite (n = 79), worry (n = 75), weakness (n = 65), diarrhoea (n = 60) and impact on work (n = 51). New concepts reported by 10 or more patients that were considered distinct from those included in the QLQ-C30 included climbing steps or stairs (n = 37), weight loss (n = 20), skin problems (n = 19), numbness (n = 15), issues with urination (n = 13), hair loss or thinning (n = 12), cough or phlegm (n = 12), swelling (n = 12), dizziness or faintness (n = 10), weight gain (n = 10) and feeling more emotional (n = 10). Impact on sexual functioning was reported by seven patients.
      Concepts reported by fewer than three patients and conceptual mapping are provided in Supplementary File 3.
      Patients were asked if there were any concepts or items missing from the questionnaire. Only seven concepts were identified as missing by a maximum of four patients each (impact on sexual functioning, issues with urination, treatment side effects, relationship with healthcare professionals, dizziness and swelling).

      3.2.2 Cognitive debriefing

      Patients (N = 112; debriefing data not available for one patient who chose to end the interview early) shared their feedback on the QLQ-C30. Due to various factors (e.g. time constraints), not all patients were asked all debriefing questions. Findings are reported out of the number of patients who were asked each debriefing question.

      3.2.3 Item relevance

      Item relevance ranged from 45% (vomiting) to 96% (tired), with an average of 78% and median of 80%. The proportion of patients reporting each item as relevant to their experience is detailed in Fig. 2.
      Fig. 2
      Fig. 2Proportion of patients reporting QLQ-C30 items as relevant to their experience of cancer (N = 112).
      Sub-group analysis to examine the relevance of QLQ-C30 items across the largest cancer-site groups (breast, lung, prostate, colorectal, haematological and skin) demonstrated that all but three items were relevant across all sub-groups, with at least one patient from each sub-group indicating the item was relevant. Items assessing lack of appetite, vomiting and diarrhoea were found to be relevant in all groups except skin cancer. However, while no patients with skin cancer reported that these QLQ-C30 items were relevant in the cognitive debriefing section, a small number of these patients reported during the initial concept elicitation discussion that they had experienced a lack of appetite (n = 2) and diarrhoea (n = 1), confirming the relevance of all but the concept of vomiting in this population. Sub-group analysis by disease stage (localised, locally advanced and metastatic) demonstrated that all QLQ-C30 items were relevant to patients across all sub-groups (relevant to between 23% and 92% of patients in each group).

      3.2.4 Item understanding

      For each QLQ-C30 item, at least 90% of patients asked demonstrated understanding. A similar proportion of patients demonstrated understanding across the language versions tested.

      3.2.5 General findings

      Almost all patients asked (n = 54/57, 95%) demonstrated understanding of the QLQ-C30 instructions, providing evidence that the instructions are sufficiently clearly worded and appropriate. No patients suggested rewording the instructions.
      Regarding the recall periods, items 1 to 5 (the physical functioning domain) have no specified recall period. Items 6 to 30 specify a seven-day recall period. Over half of the patients asked (n = 60/104, 58%) indicated that they adhered to the recall period, while 37% (n = 38/104) indicated they did not adhere to the recall period. For six patients asked, it was unclear whether they were adherent to the recall period due to the lack of information available.
      Most patients who were asked (n = 94/107, 88%) reported that it was easy to select responses to the QLQ-C30 items. A small number of patients (n = 12/107, 11%) reported that it was somewhat difficult to select responses. The reasons provided were in relation to item content, item wording or finding it difficult to choose a response.
      Most patients asked (n = 89/102, 87%) indicated that the length of the questionnaire is appropriate. A small number of patients (n = 8/102, 8%) responded with respect to the duration of the interview rather than the questionnaire. Two patients (n = 2/102, 2%) felt the questionnaire was too long, saying they would have preferred it to consist of 15–20 items. One patient felt some items were repetitive and another felt some questions could be merged. Finally, one patient felt the questionnaire length was ‘just about right’ but reported struggling to think of answers towards the end.

      4. Discussion

      Qualitative interview results from 113 patients with cancer from Europe and the US showed that concepts included in the QLQ-C30 are widely understood across language versions, and that existing items are relevant to patients across cancer types and disease stages. In this study sample, the 13 most frequently, spontaneously elicited concepts were already covered by the QLQ-C30 conceptual framework.
      The use of PROMs such as the QLQ-C30 in oncology clinical trials is well-established and there is growing evidence for their utility in enhancing clinical practice [
      • Basch E.
      • Barbera L.
      • Kerrigan C.L.
      • Velikova G.
      Implementation of patient-reported outcomes in routine medical care.
      ,
      • Campbell R.
      • Ju A.
      • King M.T.
      • Rutherford C.
      Perceived benefits and limitations of using patient-reported outcome measures in clinical practice with individual patients: a systematic review of qualitative studies.
      ]. Using PROMs for routine monitoring has been shown to improve communication, clinician awareness of symptoms and symptom management, alongside improving quality of life and even survival [
      • Basch E.
      • Barbera L.
      • Kerrigan C.L.
      • Velikova G.
      Implementation of patient-reported outcomes in routine medical care.
      ,
      • Basch E.
      • Deal A.M.
      • Dueck A.C.
      • et al.
      Overall survival results of a trial assessing patient-reported outcomes for symptom monitoring during routine cancer treatment.
      ,
      • Basch E.
      • Deal A.M.
      • Kris M.G.
      • et al.
      Symptom monitoring with patient-reported outcomes during routine cancer treatment: a randomized controlled trial.
      ,
      • Denis F.
      • Basch E.
      • Septans A.-L.
      • et al.
      Two-year survival comparing web-based symptom monitoring vs routine surveillance following treatment for lung cancer.
      ,
      • Gilbert A.
      • Sebag-Montefiore D.
      • Davidson S.
      • Velikova G.
      Use of patient-reported outcomes to measure symptoms and health related quality of life in the clinic.
      ,
      • Kotronoulas G.
      • Kearney N.
      • Maguire R.
      • et al.
      What is the value of the routine use of patient-reported outcome measures toward improvement of patient outcomes, processes of care, and health service outcomes in cancer care? A systematic review of controlled trials.
      ]. To remain credible and support medical product evaluation by regulators, PROMs need to show evidence that they comprehensively cover concepts that are important and relevant to the target patient population, known as content validity [
      • Patrick D.L.
      • Burke L.B.
      • Gwaltney C.J.
      • et al.
      Content validity—establishing and reporting the evidence in newly developed patient-reported outcomes (PRO) instruments for medical product evaluation: ISPOR PRO Good Research Practices Task Force report: part 2—assessing respondent understanding.
      ].
      Guidelines for establishing content validity are focussed on eliciting concepts from patients via interviews and assessing patient understanding of the developed PROM using cognitive debriefing interviews. Although the QLQ-C30 has been extensively used in patients with cancer for almost 30 years, it is pertinent to evaluate content validity using current recommended methods, ensuring the concepts assessed are important and relevant to patients with cancer today and supplying evidence documenting this. Due to the widespread use of the QLQ-C30, recommending changes could have been challenging yet would have been necessary to confirm content validity.
      It is important to recall that the EORTC measurement model recommends the QLQ-C30 be used alongside disease- or symptom-specific modules or standalone questionnaires to cover additional concepts important and relevant to specific populations. Items from the EORTC Item Library can be added if required concepts are not covered by the selected questionnaires [
      • Kulis D.
      • Bottomley A.
      • Whittaker C.
      • et al.
      The use of the EORTC item library to supplement EORTC quality of life instruments.
      ].
      A short list of concepts that were considered distinct from those included in the QLQ-C30 was reported by a small proportion of patients in the open-ended concept elicitation discussion. These concepts generally fit within existing QLQ-C30 conceptual domains (e.g. additional symptoms or side-effects and further impacts on emotional functioning), except for impact on sexual functioning that is not currently covered within the QLQ-C30. If required, the EORTC suite of resources can be used alongside the QLQ-C30 to support the assessment of these concepts. Since the QLQ-C30 needs to strike a balance between patient burden and ensuring relevance across a range of patients with cancer, no new generic items were recommended for inclusion but supplementing the core questionnaire with additional items or scales may be an appropriate strategy depending on likely importance and relevance in the specific target population.
      In addition to the findings from this study, the QLQ-C30 encompasses 11 of the 12 concepts recommended by Reeve et al. as the core set of symptoms to measure in adult oncology clinical trials [
      • Reeve B.B.
      • Mitchell S.A.
      • Dueck A.C.
      • et al.
      Recommended patient-reported core set of symptoms to measure in adult cancer treatment trials.
      ]. The remaining concept, neuropathy (referred to as numbness in this study), was reported by a small proportion of the sample (n = 15, 13%) and was not considered prevalent enough to warrant adding the concept to the QLQ-C30. For specific populations where this concept may be more relevant, the EORTC measurement model allows for the inclusion of a chemotherapy-induced peripheral neuropathy module, the QLQ-CIPN20 alongside the QLQ-C30 [
      • Kieffer J.M.
      • Postma T.J.
      • van de Poll-Franse L.
      • et al.
      Evaluation of the psychometric properties of the EORTC chemotherapy-induced peripheral neuropathy questionnaire (QLQ-CIPN20).
      ], or for the selection of one or more items from the Item Library.
      To assess the relevance of items across cancer types, whether a concept was elicited within that patient subgroup was evaluated. All QLQ-C30 concepts were relevant in all but skin cancer, in which vomiting was not part of the patients’ disease experience (though nausea was relevant). Though we had a small threshold for the relevance of at least one patient within the group indicating an item/concept as relevant, given the research question asks for a binary response (is the item relevant in this group or not?), this threshold was deemed suitable. No QLQ-C30 concepts were deemed irrelevant. The item assessing vomiting had the lowest proportion of patients reporting relevance, yet nearly 50% of patients across the sample endorsed this item. This may in part reflect the sample characteristics and their current treatments, rather than indicating it as a symptom that does not warrant inclusion.
      Reported adherence to recall periods was acceptable, though adding the recall period stated alongside each item could be considered to improve clarity. This aligns with how the questionnaire may be delivered electronically. Recent research highlighted that for the PROMIS physical function scores, the recall period (no recall period versus 24 h versus 7 days) did not impact scores [
      • Condon D.M.
      • Chapman R.
      • Shaunfield S.
      • et al.
      Does recall period matter? Comparing PROMIS® physical function with no recall, 24-hr recall, and 7-day recall.
      ]; therefore, the variable adherence to recall period in our study was not considered a concern for content validity.

      5. Limitations

      Although the study was multi-national and multi-lingual, there was a high proportion of English-speaking patients in the sample. Further, US patients were recruited and interviewed differently to the European patients, which may have led to differences between the US and European samples. The US patients were interviewed via telephone, whereas patients from all other countries were interviewed face-to-face. However, there is little evidence of difference in data quality when conducting interviews in these different ways [
      • Mazar I.
      • Lamoureux R.
      • Ojo O.
      • et al.
      Telephone versus face-to-face interviews for patient-reported outcome instrument development.
      ].
      As this is a qualitative study, it reflects the specific views of the patients interviewed and may not be fully representative of, or transferable to, the target patient population. A typical cancer population would consist of many different cancer sites and stages, but to capture this in a study, sample would require a much larger sample size and a more targeted recruitment approach than is typically feasible for a qualitative study. Any sampling strategy would struggle to achieve complete representation across all possible patient and disease characteristics, though in this study, many groups were represented and the consistency in findings between groups is supportive of the EORTC model to administer the QLQ-C30 with a disease/treatment module. As is common for studies in patients with cancer, younger patients and those with more limitations (Eastern Cooperative Oncology Group scores of 2 or more) may be under-represented. Additionally, few participants relative to the total sample were reported to have been previously or currently undergoing immunotherapy or targeted therapies. However, these newer therapies were available to patients at study recruitment sites and, in some instances, were standard of care. In our study, patients on these therapies were most often categorised on the case report form as being on systemic therapy; as such, the treatment name or type was not captured. Despite these limitations, the study comprised a large sample of international patients, including the most common cancer sites and widely used treatments.

      6. Conclusions

      Overall, this study demonstrates that the existing QLQ-C30 conceptual framework is appropriate in a large, international patient sample with different cancer sites and stages. The questionnaire items and subscales are relevant and important to patients across a range of cancers, disease stages and treatments. The EORTC measurement model recommends using the QLQ-C30 alongside supplementary disease- or symptom-specific modules, standalone questionnaires or items from the Item Library. The QLQ-C30 demonstrates good evidence of content validity for use with patients with localised-to-advanced cancer.

      Funding

      This work was supported by the EORTC Quality of Life Group [Grant number 006-2015].

      Data sharing

      The study protocol and aggregated and anonymised data that underlie the results reported in this article will be available immediately following publication with no end date to researchers who provide a methodologically-sound proposal to achieve aims related to the approved proposal. Proposals should be submitted to www.eortc.be/services/forms/erp/request.aspx. To gain access, data requestors will need to sign a data access agreement.

      Author contributions

      Author Kim Cocks contributed through study conceptualisation, funding acquisition, investigation, methodology, analysis, and writing the original draft manuscript.
      Author Jane R Wells contributed through data curation, formal analysis, methodology, project administration, validation, and writing the original draft manuscript.
      Author Colin Johnson contributed through study conceptualisation, funding acquisition, investigation, methodology, and review and editing the manuscript.
      Authors Michael Koller, Heike Schmidt, Simone Oerlemans, Galina Velikova, Monica Pinto, and Krzysztof Tomaszewski contributed through investigation, methodology, and review and editing the manuscript.
      Authors Elizabeth Exall and Chelsea Finbow contributed through data curation, formal analysis, project administration, validation, and writing the original draft manuscript.
      Author Laura Grant contributed through methodology, project administration, and review and editing the manuscript.
      Authors Neil Aaronson, Deborah Fitzsimmons, Mogens Groenvold, and Sally Wheelwright contributed through study conceptualisation, methodology, and review and editing the manuscript.
      Author Chloe Tolley contributed through study conceptualisation, funding acquisition, investigation, methodology, and review and editing the manuscript.
      Author Andrew Bottomley contributed through study conceptualisation, methodology, funding acquisition, and review and editing the manuscript.
      All authors meet all four criteria for authorship according to the ICMJE recommendations, and confirm they have full access to the study data and accept responsibility to submit for publication.

      Conflict of interest statement

      The authors declare the following financial interests/personal relationships which may be considered as potential competing interests:
      The following authors declare no competing interests: Andrew Bottomley, Neil K Aaronson, Deborah Fitzsimmons, Mogens Groenvold, and Sally Wheelwright.
      Colin Johnson, Krzysztof Tomaszewski, Michael Koller, Monica Pinto, Simone Oerlemans, and Heike Schmidt's respective institutions were provided funding from the EORTC Quality of Life Group to conduct the research described in this manuscript.
      Jane R Wells, Chelsea Finbow, Elizabeth Exall, Chloe Tolley, and Laura Grant are or were at the time of the study employees of Adelphi Values Ltd. Adelphi Values Ltd was provided a grant from the EORTC Quality of Life Group to conduct the research described in this manuscript and to develop this manuscript. Adelphi Values Ltd provides consultancy for a variety of pharmaceutical companies.
      Kim Cocks is an employee of Adelphi Values Ltd. However, the role of Principal Investigator on this study was not funded through the EORTC Quality of Life Group grant. Kim Cocks has received consulting fees from Endomag Ltd.
      Galina Velikova's institution was provided funding from the EORTC Quality of Life Group to conduct the research described in this manuscript and other research. Galina Velikova also declares that her institution has grants/contracts with Breast Cancer Now, Pfizer, and IQVIA. Galina Velikova discloses relationships with Novartis, Eisai, Seattle Genetics, Sanofi Advisory Board, and Roche Ester Steering Committee.

      Appendix A. Supplementary data

      The following are the Supplementary data to this article:

      References

        • Aaronson N.K.
        • Ahmedzai S.
        • Bergman B.
        • et al.
        The European Organisation for Research and Treatment of Cancer QLQ-C30: a quality-of-life instrument for use in international clinical trials in oncology.
        J Natl Cancer Inst. 1993; 85: 365-376
        • Bjordal K.
        • De Graeff A.
        • Fayers P.
        • et al.
        A 12 country field study of the EORTC QLQ-C30 (version 3.0) and the head and neck cancer specific module (EORTC QLQ-H&N35) in head and neck patients.
        Eur J Cancer. 2000; 36: 1796-1807
        • Bjelic-Radisic V.
        • Cardoso F.
        • Cameron D.
        • et al.
        An international update of the EORTC questionnaire for assessing quality of life in breast cancer patients: EORTC QLQ-BR45.
        Ann Oncol. 2020; 31: 283-288
        • Aaronson N.
        • Bullinger M.
        • Ahmedzai S.
        A modular approach to quality-of-life assessment in cancer clinical trials.
        in: Cancer clinical trials. Springer, 1988: 231-249
        • Smith A.B.
        • Cocks K.
        Content validity and legacy patient-reported outcome measures in cancer.
        Qual Life Res. 2015; 24: 1585-1586
        • Aaronson N.
        • Ahmedzai S.
        • Bullinger M.
        • et al.
        The EORTC core quality of life questionnaire: interim results of an international field study.
        Eff Cancer Qual Life. 1991; : 185-203
        • Osoba D.
        • Aaronson N.
        • Zee B.
        • Sprangers M.
        • Te Velde A.
        Modification of the EORTC QLQ-C30 (version 2.0) based on content validity and reliability testing in large samples of patients with cancer.
        Qual Life Res. 1997; 6 (0-0)
        • Haynes S.N.
        • Richard D.
        • Kubany E.S.
        Content validity in psychological assessment: a functional approach to concepts and methods.
        Psychol Assess. 1995; 7: 238
        • US Food and Drug Administration
        Guidance for Industry: patient-reported outcome measures: use in medical product development to support labeling claims.
        2009
        • Patrick D.L.
        • Burke L.B.
        • Gwaltney C.J.
        • et al.
        Content validity—establishing and reporting the evidence in newly developed patient-reported outcomes (PRO) instruments for medical product evaluation: ISPOR PRO Good Research Practices Task Force report: part 2—assessing respondent understanding.
        Value Health. 2011; 14: 978-988
        • Wheelwright Sally
        • Bjordal Kristen
        • Bottomley Andrew
        • et al.
        • EORTC Quality of Life Group
        EORTC Quality of Life Group guidelines for developing questionnaire modules. 5th. EORTC, 2021
        • Guest G.
        • Bunce A.
        • Johnson L.
        How many interviews are enough? An experiment with data saturation and variability.
        Field Methods. 2006; 18: 59-82
        • Mazar I.
        • Lamoureux R.
        • Ojo O.
        • et al.
        Telephone versus face-to-face interviews for patient-reported outcome instrument development.
        Value Health. 2015; 18: A718
        • Gale N.K.
        • Heath G.
        • Cameron E.
        • Rashid S.
        • Redwood S.
        Using the framework method for the analysis of qualitative data in multi-disciplinary health research.
        BMC Med Res Methodol. 2013; 13: 1-8
        • Basch E.
        • Barbera L.
        • Kerrigan C.L.
        • Velikova G.
        Implementation of patient-reported outcomes in routine medical care.
        Am Soc Clin Oncol Educ Book. 2018; vol. 38: 122-134
        • Campbell R.
        • Ju A.
        • King M.T.
        • Rutherford C.
        Perceived benefits and limitations of using patient-reported outcome measures in clinical practice with individual patients: a systematic review of qualitative studies.
        Qual Life Res. 2021; : 1-24
        • Basch E.
        • Deal A.M.
        • Dueck A.C.
        • et al.
        Overall survival results of a trial assessing patient-reported outcomes for symptom monitoring during routine cancer treatment.
        JAMA. 2017; 318: 197-198
        • Basch E.
        • Deal A.M.
        • Kris M.G.
        • et al.
        Symptom monitoring with patient-reported outcomes during routine cancer treatment: a randomized controlled trial.
        J Clin Oncol. 2016; 34: 557
        • Denis F.
        • Basch E.
        • Septans A.-L.
        • et al.
        Two-year survival comparing web-based symptom monitoring vs routine surveillance following treatment for lung cancer.
        JAMA. 2019; 321: 306-307
        • Gilbert A.
        • Sebag-Montefiore D.
        • Davidson S.
        • Velikova G.
        Use of patient-reported outcomes to measure symptoms and health related quality of life in the clinic.
        Gynecol Oncol. 2015; 136: 429-439
        • Kotronoulas G.
        • Kearney N.
        • Maguire R.
        • et al.
        What is the value of the routine use of patient-reported outcome measures toward improvement of patient outcomes, processes of care, and health service outcomes in cancer care? A systematic review of controlled trials.
        J Clin Oncol. 2014; 32: 1480-1510
        • Kulis D.
        • Bottomley A.
        • Whittaker C.
        • et al.
        The use of the EORTC item library to supplement EORTC quality of life instruments.
        Value Health. 2017; 20: A775
        • Reeve B.B.
        • Mitchell S.A.
        • Dueck A.C.
        • et al.
        Recommended patient-reported core set of symptoms to measure in adult cancer treatment trials.
        JNCI: J Natl Cancer Inst. 2014; 106: dju129
        • Kieffer J.M.
        • Postma T.J.
        • van de Poll-Franse L.
        • et al.
        Evaluation of the psychometric properties of the EORTC chemotherapy-induced peripheral neuropathy questionnaire (QLQ-CIPN20).
        Qual Life Res. 2017; 26: 2999-3010
        • Condon D.M.
        • Chapman R.
        • Shaunfield S.
        • et al.
        Does recall period matter? Comparing PROMIS® physical function with no recall, 24-hr recall, and 7-day recall.
        Qual Life Res. 2020; 29: 745-753