- Research article
- Open Access
- Open Peer Review
Using the Influenza Patient-reported Outcome (FLU-PRO) diary to evaluate symptoms of influenza viral infection in a healthy human challenge model
© The Author(s). 2018
- Received: 5 December 2017
- Accepted: 26 June 2018
- Published: 28 July 2018
In clinical studies involving a healthy volunteer human challenge model, a valid and reliable measure to assess the evolution of patient-reported symptom type and severity following viral exposure is necessary. This study examines the use of the InFLUenza Patient-Reported Outcome (FLU-PRO) diary as a standardized measure of symptom severity in a healthy volunteer human challenge model.
Healthy adults admitted to the NIH Clinical Center (Day − 1) underwent a 9-day inpatient quarantine after intranasal challenge with a wild-type influenza A/H1N1pdm virus (Day 0). Participants completed the 32-item FLU-PRO diary twice daily for 14 days to assess presence, severity, and duration of symptoms across six body systems. Secondary analyses included descriptive statistics to examine FLU-PRO scores over the course of illness and analysis of variance to compare scores on Day 3 post-challenge by presence of viral shedding, and pre-challenge hemagglutinin and neuraminidase inhibition (HAI and NAI) titers.
All but one subject (99%), who was lost to follow-up, completed twice daily FLU-PRO diaries on all study assessment days. FLU-PRO demonstrated that 61 of 65 subjects reported symptoms (Days: Median 5, Mean 6 ± 7), of whom 37 (61%) had viral shedding. Pre-challenge, 39 (64%) and 10 (16%) subjects had low (< 1:40) HAI and NAI titers, respectively. Nose, throat, body, and gastrointestinal (GI) symptoms reached peak intensity at Day 3, followed by chest/respiratory and eye symptoms at Day 4. Subjects with viral shedding had higher mean FLU-PRO scores compared to those without, except for Eye and GI domains (p <0.05). Mean FLU-PRO scores were significantly higher for subjects with low NAI titer (p <0.05) across all domains. No significant differences were observed between HAI titer groups. FLU-PRO scores of the low HAI-low NAI group (n = 10) were significantly higher (more severe) than the other two groups (p < 0.05) (high HAI-high NAI (n = 22), low HAI-high NAI (n = 29)).
The FLU-PRO had high adherence and low respondent burden. It can be used to track symptom onset, intensity, duration, and recovery from influenza infection in clinical research. In this human challenge study, scores were responsive to change and distinguished known clinical subgroups.
NCT01971255 First Registered October 2, 2013.
- Human challenge study
Seasonal influenza is a global public health problem that ranges in severity and can result in hospitalization and death, particularly in high risk groups . Globally, it is estimated that annual seasonal influenza epidemics result in 3 to 5 million cases of severe illness, and 290,000 to 650,000 deaths . This has a significant economic impact, with total economic burden due to annual influenza epidemics estimated at $87.1 billion in the U.S. alone .
Healthy volunteer human challenge studies are used to study the pathogenesis of influenza infection, understand the immune response, and gather information on proof-of-concept during the early stages of vaccine or therapeutic development. Although there are no U.S. Food & Drug Administration (FDA) guidelines for defining efficacy endpoints specifically for challenge studies, the FDA guidance for industry “Influenza: Developing drugs for treatment and/or prophylaxis” suggest an efficacy endpoint for influenza drug development trials that would include a combination of measures including patient-reported symptoms . In addition, there are guidance documents to inform the development, selection and use of patient-reported outcome (PRO) measures for use in drug development clinical trials  as one way of fostering patient-centered drug development, and properly developed and evaluated PRO instruments provide a unique and important endpoint measure in clinical trials . The InFLUenza Patient-Reported Outcome (FLU-PRO©) measure was developed using methods consistent with these guidelines [6, 7]. The FLU-PRO is a comprehensive, 32-item daily diary that provides a direct measure of the presence and severity of patient-reported influenza symptoms across 6 body systems affected by influenza: Nose, Throat, Eyes, Chest/Respiratory, Gastrointestinal, and Body/Systemic based on evidence from patients regarding the symptoms they find important in influenza. The FLU-PRO total score represents the “severity” of symptoms across these systems as measured by the intensity, frequency, and duration of symptoms.
Qualitative evidence of content validity  and data on the reliability, validity, and responsiveness of FLU-PRO scores in hospitalized and non-hospitalized adults with laboratory-confirmed influenza and influenza-like illness have been previously reported, with results suggesting the instrument is a valid measure for evaluating symptoms in these conditions and settings [7, 8].
The FLU-PRO was recently administered in a human influenza challenge study designed to examine the role of hemagglutination inhibition (HAI) and neuraminidase inhibition (NAI) in conferring protection against influenza A virus . In this study, the role of NAI was identified as an important correlate of protection expanding on what has been previously reported [10, 11]. This analysis by Memoli et al.  used the same questions as the finalized FLU-PRO instrument, but used an early draft FLU-PRO scoring method since the final FLU-PRO scoring methods and psychometric evaluation were still under development, and FLU-PRO had not previously been used in a healthy volunteer influenza challenge study. Even with the draft scoring method, FLU-PRO still provided valuable data used in the primary analysis. Preliminary secondary analyses were presented at IDWeek 2017 in San Diego, California . This report describes in more detail the analysis of the FLU-PRO data from the same study using the final validated FLU-PRO scoring method in the setting of a human influenza challenge study. It examines the performance of FLU-PRO scores in a healthy volunteer human challenge model, assessing the evolution of symptom frequency, intensity, and duration to define “severity” from exposure to recovery and the relationship between symptom severity and viral shedding, NAI, and HAI titer.
Post-hoc, secondary analyses were performed on data from healthy volunteers enrolled in an influenza challenge study conducted at the National Institutes of Health (NIH) Clinical Center. Study design and primary results were reported previously and adheres to CONSORT guidelines for reporting clinical trials as applicable . Briefly, the study investigated whether participants with a high pre-challenge serum HAI titer were less likely to develop mild to moderate influenza disease after intranasal inoculation with a wild-type influenza A/H1N1pdm virus compared to participants with a low pre-challenge HAI titer.
After signing written informed consent, 65 healthy men and women between 18 to 40 years of age underwent a 9-day inpatient quarantine and intranasal challenge via a nasal atomizer with an infective dose of H1N1pdm and were then monitored in an outpatient clinic periodically over 2 months. Participants were blinded to viral shedding and antibody titer results. Participants received no antiviral medications during the first 7 days post-inoculation and a small number were given other medications to control symptoms if necessary. Participants were tested at baseline to rule out the presence of other viral pathogens. Participants completed the FLU-PRO questionnaire on paper the evening of Day − 1 and morning of Day 0 (pre-inoculation), evening of Day 0 post-inoculation, and twice daily thereafter to Day 14 post-inoculation and then once more on Day 28. The FLU-PRO is a 32-item daily diary assessing influenza signs and symptoms across 6 body systems: Nose (4 items), Throat (3 items), Eyes (3 items), Chest/Respiratory (7 items), Gastrointestinal (4 items), and Body/Systemic (11 items). Respondents are asked to rate each sign or symptom on a 5-point ordinal scale, with higher scores indicating a more frequent sign or symptom. For 27 of the items, the scale is as follows: 0 (“Not at all”), 1 (“A little bit”), 2 (“Somewhat”), 3 (“Quite a bit”), and 4 (“Very much”). For 5 items, severity is assessed in terms of numerical frequency, i.e., vomiting or diarrhea (0 times, 1 time, 2 times, 3 times, or 4 or more times); with frequency of sneezing, coughing, and coughed up mucus or phlegm evaluated on a scale from 0 (“Never”) to 4 (“Always”).
The FLU-PRO total score is computed as a mean score across all 32 items comprising the instrument. Total scores can range from 0 (symptom free) to 4 (very severe symptoms). Six domain scores are also computed, representing symptom severity in each of the assessed body areas. Each domain score is calculated as the mean of all items comprising the domain, with scores ranging from 0 to 4.
The FLU-PRO has been tested as a once-daily measure . In this study, participants completed the diary twice a day. Using paired t-tests, no significant domain or total score differences were observed between the two pre-inoculation administrations (evening Day − 1 and morning Day 0) or the mornings and evenings of Days 1 through 14 post-inoculation (data not shown). Given no added discrimination of a twice daily administration and to be consistent with current use, a daily mean FLU-PRO score was calculated for each participant, taking the average of scores from the two pre-inoculation administrations, and the morning and evening administration of each day post-inoculation. All subsequent analyses used mean daily scores.
Analyses were performed on symptomatic participants only, with asymptomatic participants defined by a FLU-PRO daily total score of zero at every study assessment (Day − 1 to Day 14, and Day 28). Descriptive statistics were used to characterize demographics and clinical characteristics of the study population.
The distributional characteristics of FLU-PRO domain and total scores, including frequency of symptom occurrence within the sample (n, %) and severity of symptoms (score mean (standard deviation, SD), range, floor, ceiling) were examined pre-inoculation, on Day 3 (peak viral shedding and peak number of symptoms), and on Day 10 post-inoculation (by which time most participants were discharged). These days were selected based on viral shedding and number of clinician-assessed symptoms as reported by Memoli et al. . The trajectory of FLU-PRO domain and total scores from Days − 1 to 10 were also plotted and examined.
Analysis of variance (ANOVA) was used to test the ability of FLU-PRO severity scores to differentiate two or more clinical groups. Specifically, domain and total scores on Day 3 were compared by viral shedding status (shedding vs. no shedding), baseline HAI (high vs. low titers), baseline NAI (high vs. low titers), and the combination of baseline HAI-NAI titers (high HAI-high NAI vs. low HAI-high NAI vs. low HAI-low NAI). Day 3 was selected as it was the day of peak viral shedding and peak number of symptoms.
For this set of analyses, assuming the differences between these known-groups would be similar to the difference between influenza positive patients with no/mild symptoms [Mean = 0.98 (SD 0.47)] and moderate symptoms [1.38 (SD 0.57)] , the power for detecting significant differences between these known-groups was 0.97.
Sample demographic and clinical characteristics (N = 61)
FLU-PRO Symptomatic (N = 61)
Age at inoculation/enrollment, Mean (SD)
Gender, n (%)
Race, n (%)
Ethnicity, n (%)
Hispanic or Latino
Not Hispanic or Latino
HAI Titer, n (%) Low < 1:40
NAI Titer, n (%) Low < 1:40
high HAI-high NAI, n (%)
high HAI-low NAI, n (%)
low HAI-high NAI, n (%)
low HAI-low NAI, n (%)
Viral shedding, n (%) Yes
Number of days with symptoms, Mean (SD)
Number of days with shedding, Mean (SD)
FLU-PRO symptom prevalence
FLU-PRO scores over time
Throughout the duration of the challenge study, across all 61 subjects, mean domain scores were highest for the Nose domain and lowest for the Gastrointestinal domain. The greatest score fluctuations were observed in the Nose domain with the smallest variation in the Gastrointestinal domain. Symptoms comprising the Nose domain were the most frequently endorsed throughout the study, while Gastrointestinal domain symptoms were the least frequently endorsed.
FLU-PRO scores by clinical group
Mean (SE) FLU-PRO domain and total scores on Day 3 by baseline (Day 0) joint HAI/NAI status
FLU-PRO Total/Domain Score
High HAI – High NAIa
Low HAI – High NAIa
Low HAI – Low NAIa
Overall F value (P value)b
LS Mean (SE)
LS Mean (SE)
LS Mean (SE)
2: < 0.0001
3: < 0.0001
2: < 0.0001
3: < 0.0001
This study assessed the ability of the FLU-PRO to quantify the symptoms and severity of influenza in a human challenge study model and to monitor the evolution of symptoms from baseline/pre-challenge through symptom development and resolution. The total score provides an overview of symptom progression and severity overall, while the domain scores show how the severity of symptoms associated with affected body systems vary over time and compare with one another. The symptoms measured by FLU-PRO are due to influenza since other viral pathogens were ruled out at baseline, and participants received minimal symptomatic medications and no antiviral medications during the acute illness.
Mean FLU-PRO scores on peak symptoms day (Day 3) were less severe than values seen in the observational FLU-PRO validation study where naturally-infected subjects were enrolled during a clinic visit, tested for the influenza virus, and recovered at home or in the hospital . The peak mean FLU-PRO total score in naturally-infected patients for the home-recovered group was 1.6 (SD 0.7), with domain scores ranging from 0.6 (SD 0.8) for the GI domain to 1.9 (SD 0.9) for the Body/Systemic domain . Subjects hospitalized with influenza had FLU-PRO total score of 1.4 (SD 0.8), and domain scores ranging from 0.8 (SD 1.0) for the GI domain to 1.9 (SD 0.9) for the Chest/Respiratory domain . The lower scores reported by subjects in the current study may be a function of their youth and health prior to exposure. These subjects had no major comorbidities, while 39% of the natural history study subjects reported one or more comorbidities (Age range: mean age: 40.7 years (SD 16.6), range 18–95 years). Given these challenge participants were younger and healthy, it is likely that their disease represents that of influenza cases that are less likely to be medically attended and therefore not captured in natural history studies. Therefore it is not surprising that the scores were overall lower for these subjects. This provides further support for the sensitivity of the validated FLU-PRO scoring system even among the less symptomatic cases of influenza. In addition, in this human challenge study few concomitant medications were dispensed so resolution of symptoms can be attributed to the natural resolution of influenza infection without the influence of concomitant medications.
Even with the low mean scores/mild symptoms, FLU-PRO total and domain scores were responsive to change from usual (asymptomatic) health to the development of influenza symptoms post-challenge and resolution over time, from Days − 1 to 10. Varied patterns of severity and change were seen across body systems (domains). Applying the FLU-PRO in a challenge setting provides the unique opportunity to better understand symptom onset to resolution since the exact amount and time of exposure are both known. In natural history studies, by the time patients seek care their symptoms may already be quite severe, obviating the ability to capture the early trajectory of their symptoms. In this study of known H1N1pdm exposure, nasal symptoms were more common and more severe, while gastrointestinal symptoms occurred less frequently. Nasal, throat, body/systemic, and gastrointestinal symptoms developed and peaked earlier than chest/respiratory and eye symptoms showing that all symptoms do not occur simultaneously but have an orderly progression over time.
FLU-PRO score differences across viral shedding and both NAI and HAI titer groups offer support for the known-groups validity of FLU-PRO scores , provide evidence that the symptoms measured are due to influenza infection and provide evidence for the utility of this tool for use in challenge studies. Participants with viral shedding reported more severe symptoms of influenza than those with no viral shedding. Those with lower NAI titers reported more severe symptoms than those with high NAI titers. In current vaccine development, efficacy has been predominantly evaluated in terms of protection based on HAI titers. The results of more recent research suggest NAI titers have a role in protection that is independent of HAI titers [10, 11]. Symptoms as assessed by the FLU-PRO lasting longer than peak viral shedding also showed the importance of host immune response to symptoms in influenza, and further demonstrated the challenges of using viral shedding as an outcome to define endpoints in influenza studies. Direct measurement of patient symptoms provides the most relevant evidence of patient benefit.
The primary analyses for this challenge study showed that NAI titer was an independent predictor of protection from influenza, and that HAI and NAI titer, jointly considered, could be a better predictor of protection from influenza than either alone . Results of the analyses presented here suggest patient self-report of symptom severity, i.e., the FLU-PRO, can distinguish HAI/NAI titer groups and may be useful as a proxy measure of influenza protection in a clinical or observational study setting. When considered jointly, scores were also able to distinguish between low HAI-low NAI and high HAI-high NAI, and between low HAI-high NAI and low HAI-low NAI groups, again suggesting its applicability for use in a clinical or observational study setting. However, the evidence suggests that at least in the assessment of patient-reported symptoms severity, NAI titer alone would be a better independent predictor of symptoms severity than HAI titer or joint HAI-NAI titers. In addition, symptom measurement adds to the information obtained by titer measurement in evaluating vaccine efficacy, since the evidence shows that in patients who do become infected, their symptoms may be mitigated with an effective vaccine even when disease is not completely prevented.
The FLU-PRO was easy to administer with low respondent burden. Adherence was 99%. Therefore, the FLU-PRO instrument as constructed is suitable for use in clinical trials and observational studies. Once daily administration was sufficient and twice daily administration did not add to the instrument’s discriminative ability.
This was a secondary analysis of existing data, which is a limitation of this report. In addition, the study was not powered for other tests of instrument performance, such as estimates of internal consistency or test-retest reliability. Due to a limited sample size and individual variability, we compared mean scores on pre-inoculation, Day 3, and Day 10. Future challenge studies with a larger sample size would also include longitudinal analysis of individual responses in addition to the group comparisons we have presented. Furthermore, we did not assess patient global impression of return to usual health to define a “responder criteria” for this study, but this is planned for future challenge studies. It is clear that all symptoms do not have to return to scores of zero in order for patients to assess return to their usual state of health . Nevertheless, this is the first time FLU-PRO with a validated scoring system has been used in a human influenza challenge study and it showed responsiveness and known-groups validity in this setting.
The FLU-PRO is a useful tool with content and construct validity for tracking symptom onset, severity, and recovery from influenza infection in a human challenge study setting. Total and domain scores were responsive to change, with domain scores exhibiting differential patterns of severity and evolution. Differences in symptom severity scores were also observed across known clinical subgroups. Using the FLU-PRO as a standardized measure of influenza symptoms in human challenge studies may facilitate our understanding of the symptomatic evolution of influenza and the effects of new vaccines and drugs to prevent and treat influenza infection.
The authors would like to acknowledge Dr. H. Clifford Lane, Dr. Richard T. Davey, the LID Clinical Studies Unit staff, the NIH Clinical Center Special Clinical Studies Unit staff, and the Department of Laboratory Medicine for their support of the clinical protocol.
This work was supported in part by the intramural funds of the NIH and the NIAID, with support from the Biomedical Advanced Research and Development Authority (BARDA), and with federal funds from the National Cancer Institute, National Institutes of Health, under Contract No. HHSN261200800001E. The content of this publication does not necessarily reflect the views or policies of the Department of Health and Human Services, nor does mention of trade names, commercial products, or organizations imply endorsement by the U.S. Government.
Availability of data and materials
The datasets generated and analyzed during the primary study and secondary analysis are not publicly available due to participant confidentiality, but are available from the corresponding author on reasonable request.
AH analyzed and interpreted the data from the primary analysis and this secondary analysis and was a major contributor to the writing and editing of this manuscript. JLP, NKL, and RY analyzed and interpreted the data and were major contributors to the writing and editing of this manuscript. JLP was at Evidera during the writing and editing of this manuscript and is now at Eli Lilly and Company in Indianapolis, Indiana. NKL also developed and validated the FLU-PRO. JHP developed and validated the FLU-PRO and was a major contributor to the writing and editing of this manuscript. MJM designed and implemented the primary study, analyzed and interpreted the data from the primary and secondary analysis, and was a major contributor to the writing and editing of this manuscript. All authors read and approved the final manuscript.
Ethics approval and consent to participate
This study (ClinicalTrials.gov identifier NCT01971255) was approved by the National Institute of Allergy and Infectious Diseases institutional review board and was conducted in accordance with the provisions of the Declaration of Helsinki and Good Clinical Practice guidelines. Participants signed written informed consent prior to study enrollment.
Consent for publication
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
- World Health Organization. Media centre. Influenza (Seasonal). Fact Sheet. 2016. Available at: http://www.who.int/en/news-room/fact-sheets/detail/influenza-(seasonal). Accessed June 20, 2017.
- Molinari NM, Ortega-Sanchez IR, Messonnier ML, Thompson WW, Wortley PM, Weintraub E, Bridges CBB. The annual impact of seasonal influenza in the US: measuring disease burden and costs. Vaccine. 2007;25(27):5086–96.View ArticlePubMedGoogle Scholar
- Food and Drug Administration. Guidance for industry on influenza: developing drugs for treatment and/or prophylaxis. Fed Regist. 2011;76(71):20689–90.Google Scholar
- Food and Drug Administration. Guidance for industry on patient-reported outcome measures: use in medical product development to support labeling claims. Fed Regist. 2009;74(235):65132–3.Google Scholar
- Patrick DL, Burke LB, Powers JH, et al. Patient-reported outcomes to support medical product laveling claims: FDA perspective. Value Health. 2007;10(Suppl 2):S125–37.View ArticlePubMedGoogle Scholar
- Powers JH, Guerrero ML, Leidy NK, et al. Development of the flu-PRO: a patient-reported outcome (PRO) instrument to evaluate symptoms of influenza. BMC Infect Dis. 2016;16:1.View ArticlePubMedPubMed CentralGoogle Scholar
- Powers JH, Bacci ED, Guerrero ML, et al. Reliability, validity, and responsiveness of InFLUenza patient-reported outcome (FLU-PRO©) scores in influenza-positive patients. Value Health. 2018a;21(2):210–8.View ArticlePubMedGoogle Scholar
- Powers JH, Bacci ED, Leidy NK, et al. Performance of the InFLUenza patient-reported outcome (FLU-PRO©) diary in patients with influenza-like illness (ILI). PLoS One. 2018b;13(3):e0194180.View ArticlePubMedPubMed CentralGoogle Scholar
- Memoli MJ, Shaw PA, Han A, et al. Evaluation of Antihemagglutinin and Antineuraminidase antibodies as correlates of protection in an influenza a/H1N1 virus healthy human challenge model. MBio. 2016;7(2):e00417-16.View ArticlePubMedPubMed CentralGoogle Scholar
- Couch RB, Atmar RL, Franco LM, et al. Antibody correlates and predictors of immunity to naturally occurring influenza in humans and the importance of antibody to the neuraminidase. J Infect Dis. 2013;207(6):974–81.View ArticlePubMedPubMed CentralGoogle Scholar
- Monto AS, Petrie JG, Cross RT, et al. Antibody to influenza virus neuraminidase: an independent correlate of protection. J Infec Dis. 2015;212(8):1191–9.View ArticleGoogle Scholar
- Han A, Poon JL, Powers JH, Yu R, Leidy NK, Memoli MJ. Evaluating symptom severity of influenza viral infection using the Influena patient-reported outcomes Instruemtn (FLU-PRO) in a healthy human challenge model. Open Forum Infectious Diseases. 2017;4(suppl_1):S572–3.View ArticleGoogle Scholar