Best practices for the development and fit-for-purpose validation of biomarker methods: a conference report

Abstract

This conference report summarized a full-day workshop, “best practices for the development and fit-for-purpose validation of biomarker methods,” which was held prior to the American Association of Pharmaceutical Scientists (AAPS) PharmSci360 Congress, San Antonio, TX, November 2019. The purpose of the workshop was to bring together thought leaders in biomarker assay development in order to identify which assay parameters and key statistical measures need to be considered when developing a biomarker assay. A diverse group of more than 40 scientists participated in the workshop. The workshop and subsequent working dinner stimulated robust discussion. While a consensus on best practices was not achieved, some common themes and major points to consider for biomarker assay development have been identified and agreed on. The focus of this conference report is to summarize the presentations and discussions which occurred at the workshop. Biomarker assay validation is a complex and an evolving area with discussions ongoing.

Introduction

The importance of biomarkers in decision-making during drug development continues to increase (Menetski et al. 2019). A variety of biomarker modalities are employed during various phases of drug development. Protein biomarkers are commonly used for a wide range of indications during the development of new drugs, including, but not limited to, providing initial mechanism-of-action and proof-of-concept data, to support dose selection for clinical development, or the recommended clinical dose in the product labeling (Wang et al. 2019). Cellular biomarkers have become critical components in the development of novel immune therapies for oncology, cell and gene therapies, infectious diseases, and vaccines (Sant et al. 2017). In clinical laboratories, both protein and cellular biomarkers are used to establish a diagnosis and/or prognosis.

The term “fit-for-purpose” method validation first appeared in a publication from the AAPS Ligand Binding Analytical Focus Group in 2006 (Lee et al. 2005). This term has come to mean that assays should be validated as appropriate for the intended use of the data and the associated regulatory requirements. If the intended use of the data changes, any additional validation that may be required should be conducted using an iterative approach (Lee et al. 2005; Jani et al. 2016; Lee et al. 2006). More recently, the term “context-of-use” (COU) has been applied to define the fit-for-purpose expectations for the validation of the assay (Piccoli et al. 2019).

The pharmaceutical community and regulatory agencies have accepted the term “fit-for-purpose” method validation, which appears in the 2018 Guidance for the Industry (Islam et al. 2018; Piccoli and Garofolo 2018). Nonetheless, to date, there is no consensus on the best practices for assay development or standardized validation requirements for biomarker assays based on the COU using the fit-for-purpose approach. In an attempt to move closer to such a consensus, a full-day workshop, “best practices for the development and fit-for-purpose validation of biomarker methods” was held prior to the American Association of Pharmaceutical Scientists (AAPS) PharmSci360 Congress, San Antonio, TX, November 2019.

The purpose of the workshop was to bring together thought leaders in biomarker assay development from the pharmaceutical industry, contract research organizations (CROs), and the Food and Drug Administration (FDA) to discuss best practices. The workshop put an emphasis on identifying which assay parameters and key statistical measures need to be considered when developing a biomarker assay. The discussion addressed questions such as whether minimum validation standards can be established and if so, should they be established. The focus was on ligand binding and flow cytometry-based assays. More than 40 scientists participated in the workshop, which included presentations, breakout discussion sessions, and a working dinner for additional discussion.

A pre-workshop survey was distributed which received 41 responses. The initial questions in the survey were aimed at gathering general demographic data about the respondents, regulatory settings, and biomarker experience (Fig. 1, Figs. S1, S2, S3). Most of the respondents (61–66%) were evaluating soluble protein or cellular biomarkers, whereas only 37% were evaluating molecular targets. GCLP and GLP were the most common regulatory settings for conducting biomarker validation (35% and 33%, respectively) while 17% were conducted under CAP/CLIA, 4% GMP, and 11% other or non-regulated (Figs. S1, S2, S3). Regarding the important parameters to address during validation, the respondents were mostly in agreement, with greater than 60% selecting precision and accuracy, parallelism, stability, and specificity (Fig. 2). Moreover, 75% of the respondents agreed that there should be a minimum standard for validation (Fig. 2). Nonetheless, during the workshop itself, there was less agreement on this point, and while everyone agreed that the COU would be the primary driver of the validation design, consensus on the value of and how to set minimum standards was not reached.

Fig. 1

Pre-workshop survey—demographics of respondents. The distribution of workplace (A) and job title of the 41 survey respondents (B)

Fig. 2

Pre-workshop survey—key criteria for validation method. A large majority of the 41 survey respondents (75%) pronounced in favor of establishing a minimal requirement for method validation (center graph). The respondents also indicated the key validation criteria which should be included in a method validation (outer ring, shown as the percentage of total respondents). Of note, for protein biomarker assay (ligand binding and mass spectrometry), accuracy is often referred to as relative accuracy given that calibrators composed of recombinant protein are not typically identical to the endogenous biomarker, thus the concentration is relative to the specific assay used. However, the survey did not highlight this point. For cellular biomarkers, accuracy is even more challenging due to the lack of appropriate cellular reference materials for multiparametric flow cytometry

Workshop presentations

The first speaker in the workshop was Dr. Lakshmi Amaravadi, who focused on the impact of biomarkers on drug development, and the importance of understanding how the COU will inform appropriate assay development, assay validation, and ultimately data interpretation (Piccoli et al. 2019). She discussed important parameters to consider when establishing the COU (Table 1). The importance of understanding the limitations of the technology as well as the assay systems used in the validation was highlighted. In addition, when selecting a technology, the biomarker scientist needs to determine what assay sensitivity will be required for the particular biomarker. Other categories that were highlighted by Dr. Amaravadi include the stage of development the biomarker will be used in and how the biomarker will be used (safety, diagnostic, etc.). Goodman et al. discuss in-depth various aspects to consider when establishing the COU and provide a broad set of questions a biomarker scientist should address when defining the COU (Goodman et al. 2020).

Table 1 Categories for considerations when establishing context of use. Multiple factors need to be considered when establishing the biomarker assay context of use

Dr. Amaravadi also addressed the importance of pre-analytical variables, which are too often overlooked, yet can have a significant impact on assay performance and ultimately impact the final results considering that samples in clinical trials are typically collected at global sites and shipped to the testing facility. Included in this report is a list of potential pre-analytical variables which could affect the biomarker measurement (Table 2). This list is included primarily to provide points to consider; not all of these variables will impact every biomarker. Moreover, it is important to emphasize that the biomarker scientist must consider all potential pre-analytical variables and determine if they could have an effect on the biomarker measurement; there is no one-size-fits-all approach. In his presentation, Mr. John Allinson organized pre-analytical variables as either controllable or uncontrollable (Table 2). Controllable variables are those which the biomarker scientist has influence such as the matrix, specimen collection, processing, and transport procedures. For example, many biomarkers are secreted by activated platelets or are affected by the anticoagulant such as VEGF. An in-depth discussion of the considerations related to matrix selection is reviewed in a publication by Mathews et al. (2020). By contrast, uncontrollable variables are variables which are characteristic of the patients or study population such as gender and age, which would not be controlled by the specimen collection, processing, and transport procedures; nonetheless, the biomarker scientist must take these variables into account. For instance, when designing a trial, it may be known that the biomarker of interest is affected by obesity; thus; within the trial, obesity may be included as an exclusion criterion or the clinical team may want to ensure that those who are obese are balanced across the various study groups. Dr. Amaravadi also reviewed publications which discussed the importance of different pre-analytical variables; how samples are collected, processed, stored, and transported; and the need for standardization (Hu et al. 2015; Dakappagari et al. 2017). One publication from Hu et al. demonstrated how different protocols of sample processing impacted the results in CSF beta-amyloid measurements (Hu et al. 2015; Dakappagari et al. 2017).

Table 2 Potential pre-analytical variables*

The lack of true reference standards was also emphasized as a critical limitation for both protein biomarker assays and cell-based assays. The differences between recombinant protein calibrators used in protein biomarker assays and the endogenous biomarkers were discussed as was the need to use endogenous quality controls (QCs) instead of recombinant material for stability determination and assay performance monitoring (Cowan et al. 2017). Understanding the limitations of the validated assay is critical when deploying the assay and interpreting the data.

Dr. Lauren Stevenson’s presentation “No Context, No Assay – Biomarker Assay Validation Demands Context of Use” emphasized the need to establish a foundational understanding of COU with cross-functional partners, project teams, and the broader scientific organization (Stevenson 2019). This presentation clarified that broad terms such as “exploratory endpoint” do not constitute a COU and instead emphasized that COU is the specific purpose in “fit-for-purpose.” In addition, the selection of assay platform, assay format, and identification of key drivers for assay development are all dependent upon COU. Therefore, without a clear understanding of the intended use of the data, it is not possible to validate the assay for its intended use, or more succinctly, “no context, no validated assay.” Dr. Stevenson then provided practical suggestions for how to establish a foundational understanding of COU with project teams and presented examples that demonstrated the risks of misapplication of biomarker assays for unintended COUs and the potential misinterpretation of data that can follow.

Mr. John Allinson’s presentation used an example from his clinical background in diagnostics—where clinical utility has always been dependent upon context. He chose a biomarker that has multiple COUs, namely thyroid-stimulating hormone (TSH), and demonstrated how each different COU dictated the assay performance requirements for the same biomarker and detailed what additional validation work was required to meet the demands of each new COU in turn. To date, there have been at least five unique COUs for TSH, each carrying unique requirements. In addition, Mr. Allinson reviewed the many issues facing bioanalytical scientists today and presented some tools to assist in data interpretation. These tools include analytical quality control practices used to monitor and understand assay performance.

The next presentation by Dr. Viswanath Devanarayan focused on assay variability and emphasized that most validation parameters are influenced by the analytical variability of the method. He stated that when defining the minimum adequate precision and maximum tolerable imprecision of the analytical variability, decisions are often made based on historical preference or guidance documents; however, depending on the level of biological variability and intended use of biomarkers, the practical impact and acceptable level of analytical variability can vary greatly. Thus, consideration regarding biological variability and COU should be taken into account. The take-home message of this presentation was that fit-for-purpose is not just about exploratory vs. confirmatory vs. CDx but also about the COU and biological variability.

Dr. Yow-Ming Wang from FDA CDER presented the regulatory perspective: the necessity to validate the biomarker assays is to ensure that a method can produce accurate, reliable, and robust data to support regulatory decision-making. Therefore, the FDA’s guidance on bioanalytical method validation (BMV) recommends that the assay should be fully validated when it provides biomarker data for the pivotal determination of safety and/or effectiveness of a therapeutic or to support dosing instructions in product labeling (US Department of Health and Human Services, Food and Drug Administration. Center for Drug Evaluation and Research (CDER) 2018). Consistent with this objective, the FDA recently published a guidance entitled “Bioanalytical Methods Templates” (US Department of Health and Human Services Food and Drug Administration. Center for Drug Evaluation and Research (CDER) 2019) which provides recommendations on the reporting of information regarding the lifecycle of methods, the method performance, and cross-validations for the associated method modifications. The guidance is intended for both biomarker assays and PK assays to be included in regulatory submissions.

Three presentations focused on flow cytometric methods, which represent the most specific tool in the repertoire of bioanalytical techniques for the analysis of defined cellular subsets, cell-specific protein expression, and cellular distribution. Ms. Kelly Lundsten introduced the best practices for panel design and optimization in flow cytometry (Cossarizza et al. 2019). The first consideration presented was the need to choose the appropriate combination of markers and data analysis strategy (gating) to identify the cellular population of interest. The second was to consider the selectivity of the monoclonal antibody (mAb) clones used to identify those markers and of pairing the appropriate fluorophore to each mAb (Kalina et al. 2020). Choosing which fluorophore to pair with your mAb requires knowing the antigen expression levels, co-expression of other critical markers, and the spectral spillover properties of the fluorophores.

Dr. Steven Eck described the tiered validation approach presented in the Clinical Laboratory and Standards Institute (CLSI) new guideline, H62 1st Edition: Validation Assays Performed by Flow Cytometry (CLSI 2021). This document provides practical guidance on instrument qualification, panel design, method validation, and quality assurance processes unique to flow cytometric methods. In addition, CLSI H62 makes specific recommendations for the minimal validation requirements for a variety of validation scenarios ranging from FFP to analytical validation based on the COU: whether it is basic research, label claims, or clinical decision-driving diagnostics. The document also explains that panel design, as described in detail by Ms. Lundsten, is the primary factor influencing assay specificity and precision. For this reason, CLSI H62 suggests minimal acceptance criteria for abundant as well as rare cellular populations because the experts in the field of flow cytometry biomarkers are of the opinion that if an assay does not meet these criteria, it is likely that it is not optimally designed.

A representative from FDA CBER presented a regulator’s prospective on validation strategies applied to flow cytometric methods. Multiparameter cytometric measurements are routinely carried out for disease diagnosis, selection of appropriate treatment options, and characterization of cellular therapies. These applications have increased the need for high-quality, robust, and validated measurements to ensure appropriate diagnosis, product safety, and effectiveness. A well-designed flow cytometry validation plan is required to ensure that the obtained results are robust, credible, and reproducible. The presentation summarized the appropriate method validation strategy for flow cytometry-based assays, based on the COU. The presentation highlighted the difference between assay qualification and analytical validation, identified challenges, and phases of assay validation. It also summarized the key parameters such as accuracy, precision, linearity, specificity and selectivity, assay sensitivity and LLOQ, stability, robustness, carryover, and reference intervals and the general rules to consider when executing validation studies.

Workshop discussions

Although one objective of this workshop was to craft minimal criteria for different levels of validation, consensus on best practices regarding the development and fit-for-purpose validation of biomarker methods was not achieved during the workshop, the participants did agree that the biomarker assay COU should be defined prior to assay development and is established by considering factors such as but not limited to (1) the biology of the biomarker, (2) the pharmacodynamic effects of the drug on the biomarker, (3) the intended use of the biomarker data, (4) the phase of drug development, and (5) the size of the trial. The initial COU will impact the type of assay that is developed and the validation plan. It is important to note that once developed and validated biomarker assays can be, and are often, repurposed for a different COU. In these cases, it is essential to evaluate whether the initial validation is appropriated for the new COU or if an additional validation is required. After the initial validation is completed, other parameters may need to be evaluated such as biological variability. When interpreting study data, both the analytical variability and the biological variability should be considered (Piccoli and Sauer 2019).

A major theme which dominated the various discussions was whether a modified PK assay approach should be applied to protein biomarker validation. It was noted that the initial fit-for-purpose paper attempted to align biomarker assay validation and pharmacokinetic (PK) assay validation (Lee et al. 2006). As was discussed in this workshop, due to a multitude of differences between assays to measure drug concentration and biomarker assays, this approach would not be appropriate. There is only one COU for PK assays—to measure drug concentration in order to determine exposure and PK parameters—and this COU has well-defined a priori criteria that do not vary. Whereas during the entirety of the drug development process, a specific biomarker might have multiple COU as was shown for TSH in Dr. John Allinson’s presentation.

Recognizing that assay validation considerations for biomarkers differ from those for drug concentration assays, both the BMV guidance and the CLSI H62 guidance document provide a general framework and common principles useful as a starting point when creating a validation plan for a biomarker method. Nonetheless, the specific parameters addressed, and the approaches used to address them, would need to align with COU.

Conclusion

Fit-for-purpose, context-of-use, and iterative approaches are all familiar terms in biomarker analytical science. They describe a philosophy of assay development and validation in which the performance targets for an assay’s results and the rigor of the experiments used to demonstrate that performance is continually assessed against what the assay is intended to achieve within its then-current application.When applied properly, this approach facilitates the efficient development of assays that will deliver reproducible results. While this makes intuitive sense, its interpretive nature can lead to confusion regarding expectations, particularly with regard to what minimal validation experiments should be considered.

This is an evolving field, and additional workshops occurred and publications appeared subsequent to this 2019 AAPS Workshop where Dr. Devanarayan presented the concept of moving away from referring to the process as a validation, but rather refer to it as the characterization of the assay. This concept, of assay characterization against which COU could be evaluated with validation being claimed when assay performance, has been demonstrated to meet COU, has also been discussed at AAPS 2020 PharmSci360 and several European Bioanalysis Forum meetings as well as in the literature (Piccoli and Sauer 2019). The number of times this concept has been discussed highlights its importance for biomarker scientists.

There has been an exponential increase in the application of biomarkers to drug development in the last couple of decades. However, significant challenges remain in realizing the full promise of biomarkers. This is due to the breadth and complexity of the questions that biomarkers are intended to address as well as translating those applications to the appropriate development and validation of the biomarker assays. A holistic view of all the considerations that are included in defining COU is needed. Effective implementation of biomarkers requires consideration of not just the performance of the biomarkers assay but of the biology of the biomarker as well. There is no one-size-fits-all solution when it comes to biomarker assay validation and deployment.