CGM

A New Method to Evaluate Analytic Performance of CGM Devices Scott Pardo, PhD, PStat ; David Simmons, MD ; Sergey Zhuplatov, MD, PhD ; Marc Breton, PhD * ®1

1

1

2,

Ascensia Diabetes Care, Parsippany, NJ; University of Virginia School of Medicine, Charlottesville, VA.

Σ

N |MeterBGi – LabBGi| i=1 LabBG i

⦁⦁ Lower MARD values generally translate to better accuracy versus higher MARD values, but the relationship between ISO accuracy criteria and MARD is complex5 –– It is not possible to define a particular MARD value below which the meter is considered sufficiently accurate to satisfy ISO accuracy criteria5 –– The accuracy of a BGM/CGM device is usually evaluated by experimental comparison against the YSI Gold Standard ⦁⦁ Thus, the performance of BGMs or CGMs should not be assessed solely on MARD value, and additional measures of accuracy should be considered to obtain a more complete assessment of BGM/CGM performance5 ⦁⦁ Here, we attest that a more complete characterization of CGM performance may be obtained by computing the absolute relative difference (ARD) of each CGM and reference glucose measurement pair for each subject/sensor combination –– We show that while MARD is easily computed by using the average, the distribution of ARD may be empirically constructed and fit to known parametric probability distributions

OBJECTIVE ⦁⦁ To examine a new methodology for assessing CGM analytical performance

METHODS ⦁⦁ Data taken over a month from subjects with type 1 diabetes (ClinicalTrials.gov Identifier: NCT01835964) were used to construct histograms of the ARD of each CGM and reference method blood glucose measurement pair for each subject/sensor combination –– CGM and corresponding comparator glucose values for 11 subjects in one study (Study 1) and 27 subjects in another study (Study 2) were used

N |CGMi – compi| i=1 compi

Σ

“CGMi” refers to the i blood glucose value from the sequence of results taken from the CGM device, and “compi” is the i blood glucose value from the comparator device. In Study 1, the comparator device was a self-monitoring BGM. In Study 2, the comparator was a laboratory blood glucose instrument. th

th

⦁⦁ Multiple probability distributions were fit to the ARD data, including normal, lognormal, Weibull, Johnson (SU, SB, and SL), and gamma; distributions were fit to the ARD data separately for each sensor ⦁⦁ The percentage of CGM measurements falling within ±15% of the corresponding comparator result, and MARD, were computed, and a gamma distribution was fit to each sensor’s ARD data –– Parameter estimates of shape and scale were used to compute a “theoretical” MARD as follows when α = shape, β = scale for “Pr{In15} gamma”:

Pr{ARD ≤15} =∫

15

–α

β 0 Γ(α)

e

–

r β

0.15 0.10 0.05 0

⦁⦁ To create plots of the probabilities of errors using the gamma distribution of ARD, probabilities were plotted against MARD = shape × scale

RESULTS Measures of Accuracy ⦁⦁ After separately fitting multiple probability distributions to ARD data for each sensor, the only one for which goodness-of-fit tests failed to reject for all subject/sensor combinations was gamma; thus, the gamma distribution was chosen as the best model for ARD –– The gamma distribution has 2 parameters: shape and scale. The mean of the gamma distribution is the product of shape and scale –– Figure 1 shows a typical example of gamma fit to ARD data

Proportions of Measurements Within Percentage Ranges of Comparator Results and MARD ⦁⦁ Shape and scale estimates, “direct” MARD calculation, and the product shape × scale for all sensor/subject data aggregated are presented in Table 1 ⦁⦁ The gamma-based probability calculations and observed proportions of results within ±XX% (XX = 5, 10, 15, 20) with all data aggregated are presented in Table 2 ⦁⦁ Figure 2 shows histograms of ARD for both studies with the fitted gamma density functions –– The goodness-of-fit test for Study 2 aggregated data had a P value of 0.0384; however, the gamma error probabilities are not far from observed proportions

0

10

20

30

40

50

0.7

17.00 15.57

17.01 15.57

MARD, mean absolute relative difference.

Table 2. Gamma Distribution Probabilities and Empirical Proportions, All Data Aggregated Pr Pr Pr Pr Obs Obs Obs Obs Study {ln5} {ln10} {ln15} {ln20} {ln5} {ln10} {ln15} {ln20} 0.2140 0.4132 0.5692 0.6866 0.2241 0.3916 0.5599 0.6823 0.2488 0.4558 0.6100 0.7221 0.2694 0.4806 0.6314 0.7328

Pr, probability; obs, observed.

Study 1

Study 2

0.25

0.25

0.20

0.20

0.15 0.10

0.10 0.05

0

0

0 10 20 30 40 50 60 70 80 90 100 ARD (%) Gamma (1.2088, 14.0695, 0)

0.3

0

2

4

6

8

10 12 14 16 18 20 22 24 26 28 30 MARD (%)

X = 5%

X = 10%

X = 15%

X = 20%

MARD, mean absolute relative difference.

Figure 4. Probabilities of errors falling within ±X, X = 5%, 10%, 15%, 20%. 5

10

15

20

25

30

35

40

MARD (%) CGM, continuous glucose monitoring; MARD, mean absolute relative difference.

Figure 3. Scatter plot and linear fit: observed % CGM errors within ±15% of comparator versus observed MARD.

Utilizing the Gamma Distribution of ARD ⦁⦁ Figure 4 shows various probabilities plotted against MARD for shape 0.8 to 3.5 and scale 1.5 to 8.0 (756 combinations), highlighting that the same MARD can mask different error characteristics (eg, MARD = 12% may correspond to CGM within 5% of reference method blood glucose measurements anywhere between 10% and 25% of the time) ⦁⦁ This model and its parameters, shape and scale, easily provide the expected MARD and the probability that CGM and reference method blood glucose measurements would be within prespecified bounds

DISCUSSION ⦁⦁ MARD alone is an insufficient indicator of the probability that CGM errors will fall within ±15% of a “true” glucose value and may not be a sufficient measure of overall CGM analytical performance ⦁⦁ MARD has been used to characterize the analytical performance of CGM devices, yet it is clear that MARD does not always provide a 1-to-1 correspondence to the range of errors expected from a CGM device ⦁⦁ The statistical variation in ARD can likely be predicted using gamma distributions ⦁⦁ The gamma distribution is a desirable model because it can be used to easily determine the expected MARD and the probability that CGM and reference method blood glucose measurements would be within prespecified bounds

CONCLUSIONS ⦁⦁ Using MARD as the single measure of analytical performance for CGM is insufficient and may lead to incorrect conclusions ⦁⦁ Additional measures of analytical performance are needed to provide a more complete characterization of CGM performance ⦁⦁ The error probability calculation facilitated by the gamma distribution model may be used in addition to MARD to provide a more informative tool for assessing CGM analytical performance

0.15

0.05

0.4

0

Linear fit

0.4

0

Table 1. Gamma Distribution Parameter Estimates and MARD, All Data Aggregated Shape × Study Shape Scale N MARD (%) scale

1 2

0.5

0.1

Figure 1. Gamma fit to ARD (%).

1,218 1,804

0.5

0.1

0.2

ARD, absolute relative difference.

14.0695 13.8080

0.6

0.3

Notes: The third parameter, θ, is only used if the data have negative values. H0 = the data are from the gamma distribution. Small P values (W2 0.156483 > 0.2500

1 2

0.8

0.8

ARD (%)

dr (× 100%)

1.0 0.95 0.9

0.9

Observed

MARD =

1 N

0.20

Probability

⦁⦁ For the assessment of analytical performance of continuous glucose monitoring (CGM), the mean absolute relative difference (MARD) between CGM and reference method blood glucose measurements is commonly used,2,3 but may be insufficient to characterize CGM performance –– Blood glucose values for an individual vary over time, with strong autocorrelation between measurements taken less than 1 hour apart4; methods such as MARD eliminate the notion of ordered values from the data set, a key aspect of CGM measurements ⦁⦁ MARD has been used both to assess the accuracy of CGMs and more recently to compare the accuracy of BGMs using a single numeric value5

MARD% = 100×

1 N

*Presenting author.

⦁⦁ A scatter plot of the proportion of errors within ±15% of the comparator results versus the empirical (arithmetic average) MARD is presented in Figure 3 –– The statistical variation in ARD can likely be predicted using gamma distributions; furthermore, there appears to be a linear relationship between the probability that ARD is less than or equal to a given value (eg, 15%) and MARD

Distribution curve

0.25

Probability

⦁⦁ International Organization for Standardization (ISO) 15197:2013 accuracy criteria are used by regulatory agencies and the research community to establish the accuracy of blood glucose meters (BGMs)1

⦁⦁ To assess CGM measures of accuracy, empirical distributions of ARD for each subject were computed, and parametric forms were fit to these distributions. The mean of the ARDs was computed for each subject as follows:

Probability

INTRODUCTION

2

Probability (within ±X)

1

0 10 20 30 40 50 60 70 80 90 100 ARD (%) Gamma (1.1276, 13.8080, 0)

Distribution curve ARD, absolute relative difference.

Figure 2. ARD histograms with fitted gamma density functions.

References

Acknowledgments

1. ISO 15197:2013(E). In vitro diagnostic test systems—Requirements for blood-glucose monitoring systems for self-testing in managing diabetes mellitus. May 2013. 2. Beck RW, et al. JAMA. 2017;317(4):371-378. 3. Luijf YM, et al. Diabetes Technol Ther. 2013;15(8):722-727. 4. Breton M, Kovatchev B. J Diabetes Sci Technol. 2008;2(5):853-862. 5. Pardo S, Simmons DA. J Diabetes Sci Technol. 2016;10(5):1182-1187.

These analyses were supported by Ascensia Diabetes Care, Parsippany, NJ, USA. Editorial assistance was provided by Allison Michaelis, PhD, of MedErgy, and was funded by Ascensia Diabetes Care. D.S. was an employee of Ascensia Diabetes Care during the development of the work reported here.

An electronic version of the poster can be viewed by scanning the QR code. The QR code is intended to provide scientific information for individual reference. The PDF should not be altered or reproduced in any way. http://bgd_ada.scientificpresentations.org/ Pardo_BGD61957.pdf

POSTER PRESENTED AT THE 77TH SCIENTIFIC SESSIONS OF THE AMERICAN DIABETES ASSOCIATION (ADA); JUNE 9-13, 2017; SAN DIEGO, CALIFORNIA.