Accred Qual Assur (2014) 19:221–223 DOI 10.1007/s00769-014-1050-y
DISCUSSION FORUM
A simple method of constructing a confidence interval for the mean probability of detection in collaborative studies of binary measurement methods Hans Schneeweiß • Peter-Th. Wilrich
Received: 31 January 2014 / Accepted: 2 April 2014 / Published online: 24 April 2014 Ó Springer-Verlag Berlin Heidelberg 2014
Abstract We deal with collaborative studies where each of k laboratories performs n repeated binary measurements (measurement result x = 0: ‘‘not detected’’; measurement result x = 1: ‘‘detected’’), and present a simple method of constructing a confidence interval for the mean probability of detection of the laboratories. This method is based on an approximation of the distribution of the number y of detections among n independent measurements of a randomly chosen laboratory by a binomial distribution. The confidence interval is not only much easier to calculate but also more accurate than the profile likelihood interval presented by Uhlig et al. Keywords Probability of detection POD Interlaboratory experiments Collaborative studies Confidence interval Qualitative measurements Binary measurements Coverage
Introduction Wehling et al. [1], Uhlig et al. [2] and Wilrich [3] deal with collaborative studies where each of k laboratories performs n repeated binary measurements (measurement result x = 0: ‘‘not detected’’; measurement result x = 1: ‘‘detected’’). Wehling et al. [1] present an approximate H. Schneeweiß Institut fu¨r Statistik, Universita¨t Mu¨nchen, Akademiestrasse 1/I, 80799 Mu¨nchen, Germany e-mail:
[email protected] Peter-Th.Wilrich (&) ¨ konometrie, Freie Universita¨t Berlin, Institut fu¨r Statistik und O Garystrasse 21, 14195 Berlin, Germany e-mail:
[email protected]
confidence interval for the mean probability of detection (POD) of the laboratories, p, using the t-distribution, with coverages that often deviate very much from the nominal value. Uhlig et al. [2] propose to use a profile likelihood interval based on a statistical model with a latent random laboratory effect as a confidence interval for the mean POD. On page 370 of their paper they state: ‘‘This type of confidence interval is based on asymptotic approximation. It should be noted that the calculations require numerical integration. Web tools and software for these calculations are available’’. This approach that not only requires numerical integrations for the numerical optimisation of the likelihood function but also for two optimisation procedures (for the limits of the profile likelihood interval) under constraints is unnecessarily complicated and should be substituted by the following easier method. Two related problems shall be mentioned: Macarthur and von Holst [4] deal with the same kind of collaborative studies as Wehling et al. and Uhlig et al.; however, they are interested in a b-expectation tolerance interval that is expected to cover the fraction b of the PODs of the population of laboratories out of which the laboratories have been randomly selected. Wilrich [5] discusses this approach and presents a b-expectation tolerance interval that uses directly the estimate of the standard deviation of the laboratories and is much easier to calculate than the interval of Macarthur and von Holst. In ISO 16140-2 [6] and Wilrich and Wilrich [7], intralaboratory studies are considered in which repeated measurements at different known concentration levels are obtained. The POD curve, i.e. the relationship between the mean POD and the concentration, is modelled with a complementary log–log model. The estimated POD curve is used for the determination of limits of detection (for given PODs).
123
222
Accred Qual Assur (2014) 19:221–223
The method
p ¼ p;
A randomly chosen laboratory has an unknown probability p of detection (POD) and reports y, the number of detections among n independent measurements. The pair (p, y) is a random pair, where y is observable and p is latent. p has an unknown distribution with mean p and variance r2L , the between-laboratory variance. The conditional distribution of y given p is a binomial distribution: yjp Binðn; pÞ. Mean and variance of yjp are EðyjpÞ ¼ np and VðyjpÞ ¼ npð1 pÞ, respectively. The marginal distribution of y is unknown, but its mean is given by ly ¼ EðyÞ ¼ EðEðyjpÞÞ ¼ EðnpÞ ¼ np
ð1Þ
and its variance by r2y ¼ EðVðyjpÞÞ þ VðEðyjpÞÞ ¼ Eðnpð1 pÞÞ þ VðnpÞ ¼ n Efpð1 pÞg þ nr2L :
ð2Þ
Because of EðpÞ ¼ p we get Eðpð1 pÞÞ ¼ pð1 pÞ r2L and r2y ¼ n pð1 pÞ þ ðn 1Þr2L : ð3Þ We consider the k laboratory values ðpi ; yi Þ; i ¼ 1; . . .; k, as independent identically distributed sample values from the distribution of (p, y). We are interested in estimating p on the basis of the sample values yi ; i ¼ 1; . . .; k. We also want to construct a, possibly approximate, confidence interval for p. An unbiased estimate of p is p^ ¼
1 S; kn
S¼
k X
yi :
ð4Þ
1
Its variance is 1 1 ^ ¼ 2 2 VðSÞ ¼ 2 r2y : VðpÞ k n kn
k 1 X ðyi yÞ2 : k1 1
n ¼
n2 pð1 pÞ : r2y
ð8Þ
We can estimate n by replacing p and r2y with their estimates: n^# ¼
^ pÞ ^ n2 pð1 : 2 r^y
ð9Þ
Because of (3), r2y npð1 pÞ and hence, n in (8) cannot be larger than n. However, in (9), it is possible that ^ pÞ ^ and hence, n^# can be larger than n. In order r^2y \npð1 to avoid this, we estimate n by ! 2 ^ ^ n pð1 pÞ : ð10Þ n^ ¼ minðn; n^# Þ ¼ min n; r^2y Inference on p can now be accomplished in the following way. P ^ and r^2y from the sample values 1. Compute S :¼ k1 yi ; p, yi 2. Compute n^ ; N : ¼½kn^ , and S :¼ n^n S , where the square brackets indicate rounding to the nearest integer 3. Assuming that (approximately) S BinðN ; pÞ, construct the Pearson–Clopper confidence interval ðpL ; pU Þ for p at the confidence level 1 a (see [8], p. 107) with limits pL ¼
S S þ ðN S þ 1ÞFm1 ;m2 ;1a=2
m1 ¼ 2ðN S þ 1Þ; S þ 1 pU ¼ S þ 1 þ ðN S ÞFm1 ;m2 ;a=2 with
m1 ¼ 2ðN S Þ;
m2 ¼ 2S
m2 ¼ 2ðS þ 1Þ
ð5Þ
where Fm1 ;m2 ;p is the p-quantile of the F-distribution with m1 and m2 degrees of freedom. These quantiles are available in almost every statistical software package, in EXCEL as function FINV(). (Note that by construction N S ).
ð6Þ
Uhlig et al. [2] have obtained the coverage probabilities of their confidence interval for 1 a ¼ 95 %; k ¼ 12 laboratories, n ¼ 6 or n ¼ 18 repeated measurements and six different scenarios concerning the distribution of the PODs p of the laboratories by simulation. In Table 1, we compare their coverage probabilities with the coverage probabilities of the proposed easy method. In almost all scenarios, the coverage probabilities of the confidence intervals are closer to the nominal value 95 % than those of the profile likelihood intervals.
To construct a confidence interval for p one would need to know the distribution of p^ or, equivalently, of y, which, however, is unknown. In the particular case where p is a constant ðp ¼ pÞ, the distribution of y is known: it is Binðn; pÞ. In the general case, we could therefore try to approximate the distribution of y by a binomial, i.e. y Binðn ; p Þ such that the distribution of y =n approximates the distribution of y=n in the sense that Eðy =n Þ ¼ Eðy=nÞ and Vðy =n Þ ¼ Vðy=nÞ. This implies that
123
ð7Þ
from which follows
with
An unbiased estimate of r2y is given by r^2y ¼
1 1 p ð1 p Þ ¼ 2 r2y ; n n
Accred Qual Assur (2014) 19:221–223
223
Table 1 Simulated coverage probabilities of the confidence intervals for p Scenario
1a
Distribution of the PODs pi
Constant pi ¼ p0 ¼ 0:5
1b 2a 2b
Every pi between 0.3 and 0.7 is equally probable
3a
Every pi between 0.7 and 1 is equally probable
3b 4a
pi ¼ UðXi Þ, where Xi Nð0; 0:52 Þ
4b 5a
pi ¼ UðXi Þ, where Xi Nð1; 0:52 Þ
5b 6a
pi ¼ UðXi Þ, where Xi Nð1:5; 0:52 Þ
6b a
Copied from Table 2 of Uhlig et al. [2]
b
Obtained by simulation with 105 simulation runs
n
95 % Profile likelihood intervala
95 % Confidence intervalb
Coverage ð%Þ
Coverage ð%Þ
Too lowð%Þ
Too high ð%Þ
Too low ð%Þ
Too high ð%Þ
6
97.7
1.2
1.1
96.7
1.6
1.7
18
98.0
1.0
1.0
96.4
1.8
1.8
6 18
95.7 91.2
2.1 4.4
2.2 4.4
95.5 94.0
2.2 3.0
2.3 3.0
6
95.1
2.4
2.5
96.9
1.3
1.8
18
92.3
2.4
5.3
94.1
3.0
2.9
6
92.2
3.9
3.9
94.9
2.4
2.7
18
90.0
5.0
5.0
94.2
2.9
2.9
6
94.8
2.3
2.9
95.4
2.8
1.8
18
89.8
4.4
5.8
93.5
4.3
2.2
6
96.3
2.9
0.8
96.8
2.0
1.2
18
91.6
4.0
4.4
93.1
5.2
1.7
On page 368 of their paper, Uhlig et al. [2] present an example of a collaborative study where each of k = 17 laboratories performed n = 6 repeated measurements. The number of detections were 3, 4, 5, 6 in 2, 2, 5, 8 laboratories, respectively. We calculate S ¼ 87; p^ ¼ 0:853; r^2y ¼ 1:110; n^ ¼ 4:067; N ¼ 69; S ¼ 59 and the 95 % confidence interval (0.750, 0.928) for p that is almost equal to the profile likelihood interval (0.722, 0.921) reported on page 370 of the paper. Conclusion We deal with collaborative studies where each of k laboratories performs n repeated binary measurements (measurement result x = 0: ‘‘not detected’’; measurement result x = 1: ‘‘detected’’), and present a simple method of constructing a confidence interval for the mean POD of the laboratories. This method is based on an approximation of the distribution of the number y of detections among n independent measurements of a randomly chosen laboratory by a binomial distribution. The confidence interval is not only much easier to calculate but also more accurate than the profile likelihood interval presented by Uhlig et al. The confidence interval for the mean POD of the laboratories should be distinguished from the b-expectation tolerance interval that is expected to cover the fraction b of the PODs of the population of laboratories out of which the laboratories have been randomly selected. The determination of such a b-expectation tolerance interval is addressed in Macarthur and von Holst [4] and Wilrich [5]. If the collaborative study includes measurements at different known concentration levels, a functional relationship
between the mean POD and the concentration can be estimated and used for the determination of limits of detection. In ISO 16140-2 [6] and Wilrich and Wilrich [7], this has been done for an intralaboratory study; a publication that extends this approach to collaborative studies is in progress.
References 1. Wehling P, LaBudde RA, Brunelle SL, Nelson MT (2011) Probability of detection (POD) as a statistical model for the validation of qualitative methods. J AOAC Int 94:335–347 2. Uhlig S, Kru¨gener S, Gowik P (2013) A new profile likelihood confidence interval for the mean probability of detection in collaborative studies of binary test methods. Accred Qual Assur 18:367–372 3. Wilrich P-T (2010) The determination of precision of qualitative measurement methods by interlaboratory experiments. Accred Qual Assur 15:439–444 4. Macarthur R, von Holst C (2012) A protocol for the validation of qualitative methods for detection. Anal Methods 4:2744–2754 5. Wilrich P-T (2014) The precision of binary measurement methods. In: Beran J, Feng Y, Hebbel H (eds) Empirical economic and financial research—theory, methods and practice. Fetschrift in honour of Prof. Siegfried Heiler. Advanced sudies in theoretical and applied econometrics. Springer, Berlin, pp 213–225 6. Draft International Standard ISO 16140-2 (2013) Microbiology of food and animal feed— method validation—part 2: protocol for the validation of alternative (proprietary) methods against a reference method. Geneva, International Organization for Standardization 7. Wilrich C, Wilrich P-T (2009) Estimation of the POD function and the LOD of a qualitative microbiological measurement method. J AOAC Int 92:1763–1772 8. Graf U, Henning H-J, Stange K, Wilrich P-T (1987) Formeln und Tabellen der angewandten mathematischen Statistik. Springer, Berlin
123
