A Generative Statistical Model of Mammographic Appearance (The paper appears in its original form on the next page.) @inproceedings{CJR-MIUA-2004, author = {C. J. Rose and C. J. Taylor}, title = {A Generative Statistical Model of Mammographic Appearance}, booktitle = {Medical Image Understanding and Analysis 2004}, editor = {Rueckert, Daniel and Hajnal, Jo and Yang, Guang-Zhong}, pages = {89–92}, year = {2004}, isbn = {1 901725 27 8}, abstract = {We present a generative parametric statistical model of the appearance of entire digital x-ray mammograms. Computer-aided detection (CADe) in mammography has traditionally been treated as a pattern recognition task, where an attempt is made to emulate radiologists’ interpretation strategies. We propose instead that CADe be performed via novelty (outlier) detection [1]. Not only are there far more pathology-free than abnormal mammograms from which to learn, but novelty detection would detect all abnormal features, rather than specific classes. This requires a model of normal mammographic appearance that allows novel (unlikely) model instances to be identified, suggesting that a statistical model is appropriate. Our model addresses many of the problems associated with modelling the appearance of entire mammograms for novelty detection, but this paper does not focus on using the model in the novelty detection scenario. We propose a method for novelty detection, and offer a discussion of the work, in section 5.} }
A Generative Statistical Model of Mammographic Appearance C. J. Rose∗ and C. J. Taylor Imaging Science and Biomedical Engineering, University of Manchester, M13 9PT, UK
1
Introduction
We present a generative parametric statistical model of the appearance of entire digital x-ray mammograms. Computer-aided detection (CADe) in mammography has traditionally been treated as a pattern recognition task, where an attempt is made to emulate radiologists’ interpretation strategies. We propose instead that CADe be performed via novelty (outlier) detection [1]. Not only are there far more pathology-free than abnormal mammograms from which to learn, but novelty detection would detect all abnormal features, rather than specific classes. This requires a model of normal mammographic appearance that allows novel (unlikely) model instances to be identified, suggesting that a statistical model is appropriate. Our model addresses many of the problems associated with modelling the appearance of entire mammograms for novelty detection, but this paper does not focus on using the model in the novelty detection scenario. We propose a method for novelty detection, and offer a discussion of the work, in section 5.
2
Background
Mammograms vary considerably due to differences between women and inconsistencies in the image acquisition process. Women’s breasts vary not only in size and shape, but also in their composition. The proportion of glandular to fatty tissue—the density—is variable, with post-menopausal women usually having almost entirely fatty breasts. The number and configuration of ducts varies between women, and the imaging process may capture these in varying degree. Due to the manual placement of the breast in the x-ray equipment, features such as the nipple or pectoral muscle may be absent or only partially visible. A successful model of mammographic appearance must cope with all these sources of variability. Mammographic appearance has been modelled using physics-based models [2], where the image acquisition process is simulated by computer. However, the synthetic images are not particularly realistic and the approach cannot be used to perform image analysis. Pixel values can be used directly: a parametric model of appearance variation in small image patches is presented in [3], which can be used in both generative and analytical modes, but the explicit assumption of spatial ergodicity does not allow the appearance of the entire breast to be modelled. In [4], wavelet coefficients, computed from small mammographic patches, are statistically modelled using a tree-structured variant of a hidden Markov model. The approach can be used in both generative and analytical modes. Again, the assumption of ergodicity limits the usefulness of the method. Also, due to the structure of the model, the synthetic images have an obvious grid structure. The Active Appearance Model (AAM) [5] is a statistical approach to modelling shape and shape-free appearance. The approach has been applied successfully to a range of computer vision and computer graphics problems. The detailed structure of mammograms is important, but so variable that the AAM approach cannot be applied directly. The approach we have developed combines ideas from the AAM and models of local texture to make the task of modelling mammographic appearance more tractable.
3
Method
We decompose the problem of modelling mammograms by combining an AAM-like model with a spatially ergodic wavelet-based texture model, allowing us to bypass the ‘curse of dimensionality’. After outlining a series of preprocessing steps, we present the three components of our model, and show how they can be combined to generate synthetic mammograms. We assume a training set of mammograms, B, with non-breast regions removed, and normalised such that all breasts ‘face’ right. As in an AAM, a statistical shape model (SSM) is used to cope with size and shape variation. A set of N 2-D landmark points is required to define the breast borders. These landmarks must correspond across the training set. In the SSM, landmarks are often manually placed and chosen to correspond to intuitive image features (e.g. the corners of the mouth when modelling faces). As mammograms lack reliable features, we seek to automate annotation. A na¨ıve approach is to use landmarks placed at regular intervals on the breast borders, ∗
[email protected]
starting at a reliable location (e.g. the right-most point on the breast boundary—the approximate location of the nipple). Such landmarks serve as a good first approximation of corresponding points, but the model they produce is not sufficiently specific to be useful. We use a method proposed by Davies et al. [6] to improve correspondences across the set of training shapes, yielding an SSM that sucessfully limits illegal shape variation. The resulting Principal Components model has the form: s=¯ s + Ps bs (1) where s is a shape parameterised by bs , ¯ s is the mean shape, and Ps is a matrix whose columns are a set of eigenvectors of the shape data covariance matrix, sufficient for the model to retain a given proportion of the total variance of the original data. The mean shape ¯ s provides a natural canonical reference shape. We warp each segmented breast in B to the canonical shape, yielding a further set, N , of segmented breasts in a shape-normalised space.
3.1 3.1.1
Modelling the Shape-normalised Appearance Steerable Pyramid Decomposition
We would like to be able treat the appearance of each mammogram as a point in an appearance space. Although the number of pixels in a mammogram is very large, if there was significant redundancy in the shape-normalised appearance we might still be able to populate the appearance space sufficiently for density estimation to be successful. Unfortunately, this is not the case. To overcome this problem, we use a hierarchical decomposition called the Steerable Pyramid [7]. Images are decomposed in terms of multiple scales and orientations using directional derivative basis functions which range in size and orientation. This allows the coarse and fine structure of the images to be treated separately.
Figure 1. Block diagram for the Steerable Pyramid decomposition. Analysis is shown on the left and synthesis is shown on the right. The dark circle indicates the recursive computation of the shaded region.
Figure 2. The coefficients in the top three levels of a Steerable Pyramid decomposition of a mammogram.
The left-hand side of figure 1 shows the decomposition process (computed using recursive application of oriented bandpass filters to sub-sampled low-pass filtered images) and the right-hand side shows the inverse process of reconstruction. We can think of the decomposition as having a pyramidal structure with discrete levels that correspond to scale, and range from coarse to fine. Each level has a number of oriented sub-bands. Although there are more coefficients in the pyramid than pixels in the original image, the hierarchical structure of the pyramid allows us to decompose our modelling problem further. We can consider the top part of the pyramid (the coarse levels) separately from the bottom part of the pyramid (the fine levels). Figure 2 shows the top three pyramid levels for a mammogram. From N we form the set of pyramids P. 3.1.2
Approximating Appearance Model
For each pyramid in P, we concatenate the coefficients for the top few pyramid levels into a vector a, which describes the approximate appearance of the shape-normalised mammogram. We again perform Principal Components Analysis (PCA) 1 , yielding a model similar to Equation 1: a=¯ a + Pa ba . 1 Initially,
(2)
the coefficients in each pyramid level are effectively measured on different scales. In this work we often need to use covariance matrices to model the distribution of such data, either for its own sake, or to perform PCA. These techniques work best when we use a common
We then model the distribution of the approximating parameters using a multivariate Gaussian, parameterised by a mean vector ma and covariance matrix Σa : p(ba ) ∼ N (ma , Σa ).
(3)
To produce a synthetic approximate mammogram (in the normalised shape space), we sample a ba from our model of p(ba ), project back to the natural space via equation 2—to yield the corresponding a (synthetic coefficients for the top few pyramid levels)—and then reconstruct the pyramid. 3.1.3
Local Textural Detail Model
To model the coefficients in the lower levels—that desribe the fine detail—we assume spatial ergodicity. This makes the problem tractable and is reasonable because we might expect local detail to depend only on tissue type, which is modelled implicitly by the approximating model. A parent vector, bt , which describes local image behaviour, is the set of coefficients on a path through each pyramid level at locations corresponding to a particular pixel in the original image. We seek to model the distribution of parent vectors, p(bt ), such that coefficients in the detailing levels can be sampled conditionally upon coefficients in the approximating levels. Multivariate Gaussian, or mixture of multivariate Gaussian, representations are ideal for this purpose as there is a closed-form solution for the conditional Gaussian. We use a single multivariate Guassian for computational expediency: p(bt ) ∼ N (mt , Σt ). To compute the conditional, we partition the mean vector and covariance matrix of the Gaussian as: m1 Σ11 Σ12 mt = , Σt = m2 Σ21 Σ22
(4)
(5)
where m1 corresponds to the unknown dimensions and m2 corresponds to the known dimensions (the coefficients of the approximating levels); the partitions of the covariance matrix are denoted similarly. If x is the known part of a particular parent vector, the conditioned mean vector and covariance matrix are computed by: −1 m0 = m1 + Σ12 Σ22 (x − m2 ),
−1 Σ 0 = Σ11 − Σ12 Σ22 Σ21 .
(6)
To fully populate a pyramid, given coefficients in the approximating part, the conditional is computed and sampled for each parent vector in the pyramid. The fully-populated pyramid can then be reconstructed into the corresponding image in the shape normalised space.
3.2
Joint Shape and Approximate Appearance Model
To complete synthesis we need to be able to warp a synthetic image in the shape-free space to a plausible shape. We need to take into consideration that there may be a relationship between the appearance of a mammogram and its size and shape, for example fatty breasts tend to be large and glandular breasts tend to be small. We model the joint distribution of shape and approximating appearance parameters and condition this model on the approximating parameters to yield a model of plausible shapes for the generated mammogram. Because the number of training examples is likely to be small compared to the dimensionality of this joint space, we use a single multivariate Gaussian: p(bs , ba ) = p(bj ) ∼ N (mj , Σj ). (7) Given parameters ba describing an approximate mammogram, we can now compute p(bs |ba ) using equation 6 and sample a plausible shape from it.
4
Evaluation and Results
A model was trained with 36 normal mammograms, using 100 boundary landmark points, 7 pyramid levels and 5 orientations. The top three levels were used in the approximating model. The detail model was trained with 100,000 uniformly-sampled parent vectors. It took 24 hours to build the model and 2.5 hours to synthesise a mammogram on a 2.8GHz Intel Xeon processor with 2GB of RAM. Figure 4 shows a real mammogram and three scale for all dimensions. To achieve this we normalise the data in each dimension either to z-scores, or to a common scale using robust estimation, depending upon the characteristics of the data. For expositional simplicity, the conversion to and from these standard scales is implied in the remainder of the text.
synthetic mammograms generated using the model. Note that our method allows us to produce full-resolution synthetic mammograms. We presented a set of 13 real and 13 synthetic full resolution mammograms to an expert mammography radiologist and a digital mammography computer vision researcher. They gave positive feedback, but could identify the synthetic mammograms by their lack of vasculature, lymph nodes and calcifications. These structures exist at the boundary of the approximating and detailing models, and are not captured by our model.
Figure 3. A real mammogram is shown on the left and three synthetic mammograms are shown to its right.
We generated a set of 7 synthetic and 7 real mammograms at 200 × 140 pixel resolution. We selected real mammograms that lacked strong vascular clues. The synthetic images contained information from both the approximating and detailing models. We recruited five computer vision researchers (though not mammography experts) for a forced choice experiment. After studying a training set of 6 real mammograms, the participants were asked to identify the real mammogram from each of the 49 possible pairings of real and synthetic mammograms. None of the subjects believed that they had been able to identify the real mammograms.
χ2 analysis showed that two participants did no better than random (one at the 95% level and one at the lenient 99.9% level). The other three participants differed significantly from random, but consistently mistook the synthetic mammograms for the real ones. Between them, the participants correctly identified 75 real mammograms out of 245 (31%). If we allow consistent misclassification to count as correct identification of the real mammograms, the participants collectively identified 191 real mammograms out of 245 (78%). Although the low resolution synthetic images are not always indistinguishable from real mammograms, they are sufficiently convincing to make discrimination difficult. The fact that several subjects consistently mistook the synthetic mammograms for the real ones implies that the differences were very subtle. Although further improvement is required, the results are extremely promising.
5
Discussion
We have developed a generative statistical model of the appearance of entire mammograms. The model combines ideas from the AAM and wavelet-based ergodic texture models to bypass the ‘curse of dimensionality’. The model can be used to produce synthetic mammograms, which potentially has pedagogic application. Although our model has only been used for image synthesis so far, it has been constructed to be extended to perform CADe via novelty detection. For example, we could easily look at the likelihood of a local collection of parent vectors under our model of normal appearance. Our synthetic mammograms were evaluated by an expert radiologist and 6 computer vision researchers. Although the model cannot yet capture certain structures found in mammograms, results from psychophysical evaluations of synthetic images generated using our model suggest that the method has significant promise.
References 1. L. Tarassenko, P. Hayton, N. Cerneaz et al. “Novelty Detection for the Identification of Masses in Mammograms.” In 4th IEE International Conference on Artificial Neural Networks, pp. 442–447. Cambridge, UK, 1995. 2. P. R. Bakic, M. Albert, D. Brzakovic et al. “Mammogram synthesis using 3D simulation. I. Breast tissue model and image acquisition simulation.” Medical Physics 29, pp. 2131–2139, 2002. 3. C. J. Rose & C. J. Taylor. “A Statistical Model of Texture for Medical Image Synthesis and Analysis.” In Medical Image Understanding and Analysis, pp. 1–4. 2003. 4. C. Spence, L. Parra & P. Sajda. “Detection, Synthesis and Compression in Mammographic Image Analysis with a Hierarchical Image Probability Model.” In L. Staib (editor), IEEE Workshop on Mathematical Methods in Biomedical Image Analysis. 2001. 5. T. F. Cootes, G. J. Edwards & C. J. Taylor. “Active appearance models.” IEEE Transactions on Pattern Analysis Machine Intelligence 23(6), pp. 681–685, 2001. 6. R. H. Davies, C. J. Twining, T. F. Cootes et al. “A Minimum Description Length Approach to Statistical Shape Modelling.” IEEE Transactions on Medical Imaging 2002. 7. E. P. Simoncelli & W. T. Freeman. “The Steerable Pyramid: A Flexible Architecture for Multi-Scale Derivative Computation.” In Second International Conference on Image Processing, volume III, pp. 444–447. IEEE Signal Processing Society, Washington, DC, October 1995.
