Clustering of Gaussian distributions Przemysław Spurek∗ , Wiesław Pałka∗ ∗ Faculty
−1.5
−1.0
−0.5
0.0
0.5
1.0 0.5 0.0 −0.5
● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ●● ● ● ●●● ● ● ●● ●● ●● ●● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ●● ●● ●●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ●● ● ● ●● ● ● ●● ● ● ●●● ● ●● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ●●● ● ● ● ●●● ● ●● ● ●● ● ●● ● ● ● ● ● ● ● ●● ● ●● ● ●● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ●● ● ● ● ● ● ● ● ● ●● ● ● ●● ●● ● ● ● ● ● ●● ● ● ●●●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ● ● ●● ● ● ● ●● ● ●● ● ● ● ●● ● ● ● ● ● ●● ● ● ●● ● ● ● ● ● ●● ●● ● ● ● ●● ● ●● ● ● ● ●● ● ● ● ●● ●● ● ●● ● ● ● ● ●●● ● ●● ● ● ● ●● ●● ● ● ● ●● ● ● ● ● ●● ● ●● ● ● ● ●● ● ● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ● ●● ● ● ● ●●● ●● ● ● ● ● ●● ● ● ●● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ●● ● ● ●●● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ●● ● ● ●● ● ● ● ● ● ● ● ●● ● ● ●●● ● ● ● ●● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ●● ● ● ●● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ●● ● ● ●● ● ● ● ● ● ●● ● ●● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ●● ●● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ●● ● ● ●● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ●● ● ●● ● ● ● ● ●● ●● ● ●● ● ● ●● ● ● ● ● ● ● ● ● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ●● ● ● ●● ●● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ● ●● ● ●● ● ●● ● ●● ● ●● ● ● ● ● ● ● ● ●●● ● ● ● ●● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ●●● ● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ●●●● ● ● ● ● ●● ●● ● ●● ● ● ●● ●●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ●● ● ● ●●● ● ● ● ● ● ● ●● ● ● ● ●● ● ● ● ●● ● ●● ● ●● ● ●● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ●●● ● ● ● ● ●● ● ●● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ●● ●● ●●● ● ●●● ●● ● ● ●● ● ●●● ● ●● ●● ● ●● ● ●● ●● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ●●● ●● ● ● ● ● ●●●● ● ● ● ● ●● ●● ● ● ● ● ● ●● ● ● ● ●● ●● ● ●● ● ● ●● ●● ● ● ● ● ● ● ●● ● ● ● ● ● ● ●● ●● ● ● ●● ● ●● ● ●● ● ● ● ● ●● ●● ● ●
−1.0
0.5 0.0 −0.5 −1.0
1.0
−0.5 −1.0
Clustering [1]–[5] plays a basic role in many parts of data engineering, pattern recognition and image analysis. One of the most important clustering method is Gaussian Mixture Model (GMM) [6], [7], which uses Expectation Maximization (EM) approach. It is hard to overestimate the role of GMM in computer science, for instance in object detection [8], [9]; object tracking [10], [11]; learning and modeling [9], [12]; feature selection [13]; classification [14] or statistic background subtraction [15]. In a family of clustering methods which use Gaussian densities we can find Cross–Entropy Clustering (CEC) [16]. The algorithm uses a cost function which is a small modification of the Maximum Likelihood Estimation (MLE). The main difference between EM and CEC is that CEC is dedicated to clustering instead of density estimation. Consequently, at the small cost of minimally worse density approximation CEC gives us the speed in implementation and the ease of using more complicated density models. The advantage is given by the fact that, roughly speaking, the models do not mix with each other, since we take the maximum instead of the sum. GMM accommodates data with distributions that lie on affine subspaces of lower dimensions obtained by principal components [17]. However, as it often happens clusters are concentrated around lower dimensional manifolds which are non-linear. In such case, one non-Gaussian component is approximated by several Gaussian ones [18] and it may be seen as a form of a piecewise linear approximation, see Fig. 1b. Unfortunately, a vast number of clusters can cause problems. The EM can be unacceptably expensive when dealing with large quantities [19]. The reality is that in many learning processes which use mixture models the computational requirement is very demanding due to the large number of
0.0
I. I NTRODUCTION
−0.5
0.0
0.5
1.0
● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ●● ● ● ●●● ●● ● ● ●● ●● ● ●● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ●● ●● ●●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ●● ● ● ●● ● ● ●● ● ● ●●● ● ●● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ●●● ● ● ● ●●● ● ●● ● ●● ● ●● ● ● ● ● ● ● ● ●● ● ●● ● ●● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ●● ● ● ● ● ● ● ● ● ●● ● ● ●● ●● ● ● ● ● ● ●● ● ● ●●●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ●● ●● ● ● ● ●● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ●● ● ● ●● ● ● ● ● ● ● ● ●● ●● ● ● ● ● ● ● ●● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ●● ● ● ● ● ●● ● ●● ● ● ● ●● ● ● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ●● ● ● ● ●●● ●● ● ● ● ● ●● ● ● ●● ● ● ●● ● ●● ● ● ● ●● ● ● ●● ● ● ● ● ● ● ●● ● ●● ● ● ●●● ● ● ● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ●● ● ● ● ● ●● ● ●● ● ● ●●● ● ● ● ●● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ●● ● ●● ● ● ●● ● ●● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ●● ● ● ●● ● ● ● ● ● ●● ● ●● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ●●● ● ●● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ●● ● ● ● ● ● ● ● ●● ● ● ● ● ● ●● ● ● ● ● ● ●●● ● ●● ● ●● ● ● ● ● ●● ●● ● ●● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ●● ●● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ●●● ●● ● ● ●● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ●● ● ●● ● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ●● ●●● ● ● ● ● ●● ● ●● ● ●● ● ● ● ● ●● ● ● ● ● ●● ●● ● ●● ● ● ●● ●●● ● ● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ● ● ●●● ●● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ●●● ● ● ● ● ● ● ● ●● ● ● ● ●● ● ● ● ●● ● ● ●● ● ●● ● ●● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ●● ●● ● ● ● ●● ● ● ● ● ●● ●● ● ● ● ● ●● ●● ● ●● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●●● ●● ●● ●● ●●●● ● ●● ● ● ● ●● ●● ● ●● ● ● ●● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ●● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ●● ●●● ● ● ● ● ● ● ● ●● ● ● ● ●● ●● ●● ● ● ● ● ●● ●● ● ● ● ● ● ● ●● ● ● ● ● ● ● ●● ●● ● ● ●● ● ●● ● ●● ● ● ● ● ●● ●● ● ●
−1.5
(c) Dendrogram of hierarchical clustering.
−1.0
(b) Gaussian distributions.
1.0
(a) Original two spirals dataset which is represented by densities.
● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ●● ● ● ●●● ● ● ●● ●● ●● ●● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ●● ●● ●●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ●● ● ● ●● ● ● ●● ● ● ●●● ● ●● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ●●● ● ● ● ●●● ● ●● ● ●● ● ●● ● ● ● ● ● ● ● ●● ● ●● ● ●● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ●● ● ● ● ● ● ● ● ● ●● ● ● ●● ●● ● ● ● ● ● ●● ● ● ●●●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ● ● ●● ● ● ● ●● ● ●● ● ● ● ●● ● ● ● ● ● ●● ● ● ●● ● ● ● ● ● ●● ●● ● ● ● ●● ● ●● ● ● ● ●● ● ● ● ●● ●● ● ●● ● ● ● ● ●●● ● ●● ● ● ● ●● ●● ● ● ● ●● ● ● ● ● ●● ● ●● ● ● ● ●● ● ● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ● ●● ● ● ● ●●● ●● ● ● ● ● ●● ● ● ●● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ●● ● ● ●●● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ●● ● ● ●● ● ● ● ● ● ● ● ●● ● ● ●●● ● ● ● ●● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ●● ● ● ●● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ●● ● ● ●● ● ● ● ● ● ●● ● ●● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ●● ●● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ●● ● ● ●● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ●● ● ●● ● ● ● ● ●● ●● ● ●● ● ● ●● ● ● ● ● ● ● ● ● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ●● ● ● ●● ●● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ● ●● ● ●● ● ●● ● ●● ● ●● ● ● ● ● ● ● ● ●●● ● ● ● ●● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ●●● ● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ●●●● ● ● ● ● ●● ●● ● ●● ● ● ●● ●●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ●● ● ● ●●● ● ● ● ● ● ● ●● ● ● ● ●● ● ● ● ●● ● ●● ● ●● ● ●● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ●●● ● ● ● ● ●● ● ●● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ●● ●● ●●● ● ●●● ●● ● ● ●● ● ●●● ● ●● ●● ● ●● ● ●● ●● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ●●● ●● ● ● ● ● ●●●● ● ● ● ● ●● ●● ● ● ● ● ● ●● ● ● ● ●● ●● ● ●● ● ● ●● ●● ● ● ● ● ● ● ●● ● ● ● ● ● ● ●● ●● ● ● ●● ● ●● ● ●● ● ● ● ● ●● ●● ● ●
−1.5
0.5
Abstract—Clustering plays a basic role in many areas of data engineering, pattern recognition and image analysis. Gaussian Mixture Model (GMM) and Cross-Entropy Clustering (CEC) can approximate data of varied shapes by covering it with several clusters e.g. elliptical ones. However, it often happens that we need to extract clusters concentrated around lower dimensional non-linear manifolds. Moreover it is problematic to extract a cluster when data contains a big number of components. Here, we propose a method of solving the above problem by clustering density distribution. This approach allows to determine components of various sizes and geometry. Moreover, it is frequent in clustering problems that the data is naturally given by Gaussian distributions.
1.0
of Mathematics and Computer Science Jagiellonian University Łojasiewicza 6, 30-348 Krak´ow, Poland Email:
[email protected] [email protected]
−1.0
−0.5
0.0
0.5
1.0
(d) Final effect of clustering.
Fig. 1. Clustering of two spirals dataset with use of a general CEC and hierarchical algorithm with Cauchy-Schwarz divergence and single-linkage function.
components involved in the model [20]. Therefore, when working with a large amount of components, at first, we are interested in reducing their number so we can save on computing resources. That is important because in many reallife data examples, each input object is naturally represented by multiple samples drawn from an underlying distribution [21], [22]. Some methods of mixture model simplification have been presented in the last decade. In [20], [23] authors have proposed to simplify GMM which groups similar components together and then performing local fitting through function approximation. On the other hand, we can use a fast GMM simplification algorithm named UTAC (Unscented Transform Approximation Clustering) [24] which uses similarity measure between two mixtures of Gaussian distributions. In [21] authors have shown a hard clustering algorithm based on the decomposition of the relative entropy as the sum of a Burg matrix divergence with a Mahalanobis distance parametrized by the covariance matrices. Similar approach, which uses a generalized distance between exponential distributions, was presented in [25]. In this paper, we propose a solution to both of the above problems: finding curve type structures and reducing unnec-
2
−2
−1
0
1
2
3
(a) The mouse type dataset represented by densities. ● ●● ● ● ●●● ● ● ● ●● ●● ● ● ●● ● ●● ● ● ●● ● ● ●● ● ●● ● ●● ● ● ● ● ● ● ● ● ●●● ● ●● ● ● ● ● ●●●●●● ●● ●● ●● ●●● ● ● ●●● ●● ●● ● ● ● ● ●● ● ●● ● ●● ● ● ● ●● ● ● ● ● ● ● ●●●● ● ●● ● ●● ●● ● ●●● ●● ● ●● ● ●●● ● ●●● ● ●● ● ●● ●●● ●● ●● ● ● ● ●● ● ●●● ● ● ●● ●● ●● ●● ● ●● ●●● ● ● ● ● ● ●● ● ● ● ● ●● ●● ● ● ● ●●● ●● ● ●●● ● ●● ● ●● ● ●● ● ●● ●● ● ● ●● ●● ● ● ●● ●● ● ● ● ● ● ● ● ● ●●● ● ● ● ●● ● ● ●● ● ●● ● ● ● ● ● ● ● ● ● ●● ●● ● ●● ● ● ● ● ● ●●●●● ● ● ● ●● ●●● ●● ● ●● ● ●● ● ●● ● ● ●● ●● ● ●● ● ●● ●● ●●●● ●●● ●● ● ● ●● ●● ● ●● ● ● ●● ●●●● ●● ●● ●● ● ● ● ● ● ●● ● ●●● ● ● ●● ● ● ●●● ●●● ● ● ● ● ●●● ●●●● ● ● ●● ●● ● ● ● ● ● ● ●●● ● ●● ●● ● ●● ● ●●● ● ● ●● ● ● ●● ●● ●●●●● ● ●● ●● ●●● ● ●● ● ● ●● ●● ● ● ●● ●● ● ●● ●● ●● ● ● ●● ●● ● ● ● ● ●●● ●●● ● ●● ● ● ● ● ● ● ● ● ● ● ● ●●●● ● ● ●●● ● ● ● ●● ●● ● ● ● ● ● ●● ●● ● ●● ● ● ●●●● ●● ● ●●●●●● ● ● ●● ● ●●● ● ●● ● ●● ●● ● ● ●● ● ● ●●● ● ●●● ● ●●●● ● ● ● ● ●●●●● ● ●● ● ● ● ● ●● ● ●● ●● ● ● ●●●● ● ● ●● ●● ● ●● ● ●●●● ● ●●● ● ●● ●● ● ●● ● ● ●● ● ●● ● ● ● ● ●● ● ●●● ● ● ● ●●●● ● ● ●● ● ●● ● ● ●● ● ● ● ●● ● ● ●●● ● ● ● ● ● ● ● ●● ●● ●● ● ● ●● ● ● ●● ● ●● ● ●●●● ● ●● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ●●●●● ● ● ● ● ● ● ●● ● ● ●● ● ● ● ● ● ●●●● ● ● ● ●● ● ●● ● ● ● ●●●●● ● ●● ● ● ●● ● ● ●●● ● ●●● ●● ●● ● ● ● ●● ● ● ● ● ● ● ●● ● ● ● ●●●● ● ● ● ●● ●● ●● ●●● ●● ●● ●● ● ● ●● ●● ● ●● ● ● ● ● ●● ● ●●● ● ●●● ● ●● ● ● ●● ● ●● ● ● ● ● ● ●● ● ●● ●● ● ● ● ●● ● ● ●● ● ● ●● ● ● ●● ● ●●● ● ● ● ●● ●● ● ●● ● ● ●●● ● ●● ● ● ●● ●● ● ● ● ●● ● ● ● ● ● ●● ●● ● ●●●● ● ● ● ●●● ● ●● ● ● ●● ●● ● ●● ● ●● ● ● ●● ● ● ●● ● ●● ● ●● ●● ● ● ● ● ●●● ●● ●●●● ● ● ●● ● ●●● ●● ● ●● ● ● ●●● ● ● ●● ●● ●● ●●●● ● ●●● ●● ● ●● ● ●●● ● ● ●●● ● ●● ● ● ● ●● ● ●● ●●● ●● ● ● ●● ● ● ● ●● ●● ●●● ● ● ●● ● ● ● ●● ●●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●●●● ● ●● ●● ●● ● ● ● ● ●●● ● ●● ● ● ●● ● ●● ● ● ● ● ● ●● ● ● ● ●● ● ● ●● ● ●●● ● ●● ● ● ●●● ● ● ●●● ● ● ●● ●● ● ● ● ●●●●● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ●● ● ●● ● ●● ●●● ● ●● ● ●● ●● ● ●● ● ●● ● ●● ● ● ● ● ● ●● ● ● ●●● ● ●● ●● ● ● ● ●● ● ●● ● ● ●●● ● ●● ●● ●● ●●● ● ● ●● ● ● ●●●● ● ● ●● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●●●●● ● ● ● ●● ● ● ● ●● ● ●● ●●● ● ●● ●●●● ● ● ● ● ●● ● ● ● ● ● ●●●● ● ● ● ●●● ● ● ● ●●●●●●●●●● ● ● ●● ● ● ●● ● ●● ●●● ● ● ● ●● ●● ● ●● ●● ● ● ● ● ● ●● ● ● ●●●●●● ● ● ●● ● ●● ● ● ●●● ● ● ●● ● ●● ● ● ●● ● ● ● ●●●● ● ●● ●● ● ● ●● ● ●● ● ● ●● ● ● ●● ●● ●● ● ● ●● ● ● ● ● ●● ● ● ●● ●●● ● ●●● ●●●●● ● ●● ●●●● ● ●● ● ●● ● ●● ●●● ● ●●● ●●● ● ● ●● ● ● ●● ● ●●●● ● ● ● ● ● ●● ● ● ● ●● ● ●●● ● ●● ● ●● ● ● ●●●●● ●● ● ●● ● ● ● ●●●●● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ●●● ● ●● ●● ● ● ● ● ● ●● ● ● ● ●● ●● ● ●● ●● ● ● ●●●●● ● ● ● ● ●● ● ●● ● ● ● ●● ● ●●● ●● ●● ● ●● ● ●● ●● ● ●● ● ● ● ● ● ● ●● ● ●● ● ● ● ●● ●● ● ●● ●● ● ● ● ● ●● ●● ● ● ● ● ●●● ●● ● ● ● ●● ● ● ●● ● ● ● ● ● ● ● ●● ● ● ●● ● ●● ●● ● ●● ● ● ●● ●●●● ● ● ● ● ● ●● ● ●● ● ● ●● ● ● ● ● ● ● ● ● ●● ● ●● ●●● ●● ● ● ● ●●●● ●● ●●● ●● ● ●● ●● ●● ●●● ●●● ● ● ● ● ● ●●● ● ●● ●● ●● ●●● ● ● ●● ● ● ●● ● ● ●● ● ● ● ● ●● ●● ● ● ●●●● ●●● ● ● ● ●● ● ●● ●●●●● ● ●●● ●● ●● ● ●●● ●●● ●●● ● ●●● ● ●●● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ● ● ●●● ● ●● ●● ●● ● ●● ●●● ●● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ●● ●● ● ● ● ●●● ●● ● ● ● ●●● ●● ● ● ●●● ●● ● ● ● ●● ● ●●●● ● ● ● ● ● ● ● ●●● ● ● ● ● ●●● ● ● ●● ● ●●● ●● ●● ●●● ● ● ●● ● ●●● ● ● ●● ● ● ●● ● ● ● ●● ● ●● ● ● ●● ● ● ● ● ● ●● ● ●●● ● ●● ●● ● ●● ●●● ● ●● ● ● ●●●●●● ●● ●● ● ● ● ● ● ●● ● ● ● ●● ●● ●● ● ●●●● ● ● ● ● ● ● ● ●●● ●● ● ●●● ● ● ● ● ●● ● ●● ● ●●●● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ●●●●●●● ●● ● ●●● ● ●●●●●● ● ●● ● ● ●●●●●●● ● ● ●● ●● ● ● ●● ● ●● ● ● ● ●● ● ● ● ● ●● ●●●●● ●● ● ● ●● ● ●●● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ●●● ●●● ● ● ●● ●● ● ● ● ●● ●● ● ● ●●● ●●●● ●● ● ● ●●●● ● ● ● ● ● ●●● ● ● ●●● ●● ●●● ● ● ● ● ●● ● ●● ● ● ●● ● ● ● ● ● ● ● ● ●● ●●●● ● ● ● ● ● ●● ● ●●● ● ● ●● ● ●●● ●● ● ● ● ● ● ● ●●● ● ● ●●●●● ●●● ● ●●● ●●●●● ● ● ● ●● ● ●●●● ● ●●● ● ●● ● ●● ●● ● ●● ● ● ●● ● ●●● ●● ●● ●●●●● ●● ● ● ● ● ● ● ●● ● ● ● ●● ●● ●●● ● ●● ● ● ●● ●● ●● ●● ●● ● ●●● ● ● ● ● ● ●● ● ● ●● ●● ●● ●●●● ●● ●● ● ● ● ● ●● ● ● ●●●● ● ●●● ●● ● ● ●●● ● ●● ● ●● ● ●●● ● ● ● ●●●● ● ●●●● ● ● ●● ● ● ● ●●●● ● ●● ● ● ● ● ● ● ● ●●●● ●●● ● ●● ● ●● ●● ●● ●●● ● ● ● ● ●● ● ●●● ● ●● ● ● ● ●● ● ● ●●● ●● ● ●●● ● ●● ● ● ●●● ●●●●● ● ● ● ● ● ● ●● ● ● ● ● ●● ●●● ●●●● ●●●● ● ●● ● ●● ● ● ● ● ● ● ● ●● ●●● ● ● ● ● ● ● ● ●● ●● ● ●● ● ● ●●●●● ●●●● ● ● ● ● ● ●● ● ● ●● ●● ● ●●●●● ● ● ●●●● ●● ● ● ●●●● ● ●● ● ●●● ●● ● ●● ● ●● ● ●● ● ●● ●● ● ●● ●●●●● ● ● ● ● ●● ●● ●●● ●●●● ● ● ● ●● ● ● ● ●● ● ● ● ●● ● ● ●● ●● ●● ●● ● ●●● ●●● ● ●● ●●● ● ●●●● ●● ● ●● ● ● ●●●● ● ● ●● ● ● ● ● ● ● ● ●● ● ● ●●● ●●●● ● ●● ●● ●● ● ●● ● ● ● ● ●● ● ● ●● ● ●● ● ● ● ● ● ● ● ●●●●● ●●● ● ●● ●●●●●● ● ●● ● ● ●● ● ● ● ●● ●● ●● ● ● ●●● ●● ●●● ●●●● ● ● ●● ● ● ● ●● ●● ●●● ● ● ● ●● ●● ● ●● ●●● ●●● ● ●● ● ● ● ● ● ●● ● ● ● ●● ● ● ●● ● ● ● ● ●● ●●● ●● ● ● ● ● ●●● ●●● ● ● ●● ● ●● ● ●● ●●● ● ● ●● ● ● ● ● ● ● ● ●●●●●● ● ●● ● ● ● ● ● ●●● ●●● ● ● ● ●● ● ●● ● ●●●● ● ●● ●●●●● ●●●●● ● ● ● ● ● ●● ● ●● ●● ● ●● ●● ● ● ●●●●● ● ●● ● ●●● ● ●●● ● ● ●● ● ● ● ●●● ● ●● ● ●● ● ● ● ● ● ● ● ● ● ●●● ● ● ●●● ● ●● ● ● ● ● ● ●● ●●●●●● ●● ● ● ● ●●● ● ● ●● ●●●●● ● ● ● ● ● ●● ● ● ●● ● ●● ●●● ●● ● ● ●●●● ● ● ● ●● ●● ●● ● ● ●● ● ●● ● ● ●● ● ● ● ●● ● ● ●●●● ●● ●●● ● ●●● ● ● ● ●●●● ●● ● ●● ● ●● ●● ● ●●●● ●●● ●● ● ● ●●● ● ● ● ● ● ● ●● ● ● ● ● ● ●● ● ● ●● ● ● ● ●● ●● ●●● ●● ●● ● ● ●● ●● ● ● ●● ●● ● ● ● ●● ● ● ● ● ● ● ●●● ●● ●● ● ●● ● ●●● ● ● ● ●● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ●● ●●●● ●● ● ●●●● ● ● ● ●● ● ● ● ● ● ●● ●● ●●● ● ●● ● ● ●● ●● ●● ● ● ● ● ●● ● ● ● ● ● ● ●● ●● ● ●● ● ●● ● ● ● ● ● ● ● ● ●● ●● ● ●●● ● ● ●● ●●● ● ●●● ● ●●●● ●● ●● ● ● ●● ● ● ●● ●● ● ● ● ● ●●●●● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●●●● ● ● ● ● ●●●●● ● ● ●● ● ● ● ● ● ● ● ●●● ● ● ● ●●●● ● ●● ●● ● ● ●●●● ● ●● ● ●● ●●● ● ●●● ●● ● ●●●● ● ● ● ●●● ● ● ●●● ●●●●●●● ●●● ● ● ●●● ●● ● ● ●● ● ● ● ● ●● ●●● ● ● ● ● ● ●● ● ● ●● ● ● ● ●● ● ● ●● ● ●●● ● ● ●●●●● ● ● ●● ● ● ● ● ●● ● ● ●●● ●●●●● ● ●● ●●● ● ●● ●● ● ● ● ● ● ●● ●● ● ● ● ●● ● ● ● ● ●● ●●● ●●●● ●● ●● ● ● ●● ●● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ●● ●●● ●●●●●● ●● ● ●●● ● ●● ●● ● ● ●● ● ● ● ● ● ●●●● ● ● ● ● ● ●●● ● ●●● ●● ● ● ●● ●● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ●●● ●●● ● ● ● ●● ● ●● ● ● ● ●
−3
−2
−1
0
1
2
3 2 1 0 −1 −2
3
3
(c) Results of clustering the densities.
● ●● ● ● ●●● ● ● ● ●● ●● ● ● ●● ● ●● ● ● ●● ● ● ●● ● ●● ● ●● ● ● ● ● ● ● ● ● ●●● ● ●● ● ● ● ● ●●●●●● ●● ●● ●● ●●● ● ● ●●● ●● ●● ● ● ● ● ●● ● ●● ● ●● ● ● ● ●● ● ● ● ● ● ● ●●●● ● ●● ● ●● ●● ● ●●● ●● ● ●● ● ●●● ● ●●● ● ●● ● ●● ●●● ●● ●● ● ● ● ●● ● ●●● ● ● ●● ●● ●● ●● ● ●● ●●● ● ● ● ● ● ●● ● ● ● ● ●● ●● ● ● ● ●●● ●● ● ●● ●● ● ● ●●● ● ●● ● ●● ● ●● ● ●● ●● ● ● ●● ●● ● ● ● ● ● ● ● ● ●●● ● ● ● ●● ● ● ●● ● ●● ● ● ● ● ● ● ● ● ● ●● ●● ● ●● ● ● ● ● ● ●●●●● ● ● ● ●● ●●● ●● ● ●● ● ●● ● ●● ● ● ●● ●● ● ●● ● ●● ●● ●●●● ●●● ●● ● ● ●● ●● ● ●● ● ● ●● ●●●● ●● ●● ●● ● ● ● ● ● ●● ● ●●● ● ● ●● ● ● ●●● ●● ● ● ● ● ●●● ●●●● ● ● ●● ●● ● ● ● ● ● ● ●●● ● ●● ●● ● ●● ● ●●● ● ● ●● ● ● ●● ●● ●●●●● ● ●● ●● ● ●● ● ●● ● ● ●● ●● ● ● ●● ●● ● ●● ●● ●● ● ● ●● ●● ● ● ● ● ●●● ●●● ● ●● ● ● ● ● ● ● ● ● ● ● ● ●●●● ● ● ●●● ● ● ● ●● ●● ● ● ● ● ● ●● ●● ● ●● ● ● ●●●● ●● ● ●●●●●● ● ● ●● ● ●●● ● ●● ● ●● ●● ● ● ●● ● ● ●●● ● ●●● ● ●●●● ● ● ● ● ●●●●● ● ●● ● ● ● ● ●● ● ●● ●● ● ● ●●●● ● ● ●● ●● ● ●● ● ●●●● ● ●●● ● ●● ●● ● ●● ● ● ●● ● ●● ● ● ● ● ●● ● ●●● ● ● ● ●●●● ● ● ●● ● ●● ● ● ●● ● ● ● ●● ● ● ●●● ● ● ● ● ● ● ● ●● ●● ●● ● ● ●● ● ● ●● ● ●● ● ●●●● ● ●● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ●●●●● ● ● ● ● ● ● ●● ● ● ●● ● ● ● ● ● ●●●● ● ● ● ●● ● ●● ● ● ● ●●●●● ● ●● ● ● ●● ● ● ●●● ● ●●● ●● ●● ● ● ● ●● ● ● ● ● ● ● ●● ● ● ● ●●●● ● ● ● ●● ●● ●● ●●● ●● ●● ●● ● ● ●● ●● ● ●● ● ● ● ● ●● ● ●●● ● ●●● ● ●● ● ● ●● ● ●● ● ● ● ● ● ●● ● ●● ●● ● ● ● ●● ● ● ●● ● ● ●● ● ● ●● ● ●●● ● ● ● ●● ●● ● ●● ● ● ●●● ● ●● ● ● ●● ●● ● ● ● ●● ● ● ● ● ● ●● ●● ● ●●●● ● ● ● ●●● ● ●● ● ● ●● ●● ● ●● ● ●● ● ● ●● ● ● ●● ● ●● ● ●● ●● ● ● ● ● ●●● ●● ●●●● ● ● ●● ● ●●● ●● ● ●● ● ● ●●● ● ● ●● ●● ●● ●●●● ● ●●● ● ●● ● ●●● ● ● ●●● ● ●● ● ● ●●● ●● ● ●● ●●● ●● ● ● ● ● ● ● ●● ●● ● ●●● ● ● ●● ● ● ● ● ● ●● ● ● ● ● ● ●●●● ● ● ● ● ● ● ● ● ● ● ● ●●● ●●●●●● ● ● ●● ●● ● ● ● ● ●● ●●● ●●● ●● ●● ● ● ● ● ●●●● ●●● ●● ●● ●● ● ● ● ● ●●● ●●● ● ● ● ● ● ● ●● ● ● ●●● ●●● ● ●● ● ● ● ● ●●● ● ● ● ● ●● ●● ● ● ● ● ●● ● ● ● ●● ● ●● ●●● ● ●●● ● ●● ●● ● ● ●● ● ● ●● ● ●● ● ●● ●● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ●● ● ● ●●● ● ●● ● ● ●● ●● ● ● ● ● ●● ●● ● ● ●●● ● ●● ●● ● ●● ●● ● ● ● ●● ● ● ●● ● ●●● ● ● ●●● ● ● ● ● ● ●●●● ●●●●● ● ● ● ●● ● ●● ● ● ● ●● ● ●● ●●● ●● ● ● ●● ● ●● ● ●●●● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ●● ●● ●● ● ●● ● ● ● ● ● ●● ● ● ●●● ● ● ● ● ● ● ●● ●● ●● ● ●● ●●● ● ●● ● ●● ● ●● ● ●● ● ● ●● ●● ● ● ●● ● ●● ● ●● ●● ● ●● ●● ●● ● ● ● ● ●● ● ● ● ● ●● ● ● ●● ● ● ● ●●● ●●●●● ● ●● ●●● ● ●● ● ●● ● ●● ●●● ● ●●● ●●● ● ● ●● ● ● ●● ● ●●●● ● ● ● ●● ● ●● ● ● ● ● ● ●●● ● ●● ● ● ● ●● ● ●●●●● ●● ● ● ●●● ●● ●● ●● ●● ●● ● ●● ●● ● ● ●●●●● ● ●● ● ●● ●● ● ●● ● ● ●●● ● ●● ● ●●● ●● ● ● ●● ● ●● ● ● ● ● ● ●●● ●● ●● ●● ● ● ● ●●● ●● ● ●● ● ● ● ● ● ●● ● ●●● ●● ● ●● ●● ● ●● ● ●● ● ●● ● ●● ● ● ●● ● ●● ●●● ● ●● ● ● ●● ●● ● ● ● ● ● ●● ●● ● ● ● ● ●●● ●● ● ● ● ● ●● ● ● ●● ● ● ● ●● ● ● ● ● ●● ● ● ●● ● ●● ●● ● ●● ● ● ●● ●●●● ● ● ● ●● ● ●● ● ● ●● ● ● ● ● ● ● ● ●● ● ● ●● ● ●● ●●● ●● ● ● ● ●●●● ●● ●●● ●● ● ●● ●● ●● ●●● ● ● ●● ●●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ●●● ● ● ●● ● ● ● ● ●● ●● ● ● ● ● ● ●● ●●● ● ● ●●● ● ● ● ● ● ●●● ●●● ●●● ● ●●● ● ● ● ●● ●● ● ● ● ● ● ● ● ● ●●● ● ● ● ●● ● ●● ●●● ●● ● ● ●● ●●● ●● ●● ● ●●● ●● ●● ●● ● ● ● ● ●●●● ● ● ● ● ●● ● ●●●●●●● ● ● ● ● ● ●●● ● ● ● ● ● ●●● ●● ●● ● ●● ● ●●●● ● ●● ● ● ● ●● ● ● ● ● ●●● ● ● ● ● ●● ● ● ● ●● ● ●●● ●● ●● ● ● ● ● ● ●● ● ● ●● ● ●● ● ●● ● ● ●● ● ●● ● ● ●● ●● ●● ● ● ●●● ● ●●● ● ● ● ● ●● ● ●● ● ● ● ●● ● ●● ●● ● ● ●● ● ● ●●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ●●● ● ●●● ● ● ●●● ●● ● ● ● ● ●● ●●●● ● ● ● ● ● ● ● ● ● ●● ● ● ●● ● ●●● ●●●●●●● ●● ● ●●● ● ●● ● ●● ● ● ●●●●●●● ● ● ●● ●● ● ● ●● ● ●● ● ●● ● ● ● ●● ● ● ●● ●●●●● ●● ● ● ● ● ●● ● ● ●●●● ●● ●●●● ●● ● ●● ●● ●● ● ● ● ●● ●●● ● ● ●● ● ● ●● ● ●●● ● ●● ●● ● ● ● ●● ● ● ● ●●● ●●●● ●● ● ● ●●● ● ● ● ● ● ●●● ● ● ● ●●● ●● ●●● ● ● ● ● ●● ● ●● ● ● ●● ● ● ● ● ● ● ● ● ●● ●●●● ● ● ● ● ● ●● ● ●●● ● ● ●● ● ●●● ●● ● ● ● ● ● ● ●●● ● ● ●●●●● ● ● ●●● ● ●●● ●●●●● ● ● ● ● ●●● ● ●● ● ●● ●● ● ● ●●●● ●●●● ● ● ● ●● ●● ● ● ●● ● ●●● ●● ●● ● ● ● ●● ● ● ● ●● ●● ●●● ● ●● ● ● ●● ●● ●● ● ●●● ● ● ●● ● ● ● ● ● ● ●● ●● ●● ● ●● ●● ● ● ● ● ● ●●● ●● ● ● ●●●● ●●●● ● ●●●● ● ●●● ●● ● ● ●●● ● ●● ● ● ●● ●●● ● ● ● ● ●● ● ● ● ● ●●●● ●● ● ●● ●●●● ● ● ● ● ● ●●●● ●●● ● ●● ● ●● ●●● ● ●●● ●● ● ● ●● ● ●●● ● ●● ● ● ● ●● ●●● ●● ● ●●● ● ●● ● ● ●●● ●●●●● ● ● ● ● ● ● ●● ● ● ● ● ● ●● ●●● ●●●● ●●●● ● ●● ● ● ● ● ● ● ● ● ●● ●●● ● ● ● ● ●●● ● ● ●● ●● ● ●● ● ● ●●●●● ●●●● ● ● ● ● ●● ● ● ●● ●● ● ●●●●● ● ● ●●●● ●● ● ● ●●●● ● ●● ● ●●● ●● ● ●● ● ●● ● ●● ● ●● ●● ● ●● ●●●●● ● ● ● ● ●● ●● ●●● ●●●● ● ● ● ●● ● ● ● ●● ● ● ● ●● ● ● ●● ●● ●● ●● ● ●●● ●●● ● ●● ●●● ● ●●●● ●● ● ●● ● ● ●●●● ● ● ●● ● ● ● ● ● ● ● ●● ● ● ●●● ●●●● ● ●● ●● ●● ● ●● ● ● ● ● ●● ● ● ●● ● ●● ● ● ● ● ● ● ● ●●●●● ●●● ● ●● ●●●●●● ● ●● ● ● ●● ● ● ● ●● ●● ●● ● ● ●●● ●● ●●● ●●●● ● ● ●● ● ● ● ●● ●● ●●● ● ● ● ●● ●● ● ●● ●●● ●●● ● ●● ● ● ● ● ● ●● ● ● ● ●● ● ● ●● ● ● ● ●●●● ●●● ●● ● ● ● ● ●●● ●●● ● ● ●● ● ●● ● ●● ●●● ● ● ●● ● ● ● ● ● ● ● ●●●●● ● ●● ● ● ● ● ● ●●● ●●● ● ● ● ●● ● ●● ● ●●●● ● ●● ●●●●● ●●●●● ● ● ● ● ● ●● ● ●● ●● ● ●● ●● ● ● ●●●●● ● ●● ● ●●● ● ●●● ● ● ●● ● ● ● ●●● ● ●● ● ●● ● ● ● ● ● ● ● ● ● ●●● ● ● ●●● ● ●● ● ● ● ● ● ●● ●●●●●● ●● ● ● ● ●●● ● ● ●● ●●●●● ● ● ● ● ● ●● ● ● ●● ● ●●●●● ●● ● ● ●●●● ● ● ● ●● ●● ●● ● ● ●● ● ●● ● ● ●● ● ● ● ●● ● ●●●●● ●● ●●● ● ●●● ● ● ● ●●●● ●● ● ●● ● ●● ●● ● ●●●● ●●● ●● ● ● ●●● ● ● ● ● ● ●● ● ● ● ● ● ●● ● ● ●● ● ●● ●● ●●● ●● ●● ● ● ●● ●● ● ● ● ●● ● ●● ●● ● ● ● ●● ● ● ● ● ● ● ● ●● ● ● ●●● ●● ●● ● ●● ● ●●● ● ● ● ●● ● ● ● ● ●●● ● ● ●●● ● ● ●● ●● ● ● ●● ●●●● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ●● ● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ● ●● ● ● ●● ● ●● ● ●● ● ●● ● ● ● ● ● ● ● ●● ●● ● ●●● ● ● ●● ●●● ● ●●● ● ●●● ●● ●● ● ● ●● ● ●● ●● ● ● ● ●● ● ● ● ● ● ● ●● ● ●● ●●●●● ● ● ● ●● ● ● ● ● ●● ●● ● ● ● ● ● ● ●●●● ● ● ●● ● ●●●●●●● ● ●● ● ● ● ● ● ● ● ●●● ● ● ● ●●●● ● ●● ●● ● ● ●●●● ● ●● ● ●● ●●● ● ●●● ●● ● ●●●● ● ● ● ●●● ● ● ●●● ●●●●●●● ●●● ● ● ●●● ●● ● ● ●● ● ● ●● ●●● ● ● ●● ● ●●● ● ● ●● ● ●● ●● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●● ● ●● ●●● ● ●●●● ● ● ●●● ● ●● ●● ● ● ● ● ●● ● ● ● ●● ●● ● ● ● ● ●● ● ● ●● ●● ●● ●●●● ●● ●● ● ● ●● ●● ● ●● ● ● ● ● ● ●● ● ● ● ● ● ● ●●● ●● ●●● ●●●●●● ●● ● ●●● ● ●● ●● ● ● ●● ● ● ● ● ● ●●●● ● ● ● ● ● ●●● ● ●●● ●● ● ● ●● ●● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ●●● ●●● ● ● ● ●● ● ●● ● ● ● ●
−3
−2
−1
0
1
2
3
(b) Densities which will be clustered.
3
1
2
0
1
−1
0
−2
−1
−3
−2
3 2 1 0 −1 −2
● ●● ● ● ●●● ● ● ● ●● ●● ● ● ●● ● ●● ● ● ●● ● ● ●● ● ●● ● ●● ● ● ● ● ● ● ● ● ●●● ● ●● ● ● ● ● ●●●●●● ●● ●● ●● ●●● ● ● ●●● ●● ●● ● ● ● ● ●● ● ●● ● ●● ● ● ● ●● ● ● ● ● ● ● ●●●● ● ●● ● ●● ●● ● ●●● ●● ● ●● ● ●●● ● ●●● ● ●● ● ●● ●●● ●● ●● ● ● ● ●● ● ●●● ● ● ●● ●● ●● ●● ● ●● ●●● ● ● ● ● ● ●● ● ● ● ● ●● ●● ● ● ● ●●● ●● ● ●● ●● ● ● ●●● ● ●● ● ●● ● ●● ● ●● ●● ● ● ●● ●● ● ● ● ● ● ● ● ● ●●● ● ● ● ●● ● ● ●● ● ●● ● ● ● ● ● ● ● ● ● ●● ●● ● ●● ● ● ● ● ● ●●●●● ● ● ● ●● ●●● ●● ● ●● ● ●● ● ●● ● ● ●● ●● ● ●● ● ●● ●● ●●●● ●●● ●● ● ● ●● ●● ● ●● ● ● ●● ●●●● ●● ●● ●● ● ● ● ● ● ●● ● ●●● ● ● ●● ● ● ●●● ●●● ● ● ● ● ●●● ●●●● ● ● ●● ●● ● ● ● ● ● ● ●●● ● ●● ●● ● ●● ● ●●● ● ● ●● ● ● ●● ●● ●●●●● ● ●● ●● ●●● ● ●● ● ● ●● ●● ● ● ●● ●● ● ●● ●● ●● ● ● ●● ●● ● ● ● ● ●●● ●●● ● ●● ● ● ● ● ● ● ● ● ● ● ● ●●●● ● ● ●●● ● ● ● ●● ●● ● ● ● ● ● ●● ●● ● ●● ● ● ●●●● ●● ● ●●●●●● ● ● ●● ● ●●● ● ●● ● ●● ●● ● ● ●● ● ● ●●● ● ●●● ● ●●●● ● ● ● ● ●●●●● ● ●● ● ● ● ● ●● ● ●● ●● ● ● ●●●● ● ● ●● ●● ● ●● ● ●●●● ● ●●● ● ●● ●● ● ●● ● ● ●● ● ●● ● ● ● ● ●● ● ●●● ● ● ● ●●●● ● ● ●● ● ●● ● ● ●● ● ● ● ●● ● ● ●●● ● ● ● ● ● ● ● ●● ●● ●● ● ● ●● ● ● ●● ● ●● ● ●●●● ● ●● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ●●●●● ● ● ● ● ● ● ●● ● ● ●● ● ● ● ● ● ●●●● ● ● ● ●● ● ●● ● ● ● ●●●●● ● ●● ● ● ●● ● ● ●●● ● ●●● ●● ●● ● ● ● ●● ● ● ● ● ● ● ●● ● ● ● ●●●● ● ● ● ●● ●● ●● ●●● ●● ●● ●● ● ● ●● ●● ● ●● ● ● ● ● ●● ● ●●● ● ●●● ● ●● ● ● ●● ● ●● ● ● ● ● ● ●● ● ●● ●● ● ● ● ●● ● ● ●● ● ● ●● ● ● ●● ● ●●● ● ● ● ●● ●● ● ●● ● ● ●●● ● ●● ● ● ●● ●● ● ● ● ●● ● ● ● ● ● ●● ●● ● ●●●● ● ● ● ●●● ● ●● ● ● ●● ●● ● ●● ● ●● ● ● ●● ● ● ●● ● ●● ● ●● ●● ● ● ● ● ●●● ●● ●●●● ● ● ●● ● ●●● ●● ● ●● ● ● ●●● ● ● ●● ●● ●● ●●●● ● ●●● ● ●● ● ●●● ● ● ●●● ● ●● ● ● ●●● ●● ● ●● ●●● ●● ● ● ● ● ● ● ●● ●● ● ●●● ● ● ●● ● ● ● ● ● ●● ● ● ● ● ● ●●●● ● ● ● ● ● ● ● ● ● ● ● ●●● ●●●●●● ● ● ●● ●● ● ● ● ● ●● ●●● ●●● ●● ●● ● ● ● ● ●●●● ●●● ●● ●● ●● ● ● ● ● ●●● ●●● ● ● ● ● ● ● ●● ● ● ●●● ●●● ● ●● ● ● ● ● ●●● ● ● ● ● ●● ●● ● ● ● ● ●● ● ● ● ●● ● ●● ●●● ● ●●● ● ●● ●● ● ● ●● ● ● ●● ● ●● ● ●● ●● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ●● ● ● ●●● ● ●● ● ● ●● ●● ● ● ● ● ●● ●● ● ● ●●● ● ●● ●● ● ●● ●● ● ● ● ●● ● ● ●● ● ●●● ● ● ●●● ● ● ● ● ● ●●●● ●●●●● ● ● ● ●● ● ●● ● ● ● ●● ● ●● ●●● ●● ● ● ●● ● ●● ● ●●●● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ●● ●● ●● ● ●● ● ● ● ● ● ●● ● ● ●●● ● ● ● ● ● ● ●● ●● ●● ● ●● ●●● ● ●● ● ●● ● ●● ● ●● ● ● ●● ●● ● ● ●● ● ●● ● ●● ●● ● ●● ●● ●● ● ● ● ● ●● ● ● ● ● ●● ● ● ●● ● ● ● ●●● ●●●●● ● ●● ●●●● ● ●● ● ●● ● ●● ●●● ● ●●● ●●● ● ● ●● ● ● ●● ● ●●●● ● ● ● ●● ● ●● ● ● ● ● ● ●●● ● ●● ● ● ● ●● ● ●●●●● ●● ● ● ●●● ●● ●● ●● ●● ●● ● ●● ●● ● ● ●●●●● ● ●● ● ●● ●● ● ●● ● ● ●●● ● ●● ● ●●● ●● ● ● ●● ● ●● ● ● ● ● ● ●●● ●● ●● ●● ● ● ● ●●● ●● ● ●● ● ● ● ● ● ●● ● ●●● ●● ● ●● ●● ● ●● ● ●● ● ●● ● ●● ● ● ●● ● ●● ●●● ● ●● ● ● ●● ●● ● ● ● ● ● ●● ●● ● ● ● ● ●●● ●● ● ● ● ● ●● ● ● ●● ● ● ● ●● ● ● ● ● ●● ● ● ●● ● ●● ●● ● ●● ● ● ●● ●●●● ● ● ● ●● ● ●● ● ● ●● ● ● ● ● ● ● ● ●● ● ● ●● ● ●● ●●● ●● ● ● ● ●●●● ●● ●●● ●● ● ●● ●● ●● ●●● ● ● ●● ●●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ●●● ● ● ●● ● ● ● ● ●● ●● ● ● ● ● ● ●● ●●● ● ● ●●● ● ● ● ● ● ●●● ●●● ●●● ● ●●● ● ● ● ●● ●● ● ● ● ● ● ● ● ● ●●● ● ● ● ●● ● ●● ●●● ●● ● ● ●● ●●● ●● ●● ● ●●● ●● ●● ●● ● ● ● ● ●●●● ● ● ● ● ●● ● ●●●●●●● ● ● ● ● ● ●●● ● ● ● ● ● ●●● ●● ●● ● ●● ● ●●●● ● ●● ● ● ● ●● ● ● ● ● ●●● ● ● ● ● ●● ● ● ● ●● ● ●●● ●● ●● ● ● ● ● ● ●● ● ● ●● ● ●● ●● ● ● ●● ● ●● ● ● ●● ●● ●● ● ● ●●● ● ●●● ● ● ● ●● ●● ● ●● ● ● ● ●● ● ●●● ●● ● ● ●● ● ●●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ●●● ● ●●● ● ● ●●● ●● ● ● ● ● ●● ●●●● ● ● ● ● ● ● ● ● ● ●● ● ● ●● ● ●●● ●●●●●●● ●● ● ●●● ● ●● ● ●● ● ● ●●●●●●● ● ● ●● ●● ● ● ●● ● ●● ● ●● ● ● ● ●● ● ● ●● ●●●●● ●● ● ● ● ● ●● ● ● ●●●● ●● ●●●● ●● ● ●● ●● ●● ● ● ● ●● ●●● ● ● ●● ● ● ●● ● ●●● ● ●● ●● ● ● ● ●● ● ● ● ●●● ●●●● ●● ● ● ●●● ● ● ● ● ● ●●● ● ● ●●● ●● ●●● ● ● ● ● ●● ● ●● ● ● ●● ● ● ● ● ● ● ● ● ●● ●●●● ● ● ● ● ● ●● ● ●●● ● ● ●● ● ●●● ●● ● ● ● ● ● ● ●●● ● ● ●●●●● ● ● ●●● ● ●●● ●●●●● ● ● ● ● ●●● ● ●● ● ●● ●● ● ● ●●●● ●●●● ● ● ● ●● ●● ● ● ●● ● ●●● ●● ●● ● ● ● ●● ● ● ● ●● ●● ●●● ● ●● ● ● ●● ●● ●● ● ●●● ● ● ●● ● ● ● ● ● ● ●● ●● ●● ● ●● ●● ● ● ● ● ● ●●● ●● ● ● ●●●● ●●●● ● ●●●● ● ●●● ●● ● ● ●●● ● ●● ● ● ●● ●●● ● ● ● ● ●● ● ● ● ● ●●●● ●● ● ●● ●●●● ● ● ● ● ● ●●●● ●●● ● ●● ● ●● ●●● ● ●●● ●● ● ● ●● ● ●●● ● ●● ● ● ● ●● ●●● ●● ● ●●● ● ●● ● ● ●●● ●●●●● ● ● ● ● ● ● ●● ● ● ● ● ● ●● ●●● ●●●● ●●●● ● ●● ● ● ● ● ● ● ● ● ●● ●●● ● ● ● ● ●●● ● ● ●● ●● ● ●● ● ● ●●●●● ●●●● ● ● ● ● ●● ● ● ●● ●● ● ●●●●● ● ● ●●●● ●● ● ● ●●●● ● ●● ● ●●● ●● ● ●● ● ●● ● ●● ● ●● ●● ● ●● ●●●●● ● ● ● ● ●● ●● ●●● ●●●● ● ● ● ●● ● ● ● ●● ● ● ● ●● ● ● ●● ●● ●● ●● ● ●●● ●●● ● ●● ●●● ● ●●●● ●● ● ●● ● ● ●●●● ● ● ●● ● ● ● ● ● ● ● ●● ● ● ●●● ●●●● ● ●● ●● ●● ● ●● ● ● ● ● ●● ● ● ●● ● ●● ● ● ● ● ● ● ● ●●●●● ●●● ● ●● ●●●●●● ● ●● ● ● ●● ● ● ● ●● ●● ●● ● ● ●●● ●● ●●● ●●●● ● ● ●● ● ● ● ●● ●● ●●● ● ● ● ●● ●● ● ●● ●●● ●●● ● ●● ● ● ● ● ● ●● ● ● ● ●● ● ● ●● ● ● ● ● ●● ●●● ●● ● ● ● ● ●●● ●●● ● ● ●● ● ●● ● ●● ●●● ● ● ●● ● ● ● ● ● ● ● ●●●●●● ● ●● ● ● ● ● ● ●●● ●●● ● ● ● ●● ● ●● ● ●●●● ● ●● ●●●●● ●●●●● ● ● ● ● ● ●● ● ●● ●● ● ●● ●● ● ● ●●●●● ● ●● ● ●●● ● ●●● ● ● ●● ● ● ● ●●● ● ●● ● ●● ● ● ● ● ● ● ● ● ● ●●● ● ● ●●● ● ●● ● ● ● ● ● ●● ●●●●●● ●● ● ● ● ●●● ● ● ●● ●●●●● ● ● ● ● ● ●● ● ● ●● ● ●● ●●● ●● ● ● ●●●● ● ● ● ●● ●● ●● ● ● ●● ● ●● ● ● ●● ● ● ● ●● ● ● ●●●● ●● ●●● ● ●●● ● ● ● ●●●● ●● ● ●● ● ●● ●● ● ●●●● ●●● ●● ● ● ●●● ● ● ● ● ● ●● ● ● ● ● ● ●● ● ● ●● ● ●● ●● ●●● ●● ●● ● ● ●● ●● ● ● ● ●● ● ●● ●● ● ● ● ●● ● ● ● ● ● ● ● ●● ● ● ●●● ●● ●● ● ●● ● ●●● ● ● ● ●● ● ● ● ● ●●● ● ● ●●● ● ● ●● ●● ● ● ●● ●●●● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ●● ● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ● ●● ● ● ●● ● ●● ● ●● ● ●● ● ● ● ● ● ● ● ●● ●● ● ●●● ● ● ●● ●●● ● ●●● ● ●●● ●● ●● ● ● ●● ● ●● ●● ● ● ● ●● ● ● ● ● ● ● ●● ● ●● ●●●●● ● ● ● ●● ● ● ● ● ●● ●● ● ● ● ● ● ● ●●●● ● ● ●● ● ●●●●●●● ● ●● ● ● ● ● ● ● ● ●●● ● ● ● ●●●● ● ●● ●● ● ● ●●●● ● ●● ● ●● ●●● ● ●●● ●● ● ●●●● ● ● ● ●●● ● ● ●●● ●●●●●●● ●●● ● ● ●●● ●● ● ● ●● ●● ● ●● ● ● ●● ●●● ● ● ● ●● ● ●● ●● ● ● ●● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ●● ● ●● ●●● ● ●●●● ● ● ●●● ● ●● ●● ● ● ● ● ●● ● ● ● ●● ●● ● ● ● ● ●● ● ● ●● ●● ●● ●●●● ●● ●● ● ● ●● ●● ● ●● ● ● ● ● ● ●● ● ● ● ● ● ● ●●● ●● ●●● ●●●●●● ●● ● ●●● ● ●● ●● ● ● ●● ● ● ● ● ● ●●●● ● ● ● ● ● ●●● ● ●●● ●● ● ● ●● ●● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ●●● ●●● ● ● ● ●● ● ●● ● ● ● ●
● ●● ● ● ●●● ● ● ● ●● ●● ● ● ●● ● ●● ● ● ●● ● ● ●● ● ●● ● ●● ● ● ● ● ● ● ● ● ●●● ● ●● ● ● ● ● ●●●●●● ●● ●● ●● ●●● ● ● ●●● ●● ●● ● ● ● ● ●● ● ●● ● ●● ● ● ● ●● ● ● ● ● ● ● ●●●● ● ●● ● ●● ●● ● ●●● ●● ● ●● ● ●●● ● ●●● ● ●● ● ●● ●●● ●● ●● ● ● ● ●● ● ●●● ● ● ●● ●● ●● ●● ● ●● ●●● ● ● ● ● ● ●● ● ● ● ● ●● ●● ● ● ● ●●● ●● ● ●●● ● ●● ● ●● ● ●● ● ●● ●● ● ● ●● ●● ● ● ●● ●● ● ● ● ● ● ● ● ● ●●● ● ● ● ●● ● ● ●● ● ●● ● ● ● ● ● ● ● ● ● ●● ●● ● ●● ● ● ● ● ● ●●●●● ● ● ● ●● ●●● ●● ● ●● ● ●● ● ●● ● ● ●● ●● ● ●● ● ●● ●● ●●●● ●●● ●● ● ● ●● ●● ● ●● ● ● ●● ●●●● ●● ●● ●● ● ● ● ● ● ●● ● ●●● ● ● ●● ● ● ●●● ●● ● ● ● ● ●●● ●●●● ● ● ●● ●● ● ● ● ● ● ● ●●● ● ●● ●● ● ●● ● ●●● ● ● ●● ● ● ●● ●● ●●●●● ● ●● ●● ● ●● ● ●● ● ● ●● ●● ● ● ●● ●● ● ●● ●● ●● ● ● ●● ●● ● ● ● ● ●●● ●●● ● ●● ● ● ● ● ● ● ● ● ● ● ● ●●●● ● ● ●●● ● ● ● ●● ●● ● ● ● ● ● ●● ●● ● ●● ● ● ●●●● ●● ● ●●●●●● ● ● ●● ● ●●● ● ●● ● ●● ●● ● ● ●● ● ● ●●● ● ●●● ● ●●●● ● ● ● ● ●●●●● ● ●● ● ● ● ● ●● ● ●● ●● ● ● ●●●● ● ● ●● ●● ● ●● ● ●●●● ● ●●● ● ●● ●● ● ●● ● ● ●● ● ●● ● ● ● ● ●● ● ●●● ● ● ● ●●●● ● ● ●● ● ●● ● ● ●● ● ● ● ●● ● ● ●●● ● ● ● ● ● ● ● ●● ●● ●● ● ● ●● ● ● ●● ● ●● ● ●●●● ● ●● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ●●●●● ● ● ● ● ● ● ●● ● ● ●● ● ● ● ● ● ●●●● ● ● ● ●● ● ●● ● ● ● ●●●●● ● ●● ● ● ●● ● ● ●●● ● ●●● ●● ●● ● ● ● ●● ● ● ● ● ● ● ●● ● ● ● ●●●● ● ● ● ●● ●● ●● ●●● ●● ●● ●● ● ● ●● ●● ● ●● ● ● ● ● ●● ● ●●● ● ●●● ● ●● ● ● ●● ● ●● ● ● ● ● ● ●● ● ●● ●● ● ● ● ●● ● ● ●● ● ● ●● ● ● ●● ● ●●● ● ● ● ●● ●● ● ●● ● ● ●●● ● ●● ● ● ●● ●● ● ● ● ●● ● ● ● ● ● ●● ●● ● ●●●● ● ● ● ●●● ● ●● ● ● ●● ●● ● ●● ● ●● ● ● ●● ● ● ●● ● ●● ● ●● ●● ● ● ● ● ●●● ●● ●●●● ● ● ●● ● ●●● ●● ● ●● ● ● ●●● ● ● ●● ●● ●● ●●●● ● ●●● ●● ● ●● ● ●●● ● ● ●●● ● ●● ● ● ● ●● ● ●● ●●● ●● ● ● ●● ● ● ● ●● ●● ●●● ● ● ●● ● ● ● ●● ●●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●●●● ● ●● ●● ●● ● ● ● ● ●●● ● ●● ● ● ●● ● ●● ● ● ● ● ● ●● ● ● ● ●● ● ● ●● ● ●●● ● ●● ● ● ●●● ● ● ●●● ● ● ●● ●● ● ● ● ●●●●● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ●● ● ●● ● ●● ●●● ● ●● ● ●● ●● ● ●● ● ●● ● ●● ● ● ● ● ● ●● ● ● ●●● ● ●● ●● ● ● ● ●● ● ●● ● ● ●●● ● ●● ●● ●● ●●● ● ● ●● ● ● ●●●● ● ● ●● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●●●●● ● ● ● ●● ● ● ● ●● ● ●● ●●● ● ●● ●●●● ● ● ● ● ●● ● ● ● ● ● ●●●● ● ● ● ●●● ● ● ● ●●●●●●●●●● ● ● ●● ● ● ●● ● ●● ●●● ● ● ● ●● ●● ● ●● ●● ● ● ● ● ● ●● ● ● ●●●●●● ● ● ●● ● ●● ● ● ●●● ● ● ●● ● ●● ● ● ●● ● ● ● ●●●● ● ●● ●● ● ● ●● ● ●● ● ● ●● ● ● ●● ●● ●● ● ● ●● ● ● ● ● ●● ● ● ●● ●●● ● ●●● ●●●●● ● ●● ●●● ● ●● ● ●● ● ●● ●●● ● ●●● ●●● ● ● ●● ● ● ●● ● ●●●● ● ● ● ● ● ●● ● ● ● ●● ● ●●● ● ●● ● ●● ● ● ●●●●● ●● ● ●● ● ● ● ●●●●● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ●●● ● ●● ●● ● ● ● ● ● ●● ● ● ● ●● ●● ● ●● ●● ● ● ●●●●● ● ● ● ● ●● ● ●● ● ● ● ●● ● ●●● ●● ●● ● ●● ● ●● ●● ● ●● ● ● ● ● ● ● ●● ● ●● ● ● ● ●● ●● ● ●● ●● ● ● ● ● ●● ●● ● ● ● ● ●●● ●● ● ● ● ●● ● ● ●● ● ● ● ● ● ● ● ●● ● ● ●● ● ●● ●● ● ●● ● ● ●● ●●●● ● ● ● ● ● ●● ● ●● ● ● ●● ● ● ● ● ● ● ● ● ●● ● ●● ●●● ●● ● ● ● ●●●● ●● ●●● ●● ● ●● ●● ●● ●●● ●●● ● ● ● ● ● ●●● ● ●● ●● ●● ●●● ● ● ●● ● ● ●● ● ● ●● ● ● ● ● ●● ●● ● ● ●●●● ●●● ● ● ● ●● ● ●● ●●●●● ● ●●● ●● ●● ● ●●● ●●● ●●● ● ●●● ● ●●● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ● ● ●●● ● ●● ●● ●● ● ●● ●●● ●● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ●● ●● ● ● ● ●●● ●● ● ● ● ●●● ●● ● ● ●●● ●● ● ● ● ●● ● ●●●● ● ● ● ● ● ● ● ●●● ● ● ● ● ●●● ● ● ●● ● ●●● ●● ●● ●●● ● ● ●● ● ●●● ● ● ●● ● ● ●● ● ● ● ●● ● ● ●● ● ● ●● ● ● ● ● ● ●● ● ●●● ● ● ●● ● ●● ●● ●● ● ● ●● ●●●●●● ●● ●● ● ● ● ● ● ●● ● ● ● ●● ●● ●● ● ●●●● ● ● ● ● ● ● ● ●●● ●● ● ●●● ● ● ● ● ●● ● ●● ● ●●●● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ●●●●●●● ●● ● ●●● ● ●●●●●● ● ●● ● ● ●●●●●●● ● ● ●● ●● ● ● ●● ● ●● ● ● ● ●● ● ● ● ● ●● ●●●●● ●● ● ● ●● ● ●●● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ●●● ●●● ● ● ●● ●● ● ● ● ●● ●● ● ● ●●● ●●●● ●● ● ● ●●●● ● ● ● ● ● ●●● ● ● ● ●●● ●● ●●● ● ● ● ● ●● ● ●● ● ● ●● ● ● ● ● ● ● ● ● ●● ●●●● ● ● ● ● ● ●● ● ●●● ● ● ●● ● ●●● ●● ● ● ● ● ● ● ●●● ● ● ●●●●● ●●● ● ●●● ●●●●● ● ● ● ●● ● ●●●● ● ●●● ● ●● ● ●● ●● ● ●● ● ● ●● ● ●●● ●● ●● ●●●●● ●● ● ● ● ● ● ● ●● ● ● ● ●● ●● ●●● ● ●● ● ● ●● ●● ●● ●● ●● ● ●●● ● ● ● ● ● ●● ● ● ●● ●● ●● ●●●● ●● ●● ● ● ● ● ●● ● ● ●●●● ● ●●● ●● ● ● ●●● ● ●● ● ●● ● ●●● ● ● ● ●●●● ● ●●●● ● ● ●● ● ● ● ●●●● ● ●● ● ● ● ● ● ● ● ●●●● ●●● ● ●● ● ●● ●● ●● ●●● ● ● ● ● ●● ● ●●● ● ●● ● ● ● ●● ● ● ●●● ●● ● ●●● ● ●● ● ● ●●● ●●●●● ● ● ● ● ● ● ●● ● ● ● ● ●● ●●● ●●●● ●●●● ● ●● ● ●● ● ● ● ● ● ● ● ●● ●●● ● ● ● ● ● ● ● ●● ●● ● ●● ● ● ●●●●● ●●●● ● ● ● ● ● ●● ● ● ●● ●● ● ●●●●● ● ● ●●●● ●● ● ● ●●●● ● ●● ● ●●● ●● ● ●● ● ●● ● ●● ● ●● ●● ● ●● ●●●●● ● ● ● ● ●● ●● ●●● ●●●● ● ● ● ●● ● ● ● ●● ● ● ● ●● ● ● ●● ●● ●● ●● ● ●●● ●●● ● ●● ●●● ● ●●●● ●● ● ●● ● ● ●●●● ● ● ●● ● ● ● ● ● ● ● ●● ● ● ●●● ●●●● ● ●● ●● ●● ● ●● ● ● ● ● ●● ● ● ●● ● ●● ● ● ● ● ● ● ● ●●●●● ●●● ● ●● ●●●●●● ● ●● ● ● ●● ● ● ● ●● ●● ●● ● ● ●●● ●● ●●● ●●●● ● ● ●● ● ● ● ●● ●● ●●● ● ● ● ●● ●● ● ●● ●●● ●●● ● ●● ● ● ● ● ● ●● ● ● ● ●● ● ● ●● ● ● ● ●●●● ●●● ●● ● ● ● ● ●●● ●●● ● ● ●● ● ●● ● ●● ●●● ● ● ●● ● ● ● ● ● ● ● ●●●●● ● ●● ● ● ● ● ● ●●● ●●● ● ● ● ●● ● ●● ● ●●●● ● ●● ●●●●● ●●●●● ● ● ● ● ● ●● ● ●● ●● ● ●● ●● ● ● ●●●●● ● ●● ● ●●● ● ●●● ● ● ●● ● ● ● ●●● ● ●● ● ●● ● ● ● ● ● ● ● ● ● ●●● ● ● ●●● ● ●● ● ● ● ● ● ●● ●●●●●● ●● ● ● ● ●●● ● ● ●● ●●●●● ● ● ● ● ● ●● ● ● ●● ● ●●●●● ●● ● ● ●●●● ● ● ● ●● ●● ●● ● ● ●● ● ●● ● ● ●● ● ● ● ●● ● ●●●●● ●● ●●● ● ●●● ● ● ● ●●●● ●● ● ●● ● ●● ●● ● ●●●● ●●● ●● ● ● ●●● ● ● ● ● ● ● ●● ● ● ● ● ● ●● ● ● ●● ● ● ● ●● ●● ●●● ●● ●● ● ● ●● ●● ● ● ●● ●● ● ● ● ●● ● ● ● ● ● ● ●●● ●● ●● ● ●● ● ●●● ● ● ● ●● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ●● ●●●● ●● ● ●●●● ● ● ● ●● ● ● ● ● ● ●● ●● ●●● ● ●● ● ● ●● ●● ●● ● ● ● ● ●● ● ● ● ● ● ● ●● ●● ● ●● ● ●● ● ● ● ● ● ● ● ● ●● ●● ● ●●● ● ● ●● ●●● ● ●●● ● ●●●● ●● ●● ● ● ●● ● ● ●● ●● ● ● ● ● ●●●●● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●●●● ● ● ● ● ●●●●● ● ● ●● ● ● ● ● ● ● ● ●●● ● ● ● ●●●● ● ●● ●● ● ● ●●●● ● ●● ● ●● ●●● ● ●●● ●● ● ●●●● ● ● ● ●●● ● ● ●●● ●●●●●●● ●●● ● ● ●●● ●● ● ● ●● ● ● ● ● ●● ●●● ● ● ● ● ● ●● ● ● ●● ● ● ● ●● ● ● ●● ● ●●● ● ● ●●●●● ● ● ●● ● ● ● ● ●● ● ● ●●● ●●●●● ● ●● ●●● ● ●● ●● ● ● ● ● ● ●● ●● ● ● ● ●● ● ● ● ● ●● ●●● ●●●● ●● ●● ● ● ●● ●● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ●● ●●● ●●●●●● ●● ● ●●● ● ●● ●● ● ● ●● ● ● ● ● ● ●●●● ● ● ● ● ● ●●● ● ●●● ●● ● ● ●● ●● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ●●● ●●● ● ● ● ●● ● ●● ● ● ● ●
−3
−2
−1
0
1
2
introduces density based models i.e CEC and GMM. The third section recalls basic information about Kullback-Leibler and Cauchy-Schwarz divergence in terms of different Gaussian models. The fourth section gives basic information about clustering in respect to the above metrics. The last section presents results of the experiments. II. D ENSITY BASED CLUSTERING Let it be recalled that in general EM aims to find p1 , . . . , p k ≥ 0 :
Fig. 2. Clustering of a mouse type dataset with use of a general CEC and Ward and Cauchy-Schwarz divergence approach to densities clustering algorithm.
pi = 1,
(1)
i=1
and f1 , . . . , fk ∈ F, where F is a fixed (usually Gaussian) family of densities such that the convex combination f := p1 f1 + . . . pk fk
3
(d) The final result of the algorithm.
k X
(2)
optimally approximates the scattering of the data under consideration X = {x1 , . . . , xn }. The optimization is taken with respect to a MLE based cost function n
1 X ln p1 f1 (xj ) + . . . + pk fk (xj ) , |X| j=1 (3) where |X| denotes the cardinality of a set X. The optimization in EM is divided into the Expectation step and Maximization step. While the Expectation step is relatively simple, the Maximization usually needs a complicated numerical optimization. The goal of CEC is similar, i.e. it aims at minimizing the cost function (which is a small modification of that given in (3) by substituting the sum with a maximum): MLE(f, X) := −
essary clusters. We present two methods which use classical distances between the two Gaussian distributions. The first one utilizes hierarchical clustering with a simple linking function. An example is presented in Fig. 1. The goal is to detect two separated spirals. Our procedure includes two steps. At first, we represent data in terms of Gaussian densities by applying CEC algorithm (see Fig. 1b). This step can often be omitted because in many cases the data is naturally given by Gaussian densities (see experimental part of the paper). Later, we apply hierarchical clustering on the densities (see Fig. 1c where a dendrogram produced by the algorithm is presented). At the end, we merge points which were associated with densities into clusters, see Fig. 1d. In the second approach, we reduce components by applying Ward’s method to classical k-means algorithm [26], [27]. This method detects clusters without explicitly determining the centers of groups. It is important in the context of clustering densities distribution because it is hard to determine a means of densities [21], [22]. To visualize what was written above let us consider a simple example of a mouse type dataset presented in Fig. 2. Our goal is to detect three main components. If we do not specify the exact number of clusters (for the mouse dataset we can easily determine the correct number of components but normally doing it for a more complicated dataset with a potentially larger number of components it may be problematic) main components will be approximated by several smaller ones (see Fig. 2b). Then, we apply Ward’s method on densities and consequently merge the closest groups, see Fig. 2c. Since classical density based algorithms EM and CEC can use different models, therefore, later we will show that in special cases the clustering of Gaussian densities is reduced to dividing sets with respect to means of distributions. The paper is arranged as follows. The next section briefly
n
1 X ln max(p1 f1 (xj ), . . . , pk fk (xj )) , |X| j=1 (4) where all pi for i = 1, . . . , k satisfy the condition (1). It occurs (see [16]) that the above formula implies that, contrary to EM, it is profitable to reduce some clusters (as each cluster has its cost). Consequently, after minimization of parameters i ∈ {1, . . . , k} the probabilities pi will typically equal zero, which implies that the clusters they potentially represent have disappeared. Thus k, contrary to the case of EM, does not denote the final number of clusters obtained, but is only an upper bound of the number of clusters of interest (from a series of experiments the authors discovered that typically the good initial guess is to set k = 10). Instead of focusing on the density estimation as its first aim, CEC concerns the clustering, where, similarly to EM, the point x is assigned to the cluster i which maximizes the value pi fi (x). However, feeding CEC with the solution, a good estimation of the initial density is obtained by applying the formula (2), which in practical cases is very close (with respect to the MLE cost function given by (3)) to the one constructed by EM. Let it be remarked that the seemingly small difference in the cost function between (3) and (4) has profound further consequences, which result from the fact that the densities CEC(f, X) := −
8
2 1
● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●
−1
4
0
6
J divergence CS divergence B divergence
In this paper, we focus on multidimensional (Rd for d ∈ N) Gaussian densities 1 1 2 kx − mk exp − N (m, Σ)(x) := Σ , 2 (2π)d/2 det(Σ)1/2
where m denotes a mean, Σ is a covariance matrix, and kvk2Σ := v T Σ−1 v is the square of Mahalanobis norm. In such setting a differential entropy is given by N 1 ln (2πe) + ln (det(Σ)) . 2 2 Many distance measures, which use Shannon entropy, have been proposed and studied in the literature [32], but the most widely known is the Kullback-Leibler (K-L) divergence. The distance between two densities f (x) and g(x) can be defined as the K-L divergence between the two pdfs as HS (N (m, Σ)) =
DKL (f, g) = HS× (f ||g) − HS (f ) R where HS× (f ||g) = − f (x) ln(g(x))dx is a cross entropy of densities f and g. In [33] authors used a symmetric version of K-L divergence DJ (f, g) = DKL (f, g) + DKL (g, f ). This distance is important in the context of our investigation because CEC algorithm uses H × (f ||g) as a part of cost function. The formula for the Kullback-Leible divergence is easy to compute in the case of Gaussian distribution DKL (N (m1 , Σ1 ), N (m2 , Σ2 )) = = 21 (m1 − m2 )T Σ−1 2 (m1 − m2 ) det(Σ2 ) + 21 tr(Σ−1 Σ − I) + ln . 1 2 det(Σ1 )
2
3
4
5
6
8
10
−2
−1
0
1
2
Distance between Normal distribution
2
● ●
40
−1
60
0
80
100
120
J divergence CS divergence B divergence
(b) Level sets of Gaussian distribution used in experiments.
1
140
(a) Relation between divergences with fixed covariance matrices.
● ●
−2
20
DISTRIBUTION
In the previous section we presented how to represent data by Gaussian densities. Now, we are going to show how to measure distance between two distributions. The most common measures of distance between probability distributions are related with two concepts of entropy – Shannon and Reny. The first one is a basic definition in information theory, which was conceptualized by [28] to deal with the problem of optimally transmitting messages over noisy channels. Shannon entropy was initially defined for discrete random variables and used in a coding theory [29]– [31]. Later Shannon entropy has been extended to continuous random variables known as the differential entropy. For a continuous random variable with density f : R → R+ differential entropy is defined by Z HS (f ) = −f (x) ln(f (x))dx.
1
0
III. D ISTANCE BETWEEN G AUSSIAN PROBABILITY
0
−2
2
in (4) do not “cooperate” to build the final approximation of f . We apply CEC or EM algorithm to represent data as a collection of Gaussian densities. In the case of CEC algorithm it is easy to apply various models with a lower computational effort. Consequently, in the experimental part we use CEC.
1
2
3
4
5
6
8
10
−2
−1
0
1
2
Distance between Normal distribution
(c) Relation between divergences with fixed means.
(d) Level sets of Gaussian distribution used in experiments.
Fig. 3. Comparison of measures based on two spirals problem
It is evident that Shannon’s entropy occupies a central role in information theory, but there are alternate definitions of entropy. In this paper, we utilize Renyi’s entropy [34] for continuous random variables Z 1 HRα (f ) = ln f α (x)dx . 1−α When α = 2 the above formula becomes Z 2 HR (f ) = − ln f (x)dx , and will be called Quadratic Entropy or simply Renyi’s entropy. We utilize this entropy because it has a nice behavior and possess calculation easiness while combined with Gaussian functions. The equivalent of K-L divergence for Quadratic Entropy is Cauchy-Schwarz divergence Z 2 DCS (f, g)=ln f (x)dx + ln
Z
2
g (x)dx − 2 ln
Z
f (x)g(x)dx .
The last component in the above formula is called CrossInformation Potential (CIP). One of most important properties of Cauchy-Schwarz divergence is DCS (αf, βg) = DCS (f, g). Further, we will use the following well-known [35] formula for the scalar product of two normal densities: Z N (m1 , Σ1 )(x)·N (m2 , Σ2 )(x)dx = N (0, Σ1 +Σ2 )(m1 −m2 ), and consequently Z ln( N (m1 , Σ1 )(x) · N (m2 , Σ2 )(x)dx) =
= ln(N (0, Σ1 + Σ2 )(m1 − m2 )) = 1 1 d = − ln(2π) − ln(det(Σ1 + Σ2 )) − km1 − m2 k2Σ1 +Σ2 . 2 2 2 The formula for the Cauchy-Schwarz divergence is easy to compute in the case of Gaussian distribution 1 = − ln(det(4Σ1 Σ2 ))+ln(det(Σ1 +Σ2 ))+km1 −m2 k2Σ1 +Σ2 . 2 Another popular distance between pdfs is Bhattacharya distance [36] Z p DB (f, g) = − ln( f (x)g(x)dx).
It may be noticed that this is a divergence defined for Renyi’s entropy with α = 2. Similarly as before it is easy to calculate the formula for Gaussian distributions DB (N (m1 , Σ1 ), N (m2 , Σ2 )) = =
1 1 km1 − m2 k21 (Σ1 +Σ2 ) + ln 2 8 2
+ Σ2 )) p det(Σ1 )det(Σ2 )
!
.
At this point, it is valid to ask a question which measure is optimal for our calculations. As it may be seen in Fig. 3a the difference is quite small. In the first experiment we measure the distance between 4 0 1 4 0 −1 , ,N t · , D N , 0 1 1 0 1 −1 for t = 0.1, . . . , 1. In the second, we present the comparison between Gaussian with fixed centers and different covariances matrices −1 4 0 1 4 0 D N ,t · ,N ,t · , −1 0 1 1 0 1 for t = 0.1, . . . , 1. As we see, the difference between measures is minuscule. We also observe that the DJ is the ’biggest’ therefore it is the least sensitive to small changes. TABLE I C OMPARISON OF THE HIERARCHICAL MODEL APPLIED TO THE CEC CLUSTERING (JACCARD INDEX ).
Atom Chainlink EngyTime Hepta Lsun Target Tetra TwoDiamonds WingNut
CEC Classical DCS DJ 1 1 1 1 0.5 0.5001 1 0.8639 1 0.3746 0.4921 0.8593 0.6152 0.3185 1 1 0.504 0.4999
CEC Diagonal DCS DJ 1 0.4997 1 0.4995 0.9182 0.9182 1 0.6525 0.8588 0.4049 0.5012 0.8722 0.8644 0.2992 0.9975 0.9975 0.4999 0.5067
CEC Spherical DCS DJ 1 1 1 1 0.929 0.5001 1 0.8862 1 0.8751 0.5411 0.841 0.9975 0.3163 0.9975 0.5012 0.5001 0.4997
DJ (N (m1 , Σ1 ), N (m2 , Σ2 )) = d
DCS (N (m1 , Σ1 ), N (m2 , Σ2 )) =
det( 12 (Σ1
Observation III.1 (Diagonal model). Let Σ1 and Σ2 be positive define square matrices with elements λ11 , . . . , λ1d and λ22 , . . . , λ2d on the diagonal and let m1 , m2 ∈ Rd then
CEC Spherical with fixed r = 1 DCS DJ 1 1 1 1 0.9267 0.9267 1 0.9609 1 1 0.8253 0.8258 0.8702 0.2958 1 1 0.5028 0.4999
The CEC algorithm allows to use different types of Gaussian densities. Depending on our need we can use spherical, spherical with fixed radiance, diagonal or fixed covariance model. For each model, our formula may be reduced to a simpler and more useful form. We will present all of them beginning with a diagonal model.
1X = 2 i=1
(mi1 − mi2 )2 (λ1i + λ2i ) + (λ1i − λ2i )2 λ1i λ2i
,
DCS (N (m1 , Σ1 ), N (m2 , Σ2 )) = 1 = ln 2
Y
(λ1i + λ2i )2 4λ1i λ2i
+ km1 − m2 k2Σ1 +Σ2 .
Proof. See Appendix A As a simple corollary (by taking the some element on diagonal) we obtain the formula for spherical model. Observation III.2 (Spherical model). Let Σ1 = s1 I and Σ2 = s2 I where s1 , s2 ∈ R and let m1 , m2 ∈ Rd then DJ (N (m1 , Σ1 ), N (m2 , Σ2 )) = =
km1 − m2 k2 (s1 + s2 ) + d(s1 − s2 )2 , 2s1 s2 DCS (N (m1 , Σ1 ), N (m2 , Σ2 )) =
d = ln 2
(s1 + s2 )2 4s1 s2
+
km1 − m2 k2 . s1 + s2
In a similar way by considering some radius we obtain formulas for spherical Gaussian with fixed r. Observation III.3 (Spherical model with fixed r). Let Σ1 = Σ2 = rI where r ∈ R and let m1 , m2 ∈ Rd then DJ (N (m1 , Σ1 ), N (m2 , Σ2 )) = DCS (N (m1 , Σ1 ), N (m2 , Σ2 )) =
1 km1 − m2 k2 , r 1 km1 − m2 k2 . 2r
Distances between density distributions in such cases are reduced to a classic Euclidean norm. A similar effect is obtained for fixed covariances model, where the distance is reduced to Mahalanobis norm. Observation III.4 (Fixed covariance). Let Σ1 = Σ2 = Σ and let m1 , m2 ∈ Rd then DJ (N (m1 , Σ1 ), N (m2 , Σ2 )) = km1 − m2 k2Σ , DCS (N (m1 , Σ1 ), N (m2 , Σ2 )) = km1 − m2 k22Σ . Concluding, while clustering density distributions in spatial cases, we reduce the problem to the standard clustering with respect to means of Gaussians (with possible difference metrics).
IV. C LUSTERING ALGORITHMS In this section, we present a possible approach to clustering density distribution. By G(Rn ) we denote the set of all n dimensional Gaussian densities. For clustering probability distributions, respectively to Kullback-Leibler or CauchySchwarz divergence, it is hard to determine centroids of clusters. Therefore, we use a clustering method which uses distance between only two densities. In a natural way, we can use generalization of Ward’s approach [26] to classical k-means algorithm and hierarchical method [37]. 8
● ● ● ●
● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ●● ● ● ● ● ●● ●●● ● ● ● ● ● ●● ● ● ● ● ● ●● ● ●●● ● ● ● ●● ●● ● ●● ●● ● ●● ● ● ●●●● ●● ●● ●● ● ● ●● ●●● ●● ●● ● ● ● ●●● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ●●●● ● ● ● ● ●● ● ● ● ● ● ● ●●● ● ● ●● ●● ●●●●●●● ● ● ●● ● ● ● ●●● ● ● ●●● ●●● ● ●●● ● ● ● ●● ● ● ● ●● ● ●●●●● ● ●●●●●● ●●●● ●● ● ● ● ● ●●● ●● ● ● ● ●● ● ● ●● ● ●● ●●● ● ●●●● ● ●● ● ● ●● ● ●● ●●●●●● ● ● ●●●● ● ● ●● ●● ●● ● ● ●● ● ●● ● ●●● ●● ● ● ● ● ● ●●● ●● ●● ● ●● ●●● ● ● ●●● ●● ● ● ●●●●● ●●● ● ● ● ●● ●●● ●● ●● ●● ●● ● ●● ● ● ● ● ● ● ●● ● ●●●● ● ●●●● ●● ● ● ● ●● ●● ● ●●● ●●●● ●● ● ●● ●● ● ● ● ●● ● ● ●● ● ● ● ● ●● ●● ● ● ●● ● ●● ● ● ●● ● ● ●● ●● ●● ●●● ●● ● ●●●● ●●●●● ● ● ●●● ● ● ●● ● ● ●●● ●● ● ●●● ●● ● ● ●● ●●● ● ● ● ●●● ●● ● ● ● ● ● ● ●●● ●● ● ●● ● ●● ● ● ●● ● ● ● ●●● ●● ● ● ●● ●●●● ●● ●● ● ●● ● ● ●● ● ● ● ● ● ● ● ●● ● ●●● ● ●● ●● ● ● ●●● ● ● ● ● ● ● ●● ● ●● ●● ●●● ● ● ● ●● ● ● ● ● ● ● ● ● ●● ● ● ●● ● ● ●●● ● ● ●●●● ● ● ●●● ●●● ● ● ● ● ● ● ● ● ● ●● ●● ● ● ● ● ●● ● ● ● ● ●● ● ● ●●● ● ●● ● ●●● ● ●● ● ●● ●● ●● ● ● ● ● ● ●● ● ● ● ● ● ● ●● ●● ●● ● ●●●● ● ● ● ● ● ● ● ●●●● ●● ● ● ● ● ● ●● ●● ● ●● ● ● ● ● ● ●● ● ● ● ● ● ● ● ●● ●● ● ●● ●● ●● ● ● ● ● ● ●● ●●● ● ● ● ● ● ● ●● ● ● ● ●● ● ● ●● ● ● ●● ●● ●●●● ● ● ● ● ● ●●●● ●●● ● ● ●● ● ●●● ● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ● ● ●● ●●● ● ●● ● ● ● ● ●●● ● ●● ●● ● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ● ●● ● ●●● ● ●● ● ● ● ● ● ● ●● ● ●●● ●● ● ●● ●●● ● ●●● ● ● ● ● ● ● ● ●● ●● ● ● ● ●●● ● ●● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ●● ● ● ●● ●● ● ● ● ● ●● ● ● ●● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ●● ●● ● ● ● ● ●● ● ● ● ● ● ● ●●● ● ● ●● ● ● ● ●●● ● ●● ● ● ● ● ● ● ●●● ● ●● ● ● ● ● ● ●● ● ● ●● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ● ●● ●● ● ●● ●● ● ● ● ● ● ● ●● ● ●● ●● ● ●● ●● ● ● ● ● ● ● ●●●● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ●●● ● ● ● ●● ●● ●●● ●●● ●●●●●●● ●● ● ● ● ●● ●● ●● ●●● ● ● ●●● ● ● ● ●● ● ● ● ●●● ● ● ● ●● ● ● ●● ● ● ●●●● ● ● ● ● ●● ●● ● ●● ●●● ● ●● ●● ●● ● ●● ● ●● ● ● ●●●● ● ● ● ●● ●● ●● ● ● ● ● ●● ● ● ●●● ● ●●● ● ● ● ● ● ● ● ●● ●● ● ● ● ● ● ● ● ● ●●● ● ●●● ● ● ●● ● ● ● ● ●● ●●●● ●● ● ● ● ●● ●● ●● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ●●● ●●● ● ● ●●●● ● ● ● ●●● ● ● ●● ●● ● ● ● ● ● ● ● ●● ● ● ●● ●●● ● ● ● ● ●● ●●● ●● ●● ● ●● ●● ●● ● ● ● ● ● ● ● ●● ●● ● ● ●●● ● ●● ● ● ● ● ●● ●● ● ● ● ● ● ●● ● ●● ●●● ● ● ● ● ● ● ●● ●● ●● ●●● ● ●● ● ● ● ●● ● ● ●● ● ● ●● ●●● ●●● ● ● ● ● ●●●●●● ● ●● ●● ●●●● ●● ●●●●●● ●● ● ● ● ●● ●● ● ●● ● ● ●●● ●● ● ● ● ● ●● ● ●● ● ●● ● ●●●● ●● ● ● ● ● ● ●● ●●●●● ● ● ● ● ● ●● ●●● ●● ●● ●● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ●● ● ● ● ● ●● ●● ● ●● ● ● ● ●● ● ● ● ● ● ●●●●● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ●●● ● ● ● ●● ●● ● ●● ●● ● ● ●● ● ● ●●●● ● ●● ●● ● ●● ● ●● ●●●● ●● ● ● ● ● ● ●●● ●● ●● ●●● ● ● ●● ● ●● ● ●● ● ● ● ●●●●● ●● ●● ● ● ●● ●●● ● ● ● ● ● ● ●● ● ●● ●●● ●● ● ●● ● ● ● ●●● ● ● ●● ●● ●● ● ●● ●● ● ●● ● ● ●●●● ● ●●● ● ● ●●● ● ● ● ● ● ●● ● ● ● ●● ● ●● ●● ● ●● ●●● ● ●●●●●● ● ● ●● ● ● ●● ● ●● ●● ●●●● ●●● ● ●●●●●● ● ●●●● ●● ●● ● ● ● ● ●●● ●●● ● ●● ●● ● ● ● ●● ● ● ●● ● ●● ●● ● ● ● ● ● ● ● ●● ● ●●● ● ● ● ● ●●●● ●●●●●●● ●● ● ● ●●● ● ●● ● ●● ●● ●● ● ●● ●● ●● ● ●●● ●● ●● ● ●● ●● ●● ● ● ●● ● ● ●● ●●● ● ● ● ●● ●●● ● ● ● ●● ● ●● ● ●● ●● ● ● ●●●● ●●● ●● ● ● ● ●● ● ● ● ● ● ●● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ●●●●● ● ● ● ● ●● ● ● ● ● ●●● ● ● ● ●●● ● ● ● ● ● ●● ●● ●● ● ●● ●● ●● ● ● ● ● ● ●● ● ● ● ● ● ● ● ●● ●● ● ●● ●●● ● ● ●● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ●● ● ●● ●● ● ● ● ●● ● ● ● ● ●●●●●●● ● ● ●●● ● ● ●●● ●● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ●●●● ● ● ● ●● ● ● ●● ● ●● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ●●●● ● ● ● ● ● ● ●●●● ●● ●●●●● ● ●● ● ● ● ●●● ● ●● ● ● ●● ● ●● ●● ● ●●●● ●● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●●● ●● ●● ● ● ● ● ● ● ●● ● ● ●● ●● ●● ● ● ● ● ●● ● ● ●● ● ●● ● ●● ● ● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ●● ● ●● ●●● ●●● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ●● ●● ●● ● ●●● ● ●● ● ● ● ● ● ●●● ● ● ● ●●● ● ● ●● ●● ●● ● ● ● ●● ●● ● ● ●● ● ●● ● ● ●● ● ●● ● ● ●●● ● ● ● ● ● ● ●●● ● ●● ●● ● ● ● ● ●● ● ●● ● ● ● ●●● ● ● ● ● ● ●● ● ● ●●● ● ● ● ●● ● ● ● ● ● ●● ● ● ● ●● ●● ●● ●● ● ●● ● ● ● ● ●●● ● ● ● ● ● ● ●● ● ●●●● ● ● ● ● ● ● ● ● ●● ● ●●●● ● ● ● ● ● ●● ● ● ● ● ● ●● ● ●●● ● ● ● ●● ●● ● ● ● ● ● ● ● ● ●● ● ● ●● ●● ●● ● ●●● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●●●● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ●● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ●● ● ● ● ●●●● ● ● ● ●● ● ● ● ●●● ●● ●●●● ●● ● ● ● ● ● ●●●● ● ● ●●● ● ● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ● ● ●● ● ●● ● ● ●●● ● ●● ● ●● ● ●● ●● ● ● ● ●● ● ●●● ● ●●● ●● ● ● ● ● ● ● ●● ● ●●● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ●● ● ● ● ●● ● ●●● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ●●● ●● ● ● ● ● ●● ●●● ●●● ●● ● ●● ● ● ● ●● ● ●●● ● ● ●● ●● ● ● ● ● ●●● ● ● ● ● ●● ● ● ● ● ● ●● ●●● ● ● ● ● ● ● ● ● ●● ● ●● ● ●● ● ●●●● ● ●●● ● ● ● ● ● ●● ● ● ●● ● ● ● ●●● ● ●●●● ● ● ● ● ● ● ● ● ●● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ●● ● ● ●● ● ●● ● ● ● ● ● ● ●●●● ● ●● ● ● ● ●● ● ● ● ● ● ● ●● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ●●● ● ● ● ● ● ●● ● ● ● ● ● ● ● ●●●● ●● ● ● ●● ● ●● ● ● ●● ● ● ●● ●● ● ● ●● ● ● ● ● ●● ●● ●● ● ● ● ● ● ● ● ●● ●●● ● ● ● ●● ● ● ● ● ● ● ●● ●● ● ● ● ●●●●● ● ●● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ●● ●● ●● ● ●● ● ● ● ● ●● ● ●●●● ● ● ●●● ● ●● ● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ●● ●● ● ●●● ● ● ●● ●●● ●● ● ● ● ●● ● ● ● ● ● ●● ●● ● ● ● ● ●●●● ● ●● ●● ● ● ●● ●● ● ●● ● ●●●● ●● ● ● ●● ● ●● ● ● ● ●● ● ●● ● ● ●●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ●●● ●●● ●● ● ● ● ●● ● ● ●● ●● ●● ● ● ●● ●●● ● ● ●● ● ● ●● ● ● ● ● ● ●● ● ● ●● ●●●● ● ● ●●● ● ●●● ● ● ● ● ●● ● ● ●● ● ● ● ●● ● ● ● ● ● ●● ● ● ●● ● ● ● ● ● ● ● ●●●● ● ●●● ●●●●●● ● ● ● ● ●● ●● ●● ● ●● ● ● ● ● ● ●●● ●●●●● ● ●● ● ● ● ●● ●●●● ●● ● ● ● ●● ● ● ● ● ● ● ●● ●● ● ●●● ● ● ● ● ● ●●● ● ●● ● ●● ● ● ●●●● ● ● ● ● ● ●● ●● ●● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ●● ●● ● ●● ● ● ● ●● ● ● ● ● ● ●● ● ● ● ● ● ●
●
●
6
x∈X
Consequently, we can apply Lloyd version of k-means algorithm. All the above calculations work in Euclidean space [26]. In our paper, we applied Ward approach to the case of metrics between densities by defining cost function E(G1 , . . . , Gk ) :=
●
●
It allows us to calculate the distance between point (in our case a Gaussian density distribution) and cluster without knowledge about its center (see [26][eq. (16)]) 1 1 X 2 d (x, c) − d2 (c; X) = ssX . X |X|
● ●●
V3
−2
0
2
4
●
k X X
D2 (g, f ),
i=1 g,f ∈Gi
where G1 , . . . , Gk ⊂ G(Rn ) and D is one of Kullback-Leibler, Cauchy-Schwarz or Bhattacharya divergence.
●
● ● ●
●
−4
−2
0
2
4
6
8
● ● ●
−3
1
● ● ● ●● ●● ●● ●● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ●● ●● ● ● ● ● ●● ●● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ●● ● ● ● ●●● ● ● ● ● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ●● ● ● ● ● ● ● ●● ● ● ●●● ● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ● ●● ● ● ● ● ● ●●● ● ● ● ●● ● ● ● ●● ● ● ●● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ●● ●● ● ●● ● ●● ● ● ●●● ● ● ●●●● ● ●● ● ● ● ● ● ●● ● ● ●● ●● ● ● ● ● ● ● ● ● ●● ● ● ● ●● ● ●● ● ● ● ●● ● ●●● ●●● ● ●● ● ● ● ● ● ●●●● ● ●● ● ● ● ● ●● ● ●● ● ● ● ●● ● ●● ● ● ●● ● ● ●● ● ● ● ● ●● ●● ●● ●● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ●● ● ●● ● ●● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ● ●● ● ● ● ●● ● ● ●● ● ● ●● ● ● ● ● ● ● ● ● ●●● ●●● ● ●●● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ●● ● ●● ● ● ●● ● ● ● ● ● ● ● ● ● ●● ● ● ●● ● ● ● ● ● ●●● ● ●● ● ● ● ● ● ● ●● ●● ●● ●● ● ● ● ● ● ● ● ● ● ● ●● ● ●● ● ●● ● ● ● ● ●● ●● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ● ● ●● ● ●● ● ●● ●● ●● ● ● ●● ● ● ● ● ● ● ● ● ●● ● ● ●●●● ●●●● ● ● ● ● ●● ●● ●●● ●● ● ● ● ● ● ● ● ●●● ●● ● ●● ● ● ● ● ● ●● ●● ● ● ●●● ● ● ● ●● ● ● ● ● ● ●● ● ● ●● ● ●● ●● ● ● ● ● ● ● ● ●● ● ● ● ● ● ●●● ● ● ●●● ● ● ● ● ● ●● ●● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ●● ●●● ●● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ●● ● ●● ● ●● ●● ● ● ● ● ● ● ●●● ●● ● ● ● ● ● ●● ●● ● ● ●● ● ● ●● ●● ● ● ●● ● ●● ●● ●● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ●
V3
●
●
−1
1 0
V3
−1 −2 −3
● ● ●
● ●● ● ●● ● ● ● ●●● ● ● ● ● ●● ● ● ● ● ●●● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ● ●● ● ● ● ●● ● ● ●● ● ●● ●● ● ● ● ●● ● ●● ● ● ● ● ● ●● ●● ●● ● ● ● ●● ● ●● ● ● ● ● ● ● ●● ● ● ●● ●● ● ● ●● ● ● ● ● ● ● ● ● ●● ●● ● ● ● ● ●● ●● ● ● ● ● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ●●●● ● ●● ● ● ● ●● ●● ●● ● ●● ● ●● ●● ● ●● ● ●● ● ● ● ● ● ● ●● ●● ● ●● ●●● ●● ●● ● ● ●●● ●● ● ● ● ● ●● ● ● ● ● ●● ●●● ● ●●● ● ●●● ● ● ● ● ●● ●● ●● ● ●●●●●●●● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ●● ●● ● ● ● ●● ●●● ●● ● ●● ● ● ●●●● ● ●● ● ●●●●●● ● ● ● ● ●● ● ● ● ● ● ●●● ●● ●● ●● ●●●●●●●●●● ●● ● ●●● ● ●● ●●●●●●●●● ● ● ● ● ● ● ● ●●● ●● ●●● ● ●● ●● ●● ●● ● ●● ● ● ● ●●● ● ●●● ● ●●● ●●●● ● ● ● ● ● ● ● ● ● ●● ● ● ●● ●● ● ● ● ● ●● ● ● ● ● ●●●● ● ● ● ● ●● ● ● ●● ● ● ● ●● ● ● ● ●● ●● ● ● ● ● ●● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ●●● ● ● ●●● ● ●● ● ● ● ● ● ●●●● ●●●●● ●● ● ● ●● ● ●● ●● ● ●● ● ● ●● ● ● ● ● ● ●● ● ●● ● ●●● ● ● ● ● ● ● ● ● ● ●● ● ● ●● ● ● ● ● ● ● ● ● ● ●● ● ● ●● ● ● ● ● ● ●● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ●● ●● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ●● ●● ●●● ● ●● ● ● ● ●● ● ● ● ●● ● ●●● ● ● ● ● ● ● ● ● ●● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ●●● ● ●
0
● ● ●
2
3
V2
● ●●● ●
●
● ● ●
−2
−1
0
1
2
3
0
V2
1
2
3
4
V2
(a) The original electron-microscopy images.
(b) Mask which allowed deleted background.
(c) The curved funded by the algorithm.
(d) The original electron-microscopy images.
(e) Mask which allowed deleted background.
(f) The curved funded by the algorithm.
Fig. 4. The result of finding curve type structure by hierarchical clustering of density representation of data.
A. Ward’s clustering Let us start with a classic k-mens algorithm in Euclidean space. For P a single group X ⊂ Rn we denote mean P by 1 1 mX = |X| x and covariance matrix by ΣX = |X| (x− x∈X
x∈X
mX )(x−mX )T . The sum of squares, in such a setting, is given by X kx − mX k2 . ssX = x∈X
The k-means clustering aims to divide a set X into k ∈ N disjoint groups X1 ∪ . . . ∪ Xk so as to minimize the withincluster sum of squares (WCSS). In other words, we minimize a cost function k X k X X 2 kx − mXi k . ssXi = E(X1 , . . . , Xk ) = i=1
i=1 x∈Xi
Since k-means algorithm works only on vectors, we use generalization of the classic approach [38] which minimizes cost function which depends only on the distance between vectors k X X d2 (x, y), E(X1 , . . . , Xk ) = i=1 x,y∈Xi
where d is a metric in Euclidean space. By using Ward’s approach, (see [26][eq. (13)]) we can show that 1 X 2 d (x, y). ss(X) = 2|C| x,y∈C
Fig. 5. Determining curved type structures on electron-microscopy images.
B. Hierarchical clustering Hierarchical clustering is a technique which builds, as the name implies, a hierarchy of clusters. Within the technique we distinct two approaches; bottom-up which iteratively merges separate clusters into one cluster and top-down which iteratively breaks one cluster into separate clusters. Generally, the complexity of the first approach is bounded by O(n3 ) while for the second approach we have O(2n ). We use the bottomup approach because it is more efficient so now we will briefly recall how it works. Given a set of n ∈ N items (in our case Gaussian density distributions) to be clustered the hierarchical clustering starts by assigning each item to a cluster. Therefore, the first step produces n clusters, each containing just one item. During the second step we find the closest pair of clusters and merge them into a single cluster, so we decrease the number of clusters by one. In the third step, we compute distances between a new cluster and each of the old clusters. We iterate the second and third step until we end up with one cluster which contains n elements. The third step can be done in different ways; given
two clusters C1 and C2 , the most common linkage clusterings are • single-linkage min{d(x, y) : x ∈ C1 , y ∈ C2 } • complete-linkage max{d(x, y) : x ∈ C1 , y ∈ C2 } P P 1 • average-linkage |C |·|C | x∈C1 y∈C2 d(x, y). 1 2 The product of the hierarchical clustering is a binary tree and may be visualized on a dendrogram. C. Comparison of hierarchical clustering and Ward’s method The basic difference between above methods is connected with their applications. In the case of Ward’s clustering we are looking for groups represented by centers (in our case by Gaussian distribution). As it was mentioned this procedure does not compute centers of groups explicitly. Nevertheless, we can apply this approach to improve clustering in the case when non-Gaussian component is approximate by several small Gaussian ones. Moreover, it is frequent in clustering problems that the data is naturally given by Gaussian distributions and we are looking for clustering respectively to the same centroid (for more information see experiment section). In the case of hierarchical clustering, we concentrate on clusters which lay on lower dimensional manifolds. The method can be understood as a piecewise linear approximation of the curves and merging elements belonging to different manifolds. By using single-linkage function, we can find curves (or manifolds in higher dimensional spaces). In a natural way, when the first step of our method is corrupted the second step cannot effectively discover real complexities of data. More precisely, if there exist a cluster, created in the first step of our algorithm, which contains a point belonging to different manifolds then the second step our algorithm will merge them into a single cluster. V. A PPLICATIONS In this section, we introduce the application of our method. In the first subsection, we verify hierarchical clustering in the context of finding clusters with complicated shapes. We start from synthetic dataset from [39] which are rather simple and low-dimensional. Next we, present a possible application in the context of finding curve-type structures which consist of disconnected components. Such structures are found in electron-microscopy images of metal nano-grains, which are widely used in the field of nano-sensor technology [40]. In the second subsection, we will verify the Ward method in the context of Gaussian densities clustering. The first example shows how we reduce Gaussian component in the context of finding objects on images from [41]. The second example illustrates the algorithm when data is naturally given by densities. In the second example, we group states in the USA in respect to the distribution of people who live in each of them. A. Curve type structures In the first experiment, we present how our approach works on synthetic data from [39] which is presented in Fig. 4. In the experiment the first step is done to represent data by applying
CEC algorithm. Then, as the second step, we merge clusters by applying a hierarchical model. The comparison of different models in CEC and various distances between distributions is presented in Tab.I. The method is verified by the use of Jaccard index. As we observe, our approach works well for a synthetic data, where we are able to detect curve type structures with high precision. Now, we examine the method on real data, to be more precise on electron-microscopy images [40]. All of the elements on images can be interpreted as Gaussian densities, see Fig. 5. Our goal is to find a curve which connects two different borders of the image. At the beginning, we prepare the data by removing background, see Fig. 5b and 5e by applying standard thresholding procedure [40]. In Fig. 5c and 5f a black color marks elements which were classified as a part of the largest component obtained by hierarchical procedure. If the data contains curve type strictures we are almost always able to detect them. The main problem is connected with the first step which has a great influence on the final results. Nevertheless, in the case of electron-microscopy images we omit this problem, since we have natural representation of the data as a Gaussian distribution. B. Ward method In this subsection, we present how Ward’s method works in practice. We start from the problem of extracting objects in images. We use data repository [41] which contains images where we can specify a background and a foreground. It is difficult to determine these two classes directly. Therefore, we divided images into a large number of clusters (in our experiment we use CEC with initial number of clusters set to 20) and then merged them by applying Ward’s k-means algorithm on Gaussian representation of data (we used CauchySchwarz divergence), see Fig. 6 and Fig. 7. As we see, this approach works effectively by merging densities related with background in one large component and thus connected with object in the foreground into a second smaller one. In the second example, we consider a data set which is naturally given as Gaussian densities. Our goal is to cluster states of the USA respectively to the distribution of population. We use a Zip Code dataset for extracting Gaussian representation of the states see Fig. 8a. The result of clustering is presented in Fig. 8b. We obtain splitting of data into two groups. One group gathers large states with a low level of population while the other group aggregates smaller ones with lots of people. VI. C ONCLUSIONS In this paper, the method for clustering curved data, which uses distance between density distributions, was presented. Our algorithm consists of two basic steps. In the first one, we represent data by a mixture of Gaussian distribution. In the second one, we merge components by using distance between densities. This approach allows to determine components of various sizes and geometry. Moreover, it is frequent in clustering problems that the data is naturally given by Gaussian distributions.
merging densities. Hierarchical clustering allows us to discover groups of point which lay on curves (or manifolds in higher dimensional spaces). On the other hand, Ward’s method improves clustering by connecting groups, which appears by dividing single point cloud into separated clusters (mainly by initializing clustering algorithm with a wrong number of cluster). Practical experiments show, that this approach gives promising results. Moreover, in the context of detection curve-type structures and image segmentation, we obtain a solution to real problems.
(a) General CEC algorithm.
(b) Spherical CEC algorithm.
VII. ACKNOWLEDGMENTS The work of P. Spurek was supported by the National Centre of Science (Poland) [grant no. 2013/09/N/ST6/01178]. VIII. A PPENDIX (c) Fixed radiance spherical CEC algorithm. Fig. 6. The result of Ward k-means algorithm (right images) on density representation of data obtained by CEC with different models.
(a) General CEC algorithm.
In this section, we present the proof for Observation III.1. Let us consider DJ . By simple calculation we have DJ (N (m1 , Σ1 ), N (m2 , Σ2 )) = d 1 d 2 P P λi λi 1 1 2 + 21 = −1 + 2 2 − 1 1 − 1 2 km1 − m2 kΣ−1 λ λ +Σ i i 1 2 i=1 i=1 d 1 P (λi −λ2i )2 1 1 2 = −1 + 2 1 λ2 2 km1 − m2 kΣ−1 λ i i 1 +Σ2 i=1 d P (mi1 −mi2 )2 (λ1i +λ2i )+(λ1i −λ2i )2 1 . 2 λ1 λ2 i
i=1
i
Let us consider DCS . We have
DCS (N (m1 , Σ1 ), N (m2 , Σ2 )) = − 12 ln(det(4Σ1 Σ2 ))+ ln(det(Σ1 + Σ2 )) + km1 − m2 k2Σ1 +Σ2 = − 12 ln(4d · λ11 · . . . · λ1d · λ21 · . . . · λ2d )+
(b) Spherical CEC algorithm.
ln((λ11 + λ21 ) · . . . · (λ1d + λ2d )) + km1 − m2 k2Σ1 +Σ2 Q 1 2 2 (λi +λi ) 1 + km1 − m2 k2Σ1 +Σ2 . 2 ln 4λ1 λ2 i
(c) Fixed radiance spherical CEC algorithm. Fig. 7. The result of Ward k-means algorithm (right images) on density representation of data obtained by CEC with different models.
(a)
(b)
Fig. 8. The result of clustering states of USA. On image 8a we present a coordinates of zip codes in USA and Gaussian distribution obtained represent different states. On image 8b we present the result of clustering states with k = 2.
In this paper, we introduce two possible solutions for
i
R EFERENCES [1] J. Hartigan, Clustering algorithms. John Willey and Sons, 1975. [2] A. Jain and R. Dubes, Algorithms for clustering data. Prentice-Hall, Inc., 1988. [3] A. Jain, M. Murty, and P. Flynn, “Data clustering: A Review,” ACM Computing Surveys, vol. 31, no. 3, pp. 264–323, 1999. [4] A. Jain, “Data clustering: 50 years beyond K-means,” Pattern Recognition Letters, vol. 31, no. 8, pp. 651–666, 2010. [5] R. Xu and D. Wunsch, Clustering. Wiley-IEEE Press, 2009. [6] G. McLachlan and T. Krishnan, The EM algorithm and extensions. John Wiley & Sons, 2007, vol. 382. [7] G. McLachlan and D. Peel, Finite mixture models. John Wiley & Sons, 2004. [8] S. Kumar and M. Hebert, “Man-made structure detection in natural images using a causal multiscale random field,” in Computer Vision and Pattern Recognition, 2003. Proceedings. 2003 IEEE Computer Society Conference on, vol. 1. IEEE, 2003, pp. I–119. [9] J. Samuelsson, “Waveform quantization of speech using gaussian mixture models,” in Acoustics, Speech, and Signal Processing, 2004. Proceedings.(ICASSP’04). IEEE International Conference on, vol. 1. IEEE, 2004, pp. I–165. [10] S. J. McKenna, Y. Raja, and S. Gong, “Tracking colour objects using adaptive mixture models,” Image and vision computing, vol. 17, no. 3, pp. 225–231, 1999. [11] Z. Xiong, Y. Chen, R. Wang, and T. S. Huang, “Improved information maximization based face and facial feature detection from real-time video and application in a multi-modal person identification system,” in Proceedings of the 4th IEEE International Conference on Multimodal Interfaces. IEEE Computer Society, 2002, p. 511. [12] B. Moghaddam and A. Pentland, “Probabilistic visual learning for object representation,” Pattern Analysis and Machine Intelligence, IEEE Transactions on, vol. 19, no. 7, pp. 696–710, 1997. [13] M. H. Law, M. A. Figueiredo, and A. K. Jain, “Simultaneous feature selection and clustering using mixture models,” Pattern Analysis and Machine Intelligence, IEEE Transactions on, vol. 26, no. 9, pp. 1154– 1166, 2004. [14] R. J. Povinelli, M. T. Johnson, A. C. Lindgren, and J. Ye, “Time series classification using gaussian mixture models of reconstructed phase spaces,” Knowledge and Data Engineering, IEEE Transactions on, vol. 16, no. 6, pp. 779–783, 2004. [15] E. Hayman and J.-O. Eklundh, “Statistical background subtraction for a mobile observer,” in Computer Vision, 2003. Proceedings. Ninth IEEE International Conference on. IEEE, 2003, pp. 67–74. [16] J. Tabor and P. Spurek, “Cross-entropy clustering,” Pattern Recognition, vol. 47, no. 9, pp. 3046–3059, 2014. [17] I. Jolliffe, “Principal component analysis,” Encyclopedia of Statistics in Behavioral Science, 2002. [18] C. Fraley and A. E. Raftery, “How many clusters? which clustering method? answers via model-based cluster analysis,” The computer journal, vol. 41, no. 8, pp. 578–588, 1998. [19] K. Popat and R. W. Picard, “Cluster-based probability model and its application to image and texture processing,” IEEE Transactions on Image Processing, vol. 6, no. 2, pp. 268–284, 1997. [20] K. Zhang and J. T. Kwok, “Simplifying mixture models through function approximation,” in Advances in Neural Information Processing Systems, 2006, pp. 1577–1584. [21] J. Dhillon, “Differential entropic clustering of multivariate gaussians,” Advances in Neural Information Processing Systems, vol. 19, p. 337, 2007. [22] F. Nielsen and R. Nock, “Sided and symmetrized bregman centroids,” Information Theory, IEEE Transactions on, vol. 55, no. 6, pp. 2882– 2904, 2009. [23] K. Zhang and J. T. Kwok, “Simplifying mixture models through function approximation,” Neural Networks, IEEE Transactions on, vol. 21, no. 4, pp. 644–658, 2010. [24] J. Goldberger, H. Greenspan, and J. Dreyfuss, “Simplifying mixture models using the unscented transform,” Pattern Analysis and Machine Intelligence, IEEE Transactions on, vol. 30, no. 8, pp. 1496–1502, 2008. [25] F. Nielsen, V. Garcia, and R. Nock, “Simplifying gaussian mixture models via entropic quantization,” in 17th European Conference on Signal Processing (EUSIPCO), 2009. [26] V. Batagelj, “Generalized ward and related clustering problems,” Classification and related methods of data analysis, pp. 67–74, 1988.
[27] M. Smieja and J. Tabor, “Spherical wards clustering and generalized voronoi diagrams,” in Data Science and Advanced Analytics (DSAA), 2015. 36678 2015. IEEE International Conference on. IEEE, 2015, pp. 1–10. [28] C. E. Shannon and W. Weaver, “The mathematical theory of communication (urbana, il,” 1949. [29] T. M. Cover and J. A. Thomas, Elements of information theory. John Wiley & Sons, 2012. ´ [30] M. Smieja and J. Tabor, “Entropy approximation in lossy source coding problem,” Entropy, vol. 17, no. 5, pp. 3400–3418, 2015. [31] P. Spurek and J. Tabor, “The memory center,” Information Sciences, vol. 252, pp. 132–143, 2013. [32] J. Lin, “Divergence measures based on the shannon entropy,” Information Theory, IEEE Transactions on, vol. 37, no. 1, pp. 145–151, 1991. [33] H. Jeffreys, “An invariant form for the prior probability in estimation problems,” Proceedings of the Royal Society of London. Series A. Mathematical and Physical Sciences, vol. 186, no. 1007, pp. 453–461, 1946. [34] J. C. Principe, D. Xu, and J. Fisher, “Information theoretic learning,” Unsupervised adaptive filtering, vol. 1, pp. 265–319, 2000. [35] N. H. Timm, Applied multivariate analysis. Springer, 2002, vol. 26. [36] A. Bhattacharyya, “On a measure of divergence between two multinomial populations,” Sankhy¯a: The Indian Journal of Statistics, pp. 401– 406, 1946. [37] S. C. Johnson, “Hierarchical clustering schemes,” Psychometrika, vol. 32, no. 3, pp. 241–254, 1967. [38] P. Indyk, “A sublinear time approximation scheme for clustering in metric spaces,” in 2013 IEEE 54th Annual Symposium on Foundations of Computer Science. IEEE Computer Society, 1999, pp. 154–154. [39] A. Ultsch, “{Clustering with SOM: U* C},” 2005. [40] P. Spurek, A. Chaikouskaya, J. Tabor, and E. Zajac, “A local gaussian filter and adaptive morphology as tools for completing partially discontinuous curves,” in Computer Information Systems and Industrial Management. Springer, 2014, pp. 559–570. [41] D. Martin, C. Fowlkes, D. Tal, and J. Malik, “A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics,” in Proc. 8th Int’l Conf. Computer Vision, vol. 2, July 2001, pp. 416–423.
