He has directed 42 Ph.D. degree dissertations and 57 masters degree ... puter science, the Laboratory of Computer and Information Science, Helsinki University ...
IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 15, NO. 4, JULY 2004
789
Guest Editorial Special Issue on Information Theoretic Learning
N
ONLINEAR signal processing using neural networks inherited the dominant mean-squared error criteria from linear adaptive filtering theory due to a number of appealing properties exhibited by second-order statistical optimality criteria under the linear model. In linear filtering of Gaussian distributed signals, second order statistics such as variance and correlation completely describe the data distribution and their interactions. In the nonlinear model framework, where signals are not normally Gaussian distributed, this is not the case anymore. As the goal of adaptive system’s training is to transfer onto the parameters of the nonlinear model as much information as possible about the signals’ underlying statistical and dynamical structure, one certainly needs to consider the higher order statistics of the data. Descriptors used to quantify information, such as entropy and divergence (or its special case, mutual information) appear as leading candidates to build this new information theoretic criteria. The process of designing optimal signal processing filters based on information theoretic criteria is referred to as information theoretic filtering (ITF) and the process of learning or adaptation with the new cost function is named information theoretic learning (ITL). Information theory has a long history in the design of optimal communication systems and coding. Numerous useful theoretical results relating ITL to classical signal processing methods have also been established. For example, information theory has been crucial in the understanding of key results in signal processing and learning theories ranging from maximum entropy spectral estimators to minimum description length model order selection procedures. Especially in the last decade, there has been an explosion in information theoretic approaches in neural networks and machine learning theories motivated by the elegant mathematical intuitions and the power of the analysis. The information geometry of statistical models has also been crucially influential in learning. However, in these domains, the conventional assumptions of either a known data distributions or discrete probability spaces is often a stretch and so the methodologies should be nonparametric and involve continuous random variables. The aim of this special issue is to present the current state of the art in the application of information theoretic concepts and approaches to learning, adaptation, and neural network theories by active experts working in the broadly defined area of information theoretical learning. In this special issue, we have a broad range of contributions. Iwata et al. discuss the effective utilization of information gain to devise strategies in reinforcement learning. Honkela and Valpola overview the developments in bits-back coding in
conjuction with a variational Bayesian learning poin-of-view. Rutkowski proposes a class of probabilistic neural networks that operate efficiently in nonstationary pattern classification environments. Schraudolph derives differential learning rules to optimize entropy-based criteria. Welling et al. study the under-complete generative models for independent components analysis and present an efficient sequential algorithm. Van Hulle offers an information theoretic approach for learning topographic maps based on kernel estimates. Cruces et al. portray a unified theoretical perspective relating source separation and extraction, while considering algorithmic stability issues. Morejon and Principe examine advanced parameter search algorithms for training neural networks using information theoretic criteria. Xu reexamines the Bayesian Ying-Yang Harmony Learning from an information theoretic perspective. Wang et al. present an approach to estimate mixture models from data based on the latent maximum entropy principle. Sanchez and Corbacho propose an information processing measure that simultaneously extracts relevant information and eliminates redundancies. Choi and Lee introduce a minimum negentropy based adaptation method for finite-length equalizers using nonpolynomial expansions. Sindhwani et al. develop a mutual information based feature selection algorithm for multi-class pattern recognition problems. We would like to thank Professor Jacek Zurada for providing the opportunity to prepare this special issue and his support throughout this project. Also we wish to thank all authors who have contributed to the special issue. Unfortunately, due to limited space, many high quality contributions could not be included in this issue. Finally, we thank all reviewers for their time and effort, as their contribution was crucial to the successful realization of this special issue. JOSE C. PRINCIPE, Guest Editor University of Florida Gainesville, FL 32601 USA ERKKI OJA, Guest Editor Helsinki University of Technology Espoo, Finland LEI XU, Guest Editor Chinese University of Hong Kong Shaitin, Hong Kong, China ANDRZEJ CICHOCKI, Guest Editor RIKEN, Brain Science Institute Saitama, Japan DENIZ ERDOGMUS, Guest Editor University of Florida Gainesville, FL 32601 USA
Digital Object Identifier 10.1109/TNN.2004.833368
1045-9227/04$20.00 © 2004 IEEE
790
IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 15, NO. 4, JULY 2004
Jose C. Principe (M’83–SM’90–F’00) is a Distinguished Professor of Electrical and Computer Engineering and Biomedical Engineering at the University of Florida, Gainesville, where he teaches advanced signal processing, machine learning, and artificial neural networks (ANNs) modeling. He is BellSouth Professor and the Founder and Director of the University of Florida Computational NeuroEngineering Laboratory (CNEL). His primary area of interest is processing of time varying signals with adaptive neural models. The CNEL Lab has been studying signal and pattern recognition principles based on information theoretic criteria (entropy and mutual information). He has more than 100 publications in refereed journals, 10 book chapters, and 200 conference papers. He has directed 42 Ph.D. degree dissertations and 57 masters degree theses. He recently wrote an interactive electronic book entitled Neural and Adaptive Systems: Fundamentals Through Simulation (New York: Wiley). Dr. Principe is a member of the ADCOM of the IEEE Signal Processing Society, Member of the Board of Governors of the International Neural Network Society, and Editor in Chief of the IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING. He is a member of the Advisory Board of the University of Florida Brain Institute.
Erkki Oja (S’75–M’78–SM’90–F’00) received the Dr. Sc. degree in 1977. He is currently the Director of the Neural Networks Research Centre and Professor of computer science, the Laboratory of Computer and Information Science, Helsinki University of Technology, Finland. Previously, he was a Research Associate at Brown University, Providence, RI, and Visiting Professor at the Tokyo Institute of Technology. He is a member of the editorial boards of several journals and has been in the program committees of several recent conferences including ICANN, IJCNN, and ICONIP. His research interests are in the study of principal component and independent component analysis, self-organization, statistical pattern recognition, and applying artificial neural networks to computer vision and signal processing. He is the author or coauthor of more than 260 articles and book chapters on pattern recognition, computer vision, and neural computing. He is also the author of three books: Subspace Methods of Pattern Recognition (New York: Wiley, 1983), which has been translated into Chinese and Japanese, Kohonen Map (New York: Elsevier, 1999), and Independent Component Analysis (New York: Wiley, 2001). He is member of the Finnish Academy of Sciences, Founding Fellow of the International Association of Pattern Recognition (IAPR), and President of the European Neural Network Society (ENNS). He is a member of the editorial boards of several journals and has been in the program committees of several recent conferences including ICANN, IJCNN, and ICONIP.
IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 15, NO. 4, JULY 2004
791
Lei Xu (SM’94–F’01) received the Ph.D. degree from Tsinghua University, Beijing, China, in 1987. He is a Chair Professor with the Department of Computer Science and Engineering, the Chinese University of Hong Kong (CUHK), Hong Kong. He joined the National Key Lab on Machine Perception, Peking University, Beijing, China, in 1987, where he became one of ten university-level exceptionally promoted young Associate Professors in 1988 and was exceptionally promoted to a Full Professor in 1992. From 1989 to 1993, he worked at several universities in Finland, Canada, and the United States, including Harvard University and the Massachusetts Institute of Technology, both in Cambridge, MA. He is also currently a Guest Professor in three universities in P.R. China and U.K. He has published over 300 papers, covering the areas of statistical learning and neural networks, financial engineering, computer vision and pattern recognition, signal processing, and artificial intelligence. He has given a number of keynote/plenary/invited/tutorial talks in international major neural networks (NN) conferences, such as WCNN, IEEE-ICNN, IJCNN, ICONIP, etc. Prof. Xu is one of the past Governors of the International Neural Networks Society, a past President of Asia-Pacific NN Assembly, a past Chair of the Computational Finance Technical Committee of the IEEE NN Society, and an Associate Editor for six international journals on NN, including Neural Networks and the IEEE TRANSACTIONS ON NEURAL NETWORKS from 1994 to 1998. He was an ICONIP ’96 Program Committee Chair, a Joint-ICANN-ICONIP03 Program Committee Co-chair and a General Chair of IDEAL ’98, IDEAL ’00, and IEEE CIFER’03. He has served as one of the program committee members in international major NN conferences over the past decade, including the International Joint Conference on Neural Networks, the World Conference on Neural Networks, and the IEEE-International Conference on Neural Networks. He has received several Chinese national prestigious academic awards, including the National Nature Science Prize, as well as international awards, including the 1995 INNS Leadership Award. He is a Fellow of the International Association on Pattern Recognition and a Member of the European Academy of Sciences.
Andrzej Cichocki (M’96) was born in Poland. He received the M.Sc. (with honors), Ph.D., and Habilitate Doctorate (Doctor of Science) degrees, all in electrical engineering, from Warsaw University of Technology, Warsaw, Poland, in 1972, 1975, and 1982, respectively. Since 1972, he has been with the Institute of Theory of Electrical Engineering, Measurements and Information Systems, Warsaw University of Technology, where he became a Full Professor in 1991. He spent a few years at the University Erlangen-Nuernberg, Germany, as Alexander Humboldt Research Fellow and Guest Professor. Since 1995, he has been working in the Brain Science Institute, Riken, Japan, as a Team Leader of the Laboratory for Open Information Systems, and currently as Head of Laboratory for Advanced Brain Signal Processing. He is the coauthor of three books: MOS Switched-Capacitor and Continuous-Time Integrated Circuits and Systems (New York: Springer-Verlag, 1989), Neural Networks for Optimization and Signal Processing (New York: Teubner-Wiley, 1993/94), and Adaptive Blind Signal and Image Processing (New York: Wiley, 2003) and more than 150 research journal papers. His current research interests include signal and image processing, especially analysis and processing of multisensory, and multimodal biomedical data. Prof. Cichocki is a Member of the IEEE SP Technical Committee for Machine Learning for Signal Processing and the IEEE Circuits and Systems Technical Committee for Blind Signal Processing.
Deniz Erdogmus (M’00) received the B.S. degree in electrical and electronics engineering and mathematics in 1997 and the M.S. degree in electrical and electronics engineering, with emphasis on systems and control, in 1999, both from the Middle East Technical University, Ankara, Turkey. He received the Ph.D. degree in electrical and computer engineering from the University of Florida, Gainesville, in 2002. He was a Research Engineer at the Defense Industries Research and Development Institute (SAGE), Ankara, from 1997 to 1999. Since 1999, he has been with the Computational NeuroEngineering Laboratory, University of Florida, working with Dr. J. C. Principe. His current research interests include information theory and its applications to adaptive systems and adaptive systems for signal processing, communications, and control. Dr. Erdogmus is a member of Tau Beta Pi and Eta Kappa Nu, and the recipient of the IEEE SPS 2003 Young Author Award.
