Research Introduction / Big Data Analysis
Introduction
The world is overflowing with vast amounts of data, and search services such as Google exist to extract relevant information from them. In addition to text data, you may also use image search. Although image search is one application, analyzing such big data remains a challenging problem. With the advent of deep learning, progress has been made in big data analysis and the development of new technologies. Here, as one example, we describe facial attribute estimation, which infers attributes (age, gender, hairstyle, etc.) from a massive facial image dataset.
Neural Networks
Deep learning employs deep neural networks. We begin with a brief explanation of neural networks. A network composed of units modeled on neurons, the nerve cells of the brain, is called a neural network. As shown on the left side of Figure 1, a neuron is a simple element that fires in response to incoming electrical signals. The brain is organized by connecting these neurons in a network-like fashion. The mathematical model of neurons and neural networks shown on the right side of Figure 1 is an artificial neural network (the term “artificial” is sometimes used because it differs from biological neural circuits and their mathematical models). In the mathematical model of a neural network, neurons are called units, and units are connected to form a network.
Figure 1: Neural networks as biological neural circuits and (artificial) neural networks as mathematical models
Deep Neural Networks
To solve complex and difficult problems, it is necessary to arrange many units and construct a deep network. A network with many stacked layers (columns), as shown in Figure 2, is called a deep neural network. A neural network optimizes the weights $w$ (see Figure 1) through learning using the correct relationship between input and output. When the network becomes deeper, however, a problem arises in which optimization becomes difficult. Deep learning resolved this issue. Deep learning made it possible to train deep neural networks, but this was not sufficient when images were used as input. Because the input corresponds to the number of pixels, the number of weights to be optimized becomes enormous, and learning becomes infeasible. Convolutional neural networks, described next, solved this problem.
Figure 2: Deep neural network
Convolutional Neural Networks
In a convolutional neural network (Convolutional Neural Network: CNN), features are extracted in convolutional layers and classification is performed in fully connected layers, as shown in Figure 3. In convolutional layers, filtering extracts features that are effective for recognition. The filter coefficients (weights) are optimized through learning in the same manner as in a neural network. Fully connected layers are the same as the deep neural network described above. CNNs have greatly improved the accuracy of image recognition and related tasks.
Figure 3: Convolutional neural network
Paradigm Shift in Image Recognition by CNNs
A large-scale image recognition dataset called ImageNet has been released, and image recognition competitions using this dataset (ImageNet Large Scale Visual Recognition Challenge (ILSVRC)) have been held. As Figure 4 shows, before the introduction of CNNs in 2011, the classification error rate exceeded 20%, but when CNNs were first used in 2012, the error rate dropped by 10 percentage points. Since 2012, various CNN methods have been proposed, and since 2015, accuracy exceeding that of humans has been achieved. Conventional image recognition relied on hand-designed features, but the success of CNNs at ILSVRC 2012 made methods that automatically learn features from massive image datasets the mainstream approach.
Figure 4: Evolution of image recognition performance in ILSVRC
Facial Attribute Estimation
Although the introduction has been lengthy, we now present facial attribute estimation, which is one of the research topics pursued in our laboratory. What kinds of attributes do faces have? For example, hair color, face shape, beard, eyeglasses, and many other attributes may come to mind. Because each attribute has different characteristics, designing features for attribute estimation is difficult. This is where deep learning, as explained above, becomes useful. In attribute estimation, the CelebA dataset is used for CNN training and evaluation. As shown in Figure 5, 40 types of attributes are defined. Facial attribute estimation is the problem of using a CNN to infer whether a face possesses each of these attributes.
Figure 5: 40 facial attributes defined in the CelebA dataset
DendroNet
In our laboratory, noting that relationships exist among attributes, we clustered the 40 attributes and represented them in a dendrogram. Based on this structure, we proposed DendroNet, a CNN for facial attribute estimation (see reference [1]). As shown in Figure 6, it is designed to extract local features from global features. We also proposed methods to optimize the network structure through learning for facial attribute estimation (see references [2], [3]).
Figure 6: DendroNet considering relationships among attributes
Demo of Facial Attribute Estimation
We have developed a demo system using the methods proposed in our laboratory. For this online open campus event, we present the results here. Figure 7 shows a facial photograph of Prof. Aoki. When this photograph is submitted as input, the results are displayed as a bar chart. Red indicates that the attribute is present, and blue indicates that it is absent. For Prof. Aoki, the system predicted straight hair, black hair, bags under the eyes, a big nose, wearing a necktie, and male. Looking at the photograph, the results appear reasonable. In this way, given a large-scale image dataset and labels appropriate to the problem, deep learning can be used to solve the task.
Figure 7: Facial photograph of Prof. Aoki and attribute estimation results
Figure 7: Facial photograph of Prof. Aoki and attribute estimation results
Automatic Classification of Plankton
We introduce automatic classification of plankton, one of the big data analysis projects pursued in our laboratory. Plankton occupy the base of aquatic ecosystems in seas, lakes, and marshes, and are closely related to water quality. For example, by examining the species and numbers of plankton inhabiting a lake, the current water quality can be assessed. On the other hand, experts must spend considerable time classifying plankton, which poses a challenge for water quality forecasting. Here, deep learning offers a solution. In a joint project with the Graduate School of Life Sciences, Tohoku University, we have constructed a plankton image database such as that shown in Figure 8. This database is annotated with hierarchical labels based on plankton taxonomic ranks. Because plankton species differ by subtle characteristics, hierarchical labels enable high-accuracy classification (see reference [4]).
Figure 8: Examples of plankton images
Summary
Although this introduction has been somewhat long, we have briefly presented big data analysis using deep learning. Given a large-scale dataset and ground-truth labels, deep learning has made it possible to solve many problems, but from a research perspective, that is only the starting point. The next research challenges are how to improve performance from there and how to tackle new problems.
References
- H. Kawai et al., "Performance evaluation of face attribute estimation method using DendroNet," Proc. IEEE Global Conf. Consumer Electronics, pp. 186--187, October 2019. [PDF]
- H. Kawai et al., "Merged multi-CNN with parameter reduction for face attribute estimation," Proc. Int'l Conf. Biometrics, June 2019. [PDF]
- H. Kawai et al., "Face attribute estimation using multi-task convolutional neural network," J. Imaging, vol. 8, no. 4, p. 105, April 2022 (Open access).
- K. Ito et al., "Zooplankton classification using hierarchical attention branch network," Proc. Asian Conf. Pattern Recognition (LNCS 14407), pp. 509--419, November 2023. [PDF]