“The ever accelerating progress of technology … gives the appearance of approaching some essential singularity in the history of the race beyond which human affairs, as we know them, could not continue.” – John von Neumann
Like John von Neumann once said, we have reached a point of singularity where technological development is incomprehensibly rapid and complicated. Now, we are able to communicate with a non-existing artificial assistant in our phones and computers that can understand our speech; build self-driving cars and algorithms that recognize faces, objects like us! All of these fascinating technologies developed very rapidly when we start to use deep neural networks.
A neuron is a cell that creates chemical and electrical signals to communicate with other neurons to enable us to react to our environment. For example, when you are reading this post, the light waves coming from the screen excites your optical nerves at the back of your eye, and the neurons there transmit information to your visual cortex where earlier neurons there recognize edges of these weird patterns on the screen and later neurons there recognize that these weird patterns are letters and their combinations have a meaning. Under the inspiration of such a fascinating information processing that our brains perform in seconds, artificial neural network (ANN) technology was built.
ANNs are working “conceptually” similar to neurons in the human nervous system. Human neurons (Figure 1, A) receive input from dendrites. This input can be an electrical or chemical signal due to an environmental stimulus such as light. If the input signal is strong enough, the neuron becomes activated and the signal is transmitted through the axon and arrives at terminal axons where it triggers various chemical and electrical activities and returns an output that will be used as an input by the next neuron. Likewise, artificial neurons (Figure 1, B) receives input that we give to them (e.g., x: cat or dog image), performs operations inside (e.g., a function of f(x) ), and then returns an output (e.g., y: detection of whether the input is a cat or dog image).
However, humans have approximately 100 billion neurons and these neurons are highly connected. Therefore, our fascinating ability to learn and interact with the environment is not the result of one neuron; instead, it is the result of a network of neurons that works in an organized way to perform such complex information processing. Correspondingly, by using ANN structures, a machine learning technique called deep learning was developed to make machines process information with deep neural connected layers like humans. This network structure was called deep neural networks (DNN). Human neural networks (Figure 1, C) activate in different patterns in order to process various environmental stimuli. For example, the neural network in our primary visual cortex (V1) processes low-level visual features of the stimuli such as edges and lines; whereas the neural network in our inferior temporal gyrus (IT) processes higher-level visual features such as shape. One of the DNN types, Convolutional Neural Network (CNN) works in a similar manner too! In CNNs, there is an input and output layer and at least one hidden layer between them. Each layer is formed by several neurons and functions based on a feature hierarchy like our brain. For example, assume that we want to create a CNN model (Figure 1, D) that will detect whether a given image belongs to a cat or dog. In this case, the input layer will receive images of cats and dogs; then, neurons in the first layers of the model will detect edges whereas the later layers will detect shapes; and like human neural networks, the output layer will return a result such as detection of the image class (e.g., ‘this is a dog image’).
Figure 1. (A) A human neuron. (B) An artificial neuron. (C) Human neural network composed of 2 connected neurons. (D)Artificial deep neural network with one input, 2 hidden, 1 output layers