What Is a Neural Network?

Google's machine learning glossary defines a neural network as a model containing at least one hidden layer. A network with more than one hidden layer is called a deep neural network. Deep learning rests on these deep networks and is covered in the machine learning entry.

The name comes from biology. In 1986, David Rumelhart, Geoffrey Hinton and Ronald Williams described their learning procedure in Nature for "networks of neurone-like units". Google writes that neural networks implemented on computers are sometimes called artificial neural networks to differentiate them from those found in brains and other nervous systems. The computational model borrows from the brain the picture of units passing signals on. It is not a replica of the brain.

Neural networks are one type of model within machine learning, not the whole field. A linear model without a hidden layer is not a neural network under Google's definition, yet it also learns from data.

A network has three kinds of layer. The input layer takes the features of an example, such as numeric values standing for words or pixels. One or more hidden layers follow. The output layer delivers the prediction. The nodes in the hidden layers are called neurons.

Each neuron first calculates like a linear model: it multiplies each input value by its own weight, adds up the results and adds a bias. It then applies an activation function. An example with made-up numbers: inputs 2 and 3, weights 0.5 and −1 and a bias of 1 give 2 × 0.5 + 3 × (−1) + 1 = −1. The ReLU activation function turns any negative value into 0, so the neuron passes on 0.

The activation function is not a detail. Google explains in its Machine Learning Crash Course that linear operations performed on linear operations are still linear. Without a nonlinear function, the network could not learn nonlinear relationships, however many layers it had.

Learning happens through backpropagation. In 1986, Rumelhart, Hinton and Williams described a procedure that repeatedly adjusts the weights of the connections to reduce the difference between the actual and the desired output. To do so, the algorithm calculates for each parameter how much the error depends on it. Google calls backpropagation the most common training algorithm for neural networks. It makes gradient descent feasible for multi-layer networks.

After training, the weights stay fixed. The network then applies them to new inputs.

The language models behind AI answers are neural networks. Google's glossary informally describes a large language model as any Transformer-based language model, such as Gemini or GPT, and defines the Transformer as a neural network architecture. The developers say the same about their own models. OpenAI calls GPT-4 a Transformer-based model in its 2023 technical report. Google writes in the Gemini report (May 2025 version) that Gemini models build on top of Transformer decoders. How later models are built, these reports do not say.

This has a consequence for any brand that appears in such answers. A network stores what it knows in its weights, not as records in a database. Rumelhart and colleagues already described in 1986 that hidden units come to represent important features of the task during training, and that the regularities are captured by the interactions of these units. So there is no single place where a false sentence about a company is written down and can be corrected. Why an answer turns out a certain way cannot be looked up either. More on this in the LLM and AI hallucination entries.

Marketing itself runs on neural networks too, often without the tool saying so: in the text and image generators of generative AI and in systems that represent text as numeric vectors, known as embeddings.

• Keep the terms apart: AI is the umbrella term, machine learning one part of it, a neural network one type of model within that, and an LLM one specific neural network.
• Ask which model a tool uses and how current its data is, rather than settling for the label "AI".
• Treat a network's output as a calculated probability, not a looked-up fact, and check facts at the source.
• Do not assume that a simple relationship with few features needs a neural network. Google presents them as an architecture for nonlinear patterns, and a linear model may be enough.
• Read a developer's statements about architecture as applying to the model and version the report names.

• Taking a neural network for a replica of the brain.
• Assuming every machine learning model is a neural network.
• Reading "deep" as a judgement of quality. It only means more than one hidden layer.
• Expecting to find and change a false fact inside the model the way you would edit a record.
• Inferring from the number of parameters how useful a model will be for a specific task.