What Is a Recurrent Neural Network?

A recurrent neural network is a class of neural network designed to process sequential data by maintaining an internal memory of previous inputs. Unlike feedforward networks that treat each input independently, RNNs pass information from one step of a sequence to the next through a hidden state, allowing them to capture temporal dependencies and patterns that unfold over time.

The defining feature of a recurrent neural network is its feedback loop. At each time step, the network receives a new input and combines it with the hidden state from the previous step to produce an output and an updated hidden state. This mechanism gives RNNs the ability to model relationships between elements in a sequence, whether those elements are words in a sentence, samples in an audio signal, or data points in a time series.

RNNs belong to the broader family of deep learning architectures and were among the first neural network designs to handle variable-length sequential inputs effectively. They played a foundational role in advancing natural language processing, speech recognition, and time series forecasting before the rise of attention-based models.

Understanding how RNNs work remains essential for anyone studying artificial intelligence and sequence modeling, as many of the concepts they introduced still inform modern architectures.

How RNNs Work

The Recurrence Mechanism

The core of a recurrent neural network is a repeating computational unit that processes one element of a sequence at a time. At each time step t, the network takes two inputs: the current element of the sequence (such as a word embedding or a sensor reading) and the hidden state from the previous time step. It applies learned weight matrices to both inputs, sums the results, and passes them through an activation function, typically tanh, to produce a new hidden state.

Formally, the hidden state update follows this pattern: the new hidden state equals the activation function applied to the sum of the input-to-hidden weight matrix multiplied by the current input and the hidden-to-hidden weight matrix multiplied by the previous hidden state, plus a bias term. The output at each time step is then computed from the current hidden state through another set of weights.

This recurrence is what distinguishes RNNs from other architectures. The hidden state acts as a compressed representation of everything the network has seen so far in the sequence. In theory, this allows the network to use context from arbitrarily far back when making predictions about the current time step.

Weight Sharing Across Time Steps

A key property of RNNs is that the same set of weights is applied at every time step. The network does not learn separate parameters for position one, position two, and so on. Instead, it reuses the same weight matrices throughout the entire sequence. This weight sharing has two important consequences.

First, it makes RNNs parameter-efficient. Regardless of how long the input sequence is, the number of learnable parameters stays the same. A network processing a 10-word sentence uses the same weights as one processing a 500-word paragraph.

Second, weight sharing enables RNNs to generalize across sequence positions. A pattern learned at the beginning of a sequence can be recognized when it appears later. This is analogous to how convolutional neural networks share filter weights across spatial positions in an image.

Training with Backpropagation Through Time

RNNs are trained using a variant of the standard backpropagation algorithm called backpropagation through time (BPTT). The network is "unrolled" across all time steps of a sequence, creating a computational graph that resembles a very deep feedforward network with shared weights at each layer.

The loss function is computed at the output, and gradients flow backward through this unrolled graph using gradient descent optimization. At each time step, the gradients are accumulated and used to update the shared weight matrices. Because the same weights appear at every time step, the gradient contributions from all positions are summed together.

BPTT is conceptually straightforward but computationally demanding. For long sequences, the unrolled graph becomes very deep, and storing intermediate activations for all time steps requires substantial memory. In practice, truncated BPTT is often used, where the gradient computation is limited to a fixed window of recent time steps rather than propagating all the way back to the start of the sequence.

Types of RNN Architectures

Vanilla RNN

The standard or "vanilla" RNN uses the simple recurrence formula described above. It is the most basic form and serves as the conceptual foundation for all recurrent architectures. Vanilla RNNs work reasonably well for short sequences where the relevant context is close to the prediction point.

However, vanilla RNNs struggle with long sequences due to the vanishing gradient problem. As gradients propagate backward through many time steps, repeated multiplication by the same weight matrix causes them to shrink exponentially. This makes it nearly impossible for the network to learn dependencies that span more than roughly 10 to 20 time steps. This fundamental limitation motivated the development of gated architectures.

Long Short-Term Memory (LSTM)

The LSTM architecture, introduced by Hochreiter and Schmidhuber in 1997, was specifically designed to address the vanishing gradient problem. LSTMs add a cell state, a separate pathway that runs through the entire sequence and allows information to flow with minimal modification. Three gating mechanisms control what information enters, persists in, and exits the cell state.

- The forget gate examines the previous hidden state and the current input, then outputs a value between 0 and 1 for each element of the cell state. A value near 0 means "discard this information," while a value near 1 means "retain it."

- The input gate determines which new information should be written into the cell state. It consists of two parts: a sigmoid layer that decides which values to update and a tanh layer that creates a vector of candidate values.

- The output gate controls which parts of the cell state are exposed as the hidden state output for the current time step.

These gates are learned during training and allow LSTMs to selectively remember or forget information over long sequences. In practice, LSTMs can capture dependencies spanning hundreds of time steps, a dramatic improvement over vanilla RNNs.

They became the dominant recurrent architecture for most of the 2010s and powered breakthroughs in machine translation, speech recognition, and language modeling.

Gated Recurrent Unit (GRU)

The GRU, proposed by Cho et al. in 2014, simplifies the LSTM design by combining the forget and input gates into a single update gate and merging the cell state and hidden state into one. This reduces the number of parameters and makes the architecture faster to train while maintaining much of the LSTM's ability to capture long-range dependencies.

- The update gate controls how much of the previous hidden state is carried forward versus how much is replaced by new candidate information.

- The reset gate determines how much of the previous hidden state is used when computing the candidate for the new hidden state.

GRUs perform comparably to LSTMs on many tasks and are often preferred when computational resources are limited or when the dataset is not large enough to justify the additional parameters of an LSTM. Neither architecture is universally superior; the best choice depends on the specific task and data characteristics.

Bidirectional RNNs

Standard RNNs process sequences in one direction, from the first element to the last. Bidirectional RNNs run two separate recurrent layers: one processing the sequence forward and another processing it backward. At each time step, the outputs of both layers are concatenated to produce a representation that incorporates context from both past and future elements.

Bidirectional architectures are particularly valuable for tasks where the meaning of a current element depends on what comes after it. In natural language processing, the correct interpretation of a word often depends on subsequent words. Bidirectional LSTMs and GRUs became standard components in named entity recognition, part-of-speech tagging, and sentiment analysis pipelines.

The limitation of bidirectional RNNs is that they require the complete sequence to be available before processing can begin. This makes them unsuitable for real-time or streaming applications where data arrives incrementally.

Encoder-Decoder (Sequence-to-Sequence) Architecture

The encoder-decoder framework uses two RNNs in tandem. The encoder RNN processes the input sequence and compresses it into a fixed-length context vector, which is the final hidden state of the encoder. The decoder RNN then takes this context vector and generates an output sequence one element at a time.

This architecture was originally developed for machine translation, where the input is a sentence in one language and the output is the translation in another. The encoder-decoder pattern proved flexible enough to generalize to summarization, question answering, and other sequence-to-sequence tasks.

The addition of attention mechanisms to this framework, allowing the decoder to selectively focus on different parts of the encoder output at each step, was a pivotal development that eventually led to the transformer model.

Type	Description	Best For
Vanilla RNN	The standard or "vanilla" RNN uses the simple recurrence formula described above.	—
Long Short-Term Memory (LSTM)	The LSTM architecture.	—
Gated Recurrent Unit (GRU)	The GRU, proposed by Cho et al. in 2014.	—
Bidirectional RNNs	Standard RNNs process sequences in one direction, from the first element to the last.	Tasks where the meaning of a current element depends on what comes
Encoder-Decoder (Sequence-to-Sequence) Architecture	The encoder-decoder framework uses two RNNs in tandem.	—

RNN Use Cases

Natural Language Processing

RNNs were the backbone of NLP systems for most of the past decade. They power text classification, sentiment analysis, named entity recognition, and language generation tasks. In language modeling, RNNs learn to predict the next word in a sequence by capturing statistical patterns in large text corpora. This capability forms the foundation of autocomplete systems, chatbot response generation, and grammar correction tools.

LSTM-based language models demonstrated that neural approaches could surpass traditional n-gram models in perplexity, a standard metric for language model quality. These models also enabled transfer learning in NLP, where representations learned on large unlabeled corpora could be fine-tuned for specific downstream tasks, a paradigm that later reached its full potential with transformer architectures.

Speech Recognition

Converting spoken language to text is inherently a sequence-to-sequence problem. The input is a series of audio frames, and the output is a sequence of characters or words. RNNs, particularly deep bidirectional LSTMs, became the standard acoustic model in modern speech recognition systems throughout the 2010s.

The recurrent structure naturally accommodates the variable-length nature of speech. Different speakers say the same word at different speeds, and RNNs handle this temporal variability by processing each audio frame in context. Combined with connectionist temporal classification (CTC) loss functions, LSTM-based models achieved dramatic improvements in word error rates on standard benchmarks.

Time Series Forecasting

Financial data, weather patterns, energy consumption, and sensor readings all produce sequential data with temporal dependencies. RNNs model these dependencies by learning patterns in historical observations and extrapolating them forward. An RNN trained on monthly sales figures, for example, can learn seasonal patterns, trend directions, and the influence of lagged variables.

LSTMs are especially useful for time series tasks because they can capture both short-term fluctuations and longer-term trends through their gating mechanism. In practice, LSTM-based forecasters compete with and often outperform traditional statistical methods like ARIMA on complex, multivariate time series where nonlinear interactions between variables are present.

Music Generation

RNNs can model musical sequences by treating notes, chords, or audio samples as sequential elements. Trained on a corpus of compositions, an RNN learns the statistical structure of music: which notes tend to follow which, how rhythm patterns repeat and vary, and how harmonic progressions resolve. The network can then generate new sequences that follow similar stylistic patterns.

This application is a form of unsupervised learning in the sense that the model learns structure from the data without explicit labels. LSTM networks have been used to generate music in styles ranging from classical piano to jazz improvisation, demonstrating the flexibility of recurrent architectures in creative domains.

Reinforcement Learning

RNNs appear in reinforcement learning systems where agents must make decisions based on sequences of observations. In partially observable environments, where the agent cannot see the full state of the world at any single moment, an RNN's hidden state serves as a form of memory that accumulates information from past observations.

This allows the agent to infer aspects of the environment that are not directly visible and make better-informed decisions.

Challenges and Limitations

Vanishing and Exploding Gradients

The most fundamental challenge with RNNs is gradient instability during training. When gradients are propagated back through many time steps, they are repeatedly multiplied by the recurrent weight matrix. If the largest singular value of this matrix is less than one, gradients shrink exponentially, a phenomenon known as the vanishing gradient problem. If the largest singular value exceeds one, gradients grow exponentially, causing the exploding gradient problem.

Vanishing gradients prevent the network from learning long-range dependencies because the gradient signal from distant time steps effectively disappears before it can influence the weights. Exploding gradients cause training instability, with weight updates oscillating wildly or producing NaN values. Gradient clipping, which caps the gradient magnitude at a threshold, addresses the exploding case.

LSTMs and GRUs mitigate the vanishing case through their gating mechanisms, but they do not eliminate it entirely for very long sequences.

Sequential Processing Bottleneck

RNNs process sequence elements one at a time. Each time step depends on the output of the previous one, which means computation cannot be parallelized across positions within a sequence. This creates a fundamental throughput bottleneck, especially for long sequences and large batch sizes.

On modern GPU hardware optimized for parallel computation, this sequential dependency is a significant practical disadvantage.

Training an RNN on a dataset of long sequences takes considerably more wall-clock time than training a parallelizable architecture like a transformer model or a convolutional neural network on the same data, even when the total number of floating-point operations is comparable.

Difficulty with Very Long Sequences

Despite the improvements that LSTMs and GRUs bring, RNNs still struggle with sequences that span thousands of elements. The hidden state has a fixed dimensionality, meaning it must compress all relevant information from the entire sequence history into a vector of constant size. As sequences grow longer, the network inevitably loses information.

The attention mechanism was introduced partly to address this limitation by allowing the decoder to access all encoder hidden states directly rather than relying solely on the final compressed context vector. This innovation proved so effective that it became the central idea behind the transformer architecture, which dispenses with recurrence entirely.

Hyperparameter Sensitivity

Training RNNs effectively requires careful tuning of learning rate, hidden state size, number of layers, dropout regularization rate, and sequence truncation length. RNNs are more sensitive to hyperparameter choices than many other architectures. A learning rate that is slightly too high causes divergence, while one that is too low makes training prohibitively slow.

Choosing the right hidden state dimensionality involves balancing model capacity against overfitting risk and training speed.

Supervised learning tasks with clear performance metrics make tuning more tractable, but exploratory or generative applications often require extensive experimentation to find stable training configurations.

RNNs vs Transformers

The transformer model, introduced in 2017, replaced recurrence with self-attention as the primary mechanism for modeling sequence relationships. This shift had profound consequences for the field and largely displaced RNNs from their position as the default architecture for sequence tasks.

Parallelization. Transformers process all positions in a sequence simultaneously. Self-attention computes relationships between every pair of elements in parallel, which maps efficiently onto GPU hardware. RNNs must process positions sequentially, making them slower to train on the same hardware.

Long-range dependencies. In a transformer, every position can attend directly to every other position regardless of distance. The path length between any two elements is one attention step. In an RNN, information from a distant position must propagate through every intermediate hidden state, with increasing risk of degradation. Transformers handle long-range dependencies more reliably.

Scalability. Transformers scale more effectively with increased data and compute. The largest language models, trained on hundreds of billions of parameters and trillions of tokens, are all transformer-based. RNNs have not been scaled to comparable sizes because their sequential processing bottleneck makes training at that scale impractical.

Memory efficiency for long sequences. Self-attention has quadratic memory and time complexity with respect to sequence length. For very long sequences, this becomes expensive. RNNs have linear complexity in sequence length, which gives them an advantage in certain streaming or long-context scenarios. Recent research into state-space models and linear attention variants draws on ideas from recurrent architectures to address this limitation of transformers.

Simplicity and inductive bias. RNNs have a natural inductive bias toward sequential processing. For tasks where the data is genuinely sequential and order matters fundamentally, this bias can be beneficial, especially with limited training data. Transformers impose fewer structural assumptions, which makes them more flexible but also more data-hungry.

In practice, transformers have become the dominant choice for machine learning applications involving text, code, and multimodal data. RNNs remain relevant in edge deployment scenarios with strict latency and memory constraints, in streaming applications where data arrives continuously, and in domains where training data is limited and the sequential inductive bias helps.

Practitioners working with frameworks like PyTorch will find robust implementations of both architectures, and understanding the strengths and trade-offs of each remains an important competency.

FAQ

What is the difference between an RNN and a feedforward neural network?

A feedforward neural network processes each input independently. Data flows in one direction from input to output with no cycles or feedback connections. An RNN introduces a feedback loop where the hidden state from the previous time step is fed back as an additional input at the current step.

This allows RNNs to maintain memory of prior inputs and model dependencies within sequential data, something feedforward networks cannot do without external engineering.

What is the vanishing gradient problem in RNNs?

The vanishing gradient problem occurs when gradients shrink exponentially as they are propagated backward through many time steps during training. Because the same weight matrix is applied at every step, small gradient values get multiplied repeatedly, causing the learning signal for early time steps to effectively disappear. This prevents the network from learning long-range dependencies.

LSTM and GRU architectures mitigate this problem through gating mechanisms that create more direct paths for gradient flow.

Are RNNs still used in practice?

Yes, though their prevalence has declined since the widespread adoption of transformer architectures. RNNs remain practical for edge deployment where model size and inference latency are tightly constrained, for real-time streaming applications, and for time series tasks where the sequential inductive bias provides a structural advantage.

LSTM variants are still common in production speech recognition, sensor data processing, and embedded systems where transformer models would be too large or too slow.

How do LSTMs differ from standard RNNs?

Standard RNNs use a single recurrence equation with a tanh activation. LSTMs add a dedicated cell state and three gating mechanisms (forget, input, and output gates) that control information flow. The cell state provides a highway for information to persist across many time steps with minimal degradation. The gates, implemented as sigmoid layers, learn during training which information to retain, discard, or expose.

This design allows LSTMs to capture dependencies across hundreds of time steps where vanilla RNNs would fail.

What tasks are RNNs best suited for?

RNNs excel at tasks involving sequential data where temporal order is meaningful. Strong use cases include time series forecasting, speech recognition, streaming data analysis, music generation, and sequence labeling tasks in NLP. They are particularly well suited for scenarios with limited training data, strict latency requirements, or deployment on resource-constrained hardware.

For large-scale language tasks with abundant data and compute, transformer models generally deliver better results.