Recurrent Neural Networks

As you may see from any guide available online, RNNs are out of date. This is wrong, at least because about 30 percent of problems that data scientists will have to solve in the current year will require RNNs.

I am not trying to be controversial; rather, this follows from the understanding of the areas where RNNs still ship in production – from time series predictions on edge devices to low-latency sensor pipelines, wherever the memory burden of transformers’ attention mechanisms is too expensive to afford. The ability to tell when RNNs should be used in place of transformers is more useful than just knowing that there are RNNs at all.

In this guide, I will cover both issues: RNNs themselves, both theoretical and practical aspects, and the question whether RNNs should be considered in 2026.

What Is a Recurrent Neural Network?

Recurrent Neural Networks refer to neural networks which use their previous outputs as a component of the next step’s inputs. Period.

All other networks that you have used up till now (e.g., classifier, CNN for images) treat each of their inputs as a totally new and distinct problem. Give it an image of a cat and it will tell you “cat”. Give it the same image tomorrow, it will still tell you “cat” without having any recollection of what happened yesterday. However, RNNs purposely do not treat each input as a new and independent problem.

Consider the sentence “The clouds were getting dark, so I took out an…” It’s not like you understand just the word “an.” You have the context of the entire sentence in your mind, and you may even have guessed “umbrella” even before seeing the word. This is how an RNN works: retain the context of whatever came previously and then predict something from it.

The reason for this is the common property in language, audio signals, sensor data, and stock prices — order matters. “The dog bit the man” and “the man bit the dog” contain exactly the same words but have entirely different meanings. A feedforward network will consider both of these inputs equally similar, whereas RNNs won’t.

How RNNs Work: The Hidden State Explained

Here’s how it works, and we don’t omit the actual important step.

In each iteration, the RNN receives two inputs at once – one is a present input; the other one is called “a hidden state,” which means the sum of all processed data from previous iterations. The RNN merges them into an output and updates the hidden state. Then, the process starts again.

Hidden state is updated in the following way:

This may look intimidating, but h_t is our new hidden state, while h_(t-1) represents what the neural network learned about the previous step. x_t is the current input. W_hh and W_xh are the weight matrices – values that the neural network figures out when it’s being trained and dictate the importance of the old state and the current input to the output. The tanh function ensures that we don’t get too big or too small values because it limits the output to [-1, 1].

A worked example with real numbers. Consider that we’re working through a small sequence “go now” letter by letter, and our hidden state is represented by just one number (networks in practice have hidden states composed of vectors with scores of elements, but here we will limit it to just one number).

Start with h₀ = 0 (no memory yet). Say W_hh = 0.5 and W_xh = 0.8, and we encode ‘g’ as x₁ = 1.0.

h₁ = tanh(0.5 × 0 + 0.8 × 1.0) = tanh(0.8) ≈ 0.664

Now feed in ‘o’, encoded as x₂ = 0.6:

h₂ = tanh(0.5 × 0.664 + 0.8 × 0.6) = tanh(0.332 + 0.48) = tanh(0.812) ≈ 0.671

Take note of what just took place. Not only is the second hidden state a function of ‘o’. It is also a combination of ‘o’ and the memory of ‘g’ in the previous iteration. And such a combination of states, for each letter or word in a sequence, is what enables the network to create context. 

Now, when the network gets an output, it will compare it to the correct value and make adjustments to its weight matrices so as to minimize the error. But since the error at the last step is dependent on all the previous hidden states, the process of error correction or backpropagation through time (BPTT) needs to travel back in time through the whole sequence.

And that’s where RNNs stumble.

The Problem RNNs Were Built to Solve (And the One They Created)

An RNN solves the problem of considering the sequence important. However, it brings another issue: memory for something that happened in the distant past.

When BPTT is used, the signal telling each weight which direction to move is multiplied many times during its backpropagation through time steps. If any multiplication includes a number less than 1, then the gradient gets reduced rapidly. In some time steps, it will become zero, and the network will be unable to learn from events that happened several time steps ago. It is called the vanishing gradient problem and explains why vanilla RNN cannot learn from connecting “Tom is a cat” with “she eats fish”.

The converse error also occurs. When the repeated multiplications consist of elements greater than 1, the gradient diverges instead of collapsing to zero. The updates for the weights become very large, leading to instabilities in the training process where the loss becomes erratic or even becomes NaN. The common solution is gradient clipping, where the gradient is bounded by a threshold.

Vanilla recurrent neural networks, also known as simple or “Elman” recurrent neural networks (in recognition of the pioneering work of Jeffrey Elman in the field, who proposed the architecture back in 1990), perform well on short sequences but suffer from the vanishing gradient problem on sequences with long-range dependencies.

And this is precisely what LSTM solves.

LSTM vs GRU: Which One Should You Actually Use?

Long Short-Term Memory (LSTM) architecture was proposed by Sepp Hochreiter and Jürgen Schmidhuber in 1997 for solving the vanishing gradient problem. Their solution was to provide the model with an additional memory pathway called a cell state where information could flow without being changed and controlled by three types of gates: input, forget, and output.

The input gate determines how much of the current input will be written to memory. The forget gate determines what memories to forget. The output gate determines what part of the memory is going to be used for producing the current output. Three gates, three weight vectors, and many more parameters than a regular RNN – that’s the cost you pay for long-term memory.

Gated Recurrent Units, presented by Cho et al. in 2014, reduce the complexity of that model. GRUs combine the input and the forget gate into one “update gate” and get rid of the separate cell state – everything becomes part of the hidden state. Less gates mean less parameters, which means faster learning and less memory required.

So what should you choose? This is the real answer, and few articles give it: start with GRU, move to LSTM only when necessary based on performance on the validation set.

Factor	LSTM	GRU
Parameters	More (3 gates + cell state)	Fewer (2 gates, no separate cell state)
Training speed	Slower	~25–30% faster, typically
Performance on very long sequences	Often slightly better	Competitive, occasionally behind
Performance on short-to-medium sequences	Comparable	Comparable, sometimes better
Best default for limited data or compute	Less ideal	Better starting point

I feel that the approach of comparing GRU and LSTM is inverted in most tutorials. They give the impression that this is an arbitrary decision like flipping a coin but the truth is that you must first make sure that GRU is not able to learn long-term dependencies before you switch to LSTM.

RNN Architectures by Input/Output Shape

Beyond the gating component of an RNN (vanilla, LSTM, GRU), we may classify RNNs based on the number of inputs and outputs that each model is allowed to take.

One-to-one classification has very little recurrent aspect; a single input, followed by a single output, without any sequence processing. It is mentioned mostly to be complete since such a structure can be achieved using a conventional feedforward network.

One-to-many mapping accepts a single input and produces a series of outputs. The perfect example is image captioning, where we receive a picture and generate its description letter by letter.

Many-to-one there is a sequence of inputs, but a single output is generated. Sentiment analysis belongs here, where we have a long product review but a single positive or negative sentiment result.

Many-to-many we have a mapping from one sequence to another. The canonical example is machine translation where the sequence of one language is mapped to another sequence of a different language.

Another fifth pattern can be found which is not captured by one-to-one/many-to-many taxonomy of purely feedforward architectures: encoder-decoder, in which the first RNN encodes the whole sequence of inputs to a fixed-length context vector and the second RNN decodes the vector back to the sequence of outputs. It is precisely the type of architecture used by Google for the first neural machine translation system created by them before the appearance of attention mechanisms and transformers. And the bottleneck of this model was precisely the fixed-length of that context vector, since it encoded an arbitrarily long sentence.

RNN vs Transformer: Should You Even Use an RNN in 2026?

The one question that all other guides shy away from asking is this: If transformers are superior in almost every way, then why should one study RNNs at all?

Two straightforward answers. First, you cannot appreciate the reason for the superiority of transformers without first appreciating the problem that transformers sought to solve in the first place – attention was added to RNN encoder/decoders initially. Second, and more pragmatically, RNNs are the right choice for certain repetitive tasks.

RNNs still make sense when:

• You’re operating in constrained environments (embedded devices, wearables, some IoTs) where the attention matrix in transformers becomes too costly to implement
• Your sequences are being processed as they come in through a real-time stream, and you need a prediction immediately, without waiting for the whole sequence
• Your data sets are small enough that the larger number of parameters in transformers becomes a threat to overfitting
• Latency is of greater importance than the incremental improvement in accuracy, and your sequences are so short that vanishing gradients are not an issue

Transformers win when:

• The sequences are lengthy and dependencies exist far apart; self-attention considers all positions together rather than relying on past positions only.
• You have the resources available to parallelize the training using multiple GPUs; RNNs consider positions one by one in a sequential manner.
• You have enough computing power and datasets available to build a bigger model.

To sum up, the simple answer is that we no longer have an issue of RNN vs. Transformer; instead, what exists is constraints vs. capabilities and in 2026, most product groups will be using the transformers by default while RNNs/GRUs only on edge inference.

Building a Character-Level Text Generator: Full Code Walkthrough

Theory is all fine, but here’s how an RNN looks in action. We’ll code up a simple character-based text generation model in TensorFlow/ Keras. You give it some input, it tries to predict the rest. It’s the exact same toy problem that GFG’s tutorial works on, except we’ll also go into the rationale behind the steps they miss out on.

Step 1 — Imports.

import numpy as np

import tensorflow as tf

from tensorflow.keras.models import Sequential

from tensorflow.keras.layers import SimpleRNN, Dense

Nothing exotic. NumPy handles the array math, TensorFlow and Keras handle the network itself.

Step 2 — Prepare the character vocabulary.

text = "This is GeeksforGeeks a software training institute"

chars = sorted(list(set(text)))

char_to_index = {char: i for i, char in enumerate(chars)}

index_to_char = {i: char for i, char in enumerate(chars)}

Neural networks don’t understand letters. They understand numbers. This step builds a lookup table mapping each unique character to an integer, and back again.

Step 3 — Build training sequences.

seq_length = 3

sequences = []

labels = []

for i in range(len(text) - seq_length):

    seq = text[i:i + seq_length]

    label = text[i + seq_length]

    sequences.append([char_to_index[char] for char in seq])

    labels.append(char_to_index[label])

X = np.array(sequences)

y = np.array(labels)

That’s where most tutorials stop. We slide a 3-character window over the input, and for each window we log which character comes after it. “Thi” → “s”. “his” → ” “. This is the whole training signal; predict the next character from the last three characters.

Step 4 — One-hot encode.

X_one_hot = tf.one_hot(X, len(chars))

y_one_hot = tf.one_hot(y, len(chars))

One-hot encoding involves converting each character index to a vector of all zeroes except a 1 indicating the index of the character. This is an inefficient encoding scheme for large vocabularies, but it is the most explicit form of encoding.

Step 5 — Build the model.

model = Sequential()

model.add(SimpleRNN(50, input_shape=(seq_length, len(chars)), activation="relu"))

model.add(Dense(len(chars), activation="softmax"))

One recurrent layer with 50 hidden units, feeding into a dense output layer that produces a probability distribution over every possible next character.

Step 6 — Train it.

model.compile(optimizer="adam", loss="categorical_crossentropy", metrics=["accuracy"])

model.fit(X_one_hot, y_one_hot, epochs=100)

Step 7 — Generate new text.

start_seq = "This is G"

generated_text = start_seq

for i in range(50):

    x = np.array([[char_to_index[char] for char in generated_text[-seq_length:]]])

    x_one_hot = tf.one_hot(x, len(chars))

    prediction = model.predict(x_one_hot)

    next_index = np.argmax(prediction)

    next_char = index_to_char[next_index]

    generated_text += next_char

print(generated_text)

For each iteration, the last three characters are fed into the model for prediction of the next character, which is then appended. The model is learning text character by character by feeding itself with the output it produces.

For such a small dataset, do not expect Shakespeare. In fact, the model is more likely to be just repeating the training text than making any generalizations, as there is not sufficient data to learn from. This is not a coding mistake; it is just an honest example of why real text generation models use gigabytes of data.

Common RNN Mistakes (And How to Actually Fix Them)

Few RNN tutorials end with “here is the code”. Here are some things that might go wrong, as they do.

Loss stays flat or barely moves. Most probably you’ve set an inappropriate learning rate. Try dividing it by 10. If it helps nothing, check whether your inputs are normalized. RNNs are sensitive to the scale of input data.

Loss becomes NaN partway through training. That’s an exploding gradient problem, not a bug in your code. Implement gradient clipping; in Keras it means adding clipnorm or clipvalue parameter to your optimizer.

The model performs well on short sequences but falls apart on long ones. Vanishing gradient problem classic example. Replace your Simple RNN layer with LSTM/GRU. This is by far the most efficient solution you have, which is the reason no one uses raw RNN models for any practical purposes.

Training is painfully slow. RNNs operate in a sequential manner, hence making them unable to parallelize operations like the transformer does. In case speed becomes an issue and you have short sequences, GRU trains much faster compared to LSTM.

Frequently Asked Questions

Where This Leaves You

RNNs are not what one turns to by default these days. However, “not the default” and “completely useless” are two very different things, which is precisely the sort of confusion that makes it impossible for engineers to even understand why we have attention.

When you’re running on limited hardware, dealing with short sequences, or need to make predictions on-the-fly – start with a GRU, not a transformer. You’ll be glad you did when your memory is maxed out.