Tanh Activation Function in Neural Networks

Tanh activation function maps any value from the set of real numbers to values ranging from -1 to 1. This is achieved through the formula tanh(x) = (eˣ − e⁻ˣ) / (eˣ + e⁻ˣ), where “e” represents Euler’s number which is about 2.718, and x stands for the actual input value. Tanh creates an S-shaped graph whose center is zero, meaning that an input of 0 will always yield 0.

This article highlights the equation used, the derivative of the equation, a complete comparison of tanh with sigmoid and ReLU functions, strengths, weaknesses, weight initialization for tanh, implementation using Python, and its current applications in architecture as at 2026.

Tanh Function Formula and Key Properties

There are four mathematically equivalent definitions of the tanh function:

• Exponential form: tanh(x) = (eˣ – e⁻ˣ) / (eˣ + e⁻ˣ)
• Hyperbolic form: tanh(x) = sinh(x) / cosh(x)
• Sigmoid relationship: tanh(x) = 2 · sigmoid(2x) – 1
• LeCun variant: f(x) = 1.7159 · tanh(2x/3); a special version of tanh that was suggested by LeCun et al. (1998) for normalization purposes.

All of these yield the same result. But the sigmoid relationship is what actually makes the difference between tanh and sigmoid: tanh is a scaled and shifted sigmoid function, and this is why these functions have an S-shaped curve.

Property	Value
Output range	(-1, 1), open interval
Value at x = 0	0
Symmetry	Odd function: tanh(-x) = -tanh(x)
Domain	All real numbers
Differentiable	Yes, at every point
Monotonic	Yes, strictly increasing
Saturation points	Approaches 1 as x → +∞; approaches -1 as x → -∞

Being an odd function, tanh has symmetry around the origin. The sigmoid function does not possess this feature because the range of the sigmoid function is 0 to 1, and it is symmetric around the point (0, 0.5).

Derivative of the Tanh Function

The derivative of tanh is 1 – tanh²(x). It means that the derivative is calculated solely based on the function’s result and doesn’t require any extra calculations of the exponent. The derivative attains its maximum value 1.0 when x = 0 and tends to 0 as x → ±∞.

It plays an important role in only one case: sigmoid’s derivative has the maximum value 0.25. Thus, the maximum gradient of tanh is 4 times bigger than the maximum gradient of sigmoid for the same input.

The Practical effect: In the process of backpropagation, weight changes caused by tanh have a better signal around zero compared to weight changes caused by sigmoid function. Both functions saturate at extremes. For |x| greater than about 3, the derivative of tanh becomes less than 0.01, slowing down the weight updates based on gradients. This numerical limit forms the crux of the vanishing gradient problem first identified by Hochreiter (1991) and later formulated by Bengio et al. (1994).

Tanh vs. Sigmoid vs. ReLU vs. Softsign

Four properties separate these functions: output range, zero-centering, gradient strength, and computational cost.

Function	Output Range	Zero-Centered	Max Gradient	Computational Cost	Common Use Case
Tanh	(-1, 1)	Yes	1.0	High (exponential)	RNN/LSTM hidden states, GAN generator output
Sigmoid	(0, 1)	No	0.25	High (exponential)	Binary classification output layer
ReLU	[0, ∞)	No	1.0 (for x > 0)	Low (single comparison)	Hidden layers in CNNs and deep feedforward networks
Softsign	(-1, 1)	Yes	1.0	Medium (division, no exponential)	Alternative to tanh with slower saturation

Tanh is different from Sigmoid in that it is zero-centered. Tanh is different from ReLU because ReLU is unbounded and gives zero values only when the input is negative. Tanh is different from softsign because it approaches the asymptotes quicker due to its steep slope near zero, leading to more early training gradients at the expense of early saturation for large input values.

Advantages of the Tanh Activation Function

There are four measurable advantages to using tanh over sigmoid in the hidden layers:

• Zero-centered output. Activation functions will be centered around 0 instead of 0.5. This avoids introducing any systematic bias into the updates applied to weights, as pointed out by LeCun et al. (1998) as one of the main reasons why tanh converges faster than sigmoid.
• Stronger gradient near zero. The value of 1.0 compared to 0.25 leads to larger weight updates in the initial stages of learning, when most activations are still around zero.
• Smooth and differentiable everywhere. Unlike ReLU, which is not differentiable at x = 0, the function tanh is smooth and has its derivative defined for all inputs.
• Bounded output. The known range of (-1, 1) prevents activation from becoming infinitely large and therefore eliminates the need for clipping in the case of some networks.

Limitations of the Tanh Activation Function

Tanh has three documented limitations that restrict its use in modern deep architectures:

1. Vanishing gradients in deep networks. Once |x| becomes greater than about 3, the value of the derivative is less than 0.01. When multiplied during backpropagation in a network with multiple layers, these gradients tend to become infinitesimally small, hindering learning in early layers. The saturation problem was quantified by Glorot and Bengio (2010), who noted that in a five-layer architecture with standard initialization, the last hidden layer using sigmoid activation functions saturated near zero output within the first few training epochs.
2. No sparsity. Unlike the ReLU activation function, which gives an exact zero value for any negative input, tanh activation gives a non-zero value for any input other than x=0. All neurons are somewhat active, requiring more computations from the network on each pass.
3. Higher computational cost. Tanh requires computing two exponentials while ReLU needs to make just one comparison (max(0, x)). The tanh layer on the same hardware computes a little slower than the ReLU layer with the same number of neurons, and most modern GPU kernels are optimized for ReLU-type operations.

Hard Tanh: A Faster Variant of Tanh

The hard tanh is bounded between -1 and +1 with linear interpolation as opposed to an exponential function. The definition of the hard tanh function is provided below:

• Hardtanh(x) = -1, for x <-1
• Hardtanh(x) = x, for -1 ≤ x ≤ 1
• Hardtanh(x) = 1, for x > 1

Hard tanh does not involve any exponential computation and, hence, is computationally less expensive than regular tanh. However, the drawback with hard tanh is that it is not smooth. Hard tanh has a discontinuity point at x = -1 and x = 1 and saturates beyond this limit instantly, as opposed to regular tanh that slowly approaches its limits. Hard tanh is commonly seen when there is resource constraint while making an inference.

Where Is Tanh Used in Neural Networks?

The tanh function is used in four particular types of architectures, which are based on different mathematical principles.

Recurrent Networks (LSTM and GRU): LSTM, first introduced by Hochreiter and Schmidhuber (1997), utilizes tanh and sigmoid functions to perform two separate tasks within the same cell. Sigmoid gates regulate the passage of information by generating values from 0 to 1 that serve as filters. Tanh is responsible for generating the actual candidates and outputs, which range from -1 to 1. The TensorFlow version of GRU uses the same distinction where activation=’tanh’ for candidate values and recurrent_activation=’sigmoid’for gates.

Generative Adversarial Networks (GANs). In DCGAN, the architecture developed by Radford et al. (2015), tanh is the default activation function at the output layer of the generator. Training data is scaled between [-1, 1], which is also the output interval of tanh. As a result, the generator can output valid pixel values without any further transformation.

Shallow feedforward networks. In the case of shallow networks, vanishing gradients will not be an issue, and the zero-centering property of tanh is beneficial compared to that of the sigmoid.

Bounded regression outputs. Regression problem where outputs are limited between -1 and 1. Problems where the expected target value lies between -1 and 1, for example, in some reinforcement learning problems, do not require any extra steps after using the tanh activation function.

Weight Initialization for Tanh Networks

For neural nets that make use of tanh, the requirement is variance-preserving weight initialization. This is typically accomplished via the Xavier, also known as Glorot initialization.

The Xavier/Glorot initialization was developed to specifically address activation functions such as tanh. Tanh is an odd function, and its derivative is equal to 1 at x = 0.

Weights are initialized according to a distribution where Var(W) = 2 / (n_in + n_out). Here, n_in stands for the number of input connections to the current layer, while n_out refers to the number of output connections. In practice, this distribution is represented by a uniform distribution:

W ~ Uniform[−√(6 / (n_in + n_out)), √(6 / (n_in + n_out))].

Initialization helps preserve the variance of activations and gradients across all layers. Otherwise, tanh networks will tend to saturate too quickly because large weights would drive activations into saturation right away due to the small value of their gradient, while small weights would keep the network away from saturation and its nonlinear capabilities. The initialization is used in ReLU networks and is inappropriate for tanh layers.

How to Implement the Tanh Function in Python

Tanh is available as a built-in function in NumPy, PyTorch, and TensorFlow. All three return identical numeric results for the same input.

NumPy (from the mathematical definition):

import numpy as np

def tanh(x):
    return (np.exp(x) - np.exp(-x)) / (np.exp(x) + np.exp(-x))

x = np.array([-2.0, -1.0, 0.0, 1.0, 2.0])
print(tanh(x))
# Output: [-0.964 -0.762  0.     0.762  0.964]

# NumPy also provides a direct, faster implementation:
print(np.tanh(x))
PyTorch:
import torch
import torch.nn as nn

# As a layer inside a network
model = nn.Sequential(
    nn.Linear(10, 20),
    nn.Tanh(),
    nn.Linear(20, 1)
)

# Applied directly to a tensor
x = torch.tensor([-2.0, -1.0, 0.0, 1.0, 2.0])
output = torch.tanh(x)
print(output)
# Output: tensor([-0.9640, -0.7616, 0.0000, 0.7616, 0.9640])

TensorFlow / Keras:

import tensorflow as tf

# As a layer inside a network
model = tf.keras.Sequential([
    tf.keras.layers.Dense(20, activation='tanh', input_shape=(10,)),
    tf.keras.layers.Dense(1)
])

# Applied directly to a tensor
x = tf.constant([-2.0, -1.0, 0.0, 1.0, 2.0])
output = tf.math.tanh(x)
print(output)
# Output: [-0.9640276 -0.7615942  0.         0.7615942  0.9640276]

All three outputs match to four decimal places, since each framework computes the same mathematical function.

When to Use Tanh Instead of ReLU or Sigmoid

Condition	Recommended Function	Reason
Building an LSTM or GRU hidden layer	Tanh	Matches the architecture’s default design and keeps cell-state values bounded
Building a GAN generator output layer	Tanh	Matches the [-1, 1] normalization standard for image data
Training a network deeper than roughly 10 layers	ReLU or a ReLU variant	Avoids the vanishing gradient problem that affects tanh at scale
Building a binary classification output layer	Sigmoid	Produces a direct probability output between 0 and 1
Working with already zero-centered, normalized input data	Tanh	Preserves the zero-centering through the hidden layers
Optimizing for inference speed on limited hardware	ReLU or hard tanh	Avoids the cost of computing exponential terms

Is Tanh Still Used in 2026?

Indeed, tanh is currently in use but is not the default activation anymore in architectures that are dominating current research. In 2022, a survey was conducted on 20 language models released after BERT. It was found out that 80% of those models use GELU as their default activation while no model uses tanh (arXiv:2208.14111).

Tanh continues to affect the new activities. GELU, proposed by Hendrycks and Gimpel in 2016, is commonly calculated as tanh approximation: GELU(x) ≈ 0.5x · (1 + tanh(√(2/π) · (x + 0.044715x³))). Starting from PyTorch version 1.12, tanh approximation is used as the default implementation of GELU. Moreover, Mish, created by Misra (2019), is formulated directly in terms of tanh: f(x) = x · tanh(softplus(x)).

Tanh continues to be the default activation within LSTM and GRU cells. These types of cells can be encountered in time series predictions, speech processing, and many other tasks where a full-fledged transformer architecture is not needed. Tanh is the default output activation for GAN generators due to the convention of normalizing pixels between -1 and 1 made by Radford et al. in 2015.

Frequently Asked Questions