CNN Architecture Explained

Pass an input 224×224 colored image through a simple neural network with three tiny hidden layers, and you will need more than 300 billion trainable parameters to make your network learn something. No typo and no theoretic estimation, it is just impossible to create such a network because of all the interconnections between pixels and neurons from the next layer. That is why the architecture of the convolutional neural network (CNN) was created.

It is definitely not another tutorial on how to build “convolutions, pooling, fully-connected, done!” This one not only explains the purpose of each layer but provides you with architectures, number of parameters in each of them (LeNet-5, AlexNet, VGG-16, ResNet), and code written in TensorFlow and PyTorch. And at the end of the series, you will get not only the idea of what each layer does but also will know why it should be so.

What Is CNN Architecture? (And Why It Beats a Regular Neural Network)

CNN Architecture is the design of the layers, which includes convolutional, pooling, and dense, allowing the neural network to learn visual patterns directly from the pixels rather than interpreting each pixel individually and separately as a separate numeric value.

However, this difference is more significant than it appears. A regular multilayer perceptron (MLP) model reduces the image into a single row of pixels and connects each of them to each neuron in the first hidden layer. In the case of a relatively small 224x224x3 color image, there will be 150,528 inputs. Connecting them to only one hidden layer with 128 neurons results in creating over 19 million weights. Building three similar layers brings the total number of parameters to more than 300 billion for the network that hasn’t seen even a single image.

This problem is circumvented in CNNs using the concept of weight sharing. Rather than having a different weight for each pixel-neuron connection, CNNs have small windows of size 3×3 or 5×5 sliding throughout the image, where each window will have the same set of weights used everywhere. This means that the window which detects a vertical edge at the top left corner of an image will use the exact same weights for detecting the vertical edge anywhere else in the image. It is also what makes CNNs translation invariant.

What this means is that CNNs which have been optimized in this manner will be able to achieve great results when running on images of size 224×224×3 while only requiring just a few convolutional layers with a total parameter count of a couple of hundred thousand, not 300 billion.

The Core Layers of CNN Architecture

A typical CNN learns features from its data through six to seven different layers, where each layer modifies the data in some way before sending it on to the next layer. Here’s exactly what happens at each of those layers, and why they happen there.

Input Layer

The input layer is just that – the entry point. The input layer stores the pixel data in three dimensional form (width * height * depth). If we consider an image with colors (RGB form), the depth will be 3 (red, green, blue). A 32×32 colored image will thus be stored as 32x32x3. No learning occurs here, only storage.

Convolutional Layer

Here’s where actual feature extraction occurs. The filter, which is just a matrix of learnable parameters, is slid across the input data, and at each position, the element-by-element multiplication of this matrix and the pixels in the current position is done, and the sum of this result is produced. Thus the feature map is created.

In the early stages of a neural network, these filters learn to detect relatively simple features like edges, patches of colors, or other basic textures. You do not code this functionality. Instead, through training, the network learns to discover edges as the most useful low-level components of vision-related tasks.

Size of the output? There is an actual formula for it. And strangely enough, it’s almost never mentioned in guides:

Where n is the input size, f is the filter (kernel) size, p is padding, and s is stride. Take a 32×32 input, a 3×3 filter, padding of 1, and a stride of 1:

The spatial size doesn’t shrink. That’s “same” padding doing its job — preserving dimensions so you don’t lose information at the edges with every layer.

Activation Layer

Without this, a CNN becomes nothing more than a linear layer on top of another linear layer and, as you can guess, combining a bunch of linear layers amounts to doing a single linear transformation. This prevents learning any form of complexity. But, that is where the activation function comes in handy.

The de facto standard in all places is the ReLU (short for Rectified Linear Unit), which leaves positive numbers alone and sets negative numbers to zero. It is computationally inexpensive and quick to train. While the sigmoid and Tanh functions do have their places, especially for output layers, the popularity of ReLU comes from its freedom from vanishing gradient issues unlike Sigmoid.

Pooling Layer

The pooling technique reduces the spatial dimensions of the feature map while preserving only the important signals. In the case of maximum pooling, which is by far more popular than average pooling, a small window (say 2×2) is considered, and the maximum value of the signal is preserved. Maximum pooling ignores all the other information in that particular window.

The reason behind the ability of max pooling technique to produce translation invariance, is as follows: in case of any change in the position of the detected edges in a slight movement of the object from where the picture was taken, the signal remains the same inside the maximum pooling window and thus its location becomes irrelevant.

A 2×2 max pooling with stride 2 reduces the 32x32x12 volume to a 16x16x12 volume. Depth of the output (number of feature maps) is unaffected.

Flattening

There’s only one purpose for this layer, and that is to transform the output of the last 3D feature map into a 1D vector. The 16x16x12 cube transforms into a vector containing 3,072 elements (16 x 16 x 12). Again, nothing is being learned in this layer; it’s just a repackaging of the data.

Fully Connected (Dense) Layer

This is when the network begins moving from the question “What patterns exist?” to the question “What is the meaning of these patterns?” Each of the values in the flattened vector interacts with all neurons in this layer, and the network fuses low-level features such as edges or textures with high-level features like wheels, eyes, or faces. It’s exactly the math of the densely connected layer that MLPs use, just with a much more meaningful input.

Output Layer

This last layer transforms the raw scores into something more meaningful, which is a probability. In case of a multi-class classification problem (is this a cat, dog, or bird?), the Softmax converts the scores such that they add up to 1, and the largest score becomes your classification. In case of a binary classification problem (spam or not?), you get just one probability score.

Section takeaway: every layer in this pipeline exists to solve a specific problem — feature extraction, non-linearity, dimensionality reduction, or decision-making — and removing any one of them breaks something specific, not just “performance” in the abstract.

How Many Parameters Does a CNN Layer Actually Have?

Most explanations stop at “filters have weights.” Here’s the actual count, because it changes how you think about model size.

Parameters per convolutional layer = (kernel_height × kernel_width × input_channels + 1) × number_of_filters

The “+1” is the bias term — every filter gets exactly one. Let’s say you have a 3×3 filter applied to an RGB input (3 channels), and you’re using 16 filters:

Params = (3 × 3 × 3 + 1) × 16 = (27 + 1) × 16 = 448

Compare that to a fully connected layer trying to process the same 32×32×3 image directly: 3,072 inputs connected to even a modest 64 neurons is 196,608 weights, before bias terms. That’s the parameter-sharing payoff made concrete — not an abstract claim, an actual 400x-plus difference for this single layer.

Layer Type	Input Shape	Filters/Neurons	Trainable Parameters
Conv (3×3)	32×32×3	16	448
Conv (3×3)	16×16×16	32	4,640
Dense	3,072 (flattened)	64	196,672

What do you notice? The dense layer here contains far more parameters than the two convolutional layers together – although the convolutional layers were responsible for most of the feature extraction! This is precisely why CNN network designs always make their convolutional blocks deep and dense layers shallow.

A Real CNN Architecture, Layer by Layer: VGG-16

Toy models can help understand the mechanics, but having a look at an actual production-ready architecture provides another type of insight. The VGG-16 architecture, which was proposed by the Visual Geometry Group from Oxford in 2014, is an interesting architecture to look at as it has a very straightforward structure – consisting only of 3×3 convolutions and 2×2 max pooling layers.

This network is given an RGB image input with the size of 224x224x3, which goes through five convolutional blocks called Conv-1, Conv-2, Conv-3, Conv-4, and Conv-5, where every block contains two or three convolutional layers with the pooling layer that follows. Spatial size of data decreases at each pooling operation, while depth (the number of channels) increases. When data is passed through the Conv-5 layer, its volume changes from 224x224x3 to 7x7x512: the former size is smaller, while the latter contains much more complicated information.

From this point, the 7x7x512 volume is reshaped to become a vector consisting of 25,088 elements, which is further used as an input for two fully connected layers containing 4,096 neurons and then for the output layer, which consists of 1,000 neurons (one per each class from the ImageNet).

But the VGG-16 network can hardly be considered the most efficient architecture in the modern context – with 138 million parameters, it is relatively big for its time. However, its homogeneity and scalability were the reasons why it became a model of educational practice and transfer learning.

How CNN Architecture Evolved: From LeNet-5 to Today

This is the portion which most guides omit completely, but it is precisely this portion which shows why CNNs have their particular design. Every single one of these architectures did not undergo a redesign for no particular reason; rather, each redesign came from hitting a particular bottleneck.

LeNet-5 (1998) — LeNet-5, a network designed for handwriting recognition on bank checks and developed by Yann LeCun, is an example of the convolution-pooling-fully connected network design, which forms the backbone of everything covered in this article. LeNet-5 was small compared to today’s networks; it had only a few layers and could work on 32×32 pixel black-and-white images.

AlexNet (2012) — This is the model that ignited the machine learning revolution in computer vision applications. Using the GPU to train on the ImageNet database, AlexNet delivered an impressive 57% top-1 accuracy rate on a 1,000-category classification task, which was an unprecedented leap compared to previous models. The groundbreaking idea of AlexNet had little to do with inventing a different layer structure and everything to do with its ability to harness depth and scale.

VGG-16 (2014) — Covered above. Its contribution was showing that stacking small 3×3 filters repeatedly could match or beat larger filters, while staying simpler to reason about.

ResNet (2015) — With more layers, learning became difficult because of vanishing gradients when backpropagation was used – not due to overfitting. Skip connections were invented by ResNet in order to solve this problem – with skip connection, it is possible for a layer to send its output forward to several layers ahead and be added to the input layer. It makes it possible for neural networks to exceed 100 layers in depth; for example, ResNet-50 has about 77% accuracy on ImageNet and 26 million parameters.

EfficientNet (2019) — Unlike scaling only one of the dimensions, i.e., depth, width, and resolution (which is how earlier network architectures scaled), EfficientNet scales all three at once using a compound ratio that is kept constant. EfficientNet-B0 supposedly achieves the same performance as ResNet-50 but uses only one-fifth the number of parameters.

Vision Transformers (2020 onward) — ViTs divide images into patches and employ attention mechanisms without ever using convolutions. Is the CNN architecture outdated by now with that in mind? Not at all — in fact, CNNs are the more efficient approach in terms of both computations and data requirements, particularly when datasets are small, compared to ViTs that require large amounts of pretraining data or the combination of CNNs and attention mechanisms to succeed.

I feel that the conventional narrative of this history is incorrect. Conventional narratives tend to see the models as a set of famous architectures ordered in their accuracy. The correct perspective would be that each model is an established solution to a particular problem: vanishing gradients, over-parametrization, scaling. One needs to know the problem to comprehend the solution.

CNN Architecture Variations You’ll Actually Encounter

Beyond the named landmark models, a handful of structural variations show up constantly in modern CNN design. Here’s the practical comparison.

Variation	Key Innovation	Best Use Case	Example
Deep CNNs	More stacked conv + pooling layers	Large-scale image classification	VGG-16
Residual Networks (ResNet)	Skip connections solve vanishing gradients	Very deep networks (100+ layers)	ResNet-50/101/152
Dilated Convolutions	Gaps inserted into the kernel widen the receptive field without more parameters	Segmentation, depth estimation	DeepLab-style models
Depthwise Separable Convolutions	Splits convolution into depthwise + pointwise steps, cutting compute	Mobile and edge devices	MobileNet

The concept of dilated convolutions merits an equally brief discussion since they are the most non-intuitive of the four concepts. For a regular 3×3 filter, the area scanned would be 9 pixels. However, in the case of a dilated 3×3 filter with a dilation of 2, the same number of 9 pixels will be spread across a 5×5 area.

What Affects CNN Architecture Performance

Five design choices shape how well — and how fast — a CNN architecture actually performs.

Number of filters. However, with an increase in the number of filters, there is also an increase in computational costs. The early layers in most network architectures employ a lower number of filters (32-64), while later layers have more (256-512).

Kernel size. The small kernels (3×3) provide finer detail while keeping the number of parameters small. The larger kernels (5×5, 7×7) gain context at each step but increase the computational cost. This is partly the reason that VGG-16’s 3×3 convolution architecture became popular.

Model depth. Deeper networks build richer hierarchies of features — but depth without a fix like skip connections runs straight into the vanishing-gradient wall ResNet was built to solve.

Receptive field. This is the one that has very few names of beginner’s level guides. The receptive field is defined as the portion of the original image from where the neurons at a certain layer are really looking at. It increases with the increase in layers and dilation. It is important because if the size of the receptive field of a neuron is less then the neuron cannot perceive the pattern larger than the size of its receptive field; no matter how much you train the network.

Regularization. Dropout randomly drops some proportion of neurons during training, thus making sure that the network does not get overly dependent on any one particular route. Similarly, weight decay punishes high-weight values directly. The rationale behind both of these techniques is the same — namely, an unrestrained convolutional neural network with millions of parameters will be delighted to remember your training data.

Build a Tiny CNN From Scratch (Code Walkthrough)

However, theory alone won’t get you very far. Below is an example of a basic pipeline that takes in an image and performs convolution, activation, pooling, flattening, and dense steps, depicted both using TensorFlow and PyTorch because many of you will be using one of the two.

TensorFlow:

import tensorflow as tf

# Load and preprocess

image = tf.io.read_file("image.jpg")

image = tf.io.decode_jpeg(image, channels=1)

image = tf.image.resize(image, [300, 300])

image = tf.image.convert_image_dtype(image, tf.float32)

image = tf.expand_dims(image, axis=0)  # add batch dimension

# Define a simple edge-detection kernel

kernel = tf.constant([[-1,-1,-1],[-1,8,-1],[-1,-1,-1]], dtype=tf.float32)

kernel = tf.reshape(kernel, [3, 3, 1, 1])

# Convolution -> ReLU -> Max Pooling -> Flatten -> Dense

conv_output = tf.nn.conv2d(image, kernel, strides=[1,1,1,1], padding='SAME')

relu_output = tf.nn.relu(conv_output)

pool_output = tf.nn.max_pool2d(relu_output, ksize=[1,2,2,1], strides=[1,2,2,1], padding='SAME')

flatten_output = tf.keras.layers.Flatten()(pool_output)

dense_output = tf.keras.layers.Dense(units=64, activation='relu')(flatten_output)

print("Final shape:", dense_output.shape)  # (1, 64)

PyTorch equivalent:

import torch

import torch.nn.functional as F

from torchvision import transforms

from PIL import Image

# Load and preprocess

transform = transforms.Compose([transforms.Resize((300, 300)), transforms.ToTensor()])

image = transform(Image.open("image.jpg").convert("L")).unsqueeze(0)  # add batch dim

# Define the same edge-detection kernel

kernel = torch.tensor([[-1.,-1.,-1.],[-1.,8.,-1.],[-1.,-1.,-1.]]).view(1, 1, 3, 3)

# Convolution -> ReLU -> Max Pooling -> Flatten -> Dense

conv_output = F.conv2d(image, kernel, padding=1)

relu_output = F.relu(conv_output)

pool_output = F.max_pool2d(relu_output, kernel_size=2, stride=2)

flatten_output = pool_output.view(pool_output.size(0), -1)

dense = torch.nn.Linear(flatten_output.shape[1], 64)

dense_output = F.relu(dense(flatten_output))

print("Final shape:", dense_output.shape)  # torch.Size([1, 64])

It’s the same operations and same rules, but there are two different systems. Structure stays the same, syntax varies. And that’s something to take to heart: once you understand the structure, changing frameworks becomes just a matter of syntax.

Real-World Applications of CNN Architecture

Application	How CNN Architecture Is Used
Image classification	Assigns a single label (cat, dog, tumor-present) to an entire image
Object detection	Locates and labels multiple objects within one image, with bounding boxes
Medical imaging	Flags tumors, fractures, or abnormalities in X-rays, MRIs, and CT scans
Face recognition	Verifies or identifies faces for security and authentication systems
Autonomous systems	Reads lanes, signs, pedestrians, and obstacles for real-time navigation
OCR / text extraction	Converts handwritten or printed text into digital, searchable text
Quality inspection	Spots manufacturing defects by detecting texture or shape irregularities on a production line

Medical imaging deserves one additional mention, since the stakes in that area are quite different from, for instance, those in identifying cat images. A mistake in interpreting a tumor scan is not simply an error metric; it’s a missed diagnosis. This is one reason why the receptive fields and depths chosen in medical imaging CNNs are very different from those used in consumer applications.

Common Mistakes When Designing CNN Architecture

Most guides describe what layers do. Fewer explain what goes wrong when you size them badly. A few patterns show up again and again.

Going deep without skip connections. Adding layers on top of one another without tackling the vanishing gradient problem is the most prevalent error in custom-made models. After 20 layers, straight stackings of convolutional layers begin to underperform rather than outperform.

Ignoring the receptive field for the task. The detection of tiny objects like a screw on a circuit board and the detection of larger objects like a car in a street scene require receptive fields of varying sizes. If both have the same depth of architecture and kernel size, then at least one of the two will not perform well.

Mismatching kernel size to feature scale. The big kernel usually does not do as well as two or three smaller kernels that are stacked and cover the same receptive field area, such as the entire design strategy of VGG-16 network, because when stacking occurs, there is more non-linearity between them.

Skipping regularization on small datasets. Any CNN with just a few million parameters trained on thousands of images without any dropout and augmentation will memorize the training set in a matter of only a few epochs. The solution is not a better architecture but dropout, weight decay, or simply additional data.

Frequently Asked Questions

Where This Leaves You

And that 300 billion parameters figure from earlier? It wasn’t an interesting aside – it was the challenge each and every layer of CNN architecture had to address. Parameter sharing addressed the scale issue. Pooling addressed the invariance issue. Skip connections addressed the depth issue. Compound scaling addressed the efficiency issue.

Architecture changes. Vision Transformers, hybrid architectures, whatever follows in the coming years – all of it will be judged on the same basis that led to CNNs in the first place: can it learn from pixels while not requiring more parameters than the problem requires. Understand the challenge, and everything else you’ll ever hear about architecture in your life makes sense in thirty seconds.