How I Built an OCR Under 2 MB (Without Any Prior Experience)

Introduction

A few weeks ago, I stumbled upon a WhatsApp post informing me about my university hosting a hackathon.

After reviewing all the problem statements, I decided against participating due to their highly technical, AI/ML-focused nature. I was learning and considered myself a beginner, so I didn’t expect to reliably solve any of the problems.

With a few days to go before the first round began (Which was just a PPT submission), I still registered individually and forgot about it.

A few days after that, I got approached by some of my friends to include them in my team. A bit hesitant at first, I decided to at least research and submit a PPT, thinking it wouldn’t get selected, considering seniors and people from other colleges were also participating. A mere first-year student surely didn’t stand a chance of even getting selected.
When the results came out, we were one of the 17 teams that got shortlisted for the final round.

With virtually no choice now, and about 3-4 days before finals, I began.

Selecting The Problem Statement

There were 4 problem statements, all quite technical. Due to most problem statements needing the collection of data, I picked the one that I had some experience in: PS4: (OCR Under 10 MB).

I had previously worked on an OCR project, although it was developed using OpenCV, was extremely simple, utilized APIs, and was definitely not under 10 MB.

The Challenge

Here were the conditions of the problem statement:

• Extract text from images or scanned documents.
• Must handle low-resolution, noisy images
• Encourage algorithmic preprocessing (e.g., edge detection, thresholding) rather than heavy-weight ML models.

With the following constraints:

• Model size: All model weights ≤ 10 MB.
• No compression hacks: Quantization, pruning, or external weight manipulation is not allowed.
• Offline: Fully offline inference. No internet.
• Hardware target: Low-end devices (2 CPU cores, 2 GB RAM). Inference latency target: ≤ 2 seconds per input.

Additionally, I treated it as a learning opportunity rather than a competition. So, I made a bet with a friend not to use AI for code; I could use it for learning, and preferably to point me to resources, but not to directly give me the code. I would have to treat this friend if I lost the bet.

Researching

It was clear from the problem statement that I had to use some pre-processing and then parse it through a model.

After using Gemini for research and browsing other resources, I devised a plan.

The Approach

As mentioned above, I used a two-step approach:

1. An OpenCV pipeline that would pre-process the image, separate the characters, and parse them through the model.
2. Using a CNN model to read the characters.

Confident that I can do the first step, considering I had previously done some mini projects using OpenCV in the past, I began with the CNN model.

Creating the model

Attempt 1: CNN Model

I basically had zero experience in creating a neural network from scratch (except for one beginner tutorial I saw a few years ago on the MNIST dataset using TensorFlow). The goal was to first build an MVP that barely worked and then reiterate it to get it under 10 MB.

I used the following resources to learn:

For learning about CNNs:

CNN Introdcution By geeksforgeeks

For coding the CNN model:

Pytorch documentation

The goal for this portion is to build a CNN that detects one character at a time.

After a bit of research, I decided to stick with the EMNIST dataset. It has all the English letters and numbers.

First, I made the imports and loaded the dataset:

import numpy as np
import pandas as pd
from sklearn.metrics import confusion_matrix
import matplotlib.pyplot as plt
%matplotlib inline

import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.utils.data import DataLoader
from torchvision import datasets, transforms
from torchvision.utils import make_grid

device = "cuda" if torch.cuda.is_available() else "cpu"

transform = transforms.ToTensor()

training_data = datasets.EMNIST(root='/data', split='balanced', train=True, download=True, transform=transform)

Then I split it into training and testing data (Could add testing_data declaration perhaps):

transform = v2.Compose([v2.RandomHorizontalFlip(p=1.0),
                        v2.RandomRotation(degrees=(90, 90))]) #Necessary for EMNIST

def show_images(data, main_title=None):
    plt.figure(figsize=(10, 5))
    plt.suptitle(t=main_title, y=1.0, fontsize=14)
    for i, (im, lab) in zip(range(1, 11), data):
        plt.subplot(2, 5, i)
        plt.imshow(X=im)
        plt.title(label=lab)
    plt.tight_layout()
    plt.show()
show_images(training_data, "Data")

class CNN(nn.Module) :
    def _init_(self) -> None:
        super() ._ init_()
        self.conv1: nn.Conv2d(1,6,3,1)
        self.conv2: nn. Conv2d(6,16,3,1)

        self.fc1: nn.Linear(5*5*16, 120) #All arbitrary here, just end the last layer in number of classes
        self.fc2: nn.Linear(120, 84)
        self.fc3: nn.Linear(84, 47)

    def forward(self, X) -> Any:
        X: F.relu(self.conv1(x))
        X: F.max_pool2d(X,2,2) # 2x2 kernel, stride of 2
    #Layer 2
        X: F.relu(self.conv2(x))
        X: F.max_poo12d(X,2,2)

        X: X.view(-1, 16*5*5) #Not sure, for now

        X: F.relu(self.fc1(X))
        X: F.relu(self.fc2(X))
        X: self.fc3(x)

    return F.log_softmax(X, dim=1)

model: CNN = CNN()
model

CNN(
  (conv1): Conv2d(1, 6, kernel_size=(3, 3), stride=(1, 1))
  (conv2): Conv2d(6, 16, kernel_size=(3, 3), stride=(1, 1))
  (fc1): Linear(in_features=400, out_features=120, bias=True)
  (fc2): Linear(in_features=120, out_features=84, bias=True)
  (fc3): Linear(in_features=84, out_features=47, bias=True)
)

import time
start_time: float = time.time()

epochs = 5
train_losses: list[Any] = []
test_losses: list[Any] = []
train_correct: list[Any] = []
test_correct: list[Any] = []

for i in range(epochs):
    trn_corr = 0
    tst_corr = 0

    for batch, (X_train, y_train) in enumerate[Any](train_loader):
        batch+=1
        y_pred: Any = model(X_train) #Maybe experiment with flat?
        loss: Any = criterion(y_pred, y_train)
        predicted: Any = torch.max(y_pred.data, 1)[1] #Add up number of predictions, the first item each
        batch_corr: Any = (predicted == y_train).sum() #Correct predictions in a batch
        trn_corr += batch_corr

        optimizer.zero_grad()
        loss.backward()
        optimizer.step() #A lot can be changed and optimized here and in layers

        if batch%600 == 0:
            print(f'Epoch: {i} Batch: {batch} Loss: {loss.item()}')

    train_losses.append(loss)
    train_correct.append(trn_corr)

    #Testing
    with torch.no_grad():
        for batch, (X_test, y_test) in enumerate[Any](test_loader):
            y_val: Any = model(X_test)
            ts_corr = 0
            predicted: Any = torch.max(y_val.data, 1)[1]
            ts_corr += (predicted == y_test).sum()

    loss: Any = criterion(y_val, y_test)
    test_losses.append(loss)
    test_correct.append(tst_corr)

current_time: float = time.time()
total_time: float = current_time-start_time
print(f'Total time: {total_time/60} minutes')

Final Code For Custom CNN

This was my first time creating a model from scratch. I learnt about the architecture of CNN models after going through a bunch of resources:

1. Deep Learning With PyTorch - YouTube (One of the best resources, I watched most of the playlist and highly recommend it for beginners.)
2. Example Implementation of CNN on EMNIST: How to train Neural Network from Scratch on the EMNIST Dataset | by Anuj malviya | Medium

Here is the model:

class CNN(nn.Module):
    def __init__(self) -> None:
        super().__init__()
        self.conv1: Any = nn.Conv2d(1,6,3,1)
        self.conv2: Any = nn.Conv2d(6,16,3,1)

        self.fc1: Any = nn.Linear(5*5*16, 120) #All arbitrary here, just end the last layer in number of classes
        self.fc2: Any = nn.Linear(120, 84)
        self.fc3: Any = nn.Linear(84, 47)

    def forward(self, X) -> Any:
        X: Any = F.relu(self.conv1(X))
        X: Any = F.max_pool2d(X,2,2) # 2x2 kernel, stride of 2
        #Layer 2
        X: Any = F.relu(self.conv2(X))
        X: Any = F.max_pool2d(X,2,2)

        X: Any = X.view(-1, 16*5*5) #Not sure, for now

        X: Any = F.relu(self.fc1(X))
        X: Any = F.relu(self.fc2(X))
        X: Any = self.fc3(X)

        return F.log_softmax(X, dim=1)

Training & Testing

First of all, I realised that, unlike the MNIST dataset, the EMNIST dataset is actually rotated 90 degrees.

I applied the following transformations to feed the data into the model:

transform = transforms.Compose([
    transforms.Resize((28, 28)),
    transforms.ToTensor(),
    transforms.Lambda(lambda x: x.transpose(1, 2)), # 
    transforms.Normalize((0.5,), (0.5,))
])

train_dataset = datasets.EMNIST(root='./data', split='balanced', train=True, download=True, transform=transform)
train_loader = DataLoader(train_dataset, batch_size=BATCH_SIZE, shuffle=True)

• Resize the dataset into the standard (28,28)
• Converts the PIL images to tensor
• Transposes the matrix to flip it 90 degree
• Normalize it

I kept encountering errors and inconsistent results during training.

I trained the model on Colab’s free GPU.

The whole process from research, learning, building the model, and then training took me about a weekend (This was pre-hackathon), and I did not have much time left as I had classes the next day.

Some resources I used in this step:

1. Pytorch’s official documentation on Transformations.
2. Pytorch’s official documentation on Transforming Images
3. Pytorch’s official EMNIST documentation.
4. Implementation #1 of EMNIST (Numbers-only)
5. Implementation #2
6. Small guide on MNIST

Testing

I was getting mixed results in training, which transferred to testing.

Failing

After a lot of trying, testing, and failing, I gave up on this approach. The best and worst part of a Hackathon is the lack of time.

Attempt 2: Using transfer learning on a pre-trained model

Picking the best model

After giving up on creating my own model, I asked for advice from a senior in college, and he recommended me to use transfer learning on pre-existing models.

I looked into research papers to try to figure out the best possible model for this scenario. This is, in my opinion, one of the best use cases of modern LLMs; they are good at finding research papers and summarising them.

After a lot of research, I had a few options:

• LeNet-5,
• MobileNetV3
• ShuffleNetV2

I rejected MobileNetV3; I don’t recall the exact reason, but it may have been because it exceeded 10 MB. Eventually, I went with ShuffleNetV2, which surprisingly also detects more than a thousand objects and was 5-6 MB.

Transfer Learning

I learnt about Transfer Learning, turns out, this was actually really common, and people rarely create a model from scratch in the real world.

I found this: EMNIST CNN Implementation On Github

But it seemed to be using MNIST instead of the full EMNIST dataset, so I left this.

model = models.shufflenet_v2_x0_5(weights='DEFAULT')

Changing the number of classes

The first step was changing the dimensions of the input layers from 3 to 1. Since the original model took the colored image, however, EMNIST is a greyscale dataset.

Then, we change the last layer to reduce the number of classes from 1000+ to just 47.

Un-freezing and re-training

After trying for probably an hour, I gave up and used AI for the final code (After trying to troubleshoot it first with AI, ofcourse).
Thus, officially breaking my AI-vow and losing one part of the challenge.

#Unfreeze everything
print("Unfreezing all layers...")
for param in model.parameters():
    param.requires_grad = True

# Use a much smaller learning rate (0.0001) so we don't destroy the pre-trained weights
optimizer = optim.Adam(model.parameters(), lr=0.0001)

print("Starting 15 epochs of deep training...")
for epoch in range(20):
    model.train()
    running_loss, correct, total = 0.0, 0, 0

    for images, labels in train_loader:
        images, labels = images.to(device), labels.to(device)

        optimizer.zero_grad()
        outputs = model(images)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

        # Accuracy check
        _, predicted = torch.max(outputs.data, 1)
        total += labels.size(0)
        correct += (predicted == labels).sum().item()
        running_loss += loss.item()

    # Log progress
    print(f"Fine-Tune Epoch [{epoch+1}/15] | Loss: {running_loss/len(train_loader):.4f} | Acc: {100*correct/total:.2f}%")

torch.save(model.state_dict(), "model_final.bin")
print("Done! Final high-accuracy model saved as shufflenet_emnist_final.bin")

Results

Unfreezing all layers…

Starting 15 epochs of deep training…

Fine-Tune Epoch [1/15] | Loss: 2.7147 | Acc: 35.60%

Fine-Tune Epoch [2/15] | Loss: 1.1628 | Acc: 66.16%

Fine-Tune Epoch [3/15] | Loss: 0.7747 | Acc: 75.15%

Fine-Tune Epoch [4/15] | Loss: 0.6407 | Acc: 78.85%

Fine-Tune Epoch [5/15] | Loss: 0.5685 | Acc: 80.82%

Fine-Tune Epoch [6/15] | Loss: 0.5215 | Acc: 82.20%

Fine-Tune Epoch [7/15] | Loss: 0.4889 | Acc: 83.13%

Fine-Tune Epoch [8/15] | Loss: 0.4602 | Acc: 83.99%

Fine-Tune Epoch [9/15] | Loss: 0.4381 | Acc: 84.61%

Fine-Tune Epoch [10/15] | Loss: 0.4194 | Acc: 85.24%

Fine-Tune Epoch [11/15] | Loss: 0.4013 | Acc: 85.74%

Fine-Tune Epoch [12/15] | Loss: 0.3872 | Acc: 86.03%

Fine-Tune Epoch [13/15] | Loss: 0.3746 | Acc: 86.58%

Fine-Tune Epoch [14/15] | Loss: 0.3653 | Acc: 86.76%

Fine-Tune Epoch [15/15] | Loss: 0.3526 | Acc: 87.20%

Fine-Tune Epoch [16/15] | Loss: 0.3438 | Acc: 87.46%

Fine-Tune Epoch [17/15] | Loss: 0.3344 | Acc: 87.69%

Fine-Tune Epoch [18/15] | Loss: 0.3271 | Acc: 87.97%

Fine-Tune Epoch [19/15] | Loss: 0.3188 | Acc: 88.06%

Fine-Tune Epoch [20/15] | Loss: 0.3107 | Acc: 88.41%

Done! Final high-accuracy model saved as shufflenet_emnist_final.bin

As a result of this method, our model got reduced to 1.62 MB! Not only that, there were absolutely no shortcuts like quantization or pruning involved. (It would be interesting to see how much further we can squeeze it.)

I also got a satisfactory accuracy of 88.41%; Perhaps I could have improved it with more epochs or better techniques, but I doubted the model itself would be a bottleneck in the future.

What could have been a bottleneck, however, is:

Creating the CV Pipeline

While this part sounds easy initially, it is actually the more difficult problem of this challenge, given the constraints.

After reading and watching a bunch of videos on this, these are the steps that I applied to preprocess and separate the letters purely from openCV and then parse them to my program:

def preprocess(img_path):
    img = cv2.imread(img_path) #ex2.jpg
    img_color = cv2.imread(img_path)
    img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
    img = reduce_check(img)
    img = thresholding(img, 'otsu')
    img = cv2.bitwise_not(img) #Later, have an option to check this, if the background is black already
    character_dict, img_boxes = get_characters_v2(img, img_color)
    return character_dict, img_boxes

First was this small function called reduce_check:

def reduce_check(img):
    h, w, c = img.shape
    if w > 1000:
        new_w = 1000
        ar = w/h
        new_h = int(new_w/ar)
        img = cv2.resize(img, (new_w, new_h), interpolation = cv2.INTER_AREA)
    return img

It simply reduces the size if the width is greater than 1000 pixels. This was an arbitrary value, but I saw something similar implemented many times, so I decided to implement it as well.

The next step is thresholding,

def thresholding(image, method='gaussian'):
    img_gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY) #No need twice, slight gaussian blur first???

    #Otsu's works better than Adaptive in some of my examples!(?)

    blur = cv2.GaussianBlur(img_gray,(5,5),0)
    if method=='otsu':
        ret3,thresh = cv2.threshold(blur,0,255,cv2.THRESH_BINARY+cv2.THRESH_OTSU)
    elif method=='gaussian':
        thresh = cv2.adaptiveThreshold(blur,255,cv2.ADAPTIVE_THRESH_GAUSSIAN_C,\
            cv2.THRESH_BINARY,11,2)
    return thresh

Thresholding is basically converting an image to grayscale, so every pixel is either 0 or 1.

2 methods seemed to work best for me:
Otsu and Gaussian, though one worked better than the other for different cases.

Finally, there was the main function, which took the processed image and segmented the characters. I spent at least 3-4 hours on this, watching different videos, browsing GitHub, trying different things, but nothing worked consistently. I did have to use AI eventually during the hackathon to meet the deadline (More on that later).

Some awesome resources that helped me in this step:

• Official OpenCV documentation for Contour Detection.
• Official OpenCV documentation for Thresholding
• A great GitHub on segmentation (Attached with a YT video)

def get_characters_v2(img, img_color):
    #1. WORD SEGMENTATION (Horizontal Dilation)
    kernel = np.ones((3, 15), np.uint8)
    dilated_words = cv2.dilate(img, kernel, iterations=1)


    #Find word contours
    word_contours, _ = cv2.findContours(dilated_words, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)


    #Sort words by Y then X
    word_rects = [cv2.boundingRect(c) for c in word_contours if cv2.boundingRect(c)[2] > 5]
    if not word_rects: return [], img_color


    avg_h = sum([r[3] for r in word_rects]) / len(word_rects)
    #Sort top-to-bottom, then left-to-right
    word_rects.sort(key=lambda r: (r[1] // (avg_h * 0.7), r[0]))
    processed_data = []
    for wx, wy, ww, wh in word_rects:
        # 2. CHARACTER SEGMENTATION (Inside each word)
        word_roi = img[wy:wy+wh, wx:wx+ww]
        char_contours, _ = cv2.findContours(word_roi.copy(), cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)


        # Sort characters inside the word left-to-right
        char_rects = [cv2.boundingRect(c) for c in char_contours if cv2.boundingRect(c)[2] > 2]
        char_rects.sort(key=lambda r: r[0])


        for i, (cx, cy, cw, ch) in enumerate(char_rects):
            # Convert back to global coordinates
            gx, gy = wx + cx, wy + cy


            # Normalization/Padding logic
            sz = int(1.5 * max(cw, ch))
            char_crop = img[gy:gy+ch, gx:gx+cw]


            pad_y, pad_x = (sz - ch) // 2, (sz - cw) // 2
            padded = cv2.copyMakeBorder(char_crop, pad_y, sz-ch-pad_y, pad_x, sz-cw-pad_x,
                                        cv2.BORDER_CONSTANT, value=(0,0,0))


            final_img = cv2.resize(padded, (28, 28), interpolation=cv2.INTER_AREA)


            processed_data.append({
                'image': final_img,
                'x': gx, 'y': gy, 'w': cw, 'h': ch,
                'is_space_after': (i == len(char_rects) - 1) # Last char in word
            })


    return processed_data, img_color

Connecting the CV Pipeline with the model

This was relatively straightforward, though as a beginner I had to spend some time reading documentation and looking at other implementations.

First, we fetch the model that we previously trained:

def get_model(model_path=resource_path("model.bin")):
    model = models.shufflenet_v2_x0_5()

    model.conv1[0] = nn.Conv2d(1, 24, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
    model.fc = nn.Linear(model.fc.in_features, 47)


    model.load_state_dict(torch.load(model_path, map_location=torch.device('cpu')))
    model.eval()
    return model

model = get_model()

Then we predict each character from the extracted letters.

def predict(img, model):
    img_tensor = torch.from_numpy(img).float()
    img_tensor = (img_tensor / 255.0 - 0.5) / 0.5
    img_tensor = img_tensor.unsqueeze(0).unsqueeze(0)


    with torch.no_grad(): # Disables gradient tracking to save memory
        outputs = model(img_tensor)
        _, predicted = torch.max(outputs, 1)


    class_map = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabdefghnqrt"
    return class_map[predicted.item()]

That’s pretty much it. I then made a quick function to display the characters overlapped on the image using matplotlib.

Breaking the AI vow

In the beginning, I mentioned the challenge of not using AI for coding at all. I was, however, allowed to use AI for looking up documentation or learning concepts.

I even had a bet with a friend and thought it would be a cakewalk, considering I have never used AI for coding (At least as of July 2026).

But I failed.

More specifically, I failed at exactly one function, which I mentioned above (Under creating the CV pipeline).

So yeah, I kept trying to make it work, but it kept breaking, and I eventually used AI to rewrite that function for me.

I gave it my already written function, explained to it the problems, and made it rewrite it.

I still have my original function commented in the code.

Problems

(Small Note: By the time I am writing this section, it has already been a few months since this Hackathon took place; I have since forgotten quite a bit of the smaller problems that occurred during this phase.)

There were countless small problems that I fixed on the spot.

One of the major ones that persists but shouldn’t be hard to fix is its detection of numbers between letters.

It might detect:

JUMPS as JUMP5, and so on.
One solution is to hardcode this, since the options are limited given the constraints. One other trick I did was to set the threshold higher for numbers (Though I don’t know if it is in the final code)

Result

Well, we didn’t get shortlisted for the second round.
Before starting, I fully expected not to meet the requirements and even informed my team about it.

While we didn’t win the competition, I took it as a learning opportunity and enjoyed the challenge.
Hackathons force you to learn things that would otherwise take way longer.
I like the fact that this was a technical hackathon as opposed to one that is focused on only the idea (They are great in their own right, though).

Also, one of the judges had quite a technical background and he was impressed when we described to him our results and the method used to get those.

Final Thoughts

Not really usable, but I learnt a lot

The model was not usable in any real-world scenario. It has terrible accuracy in real-life scenarios, but I don’t think it was realistic for it to work given the constraints.
There are at least a few improvements that I have in mind, that could make it at least slightly better (Given the constraints).

If anyone wants to make any suggestions, use the code, improve it (Or just roast it), you are free to reach out to me.

I have it uploaded to GitHub here:

You can reach out to me via my email (yusuf.amin2307@gmail.com), or comment below.

Some other resources that I didn’t directly use but are worth mentioning:

1. This awesome, intuitive, and interactive website on Kernels (Helped when I was making CNN from scratch)
2. This video on CNNs
3. Introduction to CNN - GeekforGeeks.