Why this problem
India's traffic enforcement has a data problem that isn't about data volume. Cameras are everywhere - urban intersections, highways, entry points. The footage exists. What doesn't exist is a unified chain from "this plate ran the red light" to "this is who was driving" to "this person has done it before." The "One Nation, One Challan" initiative created a national framework for challan generation, but left a gap at the identity layer: number plate systems identify vehicles, not drivers. A vehicle can be shared, borrowed, or stolen. The plate alone doesn't close the accountability loop.
KAVACH-2023 - the national cybersecurity hackathon run by India's Ministry of Home Affairs and Ministry of Education - framed the problem as an enforcement infrastructure challenge. The brief wasn't "build a CCTV system." It was: build something that links evidence to identity at scale. I led a team of six on it.
What the obvious approach gets wrong
The obvious path: deploy YOLOv8 on a camera feed, extract plate numbers, log violations. This works in the sense that plates are detectable and OCR on Indian plates is tractable in good lighting. We reached 92% precision and 91% recall on plate detection using the YOLOv8-nano variant. That part isn't the hard part.
Two things break after detection.
- First, a detected plate is not a readable plate. Indian plates are a mix of standardized HSRP plates and non-standard ones - hand-painted, custom fonts, decorative scripts, inconsistent spacing. OCR behaves completely differently across the two, so we trained a separate ResNet-18 classifier on the cropped plate region to sort standard from non-standard before the OCR stage, reaching 89% accuracy on that binary decision. Routing both classes through one OCR path is the mistake that quietly caps your accuracy.
- Second, a logged plate number isn't an offender. If the registered owner didn't commit the violation - impersonation, fleet vehicle, cloned plate - the challan goes to the wrong person. Cross-state enforcement is especially fragile: a cloned Gujarat plate on a Maharashtra vehicle generates a challan that dies in the RTO system.
Adding face recognition naively doesn't fix this either. The meaningful signal lives in the driver's face region, which in Indian traffic conditions is frequently occluded by a helmet or scarf, or compressed to near-nothing by low-resolution video. And unlike plates, you cannot collect a large labelled dataset of driver faces - the enrollment population is the problem, not the annotation budget.
The core decision
The architecture makes two decisions in sequence.
First: separate detection from recognition with different models. YOLOv8 handles face detection and produces a tight crop of each detected face. Recognition runs on that crop, which is a far simpler problem than recognizing a face in a full frame full of road, background, and pedestrians.
Second - and this is the decision the project actually existed to test - we built the recognition side twice and compared the two. One path was a conventional deep learning recognizer using ResNet feature embeddings, trained in the standard data-hungry way. The other was a Siamese-style few-shot approach designed to work from a handful of reference images per person, which is what a real enrollment database would realistically give you. We used dlib as the baseline for both.
A license plate identifies a vehicle. The enforcement gap is the driver - and the honest constraint on closing it is not model architecture, it's how little labelled data you can ethically and practically collect on drivers.
The trade-off came out the way the literature suggests but the magnitude is what mattered: the deep learning path is more accurate when you have the data, the Siamese path degrades far more gracefully when you don't, and in low-illumination CCTV crops the gap narrows enough that the few-shot approach is the defensible choice for anything resembling deployment. Choosing the accurate-on-paper model here would mean choosing the one that cannot be enrolled with.
The full pipeline runs: CCTV frame → YOLOv8 plate detection → standard/non-standard classification → EasyOCR → plate identity; in parallel, YOLOv8 face detection → few-shot recognition → driver identity. The backend cross-checks plate registration against driver identity, flags mismatches, and writes timestamped violation records.
Data came from a real deployment context. The Ahmedabad West traffic police, with the support of the police commissioner of Ahmedabad West, provided high-definition footage from traffic cameras for research use under a non-disclosure agreement. Preprocessing and augmentation ran through OpenCV and torchvision transforms.
How it was built
The ANPR pipeline processes each frame through several image transformations before OCR: it crops the detected plate region, converts it to grayscale, enhances contrast, and inverts it to maximise character distinctiveness. This preprocessing sequence matters for Indian plates specifically - embossed characters on dirty or faded plates read differently under each transformation, and the pipeline takes the best reading across variants before EasyOCR runs.
The recognition side used transfer learning on ResNet backbones with pretrained weights, fine-tuned on the face crops, with the Siamese variant trained on pairs rather than classes, so that adding a new person doesn't require retraining.
Everything sits behind a decoupled API. That decoupling is the deployment argument: the heavy detection and recognition models run on a GPU machine on the local network or in the cloud, and the client stays thin enough to run on an edge device at the intersection or in an officer's hand. The front end is a React Native application, so the same client renders across platforms - upload a frame, get back recognised identities and annotated image evidence, confirm or override before the violation record is written.
The system was shortlisted through KAVACH-2023 national selection rounds - first as "VioSense: Violation Sensing," (funfact: as "VigilantEye", our other problem statement also got finalised). We were one of five teams selected from Gujarat, competed against the top 100 teams nationally, and finished as finalists.
What worked, what didn't
ANPR performance was solid for the conditions: 92% precision and 91% recall on plate detection, and 89% accuracy separating standard from non-standard plates. The preprocessing pipeline held up across real CCTV variability - rusted plates, angle distortion, motion blur. The detection confusion matrix shows a near-zero background false positive rate, which matters more than headline precision for a system that would otherwise generate phantom challans.
The honest part isn't in the results table. Both recognition paths were evaluated closed-set, on a test distribution that matches the training distribution. Closed-set metrics are optimistic by construction. Real enforcement would require an enrollment database of registered drivers - which turns the problem from recognition into retrieval, and retrieval over a city-scale population is a different system with different failure modes.
The few-shot path is the one that survives that transition, which is the useful finding. The deep learning path's accuracy advantage is real and also irrelevant if you can never collect enough labelled faces per driver to realize it.
What I'd do differently
Build the retrieval pipeline from the start. Training a closed-set classifier is the obvious approach and also the one that doesn't scale past the test set. A face embedding model plus a searchable enrollment index - structurally the same thing the ANPR side already does with plate lookup - is the right architecture for a system meant to operate over an open population. We arrived at that conclusion through the Siamese comparison rather than designing for it.
Second, I'd separate the OCR evaluation from the detection evaluation. Reporting detection precision and recall says nothing about whether the plate was read correctly, and end-to-end plate-string accuracy is the number an enforcement agency actually cares about. We measured the stages we could measure cleanly and left the composite number underspecified.
