Problem and approach
Detector, tracker, and rules that know this junction
Every class in the brief is defined by where a road user is, where it moves, and what the signal shows. So we detect and track everything, map it onto a hand-drawn scene map, and apply one explicit rule per class. Learned parts do perception; rules do the traffic law.
Why this design
Registration first. The camera is fixed within a video, but the samples are framed differently: samples 03 and 04 sit about 40 px off and 2 % zoomed out compared with 01 and 02. Each video's median background is matched to the reference view with SIFT + RANSAC, so the scene map is drawn only once.
A signal we can actually see. The vehicle heads on the gantry face away from the camera. A pedestrian head on the left pole faces it. We read its two lamps from a tiny ROI and checked the mapping against every stop-line crossing: main-road traffic moves while this lamp is green.
Few classes, carefully. Score A is a macro average over every class in the ground truth or in our predictions, so a class we predict that never occurs costs a whole class. We only emit classes whose rule we checked by eye on the samples.
The lamp that gives us the phase
One rule per class
| Class | Rule | Kind |
|---|---|---|
failure_to_yield | Vehicle drives through a crossing (front entering to rear leaving) while a pedestrian on that crossing is within ~90 px of its path. | rule |
stop_line | Vehicle stops past the stop line, on the stop zone or crossing, during red. Ends when the signal turns green. | rule |
red_light | Crosses the stop line more than 4.5 s into red, not as part of a queue discharge, while other vehicles still wait, and drives on into the junction. | rule |
jaywalking | Pedestrian clearly on the carriageway: inside the kerb, 30 px or more from every crossing, off islands. Scooter riders excluded; people crossing within 4 s of each other form one event, kept if it lasts 6 s or more. | rule |
illegal_u_turn | Heading reverses more than 150°, and the reversal is made on a pedestrian crossing or across the solid median (Rules of the Road of Uzbekistan, clause 62). | rule |
stopped_vehicle | Stationary 10 s or longer on the carriageway, not explained by the signal, a queue, pedestrians ahead, or a turn gap. | rule |
wrong_way | Moving against a strongly one-directional cell of the flow map for 2.5 s and 80 px, outside the junction. | learned map rule |
accident · near_miss | Contact with an impact-like deceleration, or hard braking toward a crossing road user with TTC under 1 s. Guarded against ID switches, occlusion and cars on opposite carriageways. | rule |
congestion | Eight or more vehicles crawling in the approach for 20 s and still crawling through a green phase. | rule |
| Detection | YOLO11s COCO-pretrained, 960 px input, every 3rd frame (~10 fps), ByteTrack with a 4 s buffer. | learned |
Not predicted: illegal_turn, solid_line_crossing, road_obstacle, fire_smoke. We had no reliable signal for them on this camera.
EDA of the sample videos
What the camera sees
Findings from our own detections on the four samples. Several of them changed the solution.
The videos
| Video | Resolution | File | Duration | Lighting (mean V) | Vehicles | People | Signal: green / red | Registration |
|---|
Brief says 25 fps; the files are 29.97 fps. Samples 01–03 are daylight; 04 is dusk (mean brightness about half). The signal cycle is about 30–35 s green and 42–46 s red for the main road.
Road users in view over time
Mean number of detected objects per frame in 5 s bins. The saw-tooth in cars is the signal cycle: the approach fills during red and empties on green.
Vehicle motion heatmap
Pedestrian heatmap
Trajectories and lane directions
Learned flow map
wrong_way.Vehicle speed
Speed of moving vehicles in reference pixels per second (1 px ≈ 9 cm near the stop line). Most moving traffic is at 40–120 px/s.
Pedestrian speed
Walking is 5–20 px/s. Tracks above ~45 px/s are riders; we use this to keep scooter riders out of jaywalking.
Findings that shaped the solution
| Finding | What we changed |
|---|---|
| Samples are framed differently (up to 40 px shift, 2 % zoom). | Per-video SIFT registration to one reference view. |
| Vehicle signals face away; a pedestrian lamp faces the camera and runs in phase with main traffic. | Signal phase from that lamp, verified against every stop-line crossing. |
| Legal clearing traffic crosses up to 3.9 s after the lamp turns red; queues move off up to 5 s before it turns green (sample 02). | red_light needs 4.5 s into red, no group departure, and a waiting queue. |
Scooter riders are detected as person and wait in traffic lanes. | Rider test (bike box under the person, or vehicle speed). |
| U-turns around the median nose are common; almost all turn inside the junction, not on a crossing. | Only U-turns on a crossing or across the median count as illegal (Uzbek rules, clause 62). |
| Normal traffic here constantly has short time-to-collision (turns across crossings). | Risk calibrated on 8,271 samples of normal traffic; alarms only beyond anything seen there. |
| 4K decoding dominates runtime. | Single threaded decode pass, 1280 px detector input, time-budget guard. |
Results on the sample videos
Every sample, annotated
Rendered with our own tooling: boxes, track tails, the stop line in the live signal colour, active events on top, the risk curve bottom-right. Click an event or the timeline to jump the video.
Examples and honest failure cases
| Case | What happened | Status |
|---|---|---|
| Car runs the red (sample 01, 79 s) | Crosses the stop line 19 s into red and drives through the junction while pedestrians are on the crossing. | detected |
| Cars stop on the crossing during red (sample 04, 79–115 s) | Two cars creep past the stop line and wait on the zebra until green. | detected |
| Bus runs the red (sample 02, 58 s) | Crosses 5.6 s into red while the lane beside it waits and pedestrians stand at the kerb. | detected |
| Scooter rider waiting in a lane | First version called him a jaywalker for 33 s. Fixed with the rider test. | fixed |
| Two cars merged into one box | The box "brakes" as it grows; looked like a near miss. Fixed with a rigid-box check. | fixed |
| Cars on opposite carriageways overlap in the image | Looked like contact (accident). Fixed: pairs split by the median are ignored. | fixed |
| Failure to yield next to people waiting on the kerb (samples 03–04) | They stand on the painted end of the upper crossing, so a passing car counts. 19 of 34 predictions in 03–04 are wrong, mostly this. | open |
| Cyclist on the zebra flagged as wrong-way (sample 04, 12.5 s) | Detected as a motorcycle and moving across the lane's flow direction. | open |
| Box jitter on a distant car flagged as near miss (sample 03, 282 s) | At the top of the frame a car is ~20 px wide, and box jitter looks like hard braking. | open |
| Queues discharging before the lamp turns green | Would be dozens of false red-light events. Handled by the group-departure test; the vehicle signal itself is never visible. | mitigated |
Live demo
Upload a clip, get the events back
Runs the same pipeline on a free cloud CPU. It accepts .mp4 up to 2 minutes and 200 MB, from this camera; other scenes run but the scene map won't fit them. Expect about 2–4 minutes for a 2-minute clip. Progress is shown while it runs.
If the frame is empty or asleep, open the demo in its own tab: junction-watch.streamlit.app. A sleeping app wakes up in about a minute.
Technical report
What worked, what did not, what next
Worked
Scene-specific rules over generic models. A hand-drawn scene map plus registration made every rule interpretable and debuggable frame by frame.
Reading the pedestrian lamp. A 9×9 px ROI gives the full signal phase for free and made stop_line and red_light possible.
Reviewing candidates by eye. Contact sheets around every candidate event found each failure mode listed in Results within minutes.
Calibrating risk on normal traffic. Mapping danger to probability through quantiles of 8,271 normal-traffic samples took false alarms from one every few seconds to none on the samples.
Did not work
Raw time-to-collision as risk. At this junction turns constantly cross pedestrians; the first risk score averaged 0.42 and alarmed every few seconds.
A fixed "near" radius for failure-to-yield. Reviewing 03–04 showed it confirms only 5 of 34 predictions: it cannot tell a person waiting at the kerb from one crossing.
Frame-to-frame kinematics. Near-miss and accident rules fired on tracker artefacts: ID switches, merged boxes, occlusion behind poles and buses.
The lamp as the whole truth. In sample 02 queues leave up to 5 s before the lamp turns green, so the red-light rule needed behavioural evidence as well.
Decoding 4K twice. Our first submission ran over the time budget on a T4 because of video decoding, not the model.
Next
Fix failure-to-yield first. Count only pedestrians who are walking on the crossing, not standing at its end; then train a small classifier on track pairs (distance, speeds, position on the crossing) from our labels to replace the fixed radius. A quick test of the walking condition removed many of the kerb cases, but it moved other events around, so we did not ship it untested.
Minimum object size for kinematic rules. Ignore near_miss and wrong_way candidates on boxes that are too small to measure speed reliably, and use a person-on-bike test before wrong_way.
Accident data. Fine-tune a clip classifier on public crash datasets (DoTA, CCD) to confirm contact candidates and to feed Part B.
GPU decoding. NVDEC would remove most of the runtime and allow 30 fps tracking.
Lane-level map. Draw lane lines and solid markings to add solid_line_crossing and illegal_turn.
Evaluation
We have two kinds of evidence, and we keep them apart. (1) Independent labels: our teammates labelled sample_01 and sample_02 by hand (13 events) without looking at the model output, and we score them with the official evaluate.py. (2) Review of our own predictions: for sample_03 and sample_04, every event the model predicted (46) was checked by eye. That gives precision per class, not a score, because a review of predictions cannot count missed events.
1 · Independent labels, samples 01–02
| Class | F1 (mean over tIoU 0.3/0.5/0.7) | TP / FP / FN at 0.5 | Reading |
|---|---|---|---|
jaywalking | 0.40 | 2 / 4 / 2 | Boundaries match well when we agree (241.9–253.0 s against the label 241.2–253.9 s). |
failure_to_yield | 0.00 | 0 / 42 / 2 | Different unit of event: the labels mark one 38–52 s block per video, we report each car–pedestrian conflict. Also our least precise class (see 2). |
congestion | 0.00 | 0 / 0 / 4 | Different definition: the labels count red-light queues; our rule requires the queue to persist through a green phase. |
illegal_u_turn | 0.00 | 0 / 1 / 1 | The labelled U-turn is made inside the junction, which we treat as legal under clause 62. |
road_obstacle | 0.00 | 0 / 0 / 2 | Not predicted (class removed). |
stop_line, red_light | — | 0 / 6 / 0, 0 / 3 / 0 | Not in the labels. Checked by eye: the sample_01 red-light run at 79 s is real (a car crosses 19 s into red while people are on the crossing). |
What the low score means: most of the gap is disagreement about what one event is (a block versus each conflict, a queue versus congestion), not events nobody saw. With 13 labelled events the number is also very noisy. We did not tune thresholds to these labels. We changed only the minimum jaywalking length and two real bugs (below).
2 · Review of every predicted event, samples 03–04
Method: three zoomed frames per event (start, middle, end), with the vehicle and nearby people boxed. AI-assisted, checked by the team. C = correct, U = unclear at this resolution, W = wrong. Verdicts per event: review_03_04.json.
| Class | Predicted | C / U / W | Precision (confirmed – with unclear) | Main error |
|---|---|---|---|---|
stop_line | 4 | 3 / 1 / 0 | 75 % – 100 % | — |
failure_to_yield | 34 | 5 / 10 / 19 | 15 % – 44 % | People waiting at the kerb end of the upper crossing stand on its painted edge, so a car passing them counts as a conflict. |
jaywalking | 6 | 0 / 3 / 3 | 0 % – 50 % | People stepping between the two traffic islands next to the side-road zebra, 30–50 px outside our crossing polygon. |
near_miss | 1 | 0 / 0 / 1 | 0 % | A distant car at the top of the frame: box jitter at small size looked like hard braking. |
wrong_way | 1 | 0 / 0 / 1 | 0 % | A cyclist crossing on the upper zebra, detected as a motorcycle, moving "against" the flow of the lane. |
Bugs the evaluation found (fixed)
Walkers treated as riders. Found through the labels: brisk walkers near the camera were taken for scooter riders and dropped from jaywalking. The rider test now uses the median speed.
Camera drift and the signal lamp. Found in our review: the camera in sample_01 pans about 8 px during its first 20 s, so the lamp ROI (placed from the first frame) slid off the lamp and the whole video read as red. The reader now re-registers at 2, 5, 10, 20, 40 s and then every 60 s.
Team
Team
Computer-vision lead
Mekhroj
- Road-analysis pipeline: detection, tracking, registration
- Signal-phase reader and scene map
- Event rules, Part B risk score, evaluation
Frontend lead
Jamshid
- Website design and layout
- Sample-video results pages
- Interactive timelines
Web and demo engineer
Nizomiddin
- Live demo app and deployment
- EDA charts
- Hosting and links
A project we are proud of
An AI camera system that takes attendance automatically: classroom cameras, face detection and recognition in Python, attendance written to a register without anyone calling names.
Links