Vision Hat

overview

This project was built at HackED 2026 to help visually impaired people navigate their surroundings through real-time audio feedback. A camera mounted on the brim of a hat streams video to a Raspberry Pi 5, which runs YOLOv8 to detect objects in each frame. A scoring algorithm then selects the most relevant object based on its type, size, and position, and classifies it as close, ahead, or far depending on how much of the frame it takes up. The result is spoken through bone conduction headphones using text-to-speech, and the entire system runs on the Pi without needing an internet connection.

Design decisions and challenges

• Assigned each object type a priority weight, so people are announced first, followed by furniture like chairs and couches, and then smaller objects like bottles and backpacks.
• Weighted objects near the center of the frame more heavily, since those are the ones directly in the user's path.
• Added an obstacle fallback for things YOLOv8 can't classify, like walls, so anything that fills more than about a third of the frame and sits near the center is still announced as an obstacle.
• Estimated distance from the size of each object's bounding box instead of adding a separate sensor, keeping external hardware down to a single camera.
• Used bone conduction headphones for audio output so the user's ears stay open to the sounds around them.
• Filtered out irrelevant detections so the user isn't overwhelmed with callouts, by limiting detection to a short list of objects, ignoring people far in the background, and announcing only one object at a time.

What I learned

A pre-trained computer vision model gets you most of the way there, but the real work is adapting it to your specific use case. Deciding what to ignore, what to prioritize, and how to turn raw detections into useful feedback had a bigger impact on the final product than the model itself.