VisionOS

AI that reaches into everyday life

Your world.
Within reach.

Step-by-step speech and tactile guidance, built to put more spatial awareness and everyday control in your hands.

Explore VisionOS Meet our mission
VisionOS prototype: a pink enclosure connected by cables to a curved wearable assembly with actuators, resting on a white chair.
The VisionOS prototype, built at Cal Hacks.

Our mission

More awareness.
More agency.

We’re building a world where blind and low-vision people have greater control over their surroundings, their choices, and their everyday lives.

An unfamiliar layout or a hard-to-locate control can leave essential information out of reach. VisionOS connects visual understanding with speech and tactile cues, helping turn “where is it?” into “what’s my next move?”

From finding it
to reaching it.

Knowing what’s nearby is a beginning. We’re building assistance that stays with you through the next movement.

Close-up of the wearable hardware’s curved black band, blue servo actuators, white actuator arms, and wiring.
Servo-driven actuators deliver directional tactile cues.

  1. Say what you need.

    Ask for an everyday object. The prototype identifies a candidate, checks the match, and tracks the target.

  2. Feel the next direction.

    Hand tracking and available depth information translate the target’s position into step-by-step speech and tactile guidance.

  3. Make the move yourself.

    You move your hand and stay in control. The wearable supplies directional cues as you reach.

Early prototype. Guidance depends on camera visibility, calibration, depth quality, and the connected hardware setup.

A little more context.
A lot more possibility.

Assistance for the scene in front of you, the moment you missed, and the people you choose to hear from.

“What’s around me?”

Understand the scene.

Hear a description of what the camera can see. Ask a follow-up when you need more detail.

“What did I miss?”

Catch up on the moment.

Get a recap of recent visible changes from a short image history. The prototype does not record ambient conversations.

Family Notes + Voice

A familiar connection.

Trusted people can leave notes to hear between tasks or when a relevant object is found. With consent, a helper’s notes can use an AI copy of their voice, with a listener opt-out.

Explore Family Notes

Intelligence, made tangible

Built to connect
perception with action.

An iPhone supplies camera and depth information. The current laptop-supported prototype turns that context into spoken responses and wearable cues.

Specialist AI, working together

A coordinator routes requests to object-finding, scene-description, and recent-context specialists. Anthropic handles intent routing; Gemini interprets visual information.

Local tracking and guidance turn verified observations into cues. AI models do not directly control the wearable.

Guidance grounded in the current view

Fresh hand and target observations inform the next movement. Missing or unreliable evidence pauses guidance rather than filling in the gaps.

Servo-driven wearable feedback adds a physical sense of direction. The user makes the movement.

Speech, connection, and control

Voice requests, spoken responses, and local stop handling keep the interaction accessible without relying on a screen. Cartesia supports conversational speech; local audio handles immediate guidance.

The project also includes a native iPhone app, a watch-only helper interface, and the Family Notes companion.

The hardware behind the idea.

Close-up of the pink VisionOS enclosure with the VisionOS name, black straps, and connected wiring.
The VisionOS enclosure.
Another angle of the connected VisionOS prototype, showing the pink enclosure alongside the wearable actuator assembly.
The connected wearable assembly.

Our story

Technology should expand
what people can do.

We started VisionOS at Cal Hacks with a question: how can AI offer useful help in the physical world?

Our answer begins with the space between understanding a task and carrying it out. We’re connecting visual context to step-by-step assistance, starting with everyday object finding and reaching.

Our broader ambition includes interacting with buttons, doors, and kiosks. Those are goals we’re working toward, not completed real-world demonstrations.

We want blind and low-vision people, accessibility practitioners, and hardware collaborators to help shape that work.

Follow the build on GitHub

Build a more
accessible everyday.

Your experience can help shape what comes next.
Share a perspective, a question, or an idea for collaboration.

Get involved

Opens a public GitHub issue. A GitHub account is required.