Vision AI is a not-new-but-hotter-than-it-used-to-be capability. The thing that changed is the opportunity in physical AI, as physical AI gets increasingly real. As a refresher, vision AI is AI that can look at images or video and understand what is in them, while physical AI is does not just understand information – it can sense, reason, and act in the real world through machines like robots, cars, drones, or factory systems. Vision AI might see a box, but physical AI can see the box, decide what to do with it, and then pick it up or move around it.
Low search accuracy is a problem with traditional video searching. This is because traditional search is limited to trained attributes. With a single embedding model, we can move search from retrieval based on trained attributes to generative. Traditional approaches might fire alters when a triggering event happens in a video (think a dog walks by your security camera and you get an alert). This is great, expect we all get “alert fatigue” amirite? At first you look at every alert and then, after a while, you look at nothing. It is difficult for a human to filter the important alerts (sketchy looking human passing the video camera in the middle of the night) from the unimportant (squirrel!). We want to find the true positives, things that need to be escalated, and things where we need to act. It’s the classic needle in a haystack problem.
Video search and summarization (VSS) is a set of vision aware tools that can connect to agents so the agents can understand what they are seeing. Think about tools for decomposing, searching, retrieving, critiquing, and summarizing context of a bunch of video. VSS is a hard problem – think about what you would have to do to summarize a video – you’d have to find it, watch it, figure out what’s important, maybe watch it again, pick out some highlights, summarize it, and then check your work. That’s what a VSS does. And it can do that for a one-hour video in about six minutes. Wow.
The KPIs here were pretty strong. 80% quicker onboarding for a training company, 80% reduction in incident reporting fatigue in a manufacturing company, 95% cost reduction for a training company. Let’s take an example, imagine you want to find all the places in a soccer game where a particular person scored a goal. You know how you would do that and can estimate how long it would take. If the game is an hour, it would take… at least an hour.
Now imagine that you could process the video in six minutes and then ask a natural language questions and get verified evidence of every time that person score a goal. All in about six minutes to process the video, and two seconds to answer the questions (with evidence!). That’s literally a 90% performance gain, conservatively.
It won’t surprise you by now that “there’s an NVIDIA solution for that” and it’s their VSS blueprint. NVIDIA VSS turns video from passive footage into an AI-readable system you can search, summarize, and act on. It is interesting because it treats video as a first-class enterprise data source, combining computer vision, VLMs, LLMs, and RAG so organizations can move from watching footage to querying and operationalizing it.
By Tela Gallagher Mathias, CTO at PhoenixTeam
