Scientists create AI that ‘watches’ videos by mimicking the brain

This brain-inspired artificial intelligence model is detailed in a study published in the journal Science Proceedings of the National Academy of Sciences November 19, 2024 It can sense moving scenes by simulating how neurons (or brain cells) perceive the world in real time. Traditional AI excels at identifying static images, but MovieNet introduces a machine learning model approach to identifying complex, ever-changing scenes—a breakthrough that could transform fields from medical diagnostics to autonomous driving, where everything is changing at any time. It’s important to recognize subtle changes over time. MovieNet is also more accurate and greener than traditional artificial intelligence.

“The brain doesn’t just see static images; it creates an ongoing visual narrative,” said senior author Hollis Klein, Ph.D., director of the Dorris Center for Neuroscience and Hahn Professor of Neuroscience at Scripps Research. Static image recognition has come a long way, but the brain’s ability to process moving scenes, such as watching a movie, requires more sophisticated forms of pattern recognition. By studying how neurons capture these sequences, we have been able to apply similar principles to humans. intelligent.

To create MovieNet, Klein and first author Masaki Hiramoto, a scientist at Scripps Research Institute, studied how the brain processes real-world scenes into short sequences, similar to movie clips. Specifically, the researchers studied how tadpole neurons respond to visual stimuli.

“Tadpoles have a very good visual system, and we know they can effectively detect and respond to moving stimuli,” Hiramoto explains.

He and Klein discovered neurons that respond to film-like features, such as brightness changes and image rotation, and can recognize moving and changing objects. These neurons, located in a visual processing area of ​​the brain called the optic tectum, assemble the parts of a moving image into a coherent sequence.

Think of this process as similar to a lenticular puzzle: each piece may not make sense on its own, but together they form a complete moving image. Different neurons process various “puzzle pieces” of real-life moving images, which the brain then integrates into a continuous scene.

The researchers also found that tadpole optic tectal neurons are able to distinguish subtle changes in visual stimuli over time, capturing information in dynamic segments of about 100 to 600 milliseconds rather than static frames. These neurons are highly sensitive to patterns of light and shadow, and each neuron’s response to specific parts of the visual field helps build a detailed map of the scene to form a “movie clip.”

Cline and Hiramoto trained MovieNet to simulate this brain-like processing and encode movie clips into a series of small, recognizable visual cues. This enables AI models to distinguish subtle differences between dynamic scenes.

To test MovieNet, the researchers showed it film clips of tadpoles swimming under different conditions. Not only did MovieNet achieve 82.3% accuracy in distinguishing normal from abnormal swimming behavior, it also outperformed the ability of trained human observers by approximately 18%. It even surpassed existing AI models such as Google’s GoogLeNet, which achieved only 72% accuracy despite its massive training and processing resources.

“This is where we see the real potential,” Klein noted.

The team determined that MovieNet was not only better at understanding changing scenes than current AI models, but it also used less data and processing time. MovieNet’s ability to simplify data without sacrificing accuracy also sets it apart from traditional artificial intelligence. By breaking down visual information into basic sequences, MovieNet can effectively compress the data like a compressed file that preserves key details.

In addition to high accuracy, MovieNet is also an environmentally friendly artificial intelligence model. Traditional artificial intelligence processing requires a lot of energy and has a serious impact on the environment. MovieNet’s reduced data requirements provide a greener alternative that saves energy while maintaining high standards of performance.

“By mimicking the brain, we have successfully reduced the requirements for artificial intelligence, paving the way for a powerful and sustainable model,” said Klein. “This efficiency also opens the door to scaling artificial intelligence in areas where traditional methods are costly.”

Additionally, MovieNet has the potential to reshape medicine. As the technology improves, it could become a valuable tool for identifying subtle changes in early disease, such as detecting irregular heartbeats or spotting the first signs of neurodegenerative diseases like Parkinson’s disease. For example, small movement changes associated with Parkinson’s disease are often difficult to discern with the human eye, and artificial intelligence can flag them early, giving clinicians valuable time to intervene.

Additionally, MovieNet is able to sense changes in tadpoles’ swimming patterns when they are exposed to chemicals, which could lead to more precise drug screening techniques as scientists can study dynamic cellular responses rather than relying on static snapshots.

“Current methods miss key changes because they can only analyze images captured at intervals,” commented Hiramoto. “Looking at cells over time means MovieNet can track the smallest changes during drug testing.”

Going forward, Cline and Hiramoto plan to continue to refine MovieNet’s ability to adapt to different environments, enhancing its versatility and potential applications.

“Drawing inspiration from biology will continue to be fertile ground for advancing artificial intelligence,” said Klein. “By engineering models to think like living organisms, we can achieve levels of efficiency that are simply not possible with traditional methods.”

This work is a study of “Identification of movie-encoding neurons enables movie-recognition artificial intelligence,” supported by grants from the National Institutes of Health (RO1EY011261, RO1EY027437, and RO1EY031597), Hahn Family Foundation and the Harold Dorris Center for Neuroscience Endowment Fund.