Summary: Researchers have developed MovieNet, an AI model inspired by the human brain, to understand and analyze moving images with extraordinary accuracy. By mimicking the way neurons process visual sequences, MovieNet can identify subtle changes in moving scenes using far less information and energy than traditional AI.
In tests, MovieNet outperformed existing AI models and even human observers in recognizing behavioral patterns, such as tadpoles swimming in different conditions. Its eco-friendly design and potential to revolutionize fields such as medicine and drug development highlight the transformative power of this breakthrough.
Key data:
- Brain-like processing: MovieNet mimics neurons to process video sequences with high accuracy, enabling it to distinguish dynamic scenes better than traditional AI models.
- High performance: MovieNet achieves high accuracy with low energy and data, making it more durable and scalable for diverse applications.
- Medical potential: AI can help detect diseases like Parkinson’s early by identifying subtle changes in movement. AI can also improve drug discovery methods.
Source: Scripps Research Institute
Imagine an artificial intelligence (AI) model that can see and understand moving images with the finesse of the human brain.
Scientists at Scripps Research have now made this a reality with the development of MovieNet: a groundbreaking AI that processes videos in the same way our brains interpret real scenes as they unfold over time.
This brain-inspired AI model, described in a study published in the Proceedings of the National Academy of Sciences , can visualize dynamic scenes by mimicking how neurons (brain cells) interpret the world in real time.
Traditional AI specializes in recognizing still images, but MovieNet introduces a method that allows machine learning models to recognize complex and changing scenes. This development could revolutionize fields like medical diagnosis and autonomous driving, where distinguishing subtle changes over time is crucial.
MovieNet is also more accurate and environmentally friendly than traditional AI.
“The brain doesn’t just see pictures; it creates an ongoing visual narrative,” says lead investigator Hollis Kline, PhD, director of the Dorris Neuroscience Center and Hahn Professor of Neuroscience at Scripps Research.
“Recognizing still images has come a long way, but the brain’s ability to process moving scenes—such as watching a movie—requires a much more sophisticated form of pattern recognition. By studying how neurons record these sequences, we have been able to apply similar principles to AI.”
To create MovieNet, Klein and first author Masaki Hiramoto, a scientist at the Scripps Research Center, investigated how the brain processes real-world scenes as short sequences, similar to film clips. Specifically, the researchers studied how tadpole neurons responded to visual stimuli.
“Tadpoles have a very good visual system and we also know that they can effectively detect and respond to moving stimuli,” explains Hiramoto.
He and Klein identified neurons that respond to cinematic features—such as changes in brightness and image rotation—and that can recognize objects with movement and change. These neurons are located in the brain region responsible for visual processing, the optic tectum, and assemble parts of a moving image into a coherent sequence.
You can compare this process to a lenticular puzzle: each piece doesn’t seem logical on its own, but together they create a moving image.
Different neurons process different ‘puzzle pieces’ of a moving image, which the brain then integrates into a coherent whole.
The researchers also discovered that neurons in the tadpoles’ optic tectum can distinguish subtle changes in visual stimuli over time. They recorded the information in moving clips of about 100 to 600 milliseconds, rather than static frames.
These neurons are highly sensitive to patterns of light and shadow. The response of each neuron to a specific part of the visual field helps to create a detailed map of a scene, which can then be used as a kind of “movie clip.”
Cline and Hiramoto trained MovieNet to mimic brain-like processing and encode video clips as a series of small, recognizable visual cues. This allowed the AI model to distinguish subtle differences between moving scenes.
To test MovieNet, the researchers showed video clips of tadpoles swimming in different conditions.
Not only did MovieNet achieve 82.3 percent accuracy in distinguishing between normal and abnormal swimming behavior, it also outperformed trained human observers by about 18 percent. It also outperformed existing AI models like Google’s GoogleNet, which achieved only 72 percent accuracy despite extensive training and processing power.
“We really saw the potential here,” says Kline.
The team found that MovieNet not only outperformed existing AI models in understanding changing scenes, but also used less data and processing time.
MovieNet’s ability to simplify data without sacrificing accuracy also sets it apart from traditional AI. By breaking down visual information into essential sequences, MovieNet effectively compresses data like a compressed file, while preserving important details.
In addition to its high accuracy, MovieNet is an environmentally friendly AI model. Traditional AI processing consumes a huge amount of energy, which has a significant impact on the environment. MovieNet’s low data requirements offer a more sustainable alternative that saves energy without sacrificing performance.
“By mimicking the brain, we’ve made our AI much less demanding, paving the way for models that are not only powerful but also sustainable,” says Klein. “This efficiency also allows us to extend AI to areas where traditional methods are expensive.”
What’s more, MovieNet has the potential to transform medicine. As the technology advances, it could become a valuable tool for identifying subtle changes in early-stage diseases, such as detecting cardiac arrhythmias or early signs of neurodegenerative diseases like Parkinson’s.
For example, AI could detect small motor changes associated with Parkinson’s disease in the early stages, which are often difficult for the human eye to detect. This would give doctors valuable time to intervene.
Additionally, MovieNet’s ability to detect changes in tadpoles’ swimming patterns when exposed to chemicals could lead to more accurate drug detection techniques, allowing scientists to study dynamic cellular responses rather than relying on static snapshots.
“Current methods cannot detect significant changes because they can only analyze images taken at intervals,” Hiramoto emphasizes.
By observing cells over time, MovieNet can detect even the most subtle changes during drug trials.
Looking ahead, Kline and Hiramoto plan to further improve MovieNet’s ability to adapt to different environments, expanding its versatility and potential applications.
“Drawing inspiration from biology will continue to be fertile ground for AI development,” says Klein. “By designing models that think like living things, we can achieve levels of performance that are not possible with traditional methods.”
Funding: This work was supported by grants from the National Institutes of Health (RO1EY011261, RO1EY027437, and RO1EY031597), the Han Family Foundation, and the Harold L. Dorcience Endowment Center for Neuroscience Fund for the study “Identifying Film-Coding Neurons Enables Film Recognition AI.”
About this AI research news
Author: Press Office
Source: Scripps Research Institute
Contact: Press Office – Scripps Research Institute
Image: The image is credited to StackZone Neuro
Original Research: Open access.
“Identification of movie encoding neurons enables movie recognition AI” by Hollis Cline et al. PNAS
Abstract
Identifying the neurons that encode movies makes it possible for AI to recognize movies.
Natural visual scenes are dominated by the spatiotemporal dynamics of images, but it is unclear how the visual system integrates information from the ‘movie’ over time.
We characterized the neuronal receptive fields of the optic tectum using scattered noise stimuli and inverse correlation analysis.
Neurons recognized videos with initial and final stimuli lasting 200 and 600 ms. The duration of the videos, from the beginning to the end of the responses, was adjusted by a classification algorithm based on sensory experience.
Neurons encoded families of image sequences according to triangular functions. The series of action potentials and the flow of information suggest that film detection is based on recurrent circuit shapes.
The principles of topographic retinotectal plasticity in frogs and simple cortical cells are being applied to machine learning networks for static image recognition. This suggests that discoveries about the principles of film coding in the brain, such as how image sequences are encoded and their duration, could be useful for film recognition technology.
We built and trained a machine learning network that mimics the neural principles of the visual system’s film encoders.
The network, called MovieNet, outperformed existing machine learning image recognition networks in classifying natural film scenes. At the same time, the data size was reduced, and fewer steps were required to perform the classification task.
This research shows how film sequence and timing are encoded in the brain. It also shows that brain-based film processing principles enable effective machine learning.
