Tags:brain decoding, functional MRI, neural representations and visual system
Abstract:
The human brain processes a vast amount of visual information daily, with complex neural mechanisms underlying the perception and interpretation of these stimuli. Recent advances in functional magnetic resonance imaging (fMRI) have allowed researchers to decode visual information from brain activity patterns in humans. We introduce a pioneering method for decoding brain activity into meaningful images and captions, with a specific emphasis on brain captioning because of increased flexibility rather than images. Our approach leverages the latest advancements in image captioning models, along with a novel image reconstruction pipeline based on latent diffusion models and depth estimation. By combining these techniques, we demonstrate significant progress in brain decoding, showcasing the enormous potential of integrating vision and language to better understand human cognition.
Brain Captioning: Decoding Human Brain Activity into Images and Text