Linking Brain Signals to Visual Concepts: CLIP based knowledge transfer for EEG Decoding and visual stimuli reconstruction

decoding visual representations from human brain activity has emerged as a thriving research domain, particularly in the context of brain-computer interfaces. this study introduces a novel approach using a convolutional neural network (CNN) to classify images from the ImageNet dataset, leveraging electroencephalography (EEG) recordings. we collected EEG data from 6 subjects, each viewing 50 images across 40 distinct semantic categories. these EEG signals were transformed into spectrograms, serving as the input for training our CNN. a unique aspect of our model is the incorporation of knowledge distillation from a pre-trained image classification teacher network. this approach enabled our model to achieve a top-5 accuracy of 80%, notably surpassing a plain CNN baseline. furthermore, we integrated an image reconstruction pipeline founded on pre-trained latent diffusion models. this innovative concatenation not only decodes images from brain activity but also provides a plausible reconstruction, facilitating rapid and subject-specific feedback experiments. our work thus represents a significant advancement in the field, bridging the gap between neural signals and visual perception.