Machine translation of cortical activity to text with an encoder–decoder framework
jinglescode opened this issue · 0 comments
Paper
Link: https://www.nature.com/articles/s41593-020-0608-8
Year: 2020
Summary
RNN decoder encoder sequence-to-sequence network that act like a language translation machine, from ECoG to words. Building a representation to map between the 2 different sources.
Methods
Temporal convolution. Extract features in sequence of ECoG data, along the entire input sequence. Temporal-convolution layers, effectively downsamples the incoming ECoG signals by a factor of about ten.
Encoder RNN. Downsampled sequences into a representation for decoder. Also, it predict/generate a representation of the speech audio signal.
Decoder RNN. Produce each token until end of sequence token.
Assess and testing. Systematically remove or cripple each components in the encoder–decoder, to quantify their contributions.
Comments
- tested on only 4 subjects, not sure if it is able to generalize across subjects to apply on unseen subjects
- limited word vocabularies (250), and limited to classifying 50 sentences. Need more data to get it to work on more vocab. Great start.