Simple Embeddings

[corranmac]

Hi,

Please could you provide a simple way to load a model and test a single audio clip to produce an embedding?

Thank you very much.

[atosystem]

@corranmac Got it!
I have updated the readme.
See example.py

[corranmac]

Thanks for such a quick response!

I was wondering if I'm able to transform these outputed embeddings to the same shape as clip, for use in speech-image retrieval and also image generation trained on clip embeddings? I can't seem to find a seperate class for encoding, eg. model.encode() like clip has.

Thanks

[atosystem]

@corranmac
Yeah, you can use the semantic embedding of speech to calculate similarity with image embeddings for speech-image retrieval. In fact, this is how we do it in our paper.
I have added a function in the model class (kwClip.py) for extracting the semantic embedding for speech input

SpeechCLIP/avssl/model/kwClip.py

Lines 1299 to 1315 in e2a572d

	def encode_speech(
	self,
	wav,
	) -> dict:
	"""encode speech
	Args:
	wav (list): input list of waveforms
	Returns:
	dict: {
	"cascaded_audio_feat" : if cascaded branch exists
	"parallel_audio_feat" : if parallel branch exists
	"vq_results" : if cascaded branch exists
	"keywords" : if cascaded branch exists
	}
	"""

[atosystem]

@corranmac If there is no further question, I will close this issue.