Magnitude of Vectors on chart not taking into account number of vectors?
Closed this issue · 3 comments
zrichz commented
Hi - as far as I can tell with the code, there needs to be a division at the end of this code snippet to take into account the number of vectors used for the embedding, to get the average magnitude across all vectors:
def get_vector_data_magnitude(data: dict[int, dict[int, Tensor]], step: int) -> float:
value = 0
for n in data[step]:
value += pow(n, 2)
value = math.sqrt(value) ##<------needs a divisor here?
return value
as current this is just squaring every data point and then sqrt the result
Zyin055 commented
So something like this?
def get_vector_data_magnitude(data: dict[int, dict[int, Tensor]], step: int) -> float:
value = 0
for n in data[step]:
value += pow(n, 2)
vectors_per_token = int(len(data[step]) / DIMS_PER_VECTOR) #ie: 1, 3, 10, etc
value = math.sqrt(value) / vectors_per_token
return value
zrichz commented
Yes I believe so. I was trying to write this change but didn't realise you had to explicitly define the vectors per token again. I'm happy to test this tonight and let you know
zrichz commented