Questions about query feature visualization in the paper

[Qrteng]

Great work and thank you for publishing the code!
I have a question about figure 4 in the paper. I am curious that how you obtain query feature visualization result. Is that by using a well-known data visualization method or your own method? I want to create a visualization result like this, thank you very much!

[ljzycmd]

Hi @Qrteng, I perform the PCA on the query features and select 3 principal components as R, G, and B channels to form the visualization. The codes are:

from sklearn.decomposition import PCA
from PIL import Image


def visualize_and_save_features_pca(feats_map, t, save_dir, layer_idx):
    """
    feats_map: [B, N, D]
    """
    B = len(feats_map)
    feats_map = feats_map.flatten(0, -2)
    feats_map = feats_map.cpu().numpy()
    pca = PCA(n_components=3)
    pca.fit(feats_map)
    feature_maps_pca = pca.transform(feats_map)  # N X 3
    feature_maps_pca = feature_maps_pca.reshape(B, -1, 3)  # B x (H * W) x 3
    for i, experiment in enumerate(feature_maps_pca):
        pca_img = feature_maps_pca[i]  # (H * W) x 3
        h = w = int(np.sqrt(pca_img.shape[0]))
        pca_img = pca_img.reshape(h, w, 3)
        pca_img_min = pca_img.min(axis=(0, 1))
        pca_img_max = pca_img.max(axis=(0, 1))
        pca_img = (pca_img - pca_img_min) / (pca_img_max - pca_img_min)
        pca_img = Image.fromarray((pca_img * 255).astype(np.uint8))
        pca_img.save(os.path.join(save_dir, f"{i}_time_{t}_layer_{layer_idx}.png"))

Hope this can help you.

[Qrteng]

Thanks a lot!