Reproduce the problem with Vulnscopper

[jr011]

Hello, I would like to ask if your config has the "-tail" suffix added to the ULTRA metric, such as mrr-tail, hits@1-tail. I added this suffix. When reproducing the CVE to CWE experiment under inductive settings, there was a big gap with your score. I can't figure out whether it is a problem with my settings or other reasons.

[daniel4x]

Hi @jr011, I haven't added the tail suffix since in the CPE->CVE case, I want to predict the head entity. In the CVE->CWE case, I want to predict the tail entity.
Both evaluations are running within the same script.

I've created my evaluation using the run.py test method. It goes like this:

1. Creating a relation to dataloader object:

for relation in RELATIONS_TO_PREDICT:  # RELATIONS_TO_PREDICT a list of relevant relation names
    relation_id = test_data.rel_vocab[relation].item()  # retrieve the relation id
    relation_triplets = test_triplets[test_triplets[:, 2] == relation_id]  # extract relevant relation triples
    # create a data loader
    sampler = torch_data.DistributedSampler(relation_triplets, world_size, rank)
    rel_testloader = torch_data.DataLoader(relation_triplets, cfg.train.batch_size, sampler=sampler)

2. Predict only the relevant side (head or tail but not both):

# RELATION_TO_TARGET maps the relation name to the relevant target.
# In my paper, RELATION_TO_TARGET['MatchingCWE'] is 't', which stands for tail prediction.
if RELATION_TO_TARGET[relation] == 'h':
    rankings += [h_ranking]
    num_negatives += [num_h_negative]
elif RELATION_TO_TARGET[relation] == 't':
    rankings += [t_ranking]
    num_negatives += [num_t_negative]
else:
    raise ValueError(f"Invalid target: {RELATION_TO_TARGET[relation]} for relation: {relation}")

[daniel4x]

@jr011 closing for now, let me know if you have any other issues.