Vision, Problem Formulation, Tasks, RQs, Writings, ....

[HamedBabaei]

Research Questions

RQ1: Can LLMs identify term/entity types? Task A
RQ2: Do LLMs comprehend relations?

• RQ2.1: Can LLMs recognize types hierarchies? (is a relations -- tree structures) Task B
• RQ2.2: Can LLMs identify Non-Is-A relations in hierarchies? (non is a relations -- graph structures) Task C

Tasks

Task A: The goal is to find out which LLMs are capable of finding terms/entities type without giving prior knowledge about types. Because we don't want to give any knowledge to LLMs about types, this task is a Generation task. Design considerations during solving this task are as follows:

1. We are interested to know the entity types at the lowest level.
2. Since one of the possible steps is fine-tuning so splitting data into train and test sets are required!
3. We only will consider entities/terms.
4. Entities in types hierarchy leaf inherent in their own parent's types as well (this is for the evaluation part since for this task we don't expect models to know the hierarchies).

---

Task B: The aim of this task and the next task (Task C) is to understand whatever LLMs could find relations without naming those relations. This relationship could be an undirected or directed relationship. These tasks are classification tasks.

For example:

Acquired Abnormality is a location of a Virus.

The location_of is a relation between mentioned two types. Our goal is to find that Acquired Abnormality and Virus have relations. Not find the name of the relation (which in this case is location of). Because naming relations refers to clustering similar relations and asking experts to name them. So in this task, we are interested to know what is the is a relation in terms/entities types.

In Task B, we want to only find types of relationships that form a hierarchy (a structure that struct types tree format from top to down where the top is a root -- it could be multiple roots -- and down is a leaf) and this type of relationships called is a relations. As an example:

C is a subclass of B.
B is a subclass of A.
D is a subclass of B.
E is a subclass of A.

---

Task C: However, in types, it is possible to find relations outside of the tree structure, and it is similar to relations between types in graph format. For example (considering Task B example):

E somehow has a relationship with C.
C somehow has another direct relation with A.

So, in this task, the goal is to analyze LLMs from this perspective.

Conference

Our Target is for ISWC 2023: https://iswc2023.semanticweb.org/call-for-research-track-papers/

Abstract submission due	May 2nd, 2023
Full paper submission due	May 9th, 2023
Objection and Response	June 13th – 16th, 2023
Notifications	July 12th, 2023
Camera ready papers due	July 31st, 2023

[HamedBabaei]

Links for Information (the list will be updated regularly

Info	date	link
Diagram of the paper	23 Feb 2023	LLMs4OL.pdf
Link to editable version of overleaf	23 Feb 2023	ISWC 2023 Overleaf Template

[HamedBabaei]

Things that are on my to do list:

1. LLaMA for Tasks
2. ORKG research field taxonomy for Task B: https://orkg.org/help-center/article/20/ORKG_Research_fields_taxonomy
3. Task D: generation of documentation for classes, properties task