Prompts as Auto-Optimized Training Hyperparameters
Closed this issue · 5 comments
Description
Training Best-in-Class IR Models from Scratch with 10 Gold Labels. This method allows for the training of small-scale (under 100M parameters) neural information retrieval (IR) models using as few as 10 gold relevance labels by generating synthetic queries for documents using a language model (LM). PATH is the automatic optimization of the LM prompt based on training quality.
Implementation Details
- Integrate a large autoregressive LM (e.g., GPT-3.5-turbo) for synthetic query generation.
- Use a small pretrained encoder model (e.g., DeBERTa, MiniLM) for the reranking task.
- Sample a subset of the document corpus and Generate synthetic queries using the LM with the provided prompt template.
- Sample hard negatives from the corpus to form training triplets.
- Train the encoder model on the generated triplets to act as a point-wise reranker.
- Evaluate the reranker using a metric such as NDCG@10 on the provided relevance judgments.
- Implement an optimization loop where the LM generates new candidate prompt instructions as stated in research paper.
- Can use tools from the DSPy framework to automate the prompt optimization process.
- Evaluate the reranker trained with each new prompt to find the prompt that maximizes the reranker’s quality.
Domain
Generative AI , NLP
Organisation
SamagraX
Mentors
Complexity
Medium
References
@xorsuyash I want to work on this issue. I have went through the task and have understood well. While doing the local setup I have a doubt regarding the ansible script, is it optional? Because for that we are required to use Hashicorp vault, which is a paid service.
@xorsuyash I want to work on this issue. I have went through the task and have understood well. While doing the local setup I have a doubt regarding the ansible script, is it optional? Because for that we are required to use Hashicorp vault, which is a paid service.
not sure which ansible script you are referring to here ? can you clarify?
@harshaharod21 You don't need to setup ai-tools for this task, this for experimentation purpose.
@xorsuyash I want to work on this issue. I have went through the task and have understood well. While doing the local setup I have a doubt regarding the ansible script, is it optional? Because for that we are required to use Hashicorp vault, which is a paid service.
not sure which ansible script you are referring to here ? can you clarify?
Actually that was given in the readme file of the code
@harshaharod21 You don't need to setup ai-tools for this task, this for experimentation purpose.
Okay thanks, I got it! So anyways, this issue is open to contribution right?