Query instruction tuning?
Closed this issue · 9 comments
I'm trying to understand more about RAG, and have been playing with the prompt a bit. Here is one a modified prompt that attempts to do a better job of tracking the various ways in which verses may relate to each other, or the ways in which users might query the database.
query_instruction = "Analyze and summarize the key theological doctrines, philosophical ideas, historical events, narrative characters, and moral concepts illustrated in this Bible verse: "
I was motivated after a friend told me that he got poor results from the query "Who was Simon?" and I wondered if I could improve the response from the app.
Current results for query "Who is Simon?" with existing embedding prompt:
PSA 127
A Song of Ascents. By Solomon. Unless Yahweh builds the house, they who build it labor in vain. Unless Yahweh watches over the city, the watchman guards it in vain. It is vain for you to rise up early, to stay up late, eating the bread of toil, for he gives sleep to his loved ones. Behold, children are a heritage of Yahweh. The fruit of the womb is his reward. As arrows in the hand of a mighty man, so are the children of youth. Happy is the man who has his quiver full of them. They won’t be disappointed when they speak with their enemies in the gate.
Similarity Score: 0.8260074853897095
1KI 5
Hiram king of Tyre sent his servants to Solomon, for he had heard that they had anointed him king in the place of his father, and Hiram had always loved David. Solomon sent to Hiram, saying, “You know that David my father could not build a house for the name of Yahweh his God because of the wars which were around him on every side, until Yahweh put his enemies under the soles of his feet. But now Yahweh my God has given me rest on every side. There is no enemy and no evil occurrence. Behold, I intend to build a house for the name of Yahweh my God, as Yahweh spoke to David my father, saying, ‘Your son, whom I will set on your throne in your place shall build the house for my name.’ Now therefore command that cedar trees be cut for me out of Lebanon. My servants will be with your servants; and I will give you wages for your servants according to all that you say. For you know that there is nobody among us who knows how to cut timber like the Sidonians.”
Similarity Score: 0.8200909495353699
2CH 1
Solomon the son of David was firmly established in his kingdom, and Yahweh his God was with him, and made him exceedingly great. Solomon spoke to all Israel, to the captains of thousands and of hundreds, to the judges, and to every prince in all Israel, the heads of the fathers’ households. Then Solomon, and all the assembly with him, went to the high place that was at Gibeon; for God’s Tent of Meeting was there, which Yahweh’s servant Moses had made in the wilderness. But David had brought God’s ark up from Kiriath Jearim to the place that David had prepared for it; for he had pitched a tent for it at Jerusalem. Moreover the bronze altar that Bezalel the son of Uri, the son of Hur, had made was there before Yahweh’s tabernacle; and Solomon and the assembly were seeking counsel there. Solomon went up there to the bronze altar before Yahweh, which was at the Tent of Meeting, and offered one thousand burnt offerings on it.
Similarity Score: 0.8174569010734558
1KI 4
King Solomon was king over all Israel. These were the princes whom he had: Azariah the son of Zadok, the priest; Elihoreph and Ahijah, the sons of Shisha, scribes; Jehoshaphat the son of Ahilud, the recorder; Benaiah the son of Jehoiada was over the army; Zadok and Abiathar were priests; Azariah the son of Nathan was over the officers; Zabud the son of Nathan was chief minister, the king’s friend; Ahishar was over the household; and Adoniram the son of Abda was over the men subject to forced labor. Solomon had twelve officers over all Israel, who provided food for the king and his household. Each man had to make provision for a month in the year. These are their names: Ben Hur, in the hill country of Ephraim; Ben Deker, in Makaz, in Shaalbim, Beth Shemesh, and Elon Beth Hanan; Ben Hesed, in Arubboth (Socoh and all the land of Hepher belonged to him); Ben Abinadab, in all the height of Dor (he had Taphath, Solomon’s daughter, as wife);
Similarity Score: 0.8169373273849487
Modified results with the same query ("Who is Simon?") after rebuilding the index using the new prompt:
ACT 8
But there was a certain man, Simon by name, who used to practice sorcery in the city and amazed the people of Samaria, making himself out to be some great one, to whom they all listened, from the least to the greatest, saying, “This man is that great power of God.” They listened to him because for a long time he had amazed them with his sorceries. But when they believed Philip preaching good news concerning God’s Kingdom and the name of Jesus Christ, they were baptized, both men and women. Simon himself also believed. Being baptized, he continued with Philip. Seeing signs and great miracles occurring, he was amazed. Now when the apostles who were at Jerusalem heard that Samaria had received the word of God, they sent Peter and John to them, who, when they had come down, prayed for them, that they might receive the Holy Spirit; for as yet he had fallen on none of them. They had only been baptized in the name of Christ Jesus.
Similarity Score: 0.8460916876792908
MRK 1
Immediately, when they had come out of the synagogue, they came into the house of Simon and Andrew, with James and John. Now Simon’s wife’s mother lay sick with a fever, and immediately they told him about her. He came and took her by the hand and raised her up. The fever left her immediately, and she served them. At evening, when the sun had set, they brought to him all who were sick and those who were possessed by demons. All the city was gathered together at the door. He healed many who were sick with various diseases and cast out many demons. He didn’t allow the demons to speak, because they knew him. Early in the morning, while it was still dark, he rose up and went out, and departed into a deserted place, and prayed there. Simon and those who were with him searched for him. They found him and told him, “Everyone is looking for you.” He said to them, “Let’s go elsewhere into the next towns, that I may preach there also, because I came out for this reason.”
Similarity Score: 0.829193651676178
LUK 4
saying, “Ah! what have we to do with you, Jesus of Nazareth? Have you come to destroy us? I know who you are: the Holy One of God!” Jesus rebuked him, saying, “Be silent and come out of him!” When the demon had thrown him down in the middle of them, he came out of him, having done him no harm. Amazement came on all and they spoke together, one with another, saying, “What is this word? For with authority and power he commands the unclean spirits, and they come out!” News about him went out into every place of the surrounding region. He rose up from the synagogue and entered into Simon’s house. Simon’s mother-in-law was afflicted with a great fever, and they begged him to help her. He stood over her and rebuked the fever, and it left her. Immediately she rose up and served them. When the sun was setting, all those who had any sick with various diseases brought them to him; and he laid his hands on every one of them, and healed them.
Similarity Score: 0.8240087032318115
JHN 1
One of the two who heard John and followed him was Andrew, Simon Peter’s brother. He first found his own brother, Simon, and said to him, “We have found the Messiah!” (which is, being interpreted, Christ). He brought him to Jesus. Jesus looked at him and said, “You are Simon the son of Jonah. You shall be called Cephas” (which is by interpretation, Peter). On the next day, he was determined to go out into Galilee, and he found Philip. Jesus said to him, “Follow me.” Now Philip was from Bethsaida, the city of Andrew and Peter. Philip found Nathanael, and said to him, “We have found him of whom Moses in the law and also the prophets, wrote: Jesus of Nazareth, the son of Joseph.” Nathanael said to him, “Can any good thing come out of Nazareth?” Philip said to him, “Come and see.” Jesus saw Nathanael coming to him, and said about him, “Behold, an Israelite indeed, in whom is no deceit!”
Similarity Score: 0.820715606212616
I intend to test more examples between a few different kinds of prompts and see which ones give the best embeddings for a variety of queries. I'm not yet sure that this is the best prompt, but I think that there's definitely room for experimentation here.
Thank you!
I think part of the problem with the current instruction prompt of simply telling it to embed for semantic search is that it's a bit too "meta", and it doesn't specifically say on which axes it should be embedded. I might be trying to do too much by including so many different senses of semantic embedding, so it may be worth trying some shorter prompts as well. We'll know more after I'm able to run some tests.
Good observations, Clint. Excited to see the results of your tests and how we can further improve the instruction prompt!
@HanClinto Wdyt, what about including tests within this project to compare different model embeddings on subsequent changes and compare them to the prior results in a programmatic way. Summing similarity scores and some other metrics. Word-for-word matching is not great given we're desiring semantically related words, but it could an interesting metric. For example, is a name like "Doeg the Edomite" returned in 1 or more chunks. Or a target scripture/phrase and validating the results include it. This way, if we generate an improved DB, we'd have the test results to validate that it's acceptable for merge/deployment.
Yeah, I think you're right -- I think that's going to be the way to go about it.
My current idea: build a list of topics and related Bible verses and measure how well the hand-picked verses match the expected list.
I pulled out one of my old Thompson Chain Reference study Bibles, with the intention of loading up some lists of verses related to different topics, and checking to see how high those particular verses are scored in the system, and then using that to rank the "score" of that particular prompt or system.
What do you think?
Sounds great!
What picture of testing did you have in mind? I'm still trying to mull it all over in my mind.
Scaffolded a basic test here with some TODO's and ideas: https://github.com/dssjon/biblos/pull/11/files
Looks really good!
Something I'm looking at is maybe having some expected verses along with the queries. Maybe something like this:
"the universal love of God": ["JOH 3:16", "JOH 10:16", "ROM 10:12", "1TI 2:4", "1JO 2:2", "1JO 4:14"],
"the universal judgement of God": ["MAT 25:31", "ACT 17:31", "ROM 2:16", "ROM 14:10", "2CO 5:10", "2TI 4:1", "HEB 9:27", "1PE 4:5", "JUD 1:6", "REV 20:12"],
"posthumous influence of faithful people": ["MAT 26:13", "MAR 14:9", "ACT 9:39", "HEB 11:4", "2PE 1:15", "REV 14:13"],
What's amazing is how good of a job Github Copilot did in filling in some of these. My Thompson Chain gave the topic (such as "Influence, Posthumous"), and Github Copilot actually did a good job of suggesting additional passages in addition to what the Thompson Chain had. The suggestions of Mark 14 and Acts 9 both came from Github Copilot.
It also suggested several other verses that had either to do with predicting the future of what would happen after the speaker had gone, or speaking to the way a persons' faithful acts helped them.
Especially on the topic of "Judgement, Universal" -- Github Copilot came up with a ton that the Thompson Chain didn't have, with very few false-positives.
Anways, once we build a few test cases like this, I feel like we might be able to use this to measure the quality of associations as well? I'll see what I can do to work this into your test harness, or something similar to it.
Yes! I think you're on the right track here.