Search results aren't good
daveloyall opened this issue · 5 comments
Using these terms def con 26 badge ama the web page I wanted was this: https://www.reddit.com/r/Defcon/comments/973jik/dc26_official_badge_hardware_ama/
(This is just a real life example...)
Google returns that page for those terms as the second result. Duck Duck Go returns the parent page, but not the desired page.
DidYouGoGo.com doesn't return anything related.
Thank's for trying out alpha software. I appreciate it.
22 days ago the index consisted of thousands of pages. So. I mean.
Oh gosh, @kreeben, I didn't mean to be negative!
Open bugs result in improvements. I was trying to help! Look at this famous bug. It was open for nine years, and it served as a rallying point... Folks would bring it up in discussions about all sorts of things. Like, "Sam, your idea has some merit, but what does it do to help close bug number 1?"
Anyway, back to my report... I didn't expect that your index corpus would include every single reddit comment, yet! That's why I picked those words as my search input. I was checking how deep your index was.
Incidentally, I like it when a search engine returns zero results when nothing matching is found. I don't know how your scoring system works... Would it be possible to exclude results where the score is below some threshold? Would you see that as a feature, as I do?
Keep up the good work!
I forgot to say: did you see where neither Google nor Duck Duck Go returned the link I wanted as the first result? This is an opportunity for your search engine to exceed their capabilities.
@daveloyall The query
def con 26 badge ama
consists of too many short words for Google, Bing, DDG and (currently) DYGG to find any relevant answers. In search, especially in search over web-sized data, short words are a nightmare. Because it is hard to encode their syntactic identity.
That is why web search engines cannot rely solely on syntactic similarity. We must also look at the semantic relevance.
If a search engine has weak semantic abilities, it must perform a second query:
select favorite_searchwords from user_data where id = 'imtheproduct@gmail.com'
in order to find real relevance.
Google does this. Bing also. Because they have trouble encoding semantic meaning into their web index. DYGG however has no user data to query because users are not a product. Search is.
I will leave this issue open until I've sorted out semantic analysis.
Search result are acctually quite excellent.