Provide local Search API option
miurla opened this issue · 6 comments
Feature Description
As an alternative to the Tavily search API, use SearcXNG to provide a local search API. Since SearcXNG returns meta-search results, make it usable from LLM by embedding.
Use Case
Use it as an option for the search API when launching locally. It has advantages in terms of cost and privacy.
Additional context
https://github.com/Snazzah/duck-duck-scrape
Possibly DDG support?
Recently tried to get data from this endpoint and our current search providers returned an error:
The site is accessible but does not have https so it gets flagged as unsafe.
Would be great to have control all the way down to the web scrapper.
What kind of data do you expect from it? For instance, this is how you'd call the API, with useful configurable options such as specifying engines or categories (useful for targeted research like academic or social media focused). Is this any different, broadly speaking, than what you receive with Tavily or Serper?
