rasa/scoop-directory

Implementing Search

rashil2000 opened this issue · 4 comments

rashil2000 commented

There's still no usable online search index for Scoop, and I'd like to change that. Both scoop.netlify.app/apps and scoop-docs.now.sh/apps are non-functional.

Since scoop-directory already creates a very usable index of all the manifests, let's not reinvent the wheel and use that index itself.

The first step is creating a SQLite database, which has already been done here #19. However, it is hosted in a different place and contains only these fields:

  "name" TEXT,
  "version" TEXT,
  "description" TEXT,
  "license" TEXT,
  "bucket_repo" TEXT

Some more fields will be required, like homepage, link to manifest file etc.

For a proof of concept implementation, I have created this simple HTML file which use SQL.js JavaScript library to run queries in the scoop-directory -
poc.txt
(simply rename the file to HTML and open it, no install required).

This works, but there are some things that need to be done to make this seamless:

  • Currently, for an app, both the name and version properties hyperlink to the homepage.
    image
    It'd be great if one of them (preferably version) points to the manifest URL instead.
  • The SQLite database is currently hosted in a different repo. With some changes to maintenance/github-crawler.py, we can have the database file (around 1.98MB file) here in this repo itself. This way, both the markdown files and database file will get updated when the scheduled GitHub Action runs. We can also add more fields into the database this way.
  • A search page needs to be created, with populate-as-you-type results fitting the query.

If this looks good to go, I'd like to work on this (though I'll need help, I've never worked with Python before).

Cheers

rashil2000 commented

search.txt

I have created an updated page based on the website's theme.

(GitHub won't let me upload HTML files, so again just rename from .txt to .html and open in browser - no install required).

rashil2000 commented

https://rashil2000.github.io/scoop-directory/search

The search page is now live on my fork

rasa commented

Untested, but this might prove faster/simpler and automatically remove dups:

SELECT name,version,bucket_url,description,manifest_url,homepage,license
FROM apps
WHERE name LIKE "%${query}%"
UNION
SELECT name,version,bucket_url,description,manifest_url,homepage,license
FROM apps
WHERE description LIKE "%${query}%"
ORDER BY 
CASE
  WHEN name LIKE "${query}" OR description LIKE "${query}" THEN 1
  WHEN name LIKE "${query}%" OR description LIKE "${query}%" THEN 2
  WHEN name LIKE "%${query}" OR description LIKE "%${query}" THEN 3
  ELSE 4
END
rashil2000 commented

It gives a syntax error, but I got the gist and did something similar to combine them into a single query.

  • 👍1