Initial database pull freezes and UI doesn't have the ability to re-fill gaps
vtpgit opened this issue · 6 comments
First, great job, this is really very nice work, definitely very useful.
I'm using the web-version at yaqwsx.github.io/jlcparts
Not sure if it's my problem, however, the initial database pull for in-stock query for many of the parts never finishes, the UI hangs and upon refresh the UI doesn't give me the opportunity to re-query the missing in-stock items, therefore the in-stock is blank for these.
Then, the top banner always says "There is an update of the component library available." however when I click on the green button it does nothing. I'm using Chromium. I tried multiple times, at different times of the day or week, and from different locations. Thanks for your attention and good work.
This seems to be a problem with empty categories - however/fortunately, it should not have an impact on the functionality of the database. It will just always show "there is an update available". We should filter those categories out.
Or is there any other problem I am missing?
About slowness; we are currently reaching the limits of IndexedDB. Since the project started, the JLC PCB offer got 20x bigger. There are already ideas on resolving this, however, they won't be implemented in the near future.
The real problem is that the queries for in-stock items never finish. And when I refresh I don't get the ability to re-query the items, the screen goes away from the component query page and I need to clear the database, and re-query, which goes back to hanging.
On the database, why don't you implement using a different small database, such as SQLite. I've used SQLite before, and it may be the answer for a database. I am willing to commit some of my time to help. I really need this because JLC's component library is becoming incredibly difficult to use, and they don't seem to care or have any plans about improving it. I've got extensive full-stack design/dev experience in very large scale (multi-terabyte) database projects. This should be a nice small useful project to work on the side.
The reason there is no server-side database is simple - I don't want to maintain and host any infrastructure. Back in the time, when there was ~20k component the solution with IndexedDB worked nicely. Now it is reaching the limits (after today's update there are near 500k components). Of course, a simple backend powered by PostgreSQL or any other database would work. But that was not the aim at the beginning of the project.
Feel free to fork the project and transform it into a solution with a proper backend. If you build such a solution, please let me know!
PS: Regarding the serverless architecture; I was considering #37, but I have never fully explored it nor verified it will scale better.
Thank you. This is very helpful. On SQLite, I've used it extensively with Java, the performance up to a few million properly indexed rows is very satisfactory. I used it for live financial data with heavy real-time transactions. Next would be MariaDB, which is not my favorite, but scales up well to very large volumes, practically no limit. The extreme, best option IMO, would be Oracle Express, which has open commercial licensing with no restrictions for databases up to about 10GB. That's probably corresponding to 1 million components, assuming 1kb/component, looking at the data structures, without compression. With compression it could be more. I'd stay away from Oracle however as much as possible for such a project.
As far as off-line back-end setup, I could dedicate a "cloud" server.
Question. Is the current source code including the part where it calls JLC and LCSC directly to build the database, or only queries a pre-built by you database? In other words, is this self-sufficient or it relies on an intermediary pre-scraped database? I see in the instructions that you need to download a zipped database for the UI to work. Do I need an API subscription to JLC or is the API open?
The most added value in this project is the part where JLC PCB and LCSC sites are scraped and the data are post-processed in order to turn them into a normal form (e.g., convert resistance into numbers so you can sort it, break composite fields into separate parts, so you can select by which component to sort). Everything works with public APIs; however, you probably don't want to scrape it too often.
BTW: The problem with update toolbar still showing there is an update should be fixed.