great minds think alike 😄
DiegoFleitas opened this issue · 4 comments
I recently deployed something similar to this just to find this repo out!
I like the aproach you took with letterboxd scraping instead of using the export like I did & you might be interested in supporting other countries like I do
@DiegoFleitas Thanks for the call out! Great to see another approach to the same problem 😁 And I didn't know about fly.io, will give it a try sometime.
Appreciate the suggestion, supporting other countries is one of my remaining TODOs for this project. Curious how you're fetching movie posters (is that also exported and stored in redis?), as that's something I'm trying to display nicely as well.
fly.io is great! I Highly recommend it, they don't even charge you if your bill is less than 5 USD.
Regarding Letterboxd posters, those are loaded using AJAX & to get them I needed to make an extra request with a special "cache-busting" key that is found on the first request HTML. Also they seem to come in 125x187 & 230x345
I'm not caching posters but maybe I should!
My biggest issue right now is big watchlists, some people have like 1800+ movies on them & that's ~60+ pages.
That's just too much for my good ol server. I'm thinking of implementing a limit of 20 pages & force the user to infinite scroll if they want more
PS: You are welcome to try my demo if you haven't already
My approach was to simply have the user specify how many pages they want to scrape 😆 Not super convenient though, I like that on yours you just fetch everything automatically.
I did take a look at your site and love the UI, great work @DiegoFleitas 👏