philomena-dev/philomena

DeviantArt scraper no longer pulls the image

Opened this issue · 3 comments

Describe the bug
Current DA URLs don't seem to work anymore in the scraper. I've tried these two:
https://www.deviantart.com/rickgriffin/art/Housepets-Spa-Movie-Night-800928049
https://www.deviantart.com/rickgriffin/art/800928049
I checked the ex file and but seems to look for a url that ends with /art.

To Reproduce
Steps to reproduce the behavior:

  1. Go to Upload
  2. Paste a deviant art deviation URL.
  3. Watch as nothing happens.

Expected behavior
Thumbnail of submission appears in image preview and artist and description are pulled.

I had tried to fix the deviantart scraper but it fails to fetch images from images-wixmp-ed30a86b8c4ca887773594c2.wixmp.com due to some ssl issues. Appears to be elixir's problem, I did not figure it out.

Try it at https://derpibooru.org/images/new
https://images-wixmp-ed30a86b8c4ca887773594c2.wixmp.com/f/982c675c-20c2-4552-8800-7b6bf7b1b3e8/dh9v3q0-091931fd-d5b9-44af-94a8-df8ce59f0dc7.png?token=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJ1cm46YXBwOjdlMGQxODg5ODIyNjQzNzNhNWYwZDQxNWVhMGQyNmUwIiwiaXNzIjoidXJuOmFwcDo3ZTBkMTg4OTgyMjY0MzczYTVmMGQ0MTVlYTBkMjZlMCIsIm9iaiI6W1t7ImhlaWdodCI6Ijw9MTQ0MCIsInBhdGgiOiJcL2ZcLzk4MmM2NzVjLTIwYzItNDU1Mi04ODAwLTdiNmJmN2IxYjNlOFwvZGg5djNxMC0wOTE5MzFmZC1kNWI5LTQ0YWYtOTRhOC1kZjhjZTU5ZjBkYzcucG5nIiwid2lkdGgiOiI8PTI1NjAifV1dLCJhdWQiOlsidXJuOnNlcnZpY2U6aW1hZ2Uub3BlcmF0aW9ucyJdfQ.xkriM8CjntGHEt5p7RE11PECu-3SbRCWpTyyhYAsw1Y

@mdashlw you might need to disable middlebox compat mode in Mint

To get a Pixiv scraper working, I had to put in a referrer. Do you think adding a referrer of DeviantArt.com would fix?

From viewlogs running in prod
Screenshot_2024-04-26-00-26-37-33_d365b52accad0f47adbc08c16219827d.jpg