OP-Engineering/link-preview-js

It doesn't pass robot protection on some websites

brootle opened this issue · 2 comments

Describe the bug
It doesn't pass robot protection on some websites. Here is a good example https://www.bloomberg.com/news/articles/2022-05-17/elon-musk-says-twitter-must-prove-bot-claims-for-deal-to-proceed

Here is what I am getting in response, see "Bloomberg - Are you a robot?"

{ "data": { "url": "https://www.bloomberg.com/tosv2.html?vid=&uuid=a0651c3d-d82c-11ec-bb03-4a7a46654c6c&url=L25ld3MvYXJ0aWNsZXMvMjAyMi0wNS0xNy9lbG9uLW11c2stc2F5cy10d2l0dGVyLW11c3QtcHJvdmUtYm90LWNsYWltcy1mb3ItZGVhbC10by1wcm9jZWVk", "title": "Bloomberg - Are you a robot?", "mediaType": "website", "contentType": "text/html", "images": [], "videos": [], "favicons": [ "https://www.bloomberg.com/favicon.ico" ] } }

To Reproduce
Steps to reproduce the behavior:
Just pass this url https://www.bloomberg.com/news/articles/2022-05-17/elon-musk-says-twitter-must-prove-bot-claims-for-deal-to-proceed to getLinkPreview function

Expected behavior
Expecting to get same results as Twitter, Facebook or Google news gets
image

Facebook, Google, and Twitter are probably trusted domains by Bloomberg. Nothing this library can do about it.

yeah, I think that's the case, thanks!