becheran/mlc

crates.io links: 403 - Forbidden

Closed this issue · 4 comments

stanislav-tkach commented

Describe the bug
MLC reports that crates.io links could not be resolved. I'm not sure if it is related to #20, but this time the error is different: "403 - Forbidden" instead of "404 - Not Found". I'm not sure if this issue is on the MLC side.

To Reproduce
Steps to reproduce the behavior:

  1. Create a md file with any crates.io link, for example https://crates.io/crates/syn.
  2. Run mlc.
  3. See error:
[Err ] ./e.md (1, 1) => https://crates.io/crates/syn. 403 - Forbidden

Result (1 links):

OK       0
Skipped  0
Warnings 0
Errors   1


The following links could not be resolved:

./e.md (1, 1) => https://crates.io/crates/syn.

Expected behavior
I expect to see no errors.

Desktop (please complete the following information):

  • OS: Ubuntu 20.04.1 LTS
  • Browser Firefox 80.0.1 (64-bit)
  • Version 0.13.6

Additional context

becheran commented

It is different to #20. The Unit-Test that I have with the crates.io link is also failing now. I will further investigate and what happened and see how I can fix this.

  • 👍1
becheran commented

Seems like I need to set a user-agent for crates.io queries now. They changed the policy. Without the User-Agent field in the header the response is:
"We require that all requests include a User-Agent header. To allow us to determine the impact your bot has on our service, we ask that your user agent actually identify your bot, and not just report the HTTP client library you're using. Including contact information will also reduce the chance that we will need to take action against your bot."

I will update this and add some user-agent info like this:

mlc (github.com/becheran/mlc)

  • 👍1
becheran commented

Released version 0.13.7 includes a fix. Thanks for reporting!

  • 🎉1
stanislav-tkach commented

Thanks for the quick response!