harp.gl should support rate limiting and not just return blank tiles

Question

harp.gl should support rate limiting and not just return blank tiles

Closed this issue 3 years ago · 21 comments

When looking at the horizon & or moving the camera quite fast I get 100s of errors like that below. Parts of the map stop loading and there are chunks left empty. This never used to be a problem in earlier versions (0.21.1) for example.

Steps to reproduce:

Download & extract zip
Run 'yarn'
Enter your API key on Line 29 in 'pages/index.tsx'
Run 'yarn run dev'
Look at the horizon & move the camera around

I'd expect the tiles to load slower but not at the rate where your API is refusing requests.

OS: Window 10
Browser Chrome
Version 91.0.4472.106

Example: harp-bug.zip

Answer 1 · 2021-06-22T08:36:38.000Z

@harrycollin , thanks for the bug ticket.

I guess you are on the freemium account? Do you have many users using this particular key? I notice you put it in the screenshot, it means that other people can use and also potentially misuse.

Have you looked in your account to see if there are any further details.

Unfortunately, if you get a 429, there isn't much we can do on our side, as the service is causing the issue.

Hope that helps,

Answer 2 · 2021-06-22T08:42:52.000Z

@nzjony thanks for the reply. This is a new API key that I made specifically just for this project. I don't this this is misuse, but I'll check anyway

Answer 3 · 2021-06-22T09:21:36.000Z

@harrycollin , ok, weird. Try to create a new API key and let me know if that makes any difference?

There could be some problem locally, but I somewhat doubt it.

Do you have a globe projection with custom far plane? Potentially you are downloading too many tiles and the server returns the 429, but I somewhat doubt it.

Answer 4 · 2021-06-22T09:35:09.000Z

@nzjony I made a new API key and still have the same problem. I don't have any exotic setup, just flat projection & MapControls controlling the camera. I could try making a new account to & generate a new API key that way but I don't see why it would make a difference.

Answer 5 · 2021-06-22T09:35:48.000Z

Did you manage to test the project I attached?

Answer 6 · 2021-06-22T11:21:36.000Z

@harrycollin , thanks for trying a new API key. Ok, weird.

Can you check in your developer network tab and see what value is given for: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Retry-After ?

I will try to test the product you have attached and get back to you.

Answer 7 · 2021-06-22T12:38:46.000Z

@nzjony It's 10

Answer 8 · 2021-06-22T12:54:17.000Z

@harrycollin , I didn't get 429, but it does download many tiles.

See:
https://user-images.githubusercontent.com/2533428/122927829-8b7a3180-d369-11eb-9038-60371b8a24ce.mp4

Can you try reseting your router to get a new IP address?

Answer 9 · 2021-06-22T13:00:45.000Z

@harrycollin , can you try looking in your account to see if you have run out of the free amount of requests?

Answer 10 · 2021-06-22T13:03:14.000Z

@harrycollin , I didn't get 429, but it does download many tiles.

See:
https://user-images.githubusercontent.com/2533428/122927829-8b7a3180-d369-11eb-9038-60371b8a24ce.mp4

Can you try reseting your router to get a new IP address?

My public IP address won't get changed unfortunately.

Answer 11 · 2021-06-22T13:04:57.000Z

Btw, you can also check an online demo here: https://cassini-hackathon-parametricos.vercel.app/
I participated in a hackathon over the weekend with my company. This is what we put together.

Answer 12 · 2021-06-22T13:06:16.000Z

@harrycollin , can you try looking in your account to see if you have run out of the free amount of requests?
We're well within our limits:

Answer 13 · 2021-06-22T13:06:53.000Z

@harrycollin , I get a 429 on that link.

Something weird is going on...

Answer 14 · 2021-06-22T13:07:20.000Z

Ahhh now it works....

Answer 15 · 2021-06-22T13:10:44.000Z

Yep something weird indeed.

Answer 16 · 2021-06-22T13:21:42.000Z

@harrycollin , I wrote to you on slack.

Answer 17 · 2021-06-23T14:59:33.000Z

The rate limit from Fermium accounts is 100 requests within 10 seconds, as it says in the headers you posted. This should be enough for panning. @nzjony for tilted maps this could be a problem since many tiles are requested at the same time.

Answer 18 · 2021-06-23T15:20:00.000Z

@pakore , thank you!!

Ok, that explains it @harrycollin.

You can then:

upgrade to a paid account
try specifying a different value for: https://www.harp.gl/docs/master/doc/interfaces/harp_mapview.datasourceoptions-1.html#storageleveloffset when creating the DataSource, this will request lower detail tiles (i.e. you will request less tiles), but the map will look worse.
You could try enabling: https://www.harp.gl/docs/master/doc/interfaces/harp_mapview.mapviewoptions-1.html#enablemixedlod (but I'm not 100% sure that this works in flat projections).
Wait for us to implement HARP-15979 (support rate limiting in harp.gl)

Answer 19 · 2021-06-23T18:58:21.000Z

@harrycollin , because the rate limiting issue is clear, I have changed the title to reflect the fact that harp.gl doesn't work with rate limiting.

However we are quite full at the moment working on internal tickets, so it may take some time to work on this, but I hope we can prioritise it, because it will also affect our paying customers.

Answer 20 · 2021-06-23T19:56:24.000Z

That's fine. At least we found the source of of the issue. Thank you for investigating :)

Answer 21 · 2021-08-28T16:55:24.000Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.