get started: microservice to communicate with IPDB
Closed this issue · 6 comments
We need to get API keys out of site's js. To prevent excessive usage by 3rd parties leading to hitting the API rate limit on IPDB. The tool would stop working for everyone then.
There's the possibility of doing nothing, launching the page, shouting YOLO, and then start creating the microservice. This could create unwanted pressure though, depending on how quick our visitors figure out how to spam IPDB with the hardcoded API credentials.
The page is super valuable even without transaction tool. Releasing the page ASAP should have priority so leaving out the transaction tool section would be possible too, but only as last resort. Or just remove the tool only once we actually have a problem.
@tim, @vrde and me decided to solve the issue short term with a proxy in form of a HTTP microservice the site will communicate with. Microservice will handle the request & response to IPDB, with the API keys probably sourced from environment variables.
Microservice scope
-
proxy injecting API credentials
-
deploy to somewhere
-
switch API url in
page-getstarted.js -
remove credentials from the PR with a rebase or something (so they won't be in merged master either)
-
[ ] create a valid transaction into IPDB -
[ ] receive a user addedmessagefrom the site and include in transaction creation -
[ ] get status of a transaction in IPDB -
[ ] serve an API endpoint for creating a transaction -
[ ] serve an API endpoint for getting a transaction via its ID
Our existing microservices can be used as starting point:
Hosting
Above projects are Node.js apps, simply using https://github.com/zeit/micro to create the service, hosted on https://zeit.co under bigchaindb team space I can invite everyone into. We're on OSS plan on there which allows 3 instances, currently using 2.
But we could put it wherever, but I doubt we can beat the minimal Zeit deployment
Site adjustments
Needs to be updated with new endpoint and all the driver-related code can be removed. To be replaced with simple fetch calls to microservice for creating a transaction and getting a transaction.
Above file only gets loaded on Get started page so no need to worry about other places. Be warned about js on the site though.
also dialing in @krish7919 cause I have no idea if there's something needed on the IPDB/3scale/nginx proxy side of things to get this to work
To prevent excessive usage by 3rd parties leading to hitting the API rate limit on IPDB. The tool would stop working for everyone then.
You will get an explicit HTTP status code 429 if you are being throttled.
I have no idea if there's something needed on the IPDB/3scale/nginx proxy side of things to get this to work
This service is just a user in terms of BigchainDB/IPDB. Once you get an app_id and app_key for IPDB, everything should work as usual; the deployment is agnostic to the application.
This service is just a user in terms of BigchainDB/IPDB. Once you get an app_id and app_key for IPDB, everything should work as usual; the deployment is agnostic to the application.
What if we have more keys and we rotate them?
@vrde The throttling is based on IP addresses, not on app_id and app_key. Basically,
3scale takes care of authentication and authorization, and
nginx takes care that you don't use all your requests in one go.
As a hypothetical example, if one subscribes to a 1TB plan on 3scale, the app_id and app_key will allow you to use 1TB. But if you decide to use all the 1Tb within 5 minutes, that will be handled by nginx.
And the numbers limiting it to 10 req/s (bursts up to 20 reqs/s) are there today as I had to put some number in there during development, and there were many unknowns.
Now, we can change the numbers if we have any data point or any reason to believe that this is too low (or too high).
Now, we can change the numbers if we have any data point or any reason to believe that this is too low (or too high).
It's too slow for @vrde to upload claims for whereonthe.net/DACS.
Let's significantly decrease the rate limiting for authenticated users (or even remove it completely for them).
ok, screw the complicated scope, @vrde knows what's up https://github.com/bigchaindb/ipdb-proxy