Notice of Awair developer API increase
anthonycorbacho opened this issue · 19 comments
Describe Your Problem:
We are seeing a massive increase of API requests to our developer-apis.awair.is.
We are wondering why this is happening as it looks like something is continuously triggering a sort of retry?
We contacted the users that are triggering the high volume of requests and they are all using this plugin.
Is there anything that we can do to understand why this is happening?
Screenshots:
Hello Anthony,
I submitted a report to hello@getawair.com on August 1 that the plugin running on my system was receiving HTTP 500 errors to check asking if there was an issue on the Awair side. This coincides with the large increase that Awair observed on their servers. Will need to look into how axios behaves in this situation and whether it was in a retry loop which would explain the increase.
Interesting on how the calls phased between regions and then suddenly decreased to normal level. Has the API requests returned to a normal level in all of the regions?
Will look into axios behavior and how to prevent in the future. I will get back to you in the next few days.
Regards, Doug
Hi Doug,
Yes, we did release a new update of the developer-apis, and somehow this update trigger a cascading effect that we did not expect, we have fixed it and we are trying to understand why this happen.
No payload or error code changed.
We are a bit worried about the scripts/apps that are retrying forever because this is out of our hands and the only action we can take to save the rest of the users is to actually block, it sucks but we are actively trying to find a solution. but we are afraid that a user action might be required (e.g. restart the script)
Hello Anthony,
Looking at my logs, the problem that occurred on August 1 is a little puzzling. Requests for account UserInfo and Devices responded correctly. As soon as Device data was requested there was an immediate rejection by the Awair servers. This was not logged as an error by Axios. The calls looped over and over in a very short period of time. Eventually there was code 400 and 500 errors followed by ECONNRESET errors. This was probably the big increase that was observed on your end.
I am investigating axios options that would check server status before executing the GET request. In the process I noticed my account tier was reduced from "Large Developer" to "Hobbyist". Any chance you could bump back up so that I do not run into limits while investigating? I do my tinkering on a second instance of Homebridge and why I could exceed limits.
Thanks,
Doug
Hello @anthonycorbacho,
I have released v5.9.9 with axios check of Awair server status before axios.get call. Also added additional logging detail. Please let me know if this seems to be a reasonable correction to address the problem which you reported.
Thanks,
Doug
Closing for now as v5.9.9 has been released. Please reopen if additional heavy traffic detected on Awair server side.
Sorry @DMBlakeley, I was having a long weekend.
Looking at my logs, the problem that occurred on August 1 is a little puzzling. Requests for account UserInfo and Devices responded correctly. As soon as Device data was requested there was an immediate rejection by the Awair servers. This was not logged as an error by Axios. The calls looped over and over in a very short period of time. Eventually there was code 400 and 500 errors followed by ECONNRESET errors. This was probably the big increase that was observed on your end.
That makes sense, since then we have found cause and pushed fixes, the traffic went back to normal after that.
I am investigating axios options that would check server status before executing the GET request. In the process I noticed my account tier was reduced from "Large Developer" to "Hobbyist". Any chance you could bump back up so that I do not run into limits while investigating? I do my tinkering on a second instance of Homebridge and why I could exceed limits.
We are going to release an update this week that re-enables tiers. I will let you know when it's ready again.
Thanks.
@anthonycorbacho, no problem, long weekends are allowed!
I have released v5.9.9 of the plug-in which I believe should help but of course all users need to update to this version.
@DMBlakeley , thank you so much for the quick fix, we are closely monitoring the infra, so far we are good.
We will let you know if something funky is happening again.
@DMBlakeley
we re-enable Tier functionalities, please let us know if you have any issues.
@anthonycorbacho, I have confirmed that the Tier level for my account has been restored to Large Developer.
Thanks, Doug
@anthonycorbacho, I recently received the notification from Awair of sunsetting of the Awair v1, Glow and Glow-C. I have a couple of questions so that I can update this plugin to sync with this changes.
Questions:
- Is Awair r2 still being supported. The announcement states Awair v1 but the pictures show an Awair r2.
- The plugin also provides support for Awair Mint. Should I maintain or drop.
- I note that the Awair iOS app has been updated to remove the "Developer Option" from the Settings page. This eliminates the ability to get the Developer token via the app which is OK as it can be obtained through the webpage. Unfortunately, this was the way to enable to Local Access. Does Awair plan on restoring this function in a future iOS update? I personally make use of this capability to put my Awair r2 in "Clock" mode at night and back to "Score" mode in the morning.
Thanks in advance for your help,
Doug
- Yes awair-r2 is still supported
- awair-mint is still supported as well
- the option should be in the device > awair+ > API section. note that the local API (if available should be enabled before the sunsetting), after the 30th, we will not allow sunset devices to access the API.
Let me know if you have more questions :)
Thanks for the clarification on where to find Local API in the iOS app. I know for a fact that the Awair v1, Glow and Glow-C do not support Local API so not an issue for the Awair2 plug-in after sunsetting.
I recently had a user who did not enter the Developer Token correctly and the plugin just kept restarting. I found that I check for no Token but do not check that it is a valid Token. I found that my token is 107 characters in length. Is this true for all tokens? I checking for token length reasonable or is there a better indication that you can suggest.
Thanks!
Hi @DMBlakeley
It's kinda hard to predict the length of the token since it will depend on the user id (its a jwt if I recall correctly).
But usually it should be a decent size (>100 chars)
@anthonycorbacho, thanks this gives me a basic check. Yes, I found it was a JWT so I can just check that the first part (before the period) is correct.
I had an issue submitted from a new user with an Element. After going back and forth and looking at his logs I found that he was getting a "Tier was not found for user 141700" with the Developer token that was issued. Suggested he submit a ticket through the Awair app or email support. At present he has not received a response after a couple of days. Is there any other channel to get help?
@anthonycorbacho, hoping that you can help me out to close a user issue 426? Requoting from above as the user has not received a response from Awair support in several weeks.
I had an issue submitted from a new user with an Element. After going back and forth and looking at his logs I found that he was getting a "Tier was not found for user 141700" with the Developer token that was issued. Suggested he submit a ticket through the Awair app or email support. At present he has not received a response after a couple of days. Is there any other channel to get help?
Thanks in advance for any help you can provide.
Hi,
Sorry I forgot to reply. But we released a patch few weeks ago and it should be working.
If not, I will ask my team to take a look and let you know the progress
Thanks!