IMS client script fails every 2 or so days, but requests run fine on days that it succeeds.
Closed this issue · 10 comments
I've been using the adobe analytics api authentication template for python for a couple months now and before now, it has worked like a charm. However, in the last week, I've started to experience authentication issues that lead to a days worth of missing data in my data flow (Causing a real nuisance for things down the line of that pipeline). It seems to be that when the function "get_first_global_company_id" in the ims_client.py file gets called, the response fails to be parsed and returned because the object isn't subscriptable because it is "Nonetype". I'm guessing the API request fails which leads to a Nonetype object, but I don't quite understand why it would fail (it's just trying to get data for the company account, and it runs perfectly most days). This authentication is embedded in azure databricks and is scheduled to run each day. I'm going to try and see what the response code is for when the authentication fails, but I suspect that rerunning the client today will just result in a successful run (That's what happened last week). So I'm going to put some code in that will log the request status code so that next time it fails, I can see what the api is doing. I'll also attach photos of the traceback.
I believe that when I run my databricks script each day, it actually re-authenticates by creating a new JWT token each time, so I don't believe expiration should be an issue, but the script has failed after two successful days of running twice in a row now. I'll attach a photo of the success/fail frequency as well.
This may very well be something small that I have introduced and I am just yet to find it, but I have been pretty thorough in looking and I wanted to get this issue in sooner than later, so if I do end up relying on you guys for an answer, it isn't being looked at for the first time a week from now after I tried everything I could possibly think of. This is also my first time posting an issue to github so I've done my best to follow the guidelines, but without experience, it could be possible that I approached this improperly, so I apologize if that is the case, I'm just trying to get to the bottom of this! Thank you in advance and let me know if I can provide you with any other information.
@evanashbrook - If you could provide us with the request ids and a timestamp of the failed API requests we can get this issue routed to the correct team to look in to the issue.
The process hasn't failed in the last six days so I'm less concerned than I was, but I'm still going to try and get you those ids and timestamps just in case it's still vulnerable to failing due to the same issue. I'm guessing those are located in the response object headers? I've got my script set up so that if it does fail again, I'll be able to see the status code, the headers, and the json output, so if the ids and timestamps are among that information, I will catch it.
Also, thank you for the quick response.
@evanashbrook - The request ids should come back as response headers from the API requests.
Ok so I just got another failure today, and I got everything you asked for. The error message is saying "Too many requests", which doesn't make too much sense to me because isn't it just making enough requests to authenticate? Here are the response headers which include the request ID and the timestamp.
Response Headers = {'Server': 'openresty', 'Date': 'Tue, 26 Jul 2022 05:05:18 GMT', 'Content-Type': 'application/json', 'Transfer-Encoding': 'chunked', 'Connection': 'keep-alive', 'Access-Control-Expose-Headers': 'Date', 'Access-Control-Allow-Methods': 'GET, POST, PATCH, PUT, DELETE, OPTIONS', 'Access-Control-Allow-Origin': '*', 'Access-Control-Allow-Credentials': 'true', 'Access-Control-Allow-Headers': 'Content-Type, X-Requested-With, accept, Origin, Access-Control-Request-Method, Access-Control-Request-Headers, api-key, Authorization, x-proxy-company, x-proxy-company-id, x-proxy-global-company-id, x-proxy-username, x-proxy-userid, x-request-id, x-api-key', 'Access-Control-Max-Age': '60', 'X-Request-Id': '4mh9cRZtCHZhXlRYpxxpC3ulHOZe7pTO', 'Retry-After': '2'}
And here is the general response message:
Response = {'error_code': '429050', 'message': 'Too many requests'}
This time there were 9 successful days of runs in a row before it failed, which seems even more random. Let me know if you guys come up with anything and thanks for looking into it.
@evanashbrook - It sounds like you are hitting the rate limits for your client. https://developer.adobe.com/analytics-apis/docs/2.0/guides/faq/#what-is-the-rate-limit-for-api-calls
You should be able to make about 120 requests per minute but the rate limit is actually enforced as 12 requests every 6 seconds so if you are making a lot of requests in a very short period of time you can get these responses.
You could add caching to any of the API calls that make sense to cache. You can also try to spread out your other requests over a longer period of time if you are making a lot of requests at once.
@mike2184 I'm not sure how it would be possible for us to hit the rate limit while we are in the process of authenticating. And in that process, there seem to only be 3 api calls, of which the 2nd is failing. Further, this is the only script using the credentials from the project we setup in adobe console, so I know of nothing else making any more api calls. Trying to understand why we would get this response. I even put in a try-except-finally clause that pauses for 30 seconds when the call fails before it tries again, and I still got the too many requests error, so we are a little stumped at this point. Please let me know if you guys think of anything. Thank you in advance.
@evanashbrook - I will reach out to the team that handles authentication to see if they can shed some light on why you would be getting a rate limiting response for those requests.
Thanks for your help @mike2184.
Hey @mike2184 , I'm sure you're not pursuing this anymore but just for ease of mind, I was able to cut the errors by just not making the request that caused the errors. It turns out I don't actually need it for authentication. I'm going to close the ticket now.