Performance Issues with List Users

[amcdnl]

Issue

When running the following command:

await oktaClient.listUsers().each((user: any) => {
   items.push(user);
});

against an Okta instance that has 800+ users, this operation takes about 52s. If I take that same query and run it directly through the REST API like:

GET https://myaccount.okta.com/api/v1/users

This returns in roughly 661ms.

I realize that this is only returning a subset and not paging, however, if you say at 200 items per query * 4 queries thats still only roughly 4s ( not 52s ).

Furthermore I added some instrumentation that Datadog reports that the okta lambda is making roughly 881 HTTP GET calls!

This makes me assume that its running these individually?

When you combine this number of calls with multiple users all making similar calls I'm also getting rate limited.

Versions

    "@okta/okta-sdk-nodejs": "^4.6.0"

[shuowu]

@amcdnl Thanks for reporting the issue!

Internal Ref: OKTA-404869

[amcdnl]

Any updates?

[oleksandrpravosudko-okta]

@amcdnl not at the moment, unfortunately.
OKTA-404869 is being worked on in the current sprint - I will report back once updates are available.

[oleksandrpravosudko-okta]

@amcdnl @apexskier does the issue manifest itself in a non-Lambda env? Are there any RequestExecutor customizations in place? Would you be able to add request logging for debugging purposes?

Unwanted requests didn't seem to occur in my repro setup (both local and Lamda, SSWS/OAuth2, ~1500 dummy users, node 12/14).

Cloudwatch events

[amcdnl]

@oleksandrpravosudko-okta - Here is a screenshot of the logs using a custom requester with logging ( some are cut off ).

and here is the span list showing 44 HTTP requests to Okta API taking around 44s ( and timing out due to Lambda timeout restrictions ).

This okta instance has roughly 800 users and the code is extremely simple:

    const opts = query
      ? { q: query }
      : {};
    
    const collection = await client.listUsers(opts);

    await collection.each((user) => users.push(user));

    users.sort((a, b) => (a.profile.email > b.profile.email) ? 1 : -1);

    return { success: true, data: users };

[oleksandrpravosudko-okta]

Thanks for sharing more details @amcdnl.

I have created a sandbox with several endpoints retrieving users with either no SDK involvement (/users/raw) or using lower level methods (/users/httpClient, /users/sdkClient) - any chance you could test them against your org and see if any of them trigger request spikes?

[oleksandrpravosudko-okta]

Hi @amcdnl - have you been able to identify the cause/work around this issue?

[amcdnl]

@oleksandrpravosudko-okta - I was actually just getting back to looking into this. I'll let you know today.

[amcdnl]

OK - here are the results. Its got about 8k users so they all seem to align to the same response. I must have been doing something wrong in my tests that got me to this.

I did find one interesting aspect is that I'd expect the SDK to cache per the documentation on the subsequent requests ( I moved the client outside of the function scope ) and it does not seem to. Any suggestions?

Raw

SDK

HTTP

[oleksandrpravosudko-okta]

Thanks for running these tests!

It looks like collections requests are exempt from caching in default middleware: https://github.com/okta/okta-sdk-nodejs/blob/master/src/default-cache-middleware.js#L17

[amcdnl]

@oleksandrpravosudko-okta - Is there a sspecific reason for this? At minimal, this should be documented.

[oleksandrpravosudko-okta]

@amcdnl There seem to be no specific mention of this in past commits although my guess is that collections' requests are not cached because of pagination cursor opaqueness.

+1 on caching limitation description - I created a story to document that:
Internal ref: OKTA-419120

[oleksandrpravosudko-okta]

Note on collections caching: 42bef7c