Batch opening and closing gateways?

[mittster]

Hello sir, very nice module :)

I am trying to make it nicer by speeding up closing and opening of gateways. Current implementation uses:

apis = awsclient.get_rest_apis()["items"] // to get the ids to close and, lists max 20
awsclient.delete_rest_api(restApiId=api["id"]) // to actually close it

Which works fine, unless you have to close 600 gateways which easily takes half an hour and you have to recursively call get_rest_apis(), because it lists only 20

The same problem happens with opening gateways. Opening gateways for 10 domains takes around 30s. I am attaching the relevant code from the project:

    `# Create simple rest API resource
    create_api_response = awsclient.create_rest_api(
        name=self.api_name,
        endpointConfiguration={
            "types": [
                "REGIONAL",
            ]
        }
    )

    # Get ID for new resource
    get_resource_response = awsclient.get_resources(
        restApiId=create_api_response["id"]
    )
    rest_api_id = create_api_response["id"]

    # Create "Resource" (wildcard proxy path)
    create_resource_response = awsclient.create_resource(
        restApiId=create_api_response["id"],
        parentId=get_resource_response["items"][0]["id"],
        pathPart="{proxy+}"
    )

    # Allow all methods to new resource
    awsclient.put_method(
        restApiId=create_api_response["id"],
        resourceId=get_resource_response["items"][0]["id"],
        httpMethod="ANY",
        authorizationType="NONE",
        requestParameters={
            "method.request.path.proxy": True,
            "method.request.header.X-My-X-Forwarded-For": True
        }
    )

    # Make new resource route traffic to new host
    awsclient.put_integration(
        restApiId=create_api_response["id"],
        resourceId=get_resource_response["items"][0]["id"],
        type="HTTP_PROXY",
        httpMethod="ANY",
        integrationHttpMethod="ANY",
        uri=self.site,
        connectionType="INTERNET",
        requestParameters={
            "integration.request.path.proxy": "method.request.path.proxy",
            "integration.request.header.X-Forwarded-For": "method.request.header.X-My-X-Forwarded-For"
        }
    )

    awsclient.put_method(
        restApiId=create_api_response["id"],
        resourceId=create_resource_response["id"],
        httpMethod="ANY",
        authorizationType="NONE",
        requestParameters={
            "method.request.path.proxy": True,
            "method.request.header.X-My-X-Forwarded-For": True
        }
    )

    awsclient.put_integration(
        restApiId=create_api_response["id"],
        resourceId=create_resource_response["id"],
        type="HTTP_PROXY",
        httpMethod="ANY",
        integrationHttpMethod="ANY",
        uri=f"{self.site}/{{proxy}}",
        connectionType="INTERNET",
        requestParameters={
            "integration.request.path.proxy": "method.request.path.proxy",
            "integration.request.header.X-Forwarded-For": "method.request.header.X-My-X-Forwarded-For"
        }
    )

    # Creates deployment resource, so that our API to be callable
    awsclient.create_deployment(
        restApiId=rest_api_id,
        stageName="ProxyStage"
    )

    # Return endpoint name and whether it show it is newly created
    return {
        "success": True,
        "endpoint": f"{rest_api_id}.execute-api.{region}.amazonaws.com",
        "new": True
    }`

Is it possible to perform opening and closing tasks in batch?

[Ge0rg3]

H! Thank you for raising the issue. I checked the docs but couldn't see any sort of bulk delete operation we could use instead. However, it is currently threading with 10 threads, maybe you can try increasing the thread count manually?

[mittster]

I checked them too, couldn't find anything on topic. The error that I am getting is "Too many requests" so even more threads won't help. I tried with multiple accounts but apparently aws operation speed limit system is also IP based so that didn't make a noticeable difference

[mittster]

Since there will be no luck with batch requests, I'll explore if there is a way to make and configure aws gateway resource and method in a single operation instead of multiple calls. The web browser interface offers an option to import swagger API. Haven't found an option to export the configured gateway so far..

[Ge0rg3]

Hi @mittster ! Did you have any progress on this? Thanks 😄

[mittster]

Unfortunately no. I did manage to import the configuration but it wasn't any faster.