Middleware for Downloader

[aaronflorey]

While having RequestMiddleware and ResponseMiddleware is awesome, what are your thoughts on a Downloader middleware that takes a request and returns a response?

An example use case is the JavascriptMiddleware, which could take the request and render the page, which then returns the response itself, instead of running it through the downloader and then overwriting the response. The other use case would be caching requests between runs.

I'm sure there's a reason why this isn't already a thing, so i was wondering what would need to be done to get this implemented? I'm happy to contribute to get it done

[ksassnowski]

That would be pretty tricky to implement because it would essentially break the entire downloader pipeline.

Right now, each request is run through each RequestMiddleware first, so that any middleware can alter or drop the request before it gets sent. Then, the response is passed through each ResponseMiddleware. Adding a kind of middleware that takes a request and immediately returns a response would completely break this pattern, unfortunately.

A possible way to deal with this could be, extending the Request class to allow setting a Response on the request. The downloader could then check if the request already has a response set and simply return that response instead of sending the request. I think this is similar to how Symfony's kernel events work, too.

You could then implement a caching middleware by simply setting the cached response on the request inside a regular RequestMiddleware. The hypothetical API could look something like this:

class CachingMiddleware implements RequestMiddlewareInterface
{
    public function handleRequest(Request $request): Request
    {
        // assuming the cache returns `null` if no cached response exists, it's safe
        // to always set the result of `forRequest` on the request.
        $cachedResponse = $this->cache->forRequest($request);
        
        return $request->setResponse($cachedResponse);
    }
}

This would still allow the other middlewares to run as usual which is something I definitely want to preserve.

[aaronflorey]

That's a great way of doing it, i'll work on adding that, Thanks!

[noplanman]

@aaronflorey @ksassnowski Do you have a more complete working version of this caching middleware?

I've not been able to get it to work and would appreciate your help, thanks.