Add support for mocks-server being a proxy server

Question

Add support for mocks-server being a proxy server

stigkj opened this issue 2 years ago · 8 comments

Is your feature request related to a problem? Please describe.
When I have a lot of different endpoints I need to mock, it can be quite a bit of work setting up these to use the mocks-server's endpoint. In addition, when adding a new endpoint in the app under test, it is not easy to discover that this lacks mock setup.

Describe the solution you'd like
mocks-server could support being a proxy server, i.e. one would just need to set PROXY_HOST and PROXY_PORT (or set a global Agent), and all the HTTP requests from the app under test would go through mocks-server.

Answer 1 · 2022-12-22T07:10:58.000Z

Hi @stigkj , Mocks Server already can act as a proxy server. There is a "proxy" route variant that you can use in several ways in order to proxy some routes to other hosts.

Answer 2 · 2022-12-22T09:13:37.000Z

Hi @javierbrea, I saw that, but as far as I can see, that does not solve my problem. I'm talking about how I get my application to use Mocks Server, not about proxying some of the calls my application does to other hosts.

So, instead of configuring the HTTP client my application uses to point to Mocks Server for all the external endpoints my application uses, I would like to configure the HTTP client to use Mocks Server as an HTTP proxy. This would make all requests made with the HTTP client go through Mocks Server without changing any configuration. An example of another library supporting this: https://github.com/httptoolkit/mockttp/blob/main/docs/setup.md#local-nodejs-setup (see point 2).

Answer 3 · 2022-12-22T09:59:47.000Z

Now I understand what you want @stigkj , but I don't know how do you pretend to change all of the requests that your application do automatically to the Mocks Server url without changing any configuration on it.
In fact, in the example that you provided, I think that it specifically says that the client has to be configured:

Local Node.JS Setup
[....]
Direct your HTTP traffic through that server, by doing one of the below:
* Change your application's configuration to make requests to the mock server's URL (mockServer.url)
* Use env vars to set your proxy settings globally, if supported by your HTTP client: process.env = Object.assign(process.env, mockServer.proxyEnv)
* Use a specific setting to reconfigure your HTTP client of choice to use mockServer.proxyEnv.HTTP_PROXY as its proxy, if it doesn't automatically use proxy configuration from the environment.
[....]
Browser setup
[....]
Direct your HTTP traffic through that server, by doing one of the below:
* Change your application's configuration to make requests to the mock server's URL (mockServer.url)
* Using a fixed port for the mock server (mockServer.start(8080)) and configuring your test browser to use that as a proxy

Do you mean to implement an "interceptor" for possible clients that would change the originals requests url by the Mocks Server one? Something like the interceptors library for using the mswjs project in Node.js?

Answer 4 · 2023-01-02T15:18:44.000Z

Sorry for the late answer; was away on vacation :-)

You are correct that you would need to make a small change to get the application under test to use an HTTP proxy, but that would typically be changing the HTTP client globally, kind of like what the interceptors library you pointed to is doing. The difference to interceptors is that an HTTP proxy is a standard that can be used by all software adhering to that standard, i.e. if Mocks Server was an HTTP proxy you could use it to test curl by setting the environment variable http_proxy=localhost:<port of Mocks Server>.

Most of the HTTP clients in node can also be easily setup to use an HTTP proxy, for example by overriding the global http.Agent with https://www.npmjs.com/package/http-proxy-agent.

I see that this would require some other changes too, i.e. the routes in Mocks Server would need to support the hostname of the original request. That would be needed to be able to differentiate 2 requests with different hostnames but equal paths.

Answer 5 · 2023-01-03T13:22:39.000Z

Hi again @stigkj ,

I'm sorry, but I still don't understand what do you specifically mean by "if Mocks Server was an HTTP proxy", because in fact, you can implement an HTTP proxy by using the proxy-route-variant. You could even implement a route listening to all "paths", and then proxy the request to the original domain by using the options of the express-http-proxy package, which is used under the hood by the proxy route variant.

As far as I understand, the http_proxy=localhost:<port of Mocks Server> environment variable that you mentioned should be affecting only to the clients, am I right? Then, the clients would perform the request to the mock server, and... what do you need more? Maybe do you want a way to know the original url, and then proxy the request to the correspondent host? Is that information passed somehow in the requests when the http_proxy environment variable is enabled? Is it a part of the "standard" proxies specification? Could you give me more specific details about what is needed in mocks-server for it to be a "real HTTP proxy"?

About hostnames, it is something that could be implemented in mock-server, but for the moment you could define a global route middleware listening to all paths, and then, depending on the request host, you could proxy the request to other internal routes with a different prefix for each different host, for example.

Answer 6 · 2023-01-06T09:23:49.000Z

Hi @javierbrea,

Sorry for my bad explanation :-) I'll see if I can explain the standard HTTP proxy spec better using an example with curl:

First a regular HTTP request:

❯ curl -v ifconfig.co
*   Trying 172.64.111.32:80...
* Connected to ifconfig.co (172.64.111.32) port 80 (#0)
> GET / HTTP/1.1
> Host: ifconfig.co
> User-Agent: curl/7.77.0
> Accept: */*

And then an HTTP request that goes through a standard HTTP proxy:

❯ http_proxy=http://localhost:8000 curl -v ifconfig.co
* Uses proxy env variable http_proxy == 'http://localhost:8000'
*   Trying ::1:8000...
* Connected to localhost (::1) port 8000 (#0)
> GET http://ifconfig.co/ HTTP/1.1
> Host: ifconfig.co
> User-Agent: curl/7.77.0
> Accept: */*
> Proxy-Connection: Keep-Alive

The 2 big differences here are (a is without proxy, b is with proxy)

curl is connecting to
a. the site directly
b. to the HTTP proxy
the GET line has
a. only the path
b. the full URL

So, as you mentioned, for Mocks Server to be a real HTTP proxy, it would need to handle the GET line properly. And if I understood you correctly, that could actually be accomplished by an Express middleware which could look at the GET line (I guess by looking at the req object?) and from that setup some mocking?

Of course, if would be nice if this middleware was provided by Mocks Server, maybe as a plugin? Then one could just add that plugin and it would make it possible to use routes/variants/collections in the usual way. Maybe the URL field in routing must be changed to support a full URL and not just the path? Or maybe the plugin could rewrite the path to include the scheme and hostname as part of the path, i.e. https://example.com/some/path --> /https/example.com/some/path.

Answer 7 · 2023-01-06T09:28:55.000Z

For more in-depth coverage of the HTTP proxy spec, look here: https://www.ietf.org/rfc/rfc2068.txt (just search for proxy).

Answer 8 · 2023-01-08T11:42:59.000Z

Thank you @stigkj , now I understood better your requirements, and I have achieved to configure the server to proxy all requests to the original hosts, except the requests to the hosts that you want to mock, adding to them a prefix to the path, so you can use different prefixes in your routes for mocking requests to different hosts.

It can be achieved by using the proxy route variant in the next way:

We can add a "proxy" route handling all paths. Using a callback in the host option allow us to determine the host of the request, so we can decide if we let the request pass to the original host, or we proxy it again to the mock server.
Using the filter option of the proxy variant and the host header in the request, we can avoid proxying again the requests that are directly performed to the Mock Server, so, the requests that we previously redirected to the mock server won't be handled again by the proxy middleware and it won't produce an infinite cycle.
Using the proxyReqPathResolver option we can modify the path of the requests that we wan't to "intercept", so we can add a prefix to them (a prefix corresponding to the host name, for example, as you suggested)
Now we can add routes handling the paths of the hosts that we want to mock, simply adding to them the path prefix corresponding to its host.

Here you have an example of routes:

module.exports = [
  {
    id: "proxy",
    url: "*",
    method: "*",
    variants: [
      {
        id: "interceptor",
        type: "proxy",
        options: {
          host: (req) => {
            if (req.hostname === "jsonplaceholder.typicode.com") {
              return `http://127.0.0.1:3100`;
            }
            if (req.hostname === "dummyjson.com") {
              return `http://127.0.0.1:3100`;
            }
            return `${req.protocol}://${req.hostname}`;
          },
          options: {
            filter: function(req) {
              return req.headers.host !== "127.0.0.1:3100";
            },
            proxyReqPathResolver: function (req) {
              const parts = req.path.split('?');
              const queryString = parts[1] ? `?${queryString}` : '';
              const originalPath = parts[0];
              if (req.hostname === "jsonplaceholder.typicode.com") {
                return `http://127.0.0.1:3100/json-placeholder${originalPath}${queryString}`;
              }
              if (req.hostname === "dummyjson.com") {
                return `http://127.0.0.1:3100/dummy-json${originalPath}${queryString}`;
              }
              return req.url;
            },
            memoizeHost: false
          }
        },
      },
    ],
  },
  {
    id: "json-placeholder-posts",
    url: "/json-placeholder/posts",
    method: ["GET"],
    variants: [
      {
        id: "success",
        type: "json",
        options: {
          status: 200,
          body: [{
            id: 1,
            title: "Json placeholder post intercepted by mock server"
          }]
        },
      },
    ],
  },
  {
    id: "dummy-json-products",
    url: "/dummy-json/products",
    method: ["GET"],
    variants: [
      {
        id: "success",
        type: "json",
        options: {
          status: 200,
          body: [{
            id: 1,
            title: "Dummy product intercepted by mock server"
          }]
        },
      },
    ],
  },
];

And the corresponding collection:

[
  {
    "id": "base",
    "routes": ["proxy:interceptor", "json-placeholder-posts:success", "dummy-json-products:success"]
  }
]

I have tested it and it works properly. For example, if you use curl to get /posts from jsonplaceholder.typicode.com, it returns the mocked response from the route "json-placeholder-posts", with url /json-placeholder/posts.

http_proxy=http://localhost:3100 curl -v http://jsonplaceholder.typicode.com/posts

And if you get /products from dummyjson.com, it will send the response defined in the "dummy-json-products" route, with url "/dummy-json/products"

http_proxy=http://localhost:3100 curl -v http://dummyjson.com/products

Obviously, the code in the example can be improved, I have kept it "simple" just for demonstration purposes. It could even mock only some specific paths of some hosts, etc. Anyway, I hope you get the idea.

As you suggested, maybe this solution should be implemented as a new route variant relying only on some kind of configuration about which hosts and/or paths should be locally handled, and the prefixes to add to each host. So, I will let the issue opened.

Please let me know if the proposed solution is valid for you until them. 😃