reqque

Promise based JavaScript library that enables you to make numerous HTTP requests without bumping into the rate limits

Features
Installation
Usage
Examples
Config
Result Schema
Resources
License

Features

Make parallel/concurrent HTTP requests by splitting your requests into batches
Make sequential HTTP requests
Put some delay between requests
Retry failed requests again until reaching the specified retry limits

Installation

Using npm:

$ npm install reqque

Using yarn:

$ yarn add reqque

Usage

reqque(requests, requestTemplate, config)

reqque has a very simple usage. You just need to pass 3 arguments to start the machine;

requests (required): The list of request items that will be passed to requestTemplate
requestTemplate (required): The template that takes request item as parameter and makes the HTTP call by using the passed request item
config (optional): custom reqque config

Examples

There are two different examples that are prepared to guide you on how to use reqque; these are basic.js and advanced.js.

In both examples, Axios is used to make HTTP requests easily configurable. You can replace Axios with any HTTP client that you prefer.

PS: You can access the executable example files under the examples folder.

Basic

Basic usage of reqque

/example/basic.js

import axios from "axios";
import reqque from "reqque";

(async () => {
  const requests = [
    "https://httpstat.us/200?sleep=100",
    /*
    "...",
    "...",
    */
  ];

  const requestTemplate = async (url) => axios.get(url);

  const config = {
    maxRetries: 2,
    batch: {
      size: {
        limit: 5,
      },
    },
    delay: {
      duration: {
        limit: 1000,
      },
    },
  };

  const results = await reqque(requests, requestTemplate, config);
})();

Advanced

This example shows you to how you can disguise your requests while making HTTP calls with reqque.

/example/advanced.js

import axios from "axios";
import UserAgent from "user-agents";
import HttpsProxyAgent from "https-proxy-agent";
import reqque from "reqque";

(async () => {
  const requests = [
    {
      method: "GET",
      code: 200,
      sleep: 100,
    },
    /*
    {...},
    {...},
    */
  ];

  const proxyList = [
    "110.164.253.85:8080",
    /*
   "...",
   "...",
    */
  ];

  const requestTemplate = async ({ method, code, sleep }) => {
    const randomProxyIndex = Math.floor(Math.random() * proxyList.length);
    const randomProxy = proxyList[randomProxyIndex];
    const httpsAgent = new HttpsProxyAgent(`http://${randomProxy}`);
    const userAgent = new UserAgent();

    return axios({
      url: `http://httpstat.us/${code}`,
      query: { sleep },
      headers: {
        "User-Agent": userAgent,
      },
      method,
      httpsAgent,
    });
  };

  const config = {
    maxRetries: 5,
    batch: {
      size: {
        limit: 5,
      },
    },
  };

  const results = await reqque(requests, requestTemplate, config);
})();

Config

These are the available options of reqque config. All of them are optional. It will use the default options unless you specify custom values

{
    // `maxRetries` defines the maximum number of retries.
    // If set to 0, the failed requests won't be retried.
    maxRetries: 5, // default

    // `batch` defines the options of batch request feature.
    batch: {
        // `active` indicates whether or not the batch request feature is active
        // If set to true, the requests will be splitted into batches and the requests in those batches will be made concurrently
        // If set to false, the requests will be made sequentially one by one
        active: true, // default

        // `size` defines the size options of batch request feature.
        size: {
            // `limit` defines the number of requests that are placed in each batch
            limit: 20, // default

            // `random` indicates whether or not the batch size is random
            // If set to true, a random batch size that is generated between 0 and limit value will be used in each iteration
            // If set to false, a fixed batch size will be used in each iteration
            random: false // default
        }
    },

    // `delay` defines the options of delay feature.
    delay: {
        // `active` indicates whether or not the delay feature is active
        // If set to true, it will be put some delay between each iterations
        // If set to false, it won't be put any delay between iterations
        active: true, // default

        // `duration` defines the duration options of delay feature.
        duration: {
            // `limit` defines the amount of delays that are put between each iterations
            limit: 20, // default

            // `random` indicates whether or not the delay duration is random
            // If set to true, a random delay duration that is generated between 0 and limit value will be used in each iteration
            // If set to false, a fixed delay duration will be used in each iteration
            random: false // default
        }
    }
}

Result Schema

The result for a reqque request contains the following information.

[
  	{
    		// request item that was passed to request template
    		request: {},

    		// response body that was provided by the server
    		response: {},

    		// status of the request
    		status: "successful",

    		// the total number of request attempts
    		tryCount: 1,
  	},
  	/*
    {...},
    {...},
    {...}
    */
];

Resources

License

MIT

furkankose/reqque