Pagination and parsing a larger number of results

[lstiebel]

I'm running the exporter in a larger Rancher setup with 500+ services. Because the Rancher API has a pagination setting of 100 by default, this leads to the exporter only scraping/displaying metrics for the first page of results, rendering it unusable for users with larger Rancher installations.

I was able to improve this by adding a ?limit=1000 to the endpoints to use:

// setEndpoint - Determines the correct URL endpoint to use, gives us backwards compatibility
func setEndpoint(rancherURL string, component string, apiVer string) string {

	var endpoint string

	if strings.Contains(component, "services") {
		endpoint = (rancherURL + "/services?limit=1000")
	} else if strings.Contains(component, "hosts") {
		endpoint = (rancherURL + "/hosts?limit=1000")
	} else if strings.Contains(component, "stacks") {

		if apiVer == "v1" {
			endpoint = (rancherURL + "/environments/")
		} else {
			endpoint = (rancherURL + "/stacks?limit=1000")
		}
	}

	return endpoint
}

A better approach would of course be to detect if the response has been paginated and to automatically follow the link to the next page of results that the Rancher API provides.

[Rucknar]

@lstiebel I'll have a look at this, i'm planning in an updated version that works against the metadata service following feedback from Rancher, that might get around this issue by design.

[mikequentel]

In case anyone else is facing a similar problem, please see #48 (comment)