Allow "forests-per-host" specification
Closed this issue · 10 comments
The current mlsound code uses separate source files for each request that will be sent to the Management API. In order to reduce the number of nearly-duplicate files that need to be maintained, allow a database's "forest" property to have an object with one property: "forests-per-host". When mlsound sees this, construct that many forests for each host in the cluster, use the Management API to create them, then in memory replace the database's forest property with the array of forest names.
Using an object here would also allow a future enhancement of specifying a Strategy for naming forests.
This would also simplify environment management: if you have one host for your local work, three in your dev cluster, seven in your integration testing cluster, and seven in production, managing the forest configurations for each would be complicated. Specifying how many forests you want would be a lot simpler.
Engineering's current recommendation is six forests per host.
@miguelrgonzalez, I can tackle this if you agree with the direction.
Hi @dmcassel,
I like the solution direction. Just keep in mind to remove the new property forest-per-host before sending the DB create/update commands (have a look at DBManager.prototype.filterUnsupportedProperties at database.js). Otherwise I believe you'll get a 400 bad request back.
And thanks for chipping in!
Just came back from my holidays and was planning to resume the work on mlsound this weekend :)
Hey guys - does the source file represent a JSON object that mlsound uses to construct a call to the mgmt API, or is it a JSON payload as defined by the mgmt API?
The approach I've taken so far with ml-gradle is to have all the files be JSON/XML payloads as defined by the mgmt API. I know the /v1/rest-apis endpoint supports a forests-per-host endpoint, but /manage/v2/databases does not. I'm guessing the source file then defines a JSON object as a spec for how to make calls to construct a database and a set of forests?
This is an area where I think the mgmt API should provide a little more power - I think it's very common to want to say "Create me a database with this config and with this many forests", and to not worry about the details of the forests themselves (i.e. what the fast data directory is).
Hi @rjrudin,
files are JSON payloads as defined by the management API. Same as ml-gradle
Okay, so I think based on your previous comment, this would be an additional field that's added to the file, and mlsound then drops that field when it sends the contents of the file to the mgmt API?
I haven't provided nice support for this yet in ml-gradle - I've assumed content databases are going to be created via /v1/rest-apis - but my hunch is I'd have a "CreateContentDatabaseCommand" with a property that specifies a number of forests. And I like Dave's suggestion for a strategy interface for defining how the forest should be created - the name, fast data dir, etc. A developer could then add a new instance of this command to the build file and specify those properties.
In mlsound, would that be equivalent to adding a new task in the gulp file that specifies number of forests, forest names, etc?
To confirm, I picture the files as API payloads, but the proposed change will be a small departure from that. The normal contents of the "forests" property will optionally be replaced with something that triggers construction of forests. Full payload approach will still work.
mlsound is not currently gulp based, though I've also thought it would help with adoption to make that an option.
At the moment mlsound is command line base very much like roxy.
It shouldn't take too long to create some sort of wrapper for gulp though.
On Fri, Aug 7, 2015 at 6:20 PM, David Cassel notifications@github.com
wrote:
To confirm, I picture the files as API payloads, but the proposed change
will be a small departure from that. The normal contents of the "forests"
property will optionally be replaced with something that triggers
construction of forests. Full payload approach will still work.mlsound is not currently gulp based, though I've also thought it would
help with adoption to make that an option.—
Reply to this email directly or view it on GitHub
#5 (comment)
.
We've explicitly designed manage rest api to be fined grained and avoid any kind of automatic behavior ... however there are a few scenarios (like the one described here) that developers consistently bump up against; I've gone and added an internal bug(34932) to track the discussion.
For now, I think its ok to add additional properties in the POST creation bodies to be used by client side scripting.
@miguelrgonzalez I propose closing this one
Documented use at https://github.com/miguelrgonzalez/mlsound/wiki/Database-configuration.