Support organization/project hierarchy

[ojohnny]

Currently laminar only supports a "flat hierarchy", presumably by deliberate choice. I can see this working well in a single-user environment, but I can also see it becoming hairy quite fast if one wants to have more users.

It would be nice if laminar could:

• Allow forward-slashes in "job names", so that one could name a job <whatever>/<job> or even <organization>/<project>/<job>.
• Each "hierarchy level" could have its own "summary/landing page", e.g. going to <base_url>/jobs/<organization>/<project> in the previous example would show me statistics only for jobs named <organization>/<project>/*.

The benefits I can see:

• If more than a handful of projects use the same laminar instance then the separate "summary/landing page" provides a filter to only view the jobs that are interesting/related to some project.
• It would allow me to have easier access control configuration in my webserver if I could just put a rule on <base_url>/jobs/<organization>/ instead of each related job individually. I think user access control is out of scope for laminar, so it would be nice if we had a structure which we could easily configure ourselves.

One thing I would not want is to have multiple instances of laminar running. I would like the all available executors to be a "shared pool", so I think than means a single laminar instance should run? (multiple laminar instances would also complicate job submission, I think)

Hopefully this feature request is not too far outside the possible scope of laminar :)

[ohwgiles]

This would be a highly complex change, especially making the statistics aware of a hierarchy and prevent the backend from leaking data about a different project to the frontend (remember it is an SPA). Also, using inotify to respond to changes to job config files gets tricky when you have nested folders.

Basically I'm not against the principle, but I am not excited about it either, and not really interested in implementing this myself. If someone can propose a relatively simple yet complete way to do this, I can provide some guidance.

I can't think of a good workaround for your first proposed benefit, but for the second, you could just pick a different separator (colon, underscore, §, go crazy) and use a regex in your reverse proxy.

[ohwgiles]

Or, regarding multiple instances of laminar, which is an elegant way to solve this:

• You could use cgroups to restrict the cpu usage of each instance
• Submitting "foojob" to the "org7" instance could be as simple as LAMINAR_HOST=unix-abstract:laminar-org7 laminarc trigger foojob

I would also consider a thoughtful proposal for sharing executors between laminar instances, à la make jobserver

[ojohnny]

Thinking about it some more, I think multiple instances would be the easiest way forward. Then I could get the "filtering" and "access control" at least, which was my primary benefits. My worries about shared executors could probably be fixed with cgroups and flock, and it seems most things should be "simple enough" to script since everything is configurable via environment variables (regarding job submission and such).

[ohwgiles]

Of course the downside with cgroups/flock is that laminar is unaware, so jobs might be started anyway which would just block on flock, or take much longer due to constrained resources/cgroups, messing up your build time stats.

A possible solution could be: consider multiple laminar instances, each with LAMINAR_HOME=/var/lib/laminar/orgname, and each with cfg/contexts/default.conf containing EXECUTORS=6 or whatever your max count is. Create a global before script ($LAMINAR_HOME/cfg/before), ideally symlinked to something common between instances, with contents something like

for instance in "$LAMINAR_HOME/../"*; do
  if [[ "$(readlink -f "$instance")" != "$LAMINAR_HOME" ]]; then
    # Decrease EXECUTORS by one of all other instances
    awk -i inplace -F= -v OFS== '/EXECUTORS/{$2-=1;print}' test.conf
  fi
done

and the reverse operation $2+=1 in a global after script. Laminar reacts instantly to this change - there is a minor race in that a job could be queued and started on instance X before it has reacted to an EXECUTORS change due to a job starting on another instance, but you are unlikely to hit this unless perhaps you are triggering jobs across multiple organizations based on the same event, which doesn't sound healthy.

[thomas-mc-work]

I've encountered the same problem and solved it by joining the group name and project name by a static separator (e.g. _). This could also be used in q regex pattern for your web server access control. Not very sophisticated, but a lightweight solution that refuses to have multiple instances.

What's your SCM service, @ojohnny?