openstreetmap/operations

Containerise small services

Opened this issue · 21 comments

grischard commented

There's a number of small services that currently live on a full server, and could live inside a container.

  • Identify small services that could be containerised
  • Identify server to host containers
  • Choose container system
  • Write container files
Firefishy commented

I am going to test OSQA as a container first.

tomhughes commented

Yes sure why not pick literally the hardest possible thing to try first.

Firefishy commented

Choose container system... examples could be:

  • k3s (unlikely, only recommended for 100% disposable. eg: CI)
  • kubernetes (no.)
  • nomad (tempting)
  • docker without any bells and whistles managed using chef with systemd
  • podman without any bells and whistles managed using chef with systemd
  • others?
tomhughes commented

Well there's actually really two separate questions - what to use to build images and what to use to deploy them.

As I understand there are other less horrible languages that Dockerfile for describing images and those images can be deployed with any of the major systems just as podman for example can deploy an image built from a Dockerfile.

tomhughes commented

Some articles on container systems and orchestration systems.

I think https://buildah.io/ is what I was thinking of as the main alternative to Dockerfile - it can actually use Dockerfiles but the native way is just based around writing a script that uses buildah commands to manipulate the image.

Firefishy commented

Dockerfiles are the beast I know and by a million miles has more adoption than the alternatives.

There are alternatives for building OCI compatible container images with varying levels completeness. Other than https://buildah.io/ there is also https://github.com/genuinetools/img (uses Dockerfile) and https://github.com/GoogleContainerTools/jib (java images)

grischard commented

The stuff that runs on naga (redirectors, blog aggregator, munin server that we're going to demise (#501)) might be good.

grischard commented

https://hardware.osm.org which runs on idris even already has a dockerfile in https://github.com/osmfoundation/osmf-server-info

Firefishy commented

The following sites are now containers:

  • 🎉1
jpds commented

You probably want to deploy https://github.com/google/cadvisor on every Docker host and point Prometheus at that,

tomhughes commented

Well that rather depends how it works - if it depends on the docker daemon then it probably won't work for us.

tomhughes commented

A quick test on naga suggests it doesn't manage to collect anything useful in our setup - all it finds is some basic host hardware metrics:

# HELP machine_cpu_cores Number of logical CPU cores.
# TYPE machine_cpu_cores gauge
machine_cpu_cores{boot_id="d474a5a2-46e3-45bb-bd70-8babbdf0fd25",machine_id="0787fff1c8bb4e53a49da871f67e3246",system_uuid="32353537-3835-584d-5136-303130324447"} 64 1681303543392
# HELP machine_cpu_physical_cores Number of physical CPU cores.
# TYPE machine_cpu_physical_cores gauge
machine_cpu_physical_cores{boot_id="d474a5a2-46e3-45bb-bd70-8babbdf0fd25",machine_id="0787fff1c8bb4e53a49da871f67e3246",system_uuid="32353537-3835-584d-5136-303130324447"} 16 1681303543392
# HELP machine_cpu_sockets Number of CPU sockets.
# TYPE machine_cpu_sockets gauge
machine_cpu_sockets{boot_id="d474a5a2-46e3-45bb-bd70-8babbdf0fd25",machine_id="0787fff1c8bb4e53a49da871f67e3246",system_uuid="32353537-3835-584d-5136-303130324447"} 2 1681303543392
# HELP machine_dimm_capacity_bytes Total RAM DIMM capacity (all types memory modules) value labeled by dimm type.
# TYPE machine_dimm_capacity_bytes gauge
machine_dimm_capacity_bytes{boot_id="d474a5a2-46e3-45bb-bd70-8babbdf0fd25",machine_id="0787fff1c8bb4e53a49da871f67e3246",system_uuid="32353537-3835-584d-5136-303130324447",type="Registered-DDR4"} 2.06158430208e+11 1681303543392
# HELP machine_dimm_count Number of RAM DIMM (all types memory modules) value labeled by dimm type.
# TYPE machine_dimm_count gauge
machine_dimm_count{boot_id="d474a5a2-46e3-45bb-bd70-8babbdf0fd25",machine_id="0787fff1c8bb4e53a49da871f67e3246",system_uuid="32353537-3835-584d-5136-303130324447",type="Registered-DDR4"} 12 1681303543392
# HELP machine_memory_bytes Amount of memory installed on the machine.
# TYPE machine_memory_bytes gauge
machine_memory_bytes{boot_id="d474a5a2-46e3-45bb-bd70-8babbdf0fd25",machine_id="0787fff1c8bb4e53a49da871f67e3246",system_uuid="32353537-3835-584d-5136-303130324447"} 2.02673983488e+11 1681303543392
# HELP machine_nvm_avg_power_budget_watts NVM power budget.
# TYPE machine_nvm_avg_power_budget_watts gauge
machine_nvm_avg_power_budget_watts{boot_id="d474a5a2-46e3-45bb-bd70-8babbdf0fd25",machine_id="0787fff1c8bb4e53a49da871f67e3246",system_uuid="32353537-3835-584d-5136-303130324447"} 0 1681303543392
# HELP machine_nvm_capacity NVM capacity value labeled by NVM mode (memory mode or app direct mode).
# TYPE machine_nvm_capacity gauge
machine_nvm_capacity{boot_id="d474a5a2-46e3-45bb-bd70-8babbdf0fd25",machine_id="0787fff1c8bb4e53a49da871f67e3246",mode="app_direct_mode",system_uuid="32353537-3835-584d-5136-303130324447"} 0 1681303543392
machine_nvm_capacity{boot_id="d474a5a2-46e3-45bb-bd70-8babbdf0fd25",machine_id="0787fff1c8bb4e53a49da871f67e3246",mode="memory_mode",system_uuid="32353537-3835-584d-5136-303130324447"} 0 1681303543392
# HELP machine_scrape_error 1 if there was an error while getting machine metrics, 0 otherwise.
# TYPE machine_scrape_error gauge
machine_scrape_error 0
tomhughes commented

Also generating metrics with timestamps (rather than letting the server add them) is generally considered bad form and can cause problems with metric ingestion.

jpds commented

generating metrics with timestamps (rather than letting the server add them) is generally considered bad form

Do you have a source for this? Because just searching for "timestamp" on my own Grafana brings up:

  • grafana_build_timestamp
  • node_boot_time_seconds
  • prometheus_config_last_reload_success_timestamp_seconds
  • prometheus_tsdb_lowest_timestamp_seconds
  • thanos_bucket_store_blocks_last_loaded_timestamp_seconds
tomhughes commented

I mean that fact that each of those metrics has a timestamp like 1681303543392 after the metric value at the end of the line - when that is there the server will use that as the timestamp to associate with the value instead of using it's own clock to get a scrape time to associate with the value.

See https://promlabs.com/blog/2022/12/15/understanding-duplicate-samples-and-out-of-order-timestamp-errors-in-prometheus/#buggy-client-side-timestamps for some discussion of the potential issues with it.

jpds commented

Ah yes - that's already discussed on cAdvisor issue #2526.

However, if you don't see one of these https://github.com/google/cadvisor/blob/master/metrics/prometheus.go#L138 metrics here, then something is misconfigured.

tomhughes commented

Well I didn't do any configuration. I just ran the executable as there didn't seem to be any clear instructions telling me to do anything else.

As I say because we are using podman if it relies on talking to dockerd to get statistics then it's probably not going to work.

Firefishy commented

dmca.osm.org now a container: openstreetmap/chef@4ac7cf5

grischard commented

Left to containerise on Ridley: tracked in #1028

Firefishy commented

https://github.com/openstreetmap/birthday20-website/ is a wordpress to static site I generated using wp2static.

wp2static did a good job. Some cleanup was required, but it does a reasonable job.

Linked issue: #1125

Firefishy commented

SoTM 2007, 2008 and 2009 are now containers.