cloudfoundry/cloud_controller_ng

CPU assignment for app instance >8G memory

stephanme opened this issue · 6 comments

stephanme commented

Issue

Currently, CPU assignment to app instance is capped at 8G instance memory, originally reported in #1178
Back then, this was more a theoretical problem but meanwhile we face user requests for 16G app instances and even more. Often, the reason is not memory but the wish for more CPU.

Context

There is a hardcoded cap at 8G: MAX_CPU_PROXY in task_cpu_weight_calculator.rb

See also discussion of other options in diego-release #865

Steps to Reproduce

  • push an 8G and a 16G app
  • ssh into each app
  • check assigned cpu shares with cat /sys/fs/cgroup/cpu/cpu.shares

Expected result

16G app should have double the cpu shares of the 8G app.

Current result

16G app has only the same cpu shares as the 8G app.

Possible Fix

Make the MIN/MAX_CPU_PROXY values that are used for capping the cpu weight in task_cpu_weight_calculator.rb configurable in CAPI

E.g.

  • cc.cpu_weight_min_mem - Min memory in MB that is used for CPU weight calculation even if app has less memory assigned. Default=128
  • cc.cpu_weight_max_mem - Max memory in MB that is used for CPU weight calculation even if app has more memory assigned. If set to 0, cpu shares are not capped. Default=8192

CPU weight is still defined as 100% for an 8G app. 16G apps would get a value of 200%.

Gerg commented

As discussed in CAPI office hours, It looks like BBS currently limits CPU weight to 100: https://github.com/cloudfoundry/bbs/blob/0007fcda733509f73717fa966d8dcd8ff92ce62c/models/desired_lrp.go#L725-L727

PlamenDoychev commented

Dear CAPI Colleagues,

We can take the Diego part, i shall create an issue in diego-release.
But i have one question with regards to the new values, it would be good to share if you have already discussed it.
Should we limit the new CPU weight value to 200 hence [0; 200] interval which covers apps up to 16GB or have [0; infinity) interval which covers any app in general?

stephanme commented

My proposal would be to not limit the cpu weight value in Diego, i.e. accept [0, infinity). Otherwise we just introduce the next hardcoded limit.

In practice, the cpu weight value is limited by the max possible instance memory defined in org and space quotas. If set to unlimited (e.g. for non-productive foundations), the max possible instance memory size is determined by the Diego cell VM size and will indirectly limit the max possible cpu weight.

  • 👍1
klapkov commented

Hello Capi colleagues,

Do you plan to change the way the cpu_weight is calculated ? From what I understand, with current way the cpu_weight will not go over 100.

100 * numerator / MAX_CPU_PROXY

Here the numerator is the requested app memory. With the current formula, even if we change MAX_CPU_PROXY to a larger number, the cpu_weight will never be over 100. This is because we aways divide by the upper boundary. Am I wrong ?

stephanme commented

The proposed new calculation would be:

mem = memory_in_mb
// cap mem used for weight calculation
mem = max(mem, cc.cpu_weight_min_mem)
mem = min(mem, cc.cpu_weight_max_mem)
// convert to cpu weight (8G = 100%)
cpu_weight = mem * 100 / 8192

I.e. weight scales linearly with memory between the configured min/max values and 100% weight equals an 8G instance.

With the proposed defaults for cc.cpu_weight_min/max_mem, the weight calculator will behave exactly as today:
memory_in_mb = 4096 -> cpu_weight = 50
memory_in_mb = 8192 -> cpu_weight = 100
memory_in_mb = 16384 -> cpu_weight = 100

But when setting cc.cpu_weight_max_mem = 16384, it would return 200% for a 16G app instance:
memory_in_mb = 4096 -> cpu_weight = 50
memory_in_mb = 8192 -> cpu_weight = 100
memory_in_mb = 16384 -> cpu_weight = 200
memory_in_mb = 32768 -> cpu_weight = 200 // capped at 16G due to configuration

PlamenDoychev commented

I just created the following for the Diego part
cloudfoundry/diego-release#886