Several quirks preventing the buildpack from working in a cf-for-k8s environment

Question

Several quirks preventing the buildpack from working in a cf-for-k8s environment

Closed this issue 4 years ago · 6 comments

ericpromislow commented 4 years ago

Problem

The staticfile buildpack does not work with the cnb bionic stack.

Steps to reproduce

We added the staticfile buildpack to cf-for-k8s using the following components:

gcr.io/paketo-community/staticfile@sha256:37dfc5e75328f3c714f060c60db89b685a70aa12c9fda95396300e28c255b82b
bionic-image with this runtime image index.docker.io/paketobuildpacks/run@sha256:c80d5fe8433aea6f3e7dbfd414b56327758f096aea96b06768dd4a172a7f9d5c

Expect Results

cf push is successful and the staticfile app is available via the cf created route

Actual Results

The app failed to start, with this error message:

mkdir() "/client_body_temp" failed (13: Permission denied)

We believe the root cause of this is that the cnb user (from the runtime image) does not have permissions to write to /.

Related cf4k8s bug: cloudfoundry/cf-for-k8s#587
Staticfile app we pushed: https://github.com/cloudfoundry/cf-acceptance-tests/assets/staticfile (we made a temporary change to this app to add a buildpack.yml file based on the instructions in your staticfile)

Subsequent investigations:

We then locally ran docker run -it -p 8080:8080 bash on the workload image that CF runs, and found these problems:

We updated the listen field in the server block to listen on port 8080 -- there was a space between listen and ; as if an environment var wasn't set.
We manually started sbin/nginx with the options -p /tmp and -c /workspace/nginx.conf . Also, we had to pre-create /tmp/logs
The app's index file index.html is in /workspace but the doc root is /workspace/public. We created .../public and copied index.html into public.

After making the updates, we were able to run curl localhost:8080 from the command-line and get output (note that when we started the docker container we specified -p 8080:8080 to expose the container's port.

Eric Promislow & @paulcwarren

Answer 1 · 2020-12-01T16:01:12.000Z

Hi,
This is still a community buildpack and yes there are few quirks but you should still be able to to do what you're trying to do.
You'll need to set a runtime env var PORT and must have a buildpack.yml that directs it to use nginx. Also the nginx buildpack provides the start command, so you don't have to start it manually.

Here are the steps I used to get your app up and running.

(1) Clone https://github.com/cloudfoundry/cf-acceptance-tests/tree/develop/assets and cd to it.
(2) Create a buildpack.yml at the root of the app:

cat <<EOF >buildpack.yml
staticfile:
  nginx:
    root: "."
EOF

(3) Use pack to build an image using the latest versions of nginx and staticfile buildpacks:

pack build staticimage -b gcr.io/paketo-buildpacks/nginx:0.0.194 -b gcr.io/paketo-community/staticfile:0.0.12

(4) Docker-run by passing a PORT env var. This is translated to the listen <> directive by the nginx buildpack.

docker run --env PORT=8080 -it -p 8080:8080 staticimage

(5) Test it

curl 0.0.0.0:8080
Hello from a staticfile

Answer 2 · 2020-12-02T00:14:28.000Z

Hey @ericpromislow, after reading this I'm still a little bit curious how you all got the permission denied error, and I was hoping you could post the build logs so we can make sure what we expect to happen is happening.

Answer 3 · 2020-12-02T01:10:56.000Z

Hey @joshzarrabi, when we inspected the final image it looked like it had a filesystem that was largely owned by root (and my assumption is that is supplied by from the bionic runtime image noted above). At runtime, the image then wants to run as uid cnb:1000 (we confirmed this by running it directly from docker) so when nginx starts up and attempts to write to /client_body_temp it is unable to. At least that is what I thought I was seeing yesterday.

Because we first observed this in CF it's also possible that the runtime user was forced to be 2000 (vcap) and that is what caused the permission denied but, as I say, we tried running the built container ourselves with docker and observed the same permission denied and in that case we were definately uid 1000.

Answer 4 · 2020-12-02T19:29:27.000Z

@paulcwarren Did you also include the nginx buildpack when you built the image?

Answer 5 · 2020-12-16T15:22:43.000Z

Hey. Just a note that I've seen the same error when doing a cf push with the staticfile and nginx buildpacks (I'm also doing this on a cf-for-k8s deployment):

2020-12-16T16:13:34.24+0100 [APP/PROC/WEB/d0c76acc-2a7e-46c0-8d07-364d9e206532] OUT 2020/12/16 15:13:34 [emerg] 7#0: mkdir() "/client_body_temp" failed (13: Permission denied)

My push command looked like this:

cf push my-app -f manifest.yml -b paketo-buildpacks/nginx -b paketo-community/staticfile

My buildpack.yml:

staticfile:
  nginx:
    root: "."

And my manifest:

---
applications:
- name: my-app
  memory: 64M

If I build the image locally with pack and then run it with docker - using the exact commands you paste above - then it works fine and I can access the site on localhost in my browser.

Answer 6 · 2020-12-17T09:50:40.000Z

Alright so I've managed to get my app deployed on cf-for-k8s with a cf push. I had to add four lines to the nginx.conf giving explicit paths to directories inside /workspace for the fields pid, client_body_temp_path, proxy_temp_path and fastcgi_temp_path. Without these lines nginx would default to other paths under the root and error when its user lacked permissions to create directories/files there.

Here's the nginx.conf that I got working:

worker_processes 1;
daemon off;
pid /workspace/logs/nginx.pid;

error_log /workspace/logs/nginx/error.log;
events { worker_connections 1024; }

http {
  client_body_temp_path /workspace/client_body_temp;
  proxy_temp_path /workspace/proxy_temp;
  fastcgi_temp_path /workspace/fastcgi_temp;

  charset utf-8;
  types {
    text/html html htm shtml;
    text/css css;
    text/xml xml;
    image/gif gif;
    image/jpeg jpeg jpg;
    application/javascript js;
    application/atom+xml atom;
    application/rss+xml rss;
    font/ttf ttf;
    font/woff woff;
    font/woff2 woff2;
    text/mathml mml;
    text/plain txt;
    text/vnd.sun.j2me.app-descriptor jad;
    text/vnd.wap.wml wml;
    text/x-component htc;
    text/cache-manifest manifest;
    image/png png;
    image/tiff tif tiff;
    image/vnd.wap.wbmp wbmp;
    image/x-icon ico;
    image/x-jng jng;
    image/x-ms-bmp bmp;
    image/svg+xml svg svgz;
    image/webp webp;
    application/java-archive jar war ear;
    application/mac-binhex40 hqx;
    application/msword doc;
    application/pdf pdf;
    application/postscript ps eps ai;
    application/rtf rtf;
    application/vnd.ms-excel xls;
    application/vnd.ms-powerpoint ppt;
    application/vnd.wap.wmlc wmlc;
    application/vnd.google-earth.kml+xml  kml;
    application/vnd.google-earth.kmz kmz;
    application/x-7z-compressed 7z;
    application/x-cocoa cco;
    application/x-java-archive-diff jardiff;
    application/x-java-jnlp-file jnlp;
    application/x-makeself run;
    application/x-perl pl pm;
    application/x-pilot prc pdb;
    application/x-rar-compressed rar;
    application/x-redhat-package-manager  rpm;
    application/x-sea sea;
    application/x-shockwave-flash swf;
    application/x-stuffit sit;
    application/x-tcl tcl tk;
    application/x-x509-ca-cert der pem crt;
    application/x-xpinstall xpi;
    application/xhtml+xml xhtml;
    application/zip zip;
    application/octet-stream bin exe dll;
    application/octet-stream deb;
    application/octet-stream dmg;
    application/octet-stream eot;
    application/octet-stream iso img;
    application/octet-stream msi msp msm;
    application/json json;
    audio/midi mid midi kar;
    audio/mpeg mp3;
    audio/ogg ogg;
    audio/x-m4a m4a;
    audio/x-realaudio ra;
    video/3gpp 3gpp 3gp;
    video/mp4 mp4;
    video/mpeg mpeg mpg;
    video/quicktime mov;
    video/webm webm;
    video/x-flv flv;
    video/x-m4v m4v;
    video/x-mng mng;
    video/x-ms-asf asx asf;
    video/x-ms-wmv wmv;
    video/x-msvideo avi;
  }

  log_format cloudfoundry '$http_x_forwarded_for - $http_referer - [$time_local] "$request" $status $body_bytes_sent';
  access_log /workspace/logs/nginx/access.log cloudfoundry;
  default_type application/octet-stream;
  sendfile on;

  gzip on;
  gzip_disable "msie6";
  gzip_comp_level 6;
  gzip_min_length 1100;
  gzip_buffers 16 8k;
  gzip_proxied any;
  gunzip on;
  gzip_static always;
  gzip_types text/plain text/css text/js text/xml text/javascript application/javascript application/x-javascript application/json application/xml application/xml+rss;
  gzip_vary on;

  tcp_nopush on;
  keepalive_timeout 30;
  port_in_redirect off; # Ensure that redirects don't include the internal container PORT - <%= ENV["PORT"] %>
  server_tokens off;

  server {
    listen 8080;
    server_name localhost;

    root /workspace/.;

    location / {
        index index.html index.htm Default.htm;
    }

      location ~ /\. {
        deny all;
        return 404;
      }

  }
}

... And here's the original generated by the buildpack:

worker_processes 1;
daemon off;

error_log /workspace/logs/nginx/error.log;
events { worker_connections 1024; }

http {
  charset utf-8;
  types {
    text/html html htm shtml;
    text/css css;
    text/xml xml;
    image/gif gif;
    image/jpeg jpeg jpg;
    application/javascript js;
    application/atom+xml atom;
    application/rss+xml rss;
    font/ttf ttf;
    font/woff woff;
    font/woff2 woff2;
    text/mathml mml;
    text/plain txt;
    text/vnd.sun.j2me.app-descriptor jad;
    text/vnd.wap.wml wml;
    text/x-component htc;
    text/cache-manifest manifest;
    image/png png;
    image/tiff tif tiff;
    image/vnd.wap.wbmp wbmp;
    image/x-icon ico;
    image/x-jng jng;
    image/x-ms-bmp bmp;
    image/svg+xml svg svgz;
    image/webp webp;
    application/java-archive jar war ear;
    application/mac-binhex40 hqx;
    application/msword doc;
    application/pdf pdf;
    application/postscript ps eps ai;
    application/rtf rtf;
    application/vnd.ms-excel xls;
    application/vnd.ms-powerpoint ppt;
    application/vnd.wap.wmlc wmlc;
    application/vnd.google-earth.kml+xml  kml;
    application/vnd.google-earth.kmz kmz;
    application/x-7z-compressed 7z;
    application/x-cocoa cco;
    application/x-java-archive-diff jardiff;
    application/x-java-jnlp-file jnlp;
    application/x-makeself run;
    application/x-perl pl pm;
    application/x-pilot prc pdb;
    application/x-rar-compressed rar;
    application/x-redhat-package-manager  rpm;
    application/x-sea sea;
    application/x-shockwave-flash swf;
    application/x-stuffit sit;
    application/x-tcl tcl tk;
    application/x-x509-ca-cert der pem crt;
    application/x-xpinstall xpi;
    application/xhtml+xml xhtml;
    application/zip zip;
    application/octet-stream bin exe dll;
    application/octet-stream deb;
    application/octet-stream dmg;
    application/octet-stream eot;
    application/octet-stream iso img;
    application/octet-stream msi msp msm;
    application/json json;
    audio/midi mid midi kar;
    audio/mpeg mp3;
    audio/ogg ogg;
    audio/x-m4a m4a;
    audio/x-realaudio ra;
    video/3gpp 3gpp 3gp;
    video/mp4 mp4;
    video/mpeg mpeg mpg;
    video/quicktime mov;
    video/webm webm;
    video/x-flv flv;
    video/x-m4v m4v;
    video/x-mng mng;
    video/x-ms-asf asx asf;
    video/x-ms-wmv wmv;
    video/x-msvideo avi;
  }

  log_format cloudfoundry '$http_x_forwarded_for - $http_referer - [$time_local] "$request" $status $body_bytes_sent';
  access_log /workspace/logs/nginx/access.log cloudfoundry;
  default_type application/octet-stream;
  sendfile on;

  gzip on;
  gzip_disable "msie6";
  gzip_comp_level 6;
  gzip_min_length 1100;
  gzip_buffers 16 8k;
  gzip_proxied any;
  gunzip on;
  gzip_static always;
  gzip_types text/plain text/css text/js text/xml text/javascript application/javascript application/x-javascript application/json application/xml application/xml+rss;
  gzip_vary on;

  tcp_nopush on;
  keepalive_timeout 30;
  port_in_redirect off; # Ensure that redirects don't include the internal container PORT - <%= ENV["PORT"] %>
  server_tokens off;

  server {
    listen ;
    server_name localhost;

    root /workspace/.;






    location / {


        index index.html index.htm Default.htm;












    }


      location ~ /\. {
        deny all;
        return 404;
      }

  }
}