shinyproxy not able to create r-apps Pod
Closed this issue · 3 comments
massaox commented
Hi,
I have followed the instructions to deploy the shinyproxy operator in a clustered manner (with some slight modification) but I was not able to get the shinyproxy Pod to create the r-apps applications. At least it seems to operator is doings its job e.g creating the shinyproxy Pod, the configMap, Service and Skipper ingress. I have tested using Minikube version 1.20 and GKE 1.18 and 1.20, all approaches returning the same error. Below you can see the full error output when I tried to instantiate the "hello" application:
Error
Status code: 500
Message: Failed to start container
Stack Trace:eu.openanalytics.containerproxy.ContainerProxyException: Failed to start containerat eu.openanalytics.containerproxy.backend.AbstractContainerBackend.startProxy(AbstractContainerBackend.java:138)at eu.openanalytics.containerproxy.service.ProxyService.startProxy(ProxyService.java:226)at eu.openanalytics.shinyproxy.controllers.AppController.getOrStart(AppController.java:106)at eu.openanalytics.shinyproxy.controllers.AppController.startApp(AppController.java:63)at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)at java.base/java.lang.reflect.Method.invoke(Unknown Source)at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190)at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138)at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:105)at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:878)at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:792)at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1040)at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943)at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909)at javax.servlet.http.HttpServlet.service(HttpServlet.java:517)at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)at javax.servlet.http.HttpServlet.service(HttpServlet.java:584)at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74)at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:320)at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:126)at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:90)at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:118)at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137)at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:158)at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilterInternal(BasicAuthenticationFilter.java:155)at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:200)at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:116)at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)at org.springframework.security.web.csrf.CsrfFilter.doFilterInternal(CsrfFilter.java:117)at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)at org.springframework.security.web.header.HeaderWriterFilter.doHeadersAfter(HeaderWriterFilter.java:92)at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:77)at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105)at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56)at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:215)at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:178)at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358)at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271)at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:93)at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68)at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)at io.undertow.server.handlers.PathHandler.handleRequest(PathHandler.java:91)at eu.openanalytics.containerproxy.util.ProxyMappingManager$ProxyPathHandler.handleRequest(ProxyMappingManager.java:160)at io.undertow.servlet.handlers.RedirectDirHandler.handleRequest(RedirectDirHandler.java:68)at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:132)at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:269)at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:78)at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:133)at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:130)at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:249)at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:78)at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:99)at io.undertow.server.Connectors.executeRootHandler(Connectors.java:370)at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:836)at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:2019)at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1558)at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1449)at java.base/java.lang.Thread.run(Unknown Source)Caused by: com.spotify.docker.client.exceptions.DockerException: java.util.concurrent.ExecutionException: javax.ws.rs.ProcessingException: java.io.IOException: No such file or directoryat com.spotify.docker.client.DefaultDockerClient.propagate(DefaultDockerClient.java:2848)at com.spotify.docker.client.DefaultDockerClient.request(DefaultDockerClient.java:2723)at com.spotify.docker.client.DefaultDockerClient.createContainer(DefaultDockerClient.java:742)at com.spotify.docker.client.DefaultDockerClient.createContainer(DefaultDockerClient.java:724)at eu.openanalytics.containerproxy.backend.docker.DockerEngineBackend.startContainer(DockerEngineBackend.java:87)at eu.openanalytics.containerproxy.backend.AbstractContainerBackend.doStartProxy(AbstractContainerBackend.java:169)at eu.openanalytics.containerproxy.backend.AbstractContainerBackend.startProxy(AbstractContainerBackend.java:135)... 106 moreCaused by: java.util.concurrent.ExecutionException: javax.ws.rs.ProcessingException: java.io.IOException: No such file or directoryat java.base/java.util.concurrent.CompletableFuture.reportGet(Unknown Source)at java.base/java.util.concurrent.CompletableFuture.get(Unknown Source)at com.spotify.docker.client.DefaultDockerClient.request(DefaultDockerClient.java:2721)... 111 moreCaused by: javax.ws.rs.ProcessingException: java.io.IOException: No such file or directoryat org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:528)at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:535)at org.glassfish.jersey.client.ClientRuntime.lambda$null$6(ClientRuntime.java:181)at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)at org.glassfish.jersey.internal.Errors.process(Errors.java:292)at org.glassfish.jersey.internal.Errors.process(Errors.java:274)at org.glassfish.jersey.internal.Errors.process(Errors.java:244)at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:288)at org.glassfish.jersey.client.ClientRuntime.lambda$createRunnableForAsyncProcessing$7(ClientRuntime.java:155)at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)at java.base/java.util.concurrent.FutureTask.run(Unknown Source)at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)... 1 moreCaused by: java.io.IOException: No such file or directoryat jnr.unixsocket.UnixSocketChannel.doConnect(UnixSocketChannel.java:127)at jnr.unixsocket.UnixSocketChannel.connect(UnixSocketChannel.java:136)at jnr.unixsocket.UnixSocketChannel.connect(UnixSocketChannel.java:223)at com.spotify.docker.client.UnixConnectionSocketFactory.connectSocket(UnixConnectionSocketFactory.java:85)at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376)at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72)at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:480)... 14 more
Looking at the logs from the shinyproxy Pod I was able to retrieve also the following entries:
2021-04-16 10:47:34.563 INFO 1 --- [sync-executor-0] o.apache.http.impl.execchain.RetryExec : I/O exception (java.io.IOException) caught when processing request to {}->unix://localhost:80: No such file or directory
2021-04-16 10:47:34.563 INFO 1 --- [sync-executor-0] o.apache.http.impl.execchain.RetryExec : Retrying request to {}->unix://localhost:80
2021-04-16 10:47:34.565 INFO 1 --- [sync-executor-0] o.apache.http.impl.execchain.RetryExec : I/O exception (java.io.IOException) caught when processing request to {}->unix://localhost:80: No such file or directory
2021-04-16 10:47:34.565 INFO 1 --- [sync-executor-0] o.apache.http.impl.execchain.RetryExec : Retrying request to {}->unix://localhost:80
2021-04-16 10:47:34.567 INFO 1 --- [sync-executor-0] o.apache.http.impl.execchain.RetryExec : I/O exception (java.io.IOException) caught when processing request to {}->unix://localhost:80: No such file or directory
2021-04-16 10:47:34.568 INFO 1 --- [sync-executor-0] o.apache.http.impl.execchain.RetryExec : Retrying request to {}->unix://localhost:80
2021-04-16 10:47:34.619 ERROR 1 --- [ XNIO-1 task-1] io.undertow.request : UT005023: Exception handling request to /app/01_hello
I could not figure it out where the unix://localhost:80 is coming from. All the manifest deployed are the ones under the clustered directory in the repo, the only change I made was on the shinyproxy manifest that looks like the following:
apiVersion: openanalytics.eu/v1alpha1
kind: ShinyProxy
metadata:
name: shinyproxy-pod
namespace: shinyproxy-apps
spec:
logging:
level:
root: DEBUG
fqdn: shinyproxy.local
image: openanalytics/shinyproxy:2.5.0
appNamespaces:
- my-namespace
proxy:
authentication: simple
containerBackend: kubernetes
heartbeatRate: 10000
heartbeatTimeout: 60000
kubernetes:
internal-networking: true
landingPage: /
logoUrl: https://www.openanalytics.eu/shinyproxy/logo.png
port: 8080
specs:
- containerCmd:
- R
- -e
- shinyproxy::run_01_hello()
containerImage: openanalytics/shinyproxy-demo
description: Application which demonstrates the basics of a Shiny app
displayName: Hello Application
id: 01_hello
kubernetes-pod-patches: |-
- op: add
path: /spec/containers/0/env
value:
- name: SOME_PASSWORD
valueFrom:
secretKeyRef:
name: some-password
key: password
title: Open Analytics Shiny Proxy
users:
- name: john
password: password
- name: jeff
password: password
spring:
redis:
host: redis
password: ${REDIS_PASSWORD}
session:
store-type: redis
I "exec'ed" into the shinyproxy Pod and could confirm the shinyproxy container was using all the info above as it was on the file /etc/shinyproxy/application.yml'. So I am not sure if there are any further parameters that need to be passed under proxy.kubernetes` in order to work or there is a bug somewhere, I had tried adding the following but to no avail:
...
kubernetes:
internal-networking: true
cert-path: /etc/certs
url: https://kubernetes.default.svc.cluster.local
...
Any insights are appreciated ;)
LEDfan commented
Hi @massaox
Since you are using the Docker image of ShinyProxy 2.5.0 , the configuration file should not be mounted under /etc/shinyproxy/application.yml, but under `/opt/shinyproxy/application.yml'. This changed since the docker image of 2.5.0. This is reflected in the latest versions of the ShinyProxy operator (at least openanalytics/shinyproxy-operator-snapshot:0.0.1-SNAPSHOT-20210302.095930) . Unfortunately this was not yet updated in the example deployments.
I'm re-writing the example deployments as we speak (to provide more re-usable examples). So I'll make sure to update the image version.
Please let me know if this fixes your issue.
LEDfan commented
Note you can use openanalytics/shinyproxy-operator-snapshot:latest as image for the operator, to ensure you are always using the latest version (we are using that tag in the new docs now as well).
massaox commented
HI @LEDfan,
Thank you very much for the quick response. You were absolutely right. Once I changed the image of the operator to run the latest one it worked flawlessly. The configMap was indeed mounted under /opt/shinyproxy/application.yaml and no further configuration was need.
I will go ahead and close down this issue. Keep up the good work!