Constantly getting Couldn't start appServer tomcat on AuctionWeb1 in Docker container deployment

Question

Constantly getting Couldn't start appServer tomcat on AuctionWeb1 in Docker container deployment

mafzal786 opened this issue 7 years ago · 25 comments

I'm getting this randomly on different container
Couldn't start appServer tomcat on AuctionWeb1
sometimes
Couldn't start appServer tomcat on AuctionCm1
Couldn't start appServer tomcat on AuctionCs1 or
Couldn't start appServer tomcat on AuctionDriver1

After looking into logs files in /ouput/9/logs/appServer/AuctionWeb1/AuctionApp1D/Tom....log

I see the following:

12-Jul-2017 19:22:34.712 SEVERE [localhost-startStop-1] org.apache.catalina.core.StandardContext.listenerStart Exception sending context initialized event to listener instance of class [org.springframework.web.context.ContextLoaderListener]
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'entityManagerFactory' defined in ServletContext resource [/WEB-INF/spring/jpa-context.xml]: Cannot resolve reference to bean 'dataSource' while setting bean property 'dataSource'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'dataSource': Invocation of init method failed; nested exception is javax.naming.NameNotFoundException: Name [jdbc/auction] is not bound in this Context. Unable to find [jdbc].
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:359)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:108)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1531)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1276)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:553)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1081)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:856)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:542)
at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:443)
at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:325)
at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:107)
at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4745)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5207)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1419)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1409)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'dataSource': Invocation of init method failed; nested exception is javax.naming.NameNotFoundException: Name [jdbc/auction] is not bound in this Context. Unable to find [jdbc].
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1628)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:555)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351)
... 24 more
Caused by: javax.naming.NameNotFoundException: Name [jdbc/auction] is not bound in this Context. Unable to find [jdbc].
at org.apache.naming.NamingContext.lookup(NamingContext.java:816)
at org.apache.naming.NamingContext.lookup(NamingContext.java:173)
at org.apache.naming.SelectorContext.lookup(SelectorContext.java:163)
at javax.naming.InitialContext.lookup(InitialContext.java:417)
at org.springframework.jndi.JndiTemplate$1.doInContext(JndiTemplate.java:155)
at org.springframework.jndi.JndiTemplate.execute(JndiTemplate.java:87)
at org.springframework.jndi.JndiTemplate.lookup(JndiTemplate.java:152)
at org.springframework.jndi.JndiTemplate.lookup(JndiTemplate.java:179)
at org.springframework.jndi.JndiLocatorSupport.lookup(JndiLocatorSupport.java:104)
at org.springframework.jndi.JndiObjectLocator.lookup(JndiObjectLocator.java:106)
at org.springframework.jndi.JndiObjectFactoryBean.lookupWithFallback(JndiObjectFactoryBean.java:231)
at org.springframework.jndi.JndiObjectFactoryBean.afterPropertiesSet(JndiObjectFactoryBean.java:217)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1687)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1624)
... 31 more

12-Jul-2017 19:22:50.370 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log Closing Spring root WebApplicationContext
12-Jul-2017 19:32:38.578 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log Closing Spring root WebApplicationContext
113

Answer 1 · 2017-07-13T13:41:38.000Z

Can you please attach the weathervane.config file that you are using for the run. That will help me understand what is happening a little better.

Answer 2 · 2017-07-13T14:42:27.000Z

{
"users" : 60,
"maxUsers" : 60,

"description" : "Simple Weathervane Config.",
"runLength" : "medium",

"numWorkloads" : 1,
"numAppInstances" : 1,
"numDrivers" : 1,
"numConfigurationManagers" : 1,
#"numElasticityServices" : 0,
"numCoordinationServers" : 1,
#"numLbServers" : 0,
"numWebServers" : 1,
"numAppServers" : 1,
"numMsgServers" : 1,
#"numFileServers" : 0,
"numNosqlServers" : 1,
#"nosqlSharded" : false,
#"nosqlReplicated" : false,
"numDbServers" : 1,

#"logLevel" : 0,
#"runStrategy" : "single",
#"powerOnVms" : false,
#"powerOffVms" : false,

#"numViHosts" : 0,

"useDocker" : true,
"dockerNamespace" : "xxxxx",

"dockerHostPin" : false,

"dockerHostPinMode" : "all",

"dockerCpuShares" : n/a,

"dockerCpuSetCpus" : n/a,

"dockerCpuSetMems" : n/a,

"dockerMemory" : n/a,

"dockerMemorySwap" : n/a,

"dockerNet" : "bridge",

"dockerRegistryName" : "vmc-perf-h11",

}

Answer 3 · 2017-07-13T18:11:22.000Z

Are you running everything on one VM or Docker host? I'm not sure why the hostname for which you get the error message keeps changing.

Can you create a tar-file from the output directory and attach it to the bug? It will be alot quicker than me asking for one file at a time.

The command would be:

tar cvf 9.tgz /root/weathervane/output/9

Then attach the file 9.tgz that is created.

Answer 4 · 2017-07-13T18:22:06.000Z

it is docker bare-metal host. log is attached. it shows jdbc connection issues.

9.zip

Answer 5 · 2017-07-13T18:54:11.000Z

It looks as though the jdbc issue is caused by this error:

java.sql.SQLException: Unable to load class: org.postgresql.Driver

If the build of the Docker images went properly, the jar file for the driver class should be in the image in the proper place.

Can you look in the file buildDockerImages.log for errors. This file is created when running the buildDockerImages.pl script, and should be in /root/weathervane.

Answer 6 · 2017-07-14T01:12:46.000Z

tried re-creating tomcat image with the Dockerfile. I'm getting the following in the end

cp: cannot create regular file '/opt/apache-tomcat-auction1/bin/': Not a directory

Answer 7 · 2017-07-14T01:38:14.000Z

look like this line is failing in creating docker image

cp /opt/apache-tomcat/bin/tomcat-juli.jar /opt/apache-tomcat-auction1/bin/

Answer 8 · 2017-07-14T03:29:55.000Z

On the side note, in zookeeper dockerfile

curl -s http://www.us.apache.org/dist/zookeeper/stable/zookeeper-3.4.9.tar.gz -o zookeeper-3.4.9.tar.gz && \

3.4.9 does not exist. it needs to be changed to 3.4.10. That curl link is invalid. Can you please verify.

Thanks

Answer 9 · 2017-07-14T15:23:41.000Z

You are right about the version of Zookeeper.

I have fixed the Dockerfiles and the buildDockerImages.pl script to always get the most recent available version of Tomcat and Zookeeper. The changes were pushed to the master branch in this commit: 7819be3

If you get Weathervane by cloning the repository, you should be able to get the latest versions by doing a git pull. If you deployed a release package, you can get the changed files from that commit link, and copy them over the respective files in your deployment.

Then try build the docker images again and hopefully everything will work.

Answer 10 · 2017-07-15T03:20:25.000Z

Version patch did not help!!! tomcat dockerfile failed as tar file failed to download. I changed the Dockerfile to use 8.5.16 in the url than the TOMCAT_VERSION variable, and it worked. but later it failed on the same cp command in Dockerfile as follows

cp Command: cp /opt/apache-tomcat/bin/tomcat-juli.jar /opt/apache-tomcat-auction1/bin/ && \

Error as below. it does not find the /bin directory
cp: cannot create regular file '/opt/apache-tomcat-auction1/bin/': Not a directory

Answer 11 · 2017-07-17T18:01:56.000Z

Can you please post the entire buildDockerImages.log file. I am not sure why it is not working.

Answer 12 · 2017-07-17T18:04:43.000Z

Never mind. I just discovered that there is a bug in the updated script. I will fix it and let you know when I have pushed up the fix.

Answer 13 · 2017-07-17T19:28:21.000Z

There was a problem with the way that I was referencing the Tomcat version number in the dockerfile. I have fixed this in commit e057cf6.

Can you please take the file dockerImages/tomcat/Dockerfile from that commit, use it to replace the file /root/weathervane/dockerImages/tomcat/Dockerfile on your system, and rety building the Docker images.

Answer 14 · 2017-07-17T20:28:37.000Z

now download worked fine for tomcat, but the real issue is not resolved. still seeing this cp command failed.

Answer 15 · 2017-07-17T20:31:02.000Z

Can you please attach the entire buildDockerImages.log file

Answer 16 · 2017-07-17T20:51:34.000Z

attached
buildDockerImages.log.zip

Answer 17 · 2017-07-17T21:27:51.000Z

Can you please confirm for me that there isn't something missing from your Weathervane installation.

Under the directory /root/weathervane/dockerImages/tomcat/apache-tomcat-auction1, there should be three subdirectories:

bin
conf
lib
All of these directories should contain at least one file.

Answer 18 · 2017-07-17T21:30:39.000Z

i see only conf and webapps folder. no other folders you mentioned.

Answer 19 · 2017-07-17T21:30:48.000Z

OK. I found the issue. These files exist on my systems, but the .gitgnore is preventing them from being added to the repository. I will fix this and create a new release.

Answer 20 · 2017-07-18T12:24:29.000Z

I have push a fix in ad27fdc that makes the missing files available.

If you grab the files in dockerImages/tomcat/apache-tomcat-auction1/bin and dockerImages/tomcat/apache-tomcat-auction1/lib and place them in the correct place on your system then hopefully everything will work.

Answer 21 · 2017-07-18T17:50:50.000Z

I'm able to create tomcat image now. also weathvane.pl ran but it ended up on this

The workload driver nodes for workload 1 did not start within 4 minutes. Exiting

Answer 22 · 2017-07-19T01:07:46.000Z

what could be the reason for that

Answer 23 · 2017-07-19T21:19:26.000Z

Sorry about the slow response.

Can you please attach a zip of the output directory: /root/weathervane/output/15

Failing proportion checks can happen for a few reasons:

The application deployment cannot handle the load placed by the selected number of users.
The run is too short, or the number of users too low, to reach the expected steady-state proportions. In short or low-load runs, random variation can cause these checks to fail.
There was some sort of error during the run that was not caught by the run harness.

I suspect some sort of error in this case, as the failing operations are all those that a user performs when it initially starts. This pattern of proportion check failures sometime occurs if the auctions did not start for some reason. I need to see the logs in order to help debug.

Answer 24 · 2017-07-21T00:30:40.000Z

it finally worked after reducing number of users and making some changes in weathervane.forward.zone file.

Is there any option to use overlay network with weathervane.config file? I see it only support bridge and host network.

Does it work with docker in swarm mode?

Thanks

Answer 25 · 2017-07-21T17:42:14.000Z

Weathervane does not currently support Docker swarm.

Adding support for Kubernetes and Docker swarm is high on our priority list, but probably won't happen for a month or more.

If all that you need from overlay networks is to have a custom network name in the --network setting, then that works already. Just set the parameter "dockerNet" to the name of your network. The documentation claims that only "host" and "bridge" are supported, but that is out of date.