Occurrence of errors while runninig workload from fb-oss-performance benchmark

Question

Occurrence of errors while runninig workload from fb-oss-performance benchmark

ZahraHeydari95 opened this issue 3 years ago · 7 comments

I have a problem running workload from fb-oss-performance benchmark.
I get the following error from every ip I use.
Even localhost has the same problem.
please guide me.
Thanks.

`root@zahra:/home/zahra# docker run --net=host --name=fb -v /home/zahra/cmd.sh:/oss-performance/cmd.sh cloudsuite/fb-oss-performance:4.0

'[' '!' -f /oss-performance/cmd.sh ']'
bash /oss-performance/cmd.sh
Warning: Permanently added '[192.168.43.204]:9801' (ECDSA) to the list of known hosts.

** 2022-05-30 03:46:50 UTC
** Configuration: MediaWikiTarget on /usr/local/bin/hhvm

** 2022-05-30 03:46:50 UTC
** Installing framework

Warning: mysql_connect(): Authentication plugin 'caching_sha2_password' cannot be loaded: /usr/local/mysql/lib/plugin/caching_sha2_password.so: cannot open shared object file: No such file or directory in /oss-performance/base/DatabaseInstaller.php on line 61

Warning: mysql_select_db() expects parameter 2 to be resource, boolean given in /oss-performance/base/DatabaseInstaller.php on line 62
Can't connect to database (mysql -h 192.168.43.204 -pmw_bench -u mw_bench mw_bench). This can be fixed for you.
ERROR 2061 (HY000): RSA Encryption not supported - caching_sha2_password plugin was built with GnuTLS support

gzip: stdout: Broken pipe

Fatal error: Uncaught exception 'Exception' with message 'Database installation failed: ' in /oss-performance/base/DatabaseInstaller.php:99
Stack trace:
#0 /oss-performance/targets/mediawiki/MediaWikiTarget.php(35): DatabaseInstaller->installDatabase()
#1 /oss-performance/base/PerfRunner.php(54): MediaWikiTarget->install()
#2 /oss-performance/base/PerfRunner.php(41): PerfRunner::RunWithOptionsAndEngine()
#3 /oss-performance/base/PerfRunner.php(16): PerfRunner::RunWithOptions()
#4 /oss-performance/perf.php(14): PerfRunner::RunWithArgv()
#5 /oss-performance/perf.php(18): perf_main()
#6 {main}
`

Answer 1 · 2022-05-30T06:56:56.000Z

Hello,
Can you show all the commands you use to run the workload? It seems that your way of setup the database is not expected. Are you also using the database docker image provided in CloudSuite?

Best,

Answer 2 · 2022-05-30T08:02:53.000Z

Hello, Can you show all the commands you use to run the workload? It seems that your way of setup the database is not expected. Are you also using the database docker image provided in CloudSuite?

Best,

Hello,
Thank you for your response.

Yes, I did all the installation and execution steps according to the document in the link https://github.com/parsa-epfl/cloudsuite/blob/main/docs/benchmarks/fb-oss-performance.md

Only in "docker run --name=siege -dt --net=host cloudsuite/siege:4.0.3rc3 <HOST_IP_OF_FB_SERVER> <HOSTNAME_OF_FB_SERVER>"
I did not consider <HOST_IP_OF_FB_SERVER> <HOSTNAME_OF_FB_SERVER>, because siege host is the same as fb-oss-performance: 4.0.

I put the content of the cmd.sh file as follows:

> .#!/bin/bash
> 
> set -eu
> 
> LOCAL_IP=$(ip route get 1.0.0.0 | grep src | cut -d' ' -f7)
> 
> echo "The IP address is $LOCAL_IP"
> 
> .# mysql server ip
> MYSQL_IP=$LOCAL_IP
> SIEGE_IP=$LOCAL_IP
> 
> 
> .# Max connections >1000 required for mediawiki workload. If mysql already has max_connections > 1000 comment the below line.
> mysql --host=$MYSQL_IP --user=root --password=root -e "SET GLOBAL max_connections = 1001;"
> 
> .# run hhvm
> hhvm perf.php --i-am-not-benchmarking --mediawiki --db-host=$MYSQL_IP --db-username=root --db-password=root --hhvm=$HHVM_BIN --remote-siege=root@$SIEGE_IP
> 
> .# run php_cgi7
> .# hhvm perf.php --i-am-not-benchmarking --mediawiki --db-host=$MYSQL_IP --db-username=root --db-password=root --php5=$PHP_CGI7_BIN --remote-siege=root@$SIEGE_IP
> 
> .# run php_fpm7
> .# hhvm perf.php --i-am-not-benchmarking --mediawiki --db-host=$MYSQL_IP --db-username=root --db-password=root --php=$PHP_FPM7_BIN --remote-siege=root@$SIEGE_IP
> 
> 
>

The error I am currently experiencing is as follows:

`root@zahra:~# docker run --net=host --name=fb -v /home/zahra/cmd.sh:/oss-performance/cmd.sh cloudsuite/fb-oss-performance:4.0

    '[' '!' -f /oss-performance/cmd.sh ']'
    bash /oss-performance/cmd.sh
    The IP address is 192.168.134.67
    ssh: connect to host 192.168.134.67 port 9801: Connection refused

Fatal error: Uncaught exception 'HH\InvariantException' with message 'Invalid ssh credentials: root@192.168.134.67' in /oss-performance/base/PerfOptions.php:396
Stack trace:
#0 /oss-performance/base/PerfOptions.php(396): HH\invariant_violation()
https://github.com/parsa-epfl/cloudsuite/pull/1 /oss-performance/base/PerfRunner.php(48): PerfOptions->validate()
https://github.com/parsa-epfl/cloudsuite/pull/2 /oss-performance/base/PerfRunner.php(41): PerfRunner::RunWithOptionsAndEngine()
https://github.com/parsa-epfl/cloudsuite/pull/3 /oss-performance/base/PerfRunner.php(16): PerfRunner::RunWithOptions()
https://github.com/parsa-epfl/cloudsuite/pull/4 /oss-performance/perf.php(14): PerfRunner::RunWithArgv()
https://github.com/parsa-epfl/cloudsuite/pull/5 /oss-performance/perf.php(18): perf_main()
https://github.com/parsa-epfl/cloudsuite/pull/6 {main}

`

Answer 3 · 2022-05-30T10:11:05.000Z

Hello,

If you carefully check the error message:

ssh: connect to host 192.168.134.67 port 9801: Connection refused

You can find that the problem is because of the server running siege. It either has the 9801 occupied by other applications, or the server is listening on the wrong IP address. In this case, you should first check the occupation, and then maybe strictly follow the guideline to set up the siege server.

Best,

Answer 4 · 2022-05-31T17:32:51.000Z

Hello,

If you carefully check the error message:
ssh: connect to host 192.168.134.67 port 9801: Connection refused
You can find that the problem is because of the server running siege. It either has the 9801 occupied by other applications, or the server is listening on the wrong IP address. In this case, you should first check the occupation, and then maybe strictly follow the guideline to set up the siege server.

Best,

I solved the port problem but now I get the following error:


> root@zahra:~# docker rm fb && docker run --net=host --name=fb -v /home/zahra/cmd.sh:/oss-performance/cmd.sh cloudsuite/fb-oss-performance:4.0
> fb
> + '[' '!' -f /oss-performance/cmd.sh ']'
> + bash /oss-performance/cmd.sh
> The IP address is 10.16.0.46
> ERROR 2061 (HY000): RSA Encryption not supported - caching_sha2_password plugin was built with GnuTLS support

Answer 5 · 2022-05-31T20:59:16.000Z

Isn't this the same problem you faced in the beginning?

Answer 6 · 2022-06-01T00:55:51.000Z

Isn't this the same problem you faced in the beginning?

Yes, it is true, but when I solve the problem of this error, I encounter the following error:

root@zahra:~# docker rm fb && docker run --net=host --name=fb -v /home/zahra/cmd.sh:/oss-performance/cmd.sh cloudsuite/fb-oss-performance:4.0
fb
+ '[' '!' -f /oss-performance/cmd.sh ']'
+ bash /oss-performance/cmd.sh
The IP address is 10.18.0.85
Warning: Permanently added '[10.18.0.85]:9801' (ECDSA) to the list of known hosts.
Permission denied, please try again.
Permission denied, please try again.
root@10.18.0.85: Permission denied (publickey,password).

Fatal error: Uncaught exception 'HH\InvariantException' with message 'Invalid ssh credentials: root@10.18.0.85' in /oss-performance/base/PerfOptions.php:396
Stack trace:
#0 /oss-performance/base/PerfOptions.php(396): HH\invariant_violation()
#1 /oss-performance/base/PerfRunner.php(48): PerfOptions->validate()
#2 /oss-performance/base/PerfRunner.php(41): PerfRunner::RunWithOptionsAndEngine()
#3 /oss-performance/base/PerfRunner.php(16): PerfRunner::RunWithOptions()
#4 /oss-performance/perf.php(14): PerfRunner::RunWithArgv()
#5 /oss-performance/perf.php(18): perf_main()
#6 {main}

While outside the docker run command, the ssh 10.18.0.85 -p 9801 command works without an access permission error and without asking for a password.

`root@zahra:~# ssh 10.18.0.85 -p 9801
Welcome to Ubuntu 20.04.4 LTS (GNU/Linux 5.4.0-113-generic x86_64)
 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

0 updates can be applied immediately.

Last login: Wed Jun  1 04:45:53 2022 from 10.18.0.85

`

Another problem I noticed is that siege container is not added to the running container (docker ps) . Unless "--net=host" removed from the command "docker run --name=siege -dt --net=host cloudsuite/siege:4.0.3rc3".

Answer 7 · 2022-06-01T09:02:18.000Z

First of all, the ssh run by hhvm is actually the siege docker. As far as I remember, our siege docker uses Debian image, so it should never give you Ubuntu prompt when you log in unless you are logging in to your own operating system.
Second, you should definitely check what happens when running siege if siege docker is not on the active list. Do you face any errors when running siege?