spcl/serverless-benchmarks

THE COMPLETE PICTURE OF MY ERRORS

jchigu opened this issue · 1 comments

jchigu commented

The following is a complete picture of the errors as of today 05 January 2023. I would be grateful for your assistance.

(python-virtualenv) justinchigu@justinchigu-VirtualBox:/serverless-benchmarks$ ls
110.dynamic-html_code install.py package-lock.json requirements.txt
benchmarks LICENSE.md python-venv scripts
benchmarks-data mc README.md sebs
cache minio regression-cache sebs.py
check-config.sh node_modules regression-output templates
CITATION.cff out_benchmark.json requirements.aws.txt tests
config out.log requirements.azure.txt third-party
docker out_storage.json requirements.gcp.txt tools
docs package.json requirements.local.txt
(python-virtualenv) justinchigu@justinchigu-VirtualBox:/serverless-benchmarks$ ./sebs.py storage start minio --port 9011 --output-json out_storage.json
10:51:45,820 INFO root: Starting storage Storage.MINIO on port 9011.
10:51:45,823 INFO minio.Minio-5465: Minio storage ACCESS_KEY=P3-bFE-8D-rRqlTuVBOS9X2VtB9zgF0L4LXN6JzVuBc
10:51:45,823 INFO minio.Minio-5465: Minio storage SECRET_KEY=b3f9d5ae67d1d53e971e653f4f951d3fbe5ba99b1c0990b66303611766cc51b2
10:51:47,98 INFO minio.Minio-5465: Starting minio instance at 172.17.0.2:9000
10:51:47,98 INFO root: Writing storage configuration to out_storage.json.
(python-virtualenv) justinchigu@justinchigu-VirtualBox:/serverless-benchmarks$ jq --argfile file1 out_storage.json '.deployment.local.storage = $file1 ' config/example.json > config/local_deployment.json
(python-virtualenv) justinchigu@justinchigu-VirtualBox:/serverless-benchmarks$ ./sebs.py local start 110.dynamic-html test out_benchmark.json --config config/local_deployment.json --deployments 1
(python-virtualenv) justinchigu@justinchigu-VirtualBox:/serverless-benchmarks$ curl 172.17.0.3:9000 --request POST --data '{"random_len": 10,"username": "testname"}' --header 'Content-Type: application/json'
{"begin": "1672911428.612681", "end": "1672911428.680205", "request_id": "7583e2fc-a113-49c8-86a9-2869b70b4a72", "is_cold": false, "result": {"output": {"result": "\n\n \n <title>Randomly generated data.</title>\n <meta name="viewport" content="width=device-width, initial-scale=1.0">\n <link href="http://netdna.bootstrapcdn.com/bootstrap/3.0.0/css/bootstrap.min.css\" rel="stylesheet" media="screen">\n <style type="text/css">\n .container {\n max-width: 500px;\n padding-top: 100px;\n }\n </style>\n \n \n <div class="container">\n

Welcome testname!

\n

Data generated at: 2023-01-05 09:37:08.680302!

\n

Requested random numbers:

\n
    \n \n
  • 907992
    • \n \n
  • 609716
    • \n \n
  • 723125
    • \n \n
  • 539621
    • \n \n
  • 227802
    • \n \n
  • 359026
    • \n \n
  • 488992
    • \n \n
  • 986015
    • \n \n
  • 539617
    • \n \n
  • 762779
    • \n \n
\n \n \n"}}}(python-virtualenv) justinchigu@justinchigu-VirtualBo
(python-virtualenv) justinchigu@justinchigu-VirtualBox:/serverless-benchmarks$ ./sebs.py benchmark invoke 110.dynamic-html test --config config/example.json --deployment aws --verbose
11:45:54,33 INFO SeBS-74e8: Created experiment output at /home/justinchigu/serverless-benchmarks
ERROR:root:Deployment aws not supported!
Traceback (most recent call last):
File "/home/justinchigu/serverless-benchmarks/./sebs.py", line 30, in call
return self.main(*args, **kwargs)
File "/home/justinchigu/python-virtualenv/lib/python3.10/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
File "/home/justinchigu/python-virtualenv/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/justinchigu/python-virtualenv/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/justinchigu/python-virtualenv/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/justinchigu/python-virtualenv/lib/python3.10/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/home/justinchigu/serverless-benchmarks/./sebs.py", line 72, in wrapper
return func(*args, **kwargs)
File "/home/justinchigu/serverless-benchmarks/./sebs.py", line 97, in wrapper
return func(*args, **kwargs)
File "/home/justinchigu/serverless-benchmarks/./sebs.py", line 215, in invoke
) = parse_common_params(**kwargs)
File "/home/justinchigu/serverless-benchmarks/./sebs.py", line 135, in parse_common_params
deployment_client = sebs_client.get_deployment(
File "/home/justinchigu/serverless-benchmarks/sebs/sebs.py", line 106, in get_deployment
raise RuntimeError("Deployment {name} not supported!".format(name=name))
RuntimeError: Deployment aws not supported!
(python-virtualenv) justinchigu@justinchigu-VirtualBox:~/serverless-benchmarks$ ./sebs.py benchmark invoke --help
Usage: sebs.py benchmark invoke [OPTIONS] BENCHMARK {test|small|large}

Options:
--repetitions INTEGER Number of experimental repetitions.
--trigger [library|http] Function trigger to be used.
--memory INTEGER Override default memory settings for the
benchmark function.
--timeout INTEGER Override default timeout settings for the
benchmark function.
--function-name TEXT Override function name for random
generation.
--image-tag-prefix TEXT Attach prefix to generated Docker image tag.
--update-code / --no-update-code
Update function code in cache and cloud
deployment.
--update-storage / --no-update-storage
Update benchmark storage files in cloud
deployment.
--deployment [azure|aws|gcp|local|openwhisk]
Cloud deployment to use.
--config PATH Location of experiment config. [required]
--output-dir TEXT Output directory for results.
--output-file TEXT Output filename for logging.
--cache TEXT Location of experiments cache.
--verbose / --no-verbose Verbose output.
--preserve-out / --no-preserve-out
Preserve current results in output
directory.
--language [python|nodejs] Benchmark language
--language-version TEXT Benchmark language version
--help Show this message and exit.
(python-virtualenv) justinchigu@justinchigu-VirtualBox:~/serverless-benchmarks$ ./sebs.py benchmark regression test --config config/example.json --deployment aws
Select test test_110.dynamic-html
Select test test_120.uploader
Select test test_210.thumbnailer
Select test test_220.video-processing
Select test test_311.compression
Select test test_411.image-recognition
Select test test_501.graph-pagerank
Select test test_502.graph-mst
Select test test_503.graph-bfs
Select test test_504.dna-visualisation

sebs.regression.AWSTestSequence.test_210.thumbnailer: fail
sebs.regression.AWSTestSequence.test_210.thumbnailer: Traceback (most recent call last):
File "/home/justinchigu/serverless-benchmarks/sebs/regression.py", line 38, in test
deployment_client = self.get_deployment(benchmark_name)
File "/home/justinchigu/serverless-benchmarks/sebs/regression.py", line 103, in get_deployment
deployment_client = self.client.get_deployment(
File "/home/justinchigu/serverless-benchmarks/sebs/sebs.py", line 106, in get_deployment
raise RuntimeError("Deployment {name} not supported!".format(name=name))
RuntimeError: Deployment aws not supported!

sebs.regression.AWSTestSequence.test_120.uploader: fail
sebs.regression.AWSTestSequence.test_120.uploader: Traceback (most recent call last):
File "/home/justinchigu/serverless-benchmarks/sebs/regression.py", line 38, in test
deployment_client = self.get_deployment(benchmark_name)
File "/home/justinchigu/serverless-benchmarks/sebs/regression.py", line 103, in get_deployment
deployment_client = self.client.get_deployment(
File "/home/justinchigu/serverless-benchmarks/sebs/sebs.py", line 106, in get_deployment
raise RuntimeError("Deployment {name} not supported!".format(name=name))
RuntimeError: Deployment aws not supported!

sebs.regression.AWSTestSequence.test_220.video-processing: fail
sebs.regression.AWSTestSequence.test_220.video-processing: Traceback (most recent call last):
File "/home/justinchigu/serverless-benchmarks/sebs/regression.py", line 38, in test
deployment_client = self.get_deployment(benchmark_name)
File "/home/justinchigu/serverless-benchmarks/sebs/regression.py", line 103, in get_deployment
deployment_client = self.client.get_deployment(
File "/home/justinchigu/serverless-benchmarks/sebs/sebs.py", line 106, in get_deployment
raise RuntimeError("Deployment {name} not supported!".format(name=name))
RuntimeError: Deployment aws not supported!

sebs.regression.AWSTestSequence.test_110.dynamic-html: fail
sebs.regression.AWSTestSequence.test_110.dynamic-html: Traceback (most recent call last):
File "/home/justinchigu/serverless-benchmarks/sebs/regression.py", line 38, in test
deployment_client = self.get_deployment(benchmark_name)
File "/home/justinchigu/serverless-benchmarks/sebs/regression.py", line 103, in get_deployment
deployment_client = self.client.get_deployment(
File "/home/justinchigu/serverless-benchmarks/sebs/sebs.py", line 106, in get_deployment
raise RuntimeError("Deployment {name} not supported!".format(name=name))
RuntimeError: Deployment aws not supported!

sebs.regression.AWSTestSequence.test_411.image-recognition: fail
sebs.regression.AWSTestSequence.test_411.image-recognition: Traceback (most recent call last):
File "/home/justinchigu/serverless-benchmarks/sebs/regression.py", line 38, in test
deployment_client = self.get_deployment(benchmark_name)
File "/home/justinchigu/serverless-benchmarks/sebs/regression.py", line 103, in get_deployment
deployment_client = self.client.get_deployment(
File "/home/justinchigu/serverless-benchmarks/sebs/sebs.py", line 106, in get_deployment
raise RuntimeError("Deployment {name} not supported!".format(name=name))
RuntimeError: Deployment aws not supported!

sebs.regression.AWSTestSequence.test_311.compression: fail
sebs.regression.AWSTestSequence.test_311.compression: Traceback (most recent call last):
File "/home/justinchigu/serverless-benchmarks/sebs/regression.py", line 38, in test
deployment_client = self.get_deployment(benchmark_name)
File "/home/justinchigu/serverless-benchmarks/sebs/regression.py", line 103, in get_deployment
deployment_client = self.client.get_deployment(
File "/home/justinchigu/serverless-benchmarks/sebs/sebs.py", line 106, in get_deployment
raise RuntimeError("Deployment {name} not supported!".format(name=name))
RuntimeError: Deployment aws not supported!

sebs.regression.AWSTestSequence.test_501.graph-pagerank: fail
sebs.regression.AWSTestSequence.test_501.graph-pagerank: Traceback (most recent call last):
File "/home/justinchigu/serverless-benchmarks/sebs/regression.py", line 38, in test
deployment_client = self.get_deployment(benchmark_name)
File "/home/justinchigu/serverless-benchmarks/sebs/regression.py", line 103, in get_deployment
deployment_client = self.client.get_deployment(
File "/home/justinchigu/serverless-benchmarks/sebs/sebs.py", line 106, in get_deployment
raise RuntimeError("Deployment {name} not supported!".format(name=name))
RuntimeError: Deployment aws not supported!

sebs.regression.AWSTestSequence.test_502.graph-mst: fail
sebs.regression.AWSTestSequence.test_502.graph-mst: Traceback (most recent call last):
File "/home/justinchigu/serverless-benchmarks/sebs/regression.py", line 38, in test
deployment_client = self.get_deployment(benchmark_name)
File "/home/justinchigu/serverless-benchmarks/sebs/regression.py", line 103, in get_deployment
deployment_client = self.client.get_deployment(
File "/home/justinchigu/serverless-benchmarks/sebs/sebs.py", line 106, in get_deployment
raise RuntimeError("Deployment {name} not supported!".format(name=name))
RuntimeError: Deployment aws not supported!

sebs.regression.AWSTestSequence.test_503.graph-bfs: fail
sebs.regression.AWSTestSequence.test_503.graph-bfs: Traceback (most recent call last):
File "/home/justinchigu/serverless-benchmarks/sebs/regression.py", line 38, in test
deployment_client = self.get_deployment(benchmark_name)
File "/home/justinchigu/serverless-benchmarks/sebs/regression.py", line 103, in get_deployment
deployment_client = self.client.get_deployment(
File "/home/justinchigu/serverless-benchmarks/sebs/sebs.py", line 106, in get_deployment
raise RuntimeError("Deployment {name} not supported!".format(name=name))
RuntimeError: Deployment aws not supported!

sebs.regression.AWSTestSequence.test_504.dna-visualisation: fail
sebs.regression.AWSTestSequence.test_504.dna-visualisation: Traceback (most recent call last):
File "/home/justinchigu/serverless-benchmarks/sebs/regression.py", line 38, in test
deployment_client = self.get_deployment(benchmark_name)
File "/home/justinchigu/serverless-benchmarks/sebs/regression.py", line 103, in get_deployment
deployment_client = self.client.get_deployment(
File "/home/justinchigu/serverless-benchmarks/sebs/sebs.py", line 106, in get_deployment
raise RuntimeError("Deployment {name} not supported!".format(name=name))
RuntimeError: Deployment aws not supported!

Succesfully executed 0 out of 10 functions
Failures when executing 10 out of 10 functions

  • 311.compression
  • 411.image-recognition
  • 503.graph-bfs
  • 220.video-processing
  • 210.thumbnailer
  • 120.uploader
  • 502.graph-mst
  • 504.dna-visualisation
  • 110.dynamic-html
  • 501.graph-pagerank
    (python-virtualenv) justinchigu@justinchigu-VirtualBox:~/serverless-benchmarks$ ./sebs.py benchmark regression test --config config/example.json --deployment azure
    Select test test_110.dynamic-html
    Select test test_120.uploader
    Select test test_210.thumbnailer
    Select test test_220.video-processing
    Select test test_311.compression
    Select test test_411.image-recognition
    Select test test_501.graph-pagerank
    Select test test_502.graph-mst
    Select test test_503.graph-bfs
    Select test test_504.dna-visualisation

sebs.regression.AzureTestSequence.test_220.video-processing: fail
sebs.regression.AzureTestSequence.test_220.video-processing: Traceback (most recent call last):
File "/home/justinchigu/serverless-benchmarks/sebs/regression.py", line 38, in test
deployment_client = self.get_deployment(benchmark_name)
File "/home/justinchigu/serverless-benchmarks/sebs/regression.py", line 122, in get_deployment
AzureTestSequence.cfg = self.client.get_deployment_config(
File "/home/justinchigu/serverless-benchmarks/sebs/sebs.py", line 127, in get_deployment_config
return Config.deserialize(config, self.cache_client, handlers)
File "/home/justinchigu/serverless-benchmarks/sebs/faas/config.py", line 126, in deserialize
assert func, "Unknown config type!"
AssertionError: Unknown config type!

sebs.regression.AzureTestSequence.test_110.dynamic-html: fail
sebs.regression.AzureTestSequence.test_110.dynamic-html: Traceback (most recent call last):
File "/home/justinchigu/serverless-benchmarks/sebs/regression.py", line 38, in test
deployment_client = self.get_deployment(benchmark_name)
File "/home/justinchigu/serverless-benchmarks/sebs/regression.py", line 122, in get_deployment
AzureTestSequence.cfg = self.client.get_deployment_config(
File "/home/justinchigu/serverless-benchmarks/sebs/sebs.py", line 127, in get_deployment_config
return Config.deserialize(config, self.cache_client, handlers)
File "/home/justinchigu/serverless-benchmarks/sebs/faas/config.py", line 126, in deserialize
assert func, "Unknown config type!"
AssertionError: Unknown config type!

sebs.regression.AzureTestSequence.test_120.uploader: fail
sebs.regression.AzureTestSequence.test_120.uploader: Traceback (most recent call last):
File "/home/justinchigu/serverless-benchmarks/sebs/regression.py", line 38, in test
deployment_client = self.get_deployment(benchmark_name)
File "/home/justinchigu/serverless-benchmarks/sebs/regression.py", line 122, in get_deployment
AzureTestSequence.cfg = self.client.get_deployment_config(
File "/home/justinchigu/serverless-benchmarks/sebs/sebs.py", line 127, in get_deployment_config
return Config.deserialize(config, self.cache_client, handlers)
File "/home/justinchigu/serverless-benchmarks/sebs/faas/config.py", line 126, in deserialize
assert func, "Unknown config type!"
AssertionError: Unknown config type!

sebs.regression.AzureTestSequence.test_210.thumbnailer: fail
sebs.regression.AzureTestSequence.test_210.thumbnailer: Traceback (most recent call last):
File "/home/justinchigu/serverless-benchmarks/sebs/regression.py", line 38, in test
deployment_client = self.get_deployment(benchmark_name)
File "/home/justinchigu/serverless-benchmarks/sebs/regression.py", line 122, in get_deployment
AzureTestSequence.cfg = self.client.get_deployment_config(
File "/home/justinchigu/serverless-benchmarks/sebs/sebs.py", line 127, in get_deployment_config
return Config.deserialize(config, self.cache_client, handlers)
File "/home/justinchigu/serverless-benchmarks/sebs/faas/config.py", line 126, in deserialize
assert func, "Unknown config type!"
AssertionError: Unknown config type!

sebs.regression.AzureTestSequence.test_411.image-recognition: fail
sebs.regression.AzureTestSequence.test_411.image-recognition: Traceback (most recent call last):
File "/home/justinchigu/serverless-benchmarks/sebs/regression.py", line 38, in test
deployment_client = self.get_deployment(benchmark_name)
File "/home/justinchigu/serverless-benchmarks/sebs/regression.py", line 122, in get_deployment
AzureTestSequence.cfg = self.client.get_deployment_config(
File "/home/justinchigu/serverless-benchmarks/sebs/sebs.py", line 127, in get_deployment_config
return Config.deserialize(config, self.cache_client, handlers)
File "/home/justinchigu/serverless-benchmarks/sebs/faas/config.py", line 126, in deserialize
assert func, "Unknown config type!"
AssertionError: Unknown config type!

sebs.regression.AzureTestSequence.test_311.compression: fail
sebs.regression.AzureTestSequence.test_311.compression: Traceback (most recent call last):
File "/home/justinchigu/serverless-benchmarks/sebs/regression.py", line 38, in test
deployment_client = self.get_deployment(benchmark_name)
File "/home/justinchigu/serverless-benchmarks/sebs/regression.py", line 122, in get_deployment
AzureTestSequence.cfg = self.client.get_deployment_config(
File "/home/justinchigu/serverless-benchmarks/sebs/sebs.py", line 127, in get_deployment_config
return Config.deserialize(config, self.cache_client, handlers)
File "/home/justinchigu/serverless-benchmarks/sebs/faas/config.py", line 126, in deserialize
assert func, "Unknown config type!"
AssertionError: Unknown config type!

sebs.regression.AzureTestSequence.test_501.graph-pagerank: fail
sebs.regression.AzureTestSequence.test_501.graph-pagerank: Traceback (most recent call last):
File "/home/justinchigu/serverless-benchmarks/sebs/regression.py", line 38, in test
deployment_client = self.get_deployment(benchmark_name)
File "/home/justinchigu/serverless-benchmarks/sebs/regression.py", line 122, in get_deployment
AzureTestSequence.cfg = self.client.get_deployment_config(
File "/home/justinchigu/serverless-benchmarks/sebs/sebs.py", line 127, in get_deployment_config
return Config.deserialize(config, self.cache_client, handlers)
File "/home/justinchigu/serverless-benchmarks/sebs/faas/config.py", line 126, in deserialize
assert func, "Unknown config type!"
AssertionError: Unknown config type!

sebs.regression.AzureTestSequence.test_502.graph-mst: fail
sebs.regression.AzureTestSequence.test_502.graph-mst: Traceback (most recent call last):
File "/home/justinchigu/serverless-benchmarks/sebs/regression.py", line 38, in test
deployment_client = self.get_deployment(benchmark_name)
File "/home/justinchigu/serverless-benchmarks/sebs/regression.py", line 122, in get_deployment
AzureTestSequence.cfg = self.client.get_deployment_config(
File "/home/justinchigu/serverless-benchmarks/sebs/sebs.py", line 127, in get_deployment_config
return Config.deserialize(config, self.cache_client, handlers)
File "/home/justinchigu/serverless-benchmarks/sebs/faas/config.py", line 126, in deserialize
assert func, "Unknown config type!"
AssertionError: Unknown config type!

sebs.regression.AzureTestSequence.test_503.graph-bfs: fail
sebs.regression.AzureTestSequence.test_503.graph-bfs: Traceback (most recent call last):
File "/home/justinchigu/serverless-benchmarks/sebs/regression.py", line 38, in test
deployment_client = self.get_deployment(benchmark_name)
File "/home/justinchigu/serverless-benchmarks/sebs/regression.py", line 122, in get_deployment
AzureTestSequence.cfg = self.client.get_deployment_config(
File "/home/justinchigu/serverless-benchmarks/sebs/sebs.py", line 127, in get_deployment_config
return Config.deserialize(config, self.cache_client, handlers)
File "/home/justinchigu/serverless-benchmarks/sebs/faas/config.py", line 126, in deserialize
assert func, "Unknown config type!"
AssertionError: Unknown config type!

sebs.regression.AzureTestSequence.test_504.dna-visualisation: fail
sebs.regression.AzureTestSequence.test_504.dna-visualisation: Traceback (most recent call last):
File "/home/justinchigu/serverless-benchmarks/sebs/regression.py", line 38, in test
deployment_client = self.get_deployment(benchmark_name)
File "/home/justinchigu/serverless-benchmarks/sebs/regression.py", line 122, in get_deployment
AzureTestSequence.cfg = self.client.get_deployment_config(
File "/home/justinchigu/serverless-benchmarks/sebs/sebs.py", line 127, in get_deployment_config
return Config.deserialize(config, self.cache_client, handlers)
File "/home/justinchigu/serverless-benchmarks/sebs/faas/config.py", line 126, in deserialize
assert func, "Unknown config type!"
AssertionError: Unknown config type!

Succesfully executed 0 out of 10 functions
Failures when executing 10 out of 10 functions

  • 220.video-processing
  • 501.graph-pagerank
  • 210.thumbnailer
  • 411.image-recognition
  • 502.graph-mst
  • 503.graph-bfs
  • 110.dynamic-html
  • 120.uploader
  • 311.compression
  • 504.dna-visualisation
    (python-virtualenv) justinchigu@justinchigu-VirtualBox:~/serverless-benchmarks$ ./sebs.py benchmark regression test --config config/example.json --deployment gcp
    Select test test_110.dynamic-html
    Select test test_120.uploader
    Select test test_210.thumbnailer
    Select test test_220.video-processing
    Select test test_311.compression
    Select test test_411.image-recognition
    Select test test_501.graph-pagerank
    Select test test_502.graph-mst
    Select test test_503.graph-bfs
    Select test test_504.dna-visualisation

sebs.regression.GCPTestSequence.test_110.dynamic-html: fail
sebs.regression.GCPTestSequence.test_110.dynamic-html: Traceback (most recent call last):
File "/home/justinchigu/serverless-benchmarks/sebs/regression.py", line 38, in test
deployment_client = self.get_deployment(benchmark_name)
File "/home/justinchigu/serverless-benchmarks/sebs/regression.py", line 145, in get_deployment
deployment_client = self.client.get_deployment(
File "/home/justinchigu/serverless-benchmarks/sebs/sebs.py", line 106, in get_deployment
raise RuntimeError("Deployment {name} not supported!".format(name=name))
RuntimeError: Deployment gcp not supported!

sebs.regression.GCPTestSequence.test_210.thumbnailer: fail
sebs.regression.GCPTestSequence.test_210.thumbnailer: Traceback (most recent call last):
File "/home/justinchigu/serverless-benchmarks/sebs/regression.py", line 38, in test
deployment_client = self.get_deployment(benchmark_name)
File "/home/justinchigu/serverless-benchmarks/sebs/regression.py", line 145, in get_deployment
deployment_client = self.client.get_deployment(
File "/home/justinchigu/serverless-benchmarks/sebs/sebs.py", line 106, in get_deployment
raise RuntimeError("Deployment {name} not supported!".format(name=name))
RuntimeError: Deployment gcp not supported!

sebs.regression.GCPTestSequence.test_220.video-processing: fail
sebs.regression.GCPTestSequence.test_220.video-processing: Traceback (most recent call last):
File "/home/justinchigu/serverless-benchmarks/sebs/regression.py", line 38, in test
deployment_client = self.get_deployment(benchmark_name)
File "/home/justinchigu/serverless-benchmarks/sebs/regression.py", line 145, in get_deployment
deployment_client = self.client.get_deployment(
File "/home/justinchigu/serverless-benchmarks/sebs/sebs.py", line 106, in get_deployment
raise RuntimeError("Deployment {name} not supported!".format(name=name))
RuntimeError: Deployment gcp not supported!

sebs.regression.GCPTestSequence.test_311.compression: fail
sebs.regression.GCPTestSequence.test_311.compression: Traceback (most recent call last):
File "/home/justinchigu/serverless-benchmarks/sebs/regression.py", line 38, in test
deployment_client = self.get_deployment(benchmark_name)
File "/home/justinchigu/serverless-benchmarks/sebs/regression.py", line 145, in get_deployment
deployment_client = self.client.get_deployment(
File "/home/justinchigu/serverless-benchmarks/sebs/sebs.py", line 106, in get_deployment
raise RuntimeError("Deployment {name} not supported!".format(name=name))
RuntimeError: Deployment gcp not supported!

sebs.regression.GCPTestSequence.test_120.uploader: fail
sebs.regression.GCPTestSequence.test_120.uploader: Traceback (most recent call last):
File "/home/justinchigu/serverless-benchmarks/sebs/regression.py", line 38, in test
deployment_client = self.get_deployment(benchmark_name)
File "/home/justinchigu/serverless-benchmarks/sebs/regression.py", line 145, in get_deployment
deployment_client = self.client.get_deployment(
File "/home/justinchigu/serverless-benchmarks/sebs/sebs.py", line 106, in get_deployment
raise RuntimeError("Deployment {name} not supported!".format(name=name))
RuntimeError: Deployment gcp not supported!

sebs.regression.GCPTestSequence.test_411.image-recognition: fail
sebs.regression.GCPTestSequence.test_411.image-recognition: Traceback (most recent call last):
File "/home/justinchigu/serverless-benchmarks/sebs/regression.py", line 38, in test
deployment_client = self.get_deployment(benchmark_name)
File "/home/justinchigu/serverless-benchmarks/sebs/regression.py", line 145, in get_deployment
deployment_client = self.client.get_deployment(
File "/home/justinchigu/serverless-benchmarks/sebs/sebs.py", line 106, in get_deployment
raise RuntimeError("Deployment {name} not supported!".format(name=name))
RuntimeError: Deployment gcp not supported!

sebs.regression.GCPTestSequence.test_501.graph-pagerank: fail
sebs.regression.GCPTestSequence.test_501.graph-pagerank: Traceback (most recent call last):
File "/home/justinchigu/serverless-benchmarks/sebs/regression.py", line 38, in test
deployment_client = self.get_deployment(benchmark_name)
File "/home/justinchigu/serverless-benchmarks/sebs/regression.py", line 145, in get_deployment
deployment_client = self.client.get_deployment(
File "/home/justinchigu/serverless-benchmarks/sebs/sebs.py", line 106, in get_deployment
raise RuntimeError("Deployment {name} not supported!".format(name=name))
RuntimeError: Deployment gcp not supported!

sebs.regression.GCPTestSequence.test_502.graph-mst: fail
sebs.regression.GCPTestSequence.test_502.graph-mst: Traceback (most recent call last):
File "/home/justinchigu/serverless-benchmarks/sebs/regression.py", line 38, in test
deployment_client = self.get_deployment(benchmark_name)
File "/home/justinchigu/serverless-benchmarks/sebs/regression.py", line 145, in get_deployment
deployment_client = self.client.get_deployment(
File "/home/justinchigu/serverless-benchmarks/sebs/sebs.py", line 106, in get_deployment
raise RuntimeError("Deployment {name} not supported!".format(name=name))
RuntimeError: Deployment gcp not supported!

sebs.regression.GCPTestSequence.test_503.graph-bfs: fail
sebs.regression.GCPTestSequence.test_503.graph-bfs: Traceback (most recent call last):
File "/home/justinchigu/serverless-benchmarks/sebs/regression.py", line 38, in test
deployment_client = self.get_deployment(benchmark_name)
File "/home/justinchigu/serverless-benchmarks/sebs/regression.py", line 145, in get_deployment
deployment_client = self.client.get_deployment(
File "/home/justinchigu/serverless-benchmarks/sebs/sebs.py", line 106, in get_deployment
raise RuntimeError("Deployment {name} not supported!".format(name=name))
RuntimeError: Deployment gcp not supported!

sebs.regression.GCPTestSequence.test_504.dna-visualisation: fail
sebs.regression.GCPTestSequence.test_504.dna-visualisation: Traceback (most recent call last):
File "/home/justinchigu/serverless-benchmarks/sebs/regression.py", line 38, in test
deployment_client = self.get_deployment(benchmark_name)
File "/home/justinchigu/serverless-benchmarks/sebs/regression.py", line 145, in get_deployment
deployment_client = self.client.get_deployment(
File "/home/justinchigu/serverless-benchmarks/sebs/sebs.py", line 106, in get_deployment
raise RuntimeError("Deployment {name} not supported!".format(name=name))
RuntimeError: Deployment gcp not supported!

Succesfully executed 0 out of 10 functions
Failures when executing 10 out of 10 functions

  • 502.graph-mst
  • 220.video-processing
  • 110.dynamic-html
  • 503.graph-bfs
  • 411.image-recognition
  • 504.dna-visualisation
  • 210.thumbnailer
  • 501.graph-pagerank
  • 311.compression
  • 120.uploader
    (python-virtualenv) justinchigu@justinchigu-VirtualBox:/serverless-benchmarks$ ./sebs.py benchmark regression test --config config/example.json --deployment local
    Succesfully executed 0 out of 0 functions
    (python-virtualenv) justinchigu@justinchigu-VirtualBox:    /serverless-benchmarks$ ./sebs.py benchmark regression test --config config/example.json --deployment aws --benchmark-name 120.uploader
    Skip test test_110.dynamic-html
    Select test test_120.uploader
    Skip test test_210.thumbnailer
    Skip test test_220.video-processing
    Skip test test_311.compression
    Skip test test_411.image-recognition
    Skip test test_501.graph-pagerank
    Skip test test_502.graph-mst
    Skip test test_503.graph-bfs
    Skip test test_504.dna-visualisation

sebs.regression.AWSTestSequence.test_120.uploader: fail
sebs.regression.AWSTestSequence.test_120.uploader: Traceback (most recent call last):
File "/home/justinchigu/serverless-benchmarks/sebs/regression.py", line 38, in test
deployment_client = self.get_deployment(benchmark_name)
File "/home/justinchigu/serverless-benchmarks/sebs/regression.py", line 103, in get_deployment
deployment_client = self.client.get_deployment(
File "/home/justinchigu/serverless-benchmarks/sebs/sebs.py", line 106, in get_deployment
raise RuntimeError("Deployment {name} not supported!".format(name=name))
RuntimeError: Deployment aws not supported!

Succesfully executed 0 out of 1 functions
Failures when executing 1 out of 1 functions

  • 120.uploader
    (python-virtualenv) justinchigu@justinchigu-VirtualBox:~/serverless-benchmarks$ ./sebs.py benchmark regression test --config config/example.json --deployment gcp --benchmark-name 120.uploader
    Skip test test_110.dynamic-html
    Select test test_120.uploader
    Skip test test_210.thumbnailer
    Skip test test_220.video-processing
    Skip test test_311.compression
    Skip test test_411.image-recognition
    Skip test test_501.graph-pagerank
    Skip test test_502.graph-mst
    Skip test test_503.graph-bfs
    Skip test test_504.dna-visualisation

sebs.regression.GCPTestSequence.test_120.uploader: fail
sebs.regression.GCPTestSequence.test_120.uploader: Traceback (most recent call last):
File "/home/justinchigu/serverless-benchmarks/sebs/regression.py", line 38, in test
deployment_client = self.get_deployment(benchmark_name)
File "/home/justinchigu/serverless-benchmarks/sebs/regression.py", line 145, in get_deployment
deployment_client = self.client.get_deployment(
File "/home/justinchigu/serverless-benchmarks/sebs/sebs.py", line 106, in get_deployment
raise RuntimeError("Deployment {name} not supported!".format(name=name))
RuntimeError: Deployment gcp not supported!

Succesfully executed 0 out of 1 functions
Failures when executing 1 out of 1 functions

  • 120.uploader
    (python-virtualenv) justinchigu@justinchigu-VirtualBox:/serverless-benchmarks$ ./sebs.py experiment invoke perf-cost --config config/example.json --deployment aws
    ERROR:root:Deployment aws not supported!
    Traceback (most recent call last):
    File "/home/justinchigu/serverless-benchmarks/./sebs.py", line 30, in call
    return self.main(*args, **kwargs)
    File "/home/justinchigu/python-virtualenv/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
    File "/home/justinchigu/python-virtualenv/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
    File "/home/justinchigu/python-virtualenv/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
    File "/home/justinchigu/python-virtualenv/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
    File "/home/justinchigu/python-virtualenv/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
    File "/home/justinchigu/serverless-benchmarks/./sebs.py", line 72, in wrapper
    return func(*args, **kwargs)
    File "/home/justinchigu/serverless-benchmarks/./sebs.py", line 97, in wrapper
    return func(*args, **kwargs)
    File "/home/justinchigu/serverless-benchmarks/./sebs.py", line 453, in experiment_invoke
    ) = parse_common_params(**kwargs)
    File "/home/justinchigu/serverless-benchmarks/./sebs.py", line 135, in parse_common_params
    deployment_client = sebs_client.get_deployment(
    File "/home/justinchigu/serverless-benchmarks/sebs/sebs.py", line 106, in get_deployment
    raise RuntimeError("Deployment {name} not supported!".format(name=name))
    RuntimeError: Deployment aws not supported!
    (python-virtualenv) justinchigu@justinchigu-VirtualBox:    /serverless-benchmarks$ ./sebs.py experiment process perf-cost --config example.json --deployment aws
    ERROR:root:[Errno 2] No such file or directory: 'example.json'
    Traceback (most recent call last):
    File "/home/justinchigu/serverless-benchmarks/./sebs.py", line 30, in call
    return self.main(*args, **kwargs)
    File "/home/justinchigu/python-virtualenv/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
    File "/home/justinchigu/python-virtualenv/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
    File "/home/justinchigu/python-virtualenv/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
    File "/home/justinchigu/python-virtualenv/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
    File "/home/justinchigu/python-virtualenv/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
    File "/home/justinchigu/serverless-benchmarks/./sebs.py", line 72, in wrapper
    return func(*args, **kwargs)
    File "/home/justinchigu/serverless-benchmarks/./sebs.py", line 97, in wrapper
    return func(*args, **kwargs)
    File "/home/justinchigu/serverless-benchmarks/./sebs.py", line 470, in experment_process
    ) = parse_common_params(**kwargs)
    File "/home/justinchigu/serverless-benchmarks/./sebs.py", line 119, in parse_common_params
    config_obj = json.load(open(config, "r"))
    FileNotFoundError: [Errno 2] No such file or directory: 'example.json'
    (python-virtualenv) justinchigu@justinchigu-VirtualBox:~/serverless-benchmarks$
mcopik commented

@jchigu Please post everything under a single issue #105