添加 jvm 启动参数gclog 后,启动失败
chncaesar opened this issue · 0 comments
chncaesar commented
byzer.properties.override 添加以下参数:
spark.driver.extraJavaOptions="-Xloggc:/tmp/byzer-gc-p%.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=200M -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintClassHistogramBeforeFullGC -XX:+PrintClassHistogramAfterFullGC -XX:+PrintGCApplicationStoppedTime"
spark.executor.extraJavaOptions="-Xloggc:/tmp/byzer-gc-p%.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=200M -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintClassHistogramBeforeFullGC -XX:+PrintClassHistogramAfterFullGC -XX:+PrintGCApplicationStoppedTime"
Byzer-lang server模式启动失败。错误日志如下:
+ nohup /mlsql/server/spark/bin/spark-submit --class streaming.core.StreamingApp --jars /mlsql/server/byzer-lang-3.3.0-2.4.0-SNAPSHOT/libs/azure-blob_3.2-1.0-SNAPSHOT.jar,/mlsql/server/byzer-lang-3.3.0-2.4.0-SNAPSHOT/libs/bqjdbc-2.3.22.jar,/mlsql/server/byzer-lang-3.3.0-2.4.0-SNAPSHOT/libs/byzer-objectstore-obs-3.3_2.12-0.1.0-SNAPSHOT.jar,/mlsql/server/byzer-lang-3.3.0-2.4.0-SNAPSHOT/libs/byzer-objectstore-s3-3.3_2.12-0.1.0-SNAPSHOT.jar,/mlsql/server/byzer-lang-3.3.0-2.4.0-SNAPSHOT/main/byzer-lang-3.3.0-2.12-2.4.0-SNAPSHOT.jar --conf spark.driver.extraClassPath=/mlsql/server/byzer-lang-3.3.0-2.4.0-SNAPSHOT/plugin/byzer-doris-3.3_2.12-0.1.0-SNAPSHOT.jar:/mlsql/server/byzer-lang-3.3.0-2.4.0-SNAPSHOT/plugin/mlsql-assert-3.3_2.12-0.1.0-SNAPSHOT.jar:/mlsql/server/byzer-lang-3.3.0-2.4.0-SNAPSHOT/plugin/mlsql-excel-3.3_2.12-0.1.0-SNAPSHOT.jar:/mlsql/server/byzer-lang-3.3.0-2.4.0-SNAPSHOT/plugin/mlsql-ext-ets-3.3_2.12-0.1.0-SNAPSHOT.jar:/mlsql/server/byzer-lang-3.3.0-2.4.0-SNAPSHOT/plugin/mlsql-mllib-3.3_2.12-0.1.0-SNAPSHOT.jar:/mlsql/server/byzer-lang-3.3.0-2.4.0-SNAPSHOT/plugin/mlsql-shell-3.3_2.12-0.1.0-SNAPSHOT.jar:/mlsql/server/byzer-lang-3.3.0-2.4.0-SNAPSHOT/libs/azure-blob_3.2-1.0-SNAPSHOT.jar:/mlsql/server/byzer-lang-3.3.0-2.4.0-SNAPSHOT/libs/bqjdbc-2.3.22.jar:/mlsql/server/byzer-lang-3.3.0-2.4.0-SNAPSHOT/libs/byzer-objectstore-obs-3.3_2.12-0.1.0-SNAPSHOT.jar:/mlsql/server/byzer-lang-3.3.0-2.4.0-SNAPSHOT/libs/byzer-objectstore-s3-3.3_2.12-0.1.0-SNAPSHOT.jar:/mlsql/server/byzer-lang-3.3.0-2.4.0-SNAPSHOT/main/byzer-lang-3.3.0-2.12-2.4.0-SNAPSHOT.jar --conf spark.executor.extraClassPath=/mlsql/server/byzer-lang-3.3.0-2.4.0-SNAPSHOT/plugin/byzer-doris-3.3_2.12-0.1.0-SNAPSHOT.jar:/mlsql/server/byzer-lang-3.3.0-2.4.0-SNAPSHOT/plugin/mlsql-assert-3.3_2.12-0.1.0-SNAPSHOT.jar:/mlsql/server/byzer-lang-3.3.0-2.4.0-SNAPSHOT/plugin/mlsql-excel-3.3_2.12-0.1.0-SNAPSHOT.jar:/mlsql/server/byzer-lang-3.3.0-2.4.0-SNAPSHOT/plugin/mlsql-ext-ets-3.3_2.12-0.1.0-SNAPSHOT.jar:/mlsql/server/byzer-lang-3.3.0-2.4.0-SNAPSHOT/plugin/mlsql-mllib-3.3_2.12-0.1.0-SNAPSHOT.jar:/mlsql/server/byzer-lang-3.3.0-2.4.0-SNAPSHOT/plugin/mlsql-shell-3.3_2.12-0.1.0-SNAPSHOT.jar:/mlsql/server/byzer-lang-3.3.0-2.4.0-SNAPSHOT/libs/azure-blob_3.2-1.0-SNAPSHOT.jar:/mlsql/server/byzer-lang-3.3.0-2.4.0-SNAPSHOT/libs/bqjdbc-2.3.22.jar:/mlsql/server/byzer-lang-3.3.0-2.4.0-SNAPSHOT/libs/byzer-objectstore-obs-3.3_2.12-0.1.0-SNAPSHOT.jar:/mlsql/server/byzer-lang-3.3.0-2.4.0-SNAPSHOT/libs/byzer-objectstore-s3-3.3_2.12-0.1.0-SNAPSHOT.jar:/mlsql/server/byzer-lang-3.3.0-2.4.0-SNAPSHOT/main/byzer-lang-3.3.0-2.12-2.4.0-SNAPSHOT.jar --driver-java-options -Dlog4j.configurationFile=file:/mlsql/server/byzer-lang-3.3.0-2.4.0-SNAPSHOT/conf/byzer-server-log4j2.properties --conf spark.kryoserializer.buffer=256k --conf spark.driver.memory=2g --conf spark.kryoserializer.buffer.max=1024m --conf spark.sql.hive.metastore.jars=builtin --conf 'spark.yarn.jars=hdfs://ubuntu-dev:9000/work/spark-3.3.0/*' --conf spark.sql.hive.thriftServer.singleSession=true --conf 'spark.master=local[*]' --conf spark.scheduler.mode=FAIR --conf spark.executor.cores=1 --conf spark.executor.memroy=1g --conf spark.serializer=org.apache.spark.serializer.KryoSerializer --conf spark.executor.extraJavaOptions=-Xloggc:/tmp/byzer-gc-p%.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=200M -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintClassHistogramBeforeFullGC -XX:+PrintClassHistogramAfterFullGC -XX:+PrintGCApplicationStoppedTime --conf spark.executor.instances=1 --conf spark.driver.extraJavaOptions=-Xloggc:/tmp/byzer-gc-p%.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=200M -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintClassHistogramBeforeFullGC -XX:+PrintClassHistogramAfterFullGC -XX:+PrintGCApplicationStoppedTime --conf spark.sql.hive.metastore.version=2.3.9 /mlsql/server/byzer-lang-3.3.0-2.4.0-SNAPSHOT/main/byzer-lang-3.3.0-2.12-2.4.0-SNAPSHOT.jar -streaming.spark.service true -streaming.driver.port 9004 -streaming.platform spark -streaming.name byzer-lang-1 -streaming.thrift false -streaming.rest true -streaming.datalake.path /work/juicefs/byzer-lang-1/delta/ -streaming.enableHiveSupport false -spark.mlsql.path.schemas '"s3a,oss,obs,abfs"'
Error: Unrecognized option: -XX:+UseGCLogFileRotation
Usage: spark-submit [options] [app arguments]
Usage: spark-submit --kill [submission ID] --master [spark://...]
Usage: spark-submit --status [submission ID] --master [spark://...]
Usage: spark-submit run-example [options] example-class [example args]