Can not work with the RikaiSparkSessionExtensions
da-liii opened this issue · 2 comments
da-liii commented
spark.sql.extensions ai.eto.rikai.sql.spark.RikaiSparkSessionExtensions,ai.eto.rikai.sql.spark.SparkVideoExtensions
create or replace temporary view frames
using video
options (
path "s3://the_bucket/big_buck_bunny_short.mp4",
frameStepSize 2
);
select frame_id, ml_image(image_data) as frame from frames where frame_id < 20 limit 3
com.databricks.backend.common.rpc.DatabricksExceptions$SQLExecutionException: org.apache.spark.sql.catalyst.parser.ParseException:
Unsupported SQL statement
== SQL ==
create or replace temporary view frames
using video
options (
path "s3://tubi-datalake-production/da/data/big_buck_bunny_short.mp4",
frameStepSize 2
)
at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.$anonfun$parsePlan$1(ParseDriver.scala:90)
at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parse(ParseDriver.scala:117)
at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parsePlan(ParseDriver.scala:85)
at ai.eto.rikai.sql.spark.parser.RikaiExtSqlParser.$anonfun$parsePlan$1(RikaiExtSparkSQLParser.scala:71)
at ai.eto.rikai.sql.spark.parser.RikaiExtSqlParser.parse(RikaiExtSparkSQLParser.scala:101)
at ai.eto.rikai.sql.spark.parser.RikaiExtSqlParser.parsePlan(RikaiExtSparkSQLParser.scala:66)
da-liii commented
Changed the order to
spark.sql.extensions ai.eto.rikai.sql.spark.SparkVideoExtensions,ai.eto.rikai.sql.spark.RikaiSparkSessionExtensions
It seems to be a Rikai SQL Parser issue.
da-liii commented
spark.sql.extensions ai.eto.rikai.sql.spark.SparkVideoExtensions
Now it works, confirmed, it is a Rikai issue.