eto-ai/spark-video

Can not work with the RikaiSparkSessionExtensions

da-liii opened this issue · 2 comments

spark.sql.extensions ai.eto.rikai.sql.spark.RikaiSparkSessionExtensions,ai.eto.rikai.sql.spark.SparkVideoExtensions
create or replace temporary view frames
using video
options (
path "s3://the_bucket/big_buck_bunny_short.mp4",
frameStepSize 2
);

select frame_id, ml_image(image_data) as frame from frames where frame_id < 20 limit 3
com.databricks.backend.common.rpc.DatabricksExceptions$SQLExecutionException: org.apache.spark.sql.catalyst.parser.ParseException: 
Unsupported SQL statement
== SQL ==
create or replace temporary view frames
using video
options (
path "s3://tubi-datalake-production/da/data/big_buck_bunny_short.mp4",
frameStepSize 2
)
	at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.$anonfun$parsePlan$1(ParseDriver.scala:90)
	at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parse(ParseDriver.scala:117)
	at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parsePlan(ParseDriver.scala:85)
	at ai.eto.rikai.sql.spark.parser.RikaiExtSqlParser.$anonfun$parsePlan$1(RikaiExtSparkSQLParser.scala:71)
	at ai.eto.rikai.sql.spark.parser.RikaiExtSqlParser.parse(RikaiExtSparkSQLParser.scala:101)
	at ai.eto.rikai.sql.spark.parser.RikaiExtSqlParser.parsePlan(RikaiExtSparkSQLParser.scala:66)

Changed the order to

spark.sql.extensions ai.eto.rikai.sql.spark.SparkVideoExtensions,ai.eto.rikai.sql.spark.RikaiSparkSessionExtensions

It seems to be a Rikai SQL Parser issue.

spark.sql.extensions ai.eto.rikai.sql.spark.SparkVideoExtensions

Now it works, confirmed, it is a Rikai issue.