[Submarine Spark Security] when run sql have expression will return ”UnresolvedAttribute.exprId“ error
Closed this issue · 4 comments
Junyewu commented
env:
spark-3.1.1
ranger-2.1
submarine-security-0.6.0
sql code:
spark-sql> create or replace temporary view user_city_temp as
select id,case when if(city_zip_active >0,city_zip_active,if(city_zip_check >0 ,city_zip_check,city_zip_phone)) in ('0100','0200','0300','0400') then if(city_zip_active >0,city_zip_active,if(city_zip_check >0 ,city_zip_check,city_zip_phone)) else 'other' end as city from user_city;
colum_name type
city_zip_phone int NULL
city_zip_check int NULL
city_zip_active int NULL
it will return a error
org.apache.spark.sql.catalyst.analysis.UnresolvedException: Invalid call to exprId on unresolved object, tree: 'city_zip_active
at org.apache.spark.sql.catalyst.analysis.UnresolvedAttribute.exprId(unresolved.scala:157)
at org.apache.spark.sql.catalyst.optimizer.SubmarineDataMaskingExtension$$anonfun$collectAllAliases$1$$anonfun$applyOrElse$2.$anonfun$applyOrElse$3(SubmarineDataMaskingExtension.scala:191)
at scala.collection.mutable.HashMap.getOrElseUpdate(HashMap.scala:86)
at org.apache.spark.sql.catalyst.optimizer.SubmarineDataMaskingExtension$$anonfun$collectAllAliases$1$$anonfun$applyOrElse$2.applyOrElse(SubmarineDataMaskingExtension.scala:191)
at org.apache.spark.sql.catalyst.optimizer.SubmarineDataMaskingExtension$$anonfun$collectAllAliases$1$$anonfun$applyOrElse$2.applyOrElse(SubmarineDataMaskingExtension.scala:189)
at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformUp$2(TreeNode.scala:352)
it is ok run in spark-3.1.1 with not spark-security
Junyewu commented
I try to make CreateViewCommand
is not doMasking,and it is run ok
org.apache.spark.sql.catalyst.optimizer.scala
override def apply(plan: LogicalPlan): LogicalPlan = plan match {
case c: Command => c match {
case c: CreateDataSourceTableAsSelectCommand => c.copy(query = doMasking(c.query))
case c: CreateHiveTableAsSelectCommand => c.copy(query = doMasking(c.query))
// case c: CreateViewCommand => c.copy(child = doMasking(c.child))
harsha-tenneti commented
Do you have any masking policies on any of the columns in the table? If yes, what are the columns and their masking strategy?
Junyewu commented
yes,table have masking
mask column:city_zip_active
masking strategy:redact
xunliu commented
The [Submarine Spark Security] functionality has been moved to the apache/incubator-kyuubi standalone project.