delta-io/delta-sharing

Get Internal server Error 500 for url: http://localhost:8080/delta-sharing/....

netapp-acheng opened this issue · 1 comments

netapp-acheng commented

I have a delta table in non-aws s3 storage, I installed delta-sharing-server-1.0.2 and delta-sharing 2.12:0.6.2, both on same Linux centos server as I want to test it works first.
conf/delta-sharing-server.yaml :
version: 1
shares:

  • name: "testshare"
    schemas:
    • name: "tables"
      tables:
      • name: "reason"
        location: "s3a://delta-lake/delta_ext/reason"

host: "localhost"
port: 8080
endpoint: "/delta-sharing"

I created a core-site.xml to define the s3 endpoint, access key and secret access key:
fs.s3a.endpoint
fs.s3a.access.key
fs.s3a.secret.key

On the delta-sharing side: I use the sg_detlalake.py and sg_deltalake.share to access the share defined above.

sg_deltalake.py:
import delta_sharing
import logging
logging.basicConfig(level=logging.DEBUG)

profile_file = "/home/hdp-user/databricks/sg_deltalake.share"

client = delta_sharing.SharingClient(profile_file)

print(client.list_all_tables())

sg_deltalake.share:
{"shareCredentialsVersion": 1,
"bearerToken": "",
"endpoint": "http://localhost:8080/delta-sharing"
}

Error I got when I ran this: spark-submit --packages io.delta:delta-sharing-spark_2.12:0.6.2 sg_deltalake.py

DEBUG:urllib3.connectionpool:http://localhost:8080 "GET /delta-sharing/shares/testshare/all-tables HTTP/1.1" 500 43
INFO:root:Sleeping 100 to retry because of error 500 Server Error: Internal Server Error for url: http://localhost:8080/delta-sharing/shares/testshare/all-tables
Response from server:
{'errorCode': 'INTERNAL_ERROR', 'message': ''}
DEBUG:urllib3.connectionpool:http://localhost:8080 "GET /delta-sharing/shares/testshare/all-tables HTTP/1.1" 500 43
INFO:root:Sleeping 200 to retry because of error 500 Server Error: Internal Server Error for url: http://localhost:8080/delta-sharing/shares/testshare/all-tables
Response from server:
{'errorCode': 'INTERNAL_ERROR', 'message': ''}
DEBUG:urllib3.connectionpool:http://localhost:8080 "GET /delta-sharing/shares/testshare/all-tables HTTP/1.1" 500 43
INFO:root:Sleeping 400 to retry because of error 500 Server Error: Internal Server Error for url: http://localhost:8080/delta-sharing/shares/testshare/all-tables
Response from server:
{'errorCode': 'INTERNAL_ERROR', 'message': ''}
^CTraceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/delta_sharing/rest_client.py", line 101, in func_with_retry
return func(self, *arg, **kwargs)
File "/usr/local/lib/python3.8/site-packages/delta_sharing/rest_client.py", line 271, in list_all_tables
with self._get_internal(f"/shares/{share.name}/all-tables", data) as lines:
File "/usr/lib64/python3.8/contextlib.py", line 113, in enter
return next(self.gen)
File "/usr/local/lib/python3.8/site-packages/delta_sharing/rest_client.py", line 441, in _request_internal
raise HTTPError(message, response=e.response) from None
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http://localhost:8080/delta-sharing/shares/testshare/all-tables
Response from server:
{'errorCode': 'INTERNAL_ERROR', 'message': ''}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/hdp-user/databricks/sg_deltalake.py", line 14, in
print(client.list_all_tables())
File "/usr/local/lib/python3.8/site-packages/delta_sharing/delta_sharing.py", line 316, in list_all_tables
return list(chain((self.__list_all_tables_in_share(share) for share in shares)))
File "/usr/local/lib/python3.8/site-packages/delta_sharing/delta_sharing.py", line 316, in
return list(chain((self.__list_all_tables_in_share(share) for share in shares)))
File "/usr/local/lib/python3.8/site-packages/delta_sharing/delta_sharing.py", line 255, in __list_all_tables_in_share
response = self._rest_client.list_all_tables(share=share, page_token=page_token)
File "/usr/local/lib/python3.8/site-packages/delta_sharing/rest_client.py", line 105, in func_with_retry
self._sleeper(sleep_ms)
File "/usr/local/lib/python3.8/site-packages/delta_sharing/rest_client.py", line 145, in
self._sleeper = lambda sleep_ms: time.sleep(sleep_ms / 1000)

Error from delta-sharing-server side:
15936 [armeria-common-worker-epoll-2-1] ERROR io.delta.sharing.server.DeltaSharingServiceExceptionHandler - java.lang.NullPointerException
io.delta.sharing.server.DeltaInternalException: java.lang.NullPointerException
at io.delta.sharing.server.DeltaSharingService.processRequest(DeltaSharingService.scala:187)
at io.delta.sharing.server.DeltaSharingService.listAllTables(DeltaSharingService.scala:234)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.linecorp.armeria.internal.server.annotation.AnnotatedService.invoke(AnnotatedService.java:338)
at com.linecorp.armeria.internal.server.annotation.AnnotatedService.lambda$serve0$6(AnnotatedService.java:313)
at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:616)
at java.util.concurrent.CompletableFuture.uniApplyStage(CompletableFuture.java:628)
at java.util.concurrent.CompletableFuture.thenApply(CompletableFuture.java:1996)
at com.linecorp.armeria.internal.server.annotation.AnnotatedService.serve0(AnnotatedService.java:317)
at com.linecorp.armeria.internal.server.annotation.AnnotatedService.serve(AnnotatedService.java:262)
at com.linecorp.armeria.server.HttpServerHandler.handleRequest(HttpServerHandler.java:386)
at com.linecorp.armeria.server.HttpServerHandler.channelRead(HttpServerHandler.java:250)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at com.linecorp.armeria.server.Http1RequestDecoder.channelRead(Http1RequestDecoder.java:216)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296)
at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.handler.flush.FlushConsolidationHandler.channelRead(FlushConsolidationHandler.java:152)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795)
at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.NullPointerException
at io.delta.sharing.server.SharedTableManager.$anonfun$listAllTables$4(SharedTableManager.scala:168)
at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:238)
at scala.collection.Iterator.foreach(Iterator.scala:941)
at scala.collection.Iterator.foreach$(Iterator.scala:941)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1429)
at scala.collection.IterableLike.foreach(IterableLike.scala:74)
at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
at scala.collection.TraversableLike.map(TraversableLike.scala:238)
at scala.collection.TraversableLike.map$(TraversableLike.scala:231)
at scala.collection.AbstractTraversable.map(Traversable.scala:108)
at io.delta.sharing.server.SharedTableManager.$anonfun$listAllTables$3(SharedTableManager.scala:162)
at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:245)
at scala.collection.Iterator.foreach(Iterator.scala:941)
at scala.collection.Iterator.foreach$(Iterator.scala:941)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1429)
at scala.collection.IterableLike.foreach(IterableLike.scala:74)
at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
at scala.collection.TraversableLike.flatMap(TraversableLike.scala:245)
at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:242)
at scala.collection.AbstractTraversable.flatMap(Traversable.scala:108)
at io.delta.sharing.server.SharedTableManager.$anonfun$listAllTables$2(SharedTableManager.scala:161)
at io.delta.sharing.server.SharedTableManager.$anonfun$listAllTables$2$adapted(SharedTableManager.scala:160)
at io.delta.sharing.server.SharedTableManager.getPage(SharedTableManager.scala:80)
at io.delta.sharing.server.SharedTableManager.listAllTables(SharedTableManager.scala:160)
at io.delta.sharing.server.DeltaSharingService.$anonfun$listAllTables$1(DeltaSharingService.scala:236)
at io.delta.sharing.server.DeltaSharingService.processRequest(DeltaSharingService.scala:180)
... 51 more

aws s3 ls s3://delta-lake/delta_ext/reason/
PRE _delta_log/
2023-10-12 16:35:48 2032 part-00000-7f7786ed-de88-49f8-bd37-ef4c2965b926-c000.snappy.parquet

netapp-acheng commented

Looks like I am using an older delta sharing server yaml template which does not has the line id after location.
id: "00000000-0000-0000-0000-000000000000"

Now, I added it, it is working as expected.

conf/delta-sharing-server.yaml :
version: 1
shares:

name: "testshare"
schemas:
name: "tables"
tables:
name: "reason"
location: "s3a://delta-lake/delta_ext/reason"
id: "00000000-0000-0000-0000-000000000000"

  • 👍2