Spin up server for aapl stock

[slinlee]

Let's spin up one instance for stock prediction manually.

The files are in this zip file:
stockexample.zip

1. saved_model.zip
2. schema.yaml

[xneyder]

I got this error:

Traceback (most recent call last):
File "generate.py", line 4, in
import config
File "/usr/src/app/config.py", line 40, in
features = observations.keys()
AttributeError: 'NoneType' object has no attribute 'keys'

I think is because in the schema.yaml file we are missing the following, can i add it ?

observations:
minute:
type: int
hour:
type: int
day:
type: int
month:
type: int
price:
type: float

[slinlee]

@maxpumperla this is a gym env trained on the webapp. We we shouldn't need more than this in the schema.yaml, right?

parameters:
  discrete: True
  tuple: False
  api_key: "1234567asdfgh"

[slinlee]

@xneyder does your deploy grab the latest code from the github repo?

[xneyder]

Yes it uses the latest from github

[xneyder]

It is deployed at stock.devpathmind.com

[slinlee]

@xneyder Please let me know when the latest version is deployed. Thanks.

[slinlee]

Okay, so the latest status is @xneyder redeployed this with the code from master and it shows the right docs at https://stocks.devpathmind.com/docs

When I try to test the endpoint using a set of observations from @johnnyL7 the response is internal server error

Am I doing something wrong here, @maxpumperla ? Thanks.

curl -X POST "https://stock.devpathmind.com/predict_raw/?access_token=1234567asdfgh" -H  "accept: application/json" -H  "Content-Type: application/json" -d "{\"obs\":[0.885329,-0.00450165,0.09615385,0.8,0.08333333,0.68333333]}"

Function call stack:
pruned
INFO:     10.20.12.127:53256 - "POST /predict_raw/?access_token=1234567asdfgh HTTP/1.1" 500 Internal Server Error
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/uvicorn/protocols/http/httptools_impl.py", line 396, in run_asgi
    result = await app(self.scope, self.receive, self.send)
  File "/usr/local/lib/python3.7/site-packages/uvicorn/middleware/proxy_headers.py", line 45, in __call__
    return await self.app(scope, receive, send)
  File "/usr/local/lib/python3.7/site-packages/fastapi/applications.py", line 190, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.7/site-packages/starlette/applications.py", line 111, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.7/site-packages/starlette/middleware/errors.py", line 181, in __call__
    raise exc from None
  File "/usr/local/lib/python3.7/site-packages/starlette/middleware/errors.py", line 159, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.7/site-packages/starlette/exceptions.py", line 82, in __call__
    raise exc from None
  File "/usr/local/lib/python3.7/site-packages/starlette/exceptions.py", line 71, in __call__
    await self.app(scope, receive, sender)
  File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 566, in __call__
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 227, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 41, in app
    response = await func(request)
  File "/usr/local/lib/python3.7/site-packages/fastapi/routing.py", line 189, in app
    dependant=dependant, values=values, is_coroutine=is_coroutine
  File "/usr/local/lib/python3.7/site-packages/fastapi/routing.py", line 135, in run_endpoint_function
    return await dependant.call(**values)
  File "./app.py", line 162, in predict_raw
    return await SERVE_HANDLE.remote(payload.obs)
ray.exceptions.RayTaskError(InvalidArgumentError): ray::RayServeReplica_PathmindPolicy.handle_request() (pid=166, ip=10.20.12.131)
  File "/usr/local/lib/python3.7/site-packages/ray/serve/backend_worker.py", line 136, in wrap_to_ray_error
    raise exception
  File "/usr/local/lib/python3.7/site-packages/ray/serve/backend_worker.py", line 256, in invoke_single
    result = await method_to_call(arg)
  File "/usr/local/lib/python3.7/site-packages/ray/async_compat.py", line 29, in wrapper
    return func(*args, **kwargs)
  File "/usr/src/app/api.py", line 50, in __call__
    prev_reward=self.prev_reward_tensor, seq_lens=self.seq_lens_tensor
  File "/usr/local/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 1655, in __call__
    return self._call_impl(args, kwargs)
  File "/usr/local/lib/python3.7/site-packages/tensorflow/python/eager/wrap_function.py", line 247, in _call_impl
    args, kwargs, cancellation_manager)
  File "/usr/local/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 1673, in _call_impl
    return self._call_with_flat_signature(args, kwargs, cancellation_manager)
  File "/usr/local/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 1722, in _call_with_flat_signature
    return self._call_flat(args, self.captured_inputs, cancellation_manager)
  File "/usr/local/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 1924, in _call_flat
    ctx, args, cancellation_manager=cancellation_manager))
  File "/usr/local/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 550, in call
    ctx=ctx)
  File "/usr/local/lib/python3.7/site-packages/tensorflow/python/eager/execute.py", line 60, in quick_execute
    inputs, attrs, num_outputs)
tensorflow.python.framework.errors_impl.InvalidArgumentError:  Matrix size-incompatible: In[0]: [1,6], In[1]: [3,256]
	 [[node default_policy/functional_1/fc_1/BiasAdd (defined at /src/app/api.py:40) ]] [Op:__inference_pruned_8677]
Function call stack:
pruned

[maxpumperla]

@xneyder the line with features is outdated (File "/usr/src/app/config.py", line 40) try to pull latest master first and redeploy.

@slinlee from Daniel's description I see that we have (minute, hour, day, month, price) as features, yet you seems to be sending 6 values with obs. but the error message indicates that the input length should be 3. I checked locally and it works, e.g. like this:

curl -X 'POST' \
  'http://127.0.0.1:8000/predict_raw/' \
  -H 'accept: application/json' \
  -H 'access_token: 1234567asdfgh' \
  -H 'Content-Type: application/json' \
  -d '{
  "obs": [
    0,1,2
  ]
}'

Also, I can't seem to authenticate in the UI. Are we sure 1234567asdfgh is the correct key used for deployment?

[slinlee]

@maxpumperla ah, that's interesting. I'll have to check with @johnnyL7 to confirm the inputs, this is from one of the trainings he did on the webapp of a python gym env.

@xneyder the deployment only seems to use the API key if it's passed as a URL param. Can we switch it to use a header token?

[xneyder]

Hi @slinlee regarding the API i think it should be done in the policy server code itself i dont really do anything about it in the deployment, @maxpumperla is it something you can do ?

[slinlee]

@xneyder okay, in the meantime, can you redeploy with this new schema.yaml (the discrete is changed). Thanks.

parameters:
  discrete: False
  tuple: False
  api_key: "1234567asdfgh"

[slinlee]

The server is up. I'll close this now and create a new issue for the access token.