Deploying Deep Learning Models
Strategies to deploy deep learning models
The slides of my OSCON Tensorflow Day 2018 are available here. I will add a link to the conference video once it is published.
Sample Project
Model Structure
Let's predict Amazon product ratings based on the comments with a small LSTM network.
model_input = Input(shape=(MAX_TOKENS,))
x = Embedding(input_dim=len(CHARS), output_dim=10, input_length=MAX_TOKENS)(model_input)
x = LSTM(128)(text_input)
output = Dense(5, activation='softmax')(x)
model = Model(inputs=text_input, outputs=output)
optimizer = RMSprop(lr=0.01)
model.compile(loss='categorical_crossentropy', optimizer=optimizer)
Testing our Model
Negative Review
>> test_sentence = "horrible book, don't buy it"
>> test_vector = clean_data(test_sentence, max_tokens=MAX_TOKENS, sup_chars=CHARS)
>> model.predict(test_vector.reshape(1, MAX_TOKENS, len(CHARS)))
[[0.5927979 0.23748466 0.10798287 0.03301411 0.02872046]]
Positive Review
>> test_sentence = "Awesome product."
>> test_vector = clean_data(test_sentence, max_tokens=MAX_TOKENS, sup_chars=CHARS)
>> model.predict(test_vector.reshape(1, MAX_TOKENS, len(CHARS)))
[[0.03493131 0.0394276 0.08326671 0.2957105 0.5466638 ]]
Steps to Deploy the Sample Project
Export the model as protobuf
import os
from keras import backend as K
import tensorflow as tf
tf.app.flags.DEFINE_integer('training_iteration', 1000, 'number of training iterations.')
tf.app.flags.DEFINE_integer('model_version', 1, 'version number of the model.')
tf.app.flags.DEFINE_string('work_dir', '/tmp', 'Working directory.')
FLAGS = tf.app.flags.FLAGS
export_path_base = '/tmp/amazon_reviews'
export_path = os.path.join(tf.compat.as_bytes(export_path_base),
tf.compat.as_bytes(str(FLAGS.model_version)))
builder = tf.saved_model.builder.SavedModelBuilder(export_path)
signature = tf.saved_model.signature_def_utils.predict_signature_def(
inputs={'input': model.input}, outputs={'rating_prob': model.output})
builder.add_meta_graph_and_variables(
sess=K.get_session(), tags=[tf.saved_model.tag_constants.SERVING],
signature_def_map={
tf.saved_model.signature_constants.DEFAULT_SERVING_SIGNATURE_DEF_KEY: signature })
builder.save()
Set up the Tensorflow Serving
$ git clone git@github.com:hanneshapke/Deploying_Deep_Learning_Models.git
$ docker build --pull -t $USER/tensorflow-serving-devel-cpu \
-f {path to repo}/\
Deploying_Deep_Learning_Models/\
examples/Dockerfile .
$ docker run -it -p 8500:8500 -p 8501:8501
-v {model_path}/exported_models/amazon_review/:/models
$USER/tensorflow-serving-devel-cpu:latest /bin/bash
$[docker bash] tensorflow_model_server --port=8500
--model_name={model_name}
--model_base_path=/models/{model_name}
Setup a client (either gRPC or REST based)
from grpc.beta import implementations
from tensorflow_serving.apis import prediction_service_pb2
def get_stub(host='127.0.0.1', port='8500'):
channel = implementations.insecure_channel(host, int(port))
stub = prediction_service_pb2.beta_create_PredictionService_stub(channel)
return stub
def get_model_prediction(model_input, stub,
model_name='amazon_reviews',
signature_name='serving_default'):
request = predict_pb2.PredictRequest()
request.model_spec.name = model_name
request.model_spec.signature_name = signature_name
request.inputs['input'].CopyFrom(
tf.contrib.util.make_tensor_proto(
model_input.reshape(1, 50),
verify_shape=True, shape=(1, 50)))
response = stub.Predict.future(request, 5.0) # wait max 5s
return response.result().outputs["rating_prob"].float_val
>>> sentence = "this product is really helpful"
>>> model_input = clean_data_encoded(sentence)
>>> get_model_prediction(model_input, stub)
[0.0250927172601223, 0.03738045319914818, 0.09454590082168579,
0.33069494366645813, 0.5122858881950378]
Deployment Options
On Premise
- Kubeflow
- Tensorflow Serving
- MLflow
- Seldon.io
Cloud base
- Google Cloud Platform
- Microsoft Azure ML
- Amazon SageMaker
More details here