Optimizing TensorFlow model serving with Kubernetes and Amazon Elastic Inference Amazon Web Services

The only aspect of the code that isn't straightforward is the need to enable EC2 instance termination protection while workers are processing videos, as shown in the following code example: After the job processes, a similar API call disables termination protection. This example application uses termination protection because the jobs are long-running, and you don't want an EC2 instance terminated during a scale-in event if it is still processing a video. You can easily modify the inference code and optimize it for your use case, so this post doesn't spend further time examining it. To review the Dockerfile for the inference code, see the amazon-elastic-inference-eks GitHub repo, under the /Dockerfile directory. The code itself is in the test.py