Maybe some overhead from Docker or some TF versions are different from my host, but that’s not the point of this article. Running this container will result in an inference time of ~0.15s. Note: Pillow is an image handling library required by Keras to load an image. GPU was made not visible by using the environment variable CUDA_VISIBLE_DEVICES.

Note that my TensorFlow is not properly compiled with AVX or MKL support. 12x slower is in the order of magnitude of what to expect between CPU and GPU. This time, on my CPU, without a container it takes ~0.12s. Running that on my Nvidia 1080 will result in an inference time of ~0.01s per image. predict(), so will not time the first call, but the second one. Keras/Tensorflow seems to do some operation in GPU upon the first call to. Let’s do a simple benchmark comparing an inference on GPU, CPU on the host, CPU on docker, and CPU on docker with restriction.

TL:DR: TensorFlow reads the number of logical CPU cores to configure itself, which can be all wrong when you have a container with CPU restriction.

By Pierre Paci How a badly configured Tensorflow in Docker can be 10x slower than expected

Tensorboard docker on mac