Deploy Tensorflow model 4x faster

Hello, For a couple of weeks I was trying to deploy the TensorFlow model (hand digit recognition) with edge devices. Within this post, we are going to discuss how we can deploy the TensorFlow model on edge devices like jetsonNano or any edge device.

Workflow for the deployment of TensorFlow model

I would advise you that always consider model optimization during your application development process. This document outlines some best practices for optimizing Tensorflow models for deployment on edge hardware

Why we need optimization?

Optimization brings your model Smaller in Size, And at the same time, it should have low computation to run inference, which will lead me to low Latency.

We have different types of optimization techniques in TensorFlow.

Types of optimization

TensorFlow currently supports optimization via

quantization

pruning

clustering.

Quantization

Quantization works by reducing the precision of numbers used to represent the model’s parameter, which is by default 32bit floating points numbers.

Dynamic range quantization

This is the simplest form of post-training quantization statically quantizes only the weights from floating-point to integer, which has 8-bits of precision.

here, I am going to show you a short code snippet that can give you an idea of how I have used TensorFlow optimization.

Create a quantized model for the TFLite backend

Now we can use these trained tflite models in my desire place on android or edge.

Tradeoffs

Of course, optimization comes with some accuracy changes depend on the individual model being optimized and is difficult to predict ahead of time. Generally, models that are optimized for size or latency will lose a small amount of accuracy.

In rare cases, certain models may gain some accuracy as a result of the optimization process.

Depending on your application, this may or may not impact your users’ experience.

Machine Learning enthusiast |Python |C++|JavaScript|Tensorflow|keras

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store