In this short post we'll take a look at how lo load Tensorflow Lite models exported as a C header file from the filesystem, be it an SD card or the built-in SPIFFS filesystem on ESP32 devices.

Load Tensorflow model from SD card

Many times you export a model from Tensorflow as a C header file and #include it in your project to run the interpreter on it.

Sometimes, however, you may want to save your model on a filesystem (external SD card or built-in SPIFFS filesystem for ESP devices) in the form of a file: maybe you want to create a "distributable" medium, maybe you simply have many models that you want to be able to run on demand.

Is it then possible to load a model from a file, instead of directly saving it as a C array?

Yes, it is possible!

And it's quite easy, too. I'll show you how.

Step 1. Save your model to a file

We'll replicate the sine model example, this time saving the model as a raw binary file.

import math
import numpy as np
import tensorflow as tf
from tensorflow.keras import layers

def get_model():
    SAMPLES = 1000
    np.random.seed(1337)
    x_values = np.random.uniform(low=0, high=2*math.pi, size=SAMPLES)
    # shuffle and add noise
    np.random.shuffle(x_values)
    y_values = np.sin(x_values)
    y_values += 0.1 * np.random.randn(*y_values.shape)

    # split into train, validation, test
    TRAIN_SPLIT =  int(0.6 * SAMPLES)
    TEST_SPLIT = int(0.2 * SAMPLES + TRAIN_SPLIT)
    x_train, x_test, x_validate = np.split(x_values, [TRAIN_SPLIT, TEST_SPLIT])
    y_train, y_test, y_validate = np.split(y_values, [TRAIN_SPLIT, TEST_SPLIT])

    # create a NN with 2 layers of 16 neurons
    model = tf.keras.Sequential()
    model.add(layers.Dense(8, activation='relu', input_shape=(1,)))
    model.add(layers.Dense(16, activation='relu'))
    model.add(layers.Dense(1))
    model.compile(optimizer='rmsprop', loss='mse', metrics=['mae'])
    model.fit(x_train, y_train, epochs=100, batch_size=16,
              validation_data=(x_validate, y_validate))
    return model

if __name__ == '__main__':
    model = get_model()

    # save to binary file
    with open('sine.bin', 'wb') as file:
        converter = tf.lite.TFLiteConverter.from_keras_model(model)
        tflite_model = converter.convert()
        file.write(tflite_model)

Copy the sine.bin file to your SD card or upload to the SPIFFS filesystem.

Step 2. Load model from file

This part is very easy to implement. We will use the EloquentTinyML library, but you're free to copy-paste from the official Tensorflow Lite for Microcontrollers documentation all the boilerplate code you need to run Tf.

The way to make it work is simple: you read the file into a byte array and feed it to the Tf interpreter. For the interpreter it makes no difference if you declared your model as an array from the beginning or loaded it from somewhere else: it will work indentically in both cases.

#include <SPI.h>
#include <SD.h>
#include <EloquentTinyML.h>
#include <eloquent_tinyml/tensorflow.h>

#define N_INPUTS 1
#define N_OUTPUTS 1
#define TENSOR_ARENA_SIZE 2*1024

uint8_t *loadedModel;
Eloquent::TinyML::TfLite<N_INPUTS, N_OUTPUTS, TENSOR_ARENA_SIZE> tf;

void loadModel(void);

/**
 *
 */
void setup() {
    Serial.begin(115200);
    SPI.begin();
    delay(3000);

    if (!SD.begin(4)) {
        Serial.println("Cannot init SD");
        delay(60000);
    }

    loadModel();

    // init Tf from loaded model
    if (!tf.begin(loadedModel)) {
        Serial.println("Cannot inialize model");
        Serial.println(ml.errorMessage());
        delay(60000);
    }
}

/**
 *
 */
void loop() {
    // pick up a random x and predict its sine
    float x = 3.14 * random(100) / 100;
    float y = sin(x);
    float input[1] = { x };
    float predicted = tf.predict(input);

    Serial.print("sin(");
    Serial.print(x);
    Serial.print(") = ");
    Serial.print(y);
    Serial.print("\t predicted: ");
    Serial.println(predicted);
    delay(1000);
}

/**
 * Load model from SD
 */
void loadModel() {
    File file = SD.open("/sine.bin", FILE_READ);
    size_t modelSize = file.size();

    Serial.print("Found model on filesystem of size ");
    Serial.println(modelSize);

    // allocate memory
    loadedModel = (uint8_t*) malloc(modelSize);

    // copy data from file
    for (size_t i = 0; i < modelSize; i++)
        loadedModel[i] = file.read();

    file.close();
}

The only significant thing this sketch does is to allocate the memory for the model (malloc(modelSize)) and copy the model content byte-by-byte, from the file to the memory.

At this point, your model is fully functional and can be given as input to the EloquentTinyML class, which does all the heavy lifting for you (please reafer to the previous TinyML posts for tutorials on how to use the EloquentTinyML library).

For the complete files refer to the Github repo

Help the blow grow

Related Posts