Are you fascinated by TinyML and Tensorflow for microcontrollers?

Do you want to run a CNN (Convolutional Neural Network) on your Arduino and STM32 boards?

Do you want to do it without pain?

EloquentTinyML is the library for you!

EloquentTinyML, my library to easily run Tensorflow Lite neural networks on Arduino microcontrollers, is gaining some popularity so I think it's time for a good tutorial on the topic.

If you're a seasoned follower of my blog, you may know that I don't really like Tensorflow on microcontrollers, because it is often "over-sized" for the project at hand and there are leaner, faster alternatives.

Nonetheless, Tensorflow is gaining much popularity in the embedded world so I'll try to give my contribute too.

In this tutorial, I'm going to show you step by step how to train a CNN in Tensorflow and deploy it to you board: I tested the code both on the Arduino Nano 33 BLE Sense and the STM32 Nucleus L432KC.

How to train a CNN in Tensorflow

I'm not an expert either in Tensorflow nor Convolutional Neural Networks, so I kept the project as simple as possible. I used an image-like dataset to create a setup where CNN should perform well: the dataset is the MNIST handwritten digits one.

It is composed by 8x8 images of handwritten digits, from 0 to 9 and can be easily imported via the scikit-learn Python package.

Regarding the CNN topology, I wanted to stay as lean as possible: the goal of this tutorial is to teach you how to deploy your own network, not about achieving 100% accuracy.

Let's see step by step how to produce a usable model.

Step 1. Import the libraries

We will need numpy and Tensorflow, of course, plus scikit-learn to load the dataset and tinymlgen to port the CNN to plain C.

import numpy as np
from sklearn.datasets import load_digits
import tensorflow as tf
from tensorflow.keras import layers
from tinymlgen import port

Step 2. Generate train, validation and test data

To train the network, we need:

• training data: this is the data the network uses to learn its weights
• validation data: this is the data the network uses to understand if it's doing well during learning
• test data: this is the data we use to test the network accuracy once it's done learning

def get_data():
    np.random.seed(1337)
    x_values, y_values = load_digits(return_X_y=True)
    x_values /= x_values.max()
    # reshape to (8 x 8 x 1)
    x_values = x_values.reshape((len(x_values), 8, 8, 1))

    # split into train, validation, test
    TRAIN_SPLIT = int(0.6 * len(x_values))
    TEST_SPLIT = int(0.2 * len(x_values) + TRAIN_SPLIT)
    x_train, x_test, x_validate = np.split(x_values, [TRAIN_SPLIT, TEST_SPLIT])
    y_train, y_test, y_validate = np.split(y_values, [TRAIN_SPLIT, TEST_SPLIT])

    return x_train, x_test, x_validate, y_train, y_test, y_validate

Step 3. Create and train the model

Now we have to create our network topology.

As I stated earlier, I wanted to keep this as simple as possible (also considering that we're using a toy dataset): I added a single convolution layer (without even max pooling) followed by the output layer.

def get_model():
    x_train, x_test, x_validate, y_train, y_test, y_validate = get_data()

    # create a CNN
    model = tf.keras.Sequential()
    model.add(layers.Conv2D(8, (3, 3), activation='relu', input_shape=(8, 8, 1)))
    # model.add(layers.MaxPooling2D((2, 2)))
    model.add(layers.Flatten())
    model.add(layers.Dense(len(np.unique(y_train))))

    model.compile(optimizer='adam', loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True), metrics=['accuracy'])
    model.fit(x_train, y_train, epochs=50, batch_size=16,
                        validation_data=(x_validate, y_validate))
    return model, x_test, y_test

Step 4. Testing the model accuracy

Do you think this topology is too simple to learn something useful in so few epochs?

Think again: it achieved 97% accuracy!

Not bad.

def test_model(model, x_test, y_test):
    x_test = (x_test / x_test.max()).reshape((len(x_test), 8, 8, 1))
    y_pred = model.predict(x_test).argmax(axis=1)

    print('ACCURACY', (y_pred == y_test).sum() / len(y_test))

Step 5. Exporting the model

Once we have a trained model that performs well, we want to deploy it to our microcontroller. Thanks to the tinymlgen packages, is as easy as a one-liner.

if __name__ == '__main__':
    model, x_test, y_test = get_model()
    test_model(model, x_test, y_test)
    c_code = port(model, variable_name='digits_model', pretty_print=True)
    print(c_code)

How to run a CNN on Arduino and STM32 boards with EloquentTinyML

Ok, now we have the content we need to create an Arduino sketch to run the CNN on our microcontroller.

We will use the EloquentTinyML library to do this without pain.

This is a library to run TinyML models on your microcontroller without messing around with complex compilation procedures and esoteric errors.

You must first install the library at its latest version (0.0.5 or 0.0.4 if not available), either via the Library Manager or directly from Github.

#include <EloquentTinyML.h>

// copy the printed code from tinymlgen into this file
#include "digits_model.h"

#define NUMBER_OF_INPUTS 64
#define NUMBER_OF_OUTPUTS 10
#define TENSOR_ARENA_SIZE 8*1024

Eloquent::TinyML::TfLite<NUMBER_OF_INPUTS, NUMBER_OF_OUTPUTS, TENSOR_ARENA_SIZE> ml;

void setup() {
    Serial.begin(115200);
    ml.begin(digits_model);
}

void loop() {
    // a random sample from the MNIST dataset (precisely the last one)
    float x_test[64] = { 0., 0. , 0.625 , 0.875 , 0.5   , 0.0625, 0. , 0. ,
                    0. , 0.125 , 1. , 0.875 , 0.375 , 0.0625, 0. , 0. ,
                    0. , 0. , 0.9375, 0.9375, 0.5   , 0.9375, 0. , 0. ,
                    0. , 0. , 0.3125, 1. , 1. , 0.625 , 0. , 0. ,
                    0. , 0. , 0.75  , 0.9375, 0.9375, 0.75  , 0. , 0. ,
                    0. , 0.25  , 1. , 0.375 , 0.25  , 1. , 0.375 , 0. ,
                    0. , 0.5   , 1. , 0.625 , 0.5   , 1. , 0.5   , 0. ,
                    0. , 0.0625, 0.5   , 0.75  , 0.875 , 0.75  , 0.0625, 0. };
    // the output vector for the model predictions
    float y_pred[10] = {0};
    // the actual class of the sample
    int y_test = 8;

    // let's see how long it takes to classify the sample
    uint32_t start = micros();

    ml.predict(x_test, y_pred);

    uint32_t timeit = micros() - start;

    Serial.print("It took ");
    Serial.print(timeit);
    Serial.println(" micros to run inference");

    // let's print the raw predictions for all the classes
    // these values are not directly interpretable as probabilities!
    Serial.print("Test output is: ");
    Serial.println(y_test);
    Serial.print("Predicted proba are: ");

    for (int i = 0; i < 10; i++) {
        Serial.print(y_pred[i]);
        Serial.print(i == 9 ? '\n' : ',');
    }

    // let's print the "most probable" class
    // you can either use probaToClass() if you also want to use all the probabilities
    Serial.print("Predicted class is: ");
    Serial.println(ml.probaToClass(y_pred));
    // or you can skip the predict() method and call directly predictClass()
    Serial.print("Sanity check: ");
    Serial.println(ml.predictClass(x_test));

    delay(1000);
}

That's it: if everything went fine, you should see that the predicted class is 8.

CNN on Arduino and STM32 figures

I'll report the figures I get for compiling and running this project on the two boards I used.

I used the Grumpyoldpizza compiler for the Nucleus, which doesn't report back the RAM usage

And you?

Were you able to deploy a CNN to your microcontroller thanks to this tutorial? Or are you having troubles?

Let me know in the comment and I will help you or share your experience with us.

You can find the whole code on Github.

L'articolo TinyML on Arduino and STM32: CNN (Convolutional Neural Network) example proviene da Eloquent Arduino Blog.

ri-elaborated from https://www.accuware.com/blog/ambient-signals-plus-video-images/

My original post abot Wifi indoor positioning is one of my top-performing post of all time (after motion detection using ESP32 camera and the introductory post on Machine Learning for Arduino). This is why I settled to revamp it and add some more details, tools and scripts to create the most complete free guide on how to implement such a system, from the beginning to the end.

This post will cover all the necessary steps and provide all the code you need: for an introduction to the topic, I point you to the original post.

Features definition

This part stays the same as the original post: we will use the RSSIs (signal strength) of the nearby Wifi hotspots to classifiy which location we're in.

Each location will "see" a certain number of networks, each with a RSSI that will stay mostly the same: the unique combination of these RSSIs will become a fingerprint to distinguish the locations from one another.

Since not all networks will be visible all the time, the shape of our data will be more likely a sparse matrix.
A sparse matrix is a matrix where most of the elements will be zero, meaning the absence of the given feature. Only the relevant elements will be non-zero and will represent the RSSI of the nth network.

The following example table should give you an idea of what our data will look like.

Even though the numbers in this table are fake, you should recognize a pattern:

• each location is characterized by a certain combination of always-visible networks
• some sample could be "noised" by weak networks (the 5 in the table)

Our machine learning algorithm should be able to extract each location's fingerprint without being fooled by this inconsistent features.

Data gathering

Now that we know what our data should look like, we need to first get it.

In the original post, this point was the one I'm unhappy with since it's not as straight-forward as I would have liked. The method I present you in this post, instead, is by far way simpler to follow.

First of all, you will need a Wifi equipped board. I will use an Arduino MKR WiFi 1010, but any ESP8266 / ESP32 or the like will work.

The following sketch will do the job: it scans the visible networks at a regular interval and prints their RSSIs encoded in JSON format.

// file DataGathering.h

#include "WiFi.h"

#define print(string) Serial.print(string);
#define quote(string) print('"'); print(string); print('"');

String location = "";

/**
 * 
 */
void setup() {
  Serial.begin(115200);
  delay(3000);
  WiFi.disconnect();
}

/**
 * 
 */
void loop() {  
  // if location is set, scan networks
  if (location != "") {
    int numNetworks = WiFi.scanNetworks();

    // print location
    print('{');
    quote("__location");
    print(": ");
    quote(location);
    print(", ");

    // print each network SSID and RSSI
    for (int i = 0; i < numNetworks; i++) {
      quote(WiFi.SSID(i));
      print(": ");
      print(WiFi.RSSI(i));
      print(i == numNetworks - 1 ? "}\n" : ", ");
    }

    delay(1000);
  }
  // else wait for user to enter the location
  else {
    String input;

    Serial.println("Enter 'scan {location}' to start the scanning");

    while (!Serial.available())
      delay(200);

    input = Serial.readStringUntil('\n');

    if (input.indexOf("scan ") == 0) {
      input.replace("scan ", "");
      location = input;
    }
    else {
      location = "";
    }
  }
}

Upload the sketch to your board and start mapping your house / office: go to the target location and type scan {location} in the serial monitor, where {location}is the name you want to give to the current location (so, for example, if you're mapping the kitchen, type scan kitchen).

Move around the room a bit so you capture a few variations of the visible hotspots: this will lead to a more robust classification later on.

To stop the recording just type stop in the serial monitor.

Now repeat this process for each location you want to classify. At this point you should have ended with something similar to the following:

{"__location": "Kitchen", "N1": 100, "N2": 50}
{"__location": "Bedroom", "N3": 100, "N2": 50}
{"__location": "Bathroom", "N1": 100, "N4": 50}
{"__location": "Bathroom", "N5": 100, "N4": 50}

In your case, "N1", "N2"... will contain the name of the visible networks.

When you're happy with your training data, it's time to convert it to something useful.

Generating the features converter

Given the data we have, we want to generate C code that can convert a Wifi scan result into a feature vector we can use for classification.

Since I'm a fan of code-generators, I wrote one specifically for this very project. And since I already have a code-generator library I use for Machine Learning code written in Python, I updated it with this new functionality.

Start by installing the library.

# be sure it installs version >= 1.1.8
pip install --upgrade micromlgen

Now create a script with the following code:

from micromlgen import port_wifi_indoor_positioning

if __name__ == '__main__':
    samples = '''
    {"__location": "Kitchen", "N1": 100, "N2": 50}
    {"__location": "Bedroom", "N3": 100, "N2": 50}
    {"__location": "Bathroom", "N1": 100, "N4": 50}
    {"__location": "Bathroom", "N5": 100, "N4": 50}
    '''
    X, y, classmap, converter_code = port_wifi_indoor_positioning(samples)
    print(converter_code)

Of course you have to replace the samples content with the output you got in the previous step.

In the console you should see a C++ class we will use later in the Arduino sketch. The class should be similar to the following example code.

// Save this code in your sketch as Converter.h

#pragma once
namespace Eloquent {
    namespace Projects {
        class WifiIndoorPositioning {
            public:
                /**
                * Get feature vector
                */
                float* getFeatures() {
                    static float features[5] = {0};
                    uint8_t numNetworks = WiFi.scanNetworks();

                    for (uint8_t i = 0; i < 5; i++) {
                        features[i] = 0;
                    }

                    for (uint8_t i = 0; i < numNetworks; i++) {
                        int featureIdx = ssidToFeatureIdx(WiFi.SSID(i));

                        if (featureIdx >= 0) {
                            features[featureIdx] = WiFi.RSSI(i);
                        }
                    }

                    return features;
                }

            protected:
                /**
                * Convert SSID to featureIdx
                */
                int ssidToFeatureIdx(String ssid) {
                    if (ssid.equals("N1"))
                    return 0;

                    if (ssid.equals("N2"))
                    return 1;

                    if (ssid.equals("N3"))
                    return 2;

                    if (ssid.equals("N4"))
                    return 3;

                    if (ssid.equals("N5"))
                    return 4;

                    return -1;
                }
            };
        }
    }

I will briefly explain what it does: when you call getFeatures(), it runs a Wifi scan and for each network it finds, it fills the corresponding element in the feature vector (if the network is a known one).

At the end of the procedure, your feature vector will look something like [0, 10, 0, 0, 50, 0, 0], each element representing the RSSI of a given network.

Generating the classifier

To close the loop of the project, we need to be able to classify the features vector into one of the recorded location. Since we already have micromlgen installed, it will be very easy to do so.

Let's update the Python code we already have: this time, instead of printing the converter code, we will print the classifier code.

# install ml package first
pip install scikit-learn

from sklearn.tree import DecisionTreeClassifier
from micromlgen import port_wifi_indoor_positioning, port

if __name__ == '__main__':
    samples = '''
    {"__location": "Kitchen", "N1": 100, "N2": 50}
    {"__location": "Bedroom", "N3": 100, "N2": 50}
    {"__location": "Bathroom", "N1": 100, "N4": 50}
    {"__location": "Bathroom", "N5": 100, "N4": 50}
    '''
    X, y, classmap, converter_code = port_wifi_indoor_positioning(samples)
    clf = DecisionTreeClassifier()
    clf.fit(X, y)
    print(port(clf, classmap=classmap))

Here I chose Decision tree because it is a very lightweight algorithm and should work fine for the kind of features we're working with.
If you're not satisfied with the results, you can try to use SVM or Gaussian Naive Bayes, which are both supported by micromlgen.

In the console you will see the generated code for the classifier you trained. In the case of DecisionTree the code will look like the following.

// Save this code in your sketch as Classifier.h

#pragma once
namespace Eloquent {
    namespace ML {
        namespace Port {
            class DecisionTree {
                public:
                    /**
                    * Predict class for features vector
                    */
                    int predict(float *x) {
                        if (x[2] <= 25.0) {
                            if (x[4] <= 50.0) {
                                return 1;
                            }

                            else {
                                return 2;
                            }
                        }

                        else {
                            return 0;
                        }
                    }

                    /**
                    * Convert class idx to readable name
                    */
                    const char* predictLabel(float *x) {
                        switch (predict(x)) {
                            case 0:
                            return "Bathroom";
                            case 1:
                            return "Bedroom";
                            case 2:
                            return "Kitchen";
                            default:
                            return "Houston we have a problem";
                        }
                    }

                protected:
                };
            }
        }
    }

Wrapping it all together

Now that we have all the pieces together, we only need to merge them to get a complete working example.

// file WifiIndoorPositioning.h

#include "WiFi.h"
#include "Converter.h"
#include "Classifier.h"

Eloquent::Projects::WifiIndoorPositioning positioning;
Eloquent::ML::Port::DecisionTree classifier;

void setup() {
  Serial.begin(115200);
}

void loop() {
  Serial.print("You're in ");
  Serial.println(classifier.predictLabel(positioning.getFeatures()));
  delay(3000);
}

To the bare minimum, the above code runs the scan and tells you which location you're in. That's it.

Disclaimer

This system should be pretty accurate and robust if you properly gather the data, though I can quantify how much accurate.

This is not an indoor navigation system: it can't tell you "the coordinates" of where you are, it can only detect in which room you're in.

If your location lack of nearby Wifi hotspots, an easy and cheap solution would be to spawn a bunch of ESP8266 / ESP32 boards around your house each acting as Access Point: with this simple trick you should be able to be as accurate as needed by just adding more boards.

With this in-depth tutorial I hope I helped you going from start to end of setting up a Wifi indoor positioning system using cheap hardware as ESP8266 / ESP32 boards and the Arduino IDE.

As you can see, Machine learning has not to be intimidating even for beginners: you just need the right tools to get the job done.

If this guide excited you about Machine learning on microcontrollers, I invite you to read the many other posts I wrote on the topic and share them on the socials.

You can find the whole project on Github. Don't forget to star the repo if you like it.

L'articolo The Ultimate Guide to Wifi Indoor Positioning using Arduino and Machine Learning proviene da Eloquent Arduino Blog.