Decision Tree

Decision Tree is without doubt one of the most well-known classification algorithms out there. It is so simple to understand that it was probably the first classifier you encountered in any Machine Learning course.

I won't go into the details of how a Decision Tree classifier trains and selects the splits for the input features: here I will explain how a RAM-efficient porting of such a classifier is implemented.

To an introduction visit Wikipedia; for a more in-depth guide visit KDNuggets.

Since we're willing to sacrifice program space (a.k.a flash) in favor of memory (a.k.a RAM), because RAM is the most scarce resource in the vast majority of microcontrollers, the smart way to port a Decision Tree classifier from Python to C is "hard-coding" the splits in code, without keeping any reference to them into variables.

Here's what it looks like for a Decision tree that classifies the Iris dataset.

As you can see, we're using 0 bytes of RAM to get the classification result, since no variable is being allocated. On the other side, the program space will grow almost linearly with the number of splits.

Since program space is often much greater than RAM on microcontrollers, this implementation exploits its abundance to be able to deploy larger models. How much large? It will depend on the flash size available: many new generations board (Arduino Nano 33 BLE Sense, ESP32, ST Nucleus...) have 1 Mb of flash, which will hold tens of thousands of splits.

Random Forest

Random Forest is just many Decision Trees joined together in a voting scheme. The core idea is that of "the wisdom of the corwd", such that if many trees vote for a given class (having being trained on different subsets of the training set), that class is probably the true class.

Towards Data Science has a more detailed guide on Random Forest and how it balances the trees with thebagging tecnique.

As easy as Decision Trees, Random Forest gets the exact same implementation with 0 bytes of RAM required (it actually needs as many bytes as the number of classes to store the votes, but that's really negligible): it just hard-codes all its composing trees.

XGBoost (Extreme Gradient Boosting)

Extreme Gradient Boosting is "Gradient Boosting on steroids" and has gained much attention from the Machine learning community due to its top results in many data competitions.

1. "gradient boosting" refers to the process of chaining a number of trees so that each tree tries to learn from the errors of the previous
2. "extreme" refers to many software and hardware optimizations that greatly reduce the time it takes to train the model

You can read the original paper about XGBoost here. For a discursive description head to KDNuggets, if you want some more math refer to this blog post on Medium.

Porting to plain C

If you followed my earlier posts on Gaussian Naive Bayes, SEFR, Relevant Vector Machine and Support Vector Machines, you already know how to port these new classifiers.

If you're new, you will need a couple things:

1. install the micromlgen package with

pip install micromlgen

2. (optionally, if you want to use Extreme Gradient Boosting) install the xgboost package with

pip install xgboost

3. use the micromlgen.port function to generate your plain C code

from micromlgen import port
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import load_iris

clf = DecisionTreeClassifier()
X, y = load_iris(return_X_y=True)
clf.fit(X, y)
print(port(clf))

You can then copy-past the C code and import it in your sketch.

Using in the Arduino sketch

Once you have the classifier code, create a new project named TreeClassifierExample and copy the classifier code into a file named DecisionTree.h (or RandomForest.h or XGBoost.h depending on the model you chose).

The copy the following to the main ino file.

#include "DecisionTree.h"

Eloquent::ML::Port::DecisionTree clf;

void setup() {
    Serial.begin(115200);
    Serial.println("Begin");
}

void loop() {
    float irisSample[4] = {6.2, 2.8, 4.8, 1.8};

    Serial.print("Predicted label (you should see '2': ");
    Serial.println(clf.predict(irisSample));
    delay(1000);
}

Bechmarks

How do the 3 classifiers compare against each other?

We will evaluate a few keypoints:

• training time
• accuracy
• needed RAM
• needed Flash

for each classifier on a variety of datasets. I will report the results for RAM and Flash on the Arduino Nano old generation, so you should consider more the relative figures than the absolute ones.

* all datasets are taken from the UCI Machine Learning datasets archive

I'm collecting more data for a complete benchmark, but in the meantime you can see that both Random Forest and XGBoost are on par: if not that XGBoost takes 5 to 25 times longer to train.

I've never used XGBoost, so I may be missing some tuning parameters, but for now Random Forest remains my favourite classifier.

Code listings

// example IRIS dataset classification with Decision Tree
int predict(float *x) {
  if (x[3] <= 0.800000011920929) {
      return 0;
  }
  else {
      if (x[3] <= 1.75) {
          if (x[2] <= 4.950000047683716) {
              if (x[0] <= 5.049999952316284) {
                  return 1;
              }
              else {
                  return 1;
              }
          }
          else {
              return 2;
          }
      }
      else {
          if (x[2] <= 4.950000047683716) {
              return 2;
          }
          else {
              return 2;
          }
      }
  }
}

// example IRIS dataset classification with Random Forest of 3 trees

int predict(float *x) {
  uint16_t votes[3] = { 0 };

  // tree #1
  if (x[0] <= 5.450000047683716) {
      if (x[1] <= 2.950000047683716) {
          votes[1] += 1;
      }
      else {
          votes[0] += 1;
      }
  }
  else {
      if (x[0] <= 6.049999952316284) {
          if (x[3] <= 1.699999988079071) {
              if (x[2] <= 3.549999952316284) {
                  votes[0] += 1;
              }
              else {
                  votes[1] += 1;
              }
          }
          else {
              votes[2] += 1;
          }
      }
      else {
          if (x[3] <= 1.699999988079071) {
              if (x[3] <= 1.449999988079071) {
                  if (x[0] <= 6.1499998569488525) {
                      votes[1] += 1;
                  }
                  else {
                      votes[1] += 1;
                  }
              }
              else {
                  votes[1] += 1;
              }
          }
          else {
              votes[2] += 1;
          }
      }
  }

  // tree #2
  if (x[0] <= 5.549999952316284) {
      if (x[2] <= 2.449999988079071) {
          votes[0] += 1;
      }
      else {
          if (x[2] <= 3.950000047683716) {
              votes[1] += 1;
          }
          else {
              votes[1] += 1;
          }
      }
  }
  else {
      if (x[3] <= 1.699999988079071) {
          if (x[1] <= 2.649999976158142) {
              if (x[3] <= 1.25) {
                  votes[1] += 1;
              }
              else {
                  votes[1] += 1;
              }
          }
          else {
              if (x[2] <= 4.1499998569488525) {
                  votes[1] += 1;
              }
              else {
                  if (x[0] <= 6.75) {
                      votes[1] += 1;
                  }
                  else {
                      votes[1] += 1;
                  }
              }
          }
      }
      else {
          if (x[0] <= 6.0) {
              votes[2] += 1;
          }
          else {
              votes[2] += 1;
          }
      }
  }

  // tree #3
  if (x[3] <= 1.75) {
      if (x[2] <= 2.449999988079071) {
          votes[0] += 1;
      }
      else {
          if (x[2] <= 4.8500001430511475) {
              if (x[0] <= 5.299999952316284) {
                  votes[1] += 1;
              }
              else {
                  votes[1] += 1;
              }
          }
          else {
              votes[1] += 1;
          }
      }
  }
  else {
      if (x[0] <= 5.950000047683716) {
          votes[2] += 1;
      }
      else {
          votes[2] += 1;
      }
  }

  // return argmax of votes
  uint8_t classIdx = 0;
  float maxVotes = votes[0];

  for (uint8_t i = 1; i < 3; i++) {
      if (votes[i] > maxVotes) {
          classIdx = i;
          maxVotes = votes[i];
      }
  }

  return classIdx;
}

L'articolo Decision Tree, Random Forest and XGBoost on Arduino proviene da Eloquent Arduino Blog.

In this post I propose a partial, naive linear-time implementation of the Fourier Transform you can use to extract features from your data for Machine Learning models.

DISCLAIMER

The contents of this post represent my own knowledge and are not supported by any academic work (as far as I know). It may really be the case that the findings of my work don't apply to your own projects; yet, I think this idea can turn useful in solving certain kind of problems.

Training-aware FFT

Fourier transform is used to describe a signal over its entire frequency range. This is useful in a number of applications, but here we're focused on the FT for the sole purpose of extracting features to be used with Machine learning models.

For this reason, we don't actually need a full description of the input signal: we're only interested in extracting some kind of signature that a ML model can use to distinguish among the different classes. Noticing that in a signal spectrum most frequencies have a low magnitude (as you can see in the picture above), the idea to only keep the most important frequencies came to my mind as a mean to speed up the computation on resource constrained microcontrollers.

I was thinking to a kind of PCA (Principal Component Analysis), but using FFT spectrum as features.

Since we will have a training set with the raw signals, we would like to select the most prominent frequencies among all the samples and apply the computation only on those: even using the naive implementation of FFT, this will yield a linear-time implementation.

Accuracy comparison

How does this Principal FFT compare to, let's say, PCA as a dimensionality reduction algorithm w.r.t model accuracy? Let's see the numbers!

Download the Principal FFT benchmark spreadsheet

I couldn't find many examples of the kind of datasets I wished to test, but in the image you can see different types of data:

• human activity classification from smartphone data
• gesture classification by IMU data
• MNIST handwritten digits image data
• free speech audio data

We can note a couple findings:

1. Principal FFT is almost on par with PCA after a certain number of components
2. PrincipalFFT definitely leaves PCA behind on audio data

From even this simple analysis you should be convinced that Principal FFT can be (under certain cases) a fast, performant features extractor for your projects that involve time-dependant data.

How to use Principal FFT in Python

I created a Python package to use Principal FFT, called principal-fft.

pip install principal-fft

The class follows the Transformer API from scikit-learn, so it has fit and transform methods.

from principalfft import PrincipalFFT
from sklearn.model_selection import train_test_split
from sklearn.datasets import load_digits
from sklearn.ensemble import RandomForestClassifier

mnist = load_digits()
X, y = mnist.data, mnist.target
Xfft = PrincipalFFT(n_components=10).fit_transform(X)

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3)
Xfft_train, Xfft_test, y_train, y_test = train_test_split(Xfft, y, test_size=0.3)

clf = RandomForestClassifier(50, min_samples_leaf=5).fit(X_train, y_train)
print("Raw score", clf.score(X_test, y_test))

clf = RandomForestClassifier(50, min_samples_leaf=5).fit(Xfft_train, y_train)
print("FFT score", clf.score(Xfft_test, y_test))

My results are 0.09 for raw data and 0.78 for FFT transformed: quite a big difference!

As with any dimensionality reduction, n_components is an hyperparameter you have to tune for your specific project: from my experiments, you shouldn't go lower than 8 to achieve a reasonable accuracy.

How to use Principal FFT in C

So, now that we tested our Principal FFT transformer in Python and achieved good results, how do we use it on our microcontroller? Of course with the micromlgen porter: it is now (version 1.1.9) able to port PrincipalFFT objects to plain C.

pip install micromlgen==1.1.9

What does the C code look like?

void principalFFT(float *features, float *fft) {
    // apply principal FFT (naive implementation for the top 10 frequencies only)
    const int topFrequencies[] = { 0, 8, 17, 16, 1, 9, 2, 7, 15, 6 };

    for (int i = 0; i < 10; i++) {
        const int k = topFrequencies[i];
        const float harmonic = 0.09817477042468103 * k;
        float re = 0;
        float im = 0;

        // optimized case
        if (k == 0) {
            for (int n = 0; n < 64; n++) {
                re += features[n];
            }
        }

        else {
            for (int n = 0; n < 64; n++) {
                const float harmonic_n = harmonic * n;
                const float cos_n = cos(harmonic_n);
                const float sin_n = sin(harmonic_n);
                re += features[n] * cos_n;
                im -= features[n] * sin_n;
            }
        }

        fft[i] = sqrt(re * re + im * im);
    }
}

This is the most direct porting available.

In the Benchmarks section, we'll see how this implementation can be speed-up with alternative implementations.

Benchmarking

The following table reports the benchmark on the MNIST dataset (64 features) with 10 principal FFT components vs various tecniques to decrease the computation time at the expense of memory usage.

*all the benchmarks were run on the Arduino 33 Nano BLE Sense

Some thoughts:

2. principalFFT w/ cos+sin LUT means I pre-compute the values of sin and cos at compile time, so there's no computation on the board; of course these lookup tables will eat some memory
3. principalFFT w/ cos LUT + sin sign LUT means I pre-compute the cos values only and compute sin using sqrt(1 - cos(x)^2); it adds some microseconds to the computation, but requires less memory
4. arduinoFFT library is faster than principalFFT in the execution time and requires less memory, even if principalFFT is only computing 10 frequencies: I need to investigate how it can achieve such performances

You can activate the LUT functionality with:

from micromlgen import port
from principalfft import PrincipalFFT

fft = PrincipalFFT(n_components=10).fit(X)

# cos lookup, sin computed
port(fft, lookup_cos=True)

# cos + sin lookup
port(fft, lookup_cos=True, lookup_sin=True)

Here's how the C code looks like with LUT.

void principalFFT(float *features, float *fft) {
    // apply principal FFT (naive implementation for the top N frequencies only)
    const int topFrequencies[] = { 0, 8, 17, 16, 1, 9, 2, 7, 15, 6 };
    const float cosLUT[10][64] = {
        {  1.0,  1.0,  1.0,  1.0,  1.0,  1.0,  1.0,  1.0,  1.0,  1.0,  1.0,  1.0,  1.0,  1.0,  1.0,  1.0,  1.0,  1.0,  1.0,  1.0,  1.0,  1.0,  1.0,  1.0,  1.0,  1.0,  1.0,  1.0,  1.0,  1.0,  1.0,  1.0,  1.0,  1.0,  1.0,  1.0,  1.0,  1.0,  1.0,  1.0,  1.0,  1.0,  1.0,  1.0,  1.0,  1.0,  1.0,  1.0,  1.0,  1.0,  1.0,  1.0,  1.0,  1.0,  1.0,  1.0,  1.0,  1.0,  1.0,  1.0,  1.0,  1.0,  1.0,  1.0},
        {  1.0,  0.7071,  6.1232e-17,  -0.7071,  -1.0,  -0.7071,  -1.8369e-16,  0.7071,  1.0,  0.7071,  3.0616e-16,  -0.7071,  -1.0,  -0.7071,  -4.2862e-16,  0.7071,  1.0,  0.7071,  5.5109e-16,  -0.7071,  -1.0,  -0.7071,  -2.4499e-15,  0.7071,  1.0,  0.7071,  -9.8033e-16,  -0.7071,  -1.0,  -0.7071,  -2.6948e-15,  0.7071,  1.0,  0.7071,  -7.3540e-16,  -0.7071,  -1.0,  -0.7071,  -2.9397e-15,  0.7071,  1.0,  0.7071,  -4.9047e-16,  -0.7071,  -1.0,  -0.7071,  -3.1847e-15,  0.7071,  1.0,  0.7071,  -2.4554e-16,  -0.7071,  -1.0,  -0.7071,  -3.4296e-15,  0.7071,  1.0,  0.7071,  -6.1898e-19,  -0.7071,  -1.0,  -0.7071,  -3.6745e-15,  0.7071},   ... };
    const bool sinLUT[10][64] = {
        {  false,  false,  false,  false,  false,  false,  false,  false,  false,  false,  false,  false,  false,  false,  false,  false,  false,  false,  false,  false,  false,  false,  false,  false,  false,  false,  false,  false,  false,  false,  false,  false,  false,  false,  false,  false,  false,  false,  false,  false,  false,  false,  false,  false,  false,  false,  false,  false,  false,  false,  false,  false,  false,  false,  false,  false,  false,  false,  false,  false,  false,  false,  false,  false},
        {  false,  true,  true,  true,  true,  false,  false,  false,  false,  true,  true,  true,  true,  false,  false,  false,  false,  true,  true,  true,  true,  false,  false,  false,  false,  true,  true,  true,  true,  false,  false,  false,  false,  true,  true,  true,  true,  false,  false,  false,  false,  true,  true,  true,  true,  false,  false,  false,  false,  true,  true,  true,  false,  false,  false,  false,  false,  true,  true,  true,  true,  false,  false,  false},  ...};

    for (int i = 0; i < 10; i++) {
        const int k = topFrequencies[i];
        const float harmonic = 0.09817477042468103 * k;
        float re = 0;
        float im = 0;
        // optimized case
        if (k == 0) {
            for (int n = 0; n < 64; n++) {
                re += features[n];
            }
        }

        else {
            for (int n = 0; n < 64; n++) {
                const float cos_n = cosLUT[i][n];
                const float sin_n = sinLUT[i][n] ? sqrt(1 - cos_n * cos_n) : -sqrt(1 - cos_n * cos_n);
                re += features[n] * cos_n;
                im -= features[n] * sin_n;
            }
        }

        fft[i] = sqrt(re * re + im * im);
    }
}

This post required much work to be produced, so I hope I didn't forgot anything and you found these information useful.
As always, there's a Github repo with all the code of this post.

L'articolo “Principal” FFT components as efficient features extrator proviene da Eloquent Arduino Blog.

from https://www.udemy.com/course/learn-audio-processing-complete-engineers-course/

Updated on 16 October 2020: step by step explanation of the process with ready-made sketch code

What you'll learn

This tutorial will teach you how to capture audio from the Arduino Nano 33 BLE Sense microphone and classify it: at the end of this post, you will have a trained model able to detect in real-time the word you tell, among the ones that you trained it to recognize. The classification will occur directly on your Arduino board.

This is not a general-purpose speech recognizer able to convert speech-to-text: it works only on the words you train it on.

What you'll need

• Hardware

  1. Arduino Nano 33 BLE Sense

• Software

  1. Python
  2. Python's module scikit-learn
  3. Python's module micromlgen

To install the software, open your terminal and install the libraries.

pip install -U scikit-learn
pip install -U micromlgen

Step 1. Capture audio samples

First of all, we need to capture a bunch of examples of the words we want to recognize.

In the original post, we used an analog microphone to record the audio. It is for sure the easiest way to interact with audio on a microcontroller since you only need to analogRead() the selected pin to get a value from the sensor.

This semplicity, however, comes at the cost of a nearly inexistent signal pre-processing from the sensor itself: most of the time, you will get junk - I don't want to be rude, but that's it.

Theory: Pulse-density modulation (a.k.a. PDM)

The microphone mounted on the Arduino Nano 33 BLE Sense (the MP34DT05) is fortunately much better than this: it gives you access to a modulated signal much more suitable for our processing needs.

The modulation used is pulse-density: I won't try to explain you how this works since I'm not an expert in DSP and neither it is the main scope of this article (refer to Wikipedia for some more information).

What matters to us is that we can grab an array of bytes from the microphone and extract its Root Mean Square (a.k.a. RMS) to be used as a feature for our Machine Learning model.

I had some difficulty finding examples on how to access the microphone on the Arduino Nano 33 BLE Sense board: fortunately, there's a Github repo from DelaGia that shows how to access all the sensors of the board.

I extracted the microphone part and incapsulated it in an easy to use class, so you don't really need to dig into the implementation details if you're not interested.

Practice: the code to capture the samples

When loaded on your Arduino Nano 33 BLE Sense, the following sketch will await for you to speak in front of the microphone: once it detects a sound, it will record 64 audio values and print them to the serial monitor.

From my experience, 64 samples are sufficient to cover short words such as yes, no, play, stop: if you plan to classify longer words, you may need to increase this number.

Download the Arduino Nano 33 BLE Sense - Capture audio samples sketch, open it the Arduino IDE and flash it to your board.

Here's the main code.

#include "Mic.h"

// tune as per your needs
#define SAMPLES 64
#define GAIN (1.0f/50)
#define SOUND_THRESHOLD 2000

float features[SAMPLES];
Mic mic;

void setup() {
    Serial.begin(115200);
    PDM.onReceive(onAudio);
    mic.begin();
    delay(3000);
}

void loop() {
    // await for a word to be pronounced
    if (recordAudioSample()) {
        // print features to serial monitor
        for (int i = 0; i < SAMPLES; i++) {
            Serial.print(features[i], 6);
            Serial.print(i == SAMPLES - 1 ? '\n' : ',');
        }

        delay(1000);
    }

    delay(20);
}

/**
 * PDM callback to update mic object
 */
void onAudio() {
    mic.update();
}

/**
 * Read given number of samples from mic
 */
bool recordAudioSample() {
    if (mic.hasData() && mic.data() > SOUND_THRESHOLD) {

        for (int i = 0; i < SAMPLES; i++) {
            while (!mic.hasData())
                delay(1);

            features[i] = mic.pop() * GAIN;
        }

        return true;
    }

    return false;
}

Now that we have the acquisition logic in place, it's time for you to record some samples of the words you want to classify.

Action: capture the words examples

Now you have to capture as many samples of the words you want to classify as possible.

Open the serial monitor and pronounce a word near the microphone: a line of numbers will be printed on the monitor.

This is the description of your word.

You need many lines like this for an accurate prediction, so keep repeating the same word 15-30 times.

After you repeated the same words many times, copy the content of the serial monitor and save it in a CSV file named after the word, for example yes.csv.

Then clear the serial monitor and repeat the process for each word.

Keep all these files in a folder because we need them to train our classifier.

Step 2. Train the machine learning model

Now that we have the samples, it's time to train the classifier.

Create a Python project in your favourite IDE or use your favourite text editor, if you don't have one.

As described in my post about how to train a classifier, we create a Python script that reads all the files inside a folder and concatenates them in a single array you feed to the classifier model.

Be sure your folder structure is like the following:

ArduinoWordClassification
  |-- train_classifier.py
  |-- data/
  |---- yes.csv
  |---- no.csv
  |---- play.csv
  |---- any other .csv file you recorded

# file: train_classifier.py

import numpy as np
from os.path import basename
from glob import glob
from sklearn.svm import SVC
from micromlgen import port
from sklearn.model_selection import train_test_split

def load_features(folder):
    dataset = None
    classmap = {}
    for class_idx, filename in enumerate(glob('%s/*.csv' % folder)):
        class_name = basename(filename)[:-4]
        classmap[class_idx] = class_name
        samples = np.loadtxt(filename, dtype=float, delimiter=',')
        labels = np.ones((len(samples), 1)) * class_idx
        samples = np.hstack((samples, labels))
        dataset = samples if dataset is None else np.vstack((dataset, samples))
    return dataset, classmap

np.random.seed(0)
dataset, classmap = load_features('data')
X, y = dataset[:, :-1], dataset[:, -1]
# this line is for testing your accuracy only: once you're satisfied with the results, set test_size to 1
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

clf = SVC(kernel='poly', degree=2, gamma=0.1, C=100)
clf.fit(X_train, y_train)

print('Accuracy', clf.score(X_test, y_test))
print('Exported classifier to plain C')
print(port(clf, classmap=classmap))

Among the classifiers I tried, SVM produced the best accuracy at 96% with 32 support vectors: it's not a super-tiny model, but it's quite small nevertheless.

If you're not satisifed with SVM, you can use Decision Tree, Random Forest, Gaussian Naive Bayes, Relevant Vector Machines. See my other posts for a detailed description of each.

In your console, after the accuracy score, you will have the plain C implementation of the classifier you trained. The following reports my SVM model.

// File: Classifier.h

#pragma once
namespace Eloquent {
    namespace ML {
        namespace Port {
            class SVM {
            public:
                /**
                * Predict class for features vector
                */
                int predict(float *x) {
                    float kernels[35] = { 0 };
                    float decisions[6] = { 0 };
                    int votes[4] = { 0 };
                    kernels[0] = compute_kernel(x,   33.0  , 41.0  , 47.0  , 54.0  , 59.0  , 61.0  , 56.0  , 51.0  , 50.0  , 51.0  , 44.0  , 32.0  , 23.0  , 15.0  , 12.0  , 8.0  , 5.0  , 0.0  , 0.0  , 0.0  , 0.0  , 0.0  , 0.0  , 0.0  , 5.0  , 3.0  , 5.0  , 0.0  , 0.0  , 0.0  , 0.0  , 0.0  , 0.0  , 0.0  , 0.0  , 0.0  , 0.0  , 0.0  , 0.0  , 0.0  , 0.0  , 0.0  , 0.0  , 0.0  , 0.0  , 0.0  , 0.0  , 0.0  , 0.0  , 0.0  , 0.0  , 0.0  , 0.0  , 0.0  , 0.0  , 0.0  , 0.0  , 0.0  , 0.0  , 0.0  , 0.0  , 0.0  , 0.0  , 0.0 );
                    kernels[1] = compute_kernel(x,   40.0  , 50.0  , 51.0  , 60.0  , 56.0  , 57.0  , 58.0  , 53.0  , 50.0  , 45.0  , 42.0  , 34.0  , 23.0  , 16.0  , 10.0  , 7.0  , 3.0  , 3.0  , 0.0  , 0.0  , 0.0  , 0.0  , 0.0  , 0.0  , 0.0  , 14.0  , 3.0  , 8.0  , 0.0  , 0.0  , 3.0  , 0.0  , 0.0  , 0.0  , 0.0  , 0.0  , 0.0  , 0.0  , 3.0  , 0.0  , 0.0  , 5.0  , 3.0  , 0.0  , 0.0  , 0.0  , 0.0  , 0.0  , 0.0  , 3.0  , 0.0  , 5.0  , 3.0  , 0.0  , 0.0  , 0.0  , 0.0  , 0.0  , 0.0  , 3.0  , 0.0  , 0.0  , 0.0  , 3.0 );
                    kernels[2] = compute_kernel(x,   56.0  , 68.0  , 78.0  , 91.0  , 84.0  , 84.0  , 84.0  , 74.0  , 69.0  , 64.0  , 57.0  , 44.0  , 33.0  , 18.0  , 12.0  , 8.0  , 5.0  , 9.0  , 15.0  , 12.0  , 12.0  , 9.0  , 12.0  , 7.0  , 3.0  , 10.0  , 12.0  , 6.0  , 3.0  , 0.0  , 0.0  , 0.0  , 0.0  , 6.0  , 3.0  , 6.0  , 10.0  , 10.0  , 8.0  , 3.0  , 9.0  , 9.0  , 9.0  , 8.0  , 9.0  , 9.0  , 11.0  , 3.0  , 8.0  , 9.0  , 8.0  , 8.0  , 8.0  , 6.0  , 7.0  , 3.0  , 3.0  , 8.0  , 5.0  , 3.0  , 0.0  , 3.0  , 0.0  , 0.0 );

                    // ...many other kernels computations...

                    decisions[0] = 0.722587775297
                                   + kernels[1] * 3.35855e-07
                                   + kernels[2] * 1.64612e-07
                                   + kernels[4] * 6.00056e-07
                                   + kernels[5] * 3.5195e-08
                                   + kernels[7] * -4.2079e-08
                                   + kernels[8] * -4.2843e-08
                                   + kernels[9] * -9.994e-09
                                   + kernels[10] * -5.11065e-07
                                   + kernels[11] * -5.979e-09
                                   + kernels[12] * -4.4672e-08
                                   + kernels[13] * -1.5606e-08
                                   + kernels[14] * -1.2941e-08
                                   + kernels[15] * -2.18903e-07
                                   + kernels[17] * -2.31635e-07
                            ;
                    decisions[1] = -1.658344586719
                                   + kernels[0] * 2.45018e-07
                                   + kernels[1] * 4.30223e-07
                                   + kernels[3] * 1.00277e-07
                                   + kernels[4] * 2.16524e-07
                                   + kernels[18] * -4.81187e-07
                                   + kernels[20] * -5.10856e-07
                            ;
                    decisions[2] = -1.968607562265
                                   + kernels[0] * 3.001833e-06
                                   + kernels[3] * 4.5201e-08
                                   + kernels[4] * 1.54493e-06
                                   + kernels[5] * 2.81834e-07
                                   + kernels[25] * -5.93581e-07
                                   + kernels[26] * -2.89779e-07
                                   + kernels[27] * -1.73958e-06
                                   + kernels[28] * -1.09552e-07
                                   + kernels[30] * -3.09126e-07
                                   + kernels[31] * -1.294219e-06
                                   + kernels[32] * -5.37961e-07
                            ;
                    decisions[3] = -0.720663029823
                                   + kernels[6] * 1.4362e-08
                                   + kernels[7] * 6.177e-09
                                   + kernels[9] * 1.25e-08
                                   + kernels[10] * 2.05478e-07
                                   + kernels[12] * 2.501e-08
                                   + kernels[15] * 4.363e-07
                                   + kernels[16] * 9.147e-09
                                   + kernels[18] * -1.82182e-07
                                   + kernels[20] * -4.93707e-07
                                   + kernels[21] * -3.3084e-08
                            ;
                    decisions[4] = -1.605747746589
                                   + kernels[6] * 6.182e-09
                                   + kernels[7] * 1.3853e-08
                                   + kernels[8] * 2.12e-10
                                   + kernels[9] * 1.1243e-08
                                   + kernels[10] * 7.80681e-07
                                   + kernels[15] * 8.347e-07
                                   + kernels[17] * 1.64985e-07
                                   + kernels[23] * -4.25014e-07
                                   + kernels[25] * -1.134803e-06
                                   + kernels[34] * -2.52038e-07
                            ;
                    decisions[5] = -0.934328303475
                                   + kernels[19] * 3.3529e-07
                                   + kernels[20] * 1.121946e-06
                                   + kernels[21] * 3.44683e-07
                                   + kernels[22] * -6.23056e-07
                                   + kernels[24] * -1.4612e-07
                                   + kernels[28] * -1.24025e-07
                                   + kernels[29] * -4.31701e-07
                                   + kernels[31] * -9.2146e-08
                                   + kernels[33] * -3.8487e-07
                            ;
                    votes[decisions[0] > 0 ? 0 : 1] += 1;
                    votes[decisions[1] > 0 ? 0 : 2] += 1;
                    votes[decisions[2] > 0 ? 0 : 3] += 1;
                    votes[decisions[3] > 0 ? 1 : 2] += 1;
                    votes[decisions[4] > 0 ? 1 : 3] += 1;
                    votes[decisions[5] > 0 ? 2 : 3] += 1;
                    int val = votes[0];
                    int idx = 0;

                    for (int i = 1; i < 4; i++) {
                        if (votes[i] > val) {
                            val = votes[i];
                            idx = i;
                        }
                    }

                    return idx;
                }

                /**
                * Convert class idx to readable name
                */
                const char* predictLabel(float *x) {
                    switch (predict(x)) {
                        case 0:
                            return "no";
                        case 1:
                            return "stop";
                        case 2:
                            return "play";
                        case 3:
                            return "yes";
                        default:
                            return "Houston we have a problem";
                    }
                }

            protected:
                /**
                * Compute kernel between feature vector and support vector.
                * Kernel type: poly
                */
                float compute_kernel(float *x, ...) {
                    va_list w;
                    va_start(w, 64);
                    float kernel = 0.0;

                    for (uint16_t i = 0; i < 64; i++) {
                        kernel += x[i] * va_arg(w, double);
                    }

                    return pow((0.1 * kernel) + 0.0, 2);
                }
            };
        }
    }
}

Step 3. Deploy to your microcontroller

Now we have all the pieces we need to perform word classification on our Arduino board.

Download the Arduino Nano 33 BLE Sense - Audio classification sketch, open it in the Arduino IDE and paste the plain C code you got in the console inside the Classifier.h file (delete all its contents before!).

Fine: it's time to deploy!

Hit the upload button: if everything went fine, open the serial monitor and pronounce one of the words you recorded during Step 1.

Hopefully, you will read the word on the serial monitor.

Here's a quick demo (please forgive me for the bad video quality).

If you liked this tutorial and it helped you successfully implement word classification on your Arduino Nano 33 BLE Sense, please share it on your social media so others can benefit too.

If you have troubles or questions, don't hesitate to leave a comment: I will be happy to help you.

L'articolo Better word classification with Arduino Nano 33 BLE Sense and Machine Learning proviene da Eloquent Arduino Blog.

(Gaussian) Naive Bayes

Naive Bayes classifiers are simple models based on the probability theory that can be used for classification.

They originate from the assumption of independence among the input variables. Even though this assumption doesn't hold true in the vast majority of the cases, they often perform very good at many classification tasks, so they're quite popular.

Gaussian Naive Bayes stack another (mostly wrong) assumption: that the variables exhibit a Gaussian probability distribution.

I (and many others like me) will never understand how it is possible that so many wrong assumptions lead to such good performances!

Nevertheless, what is important to us is that sklearn implements GaussianNB, so we easily train such a classifier.
The most interesting part is that GaussianNB can be tuned with just a single parameter: var_smoothing.

Don't ask me what it does in theory: in practice you change it and your accuracy can boost. This leads to an easy tuning process that doesn't involves expensive grid search.

import sklearn.datasets as d
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import normalize
from sklearn.naive_bayes import GaussianNB

def pick_best(X_train, X_test, y_train, y_test):
    best = (None, 0)
    for var_smoothing in range(-7, 1):
        clf = GaussianNB(var_smoothing=pow(10, var_smoothing))
        clf.fit(X_train, y_train)
        y_pred = clf.predict(X_test)
        accuracy = (y_pred == y_test).sum()
        if accuracy > best[1]:
            best = (clf, accuracy)
    print('best accuracy', best[1] / len(y_test))
    return best[0]

iris = d.load_iris()
X = normalize(iris.data)
y = iris.target
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3)
clf = pick_best(X_train, X_test, y_train, y_test)

This simple procedure will train a bunch of classifiers with a different var_smoothing factor and pick the best performing one.

EloquentML integration

Once you have your trained classifier, porting it to C is as easy as always:

from micromlgen import port

clf = pick_best()
print(port(clf))

Always remember to run

pip install --upgrade micromlgen

port is a magic method able to port many classifiers: it will automatically detect the proper converter for you.

What does the exported code looks like?

#pragma once
namespace Eloquent {
    namespace ML {
        namespace Port {
            class GaussianNB {
                public:
                    /**
                    * Predict class for features vector
                    */
                    int predict(float *x) {
                        float votes[3] = { 0.0f };
                        float theta[4] = { 0 };
                        float sigma[4] = { 0 };
                        theta[0] = 0.801139789889; theta[1] = 0.54726920354; theta[2] = 0.234408773313; theta[3] = 0.039178084094;
                        sigma[0] = 0.000366881742; sigma[1] = 0.000907992556; sigma[2] = 0.000740960787; sigma[3] = 0.000274925514;
                        votes[0] = 0.333333333333 - gauss(x, theta, sigma);
                        theta[0] = 0.748563871324; theta[1] = 0.349390892644; theta[2] = 0.536186138345; theta[3] = 0.166747384117;
                        sigma[0] = 0.000529727082; sigma[1] = 0.000847956504; sigma[2] = 0.000690057342; sigma[3] = 0.000311828658;
                        votes[1] = 0.333333333333 - gauss(x, theta, sigma);
                        theta[0] = 0.704497203305; theta[1] = 0.318862439835; theta[2] = 0.593755956917; theta[3] = 0.217288784452;
                        sigma[0] = 0.000363782089; sigma[1] = 0.000813846722; sigma[2] = 0.000415475678; sigma[3] = 0.000758478249;
                        votes[2] = 0.333333333333 - gauss(x, theta, sigma);
                        // return argmax of votes
                        uint8_t classIdx = 0;
                        float maxVotes = votes[0];

                        for (uint8_t i = 1; i < 3; i++) {
                            if (votes[i] > maxVotes) {
                                classIdx = i;
                                maxVotes = votes[i];
                            }
                        }

                        return classIdx;
                    }

                protected:
                    /**
                    * Compute gaussian value
                    */
                    float gauss(float *x, float *theta, float *sigma) {
                        float gauss = 0.0f;

                        for (uint16_t i = 0; i < 4; i++) {
                            gauss += log(sigma[i]);
                            gauss += pow(x[i] - theta[i], 2) / sigma[i];
                        }

                        return gauss;
                    }
                };
            }
        }
    }

As you can see, we need a couple of "weight vectors":

• theta is the mean of each feature
• sigma is the standard deviation

The computation is quite thin: just a couple of operations; the class with the highest score is then selected.

Benchmarks

Following there's a recap of a couple benchmarks I run on an Arduino Nano 33 Ble Sense.

We can see that the accuracy is on par with a linear SVM, reaching up to 97% on some datasets. Its semplicity shines with high-dimensional datasets (breast cancer) where execution time is half of the LinearSVC: I can see this pattern repeating with other real-world, medium-sized datasets.

This is it, you can find the example project on Github.

L'articolo EloquentML grows its family of classifiers: Gaussian Naive Bayes on Arduino proviene da Eloquent Arduino Blog.

A few weeks ago I was wandering over arxiv.org looking for insipiration relative to Machine learning on microcontrollers when I found exactly what I was looking for.

SEFR: A Fast Linear-Time Classifier for Ultra-Low Power Devices is a paper from Hamidreza Keshavarz, Mohammad Saniee Abadeh, Reza Rawassizadeh where the authors develop a binary classifier that is:

• fast during training
• fast during prediction
• requires minimal memory

It has been specifically designed for embedded machine learning, so no optimization is required to run in on microcontrollers: it is tiny by design. In short, it uses a combination of the averages of the features as weights plus a bias to distinguish between positive and negative class. If you read the paper you will sure understand it: it's very straightforward.

How to use

The authors both provided a C and Python implementation on Github you can read. I ported the C version "manually" to my Eloquent ML library and created a Python package called sefr copy-pasting from the original repo.

Here's a Python example.

from sefr import SEFR
from sklearn.datasets import load_iris
from sklearn.preprocessing import normalize
from sklearn.model_selection import train_test_split

if __name__ == '__main__':
    iris = load_iris()
    X = normalize(iris.data)
    y = iris.target
    X = X[y < 2]
    y = y[y < 2]
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3)
    clf = SEFR()
    clf.fit(X_train, y_train)
    print('accuracy', (clf.predict(X_test) == y_test).sum() / len(y_test))

How good is it?

Considering that the model only needs 1 weight per feature, I think this results are impressive!

Micromlgen integration

The Python porting was done so I could integrate it easily in my micromlgen package.

How to use it?

from sefr import SEFR
from sklearn.datasets import load_iris
from micromlgen import port

if __name__ == '__main__':
    iris = load_iris()
    X = iris.data
    y = iris.target
    X = X[y < 2]
    y = y[y < 2]
    clf = SEFR()
    clf.fit(X_train, y_train)
    print(port(clf))

The produced code is so compact I will report it here.

#pragma once
namespace Eloquent {
    namespace ML {
        namespace Port {
            class SEFR {
                public:
                    /**
                    * Predict class for features vector
                    */
                    int predict(float *x) {
                        return dot(x,   0.084993602632  , -0.106163278477  , 0.488989863684  , 0.687022900763 ) <= 2.075 ? 0 : 1;
                    }

                protected:
                    /**
                    * Compute dot product between features vector and classifier weights
                    */
                    float dot(float *x, ...) {
                        va_list w;
                        va_start(w, 4);
                        float kernel = 0.0;

                        for (uint16_t i = 0; i < 4; i++) {
                            kernel += x[i] * va_arg(w, double);
                        }

                        return kernel;
                    }
                };
            }
        }
    }

In your sketch:

#include "IrisSEFR.h"
#include "IrisTest.h"

void setup() {
    Serial.begin(115200);
}

void loop() {
    Eloquent::ML::Port::SEFR clf;
    Eloquent::ML::Test::IrisTestSet testSet;

    testSet.test(clf);
    Serial.println(testSet.dump());
    delay(5000);
}

You have to clone the Github example to compile the code.

That's all for today, I hope you will try this classifier and find a project it fits in: I'm very impressed by the easiness of implementation yet the accuracy it can achieve on benchmark datasets.

In the next weeks I'm thinking in implementing a multi-class version of this and see how it performs, so stay tuned!

L'articolo SEFR: A Fast Linear-Time Classifier for Ultra-Low Power Devices proviene da Eloquent Arduino Blog.

Why dimensionality reduction on Arduino microcontrollers?

Dimensionality reduction is a tecnique you see often in Machine Learning projects. By stripping away "unimportant" or redundant information, it generally helps speeding up the training process and achieving higher classification performances.

Since we now know we can run Machine Learning on Arduino boards and embedded microcontrollers, it can become a key tool at our disposal to squeeze out the most out of our boards.

In the specific case of resource-constrained devices as old Arduino boards (the UNO for example, with only 2 kb of RAM), it can become a decisive turn in unlocking even more application scenarios where the high dimensionality of the input features would not allow any model to fit.

Let's take the Gesture classification project as an example: among the different classifiers we trained, only one fitted on the Arduino UNO, since most of them required too much flash memory due to the high dimension of features (90) and support vectors (25 to 61).

In this post I will resume that example and see if dimensionality reduction can help reduce this gap.

If you are working on a project with many features, let me know in the comments so I can create a detailed list of real world examples.

How to export PCA (Principal Component Analysis) to plain C

Among the many algorithms available for dimensionality reduction, I decided to start with PCA (Principal Component Analysis) because it's one of the most widespread. In the next weeks I will probably work on porting other alternatives.

If you never used my Python package micromlgen I first invite you to read the introduction post to get familiar with it.

Always remember to install the latest version, since I publish frequent updates.

pip install --upgrade micromlgen

Now it is pretty straight-forward to convert a sklearn PCA transformer to plain C: you use the magic method port. In addition to converting SVM/RVM classifiers, it is now able to export PCA too.

from sklearn.decomposition import PCA
from sklearn.datasets import load_iris
from micromlgen import port

if __name__ == '__main__':
    X = load_iris().data
    pca = PCA(n_components=2, whiten=False).fit(X)

    print(port(pca))

How to deploy PCA to Arduino

To use the exported code, we first have to include it in our sketch. Save the contents to a file (I named it pca.h) in the same folder of your .ino project and include it.

#include "pca.h"

// this was trained on the IRIS dataset, with 2 principal components
Eloquent::ML::Port::PCA pca;

The pca object is now able to take an array of size N as input and return an array of size K as output, with K < N usually.

void setup() {
    float x_input[4] = {5.1, 3.5, 1.4, 0.2};
    float x_output[2];

    pca.transform(x_input, x_output);
}

That's it: now you can run your classifier on x_output.

#include "pca.h"
#include "svm.h"

Eloquent::ML::Port::PCA pca;
Eloquent::ML::Port::SVM clf;

void setup() {
    float x_input[4] = {5.1, 3.5, 1.4, 0.2};
    float x_output[2];
    int y_pred;

    pca.transform(x_input, x_output);

    y_pred = clf.predict(x_output);
}

A real world example

As I anticipated, let's take a look at how PCA dimensionality reduction can help in fitting classifiers that would otherwise be too large to fit on our microcontrollers.

This is the exact table from the Gesture classification project.

The dataset has 90 features (30 samples x 3 axes) and achieves 99% accuracy.

Let's pick the poly kernel with degree 2 and see how much we can decrease the number of components while still achieving a good accuracy.

We clearly see a couple of things:

1. we still achieve 99% accuracy even with only 40 out of 90 principal components
2. we get a satisfactory 90% accuracy even with only 15 components
3. (this is a bit unexpected) it looks like there's a sweet spot at 10 components where the accuracy skyrockets to 99% again. This could be just a contingency of this particular dataset, don't expect to replicate this results on your own dataset

What do these numbers mean to you? It means your board has to do many less computations to give you a prediction and will probably be able to host a more complex model.

Let's check out the figures with n_components = 10 compared with the ones without PCA.

A couple notes:

1. accuracy increased (on stayed the same) for all kernels
2. with one exception, flash size decreased in the range 20 - 50%
3. now we can fit 3 classifiers on our Arduino UNO instead of only one

I will probably spend some more time investingating the usefulness of PCA for Arduino Machine Learning projects, but for now that's it: it's a good starting point in my opinion.

There's a little example sketch on Github that applies PCA to the IRIS dataset.

Tell me what you think may be a clever application of dimensionality reduction in the world of microcontrollers and see if we can build something great together.

L'articolo Arduino dimensionality reduction (PCA) for Machine Learning projects proviene da Eloquent Arduino Blog.

One vs One

Many classifiers are, by nature, binary: they can only distinguish the positive class from the negative one. Many of real-world problems, however, are multiclass: you have 3 or more possible outcomes to distinguish from.

There are a couple of ways to achieve this:

1. One vs All: if your classifier is able to output a confidence score of its prediction, for N classes you train N classifiers, each able to recognize a single class. During inference, you pick the "most confident" one.
2. One vs One: for N classes, you train N * (N-1) / 2 classifiers, one for each couple of classes. During inference, each classifier makes a prediction and you pick the class with the highest number of votes.

Since SGD and Passive-Aggressive don't output a confidence score, I implemented the One vs One algorithm to tackle the multiclass classification problem on microcontrollers.

Actually, One vs One is not a new type of classifier: it is really a "coordinator" class that sorts which samples go to which classifier. You can still choose your own classifier type to use.

As SGD and Passive-Aggressive, OneVsOne implements the classifier interface, so you will use the well known fitOne and predict methods.

Example code

// Esp32 has some problems with min/max
#define min(a, b) (a) < (b) ? (a) : (b)
#define max(a, b) (a) > (b) ? (a) : (b)
// you will actually need only one of SGD or PassiveAggressive
#include "EloquentSGD.h"
#include "EloquentPassiveAggressive.h"
#include "EloquentOneVsOne.h"
#include "EloquentAccuracyScorer.h"
// this file defines NUM_FEATURES, NUM_CLASSES, TRAIN_SAMPLES and TEST_SAMPLES
#include "dataset.h"

using namespace Eloquent::ML;

void setup() {
  Serial.begin(115200);
  delay(3000);
}

void loop() {
  AccuracyScorer scorer;
  // OneVsOne needs the actual classifier class, the number of features and the number of classes
  OneVsOne<SGD<FEATURES_DIM>, FEATURES_DIM, NUM_CLASSES> clf;

  // clf.set() propagates the configuration to the actual classifiers
  // if a parameter does not exists on the classifier, it does nothing
  // in this example, alpha and momentum refer to SGD, C to Passive-Aggressive
  clf.set("alpha", 1);
  clf.set("momentum", 0.7);
  clf.set("C", 0.1);

  // fit
  // I noticed that repeating the training a few times over the same dataset increases performance  to a certain extent: if you re-train it too much, performance will decay
  for (unsigned int i = 0; i < TRAIN_SAMPLES * 5; i++) {
      clf.fitOne(X_train[i % TRAIN_SAMPLES], y_train[i % TRAIN_SAMPLES]);
  }

  // predict
  for (int i = 0; i < TEST_SAMPLES; i++) {
      int y_true = y_test[i];
      int y_pred = clf.predict(X_test[i]);

      Serial.print("Predicted ");
      Serial.print(y_pred);
      Serial.print(" vs ");
      Serial.println(y_true);
      scorer.scoreOne(y_true, y_pred);
  }

  Serial.print("Accuracy = ");
  Serial.print(scorer.accuracy() * 100);
  Serial.print(" out of ");
  Serial.print(scorer.support());
  Serial.println(" samples");
  delay(30000);
}

If you refer to the previous posts on SGD and Passive-Aggressive, you'll notice that you would be able to replace one with the other and your code will change by 1 single line only. This let's you experiment to find the best configuration for your project without hassle.

Accuracy

Well, accuracy vary.

In my tests, I couldn't get predictable accuracy on all datasets. I couldn't even get acceptable accuracy on the Iris dataset (60% max). But I got 90% accuracy on the Digits dataset from scikit-learn with 6 classes.

You have to experiment. Try Passive-Aggressive with many C values. If it doesn't work, try SGD with varying momentum and alpha. Try to repeat the training over the dataset 5, 10 times.

In a next post I'll report my benchmarks so you can see what works for you and what not.
This is an emerging field for me, so I will need time to master it.

As always, you can find the examle on Github with a the dataset to experiment with.

L'articolo Incremental multiclass classification on microcontrollers: One vs One proviene da Eloquent Arduino Blog.

A brief recap on Stochastic Gradient Descent

If you ever worked with Machine learning, you surely know about Gradient descent: it is an iterative algorithm to optimize a loss function.

It is much general-purpose, in the sense that it is not bound to a particular application, but it has been heavily used in Neural networks in the recent years.

Yet, it can be used as a classifier on its own if you set its loss function as the classification error.

This is the core update rule of Gradient descent: quite simple.

As you see, there's a summation in the formula: this means we need to cycle through the entire training set to compute the update to the weights.

In case of large datasets, this can be slow or not possible at all.

And requires a lot of memory.

And we don't have memory on microcontrollers.

So we need Stochastic gradient descent.

Stochastic gradient descent has the same exact update rule, but it is applied on the single training sample.

Imagine the summation goes from 1 to 1, instead of m.

That's it.

How to use

The pattern of use is similar to that of the Passive Aggressive classifier: you have the fitOne and predict methods.

First of all, download the library from Github.

#include <EloquentSGD.h>
#include <EloquentAccuracyScorer.h>
#include "iris.h"

#define VERBOSE

using namespace Eloquent::ML;

void setup() {
    Serial.begin(115200);
    delay(3000);
}

void loop() {
    int trainSamples;
    int retrainingCycles;
    SGD<FEATURES_DIM> clf;
    AccuracyScorer scorer;

    // ....

    // train
    for (uint16_t cycle = 0; cycle < retrainingCycles; cycle++)
        for (uint16_t i = 0; i < trainSamples; i++)
            clf.fitOne(X[i], y[i]);

    // predict
    for (uint16_t i = trainSamples; i < DATASET_SIZE; i++) {
        int predicted = clf.predict(X[i]);
        int actual = y[i];

        scorer.scoreOne(actual, predicted);
    }

    Serial.print("Accuracy: ");
    Serial.print(round(100 * scorer.accuracy()));
    Serial.print("% out of ");
    Serial.print(scorer.support());
    Serial.println(" predictions");
}

In this case we're working with known datasets, so we cycle through them for the training, but if you're learning "on-line", from samples generated over time, it will work exactly the same.

A bit of momentum

Stochastic gradient descent works quite well out of the box in most cases.

Sometimes, however, its updates can start "oscillating".

To solve this problem, it has been proposed the momentum technique, which can both speed up learning and increase the accuracy.

In my personal tests, I was able to achieve up to +5% in accuracy on the majority of datasets.

To use it, you only need to set a decay factor between 0 and 1.

SGD clf;

clf.momentum(0.5);

Run on your own

On Github you can find the full example with some benchmark datasets to try on your own.

The example is interactive an will ask you how many samples to use for the training and how many times to cycle through them.

This is something you should consider: if you have a training set and can store it somehow (in memory or on Flash for example), re-presenting the same samples to the SGD classifier could (and probably will) increase its performance if done correctly.

This happens because the algorithm needs some time to converge and if it doesn't receive enough samples it won't learn properly.

Of course, if you re-use the same samples over and over again, you're likely to overfit.

L'articolo Stochastic Gradient Descent on your microcontroller proviene da Eloquent Arduino Blog.

Batch learning

A couple weeks ago I started exploring the possibility to train a machine learning classifier directly on a microcontroller. Since I like SVM, I ported the simplified SVM SMO (Sequential Minimal Optimization) algorithm to plain C, ready to be deployed to embedded devices.

Now, that kind of algorithm works in the so-called "batch-mode": it needs all the training data to be available in memory to learn.

This may be a limiting factor on resource-constrained devices, since it poses an upper bound to the number of samples you can train on. And when working with high-dimensional datasets, the number of samples could be not enough to achieve good accuracy.

Enter incremental learning

To solve this limitation, you need a totally different kind of learning algorithms: you need incremental (a.k.a online a.k.a out of core) learning.

Incremental learning works by inspecting one training sample at a time, instead of all at once.

The clear advantage is that you have a tiny memory footprint. And this is a huge advantage.

The clear disadvantage is that you don't have the "big picture" of your data, so:

• the end result will probably be affected by the order of presentation of the samples
• you may not be able to achieve top accuracy

Passive-aggressive classifier

Passive-aggressive classification is one of the available incremental learning algorithms and it is very simple to implement, since it has a closed-form update rule.

Please refer to this short explanation on Passive-aggressive classifiers for a nice description with images.

The core concept is that the classifier adjusts it weight vector for each mis-classified training sample it receives, trying to get it correct.

Benchmarks

I run a couple benchmark on my Esp32 to assess both accuracy and training time.

First of all: it is fast!. When I say it is fast I mean it takes ~1ms to train on 400 samples x 30 features each.

Talking about accuracy instead... Uhm...

Accuracy vary. Greatly.

You can achieve 100% on some datasets.

And 40% on others. But on those same datasets you can achieve >85% if training on a different number of samples. Or in a different order.

I guess this is the tradeoff for such a simple and space-efficient algorithm.

I report my results in the following table. It is not meant to be an exhaustive benchmark of the classifier, since those number will vary based on the order of presentation, but still you can get an idea of what it is able to achieve.

Time to code

Here I'll report an extract of the example code you can find on Github for this classifier.

#include "EloquentPassiveAggressiveClassifier.h"
#include "EloquentAccuracyScorer.h"
#include "iris.h"

using namespace Eloquent::ML;

void loop() {
    int trainSamples;
    PassiveAggressiveClassifier<FEATURES_DIM> clf;
    AccuracyScorer scorer;

    trainSamples = readSerialNumber("How many samples will you use as training?", DATASET_SIZE - 2);

    if (trainSamples == 0)
        return;

    clf.setC(1);

    // train
    for (uint16_t i = 0; i < trainSamples; i++)
        clf.fitOne(X[i], y[i]);

    // predict
    for (uint16_t i = trainSamples; i < DATASET_SIZE; i++) {
        int predicted = clf.predict(X[i]);
        int actual = y[i] > 0 ? 1 : -1;

        scorer.scoreOne(actual, predicted);
    }

    Serial.print("Accuracy: ");
    Serial.print(round(100 * scorer.accuracy()));
    Serial.print("% out of ");
    Serial.print(scorer.support());
    Serial.println(" predictions");
}

On the project page you will find the code to reproduce these numbers.

L'articolo Passive-aggressive classifier for embedded devices proviene da Eloquent Arduino Blog.

This will be an hands-on guide, so let's walk throughout each step you need to complete to run the example.

I setup this very example as a basis for your future projects, so you can easily swap the color classification task for any other one you could think of.

Definitions

#ifdef ESP32
#define min(a, b) (a) < (b) ? (a) : (b)
#define max(a, b) (a) > (b) ? (a) : (b)
#define abs(x) ((x) > 0 ? (x) : -(x))
#endif

#include <EloquentSVMSMO.h>
#include "RGB.h"

#define MAX_TRAINING_SAMPLES 20
#define FEATURES_DIM 3

using namespace Eloquent::ML;

int numSamples;
RGB rgb(2, 3, 4);
float X_train[MAX_TRAINING_SAMPLES][FEATURES_DIM];
int y_train[MAX_TRAINING_SAMPLES];
SVMSMO<FEATURES_DIM> classifier(linearKernel);

When training a classifier on your microcontroller there are some things that are mandatory:

1. #include <EloquentSVMSMO.h>: this is the library that implements the SVM learning algorithm
2. X_train: this is a matrix where each row represents a training sample. You will need to keep this data always with you, since it's required also during the inference
3. y_train: this array contains, for each training sample, the class it belongs to: 1 or -1
4. linearKernel: this is the kernel function for the SVM classifier (you can read more here). You can pass your own kernel other than linear (for example poly or rbf)

In this specific example, we're using the RGB class to handle the TCS3200 sensor reading, but this will change based on the dataset you want to train on. Also, since our features are going to be the R, G and B components of a color, FEATURES_DIM is set to 3.

Setup

void setup() {
    Serial.begin(115200);
    rgb.begin();

    classifier.setC(5);
    classifier.setTol(1e-5);
    classifier.setMaxIter(10000);
}

The setup does not contain any logic really. You can use this part to configure the parameters of the classifier:

• C: "The C parameter tells the SVM optimization how much you want to avoid misclassifying each training example. For large values of C, the optimization will choose a smaller-margin hyperplane if that hyperplane does a better job of getting all the training points classified correctly. Conversely, a very small value of C will cause the optimizer to look for a larger-margin separating hyperplane, even if that hyperplane misclassifies more points" (quoted from stackexchange)
• tol: "The tol parameter is a setting for the SVM's tolerance in optimization. Recall that yi(xi.w+b)-1 >= 0. For an SVM to be valid, all values must be greater than or equal to 0, and at least one value on each side needs to be "equal" to 0, which will be your support vectors. Since it is highly unlikely that you will actually get values equal perfectly to 0, you set tolerance to allow a bit of wiggle room." (quoted from pythonprogramming)
• maxIter: set an upper bound to the number of iterations the algorithm can take to converge
• passes: max # of times to iterate over α’s without changing
• alphaTol: alfpha coefficients determine which samples from the training set are to be considered support vectors and so be included during the inference procedure. This value discards support vectors with an alpha too small to be noticeable.

Fit

else if (command == "fit") {
        Serial.print("How many samples will you record? ");
        numSamples = readSerialNumber();

        for (int i = 0; i < numSamples; i++) {
            Serial.print(i + 1);
            Serial.print("/");
            Serial.print(numSamples);
            Serial.println(" Which class does the sample belongs to, 1 or -1?");
            y_train[i] = readSerialNumber() > 0 ? 1 : -1;
            getFeatures(X_train[i]);
        }

        Serial.print("Start training... ");
        classifier.fit(X_train, y_train, numSamples);
        Serial.println("Done");
    }

This is the core of the project. Here we are loading the samples to train our classifier "live" on the board.

Since this is an interactive demo, the program prompts us to define how many samples we'll load and, one by one, which class they belong to.

Now there are a few important things to keep in mind:

• numSamples: sadly, C has no easy way to know the size of an array, so we have to be explicit about it. To train the classifier, it is mandatory that you do know I many samples you're passing to it
• getFeatures() is the function that reads the training sample. It is actually a "proxy" to your own custom logic: in this example it reads the TCS3200, in your project it could read an accelerometer or the like.
• fit(): this is where the magic happens. With this single line of code you're training the SVM on the training data; when the functions ends, the classifier will have updated its internal state with the coefficients it needs to classify new samples

Predict

ColorClassificationTrainingExample.ino

else if (command == "predict") {
        int label;
        float x[FEATURES_DIM];

        getFeatures(x);
        Serial.print("Predicted label is ");
        Serial.println(classifier.predict(X_train, x));
    }

Now that our classifier has been trained, we can finally make use of it to classify new samples.

As easy as it can be, you just call its predict method.

And that's it: you can now complete your Machine learning task on your microcontroller from start to end, without the need of a PC.

Check the full project code on Github

Full example

ColorClassificationTrainingExample.ino

#include <EloquentSVMSMO.h>
#include "RGB.h"

#define MAX_TRAINING_SAMPLES 20
#define FEATURES_DIM 3

using namespace Eloquent::ML;

int numSamples;
RGB rgb(2, 3, 4);
float X_train[MAX_TRAINING_SAMPLES][FEATURES_DIM];
int y_train[MAX_TRAINING_SAMPLES];
SVMSMO<FEATURES_DIM> classifier(linearKernel);

void setup() {
    Serial.begin(115200);
    rgb.begin();

    classifier.setC(5);
    classifier.setTol(1e-5);
    classifier.setMaxIter(10000);
}

void loop() {
    if (!Serial.available()) {
        delay(100);
        return;
    }

    String command = Serial.readStringUntil('\n');

    if (command == "help") {
        Serial.println("Available commands:");
        Serial.println("\tfit: train the classifier on a new set of samples");
        Serial.println("\tpredict: classify a new sample");
        Serial.println("\tinspect: print X_train and y_train");
    }
    else if (command == "fit") {
        Serial.print("How many samples will you record? ");
        numSamples = readSerialNumber();

        for (int i = 0; i < numSamples; i++) {
            Serial.print(i + 1);
            Serial.print("/");
            Serial.print(numSamples);
            Serial.println(" Which class does the sample belongs to, 1 or -1?");
            y_train[i] = readSerialNumber() > 0 ? 1 : -1;
            getFeatures(X_train[i]);
        }

        Serial.print("Start training... ");
        classifier.fit(X_train, y_train, numSamples);
        Serial.println("Done");
    }
    else if (command == "predict") {
        int label;
        float x[FEATURES_DIM];

        getFeatures(x);
        Serial.print("Predicted label is ");
        Serial.println(classifier.predict(X_train, x));
    }
    else if (command == "inspect") {
        for (int i = 0; i < numSamples; i++) {
            Serial.print("[");
            Serial.print(y_train[i]);
            Serial.print("] ");

            for (int j = 0; j < FEATURES_DIM; j++) {
                Serial.print(X_train[i][j]);
                Serial.print(", ");
            }

            Serial.println();
        }
    }
}

/**
 *
 * @return
 */
int readSerialNumber() {
    while (!Serial.available()) delay(1);

    return Serial.readStringUntil('\n').toInt();
}

/**
 * Get features for new sample
 * @param x
 */
void getFeatures(float x[FEATURES_DIM]) {
    rgb.read(x);

    for (int i = 0; i < FEATURES_DIM; i++) {
        Serial.print(x[i]);
        Serial.print(", ");
    }

    Serial.println();
}

RGB.h

#pragma once

/**
 * Wrapper for RGB color sensor
 */
class RGB {
    public:
        RGB(uint8_t s2, uint8_t s3, uint8_t out) :
            _s2(s2),
            _s3(s3),
            _out(out) {

        }

        /**
         *
         */
        void begin() {
            pinMode(_s2, OUTPUT);
            pinMode(_s3, OUTPUT);
            pinMode(_out, INPUT);
        }

        /**
         *
         * @param x
         */
        void read(float x[3]) {
            x[0] = readComponent(LOW, LOW);
            x[1] = readComponent(HIGH, HIGH);
            x[2] = readComponent(LOW, HIGH);
        }

    protected:
        uint8_t _s2;
        uint8_t _s3;
        uint8_t _out;

        /**
         *
         * @param s2
         * @param s3
         * @return
         */
        int readComponent(bool s2, bool s3) {
            delay(10);
            digitalWrite(_s2, s2);
            digitalWrite(_s3, s3);

            return pulseIn(_out, LOW);
        }
};

L'articolo How to train a color classification Machine learning classifier directly on your Arduino board proviene da Eloquent Arduino Blog.