![\"PCA](\"https://setosa.io/ev/principal-component-analysis/fb-thumb.png\")
When working with Machine Learning projects on microcontrollers and embedded devices the dimension of features can become a limiting factor due to the lack of RAM: dimensionality reduction (eg. PCA) will help you shrink your models and even achieve higher prediction accuracy.
\n
\n
Dimensionality reduction is a tecnique you see often in Machine Learning projects. By stripping away "unimportant" or redundant information, it generally helps speeding up the training process and achieving higher classification performances.
\nSince we now know we can run Machine Learning on Arduino boards and embedded microcontrollers, it can become a key tool at our disposal to squeeze out the most out of our boards.
\nIn the specific case of resource-constrained devices as old Arduino boards (the UNO for example, with only 2 kb of RAM), it can become a decisive turn in unlocking even more application scenarios where the high dimensionality of the input features would not allow any model to fit.
\nLet's take the Gesture classification project as an example: among the different classifiers we trained, only one fitted on the Arduino UNO, since most of them required too much flash memory due to the high dimension of features (90) and support vectors (25 to 61).
\nIn this post I will resume that example and see if dimensionality reduction can help reduce this gap.
\nIf you are working on a project with many features, let me know in the comments so I can create a detailed list of real world examples.
\nAmong the many algorithms available for dimensionality reduction, I decided to start with PCA (Principal Component Analysis) because it's one of the most widespread. In the next weeks I will probably work on porting other alternatives.
\nIf you never used my Python package micromlgen I first invite you to read the introduction post to get familiar with it.
\nAlways remember to install the latest version, since I publish frequent updates.
\npip install --upgrade micromlgenNow it is pretty straight-forward to convert a sklearn PCA transformer to plain C: you use the magic method port. In addition to converting SVM/RVM classifiers, it is now able to export PCA too.
from sklearn.decomposition import PCA\nfrom sklearn.datasets import load_iris\nfrom micromlgen import port\n\nif __name__ == '__main__':\n X = load_iris().data\n pca = PCA(n_components=2, whiten=False).fit(X)\n\n print(port(pca))To use the exported code, we first have to include it in our sketch. Save the contents to a file (I named it pca.h) in the same folder of your .ino project and include it.
#include "pca.h"\n\n// this was trained on the IRIS dataset, with 2 principal components\nEloquent::ML::Port::PCA pca;The pca object is now able to take an array of size N as input and return an array of size K as output, with K < N usually.
void setup() {\n float x_input[4] = {5.1, 3.5, 1.4, 0.2};\n float x_output[2];\n\n pca.transform(x_input, x_output);\n}That's it: now you can run your classifier on x_output.
#include "pca.h"\n#include "svm.h"\n\nEloquent::ML::Port::PCA pca;\nEloquent::ML::Port::SVM clf;\n\nvoid setup() {\n float x_input[4] = {5.1, 3.5, 1.4, 0.2};\n float x_output[2];\n int y_pred;\n\n pca.transform(x_input, x_output);\n\n y_pred = clf.predict(x_output);\n}As I anticipated, let's take a look at how PCA dimensionality reduction can help in fitting classifiers that would otherwise be too large to fit on our microcontrollers.
\nThis is the exact table from the Gesture classification project.
\n| Kernel | \nC | \nGamma | \nDegree | \nVectors | \nFlash size | \nRAM (b) | \nAvg accuracy | \n
|---|---|---|---|---|---|---|---|
| RBF | \n10 | \n0.001 | \n- | \n37 | \n53 Kb | \n1228 | \n99% | \n
| Poly | \n100 | \n0.001 | \n2 | \n12 | \n25 Kb | \n1228 | \n99% | \n
| Poly | \n100 | \n0.001 | \n3 | \n25 | \n40 Kb | \n1228 | \n97% | \n
| Linear | \n50 | \n- | \n1 | \n40 | \n55 Kb | \n1228 | \n95% | \n
| RBF | \n100 | \n0.01 | \n- | \n61 | \n80 Kb | \n1228 | \n95% | \n
The dataset has 90 features (30 samples x 3 axes) and achieves 99% accuracy.
\nLet's pick the poly kernel with degree 2 and see how much we can decrease the number of components while still achieving a good accuracy.
| PCA components | \nAccuracy | \nSupport vectors | \n
|---|---|---|
| 90 | \n99% | \n31 | \n
| 50 | \n99% | \n31 | \n
| 40 | \n99% | \n31 | \n
| 30 | \n90% | \n30 | \n
| 20 | \n90% | \n28 | \n
| 15 | \n90% | \n24 | \n
| 10 | \n99% | \n18 | \n
| 5 | \n76% | \n28 | \n
We clearly see a couple of things:
\nWhat do these numbers mean to you? It means your board has to do many less computations to give you a prediction and will probably be able to host a more complex model.
\nLet's check out the figures with n_components = 10 compared with the ones without PCA.
| Kernel | \nPCA support vectors | \nPCA flash size | \nAccuracy | \n
|---|---|---|---|
| RBF C=10 | \n46 (+24%) | \n32 Kb (-40%) | \n99% | \n
| RBF C=100 | \n28 (-54%) | \n32 Kb (-60%) | \n99% | \n
| Poly 2 | \n13 (-48%) | \n28 Kb (+12%) | \n99% | \n
| Poly 3 | \n24 (-4%) | \n32 Kb (-20%) | \n99% | \n
| Linear | \n18 (-64%) | \n29 Kb (-47%) | \n99% | \n
A couple notes:
\nI will probably spend some more time investingating the usefulness of PCA for Arduino Machine Learning projects, but for now that's it: it's a good starting point in my opinion.
\nThere's a little example sketch on Github that applies PCA to the IRIS dataset.
\nTell me what you think may be a clever application of dimensionality reduction in the world of microcontrollers and see if we can build something great together.
\nL'articolo Arduino dimensionality reduction (PCA) for Machine Learning projects proviene da Eloquent Arduino Blog.
\n", "content_text": "When working with Machine Learning projects on microcontrollers and embedded devices the dimension of features can become a limiting factor due to the lack of RAM: dimensionality reduction (eg. PCA) will help you shrink your models and even achieve higher prediction accuracy.\n\n\nWhy dimensionality reduction on Arduino microcontrollers?\nDimensionality reduction is a tecnique you see often in Machine Learning projects. By stripping away "unimportant" or redundant information, it generally helps speeding up the training process and achieving higher classification performances.\nSince we now know we can run Machine Learning on Arduino boards and embedded microcontrollers, it can become a key tool at our disposal to squeeze out the most out of our boards.\nIn the specific case of resource-constrained devices as old Arduino boards (the UNO for example, with only 2 kb of RAM), it can become a decisive turn in unlocking even more application scenarios where the high dimensionality of the input features would not allow any model to fit.\nLet's take the Gesture classification project as an example: among the different classifiers we trained, only one fitted on the Arduino UNO, since most of them required too much flash memory due to the high dimension of features (90) and support vectors (25 to 61).\nIn this post I will resume that example and see if dimensionality reduction can help reduce this gap.\nIf you are working on a project with many features, let me know in the comments so I can create a detailed list of real world examples.\nHow to export PCA (Principal Component Analysis) to plain C\nAmong the many algorithms available for dimensionality reduction, I decided to start with PCA (Principal Component Analysis) because it's one of the most widespread. In the next weeks I will probably work on porting other alternatives.\nIf you never used my Python package micromlgen I first invite you to read the introduction post to get familiar with it.\nAlways remember to install the latest version, since I publish frequent updates.\npip install --upgrade micromlgen\nNow it is pretty straight-forward to convert a sklearn PCA transformer to plain C: you use the magic method port. In addition to converting SVM/RVM classifiers, it is now able to export PCA too.\nfrom sklearn.decomposition import PCA\nfrom sklearn.datasets import load_iris\nfrom micromlgen import port\n\nif __name__ == '__main__':\n X = load_iris().data\n pca = PCA(n_components=2, whiten=False).fit(X)\n\n print(port(pca))\nHow to deploy PCA to Arduino\nTo use the exported code, we first have to include it in our sketch. Save the contents to a file (I named it pca.h) in the same folder of your .ino project and include it.\n#include "pca.h"\n\n// this was trained on the IRIS dataset, with 2 principal components\nEloquent::ML::Port::PCA pca;\nThe pca object is now able to take an array of size N as input and return an array of size K as output, with K < N usually.\nvoid setup() {\n float x_input[4] = {5.1, 3.5, 1.4, 0.2};\n float x_output[2];\n\n pca.transform(x_input, x_output);\n}\nThat's it: now you can run your classifier on x_output.\n#include "pca.h"\n#include "svm.h"\n\nEloquent::ML::Port::PCA pca;\nEloquent::ML::Port::SVM clf;\n\nvoid setup() {\n float x_input[4] = {5.1, 3.5, 1.4, 0.2};\n float x_output[2];\n int y_pred;\n\n pca.transform(x_input, x_output);\n\n y_pred = clf.predict(x_output);\n}\n\r\n\r\n\r\n \r\n\tFinding this content useful?\r\n\r\n\t\r\n\r\n\t\r\n\t\t\r\n\t\t\r\n\t \r\n \r\n \r\n \r\n\r\n\r\n\r\n\nA real world example\nAs I anticipated, let's take a look at how PCA dimensionality reduction can help in fitting classifiers that would otherwise be too large to fit on our microcontrollers.\nThis is the exact table from the Gesture classification project.\n\n\n\nKernel\nC\nGamma\nDegree\nVectors\nFlash size\nRAM (b)\nAvg accuracy\n\n\n\n\nRBF\n10\n0.001\n-\n37\n53 Kb\n1228\n99%\n\n\nPoly\n100\n0.001\n2\n12\n25 Kb\n1228\n99%\n\n\nPoly\n100\n0.001\n3\n25\n40 Kb\n1228\n97%\n\n\nLinear\n50\n-\n1\n40\n55 Kb\n1228\n95%\n\n\nRBF\n100\n0.01\n-\n61\n80 Kb\n1228\n95%\n\n\n\nThe dataset has 90 features (30 samples x 3 axes) and achieves 99% accuracy. \nLet's pick the poly kernel with degree 2 and see how much we can decrease the number of components while still achieving a good accuracy.\n\n\n\nPCA components\nAccuracy\nSupport vectors\n\n\n\n\n90\n99%\n31\n\n\n50\n99%\n31\n\n\n40\n99%\n31\n\n\n30\n90%\n30\n\n\n20\n90%\n28\n\n\n15\n90%\n24\n\n\n10\n99%\n18\n\n\n5\n76%\n28\n\n\n\nWe clearly see a couple of things:\n\nwe still achieve 99% accuracy even with only 40 out of 90 principal components\nwe get a satisfactory 90% accuracy even with only 15 components\n(this is a bit unexpected) it looks like there's a sweet spot at 10 components where the accuracy skyrockets to 99% again. This could be just a contingency of this particular dataset, don't expect to replicate this results on your own dataset\n\nWhat do these numbers mean to you? It means your board has to do many less computations to give you a prediction and will probably be able to host a more complex model.\nLet's check out the figures with n_components = 10 compared with the ones without PCA.\n\n\n\nKernel\nPCA support vectors\nPCA flash size\nAccuracy\n\n\n\n\nRBF C=10\n46 (+24%)\n32 Kb (-40%)\n99%\n\n\nRBF C=100\n28 (-54%)\n32 Kb (-60%)\n99%\n\n\nPoly 2\n13 (-48%)\n28 Kb (+12%)\n99%\n\n\nPoly 3\n24 (-4%)\n32 Kb (-20%)\n99%\n\n\nLinear\n18 (-64%)\n29 Kb (-47%)\n99%\n\n\n\nA couple notes:\n\naccuracy increased (on stayed the same) for all kernels\nwith one exception, flash size decreased in the range 20 - 50%\nnow we can fit 3 classifiers on our Arduino UNO instead of only one\n\nI will probably spend some more time investingating the usefulness of PCA for Arduino Machine Learning projects, but for now that's it: it's a good starting point in my opinion.\n\nThere's a little example sketch on Github that applies PCA to the IRIS dataset.\nTell me what you think may be a clever application of dimensionality reduction in the world of microcontrollers and see if we can build something great together.\nL'articolo Arduino dimensionality reduction (PCA) for Machine Learning projects proviene da Eloquent Arduino Blog.", "date_published": "2020-06-07T09:24:20+02:00", "date_modified": "2020-06-07T11:26:25+02:00", "authors": [ { "name": "simone", "url": "https://eloquentarduino.github.io/author/simone/", "avatar": "http://1.gravatar.com/avatar/d670eb91ca3b1135f213ffad83cb8de4?s=512&d=mm&r=g" } ], "author": { "name": "simone", "url": "https://eloquentarduino.github.io/author/simone/", "avatar": "http://1.gravatar.com/avatar/d670eb91ca3b1135f213ffad83cb8de4?s=512&d=mm&r=g" }, "tags": [ "microml", "pca", "Arduino Machine learning" ] } ] }