{ "version": "https://jsonfeed.org/version/1.1", "user_comment": "This feed allows you to read the posts from this site in any feed reader that supports the JSON Feed format. To add this feed to your reader, copy the following URL -- https://eloquentarduino.github.io/tag/rvm/feed/json/ -- and add it your reader.", "home_page_url": "https://eloquentarduino.github.io/tag/rvm/", "feed_url": "https://eloquentarduino.github.io/tag/rvm/feed/json/", "language": "en-US", "title": "rvm – Eloquent Arduino Blog", "description": "Machine learning on Arduino, programming & electronics", "items": [ { "id": "https://eloquentarduino.github.io/?p=893", "url": "https://eloquentarduino.github.io/2020/02/even-smaller-machine-learning-models-for-your-mcu/", "title": "Even smaller Machine learning models for your MCU: up to -82% code size", "content_html": "

So far we've used SVM (Support Vector Machine) as our main classifier to port a Machine learning model to a microcontroller: but recently I found an interesting alternative which could be waaaay smaller, mantaining a similar accuracy.

\n

\"RVM

\n

\n

Table of contents
  1. The current state
  2. A new algorithm: Relevance Vector Machines
  3. Training a classifier
  4. Porting to C
  5. Performace comparison
  6. Size comparison
  7. Disclaimer

\n

The current state

\n

I chose SVM as my main focus of intereset for the MicroML framework because I knew the support vector encoding could be very memory efficient once ported to plain C. And it really is.

\n

I was able to port many real-world models (gesture identification, wake word detection) to tiny microcontrollers like the old Arduino Nano (32 kb flash, 2 kb RAM).

\n

The tradeoff of my implementation was to sacrifice the flash space (which is usually quite big) to save as much RAM as possible, which is usually the most limiting factor.

\n

Due to this implementation, if your model grows in size (highly dimensional data or not well separable data), the generated code will still fit in the RAM, but "overflow" the available flash.

\n

In a couple of my previous post I warned that model selection might be a required step before being able to deploy a model to a MCU, since you should first check if it fits. If not, you must train another model hoping to get fewer support vectors, since each of them contributes to the code size increase.

\n

A new algorithm: Relevance Vector Machines

\n

It was by chance that I came across a new algorithm that I never heard of, called Relevance Vector Machine. It was patented by Microsoft until last year (so maybe this is the reason you don't see it in the wild), but now it is free of use as far as I can tell.

\n

Here is the link to the paper if you want to read it, it gives some insights into the development process.

\n

I'm not a mathematician, so I can't describe it accurately, but in a few words it uses the same formulation of SVM (a weightened sum of kernels), applying a Bayesan model.

\n

This serves in the first place to be able to get the probabilities of the classification results, which is something totally missing in SVM.

\n

In the second place, the algorithm tries to learn a much more sparse representation of the support vectors, as you can see in the following picture.

\n

\"RVM

\n

When I first read the paper my first tought was just "wow"! This is exactly what I need for my MicroML framework: a ultra-lightweight model which can still achieve high accuracy.

\n

Training a classifier

\n

Now that I knew this algorithm, I searched for it in the sklearn documentation: it was not there.

\n

It seems that, since it was patented, they didn't have an implementation.

\n

Fortunately, there is an implementation which follows the sklearn paradigm. You have to install it:

\n
pip install Cython\npip install https://github.com/AmazaspShumik/sklearn_bayes/archive/master.zip
\n

Since the interface is the usual fit predict, it is super easy to train a classifier.

\n
from sklearn.datasets import load_iris\nfrom skbayes.rvm_ard_models import RVC\nimport warnings\n\n# I get tons of boring warnings during training, so turn it off\nwarnings.filterwarnings("ignore")\n\niris = load_iris()\nX = iris.data\ny = iris.target\nclf = RVC(kernel='rbf', gamma=0.001)\nclf.fit(X, y)\ny_predict = clf.predict(X)
\n

The parameters for the constructor are similar to those of the SVC classifier from sklearn:

\n\n

You can read the docs from sklearn to learn more.

\n

Porting to C

\n

Now that we have a trained classifier, we have to port it to plain C that compiles on our microcontroller of choice.

\n

I patched my package micromlgen to do the job for you, so you should install the latest version to get it working.

\n
 pip install --upgrade micromlgen
\n

Now the export part is almost the same as with an SVM classifier.

\n
 from micromlgen import port_rvm\n\n clf = get_rvm_classifier()\n c_code = port_rvm(clf)\n print(c_code)
\n

And you're done: you have plain C code you can embed in any microcontroller.

\n

Performace comparison

\n

To test the effectiveness of this new algorithm, I applied it to the datasets I built in my previous posts, comparing side by side the size and accuracy of both SVM and RVM.

\n

The results are summarized in the next table.

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
DatasetSVMRVMDelta
Flash(byte)Acc. (%)Flash(byte)Acc. (%)FlashAcc.
RGB colors45841003580100-22%-0%
Accelerometer gestures(linear kernel)3688892705685-80%-7%
Accelerometer gestures(gaussian kernel)4534895776695-82%-0%
Wifi positioning46411003534100-24%-0%
Wake word(linear kernel)1809886360253-80%-33%
Wake word(gaussian kernel)2178890482662-78%-28%
\n

** the accuracy reported are with default parameters, without any tuning, averaged in 30 runs

\n

As you may see, the results are quite surpising:

\n\n

As in any situation, you should test which one of the two algorithms works best for your use case, but there a couple of guidelines you may follow:

\n\n

Size comparison

\n

As a reference, here is the codes generated for an SVM classifier and an RVM one to classify the IRIS dataset.

\n
uint8_t predict_rvm(double *x) {\n    double decision[3] = { 0 };\n    decision[0] = -0.6190847299428206;\n    decision[1] = (compute_kernel(x,  6.3, 3.3, 6.0, 2.5) - 72.33233 ) * 0.228214 + -2.3609625;\n    decision[2] = (compute_kernel(x,  7.7, 2.8, 6.7, 2.0) - 81.0089166 ) * -0.29006 + -3.360963;\n    uint8_t idx = 0;\n    double val = decision[0];\n    for (uint8_t i = 1; i < 3; i++) {\n        if (decision[i] > val) {\n            idx = i;\n            val = decision[i];\n        }\n    }\n    return idx;\n}\n\nint predict_svm(double *x) {\n    double kernels[10] = { 0 };\n    double decisions[3] = { 0 };\n    int votes[3] = { 0 };\n        kernels[0] = compute_kernel(x,   6.7  , 3.0  , 5.0  , 1.7 );\n        kernels[1] = compute_kernel(x,   6.0  , 2.7  , 5.1  , 1.6 );\n        kernels[2] = compute_kernel(x,   5.1  , 2.5  , 3.0  , 1.1 );\n        kernels[3] = compute_kernel(x,   6.0  , 3.0  , 4.8  , 1.8 );\n        kernels[4] = compute_kernel(x,   7.2  , 3.0  , 5.8  , 1.6 );\n        kernels[5] = compute_kernel(x,   4.9  , 2.5  , 4.5  , 1.7 );\n        kernels[6] = compute_kernel(x,   6.2  , 2.8  , 4.8  , 1.8 );\n        kernels[7] = compute_kernel(x,   6.0  , 2.2  , 5.0  , 1.5 );\n        kernels[8] = compute_kernel(x,   4.8  , 3.4  , 1.9  , 0.2 );\n        kernels[9] = compute_kernel(x,   5.1  , 3.3  , 1.7  , 0.5 );\n        decisions[0] = 20.276395502\n                    + kernels[0] * 100.0\n                    + kernels[1] * 100.0\n                    + kernels[3] * -79.351629954\n                    + kernels[4] * -49.298850195\n                    + kernels[6] * -40.585178082\n                    + kernels[7] * -30.764341769\n        ;\n        decisions[1] = -0.903345464\n                    + kernels[2] * 0.743494115\n                    + kernels[9] * -0.743494115\n        ;\n        decisions[2] = -1.507856504\n                    + kernels[5] * 0.203695177\n                    + kernels[8] * -0.160020702\n                    + kernels[9] * -0.043674475\n        ;\n        votes[decisions[0] > 0 ? 0 : 1] += 1;\n        votes[decisions[1] > 0 ? 0 : 2] += 1;\n        votes[decisions[2] > 0 ? 1 : 2] += 1;\n                int classVal = -1;\n        int classIdx = -1;\n        for (int i = 0; i < 3; i++) {\n            if (votes[i] > classVal) {\n                classVal = votes[i];\n                classIdx = i;\n            }\n        }\n        return classIdx;\n}
\n

As you can see, RVM actually only computes 2 kernels and does 2 multiplications. SVM, on the other hand, computes 10 kernels and does 13 multiplications.

\n

This is a recurring pattern, so RVM is much much faster in the inference process.

\n

Disclaimer

\n

micromlgen and in particular port_rvm are work in progress: you may experience some glitches or it may not work in your specific case. Please report any issue on the Github repo.

\n

L'articolo Even smaller Machine learning models for your MCU: up to -82% code size proviene da Eloquent Arduino Blog.

\n", "content_text": "So far we've used SVM (Support Vector Machine) as our main classifier to port a Machine learning model to a microcontroller: but recently I found an interesting alternative which could be waaaay smaller, mantaining a similar accuracy.\n\n\nTable of contentsThe current stateA new algorithm: Relevance Vector MachinesTraining a classifierPorting to CPerformace comparisonSize comparisonDisclaimer\nThe current state\nI chose SVM as my main focus of intereset for the MicroML framework because I knew the support vector encoding could be very memory efficient once ported to plain C. And it really is.\nI was able to port many real-world models (gesture identification, wake word detection) to tiny microcontrollers like the old Arduino Nano (32 kb flash, 2 kb RAM).\nThe tradeoff of my implementation was to sacrifice the flash space (which is usually quite big) to save as much RAM as possible, which is usually the most limiting factor.\nDue to this implementation, if your model grows in size (highly dimensional data or not well separable data), the generated code will still fit in the RAM, but "overflow" the available flash.\nIn a couple of my previous post I warned that model selection might be a required step before being able to deploy a model to a MCU, since you should first check if it fits. If not, you must train another model hoping to get fewer support vectors, since each of them contributes to the code size increase.\nA new algorithm: Relevance Vector Machines\nIt was by chance that I came across a new algorithm that I never heard of, called Relevance Vector Machine. It was patented by Microsoft until last year (so maybe this is the reason you don't see it in the wild), but now it is free of use as far as I can tell.\nHere is the link to the paper if you want to read it, it gives some insights into the development process.\nI'm not a mathematician, so I can't describe it accurately, but in a few words it uses the same formulation of SVM (a weightened sum of kernels), applying a Bayesan model.\nThis serves in the first place to be able to get the probabilities of the classification results, which is something totally missing in SVM.\nIn the second place, the algorithm tries to learn a much more sparse representation of the support vectors, as you can see in the following picture.\n\nWhen I first read the paper my first tought was just "wow"! This is exactly what I need for my MicroML framework: a ultra-lightweight model which can still achieve high accuracy.\nTraining a classifier\nNow that I knew this algorithm, I searched for it in the sklearn documentation: it was not there.\nIt seems that, since it was patented, they didn't have an implementation.\nFortunately, there is an implementation which follows the sklearn paradigm. You have to install it:\npip install Cython\npip install https://github.com/AmazaspShumik/sklearn_bayes/archive/master.zip\nSince the interface is the usual fit predict, it is super easy to train a classifier.\nfrom sklearn.datasets import load_iris\nfrom skbayes.rvm_ard_models import RVC\nimport warnings\n\n# I get tons of boring warnings during training, so turn it off\nwarnings.filterwarnings("ignore")\n\niris = load_iris()\nX = iris.data\ny = iris.target\nclf = RVC(kernel='rbf', gamma=0.001)\nclf.fit(X, y)\ny_predict = clf.predict(X)\nThe parameters for the constructor are similar to those of the SVC classifier from sklearn:\n\nkernel: one of linear, poly, rbf\ndegree: if kernel=poly\ngamma: if kernel=poly or kernel=rbf\n\nYou can read the docs from sklearn to learn more.\nPorting to C\nNow that we have a trained classifier, we have to port it to plain C that compiles on our microcontroller of choice.\nI patched my package micromlgen to do the job for you, so you should install the latest version to get it working.\n pip install --upgrade micromlgen\nNow the export part is almost the same as with an SVM classifier.\n from micromlgen import port_rvm\n\n clf = get_rvm_classifier()\n c_code = port_rvm(clf)\n print(c_code)\nAnd you're done: you have plain C code you can embed in any microcontroller.\nPerformace comparison\nTo test the effectiveness of this new algorithm, I applied it to the datasets I built in my previous posts, comparing side by side the size and accuracy of both SVM and RVM.\nThe results are summarized in the next table.\n\n\n\n\nDataset\nSVM\nRVM\nDelta\n\n\n\nFlash(byte)\nAcc. (%)\nFlash(byte)\nAcc. (%)\nFlash\nAcc.\n\n\n\n\nRGB colors\n4584\n100\n3580\n100\n-22%\n-0%\n\n\nAccelerometer gestures(linear kernel)\n36888\n92\n7056\n85\n-80%\n-7%\n\n\nAccelerometer gestures(gaussian kernel)\n45348\n95\n7766\n95\n-82%\n-0%\n\n\nWifi positioning\n4641\n100\n3534\n100\n-24%\n-0%\n\n\nWake word(linear kernel)\n18098\n86\n3602\n53\n-80%\n-33%\n\n\nWake word(gaussian kernel)\n21788\n90\n4826\n62\n-78%\n-28%\n\n\n\n** the accuracy reported are with default parameters, without any tuning, averaged in 30 runs\nAs you may see, the results are quite surpising:\n\nyou can achieve up to 82% space reduction on highly dimensional dataset without any loss in accuracy (accelerometer gestures with gaussian kernel)\nsometimes you may not be able to achieve a decent accuracy (62% at most on the wake word dataset)\n\nAs in any situation, you should test which one of the two algorithms works best for your use case, but there a couple of guidelines you may follow:\n\nif you need top accuracy, probably SVM can achieve slighter better performance if you have enough space\nif you need tiny space or top speed, test if RVM achieves a satisfiable accuracy\nif both SVM and RVM achieve comparable performace, go with RVM: it's much lighter than SVM in most cases and will run faster\n\nSize comparison\nAs a reference, here is the codes generated for an SVM classifier and an RVM one to classify the IRIS dataset.\nuint8_t predict_rvm(double *x) {\n double decision[3] = { 0 };\n decision[0] = -0.6190847299428206;\n decision[1] = (compute_kernel(x, 6.3, 3.3, 6.0, 2.5) - 72.33233 ) * 0.228214 + -2.3609625;\n decision[2] = (compute_kernel(x, 7.7, 2.8, 6.7, 2.0) - 81.0089166 ) * -0.29006 + -3.360963;\n uint8_t idx = 0;\n double val = decision[0];\n for (uint8_t i = 1; i < 3; i++) {\n if (decision[i] > val) {\n idx = i;\n val = decision[i];\n }\n }\n return idx;\n}\n\nint predict_svm(double *x) {\n double kernels[10] = { 0 };\n double decisions[3] = { 0 };\n int votes[3] = { 0 };\n kernels[0] = compute_kernel(x, 6.7 , 3.0 , 5.0 , 1.7 );\n kernels[1] = compute_kernel(x, 6.0 , 2.7 , 5.1 , 1.6 );\n kernels[2] = compute_kernel(x, 5.1 , 2.5 , 3.0 , 1.1 );\n kernels[3] = compute_kernel(x, 6.0 , 3.0 , 4.8 , 1.8 );\n kernels[4] = compute_kernel(x, 7.2 , 3.0 , 5.8 , 1.6 );\n kernels[5] = compute_kernel(x, 4.9 , 2.5 , 4.5 , 1.7 );\n kernels[6] = compute_kernel(x, 6.2 , 2.8 , 4.8 , 1.8 );\n kernels[7] = compute_kernel(x, 6.0 , 2.2 , 5.0 , 1.5 );\n kernels[8] = compute_kernel(x, 4.8 , 3.4 , 1.9 , 0.2 );\n kernels[9] = compute_kernel(x, 5.1 , 3.3 , 1.7 , 0.5 );\n decisions[0] = 20.276395502\n + kernels[0] * 100.0\n + kernels[1] * 100.0\n + kernels[3] * -79.351629954\n + kernels[4] * -49.298850195\n + kernels[6] * -40.585178082\n + kernels[7] * -30.764341769\n ;\n decisions[1] = -0.903345464\n + kernels[2] * 0.743494115\n + kernels[9] * -0.743494115\n ;\n decisions[2] = -1.507856504\n + kernels[5] * 0.203695177\n + kernels[8] * -0.160020702\n + kernels[9] * -0.043674475\n ;\n votes[decisions[0] > 0 ? 0 : 1] += 1;\n votes[decisions[1] > 0 ? 0 : 2] += 1;\n votes[decisions[2] > 0 ? 1 : 2] += 1;\n int classVal = -1;\n int classIdx = -1;\n for (int i = 0; i < 3; i++) {\n if (votes[i] > classVal) {\n classVal = votes[i];\n classIdx = i;\n }\n }\n return classIdx;\n}\nAs you can see, RVM actually only computes 2 kernels and does 2 multiplications. SVM, on the other hand, computes 10 kernels and does 13 multiplications.\nThis is a recurring pattern, so RVM is much much faster in the inference process.\nDisclaimer\nmicromlgen and in particular port_rvm are work in progress: you may experience some glitches or it may not work in your specific case. Please report any issue on the Github repo.\nL'articolo Even smaller Machine learning models for your MCU: up to -82% code size proviene da Eloquent Arduino Blog.", "date_published": "2020-02-15T17:37:32+01:00", "date_modified": "2020-05-31T18:50:57+02:00", "authors": [ { "name": "simone", "url": "https://eloquentarduino.github.io/author/simone/", "avatar": "http://1.gravatar.com/avatar/d670eb91ca3b1135f213ffad83cb8de4?s=512&d=mm&r=g" } ], "author": { "name": "simone", "url": "https://eloquentarduino.github.io/author/simone/", "avatar": "http://1.gravatar.com/avatar/d670eb91ca3b1135f213ffad83cb8de4?s=512&d=mm&r=g" }, "tags": [ "rvm", "svm", "Arduino Machine learning" ] } ] }