Chapter 7 - Integrating Teachable machine with physical hardware

Freeing Teachable Machine from the confine of your browser:

In this chapter, we are going use the trained model exported from Teachable Machine to control our C3 CoreModule via the WebSocket server.

Load the ESP32-C3 firmware from Chapter 3. We will be using the WebSocket server to control the LED on the CoreModule.

Add in additional payload comparison under
case WStype_TEXT:
The example code only does matching for "toggle". We could add in for instance "LED_on" & "LED_off".
Create a new TeachableMachine "Image project" with the recognition class names "LED_on" & "LED_off" (or you can reuse your Test Project from the previous chapter, make sure you rename your classes name accordingly).
For this example to work properly, we need to add an additional recognition class name "background", as a control (to detect when there is no object in front of the camera).
Create a new p5.js sketch which incorporate both WebSocket & ml5.js to gain the ability to access the pre-trained Tensorflow models generated by TeachableMachine.

Additionally you will need to add in an extra line of Code in the index.html within your p5.js project to include the javascript library for parsing the tensorflow.js trained model.

<script type="text/javascript" src="https://unpkg.com/ml5@0.6.1/dist/ml5.min.js"></script>

Project Name

Source Code

/* 
 *  CB05 - p5.js WebSocket LED with Teachable Machine 
 *         Tensorflow.js model support.
 *  Creating a fancy interactive user interface 
 *  capable of communicating via WebSocket in p5.js
 *  
 *  
 */

const modelURL = 'https://teachablemachine.withgoogle.com/models/XXXX-XXXX/';

// where the cockroach SocketServer is (your local machine):
var host = '192.168.0.XXX:81';  // Change this to your CockroachBot IP Address 
                    
var socket; // the websocket
var classifier;
var video;
var flippedVideo;
var label;
var previousLabel;
var ledValue;

function preload() {
    classifier = ml5.imageClassifier(modelURL + 'model.json');
}

function setup() {
    createCanvas(320, 260);
  
    // connect to server:
    socket = new WebSocket('ws://' + host);
    // socket connection listener:
    socket.onopen = sendIntro;
    // socket message listener:
    socket.onmessage = readMessage;
  
    video = createCapture(VIDEO);
    video.size(320, 240);
    video.hide();
    flippedVideo = ml5.flipImage(video);
    classifyVideo();
}

function draw() {
    background(0);
    image(flippedVideo, 0, 0);
    fill(255);
    textSize(16);
    textAlign(CENTER);
    text(label, width / 2, height - 4);
}

function classifyVideo() {
    flippedVideo = ml5.flipImage(video)
    classifier.classify(flippedVideo, gotResult);
    flippedVideo.remove();
}

function sendIntro() {
  // convert the message object to a string and send it:
  socket.send("Hello");
}

function readMessage(event) {
  var msg = event.data; // read data from the onmessage event
  if (event.data == "LED=1"){
    ledValue = 1; 
  }
  else if (event.data == "LED=0"){
    ledValue = 0; 
  }
}

function gotResult(error, results) {
    if (error) {
        console.error(error);
        return;
    }
    label = String(results[0].label);
    console.log(label);
  
    if ((label != "Background") && (previousLabel != label)) {
        socket.send(label);
    }
    previousLabel = label;
  
    classifyVideo();
}

Todo:

Instead of turning the LED on & off, is it possible to have a new recognition class to blink the LED instead? "LED_blink"?

Chapter 8 - CockroachBot with Vision/Voice recognition upgrade