Demonstrator: Machine vision for a magic marble machine

This magic marble demonstrator was proposed by CTRL-Engineering and created through several student projects: two student projects from electromechanical bachelor's in Vives and KU Leuven and a master thesis from the KU Leuven master in technological engineering electronics-ICT. The students from the bachelor projects created the mechanical setup while the master thesis consisted of the creation of the machine vision application.

The setup

This setup was created by four students from KU Leuven and Vives. The goal of the setup is to demonstrate real-time machine vision applications on a PLC in an industrial setup. The setup consists of three main parts. The first part is the elevator where marbles are lifted from the bottom of the setup to the top. A camera is looking at a row in this elevator whose image is used as input of a machine vision model which classifies the different marbles in the row by color. Solenoids are then used to shoot the incorrectly colored marbles out of the elevator. The real-time aspects of the machine learning model can be demonstrated by changing the distance between the camera and the solenoids to be shorter or by making the elevator faster. The second part of the machine consists of a track which displays the marbles which were let through by the solenoids. The last main part of the magic marble machine consists of a marble collector which combines the previous two parts at the bottom guiding the marbles from the track back to the elevator. All electronics, including the camera, motors and solenoids are driven by a PLC underneath the setup where the machine learning model is also located. Because this setup is a demonstrator, it is easily transportable with a car and is not too difficult to set up and take down.

Figure 1: The full magic marble machine setup.

Machine vision application

The machine vision part of this demo setup was done through a master thesis at KU Leuven. The main research question of this thesis was how to implement a machine vision application on a PLC so that it can run in the real-time machine cycle. The goal of the machine vision application itself is to classify the marble images based on color. The thesis student evaluated several machine learning models and evaluated their timing, accuracy and memory usage. Four types of machine learning models were evaluated of which three were supervised models and one was unsupervised. The chosen models were support vector machines (SVM), K-Means, multi-layer perceptron (MLP) and the convolutional neural network (CNN). The first three machine learning models were trained by the student itself and implemented on the PLC using ONNX and later the Beckhoff XML model format. The CNN was created with a beta machine vision creation application from Beckhoff which also produced a model in XML format. After the evaluation of the different models on the PLC, the color sorting mechanism was implemented and a visualizer was made to demonstrate and compare the results of the different machine learning models. Later, the functionality for creating the images, including driving the solenoids and deciding which marbles to shoot based on an image matrix, was implemented as well.

Result and future work

The resulting setup is now ready to be used as a demonstrator although there are still further improvements to be made for optimal usage. The demonstrator will be extended in the future to showcase MLOps techniques like drift detection and model adaptation. The first steps for this have already been taken in the LSDD use case where the dataset created from this setup is used to evaluate a drift detection algorithm. Next, the goal will be to demonstrate a full MLOps lifecycle on this setup, which includes the usage of common MLOps tools to create new machine learning models from data collected from the setup.