Use Cases
Docs
Blog Articles
BlogResources
How to send data reliably from IOT edge devices using Fluvio Mirroring
Contributor, InfinyOn
How to send data reliably from IOT edge devices using Fluvio Mirroring
Introduction
Edge, sensors, and the connected IoT ecosystem typically operate with limited storage, memory, and compute. To operate reliably, we assume perfect network performance. But in reality, network performance issues and disruptions cause data loss. All modern sensors operating with protocols like LoRaWAN require a robust edge-to-core event streaming infrastructure.
With this blog post, you will learn how to leverage mirroring to optimize your IoT infrastructure and stay ahead in the fast-paced world of connected devices.
What is Mirroring?
Mirroring is a new Fluvio feature which enables connecting multiple devices running Fluvio with a home-remote relationship.
Home and Remote
To understand mirroring, we also need to understand what is a Home and a Remote cluster:
- Home cluster is the target cluster that will manage, receive and consume data.
- Remote cluster is the source cluster that will send the data.
For this guide we’ll use the InfinyOn Cloud as the Home cluster, also an Orange Pi Zero 3, a tiny ARM device running Linux as the Remote cluster, but you can use any device compatible with Fluvio instead.
This flowchart below is a demonstration of how this guide should use the Orange Pi and the InfinyOn Cloud to sync its data:
Benefits of using Fluvio Mirroring
- Reliable synchronization:
- If connected, the remote sends telemetry and events to the home in real-time using mirroring.
- If disconnected, the Remote stream processor caches events locally. When the connection resumes, the Remote stream processor brings the Home Cluster up to date and continues mirroring until the subsequent connection loss.
- Intelligent processing with InfinyOn SmartModules.
- Hierarchical processing, where you decide where to apply the transformations.
- Built-in connectors to push events to databases and other core products.
- Runs on tiny devices.
- It just requires the Fluvio simple installation.
- Easy to set up.
Temperature track guide:
In this guide, we will set up an IoT device that sends its temperature continuously, as an example of what you can build and integrate with Fluvio and IoT devices.
Create an InfinyOn Cloud Account
If you don’t have an InfinyOn Cloud Account, using your web browser, navigate to https://infinyon.cloud/signup, and create one.
After the account is created, you will be placed on the Dashboard. You may choose to create a cluster in the GUI. In this guide, we’ll create a cluster using the CLI.
Install fluvio binary
Use curl to download and install:
curl -fsS https://hub.infinyon.cloud/install/install.sh | bash
Make sure to add .fluvio/bin to the $PATHas specified in the installation script.
Login to InfinyOn Cloud
Login to InfinyOn Cloud:
fluvio cloud login --use-oauth2
Leave out --use-oauth2 if you prefer the username/password method.
Create Cloud Cluster
Let’s provision a new cloud cluster:
fluvio cloud cluster create
Check the result with:
fluvio cluster status
Register Remote cluster
Let’s register the edge cluster orangepi to inform our home cluster from the cloud to accept connection requests from this remote device:
fluvio remote register orangepi
Create mirror topic
Each edge cluster mirror connects to a partition of a topic, where each partition has a 1-to-1 relationship with the remote cluster.
Create a partition assignment file with an array of remotes that we expect to connect to this cluster:
echo '["orangepi"]' > devices.json
fluvio topic create mirror-topic --mirror-apply devices.json
or just create the mirror topic, then add the remote:
fluvio topic create mirror-topic --mirror
fluvio topic add-mirror mirror-topic orangepi
Generate metadata for the Remote cluster:
Each Remote cluster requires a unique metadata file that informs how to connect with the home cluster. Create the config file by running the following command:
fluvio cloud remote export orangepi --file orangepi.json
The Cloud cluster configuration is now complete. Next, we’ll create an edge cluster that synchronizes data to the Cloud.
Install Remote cluster on Orange Pi
We’ll start a remote cluster on our tiny computer. But first, we also need Fluvio installed here:
curl -fsS https://hub.infinyon.cloud/install/install.sh | bash
Again, make sure to add .fluvio/bin to the $PATHas specified in the installation script.
Start Remote cluster
First we will start the cluster:
fluvio cluster start
Then, we’ll connect to the Home with the metadata orangepi:
fluvio home connect --file orangepi.json
Let’s check the partitions:
fluvio partition list
The remote device should show the following partition:
TOPIC PARTITION LEADER MIRROR REPLICAS RESOLUTION SIZE HW LEO LRS FOLLOWER OFFSETS
mirror-topic 0 5001 7e7fcb8b-6e47-4eef-a207-3199259f3771:0:router.infinyon.cloud:9005 [] Online 0 B 0 0 0 0 []
Produce data with the script
In this simple example, we’ll produce data using this simple bash script to get the temperature of the Orange Pi CPU and produce it on the mirror topic once per second:
#!/bin/bash
while true; do
temp=$(cat /sys/class/thermal/thermal_zone0/temp)
temp_c=$(echo "scale=1; $temp / 1000" | bc)
echo "CPU Temperature: $temp_c°C"
echo $temp_c | fluvio produce mirror-topic
sleep 1
done
Copy this script to a file, give it permission and run it:
sudo chmod +x ./temp_producer.sh
./temp_producer.sh
CPU Temperature: 44.9°C
CPU Temperature: 45.0°C
CPU Temperature: 44.6°C
CPU Temperature: 44.7°C
CPU Temperature: 44.4°C
CPU Temperature: 44.9°C
CPU Temperature: 45.0°C
CPU Temperature: 45.2°C
CPU Temperature: 45.0°C
...
Consume it!
Consume it on any device connected to the InfinyOn Cloud:
fluvio consume mirror-topic -B
44.9
45.0
44.6
44.7
44.4
44.9
45.0
45.2
45.0
...
Conclusion
🎉 Congratulations! You now know the basics of how to set up IoT infrastructure using Mirroring.
Join us on Discord if you have questions or would like to suggest new improvements.