Stateful Data Flow Beta Build composable event-driven data pipelines in minutes.

Get Started Now

IoT Mirroring - Raspberry Pi to Local Cluster

Skill -🛠️ advanced

This advanced tutorial reuqires a Raspberry Pi and a local installation of your collector cluster running on Kubernetes. Checkout the basic versino at: “IoT Mirroring - Edge VM to Cloud”.

 

Raspberry Pi to Local Cluster

This section will use Raspberry Pi v3 running Ubuntu 32-bit as the edge device and our local machine for the target cluster. Let’s start with installing and configuring the target cluster.

 

Install Target Cluster on Local Machine

Installing the target cluster on Linux or Mac requires Kubernetes. Use the following instructions to set up Kubernetes on your local machine.

 

Create a new directory

Create a clean directory for the configuration and metadata files:

mkdir -p local/projects/mirror; cd local/projects/mirror
 

Download fluvio binary

Download and install mirroring binary.

Use curl to download and install:

curl -fsS https://hub.infinyon.cloud/install/install.sh | VERSION='0.10.15-dev-2+mirroring-9961bdb' bash

Make sure to add .fluvio/bin to the $PATHas specified in the installation script.

 

Start target cluster

Use the fluvio binary to start the cluster:

fluvio cluster start --local

Check the result with:

fluvio cluster status
 

Create the mirror topic

Mirror topics on the upstream clusters has multiple partitions, where each partition has a 1-to-1 relationship with the edge cluster.

Create a partition assignment file to define the edge devices:

echo '[
    "edge1", "edge2"
]' > assignment_file.json

Apply the configuration file to create the topic:

fluvio topic create edge-topic --mirror-assignment assignment_file.json

List partitions to check the assignment:

fluvio partition list

It should display all partitions:

  TOPIC       PARTITION  LEADER  MIRROR        REPLICAS  RESOLUTION  SIZE  HW  LEO  LRS  FOLLOWER OFFSETS     
  edge-topic  0          5001    edge1           []        Online      0 B   0   0    0    0                 [] 
  edge-topic  1          5001    edge2           []        Online      0 B   0   0    0    0                 [] 
 

Register Edge clusters

Use the remote-cluster CLI to register the edge clusters (edge1 and edge2) with the upstream cluster:

Edge 1:

fluvio cluster remote-cluster register --type mirror-edge edge1

Edge 2:

fluvio cluster remote-cluster register --type mirror-edge edge2

List remote clusters to check their status:

fluvio cluster remote-cluster list

It should show the following:

  RemoteCluster  RemoteType   Paired  Status  Last Seen 
  edge1          mirror-edge  -       -       -         
  edge2          mirror-edge  -       -       -         
 

Generate Metadata for Edge Clusters

Each edge cluster requires a unique metadata file that gives the edge cluster the information to connect to the upstream cluster and the topic/mirror where the data is synchronized.

Generate a metadata file for each cluster:

Edge 1:

The target edge device is a Virtual Machine emulating an IoT device:

fluvio cluster remote-cluster metadata export --topic edge-topic --mirror edge1 --upstream host.orb.internal --file edge1.json

Edge 2:

The target edge device is a Raspberry Pi device. You may skip this if you don’t have such a device.

fluvio cluster remote-cluster metadata export --topic edge-topic --mirror edge2 --upstream 192.168.79.252 --file edge2.json

We’ll transfer these files to edge devices in the following sections.

Install Edge Cluster on Raspberry Pi (optional)

We’ll use the same procedure as before to mirror from Raspberry Pi to the same upstream cluster. The test below was performed on a Raspberry Pi v3 running Ubuntu 32-bit image.

 

Download metadata file

We’ll use the metadata file edge2.json that we’ve exported above to provision this device.

Using the upstream terminal, let’s scp the edge2.json file to the edge device:

scp edge2.json [email protected]:~
 

Login into the edge device

Spawn a new terminal and login into the Raspberry Pi:

 

Download fluvio binaries

On the raspberry pi, run the following command:

curl -fsS https://hub.infinyon.cloud/install/install.sh | VERSION='0.10.15-dev-2+mirroring-9961bdb' bash

Run fluvio version to double check.

 

Start cluster

We’ll use the metadata file to start the edge cluster on the Raspberry Pi:

fluvio cluster start --read-only edge2.json

Let’s check the partitions:

fluvio partition list

The edge device should show the following partition::

  TOPIC       PARTITION  LEADER  MIRROR                REPLICAS  RESOLUTION  SIZE  HW  LEO  LRS  FOLLOWER OFFSETS     
  edge-topic  0          5001    Source:upstream:5001  []        Online      0 B   11  11   11   0                 [] 

 

Test 1: Mirror from Raspberry Pi Edge to Upstream

Let’s produce on the Raspberry Pi and consume from the upstream cluster.

 

Produce to edge cluster

Produce on the pi terminal:

fluvio produce edge-topic
> A
Ok!
> B
Ok!
 

Consume from upstream

Consume on the upstream terminal:

fluvio consume edge-topic --mirror edge2 -B 
A
B

Mirror test is successful.

 

Test2: Upstream Cluster Offline

Shutdown the upstream cluster and check that the edge cluster can continue receiving records. Then, resume the upstream cluster and ensure the data is synchronized and can consumed on both sides.

 

Shutdown upstream cluster

On the upstream terminal, shutdown the cluster:

fluvio cluster shutdown --local
kubectl delete spu custom-spu-5001

Ensure the cluster is shutdown:

 fluvio cluster status
 

On edge cluster

Produce a few more records on the pi terminal:

fluvio produce edge-topic
C
D
E
 

Reconnect upstream cluster & consume from topic

On the upstream terminal, restart the cluster:

fluvio cluster upgrade --local

The topic on the upstream cluster should automatically synchronize with the edge cluster.

Let’s consume:

fluvio-0.10.15 consume edge-topic --mirror edge2 -B
A
B
C
D
E

The disconnect test was successful.

 

Test 3: Edge Cluster Offline

This test ensures that the edge cluster will not lose data following a power loss.

 

Restart edge cluster

On the edge terminal, shutdown the cluster:

fluvio cluster shutdown --local

Restart the cluster:

fluvio cluster upgrade --read-only edge2.json 
 

Consume from edge cluster

On the pi terminal, consume from the cluster:

fluvio consume edge-topic -B
A
B
C
D
E

Produce records and observe that the mirror will resume the synchronization.

🎉 Congratulations! You have successfully tested edge mirroring using Rapsberry Pi. It is now time to roll it out in your environment.