Stateful Services (private release) Build composable event-driven data pipelines in minutes.

Request access

Install Fluvio Cluster in Docker

Installing Fluvio on Docker requires two containers, one for the Streaming Controller (SC) and the other for the Streaming Processing Unit (SPU).

 

Prerequisites

 
Docker & Docker Compose
  • We used Colima, but any Docker environment will do.
 
Fluvio CLI
  • Install fvm and follow the instructions:

    $ curl -fsS https://hub.infinyon.cloud/install/install.sh | bash
    

You are all set; let’s get started.

 

Start Fluvio Cluster

To start a fluvio cluster, we’ll need to perform the following steps:

  1. Start SC and SPU containers
  2. Add the cluster profile to Fluvio
  3. Authorize the SPU with the SC
 

Start SC and SPU Containers

Open a terminal, and add the following sub-directories:

$ mkdir metadata; mkdir data

The SC uses metadata to store the metadata configurations and the SPU data to persist the data records.

Create a file called docker-compose.yaml and paste the following configuration:

version: '3.8'
services:
  sc:
    image: infinyon/fluvio:latest
    hostname: sc
    container_name: sc
    volumes:
      - ./metadata:/fluvio/metadata
    environment:
      - RUST_LOG=info
    ports:
    - "9103:9003"
    command:
      - ./fluvio-run
      - sc
      - --local
      - /fluvio/metadata
  spu-5001:
    image: infinyon/fluvio:latest
    hostname: spu-5001
    container_name: spu-5001
    volumes:
      - ./data:/fluvio/data
    environment:
      - RUST_LOG=info
    ports:
    - "9110:9010"
    command:
      - ./fluvio-run
      - spu
      - -i
      - "5001"
      - -p
      - spu-5001:9010
      - -v
      - spu-5001:9011
      - --sc-addr
      - sc:9004
      - --log-base-dir
      - /fluvio/data

Note

  • The SC & SPU default ports are mapped to avoid colisions if you are also running an local cluster:
    • SC public port 9103
    • SPU public 9110
  • The compose file uses the metadata and data directories we’ve created above.

Run the compose file to start containers (add -d to run in the background):

$ docker compose up 

With the containers running, we’ll need to link them in a cluster.

 

Add the cluster profile to Fluvio

Fluvio uses profiles to switch between clusters. For example, a Local cluster, an InfinyOn Cloud cluster, and a Docker cluster. Let’s create a Docker cluster:

$ fluvio profile add docker 127.0.0.1:9103 docker

Fluvio automatically switches to the new profile.

 

Authorize the SPU with the SC

The SC and SPU are both running. However they are not yet talking to each other, as the SC does not allow the SPU to attach to the cluster without prior authorization.

Register the SPU with the SC:

$ fluvio cluster spu register \
  --id 5001 \
  --public-server 0.0.0.0:9110 \
  --private-server  0.0.0.0:9011

Finally, check the cluster status:

$ fluvio cluster status

🎉 Congratulations! Your cluster is now ready for use.

 

Advanced

If you want to start the SC and the SPU independently, you may use docker run.

Run the SC:

$ docker run --name sc \
  -v ./metadata:/fluvio/metadata \
  -e RUST_LOG=info \
  --network host \
  infinyon/fluvio \
  /fluvio-run sc \
    --local /fluvio/metadata

Run the SPU:

$ docker run --name spu-5001 \
    -v ./data:/fluvio/data \
    -e RUST_LOG=info \
    --network host \
    infinyon/fluvio \
    /fluvio-run spu \
      -i 5001 \
      -p 0.0.0.0:9010 \
      -v 0.0.0.0:9011 \
      --log-base-dir /fluvio/data