InfinyOn Cloud - User Guide

This guide will outline using the fluvio CLI with InfinyOn Cloud.

We will cover:

• Using Topics to store data
• Using Producers and Consumers to send and read data
• Creating data Connectors
• Developing and using SmartModules

• An InfinyOn Cloud account
  • You need to log in via InfinyOn Cloud WebUI once, to start your cluster for this guide.
• A little bit of command-line experience for using the fluvio CLI
• A basic Rust development environment
  • For SmartModule development
• The fluvio CLI
  • We’ll cover installation in the next section.

You’ll need to download and install the CLI.

curl -fsS https://packages.fluvio.io/v1/install.sh | bash

This will download the CLI and related tools and config files into $HOME/.fluvio, with the executables in $HOME/.fluvio/bin.

To complete the installation, you will need to add the executables to your shell $PATH

You can run the following to set your PATH on shell startup:

echo 'export PATH="${HOME}/.fluvio/bin:${PATH}"' >> ~/.bashrc

echo 'export PATH="${HOME}/.fluvio/bin:${PATH}"' >> ~/.zshrc

fish_add_path "$HOME/.fluvio/bin

After adding to $PATH you should open a new terminal window.

To log in, run fluvio cloud login. You will be prompted for your email and password. These values are what you used for the WebUI.

$ fluvio cloud login
InfinyOn Cloud email: [email protected]e.com
Password: <Your InfinyOn Cloud password>

To verify login, you can run this command. The value returned should be cloud.

$ fluvio profile current
cloud

Topics are used to store data and send data streams.

You can create a topic with the following command:

$ fluvio topic create quickstart-topic

Where quickstart-topic is the name of your topic

Read more about Topics in the Fluvio docs

You can send topic data, or produce to your topic.

Let’s try to interactively produce text to your topic

$ fluvio produce quickstart-topic
> hello world!
Ok!

Typing anything then pressing Enter will send a record to your topic.

Press Ctrl+C to exit the interactive producer prompt.

Read more about Producers in the Fluvio docs

You can read topic data, or consume from your topic.

This command will create a consumer that listens to your topic for new records, then prints it to screen

$ fluvio consume quickstart-topic
Consuming records from the end of topic 'quickstart-topic'. This will wait for new records

To see this in action, open another terminal, and produce new data.

To see previously sent data, you can add an option to your consume command to request a starting offset with the -B <offset> flag.

The -B flag will default to offset value 0 if not supplied.

The -d flag will close the consumer connection after all data has been sent.

$ fluvio consume quickstart-topic -B -d
hello world!

Read more about Consumers in the Fluvio docs

SmartConnectors are active components running in InfinyOn Cloud.

They are used for importing data into or export data from your cluster with Source or Sink connectors, respectively.

They are customized by a YAML config file. We support a growing number of connectors, but we will be covering the HTTP Source connector.

This is an example config file for an HTTP source connector. It will poll data from an HTTP endpoint that returns a random cat fact every 30 seconds to a topic cat-facts.

# cat-facts-source-connector.yml
name: http-random-cat-facts
type: http
topic: cat-facts 
create_topic: true
direction: source
parameters:
  endpoint: https://catfact.ninja/fact
  interval: 30

You create the connector with fluvio connector create -c <path to connector config>

$ fluvio connector create -c ./path/to/cat-facts-source-connector.yml

After a moment, we can see our connector status with fluvio connector list.

$ fluvio connector list
 fluvio connector list
  NAME                   TYPE  VERSION  STATUS
  http-random-cat-facts  http  latest   Running

We can monitor new data in the connector’s topic with fluvio consume cat-facts

$ fluvio consume cat-facts
Consuming records from the end of topic 'cat-facts'. This will wait for new records
{"fact":"People who are allergic to cats are actually allergic to cat saliva or to cat dander. If the resident cat is bathed regularly the allergic people tolerate it better.","length":165}

Read more about SmartConnectors in the Fluvio docs

SmartModules are user-defined functions that are used for inline data stream manipulation.

They are compiled into WebAssembly, which enables them to be portable and re-usable at many points of the streaming pipeline.

You can also use SmartModules with Connectors. Check out the docs for more details.

At the moment, SmartModule logic can only be written in Rust.

In addition to the basic Rust development environment, we need to install a few more tools to help build the SmartModule code.

First, we need to add the wasm32-unknown-unknown toolchain target, to allow us to compile our Rust code into WebAssembly (or Wasm).

$ rustup target add wasm32-unknown-unknown

Next, we need cargo-generate, to help build the SmartModule Rust project template.

$ cargo install cargo-generate

The following command will guide you through a few prompts and generate a SmartModule template project.

We will cover writing a filter type, but for more examples check out https://github.com/infinyon/fluvio-smartmodule-examples

$ cargo generate --git https://github.com/infinyon/fluvio-smartmodule-template
🤷   Project Name : quickstart
🔧   Generating template ...
✔ 🤷   Which type of SmartModule would you like? · filter
✔ 🤷   Want to use SmartModule parameters? · false
[1/7]   Done: .cargo/config.toml
[2/7]   Done: .cargo
[3/7]   Done: .gitignore
[4/7]   Done: Cargo.toml
[5/7]   Done: README.md
[6/7]   Done: src/lib.rs
[7/7]   Done: src
🔧   Moving generated files into: `/code/quickstart`...
💡   Initializing a fresh Git repository
✨   Done! New project created /code/quickstart

In the default filter code, we read in the record and evaluate if the contents contain the letter a. If it does, we return a value. Otherwise we don’t return a value.

use fluvio_smartmodule::{smartmodule, Result, Record};

#[smartmodule(filter)]
pub fn filter(record: &Record) -> Result<bool> {
    let string = std::str::from_utf8(record.value.as_ref())?;
    Ok(string.contains('a'))
}

From the project directory, we can compile the code into Wasm.

$ cargo build --release

After a moment of code compiling, we can see the resulting Wasm binary, quickstart.wasm, which is our SmartModule.

$ ls ./target/wasm32-unknown-unknown/release/
build  deps  examples  incremental  quickstart.d  quickstart.wasm

You can load the SmartModule and give it a name so it can be used by producers, consumers and connectors.

Here we are naming our filter contains-a

$ fluvio smartmodule create contains-a --wasm-file ./target/wasm32-unknown-unknown/release/quickstart.wasm

And we can confirm our SmartModule has been loaded with fluvio smartmodule list

$ fluvio sm list
  NAME        STATUS             SIZE
  contains-a  SmartModuleStatus  90224

In this example, we have a topic fruits and using 2 terminals.

In one terminal, we’ll create a Consumer using our contains-a filter.

Consumer

$ fluvio consume fruits --filter="contains-a"

And in another, our Producer. We’ll add in 5 example fruit names.

$ fluvio produce fruits
> grape
Ok!
> strawberry
Ok!
> plum
Ok!
> raspberry
Ok!
> mango
Ok!
> ^C

The output of our Consumer should only have 4 fruits, since plum does not have the letter a.

grape
strawberry
raspberry
mango

Now you’re familiar with using InfinyOn Cloud with the fluvio CLI.

If you need more flexibility, you can write code instead.

The Fluvio client is available with support for several programming languages.