InfinyOn Cloud - Quick Start

This guide will outline using the Fluvio CLI with InfinyOn Cloud.

We will cover:

Using Topics to store data
Using Producers and Consumers to send and read data
Creating data Connectors
Developing and using SmartModules

Requirements

An InfinyOn Cloud account
- You need to log in via InfinyOn Cloud WebUI once, to start your cluster for this guide.
A little bit of command-line experience for using the fluvio CLI
A basic Rust development environment
- For SmartModule development
The fluvio and smdk CLIs
- We’ll cover installation in the next section.

Installing Fluvio Client

You’ll need to download and install the CLI.

$ curl -fsS https://hub.infinyon.cloud/install/install.sh | bash

This command will download the Fluvio Version Manager (fvm), Fluvio CLI (fluvio) and config files into $HOME/.fluvio, with the executables in $HOME/.fluvio/bin. To complete the installation, you will need to add the executables to your shell $PATH.

Create InfinyOn Cloud Account

Head over to the InfinyOn Cloud sign up page.

Currently, there are two ways to create your Fluvio account:

Google sign in (Oauth2)
Email registration

Log in to InfinyOn Cloud via CLI

Depending which method you chose in account creation, you will have the option of logging in with OAuth2, or username/password.

Login with Google Oauth

Use the command fluvio cloud login --use-oauth2 to connect to the InfinyOn Cloud.

$ fluvio cloud login --use-oauth2
A web browser has been opened at https://infinyon-cloud.us.auth0.com/activate?user_code=GLMC-QDDJ.
Please proceed with authentication.

Fluvio will open a login screen in your browser. If it is unable to do so, copy the URL provided in the command prompt and enter it into your web browser manually.

Login with Username & Password

Use the command fluvio cloud login to connect the InfinyOn Cloud to your Fluvio CLI. It will ask for your account credentials, as seen below.

$ fluvio cloud login
InfinyOn Cloud email: [email protected]
Password:

Verify Login

To verify login, you can run this command. The value returned should be cloud.

$ fluvio profile current
cloud

Create your first topic

Topics are used to store data and send data streams.

You can create a topic with the following command:

$ fluvio topic create quickstart-topic

Where quickstart-topic is the name of your topic

Produce data to your topic

You can send topic data, or produce to your topic.

Let’s try to interactively produce text to your topic

$ fluvio produce quickstart-topic
> hello world!
Ok!

Typing anything then pressing Enter will send a record to your topic.

Press Ctrl+C to exit the interactive producer prompt.

Consume data from your topic

You can read topic data, or consume from your topic.

This command will create a consumer that listens to your topic for new records, then prints it to screen

$ fluvio consume quickstart-topic
Consuming records from the end of topic 'quickstart-topic'. This will wait for new records

To see this in action, open another terminal, and produce new data.

To see previously sent data, you can add an option to your consume command to request a starting offset with the -B <offset> flag.

The -B flag will default to offset value 0 if not supplied.

The -d flag will close the consumer connection after all data has been sent.

$ fluvio consume quickstart-topic -B -d
hello world!

Create your first SmartConnector

SmartConnectors are active components running in InfinyOn Cloud.

They are used for importing data into or export data from your cluster with Source or Sink connectors, respectively.

They are customized by a YAML config file. We support a growing number of connectors, but we will be covering the HTTP Source connector.

Example: HTTP Source Cat Facts

This is an example config file for an HTTP source connector. It will poll data from an HTTP endpoint that returns a random cat fact every 30 seconds to a topic cat-facts.

# cat-facts-source-connector.yml
meta:
  version: 0.1.0
  name: http-random-cat-facts
  type: http-source
  topic: cat-facts
http:
  endpoint: https://catfact.ninja/fact
  interval: 30

You create the connector with fluvio cloud connector create -c <path to connector config>

$ fluvio cloud connector create -c ./path/to/cat-facts-source-connector.yml

After a moment, we can see our connector status with fluvio connector list.

$ fluvio cloud connector list
 NAME                   TYPE         VERSION  CDK  STATUS
 http-random-cat-facts  http-source  0.1.0    V3   Running

We can monitor new data in the connector’s topic with fluvio consume cat-facts

$ fluvio consume cat-facts
Consuming records from the end of topic 'cat-facts'. This will wait for new records
{"fact":"People who are allergic to cats are actually allergic to cat saliva or to cat dander. If the resident cat is bathed regularly the allergic people tolerate it better.","length":165}

Create your first SmartModule

SmartModules are user-defined functions that are used for inline data stream manipulation.

They are compiled into WebAssembly, which enables them to be portable and re-usable at many points of the streaming pipeline. You can also use SmartModules with Connectors.

Install SMDK

SmartModules Development Kit (SMDK) is an independent executable downloadable via Fluvio CLI to help developers build and test SmartModules, and publish them to the SmartModule Hub.

Currently, SMDK is limited to the Rust programming language. Install the following prerequisites:

Install Rust
Add cargo target wasm32-unknown-unknown install [cargo-generate] for compiling SmartModule Rust code into WebAssembly modules.

$ rustup target add wasm32-unknown-unknown

Install cargo-generate for building the SmartModule Rust project template.

$ cargo install cargo-generate

4 Install smdk

$ fluvio install smdk

Build a Filter SmartModule

The following command will guide you through a few prompts and generate a SmartModule template project.

Writing the SmartModule logic

We will cover writing a filter type, but for more examples check out https://github.com/infinyon/fluvio-smartmodule-examples

$ smdk generate
🤷   Project Name: quickstart
✔ 🤷   Will your SmartModule use init parameters? · false
✔ 🤷   Which type of SmartModule would you like? · filter
✔ 🤷   Will your SmartModule be public? · false
[1/7]   Done:. gitignore 
[2/7]   Done: Cargo.toml
[3/7]   Done: README.md
[4/7]   Done: SmartModule.toml
[5/7]   Done: rust-toolchain.toml 
[6/7]   Done: src/lib.rs
[7/7]   Done: src

In the default filter code, we read in the record and evaluate if the contents contain the letter a. If it does, we return a value. Otherwise we don’t return a value.

Let’s check it out:

$ cd quickstart; cat src/lib.rs

use fluvio_smartmodule::{smartmodule, Result, Record};

#[smartmodule(filter)]
pub fn filter(record: &Record) -> Result<bool> {
    let string = std::str::from_utf8(record.value.as_ref())?;
    Ok(string.contains('a'))
}

Build and test the SmartModule

From the project directory, we can compile the code into WASM.

$ smdk build

We can test the SmartModule at the command line:

$ smdk test --text "cats"
cats
$ smdk test --text "dogs"

We ensured that it works, let’s load it on the cluster.

Load the SmartModule into cluster

Uploading the SmartModule to the cluster allows us to use it with producers, consumers and connectors.

$ smdk load
Loading package at: ~/quickstart
Found SmartModule package: quickstart
Trying connection to fluvio router.infinyon.cloud:9003
Creating SmartModule: quickstart

Let’s confirm our SmartModule has been loaded with fluvio smartmodule list

$ fluvio sm list
  SMARTMODULE                    SIZE     
  john/[email protected]          28.1 KB

Using SmartModules with Producers and Consumers

In this example, we’ll create a topic fruits:

$ fluvio topic create fruits

In one terminal, we’ll create a Consumer using our smartmodule filter.

Let’s start the consumer

$ fluvio consume fruits --smartmodule=john/[email protected]

And in another, our Producer. We’ll add in 5 example fruit names.

$ fluvio produce fruits
> grape
Ok!
> strawberry
Ok!
> plum
Ok!
> raspberry
Ok!
> mango
Ok!
> ^C

The output of our Consumer should only have 4 fruits, since plum does not have the letter a.

grape
strawberry
raspberry
mango

Checkout Smartmodule Hub

SmartModule Hub allows you to download and run pre-built SmartModules. You also have the option to upload your own as private or pubolic packages.

Let’s see what SmartModules are availble for us:

$ fluvio hub sm list
  SMARTMODULE                              Visibility 
  infinyon-labs/[email protected]       public     
  infinyon-labs/[email protected]         public     
  infinyon-labs/[email protected]         public     
  infinyon-labs/[email protected]       public     
  infinyon-labs/[email protected]            public     
  infinyon-labs/[email protected]             public     
  infinyon-labs/[email protected]  public     
  infinyon/[email protected]                      public     
  infinyon/[email protected]                  public     
  infinyon/[email protected]              public

Next Steps

Now you’re familiar with using InfinyOn Cloud with the fluvio CLI, Checkout our Examples and Tutorials.

Company

Resources

InfinyOn Cloud - Quick Start

Requirements

Installing Fluvio Client

Create InfinyOn Cloud Account

Log in to InfinyOn Cloud via CLI

Login with Google Oauth

Login with Username & Password

Verify Login

Create your first topic

Produce data to your topic

Consume data from your topic

Create your first SmartConnector

Example: HTTP Source Cat Facts

Create your first SmartModule

Install SMDK

Build a Filter SmartModule

Writing the SmartModule logic

Build and test the SmartModule

Load the SmartModule into cluster

Using SmartModules with Producers and Consumers

Checkout Smartmodule Hub

Next Steps