InfinyOn Cloud - Quick Start

This guide will outline using the Fluvio CLI with InfinyOn Cloud.

We will cover:

• Using Topics to store data
• Using Producers and Consumers to send and read data
• Creating data Connectors
• Developing and using SmartModules

• An InfinyOn Cloud account
  • You need to log in via InfinyOn Cloud WebUI once, to start your cluster for this guide.
• A little bit of command-line experience for using the fluvio CLI
• A basic Rust development environment
  • For SmartModule development
• The fluvio and smdk CLIs
  • We’ll cover installation in the next section.

You’ll need to download and install the CLI.

curl -fsS https://packages.fluvio.io/v1/install.sh | bash

This will download the CLI and related tools and config files into $HOME/.fluvio, with the executables in $HOME/.fluvio/bin. To complete the installation, you will need to add the executables to your shell $PATH.

Head over to the InfinyOn Cloud sign up page.

Currently, there are two ways to create your Fluvio account:

• Google sign in (Oauth2)
• Email registration

Depending which method you chose in account creation, you will have the option of logging in with OAuth2, or username/password.

Use the command fluvio cloud login --use-oauth2 to connect to the InfinyOn Cloud.

$ fluvio cloud login --use-oauth2
A web browser has been opened at https://infinyon-cloud.us.auth0.com/activate?user_code=GLMC-QDDJ.
Please proceed with authentication.

Fluvio will open a login screen in your browser. If it is unable to do so, copy the URL provided in the command prompt and enter it into your web browser manually.

Use the command fluvio cloud login to connect the InfinyOn Cloud to your Fluvio CLI. It will ask for your account credentials, as seen below.

$ fluvio cloud login
InfinyOn Cloud email: [email protected]
Password:

To verify login, you can run this command. The value returned should be cloud.

$ fluvio profile current
cloud

Topics are used to store data and send data streams.

You can create a topic with the following command:

$ fluvio topic create quickstart-topic

Where quickstart-topic is the name of your topic

Read more about Topics in the Fluvio docs

You can send topic data, or produce to your topic.

Let’s try to interactively produce text to your topic

$ fluvio produce quickstart-topic
> hello world!
Ok!

Typing anything then pressing Enter will send a record to your topic.

Press Ctrl+C to exit the interactive producer prompt.

Read more about Producers in the Fluvio docs

You can read topic data, or consume from your topic.

This command will create a consumer that listens to your topic for new records, then prints it to screen

$ fluvio consume quickstart-topic
Consuming records from the end of topic 'quickstart-topic'. This will wait for new records

To see this in action, open another terminal, and produce new data.

To see previously sent data, you can add an option to your consume command to request a starting offset with the -B <offset> flag.

The -B flag will default to offset value 0 if not supplied.

The -d flag will close the consumer connection after all data has been sent.

$ fluvio consume quickstart-topic -B -d
hello world!

Read more about Consumers in the Fluvio docs

SmartConnectors are active components running in InfinyOn Cloud.

They are used for importing data into or export data from your cluster with Source or Sink connectors, respectively.

They are customized by a YAML config file. We support a growing number of connectors, but we will be covering the HTTP Source connector.

This is an example config file for an HTTP source connector. It will poll data from an HTTP endpoint that returns a random cat fact every 30 seconds to a topic cat-facts.

# cat-facts-source-connector.yml
meta:
  version: 0.1.0
  name: http-random-cat-facts
  type: http-source
  topic: cat-facts
http:
  endpoint: https://catfact.ninja/fact
  interval: 30

You create the connector with fluvio cloud connector create -c <path to connector config>

$ fluvio cloud connector create -c ./path/to/cat-facts-source-connector.yml

After a moment, we can see our connector status with fluvio connector list.

$ fluvio cloud connector list
 NAME                   TYPE         VERSION  CDK  STATUS
 http-random-cat-facts  http-source  0.1.0    V3   Running

We can monitor new data in the connector’s topic with fluvio consume cat-facts

$ fluvio consume cat-facts
Consuming records from the end of topic 'cat-facts'. This will wait for new records
{"fact":"People who are allergic to cats are actually allergic to cat saliva or to cat dander. If the resident cat is bathed regularly the allergic people tolerate it better.","length":165}

Read more about SmartConnectors in the Fluvio docs

SmartModules are user-defined functions that are used for inline data stream manipulation.

They are compiled into WebAssembly, which enables them to be portable and re-usable at many points of the streaming pipeline. You can also use SmartModules with Connectors.

SmartModules Development Kit (SMDK) is an independent executable downloadable via Fluvio CLI to help developers build and test SmartModules, and publish them to the SmartModule Hub.

Currently, SMDK is limited to the Rust programming language. Install the following prerequisites:

1. Install Rust

2. Add cargo target wasm32-unknown-unknown install [cargo-generate] for compiling SmartModule Rust code into WebAssembly modules.

$ rustup target add wasm32-unknown-unknown

3. Install cargo-generate for building the SmartModule Rust project template.

$ cargo install cargo-generate

4 Install smdk

$ fluvio install smdk

The following command will guide you through a few prompts and generate a SmartModule template project.

Writing the SmartModule logic

We will cover writing a filter type, but for more examples check out https://github.com/infinyon/fluvio-smartmodule-examples

$ amdk generate
🤷   Project Name: quickstart
✔ 🤷   Will your SmartModule use init parameters? · false
✔ 🤷   Which type of SmartModule would you like? · filter
✔ 🤷   Will your SmartModule be public? · false
[1/7]   Done:. gitignore 
[2/7]   Done: Cargo.toml
[3/7]   Done: README.md
[4/7]   Done: SmartModule.toml
[5/7]   Done: rust-toolchain.toml 
[6/7]   Done: src/lib.rs
[7/7]   Done: src 

In the default filter code, we read in the record and evaluate if the contents contain the letter a. If it does, we return a value. Otherwise we don’t return a value.

Let’s check it out:

$ cd quickstart; cat src/lib.rs 

use fluvio_smartmodule::{smartmodule, Result, Record};

#[smartmodule(filter)]
pub fn filter(record: &Record) -> Result<bool> {
    let string = std::str::from_utf8(record.value.as_ref())?;
    Ok(string.contains('a'))
}

Build and test the SmartModule

From the project directory, we can compile the code into WASM.

$ smdk build

We can test the SmartModule at the command line:

$ smdk test --text "cats"
cats
$ smdk test --text "dogs"

We ensured that it works, let’s load it on the cluster.

Load the SmartModule into cluster

Uploading the SmartModule to the cluster allows us to use it with producers, consumers and connectors.

$ smdk load
Loading package at: ~/quickstart
Found SmartModule package: quickstart
Trying connection to fluvio router.infinyon.cloud:9003
Creating SmartModule: quickstart

Let’s confirm our SmartModule has been loaded with fluvio smartmodule list

$ fluvio sm list
  SMARTMODULE                    SIZE     
  john/[email protected]          28.1 KB 

In this example, we’ll create a topic fruits:

$ fluvio topic create fruits

In one terminal, we’ll create a Consumer using our smartmodule filter.

Let’s start the consumer

$ fluvio consume fruits --smartmodule=john/[email protected]

And in another, our Producer. We’ll add in 5 example fruit names.

$ fluvio produce fruits
> grape
Ok!
> strawberry
Ok!
> plum
Ok!
> raspberry
Ok!
> mango
Ok!
> ^C

The output of our Consumer should only have 4 fruits, since plum does not have the letter a.

grape
strawberry
raspberry
mango

SmartModule Hub allows you to download and run pre-built SmartModules. You also have the option to upload your own as private or pubolic packages.

Let’s see what SmartModules are availble for us:

$ fluvio hub list
  SMARTMODULE                              Visibility 
  infinyon-labs/[email protected]       public     
  infinyon-labs/[email protected]         public     
  infinyon-labs/[email protected]         public     
  infinyon-labs/[email protected]       public     
  infinyon-labs/[email protected]            public     
  infinyon-labs/[email protected]             public     
  infinyon-labs/[email protected]  public     
  infinyon/[email protected]                      public     
  infinyon/[email protected]                  public     
  infinyon/[email protected]              public     

Now you’re familiar with using InfinyOn Cloud with the fluvio CLI, Checkout our Examples and Tutorials.