InfinyOn Cloud - Quick Start
This guide will outline using the Fluvio CLI
with InfinyOn Cloud.
We will cover:
- Using Topics to store data
- Using Producers and Consumers to send and read data
- Creating data Connectors
- Developing and using SmartModules
- An InfinyOn Cloud account
- You need to log in via InfinyOn Cloud WebUI once, to start your cluster for this guide.
- A little bit of command-line experience for using the
fluvio
CLI - A basic Rust development environment
- For SmartModule development
- The
fluvio
andsmdk
CLIs- We’ll cover installation in the next section.
You’ll need to download and install the CLI.
$ curl -fsS https://hub.infinyon.cloud/install/install.sh | bash
This command will download the Fluvio Version Manager (fvm), Fluvio CLI (fluvio) and config files into $HOME/.fluvio
, with the executables in $HOME/.fluvio/bin
. To complete the installation, you will need to add the executables to your shell $PATH
.
Head over to the InfinyOn Cloud sign up page.
Currently, there are two ways to create your Fluvio account:
- Google sign in (Oauth2)
- Email registration
Depending which method you chose in account creation, you will have the option of logging in with OAuth2, or username/password.
Use the command fluvio cloud login --use-oauth2
to connect to the InfinyOn Cloud.
$ fluvio cloud login --use-oauth2
A web browser has been opened at https://infinyon-cloud.us.auth0.com/activate?user_code=GLMC-QDDJ.
Please proceed with authentication.
Fluvio will open a login screen in your browser. If it is unable to do so, copy the URL provided in the command prompt and enter it into your web browser manually.
Use the command fluvio cloud login
to connect the InfinyOn Cloud to your
Fluvio CLI. It will ask for your account credentials, as seen below.
$ fluvio cloud login
InfinyOn Cloud email: [email protected]
Password:
To verify login, you can run this command. The value returned should be cloud
.
$ fluvio profile current
cloud
Topics are used to store data and send data streams.
You can create a topic with the following command:
$ fluvio topic create quickstart-topic
Where quickstart-topic
is the name of your topic
Read more about Topics in the Fluvio docs
You can send topic data, or produce to your topic.
Let’s try to interactively produce text to your topic
$ fluvio produce quickstart-topic
> hello world!
Ok!
Typing anything then pressing Enter
will send a record to your topic.
Press Ctrl+C
to exit the interactive producer prompt.
Read more about Producers in the Fluvio docs
You can read topic data, or consume from your topic.
This command will create a consumer that listens to your topic for new records, then prints it to screen
$ fluvio consume quickstart-topic
Consuming records from the end of topic 'quickstart-topic'. This will wait for new records
To see this in action, open another terminal, and produce new data.
To see previously sent data, you can add an option to your consume command to request a starting offset with the -B <offset>
flag.
The -B
flag will default to offset value 0
if not supplied.
The -d
flag will close the consumer connection after all data has been sent.
$ fluvio consume quickstart-topic -B -d
hello world!
Read more about Consumers in the Fluvio docs
SmartConnectors are active components running in InfinyOn Cloud.
They are used for importing data into or export data from your cluster with Source or Sink connectors, respectively.
They are customized by a YAML config file. We support a growing number of connectors, but we will be covering the HTTP Source connector.
This is an example config file for an HTTP source connector. It will poll data from an HTTP endpoint that returns a random cat fact every 30
seconds to a topic cat-facts
.
# cat-facts-source-connector.yml
meta:
version: 0.1.0
name: http-random-cat-facts
type: http-source
topic: cat-facts
http:
endpoint: https://catfact.ninja/fact
interval: 30
You create the connector with fluvio cloud connector create -c <path to connector config>
$ fluvio cloud connector create -c ./path/to/cat-facts-source-connector.yml
After a moment, we can see our connector status with fluvio connector list
.
$ fluvio cloud connector list
NAME TYPE VERSION CDK STATUS
http-random-cat-facts http-source 0.1.0 V3 Running
We can monitor new data in the connector’s topic with fluvio consume cat-facts
$ fluvio consume cat-facts
Consuming records from the end of topic 'cat-facts'. This will wait for new records
{"fact":"People who are allergic to cats are actually allergic to cat saliva or to cat dander. If the resident cat is bathed regularly the allergic people tolerate it better.","length":165}
Read more about SmartConnectors in the Fluvio docs
SmartModules are user-defined functions that are used for inline data stream manipulation.
They are compiled into WebAssembly, which enables them to be portable and re-usable at many points of the streaming pipeline. You can also use SmartModules with Connectors.
SmartModules Development Kit (SMDK) is an independent executable downloadable via Fluvio CLI to help developers build and test SmartModules, and publish them to the SmartModule Hub.
Currently, SMDK is limited to the Rust programming language. Install the following prerequisites:
-
Add cargo target
wasm32-unknown-unknown
install [cargo-generate] for compiling SmartModule Rust code into WebAssembly modules.
$ rustup target add wasm32-unknown-unknown
- Install
cargo-generate
for building the SmartModule Rust project template.
$ cargo install cargo-generate
4 Install smdk
$ fluvio install smdk
The following command will guide you through a few prompts and generate a SmartModule template project.
We will cover writing a filter
type, but for more examples check out https://github.com/infinyon/fluvio-smartmodule-examples
$ smdk generate
🤷 Project Name: quickstart
✔ 🤷 Will your SmartModule use init parameters? · false
✔ 🤷 Which type of SmartModule would you like? · filter
✔ 🤷 Will your SmartModule be public? · false
[1/7] Done:. gitignore
[2/7] Done: Cargo.toml
[3/7] Done: README.md
[4/7] Done: SmartModule.toml
[5/7] Done: rust-toolchain.toml
[6/7] Done: src/lib.rs
[7/7] Done: src
In the default filter
code, we read in the record and evaluate if the contents contain the letter a
. If it does, we return a value. Otherwise we don’t return a value.
Let’s check it out:
$ cd quickstart; cat src/lib.rs
use fluvio_smartmodule::{smartmodule, Result, Record};
#[smartmodule(filter)]
pub fn filter(record: &Record) -> Result<bool> {
let string = std::str::from_utf8(record.value.as_ref())?;
Ok(string.contains('a'))
}
From the project directory, we can compile the code into WASM.
$ smdk build
We can test the SmartModule at the command line:
$ smdk test --text "cats"
cats
$ smdk test --text "dogs"
We ensured that it works, let’s load it on the cluster.
Uploading the SmartModule to the cluster allows us to use it with producers, consumers and connectors.
$ smdk load
Loading package at: ~/quickstart
Found SmartModule package: quickstart
Trying connection to fluvio router.infinyon.cloud:9003
Creating SmartModule: quickstart
Let’s confirm our SmartModule has been loaded with fluvio smartmodule list
$ fluvio sm list
SMARTMODULE SIZE
john/[email protected] 28.1 KB
In this example, we’ll create a topic fruits
:
$ fluvio topic create fruits
In one terminal, we’ll create a Consumer using our smartmodule filter.
Let’s start the consumer
$ fluvio consume fruits --smartmodule=john/[email protected]
And in another, our Producer. We’ll add in 5 example fruit names.
$ fluvio produce fruits
> grape
Ok!
> strawberry
Ok!
> plum
Ok!
> raspberry
Ok!
> mango
Ok!
> ^C
The output of our Consumer should only have 4 fruits, since plum
does not have the letter a
.
grape
strawberry
raspberry
mango
SmartModule Hub allows you to download and run pre-built SmartModules. You also have the option to upload your own as private or pubolic packages.
Let’s see what SmartModules are availble for us:
$ fluvio hub sm list
SMARTMODULE Visibility
infinyon-labs/[email protected] public
infinyon-labs/[email protected] public
infinyon-labs/[email protected] public
infinyon-labs/[email protected] public
infinyon-labs/[email protected] public
infinyon-labs/[email protected] public
infinyon-labs/[email protected] public
infinyon/[email protected] public
infinyon/[email protected] public
infinyon/[email protected] public
Now you’re familiar with using InfinyOn Cloud with the fluvio
CLI, Checkout our Examples and Tutorials.