This guide will outline using the fluvio
CLI with InfinyOn Cloud.
We will cover:
- Using Topics to store data
- Using Producers and Consumers to send and read data
- Creating data Connectors
- Developing and using SmartModules
- An InfinyOn Cloud account
- A little bit of command-line experience for using the
fluvio
CLI
- A basic Rust development environment
- For SmartModule development
- The
fluvio
CLI
- We’ll cover installation in the next section.
You’ll need to download and install the CLI.
curl -fsS https://packages.fluvio.io/v1/install.sh | bash
This will download the CLI and related tools and config files into $HOME/.fluvio
, with the executables in $HOME/.fluvio/bin
.
To complete the installation, you will need to add the executables to your shell $PATH
You can run the following to set your PATH
on shell startup:
For Bash
echo 'export PATH="${HOME}/.fluvio/bin:${PATH}"' >> ~/.bashrc
For Zsh
echo 'export PATH="${HOME}/.fluvio/bin:${PATH}"' >> ~/.zshrc
For Fish
fish_add_path "$HOME/.fluvio/bin
After adding to $PATH
you should open a new terminal window.
Log in to InfinyOn Cloud via CLI
To log in, run fluvio cloud login
. You will be prompted for your email and password. These values are what you used for the WebUI.
$ fluvio cloud login
InfinyOn Cloud email: [email protected]e.com
Password: <Your InfinyOn Cloud password>
To verify login, you can run this command. The value returned should be cloud
.
$ fluvio profile current
cloud
Topics are used to store data and send data streams.
You can create a topic with the following command:
$ fluvio topic create quickstart-topic
Where quickstart-topic
is the name of your topic
Read more about Topics in the Fluvio docs
Produce data to your topic
You can send topic data, or produce to your topic.
Let’s try to interactively produce text to your topic
$ fluvio produce quickstart-topic
> hello world!
Ok!
Typing anything then pressing Enter
will send a record to your topic.
Press Ctrl+C
to exit the interactive producer prompt.
Read more about Producers in the Fluvio docs
Consume data from your topic
You can read topic data, or consume from your topic.
This command will create a consumer that listens to your topic for new records, then prints it to screen
$ fluvio consume quickstart-topic
Consuming records from the end of topic 'quickstart-topic'. This will wait for new records
To see this in action, open another terminal, and produce new data.
To see previously sent data, you can add an option to your consume command to request a starting offset with the -B <offset>
flag.
The -B
flag will default to offset value 0
if not supplied.
The -d
flag will close the consumer connection after all data has been sent.
$ fluvio consume quickstart-topic -B -d
hello world!
Read more about Consumers in the Fluvio docs
Create your first SmartConnector
SmartConnectors are active components running in InfinyOn Cloud.
They are used for importing data into or export data from your cluster with Source or Sink connectors, respectively.
They are customized by a YAML config file. We support a growing number of connectors, but we will be covering the HTTP Source connector.
Example: HTTP Source Cat Facts
This is an example config file for an HTTP source connector. It will poll data from an HTTP endpoint that returns a random cat fact every 30
seconds to a topic cat-facts
.
# cat-facts-source-connector.yml
name: http-random-cat-facts
type: http
topic: cat-facts
create_topic: true
direction: source
parameters:
endpoint: https://catfact.ninja/fact
interval: 30
You create the connector with fluvio connector create -c <path to connector config>
$ fluvio connector create -c ./path/to/cat-facts-source-connector.yml
After a moment, we can see our connector status with fluvio connector list
.
$ fluvio connector list
fluvio connector list
NAME TYPE VERSION STATUS
http-random-cat-facts http latest Running
We can monitor new data in the connector’s topic with fluvio consume cat-facts
$ fluvio consume cat-facts
Consuming records from the end of topic 'cat-facts'. This will wait for new records
{"fact":"People who are allergic to cats are actually allergic to cat saliva or to cat dander. If the resident cat is bathed regularly the allergic people tolerate it better.","length":165}
Read more about SmartConnectors in the Fluvio docs
Create your first SmartModule
SmartModules are user-defined functions that are used for inline data stream manipulation.
They are compiled into WebAssembly, which enables them to be portable and re-usable at many points of the streaming pipeline.
You can also use SmartModules with Connectors. Check out the docs for more details.
At the moment, SmartModule logic can only be written in Rust.
In addition to the basic Rust development environment, we need to install a few more tools to help build the SmartModule code.
First, we need to add the wasm32-unknown-unknown
toolchain target, to allow us to compile our Rust code into WebAssembly (or Wasm).
$ rustup target add wasm32-unknown-unknown
Next, we need cargo-generate
, to help build the SmartModule Rust project template.
$ cargo install cargo-generate
Writing the SmartModule logic
The following command will guide you through a few prompts and generate a SmartModule template project.
We will cover writing a filter
type, but for more examples check out
https://github.com/infinyon/fluvio-smartmodule-examples
$ cargo generate --git https://github.com/infinyon/fluvio-smartmodule-template
🤷 Project Name : quickstart
🔧 Generating template ...
✔ 🤷 Which type of SmartModule would you like? · filter
✔ 🤷 Want to use SmartModule parameters? · false
[1/7] Done: .cargo/config.toml
[2/7] Done: .cargo
[3/7] Done: .gitignore
[4/7] Done: Cargo.toml
[5/7] Done: README.md
[6/7] Done: src/lib.rs
[7/7] Done: src
🔧 Moving generated files into: `/code/quickstart`...
💡 Initializing a fresh Git repository
✨ Done! New project created /code/quickstart
In the default filter
code, we read in the record and evaluate if the contents contain the letter a
. If it does, we return a value. Otherwise we don’t return a value.
use fluvio_smartmodule::{smartmodule, Result, Record};
#[smartmodule(filter)]
pub fn filter(record: &Record) -> Result<bool> {
let string = std::str::from_utf8(record.value.as_ref())?;
Ok(string.contains('a'))
}
From the project directory, we can compile the code into Wasm.
After a moment of code compiling, we can see the resulting Wasm binary, quickstart.wasm
, which is our SmartModule.
$ ls ./target/wasm32-unknown-unknown/release/
build deps examples incremental quickstart.d quickstart.wasm
Load the SmartModule into cluster
You can load the SmartModule and give it a name so it can be used by producers, consumers and connectors.
Here we are naming our filter contains-a
$ fluvio smartmodule create contains-a --wasm-file ./target/wasm32-unknown-unknown/release/quickstart.wasm
And we can confirm our SmartModule has been loaded with fluvio smartmodule list
$ fluvio sm list
NAME STATUS SIZE
contains-a SmartModuleStatus 90224
Using SmartModules with Producers and Consumers
In this example, we have a topic fruits
and using 2 terminals.
In one terminal, we’ll create a Consumer using our contains-a
filter.
Consumer
$ fluvio consume fruits --filter="contains-a"
And in another, our Producer. We’ll add in 5 example fruit names.
$ fluvio produce fruits
> grape
Ok!
> strawberry
Ok!
> plum
Ok!
> raspberry
Ok!
> mango
Ok!
> ^C
The output of our Consumer should only have 4 fruits, since plum
does not have the letter a
.
grape
strawberry
raspberry
mango
Now you’re familiar with using InfinyOn Cloud with the fluvio
CLI.
If you need more flexibility, you can write code instead.
The Fluvio client is available with support for several programming languages.