Stateful Data Flow Beta Build composable event-driven data pipelines in minutes.

Get Started Now

SmartModule Basics

This tutorial assumes that fluvio is installed, and logged-in to InfinyOn Cloud. Follow the Quick Start to get set up.

SmartModules are the basic building blocks for transformations in Fluvio, allowing users to define custom functions for processing or transforming streaming data. They provide a flexible way to tailor data handling to meet particular needs, enhancing Fluvio’s capabilities.

 

See list of available SmartModules

$ fluvio hub sm list
  SMARTMODULE                              Visibility 
  infinyon-labs/[email protected]       public     
  infinyon-labs/[email protected]         public     
  infinyon-labs/[email protected]       public     
  infinyon-labs/[email protected]         public     
  infinyon-labs/[email protected]       public     
  infinyon-labs/[email protected]            public     
  infinyon-labs/[email protected]  public     
  infinyon/[email protected]                      public     
  infinyon/[email protected]                  public     
  infinyon/[email protected]              public
 

Download SmartModules

SmartModules must be downloaded before they can be used. Afterwards, downloaded SmartModules are available for your Producers and Consumers.

$ fluvio hub sm download infinyon/[email protected]
downloading infinyon/[email protected] to infinyon-regex-filter-0.1.0.ipkg
... downloading complete
... checking package
trying connection to fluvio router.infinyon.cloud:9003
... cluster smartmodule install complete
 

Use SmartModule with Producer and Consumer

You can specify a SmartModule to use with fluvio produce and fluvio consume

This consumer is using the SmartModule we just downloaded with --smartmodule, and is also configuring it with --params/-e.

$ fluvio consume --smartmodule infinyon/[email protected] --params regex='[Cc]at' cat-facts

The --transforms-file flag are for more complex transformations defined in a YAML file. See the Transformation Chaining page for more detail.

Example transforms.yaml with multiple transformations. Order matters here, so infinyon/[email protected] is first and infinyon/[email protected] is second.

transforms:
  - uses: infinyon/[email protected]
    with:
      spec:
        - operation: shift
          spec:
            fact: "animal.fact"
            length: "length"
  - uses: infinyon/[email protected]
    with:
      regex: "[Cc]at"

$ fluvio consume --transforms-file ./my-transforms.yaml my-topic
 

See list of Downloaded SmartModules

$ fluvio sm list
  SMARTMODULE                  SIZE     
  infinyon/[email protected]          611.5 KB 
  infinyon/[email protected]      558.4 KB 
  infinyon/[email protected]  312.7 KB 
  infinyon/[email protected]          564.0 KB 
  infinyon/[email protected]      559.6 KB
 

Delete SmartModules

$ fluvio sm delete infinyon/[email protected] 
smartmodule "infinyon/[email protected]" deleted
 

Use SmartModule with Connector

You can define transforms when you create connectors with transforms

# connector-example.yaml
apiVersion: 0.1.0
meta:
  version: 0.2.5
  name: cat-facts
  type: http-source
  topic: cat-facts
  create-topic: true
  secrets:
    - name: AUTHORIZATION_TOKEN
http:
  endpoint: "https://catfact.ninja/fact"
  interval: 10s  
  headers:
    - "Authorization: token ${{ secrets.AUTHORIZATION_TOKEN }}"
    - "Cache-Control: no-cache"
transforms:
  - uses: infinyon/[email protected]
    with:
      spec:
        - operation: shift
          spec:
            fact: "animal.fact"
            length: "length"
  - uses: infinyon/[email protected]
    with:
      regex: "[Cc]at"

Everything is in the config file. You create a connector as usual.

$ fluvio cloud connector create --config connector-example.yaml
 

Use SmartModule with Webhook

# example-webhook.yaml
meta:
  name: my-webhook
  topic: my-topic 
transforms:
  - uses: infinyon/[email protected]
    with:
      spec:
        - operation: shift
          spec:
            fact: "animal.fact"
            length: "length"
  - uses: infinyon/[email protected]
    with:
      regex: "[Cc]at"
webhook:
  outputParts: body

Just like Connectors, everything is in the config file. You create a webhook as usual.

fluvio cloud webhook create --config example-webhook.yaml