Transformation Chaining

In Fluvio, data modification is done using SmartModules, user-defined functions converted to WebAssembly (WASM). Several SmartModules can form a Transformation Chain, working in sequence—each one modifies the data and passes it to the next. Both the sending (Producer) and receiving (Consumer) ends can use these chains; for the Producer, modification happens before the data is saved to the topic, while for the Consumer, it occurs before sending the data.

Transformation Chaining is available for:

1. Fluvio Client
2. Fluvio CLI
3. SmartConnectors

Each transformation in the chain is a SmartModule, paired with some specific instructions. Typically, you’d set this up in a yaml file.

Here’s a basic example:

transforms:
  - uses: infinyon/[email protected]
    with:
      spec:
        - operation: default
          spec:
            source: "http"

In this example, there’s one transformation done by a SmartModule named infinyon/[email protected]. Ensure this SmartModule is available in your cluster:

fluvio sm list
  SMARTMODULE              SIZE
  infinyon/[email protected]      608.4 KB

The section under with in the config is the special note to the SmartModule on how to modify the data. In this case, it receives this bit of info:

"spec" : "[{\"operation\":\"default\",\"spec\":{\"source\":\"http\"}}]"

You can arrange multiple SmartModules to work in sequence. The output from one SmartModule is used as the input for the next, so their order is important.

transforms:
  - uses: infinyon/[email protected]
    with:
      spec:
        - operation: shift
          spec:
            fact: "animal.fact"
            length: "length"
  - uses: infinyon/[email protected]
    with:
      regex: "[Cc]at"

In this setup, the jolt transformation takes place first, then its output is used as the input for regex-filter.