Stateful Data Flow Beta Build composable event-driven data pipelines in minutes.

Get Started Now

Inline Dependencies

For inline operators, you can add dependencies section to include external language specific libraries. The dependencies are defined in the dependencies section of the operator definition. The dependencies are resolved by the Fluvio engine and made available to the operator at runtime.

 

External Dependencies

For Rust based operators, the dependencies are based on Rust Cargo package manager similar to dependencies section in Cargo.

For example, following variations are possible:

Adding a crate with version:

 - operator: map
   dependencies:
     - name: regex
       version: "1.10.0"

Adding a git repository:

 - operator: map
   dependencies:
     - name: regex
       git: "https://github.com/rust-lang/regex"
       tag: "1.10.0"

Adding multiple dependencies:

 - operator: map
   dependencies:
     - name: regex
       version: "1.10.0"
     - name: iter_tools
       version: "0.15.0"

Once the dependencies are added, inline operator can use the dependencies in the run section as in the typical Rust code as example below which tries to validate the year format.

- operator: filter
  dependencies:
    - name: regex
      version: "1.10.0"
  run: |
    fn validate_year(input: String) -> Result<bool, String> {
        use std::sync::OnceLock;
        use regex::Regex;

        static REGEX: OnceLock<Regex> = OnceLock::new();

        let re = REGEX.get_or_init(|| Regex::new(r"^\d{4}-\d{2}-\d{2}$").unwrap());
        Ok(re.is_match(&input))
    }
 

InfinyOn Dependencies

The Stateful Dataflow requires external dependencies to invoke an HTTP call-out and other operations.

 

HTTP Callout

The http callout function makes it easy to make HTTP requests from your service operator. The call-out library is based on the reqwest library. To use the http callout, add a git dependency in your operator definition.

- operator: map
  dependencies:
      - name: sdf-http
        git: "https://github.com/infinyon/sdf-guest-http"
        tag: "v0.3.0"

Following snippet takes a sentence and translate english to spanish using external API. The Request is part of reqwest library and sdf_http::blocking::send is used to send the request.

- operator: map
  dependencies:
      - name: sdf-http
        git: "https://github.com/infinyon/sdf-guest-http"
        tag: "v0.3.0"
  run: |
    fn english_to_spanish(sentence: String) -> Result<String, String> {
        use  sdf_http::http::Request;

        let url = format!("https://acme.com/translate?text={}", sentence);
        let request = Request::builder().uri(url).body("").map_err(|e| e.to_string())?;
        let response = sdf_http::blocking::send(request).map_err(|e| e.to_string())?;
        Ok(response.into_body())
    }