Stateful Data Flow Beta Build composable event-driven data pipelines in minutes.

Get Started Now

Hackernews Reader

Hackernews Reader helps you build an XML reader that ingests hackernews articles, converts them to json, divides them into records, and publishes each record to a topic. This guide uses the following connector:

  • http-source: to read periodically from a hackernews, parse the XML result into json records, and publish the result to a topic.
 

Prerequisites

 

Step-by-Step

  1. Create http-source configuration file
  2. Download smartmodules
  3. Start Connector
  4. Check Results
 

Create http-source configuration file

Create an HTTP source connector configuration file called hackernews.yaml :

apiVersion: 0.1.0
meta:
  version: 0.2.5
  name: hackernews 
  type: http-source
  topic: hackernews
http:
  method: GET
  endpoint: 'https://hnrss.org/newest'
  interval: 600s
transforms:
  - uses: infinyon-labs/[email protected]
  - uses: infinyon/[email protected]
    with:
      spec:
      - operation: shift
        spec:
          items: ""
  - uses: infinyon-labs/[email protected]
 

Download startmodules

Download the smartmodules used by the connectors to your cluster:

$ fluvio hub sm download infinyon/[email protected]
$ fluvio hub sm download infinyon-labs/[email protected]
$ fluvio hub sm download infinyon-labs/[email protected]
 

Start Connector

$ fluvio cloud connector create -c hackernews.yaml
 

Check Results

Connector logs:

$ fluvio cloud connector log hackernews

Records produced:

$ fluvio consume hackernews -T 10
 

References