Partnership Announcement InfinyOn and MotherDuck Announce Strategic Partnership

Press Release

Hackernews Reader

Hackernews Reader helps you build an XML reader that ingests hackernews articles, converts them to json, divides them into records, and publishes each record to a topic. This guide uses the following connector:

  • http-source: to read periodically from a hackernews, parse the XML result into json records, and publish the result to a topic.
 

Prerequsites

 

Step-by-Step

  1. Create http-source configuration file
  2. Download smartmodules
  3. Start Connector
  4. Check Results
 

Create http-source configuration file

Create an HTTP source connector configuration file called hackernews.yaml :

apiVersion: 0.1.0
meta:
  version: 0.2.4
  name: hackernews 
  type: http-source
  topic: hackernews
http:
  method: GET
  endpoint: 'https://hnrss.org/newest'
  interval: 600s
transforms:
  - uses: infinyon-labs/[email protected]
  - uses: infinyon/[email protected]
    with:
      spec:
      - operation: shift
        spec:
          items: ""
  - uses: infinyon-labs/[email protected]
 

Download startmodules

Download the smartmodules used by the connectors to your cluster:

fluvio hub download infinyon/[email protected]
fluvio hub download infinyon-labs/[email protected]
fluvio hub download infinyon-labs/[email protected]
 

Start Connector

fluvio cloud connector create -c hackernews.yaml
 

Check Results

Connector logs:

fluvio cloud connector log hackernews

Records produced:

fluvio consume hackernews -T 10
 

References