@wholebuzz/mapreduce-example

Example project for @wholebuzz/mapreduce

This project illustrates running a custom Mapper and Reducer in various scheduling scenarios:

airflow: Various Apache Airflow operators and examples.
docker: Examples of running the tasks in local Docker containers.
helm: Example running on Kubernetes with Helm.
gce: Example starting directly on Google Compute Engine.
local: Examples of running the tasks locally.

For example tasks, we'll:

Count the words in this README.md
Count the words in the title property of records in the test dataset
Sort the test dataset by date, guid and id
Join two separate datasets, each sorted (and sharded) by guid

The supplied test dataset consists of a collection of 10,000 headlines of the form:

{
  "id": 28,
  "date": "2021-10-17T08:52:05.000Z",
  "guid": "https://metro.co.uk/?p=15435904",
  "link": "https://metro.co.uk/2021/10/17/mum-of-six-creates-hallway-tribute-to-her-kids-for-less-than-80-15435904/",
  "feed": "https://metro.co.uk/feed/",
  "props": {
    "title": "Mum-of-six creates sweet hallway tribute to her kids – for less than £80",
    "summary": "'It proves you don’t need to spend lots of money to transform your home.'",
    "imageUrl": "https://wholenews-images.storage.googleapis.com/261d2f38dfd584f5e83130fe504934fb.png"
  },
  "tags": {
    "topic": "PARENTS",
    "locale": "en-GB",
    "topics": [
      "Interiors",
      "DIY"
    ],
    "category": "national",
    "classifiedCategory": "national"
  }
}