Holmes7/MapReduce

A simplified distributed MapReduce system built from scratch in Go, with the goal of gaining a deep understanding of the fundamental challenges it addresses. Handles task distribution, fault tolerance, and parallel processing, resembling the concepts from the classic MapReduce paper.

MapReduce

A simplified distributed MapReduce system built from scratch in Go, featuring a coordinator and multiple workers. Handles task distribution, fault tolerance, and parallel processing, resembling the concepts from the classic MapReduce Paper. This is a lab in the MIT distributed systems course.

My motivation behind building this was to understand the inner workings of MapReduce.

Usage

Build the plugin

go build -buildmode=plugin wc.go

Run the master node in a terminal window

go run mrcoordinator.go input/pg*.txt

Run the worker node in another terminal window. You can run multiple such workers

go run mrworker.go wc.so

You can similarly run other map reduce applications by writing a map and reduce function. Refer wc.go to see how to write it.