geftimov/hadoop-map-reduce-patterns

Hadoop Map-Reduce Design Patterns

MapReduce

Hadoop Map-Reduce Design Patterns

How to use the MapReduce:

Clone the repository:

git clone git@github.com:geftimov/MapReduce.git

Go in to the folder:

cd MapReduce

Build it with Maven:

mvn clean install

Example run in each individual pattern example.

Summarization Patterns

1. Numerical Summarization ReadMe

• CommentWordCount
• MinMaxCount
• Average
• MedianStdDev (With In-Memory Map)
• MedianAndStandardDeviationCommentLengthByHour (Without the Map, more efficient)

2. Inverted Index Summarization ReadMe

• WikipediaExtractor

3. Counting with Counters ReadMe

• CountNumUsersByState

Filtering Patterns

1. Filtering ReadMe

• DistributedGrep
• SimpleRandomSampling

2. Bloom Filtering ReadMe

3. Top Ten ReadMe

• TopTenUsersByReputation

4. Distinct ReadMe

• DistinctUser

Data Organization Patterns

1. Structured to Hierarchical ReadMe

• PostCommentHierarchy
• QuestionAnswerBuilder

2. Partitioning ReadMe

• LastAccessDate

3. Binning ReadMe

• BinningTags

4. TotalOrderSorting ReadMe

• TotalOrderSortingStage

5. Shuffling ReadMe

• Anonymize

Join Patterns

1. Reduce Side Join ReadMe

• UserJoin
• UserJoinBloomFilter

2. Replicated Join ReadMe

• ReplicatedUserJoin

3. Composite Join ReadMe

• CompositeUserJoin

4. Cartesian Product ReadMe

• CartesianCommentComparison

About Me

Georgi Kalinov Eftimov
jokatavr@gmail.com