This is an accompanying website for MLPrague 2017 workshop Advanced data analysis on Hadoop clusters. Specifically, source codes for the machine learning part are provided. Description of the used data can be found here too, see below.
Source codes are created for Spark.
Tha practical part of machine learning can be divided into two parts:
- Community detection in telecommunication networks
- Churn prediction in telecommunication industry
As churn prediction part assumes results from community detection, it is necesarry to run the codes described in Community Detection section first. Then the churn prediction part can be executed by running the main.py script.
Given the phone call records, the task is to find communities in a network created from these phone calls. Customers represent vertices in such a network and edges link customers who called to each other.
The presented solution creates a graph from one-month call records. Only customers with at least 10 calls are linked together. Label Propagation Algorithm is used for community detection.
Scala source codes for Spark can be found in phase_0_community_detection/ directory. The scala script assumes mlp_sampled_cdr_records.parquet data available.
This script will create two new data files: lpa_20160301_20160401.parquet and lpa_20160401_20160501.parquet.
In this part, the task is to predict customers who are likely to churn. All source codes for this part are written in python are assumed to be run by PySpark. Created machine learning model uses features extracted from one month and predicts potential churners for the next month. For example, it takes phone call records from March and predicts which customers are likely to churn in April. Features are built from the input data described below.
This part is divided into three phases:
- Data preparation - creates various features from the input data
- Data preprocessing - imputing and trasforming features; it also adds some new derived features
- Classification - trains a classification model on a train dataset and applies it on a test dataset
Evaluation of the model is performed outside of those phases for the sake of detailed illustration.
mlp_sampled_cdr_records.parquet - phone call records from two months
- record_type: string - type of voice records
- date_key: string - date of the call
- duration: integer - duration of the call in seconds
- frommsisdn_prefix: string - operator prefix
- frommsisdn: long - home operator number (either receiving or calling - according to the record type)
- tomsisdn_prefix: string - operator prefix
- tomsisdn: long - number of the second customer (can be either of the home operator or not)
mlp_sampled_ebr_base_20160401.parquet, mlp_sampled_ebr_base_20160501.parquet - information about home operator customers
- msisdn: long - number of the customer
- customer_type: string - either private or business
- commitment_from_key: string - date of the commitment start
- commitment_to_key: string - date of the commitment end
- rateplan_group: long - name of the rateplan group
- rateplan_name: long - name of the raplan
mlp_sampled_ebr_churners_20151201_20160630.parquet - list of churned customers from two months
- msisdn: long - number of the customer
- date_key: string - date of the churn
Directory scripts contains various python scripts for data exploration. Script move_data.py illustrates how to save parquet data from a remote AWS S3 repository to local repository.
NOTE: 'callcenters' are numbers behaving like callcenters - i.e. they call to a huge number of phone numbers. We select TOP 12 such 'callcenters' from data.
churned - binary label attribute msisdn customer_type rateplan_group rateplan_name committed - whether the customer is committed at this point committed_days - for how long is the customer committed commitment_remaining - how many days till the end of the commitment callcenter_calls_count - count of phone calls with so called 'callcenters' callcenter_calls_duration - total duration of phone calls with so called 'callcenters' cc_\cnt_X1 - count of phone calls with call center X1, where X1 is the number of the callcenter cc_\dur_X1 - duration of phone calls with call center X1, where X1 is the number of the callcenter cc_\avg_X1 - average duration of phone calls with call center X1, where X1 is the number of the callcenter cc_\std_X1 - standard deviation of duration of phone calls with call center X1, where X1 is the number of the callcenter com_degree com_degree_total com_count_in_group com_degree_in_group com_score com_group_leader
'com_group_follower', 'com_churned_cnt', 'com_leader_churned_cnt