/lab-cleaning-numerical-data

Primary LanguageJupyter Notebook

logo_ironhack_blue 7

Lab | Cleaning numerical data

For this lab, we will be using the dataset in the Customer Analysis Business Case. This dataset can be found in files_for_lab folder.

Context

An auto insurance company has collected some data about its customers including their demographics, education, employment, policy details, vehicle information on which insurance policy is, and claim amounts. You will help the senior management with some business questions that will help them to better understand their customers, improve their services, and improve profitability.

Some business Objectives:

  • Retain customers,
  • Analyze relevant customer data,
  • Develop focused customer retention programs.
  • Based on the analysis, take targeted actions to increase profitable customer response, retention, and growth.

Data Analysis Process

Remember the process:

  • Case Study
  • Get data
  • Cleaning/Wrangling/EDA
  • Processing Data
  • Modeling -Validation
  • Reporting

Instructions

  1. Import the necessary libraries.

  2. Load the we_fn_use_c_marketing_customer_value_analysis.csv into the variable customer_df

  3. First look at its main features (head, shape, info). NOTE The target column is total_claim_amount

  4. Rename the columns so they follow the PE8 (snake case) (lower_case_with_underscores)

  5. Change effective to date column to datetime format.

  6. Create a NEW dataframe using only the numerical data called numerical_df. Use this for the following questions.

  7. Define a function that differentiates between continuous and discrete variables. Hint: The number of unique values might be useful. Store continuous data into a continuous_df variable and do the same for discrete_df

  8. Plot a correlation matrix, comment what you see.

  9. Create a function to plot every discrete variable. Do the same with the continuous variables (Be Careful, you may need to change the plot type to one better suited for continuous data!)

  10. Comment what you can see in the plots.

  11. Look for outliers in the continuous variables. (HINT: There’s a good plot to do that!)

  12. Did you find outliers? Comment what you will do with them. If you decide to remove them, do that now. Also, remove them from your main dataframe.

  13. Check all columns for NaN values. Decide what (if anything) you will need to do with them, do that now.

  14. Save the numerical_df to csv as numerical.csv You will use this file again this week.