Objectives
The primary goal is to get the students comfortable enough with pandas
to apply it to their data processing and analysis tasks.
Students will be able to use pandas
to
- perform data analysis
- read in CSV data
- perform data munging and cleaning on dirty data
- apply advanced pandas techniques such as grouping, aggregation and pivot tables
Agenda
- Set up programming environment
- Python Warm Up
- Using Documentation
- Go Deep into Pandas
- Wrap Up & Next Steps
Setup
- Go to tmpnb.org.
- Select
New
>Python 2
to create a Python notebook. - Follow along with me:
>>> import pandas
>>> import sklearn
>>> import matplotlib
Caveat: Don't hard refresh otherwise you'll lose your code. Note: to execute a cell, run Shift + Enter
Python Warm-Up / Review (20)
This problem will give us a review of lists, for loops and lambda functions
Given the following list,
names = ["Michael Fassbender", "Karlie Kloss", "Taylor Swift", "Justin Bieber"]
- print out the names that contain the letter "l"
- turn all of the names lowercase
- sort the list of names alphabetically using the built-in
sorted
function (HINT: Use Google) - sort the list of names by length using the built-in
sorted
function
Warming Up to Documentation (20)
What is a library? A reusable, collection of code that someone else (or you) has already written. Some great built-in libraries:
random
csv
collections
datetime
To use other libraries, we need to be able to:
- understand the documentation of that library
- understand what the inputs and outputs of their functions are
- how to call those functions correctly
Let's start with the random library.
random
library.
The Our first step is to locate the documentation. Google "python random". It should take you here
Let's import the library
>>> import random
>>> dir(random)
How does python know what random
is and how to find the code? Because it comes built-in to the Python language. Other libraries such as pandas
will have to be installed prior to use.
Exercises
- Find the
randint
function in the documentation and explain to your neighbor what it does and how to use it. - Again, with your partner, use the
randint
function to generate a random number between 1 and 125.
as
?
What is Let's see what happens:
>>> import random as rd
>>> random.randint
>>> rd.randint
Pandas (65)
Question What is pandas
?
Here is our first data set. Let's download it and upload it to the datasets folder within the notebook.
Preliminary Exercise
With your partner 0. Download and open the dataset
- What is this dataset about?
- What are some questions you might ask about the data?
Basic Manipulation (75)
- Let's read in the data.
- How do we see what columns are available?
- How do we look at just the head or tail of the dataset?
- How do we look at only a few rows?
- How do we only look at certain columns?
- How do we pull out a column and look at it as a series?
Filtering DataFrames (80)
- How do we look at only those rows that have Status = won
- Exercise: How many accounts have a price greater than $12,000?
- How do we get the maximum value of a certain column?
- Exercise: What is the minimum account price? The mean? The sum? The standard deviation?
Aggregating data (120)
What is the total dollar amount pending?
- How do we add columns?
- Let's add a column called Amount that is equal to Quantity * Price
- Exercise: Let's select just those rows where status is pending and sum up those amounts.
Pivot tables (130)
Question: What are pivot tables? Why are they useful?
Let's take a look at the documentation here.
- Let's pivot using one index.
- Let's pivot on multiple indexes
- Let's reverse those indexes
- Let's specify which values we care about
- Let's specify which columns we want broken down
- Let's specify how we want the values to be aggregated (
aggfunc
) - Let's fill N/A values
- Let's get subtotals
(Creative) Exercise: with a partner, use pivot tables to play around with the data. What pivots do you find particularly interesting or useful for this dataset?
More Pandas (120)
Let's expand our Pandas knowledge and practice it with another dataset.
Download this dataset
Preliminary Exercise
- Take 2 minutes to download this dataset and examine it.
- Take 4 minutes to talk with a partner about the dataset and what kinds of insights you might extract from the data.
- Share with the class.
Exercises (125)
- Read in the dataset from the url
- Solo: How many songs were released in 1981
- What is the earliest release year in the data (Hint: there might be dirty data...)
Data munging (135)
.apply()
to clean up the year.
Using Let's create a new column that contains the clean year
lambda functions (150)
Sometimes the data manipulation we'll want to do on a column will be pretty simple, so we can apply it in place. Say for example we wanted to lowercase an entire column
>>> df["lowercase_title"] = df["Song Clean"].apply(lambda val: val.lower())
lambda function exercises
- Create a new column called "contains_Rock" and the value should be True if the title contains the word "Rock" and False otherwise
- Create a new column called "contains_rock" and the value should be True if the title contains "rock" regardless of case (so it could contain "Rock", "ROCK", "roCK", etc) and False otherwise
value_counts
(175)
Using Pull up the pandas value_counts documentation. We'll answer the following: What are the top 20 songs by play count?
>>> df["ARTIST CLEAN"].value_counts()[:20]
groupby
(180)
Using Pull up the Pandas groupby
documentation.
We'll look at play count grouped by artist.
>>> df.groupby("ARTIST CLEAN")["PlayCount"].mean()
Even More Pandas (190)
Let's look at this dataset from Crunchbase
Preliminary Exercises
- Solo: Read in the data. What is this data about? What are some questions you want to answer about the data?
- What is the max funding total?
- Partner: Anything interesting/strange about the column names?
- Class altogether: Let's rename the column names accordingly.
.describe()
to get some descriptive statistics
Using Pull up the pandas DataFrame describe
documentation.
Exercises (200)
- Construct a pivot table that indexes on region and shows the total funding amounts by region
- What is the average number of funding rounds for companies in NYC? How does that compare to SF?
- What are the top 3 markets with the highest average funding total per company?
- How many companies have Games as a category? What's their average funding total? Using a pivot table, what's the average funding total of those companies that have Games as a category, broken down by Region?
- What is the most popular category of company?
Sharing Our Analysis
- Download the iPython notebook
- Paste it into a GitHub gist
- Copy the URL of the gist and paste it here
- VoilĂ , now you have a shareable analysis!
Next Steps
To continue your Python/Pandas/Data Science education, recommend the following: