Project 2

Exploratory Data Analysis (EDA)

Your hometown mayor just created a new data analysis team to give policy advice, and the administration recruited you via LinkedIn to join it. Unfortunately, due to budget constraints, for now the “team” is just you…

The mayor wants to start a new initiative to move the needle on one of two separate issues: high school education outcomes, or drug abuse in the community.

Also unfortunately, that is the entirety of what you’ve been told. And the mayor just went on a lobbyist-funded fact-finding trip in the Bahamas. In the meantime, you got your hands on two national datasets: one on SAT scores by state, and one on drug use by age. Start exploring these to look for useful patterns and possible hypotheses!

This project is focused on exploratory data analysis, aka “EDA”. EDA is an essential part of the data science analysis pipeline. Failure to perform EDA before modeling is almost guaranteed to lead to bad models and faulty conclusions. What you do in this project are good practices for all projects going forward, especially those after this bootcamp!

This lab includes a variety of plotting problems. Much of the plotting code will be left up to you to find either in the lecture notes, or if not there, online. There are massive amounts of code snippets either in documentation or sites like Stack Overflow that have almost certainly done what you are trying to do.

Get used to googling for code! You will use it every single day as a data scientist, especially for visualization and plotting.

Package imports

import numpy as np
import scipy.stats as stats
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns




# this line tells jupyter notebook to put the plots in the notebook rather than saving them to file.
%matplotlib inline

# this line makes plots prettier on mac retina screens. If you don't have one it shouldn't do anything.
%config InlineBackend.figure_format = 'retina'

1. Load the sat_scores.csv dataset and describe it

NOTE: All CSVs are in the same directory as this notebook

# Gave the dataset a variable name "sat"
sat = 'sat_scores.csv'

1.1 Make a pandas DataFrame object with pandas .read_csv() function

Take a look at the .dtypes attribute in the DataFrame.

# Loaded the data set by using pd.read_csv and called the sat variable.
# Gave the data set the variable name "sat_scores".
# Called the variable "sat_scores" to view the data.

sat_scores = pd.read_csv(sat)

sat_scores

# Used .dtypes to see the types of data.

sat_scores.dtypes

State     object
Rate       int64
Verbal     int64
Math       int64
dtype: object

1.2 Look at the first ten rows of the DataFrame: what does our data describe?

From now on, use the DataFrame loaded from the file using the .read_csv() function.

Use the .head(num) built-in DataFrame function, where num is the number of rows to print out.

You are not given a “codebook” with this data, so you will have to make some (very minor) inference.

# used .head(10) in order to view the first 10 rows of the DataFrame.

sat_scores.head(10)

2. Create a “data dictionary” based on the data

A data dictionary is an object that describes your data. This should contain the name of each variable (column), the type of the variable, your description of what the variable is, and the shape (rows and columns) of the entire dataset.

# Used .columns to view the names of all the columns in the DataFrame.

sat_scores.columns

Index(['State', 'Rate', 'Verbal', 'Math'], dtype='object')

# Used .describe to see the attributes of the DataFrame

sat_scores.describe()

# Used .shape to show the shape (rows x columns) of the DataFrame.

sat_scores.shape

(52, 4)

# Used .dytpes to show the types(object, integer, float, e.g.) of the DataFrame.

sat_scores.dtypes

State     object
Rate       int64
Verbal     int64
Math       int64
dtype: object

# Created a Data dictionary after calling .describe, .shape and .dtypes

# Data Dictionary:

# Variable  Definition        Type
# State      State             object
# Rate       Rate              int64
# Verbal     Verbal Score      int64
# Math       Math Score        int64

# The shape of the DataFrame is 52 rows by 4 columns

3. Plot the data using seaborn

3.1 Using seaborn’s distplot, plot the distributions for each of Rate, Math, and Verbal

Set the keyword argument kde=False. This way you can actually see the counts within bins. You can adjust the number of bins to your liking.

Please read over the distplot documentation to learn about the arguments and fine-tune your chart if you want.

# Used Seaborn's distplot with each variable separately. This one is for "Rate".

sns.distplot(sat_scores["Rate"], kde=False)

<matplotlib.axes._subplots.AxesSubplot at 0x1186505c0>

# Used Seaborn's distplot with each variable separately. This one is for "Verbal".

sns.distplot(sat_scores["Verbal"], kde=False)

<matplotlib.axes._subplots.AxesSubplot at 0x11872db70>

# Used Seaborn's distplot with each variable separately. This one is for "Math".

sns.distplot(sat_scores["Math"], kde=False)

<matplotlib.axes._subplots.AxesSubplot at 0x1187430b8>

3.2 Using seaborn’s pairplot, show the joint distributions for each of Rate, Math, and Verbal

Explain what the visualization tells you about your data.

Please read over the pairplot documentation to fine-tune your chart.

# Used Seaborn's pairplot with the 3 variables ("Rate", "Math", "Verbal") in one double bracket.

sns.pairplot(sat_scores[["Rate", "Math", "Verbal"]])

<seaborn.axisgrid.PairGrid at 0x1185c1550>

4. Plot the data using built-in pandas functions.

Pandas is very powerful and contains a variety of nice, built-in plotting functions for your data. Read the documentation here to understand the capabilities:

http://pandas.pydata.org/pandas-docs/stable/visualization.html

4.1 Plot a stacked histogram with Verbal and Math using pandas

# Plotted Stacked Histogram for Verbal and Math
sat_scores[['Math', 'Verbal']].plot.hist(grid=True,by=None)

<matplotlib.axes._subplots.AxesSubplot at 0x11b1632e8>

4.2 Plot Verbal and Math on the same chart using boxplots

What are the benefits of using a boxplot as compared to a scatterplot or a histogram?

What’s wrong with plotting a box-plot of Rate on the same chart as Math and Verbal?

# Plotted Box Blot for Verbal and Math
sat_scores[['Math', 'Verbal']].plot.box(grid=True,by=None)

<matplotlib.axes._subplots.AxesSubplot at 0x119e24278>

4.3 Plot Verbal, Math, and Rate appropriately on the same boxplot chart

Think about how you might change the variables so that they would make sense on the same chart. Explain your rationale for the choices on the chart. You should strive to make the chart as intuitive as possible.

5. Create and examine subsets of the data

For these questions you will practice masking in pandas. Masking uses conditional statements to select portions of your DataFrame (through boolean operations under the hood.)

Remember the distinction between DataFrame indexing functions in pandas:

.iloc[row, col] : row and column are specified by index, which are integers
.loc[row, col]  : row and column are specified by string "labels" (boolean arrays are allowed; useful for rows)

For detailed reference and tutorial make sure to read over the pandas documentation:

http://pandas.pydata.org/pandas-docs/stable/indexing.html

5.1 Find the list of states that have an average Verbal score greater than the average of Verbal scores across the entire dataset

How many states are above the mean? What does this tell you about the distribution of Verbal scores?

# Gave the mean of the "Verbal" column throughout the total DataFrame the variable name "sat_verbal_mean".
sat_verbal_mean = sat_scores['Verbal'].mean()

# Gave the mean of the states, whose mean was greater than the mean of the entire DataFrame, the variable name "state_verbal_mean"
state_verbal_mean = sat_scores[sat_scores['Verbal'] > sat_verbal_mean]

# Only showed the Index of the State, the State and the Mean Verbal score.
state_verbal_mean[["State" , "Verbal"]]

# 24 States are above the Mean, which means 26 States are at or below the mean.  That means the data is normally distributed.

state_verbal_mean.count()

State     24
Rate      24
Verbal    24
Math      24
dtype: int64

5.2 Find the list of states that have a median Verbal score greater than the median of Verbal scores across the entire dataset

How does this compare to the list of states greater than the mean of Verbal scores? Why?

# Gave the median of the "Verbal" column throughout the total DataFrame the variable name "sat_verbal_median".
sat_verbal_median = sat_scores['Verbal'].median()

# Gave the median of the states, whose median was greater than the median of the entire DataFrame, the variable name "state_verbal_median"
state_verbal_median = sat_scores[sat_scores['Verbal'] > sat_verbal_median]

# Only showed the Index of the State, the State and the Median Verbal score.
state_verbal_median[["State" , "Verbal"]]

# Washington & West Virginia appear on the Median list, but not on the Mean List. They have outliers in their data.

state_verbal_median.count()

State     26
Rate      26
Verbal    26
Math      26
dtype: int64

5.3 Create a column that is the difference between the Verbal and Math scores

Specifically, this should be Verbal - Math.

# Created a column called and "Verbal_Less_Math" and assigned it to "sat_scores" Assigned the formula of Verbal scores - Math scores.

sat_scores['Verbal_Less_Math'] = sat_scores['Verbal'] - sat_scores['Math']
sat_scores

5.4 Create two new DataFrames showing states with the greatest difference between scores

1. Your first DataFrame should be the 10 states with the greatest gap between Verbal and Math scores where Verbal is greater than Math. It should be sorted appropriately to show the ranking of states.
2. Your second DataFrame will be the inverse: states with the greatest gap between Verbal and Math such that Math is greater than Verbal. Again, this should be sorted appropriately to show rank.
3. Print the header of both variables, only showing the top 3 states in each.

# Used .sort_values and assigned it to "Verbal_Less_Math". Made it sort descending by the top 10 States that have a verbal - math score difference 

sat_scores.sort_values('Verbal_Less_Math',ascending=False).head(10)

6. Examine summary statistics

Checking the summary statistics for data is an essential step in the EDA process!

6.1 Create the correlation matrix of your variables (excluding State).

• Use seaborn’s .heatmap method to add some color to the matrix
• Set annot=True

# Created the correlation matrix using .corr and excluded the "State" column.

corr_sat_scores = sat_scores[['Math', 'Verbal']].corr()
corr_sat_scores

# Created a heatmap with the correlation matrix.

sns.heatmap(corr_sat_scores, annot=True)

<matplotlib.axes._subplots.AxesSubplot at 0x119e6a630>

6.2 Use pandas’ .describe() built-in function on your DataFrame

Write up what each of the rows returned by the function indicate.

# Count counts the rows
# Mean returns the average of each column
# STD returns the standard deviation of each column
# Min returns the minimun value of each column
# Max returns the maximun value of each column
# The 25%, 50%, and 75% are percentiles : array-like, optional
# The percentiles to include in the output. Should all be in the interval [0, 1]. By default percentiles is [.25, .5, .75], returning the 25th, 50th, and 75th percentiles.

sat_scores.describe()

6.3 Assign and print the covariance matrix for the dataset

1. Describe how the covariance matrix is different from the correlation matrix.
2. What is the process to convert the covariance into the correlation?
3. Why is the correlation matrix preferred to the covariance matrix for examining relationships in your data?

# Covariance
# You tend to use the covariance matrix when the variable scales are similar and the correlation matrix when variables are on different scales.

sat_scores.cov()

7. Performing EDA on “drug use by age” data.

You will now switch datasets to one with many more variables. This section of the project is more open-ended - use the techniques you practiced above!

We’ll work with the “drug-use-by-age.csv” data, sourced from and described here: https://github.com/fivethirtyeight/data/tree/master/drug-use-by-age.

7.1

Load the data using pandas. Does this data require cleaning? Are variables missing? How will this affect your approach to EDA on the data?

# Loaded the dataset

drug_use = pd.read_csv('drug-use-by-age.csv')
drug_use

# Cleaned up values

drug_use.replace(to_replace="-", value=0.0, inplace=True)

# viewed the head of the dataset
drug_use.head()

# The shape of the dataframe is 17 rows by 28 columns
drug_use.shape

(17, 28)

# all the objects have been converted to float64
drug_use.dtypes

age                         object
n                            int64
alcohol-use                float64
alcohol-frequency          float64
marijuana-use              float64
marijuana-frequency        float64
cocaine-use                float64
cocaine-frequency           object
crack-use                  float64
crack-frequency             object
heroin-use                 float64
heroin-frequency            object
hallucinogen-use           float64
hallucinogen-frequency     float64
inhalant-use               float64
inhalant-frequency          object
pain-releiver-use          float64
pain-releiver-frequency    float64
oxycontin-use              float64
oxycontin-frequency         object
tranquilizer-use           float64
tranquilizer-frequency     float64
stimulant-use              float64
stimulant-frequency        float64
meth-use                   float64
meth-frequency              object
sedative-use               float64
sedative-frequency         float64
dtype: object

# Change objects to floats


drug_use['cocaine-frequency'] = drug_use['cocaine-frequency'].astype(float) 
drug_use['crack-frequency'] = drug_use['crack-frequency'].astype(float) 
drug_use['heroin-frequency'] = drug_use['heroin-frequency'].astype(float) 
drug_use['inhalant-frequency'] = drug_use['inhalant-frequency'].astype(float) 
drug_use['oxycontin-frequency'] = drug_use['oxycontin-frequency'].astype(float) 
drug_use['meth-frequency'] = drug_use['meth-frequency'].astype(float) 

drug_use.dtypes

age                         object
n                            int64
alcohol-use                float64
alcohol-frequency          float64
marijuana-use              float64
marijuana-frequency        float64
cocaine-use                float64
cocaine-frequency          float64
crack-use                  float64
crack-frequency            float64
heroin-use                 float64
heroin-frequency           float64
hallucinogen-use           float64
hallucinogen-frequency     float64
inhalant-use               float64
inhalant-frequency         float64
pain-releiver-use          float64
pain-releiver-frequency    float64
oxycontin-use              float64
oxycontin-frequency        float64
tranquilizer-use           float64
tranquilizer-frequency     float64
stimulant-use              float64
stimulant-frequency        float64
meth-use                   float64
meth-frequency             float64
sedative-use               float64
sedative-frequency         float64
dtype: object

7.2 Do a high-level, initial overview of the data

Get a feel for what this dataset is all about.

Use whichever techniques you’d like, including those from the SAT dataset EDA. The final response to this question should be a written description of what you infer about the dataset.

Some things to consider doing:

• Look for relationships between variables and subsets of those variables’ values
• Derive new features from the ones available to help your analysis
• Visualize everything!

drug_use[["alcohol-use", "marijuana-use", "cocaine-use", "crack-use","heroin-use","hallucinogen-use","inhalant-use","pain-releiver-use", "oxycontin-use","tranquilizer-use","stimulant-use","meth-use","sedative-use"]].plot.box(rot=45)

<matplotlib.axes._subplots.AxesSubplot at 0x11a662ba8>

7.3 Create a testable hypothesis about this data

Requirements for the question:

1. Write a specific question you would like to answer with the data (that can be accomplished with EDA).
2. Write a description of the “deliverables”: what will you report after testing/examining your hypothesis?
3. Use EDA techniques of your choice, numeric and/or visual, to look into your question.
4. Write up your report on what you have found regarding the hypothesis about the data you came up with.

Your hypothesis could be on:

• Difference of group means
• Correlations between variables
• Anything else you think is interesting, testable, and meaningful!

Important notes:

You should be only doing EDA relevant to your question here. It is easy to go down rabbit holes trying to look at every facet of your data, and so we want you to get in the practice of specifying a hypothesis you are interested in first and scoping your work to specifically answer that question.

Some of you may want to jump ahead to “modeling” data to answer your question. This is a topic addressed in the next project and you should not do this for this project. We specifically want you to not do modeling to emphasize the importance of performing EDA before you jump to statistical analysis.

** Question and deliverables**

…

# Code

drug_use.plot('age', 'marijuana-use', kind='bar')
plt.ylabel('Marijuana Use')
plt.title('Marijuana Use')
plt.plot()

drug_use.plot('age', 'marijuana-frequency', kind='bar')
plt.ylabel('Marijuana Frequency')
plt.title('Marijuana Frequency')
plt.plot()

# Marijuana use increases from age 12 to 20, then almost symetrically decreases until 65+.
# Marijuana frequency rapidly increases from age 12 to 20, levels off until 29. It spikes up from 30-34
# and then settles back down from age 50 to 65+. 

[]

Report

…

8. Introduction to dealing with outliers

Outliers are an interesting problem in statistics, in that there is not an agreed upon best way to define them. Subjectivity in selecting and analyzing data is a problem that will recur throughout the course.

1. Pull out the rate variable from the SAT dataset.
2. Are there outliers in the dataset? Define, in words, how you numerically define outliers.
3. Print out the outliers in the dataset.
4. Remove the outliers from the dataset.
5. Compare the mean, median, and standard deviation of the “cleaned” data without outliers to the original. What is different about them and why?

9. Percentile scoring and spearman rank correlation

9.1 Calculate the spearman correlation of sat Verbal and Math

1. How does the spearman correlation compare to the pearson correlation?
2. Describe clearly in words the process of calculating the spearman rank correlation.
   • Hint: the word “rank” is in the name of the process for a reason!

9.2 Percentile scoring

Look up percentile scoring of data. In other words, the conversion of numeric data to their equivalent percentile scores.

http://docs.scipy.org/doc/numpy-dev/reference/generated/numpy.percentile.html

http://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.percentileofscore.html

1. Convert Rate to percentiles in the sat scores as a new column.
2. Show the percentile of California in Rate.
3. How is percentile related to the spearman rank correlation?

9.3 Percentiles and outliers

1. Why might percentile scoring be useful for dealing with outliers?
2. Plot the distribution of a variable of your choice from the drug use dataset.
3. Plot the same variable but percentile scored.
4. Describe the effect, visually, of coverting raw scores to percentile.