Intro to Numpy + Pandas

Lecture 3

Dr. Greg Chism

University of Arizona
INFO 511 - Fall 2024

Intro to NumPy

What is NumPy?

NumPy = Numerical Python
Foundational package for scientific computing
High-performance multidimensional arrays
Tools for working with arrays

Start with

pip install numpy

Why NumPy for data science?

Essential for data processing, manipulation, and analysis.
Underpins advanced data science algorithms implemented in Python.
Fast and memory-efficient with powerful data structures.

NumPy Arrays

import numpy as np

# Creating a simple NumPy array
arr = np.array([1, 2, 3, 4])

# Multidimensional array
multi_arr = np.array([[1, 2, 3], [4, 5, 6]])

# Range of values
range_arr = np.arange(10)

# Array of zeros
zeros_arr = np.zeros((3, 3))

# Array of ones
ones_arr = np.ones((2, 2))

# Identity matrix
identity_matrix = np.eye(3)

arr: [1 2 3 4] 

multi_arr: [[1 2 3]
 [4 5 6]] 

range_arr: [0 1 2 3 4 5 6 7 8 9] 

zeros_arr: [[0. 0. 0.]
 [0. 0. 0.]
 [0. 0. 0.]] 

ones_arr: [[1. 1.]
 [1. 1.]] 

identity_matrix: [[1. 0. 0.]
 [0. 1. 0.]
 [0. 0. 1.]]

Array attributes

# Array dimensions
print("Dimensions:", multi_arr.ndim)

# Shape of array
print("Shape:", multi_arr.shape)

# Size of array
print("Size:", multi_arr.size)

# Data type of array elements
print("Data Type:", multi_arr.dtype)

Dimensions: 2
Shape: (2, 3)
Size: 6
Data Type: int64

Array operations

Arithmetic operations

# Element-wise addition
addition = arr + 2

# Element-wise subtraction
subtraction = arr - 2

# Element-wise multiplication
multiplication = arr * 2

# Element-wise division
division = arr / 2

addition: [3 4 5 6] 

subtraction: [-1  0  1  2] 

multiplication: [2 4 6 8] 

division: [0.5 1.  1.5 2. ]

Aside

Why do my outputs look than different than Python?

Python
My slides

print(addition)
print(subtraction)
print(multiplication)
print(division)

[3 4 5 6]
[-1  0  1  2]
[2 4 6 8]
[0.5 1.  1.5 2. ]

print("addition:", addition, "\n")

print("subtraction:", subtraction, "\n")

print("multiplication:", multiplication, "\n")

print("division:", division, "\n")

addition: [3 4 5 6] 

subtraction: [-1  0  1  2] 

multiplication: [2 4 6 8] 

division: [0.5 1.  1.5 2. ]

Statistical operations

# Sum of elements
total = arr.sum()

# Mean of elements
mean_value = arr.mean()

# Standard deviation
std_dev = arr.std()

# Correlation coefficient
corr = np.corrcoef(multi_arr)

total: 10 

mean_value: 2.5 

std_dev: 1.118033988749895 

corr: [[1. 1.]
 [1. 1.]]

Advanced operations

Reshaping and transposing

# Reshaping an array
reshaped = np.reshape(range_arr, (2, 5))

# Transpose of an array
transposed = multi_arr.T

range_arr: [0 1 2 3 4 5 6 7 8 9] 

reshaped: [[0 1 2 3 4]
 [5 6 7 8 9]] 

multi_arr: [[1 2 3]
 [4 5 6]] 

transposed: [[1 4]
 [2 5]
 [3 6]]

Indexing and slicing

# Accessing a specific element
element = multi_arr[0, 1]

# Slicing a row
row = multi_arr[1, :]

# Slicing a column
column = multi_arr[:, 2]

multi_arr: [[1 2 3]
 [4 5 6]] 

element: [[1 2 3]
 [4 5 6]] 

row: [4 5 6] 

column: [3 6]

Broadcasting

# Broadcasting allows arithmetic operations on arrays of different sizes
broadcasted_addition = multi_arr + np.array([1, 0, 1])

multi_arr: [[1 2 3]
 [4 5 6]] 

broadcasted_addition: [[2 2 4]
 [5 5 7]]

Linear algebra in NumPy

Matrix operations

Dot product: take two equal-length sequences and return a single number

2 • (1, 2, 3) = 2x1 = 2; 2x2 = 4; 2x3 = 6

Matrix multiplication:

(1, 2, 3) • (7, 9, 11) = (1×7 + 2×9 + 3×11) = 58

# Dot product
dot_product = np.dot(arr, arr)

# Matrix multiplication
matrix_mul = np.dot(multi_arr, identity_matrix)

dot_product: 30 

matrix_mul: [[1. 2. 3.]
 [4. 5. 6.]]

Eigenvalues and Eigenvectors

# Eigenvalues and eigenvectors
eigenvalues, eigenvectors = np.linalg.eig(identity_matrix)

eigenvalues: [1. 1. 1.] 

eigenvectors: [[1. 0. 0.]
 [0. 1. 0.]
 [0. 0. 1.]]

NumPy for data science

Application in Algorithms

NumPy arrays are used in various data science algorithms like clustering, classification, and neural networks.

Performance

NumPy operations are implemented in C, which makes them much faster than standard Python.

Conclusion

NumPy is integral to data science and analysis in Python.
It provides efficient and fast operations for array and matrix manipulation.
Understanding NumPy is crucial for implementing and customizing data science algorithms.

Introduction to

What is Pandas?

High-Performance Library: Pandas is a Python library for fast data manipulation.
Core Structures: It introduces DataFrame and Series for data handling.
Data Processing: Ideal for cleaning and analyzing datasets.
Versatile I/O: Offers extensive file format compatibility for data I/O.

Why Pandas for data mining?

Streamlines Data Prep: Optimizes data manipulation for mining readiness.
Built-in Analysis: Includes essential tools for quick data exploration.
Handles Large Data: Efficiently processes and analyzes big datasets.

Pandas data structures

Series

One-dimensional array-like object containing a sequence of values with an associated array of labels, its index.

import pandas as pd

# Creating a Series
ser = pd.Series([4, 7, -5, 3], index=['d', 'b', 'a', 'c'])

# Accessing elements
print("a:", ser['a'])

a: -5

DataFrame

A rectangular table of data with an ordered collection of columns

# Creating a DataFrame
data = {
    'state': ['Ohio', 'Ohio', 'Ohio', 'Nevada', 'Nevada'],
    'year': [2000, 2001, 2002, 2001, 2002],
    'pop': [1.5, 1.7, 3.6, 2.4, 2.9]
}
frame = pd.DataFrame(data)

# Selecting columns
print(frame['state'])

0      Ohio
1      Ohio
2      Ohio
3    Nevada
4    Nevada
Name: state, dtype: object

Index objects

Immutable, can’t be modified by a user

# Index objects
obj = pd.Series(range(3), index=['a', 'b', 'c'])
index = obj.index

print(index)

Index(['a', 'b', 'c'], dtype='object')

Case study: average rent costs

💰📈

Reading & Inspecting Data

First, why .CSVs?

More reproducible - can see changes on GitHub
Simple file structure
Standardized
Non-proprietary (e.g., Excel)

Reading in .CSV files

# Loading data from CSV
df = pd.read_csv('data/rent_avg.csv')

df

	RegionID	SizeRank	RegionName	RegionType	StateName	1/31/15	2/28/15	3/31/15	4/30/15	5/31/15	...	2/28/23	3/31/23	4/30/23	5/31/23	6/30/23	7/31/23	8/31/23	9/30/23	10/31/23	11/30/23
0	102001	0	United States	country	NaN	1266.059583	1272.748070	1281.390109	1291.808026	1301.544232	...	2072.346516	2084.944938	2100.570959	2113.158286	2123.032953	2132.040398	2141.677753	2147.835795	2148.939168	2147.563754
1	394913	1	New York, NY	msa	NY	2233.133615	2255.035180	2272.077073	2291.645864	2297.479956	...	3336.961840	3402.179034	3470.949450	3492.930681	3502.786897	3503.650740	3509.480232	3493.537322	3460.942251	3445.696877
2	753899	2	Los Angeles, CA	msa	CA	2571.296547	2586.050819	2604.348963	2616.104497	2637.303435	...	4073.810818	4100.234089	4134.999768	4148.832674	4182.522537	4202.376930	4230.003153	4224.038689	4213.465460	4209.636932
3	394463	3	Chicago, IL	msa	IL	1504.096116	1510.879827	1522.416987	1534.343702	1547.516576	...	2112.639599	2123.617285	2145.692631	2163.843271	2183.541315	2196.506806	2213.427631	2225.237153	2225.798038	2221.960828
4	394514	4	Dallas, TX	msa	TX	1363.557414	1371.136919	1381.114797	1394.643185	1408.025840	...	2222.442779	2229.474165	2239.462332	2253.004851	2267.408242	2280.056798	2285.063932	2284.569053	2274.785931	2277.403767
...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...
467	753871	811	Breckenridge, CO	msa	CO	NaN	NaN	NaN	NaN	NaN	...	3848.649613	4327.112762	4414.482838	4438.619592	4535.664564	4599.237795	4558.170264	4667.043338	4671.081209	4472.222222
468	394751	821	Kirksville, MO	msa	MO	NaN	NaN	NaN	NaN	NaN	...	NaN	NaN	NaN	NaN	NaN	NaN	NaN	1000.733430	981.701456	951.458333
469	753923	849	The Dalles, OR	msa	OR	NaN	NaN	NaN	NaN	NaN	...	NaN	NaN	NaN	NaN	NaN	NaN	1757.739178	1788.943167	1885.904065	1838.888889
470	394584	863	Fallon, NV	msa	NV	NaN	NaN	NaN	NaN	NaN	...	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	1295.000000
471	394996	915	Portales, NM	msa	NM	NaN	NaN	NaN	NaN	NaN	...	NaN	NaN	NaN	NaN	1122.195307	1101.628147	1057.654564	1071.572024	1046.671512	1099.305556

472 rows × 112 columns

Inspecting data

Any issues?

Head
Tail
Data types
Describe
Stats
Unique values

df.head()

	RegionID	SizeRank	RegionName	RegionType	StateName	1/31/15	2/28/15	3/31/15	4/30/15	5/31/15	...	2/28/23	3/31/23	4/30/23	5/31/23	6/30/23	7/31/23	8/31/23	9/30/23	10/31/23	11/30/23
0	102001	0	United States	country	NaN	1266.059583	1272.748070	1281.390109	1291.808026	1301.544232	...	2072.346516	2084.944938	2100.570959	2113.158286	2123.032953	2132.040398	2141.677753	2147.835795	2148.939168	2147.563754
1	394913	1	New York, NY	msa	NY	2233.133615	2255.035180	2272.077073	2291.645864	2297.479956	...	3336.961840	3402.179034	3470.949450	3492.930681	3502.786897	3503.650740	3509.480232	3493.537322	3460.942251	3445.696877
2	753899	2	Los Angeles, CA	msa	CA	2571.296547	2586.050819	2604.348963	2616.104497	2637.303435	...	4073.810818	4100.234089	4134.999768	4148.832674	4182.522537	4202.376930	4230.003153	4224.038689	4213.465460	4209.636932
3	394463	3	Chicago, IL	msa	IL	1504.096116	1510.879827	1522.416987	1534.343702	1547.516576	...	2112.639599	2123.617285	2145.692631	2163.843271	2183.541315	2196.506806	2213.427631	2225.237153	2225.798038	2221.960828
4	394514	4	Dallas, TX	msa	TX	1363.557414	1371.136919	1381.114797	1394.643185	1408.025840	...	2222.442779	2229.474165	2239.462332	2253.004851	2267.408242	2280.056798	2285.063932	2284.569053	2274.785931	2277.403767

5 rows × 112 columns

df.tail()

	RegionID	SizeRank	RegionName	RegionType	StateName	1/31/15	2/28/15	3/31/15	4/30/15	5/31/15	...	2/28/23	3/31/23	4/30/23	5/31/23	6/30/23	7/31/23	8/31/23	9/30/23	10/31/23	11/30/23
467	753871	811	Breckenridge, CO	msa	CO	NaN	NaN	NaN	NaN	NaN	...	3848.649613	4327.112762	4414.482838	4438.619592	4535.664564	4599.237795	4558.170264	4667.043338	4671.081209	4472.222222
468	394751	821	Kirksville, MO	msa	MO	NaN	NaN	NaN	NaN	NaN	...	NaN	NaN	NaN	NaN	NaN	NaN	NaN	1000.733430	981.701456	951.458333
469	753923	849	The Dalles, OR	msa	OR	NaN	NaN	NaN	NaN	NaN	...	NaN	NaN	NaN	NaN	NaN	NaN	1757.739178	1788.943167	1885.904065	1838.888889
470	394584	863	Fallon, NV	msa	NV	NaN	NaN	NaN	NaN	NaN	...	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	1295.000000
471	394996	915	Portales, NM	msa	NM	NaN	NaN	NaN	NaN	NaN	...	NaN	NaN	NaN	NaN	1122.195307	1101.628147	1057.654564	1071.572024	1046.671512	1099.305556