MovieLens Case Study with Python
Problem Objective :
Here, we ask you to perform the analysis using the Exploratory Data Analysis technique. You need to find features affecting the ratings of any particular movie and build a model to predict the movie ratings. Domain: Entertainment
Dataset Description :
These files contain 1,000,209 anonymous ratings of approximately 3,900 movies made by 6,040 MovieLens users who joined MovieLens in 2000.
Analysis Tasks to be performed:
1. Import the three datasets
In [1]:
import numpy as np #import the numpy library for numerical operations
import pandas as pd #import the pandas library for data wrangling operations
In [2]:
colnames_movies = ['MovieID','Title','Genres']
df_movies = pd.read_csv('movies.dat', delimiter='::', engine='python', names=colnames_movies) #open the movies.dat datafile
df_movies.head()
Out[2]:
In [3]:
df_movies.shape #movies.dat has 3882 observations & 3 variables
Out[3]:
In [4]:
colnames_ratings=['UserID','MovieID','Rating','Timestamp']
df_ratings = pd.read_csv('ratings.dat', delimiter='::', engine='python', names=colnames_ratings) #open the ratings.dat datafile
df_ratings.head()
Out[4]:
In [5]:
df_ratings.shape
Out[5]:
In [6]:
colnames_users=['UserID','Gender','Age','Occupation','Zip-code']
df_users = pd.read_csv('users.dat', delimiter='::', engine='python', names=colnames_users) #open the users.dat datafile
df_users.head()
Out[6]:
In [7]:
df_users.shape
Out[7]:
In [8]:
#merge the ratings dataframe with movies dataframe on column MovieID
df_ratings_movies = pd.merge(df_ratings, df_movies, how='inner', on='MovieID', left_on=None, right_on=None,
left_index=False, right_index=False, sort=True,
suffixes=('_x', '_y'), copy=True, indicator=False)
df_ratings_movies.shape
Out[8]:
In [9]:
df_ratings_movies.head()
Out[9]:
In [10]:
#now merge the users dataframe with the ratings_movies dataframe
df_ratings_movies_users = pd.merge(df_ratings_movies, df_users, how='inner', on='UserID', left_on=None, right_on=None,
left_index=False, right_index=False, sort=True,
suffixes=('_x', '_y'), copy=True, indicator=False)
df_ratings_movies_users.shape
Out[10]:
In [11]:
df_ratings_movies_users.head()
Out[11]:
In [12]:
df_master_data = df_ratings_movies_users[['MovieID', 'Title', 'UserID', 'Age', 'Gender', 'Occupation', 'Rating','Genres']]
In [13]:
df_master_data.head()
Out[13]:
In [14]:
df_master_data.shape
Out[14]:
Explore the datasets using visual representations (graphs or tables), also include your comments on the following:¶
1. User Age Distribution¶
In [15]:
from matplotlib import pyplot as plt #import the matpllotlib pyplot subpackage
In [16]:
plt.hist(df_master_data.Age, bins = 7)
plt.show()
#below histogram shows that age group 25 (i.e. 25-34 years) have voted the maximum times
2.User rating of the movie “Toy Story”¶
In [17]:
df_bytitle = df_master_data.groupby('Title')
In [18]:
df_bytitle.get_group('Toy Story (1995)')
Out[18]:
In [19]:
#Get the average ratings for all the movies
df_master_data.groupby(['Title'])['Rating'].mean()
Out[19]:
In [20]:
#get the average user rating for the movie - Toy Story (1995)
np.average(df_bytitle.get_group('Toy Story (1995)').Rating)
#shows that Toy Story (1995) has an average user rating of 4.14
Out[20]:
3.Top 25 movies by viewership rating¶
In [21]:
Top25_Movies = df_master_data.groupby(['MovieID','Title'])['Rating'].mean().sort_values(ascending=False)
In [22]:
Top25_Movies
Out[22]:
In [ ]:
4.Find the ratings for all the movies reviewed by for a particular user of user id = 2696¶
In [23]:
df_byuserID = df_master_data.groupby('UserID')
In [24]:
df_byuserID.get_group(2696) #output is not much impressive. Let us try an alternate method
Out[24]:
In [25]:
#alternative method
df_master_data[df_master_data['UserID'] == 2696][['Title','Rating']]
Out[25]:
Feature Engineering¶
Find out all the unique genres (Hint: split the data in column genre making a list and then process the data to find out only the unique categories of genres)
In [26]:
df_master_data.head()
Out[26]:
In [27]:
#below is an inefficient way of getting the genres
genres = ['Action','Adventure','Animation','Children''s','Comedy','Crime','Documentary','Drama','Fantasy','Film-Noir',
'Horror','Musical','Mystery','Romance','Sci-Fi','Thriller','War','Western']
Create a separate column for each genre category with a one-hot encoding ( 1 and 0) whether or not the movie belongs to that genre.
In [28]:
#below is an efficient way by python. It will add dummy columns
df_genres = df_master_data['Genres'].str.get_dummies("|")
In [29]:
df_genres
Out[29]:
In [30]:
df_master_data = pd.merge(df_master_data, df_genres, how='inner', left_on=None, right_on=None,
left_index=True, right_index=True, sort=True,
suffixes=('_x', '_y'), copy=True, indicator=False) #Merge by Index; however it is not a good practice
In [31]:
df_master_data.head()
Out[31]:
In [32]:
#This is a classification problem.
#We need to predict the Rating of a movie and find out variables that are significant
In [33]:
df_master_data['Rating'].hasnans
Out[33]:
In [34]:
df_master_data['Rating']=df_master_data.Rating.astype('int') #convert Rating to integer
In [35]:
df_master_data['Age']=df_master_data.Age.astype('int') #convert Age to integer
In [36]:
df_master_data['Occupation']=df_master_data.Occupation.astype('int') #convert Occupation to integer
Model Selection Process¶
In [37]:
from sklearn.model_selection import train_test_split #used to train and test existing dataset
Perform Exploratory Data Analysis (EDA) for the Master Data Set
In [38]:
# visualization
import seaborn as sns
import matplotlib.pyplot as plt
%matplotlib inline
In [39]:
# machine learning
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.svm import SVC, LinearSVC
from sklearn.ensemble import RandomForestClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.naive_bayes import GaussianNB
from sklearn.linear_model import Perceptron
from sklearn.linear_model import SGDClassifier
from sklearn.tree import DecisionTreeClassifier
In [40]:
#Visualize user age distribution
df_master_data['Age'].value_counts().plot(kind='barh',alpha=0.7,figsize=(10,10))
plt.show()
In [41]:
df_master_data.Age.plot.hist(bins=25)
plt.title("Distribution of users' ages")
plt.ylabel('count of users')
plt.xlabel('Age')
Out[41]:
In [42]:
#Visualize overall rating by users
df_master_data['Rating'].value_counts().plot(kind='bar',alpha=0.7,figsize=(10,10))
plt.show()
Perform Machine Learning Algorithms
In [43]:
#Use the following features:movie id,age,occupation
features = df_master_data[['MovieID','Age','Occupation']].values
In [44]:
#Use rating as label i.e response variable
labels = df_master_data[['Rating']].values
In [45]:
#Create train and test data set
train, test, train_labels, test_labels = train_test_split(features,labels,test_size=0.25,random_state=1)
In [46]:
#Create a histogram for movie
df_master_data.Age.plot.hist(bins=25)
plt.title("Movie & Rating")
plt.ylabel('MovieID')
plt.xlabel('Ratings')
Out[46]:
In [47]:
#Create a histogram for age
df_master_data.Age.plot.hist(bins=25)
plt.title("Age & Rating")
plt.ylabel('Age')
plt.xlabel('Ratings')
Out[47]:
In [48]:
#Create a histogram for occupation
df_master_data.Age.plot.hist(bins=25)
plt.title("Occupation & Rating")
plt.ylabel('Occupation')
plt.xlabel('Ratings')
Out[48]:
In [49]:
# Logistic Regression
logreg = LogisticRegression()
logreg.fit(train, train_labels)
Y_pred = logreg.predict(test)
acc_log = round(logreg.score(train, train_labels) * 100, 2)
acc_log
Out[49]:
In [50]:
# K Nearest Neighbors Classifier
knn = KNeighborsClassifier(n_neighbors = 3)
knn.fit(train, train_labels)
Y_pred = knn.predict(test)
acc_knn = round(knn.score(train, train_labels) * 100, 2)
acc_knn
Out[50]:
In [51]:
# Gaussian Naive Bayes
gaussian = GaussianNB()
gaussian.fit(train, train_labels)
Y_pred = gaussian.predict(test)
acc_gaussian = round(gaussian.score(train, train_labels) * 100, 2)
acc_gaussian
Out[51]:
In [52]:
# Perceptron
perceptron = Perceptron()
perceptron.fit(train, train_labels)
Y_pred = perceptron.predict(test)
acc_perceptron = round(perceptron.score(train, train_labels) * 100, 2)
acc_perceptron
Out[52]:
In [54]:
# Decision Tree
decision_tree = DecisionTreeClassifier()
decision_tree.fit(train, train_labels)
Y_pred = decision_tree.predict(test)
acc_decision_tree = round(decision_tree.score(train, train_labels) * 100, 2)
acc_decision_tree
Out[54]:
In [58]:
models = pd.DataFrame({
'Model': ['KNN', 'Logistic Regression',
'Naive Bayes', 'Perceptron',
'Decision Tree'],
'Score': [acc_knn, acc_log,
acc_gaussian, acc_perceptron, acc_decision_tree]})
models.sort_values(by='Score', ascending=False)
Out[58]:
from the above accuracy scores, Decision Tree seems to be the most suitable Model with 55.68% accuracy
In [ ]:
Comments
Post a Comment