What is Data Science?

-

Data Science

It requires below skills:

- Domain Knowledge (Finance/Banking/Retail/Supply Chain/Healthcare/Insurance)

- Statistical Skills (Measures of Central Tendencies/Non-Central Tendencies)

- Programming (Engineering Skills)

- Mathematical Skills (Average)

Purpose of Data Science is to use modern technology, processing power to use above skills.

Statistical knowledge has been existing for years, the algorithms are existing for decades.

However, the term Data Science is merely combining technology with existing statistical skills.

Data Science enables business problem solving & Decision making process.

Role of a Data Scientist:

- Ask "right" Questions

- Acquire Data from various Organizational Sources (File Handling/File Formats)

- Structured Data (tabular/csv)

- Semi-structured Data (e.g. json/html/xml file format)

- Unstructured Data (Text files/Audio/Video files)

- Web Scraping (Getting Data through Web/View Source HTML)

- Wrangle Data

- Data Cleansing (NULL Removal/Duplicate Removals etc.)

- Data Manipulation/Transformation (bringing data in structured format)

-Impute missing Values

-Create New Variables

        - Challenges: - Takes upto 70% of Data Scientist's time

- Unexpected Data Format (eg.XML instead of CSV)

- Erroneous Data

- Voluminous data to be manipulated

- Classifying data into linear (eg.Age) or clustered (eg.Age range/slab)

- Explore Data

- Data Discovery

- Data Pattern (Plot the Data/Bar Charts/Histogram/Scatter Plot etc.)

- Challenges:

-Determining relationship between Observation, Feature & Response

Observation    : Rows in Table

Variables     : Columns in Table

Features (Variable): Column(s) which are thrown into Models. (Also called Independent Variables)

Response (Variable): Column which is to be studied. (Also called Dependent Variable)  

- Apply Mathematical/Statistical Model

- Based on the nature of problem we are trying to solve.

- Regression/Classification etc.

- Machine Learning

- Wrangled Data can be input to Machine Learning Algorithm to improve results

- Present Outcomes/Communicate to Stakeholders/Conclude or Prediction

- Data Visualization (Tableau/Dashboards/Charts)

- Data Report

- Data Products (Recommendations based on previous Data/Choices e.g. Youtube/Facebook)


Comments

Post a Comment

Popular posts from this blog

Types of Data Analysis

-
Descriptive: - We just describle the existing Data merely by observing the Data. - We do not predict or forecast anything using the data - It is Exploratory Data Analysis (EDA) Predictive: - Classification based on existing Data Sample eg. - Titanic Survival based on variables viz. age/no.of family members/boarding point - Banking: Will the person default Loan based on variables viz. Age/Annual Income/CIBIL Score - Predict future rows based on existing data - Test Model (80:20 approach) - Forecast based on historic data (Forecasting is time-based) Prescriptive: - Recommend changes to the variables based on study of the Data (identifying relation between variables) - Provide Leavers
Read more

MovieLens Case Study with Python

-
Image
Problem Objective : Here, we ask you to perform the analysis using the Exploratory Data Analysis technique. You need to find features affecting the ratings of any particular movie and build a model to predict the movie ratings. Domain: Entertainment Dataset Description : These files contain 1,000,209 anonymous ratings of approximately 3,900 movies made by 6,040 MovieLens users who joined MovieLens in 2000. Analysis Tasks to be performed: 1. Import the three datasets In [1]: import numpy as np #import the numpy library for numerical operations import pandas as pd #import the pandas library for data wrangling operations In [2]: colnames_movies = [ 'MovieID' , 'Title' , 'Genres' ] df_movies = pd . read_csv ( 'movies.dat' , delimiter = '::' , engine = 'python' , names = colnames_movies ) #open the movies.dat datafile df_movies . head () Out[2]: MovieID Title Genres 0 1 Toy Story (1995) Animation|Children's|Comedy 1 2 ...
Read more

Data Pre-processing for Machine Learning

-
Image
  Traditional Computer Computation: ¶ (Data & Program) -> Computer -> Output Machine Learning: ¶ (Data & Output) -> Computer -> Program/Model The model that is generated is used to predict the results for the future Machine Learning (ML) is subset of Artificial Intelligence (AI) In this exercise, let us learn below Data Pre-processing Techniques: Data Acquisition Data Exploration Data Wrangling Data Manipulation In [3]: import os os . getcwd () #gets current working directory Out[3]: 'C:\\Users\\ns45237\\working_neeraj\\2_machine_learning' In [38]: #Data Acquisition #Loading the CSV file in Python import pandas as pd #pandas is python data wrangling package df = pd . read_csv ( 'BostonHousing.csv' ) #read the CSV file into a Python DataFrame #df.to_csv("/home/neerajshinde/Data/BostonHousing.csv") #loading data into an existing CSV file #df = pd.read_excel('BostonHousing.xlsx') #read the XLSX file into a Python...
Read more