rasika1206/ODD2023-Datascience-Ex01

Ex-01_DS_Data_Cleansing

AIM

To read the given data and perform data cleaning and save the cleaned data to a file.

Explanation

Data cleaning is the process of preparing data for analysis by removing or modifying data that is incorrect ,incompleted , irrelevant , duplicated or improperly formatted. Data cleaning is not simply about erasing data ,but rather finding a way to maximize datasets accuracy without necessarily deleting the information.

ALGORITHM

STEP 1

Read the given Data

STEP 2

Get the information about the data

STEP 3

Remove the null values from the data

STEP 4

Save the Clean data to the file

STEP 5

Remove outliers using IQR

STEP 6

Use zscore of to remove outliers

CODE and OUTPUT

import pandas as pd

df=pd.read_csv("/content/SAMPLEIDS.csv")

df

df.isnull().sum()

df.isnull().any()

df.dropna()

df.fillna(0)

df.fillna(method="ffill")

df.fillna(method='bfill')

df_dropped = df.dropna()

df_dropped

df.fillna({'GENDER':'MALE','NAME':'SRI','ADDRESS':'POONAMALEE','M1':98,'M2':87,'M3':76,'M4':92,'TOTAL':305,'AVG':89.999999})

import pandas as pd

ir=pd.read_csv('/content/iris.csv')

ir

ir.describe()

import seaborn as sns

sns.boxplot(x='sepal_width',data=ir)

c1=ir.sepal_width.quantile(0.25)

c3=ir.sepal_width.quantile(0.75)

iq=c3-c1

print(c3)

3.3

rid=ir[((ir.sepal_width<(c1-1.5iq))|(ir.sepal_width>(c3+1.5iq)))]

rid['sepal_width']

delid=ir[~((ir.sepal_width<(c1-1.5iq))|(ir.sepal_width>(c3+1.5iq)))]

delid

sns.boxplot(x='sepal_width',data=delid)

import matplotlib.pyplot as plt

import pandas as pd

import numpy as np

import scipy.stats as stats

dataset = pd.read_csv('/content/heights.csv')

dataset

df = pd.read_csv('/content/heights.csv')

q1=df['height'].quantile(0.25)

q2=df['height'].quantile(0.5)

q3=df['height'].quantile(0.75)

iqr=q3-q1

iqr

0.9249999999999998

low=q1-1.5*iqr

print(low)

3.8625000000000003

high= q3+1.5*iqr

print(high)

7.5625

df1=df[(df['height']>=low) & (df['height']<=high)]

df1

z = np.abs(stats.zscore(df['height']))

z

df1=df[z<3]

df1

RESULT

Thus we have cleaned the data and removed the outliers by detection using IQR and Z-score method.