QuickDataScience | Quick & Easy Data Science

Welcome back, friends..!!!  Till now we have seen quite a few techniques, that we can use for Imputing Missing values in the dataset.  We have studied the theory for them and have seen a basic code for using those techniques. But as said there are other libraries that we can use to implement these techniques. So, we are going to study one such library, i.e. skLearn for Imputation. So let's begin and get our hands dirty and learn these libraries.  *Please Note:- Theory is already covered in previous articles, we will be directly moving to use libraries for the methods. For demo purposes, we will be using the COVID-19 cases dataset. 

Till now, we have seen imputation techniques that could only be used for Numerical variables but didn't say anything about the Categorical variables/column.   So now, we are going to discuss a technique that is mostly used for imputing categorical variables. Missing Category Imputation is the technique in which we add an additional category for the missing value, as "Missing" in the variable/column. In simple terms we do not take the load of predicting or calculating the value(like we did for Mean/Median or End tail Imputation ), we simply put "Missing" as the value.  Now, we may have a doubt that if we are only replacing the value with "Missing" then why it is said that this method can be used for Categorical variables only?  Here is the answer, we can use it for Numerical variables also, since we can't introduce a categorical value in the Numerical variables/column, we will be required to introduce some Numerical value that is unique for the va

End of Tail Imputation is another important Imputation technique. This technique was developed as an enhancement or to overcome the problems in the Arbitrary value Imputation technique. In the Arbitrary values Imputation method the biggest problem was selecting the arbitrary value to impute for a variable. Thus, making it hard for the user to select the value and in the case of a large dataset, it became more difficult to select the arbitrary value every time and for every variable/column.  So, to overcome the problem of selecting the arbitrary value every time, End of Tail Imputation was introduced where the arbitrary value is automatically selecting arbitrary values from the variable distributions. Now the question comes How do we select the values? & How to Impute the End value? There is a simple rule to select the value given below:-  In the case of normal distribution of the variable, we can use Mean plus/minus 3 times the standard deviation.  In the case variable is skewed,

Welcome to a series of articles about Imputation techniques. We will be publishing small articles(Quick Notes) about the various Imputation techniques used, their advantages, disadvantages, when to use and coding involved for them.  Not Sure What is Imputation?   &   What is Missing Data?    Why they are important. Click on the links to know more about them.   1. Mean Or Median Imputation 2.  End of tail Imputation  3. Missing Category Imputation 4. Random Sample Imputation 5. Missing Indicator Imputation 6. Mode Imputation 7. Arbitrary Value Imputation 8. Complete Case Analysis(CCA)  Python Libraries used for Quick & Easy Imputation.  09.  SimpleImputer   10. Feature Engine 11. Multi-Variate Imputation

To understand Mean or Median Imputation, we need to first revise the concepts of Mean & Median. Then it would be easy for us to know why this is a widely used method for imputation and can easily identify its issues.  We already studied the Missing Data and defined Imputation & its basics in previous articles. What is Mean?  Mean is nothing but the arithmetic average of numbers. That is why it is also referred to as Average or Arithmetic Average. The process of finding average/mean is quite simple, we just add all the given values irrespective of the magnitude(+/-) and then divide the total sum by the no. of observations.                                  Sum of all the observation  Average/Mean =  ---------------------------------------                                    No. of Observations