In the figure below we get a nice normally-shaped distribution without having to filter out outliers. It manages to "draws in" big values which often makes the data easier to look at and sometimes normalizes the variance across observations. That magic operation is called the logarithm. This is doable if and only if large values are shrinked a lot and smaller values a little. The ultimate goal is to transform the distribution of the target variable resembling that of a narrow “bell curve” distribution without a tail. This can lead to wildly skewed predictions (predictions could be very far off) if outliers are present leading to poor models. Tree-based models makes predictions by averaging similar record's target values. Remember that the log-transformation can only be applied when the target variable takes only non-negative values.
That said if you are sure that those points which skewed the distribution are outliers then they should be filtered out. This occurs when there are outliers that can't be filtered out as they are important to the model. It is useful if and only if the distribution of the target variable is right-skewed which can be observed by a simply histogram plot. Photo by Glenn Carstens-Peters / Unsplash 🛠When to log-transform the target variable? When to log-transform the target variable?.Note: For the analysis below I used the House Prices: Advanced Regression Techniques Kaggle dataset. In this article, I will try answering my initial question of how log-transforming the target variable into a more uniform space boost model performance. By googling it I found out that log transformation can help a lot. As data scientist working on regression problems I have faced a lot of times datasets with right-skewed target's distributions.
0 kommentar(er)
