Data wrangling and exploratory data analysis explained
Novice data scientists sometimes have the notion that all they need to do is to find the right model for their data and then fit it. Nothing could be farther from the actual practice of data science. In fact, data wrangling (also called data cleansing and data munging) and exploratory data analysis often consume 80% of a data scientist’s time.
Despite how easy data wrangling and exploratory data analysis are conceptually, it can be hard to get them right. Uncleansed or badly cleansed data is garbage, and the GIGO principle (garbage in, garbage out) applies to modeling and analysis just as much as it does to any other aspect of data processing.
What is data wrangling?
Data rarely comes in usable form. It’s often contaminated with errors and omissions, rarely has the desired structure, and usually lacks context. Data wrangling is the process of discovering the data, cleaning the data, validating it, structuring it for usability, enriching the content (possibly by adding information from public data such as weather and economic conditions), and in some cases aggregating and transforming the data.