Maintaining the highest quality of data is a necessary condition for the growth of any organization. There is a reason why the concept of a data – Data Driven Company – is gaining more and more popularity. Without certain information and good-quality data, it is difficult to expect managers and analysts to draw logically correct conclusions, allowing them to develop the company and improve their daily work.
In our previous publications, you may have learned about the principles that shape data-driven companies. Caring for the proper quality of data is definitely one of them. Therefore, it is worth knowing the answer to the question: how to obtain high quality data? There are really many ways and in this material we present the most important and the most interesting methods of combating the insufficient data quality. The basic tool useful in this area are data metrics that allow you to keep track of the state we are dealing with. Depending on the results, we can target further actions towards a quick repair, acceptance of the current state or enrichment.
1. To get us started: classic data quality metrics
First, let’s verify the quality of the data we work with. First, let’s answer the question: when do we actually know that we are dealing with good quality data? We have this certainty only when it is up-to-date, complete and properly reflect the actual state of affairs. Additionally, if it comes from multiple sources, it should be mutually consistent. At the same time, individual cases that characterize a specific data set should be considered.
Having this information in place, we are one step away from assessing the quality of the data. Now it is enough to regularly verify the previously determined attributes to be sure that all analyzes, at the very source, are based on correct and reliable information.
2. Automation of metrics – factor out repeating problems
Naturally, creating metrics is a laborious and time-consuming process that requires additional work. However, nothing prevents you from making it easier and supporting it with general rules. The analysis of the aforementioned data quality attributes comes down to similar activities for the majority of sets, enabling the quality examination “at the core”. Therefore, why not create a mechanism that will simplify the registration of the metrics to a minimum? Once developed, the target solutions may only require basic information to quickly create a complete data quality verification process. Thanks to this approach, the data quality verification process does not require additional processing.
After all, each organization has its own rules, so it is worth analyzing carefully which cases we encounter most often. All this will make the automation come naturally and the data quality testing process itself will significantly speed up.
3. Artificial Intelligence – any deviation from the norm should be taken into account
The previously discussed cases of measuring data quality refer to obvious errors or gaps in the data, the validity of which cannot be questioned. Therefore, it is enough to correct or complete the data. These are elementary problems that need to be limited. Additionally, it is worth studying trends and identifying anomalies based on it. In this case, artificial intelligence and algorithms for detecting deviations turn out to be very useful. “Abnormal values” can often spell problems, but they don’t have to. They may well turn out to be an interesting starting point for further analyzes.
The use of artificial intelligence requires taking into account several important factors, such as: selection of the appropriate algorithm, proper model training, taking into account the specificity of the data and the involvement of the right people for the final assessment of the anomaly and further tuning of the model. Nevertheless, the effects can often be more than surprising and satisfactory.
4. Data profiling – data knowledge is a key factor
Let’s focus on simplifying the data quality measurement process itself. The previous points relied heavily on stakeholders knowledge. If we ignore their involvement in the process, we can expect worse care for the area, but in return, we benefit significantly from automation. It is enough for us to take into account data profiling. To put it simply, it will allow us to automatically apply previously established rules. For example, thanks to this approach, we are able to use algorithms to detect whether we are dealing with names and surnames, and therefore their verification can occur automatically.
There are many more applications of data profiling, but undoubtedly the main factor that encourages the use of this type of solutions is the aforementioned simplification and automation support.
5. Data enrichment and cleaning – why not use different sources?
The last point really comes down to a situation where we cannot complete or correct the data that we use. Should the set and related objects be ignored in this case? Definitely not – it doesn’t make sense in any way. Therefore, this is where the concept of Data Cleansing, i.e., the data cleansing process that detects and removes or corrects erroneous information, comes into play. It often happens that specific data linked to other data from a different source make more sense and become of correspondingly good quality. Thanks to this combination, we will be able to prepare appropriate analyzes.
Finally, it is worth remembering that once invested time and funds in the development and research on data quality, can pay off when the first problems with it are discovered. However, the benefits are huge. Thanks to this approach, we will find out about the inaccessibility or lack of completeness of data at the very beginning, after noticing the errors, i.e. right after loading, instead of in the final reports, where they may even pass unnoticed and lead to false conclusions.
Łukasz Pająk, Senior Programmer / Designer
Łukasz has been working with data since the beginning of his professional career. He is closely associated with the telecommunication industry, where he cares about Data Quality & Data Governance, and at the same time about a good working atmosphere. Privately, a huge fan of new technologies, automotive and unconventional solutions.
