In this topic I would like to show how to manage a dataset with many features (especially numeric data with completely unclear meaning and influence on whole dataset)

The dataset contains speech features and clinical variables from participants of a depression related study. Based on speech recordings, vocal features have been derived from different categories. Each feature contains a tag _{pos,neg}, which refers to the vocal task it was extracted from. Clinical and demographic variables of participants can be found at the beginning. The study was meant to show a relation between voice patterns and depression scale (variable ADS).

In this topic I want to describe some basic data transformation and analysing techniques to prepare data for modelling. For demonstration I took Medical Cost Personal Datasets from Kaggle.

In this article I want to show some basic data preparation and analytics tasks. Also I want to shown Plotly library for data visualisation.

I have been looking for some datasets that I can easily manipulate to demonstrate the first pre-step of the Machine Learning - data preparation and cleaning. So I started to surf the Internet for datasets I can somehow explain and easily interpret based on my experience.

In the previous article I described how to prepare training data (corpus) for training with OpenNLP for a specific domain and custom entities.

MLFlow helps to data scientists to track their machine learning experiments. However, it provides more than that. Some of the main features:

Why NLP?

Natural language processing is a component of Text Mining that performs a special kind of linguistic analysis that essentially helps a machine to read text. So this part of Data Science provides methods and approaches to mining patterns in texts. Thus NER is just one in the list of NLP tasks.

As I am working with Java code, I also find static types in my project as a useful thing. They help me to reason about my code and avoid simple bugs, for example if I pass incorrect argument to some method or function.

Apache Spark is an abstract query engine that allows to process data at scale. Spark provides an API in several languages such as Scala, Java and Python. Today I would like to show you how to use Python and PySpark to do data analytics in Spark SQL API. I will also use Plotly library to visualise processed data.