{"id":3718,"date":"2017-11-06T08:12:22","date_gmt":"2017-11-06T08:12:22","guid":{"rendered":"https:\/\/datakeen.co\/8-machine-learning-algorithms-explained-in-human-language\/"},"modified":"2021-11-18T08:19:50","modified_gmt":"2021-11-18T08:19:50","slug":"8-machine-learning-algorithms-explained-in-human-language","status":"publish","type":"post","link":"https:\/\/legacy-wp.datakeen.co\/en\/8-machine-learning-algorithms-explained-in-human-language\/","title":{"rendered":"8 Machine Learning Algorithms explained in Human language"},"content":{"rendered":"

\"\"Little schoolgirl with Pepper Robot. Credits : Andy Kelly (Unsplash)<\/a><\/p>\n

What we call\u00a0“Machine Learning” is none other than the meeting of statistics and\u00a0the incredible computation power available today (in terms of memory, CPUs, GPUs). This domain\u00a0has become increasingly visible\u00a0important because of the digital revolution of companies leading\u00a0to the production of massive data of different forms and types, at ever increasing rates: Big Data. On a purely mathematical level most of the algorithms used today are already several decades old. In this article I will explain the underlying logic\u00a0of 8 machine learning algorithms in the simplest possible terms.<\/span><\/span><\/span><\/p>\n

I. Some global\u00a0concepts before describing the algorithms <\/span><\/span><\/p>\n

1. Classification and Prediction \/ Regression<\/span><\/span><\/strong><\/span><\/p>\n

Classification<\/span><\/span><\/strong><\/span><\/p>\n

Assigning a class \/ category to each of the observations in a dataset is called classification. It is done a\u00a0posteriori<\/em>, once the data is recovered. <\/span><\/span><\/span><\/p>\n

Example:<\/span><\/span><\/span> classifying consumers reasons of visit in store in order to send them a personalized campaign. <\/span><\/span><\/span><\/p>\n

Prediction<\/strong><\/span><\/span><\/span><\/p>\n

A prediction is made on a new observation. When it comes to a numerical variable (continuous) we speak of regression. <\/span><\/span><\/span><\/p>\n

Example:<\/span><\/span><\/span> predicting a heart attack\u00a0based on data from an electro cardiogram. <\/span><\/span><\/span><\/p>\n

2. Supervised and unsupervised learning<\/span><\/span><\/span><\/b><\/p>\n

Supervised<\/b><\/span><\/span><\/span><\/p>\n

You already have tags on historical data and want to\u00a0classify new data according to these tags. The number of classes is known. <\/span><\/span><\/span><\/p>\n

Example:<\/span><\/span><\/span> in botany you made measurements (length of the stem, petals, …) on 100 plants of 3 different species. Each of the measurements is labeled with the species of the plant. You want to build a model that will automatically tell which species a new plant belongs to thanks to\u00a0the same measurements.<\/span><\/span><\/span><\/p>\n

Unsupervised<\/b><\/span><\/span><\/span><\/p>\n

On the contrary, in unsupervised learning, you have no labels, no predefined classes. You want to identify common patterns in order to form homogeneous groups based on your observations. <\/span><\/span><\/span><\/p>\n

Examples:<\/b><\/span><\/span><\/span> You want to classify your customers based on their browsing history on your website but you have not formed groups and are in an exploratory approach to see what would be the common points between them. In this case a clustering algorithm is adapted.<\/span><\/span><\/span><\/p>\n

Some neural network algorithms will be able to differentiate between human and animal images without prior labeling. <\/span><\/span><\/span><\/p>\n

II. Machine Learning Algorithms<\/span><\/span><\/span><\/p>\n

We will describe <\/span><\/span><\/span>8<\/span><\/span><\/span> algorithms used in Machine Learnin<\/span><\/span><\/span>g<\/span><\/span><\/span>. The objective here is not to go into the details of the models but rather to give the reader elements of understanding on each of them. <\/span><\/span><\/span><\/p>\n

1. “The Decision Tree”<\/span><\/span><\/span><\/b><\/p>\n

A decision tree is used to classify future observations given a body of already labeled observations. This is the case of our botanical example where we already have 100 sightings classified in species A, B and C. <\/span><\/span><\/span><\/p>\n

The tree begins with a root (where we still have all our observations) then comes a series of branches whose intersections are called nodes and ends are called leaves, each corresponding to one of the classes to predict. The depth of the tree is refers to\u00a0the maximum number of nodes before reaching a leaf. Each node of the tree represents a rule (example: length of the petal greater than 2.5 cm). To browse the tree is to check a series of rules. The tree is constructed in such a way that each node corresponds to the rule that best divides the set of initial observations (variable and threshold).<\/span><\/span><\/span><\/p>\n

Example:<\/span><\/span><\/span>
\n<\/span><\/span><\/span><\/b><\/p>\n

\"Decsion<\/p>\n

\"Decision<\/p>\n

The tree has a depth of 2 (one node plus the root). The length of the petal is the first measure that is used because it best separates the 4 observations according to class membership (here class B). <\/span><\/span><\/span><\/p>\n

2. “Random Forests”<\/b><\/span><\/span><\/span><\/p>\n

As the name might suggest, the random forest algorithm is based on a multitude of decision trees. <\/span><\/span><\/span><\/p>\n

In order to\u00a0understand better the advantages\u00a0and logic\u00a0of this algorithm, let’s start with an example:<\/span><\/span><\/span><\/p>\n

You are looking for a good travel destination for your next vacation. You ask your best friend for his opinion. He asks you questions about your previous trips and makes a recommendation. <\/span><\/span><\/span><\/p>\n

You decide to ask a group of friends who ask you questions randomly. They each make a recommendation. The chosen destination is the one that has been the most recommended by your friends. <\/span><\/span><\/span><\/p>\n

The recommendations made by your best friend and the group will both make good destination choices. But when the first recommendation method works very well for you, the second will be more reliable for other people. <\/span><\/span><\/span><\/p>\n

This comes from the fact that your best friend, who builds a decision tree to give you a destination recommendation, knows you very well what making the decision tree over-learned about you (we talk about overfitting). <\/span><\/span><\/span><\/p>\n

Your friend group represents the random forest of multiple decision trees and it’s a model, when used properly, avoids the pitfall of overfitting. How is this forest built? <\/span><\/span><\/span><\/p>\n

Here are the main steps: <\/span><\/span><\/span><\/p>\n

1. <\/span><\/span><\/span>We take a<\/span><\/span><\/span> number <\/span><\/span><\/span>X <\/span><\/span><\/span>of observations from the starting dataset (with discount).<\/span><\/span><\/span><\/p>\n

2. <\/span><\/span><\/span>We take a number K of the M variables available (features), for example: only temperature and population density<\/span><\/span><\/span><\/p>\n

3. <\/span><\/span><\/span>We create\u00a0a decision tree on this dataset.<\/span><\/span><\/span><\/p>\n

4. <\/span><\/span><\/span>Steps 1. to 4. are repeated N times so as to obtain N trees.<\/span><\/span><\/span><\/p>\n

To obtain the class of a new observation we\u00a0go down the N trees. Each tree will predict\u00a0a different class. The class chosen is the one that is most represented among all the trees in the forest. (Majority vote \/ ‘Ensemble’).<\/span><\/span><\/span><\/p>\n

3. The “Gradient Boosting” \/ “XG Boost”<\/b><\/span><\/span><\/span><\/p>\n

The boosting gradient method is used to reinforce a model that produces weak predictions, such as a decision tree (see below how do we judge the\u00a0quality of a model). <\/span><\/span><\/span><\/p>\n

We will explain the principle of boosting gradient with the decision tree but this could be with another model. <\/span><\/span><\/span><\/p>\n

You have an individual database with demographics information and past activities. You have 50% of individuals their age but the other half is unknown.<\/span><\/span><\/span><\/p>\n

You want to get the age of a person according to his activities: food shopping, television, gardening, video games … You choose as a model a decision tree, in this case it is a regression tree because the value to predict is numeric.<\/span><\/span><\/span><\/p>\n

Your first regression tree is satisfying but can be improved: it predicts, for example, that an individual is 19 years old when in fact he is 13 years old, and for another 55 years old instead of 68 years old. <\/span><\/span><\/span><\/p>\n

The principle of the gradient boosting is that you will redo a model on the difference between the predicted value and the true value to be predicted.<\/span><\/span><\/span><\/p>\n\n\n\n\n\n<\/colgroup>\n\n\n\n\n
Age<\/span><\/span><\/span><\/td>\nPrediction Tree 1<\/span><\/span><\/span><\/td>\nDifference<\/span><\/span><\/span><\/td>\nPrediction Tree 2<\/span><\/span><\/span><\/td>\n<\/tr>\n
13<\/span><\/span><\/span><\/td>\n19<\/span><\/span><\/span><\/td>\n-6<\/span><\/span><\/span><\/td>\n15<\/span><\/span><\/span><\/td>\n<\/tr>\n
68<\/span><\/span><\/span><\/td>\n55<\/span><\/span><\/span><\/td>\n+13<\/span><\/span><\/span><\/td>\n63<\/span><\/span><\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

This step N is repeated where N is determined by successively minimizing the error between the prediction and the true value. <\/span><\/span><\/span><\/p>\n

The method to optimize is the gradient descent method that we will not explain here. The XG Boost (eXtreme Gradient Boosting) model is one of the implementations of the boosting gradient founded by Tianqi Chen and has seduced the Kaggle datascientist community with its efficiency and performance. The publication explaining the algorithm is here. <\/span><\/span><\/span><\/p>\n

4. “Genetic Algorithms”<\/b><\/span><\/span><\/span><\/p>\n

As their name suggests genetic algorithms are based on the process of genetic evolution that has made us who we are … <\/span><\/span><\/span><\/p>\n

More prosaically they are mainly used when there are no observations of departure and it is hoped that a machine will learn to learn as and when testing. <\/span><\/span><\/span><\/p>\n

These algorithms are not the most effective for a specific problem but rather for a set of subproblems (eg learning balance and walking in robotics). <\/span><\/span><\/span><\/p>\n

Let’s take a simple example: We want to find the code of a safe that is made of 15 letters: “MACHINELEARNING” <\/span><\/span><\/span><\/p>\n

The genetic algorithm approach will be as follows: <\/span><\/span><\/span><\/p>\n

We start from a population of 10,000 “chromosomes” of 15 letters each. We say that the code is a word or a set of words pro <\/span><\/span><\/span><\/p>\n

“DEEP-LEARNING”<\/span><\/span><\/span><\/p>\n

“STATISTICAL INFERENCE-“<\/span><\/span><\/span><\/p>\n

“HUMAN MACHINE INTERFACE” etc. <\/span><\/span><\/span><\/p>\n

We will define a method of reproduction: for example, to combine the beginning of one chromosome with the end of another. <\/span><\/span><\/span><\/p>\n

Ex: “DEEP-LEARNING” + “STATISTICAL-INFERENCE” = “DEEP-INFERENCE” <\/span><\/span><\/span><\/p>\n

Then we will define a mutation method which allows to change a progeny that is blocked. In our case it could be to vary one of the letters randomly. Finally we define a score that will reward such or such descendants of chromosomes. In our case where the code is hidden we can imagine a sound that the trunk would do when 80% of the letters are similar and that would become stronger as we approach the right code. <\/span><\/span><\/span><\/p>\n

Our genetic algorithm will start from the initial population and form chromosomes until the solution has been found.<\/span><\/span><\/span><\/p>\n

5. “Support Vector Machines”<\/span><\/span><\/span><\/b><\/p>\n

Also known as “SVM” this algorithm is mainly used for classification problems even though it has been extended to regression problems (Drucker et al., 96).<\/span><\/span><\/span><\/p>\n

Let’s take our example of ideal holiday destinations. For the simplicity of our example consider only 2 variables to describe each city: the temperature and the density of population. We can therefore represent cities in 2 dimensions.<\/span><\/span><\/span><\/p>\n

We represent by circles\u00a0cities which\u00a0you very much appreciated and by squares those which you least appreciated. When you consider new cities you want to know which group this new city is closest to.<\/span><\/span><\/span><\/p>\n

\"SVM

SVM Example<\/p><\/div>\n

As we see in the graph on the right, there are many plans (straight lines when you only have 2 dimensions) that separate the two groups.<\/span><\/span><\/span><\/p>\n

We will choose the line that is at the maximum distance between the two groups. To build it we already see that we do not need all the points, it is enough to take the points which are at the border of their group we call these points or vectors, the support vectors. The planes passing through these support vectors are called support planes. The separation plan will be the one that will be equidistant from the two supporting planes.<\/span><\/span><\/span><\/p>\n

What to do if the groups are not so easily separable, for example if by\u00a0one of the dimensions circles are mixed up with squares or vice-versa?<\/span><\/span><\/span><\/p>\n

We will proceed to a transformation of these points by a function to be able to separate them. As in the example below:<\/span><\/span><\/span><\/p>\n

\"SVM

SVM transformation example<\/p><\/div>\n

The SVM algorithm will therefore consist of looking for both the optimal hyperplane and minimizing classification errors.<\/span><\/span><\/span><\/p>\n

6. The “K nearest neighbors”<\/b><\/span><\/span><\/span><\/p>\n

Pause. After 5 relatively technical models the algorithm of the K nearest neighbors will appear to you as a formality. Here’s how it works:<\/span><\/span><\/span><\/p>\n

An observation is assigned the class of its nearest K neighbors. <\/span><\/span><\/span><\/p>\n

“That’s it ?!” you might ask\u00a0me. <\/span><\/span><\/span><\/p>\n

Yes that’s all. Only as the following example shows: K’s choice can matter\u00a0a lot.<\/span><\/span><\/span><\/p>\n

\"K

K nearest neighbours<\/p><\/div>\n

We will typically\u00a0try different values \u200b\u200bof K to obtain the most satisfactory separation.<\/span><\/span><\/span><\/p>\n

7. “Logistic Regression”<\/span><\/span><\/strong><\/span><\/p>\n

Let’s start by a reminder of\u00a0linear regression. Linear regression is used to predict a numerical variable, e.g the price of cotton in relation to other numeric or binary variables: the number of cultivable hectares, the demand for cotton from various industries, and so on.<\/span><\/span><\/span><\/p>\n

It is a question of finding the coefficients a1, a2, … in order to have the best estimate:<\/span><\/span><\/span><\/p>\n

Cotton price = a1 * Number of hectares + a2 * Demand for cotton + …<\/span><\/span><\/span><\/p>\n

Logistic regression is used in classification in the same way as the algorithms exposed so far. Once again let’s take the example of trips considering only two classes: good destination (Y = 1) and bad destination (Y = 0).<\/span><\/span><\/span><\/p>\n

P (1): probability the city is a good destination.<\/span><\/span><\/span><\/p>\n

P (0): probability that the city is a bad destination.<\/span><\/span><\/span><\/p>\n

The city is represented by a number of variables, we will only consider two: the temperature and population density.<\/span><\/span><\/span><\/p>\n