Gaussian and Normal distribution

March 29, 2020

Gaussian distribution is a bell-shaped curve, it follows the normal distribution with the equal number of measurements right side and left side of the mean value. Mean is situated in the center of the curve, the right side values from the mean are greater than mean value and the left side values from mean are smaller than the mean. It is used for mean, median, and mode for continuous values. You all know the basic meaning of mean, median, and mod. The mean is an average of the values, the median is the center value of the distribution and the mode is the value of the distribution which is frequently occurred. In the normal distribution, the values of mean, median, and are all same. If the values generate skewness than it is not normally distributed. The normal distribution is very important in statistics because it fits for many occurrences such as heights, blood pressure, measurement error, and many numerical values.

A gaussian and normal distribution is the same in statistics theory. Gaussian distribution is also known as a normal distribution. The curve is made with the help of probability density function with the random values. F(x) is the PDF function and x is the value of gaussian & used to represent the real values of random variables having unknown distribution.

There is a property of Gaussian distribution which is known as Empirical formula which shows that in which confidence interval the value comes under. The normal distribution contains the mean value as 0 and standard deviation 1.

The empirical rule also referred to as the three-sigma rule or 68-95-99.7 rule, is a statistical rule which states that for a normal distribution, almost all data falls within three standard deviations (denoted by σ) of the mean (denoted by µ). Broken down, the empirical rule shows that 68% falls within the first standard deviation (µ ± σ), 95% within the first two standard deviations (µ ± 2σ), and 99.7% within the first three standard deviations (µ ± 3σ).

Python code for plotting the gaussian graph:

import matplotlib.pyplot as plt
import numpy as np
import scipy.stats as stats
import math

mu = 0
variance = 1
sigma = math.sqrt(variance)
x = np.linspace(mu - 3*sigma, mu + 3*sigma, 100)
plt.plot(x, stats.norm.pdf(x, mu, sigma))
plt.show()

The above code shows the Gaussian distribution with 99% of the confidence interval with a standard deviation of 3 with mean 0.

Learnbay provides industry accredited data science courses in Bangalore. We understand the conjugation of technology in the field of Data science hence we offer significant courses like Machine learning, Tensor Flow, IBM Watson, Google Cloud platform, Tableau, Hadoop, time series, R and Python. With authentic real-time industry projects. Students will be efficient by being certified by IBM. Around hundreds of students are placed in promising companies for data science roles. Choosing Learnbay you will reach the most aspiring job of present and future.
Learnbay data science course covers Data Science with Python, Artificial Intelligence with Python, Deep Learning using Tensor-Flow. These topics are covered and co-developed with IBM.

What is Supervised, unsupervised learning, and reinforcement learning in Machine learning

March 23, 2020

The supervised learning algorithm is widely used in the industries to predict the business outcome, and forecasting the result on the basis of historical data. The output of any supervised learning depends on the target variables. It allows the numerical, categorical, discrete, linear datasets to build a machine learning model. The target variable is known for building the model and that model predicts the outcome on the basis of the given target variable if any new data point comes to the dataset.

The supervised learning model is used to teach the machine to predict the result for the unseen input. It contains a known dataset to train the machine and its performance during the training time of a model. And then the model predicts the response of testing data when it is fed to the trained model. There are different machine learning models that are suitable for different kinds of datasets. The supervised algorithm uses regression and classification techniques for building predictive models.

For example, you have a bucket of fruits and there are different types of fruits in the bucket. You need to separate the fruits according to their features and you know the name of the fruits follow up its corresponding features the features of the fruits are independent variables and name of fruits are dependent variable that is out target variable. We can build a predicting model to determine the fruit name.

There are various types of Supervised learning:

Linear regression
Logistic regression
Decision tree
Random forest
support vector machine
k-Nearest neighbors

Linear and logistic regression is used when we have continuous data. Linear regression defines the relationship between the variables where we have independent and dependent variables. For example, what would be the performance percentage of a student after studying a number of hours? The numbers of hours are in an independent feature and the performance of students in the dependent features. The linear regression is also categorized in types
those are simple linear regression, multiple linear regression, polynomial regression.

Classification algorithms help to classify the categorical values. It is used for the categorical values, discrete values, or the values which belong to a particular class. Decision tree and Random forest and KNN all are used for the categorical dataset. Popular or major applications of classification include bank credit scoring, medical imaging, and speech recognition. Also, handwriting recognition uses classification to recognize letters and numbers, to check whether an email is genuine or spam, or even to detect whether a tumor is benign or cancerous and for recommender systems.

The support vector machine is used for both classification and regression problems. It uses the regression method to create a hyperplane to classify the category of the datapoint. sentiment analysis of a subject is determined with the help of SVM whether the statement is positive or negative.

Unsupervised learning algorithms

Unsupervised learning is a technique in which we need to supervise the model as we have not any target variable or labeled dataset. It discovers its own information to predict the outcome. It is used for the unlabeled datasets. Unsupervised learning algorithms allow you to perform more complex processing tasks compared to supervised learning. Although, unsupervised learning can be more unpredictable compared with other natural learning methods. It is easier to get unlabeled data from a computer than labeled data, which needs manual intervention.

For example, We have a bucket of fruits and we need to separate them accordingly, and there no target variable available to determine whether the fruit is apple, orange, or banana. Unsupervised learning categorizes these fruits to make a prediction when new data comes.

Types of unsupervised learning:

Hierarchical clustering
K-means clustering
K-NN (k nearest neighbors)
Principal Component Analysis
Singular Value Decomposition
Independent Component Analysis

Hierarchical clustering is an algorithm that builds a hierarchy of clusters. It begins with all the data which is assigned to a cluster of their own. Here, two close clusters are going to be in the same cluster. This algorithm ends when there is only one cluster left.

K-means and KNN is also a clustering method to classify the dataset. k-means is an iterative method of clustering and also used to find the highest value for every iteration, we can select the numbers of clusters. You need to define the k cluster for making a good predictive model. K- nearest neighbour is the simplest of all machine learning classifiers. It differs from other machine learning techniques, in that it doesn’t produce a model. It is a simple algorithm that stores all available cases and classifies new instances based on a similarity measure.

PCA(Principal component analysis) is a dimensionality reduction algorithm. For example, you have a dataset with 200 of the features/columns. You need to reduce the number of features for the model with only an important feature. It maintains the complexity of the dataset.

Reinforcement learning is also a type of Machine learning algorithm. It provides a suitable action in a particular situation, and it is used to maximize the reward. The reward could be positive or negative based on the behavior of the object. Reinforcement learning is employed by various software and machines to find the best possible behavior in a situation.

Main points in Reinforcement learning –

Input: The input should be an initial state from which the model will start
Output: There are much possible output as there are a variety of solution to a particular problem
Training: The training is based upon the input, The model will return a state and the user will decide to reward or punish the model based on its output.
The model keeps continues to learn.
The best solution is decided based on the maximum reward.

XGBoost Classifier

February 4, 2020

What is the XGboost classifier?

XGBoost classifier is a Machine learning algorithm that is applied for structured and tabular data. XGBoost is an implementation of gradient boosted decision trees designed for speed and performance. XGBoost is an extreme gradient boost algorithm. And that means it’s a big Machine learning algorithm with lots of parts. XGBoost works with large complicated datasets. XGBoost is an ensemble modeling technique.

What is ensemble modeling?

XGBoost is an ensemble learning method. Sometimes, it may not be sufficient to rely upon the results of just one machine learning model. Ensemble learning offers a systematic solution to combine the predictive power of multiple learners. The resultant is a single model that gives the aggregated output from several models.

The models that form the ensemble, also known as base learners, could be either from the same learning algorithm or different learning algorithms. Bagging and boosting are two widely used ensemble learners. Though these two techniques can be used with several statistical models, the most predominant usage has been with decision trees.

Unique features of XGBoost:

XGBoost is a popular implementation of gradient boosting. Let’s discuss some features of XGBoost that make it so interesting.

- Regularization: XGBoost has an option to penalize complex models through both L1 and L2 regularization. Regularization helps in preventing overfitting
- Handling sparse data: Missing values or data processing steps like one-hot encoding make data sparse. XGBoost incorporates a sparsity-aware split finding algorithm to handle different types of sparsity patterns in the data
- Weighted quantile sketch: Most existing tree based algorithms can find the split points when the data points are of equal weights (using a quantile sketch algorithm). However, they are not equipped to handle weighted data. XGBoost has a distributed weighted quantile sketch algorithm to effectively handle weighted data
- Block structure for parallel learning: For faster computing, XGBoost can make use of multiple cores on the CPU. This is possible because of a block structure in its system design. Data is sorted and stored in in-memory units called blocks. Unlike other algorithms, this enables the data layout to be reused by subsequent iterations, instead of computing it again. This feature also serves useful for steps like split finding and column sub-sampling
- Cache awareness: In XGBoost, non-continuous memory access is required to get the gradient statistics by row index. Hence, XGBoost has been designed to make optimal use of hardware. This is done by allocating internal buffers in each thread, where the gradient statistics can be stored
- Out-of-core computing: This feature optimizes the available disk space and maximizes its usage when handling huge datasets that do not fit into memory.

Solve the XGBoost mathematically:

Here we will use simple Training Data, which has a Drug dosage on the x-axis and Drug effectiveness in the y-axis. These above two observations(6.5, 7.5) have a relatively large value for Drug Effectiveness and that means that the drug was helpful and these below two observations(-10.5, -7.5) have a relatively negative value for Drug Effectiveness, and that means that the drug did more harm than good.

The very 1st step in fitting XGBoost to the training data is to make an initial prediction. This prediction could be anything but by default, it is 0.5, regardless of whether you are using XGBoost for Regression or Classification.

The prediction 0.5 corresponds to the thick black horizontal line.

Unlike unextreme Gradient Boost which typically uses regular off-the-shelf, Regression Trees. XGBoost uses a unique Regression tree that is called an XGBoost Tree.

Now we need to calculate the Quality score or Similarity score for the Residuals.

Here λ is a regularization parameter.

So we split the observations into two groups, based on whether or not the Dosage<15.

The observation on the left is the only one with Dosage<15. All of the other residuals go to the leaf on the right.

When we calculate the similarity score for the observations –10.5,-7.5,6.5,7.5 while putting λ =0
we got similarity =4 and

Hence the result we got is:

Data Science at Intern level

January 7, 2020

As Data Science holds the grand title of sexiest job of the decade the idea of pursuing it becomes still more uneasy, more to that there is the viral notion of “Data Science is the toughest to build career in”, but it is infact much more sorted to become a part of it. All you need to have is smartness and consistency in your efforts. Let us know briefly about it down in the blog.

Quick fact: A Data Scientist is not obliged to have knowledge of all the languages that has ever existed, this is yet another notion, you will only be asked for at least any 2 language to maximum 4 languages. Just get confident with any two of the important languages, then further get into other ones one after the other. Make sure the initial languages that you start with you will possess absolute knowledge over it along with constant practice of various skills in it.

All the insights given on further are provided directly by Data Scientists in various interviews, merging all of the information we can understand the responsibilities, environment, requirements in the field of Data Science.

1. Ways of getting into the field of Data Science
When asked about how to easily get into the field, a data scientist told that there are many ways to do that by coming up with any one of important processes of the field. All that is required is right knowledge upon whatsoever process you choose and must be swift in pacing up with the levels of the processes. There are Data science course available in various colleges and also in education centers that will not only help you have the knowledge on the fundamentals of the field but also support you in finding a deserving company.

2. At the intern level as a Data Scientist
Because of the high popularity the field has gained it would nervous even the brightest scholar on the pressure of being the best, but all one must do is relax because the Data Science activity happens in a team and everything an intern would need is the absolute knowledge on the language and tools they have preferred. A Data Scientist explained how daunting it will be especially as an intern because no matter how many different languages they learn the insecurity of other person is more talented than me would haunt now and then, maybe this is because the dynamic behavior of the field. Every Data Scientist will be put in a team, for the sole reason of it is impossible for any individual to be skilled in all the programming languages.

If we interpret the importance of teamwork in a Venn diagram there will be intersection and overlapping of various languages one among another. This way one pack of team will be knowledgeable of all the required languages but be always humble towards what you do not know, social etiquette is also necessary. As an intern your focus must be on following the patterns of how the activity works, analyse which language will be appropriate to learn because ones journey in Data Science will not end until they get a job, but it will start from there.

3. Different education background and domain
This is another issue seen among the aspirants of being from an education background that is nowhere related to technical field. Addressing to this issue a Data Scientist revealed that there are data scientists from fields like biology, physics, psychology, business and are still triumphing through their way. Even if you are from technical background you will still have to study Data Science because the concepts of it are different from any other technical field, there are specific concepts that must be learnt and practiced, also since it is a dynamic the requirements and essentials will be changing regularly so it is necessary to be well groomed before stepping into the field.

For the people of domain or education background different to it must find themselves a training center that will ease them into the field by providing knowledge of the concepts right from its beginning. Getting into the field is easy but for sustaining in it you will have to pace up your game by handling the toughness and by updating yourself according to the trend in the air.

Useful information by a Data Scientist of sources where you can learn Data Science easy and efficiently:
Podcasts: Data Skeptic
Blogs: Data Science Central

Exploratory Data Analysis on Iris dataset

January 3, 2020

What is EDA?

Exploratory Data Analysis refers to the critical process of performing initial investigations on data so as to discover patterns, spot anomalies, to test hypotheses and to check assumptions with the help of summary statistics and graphical representations.

It is always good to explore and compare a data set with multiple exploratory techniques. After the exploratory data analysis, you will get confidence in your data to point where you’re ready to engage a machine learning algorithm and another benefit of EDA is to the selection of feature variables that will be used later for Machine Learning.
In this blog, we take Iris Dataset to get the process of EDA.

Importing libraries:

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt Loading the Iris data iris_data= pd.read_csv("Iris.csv")  Understand the data: iris_data.shape
(150,5)
iris_data['Species'].value_counts()
setosa        50
virginica     50
versicolor    50
Name: species, dtype: int64 iris_data.columns() Index(['sepal_length', 'sepal_width', 'petal_length', 'petal_width','species'],dtype='object') 1D scatter plot of the iris data: iris_setso = iris.loc[iris["species"] == "setosa"];
iris_virginica = iris.loc[iris["species"] == "virginica"];
iris_versicolor = iris.loc[iris["species"] == "versicolor"];
plt.plot(iris_setso["petal_length"],np.zeros_like(iris_setso["petal_length"]), 'o')
plt.plot(iris_versicolor["petal_length"],np.zeros_like(iris_versicolor["petal_length"]), 'o')
plt.plot(iris_virginica["petal_length"],np.zeros_like(iris_virginica["petal_length"]), 'o')
plt.grid()
plt.show()   2D scatter plot: iris.plot(kind="scatter",x="sepal_length",y="sepal_width")
plt.show()  2D scatter plot with the seaborn library : import seaborn as sns
sns.set_style("whitegrid");
sns.FacetGrid(iris,hue="species",size=4) \
.map(plt.scatter,"sepal_length","sepal_width") \
.add_legend()
plt.show()

Conclusion

Blue points can be easily separated from red and green by drawing a line.
But red and green data points cannot be easily separated.
Using sepal_length and sepal_width features, we can distinguish Setosa flowers from others.
Separating Versicolor from Viginica is much harder as they have considerable overlap.

Pair Plot:

A pairs plot allows us to see both the distribution of single variables and relationships between two variables. For example, let’s say we have four features ‘sepal length’, ‘sepal width’, ‘petal length’ and ‘petal width’ in our iris dataset. In that case, we will have 4C2 plots i.e. 6 unique plots. The pairs, in this case, will be :

Sepal length, sepal width
sepal length, petal length
sepal length, petal width
sepal width, petal length
sepal width, petal width
petal length, petal width

So, here instead of trying to visualize four dimensions which are not possible. We will look into 6 2D plots and try to understand the 4-dimensional data in the form of a matrix.

sns.set_style("whitegrid");
sns.pairplot(iris,hue="species",size=3);
plt.show()

Conclusion:

petal length and petal width are the most useful features to identify various flower types.
While Setosa can be easily identified (linearly separable), virginica and Versicolor have some overlap (almost linearly separable).
We can find “lines” and “if-else” conditions to build a simple model to classify the flower types.

Cumulative distribution function:

iris_setosa = iris.loc[iris["species"] == "setosa"];
iris_virginica = iris.loc[iris["species"] == "virginica"];
iris_versicolor = iris.loc[iris["species"] == "versicolor"];
counts, bin_edges = np.histogram(iris_setosa['petal_length'], bins=10, density = True)
pdf = counts/(sum(counts))
print(pdf);
>>>[0.02 0.02 0.04 0.14 0.24 0.28 0.14 0.08 0.   0.04]
print(bin_edges);
>>>[1.   1.09 1.18 1.27 1.36 1.45 1.54 1.63 1.72 1.81 1.9 ]
cdf = np.cumsum(pdf)
plt.grid()
plt.plot(bin_edges[1:],pdf);
plt.plot(bin_edges[1:], cdf)

Mean, Median, and Std-Dev:

print("Means:")
print(np.mean(iris_setosa["petal_length"]))
print(np.mean(np.append(iris_setosa["petal_length"],50)));
print(np.mean(iris_virginica["petal_length"]))
print(np.mean(iris_versicolor["petal_length"]))
print("\nStd-dev:");
print(np.std(iris_setosa["petal_length"]))
print(np.std(iris_virginica["petal_length"]))
print(np.std(iris_versicolor["petal_length"]))
OutPut: -
Means:
1.464
2.4156862745098038
5.5520000000000005
4.26

Std-dev:
0.17176728442867112
0.546347874526844
0.4651881339845203

Linear Regression in Machine Learning

December 30, 2019

What is Regression?

In statistical modeling, regression analysis is a statistical process for estimating the relationships among variables. It includes many techniques for modeling and analyzing several variables when the focus is on the relationship between a dependent variable and one or more independent variables (or ‘predictors’). Regression is a predictive modeling analysis technique. It estimates a relationship between the dependent and an independent variable.

Use of Regression:

Determine the strength of predictors.
Forecasting an effect.
Trend forecasting.

Linear Regression:

Linear regression is a basic and commonly used type of predictive analysis. The overall idea of regression is to examine two things, it does a set of predictor variables do a good job in predicting an outcome (dependent) variable? in Which variables, in particular, are significant predictors of the outcome variable, and in what way do they–indicated by the magnitude and sign of the beta estimates–impact the outcome variable? These regression estimates are used to explain the relationship between one dependent variable and one or more independent variables. The simplest form of the regression equation with one dependent and one independent variable is defined by the formula y = c + b*x, where y = estimated dependent variable score, c = constant, b = regression coefficient, and x = score on the independent variable.

Linear Regression Selection Criteria:

Classifiaction & Regression capabalities.
Data quality.
Computational complexity.
Comprehensive & transport.

When will we use Linear Regression?

Evaluating trends & sales estimates.
Analyzing the impact of price changes.
Assessment of risk in financial services and insurance domain.

for example, a group of creative Tech enthusiasts started a company in Silicon Valley. This start-up — called Banana — is so innovative that it has been growing constantly since 2016. You, the wealthy investor, would like to know whether to put your money on Banana’s success in the next year or not. Let’s assume that you don’t want to risk a lot of money, especially since the stakes are high in Silicon Valley. So you decide to buy a few shares, instead of investing in a big portion of the company.

Well, you can definitely see the trend. Banana is growing like crazy, kicking up their stock price from 100 dollars to 500 in just three years. You only care about how the price is going to be like in the year 2021 because you want to give your investment some time to blossom along with the company. Optimistically speaking, it looks like you will be growing your money in the upcoming years. The trend is likely not to go through a sudden, drastic change. This leads to you hypothesizing that the stock price will fall somewhere above the $500 indicator.

Here’s an interesting thought. Based on the stock price records of the last couple of years you were able to predict what the stock price is going to be like. You were able to infer the range of the new stock price (that doesn’t exist on the plot) for a year that we don’t have data for (the year 2021). Well — kinda.

What you just did is infer your model (that head of yours) to generalize — predict the y-value for an x-value that is not even in your knowledge. However, this is not accurate in any way. You couldn’t specify what exactly is the stock price most likely going to be. For all you know, it is probably going to be above 500 dollars.

Here is where Linear Regression (LR) comes into play. The essence of LR is to find the line that best fits the data points on the plot so that we can, more or less, know exactly where the stock price is likely to fall in the year 2021.

Let’s examine the LR-generated line (in red) above, by looking at the importance of it. It looks like, with just a little modification, we were able to realize that Banana’s stock price is likely to be worth a little bit higher than $600 by the year 2021. Obviously, this is an oversimplified example. However, the process stays the same. Linear Regression as an algorithm relies on the concept of lowering the cost to maximize the performance. We will examine this concept, and how we got the red line on the plot next.

Finding the best fit line:

To check the goodness of fit we use the R-squared method.

What is the R-squared method?

R-squared value is a statistical measure of how close the data to the fitted linear regression line. It is also known as COD(coefficient of determination), or the coefficient of multiple determination.

What are overfitting and underfitting?

Overfitting: Good performance on the training data, poor generalization to other data.

Underfitting: Poor performance on the training data & poor generalization to other data.

Linear Regression with python:

1.Importing required libraries:

import numpy as np from sklearn.linear_model import LinearRegression

2. Provide data:

x = np.array([5, 15, 25, 35, 45, 55]).reshape((-1, 1)) y = np.array([5, 20, 14, 32, 22, 38])
print(x)
print(y)

Output:
>>> print(x)
[[ 5]
 [15]
 [25]
 [35]
 [45]
 [55]]
>>> print(y)
[ 5 20 14 32 22 38]

3. Create a model and fit it:

model = LinearRegression().fit(x, y)

4. Get Result:

>> r_sq = model.score(x, y)
>>> print('coefficient of determination:', r_sq) coefficient of determination: 0.715875613747954

5. Predict response:

>>> y_pred = model.predict(x)
>>> print('predicted response:', y_pred, sep='\n')
predicted response:
[ 8.33333333 13.73333333 19.13333333 24.53333333 29.93333333 35.33333333]

Data Science is important!

December 30, 2019

Data Science is important, you do need it

Yes you read that right, you need Data Science and so does everyone! Irrespective of the domain you work in, your education background or if you are a tech head or a non-tech person, you will still be in need of Data Science. Ask me how. There is a typical notion of one will end up being only a Data Scientist by pursuing Data Science, Well, this is half a truth, while the fact is, by pursuing Data Science you could become anything. (anything is only limited to technical growth, ofcourse)

‘Anything’ does not count superpowers ofcourse, but it means that you could survive anytime in the technical world. If you just notice the definition of the Data Science you may get my point. The definition goes by, Data science is a multi-disciplinary field that uses scientific methods, processes, algorithms and systems to extract knowledge and insights from structured and unstructured data. Since it is a multidisciplinary field the aspirants will get to have knowledge on diverse languages and various tools, which will not only work inside the field of Data Science but also in any field of technology. Because Field of Technology is a vast land that bears all the technologies that has ever existed so far and Data Science is only one of its part.

Lets first get what will one get to study in Data Science, that is:

Python, R programming, Java, SQL, Scala, Tensorflow, Matlab.

The above tools and languages are the essentially used tools and languages in the field of Data Science, now I will tell you the uses of each of these tools outside the field of DS.

Python:

Used in: It is generally used in developing desktop GUI applications, websites and web applications.
Reason: Python is a highly productive language with simple programming syntax’s and easy code readability, English like commands makes it easily understandable.

There are many companies and startups working especially as website developers or application developers for several different clients. Every Independent organisations like hospitals, education sectors, business industries, or even the government sectors will need a personalized stand alone application that would keep a better track of the data and a good interface to help them ease out with the typical general tasks they get to handle.

Java

Used in: It is used in e-commerce website to android applications, is also majorly used in electronic trading system.
Reason: Java is object oriented programming language that has easy and simple methods to work with, it runs in every machine that has JVM(Java Virtual Machine), has helpful features like being platform independent, dynamic and easily portable.

Unless you weren’t under any cave, you would be aware of how much e-commerce websites like Amazon, Flipkart, banking sites, etc are growing popular everyday, which means the demand for Java will not see downfall but only get stronger.

R programming

Used in: It is generally used by Statisticians for data mining, data analysis and for major statistical approach.
Reason: It is a free software environment for statistical computing, the graphics for statistical computing are supported by R foundation.

With the knowledge of R programming one could easily merge with Statistics oriented fields.

SQL

Used in: SQL is all about working in Databases, its statements are used to perform tasks like updating data on databases, or retrieving data from various databases. Oracle, Sybase, Microsoft, SQL server, all such uses SQL.
Reason: SQL is the mainstream language that is used to access database because it can work with any database. It will help to store the data in an organised and logical manner.

Database is the main backbone for anything in the technical world, as data is the new fuel to the current generation, Database handlers are in high demand in every field that involves technology.

TensorFlow

It is mainly used for Classification, Perception, Understanding, Discovering, Prediction and Creation of Data. It is an open source artificial intelligence library, uses data flow graphs to build models. It allows developers to create large-scale neural networks with various layers.

Artificial Intelligence is the field you will attract if you get handy with TensorFlow. As per the demand and exponential growth that AI is reaching to, we can say TensorFlow will help you out the way.

These are the tools that are essentially used in DS, there are still other concepts one will get to study in the course of Data Science which will also work in different fields of the IT field.

By studying Data Science you will not only Learn that concept but also all the important languages and tools that currently runs in the technical field. You will have wide and far better opportunities even outside the boundary of Data Science. All you need is a good place where you can learn Data Science as it will act as the foundation for your journey(know why). Data Science is so much of a diverse field that it will provide diverse opportunities even for those who are a part of it, don’t hesitate to take up Data Science, it is worth it. I hope now you know why Data Science is necessary and how much you need it.

Young Data Scientists

December 17, 2019

Data Science is like a wildlife jungle, looks amazing and interesting in first sight but just as one tries to enter into it the vastness of Data Science will be visible, which is so vast that it will scare the one to drop the passion of Data Science. Even if efforts are put, the heavy syllabus of concepts would weigh too much on ones sanity. This is the one major reason why Data Science seems not a cup of tea for most of the people. In a ratio of 10 only 6 percent of people would make it as Data Scientists. On the other hand, two youngsters of the age 12 and 14 are currently on their internship as Data Scientists in massive huge IT companies.

Now the question is, if Data Science is so much daunting to learn by adults how could the young kids triumph in it?

The answer to this question lays in the journey of these two young Data Scientists. Lets get it now, but first, lets have a brief intro of them.

Tanmay Bakshi:

Tanmay is from India but has been raised in Canada, who is on his internship as Data Scientist at IBM. He grew up in a coding environment because his father was a programmer, it is no wonder he got into programming at the tender age of 5. As far as I could recall I was playing with mud at that age *sigh*. His first model was a timetable managing stand-alone application which got the eyes of IBM, it was accepted by them while he was just 11 years old. Later, Tanmay collaborated with the IBM at the teenage of 14, but his work doesn’t stop till here, it was just a beginning.

Tanmay Bakshi

Tanmay perceives technology more like a tool to change the world, as he is currently focusing upon detecting Mental health issues like depression, suicidal emotions with the help of Machine Learning and AI. He has completely indulged himself to achieve his aims and goals through technology, one of his goal is to make the coding aspirants be able to attain effective and authentic coding, for which he himself got into teaching how to code. He has so far helped 11,000 coders in which the youngest of aspirant is of 8 years and the eldest of aspirant is 81 years old.

Sidharth Srivatsav Pillai:

As a Hyderabad based 12 year old tech expert, Sidharth has been hired by an IT company. As all other kids of his age he also loved playing mobile games, but his creative mind rather got curious on how the gaming works. When he was told by his father that gaming is made by codes of programming languages, he was thrilled to learn them to create one gaming app. He began with C language, learnt the fundamentals of it and started his coding journey from then. He further learnt Java and SQL but his thirst of knowledge was still parched to quench which he also learnt HTML and CSS.

Sidharth Pillai

Finally after learning Python he got into authentic coding in Python, this lead him to Machine Learning libraries like Sikit learn and Spicy. He took some courses upon the topics, downloaded free sources and materials from web, focused on statistical approach in Data Science. All this helped him to quickly master in ML and Python. His knowledge of skills couldn’t get hidden as he is currently working in part time in an IT company.

Even after being so much knowledgeable in the technical field he has not looked down upon the importance of schooling, he attends school 3 days a week, in the rest weekdays he works for the company he has been hired in.

Think twice before you say Data Science is tough!

Also, Data Science is still not an easy field to cake walk in, then how could the young minds got Data Science but not you? Lets us know what they have for Data Science that generally adults lack.

1. Curiosity, it was their only fuel!

In both cases of these young data scientists, curiosity is the initial zest that made them dig into deep layers of Data Science. A meaningful analysis came out of Tanmay when he was asked how he could manage the toughness of the concepts, he said he could do it only because he got them learning at the age where going behind the subject that we are curious about is the first and the only goal. This is so much of a truth, usually adults lose their sense of curiosity for the notion of “I know it all”, they expect things to workout in the first attempt but it is absolute contrary in matter of kids, they do things out of pure zest to learn what amazes them. Tanmay got into coding at the very age of 5, which helped him create a simple application but it was his curiosity that made him to try out for more innovative coding techniques. In the case of Sidharth it was all about absolute curiosity and passion, he got so much amazed at video games that he couldn’t resist to learn coding and its techniques behind the animations and characters. Only if the Data Science seekers shift their perception of “burden” to “only learning to quench the curiosity” the tough and heavy concepts would no longer bother them to become a Data Scientist.

2. Baby steps

One main thing we should not neglect in these two’s journey is that they eased themselves into the field, Tanmay started with VB, whens his application was recognized by IBM he continued with further more programming languages like C, Python, Machine Learning and got into Data Science. Sidharth was first introduced to C, then Java, HTML, CSS, SQL happened. He then turned towards Python and Machine Learning. When it comes to adults I have seen many people asking how to become a Data Scientist in 3 months or 4 months, they tend to hurry into the field. They only effort on acquiring a certificate in Data Science but only if they know that Data Science is like no other field and only skills can make them a Data Scientist, they would effort on understanding the fied. Since the field is multi disciplinary, it is important to follow up and understand the stance of the essential programming languages one after the other.

3. Aim was to only learn the languages

There many are people who desires the data scientist job position merely for the mighty amount of money it acquires but only if they understood that it is the large amount of responsibility that is being paid for, the notion of “Data Science is easy” would fade away. While yes, money is good in Data Science but it is just an upshot of the field. In both cases of Tanmay and Sidharth, none weren’t even aware that they are in the highest paid job, their aim was to just be one in the field. Tanmay seeks Data Science more than just a career and money, he is working to make Data Science a powerful tool to use in fixing the issues of the society, he is working on Machine Learning and AI to identify the victims of depression and weak mental health. He also understands the necessity of the tools to learn Data Science for which he hustles to teach coding to as much people as possible, he has so far taught 11,000 aspirants and is expecting to reach 100,000 aspirants. If the aim to learn Data Science is kept higher than just money and fame then one can achieve more quickly than expected.

Quick tip: Data Science is indeed all about coding but along with it the non-technical aspect also matters as equally as technical features, follow this link to know more about it.

These young data scientists are nothing but exemplars of how to take part in the field of Data Science. I hope this article has helped you regarding Data Science.

Decision Tree

November 25, 2019

Decision tree:

The decision tree is the classification algorithm in ML(machine learning). A decision tree is a decision support tool that uses a tree-like model of decisions and their possible consequences, including chance event outcomes, resource costs, and utility. It is one way to display an algorithm that only contains conditional control statements.

To understand the algorithm of the decision tree we need to know about the classification.

What is Classification?

Classification is the process of dividing the datasets into different categories or groups by adding a label. It adds the data point to a particular labeled group on the basis of some condition.

As we see in daily life there are three categories in an email(Spam, Promotions, Personal) they are classified to get the proper information. Here decision tree is used to classify the mail type and fix it the proper one.

Types of classification

DECISION TREE
RANDOM FOREST
NAIVE BAYES
KNN

Decision tree:

Graphical representation of all the possible solutions to a decision.
A decision is based on some conditions.
The decision made can be easily explained.

There are following steps to get a decision with the decision tree

1. Entropy:

Entropy is basically used to create a tree. We find our entropy from attribute or class. A decision tree is built top-down from a root node and involves partitioning the data into subsets that contain instances with similar values (homogeneous). ID3 algorithm uses entropy to calculate the homogeneity of a sample.

2.Information Gain:

The information gain is based on the decrease in entropy after a data-set is split on an attribute. Constructing a decision tree is all about finding an attribute that returns the highest information gain.

The information gain is based on the decrease in entropy after a dataset is split on an attribute.
Constructing a decision tree is all about finding an attribute that returns the highest information gain (i.e., the most homogeneous branches).
Gain(S, A) = Entropy(S) – ∑ [ p(S|A) . Entropy(S|A) ]
We intend to choose the attribute, splitting by which information gain will be the most
Next step is calculating information gain for all attributes

Decision Tree from RachitVerma25

Here the short example of a Decision tree:

import pandas as pd import numpy as np import matplotlib.pyplot as plt %matplotlib inline play_data=pd.read_csv('data/tennis.csv.txt') print(play_data) play_data=pd.read_csv('data/tennis.csv.txt') play_data

Output:

outlook temp humidity windy play 0 sunny hot high False no 1 sunny hot high True no 2 overcast hot high False yes 3 rainy mild high False yes 4 rainy cool normal False yes 5 rainy cool normal True no 6 overcast cool normal True yes 7 sunny mild high False no 8 sunny cool normal False yes 9 rainy mild normal False yes 10 sunny mild normal True yes 11 overcast mild high True yes 12 overcast hot normal False yes 13 rainy mild high True no

Entropy of play:

Entropy(play) = – p(Yes) . log2p(Yes) – p(No) . log2p(No)

play_data.play.value_counts() Entropy_play=-(9/14)*np.log2(9/14)-(5/14)*np.log2(5/14) print(Entropy_play)

output:
0.94028595867063114

Information Gain on splitting by Outlook

Gain(Play, Outlook) = Entropy(Play) – ∑ [ p(Play|Outlook) . Entropy(Play|Outlook) ]
Gain(Play, Outlook) = Entropy(Play) – [ p(Play|Outlook=Sunny) . Entropy(Play|Outlook=Sunny) ] – [ p(Play|Outlook=Overcast) . Entropy(Play|Outlook=Overcast) ] – [ p(Play|Outlook=Rain) . Entropy(Play|Outlook=Rain) ]

play_data[play_data.outlook == 'sunny']

# Entropy(Play|Outlook=Sunny) Entropy_Play_Outlook_Sunny =-(3/5)*np.log2(3/5) -(2/5)*np.log2(2/5) Entropy_Play_Outlook_Sunny play_data[play_data.outlook == 'overcast'] # Entropy(Play|Outlook=overcast) # Since, it's a homogenous data entropy will be 0 play_data[play_data.outlook == 'rainy'] # Entropy(Play|Outlook=rainy) Entropy_Play_Outlook_Rain = -(2/5)*np.log2(2/5) - (3/5)*np.log2(3/5) print(Entropy_play_Outlook_Rain) # Entropy(Play_Sunny|) Entropy_Play_Outlook_Sunny =-(3/5)*np.log2(3/5) -(2/5)*np.log2(2/5) #Gain(Play, Outlook) = Entropy(Play) – [ p(Play|Outlook=Sunny) . Entropy(Play|Outlook=Sunny) ] – #[ p(Play|Outlook=Overcast) . Entropy(Play|Outlook=Overcast) ] – [ p(Play|Outlook=Rain) . Entropy(Play|Outlook=Rain) ]

Other gains

Gain(Play, Temperature) – 0.029
Gain(Play, Humidity) – 0.151
Gain(Play, Wind) – 0.048

Conclusion – Outlook is winner & thus becomes root of the tree

Time to find the next splitting criteria¶

play_data[play_data.outlook == 'overcast'] play_data[play_data.outlook == 'sunny'] # Entropy(Play_Sunny|) Entropy_Play_Outlook_Sunny =-(3/5)*np.log2(3/5) -(2/5)*np.log2(2/5) print(Entropy_Play_Outlook_Sunny) # Entropy(Play_Sunny|) Entropy_Play_Outlook_Sunny =-(3/5)*np.log2(3/5) -(2/5)*np.log2(2/5) print(Entropy_Play_Outlook_Sunny)

Information Gain for humidity

#Entropy for attribute high = 0, also entropy for attribute normal = 0 Entropy_Play_Outlook_Sunny - (3/5)*0 - (2/5)*0

Information Gain for windy

False -> 3 -> [1+ 2-]
True -> 2 -> [1+ 1-]

Entropy_Wind_False = -(1/3)*np.log2(1/3) - (2/3)*np.log2(2/3) print(Entropy_Wind_False) Entropy_Play_Outlook_Sunny - (3/5)* Entropy_Wind_False - (2/5)*1

Information Gain for temperature

hot -> 2 -> [2- 0+]
mild -> 2 -> [1+ 1-]
cool -> 1 -> [1+ 0-]

Entropy_Play_Outlook_Sunny - (2/5)*0 - (1/5)*0 - (2/5)* 1]

Conclusion : Humidity is the best choice on sunny branch:

play_data[(play_data.outlook == 'sunny') & (play_data.humidity == 'high')]

Output:

outlook temp humidity windy play 0 sunny hot high False no 1 sunny hot high True no 7 sunny mild high False no

play_data[(play_data.outlook == 'sunny') & (play_data.humidity == 'normal']

Output:
outlook temp humidity windy play 8 sunny cool normal False yes 10 sunny mild normal True yes

Splitting the rainy branch:

play_data[play_data.outlook == 'rainy'] # Entropy(Play_Rainy|) Entropy_Play_Outlook_Rainy =-(3/5)*np.log2(3/5) -(2/5)*np.log2(2/5)outlook temp humidity windy play 3 rainy mild high False yes 4 rainy cool normal False yes 5 rainy cool normal True no 9 rainy mild normal False yes 13 rainy mild high True no

Information Gain for temp

mild -> 3 [2+ 1-]
cool -> 2 [1+ 1-]

Entropy_Play_Outlook_Rainy - (3/5)*0.918 - (2/5)*1

Output:
0.020150594454668602

Information Gain for Windy:

Entropy_Play_Outlook_Rainy - (2/5)*0 - (3/5)*0

Output:
0.97095059445466858

Information Gain for Humidity

High -> 2 -> [1+ 1-]
Normal -> 3 -> [2+ 1-]

Entropy_Play_Outlook_Rainy_Normal = -(1/3)*np.log2(1/3) - (2/3)*np.log2(2/3) Entropy_Play_Outlook_Rainy_Normal Entropy_Play_Outlook_Rainy - (2/5)*1 - (3/5)*Entropy_Play_Outlook_Rainy_Normal Entropy_Play_Outlook_Rainy_Normal Entropy_Play_Outlook_Rainy_Normal

Output:0.91829583405448956 0.019973094021974891

Final tree:

Decision trees are popular among non-statisticians as they produce a model that is very easy to interpret. Each leaf node is presented as an if/then rule. Cases that satisfy the if/then the statement is placed in the node. Are non-parametric and therefore do not require normality assumptions of the data. Parametric models specify the form of the relationship between predictors and response. An example is a linear relationship for regression. In many cases, however, the nature of the relationship is unknown. This is a case in which non-parametric models are useful. Can handle data of different types, including continuous, categorical, ordinal, and binary. Transformations of the data are not required. It can be useful for detecting important variables, interactions, and identifying outliers. It handles missing data by identifying surrogate splits in the modeling process. Surrogate splits are splitting highly associated with the primary split. In other models, records with missing values are omitted by default.

Understanding different job positions of Data Science

November 7, 2019

Job opportunities in Data Science

To maintain and sustain any job role in the field of Data Science is a bliss in life, not only because it rains money but for the professional stability and experience one gets about technology. Generally people assume only “Data Scientist” is to become by getting in Data Science, but that is completely false. Data Science is vast of a field containing multiple disciplined technologies working efficiently with each other, to each specific technology requires a respective specialist. Which means, working of various specialists of different sectors is mandatory in Data Science.

Let’s see some job roles that Data Science provides:

Data Scientist
Data Analyst
Data Engineer
Business Analyst
Applied Machine Learning Engineer
Analytics Manager
Researcher
Data Product Manager
Big Data specialist
Project Manager

Trust me, even though they all sound almost same, are very individual in their specific terms.

Position	Job Description	Salary in Rs	Salary in $	Tools used
Data scientist	They find the rational value by processing the datasets. Mostly they resolve the problems regarding businesses.	16 lakh rs	$113,435	SAS for statistics, Apache to handle datasets, MATLAB, Tableau, R programming and Python
Data Analyst	They inspect, clean, transform and structure the data to goal discovering the concealed information between patterns of data.	4-8 lakh rs	$67,445	SAS, Rapidminer, R
Data Engineer	they handle large scale processing systems like databases, while also maintaining the analytical infrastructure.	8,28,244 rs	$127,878	Hadoop, Scala, Java, C++, SQL
Big Data Engineer	One who creates and manages company’s Big Data in infrastructure and tools. Should jumble with datasets and should know the methods to reduce the mountain size data into solution.	7,22,422 rs	$139,463	Hadoop, Hive, ML, scala, Java, Sql, NoSql, C++
Business Analyst	They dwell into maintaining Business domain. They document and assess the business related queries with the help of technology. They overall focus on keeping the business steady with the help of data.	6 lakh rs	$74,751	Ms office suite, Google Docs, SmartDraw, SWOT
Machine Learning Engineer	They basically be software engineer specialized in machine learning. Their job is to enhance the intelligence of a machine by upgrading them, testing several numerous problems on them.	1,103.K rs	$187,362	Probability statistics, Data modelling, algorithms of AI and ML
Analytics Manager	Utilizing the analytics in the business efficiently, working on creative strategy in collecting, modifying, modelling, implying data for products and services of business.	9,12,174 rs	$108,531	AI, Big Data, Cloud, communication skills
Research officer	They are required to find rational solutions for any problem bu using tools to research. They should father, build and provide final model of solution for the problems. They schedule the projects regarding the process of researching.	3 lakh – 8lakh in rs	$84,180	Data collection and analysis, qualitative and quantitative, data research, project management.
Product Data Manager	They will look after management and publication of product data for every insights regarding the product, they will be responsible.	Average 3.5 lakhs	$85,751	Microsoft visio, Google Drive, analysis tools like pendo, customer survey tools.

I hope this would help you in understanding about the diversity in one Data Science field.

The salary mentioned in here increases as days passes by, but it is not about the money, it is about the proficiency one would get in fields of authentic working of technology. If you want to be a part of Data Science then Learnbay is one of the best options.

Learnbay is located in the IT hub of India, Bangalore. This feature will help students to invent their future in best place possible, Learnbay is a Data Science training institute along with Machine Learning, Big Data and Artificial Intelligence are also taught. The course structure of Data Science has all the updated essential tools and techniques of field, IBM will certify the aspirants of here.

Program Application Form
First Name
Last Name*
Email*
Phone*
Select Certification Program*
Mention Preferred time for Telephonic Discussion*