Two Underutilized Heroes of Data & Innovation: Correlation & Covariance

Two Underutilized Heroes of Data & Innovation: Correlation & Covariance
Two Underutilized Heroes of Data & Innovation: Correlation & Covariance

Yes, Data driven innovation is fun and it gets most done in less. But let’s talk about a math that is not as much known as it should be in the enterprise world. Correlation & Covariance are two such values that are most underutilized and have the tendency to cause maximum impact and disruption to any complicated business model.

First, a quick high level math primer (picked from Wiki): In probability theory and statistics, the mathematical descriptions of covariance and correlation are very similar.[1][2] Both describe the degree of similarity between two random variables or sets of random variables.
Correlation refers to any of a broad class of statistical relationships involving dependence.
Whereas, Covariance is a measure of how much two random variables change together. If the greater values of one variable mainly correspond with the greater values of the other variable, and the same holds for the smaller values, i.e., the variables tend to show similar behavior, the covariance is positive.[1] In the opposite case, when the greater values of one variable mainly correspond to the smaller values of the other, i.e., the variables tend to show opposite behavior, the covariance is negative. Anyways, over with the math talk, you could find more information by searching for covariance & correlations and if you are not blown away by it’s capabilities, do take out some extra time for reading about cross-correlation & cross covariance. You will get into the world of predictive modeling and so much more savvy stuff that you could do with these two interesting and powerful concepts.

On a traditional note, a company is analytically as smart as the analytics team it entails. But, on an interesting note, it does not have to be like this. A smarter business model like utilizing correlation & covariance on your captured data could do the heavy lifting for you and help you focus on the areas that are really causing some significant impact to your business. As you must have already read, by definition covariance and correlation can help you understand relationship between 2 random sets of data.

What is happening in most of the companies that I spoke with is that most of us have utilized this math while handling known sets of data within the periphery of a project. For an example, a project data and its variables can be correlated together for finding any hidden relations. If these relationships are not determined, it could cost your businesses a significant impact. If you are not at this yet, stop reading now and get your correlation & covariance mojo active at the least within the projects.

If your organization is already doing it within projects, you are part of that savvy organization which takes success and failures of their projects too seriously for them to be left with professionals. Now, you might need to ask, what next. Where is the next big wave? Innovation is the next big thing that is riding on the data that correlation/covariance could provide your organization. How about doing it within different projects, departments, silos etc. Consider for a case where one project is impacting the other. So, one tiny dependency on a remote department could cause a significant impact to totally unrelated department in the business.

Yes, you guessed right, we are talking about a big-data problem, or may be one of the biggest big-data problems for your organization.

Correlation and covariance have the power to identify those hidden relationships that you would have never guessed existed and then helps you find the extent of their dependency. How much one variable varies with the other. Once you have a model in place to comb your organization’s data for any correlations and thereby finding their covariance, you would understand how much one event is linked to other and by what degree. This would help your business identify high impact areas that you could then map to high performance. All you need to do is understand if the identified relationship is known or unknown. If it’s known, yes, you have validated that sometimes world is as sane as you expect it to be, and If not, wallah, you just identified a potential area to investigate and worry about, to make sure all relationships in your business are accounted for.

If data combing is done properly for any possible correlations and covariance, you could assure nothing will ever fall through the crack again. Your radar will always pick potential areas as soon as their relationship is established. And yes, that will save your business some cash and help it run optimally.

So, to do a quick recap:
1. Make sure you understand what correlation/covariance is, and for added bonus, read about cross correlation & cross covariance.
2. Make sure your project or projects in your company are leveraging correlation/covariance in finding hidden dependencies that could jeopardize the success of your project.
3. Make sure, you have big-data setup that could help connect data across various projects, departments & business units for finding possible correlations and their covariance.
4. Make sure you have right triggers, alarms and action plan setup for investigating any identified relationships further.
5. Make sure you have an automated system that combs the business data and help identifies possible cracks in real time.

If you are done with those 5 steps, your business is destined for consistent improvements and sustained data driven innovations.

And yes, as I always rant, you don’t have to do it in-house. Probably, for better business sense, get it made outside and then once it is validated, bring it in-house. All you need is a good data analytics/visualization platform that could take any number of structured and un-structured data and find correlations between them.

Originally Posted at: Two Underutilized Heroes of Data & Innovation: Correlation & Covariance

Leave a Reply

Your email address will not be published. Required fields are marked *