AnalyticsCube

The direct hard costs of a data breach are typically easy to calculate. An organization can assign a value to the human-hours and equipment costs it takes to recover a breached system. Those costs, however, are only a small part of the big picture.

Every organization that has experienced a significant data breach knows this firsthand. Besides direct financial costs, there are actually lost business, third-party liabilities, legal expenses, regulatory fines, and damaged goodwill. The true cost of a data breach encompasses much more than just direct losses.

Forensic Analysis. Hackers have learned to disguise their activity in ways that make it difficult to determine the extent of a breach. An organization will often need forensic specialists to determine how deeply hackers have infiltrated a network. Those specialists charge between $200 and $2,000 per hour.

Customer Notifications. A company that has suffered a data breach has a legal and ethical obligation to send written notices to affected parties. Those notices can cost between $5 and $50 apiece.

Credit Monitoring. Many companies will offer credit monitoring and identity theft protection services to affected customers after a data breach. Those services cost between $10 and $30 per customer.

Legal Defense Costs. Customers will not hesitate to sue a company if they perceive that the company failed to protect their data. Legal costs between $500,000 and $1 million are typical for significant data breaches affecting large companies. Companies often mitigate these high costs with data breach insurance because it covers liability and notification costs, among others.

Regulatory Fines and Legal Judgments. Target paid $18.5 million after a 2013 data breach that exposed the personal information of more than 41 million customers. Advocate Health Care paid a record $5.5 million fine after thieves stole an unsecured hard drive containing patient records. Fines and judgments of this magnitude can be ruinous for a small or medium-sized business.

Reputational Losses. Quantifying the value of lost goodwill and standing within an industry after a data breach is impossible. That lost goodwill can translate into losing more than 20 percent of regular customers, plus revenue depletions exceeding 30 percent. There’s also the cost of missing new business opportunities.

The total losses that a company experiences following a data breach depend on the number of records lost. The average per-record loss in 2017 was $225. Thus, a small or medium-sized business that loses as few as 1,000 customer records can expect to realize a loss of $225,000. This explains why more than 60 percent of SMBs close their doors permanently within six months of experiencing a data breach.

Knowing the risks, companies can focus on devoting their cyber security budget to prevention and response. The first line of defense is technological, including network firewalls and regular employee training. However, hackers can still slip through the cracks, as they’re always devising new strategies for stealing data. A smart backup plan includes a savvy response and insurance to cover the steep costs if a breach occurs. After all, the total costs are far greater than just business interruption and fines; your reputation is at stake, too.

Originally Posted at: What’s the True Cost of a Data Breach? by thomassujain

Historically, cartographers would place dragons or other imaginary great beasts to fill in areas of a map that were unexplored. It was a simple placeholder for all the unknowns. Unfortunately, shipwrights did not have the luxury of ambiguity and had to imagine what could happen, in order to build ships that could withstand all.

>>Related: Best Practices for Deploying and Scaling Embedded Analytics<<

Similarly, as you are designing an analytics application, you have to understand potential performance problems and prepare for them. Building a highly performant application means navigating the perils of the high seas in a largely distributed IT infrastructure environment. It requires having full visibility into everything that affects your end-user experience, including web code, infrastructure, database connectivity, and external dependencies.

Let’s take a different lens to determine what performant means and how to decide what to measure to get there. We need to explore why it’s important to create a baseline for user experience, monitor actual performance against the baseline, and tie performance directly to your business goals.

The first step is determining whether your analytics application is “available” to its end users. Availability may mean different things to various stakeholders. From a DevOps perspective, a successful ping to the server means the application is up and running. That particular metric, however, may be meaningless to a CEO who considers availability to mean accessing their dashboard without a problem.

The most successful applications implement performance plans with their end users in mind. End users have high expectations of interactions with applications and the seamlessness of their experience. They expect reliable application access and predictable performance. A research study on user experience revealed that one second is the longest wait time for end-users to stay within their flow. After ten seconds a user’s attention will be lost unless there is feedback to keep them engaged. A focus on the end-user experience allows you to identify meaningful indicators of application availability. Start by defining the minimum user experience and ensuring their expectations are met, and then use data from that point to improve your application.

The second step to creating a performant application is to figure out what to measure. Each application has several layers (application, components, services, and infrastructure), and each stack has its own metrics and monitoring methodologies. A lot of “noise” is generated in the monitoring data, and it can seem daunting to know what to measure and monitor. Figuring out what is important to your end-users gives you a starting point and a goal. Keep in mind that the purpose of performance monitoring is to detect issues in a system and minimize their impact on the end-user.

The final step will be to keep an eye on our overall process by defining Key Performance Indicators. Two important KPIs to consider are:
A. Mean Time to Detect (MTTD): The average amount of time it takes to discover an issue.
B. Mean Time to Resolve (MTTR): The average amount of time it takes to resolve an issue.

As application developers, we are required to maintain visibility into all of the areas that influence the end user’s experience. This can mean navigating competing priorities, budgetary issues, and stakeholder requirements. However, maintaining the primacy of the end user’s experience can be our guide through murky waters. There will always be unexpected performance problems, our proverbial dragons, but we can plan for them and navigate our way through.

Every data-driven project calls for a review of your data architecture. Download the white paper to see the 7 most common approaches to building a high-performance data architecture for embedded analytics.

 

 

Originally Posted at: Mapping the Path to a Performant Application

Large-scale computing power, combined with input from millions of fitness trackers, could help unlock the mysteries of our national insomnia.

I’m playing tennis with Marissa Mayer, and oddly, the Yahoo YHOO -1.75% CEO is wearing a pearlescent purple gown and sipping from a teacup. Her dress is just long enough to obscure her feet, so she appears to be floating across the baseline. As she strikes the ball, she tips her chin skyward and laughs in slow motion.

Meanwhile, I’m perched in the lotus position atop a manta ray that’s hovering above the ground like some kind of Landspeeder. And I’m panicking. How can I keep my balance and still hit the ball—especially with my shirt collar pulling at my neck the way it is? Can’t swing my racket. I jerk my head left. Then right. I claw at my jawline. The ball has cleared the net, and it’s headed my way. If only. I could. Just. Move. My head.

And poof. She’s gone. I open my eyes in a strange room. It’s pitch dark and completely silent, but I manage to find my bearings. Santa Cruz, Calif. Breathing heavily, I carefully disentangle a gaggle of wires twisted around my neck and roll over to glance at the clock. Just after 3 a.m.

---

This scene, I now know, was merely one of 18 REM-sleep interruptions that I experienced between 11:18 p.m. and 6:16 a.m. during one long February night. What a strange setting for the only dream I’ve ever had about a chief executive: in a laboratory, tethered to a byzantine apparatus designed to monitor my brain activity as well as every breath, eye movement, muscle twitch, and heartbeat.

Let me explain. Like you and probably everyone you know, I’ve always been confounded by my sleep routine. Why do I one morning rise ready to tackle the day and the next seem barely able to lift my head? How much rest can I be getting if I wake up sideways with the covers on the floor and my wife in the guest room? Most important, what can I do better? I don’t want a magic pill. I’ve tried those. I know the rules of thumb: less stress, more exercise, better diet, no afternoon caffeine, put down the damn phone. But I’d kill for a personalized formula.

So I subjected myself to a polysomnography test, or PSG, hoping to unravel some of the mysteries of the night. My procedure was administered in the offices of Fullpower Technologies, one floor down from where I had spent most of the evening talking with the company’s founder and CEO, Philippe Kahn.

A French expatriate who grew up in Paris, Kahn, 63, is a Silicon Valley oracle whose track record predates the web. He founded Borland Software (acquired by Micro Focus) MCFUF -1.09% in the mid-1980s, followed by Starfish software (Motorola) and LightSurf Technologies (VeriSign) VRSN -1.14% . In 1997, while anticipating the birth of his daughter, he paired a state-of-the-art Casio CSIOY 0.51% camera with a Motorola Startac and became, he claims, the first person to transmit a digital photo over cellular airwaves. He’s also been a leader in wearable technologies.

That’s precisely the focus of Fullpower, which licenses its software to other companies. Nearly five dozen framed patents for wearable-related software and devices hang on the wall in the company’s lobby. The oldest dates to 2005, long before tracking steps became such a phenomenon. In the conference room there’s an assembly of chairs and tables around a full-size bed, making obvious Kahn’s latest obsession.

Fullpower built the lab about a decade ago to capture data from sleep patterns. Of course, test subjects don’t typically snooze deeply with wires glued to their skulls, chests, legs, and arms. But almost everyone manages to at least nod off for a while, and the data that subjects generate are valuable and often surprising. “What we found early on is that sometimes you sleep less and feel more refreshed,” Kahn says. “It’s because you woke up in the light part of the sleep cycle.” The insight led him to develop a sleep-cycle alarm that could determine the best time to alert a person within a certain window. “Sometimes it’s better to get up at 10 of seven than at seven,” he says.

Kahn insists that he’s on the cusp of many more such discoveries, and he’s intent on dispelling some of the conventional wisdom that stresses people out. “People say that if you can’t sleep for eight hours without waking up, something’s wrong with you. That’s such a fallacy,” he says. “Before electricity, people used to sleep in two shifts. That’s how I behave. Sleep for four hours, get up and do an hour and a half of work, and then another four.” He’s also skeptical of the notion that a quiet room is the best environment for shut-eye and dismisses the perceived deleterious effects of repeated rousing. “The sign of good sleep hygiene may not be how many times you wake up, but rather how rapidly you fall back to sleep. Sleep should be like hunger. Eat only when you’re hungry and until you’re satisfied.”

Fullpower has oceans of data to back Kahn’s theories. The company provides the sleep-tracking and activity-monitoring software for the Jawbone UP and Nike Fuel NKE -0.85% wearable devices as well as a new line of Swiss-made smartwatches and the forthcoming Simmons Sleeptracker Smartbed. The products transmit a mother lode of information (with users’ consent) to Kahn’s team. He thinks that by combining qualitative lab data and quantitative real-world data with machine learning, artificial intelligence, and other analytics technologies, he can unlock the secrets that so many of us walking dead are looking for: a better night’s sleep. “We’re operating a huge sleep experiment, worldwide, unlike anything anyone has ever done,” he says. “We have 250 million nights of sleep in our database, and we’re using all the latest technologies to make sense of it.”

Kahn is not alone. He’s part of a movement of brilliant entrepreneurs, data scientists, engineers, and academics who are looking at demographics, geographies, and lifestyles, and even into our genomes. They’re the beneficiaries of a historic explosion in sleep data, and they’re using many of the same technologies that are busily decoding some of the world’s other great mysteries. Tiny sensors, big data, analytics, and cloud computing can predict machine breakage, pinpoint power outages, and build better supply chains. Why not put them to work to optimize the most valuable complex system of all, the human body?

It’s not an exaggeration to say lack of sleep is killing us. The Centers for Disease Control and Prevention calls it a public health epidemic and estimates that as many as 70 million Americans have a sleep disorder. Sleep deprivation has been linked to clinical depression, obesity, Type 2 diabetes, and cancer. The National Highway Traffic Safety Administration estimates that drowsy driving causes 1,550 deaths and 40,000 injuries annually in the U.S.  There are 84 sleep disorders, and some 100 million people—80% of them undiagnosed—suffer from one of them in particular: Obstructive sleep apnea, generally indicated by snoring, costs the U.S. economy as much as $165 billion a year, according to a Harvard Medical School study. That’s more than asthma, heart failure, stroke, hypertension, or drunk driving. And the study doesn’t account for tangential effects, like loss of intimacy and divorce. BCC Research predicts that the global market for sleep-aid products—everything from specialty mattresses and high-tech pillows to drugs and at-home tests—will hit $76.7 billion by 2019.

The financial upside for anyone who can crack the sleep code is obvious. And so the race is on. “I believe that 15 years from now, if we do this right, we can actually tackle epidemics like obesity, diabetes, and high blood pressure, and any number of lifestyle diseases,” says Kahn. “We’re going to help people live longer and better lives.”

---

In 1934 this magazine ran a sprawling cover story under the simple headline “Sleep.” Published about three decades after the dawn of sleep research, it explored a litany of the day’s devices, concoctions, drugs, and homespun remedies designed, or at least purported, to help insomniacs. The article chronicled the merits of everything from Ovaltine to morphine and Barbital, a synthetic hypnotic that was used as a sleep aid until the mid-1950s despite being habit-forming and occasionally deadly. The story also unpacked the relationship between Zalmon G. Simmons and Harry M. Johnson. Simmons was the president of the eponymous company his father founded, which has by now sold nearly 100 million mattresses, most notably under the Beautyrest brand. Johnson was a psychology professor at Ohio State who was conducting sleep research on the side.

At the time, old Zalmon was looking to solve two problems: He needed to sell more mattresses. And he needed sleep. The head of the iconic mattress maker was an insomniac! So he gave Johnson $25,000 to set up a sleep lab at the university now known as Carnegie Mellon. At the time it was believed that normal sleepers lay motionless throughout the night. Johnson collected 2.5 million data points from 160 subjects and demonstrated the folly of that theory. Normal sleepers, he determined, change positions from 20 to 85 times a night, with 35 being a sort of sweet spot. When we toss and turn more, we wake up groggy; any less, and we’re stiff and sore. (The insight would pay dividends for Simmons, the company. Spurred by the revelation that proper sleep required movement, it became the first to introduce king and queen-size mattresses. Unfortunately, Zalmon didn’t solve his personal problem. His insomnia devolved into depression and, in the words of the article, his “fortune began to wobble. Soon after that, sleep ceased to be a problem. For Zalmon G. Simmons was dead.”)

Those movements are what enable today’s wearable devices to monitor sleep. Gartner Group IT -1.08% projects sales of 36 million smart wristbands, a category that includes most Fitbit FIT -2.50% devices, Jawbone UP, and Nike Fuel, in 2015 and 2016. Fitbit is definitely king. NPD Group estimates that it has an 85% market share. Fitbit raised $732 million in a highly anticipated IPO stoked by the impressive metrics revealed in its S-1 filing: nearly $132 million in net income on $745 million in revenue last year, with $336 million in revenue in the first quarter of this year. Fitbit has sold more than 21 million devices since 2008 and has 9.5 million active users.

That means Fitbit is gathering far more data in one night than Johnson did in his career. Thanks to its sleep-monitoring technology, the company can distinguish among variations in movements and has determined that the majority of its users actually wake up between three and 17 times a night—9.3 times being average. Most awakenings are short and probably not remembered. Fitbit’s data suggest that the number of awakenings drops with age until about 45, and then remains flat. Men under 50 have, on average, one more awakening per night than women of the same age, but the discrepancy largely disappears after 50.

Fitbit’s research team developed the algorithms to gather and analyze all these data under the guidance of Shelten Yuen. Yuen, 36, who has a Ph.D. in engineering sciences, has worked in missile defense and has done research in surgical robotics. He was approached by Fitbit’s co-founders back in 2010 to write the software underlying the wearable device they were envisioning. “Crazy sensor thing?” Yuen remembers thinking. “That’s my deal.” He signed on as employee No. 4.

The first goal for Fitbit was tracking steps. Yuen figured it would be a snap—until he realized that a regular accelerometer can’t distinguish between a wrist that’s moving with a walking gait and a wrist that’s, say, shoveling food. He delivered what he thought was the perfect device. “They said, ‘Did you try scratching your head? Did you put it in your pocket? How about strapped to a bra?’ Suddenly it became a crazy, phenomenally hard problem,” he remembers.

Steps are easy compared with sleep. For starters, what’s the difference between an absence of movement while trying to sleep and actual sleep? Wearables are beginning to incorporate heart-rate monitoring, which helps correlate sleep stages, but they don’t measure respiration or neural activity like a polysomnograph and so rely more heavily on inference.

Fitbit encounters skepticism from users, including me, about the accuracy of its sleep measurement. Yuen admits he hasn’t achieved perfection, but he largely brushes off the incredulity, saying it stems from the fact that our instincts for how well we’re sleeping don’t typically match reality. For steps, users can monitor their progress, but not so with sleep. (For his part, Kahn insists that he has the formula for maximum accuracy: He says his huge quantity of rock-solid polysomnography data and wristband sleep-tracking information have helped him develop an algorithm that adjusts for the imperfections of wristband data.)

Whatever the accuracy of Fitbit’s data, the company has a ton of it, and is seeing some interesting patterns. Yuen calls one of his data scientists, Jacob Arnold, 31, into the conference room at Fitbit headquarters in San Francisco’s South of Market neighborhood. Arnold also has a Ph.D., his in astronomy and astrophysics. Together they unveil some graphs demonstrating that people around the world get more sleep in winter and less in summer. Daylight clearly has an effect on sleep patterns; temperature might too. We sleep less on the hottest day of the year than on the longest day. The data show that the longest average duration of sleep usually occurs on the weekend following New Year’s Day. Yuen also makes it clear that getting a lot of sleep is not equivalent to having a restful night.

Yuen and Arnold then reveal a graph demonstrating that, statistically speaking, no one gets less sleep than the demographic group to which I belong, forty-something males. I’m not sure how to take the news. Is it a body blow or a hopeful sign of more rest in my future?

Establishing the norm the way these charts do is an important step from a scientific standpoint. But to be indispensable in the marketplace, Fitbit needs to empower users with real guidance. “We need to take the veil off of sleep,” says Yuen. “We need to make this data much more understandable so they don’t feel powerless.”

Fitbit co-founder and CEO James Park echoes the sentiment. Some analysts are predicting that fitness trackers have peaked, especially because of the launch of the Apple Watch (which doesn’t track slumber because it needs to be charged every night). Park knows that if he can help his customers sleep better, Fitbit will become irreplaceable. “A lot of clinical studies show that certain behaviors might affect sleep: light levels, noise levels, caffeine, time of exercise,” says Park. “The next step is to tie prescriptive elements into it. That’s one of the big opportunities for this category. Can companies actually deliver coaching services to help people sleep that consumers actually want to pay for?”

---

Not long after meeting Steve Kay, dean of the Dornsife school of arts and sciences at the University of Southern California, I notice he’s wearing a Fitbit. Curious about what kind of a sleeper one of the world’s leading experts in circadian rhythm would be, I ask how he’d fared the night before. He looks at the app on his phone. “I went to bed at a quarter past 11 and got up at 20 past four,” he says with a shake of the head. Count Kay among the many of us getting less rest than we need.

I’m in Kay’s conference room with Ross Bersot, a venture partner at Bay City Capital turned entrepreneur. Bersot, 40, and Kay, 53, are a latter-day Zalmon Simmons and Harry Johnson. A few years ago, Kay published research demonstrating that it’s possible to isolate and manipulate the speed of an organism’s circadian rhythm using molecular compounds. Bersot provided funding to Kay’s lab in hopes of potentially starting a company based on more detailed findings. That company, Reset Therapeutics, is now real and based in the Bay Area.

Like Fullpower and Fitbit, Reset is intent on getting to the heart of the nature of sleep. But while others duke it out for wrist real estate, Reset is going deep into our bodies with a different set of tools. It’s pairing Kay’s research with the latest advances in genomics and high-throughput screening, and partnering with personal-genetics startup 23andMe to try to develop a drug that can manipulate our circadian rhythm.

Almost every living thing on earth has an internal clock—or more accurately, a symphony of clocks ticking away at the cellular level—that tells its owner when to rest and when to produce. It’s not just humans. Monsanto recently published a paper demonstrating that it could boost soybean yields 5% by tweaking one of the plant’s clock-associated genes. “We’re trying to help them figure out why,” says Kay, whose lab is working with Monsanto. “And all the major ag and biotech companies are interested. When you’re a plant stuck in the ground, everything is circadian. Biomass, yield, response to extreme temperatures—it’s all clock regulated.”

Human clocks are equally powerful, but we aren’t quite as enslaved by them because we don’t depend on photosynthesis. By some estimates, up to 70% of us live in spite of our internal clocks, or in the scientific jargon, outside our chronotype. Your chronotype predisposes you to being a night owl or a morning person or even being hungry at certain times, and many of us are guilty of ignoring our chronotypes, whether because of our lifestyle or just the call of duty.

When we live largely out of sync with our chronotypes, we experience everything from grogginess and reduced productivity to digestion issues, weight gain, and accelerated aging. We also increase our risk of obesity and Type 2 diabetes. Kay cites studies that link such lifestyles to an increased rate of breast cancer. “We evolved to adapt on this planet that has periods of light and dark. But modern life is really clashing with that,” he says. “The key is being internally synchronized. Your sleep-wake cycles, your food-intake cycles, your metabolic cycles are all working together. When they’re out of sync, that’s when (a) you feel lousy, and (b) you start seeing the disturbance of all kinds of markers. Sleep efficiency collapses. Leptin plummets. Insulin goes up.”

Reset wants to tweak our clocks with a drug. Such a drug would help reset the chronotypes of people who choose or need to live a different lifestyle or whose clocks are impaired. For now the development phase is limited to two so-called orphan diseases, narcolepsy—a neurological disorder that causes significant daytime sleepiness and in some cases cataplectic attacks—and Cushing’s disease, a neuroendocrine disorder characterized by excess blood sugar, obesity, and pituitary tumors.

Working with orphan diseases enables the company to help people sooner and get on the fast track to regulatory approval. “With the Cushing’s program, we’re focused on a protein that makes a clock run at a 24-hour cycle,” says Bersot. And so far, so good: “We’re restoring the rhythms to a 24-hour period, and in our early work with animals, we’ve seen blood glucose and insulin levels return to normal as a result.” Reset plans to take the Cushing’s drug to clinical trials next year and the narcolepsy treatment after that. Bersot hopes that once the drug establishes its ability to reset the body clock for sufferers of those diseases, Reset will develop it more broadly as a sleep aid for shift workers and anyone else whose clock and lifestyle are misaligned.

In the meantime the company is working with 23andMe to explore correlations between chronotypes and various genetic expressions. According to Emily Drabant Conley, 23andMe’s director of business development and a Ph.D. in neuroscience, 80% of 23andMe’s customers allow their genotypes to be used in research. She’s scouring their files in hopes of finding correlations between genotypes and sleep-related survey data. For example, how do people of a given genetic expression most commonly answer these questions: How many hours of sleep do you average? Do you snore? Do you take sleep medication? “Sleep is interesting because it’s not like eye color, where you have only five choices. Having a lot of data that you can mine is important,” she says. “But we’re just at the beginning. We’ve done the first pass at the genetics of being a morning person. We know that genetics plays a role in whether you need six hours or eight hours. The prescriptive aspect will come.”

---

It’s a common refrain, and despite the way it may sound to someone yearning for rest, it’s hardly a cop-out. In the realm of data science there’s a commonly accepted notion of how progress occurs. First comes descriptive analytics, next predictive analytics, and finally prescriptive analytics. In other words, describe the system, predict outcomes based on those descriptions, and prescribe actions to attain desired outcomes. To the layperson, prescription may sound like the most difficult phase, but in reality the hardest part is often where we are today, measuring and mapping behavior just to get to the point where it’s conceivable to predict behavior and prescribe actions.

Science is clearly advancing in the quest to understand sleep. We’re adding data in quantities that were unimaginable even a decade ago, much less during the time that Fortune first considered the subject. Big Pharma and biotech companies are drilling into our genomes to find links between DNA and nighttime routines; they’re also exploring medications that adjust the circadian rhythm.

Wearable-device companies are casting nets far and wide. Fitbit recently enabled its devices to track sleep automatically, rather than as a user-triggered behavior, and some of its devices now monitor heart rate, which helps Yuen and Arnold make finer distinctions between lack of motion and restful sleep. Full-power just entered a partnership with Simmons to provide sleep-tracking technology for the mattress maker’s new Sleeptracker Smartbed, which will provide noninvasive sleep-monitoring technology and is scheduled to hit the market in 2016.

On top of all this, scientists around the world are using cheap wearables to conduct their own exploratory research. A quick search of Google Scholar shows dozens of wearable-powered experiments and papers detailing the efforts to unravel the mysteries of the night. All of which will—one day soon, let’s hope—translate to a more personalized analysis and formula for how each of us can get a better night’s sleep.

Which brings me back to my night in Fullpower’s sleep lab. A few days after my test I received a PSG report that shines a light on my own personal darkness. The prognosis was mixed. The good news: My sleep-efficiency score was a robust 93.5%—“You should be proud of that,” says Kahn—which means that most of the time I’m in bed, I’m sleeping. And I’m quick. I drifted off in 2.5 minutes, even with all those wires attached. The bad: I snored. In fact, I was diagnosed with “mild to moderate” apnea. That officially makes me part of the $165 billion problem. I was advised to see a doctor.

That’s not the prescription I was looking for. I’m not crazy about doctors, and I’m afraid of being persuaded to buy a machine that I won’t use. So for now, I think I’ll just keep trying to do what I know to be right. Exercise. Keep a consistent routine. Restrict the hours that I eat, try to lay off the junk food and booze, and hope that before long someone will offer me a more personalized regimen.

Until then, I’ll always have the memory of a laughing, volleying Marissa Mayer and the knowledge that together, on one strange night in Santa Cruz, we—and whatever caused my 17 other REM interruptions—slept, or at least tried to, in the name of progress.

To read the original article on Fortune, click here.

Source: Can big data help you get a good night’s sleep?

Devo is a sponsor of TechSpective Cybersecurity is challenging. It is a daunting exercise to protect a complex hybrid cloud infrastructure from a rapidly evolving and expanding threat landscape. Organizations invest significant time, money and resources to deploy and manage a suite of firewalls, endpoint security, intrusion detection, and other cybersecurity tools, and yet network […]

The post Improving and Automating Threat Intelligence for Better Cybersecurity appeared first on TechSpective.

Originally Posted at: Improving and Automating Threat Intelligence for Better Cybersecurity

Big Data in Healthcare Today

A number of use cases in healthcare are well suited for a big . Some academic- or research-focused healthcare institutions are either experimenting with big data or using it in advanced research projects. Those institutions draw upon data scientists, statisticians, graduate students, and the like to wrangle the complexities of big data. In the following sections, we’ll address some of those complexities and what’s being done to simplify big data and make it more accessible.

A Brief History of Big Data in Healthcare

In 2001, Doug Laney, now at Gartner, coined the term “the 3 V’s” to define big data–Volume, Velocity, and Variety. Other analysts have argued that this is too simplistic, and there are more things to think about when defining big data. They suggest more V’s such as Variability and Veracity, and even a C for Complexity. We’ll stick with the simpler 3 V’s definition for this piece.

In healthcare, we do have large volumes of data coming in. EMRs alone collect huge amounts of data. Most of that data is collected for recreational purposes according to Brent James of Inter-mountain Healthcare. But neither the volume nor the velocity of data in healthcare is truly high enough to require big data today. Our work with health systems shows that only a small fraction of the tables in an EMR database (perhaps 400 to 600 tables out of 1000s) are relevant to the current practice of medicine and its corresponding analytics use cases. So, the vast majority of the data collection in healthcare today could be considered recreational. Although that data may have value down the road as the number of use cases expands, there aren’t many real use cases for much of that data today.

There is certainly variety in the data, but most systems collect very similar data objects with an occasional tweak to the model. That said, new use cases supporting genomics will certainly require a big data approach.

Health Systems Without Big Data

Most health systems can do plenty today without big data, including meeting most of their analytics and reporting needs. We haven’t even come close to stretching the limits of what healthcare analytics can accomplish with traditional relational databases—and using these databases effectively is a more valuable focus than worrying about big data.

Currently, the majority of healthcare institutions are swamped with some very pedestrian problems such as regulatory reporting and operational dashboards. Most just need the proverbial “air and water” right now, but once basic needs are met and some of the initial advanced applications are in place, new use cases will arrive (e.g. wearable medical devices and sensors) driving the need for big-data-style solutions.

Barriers Exist for Using Big Data in Healthcare Today

Several challenges with big data have yet to be addressed in the current big data distributions. Two roadblocks to the general use of big data in healthcare are the technical expertise required to use it and a lack of robust, integrated security surrounding it.

Expertise

The value for big data in healthcare today is largely limited to research because using big data requires a very specialized skill set. Hospital IT experts familiar with SQL programming languages and traditional relational databases aren’t prepared for the steep learning curve and other complexities surrounding big data.

In fact, most organizations need data scientists to manipulate and get data out of a big data environment. These are usually Ph.D.-level thinkers with significant expertise—and typically, they’re not just floating around an average health system. These experts are hard to come by and expensive, and only research institutions usually have access to them. Data scientists are in huge demand across industries like banking and internet companies with deep pockets.

The good news is thanks to changes with the tooling, people with less-specialized skillsets will be able to easily work with big data in the future. Big data is coming to embrace SQL as the lingua franca for querying. And when this happens, it will become useful in a health system setting.

Microsoft’s Polybase is an example of a query tool that enables users to query both Hadoop Distributed File System (HDFS) systems and SQL relational databases using an extended SQL syntax. Other tools such as Impala, enable the use of SQL over a Hadoop database. These types of tools will bring big data to a larger group of users.

Security

In healthcare, HIPAA compliance is non-negotiable. Nothing is more important than the privacy and security of patient data. But, frankly, there aren’t many good, integrated ways to manage security in big data. Although security is coming along, it has been an afterthought up to this point. And for good reason. If a hospital only has to grant access to a couple of data scientists, it really doesn’t have too much to worry about. But when opening up access to a large, diverse group of users, security cannot be an after thought.

Healthcare organizations can take some steps today to ensure better security of big data. Big data runs on open source technology with inconsistent security technology. To avoid big problems, organizations should be selective about big data vendors and avoid assuming that any big data distribution they select will be secure.

The best option for healthcare organizations looking to implement big data is to purchase a well-supported, commercial distribution rather than starting with a raw Apache distribution. Another option is to select a cloud-based solution like Azure HDInsight to get started quickly. An example of a company with a well supported, secure distribution is Cloudera. This company has created a Payment Card Industry (PCI) compliant Hadoop environment supporting authentication, authorization, data protection, and auditing. Surely other commercial distributions are working hard to add more sophisticated security that will be well-suited for HIPAA compliance and other security requirements unique to the healthcare industry.

Big Data Differs from the Databases Currently Used in Healthcare

Big data differs from a typical relational database. This is obvious to a CIO or an IT director, but a brief explanation of how the two systems differ will show why big data is currently a work in progress—yet still holds so much potential.

Big Data Has Minimal Structure

The biggest difference between big data and relational databases is that big data doesn’t have the traditional table-and-column structure that relational databases have. In classic relational databases, a schema for the data is required (for example, demographic data is housed in one table joined to other tables by a shared identifier like a patient identifier). Every piece of data exists in its well-defined place. In contrast, big data has hardly any structure at all. Data is extracted from source systems in its raw form stored in a massive, somewhat chaotic distributed file system. The Hadoop Distributed File System (HDFS) stores data across multiple data nodes in a simple hierarchical form of directories of files. Conventionally, data is stored in 64MB chunks (files) in the data nodes with a high degree of compression.

Big Data Is Raw Data

By convention, big data is typically not transformed in any way. Little or no “cleansing” is done and generally, no business rules are applied. Some people refer to this raw data in terms of the “Sushi Principle” (i.e. data is best when it’s raw, fresh, and ready to consume). Interestingly, the Health Catalyst Late-Binding Data Warehouse follows the same principles. This approach doesn’t transform data, apply business rules, or bind the data semantically until the last responsible moment–in other words, bind as close to the application layer as possible.

Big Data Is Less Expensive

Due to its unstructured nature and open source roots, big data is much less expensive to own and operate than a traditional relational database. A Hadoop cluster is built from inexpensive, commodity hardware, and it typically runs on traditional disk drives in a direct-attached (DAS) configuration rather than an expensive storage area network (SAN). Most relational database engines are proprietary software and require expensive licensing and maintenance agreements. Relational databases also require significant, specialized resources to design, administer, and maintain. In contrast, big data doesn’t need a lot of design work and is fairly simple to maintain. A lot of storage redundancy allows for more tolerable hardware failures. Hadoop clusters are designed to simplify rebuilding of failed nodes.

Big Data Has No Roadmap

The lack of pre-defined structure means a big data environment is cheaper and simpler to create. So what’s the catch? The difficulty with big data is that it’s not trivial to find needed data within that massive, unstructured data store. A structured relational database essentially comes with a roadmap—an outline of where each piece of data exists. On the big data side, there are no traditional schemas, and therefore not much guidance. With a relational database, a simple, structured query language (i.e. SQL) pulls the needed data using a sophisticated query engine optimized for finding data.

With big data, the query languages are much more complicated. A sophisticated data user—such as a data scientist—is needed to find the subset of data required for applications. Creating the required MapReduce algorithms for querying big data instances isn’t for the faint of heart. Fortunately, that’s changing at a fairly rapid pace with tools like SparkSQL and other query tools that leverage conventional SQL for querying. Big data query engines can now convert SQL queries into MapReduce jobs while others like the aforementioned Microsoft PolyBase can join queries from a traditional relational database and Hadoop then return a single result set.

In short, big data is cheap but more difficult to use. Relational databases are expensive but very usable. The maturity level of big data technology is low–after all the big data journey only began a few short years ago. So, as the tooling and security catches up with its potential, health systems will be able to do exciting things with it.

It’s Coming: Big Data Will Be Important in Healthcare

When healthcare organizations envision the future of big data, they often think of using it for analyzing text-based notes. Current analytics technologies for the most part make use of discrete data and struggle to capitalize on all of the valuable clinical information captured in physicians’ and nurses’ notes. Big data indexing techniques, and some of the new work finding information in textual fields, could indeed add real value to healthcare analytics in the future.

Big Data and the Internet of Things

Big data will really become valuable to healthcare in what’s known as the internet of things (IoT). SAS describes the IoT as:

The Internet of Things is a growing network of everyday objects from industrial machines to consumer goods that can share information and complete tasks while you are busy with other activities, like work, sleep, or exercise. Soon, our cars, our homes, our major appliances, and even our city streets will be connected to the Internet–creating this network of objects that is called the Internet of Things, or IoT for short. Made up of millions of sensors and devices that generate incessant streams of data, the IoT can be used to improve our lives and our businesses in many ways.

The analyst firm Gartner projects that by 2020 there will be more than 25 billion connected devices in the IoT. For healthcare, any device that generates data about a person’s health and sends that data into the cloud will be part of this IoT. Wearables are perhaps the most familiar example of such a device. Many people now can wear a fitness device that tracks how many steps they’ve taken, their heartrate, their weight, and how it’s all trending. Apps are available on smart phones that track how often and how intensely a user exercises. There are also medical devices that can also send data into the cloud: blood pressure monitors, pulse oximeters, glucose monitors, and much, much more.

Big Data and Care Management

ACOs focus on managed care and want to keep people at home and out of the hospital. Sensors and wearables will collect health data on patients in their homes and push all of that data into the cloud. Electronic scales, BP monitors, SpO2 sensors, proximity sensors like iBeacon, and soon-to-be-invented sensors will blast data from millions of patients continually. Healthcare institutions and care managers, using sophisticated tools, will monitor this massive data stream and the IoT to keep their patients healthy.

And all of this disparate sensor data will come into healthcare organizations at an unprecedented volume and velocity. In a healthcare future predicated on keeping people out of the hospital, a health system’s ability to manage all this data will be crucial. These volumes of data are best managed as streams coming into a big data cluster. As the data streams in, organizations will need to be able to identify any potential health issues and alert a care manager to intervene. For example, if a patient’s blood pressure spikes, the system will send an alert in real time to a care manager who can then interact with the patient to get his blood pressure back into a healthy range.Big data is the only hope for managing the volume, velocity, and variety of this sensor data.

The Fun Stuff: Using Big Data for Predictive Analytics, Prescriptive Analytics, and Genomics

Real-time alerting is just one important future use of big data. Another is predictive analytics. The use cases for predictive analytics in healthcare have been limited up to the present because we simply haven’t had enough data to work with. Big data can help fill that gap.

One example of data that can play a role in predictive analytics is socioeconomic data. Socioeconomic factors influence patient health in significant ways. Socioeconomic data might show that people in a certain zip code are unlikely to have a car. There is a good chance, therefore, that a patient in that zip code who has just been discharged from the hospital will have difficulty making it to a follow-up appointment at a distant physician’s office. (Health systems have, in fact, found that it is cheaper to send a taxi to pick a patient up for an appointment than it is for her to miss the appointment and be readmitted to the hospital.)

This and similar data can help organizations predict missed appointments, noncompliance with medications, and more. That is just a small example of how big data can fuel predictive analytics. The possibilities are endless.

Patient Flight Paths and Prescriptive Analytics

Another use for predictive analytics is predicting the “flight path” of a patient. Leveraging historical data from other patients with similar conditions, predictive algorithms can be created using programming languages such as R and big data machine learning libraries to faithfully predict the trajectory of a patient over time.Once we can accurately predict patient trajectories, we can shift to the Holy Grail–Prescriptive Analytics. Intervening to interrupt the patient’s trajectory and set him on the proper course will become a reality very soon. Big data is well suited for these futuristic use cases.

Genomic Sequencing and Big Data

As someone who’s spent many years working on the Human Genome project, I am personally very excited about the increasing use of genomic data in patient treatment. The cost of sequencing an individual’s full genome has plunged in recent years. Sequencing, once an art, will soon become commonplace and eventually become a commodity lab test. Genomic sequences are huge files and the analysis of genomes generates even more data. Again, big data serves this use case well. Loading a genetic sequence into a relational database would require a huge Character Large Object (CLOB) or a separate storage just to manage the sequence. With big data, just toss it in the Hadoop cluster, and it’s ready for analysis.

The Future of Healthcare Data Warehousing and the Transition to Big Data

I’ve talked about the present limitations for big data in healthcare and the truly fascinating future possibilities that big data enables. An important question to address at this point is, of course, this: What should a health system do in the meantime? Today, health systems’ need for data-driven quality and cost improvement is urgent. Healthcare organizations cannot afford to wait for big data technology to mature before diving into analytics. The important factor will be choosing a data warehousing solution that can easily adapt to the future of big data.

A Late-Binding enterprise data warehouse (EDW) architecture is ideal for making the transition from relational databases to unstructured big data. As stated earlier, the late-binding approach is, in fact, very similar to the big data approach. In a Late-Binding EDW like Health Catalyst’s, data from source systems (EHRs, financial systems, etc.) are placed into source marts. In this process—as in big data—it is best practice to keep the data as raw as possible, relying on the natural data models of the source systems. As much as possible, late-binding methods minimize remodeling data in the source marts until the analytic use case requires it. The data remains in its raw state until someone needs it. At that point, analysts package the data into a separate data mart and apply meaning and semantic context so that effective analysis can occur. Because this approach is so similar to big data, it is a natural transition to replace the source-mart layer of the EDW architecture with a big data cluster.

Real World Example Healthcare’s Transition to Big Data

In conclusion, here is a brief example of how the transition from relational databases to big data is happening in the real world. We, at Health Catalyst, are working with one of our large health system clients and Microsoft to create a massively parallel data warehouse in a Microsoft APS Appliance that also includes a Horton works Hadoop Cluster. This means we can run a traditional relational database and a big data cluster in parallel. We can query both data stores simultaneously, which significantly improves our data processing power. Together, we are beginning to experiment with big data in important ways, such as performing natural language processing (NLP) with physician notes, predictive analytics, and other use cases.

The progression from today’s symmetric multiprocessing (SMP) relational databases to massively parallel processing (MPP) databases to big data in healthcare is underway.

 

—

(Source: http://bit.ly/2v5Wk8A)

 

Source

We are excited to announce that Azure Databricks is now in Microsoft’s Azure Government region, enabling new data and AI use cases for federal agencies, state and local governments, public universities, and government contractors to enable faster decisions, more accurate predictions, and unified and collaborative data analytics. More than a dozen federal agencies are building cloud data lakes and are looking to use Delta Lake for reliability.

Azure Databricks is trusted by organizations such as Credit Suisse, Starbucks, AstraZeneca, McKesson, Deutsche Telekom, ExxonMobil, H&R Block, and Dell for business-critical data and AI use cases. Databricks maintains the highest level of data security by incorporating industry leading best practices into our cloud computing security program. Azure Government public preview provides customers the assurance that Azure Databricks is designed to meet United States Government security and compliance requirements to support sensitive analytics and data science use cases. Azure Government is a gold standard among public sector organizations and their partners who are modernizing their approach to information security and privacy.

Enabling government agencies and partners to accelerate mission-critical workloads on Azure Government

High Impact data are frequently stored and processed in emergency services systems, financial systems, department of defense and healthcare systems. For example, Azure Databricks enables government agencies and their contractors to analyze public records data such as tax history, financial records, welfare and healthcare claims to improve processing times, reduce operating costs, and reduce claims fraud. In addition, government agencies and contractors need to analyze large geospatial datasets from GPS satellites, cell towers, ships and autonomous platforms for marine mammal and fish population assessments, highway construction, disaster relief, and population health. State and local governments who utilize federal data also depend on Azure Government to ensure they meet these same high standards of security and compliance.

Learn more about Azure Government and Azure Databricks

You can learn more about Azure Databricks and Azure Government by visiting the Azure Government website, see the full list of Azure services available in Azure Government, compare Azure Government and global Azure and read Microsoft’s documentation here.

Get started with Azure Databricks by attending this free, 3-part training series. Learn more about Azure Databricks security best practices by attending this webinar and reading this blog post.

As always, we welcome your feedback and questions and commit to helping customers achieve and maintain the highest standard of security and compliance. Please feel free to reach out to the team through Azure Support.

Follow us on Twitter, LinkedIn, and Facebook for more Azure Databricks security and compliance news, customer highlights, and new feature announcements.

—

Try Databricks for free. Get started today.

The post Azure Databricks Now Available in Azure Government (Public Preview) appeared first on Databricks.

Source: Azure Databricks Now Available in Azure Government (Public Preview) by analyticsweekpick