Maybe Too Big
Today, people have a few needs – just-in-time-information, constant feedback, things that do things for us and privacy.
That’s tough to achieve in our always-on world. O.K., it’s impossible.
We have our IoT (Internet of Things) and IoE (Internet of Everything) that will soon have 50B things sending information to each other and humongous data crunching machines in the cloud.
But the industry has it all wrong. It’s not about the IoT or IoE. It’s the Internet of Data.
Things Data – The IoT and IoE promise great things for just about everyone, giving them an even better, easier life. To do that, the data has to be gathered, digested and distributed. The money isn’t in the things it is in the data … once we know how to use it, protect it.
That’s bad enough but admit it, you just can’t keep your mouth shut about how awesome you really are.
Take any given day:
- Enough data is created to fill 168 Million DVDs
- 294 Million emails are sent (it would take 2 years to process that much paper mail)
- 180 Million people visit Facebook
- 390M Twitter, 3.5M LinkedIn, 60M Google, 25M Printerest
- 7B minutes are spent on facebook, 532M updates
- 250M photos are uploaded to Facebook, a stack as tall as 80 Eifel Towers
- 22M hours of old TV and movies are watched
- Equivalent of 89 years of non-stop cat movies are uploaded annually
- 7M hours of Pandora music is streamed
- 1288 New Apps are offered, 35M downloaded
- iPhone sales outpace births – 378,000 iPhones sold, 371,000 kids born
As if that isn’t enough, the population is about 7.250B people and they’re online with all of the social media and all of their devices creating more traffic (data):
Google’s:
- 300PB
- 5T indexed URLs
- 4M servers
- 5B page views/day
Facebook:
- 5B users
- 400PB
- 750TB/day
- 40 percent of world’s photos
YouTube
- 1500PB
- 72 hrs/min uploaded
- 37M hours/year uploaded
- 4B views/day
- 124B Tweets/year
- 390M Tweets/day
- 4500 Tweets/sec
Global text messages:
- 1T/year
- 193,000/sec
- 876 per person per year
Cell calls:
- 3T min/year
- 19 min/person/day
As Phil Bergmann said, “Ratings were off the chart.”
It’s little wonder that we have created 35-40 times more data/information in the past five years than in the previous 2000 plus years.
Plugged Along, then BAM! – The world of data just plugged along for thousands of years until the Internet and Web caught on. Today, we can’t shut people up. They keep handing out more and more information about more and more things to the point that in a few years, the volume of data will have increased 35-40 x everything that came before.
All you can do is look at the stats and agree with Phil Bergman, “You’re officially scaring the shit out of me right now.”
As an employee lawsuit recently pointed out, you’re already a walking sensor platform. Even if your boss didn’t have you add a tracking app on your phone, all of the information captured on your smartphone makes it possible for folks to know who you are and where you are. If someone else has your phone, it can tell that as well … even if it’s turned off.
Despite all of the noise and information put out there every day, organizations around the globe – and elected officials who don’t really understand the meaning of the words – are demanding individual privacy.
Being deaf to the noise, agencies and companies collect data knowing that their requirements will almost always be hindsight, so more is better.
Because you simply don’t know what information is going to be needed, useful.
Besides … it’s just too much to resist.
Pile it On – Even though marketers aren’t certain what the information they accumulate means or how it can be used, they continue to capture everything possible about customers and prospective customers. Unfortunately, the piles of data are usually less than secure, making it blindly easy for hackers and cyberthieves to rummage through and find the good stuff.
Over the last few years, the volume of data has exploded.
McKinsey reported that in most US economy sectors (15 out of 17), companies were storing an average of 235TB of data (more than in the Library of Congress). The data consists of ordinary stuff like financial data, customer transactions and customer profiles/info that will help them understand what people are doing, what they want, what they could sell them.
According to Connotate, more than half of companies they studied focused their big data work on monitoring the competition or their own brands (60 percent and 52 percent).
Driving Forces – Almost every business division and operating group has a big data initiative going to determine how they can leverage customer and business data to improve their operations and profits. The problem is they don’t share and they don’t understand security.
Their marketing study was focused on product/pricing info and revenue-generated data services.
Of course they also kept a watchful eye on what new things government departments/agencies might come up with that would mess over their business.
They gather the info from inside and outside the organization in some vague hope that they will suddenly gain insight into consumer interest/preferences so they can get more customers.
The biggest challenge in most companies?
Departments don’t share.
Stumbling Blocks – Capturing big data is the easy part. Making intelligent use of it and sharing it across the organization is still a slow work-in-progress.
Just because service/support has the information, it doesn’t mean engineering can access it; or if marketing has the data, that they are willing to share their precious resource.
Of course, you can’t blame either group because even if they have the info someone else needs, neither side would probably recognize it since it is too cumbersome to dig through the piles to find the good, meaningful stuff.
Today big data is right up there with IoT (Internet of things) and cloud everything. They’re all new, hot and sexy. So what’s not to like?
But if it’s going to be a worthwhile investment for your organization, data has to be collected across all business units and available to all units so each team can examine it and use it from their perspective with their goals in mind.
In the next few years, companies will waste a lot of time, money and effort in the area and will buy a lot of great snake oil solutions that look good but don’t do diddle for you.
Only folks who have had to dig deep into the big data (government agencies) will tell you right off the bat that:
- The analytics and tools are hell to use.
- Specialists (yet to be well-defined) are needed to get valuable info out of the data and they are in extremely short supply.
- We’re using old computer algorithms to chug through the data right now and the old stuff just doesn’t work.
- Folks are doing a lot of it by hand right now and that’s really tough.
- It sorta’, kinda’ works for business but it’s still a work-in-progress.
- Systems are needed that will do the heavy lifting, learn from users and correlate, not simply search.
The ultimate key to the success of big data is to deal with the total universe of information as well as the individual.
John Koestler insisted, “This is not a crank call.”
With successful big data handling/analysis comes the ability to see macro and micro trends to make better decisions.
Closer Examination – Increasingly sophisticated information monitoring/capture tools and consumer mobile/communications devices make it easier and easier for companies to know more about customers than they know about themselves–all in the name of delivering better customer service. Despite stumbling starts, organizations and governments are going to gain great insights into macro and micro tastes and changes.
Historical data can be just as helpful in enabling tomorrow’s opportunities.
Or as George Santayana’s reportedly said in 1863, “Those who cannot remember the past are condemned to repeat it”’
Management shouldn’t invest in big data programs with the idea they are going to make an enterprise-changing decision immediately. Instead, they should first focus on smaller scale efforts – controlled experiments – to test the hypotheses and analyze the results. With this approach, you can build on your successes and grow from there … slowly.
Over time, the success will accumulate and grow as management and team members learn how to analyze and make decisions based on the data.
Firms that connect with their customers will be able to track customers from clicks, update preferences and model their likely behavior in real time, such as offering a bundle of products that complement the product they were going to buy at a special price.
By combining data from online purchases, social network conversations and now location- specific smartphone interactions, customers can be more easily divided into micro segments to more effectively pinpoint offers and results.
Taking big data and analysis even further into the IoT area, production line sensors could result in self-regulating processes to minimize/eliminate human intervention, cut waste and produce more efficient output.
Firms are beginning to see that some of their data could be of interest/use to other firms. Often called “exhaust data,” it is residual data they develop from tracking their own workflow that could be sold to other organizations.
There’s a whole new industry cropping up today of data aggregation where information is gathered from multiple sources and sold to other firms. For example, insurance companies have long been anxious to obtain general and specific healthcare data and trends.
Of course, all of this produces a major concern with data security and privacy which is going to become a major hurdle for most organizations, whether they want to admit it or not.
There is a fine line between firm’s providing greater convenience to the consumer and the individual’s right to privacy.
Then too, every day there is a significant security breach somewhere in the world and it will only get worse as we approach Cisco’s dream of 50B sensors connected around the globe and stored in something as unreliable and unsecure as the cloud.
Cyber privacy and cyber security just may be bigger, more profitable businesses than big data.
Phil Bergman – “Right now, my scientific mind is telling me to have nothing more to do with this. And yours should, too.”
# # #