#bigdata and #smalldata

Maybe you've heard ...

... "Big Data" is going to save the world.  And generate profit for your business, the kind of profit that only omnichannel solutions could theoretically generate.  "Big Data" will decide who our President will be, it will decide the future of health care, and it will play a crucial role in protecting and/or violating our privacy.

There are at least three components to big data these days.

1. Vendors who are offering solutions not all that dissimilar to the solutions offered over the past twenty-five years.
2. Pundits who talk about big data, hoping to garner page views or followers on Twitter.
3. People actually doing amazing work, not talking publicly about "Big Data" (hint, there's many people in this category).

If you're in the catalog world, then you've been dealing with "Big Data" for at least fifteen years ... you happily volunteered your most valuable asset (your customer list) to the co-ops.  Co-ops are the very definition of big data, they just never called themselves "Big Data".

If you're in the online world, then you've been dealing with "Big Data" for more than a decade.  Do you have any idea where your retargeting campaigns are being deployed?  Big Data is deciding that ... and has been for the past four thousand days.

Here are seven "Big Data" themes that are actually recycled and repackaged concepts from the past twenty-five years.

1. You need a new database infrastructure to handle the volume of data.
2. You need to combine data across all channels to obtain a "360 degree" view of the customer.
3. Your database infrastructure needs to be "fast" ... in the parlance of the day, providing results in "real time".
4. You need sophisticated data mining algorithms to find "nuggets of gold" in the data that humans cannot detect.
5. You need ad-hoc query tools that allow all employees to query the database on their own, obtaining their own answers.
6. You will need a testing platform, so that you can "optimize" your business results and become "data driven".
7. You will self-actualize, with business performance at the top of Maslow's Hierarchy of Needs.

Now, let's be honest.  There is something to "Big Data", it's not a fad, it's the hype that is out of control.

But it doesn't change where we spend the majority of our time, and it doesn't change these fundamental concepts:

1. You will always need a new database infrastructure, as the volume of data you'll manage will always increase.  This hasn't changed in 25 years.
2. We've been combining data from all channels since the advent of e-commerce.  The 360 degree view of customers didn't fundamentally change anything, it simply revealed that business is complicated.
3. Database infrastructures have always needed to be fast, going back to SB37 memory errors on IBM mainframe computers running Easytrieve Plus.
4. We've been told for decades that data mining algorithms would find "nuggets of gold" in the data that humans couldn't detect.  Remember the Neural Network craze of the early 1990s?  How many "nuggets of gold" do you recall unearthing from your exploration of Neural Networks and Genetic Algorithms?  A half-dozen in twenty years, if you're lucky?  Certainly not a half-dozen a week.
5. The business intelligence phase of vendor hype gave us ad-hoc query tools (Business Objects, Microstrategy).  How many garden-variety, non-data employees used these tools for anything other than producing simple reports?  Only a small fraction of employees using web analytics tools ever pushed this front forward.  Now, these poor web analytics folks are being told they are "outdated" in a "Big Data" world, so they are rebranding themselves as "digital" analysts.  These folks will always be valuable, regardless of the evolution of Big Data.  Their smarts matter, folks.
6. Folks have been testing/optimizing for thousands of years.  Complex multivariate testing has been documented as far back as the 1930s - 1940s ... Google "Snedecor and Cochrane" for details.
7. You will not self-actualize.  If anything, you'll implode.  "Big Data" will likely give you insights that are worthwhile, however, you have to pull the rest of your business with you, in the direction you want them to go.  Have you ever tried to align the thoughts of 50,000 employees at Nordstrom, for instance?  Hint - it's not easy!

I'd like to introduce a new concept to you.

Let's call it "small data", or as they say on Twitter, #smalldata.

"Big Data" seems to be about tools and techniques and data integration and hardware and software and automation and post-CRM theory and real-time analytics and KPIs and reporting.

"small data" is all about teaching people what you learn, evangelizing ideas, encouraging employees to be great via information.

#smalldata

1. Is all about analyzing the data you have to make decisions today.
2. Is about the "99%".  99% of the decisions you make on a daily basis have nothing to do with "Big Data".
3. Doesn't require integrated data across all sources compiled in real-time.
4. Can easily be accomplished in Excel.  Or Google Analytics.
5. Requires the analyst to be a great communicator.
6. Is about the message, not about the hardware, software, or database platforms.
7. Is actionable.
8. Is not geeky.
9. Is about teaching.
10. Is about profit.
11. Values and honors "what came before" Big Data.
12. Gladly leverages "Big Data" when appropriate.

#smalldata

Go evangelize it ... you use #smalldata every single day ... always have.