Follow Splunk Into The Big Data Revolution That Changes The Jobs Equation

A recent AP story opens with a sobering line: "The college class of 2012 is in for a rude welcome to the world of work." Opportunities are weak, the article notes, and those with ’soft’ degrees are having the most trouble. Hardly news to the Millennial generation. And it’s no comfort that their parents (count me amongst them) faced the same type of dire job market when they graduated.

We boomers didn’t know it then, but good times were just around the corner. The burgeoning revolution of personal computing and the Internet (together with favorable government policies) unleashed several decades of growth that was bullish not just for the tech dweebs that built the revolution, but for the broad economic engine that the revolution unleashed. Intel, Microsoft, AOL, Oracle, Sun, Apple and others changed the world.

And it was not as if there weren’t already big competing tech companies at that time, like IBM, HP, Digital Equipment, Burroughs, Sperry Rand, Control Data, Wang, and many others (many gone now). But much of the new hardware and software that made computing personal – - made it the domain of everyone — emerged from the new guys on the block. It fundamentally changed and expanded our economy creating growth and jobs across the board.

That was then. Can such a thing possibly happen again? You bet. Bear with me as we find the evidence in a specific company like Splunk.

Splunk [NASDAQ:SPLK], founded just eight years ago and expected to hit $100 million revenues this year, went public this past Thursday in a smoking hot IPO with its stock doubling that day giving it a market value of $3 billion. Splunk is amongst the first of many more such companies and IPOs yet to come, riding the wave of the Big Data revolution.

First, for those not daily surfing the techno-sphere, Splunk’s magic sauce is that they are "Google for machine data." That’s how Erik Swan, Splunk’s CTO and co-founder, presciently described it in an interview. All machines and most especially the machines that comprise the entire information economy, generate staggering quantities of data about what they’re doing. Data traffic is counted, logged, and categorized by all manner of machines in a network.

Consider what’s under the hood of your screen right now. There is a nearly indescribable quantity of equipment, devices, connections, servers, routers, links, switches, microprocessors, drivers, illuminators and more working so that you can read some text on your screen. All of that equipment – times billions in the network – must work in concert, perfectly and constantly to render your screen. Much of that machinery is tracking and logging (or reporting) about it’s own operations – not about the content per se, but about how the content is being managed.

Machine data about the Internet is growing at a faster rate than even the explosive growth of information on the Internet, the people data. And beyond the sheer quantity of machine data, there is the disordered nature of how it is generated, accessed and stored that makes control and forensics daunting. Splunk is about taking this kind of big data to an entirely new level of functionality, much as Apple [NASDAQ:AAPL] and Google [NASDAQ:GOOG] did in the previous two information revolutions.

The creation of intuitively simple systems/software that lets people manipulate structured data is what kicked off the first computing revolution. Accessible computing (think Apple’s operating system, quickly copied by Microscoft [NASDAQ:MSFT) enabled the Internet to balloon, quickly growing into a massive domain. Then came the search engines to make sense of that vast landscape of unstructured information – Netscape, Yahoo, Explorer, Firefox, and Google.

The Google family of companies came up with entirely new software and search paradigms to make sense of the unstructured Web. There’s no Dewey Decimal system for you to find a book on the Web. Everyone knows how easy it is to just "Google it" to find something on the Internet.

Splunk lets companies Google search what’s going on under the hood of the Web in order to improve the operation of the Big Data machinery itself. For the machines that make the Net, and for machines in general, gleaning trends and seeing patterns in unstructured data is deeply useful for improving operations, reliability, costs, security and even safety. And, it is also profoundly useful for seeing patterns about what the humans using the machines are doing, planning and even thinking – what they’re clicking on, how long, how often, etc.

Think what this means. Instead of the heretofore opaque domain of the IT infrastructure being managed and understood by a cadre (some say cult) of technical specialists, the generalists in the company can spelunk — hence Splunk’s name of course — where data traffic is coming from and going, what parts of networks are busy, over-taxed, or need attention. Splunk’s capabilities are powerful because it can make intuitive sense out of vast quantities of unstructured data – permitting simple Google-like queries and answers about how any IT system is operating. It permits logical, intuitive, human-centric analytics.

In the 21st century every business or government operation in any field uses a huge and wildly growing data set, from education and research to retail and manufacturing. Splunk can make their data work better, smoother, faster, cheaper – whether it’s their own IT infrastructure or a Cloud service.

Extracting patterns from machine data tells you much more than just the health of the machines themselves – as important as that is given the complexities, and how reliant everything is on them. It tells you a lot about why the humans are using those machines too. This is where the unstructured and meta-data analytics gets interesting.

Splunk’s own testimonials provide instructive examples. Macy’s solved black Friday data outage problems using Splunk, and also uses it to analyze customer shopping behavior illuminated by how traffic moves on their network. Salesforce.com, itself a database company (but the ’old’ structured kind) uses Splunk not just to improve operations but also to see how customers use their product’s myriad tools. Vodafone uses Splunk to make sure its routers and system behaves, and to watch for and deal with virus attacks that can be notoriously sneaky and hard to find with traditional monitoring. Most of Splunk’s 3,700 customers are likely doing similar things, extracting subtle, hidden, patterns.

For example, you may learn more, or sometimes more importantly you may learn sooner, about mortgage refinancing trends by tracking the records of people’s clicks on links about mortgages, than you can by collecting the (substantially delayed) data from the mortgage company applications themselves. The same can be said about many subjects, economic, business and disease indicators. One learns about flu outbreaks from rise in machine traffic linking to sites about the flu – - never mind what the questions are. The metadata itself tells you it’s happening.

The field of unstructured and meta-data analytics opens up the next great cycle in what computing enables. Splunk is a leader in this burgeoning domain.

Big Data analytics span every single part of human existence: vehicle traffic flows, health monitoring and treatment, commerce, entertainment, education, construction, weather and of course, warfare. Behind the structure of what you see on Facebook, Twitter, YouTube, Amazon, LinkedIn, your bank or school’s web site, everything, resides a hidden universe of unstructured data. It is, to use the astrophysics term, the "dark matter" of the Internet.

Data is associated not just with information-system machines (servers, routers, switches, power supplies), but also every other kind of machine (fetal heartbeat monitor and engine valves) or even inanimate objects (doors and lights). Data resides any and everywhere there’s a sensor of any kind measuring, tracking or logging anything and everything from temperature to state (open, closed). Nearly all of this data is unstructured, unorganized, in various so-called "silos," available at different times and in different ways.

A MarkLogic survey found unstructured data growing far faster than the structured (or "relational") data world. Credit goes to Joe Dalton, MarkLogic CMO, for the most telegraphic analogy of the importance of unstructured data analytics. He says that "using [a structured database] to try to tackle today’s information challenges ... is like trying to build a website with a typewriter."

I like to think of this as even more fundamental. Until humans had sorted out the ideas of smelting and forges, the scattered particles of iron ore dug up from the earth couldn’t be collected and organized into steel for surgical instruments and cars. Bringing order and structure out of unstructured chaos is what humans have been about for a very long time. The big shifts come only episodically.

Splunk is just one, albeit impressive, player in the rapidly growing field of Big Data and analytics estimated by Wikibon to be a $5 billion business today that will vault to $50 billion globally within five years. That kind of growth in just five years ought to be news in this era of business malaise.

So far, pure-play companies like Splunk have collectively captured barely 10 percent of that market, with the rest owned by the usual suspects like IBM, SAS, Oracle and HP. It’s early, and we’ve seen this pattern before; lots of potential for growth, innovation, and disruption.

This is an American-centric trend, as it was for each of the previous inflections in computing and communications. It’s bullish not just for the companies and people working for them, but for the economic power Big Data analytics unleashes. No surprise that Big Data analytics is one of the hottest graduate degrees. For example, Northwestern University (where, full disclosure, I sit on the Engineering School’s Advisory Board), in partnership with IBM, the SAS Institute and our recurring fave Teradata [NYSE:TDC], offers, appropriately enough, the first on-line Masters of Science in analytics. Anyone with such a degree is walking in to a job.

But take heart business, English and non-tech majors. This is bullish for you too. The tech dweebs are, finally, making complexity accessible and thus useful. That’s what’s exciting about what Splunk is doing, and all the others similar.

Time was once that computers themselves were the domain of specialists. Now kindergartners use PCs. The unstructured Cloud and Big Data infrastructure is set to march through the same transition because of what companies like Splunk are doing. Bringing simplicity to unstructured data took longer because it was harder.

You don’t need to know how to build a violin or guitar to be a musician and convert noise into music, anymore now than you have to be a computer scientist to extract patterns from unstructured data. Once the artisans build a useful instrument, the musicians take it from there. In the future, Big Data tools will mean that companies will need more creative, well educated employees – dare we say, educated in "unstructured’ ways, but ways that have imbued one with knowledge and insight, and all the attributes of a classic liberal arts education.

Oh, and for the lamentable boomers who think innovation and entrepreneurship in the tech field is the exclusive domain of the young; Splunk CEO Godfrey Sullivan is 58. For the record, a recent study found that twice as many founders of companies were older than 50 than were younger than 25, and the highest rate of entrepreneurship is in the 55 – 64 age group.

Splunk is just one, important, bellwether of the now unfolding next tech revolution. Bringing order out of chaos, seeing patterns in vast swaths of unstructured data is powerful, nearly magical, even theological. It’s not just consumers, but nature hides her secrets in patterns buried in unstructured data. Check out what Verizon is doing with Big Data to help cancer treatment, or IBM in the hunt for the origins of the universe.

But is Splunk a buy? I leave that to others. They have competition of course. (You can Google that.) Are we heading for a stock market bubble and an over-supply of companies chasing the same Big Data markets and technologies? We should all hope so. Big Data analytics is such a deep and broad trend that the bubble will take a long time to inflate. It will be a lot like the 1984 to Y2K run. There’ll be lots of winners along the way, and more jobs for everyone.

This piece originally appeared in Forbes

This piece originally appeared in Forbes