It happened to me once very early in my career. I arrived in a town to give a presentation the following morning but the airline had lost my luggage. My carefully planned “power outfit” of fancy suit and sassy shoes were nowhere to be found. I could have gone out to buy new items, but my resources at the time didn’t allow for that. I had plenty of other professional clothes, but they were another airline flight away. I had the clothes on my back, but I didn’t have what I wanted to help me look the part and perform my best. As a result, I felt only “good enough” instead of ready to soar ahead as a young professional.
The Problem with Inaccessible Data
Fast forward a decade or two, and I noticed a similar predicament at some of our clients: they had good data that could incredibly beneficial for them. However, it was located in separate environments making it, at best, a laborious process to unite and, at worst, inaccessible when it was needed in a pinch.
Client #1
A hospitality client used one proprietary tool to gather customer feedback via social media, another to gather customer feedback via a survey and a third internally managed tool to collect customer feedback via direct response. To gather the insights needed to maximize their business required multi-step manual extraction, labelling, and merging of themes, taking valuable person time from other projects, and leaving valuable insights undetected.
Good data was out there, but inaccessible.
Client #2
A technology client had their B2B customer information in a proprietary CRM tool and the end users’ details in their developed platform. Our client wanted to see how end users access their tool across a lifetime, and their customer success team fielded requests from the businesses to see how their team’s stats compared to similar businesses. Because the data lived in two environments, getting to the heart of these very reasonable business questions required several manual steps. An entire team had been built, in this case, with a focus to report across this complex environment.
Again, good data was out there, but inaccessible.
The Challenge of “Data Gravity”
“Good enough” will help many businesses survive, but let’s face it—we usually want our businesses to thrive! Good enough isn’t always good enough when the competition is close.
One way that many refer to this conundrum of data being inaccessible is that it’s demonstrating “data gravity.” Described by Dave McCrory in a 2010 blog post, the concept refers to the way in which data and applications are attracted; as data size increases, it becomes more and more difficult to move. Moving large datasets within and between applications is clumsy and expensive in time and money. If the gravity is Jupiter strong, data most critical to your business can be trapped and never fulfill its greatest potential. This makes it more difficult to combine data across sets and to analyze a single dataset in innovative ways.
The Plan to Defy “Data Gravity”
The easy answer? Have a plan to defy gravity—data gravity at least—and keep what you need readily accessible.
Think big.
While it’s tempting to employ a single use case to design a data strategy that defies gravity, this can lead to a hyper-focused solution that is not scalable. Think about where silos have prevented good things in the past and where you’re struggling to get at data right now. Think about what data is business critical, and why.
Think ahead.
Don’t just consider the challenges you face today with data gravity, but understand your organization's business strategy enough to plan your road map to avoid or minimize future problems. Consider structure, size, compute power and extraction ability when selecting data storage and technology. The caution here is to recognize that the world moves quickly these days. It may not be prudent to wait for it to stop to catch up but do keep a cautious eye on the direction things are heading.
Plan for movement.
Whether built internally or purchased from a third-party vendor, seek software and storage that allow you to do things with your data. Consider tools with APIs that integrate with your current environment. Build pipelines able to move between data sources to enable analysis by the “data folks” and create useful visualizations for other business experts to make informed decisions.
Pull in other experts.
In defying data gravity, we see a common tension in our clients between wanting to democratize the data, fully leveraging its analytic potential, and minimizing the risk of that data being misused unintentionally or intentionally by bad actors. This tension is necessary and addressing it makes your business better. In addition to protecting PII and proprietary information, information security, data governance, and legal experts can keep you off the front page of the papers for all the wrong reasons. As creators, movers, analysts and recipients of data, it is imperative we do it responsibly.
Making Data Science Responsible
So, don’t be like early-career me. Keep things accessible so you can be nimble and prepared for anything that comes your way. Defy data gravity and watch your business soar.
Here are some other related articles on the topic that we also like:
We Must Find A Way To Defy Data Gravity In The Cloud
Written by Steve McDowell
Featured on Forbes.com
What is data gravity anyway?
Written by Marshall Daly
Featured on Tableau.com
Harness the power of your data with AWS Analytics
Written by Rahul Pathak
Featured on AWS Big Data Blog
And for those of you who have had a certain showtune running through your head since you read the title of this, you’re welcome!
