The State of Data Engineering: Opportunities to Unlock the True Potential of Data
We continued our “State of Enterprise Tech” Webinar Series this week with a “State of Data Engineering” fireside chat with experts from both the startup and corporate side of the table, including Tim Delisle, CEO and Co-Founder of Datalogue, Steve Turk, Managing Director at JPMorgan Chase, and our moderator Kelley Mak, Principal at Work-Bench.
During the chat, we focused on the rise of data engineering within the Fortune 500, biggest barriers to successful data programs, as well as what the modern data landscape looks like and where it’s heading. Check out our YouTube here for a recording of the full webinar and see our recap of the top challenges and opportunities discussed:
CHALLENGE: There is a disconnect between the people, processes, and technologies.
For many organizations, innovation tends to exist within silos. There is often a gap between the day-to-day business decision makers and the data engineers actually working with data and extracting solutions. This creates miscommunications and inefficiencies through the organization, costing valuable time and money. It also leads to a middle layer of information that exists as a one-off solution that may look good to one end user at that time, but isn’t a replicable, “rinse and repeat” process.
OPPORTUNITY: Bridge the gap and create a continuous improvement cycle.
When organizations are able to eliminate handovers between these decision makers and data engineers, as a whole, they are able to better understand their data and unlock its full potential. To do this, decision makers and data engineers need to work together to minimize inconsistencies across the organization and find the common denominators among problems. This compounding approach will better allow your data to solve the problem at hand, adjacent problems, and new problems that will undoubtedly pop up in the future.
CHALLENGE: Trust in data needs to be built.
“The biggest challenge is that executives don’t always understand the principles of data engineering, so it’s important to make data transparent…When you think about building a data ecosystem and setting up an environment for exponential growth, the decision makers need to care.” — Steve Turk
In complex organizations there are multiple sources of the truth, and because of this ambiguity, the average business executive will challenge their data engineering team over and over again to get the most confident answer to bring to their peer group in fear someone else will come up with evidence proving them wrong.
OPPORTUNITY: Create more transparent and democratized data systems.
Business executives need a bird’s eye view of where the numbers come from and what processes were done to that number. It’s still a challenge of demonstrating the lineage of operations of not only data points, but the higher-level insights derived from them. Datalogue is helping solve this problem with their self-service integration platform, which enables cross functional teams to get access to data and integrate it into their analytics systems. According to Tim, Datalogue believes that in order to drive exponential returns with data, you need a subject matter expert to be tightly integrated into the supply chain of your data and to empower technologists with the right tools. Together, this ensures the business decision maker is equipped to best analyze the best data solutions. This will generate more value for your organization as a whole and end up decentralizing your data more organically.
CHALLENGE: To build or to buy?
“Especially in data engineering, there is a fine line between digital disruption and digital distraction.” — Tim Delisle
Organizations need to weigh if they have enough resources to build their own tech, if they have a good understanding of that tech market at-large, and if they have the talent to maintain the technology. If the answer to any of the above is “no,” building tech in-house drastically limits the organization’s ability to solve the problem effectively and can become a resource suck that limits their ability to then source third-party vendors later. Additionally, data engineering talent is in all-time high demand right now, and constantly needing to hire unexpected talent to keep up with the products you build in-house can become a huge undertaking.
OPPORTUNITY: Make the case for your startup.
Tim states that he often sees organizations mistake themselves as tech-first companies, who build their tech in-house and get a lot of perceived value upfront, but then get overwhelmed by production maintenance (cost and upkeep). This is an opportunity for startups to showcase how their technology and solution provides long-term ROI and can easily integrate into pre-existing platforms.
For more insights on data engineering in the enterprise and opportunities for data startups, see a full recording of our webinar below:
Stay up-to-date on upcoming “State of Enterprise Tech” Webinar Series’ topics and speakers by signing up for our Enterprise Weekly Newsletter every Friday. Future webinar topics include Cloud, DevOps, and more.
