Keep up the Great Work Jeff Meyerson
The most popular stories in the media are not necessarily the ones to spend time reading. Often they are just trolling for clicks by rehashing the same story from yesterday. Fortunately, we have careful, hard working writers doing important work serving up stories we may otherwise have missed.
In the world of software engineering, Jeff Meyerson’s Software Engineering Daily Podcast is one of the good ones. He pursues interesting people and technologies, not just the latest and greatest fads.
A good set up and introduction is essential when covering a wide variety of technologies. Fortunately, one of Jeff’s superpowers is the way he introduces his listeners to a new topic. He manages to start wide and general, and then narrow down to the subject — all while building a strong case for the listener to keep listening. He does this at the beginning of every show and somehow it only takes him about a minute. Here is his set up about data pipelines from a few days ago:
A data pipeline is a series of steps that takes large data sets and creates usable results from them. At the beginning of a data pipeline, a data set might be pulled from a database, a distributed file system, or a Kafka topic. Throughout a data pipeline, different data sets are joined, filtered, and statistically analyzed.
At the end of a data pipeline, data might be put into a data warehouse or Apache Spark for ad-hoc analysis and data science. At this point, the end-user of the data set expects that data to be clean and accurate. But how do we have any guarantees about the correctness?
-Jeff Meyerson, Software Engineering Daily podcast Great Expectations: Data Pipeline Testing with Abe Gong February 17, 2020.
Another thing Jeff does is interview guests that are deep in experience but not that deep on polish or canned talking points. CEOs can be good sometimes, but a steady diet of carefully tested talking points can leave the listener still hungry for more.
Also this week Jeff interviewed venture capitalist Eric Anderson about investing in data infrastructure companies. Eric is a hands on and experienced industry participant with no canned messaging. Their wide ranging conversation covered many angles on the industry landscape and ultimately pointed out that even though companies seem to have been modernizing their data management practices for more than a decade, some of the most basic problems have still not been solved.
At about 1:03 in the podcast, Eric says the following about how many companies still struggle with the most fundamental data preparation tasks:
I'll just flag that we've talked about data engineering in part mostly because it’s my background, but I also think that this is like – If you want digital transformation, you want machine learning to work in your company, you’re going to have to solve the data problem. The data problem is surprisingly unsolved, like despite all the tools we’ve talked about, people feel like business users can’t get the answers they want fast enough. Do I have a churn problem? I don't know. How do I find that out? I find an analyst in SQL query and they’re going to go find a data engineer who can populate tables, and six months later it’s still unclear. I’m bullish on the broad data engineering landscape in order to address the stuff we discussed.
-Eric Anderson, Software Engineering Daily podcast Data Infrastructure Investing with Eric Anderson, February 20, 2020
These two episodes together give the listener a good look at an important market not often covered adequately by the media. While podcasts can end up talking over and again about the same five hot topics, Jeff is giving us insight into a wide range of technical topics. I am grateful for Jeff and recommend his work whenever I can.