From Software Engineer to Data Engineer -- the Basics

Ever walked around exploring an area you have never been to and then you somehow end up in a street you know? I'm sure you know that feeling. That's exactly how I think about theoretical concepts you've heard before or read in passing and then one day someone explains it to you and relates it to a practical use case and you have that satisfying "Aha" moment. As a software engineer you would have worked with user-facing or backend applications and at most had some interaction with a database, maybe you've sent events/messages to a stream or an API but in general, the whole data pipeline and how data is used is a black box to you, an abstraction you don't need to understand. You might have heard about warehouses, lakes, Kafka etc, but there are enough languages, frameworks, and user requirements to think about so why dig deeper into something that is someone else's responsibility.