Baskadia

Being a software engineer at EXL MNC, I'm used to the comfort of familiar tools. So when I signed up to learn Databricks, I expected a mix of curiosity and confusion. What I got was a session that made the whole platform feel approachable.

A big thank you to Bibhusisa Pattanaik, who explained the fundamentals with clarity and patience, and to Anubha Mathew for organising the session.

Here's what clicked for me :-

1. The Lakehouse idea

Data lakes are cheap and flexible but messy. Data warehouses are clean and fast but rigid. Databricks' lakehouse combines both: one platform for raw data, analytics, and machine learning, without copying data between systems.

2. Apache Spark under the hood

Databricks is built around Spark, which splits big workloads across many machines and runs them in parallel. Once I understood that, things like "why is this lazy?" and "why did my job take so long?" started making sense.

3. Workspaces, clusters, and notebooks

Workspace: your home base for everything.

Clusters: the compute that does the heavy lifting. Start one, run your work, shut it down to save cost.

Notebooks: where you write code (Python, SQL, Scala, R) and see results right away. Switching between SQL and Python in the same notebook felt like a small superpower.

4. Delta Lake

This was my favourite part. Delta Lake adds reliability to a data lake with ACID transactions, schema enforcement, and time travel, so you can query an older version of your data. As someone who has broken a table or two in the past, I appreciated that.

5. Governance and the medallion architecture

Bronze (raw), Silver (cleaned), Gold (business-ready) is a simple way to organise data pipelines. Paired with Unity Catalog for access control and governance, it shows how data work can scale without turning into chaos.

What I'm taking away

The biggest lesson is that Databricks isn't just a tool but a way of thinking about data: unified, scalable, and collaborative. As developers, we often focus on code. This session reminded me that good data foundations make good code matter even more.

I'm still at the beginning of this journey, but I'm excited. Next up: building a small end-to-end pipeline of my own and breaking things on purpose (the best way to learn).

If you're thinking about exploring Databricks, start with the basics, get hands-on early, and find a great mentor. I was lucky to have one & thankful for the mentor named Bibhusisa Pattanaik.