Databricks Archives - DWgeek.com

Databricks Recursive CTE Example: Querying Hierarchical Data

If you have ever tried to query hierarchical data in Databricks like an employee organizational chart, a bill of materials, or category trees you know it used to be a headache. Historically, data engineers had to write complex Python loops in PySpark to walk through parent-child relationships step-by-step. Thankfully, that is no longer necessary. Databricks now natively supports recursive Common Table Expressions (CTEs). Using the WITH RECURSIVE clause, you can easily query hierarchies and graph data directly in Databricks SQL. Let’s break down exactly how recursive CTEs work and look…

0 Comments

July 6, 2026

Databricks

Databricks vs Snowflake: Which is Right for You in 2026?

Choosing between Databricks and Snowflake used to be a simple decision, but massive platform updates in 2026 have completely blurred the lines between the two. With both platforms aggressively expanding into each other's territory, from AI workloads to business intelligence, picking the right one requires a deep understanding of their modern architectures. Here is a complete, up-to-date breakdown of how they compare on performance, pricing, and ideal use cases. Page Contents Introduction What is Snowflake? (The Cloud Data Warehouse) What is Databricks? (The Data Lakehouse) Core Differences: Architecture and Pricing…

0 Comments

July 6, 2026

Databricks

Databricks Common Table Expressions (CTEs): A Simple Guide

If you read our recent guide on Databricks SQL subqueries, you know that nesting queries inside one another can get messy very quickly. If you nest them three or four levels deep, your code becomes incredibly hard to read and debug. This is where Databricks Common Table Expressions (CTEs) save the day. CTEs are the secret to writing clean, professional, and easily maintainable SQL. In this guide, we will look at what CTEs are, why you should use them over subqueries, and how to write them in Databricks. Page Content…

0 Comments

July 6, 2026

Databricks

Databricks SQL Subqueries: Examples and Best Practices

If you write SQL in Databricks, you are eventually going to need a subquery. Whether you are trying to filter data based on another table, calculate a moving average, or just organize a massive query into readable chunks, subqueries are an essential tool for any data engineer. In this guide, we will break down exactly how subqueries work in Databricks SQL, look at the specific types supported, and walk through practical examples you can use right away. Page Content Introduction What is a Subquery in Databricks? Types of subqueries in…

0 Comments

July 6, 2026

Databricks

Databricks Temporary Tables Support

If you have been waiting for Databricks to add temporary tables, good news—they are finally here in Databricks Runtime 16.4. If you are migrating from databases like Snowflake, Oracle, or Netezza, you already know how incredibly useful these are. Let's break down how to create Databricks temporary tables, why you should use them, and how they actually differ from temporary views. Page Contents Introduction What Are Temporary Tables? Why Use Temporary Tables in Databricks? Temporary Tables vs Temporary Views Databricks Temporary Tables Examples Conclusion Introduction With the release of Databricks Runtime…

Comments Off

June 23, 2025

Databricks

Lateral Column Alias in Databricks – Example

If you have ever written a complex SQL query, you have probably run into this frustrating problem: you calculate a new column using an alias, but when you try to use that alias again in the exact same SELECT statement, the database throws an error. Historically, the only way around this was to write a subquery or a Common Table Expression (CTE). That is no longer the case. The lateral column alias in Databricks allows you to reuse an expression specified earlier in the same SELECT list. This is a…

Comments Off

September 24, 2024

General

How to Connect to Databricks SQL Endpoint from Azure Data Factory?

A Databricks SQL Endpoint is a compute cluster, quite similar to the cluster we have known in the Databricks that allows you execute SQL commands on data objects within the Databricks environment. The Databricks allows you to connect using various tools such as DBT, connect to Notebook using Azure Data Factory, etc. But there is no direct method to connect Databricks SQL endpoint warehouse. In this article, we will check how to connect to Databricks SQL endpoint from Azure Data Factory (ADF). Connect to Databricks SQL Endpoint from Azure Data…

Comments Off

January 27, 2023

Apache Spark

How to Access Azure Blob Storage Files from Databricks?

Azure blob storage is a Microsoft Azure cloud service to store large amount of structured and unstructured data such as text files, database export files, json files, etc. Azure blob storage allows you to store data publicly or you can store application data privately. You can access public Azure blob data without using any additional credentials. But, to access private data, you need to generate access key. In this article, we will check how to access Azure Blob storage files from Databricks? Access Azure Blob Storage Files from Databricks Similar…

Comments Off

May 4, 2022

Snowflake

How to Connect to Snowflake from Databricks?

Many organizations use the hybrid model to process the data. They use databricks to perform operations such as Machine Learning tasks and copy end results to Snowflake for reporting or further analysis. In this article, we will check how to connect to Snowflake from databricks to build hybrid architecture. Connect to Snowflake from Databricks The Snowflake is one of the relational databases that provide connector for Spark. You can use the Snowflake Spark connector to connect to Snowflake server and copy data from databricks to Snowflake. Test Data We will…

Comments Off

June 16, 2021