DWgeek.com

How to Call BigQuery Stored Procedure from Cloud Composer?

Automating your data warehouse workflows in Google Cloud is critical for scaling your analytics. If you want to orchestrate complex SQL logic seamlessly, knowing how to call a BigQuery stored procedure from Cloud Composer is an essential skill for any data engineer. Here is a complete, step-by-step guide and Airflow DAG example to get your pipelines running smoothly. Page Content Introduction Step 1: Create the BigQuery Stored Procedure Step 2: Write the Cloud Composer (Airflow) DAG Best Practices for Orchestrating BigQuery Conclusion Introduction Google Cloud offers several ways to orchestrate…

0 Comments

July 7, 2026

Databricks

Databricks Recursive CTE Example: Querying Hierarchical Data

If you have ever tried to query hierarchical data in Databricks like an employee organizational chart, a bill of materials, or category trees you know it used to be a headache. Historically, data engineers had to write complex Python loops in PySpark to walk through parent-child relationships step-by-step. Thankfully, that is no longer necessary. Databricks now natively supports recursive Common Table Expressions (CTEs). Using the WITH RECURSIVE clause, you can easily query hierarchies and graph data directly in Databricks SQL. Let’s break down exactly how recursive CTEs work and look…

0 Comments

July 6, 2026

Databricks

Databricks vs Snowflake: Which is Right for You in 2026?

Choosing between Databricks and Snowflake used to be a simple decision, but massive platform updates in 2026 have completely blurred the lines between the two. With both platforms aggressively expanding into each other's territory, from AI workloads to business intelligence, picking the right one requires a deep understanding of their modern architectures. Here is a complete, up-to-date breakdown of how they compare on performance, pricing, and ideal use cases. Page Contents Introduction What is Snowflake? (The Cloud Data Warehouse) What is Databricks? (The Data Lakehouse) Core Differences: Architecture and Pricing…

0 Comments

July 6, 2026

Databricks

Databricks Common Table Expressions (CTEs): A Simple Guide

If you read our recent guide on Databricks SQL subqueries, you know that nesting queries inside one another can get messy very quickly. If you nest them three or four levels deep, your code becomes incredibly hard to read and debug. This is where Databricks Common Table Expressions (CTEs) save the day. CTEs are the secret to writing clean, professional, and easily maintainable SQL. In this guide, we will look at what CTEs are, why you should use them over subqueries, and how to write them in Databricks. Page Content…

0 Comments

July 6, 2026

Databricks

Databricks SQL Subqueries: Examples and Best Practices

If you write SQL in Databricks, you are eventually going to need a subquery. Whether you are trying to filter data based on another table, calculate a moving average, or just organize a massive query into readable chunks, subqueries are an essential tool for any data engineer. In this guide, we will break down exactly how subqueries work in Databricks SQL, look at the specific types supported, and walk through practical examples you can use right away. Page Content Introduction What is a Subquery in Databricks? Types of subqueries in…

0 Comments

July 6, 2026

Databricks

Databricks Temporary Tables Support

If you have been waiting for Databricks to add temporary tables, good news—they are finally here in Databricks Runtime 16.4. If you are migrating from databases like Snowflake, Oracle, or Netezza, you already know how incredibly useful these are. Let's break down how to create Databricks temporary tables, why you should use them, and how they actually differ from temporary views. Page Contents Introduction What Are Temporary Tables? Why Use Temporary Tables in Databricks? Temporary Tables vs Temporary Views Databricks Temporary Tables Examples Conclusion Introduction With the release of Databricks Runtime…

Comments Off

June 23, 2025

Databricks

Lateral Column Alias in Databricks – Example

If you have ever written a complex SQL query, you have probably run into this frustrating problem: you calculate a new column using an alias, but when you try to use that alias again in the exact same SELECT statement, the database throws an error. Historically, the only way around this was to write a subquery or a Common Table Expression (CTE). That is no longer the case. The lateral column alias in Databricks allows you to reuse an expression specified earlier in the same SELECT list. This is a…

Comments Off

September 24, 2024

Snowflake

Database Migration to Snowflake: Best Practices and Tips

Moving your legacy data warehouse to the Snowflake Data Cloud is one of the best decisions you can make for performance, scalability, and cost-efficiency. However, migrating from traditional systems like Oracle, Netezza, Teradata, or SQL Server is rarely a simple "lift and shift" operation. Without a solid strategy, you risk data loss, security gaps, and unexpected cloud computing costs. In this guide, we will break down the essential steps for a successful Snowflake migration and the best practices you need to follow to optimize your new environment. Page Content Introduction…

Comments Off

April 20, 2023

GCP BigQuery

Reuse Column Aliases in BigQuery – Lateral Column alias

BigQuery lateral Column alias are columns that are derived from the previously computed columns in same SELECT statement. Derived columns or lateral Column alias are virtual columns that are not physically stored in the table. Their values are re-calculated every time they are referenced in a query. Many PostgreSQL relational databases such as Netezza supports reuse of the column aliases within the same SELECT statement but GCP BigQuery does not support reuse of calculated derived columns. In this article, we will identify the alternate methods reuse column aliases in Google…

Comments Off

April 18, 2023

Redshift

How to use Amazon Redshift Replace Function?

The Amazon Redshift REPLACE function is one of the important string functions. The replace function allows you to manipulate the string in Amazon Redshift. This string function is similar to translate and regexp_replace functions. Amazon Redshift Replace Function Page Contents Introduction to Amazon Redshift Introduction to Amazon Redshift Syntax of the Redshift Replace Function Redshift Replace Function Example Usage of the Redshift Replace Function Best practices for using the Redshift Replace Function Conclusion Introduction to Amazon Redshift Amazon Redshift is a fully managed, cloud-based data warehouse service offered by Amazon…

Comments Off

March 13, 2023