A quick tutorial on how to access your DBFS instance to download files solely via your browser. More often than not, you may be interested in downloading data from your Databricks instance. And whilst Databricks provides a UI for retrieving your DataFrame result, sometimes you are interested in generating data…

A step-by-step guide on how to connect (query) Azure Synapse Analytics Delta Lake data from Databricks for both dedicated and serverless pools. Problem Statement In recent years, especially with the advent of Delta lake, it became ever more common to use Databricks as a main source of processing and use internal services…

Apache Airflow has become the de facto standard for Data Orchestration. However, throughout the years and versions, it accumulated a set of nuances and bugs which can hinder its production usage. This series of articles aims at walking Apache Airflow users through the process of overcoming these issues, the same…

Apache Airflow has become the de facto standard for Data Orchestration. However, throughout the years and versions, it accumulated a set of nuances and bugs which can hinder its production usage. This series of articles aims at walking Apache Airflow users through the process of overcoming these issues, the same…

A layman’s overview of the Data ecosystem in a few paragraphs — The Data ecosystem has grown exponentially in the last years, with emerging technologies and increasing vendor-hype clouding the way for newcomers. This post is my brief and simple attempt to explain the origin and path from OLTP to Data Lakehouse in a few short paragraphs. Bear with me, hopefully, in…

Avoiding running on the heap and CTEs vs Temporary Tables — This post is the second in a series of small groups of two tips on improving the performance of SQL queries, in addition to useful concepts usually overlooked in the modern SQL world: SQL Performance Tips # 1: Avoiding self joins and operations/functions for joins Similarly to previous iterations, an…

Avoiding self joins and join on operations — Many consider SQL to be data science’s ugly brother, often dedicating little time exploring its nuances and capabilities. And yet, SQL is often the primary means of extracting and preparing information (ETL) to be consumed further on. This series of posts attempts to shed light on some of the often…

A (long) primer on a growing requirement for Data Scientists — Knowledge on NoSQL databases seems to be an increasing requirement in data science applications, yet, the taxonomy is so diverse and problem-centered that it can be a challenge to grasp them. This post attempts to shed light on some of the concepts, often delving into each design’s specificities. We start…

Creating a data validator with dynamic and interchangeable business rules or why (T-) SQL is more powerful than we like to admit — Hardcoding. I’m sure at this point you’ve heard all about how using it will take you to coder’s hell. Experts seem to recommend avoiding it whenever possible. So why are most of our queries hardcoded? What if we could dictate the query’s predicate in a centralized, interchangeable fashion? …