Introducing the Databricks Spark SQL Adapter

The Tidal Automation Adapter for Databricks Spark SQL integrates Tidal Automation with Databricks SQL and Apache Spark SQL so that you can create, schedule, and run Databricks Spark jobs through TA. Databricks Spark SQL is a module for structured data processing and acts as a distributed SQL query engine. Databricks provides a programming abstraction called DataFrames, which organizes data into a table of rows and columns. DataFrames are standard data structures that offer flexible, intuitive methods for working with and storing data. Every DataFrame contains a schema that defines each column's name and data type. Spark DataFrames can have universal data types like StringType, IntegerType, and data types specific to Spark, such as StructType. The DataFrame stores missing or incomplete values as null values.