Integrating Azure Synapse with ClickHouse
Azure Synapse is an integrated analytics service that combines big data, data science and warehousing to enable fast, large-scale data analysis. Within Synapse, Spark pools provide on-demand, scalable Apache Spark clusters that let users run complex data transformations, machine learning, and integrations with external systems.
This article will show you how to integrate the ClickHouse Spark connector when working with Apache Spark within Azure Synapse.
Add the connector's dependencies
Azure Synapse supports three levels of packages maintenance:
- Default packages
- Spark pool level
- Session level
Follow the Manage libraries for Apache Spark pools guide and add the following required dependencies to your Spark application
clickhouse-spark-runtime-{spark_version}_{scala_version}-{connector_version}.jar- official mavenclickhouse-jdbc-{java_client_version}-all.jar- official maven
Please visit our Spark Connector Compatibility Matrix docs to understand which versions suit your needs.
Add ClickHouse as a catalog
There are a variety of ways to add Spark configs to your session:
- Custom configuration file to load with your session
- Add configurations via Azure Synapse UI
- Add configurations in your Synapse notebook
Follow this Manage Apache Spark configuration and add the connector required Spark configurations.
For instance, you can configure your Spark session in your notebook with these settings:
Make sure it will be in the first cell as follows:
Please visit the ClickHouse Spark configurations page for additional settings.
When working with ClickHouse Cloud Please make sure to set the required Spark settings.
Setup Verification
To verify that the dependencies and configurations were set successfully, please visit your session's Spark UI, and go to your Environment tab.
There, look for your ClickHouse related settings:
