Skip to content

Server side: Databricks Testing Tools Runner

The runner enable to run test notebooks, from a remote repository (e.g. Azure Devops), on Databricks environment. In order to do that, you must configure the Databricks workspace to access Azure Devops and install Databricks Testing Tools.

How to install Runner on a cluster

Before running the following steps, make sure the cluster respect the following prerequisites

Prerequisites

  • Edit cluster configuration to add these two environment variables. These variables reference Pernod Ricard Data Feed for package sharing. CLuster_env_variables

  • Edit cluster configuration to add an initialization script that points to: dbfs:/databricks/scripts/init-scripts/set-pip-conf.sh. This script uses previously set environment variables to set the index url where the packages will be downloaded from. CLuster_env_variables

    The content of the script: 

    #!/bin/bash

printf "[global]\n" > /etc/pip.conf
printf "index-url=https://$PYPI_LOGIN:$PYPI_PASSWORD@pkgs.dev.azure.com/pernod-ricard-data/_packaging/pernod-ricard-python-data/pypi/simple/\n" >> /etc/pip.conf

Install Databricks Testing Tools Runner

The Runner can be installed as a cluster library, via PyPI. Install the package named pr_databricks_testing_tools.

Cluster_Installation