Databricks unit testing

 A simple way to unit test databricks notebooks is to make the notebooks parameterized.

To test if contiguous months are generated we run the contiguous notebook with test input and specify an output location.We then assert to see if the output is expected

Step 1

To run blog_contiguous notebook from test notebook

outputTable = "test_output"
dbutils.notebook.run("blog_contiguous",
timeout_seconds=180, arguments={
"product": "dbfs:/FileStore/tables/test_sales-1.csv",
"output": outputTable,
})

Step 2

Assert values in the test notebook

spark.catalog.refreshTable(outputTable)
output = table(outputTable).cache()
assert output.count() == 12
assert output.groupby(["Year"]).count().collect()[0]['count']==12
assert [output.select("Month").collect()[i]['Month'] for i in range(0,output.count())]==[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]







Git hub unit test