Administering Clusters and Configuring Policies with Databricks
Databricks architecture, cluster types and runtimes, autoscaling, cluster policies, pools and init scripts.
Cluster Policies address all these problems by defining rules that govern cluster creation. They act as configuration templates that users must follow, while still allowing controlled flexibility.
Sign in to read this course
A free account unlocks all 514 courses. 20 are readable without one.
What's inside
29 sections- 1 Table of Contents
- 2 Overview and Objectives
- 3 Databricks Architecture — Foundations
- 4 Databricks Cluster Types
- 5 Databricks Runtime (DBR) Versions
- 6 Autoscaling and Capacity Management
- 7 Cluster Modes: Standard vs High Concurrency
- 8 Defining Cluster Policies on Azure Databricks
- 9 Constraint Types in a Cluster Policy
- 10 Policy Deployment Best Practices
- 11 Demonstrations: Cluster Policies in Practice
- 12 Configuring Cluster Resource Access
- 13 Entitlements and Permissions on Resources
- 14 Instance Pools — Reducing Startup Time
- 15 Cluster Tags and Cost Attribution
- 16 Init Scripts — Cluster Customization
- 17 Spark Configuration Parameters
- 18 Termination Policies and Timeouts
- 19 Unity Catalog and Cluster Access
- 20 User and Group Management
- 21 Service Principals — Application Accounts
- 22 Azure Active Directory and SCIM Integration
- 23 Managing and Using Personal Access Tokens
- 24 Databricks REST API — Complete Reference
- 25 Databricks CLI — Command-Line Management
- 26 Terraform / Infrastructure as Code for Databricks
- 27 Summary and Key Points
- 28 Review Questions
- 29 Glossary
More Azure Databricks & Spark courses
View all 14ETL Pipelines with Azure Databricks and Data Factory
Build ETL with Spark and PySpark, Unity Catalog governance, Delta Lake and Databricks vs Data Factory.
Manage Data with Azure Databricks and Azure Data Lake
Connect Databricks to ADLS Gen2 securely, ingest with Auto Loader and govern with Unity Catalog.
Optimize Storage and Performance with Delta Lake
Delta Lake internals, ACID, OPTIMIZE, Z-Order, liquid clustering, caching and Photon acceleration.
Real-Time Data Processing with Azure Databricks
Spark Structured Streaming with Event Hubs — windowing, stateful processing and real-time anomaly detection.
Machine Learning with Azure Databricks
The Databricks ML lifecycle: MLflow tracking, tuning with Ray, the model registry, serving and AutoML.
Building Deep Learning Models on Databricks
Build, train, tune and serve deep-learning models on Databricks with TensorBoard integration.
Interested in this course?
Contact us to book it or get a custom training plan for your team.