Databricks Data Engineer

i4DM is seeking a Databricks Data Engineer to support ATLAS, the VA’s enterprise data management and analytics platform. The role designs, builds, and operates scalable data pipelines and governed lakehouse solutions, while enabling analytics and machine-learning use cases. Responsibilities Design, develop, and maintain scalable batch and streaming data pipelines using Databricks, Apache Spark, PySpark, and Spark SQLBuild and manage Delta Lake tables using the medallion (bronze/silver/gold) architecture to deliver reliable, analytics-ready dataDevelop real-time and near-real-time data ingestion solutions using Spark Structured Streaming and messaging platforms such as KafkaConfigure and manage Databricks clusters, jobs, and workflows in production environmentsImplement data governance, access controls, and security best practices using Unity CatalogIntegrate data from a variety of source systems and destinations, supporting ETL/ELT and pipeline orchestration activitiesOptimize existing