Description
Sr. Databricks Engineer Mississauga, ON Long term Contract Need minimum 10 years of experience. Key Responsibilities: 1. Data Pipeline Development: o Build and maintain scalable ETL/ELT pipelines using Databricks. o Leverage PySpark/Spark and SQL to transform and process large datasets. o Integrate data from multiple sources including Azure Blob Storage, ADLS and other relational/non-relational systems. 2. Collaboration & Analysis: o Work Closely with multiple teams to prepare data for dashboard and BI Tools. o Collaborate with cross-functional teams to understand business requirements and deliver tailored data solutions. 3. Performance & Optimization: o Optimize Databricks workloads for cost efficiency and performance. o Monitor and troubleshoot data pipelines to ensure reliability and accuracy. 4. Governance & Security: o Implement and manage data security, access controls and governance standards using Unity Catalog. o Ensure compliance with organizational and regulatory data policies. 5. Deployment: o Leverage Databricks Asset Bundles for seamless deployment of Databricks jobs, notebooks and configurations across environments. o Manage version control for Databricks artifacts and collaborate with team to maintain development best practices. Technical Skills: • Strong expertise in Databricks (Delta Lake, Unity Catalog, Lakehouse Architecture, Table Triggers, Delta Live Pipelines, Databricks Runtime etc.) • Proficiency in Azure Cloud Services. • Solid Understanding of Spark and PySpark for big data processing. • Experience in relational databases. • Knowledge on Databricks Asset Bundles and GitLab. Preferred Experience: • Familiarity with Databricks Runtimes and advanced configurations. • Knowledge of streaming frameworks like Spark Streaming. • Experience in developing real-time data solutions. Certifications: • Azure Data Engineer Associate or Databricks certified Data Engineer Associate certification. (Optional )
