Location: Bay Area, CA (or) Portland, OR (or) Denver, CO
Key Skills: Cloud Engineer (Azure) with experience in Databricks and data platform architecture.
Responsibilities: • Design and deploy scalable and secure Azure infrastructure solutions for our data platform, including Databricks, Azure Data Lake Storage, Azure Synapse Analytics, and other related services. • Collaborate with data engineers and data scientists to understand their requirements and ensure the cloud infrastructure meets their needs. • Implement best practices for infrastructure as code (IaC) using tools like Azure Resource Manager templates, Terraform, or Bicep. • Configure and manage Azure networking components, such as Virtual Networks, Subnets, Network Security Groups, and Azure Firewalls. • Implement and maintain security and compliance measures, including Azure Active Directory integration, role-based access control (RBAC), and data encryption. • Automate deployment and maintenance processes using Azure DevOps or other CI/CD tools. • Monitor and optimize the performance and cost of Azure resources, including Databricks clusters and Azure Data Lake Storage. • Stay up-to-date with the latest Azure services, features, and best practices, and provide guidance to the team. • Participate in on-call rotations and respond to incidents and alerts related to the Azure infrastructure.
Requirements: • Bachelor's degree in Computer Science, Information Technology, or a related field, or equivalent work experience. • 3+ years of experience working with Azure cloud services, with a focus on data and analytics workloads. • Strong understanding of Azure infrastructure services, such as Virtual Machines, Storage Accounts, Azure Data Lake Storage, Azure Synapse Analytics, and Azure Databricks. • Experience with infrastructure as code (IaC) tools like ARM templates, Terraform, or Bicep. • Proficiency in scripting languages like PowerShell, Bash, or Python for automation and management tasks. • Knowledge of networking concepts and experience with Azure networking services (Virtual Networks, Network Security Groups, Azure Firewalls). • Familiarity with security and compliance best practices in Azure, including Azure Active Directory, RBAC, and data encryption. • Experience with CI/CD pipelines and automation tools like Azure DevOps or GitHub Actions. • Strong problem-solving and analytical skills, with the ability to troubleshoot and resolve complex issues. • Excellent communication and collaboration skills, with the ability to work effectively with cross-functional teams.
Preferred Qualifications: • Experience working with Apache Spark and Databricks environments. • Knowledge of data engineering and data science concepts and workflows. • Familiarity with containers and container orchestration platforms like Azure Kubernetes Service (AKS). • Understanding of cloud cost optimization strategies and resource management best practices. • Relevant cloud certifications, such as Azure Solutions Architect or Azure Data Engineer Associate.