Senior Data Engineer at Horizontal Talent in Woodbury, Minnesota

Job Description:

Overview:

• Architect and refine our Databricks-centric data platform, emphasizing scalable integrations and advanced entity resolution strategies.


• Lead the charge towards a data mesh infrastructure, promoting domain-centric design and decentralized data management.


• Enhance our DataOps capabilities, emphasizing data observability, discovery, and lineage to maintain data integrity and accuracy.


• Pioneer the adoption of Databricks data lakehouse architectures, focused on simplifying data management and enhancing data processing capabilities.


• Serve as a technical liaison among analytics engineers, BI developers, and analysts within the Databricks environment, integrating solutions like Segment.io.


• Develop and optimize SQL and Python/Scala scripts within Databricks for complex data transformation and integration tasks.


• Work closely with cross-functional teams to align data strategies with operational needs and objectives, incorporating advanced data resolution methodologies.


• Stay at the forefront of industry trends and technological advancements to ensure our Databricks platform remains innovative and effective.

• Bachelor's


• Strong leadership skills and the ability to advocate for best practices in data management and processing.


• 5+ years of experience in a data engineering role, with history of implementing data platform modernization


• 2+ years of hands on databricks experience.


• In-depth understanding of Databricks, along with experience in other cloud data warehouses like Snowflake, Redshift, Big Query.


• Building data integration and orchestration within the databricks environment, with hands-on experience in technologies such as Delta Live Tables, CDC, dbt, airflow, Segment.io.


• Advanced skills in SQL, Python, and Scala, with an emphasis on their application within Databricks for complex data tasks.


• Knowledge of best practices in version control and CI/CD with Git and GitHub Actions; Agile project management with Jira and Confluence.


• Proficiency with the Azure data suite (Azure SQL, Data Factory, Synapse Analytics, Power BI).


• Solid understanding of data observability, discovery, and lineage, and their application in maintaining high data quality standards.


• Additional experience with AI and ML capabilities in the context of Databricks is a plus.