Data Engineer - Implementation Consultant at Horizontal Talent in Boston, Massachusetts

Job Description:

Hybrid - 3 days on site weekly - Irving, TX


Data Engineer : Implementation Consultant

Summary

We are currently onboarding North America based OPCO' s (Operating companies) to their new digital platform (Spark) which is built on Kubernetes running in AZURE, the customer and associate experience (CX/AX) on their new SPARK platform in EMEA. They are planning to migrate North American operating companies (OPCOs) to the platform, beginning in 2024.
With this integration work the engineer will be responsible for designing, configuring, and implementing data validation tools which will validate the data transferred between. You will ensure best practices are followed through the implementation lifecycle and help construct data sets for use by the team and other business stakeholders to power both reporting and validation.

Job responsibilities

• Highly skilled and motivated Data Engineer
• As a Data Engineer, you will be responsible for validating the data between two systems, ensuring accuracy and consistency.
• Implement data quality checks and monitoring processes.
• Design and Develop data entity validation scripts using python.
• Validate and reconcile data between two systems, ensuring accuracy and consistency.
• Collaborate with cross-functional teams to identify and resolve data discrepancies and issues.
• Develop and implement data quality checks and monitoring processes to ensure data integrity.
• Design and maintain data pipelines and ETL processes to extract, transform, and load data from various sources.
• Optimize data storage and retrieval processes to improve performance and efficiency.
• Perform data analysis and profiling to identify data quality issues and recommend solutions.
• Develop and maintain documentation for data validation processes, data mappings, and data lineage.
• Stay up to date with industry trends and best practices in data engineering and data validation.
• Support migration of North American OPCOs, enabling data validation on the new platform.
• Gather business requirements from key client stakeholders and translate these into technical tracking specifications based on standards and industry best practice.

• Strong knowledge of IPAAS and data validation for migration of applications.
• Demonstrable experience in delivering end to end data validation.
• Confident understanding of web development and its languages and possess Python proficiency.
• Proactive and highly organised with strong time management and planning skills. Able to change direction and work on multiple projects across a wide range of topics.
• [Desired] Knowledge of Python/Google BigQuery architecture.