Looking for an AWS Data Engineer that has AI understanding and wants to build on this knowledge!
Design, build & maintain scalable data pipelines in AWS to ensure their compliance with up-time, security, and privacy.
Actively contribute to both data & application governance initiatives by designing and implementing policies, procedures, and best practices
Implement API-based integrations to support continuing increases in data volume and complexity
Collaborate with report developers/analysts and business teams to improve data models that feed BI tools and increase data accessibility.
Build and maintain the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using Python and SQL
Assist in the design of processes and systems to monitor data quality, ensuring production data is always accurate and available for key stakeholders and business processes that depend on it
Troubleshoot data pipeline related issues and oversee their resolution.
Identify ways to improve data reliability, efficiency, and quality.
Collaborate with all business units and development teams to develop strategy for long term data platform architecture.
Familiarity with one or more DevOps tools (Git, Jenkins, CI/CD, Jira)
Identify and integrate external data sources from various source formats.
Assist and oversee other projects as assigned
Job Requirements:
AI experience or understanding
Advanced working SQL knowledge and experience working with relational databases, query authoring (SQL) as well as working familiarity with a variety of databases.
Experience building and optimizing big data data pipelines, architectures, and data sets.
Experience performing root cause analysis on internal and external data and processes to answer specific business questions and identify opportunities for improvement.
Strong analytic skills related to working with structured and unstructured datasets.
Build processes supporting data transformation, data structures, metadata, dependency, and workload management.
A successful history of manipulating, processing, and extracting value from large, disconnected datasets.
Working knowledge of message queuing, stream processing, and highly scalable data stores.
We are looking for a candidate with 5 years of experience in a Data Engineer role, who has attained a degree in Computer Science, Statistics, Information Systems, or another quantitative field.
Experience with data pipeline and workflow management tools.
Experience with AWS cloud services: EC2, Lake Formation, Athena, Glue, Redshift Spectrum, S3 Storage, IAM, and KMS