We are seeking a highly skilled and experienced GCP (Google Cloud Platform) Data Engineer with more than 6 years of expertise to join our dynamic team. The GCP Data Engineer will play a pivotal role in designing, implementing, and optimizing data solutions on Google Cloud Platform. The ideal candidate should have a deep understanding of GCP services, strong data engineering skills, and a proven track record of delivering robust and scalable data solutions.
Key Responsibilities:
Architecture Design: Design and implement scalable and efficient data architectures on Google Cloud Platform (GCP) to meet business requirements and performance goals.
Data Integration: Integrate data from various sources into GCP using services like Cloud Storage, Cloud Pub/Sub, Dataflow, and BigQuery.
Data Processing: Develop and implement data processing pipelines using Apache Beam, Apache Spark, or other suitable frameworks to transform and cleanse data for analysis.
Data Modeling: Design and implement data models for optimal performance and efficiency in BigQuery or other GCP data storage solutions.
Streaming Analytics: Implement real-time data processing and analytics solutions using GCP streaming services such as Cloud Dataflow and Cloud Pub/Sub.
Data Security and Governance: Ensure data security and compliance with regulatory requirements by implementing appropriate access controls, encryption mechanisms, and data governance policies.
Performance Optimization: Optimize data pipelines, queries, and data storage configurations to maximize performance and minimize costs.
Monitoring and Logging: Set up monitoring and logging solutions to track system performance, detect anomalies, and troubleshoot issues in GCP data environments.
Automation and DevOps: Implement automation scripts and workflows using tools like Terraform, Deployment Manager, or Google Cloud Deployment Manager to automate provisioning and management of GCP resources.
Collaboration: Collaborate with cross-functional teams including data scientists, analysts, and business stakeholders to understand data requirements and deliver data solutions that meet business needs.
Documentation and Knowledge Sharing: Create and maintain technical documentation, best practices, and guidelines for data engineering on GCP. Share knowledge and mentor junior team members.
Qualifications:
Bachelor's or Master's degree in Computer Science, Engineering, or a related field.
Minimum of 6 years of experience in data engineering, with at least 3 years of hands-on experience working with Google Cloud Platform.
Strong understanding of GCP services such as BigQuery, Cloud Storage, Dataflow, Pub/Sub, and Dataproc.
Proficiency in programming languages such as Python, Java, or Scala for data processing and scripting tasks.
Experience with data warehouse design, implementation, and optimization.
Knowledge of data streaming and real-time analytics technologies.
Familiarity with machine learning concepts and frameworks (e.g., TensorFlow, scikit-learn) is a plus.
Excellent problem-solving skills and the ability to troubleshoot complex data engineering issues.
Strong communication and collaboration skills, with the ability to work effectively in a team environment.