Lead Big-Data Engineer (JB-329)

A leading, Global Consulting Company is seeking a Lead Big-Data Engineer to join the team and lead the implementation of its vision to build Data Lake House & MLOps on the AWS cloud.

The Lead Big-Data Engineer will be part of the Generative AI revolution and build Generative AI products and services.

Responsibilities:

Plan and build Data Lake House in the AWS cloud based on future technologies most companies will use in only 3-5 years.

Have end to end ownership: Design, develop, deploy, measure and maintain the data platform.

Solve production issues in the framework code.

Collaborate with 40+ data engineers/scientists, understand their functional needs and build products to accelerate their productivity and quality of work.

Influence directly on the way millions of people consume insurance and financial services.

Research and build Generative AI services and applications.

Lead and mentor data engineers, providing guidance and fostering a culture of continuous learning and professional development.

Stay up to date with emerging technologies, industry trends, and best practices related to new technology, and proactively recommend their adoption when appropriate.

Participate in the software engineering, processes and standards with an impact on the whole delivery organization.

Communicate complex architectural concepts and recommendations effectively to both technical and non-technical stakeholders.

Tech Stack:

Python, Java, Spark, Kafka, Parquet, IceBerg, Avro, AWS S3, AWS Glue, AWS Athena, AWS DataZone, AWS Lake Formation, AWS SageMaker, AWS DocumentDB, AWS ElasticCahe, AWS OpenSearch, AWS DRS, AWS QuickSight, AWS EKS, AWS Managed Airflow.

· Previous experience in data engineering and familiarity with AWS databases services like RDS, SPARK, Glue, OpenSearch and more.

· 5+ years of experience developing large scale distributed data systems and services, experience in Big Data tools.

· 5+ years of working experience with AWS Cloud and services.

· 5+ years of expertise in the Software design and architecture process.

· Experience with SQL and No-SQL.

· 3+ years of software or data engineering development experience in one or more of the following programming languages: Python, Java, or Scala.

· Production systems understanding.

· Has good understanding and experience of CI/CD practices and Git, including MLOps practices.

· Proven experience in gathering and converting internal customer requirements into working solutions for large enterprises.

· Engineering leadership capabilities such as leading engineering discussions, presenting to other engineers and cross-functional leader.

· A helpful, can-do attitude and a willingness to take ownership of problems.