Lead Data Engineer - AI / ML

Description:

Key projects:

Exploration of generative AI

Technology requirements:

Proficiency in Apache Spark, including Spark Core, Spark SQL and Spark Streaming.

Proficiency in languages such as Python for data processing and scripting.

Degree or certifications required:

Bachelorâ€s degree in computer science, management information systems, or related discipline

Years experience:

5+

Required background? Skills?:

Experience in designing and implementing ETL processes using Databricks notebooks for efficient data extraction, transformation, and loading.

In-depth knowledge of the Databricks Unified Analytics Platform and its features for collaborative big data analytics.

Understanding of data modeling concepts for designing database structures.

Proficiency in working with both relational databases and NoSQL databases.

Integration of data from diverse sources, including APIs, databases, streaming data, and external data feeds.

Implementation of processes to ensure data quality, including data validation, cleansing, and error handling.

Knowledge of cluster management, optimization, and scaling for efficient data processing.

Optimization of Spark jobs and Databricks clusters for better performance.

Proficiency in cloud platforms such as Azure for building scalable and flexible data architectures.

Use of tools like Apache Airflow, ADF, or Databricks to orchestrate and schedule data workflows.

Nice-to-Haves:

Knowledge of data security best practices and the implementation of measures to ensure data privacy and compliance.

Implementation of monitoring and logging solutions to track the health and performance of pipelines.

Familiarity with monitoring platforms like DataDog and New Relic

Azure

Top 3:

Data Engineering5+

Databricks5+

Spark/DevOps practices5+

Education:

Bachelors Degree

Additional client information:
#J-18808-Ljbffr