Job Description

JustinBradley’s client, a leading source of mortgage financing, is seeking a Senior Data Engineer to join our team. This role is critical for setting up and managing Change Data Capture (CDC) for multiple types of databases to hydrate a data lake. You will work closely with teams to orchestrate the flow of raw CDC data and perform ETL transformations to ensure the data is transformed into a usable, query-able form for analytics. The ideal candidate will have hands-on experience with Apache Spark for both batch and streaming data processing and will be well-versed in performance tuning and Big Data concepts.

Responsibilities:

• Set up and manage Change Data Capture (CDC) for various databases to ensure data flows seamlessly into a data lake.
• Implement ETL transformations using Apache Spark, handling both streaming and batch processing of data.
• Work with Apache Spark DataFrames, Spark SQL, and Spark Streaming to design and develop robust data pipelines.
• Orchestrate the transformation of raw CDC data into structured, analytics-ready datasets.
• Collaborate with cross-functional teams to understand data requirements and ensure data is correctly transformed and made available for downstream analysis.
• Optimize performance of data pipelines, ensuring efficient data processing and storage.
• Work with AWS services, including EMR, Glue Data Catalog, Lambda, and S3 to integrate, store, and manage data.
• Utilize Apache Airflow to orchestrate and automate workflows for data processing.
• Keep up-to-date with the latest trends and technologies in Big Data and cloud computing to improve system performance and scalability.

Requirements:

• Bachelor’s degree in Computer Science, Engineering, or related field, or equivalent work experience.
• Strong problem-solving skills with an ability to work independently and as part of a team.
• Ability to work in an agile environment and handle multiple tasks simultaneously.
• Excellent communication skills, both written and verbal.
• Extensive knowledge and experience with S3 and S3 operations (CRUD).
• Proficiency with EMR & EMR Serverless for large-scale data processing.
• Experience with Glue Data Catalog for managing metadata.
• Hands-on experience with Step Functions and Managed Workflows for Apache Airflow (MWAA) for workflow orchestration.
• Proficiency in Lambda (Python) for serverless applications.
• Experience with AWS Batch for running batch processing jobs.
• Familiarity with AWS Deequ (optional, but a plus) for data quality validation.
• Java: Mid to Senior level experience.
• Python: Mid-level experience (preferably with PySpark).
• Apache Spark: Experience with DataFrames, Spark SQL, Spark Streaming, and building ETL pipelines.
• Apache Airflow: Strong experience in using Airflow for orchestration.
• Experience with Scala.
• Experience with Apache Hudi.
• Experience with Apache Griffin.

JustinBradley is an EO employer - Veterans/Disabled and other protected employees.

Job Tags

Similar Jobs