Data Engineer - Streaming
Location
Buckinghamshire | United Kingdom
Job description
Sector: Analytics / Data
Location: Buckinghamshire
Type: Interim
Ref: BBBH70273_1706878165
Salary: Negotiable
Our client is looking for a Data Engineer with streaming date experience to work for their client on a major transformation project.
Responsibilities
- Participate in requirement gathering, analysis, and solution design to build big data engineering applications on AWS Data services such as AWS EMR, Snowflake using Spark and AWS Glue as ETL framework.
- Migrate and build Cloudera based hadoop, hive, impala, Kafka, Sqoop, Spark based data, jobs and security policies to AWS EMR , S3 , Snowflake and other AWS services such AWS Glue, DMS, IAM.
- Hands on involvement in Low Level Design, Development and architecting of large Data projects leading developer and testing teams.
- Job Scheduling and Automation.
- Data Validation, Quality checks, profiling and data reconciliation testing.
- Work as an individual contributor as well lead teams when required.
- Mentor junior members in the team by improving their skills and knowledge and have the ability to get things delivered.
- Work with both senior and junior team members like Project Manager, Hadoop Architect, other data engineers, data modellers, report developers, testers to complete the task.
- Setting up security and governance policies on data, users, data pipelines on AWS data services.
- Troubleshooting application errors and ensuring that they do not occur again.
- Apply agile and CI/CD methodologies and tools for development and deployment in agile mode.
Must have: Streaming Data Engineering and Analytics over it. Kafka, AWS Data Services, AWS, ETL, Spark/Scala, Java, Python, EMR, AWS Glue, AWS Athena
Good to have: Python, Spark, Hive, HDFS, Impala, Sqoop. Informatica
Job tags
Salary
