Location
Noida | India
Job description
GCP/Azure Data Engineer – Job Description
Data Engineer will be responsible for developing large-scale data structure and pipelines to organize, collect and standardize data that helps generate insights and address reporting needs. Use strong programming skills in Python, Java or any of the major languages to build robust data pipelines and dynamic systems. You will be considered a SME working directly with external clients and responsible for defining objectives, determining timelines and advanced business analysis and presentation of results to involved stakeholders.
Role And Responsibilities - Design Extract/Load/Transform (ETL) workflows to develop large scale data structures and pipelines to organize, collect and standardize data
- Uses knowledge of Hadoop architecture, HDFS commands and experience in designing, optimizing queries to build data pipelines
- Integrate data from a variety of sources, assuring that they adhere to data quality and accessibility standards
- Implemented endpoints to read data from BigQuery and Hive
- To provide user-defined functions for clients to read data from Hive and BigQuery directly without using any external connector jars
- Collaborate with data science team to transform data and integrate algorithms and models into automated processes using programming skills - Python or Hive
- Help in optimizing queries, and work to integrate the business requirements into existing campaign reporting data workflows
- Design and implement scalable, configurable and self-learning marketing campaign platform
- Design low level implementation plan of all GCP/Azure activities
- Develop framework including data extraction, ingestion, audit control, error, archival
- Develop code using GCP/Azure native or other languages/solutions and perform unit testing
- Work closely with GCP/Azure Architect, leads and other developers to implement the best practices
Candidate Profile - Bachelor's degree in Engineering/Computer Science or related quantitative field
- Significant of advanced experience with data analysis, reporting and programming logic (Python, PySpark, Hive, SQL, shell scripting)
- Experience interacting with decision-making business audience
- Analyzes current information technology environment to identify and access critical capabilities and recommend solutions
- Experiments with available tools and advices on new tolls in order to determine optimal solutions given the requirements dictated by the model/use case
- Good software engineering fundamentals, strong problem solving and critical thinking ability.
Skills - Proficiency in HIVE, PySpark, Pandas, Shell Scripting.
- Good Knowledge of Hadoop, MapReduce, HDFS, GCP(Big Query)
- Good knowledge of SQL and MS Excel
- EXL Health offers an exciting, fast paced and innovative environment, which brings together a group of sharp and entrepreneurial professionals who are eager to influence business decisions. From your very first day, you get an opportunity to work closely with highly experienced, world class analytics consultants.
- You can expect to learn many aspects of businesses that our clients engage in. You will also learn effective teamwork and time-management skills - key aspects for personal and professional growth
- Analytics requires different skill sets at different levels within the organization. At EXL Health, we invest heavily in training you in all aspects of analytics as well as in leading analytical tools and techniques.
- We provide guidance/ coaching to every employee through our mentoring program wherein every junior level employee is assigned a senior level professional as advisors.
- Sky is the limit for our team members. The unique experiences gathered at EXL Health sets the stage for further growth and development in our company and beyond.
Job tags
Salary
