Data Engineer

Location

Gurgaon | India

Job description

Position Summary :- Data Engineers are primarily responsible for designing, building, managing, and operationalizing data pipelines in support of key data and analytics use cases. - They are accountable for guaranteeing compliance with data governance, data privacy, and data security requirements.- Data Engineers are also responsible for the Enterprise Data Warehouse, ensuring its architecture is optimized for the organization current and future needs.Duties and responsibilities :- Build data pipelines : Managed data pipelines consist of a series of stages through which data flows (i.e. from endpoints of data acquisition to integration to consumption for analytics). Architecting, creating, and maintaining data pipelines will be the primary responsibility of the data engineer. Ensuring necessary mechanisms to guarantee data quality, completeness, and accuracy are part of the data pipeline design is accountability of the Data Engineers.- Drive Automation : The data engineer will be responsible for using innovative and modern tools, techniques and architectures to partially or completely automate the most-common, repeatable data preparation and integration tasks in order to minimize manual processes, as well as improve productivity. The data engineer will also assist with renovating the data management infrastructure to drive automation in data management and integration.- Maintain the Enterprise Data Warehouse : Data engineers are accountable for the Enterprise Data Warehouse (EDW) architecture, its implementation, optimization, and maintenance. Data engineers are responsible for designing, creating, and delivering datasets within the EDW to support analytics initiatives.- Collaborate across departments : The data engineer will need strong collaboration skills in order to work with various stakeholders within the organization. In particular, the data engineer will work in close relationship with Data analysts, Business analysts, and Data science teams in refining their data requirements for various data and analytics initiatives and their data consumption requirements.- Educate and train : The data engineer should be curious and knowledgeable about new data initiatives and how to address them. This includes applying their data and/or domain understanding in addressing new data requirements. They will also be responsible for proposing appropriate (and innovative) data ingestion, preparation, integration and operationalization techniques in addressing these data requirements. The data engineer will be required to train counterparts in these data pipelining and preparation techniques.- Ensure compliance with data governance and security : The data engineer is responsible to ensure that the data sets provided to users are compliant with established governance and security policies. Data engineers should work with data governance and data security teams while creating new and maintaining existing data pipelines to guarantee alignment and compliance.Qualifications :Education :- Bachelor or Masters in Computer Science, Information Management, Software Engineering, or equivalent work experience.Work Experience :- At least four years or more of working in data management disciplines including: data integration, modeling, optimization and data quality, and/or other areas directly relevant to data engineering responsibilities and tasks.- At least three years of experience working in cross-functional teams and collaborating with business stakeholders in support of a departmental and/or multi-departmental data management and analytics initiative.Technical knowledge, Abilities, and skills :- Ability to design, build and manage data pipelines for data structures encompassing data transformation, data models, schemas, metadata, and workload management. The ability to work with both IT and business in integrating analytics and data science output into business processes and workflows.- Strong knowledge of database programming languages including DB2, MS SQL,- Ability to work with large, heterogeneous datasets, build and optimize data pipelines, pipeline architectures and integrated datasets using traditional data integration technologies. These should include ETL/ELT, data replication, CDC, API design and access, and other data ingestion and integration technologies such as data streaming and data virtualization.- Good knowledge of advanced analytics languages like R, Python, and others.- Basic knowledge of popular data discovery, analytics and BI tools like Power BI, Tableau, Qlik, MicroStrategy, and alike.- Ability to work with data science teams in refining and optimizing data science and machine learning models and algorithms.- Able to work with large, heterogeneous datasets to extract business value using popular data preparation tools.- Strong knowledge of data warehousing architecture, able to design, build, and implement relational and multi-dimensional data models.- Strong understanding of data governance, data stewardship, data quality, data privacy, and data security.- Ability to work across multiple deployment environments including cloud, on-premises and hybrid.- Strong understanding of agile methodologies and capable of applying it.- Strong problem solving skills.Interpersonal Skills and Characteristics :- Able to collaborate with both the business and IT teams to define the business problem, refine the requirements, and design and develop data deliverables accordingly.- Good judgment, a sense of urgency, and commitment to high standards of ethics, regulatory compliance, customer service and business integrity.- Strong drive to stay current with industry best practices and trends on data acquisition, data modeling, data warehousing, and Big Data technologies.Core Competencies :- Ensures accountability- Collaborates- Courage- Customer focus- Being resilient- Drives results- Drives engagement (ref:hirist.tech)

Job tags

Salary