Sr Data Engineer/Business Analyst
Capgemini Technology Services India Limited
Location
Secunderabad | India
Job description
skill- Sr Data Engineer/Business Analyst
Experience- 5 yrs- 9yrs
Location- Hyderabad
'Primary Responsibilities:
â— Sourcing / loading / transforming and storing data:
â—‹ Managed data pipelines consist of a series of stages through which data flows (for example, from data sources or endpoints of acquisition to integration to consumption for specific use cases). These data pipelines have to be created, maintained and optimized as workloads move from development to production for specific use cases. Architecting, creating and maintaining data pipelines will be the primary responsibility of the data engineer.
â— Drive Automation through effective metadata management:
â—‹ The data engineer will be responsible for using innovative and modern tools, techniques and architectures to partially or completely automate the most-common, repeatable and tedious data preparation and integration tasks in order to minimize manual and error-prone processes and improve productivity. The data engineer will also need to assist with renovating the data management infrastructure to drive automation in data integration and management.
â—‹ Learning and using modern data preparation, integration and AI-enabled metadata management tools and techniques.
â—‹ Tracking data consumption patterns.
â—‹ Performing intelligent sampling and caching.
â—‹ Recommending and automating â€' existing and future integration flows.
â— Data compliance and governance:
â—‹ It will be the responsibility of the data engineer to ensure that the data users and consumers use the data provisioned to them responsibly through data governance and compliance initiatives. Data engineers should work with data governance teams (and information stewards within these teams) and participate in vetting and promoting content created in the business and by data scientists to the curated data catalog for governed reuse.
Essential:
â— 5-8 years of work experience in data management disciplines including [data integration, modeling, optimization and data quality], and/or other areas directly relevant to data engineering responsibilities and tasks.
â— At least three years of experience working in cross-functional teams and collaborating with business stakeholders in support of a departmental and/or multi-departmental data management and analytics initiative.
â— Strong experience with advanced analytics tools for Object-oriented/object function scripting using languages such as [R, Python, Java, C++, Scala, others].
â— Strong ability to design, build and manage data pipelines for data structures encompassing data transformation, data models, schemas, metadata and workload management. The ability to work with both IT and business in integrating analytics and data science output into business processes and workflows.
â— Strong experience with popular database programming languages including [SQL, PL/SQL, others] for relational databases and certifications on upcoming [NoSQL/Hadoop oriented databases like MongoDB, Cassandra, others] for non-relational databases.
â— Strong experience in working with large, heterogeneous datasets in building and optimizing data pipelines, pipeline architectures and integrated datasets using traditional data integration technologies. These should include [ETL/ELT, data replication/CDC, message-oriented data movement, API design and access] and upcoming data ingestion and integration technologies such as [stream data integration, CEP and data virtualization].
â— Strong experience in working with SQL on Hadoop tools and technologies including [HIVE, Impala, Presto, others] from an open source perspective and [Hortonworks Data Flow (HDF), Dremio, Informatica, Talend, others] from a commercial vendor perspective.
â— Basic experience in working with [data governance/data quality] and [data security] teams and specifically [information stewards] and [privacy and security officers] in moving data pipelines into production with appropriate data quality, governance and security standards and certification.
'
Job tags
Salary
