Engineering Manager - Data Engineer
Location
Bangalore | India
Job description
- Collaborate with Tech and Analytics team to build and maintain the infrastructure required for optimal extraction, transformation, and loading of data from a variety of data sources
- Oversee and govern the expansion of the current data architecture as business grows and ensure best practices are followed
- Design and build best in class architecture for data tables to ensure optimal querying performance in relational databases
- Create and maintain connectors that expose the data securely for consumption by downstream systems and services in near real-time.
- Create and maintain a data architecture docs to communicate data requirements that are important to business stakeholders and work on acquiring external data sets through APIs and/or Websockets and prepare physical data models on top of that
- Build data governance and security protocols and ensure adherence from analytics, tech and business teams
- Build and lead the data engineering team, recognize their strengths, and lead them to take ownership of end to end data architecture. Stay on top of latest developments in tech stack and propose potential upgrades to existing systems.
What Are We Looking For
- 6 to 10 years of experience in Data Engineering - Designing databases, building data pipelines, and maintaining data governance protocols in cloud platforms
- A visionary in technical architecture, with experience building and maintaining Data Engineering Products, along with demonstrated ability to take accountability for achieving results
- Hands-on working experience with Python, ETL pipelines, advanced SQL
- Strong understanding of AWS Services - Redshift, Lambda, Glue, Athena, security protocols
- Good understanding of ETL/ELT technology and processes with experience in building and scaling Cloud DW Redshift/Snowflake/BigQuery
- Working with data layer solutions like Apache Hudi, DeltaLake, iceberg, has experience in setting up a real time data processing system with Apache Spark/ Apache Flink , pySpark.
- Experience in gathering and processing raw data at scale including writing scripts and spark jobs.
Mandatory Skills
Apache Spark/ Apache Flink , pySpark, Apache Hudi, DeltaLake, iceberg, ETL pipelines, advanced SQL Strong understanding of AWS Services - Redshift, Lambda, Glue, Athena, security protocols
Job tags
Salary
