Location
Pune | India
Job description
As a Data Engineer your mission will be to build and maintain enterprise , production-grade data pipelines, considering business requirements and data platform standards. You must have hands on experience in building ETL flows , using CI CD pipelines , using cloud technologies such as Azure Data lake, ADF, DataBricks, Visualisations tools such as Power BI. Additioanally SQL is a must. Knowledge of Python will be added advantage.
Examples of technologies you will be working with: Azure Data Lake Storage, Delta Lake, Azure Data Factory, Databricks, Azure SQL, Azure Dev Ops.
Key responsibilities and responsibility:
Build, configure, deploy, and support a production-grade, scalable and secure data platform
Build and support long-term, production-grade data pipelines that ingest, store, transform, process, and expose data from producer to consumer, based on business requirements
Develop data pipelines using a code-first approach in line with the platform standards and practices
Translate functional requirements into a technical design
Provide and maintain technical documentation. You will develop the visuals in Power BI for as expected by Stake holders.
Train and handover common data engineering related maintenance and monitoring activities to the Data Operations & Quality Team
Share technology and platform expertise with colleague data engineers, product owners and other people in the organization
To succeed, you will need
Skills & Experience:
The ideal candidate will have a good blend of business and technical skills. Specific requirements for this position include:
General skills:
- You have a bachelor s or master s degree in computer science, engineering, or equivalent by experience
- You have a minimum work experience of 3 years with data engineering in relavent domain
- The ideal candidate has strong and demonstrable hands-on experience with data engineering on cloud-based data platforms using Databricks (Spark), preferably based on MS Azure technology: Azure Storage, Azure SQL database, Azure data factory, Azure Databricks
- Good coding skills (Python, SQL, or Scala)
- Experience with following concepts or technologies is considered a plus: Data Lake Storage, SQL, Hive, Presto, Databricks, Spark, Delta Lake, Hadoop, HDInsights, Cloudera, Hortonworks, MapR, Azure Functions, Azure Synapse, Azure Analysis Services, Power BI, Python, SQL, Scala, Java, C#, Azure AD, Ranger, Docker, Kubernetes, Azure, AWS, Kafka, StreamSets, REST
- Experience of working on MS Power BI
- Experience and understanding of agile methodologies and the SCRUM framework are a plus.
- Experience with tools like Jira and confluence is a plus
Important areas of Expertise:
- Storage: Best of breed (Cloud) storage solutions, both for unstructured and structured data
- Ingestion: Streaming and batch ingestion from source systems both in the cloud and on-premises, private and public
- Transformation: Code-first approach using Spark (Databricks)
- Analytic engines: Deploy, configure, and operationalize analytics engines and its clustered infrastructure
- Exposure/Integration: Expose data based on consumption use-cases like data science, self-service analytics, business intelligence and operational APIs for process integration
- Data Platform: Health, logging, monitoring, debugging, automation
Competences:
- You have a passion for innovation and technologies, combined with strong technical and analytical skills.
- You are customer oriented, enthusiastic, and professional
- You have a can-do mindset, hands-on approach and are decisive. You are not afraid of making errors and are willing to learn by doing
- You are a strong communicator with excellent collaboration skills and are customer focused
- You have self-drive and passion
- You are flexible and prepared to work outside of business hours if required to meet a deadline.
- You have a proactive attitude and always strive for continuous improvement
- You are able to cooperate with different levels in the organization, with different people and cultures
- You can maintain good relations with external parties
- You are stress resistant.
- You are result oriented and quality focused on terms of processes.
Job tags
Salary
