Au Finance Bank
Location
Jaipur | India
Job description
Spark Engineer Responsibilities:1. Design and Develop Spark Applications:- Understand the current Apache Spark platform code and nuances related to the same- Optimize and fine-tune Spark jobs for performance and efficiency.2. Python Programming:- Utilize Python for scripting, data manipulation, and integration with Spark applications.- Collaborate with cross-functional teams to integrate Python-based solutions into Spark workflows.3. Big Data Processing:- Work with large-scale distributed systems for processing and analysing big data.- Implement best practices for data processing, storage, and retrieval.4. Performance Optimization:- Identify bottlenecks and optimize Spark applications for enhanced performance.- Implement caching and partitioning strategies to improve data processing speed.5. Collaboration and Communication:- Collaborate with data scientists, analysts, and other engineering teams to understand business requirements and translate them into technical solutions.- Communicate effectively with team members and stakeholders.6. Monitoring and Troubleshooting:- Implement monitoring solutions for Spark jobs and proactively address any issues.- Troubleshoot and resolve performance and operational issues.7. Documentation:- Maintain comprehensive documentation for Spark applications, workflows, and processes.- Provide training and knowledge transfer to junior team members.Qualifications:- Bachelor's or Master's degree in Computer Science, Engineering, or a related field.- 5-7 years of experience working with Apache Spark.- Familiarity with AWS Cloud platform- Strong proficiency in Python programming.- Experience with big data processing frameworks and distributed computing.- Solid understanding of data structures, algorithms, and data modeling.- Excellent problem-solving and analytical skills.- Effective communication and collaboration skills (ref:hirist.tech)
Job tags
Salary
