Location
Kakinada | India
Job description
Role Purpose and Key Accountabilities:
The primary role of a Lead Data Engineer is to support development of various data engineering components of Prevalent s Security Insight Platform (SIP) across data collection, transport, transformation, ingestion and analytics. SIP is based on open source big data technologies designed to ingest, process and analyse peta byte scale data on a distributed architecture.
The ideal candidate would be a self-motivated individual with strong technology skills, commitment to quality and positive work ethic, who can
- Participate in design, development, and deployment of SIP data management modules.
- Develop data ingestion, parsing, scheduling, processing and other data management components based on agile development practices.
- Work in a fast-paced agile development environment.
- Conduct unit, system and integration testing of developed modules.
- Select and integrate any Big Data tools and frameworks required to provide requested capabilities.
- Collaborate with client and partner teams to ensure high quality data solutions are delivered.
- Troubleshoot operational escalations and resolve client implementation issues in a timely manner with strong collaboration and care for business priorities.
- Develop an understanding of industry trends and best practices related to Data Engineering and Open Source Big Data technologies.
- Create and follow personal education plan in the technology stack and solution architecture.
- Collaborate and contribute to agile teamwork.
- Build processes supporting data transformation, data structures, metadata, dependency and workload management.
Skills Experience
- Significant experience in data engineering on open source big data technologies.
- Experience with big data tools: Hadoop, Spark, Kafka, etc. and NoSQL databases like Postgres, Mongo DB, Elastic.
- Experience with data pipeline and workflow management tools: Airflow, etc.
- Experience with AWS cloud services: EC2, EMR, RDS.
- Experience with stream-processing systems: Spark-Streaming, etc.
- Experience supporting and working with cross-functional teams in a dynamic environment.
- Experience building and optimizing big data data pipelines, architectures and data sets.
- Great verbal and written communication skills.
- Self-motivated individual with an analytical mind-set capable of working in a fast-paced environment.
- Strong project management and organizational skills.
Knowledge:
- Good knowledge of object-oriented/object function scripting languages: Python, Java, Scala.
- Good understanding of Data lake concepts such as data ingestion and transformation.
- Understanding of Hadoop Framework including MapReduce concepts and parallel processing.
- In depth understanding of Spark Framework.
- Good understanding of Agile methodology.
- Working knowledge of message queuing, stream processing, and highly scalable big data data stores.
- Great verbal and written communication skills.
- Self-motivated individual with an analytical mind-set capable of working in a fast-paced environment.
- Strong project management and organizational skills.
Education:
Masters/Bachelor s Degree in Computer Science Engineering
Job tags
Salary
