BigData Integration Consultant
Location
Bangalore | India
Job description
- The most popular ones are -
- Building capabilities that address a market need, basis our ongoing research efforts
- Solving a specific use case for a current or potential client based on challenges on-ground
- Developing new systems that help be a better employer and a better partner to clients
We are invested in both, employee growth and client success!
- As a Big Data Engineer, you will be responsible for designing, developing, and maintaining our big data infrastructure.
- You will work with large datasets, perform data processing, and support various business functions by creating data pipelines, data processing jobs, and data integration solutions.
- You will be working in a dynamic and collaborative environment, leveraging your expertise in Hive, Hadoop, and PySpark to unlock valuable insights from our data.
Key Responsibilities:
- Data Ingestion and Integration:
- Develop and maintain data ingestion processes to collect data from various sources.
- Integrate data from different platforms and databases into a unified data lake.
- Data Processing:
- Create data processing jobs using Hive and PySpark for large-scale data transformation.
- Optimize data processing workflows to ensure efficiency and performance.
- Data Pipeline Development:
- Design and implement ETL pipelines to move data from raw to processed formats.
- Monitor and troubleshoot data pipelines, ensuring data quality and reliability.
- Data Modeling and Optimization:
- Develop data models for efficient querying and reporting using Hive.
- Implement performance tuning and optimization strategies for Hadoop and Spark.
- Data Governance:
- Implement data security and access controls to protect sensitive information.
- Ensure compliance with data governance policies and best practices.
- Collaboration:
- Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and provide data support.
Qualifications:
- Bachelors degree in Computer Science, Information Technology, or a related field.
- 3+ years of experience in big data engineering and data processing.
- Proficiency in Hive, Hadoop, and PySpark.
- Strong SQL and NoSQL database experience.
- Experience with data warehousing and data modeling.
- Knowledge of data integration, ETL processes, and data quality.
- Strong problem-solving and troubleshooting skills.
- Excellent communication and teamwork skills.
Preferred Qualifications:
- Experience with cloud-based big data technologies (e.g., AWS EMR, Azure HDInsight, Google Dataprep).
- Certification in Hadoop, Hive, or PySpark.
- Familiarity with containerization and orchestration tools (e.g., Docker, Kubernetes).
- Knowledge of data visualization tools (e.g., Tableau, Power BI).
Job tags
Salary