BigData Integration Consultant

Location

Bangalore | India

Job description

• The most popular ones are -

• Building capabilities that address a market need, basis our ongoing research efforts
  
• Solving a specific use case for a current or potential client based on challenges on-ground
  
• Developing new systems that help be a better employer and a better partner to clients

We are invested in both, employee growth and client success!

• As a Big Data Engineer, you will be responsible for designing, developing, and maintaining our big data infrastructure.
• You will work with large datasets, perform data processing, and support various business functions by creating data pipelines, data processing jobs, and data integration solutions.
• You will be working in a dynamic and collaborative environment, leveraging your expertise in Hive, Hadoop, and PySpark to unlock valuable insights from our data.

Key Responsibilities:

1. Data Ingestion and Integration:
   • Develop and maintain data ingestion processes to collect data from various sources.
   • Integrate data from different platforms and databases into a unified data lake.
2. Data Processing:
   • Create data processing jobs using Hive and PySpark for large-scale data transformation.
   • Optimize data processing workflows to ensure efficiency and performance.
3. Data Pipeline Development:
   • Design and implement ETL pipelines to move data from raw to processed formats.
   • Monitor and troubleshoot data pipelines, ensuring data quality and reliability.
4. Data Modeling and Optimization:
   • Develop data models for efficient querying and reporting using Hive.
   • Implement performance tuning and optimization strategies for Hadoop and Spark.
5. Data Governance:
   • Implement data security and access controls to protect sensitive information.
   • Ensure compliance with data governance policies and best practices.
6. Collaboration:
   • Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and provide data support.

Qualifications:

• Bachelors degree in Computer Science, Information Technology, or a related field.
  
• 3+ years of experience in big data engineering and data processing.
  
• Proficiency in Hive, Hadoop, and PySpark.
  
• Strong SQL and NoSQL database experience.
  
• Experience with data warehousing and data modeling.
  
• Knowledge of data integration, ETL processes, and data quality.
  
• Strong problem-solving and troubleshooting skills.
  
• Excellent communication and teamwork skills.
  

Preferred Qualifications:

• Experience with cloud-based big data technologies (e.g., AWS EMR, Azure HDInsight, Google Dataprep).
  
• Certification in Hadoop, Hive, or PySpark.
  
• Familiarity with containerization and orchestration tools (e.g., Docker, Kubernetes).
  
• Knowledge of data visualization tools (e.g., Tableau, Power BI).

Job tags

Salary