BigData Integration Consultant
Location
Bangalore North | India
Job description
At Thoucentric, we work on various problem statements.
- The most popular ones are -
- Building capabilities that address a market need, basis our ongoing research efforts
- Solving a specific use case for a current or potential client based on challenges on-ground
- Developing new systems that help be a better employer and a better partner to clients
All of these need the best of minds to work on them day-to-day ; and we do exactly that! Your contribution to organization development is as important as outward facing consulting. We are invested in both, employee growth and client success!
As a Big Data Engineer, you will be responsible for designing, developing, and maintaining our big data infrastructure. You will work with large datasets, perform data processing, and support various business functions by creating data pipelines, data processing jobs, and data integration solutions. You will be working in a dynamic and collaborative environment, leveraging your expertise in Hive, Hadoop, and PySpark to unlock valuable insights from our data.
Key Responsibilities:
- Data Ingestion and Integration:
- Develop and maintain data ingestion processes to collect data from various sources.
- Integrate data from different platforms and databases into a unified data lake.
- Data Processing:
- Create data processing jobs using Hive and PySpark for large-scale data transformation.
- Optimize data processing workflows to ensure efficiency and performance.
- Data Pipeline Development:
- Design and implement ETL pipelines to move data from raw to processed formats.
- Monitor and troubleshoot data pipelines, ensuring data quality and reliability.
- Data Modeling and Optimization:
- Develop data models for efficient querying and reporting using Hive.
- Implement performance tuning and optimization strategies for Hadoop and Spark.
- Data Governance:
- Implement data security and access controls to protect sensitive information.
- Ensure compliance with data governance policies and best practices.
- Collaboration:
- Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and provide data support.
Requirements
Qualifications:
- Bachelors degree in Computer Science, Information Technology, or a related field.
- 3+ years of experience in big data engineering and data processing.
- Proficiency in Hive, Hadoop, and PySpark.
- Strong SQL and NoSQL database experience.
- Experience with data warehousing and data modeling.
- Knowledge of data integration, ETL processes, and data quality.
- Strong problem-solving and troubleshooting skills.
- Excellent communication and teamwork skills.
Preferred Qualifications:
- Experience with cloud-based big data technologies (e.g., AWS EMR, Azure HDInsight, Google Dataprep).
- Certification in Hadoop, Hive, or PySpark.
- Familiarity with containerization and orchestration tools (e.g., Docker, Kubernetes).
- Knowledge of data visualization tools (e.g., Tableau, Power BI).
Benefits
What is in it for You:
Be part of the exciting Growth Story of Thoucentric! Work on projects that help you stay ahead of the curve. Not just exciting projects, if you are a self-starter, you will also get multiple opportunities to design, drive and contribute to the organizational and practice initiatives. Challenge yourself in an environment with higher expectations ensuring constant learning curve and steep growth opportunities Be part of One Extended Family. We bond beyond work - sports, get-togethers, common interests etc. Work in a very enriching environment with Open Culture, Flat Organization and Excellent Peer Group.
Job tags
Salary
