TPG Search
Location
Bangalore | India
Job description
Job Description :We are seeking a highly skilled and experienced Staff Engineer - Data Engineer, to join our data engineering team. The ideal candidate will have a strong background in designing, building, and maintaining data pipelines, with expertise in setting up and optimizing data processing frameworks such as Apache Spark or similar, Kafka/Pulsar and data lakeResponsibilities :- Design and develop robust, scalable, and efficient data pipelines to collect, process, and store large volumes of data from various sources.- Collaborate with platform and AI/ML teams to understand data requirements and supply data solutions like feature stores.- Architect data pipelines platform for product in a multitenant design realm- Design data solutions that scale dynamically based on the tenant data size, ranging from few GBs to TBs of data- Manage and perfect data streaming and messaging platforms such as Kafka and Pulsar for real-time data ingestion.- Work on data quality monitoring and ensure data accuracy, consistency, and reliability.- Keep abreast of industry trends and emerging technologies related to data pipelines and recommend improvements.- Build a centralized data lake that consolidates various data assets in the organization.- Build and maintain the runtime environment for data processing, including platforms like SQL, Python, and PySpark.Qualifications :- Bachelor's or higher degree in Computer Science, Data Engineering, or a related field.- Proven experience (6+ years) in data engineering in a production environment.- Strong ability in setting up and configuring Apache Spark Flink or similar, Kafka/Pulsar, for data streaming, transformation, and processing.- Hands-on experience with AWS services, including S3, Lambda, EMR, Glue, Redshift, Athena. Equivalent skills on GCP will also be highly rewarding- Proven expertise across the entire lifecycle of implementing computer and orchestration tools like Databricks, Airflow, Dagster, and others.- Proven expertise in setting data pipeline as platform for multi-tenant systems- Accomplished in managing large-scale data sets, handling Terabytes of data and billions of records effectively.- Skilled in working with streaming OLAP engines like Druid, ClickHouse, Pinot and similar technologies. Expertise with Apache Trino is a big plus- Strong programming skills in languages such as Python or Scala- Familiarity with service-oriented architecture and gRPC or similar- Familiarity with high-security environments like SOC-2, HIPAA, PCI, or similar contexts, highlighting a commitment to data privacy and security.- Strong team player and ability to collaborate with cross-functional teams.- Results-oriented and able to work independently with minimal supervision.- Excellent problem-solving and communication skills (ref:hirist.tech)
Job tags
Salary
