Pontoon Solutions
Location
Delhi | India
Job description
The Data Engineer – Databricks is responsible for assembling large, complex sets of data from various data sources using APIs. Primary responsibilities include designing and developing new data pipelines, improve performance to existing data pipelines, support and maintain data pipelines, and provide expertise and best practices regarding data bricks. Partners with other BI team members to support data infrastructure needs while assisting with data-related technical issues. Responsible for identifying, designing, and implementing internal process improvements including re-designing infrastructure for greater scalability, optimizing data delivery, and automating manual processes. Maintains all tools and processes in conformance with the organization’s policies and objectives.
What you’ll be doing Partners with global team of BI Developers, SMEs and BI Leaders to evaluate the feasibility of and provide timetables for executing requirements to deliver data & API solutions in Agile manner. Translates business requirements into meaningful, clean, and scalable enterprise-grade data solutions built according to established Pontoon Business Intelligence & Analytics best practices. Design and build data integrations using APIs, REST APIs(JSON) and web services. Ingest, transform, and publish data from sources and to destinations via an API.Build out new API integrations to support continuing increases in data volume and complexity. Plan, design, develop, test, and maintain REST APIs. Propose Data load optimizations and continuously implement to improve the performance of the Data loads. Implements processes and systems to monitor data quality, ensuring production data is always accurate and available for key stakeholders and business processes that depend on it. Performs data analysis required to troubleshoot data-related issues and assist in the resolution of data issues. Develop and maintain scalable data pipelines in data bricks to integrate with enterprise data streams. Build data pipelines that feed business intelligence tools and expose data to end users using Power-BI. Design and implement Data Architectures. Works closely with other team members to develop a strategy for long term data platform architecture. Responsible to develop data architecture that is modular and cloud centric for agile delivery. Develop POCs and other technical evaluations of technologies, designs, and solutions to influence platform architects. Drive system and platform thinking within BI development team. Assists in estimation and assessment of feasibility of features. Support the planning and implementation of data design services, providing sizing and configuration assistance and performing needs assessments. Analyze current business practices, processes, and procedures as well as identify future business opportunities for leveraging cloud services using Databricks. Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc. Lead multiple activities including but not limited to data models, APIs, partner integrations, and cloud infrastructure. About you Bachelor’s degree in computer science, statistics, applied math, or information systems with a minimum of three years of experience as Data Engineer, API Developer, Databricks Developer/Engineer, Software Engineer, or Backend software engineer/developer. Three plus (3+)years of experience with extracting and modeling data from API& JSON required. Two plus (2+)years of working experience with SQL required.Three plus (3+)years of experience designing, developing, validating, and automating ETL/ELT processes. Three plus (3+)years of experience with Databricks platform and its tools including Platform (notebooks, clusters, Jobs, Databricks SQL, relational entities, Repos) Apache Spark (PySpark, DataFrame API, basic architecture) Delta Lake (SQL-based Delta APIs, basic architecture, core functions) Databricks CLI (deploying notebook-based workflows) Databricks REST API (configure and trigger production pipelines) required. Three plus (3+)years of experience in Python and common python libraries required. Experience working with Power BI and/or other Business Intelligence visualization tools preferred. Experience managing multiple development projects at one time is required. Skilled in communicating effectively verbally and in writing. Ability to establish and maintain effective working relationships. Ability to work effectively both independently and in a group or team environment. Ability to prioritize, organize and perform multiple work assignments simultaneously while meeting assigned deadlines. Ability to understand end-user needs and assist in the development of data products. Experience designing and building modern data pipelines and data streams. Experience with API (JSON) and Streaming technologies. Experience with Databricks (Azure Databricks REST API, Delta Tables).Experience with Python. Experience in data architecture for transformation and modernization of enterprise data solutions using cloud data technologies. Advanced SQL Programming with structured and unstructured data and object-oriented/object function scripting languages (Scala). Strong knowledge of Data Integration (e.g.,Streaming, Batch, Error,and Replay) and data analysis techniques. Experience with Security Concepts (e.g.,Encryption, Identity, etc.). Good problem-solving skills. Attention to detail. Self-directed and can stand on their own two feet and get stuff done. Experience with agile development methodologies. Proficient with Microsoft Office products such as Word, Excel, PowerPoint, and Outlook. Open to working in US hours ( 8am EST - 5pm EST)Job tags
Salary
