MG Info | Data Driven Culture
Location
Home office | Brazil
Job description
Requirements
Technical Expertise:
Azure Proficiency: In-depth knowledge of Azure services, particularly Azure Data Lake Storage, Azure Databricks, Azure Data Factory, and Azure SQL Database.
Databricks Mastery: Proficiency in utilizing Databricks features such as notebooks, clusters, jobs, and libraries.
Data Architecture and Design: Strong capabilities in architecting data lake and data warehousing infrastructure, including knowledge of storage, data ingestion, data processing, and analytics components.
Data Integration and ETL: Expertise in integrating data from various sources, ensuring data quality and integrity, and designing and implementing efficient ETL (Extract, Transform, Load) pipelines.
Performance Optimization: Skills in monitoring and enhancing the performance of data processing pipelines, Databricks clusters, and Azure resources. Ability to identify and resolve performance bottlenecks.
Data Governance and Security: Knowledge of data governance practices and security measures, including access controls, encryption protocols, and data protection mechanisms.
Troubleshooting and Problem-solving: Proficiency in troubleshooting, conducting root cause analysis, and resolving technical issues within the Azure and Databricks ecosystem.
Technology Upgradation and Continuous Improvement: Keen awareness of the latest advancements in Azure and Databricks
Leadership and Management:
Technical Guidance and Standard Setting: Ability to provide technical guidance, establish technical standards, guidelines, and best practices for the Azure and Databricks ecosystem.
Collaboration and Team Management: Skilled in collaborating with cross-functional teams and leading the data lake team. Ability to mentor junior members and facilitate knowledge sharing.
Stakeholder Management: Competence in managing and aligning with various stakeholders, understanding their requirements, and delivering technical solutions that meet their needs.
Habilidades que farão você se destacar
Living or moving to Portugal.
Activities
Architecture and Design: The Technical Lead is responsible for designing and architecting the Azure-based data lake infrastructure using Databricks technologies. They work closely with the Platform Architects to define the overall data lake architecture, including storage, data ingestion, data processing, and analytics components.
Azure and Databricks Expertise: The Technical Lead needs to have deep expertise in Azure services, such as Azure Data Lake Storage, Azure Databricks, Azure Data Factory, Azure SQL Database, and other relevant Azure components. They should also be proficient in using Databricks features, including notebooks, clusters, jobs, and libraries.
Technical Guidance: The Technical Lead provides technical guidance and expertise to the data lake team in leveraging Azure and Databricks technologies effectively. They assist in defining technical standards, guidelines, and best practices specific to the Azure and Databricks ecosystem to ensure optimal performance, security, and scalability.
Data Governance and Security: The Technical Lead is responsible for implementing and maintaining data governance and security measures within the data lake. They establish access controls, encryption protocols, and data protection mechanisms as per Azure and Databricks security guidelines to ensure compliance and safeguard sensitive data.
Data Integration and ETL: The Technical Lead oversees the integration of data from various sources into a data lake, data warehouse and semantic layer. They work closely with data engineers and the Platform team to design and implement efficient pipelines. They ensure data quality, integrity, and proper transformation of data before loading it into the data lake.
Performance Optimization: The Technical Lead monitors and optimizes the performance of the Azure-based data lake implemented using Databricks technologies. They analyze system metrics, identify performance bottlenecks, and optimize data processing pipelines, Databricks clusters, and Azure resources to enhance performance, scalability, and cost efficiency.
Collaboration and Team Management: The Technical Lead collaborates with cross-functional teams, such as data engineers, data scientists, Azure architects, and business stakeholders, to understand their requirements and provide technical solutions within the Azure and Databricks environment. They provide technical leadership and guidance to the data platform team, mentor junior members, and facilitate knowledge sharing within the team.
Troubleshooting and Issue Resolution: In case of technical issues or failures within the Azure and Databricks ecosystem, the Technical Lead takes a lead role in troubleshooting and resolving problems. They conduct root cause analysis, coordinate with support teams, and implement appropriate solutions to restore the data lake s functionality.
Continuous Improvement: The Technical Lead keeps up to date with the latest advancements in Azure and Databricks technologies and identifies opportunities for continuous improvement. They evaluate new Azure services, Databricks features, and relevant tools and frameworks to enhance the data lake s capabilities, scalability, and performance.
Job tags
Salary
