Senior Data Engineer - New Delhi
Location
Delhi | India
Job description
Senior Data Engineer - Delhi
We are looking for a Senior Data Engineer with 2+ years of experience.
Who we are
We at CivicDataLab, work with the goal to use data, tech, design and social science to strengthen the course of civic-engagements in India. We work to harness the potential of the open-source movement to enable citizens to engage better with public reforms. Our work is centered around building data strategy, data platforms and data science applications to push data-driven decision-making at scale. Moreover, we work closely with governments, non-profits, think tanks, media houses, academia and more to build overall data and tech capacity.
What We are looking for
At CivicDataLab, we are building robust and automated tools focusing on data analytics, encompassing data curation, data cleansing, data standardization, and sophisticated data wrangling. These products would support our projects across different sectors – public finance, digital public goods, climate change etc. We are also standardizing open datasets to adhere to the Data Catalog Vocabulary (DCAT) metadata standards, making them searchable and more user-friendly.
To assist us with this endeavor, we are seeking an experienced data engineer (with 2+ years of experience as a Data Engineer). This role requires the candidate to be based in Delhi.
We are seeking people who are strongly aligned with our values and have an innate sense of problem-solving, automating processes and adapting well to dynamic environments. These individuals will collaborate with data strategists, public policy researchers, and other stakeholders to design and develop systems that make public datasets more accessible. They should also possess the skills to effectively model, store, and manage large datasets. This collaborative effort will enable us to co-create comprehensive data analytics tools and dashboards catering to our diverse range of stakeholders.
Our Commitment to Diversity
We are committed to inclusive hiring and strongly encourage applicants from diverse and underrepresented gender and caste identities and/or socio-cultural backgrounds to apply for this role. Our organizational policies are gender neutral, including POSH policy and leave policy.
Requirements - What You'll Be Doing
- Design and develop scalable data orchestration pipelines using Prefect and Apache Airflow
- Create and oversee data APIs responsible for collecting, managing, and analysing data from diverse public data sources.
- Standardize metadata of open datasets by ensuring compliance with the DCAT metadata standard.
- Collaborate with our partners to perform in-depth Exploratory Data Analysis (EDA) of various datasets.
- Engage in the development of database models in accordance with the specific project requirements.
- Maintain and monitor our existing open data platforms like Open Budgets India, Justice Hub, Open Contracting India.
- Engage regularly with our diverse stakeholders and open-source communities to discuss and create reusable resources around use-cases of public data, data engineering best practices, and guidebooks.
- Thoroughly document code, processes, and all activities performed by the data team, ensuring clarity and comprehensiveness. This includes documenting algorithms, methodologies, data transformations, and the overall workflow.
- Skills You Should Bring
- 2+ years of thorough experience working with Python and SQL.
- Understanding of message brokers such as RabbitMQ.
- Knowledge of open-source data scraping frameworks and tools such as Selenium and Scrapy.
- Experience with building an end to end ETL pipeline.
- Familiarity with building database systems.
- Knowledge of API or Stream-based data extraction processes.
- Comprehensive knowledge of a Git-based workflow
- Comprehensive knowledge of metadata standards such as DCAT.
- Good to have
- An understanding of data privacy principles and practices, as well as experience in implementing data privacy algorithms, would be valuable in maintaining the confidentiality and integrity of sensitive datasets.
- Prior experience collaborating with government or social sector research-based organizations.
- Prior experience in analysing and presenting data using tools such as Apache Superset, Metabase etc.
- Proficiency in working with spreadsheets
- Knowledge of reading and writing code in R and other statistical programming environments.
- Knowledge of working with geospatial data processing tools such as QGIS.
- Prior experience in actively contributing to FOSS (Free and Open-Source Software) projects.
- Familiarity in working with Agile methodologies and Scrum processes.
BenefitsWhy work with us
We help you not just define your impact but also work with you towards finding a path to learn, realise and quantify its effect on our ecosystem.
Our past work and experience of working with communities and civic tech, in general, has connected us as a branch to a network of civil society actors and organizations. You'll have the opportunity to leverage this network, to work on pressing, yet thought-provoking issues, in sectors like Judiciary, Public Finance, Economics, Public Education and Urban Development.
We also feel that this is our biggest strength, what we can offer you is not a feature to work on but a passage to an infinitely long road of people, problems, ideas and opportunities that may help you find your place amidst the chaos.
How we work
CivicDataLab has 3 base locations in India namely Delhi, Hyderabad and Guwahati. We follow a hybrid model where our bandhus work out of office for a minimum of 10 days per month (or) 3 weeks a quarter. We use open-source tools and agile methodologies in organising our work.
Perks of Working with Us
Wellness Allowance
At CivicDataLab, we always emphasize the wellness of our bandhus. This includes any Expenditure done for the purpose of Wellness Setup except Any financial instrument, any expense that can be claimed as a deductible expense under Income Tax rules, any goods and services that attract a combined tax, cess or duty of more than 28%. If youre interested in taking classes that enhance your overall physical or mental well-being, you have an INR 60,000 annual stipends to do so. For some people, that might mean a monthly massage. Some take photography lessons or learn a musical instrument or buy a gym membership. Its up to you; the point is to learn something that you feel enriches you as a person.
Professional growth and development Allowance
At CivicDataLab, we encourage everyone to take up things that help one grow professionally, and you get an annual kitty of INR 60,000 to do so. This includes setting-up your entire home office, attending or speaking at conferences and workshops, taking courses, acquiring hardware or software licenses or even joining summer schools. We feel that learning a skill should never be a hurdle to solve important problems for the community.
Remuneration:
₹ 9-12L per annum
Our Hiring Process
The entire hiring process averages between 3-4 weeks and comprises simple five simple steps:
- You apply with your detailed portfolio/CV and a cover letter
- We have an Introductory discussion
- Based on how our discussion goes, we'll give you a take-home assignment
- We meet, ideally in a week's time, to discuss the assignment
- If all goes well, we'll have a final 'Culture Discussion' round, and you get to meet the rest of the team
Note: Our hiring process works in weekly cycles, where we will collect the applications every week and start the funnel for the selected candidates together in subsequent weeks. Please expect a response within 7-14 days of your application.
Job tags
Salary
