Data-Driven Aid: Scaling Data Management for Global Nonprofit Impact
Table of Contents
About the Customer
The customer is a nonprofit organization dedicated to supporting other nonprofits by connecting them with donors and companies worldwide, through global crowdfunding and aid.
The organization focuses on empowering local groups around the world by providing them with the tools, support, and training they need to make a difference and improve lives. They believe in the power of small, local organizations to drive worldwide change and create a better future for everyone.
The organization operates in over 175 countries, managing 37,000 projects, and has raised close to $1 billion through their donation platform. They also regularly support natural disasters and humanitarian crises through their network, which supports community-led relief by sending aid to local organizations that are best-suited to provide help.
The customer’s platform is the largest database of validated nonprofits, charities and NGOs worldwide that consolidates government data, public information and other pertinent data.
It assembles this information in an easy-to-use and always up-to-date virtual map, enhancing aid and disaster relief through visualization, mobilizing volunteers, managing donor and grant programs and ensuring validation of organizations.
The Challenge
The main goal of the project was to create a more efficient and scalable system to manage and update data from government and reliable sources in a timely manner. The platform API allows companies to integrate grassroots project information, organization data, and donation options in their apps or on their websites by giving direct access to the information in the platform’s database. Companies can use this information to create workplace giving programs that connect users with vetted nonprofits that support specific causes, regions, or countries, as well as to learn more about nonprofits and create more ways to support community-led change
Data importers and API issues
- The existing data importers required an overhaul to ensure data could be refreshed more effectively and accurately.
- The API needed rewriting to handle larger volumes of data more efficiently, enabling quicker updates across multiple countries.
- The data exporter required corrections to ensure data could be exported seamlessly.
- A data promoter was needed to enhance the visibility and usability of the data within the system.
Critical bugs
- Common Crawl, an open repository of web crawl data, had outages that led to the need for manual intervention.
- The website collector needed assistance with large files and sites.
- Issues with organization names with non-Latin characters in their ID and empty step functions caused blocks, requiring manual intervention.
- Reduction of manual steps in the import process and documentation updates were crucial for success.
Documentation updates
Prior to the project, no country had its data fully updated within the past 90 days, highlighting a significant lag in maintaining current and accurate data records. As a result, the following improvements were necessary:
- Finding new nonprofit data formats and locations for the data importers that no longer work, in addition to updating the platform’s documentation which required contacting country representatives and offices.
- Updating documentation on all countries, ensuring the data locations they had were the most accurate.
- Developing a strategy to proactively monitor nonprofit data format and data location changes in the future
The Solution
The solution included rewriting the data importer APIs, enhancing data exporters, and implementing robust error-handling mechanisms.
Several practices impacted the end result:
- Where possible, the gathering of information was done programmatically, which included a digital scraping of various trusted sources for new or updated information.
- An updated spreadsheet of countries, their data import status, and any blocking issues or difficulties in gathering information were kept to track project progress.
- The Kanda team met daily to review the progress being made in importing and updating data from various sources, prioritizing those countries that are most likely to be desired by potential customers of the platform.
The Project Team
The project was a collaborative effort between Kanda and the customer. The Kanda team was comprised of a project manager, a business analyst, three Python developers, and a solution architect, all contributing to significant optimization. The Kanda and customer teams blended to focus on daily operations, access management, pull request reviews and providing guidance for each other throughout the project.
The project employed agile scrum methodology, with daily stand-ups, periodic demos, and continuous feedback integration, ensuring that all team members remained aligned and responsive to project needs.
The Results
The improved system now supports more efficient and scalable data management, enabling the customer to provide its users with timely and accurate data on nonprofits worldwide.
The changes led to significant improvements, such as reducing the data import time from 4-5 days to 4-6 hours. Gathering data for all countries combined now takes 110-120 hours, down from around 4.5 months initially.
Throughout the project, numerous lessons were learned, including the importance of proactive team management and the critical role of access and authentication protocols in maintaining project security.
The project is currently ongoing, with continued efforts to add data for additional countries and refine existing scripts.
