Data-Driven Aid: Scaling Data Management for Global Nonprofit Impact

About the Customer

The customer is a nonprofit organization dedicated to supporting other nonprofits by connecting them with donors and companies worldwide, through global crowdfunding and aid.

The organization focuses on empowering local groups around the world by providing them with the tools, support, and training they need to make a difference and improve lives. They believe in the power of small, local organizations to drive worldwide change and create a better future for everyone.

The organization operates in over 175 countries, managing 37,000 projects, and has raised close to $1 billion through their donation platform. They also regularly support natural disasters and humanitarian crises through their network, which supports community-led relief by sending aid to local organizations that are best-suited to provide help.

The customer’s platform is the largest database of validated nonprofits, charities and NGOs worldwide that consolidates government data, public information and other pertinent data.

It assembles this information in an easy-to-use and always up-to-date virtual map, enhancing aid and disaster relief through visualization, mobilizing volunteers, managing donor and grant programs and ensuring validation of organizations.

Two people packing food items into donation boxes, which include canned goods, eggs, and grains. They are wearing gloves and surrounded by other packing materials. As part of a global nonprofit initiative, their efforts are data-driven to efficiently maximize the impact of donations.

The Challenge

The main goal of the project was to create a more efficient and scalable system to manage and update data from government and reliable sources in a timely manner. The platform API allows companies to integrate grassroots project information, organization data, and donation options in their apps or on their websites by giving direct access to the information in the platform’s database. Companies can use this information to create workplace giving programs that connect users with vetted nonprofits that support specific causes, regions, or countries, as well as to learn more about nonprofits and create more ways to support community-led change

Data importers and API issues

The existing data importers required an overhaul to ensure data could be refreshed more effectively and accurately.
The API needed rewriting to handle larger volumes of data more efficiently, enabling quicker updates across multiple countries.
The data exporter required corrections to ensure data could be exported seamlessly.
A data promoter was needed to enhance the visibility and usability of the data within the system.

Critical bugs

Common Crawl, an open repository of web crawl data, had outages that led to the need for manual intervention.
The website collector needed assistance with large files and sites.
Issues with organization names with non-Latin characters in their ID and empty step functions caused blocks, requiring manual intervention.
Reduction of manual steps in the import process and documentation updates were crucial for success.

Documentation updates

Prior to the project, no country had its data fully updated within the past 90 days, highlighting a significant lag in maintaining current and accurate data records. As a result, the following improvements were necessary:

Finding new nonprofit data formats and locations for the data importers that no longer work, in addition to updating the platform’s documentation which required contacting country representatives and offices.
Updating documentation on all countries, ensuring the data locations they had were the most accurate.
Developing a strategy to proactively monitor nonprofit data format and data location changes in the future

The Solution

The solution included rewriting the data importer APIs, enhancing data exporters, and implementing robust error-handling mechanisms.
Several practices impacted the end result:

Where possible, the gathering of information was done programmatically, which included a digital scraping of various trusted sources for new or updated information.

An updated spreadsheet of countries, their data import status, and any blocking issues or difficulties in gathering information were kept to track project progress.

The Kanda team met daily to review the progress being made in importing and updating data from various sources, prioritizing those countries that are most likely to be desired by potential customers of the platform.

The Project Team

The project was a collaborative effort between Kanda and the customer. The Kanda team was comprised of a project manager, a business analyst, three Python developers, and a solution architect, all contributing to significant optimization. The Kanda and customer teams blended to focus on daily operations, access management, pull request reviews and providing guidance for each other throughout the project.

The project employed agile scrum methodology, with daily stand-ups, periodic demos, and continuous feedback integration, ensuring that all team members remained aligned and responsive to project needs.

The Results

The improved system now supports more efficient and scalable data management, enabling the customer to provide its users with timely and accurate data on nonprofits worldwide.

The changes led to significant improvements, such as reducing the data import time from 4-5 days to 4-6 hours. Gathering data for all countries combined now takes 110-120 hours, down from around 4.5 months initially.

Throughout the project, numerous lessons were learned, including the importance of proactive team management and the critical role of access and authentication protocols in maintaining project security.

The project is currently ongoing, with continued efforts to add data for additional countries and refine existing scripts.

Three volunteers in light blue shirts pack items into brown paper bags. One handles a tablet, focusing on data management while the others work together and smile. They are inside a well-lit area with a red door in the background, embodying the spirit of a global nonprofit organization.