Kudos to all reviewers: Jonas(Optimism), Carl(Open Source Observer), Holke(Hypercerts Foundation), LauNaMu(Impact Garden) and Mahesh(Karma)
As mentioned in previous reports, public goods funding is crucial for sustainably providing and maintaining public goods. Currently, in the public goods space, there is growing interest in how to provide funding for public goods fairly and effectively. However, questions such as "which projects (public goods) should receive more funding?" and "were these projects (public goods) truly effective for certain ecosystem?" should be investigated, but remain largely unaddressed.
For effective funding distribution, led by projects like Optimism, there is a trend toward retrospective funding based on past achievements rather than funding based on future expectations, as this is considered more reliable. However, the earlier questions still remain unanswered. To address these questions, there is increasing focus on effect verification and impact evaluation, with many projects beginning to work in this area.
We are also analyzing grant program data and reviewing which grant programs are providing what amount of funding to which projects to understand and analyze the current situation. In our previous report, we organized the steps for impact evaluation and clarified commonly misunderstood points, preparing ourselves for future impact evaluation (more precisely, program evaluation).
One member of Fracton research has developed the IES(Impact Evaluation Service), a protocol that provides continuous incentives to impact evaluators, and won a prize from Funding the Commons at a hackathon. Also, in our latest report, we mentioned the importance of evidence based process.
Besides us, there are many other projects that are contribute to impact evaluation, and this report will introduce them and explore the current state of impact evaluation in crypto.
Projects working on Impact Evaluation
In crypto, one of the reasons for the focus on “impact evaluation” may be Optimism's “Impact = Profit” principle. Indeed, impact evaluation efforts are centered around the Optimism ecosystem. In this report, we focus on Optimism's impact evaluation initiatives and projects working on impact evaluation around Optimism: Open Source Observer, Hypercerts, Impact Garden and Karma GAP.
Optimism
Following the principle of “Impact = Profit”, Optimism provides funding to projects that contribute to the Ethereum and Superchain ecosystem through Retro Funding(prev PGF)[1]. As of January 2025, when this report is being written, 6 previous rounds have been completed, with a total of over $60M in $OP tokens distributed to projects including digital public goods and OSS.
In Round 4, values based on onchain information and github activities are collected as impact metrics through Open Source Observer. These metrics can be found at RetroPGF Hub.
Each badge holder sets weights for the metrics so that the total is 100%. Through this process, each badge holder quantifies how important each metric is to the impact evaluation.
Each badge holder then sets a multiplier(called Open Source Multiplier) if the target project was OSS. Through this process, we quantify how much weight each badge holder places on funding OSS.
These quantified data are used to score each project and determine the final distribution amount for each project.
The data is available on Github, not only for Round 4 as introduced here, but also up to the most recent Round 6. Please refer to their Github for details on each calculation method and voting results.
Open Source Observer
Open Source Observer (OSO) is an open-source tool for measuring open source project activity. All OSO code is public, and anyone can access public data through their API.
OSO visualizes activity by tracking Github, smart contracts, and project EOA addresses, serving as a reliable source for grant operators reviewing projects. Optimism's RPGF uses OSO's data sources in their distribution decision process.
In addition, OSO and Eval.Science(Counterfactually) are working on impact measurement utilizing the Synthetic Control Method, a statistical method used to evaluate the impact of interventions.
In this example, they synthesized and weighted on-chain data, excluding Optimism, and compared changes in monthly active developers before and after the intervention to measure the impact of Retro Funding as an intervention [2].
Karma GAP
The Karma Grantee Accountability Protocol (GAP) enables grant operators to track grantee progress and hold them accountable. Grantees also build up reputation by maintaining an onchain resume of their projects including all the grants they have received and progress updates. Karma has also implemented Impact measurement based on Common Impact Data Standard framework. Karma GAP is used by various grant programs including Celo PG, Octant, Arbitrum, Gitcoin, Public Nouns and more.
GAP is an onchain protocol built on Ethereum Attestation Service. It is multichain and has been deployed on many popular EVM L2 networks including Arbitrum and Optimism. Grantees can register their projects through Karma GAP's frontend, set milestones, report progress, and declare received grants.
These actions are recorded on-chain as attestations, and GAP can be integrated into existing applications through the GAP SDK. This integration enables observation of grantee outputs and assists in project evaluation.
Hypercerts
Hypercerts is the token standard for Impact Certificate, it extends(and is backward compatible) ERC1155 to store information regarding the work that is supposed to be impactful. These hypercerts can then receive multiple evaluations, incl. self-reports, to establish and verify the claims about the impact of the workrelated to Impact.
In this way, Hypercerts enables impact tracking and accumulation as a data layer.
Hypercerts make it simple to link impact claims and evaluations to selling impact on a marketplace, and makes it easier for retrospective funding to fund projects when the impact is verified with an Impact Certificate.
Thus, the project owner can expect funding from Prospective Funder in exchange for a portion of the Hypercerts, and Prospective Funder can sell their Hypercerts to Retrospective Funder through the Marketplace. Prospective Funder can exit by selling its Hypercerts to Retrospective Funder through the Marketplace [3].
Hypercerts' docs introduce this as Impact Funding Systems (IFS), one use case of Hypercerts. Hypercerts has a SDK, called hypercerts marketplace SDK, for application builders to integrate hypercerts into their own funding platform or build new funding mechanisms on top of the hypercerts protocol. VoiceDeck and Ecocertain are the great examples built with hypercerts marketplace SDK.
Hypercerts' evaluation mechanism is based on the idea of an “Open evaluation system".
Multiple evaluations can be submitted from different evaluators with different methodologies. The evaluations can support each other to strengthen the trust in the evaluations. Or they can conflict with each other. When they conflict, we want to understand if the evaluators (1) define impact differently, (2) measure it differently, and/or (3) made a mistake in measuring it. Surfacing these conflicts is crucial to strengthen the trust in the overall system.
At this time, a few number of evaluators, including GainForest and the Hypercerts Foundation, are registered as evaluators. Hypercerts will become more valuable as evaluation mechanisms in the ecosystem are established.
Impact Garden
Impact Garden, developed by Metrics Garden Labs, provides quantified and standardized impact metrics derived from reputable community member's qualitative insights. Instead of using only quantitative indicators to evaluate impact, projects can be evaluated in a more comprehensive way by including qualitative information such as reputation.
Users can connect their Farcaster accounts and add reviews to projects registered on Impact Garden. Impact Garden aggregates this data to create impact metrics.
To evaluate projects, users must log in with their Farcaster account and select the project they would like to review. After selecting a project they are asked to rate it in three different verticals that are later used to create impact metrics. Impact Garden aggregates this data to create impact metrics and leverages the reputation linked to the Farcaster account to weight the data provided in accordance to the role of the reviewer in the community.
Impact Garden soft launched in RetroRound 5 and generated parallel data for OP Stack contributions. In Retro Funding Round6, Impact Garden was used to experiment with the creation of standardized qualitative data voters could leverage in their voting process[4]. This integration can be found at RetroPGF Hub and Retrolist.
These data can be accessed through their public API which makes these data easy for anyone to leverage on their platforms. The public API is what enabled multiple platforms to integrate the data and also enables projects to display their information.
Our Path to Impact Evaluation
It is becoming more common for protocols to fund projects from their own treasuries with a specific purpose.
People are starting to pay attention to questions such as “Did the funding really make sense?” and "What kind of projects are most aligned with the purpose?"
As described in this report, the number of projects working on impact evaluation, including us, are increasing, and we are developing an environment in which impact evaluation can take place.
We introduced OSO, which measures OSS activity, Karma GAP, which focuses on output management, Impact Garden, which aggregates qualitative evaluation of projects, and Hypercerts, which serves as a impact certificate, but these tools by themselves are not enough to conduct highly accurate impact evaluation. However, the collaboration of these tools will create an environment that will make impact evaluation possible.
Many of the impact evaluation projects introduced here focus on OSS, but we are planning to focus not only on OSS but also on community activities to conduct impact evaluation. Unlike OSS, community activities are difficult to assess because they are often qualitative, subjective, and offline, but they are an essential component of a crypto space, where community-driven governance is a prerequisite.