Background
The need for scientific approaches has been growing across society, as symbolized by the widespread adoption of Evidence-based Practice (EBP). This trend is not limited to merely gathering data, but also extends to the increasing importance of scientifically grounded methods of evaluating effectiveness and inferring causality. For example, companies use A/B testing and surveys to assess the impact of marketing initiatives, governments draft policies based on data, and educational institutions regularly measure learning outcomes through tests.
In the crypto space, there has been a similar increase in initiatives that prioritize evidence-based approaches. Specifically, there are growing efforts to examine the outcomes of grants and programs based on data and to use those findings to guide improvements. As the world becomes more digital, Evidence-based Practice is becoming ever more accessible, and this trend is particularly pronounced in the crypto sphere, where public data is widely available.
So what should we consider, and what should we be careful about, when applying Evidence-based Practice to the crypto domain? This article will review traditional discussions from other fields and explore the potential of Evidence-based Practice in the context of crypto.
Evidence – Understanding Complex Phenomena
“Evidence” is a concept employed in a wide variety of settings. In the early 1990s, “Evidence-based Medicine (EBM)” gained considerable traction, and this success paved the way for the adoption of evidence-based approaches in social welfare, education, and other fields in the late 1990s.
At the same time that evidence began to attract attention within the social sciences, there arose some confusion over what exactly “evidence” entails. One common debate equates evidence with data: that is, using data to verify whether a given initiative—which might otherwise be driven by intuition—actually has the intended effect. In contrast, fields such as medicine and education often view evidence as “proof that an intervention is effective,” placing stronger emphasis on rigorously testing the impact of that intervention. This perspective introduces the concept of an “evidence level” to classify the reliability of evidence, ensuring appropriate use in decision-making.
Generally speaking, Evidence-based Practice (EBP) involves using evidence to plan and improve real-world activities. David L. Sackett, one of the leading figures in EBM, defined it as “the conscientious, explicit, and judicious use of the current best evidence in making decisions about the care of individual patients.” [1] EBM, and by extension EBP, is often described as encompassing three stages: production, communication, and utilization of evidence. In other words, evidence must be generated—using methods such as randomized controlled trials or propensity score matching—and then applied to decision-making in a given context. This evidence can be both quantitative and qualitative, and it is common for the creators and users of evidence to be different entities.
From this perspective, it becomes clear that evidence is not simply the accumulation of data. Instead, beyond collecting data, one must consider how to analyze it, how to verify effectiveness against a target goal, and how to interpret the results. For instance, if a patient’s health improves after receiving a certain medication, was it due to the medication, natural recovery, or diet? If you cannot make that distinction, you do not have evidence. By running a clinical trial on multiple patients under similar conditions, you can obtain higher-level evidence.
Value – Where Do We Stand?
As the emphasis on evidence has grown and its application has expanded, many fields have reaped tangible benefits. At the same time, however, it is impossible to ignore the critical role of value systems underlying that evidence. Measuring and testing something are acts that reflect a specific value framework. For instance, if you measure student progress using test scores, you implicitly accept that doing well on tests is “good,” thereby excluding attributes like running speed or height from evaluation.
EBP aims to steer us toward a “better direction” by utilizing evidence. Yet the phrase “better direction” itself is rooted in a particular set of values. In other words, the production of evidence and its use in decision-making both reflect certain value judgments. These values function as a kind of operating system in EBP, and no practice can fully escape them. In fields such as social welfare and education, which involve diverse stakeholders, this issue often surfaces.
Suppose, for example, a particular policy’s impact has been measured and communicated as a form of progress. From the perspective of a community with different values, that “progress” might be undesirable. Furthermore, if funding flows are designed around such progress metrics, what sorts of incentives result from that? As mentioned, measurement and evaluation will inherently promote certain values, and linking them to incentive mechanisms only intensifies this effect.
Hence, in a community where multiple value systems coexist and collective decisions must be made, it can be difficult to establish an absolute standard of what is “correct” or “good.” This challenge is the backdrop for why values play such a pivotal role in discussions of evidence.
Application to the Crypto Domain
Given the above discussion, how should the crypto domain approach evidence?
As noted, evidence is a powerful tool, and further developments in evidence-based methods are expected. The expanding use of data, advances in machine learning, and increasingly sophisticated computing resources are all propelling new methodologies forward. In crypto in particular, on-chain data and oracles make a relatively large pool of data available, suggesting even greater momentum for evidence-based initiatives. Just as concepts like prediction markets or commons governance found fresh vitality in crypto after struggling in traditional markets, innovative approaches grounded in causal discovery, causal inference, and impact evaluation may emerge uniquely in crypto.
On the other hand, crypto protocols and DAOs are inherently decentralized and collective in nature. Historically, this has spurred interest in governance methodologies and tools that encourage a more democratic process. Such an environment gathers people with diverse values, and even a single protocol or DAO can attract conflicting viewpoints. This is especially critical for protocols or DAOs serving a public function. If evidence is generated through a limited process, it may reflect only certain values, and using that evidence to dictate funding or decisions runs the risk of excluding other important perspectives.
It is fair to say that the usefulness of evidence has been demonstrated in virtually every industry. Yet the endeavor of generating, accumulating, and applying evidence is always underpinned by the diversity and plurality of values. In the crypto domain, which is deeply rooted in philosophies of decentralization and self-sovereignty, these considerations are particularly essential.
Twinning Value with Evidence
Evidence must be considered alongside its underlying values if it is to be used effectively. This is especially true in the crypto domain, where a variety of value systems coexist. In such a space, it is necessary to discuss what constitutes a “good state” for a protocol, DAO, public good, or commons. It is also essential to examine the causal relationships behind how specific interventions or programs produce their effects, and to reconcile different perspectives on these outcomes.
From this standpoint, two key points stand out:
Measurement vs. Evaluation Measurement involves collecting data, visualizing it, and analyzing it to understand reality. Evaluation, by contrast, entails making a value judgment—deciding whether a given initiative was successful or not—and then feeding those insights back into practice. While the two processes may appear similar, it is crucial to recognize their differences.
Outputs vs. Outcomes Outputs refer to the direct results of an initiative—for example, the number of participants at an event or a new product that was created. Outcomes, on the other hand, are the changes brought about by those outputs—such as how participants’ behavior evolved after attending an event. Outputs are often easier to measure and thus more commonly tracked. However, to truly assess an intervention’s effectiveness, one must prioritize outcomes, which often require gathering and analyzing more complex information. Formulating hypotheses about how your outputs lead to specific outcomes, then collecting data to test those hypotheses, is crucial. If the hypothesis is validated, allocating additional resources may be warranted; if it is disproven, a rethink is required. By iterating through a PDCA (Plan-Do-Check-Act) cycle, you can deepen your understanding of impact.
Looking ahead, scientific approaches and evidence-based practices in the crypto field are only going to grow. Ideally, these practices will evolve in ways that respect the philosophy of decentralization, leading to ever more effective and equitable endeavors.