All four NGOs had operated in their respective areas for at least ten years prior and had been involved in past initiatives ranging from agriculture to health and nutrition to savings and credit. As a result, local implementers had preexisting relationships with study participants before the inception of the pulses program studied here. From summer 2017 through spring 2019, implementers procured and distributed certified seeds of improved pulse varieties at subsidized prices, conducted local training and extension sessions to demonstrate best practices, gave individualized feedback to program participants through the cropping season, and assisted with the sale of output. The NGO seed distribution network, which brought in higher quality inputs than were previously available in local markets, remained in place after other activities concluded. Program activities were carried out in a randomized controlled trial with the intent of using lessons from the pilot to design a statewide policy that could be adopted and run by government agencies. Both treated and control farmers knew they were part of a pilot evaluation that may scale up if successful. Full program evaluation results are reported by Lybbert et al. . In this paper, we quantify how preexisting relationships with the implementer may affect impact evaluation in pilot experimentation. A key evaluation outcome for the trial was sustained production of pulses after main program activities had concluded. As part of this evaluation,strawberry gutter system demand for certified pulse seeds was elicited among a sample of study farmers ahead of each planting season in the third year.
Specifically, we elicited demand for pigeon pea and black gram in the summer season and for red lentil in the winter season of 2019. All three crops were initially promoted by the pulses program, but second-year implementation predominantly focused on black gram and red lentil. Farmers’ willingness-to-pay for these inputs constitute the primary data analyzed in this study. Each demand elicitation included farmers from the same village who placed orders for certified seed from the NGO supplier through a price list BDM revelation mechanism. For each variety, participants were given list of possible prices and asked to report the seed quantity they wanted to order at each price. At the end of the session, one price at random was selected from the list. Participants purchased their stated demand at that price, and they did not have the option to adjust their quantity demanded after the transaction price was revealed. The purchase transaction ensures that each demand decision was incentive-compatible as reported demand was, in effect, the farmer’s seed order from the NGO. After the elicitation exercise, participants were asked a short set of questions related to demographics and intended pulse cultivation. Participants in the winter season are also asked about their involvement with the pulses pilot program, their subjective perception of whether the pulse program had been beneficial, and their past involvement with the local implementer’s other activities. Implementer desirability bias manifests as a real shift in the agricultural portfolio of evaluation participants. The estimated difference of a half kilogram of purchased pulse seed corresponds to spending about Rs. 70, or $1.00, more at supplier prices.
While this expenditure by itself may be small, program implementers were active in ensuring farmers planted and cultivated what they ordered. As a result, the main cost to participants adjusting their seed demand came in the acreage and effort they subsequently devoted to pulse farming. Purchase quantity and area cultivated were documented by the NGO supplier for all farmers from villages that received the pulses intervention. Table 3 reports the relationship between these outcomes in administrative records. The table shows a strong, positive correlation with a regression R-squared of between 0.6 and 0.8. Importantly, there is no systematic pattern in deviations from this relationship by evaluation salience. After controlling for seed quantity, the effect of evaluation salience on acreage is quantitatively small and statistically indistinguishable from zero. This evidence indicates that differences in elicited seed demand correspond to real differences in area planted, and are not merely performative purchases made at low cost and subsequently discarded or given to another farmer. For black gram and red lentil—the two crops where evaluation salience has the greatest impact—one kilogram of seeds purchased corresponds to roughly 0.07 acres cultivated. Multiplying this by the effect on seed purchases, we estimate implementer desirability bias led farmers to reallocate around 0.03 crop acres, or 2% of their total landholdings. While this represents a small portion of their total agricultural portfolio, the investment of land—and associated labor and other inputs—nevertheless reflects a real and sustained contribution of costly on-farm resources when program evaluation is made salient. This quantification must be interpreted with two caveats. First, administrative records were only kept in villages treated in the pilot evaluation, and no comparable records exist for untreated villages.
This omission, while unfortunate, is not so concerning because untreated farmers reduced their seed order in response to evaluation salience, so there is no risk of performative over purchasing as there may have been among treated farmers. Second, transaction records cannot be linked to individuals in the demand elicitation, so salience treatment status is assigned at the village level. Experimental elicitations accounted for only a quarter to half of seed purchases, with the remainder being regular orders from non-participants who were not influenced by the program evaluation. Despite these caveats, the evidence is consistent with a modest but real reallocation of resources in accordance with implementers’ desires. Prior engagement with the local implementer also influences seed demand, but this channel neither explains nor dilutes the relationship between treatment status and evaluation salience. After the second demand elicitation, farmers were asked about their participation in the pulses program, their beliefs about how beneficial the program had been, and their engagement with other NGO development activities in the past. Columns and of Table 2 explore how these variables affect experimental results. It is worth noting that self-reported beneficiary status may be an endogenous outcome of evaluation salience. Table 4 reports how the three variables relating to program benefits differ with experimental assignment. The first two columns show that while those treated in the pilot experiment are more likely to self-report participation the pulses program, a majority claim to have participated even in villages assigned to control. This is likely because all study farmers were initially recruited for a pulses-related evaluation prior to treatment assignment, so those even those who did not directly receive the intervention package may have considered themselves to be part of the pulses program. The final two columns show that those who received the pulses intervention are also more likely to self-report having participated in other NGO activities in the past despite random assignment of treatment. Interestingly, evaluation salience seems to prime participants to recall any NGO engagement,hydroponic nft gully raising the likelihood of self-reporting participation in both the pulses program and in past NGO activities.The middle two columns of Table 4 reveal three quarters of participants claim to believe the pulses intervention was beneficial. There is little variation in this rate with either treatment assignment or evaluation salience. A favorable response to this question is cheap talk and likely less indicative of true beliefs than real-stakes purchase and cultivation decisions. Evaluation salience differentially affects the input demand choices of those who self-report prior NGO engagement. As Column 4 of Table 2 demonstrates, self-identified pulse program participants have sightly lower demand on average, but their demand substantially increases when the evaluation is made salient. The inverse is true of prior engagement with other NGO actives: seed demand is slightly greater among past beneficiaries, but evaluation salience substantially lowers it. This latter pattern is consistent with a model in which mentioning the NGO at the start of the evaluation leads participants to anchor expectations around the organization’s typical strategy of providing benefits that are heavily subsidized or free,5 though other explanations are also possible. In any case, the coefficient estimates on treatment assignment, salience, and their interaction remain equally strong after controlling for self-reported measures of NGO engagement.
When evaluating a program run by a sympathetic implementer, the salience of the evaluation itself can positively bias the estimated treatment effect. We establish this fact in the context of an unsuccessful pilot agricultural intervention that actually reduced farmer demand for the target crops but was implemented by organizations with strong community ties. When the evaluation was made salient, participants altered their behavior in a real-stakes demand elicitation with binding input decisions for the coming crop season. Making the evaluation salient effectively closed the gap between treatment and control, masking evidence of the negative program impact on seed demand. This outcome is consistent with participants adjusting their seed purchases to align with the desires of the program implementer. One possibility is that program beneficiaries understand that NGOs’ funding and continued operation depend crucially on their ability to demonstrate success, and strategically act to deliver positive evaluation results. In this case, the costly actions taken by evaluation participants can be seen as investments in anticipation of a future stream of benefits that will remain intact as long as the implementing NGO remains active. Alternatively, invoking program evaluation at the start of data collection may induce feelings of reciprocity toward the implementer. Participants may wish to reward the implementer in exchange for the benefits of the program itself, goodwill generated by the effort put in by implementing agents, or other aspects of service delivery. Whatever the catalyst, beneficiaries of development programs can reciprocateat the evaluation stage by behaving in accordance with program goals. Our investigation focuses on identifying and quantifying the size of implementer desirability bias, but isolating the exact channel through which it operates remains an open topic for future research. The bias identified in this study represents a specific threat to reproducibility with clear implications for translating experience from pilot programs into broader policy lessons. When introducing new development initiatives, it is common practice to partner with established organizations that have strong community ties to leverage their local knowledge and institutional capacity. However, our findings suggest the features that enable successful implementation are precisely those that can undermine accurate evaluation. When faced with the prospect of evaluation, beneficiaries sympathetic to the implementer may take costly actions that help the implementer achieve its goal of demonstrating success. As a consequence, pilot evaluation can uncover misleadingly optimistic results that overstate a policy’s true impacts and therefore do not replicate at scale. Bacteria in the genus Frankia form nitrogen-fixing root nodule symbioses with a group of host plants, the actinorhizal plants. The actinorhizal plants are evolutionarily closely related to the legumes within the Nitrogen-fixing Clade . The bacterial symbionts are only distantly related to each, however, Frankia belongs to the phylum Actinobacteria, comprised of gram-positive bacteria with high-GC genomes. By contrast the rhizobia, symbionts of the legumes, are gramnegative proteobacteria with genomes generally closer to 50% GC. Additionally, Frankia is a multicellular, hyphal organism with a complex life cycle. In nitrogen limiting conditions, Frankia can produce stalked, ovoid to spherical structures that develop from branch hyphae called vesicles . Frankia vesicles, known to be the sites of nitrogen fixation, are surrounded by a lamellar envelope comprised of hopanoid lipids and have high rates of internal oxygen consumption through respiration. The vesicle envelope increases in number of layers in response to oxygen tension, thereby likely reducing the flow of oxygen into the vesicle interior to protect the nitrogenase complex from deactivation . Thus, unlike the rhizobia-legume symbioses, where the host nodule tissue regulates the flow of oxygen during nitrogen fixation , most Frankia are capable of fixing nitrogen in the vesicles in atmospheric oxygen conditions under nitrogen limitation . Finally, in symbiosis and in vitro, Frankia is capable of forming sporangia containing thick-walled spores , which apparently serve as a resting stage and as a source of propagules that can enhance nodule numbers . Biological nitrogen fixation has potential applications in the development of minimal input strategies for more sustainable agriculture globally, and in nutrient-deficient soils . Genetic improvement has the potential to introduce nitrogen-fixing symbioses into a wider range of crop plants once a broader understanding of the mechanisms involved in these symbioses is achieved .