Full Evaluation Results

Evaluation has been an integral component of everything the Case Foundation undertakes, and the MIYO program is no exception. Since the beginning of the process, the Foundation has collected data about nearly every step of it—from asking external reviewers for their opinions to surveying groups of winners to see how they were progressing.

In the following section of the report, authors and researchers provide detailed analysis as to whether the MIYO process, grants, and other benefits to the applicants had positive effects and, especially, had helped to support high-quality citizen-centered work that would not have occurred without the MIYO initiative. The data collected in this evaluation have also had the additional benefit of providing an unprecedented picture of citizen-centered efforts occurring in America during the later part of the ‘00s—information that had previously been difficult to obtain and will be of considerable use to the field.

What Was The Impact Of The MIYO Process On The Winners?

As noted, MIYO winners not only received financial resources but also technical support and advice (especially about marketing and social media), web tools, and access to a closed social network—all resources they likely would not otherwise have had.

To test whether these resources had any effects on applicants’ efforts, researchers compared the 20 MIYO grantees to a group of 17 projects that made the “Top 100” list but ultimately did not win a grant. The assumption was that if grantees were more successful than those who weren’t selected as winners, it would suggest that the MIYO program had contributed to this difference. (This assumption was based on both groups receiving almost identical average scores by the Case Foundation’s outside peer reviewers.)

Members of both groups answered 31 survey questions about their work since the MIYO program. Although there were not many significant differences in the groups, a few did appear.

• Winners were able to conduct real public meetings that would otherwise have been too expensive or difficult to pull off. (See Figures 1 & 2.) As one grantee said, “A three-day meeting with food would have been too expensive for any local public agency, so it never would have happened without the Case grant.” That meeting led to a detailed plan that community participants are now working to implement. There was also evidence, albeit less pronounced, that winners were able to attract more people to these “civic spaces” than non-winners. (The former difference is statistically significant; the latter is not.) Note: 4 = very successful; 3 = somewhat successful; 2 = mostly unsuccessful; and 1 = unsuccessful

• The grant award helped several winners to diversify their funding bases and, in turn, sustain their work. This benefit emerged in interviews rather than from surveys. One interviewee noted, “The Case-funded project kept my organization going long enough for it to diversify its funding base.” This organization has since prospered and expanded. Another grantee attracted enough funding from other sources to expand its efforts to several other communities beyond the original site.

• Grantees said that participants in their projects and events gained civic work skills and motivation to participate in these kinds of efforts in the future. Although the survey did not measure this outcome, it emerged during the interview stage of the evaluation, with several grantees stating they believed the MIYO program had been successful in helping to motivate participants in their efforts to continue their engagement or gain confidence in their leadership skills. One grantee who works with low-income urban youth, for example, reported how a community member who had been reluctant to participate (and “only joined for the free pizza”) became highly involved over time. In addition to recruiting both his brothers to join the effort, he has since presented at panels before state legislators about the issue the MIYO-awarded project was addressing.

What Was The Impact Of The MIYO Process On The Non-Winners?

One of the original purposes of the MIYO process was to generate citizen-centered work by encouraging people to develop proposals which they might implement even if they were not selected. Research indicates that 27 percent of the survey respondents from the original applicant pool (mostly non-winners) reported that they either did the project they proposed or completed that project and went well beyond it. Only 18 percent said that they had gone nowhere, with the rest in between, giving answers like “we have managed to do some of what we proposed.” (See Figure 3.)

What Did Applicants Think About The MIYO Process?

• Survey respondents were generally positive about the MIYO application process. On a scale from 1 (“a complete waste of my time”) to 4 (“very helpful to my work”), the average responses were mostly between 3 and 4. (A score of 3 meant “somewhat helpful, but could have been better”). (See Figure 4.)

• The most beneficial part of the MIYO process to all applicants was “learning more about the citizen-centered concept”—which many had not heard of before (or not heard it described as such)—and having the opportunity to outline the details of their own projects. Almost half (46%) of applicants considered hearing about citizen-centered approaches to be “very helpful” to their work and found it “exciting” because “it completely fits what we do.” They also said having the opportunity to delineate their plans via the application process, particularly the essay questions, was beneficial and useful.

• Participants liked the feedback provided by the coaches. Overall, finalists appreciated the coaches the Foundation provided to help them prepare their proposals for judging. Most of those interviewed said they were impressed by the individuals with whom they’d been paired, suggesting that the effort to offer more personalized technical assistance to finalists was an important aspect of the process.

• Many grantees were dissatisfied with the detail and paperwork required for their proposal submissions and the process itself “for such a small final grant.” One grantee estimated that the time required to prepare her final proposal took “at least 100 hours,” which set back his/her organization’s planning process. Others complained that the “amount of work required for this program wasn’t justified by the amount of the grants.”

• Grantees did not feel that the various opportunities provided to them by the Foundation, including widgets and social media trainings, were really optional. As one grantee noted, s/he felt that they had to post a video once that opportunity was offered. This was largely due to grantees’ fears of disappointing funders because “they are gods,” as one said.

• Applicants had mixed feelings about the Case Foundation’s online trainings and webinars. The online trainings and webinars the Foundation held throughout the process covered a wide range of topics such as citizen-centered approaches; “how tos” for online tools; and project management; and financial matters related to grant awards. Applicants differed widely in their knowledge of each of these three areas; as a result, some thought that the information was too basic, and others, too difficult. One interviewee said the webinars and other trainings were useful, but they were “too hard.” Another remarked that “too much new information was presented too quickly,” although she also described the training as “much better than nothing.” Another thought that the material was “too elementary” and should have addressed harder questions such as how this work could be sustained over time.

Make It Your Own Grants

Survey respondents were asked to rate various methods used by the Case Foundation to select MIYO winners (see figure 5). On the rating scale provided, 1 meant “simply wastes time or resources”; 2 meant “not worthwhile”; 3 was “useful”; and 4 was “essential or highly valuable.”

Most aspects of the process were rated useful, but the final public voting process was least popular. Respondents rated it 2.64 on a 4-point scale. The Top 100 rated it slightly lower (2.58), as did the Top 20, the applicants who were actually subject to public voting (2.56). (Note: These differences are not statistically significant.) (See Figure 5.)

Interviews revealed certain specific complaints about the voting process. Some applicants felt it was biased in favor of projects in large cities, appearing to be a “popularity contest” for “projects that were able to mobilize supporters to vote for them.” Some interviewees said the work required to compete in the voting round was not worth the amount of money provided to the winners.

Another complaint surfaced several times: There were apparently technical glitches that prevented some people from voting, especially if they had dial-up connections. It was frustrating to persuade an individual to vote and have the technology not work.

Although the applicants were not especially enthusiastic about the voting process, it may have had benefits they overlooked, including exposing interesting citizen-centered projects to the general public. During the MIYO process, for example, the Foundation website received approximately 45,000 unique visits to the online ballot, with more than 60,000 votes cast by 15,232 individuals (each person had to vote for four projects.)