The Operational Evaluation: Expanding Our Definitions of Non-Profit Project Evaluations

Non-Profit project evaluations always seemed to me to be a little like a science fair project. You identify your theory of change or hypothesis, and some variables that you are hoping to impact. You reference some past research. You then identify some biases, discuss your methodology, collect your data, write a report and draw some conclusions to support or not support your hypothesis. You submit that report and apply for the next round of funding.

As I started to work on non-profit evaluations, something always seemed to be missing to me. Perhaps it was the accountant in me. Perhaps it was the maverick in me or the nonconformist, but this idea of an evaluation that lived in a bubble made zero sense to me. We were not observing widgets in a bubble, we were talking about programs that impacted people, who are run by people, funded by donations, donors or funders, that live and operate in a sociopolitical environment. The sustainability, technological feasance and HR effort required to run programs, along with a million other variables impacted the OPERATION AND OUTCOME of the program, just as much as the variable or two that we identified and put under the microscope.

I would ask my nonprofit partners for the financials of the program, and they would look at me like I was crazy. I would make observations about the changing sociopolitical climate that reduced or enhanced the ability to get funding for this type of program and they would ask for those sections to be removed or edited. I would make observations about the capital or human resources requirements of a project and would get question marks. None of this mattered to funders and little of it mattered to nonprofits and their managers and staff.

Truth be told, there were a few who GOT it. Mainly those who were running social enterprise or market based programs. They understood that the impact of a program was not solely dependent upon funding and implementing some type of intervention, but rather without funding or with less funding, could the program still run and be successful, or was it too HR or Tech intensive, did it go against the current funding regimes/trends or go with or against current training theories of change.

We, rightfully, have a very systematic way to evaluate our interventions and programs, but we borrow too much from social science and not enough from the business sector. While adopting strong methods is important, if we want to achieve impact and scale, we also need to learn some lessons from the private sector.

What Needs to Change?

Namely, there are few things that need to change. An Operational Evaluation is an innovative approach that does just this, transcending conventional evaluation boundaries to encompass the critical operational factors that support and drive non-profit projects. This approach not only gauges the success of individual projects but also examines the underlying systems and processes that form the backbone of an organization, the ecosystem it operates within and predicts its future likelihood of success by examining and evaluating inputs, requirements and capital required to run the project. So how do we do this? Below, I give you my recommendations.

1. Run a PESTLE

A PESTLE analysis is a strategic management tool used to identify and analyze the macro-environmental factors that may impact an organization. The acronym PESTLE stands for:

• Political: These factors determine the extent to which a government may influence the economy or a certain industry. This includes government policy, political stability or instability, tax policy, labor laws, environmental laws, trade restrictions, tariffs, and political trends.
• Economic: These factors affect the working of an organization depending on the economic environment. This includes economic growth, interest rates, exchange rates, inflation, economic trends, the general economic cycle, and consumer confidence and spending power.
• Social: These factors focus on the social environment and identify emerging trends. This includes cultural aspects, health consciousness, population growth rates, age distribution, career attitudes, and emphasis on safety, societal norms, and lifestyle changes.
• Technological: These factors pertain to innovations in technology that may affect the operations of the industry and the market favorably or unfavorably. This includes automation, research and development activity, technology incentives, the rate of technological change, and advancements in digital or mobile technology.
• Legal: These factors include all legal aspects like employment law, consumer law, health and safety law, international as well as trade regulation and restrictions.
• Environmental: These factors include ecological and environmental aspects such as weather, climate, climate change policies, environmental offsets, carbon footprint targeting, and waste disposal laws.

Organizations use PESTLE analysis to track the environment they’re operating in or are planning to launch a new project/product/service etc. It helps businesses to understand market growth or decline, business position, potential and direction for operations. The same should be done for nonprofit project evaluations.

2. Conduct a SWOT

SWOT stands for Strengths, Weaknesses, Opportunities, and Threats. It is a structured planning method used to evaluate these four aspects of a project or business venture.

• Strengths: characteristics of the business or project that give it an advantage over others
• Weaknesses: characteristics that place the project or business at a disadvantage relative to others
• Opportunities: elements that the project or business could exploit to its advantage
• Threats: elements in the environment that could cause trouble for the business or project and brainstorm how to mitigate

A SWOT analysis is often used at the inception of or during a strategic review of a company’s processes, to understand where the company stands and what strategic changes it could make to improve its position.

3. Run A Post Project Audit

1. Post-Project Review: Hold a meeting with the project team to discuss what went well and what didn’t. Document lessons learned for future reference.
2. Project Scope: did the project stay on scope or did it change for some reason? If so, why? What was the impact?
3. Project Schedule: Did the project stay on schedule, did some parts take more or less time?
4. Team Goals: Did this project meet the team goals? Sometimes our interventions are successful but for some reason they do not meet team expectations. Why or Why not?
5. Human Resource Dynamics: Shifting focus to the people behind the projects, this assessment analyzes team composition, skill sets, training needs, and staff satisfaction. It explores how human resources practices affect project execution and organizational culture. If the team was different, could the same outcomes be achieved?
6. Technology and Infrastructure Review: By evaluating the technology and infrastructure in place, this aspect ensures that organizations are equipped with the right tools to efficiently manage their operations and adapt to changing landscapes. Could the project be done with different technology? More or less?
7. Risk Management and Compliance: This crucial element looks at how well an organization identifies, mitigates, and manages risks, along with its adherence to legal and ethical standards. It’s about safeguarding the organization’s reputation and ensuring resilience. If the project was replicated, does it put the organization at risk in some way?
8. Organizational Process Optimization: Here, the focus is on streamlining processes for better resource management, reducing redundancies, and improving overall workflow efficiency within the organization.
9. Stakeholder Engagement Effectiveness: This component measures the effectiveness of engagement strategies with key stakeholders — from donors to beneficiaries — assessing how these relationships impact organizational success.
10. Performance Metrics: Compare the project’s performance against key performance indicators (KPIs) established at the beginning.
11. Financial Analysis: Perform a financial review to assess the project financials. How did it perform, were the financial inputs at the appropriate times or should those have been changed?
12. Audit of Project Documentation: Review all project documentation to ensure that all processes were followed and deliverables were as per the scope.
13. Quality Control Checks: Ensure that the final product or service meets the quality standards through testing or audits.
14. Benchmarking: Compare the outcomes of the project against industry standards or similar projects if the numbers exist
15. Resource Utilization Analysis: Look at how effectively resources were managed by comparing planned versus actual resource use.
16. Sustainability: Given what you know from a PESTLE, will there be continued funding, is the political climate changing? Given the resource utilization of people, tech and funds, what will it take to make this project sustainable? Could it run on less?
17. Scalability: Given all of the factors here, how could this project, if it was successful, be scaled? Do we not owe it to our funders, donors and clients to replicate and grow our impact? How can it be scaled successfully?
18. Evaluate Bias: Evaluate the systems that make up this project and identify bias–all the different types of it (Check out our other post on Bias) There are more types than you think and you need to ensure that you check yourself to make sure the project, you as the evaluator and the processes are not biased.

I am sure by now I have overwhelmed you. I hope you are a little scared, that is the point. We are not doing a great job evaluating our current projects. Truth be told, a different multidisciplinary skill set is required. However, if we want to move to a true systems evaluation, we need to move to an Operational Evaluation, that evaluates not just project, but the systems it operates within, the organization, the people it helps and that are doing the work, the funding regime and all the other moving parts that make up the SYSTEM our nonprofits operate within.

We need better approaches to evaluating projects. Even if you just add one or two of these, it will make for a better, more holistic and accurate evaluation. I am sure some of you are hoping to do less evaluation work, as most organizations hate it. I would challenge you that you will never grow as an organization, nor will we truly implement systems change if we do not begin to challenge ourselves to change the system and how we view, fund and evaluate our success within it.

If you found value in this blog, we would love to hear from you. Please feel free to contact hello@pharononprofit.com to give us feedback, ask questions or leave your comments.
You can also access more content on this and other issues facing nonprofits by joining our community at: https://nonprofitportal.com/join-n