Transforming Evaluation to Evaluation Transformation - Blue Marble Operating Principles 9-12

Leading up to the launch of Michael’s new book, Blue Marble Evaluation: Premises and Principles, we shared one Blue Marble Evaluation principles each week. We got through twelve of the sixteen principles, which you can read here. The final four principles all relate to the concept of transformation, so we have grouped them together. Here are the final four operating principles for Blue Marble Evaluation:

• Theory of Transformation Principle: Design, implement and evaluate transformation based on an evidence-supported theory of transformation.

• Transformation Fidelity Principle: Ensure that what is called transformation constitutes transformation.

• Transformational Alignment Principle: Transform evaluation to evaluate transformation.

• Evaluation as Intervention Principle: Integrate and network evaluations to inform and energize transformation.

With the launch of the Blue Marble Evaluation book, we have also created a website that will host Blue Marble Evaluation going forward. We invite you to visit https://bluemarbleeval.org/ to learn more about all of the Blue Marble Evaluation Principles and to join the growing network of Blue Marble Evaluators.

Blue Marble Operating Principle 8: Skin in the Game

Principle: Acknowledge and act on your stake in how the Anthropocene unfolds.

Premise: When it comes to the survival of humanity and the planet, we all have skin in the game, we and our loved ones are in the world that is under threat. We are not outside looking in. We are part of the global system and, there’s a good chance that we are each, in our own way, part of the problem. This gives us a quite different stance than is typically expected. Evaluators are virtually always outside the programs or projects they evaluate. Acknowledging and facing the realities of the Anthropocene transforms the stance of evaluators from external observers of change to internal participants in change.

Implications:

• A shift in evaluator stance from independence to interdependence.

• Making the evaluators’ values explicit and transparent.

• Identifying your stake, whatever role you play – evaluator, designer, implementer, funder, commissioner of evaluations, intended user, policy maker – then sharing how you view your stake and the implications of that view for how you engage and fulfill your role. As a utilization-focused evaluator, we always have a stake in whether and how an evaluation is used.

• Reality-test for yourself to be valuable as a reality-tester for others. So, how good are you at reality-testing for yourself? For example, to what extent are you practicing in your life the things you know you ought to be doing (exercise, eating right, getting enough sleep…)?

Photo Credit: Ellen Harasimowicz

Blue Marble Evaluation Operating Principle 7: World Savvy Principle

World Savvy Principle: Engage in ongoing learning relevant to Blue Marble principles and practices.

Premise: Being a World Savvy evaluator requires a blend of competencies, knowledge, understandings, skills, and sensitivities. Blue Marble evaluators should be globally and cross-culturally competent as well as competent evaluators and knowledgeable about human and ecosystem interdependencies and stewardship. What competence means and what competencies are valued will vary by context.

Implications:

• Be thoughtful about what you know, don’t know, and need to know; what skills you have, don’t have, and need to development; and your strengths and weaknesses to engage as a Blue Marble evaluator.

• Be intentional and systematic about ongoing learning; have a learning agenda while also being open to emergent learning opportunities.

• Develop capabilities to engage as part of a Blue Marble team since no individual is likely to possess the full range of knowledge, skills, and competencies to engage globally, GLOCALY, and across silos in diverse contexts on varied initiatives and interventions.

• Be a creative methods bricoleur, astute in matching methods to situations.

• Be savvy about being World Savvy.

• Being World Savvy is a journey not a destination.

This principle was inspired by our work with World Savvy, an organization focused on empowering educators to make school inclusive, relevant, and engaging for all students, inspiring them to learn, work, and thrive as responsible global citizens.

Blue Marble Evaluation Operating Principle 6: Bricolage Methods Principle

Bricolage Methods Principle: Conduct utilization-focused evaluations incorporating Blue Marble principles to match methods to the evaluation situation.

Premise: The variety of possible Blue marble evaluation situations is so vast that no predetermined set will be adequate. There can be no Blue Marble methods toolbox, a popular metaphor for evaluators offering a limited and definitive set of “tools.” Context matters in designing evaluations. Intended purposes and uses matter, as does identifying and working with primary intended users. Standardization is anathema; customization and contextualization rule.

Implications:

• Blue Marble bricoleur teams will likely be needed to access a variety of possible methods, measures, analytical approaches, and methodological specializations.

• Newer technologies like Big Data, GIS, remote sensing, artificial intelligence (AI), social systems network mapping techniques, and blockchain innovations will likely have Blue Marble evaluation applications.

• Blue Marble evaluation designs may be emergent and adaptive given the diversity of worldwide situations and applications and the dynamic nature of the global environment.

• Qualitative, quantitative, and mixed methods will likely be needed to do justice to the complexity and multi-dimensional nature of Blue Marble evaluations.

• The GLOCAL principle means that Blue Marble analysis will include national data disaggregated within countries and global data aggregated and synthesized across countries.

• The time is of the essence principle guides evaluation bricoleurs toward longitudinal designs that can capture and assess trajectories toward sustainability and transformation over time.

• Blue Marble evaluations must be utilization-focused.

Photo credit Ellen Harasimowicz

Blue Marble Evaluation Operating Principle 5: Yinyang

Each week, leading up to the Blue Marble Evaluation book and website launch, we will be introducing a new principle of Blue Marble Evaluation. The first four weeks introduced the four overarching principles. This week we turn to the fifth of twelve operating principles. Click here to learn about the difference.

Principle: Harmonize conceptual opposites.

Premise: We live in a divided world. What is striking about the iconic Blue Marble photo from space is its wholeness. No nation-state divisions. No sector silos. No local-global boundaries. The image is neither long-term nor short-term, but now. Blue Marble evaluation aims for that wholeness of perspective as a guide to wholeness of understanding to inform holistic action. To achieve that sense of wholeness necessitates seeing and engaging with different perspectives, harmonizing opposites, integrating divisions, transcending boundaries, and overcoming polarities.

Implications:

• How you harmonize depends on circumstance, the nature of polarities you are encountering, the depth and degree of opposition, and myriad other factors.

• The yin-yang operating principle provides specific guidance for adhering to and applying the overarching Blue Marble Integration Principle: Integrate the Blue Marble principles in the design, engagement with, and evaluation of systems change and transformation initiatives.

• The yin-yang principle is a philosophical mindset not a procedural technique; it provides conceptual guidance for harmonizing opposites as appropriate and useful, but is not a rule that all opposites must be harmonized.

• Design and evaluation are not intrinsic opposites, but are typically treated as separate and sequential. The Blue marble perspective of wholeness and integration applies to design and evaluation.

Blue Marble Evaluation Operating Principle 4: Time Being of the Essence

Each week, leading up to the Blue Marble Evaluation book and website launch, we will be introducing a new principle of Blue Marble Evaluation. The first four weeks introduced the four overarching principles. This week we turn to the fourth of twelve operating principles. Click here to learn about the difference.

Principle: Act with a sense of urgency in the present, support adaptive sustainability long-term, grounding both in understanding the past.

Premise: Understanding the past provides a perspective on present realities and future possibilities. Future scenarios include both doomsday and utopian possibilities. Evaluative thinking applied to the present can be joined with futuristic thinking to illuminate forward-looking patterns and trajectories. Monitoring how interventions and initiatives unfold can inform adaptations along the way.

Implications:

The Blue Marble Evaluation approach to time reframes traditional evaluation thinking as follows:

1. beyond project and program time boundaries to a global ecological sustainability timeframe;

2. from a static (continuity) to a dynamic (resilience) approach to sustainability;

3. from short-term to long-term thinking; and

4. from past time (Holocene) complacency to real-time (Anthropocene) urgency.

Implications for design include:

1. Design interventions to accomplish shorter-term results while enhancing capacity for long-term resilient sustainability of the intervention through ongoing adaptation.

2. Design initiatives with attention to how larger systems dynamics, both human and environmental systems, will affect the development and adaptation of the intervention over time.

3. Design interventions to contribute to global planetary sustainability.

4. Include in the intervention design expected and evaluable milestones of resilient sustainability progress?

Evaluation Questions:

To what extent and in what ways does an initiative systemically address the relationship between shorter-term results and longer-term resilient sustainability?

How are larger system dynamics, both human and environmental, tracked to support initiative adaptations over time?

To what extent, in what ways, and within what timeframe does a particular initiative contribute to global planetary sustainability?

How and when are evaluation activities and reporting timed to ensure timeliness?

Blue Marble Evaluation Operating Principle 3: Cross Silos

Each week, leading up to the Blue Marble Evaluation book and website launch, we will be introducing a new principle of Blue Marble Evaluation. The first four weeks introduced the four overarching principles. This week we turn to the third of twelve operating principles. Click here to learn about the difference.

Principle: Engage across sectors and issues for systems change.

Basic Premise: Problems are embedded in systems. To target the problem without changing the system of which it is a part is to provide only a partial solution and one unlikely to endure. Moreover, problems are often intertwined. Interconnected problems within and across systems require systems change strategies to have lasting impact. Solving problems piecemeal leads to piecemeal solutions.

Implications:

• Changing systems is different from implementing projects. Evaluating systems change is different from evaluating projects and programs. Programs and projects are based on a linear logic of causality. Evaluation of programs and projects follows that linear thinking.

• Systems consist of interdependent elements interconnected in such a way that a change in one element changes connections with other elements and, reverberating through the set of system interconnections, may change the system. Tracking those changes for evaluation purposes requires mapping methods and ways of capturing changes in system interconnections and their dynamics.

• Systems thinking applies to situation analysis, intervention design, engagement, implementation, adaptations, and developmental evaluation.

• Cross-silos design and evaluation tackles multiple issues at once and likely requires a Blue marble team with diverse knowledge specializations and interdisciplinary capabilities.

Photo Credit: Leaflet [CC BY-SA 3.0 (https://creativecommons.org/licenses/by-sa/3.0)]

Blue Marble Evaluation Operating Principle 2: GLOCAL

Each week, leading up to the Blue Marble Evaluation book and website launch, we will be introducing a new principle of Blue Marble Evaluation. The first four weeks introduced the four overarching principles. This week we turn to the second of twelve operating principles. Click here to learn about the difference.

Principle: Integrate complex interconnections across levels.

Basic Premise: Global systems change must be contextually sensitive and grounded in the interactions between local and global processes and scales of change. The term that has emerged to capture this way of thinking is GLOCAL, or glocalization.

Implications: When designing an intervention or initiative, look at the interactions, interdependencies, and interconnections across levels (micro, meso, macro). Take into account how people, information, and resources flow from local to global, and global to local.

Evaluation Questions:

• In what ways is an initiative or intervention truly GLOCAL? In both processes and results?

• What are the interactions, interdependencies, and interconnections across levels? How do they intersect for mutually reinforcing systems change? Look for both anticipated and unanticipated interactions, both positive (mutually reinforcing) and negative (disjointed and nonaligned).

Photo Credit: Ellen Harasimowicz

Blue Marble Evaluation Operating Principle 1: Transboundary Engagement

Each week, leading up to the Blue Marble Evaluation book and website launch, we will be introducing a new principle of Blue Marble Evaluation. The first four weeks introduced the four overarching principles This week, we turn to the first of the twelve operating principles. Click here to learn about the difference.

Principle: Act at a global scale.

Basic Premise: To address global problems, look beyond nation-state borders and boundaries to affect transnational, regional, and global patterns, interactions, and dynamics.

Implications: The design, implementation and evaluation of a global initiative should be global. This does not mean operating in a few different countries, but operating across countries and taking into account the global and local contexts.

Suggested Evaluation Question: To what extent and in what ways is an initiative or intervention acting at a truly global scale.

Blue Marble Evaluation Principle 4: Overarching Integration

Each week, leading up to the Blue Marble Evaluation book and website launch, we will be introducing a new principle of Blue Marble Evaluation. We will start with the four overarching principles. This week, Principle 4: Overarching Integration.

Principle: Integrate the Blue Marble principles in the design, engagement with, and evaluation of systems change and transformation initiative.

Basic Premise: Transformation requires multiple interventions and actions on many fronts undertaken by diverse but interconnected actors.

Implications: This fourth principle integrates the previous three, making it clear that this is not a pick and-choose menu of options but rather an integrated and comprehensive approach in which all the principles are important and, together, constitute a complete package. As we proceed in subsequent weeks to introduce and explain the 12 operating principles, the Integration principle will apply to those as well.

Photo Credit: Ellen Harasimowicz

Blue Marble Evaluation Principle 3: Transformative Engagement

Each week, leading up to the Blue Marble Evaluation book and website launch, we will be introducing a new principle of Blue Marble Evaluation. We will start with the four overarching principles. This week, Principle 3: Transformative Engagement.

Principle: Engage consistent with the magnitude, direction, and speed of transformations needed and envisioned.

Basic Premise: Global, anthropogenic problems are so severe, threatening the future sustainability of the planet and humanity, that major and rapid systems transformations are needed.

Implications:

• Base transformational interventions on a research-informed theory of transformation knitting together relevant theories of change.

• Ensure that what is called transformation IS transformational.

• Catalyze, connect, track, map, and evaluate networks and initiatives worldwide to generate critical mass tipping points toward global transformation.

• Apply systems thinking and complexity theory to transformational engagements.

• Transform evaluation to evaluate transformation.

Blue Marble Evaluation Principle 1: Global Thinking

Each week, leading up to the Blue Marble Evaluation book and website launch, we will be introducing a new principle of Blue Marble Evaluation. We will start with the four overarching principles. This week, Principle 1: Global Thinking.

Principle: Apply whole Earth, big picture thinking to all aspects of systems change.

Basic premise: Global problems like climate change, worldwide pollution, and global disparities require global interventions and, correspondingly, globally-oriented and world savvy evaluators.

Implications:

• Whatever is done, or evaluated, at all levels and for all types of interventions and initiatives, consider its global context and implications both within and beyond nation-state boundaries.

• Think systemically. Conceptualize systems and evaluate systems changes, not just focus on projects and programs. Connect the local to the global, and the global to the local.

• Think across silos by examining how issues, problems, and specific interventions may be interconnected. Unpack and bring fidelity to initiatives, organizations, and projects calling themselves “global”. Working on one issue in three countries is not global. What are the various ways in which the designation “global” has meaning?

• Select appropriate methods for the situation and nature of the targeted systems changes.

• Time being of the essence, be attentive to varying time horizons by integrating short-term, medium-term, and long-term sustainability considerations while acting with a sense of urgency given climate change and related global trends.