Blue Marble Operating Principle 8: Skin in the Game

Principle: Acknowledge and act on your stake in how the Anthropocene unfolds.

Premise: When it comes to the survival of humanity and the planet, we all have skin in the game, we and our loved ones are in the world that is under threat. We are not outside looking in. We are part of the global system and, there’s a good chance that we are each, in our own way, part of the problem. This gives us a quite different stance than is typically expected. Evaluators are virtually always outside the programs or projects they evaluate. Acknowledging and facing the realities of the Anthropocene transforms the stance of evaluators from external observers of change to internal participants in change.

Implications:

  • A shift in evaluator stance from independence to interdependence.

  • Making the evaluators’ values explicit and transparent.

  • Identifying your stake, whatever role you play – evaluator, designer, implementer, funder, commissioner of evaluations, intended user, policy maker – then sharing how you view your stake and the implications of that view for how you engage and fulfill your role. As a utilization-focused evaluator, we always have a stake in whether and how an evaluation is used.

  • Reality-test for yourself to be valuable as a reality-tester for others. So, how good are you at reality-testing for yourself? For example, to what extent are you practicing in your life the things you know you ought to be doing (exercise, eating right, getting enough sleep…)?

Photo Credit: Ellen Harasimowicz

Fools’ Gold: The Widely Touted Methodological “Gold Standard” Is Neither Golden Nor a Standard

Fools’ Gold: The Widely Touted Methodological “Gold Standard” Is Neither Golden Nor a Standard

In response to yesterday’s announcement of the 2019 Nobel Prize in Economics being awarded for development RCTs, we offer this rumination from Michael Quinn Patton, originally published in his book Qualitative Evaluation & Research Methods, cautioning against promoting RCTs as the gold standard in evaluation.

Blue Marble Evaluation Operating Principle 7: World Savvy Principle

World Savvy Principle: Engage in ongoing learning relevant to Blue Marble principles and practices.

Premise: Being a World Savvy evaluator requires a blend of competencies, knowledge, understandings, skills, and sensitivities. Blue Marble evaluators should be globally and cross-culturally competent as well as competent evaluators and knowledgeable about human and ecosystem interdependencies and stewardship. What competence means and what competencies are valued will vary by context.

Implications:

  • Be thoughtful about what you know, don’t know, and need to know; what skills you have, don’t have, and need to development; and your strengths and weaknesses to engage as a Blue Marble evaluator.

  • Be intentional and systematic about ongoing learning; have a learning agenda while also being open to emergent learning opportunities.

  • Develop capabilities to engage as part of a Blue Marble team since no individual is likely to possess the full range of knowledge, skills, and competencies to engage globally, GLOCALY, and across silos in diverse contexts on varied initiatives and interventions.

  • Be a creative methods bricoleur, astute in matching methods to situations.

  • Be savvy about being World Savvy.

  • Being World Savvy is a journey not a destination.

This principle was inspired by our work with World Savvy, an organization focused on empowering educators to make school inclusive, relevant, and engaging for all students, inspiring them to learn, work, and thrive as responsible global citizens.

Blue Marble Evaluation Operating Principle 6: Bricolage Methods Principle

Bricolage Methods Principle: Conduct utilization-focused evaluations incorporating Blue Marble principles to match methods to the evaluation situation.

Premise: The variety of possible Blue marble evaluation situations is so vast that no predetermined set will be adequate. There can be no Blue Marble methods toolbox, a popular metaphor for evaluators offering a limited and definitive set of “tools.” Context matters in designing evaluations. Intended purposes and uses matter, as does identifying and working with primary intended users. Standardization is anathema; customization and contextualization rule.

Implications:

  • Blue Marble bricoleur teams will likely be needed to access a variety of possible methods, measures, analytical approaches, and methodological specializations.

  • Newer technologies like Big Data, GIS, remote sensing, artificial intelligence (AI), social systems network mapping techniques, and blockchain innovations will likely have Blue Marble evaluation applications.

  • Blue Marble evaluation designs may be emergent and adaptive given the diversity of worldwide situations and applications and the dynamic nature of the global environment.

  • Qualitative, quantitative, and mixed methods will likely be needed to do justice to the complexity and multi-dimensional nature of Blue Marble evaluations.

  • The GLOCAL principle means that Blue Marble analysis will include national data disaggregated within countries and global data aggregated and synthesized across countries.

  • The time is of the essence principle guides evaluation bricoleurs toward longitudinal designs that can capture and assess trajectories toward sustainability and transformation over time.

  • Blue Marble evaluations must be utilization-focused.

Photo credit Ellen Harasimowicz

Blue Marble Evaluation Operating Principle 5: Yinyang

Each week, leading up to the Blue Marble Evaluation book and website launch, we will be introducing a new principle of Blue Marble Evaluation. The first four weeks introduced the four overarching principles. This week we turn to the fifth of twelve operating principles. Click here to learn about the difference.

Principle: Harmonize conceptual opposites.

Premise: We live in a divided world. What is striking about the iconic Blue Marble photo from space is its wholeness. No nation-state divisions. No sector silos. No local-global boundaries. The image is neither long-term nor short-term, but now. Blue Marble evaluation aims for that wholeness of perspective as a guide to wholeness of understanding to inform holistic action. To achieve that sense of wholeness necessitates seeing and engaging with different perspectives, harmonizing opposites, integrating divisions, transcending boundaries, and overcoming polarities.

Implications:

  • How you harmonize depends on circumstance, the nature of polarities you are encountering, the depth and degree of opposition, and myriad other factors.

  • The yin-yang operating principle provides specific guidance for adhering to and applying the overarching Blue Marble Integration Principle: Integrate the Blue Marble principles in the design, engagement with, and evaluation of systems change and transformation initiatives.

  • The yin-yang principle is a philosophical mindset not a procedural technique; it provides conceptual guidance for harmonizing opposites as appropriate and useful, but is not a rule that all opposites must be harmonized.

  • Design and evaluation are not intrinsic opposites, but are typically treated as separate and sequential. The Blue marble perspective of wholeness and integration applies to design and evaluation.

Blue Marble Evaluation Operating Principle 4: Time Being of the Essence

Each week, leading up to the Blue Marble Evaluation book and website launch, we will be introducing a new principle of Blue Marble Evaluation. The first four weeks introduced the four overarching principles. This week we turn to the fourth of twelve operating principles. Click here to learn about the difference.

Principle: Act with a sense of urgency in the present, support adaptive sustainability long-term, grounding both in understanding the past.

Premise: Understanding the past provides a perspective on present realities and future possibilities. Future scenarios include both doomsday and utopian possibilities. Evaluative thinking applied to the present can be joined with futuristic thinking to illuminate forward-looking patterns and trajectories. Monitoring how interventions and initiatives unfold can inform adaptations along the way.

Implications:

The Blue Marble Evaluation approach to time reframes traditional evaluation thinking as follows:

  1. beyond project and program time boundaries to a global ecological sustainability timeframe;

  2. from a static (continuity) to a dynamic (resilience) approach to sustainability;

  3. from short-term to long-term thinking; and

  4. from past time (Holocene) complacency to real-time (Anthropocene) urgency.

Implications for design include:

  1. Design interventions to accomplish shorter-term results while enhancing capacity for long-term resilient sustainability of the intervention through ongoing adaptation.

  2. Design initiatives with attention to how larger systems dynamics, both human and environmental systems, will affect the development and adaptation of the intervention over time.

  3. Design interventions to contribute to global planetary sustainability.

  4. Include in the intervention design expected and evaluable milestones of resilient sustainability progress?

Evaluation Questions:

To what extent and in what ways does an initiative systemically address the relationship between shorter-term results and longer-term resilient sustainability?

How are larger system dynamics, both human and environmental, tracked to support initiative adaptations over time?

To what extent, in what ways, and within what timeframe does a particular initiative contribute to global planetary sustainability?

How and when are evaluation activities and reporting timed to ensure timeliness?

Blue Marble Evaluation Operating Principle 3: Cross Silos

Each week, leading up to the Blue Marble Evaluation book and website launch, we will be introducing a new principle of Blue Marble Evaluation. The first four weeks introduced the four overarching principles. This week we turn to the third of twelve operating principles. Click here to learn about the difference.

Principle: Engage across sectors and issues for systems change.

Basic Premise: Problems are embedded in systems. To target the problem without changing the system of which it is a part is to provide only a partial solution and one unlikely to endure. Moreover, problems are often intertwined. Interconnected problems within and across systems require systems change strategies to have lasting impact. Solving problems piecemeal leads to piecemeal solutions.

Implications:

  • Changing systems is different from implementing projects. Evaluating systems change is different from evaluating projects and programs. Programs and projects are based on a linear logic of causality. Evaluation of programs and projects follows that linear thinking.

  • Systems consist of interdependent elements interconnected in such a way that a change in one element changes connections with other elements and, reverberating through the set of system interconnections, may change the system. Tracking those changes for evaluation purposes requires mapping methods and ways of capturing changes in system interconnections and their dynamics.

  • Systems thinking applies to situation analysis, intervention design, engagement, implementation, adaptations, and developmental evaluation.

  • Cross-silos design and evaluation tackles multiple issues at once and likely requires a Blue marble team with diverse knowledge specializations and interdisciplinary capabilities.


Photo Credit: Leaflet [CC BY-SA 3.0 (https://creativecommons.org/licenses/by-sa/3.0)]

Contribute to the next edition of Utilization-Focused Evaluation

We are writing a 5th edition of Utilization-Focused Evaluation (U-FE) and seek your input. We welcome any of the following sent directly to us:

  1. 1A U-FE example to share.

  2. A question or questions about U-FE you'd like to have answered.

  3. A U-FE experience you'd be willing to share (positive or negative)

  4. A critique, cautionary tale, or concern you have about U-FE

  5. Anything at all you'd like us to consider for the revision

E-mail your responses by September 15th to Charmagne.

History of U-FE:

  • 1st edition, 1978

  • 2nd ed., 1986

  • 3rd ed., 1997.

  • 4th ed., 2008

  • Essentials of U-FE, 2012

  • 5th ed., forthcoming 2020

Blue Marble Evaluation Operating Principle 2: GLOCAL

Each week, leading up to the Blue Marble Evaluation book and website launch, we will be introducing a new principle of Blue Marble Evaluation. The first four weeks introduced the four overarching principles. This week we turn to the second of twelve operating principles. Click here to learn about the difference.

Principle: Integrate complex interconnections across levels.

Basic Premise: Global systems change must be contextually sensitive and grounded in the interactions between local and global processes and scales of change. The term that has emerged to capture this way of thinking is GLOCAL, or glocalization.

Implications: When designing an intervention or initiative, look at the interactions, interdependencies, and interconnections across levels (micro, meso, macro). Take into account how people, information, and resources flow from local to global, and global to local.

Evaluation Questions:

  • In what ways is an initiative or intervention truly GLOCAL? In both processes and results?

  • What are the interactions, interdependencies, and interconnections across levels? How do they intersect for mutually reinforcing systems change? Look for both anticipated and unanticipated interactions, both positive (mutually reinforcing) and negative (disjointed and nonaligned).

Photo Credit: Ellen Harasimowicz

Blue Marble Evaluation Operating Principle 1: Transboundary Engagement

Each week, leading up to the Blue Marble Evaluation book and website launch, we will be introducing a new principle of Blue Marble Evaluation. The first four weeks introduced the four overarching principles This week, we turn to the first of the twelve operating principles. Click here to learn about the difference.

Principle: Act at a global scale.

Basic Premise: To address global problems, look beyond nation-state borders and boundaries to affect transnational, regional, and global patterns, interactions, and dynamics.

Implications: The design, implementation and evaluation of a global initiative should be global. This does not mean operating in a few different countries, but operating across countries and taking into account the global and local contexts.

Suggested Evaluation Question: To what extent and in what ways is an initiative or intervention acting at a truly global scale.

Blue Marble Evaluation Principle 4: Overarching Integration

Each week, leading up to the Blue Marble Evaluation book and website launch, we will be introducing a new principle of Blue Marble Evaluation. We will start with the four overarching principles. This week, Principle 4: Overarching Integration.

Principle: Integrate the Blue Marble principles in the design, engagement with, and evaluation of systems change and transformation initiative.

Basic Premise: Transformation requires multiple interventions and actions on many fronts undertaken by diverse but interconnected actors.

Implications: This fourth principle integrates the previous three, making it clear that this is not a pick and-choose menu of options but rather an integrated and comprehensive approach in which all the principles are important and, together, constitute a complete package. As we proceed in subsequent weeks to introduce and explain the 12 operating principles, the Integration principle will apply to those as well.

Photo Credit: Ellen Harasimowicz

Blue Marble Evaluation Principle 3: Transformative Engagement

Each week, leading up to the Blue Marble Evaluation book and website launch, we will be introducing a new principle of Blue Marble Evaluation. We will start with the four overarching principles. This week, Principle 3: Transformative Engagement.

Principle: Engage consistent with the magnitude, direction, and speed of transformations needed and envisioned.

Basic Premise: Global, anthropogenic problems are so severe, threatening the future sustainability of the planet and humanity, that major and rapid systems transformations are needed.

Implications:

  • Base transformational interventions on a research-informed theory of transformation knitting together relevant theories of change.

  • Ensure that what is called transformation IS transformational.

  • Catalyze, connect, track, map, and evaluate networks and initiatives worldwide to generate critical mass tipping points toward global transformation.

  • Apply systems thinking and complexity theory to transformational engagements.

  • Transform evaluation to evaluate transformation.

Blue Marble Evaluation Principle 1: Global Thinking

Each week, leading up to the Blue Marble Evaluation book and website launch, we will be introducing a new principle of Blue Marble Evaluation. We will start with the four overarching principles. This week, Principle 1: Global Thinking.

Principle: Apply whole Earth, big picture thinking to all aspects of systems change.

Basic premise: Global problems like climate change, worldwide pollution, and global disparities require global interventions and, correspondingly, globally-oriented and world savvy evaluators.

Implications:

  • Whatever is done, or evaluated, at all levels and for all types of interventions and initiatives, consider its global context and implications both within and beyond nation-state boundaries.

  • Think systemically. Conceptualize systems and evaluate systems changes, not just focus on projects and programs. Connect the local to the global, and the global to the local.

  • Think across silos by examining how issues, problems, and specific interventions may be interconnected. Unpack and bring fidelity to initiatives, organizations, and projects calling themselves “global”. Working on one issue in three countries is not global. What are the various ways in which the designation “global” has meaning?

  • Select appropriate methods for the situation and nature of the targeted systems changes.

  • Time being of the essence, be attentive to varying time horizons by integrating short-term, medium-term, and long-term sustainability considerations while acting with a sense of urgency given climate change and related global trends.

Join the Blue Marble Evaluation Network

The emerging Blue Marble Evaluation Network invites you to join us to learn more about how to evaluate large scale systems change and transformation and to build a community of practice so we can collectively contribute to its ongoing development. We have created a survey to help grow our network and invite you to complete and share it! Click here to access the survey.

By completing this survey, you will be joining this global network at its earliest stages, informed of events such as webinars and training opportunities and invited to participate in a wide range of activities. This invitation is being sent to over 400 leaders in the field of evaluation, large scale systems change and transformation. The name "Blue Marble" refers to the iconic image of the Earth as seen from space. Blue Marble Evaluation, then, treats the whole Earth as its focus, people and planet. This means evaluating the well-being of humanity and the health and sustainability of our the Earth globally, holistically, and systemically.

The practice is rooted in three basic principles: rigorously apply Blue Marble thinking to all aspects of global systems change design, implementation, and evaluation; embed understanding of global sustainability in all systems change initiatives and their evaluation; and, ensure that what is called transformational is transformational. The first principle addresses how to conceptualize evaluation at a global level. The second principle asserts that there are core competencies regarding the evaluation of large scale systems change and transformation. The third principle addresses the scope and scale of systems change needed for global sustainability and the role of evaluators in contributing to the transformations needed. These three principles encompass the thinking, knowing, and doing of Blue Marble Evaluation. Your participation in the global network will help to transform the field of evaluation and apply, further develop, and evolve these core principles in both theory and practice.

The development of Blue Marble Evaluation has benefited from the contributions, insights and experiences of many individuals and will require a far wider global network to better "see" together the challenges and opportunities in transforming evaluation to better contribute to the wellbeing of people and planet. Networks such as the SDG Transformations Forum and the Global Alliance for the Future of Food are serving as early adopters of this concept and have contributed significantly to the development of Blue Marble Evaluation. This global community of practice will serve to spotlight and help accelerate those efforts and contribute to wider uptake of this rapidly evolving theory and practice.

50th Anniversary of the Earthrise Shot

December 24 is the 50th anniversary of the first photo of Earthrise from the moon. Apollo 8, the first manned mission to the moon, entered lunar orbit on Christmas Eve, Dec. 24, 1968. That evening, the astronauts -- Commander Frank Borman, Command Module Pilot Jim Lovell, and Lunar Module Pilot William Anders -- held a live broadcast from lunar orbit, in which they showed pictures of the Earth and moon as seen from their spacecraft. One of those photos showed Earthrise for the first time.

On this celebratory occasion, we are doubling down on the idea of Blue Marble Evaluation, an approach and framework for evaluating global systems change initiatives. Global challenges like climate change, global economic interdependence, and the global food system operate beyond national borders. Global systems change initiatives are intervening from the perspective of a complex, dynamic, and interconnected world system. Blue Marble evaluation is needed to assess the effectiveness of global systems change initiatives. If you are interested in becoming part of the Blue Marble Evaluation Network, please take a few moments to complete a brief survey to help us grow this important work.

Click here to take the Blue Marble Evaluation survey.

Reflections from IPDET 2016

Michael and I recently spent two days with twenty international development practitioners and evaluators at IPDET in Ottawa. They were pioneers in our first two-day workshop on evaluating global systems change, the result of our six-month grant from Faster Forward Fund in support of Blue Marble Evaluators. We started our journey with this video, documenting human migration over time, followed by an icebreaker to get to know one another and share our own migration stories. Participants located themselves in the room based on where they were born and had to introduce themselves without saying the name of the country. Here are some photos and a video of the exercise:

 

Blue Marble Evaluators describing the place where they were born without using country names. Camera is standing in North America, looking SE towards Australia. 

Blue Marble Evaluators describing the place where they were born without using country names. Camera is standing in North America, looking SE towards Australia. 

Blue Marble Evaluators where they current live or work.

Blue Marble Evaluators where they current live or work.

With this exercise, we were challenging the primacy of national borders, which we believe often get in the way of systems change. In the words of Albert Einstein, "you can never solve a problem on the level on which it was created." Next we watched a video of the changing borders of Europe to drive home the idea that national borders are artificial and the result of war, colonialism. We also watched the DNA Journey, a powerful video by Momondo, which shows what our DNA reveals about where we come from. All of this was intended to re-frame our way of thinking from a global perspective. We ended this introduction with the Blue Marble Shot, the first complete photograph of the Earth taken from space in 1972. Each participant also receive a blue marble as a reminder to take the blue marble perspective into their work.

After this introduction and re-framing, Michael walked through the evolution of evaluation into an international field, culminating with 2015 as the International Year of Evaluation. While the focus on building evaluation at the country-level is important, we believe that the next frontier for evaluation is for exaluators to apply a global perspective to their work and build our capacity to evaluation global systems change efforts. At its core, evaluating global systems change involves looking across both sector and national boundaries. 

Over the course of two days, we explored what this might look like through case examples and exercises mapping global issues, diving into systems thinking and complexity concepts, and working with a real-world example from the Global Centre for Pluralism.  Check out the gallery below for some photos of this work.

There are many challenges to evaluating global systems change, but this workshop reinforced our belief that this approach is lacking and necessary, especially in the field of international development. We hope to increase both demand for Blue Marble Evaluators and the supply/availability of them. On one hand, there are already many global initiatives operating, but few have evaluators at the table to inform decision-making and encourage the use of systems thinking and complexity concepts in their evaluation approach. On the other hand, there are even more development projects being undertaken without a global perspective despite the fact that the issues they are addressing are undeniably global. We believe Blue Marble Evaluators can play a role in helping to re-frame these projects and design evlauations that capture the complexity and system-wide implications of this important work. 

Ultimately, what came out of this workshop was the recognition that evaluating global systems change requires Blue Marble Evaluators who understand the historical forces that have shaped the current world system, are committed to developing their cultural competence in order to effectively collaborate and communicate with diverse populations, and bring a unique combination of empathy, humility and the capacity and willingness to speak truth to power in highly politically-charged situations. The first cohort of IPDET Blue Marble Evaluators are tackling a range of complex issues from land rights in the Mekong delta to migration in West Africa, access to anti-retroviral drugs to human security. We hope this workshops laid the foundation for these pioneers to bring a blue marble perspective to their work. 

By Charmagne Campbell-Patton

MQP's Top Ten List of the Future of Evaluation

One June 8th, I gave the closing keynote address at the Canadian Evaluation Society's annual conference in Saint John's, Newfoundland, where I listed my top ten trends for the future of evaluation. They've since gotten some traction on twitter, but a few are out of order, so I thought I'd share my official list here for all to see: 

10. Integration of mixed methods

9. Increasing importance of data visualization

8. Increased us of social media for all aspects of evaluation

7. Learning from failure

6. Getting serious about unanticipated consequences and investigating unknown unknowns

5. Evaluation as the alpha trans-discipline (Scriven)

4. Integrated, holistic multi-disciplinary complexity-informed systems thinking

3. Real-time evaluation for a fast-paced world

2. Speaking truth to power

1. Global systems change evaluation

Click here to download the slides of my full keynote, "The Future of Evaluation: Beyond here there be dragons, or are those just icebergs?"