Artificial Intelligence to create a Theory of Change?

A complete theory of change, within a few minutes?

Click to try our AI Theory of Change

Could artificial intelligence (AI) create a comprehensive theory of change (ToC) – the logic how an innovation creates positive and negative social, environmental and economic impact – in minutes? Including a consideration of short, medium and long-term outcomes, customized and relevant metrics, matching the outcomes with the appropriate Sustainable Development Goals (SDGs), the 169 targets and 247 country level indicators and ESG categorization? And if so, how would this relate to the traditional non AI ToC development? We first extract key findings from 500+ startups who we helped develop their ToC without AI. We then make predictions how AI will change ToC development, based on our findings from our newly, carefully designed AI ToC software (beta).

5 key takeaways from our work with 500+ startups without using AI

Over the past years we have worked with 500+ startups at Massachusetts Institute of Technology (2017-today), Harvard University (2019-today) as well accelerators such as Yunus & Youth (2018-today) and German Science Park WISTA Management GmbH (2023-today). The underlying commonality was that all ventures had some level of sustainability impact in their innovations, activities, products or business models, which we would see as a prerequisite for crafting a ToC. We share 5 key take aways.

1. "Theory of Change is a key tool to identify the sustainability value of a startup, product or innovation"

Looking at best practices in measuring innovation impact from startups that have sustainability value in their products and/or business model, we see ToC as the most widely used and most appropriate tool. ToC is taught at almost all venture acceleration programs in the Harvard and MIT ecosystem, and as far as we experienced, also in the broader "sustainability impact management" community worldwide. Some key benefits that might explain its popularity:

• Holistic, systems approach: All sustainability outcomes can be mapped with a systems perspective, allowing to analyze positive and negative interaction effects, as well as unintended consequences. This is relevant, as outcome on the social impact dimension such e.g. food security, can have negative environmental effects on e.g. water consumption, use of pesticides etc. Silo approaches considering only one dimension, or disregard systems and interdependencies are flawed and dangerous.

  
  

• Stakeholder centric approach: ToC considers the actual outcomes for beneficiaries, i.e. the change that occurs on stakeholder level, reducing the risks of false assumptions or potentially irrelevant metrics, such as the number of stakeholders reached, without knowing what type of outcome occurs. Stakeholders from environmental outcomes can be individuals, or, in the case of e.g. carbon emissions/climate change "the global community / planet".

Other dominant tools such as life cycle assessment or randomized control trials can be used to verify assumptions and provide evidence for metrics identified in a ToC, but are often too expensive before the stage of series A, or series B investment rounds and once a business model, innovation or product is more stable and financial resources become available. ESG approaches with the valuable "materiality analysis" help to identify and manage all sustainability risks and should complement a ToC. We will discuss in a future article how they complement each other and why we recommend to either integrate them, or focus first on ToC and sustainability impact management and second on ESG sustainability risk management.

2. "There is a disconnect between a theory of change and sustainability impact management"

Whereas an increasing number of startups undergo training in crafting a solid theory of change, this does typically not transform into actual management of the relevant key metrics. On the positive end, going through the exercise clearly influences their decisions at one point in time, but the relevant metrics are often not tracked and as a result key sustainability outcomes are simply "forgotten". Tracking a system with all interdependencies and additionally fast changing business models is overwhelming and time consuming. As a result, many ventures have a theory of change, but it does not translate into actual monitoring of progress and thus validation of the assumptions underlying their ToC.

• Time and resource constraints: Early-stage ventures often lack the bandwidth for in-depth ToC monitoring, prioritizing action over structured learning tied to their ToC outcome metrics and monitoring.

• Complexity: Monitoring complex cause-and-effect relationships, especially in projects with long-term or indirect outcomes is in most cases out of scope for startups.

3. "(Most) startups cannot measure complex systems. But they can focus on the most important impact outcomes from within their system"

During the work of our CEO Ingo Michelfelder as postdoctoral fellow at the MIT Sloan Sustainability Initiative (2 years) and the Social Innovation + Change Initiative at Harvard Kennedy School (3 years), careful iteration with over 500 startups led to the recommendation that most startups can communicate their impact using their 3-5 most important positive key impact areas and 2-3 most important negative impact areas. Applying the logic of the pareto principle that roughly 80% of consequences come from 20% of causes, identifying the relevant 20% of their actions and tracking their outcomes is key. Whenever possible, a complex systems map with comprehensive consideration of cause and effects and their interactions should be built. But in order to move from this complex systems map to measurement, we recommend to zoom into this system and identify the most relevant actions/innovations. As mentioned above, our experience shows the top 3-5 positive outcomes and 1-3 negative outcomes is the right level of granularity.

• Focus and prioritization is key: Identifying the top 3-5 most important positive impact areas and 2-3 most important negative impact areas is the right level of granularity up to series A investment (after which more, less important metrics can be added to also cover the remaining 20% of relevant outcomes).

• Linear relationships are an acceptable simplification: A traditional ToC contains typically many interdependencies between variables. In early stages, if the most relevant and most important variables are identified, linear relationships can be an acceptable simplification

4. "Managing sustainability via target setting changes actions"

Whenever we came to the level where startups had identified their top 3-5 positive and negative impact areas, relevant indicators AND set targets (typically yearly, sometimes monthly), we frequently saw a change in actions and activities. It is surprising how even the most impact oriented innovators forget about their sustainability targets - but monitoring reminds them about ways to create sustainability outcomes. There is a very clear value to go beyond the initial ToC crafting to actual monitoring of progress. This is why the disconnect in crafting a ToC and actually using it for management purposes is bad news.

• What get's measured get's managed: This claim might be somewhat obvious and overused. But looking at the data of over 500 ventures with innovators with intrinsic level of sustainability this is simply true - those who measure, change their actions to create significantly higher sustainability value.

• Measure as early as possible, refine over time: There is certainly risk involved if a venture starts to rely on poor data. Our experience is that refinement can be achieved over time, and not measuring and managing results in an even higher risk of taking the wrong decisions.

  
  
  
  

5. "Tracking sustainability outcomes needs to be a requirement or strongly incentivized for early stage ventures - or there will be no monitoring"

The founders and management teams we worked with clearly had a strong intrinsic motivation to create sustainability outcomes with their innovations. Yet, once our founding teams left their acceleration programs, they typically did not implement any learning oriented sustainability management. Typical responses are "we prioritize creating impact through our actions, rather then spending time to manage it", or "we know we have a positive impact, so why should we spend time to provide a proof for something that is obvious"? Our analyses showed however, tremendous potential was lost - primarily by taking actions that either were not aligned with their previously identified sustainability metrics, or by loosing focus and not being able to focus and prioritize on those actions with the highest impact potential. A simple prioritization of key topics would have solved this problem. For this reason we see two needs.

• Policies or investors need to incentivize or make sustainability management a requirement from early stages onwards: In Europe, new policies such as the "Sustainable Finance Disclosure Regulation" (SFDR) for investors and "Corporate Sustainability Reporting Directive" (CSRD) for companies make sustainability reporting more mainstream. But due to thresholds, this is not the case for early stage impact innovators. During times when the business model and products are being created - an understanding of sustainability impact should be built.

• AI and automation greatly reduce the burden - customization is key. Without solutions that automate large parts of the burden, sustainability management will not take place for startups, at least not before series A or B investment rounds, when most products and business models are too stable to be changed easily. Standardized metrics will not be the response. What is needed is highly customized, flexible management systems. And AI is able to take out the complexity of these processes and tasks.

  
  

AI Theory of Change - the perfect response to takeaways 1 - 5 above

Can artificial intelligence complement or outperform traditional theory of change development? In our first few tests of our AI ToC, we asked our AI model to create a ToC for a client for whom we had for many months problems to build a quality ToC "manually". We were stunned to see that our AI ToC crafted a excellent ToC that met all our quality evaluation criteria - the AI ToC beta model we had built outperformed our human expertise. 

"AI ToC should complement every manually developed ToC"

With our current AI ToC beta version, we estimate that in about 90% of the cases, using our AI ToC improves manually generated ToCs. In most cases users found at least some better formulated indicators, more clearly formulated impact areas or even in many cases impact areas the teams had not thought about before (but that the teams decided to include). However, if we start directly with our AI ToC beta without crafting first a manual ToC, we estimate that in 90% of all cases, expert knowledge how to craft a high quality ToC and relevant sustainability domain expertise will increase the quality of the AI ToC. Some sustainability outcomes suggested by AI may not fit perfectly to the innovation. In about 5% of cases our AI ToC beta does a poor job or it fails, primarily due to too little input for AI to work with. This means human expert knowledge and training on crafting a good ToC is still needed. Though we see already improvement levers of our AI ToC beta that will help to leverage the capability of our AI model further - replacing some more elements that currently need to be performed manually.

"AI ToC will increase the number of startups using a ToC in their innovation process"

The most dramatic effect of our AI ToC beta is the time saving and efficiency aspect. Our initial intention was to add our AI ToC to complement the manually crafted ToC. But our current learning (though we need to test and iterate the AI ToC beta version) is that the quality of the outcomes will be so good and time savings so important, that we now recommend starting a ToC with AI - and refining it afterwards manually. This aligns well with a recent study, suggesting that using AI to generate academic research ideas, that are then sorted by experts, will outperform ideas generated by humans without AI.

Although AI may lie and produce false claims, it is capable of performing highly complex (such as complex interdependencies relevant for an innovation, even idea generation) as well as repetitive, simple tasks like assigning appropriate SDGs, targets and indicators (SDG indicators are typically on country level on not appropriate for progress monitoring for individual ventures). With increasing quality of our AI ToC model and surely similar models that will emerge, we expect that many more startups and innovators can use it, and the "clients per expert" ratio can be increased, allowing coaches to coach more innovators.

"AI ToC will help increase early stage innovators move from one time ToC generation to continuous sustainability management and learning".

Our AI ToC beta results in extremely customized outcome areas and indicators that can be easily and continuously adjusted over time. This is good news, as we identified a theory of change as the most appropriate tool for innovations with some level of intended sustainability impact - but one key criteria is the need to adapt each metrics system to the specific innovation. Highly flexible and adapted, thus meaningful metrics systems, combined with the time savings will allow innovators move from a "one time ToC development", to continuous learning and monitoring. For this any AI ToC needs to be closely linked with a sustainability monitoring system, which we also provide at ImpactForesight. The time savings and appropriate measurement systems can also lower the hesitation of investors and policy makers to make sustainability management a requirement in the journey to making sustainability tracking and learning a management tool suitable right from idea generation to growth of the innovations.

What are the risks and unintended consequences of our AI ToC beta?

There are certainly risks, such as loosing the actual learnings resulting from crafting a ToC manually, innovators showing beautifully crafted ToCs to the public, that are auto generated but may never influence innovators decision making, relying on false or irrelevant outcomes, or skipping stakeholder validation of the assumptions, not at last the negative environmental footprint of running our AI applications. We have also not yet implemented exclusion criteria for clearly non-sustainable businesses models or products (although - if implemented - the positive outcomes suggested by our AI even for these companies will still make sense). But as with any new AI application, thoughtful use and adapting the tools to match each individual’s needs, we are sure that AI will improve sustainability impact management.

Click to try our AI Theory of Change

1. We appreciate feedback.

2. If you share our article, we will increase our own impact

3. Follow us if you want to receive updates and learnings from our AI ToC (beta) and beyond.