Generative AI is reshaping the landscape of design, ushering us into an era where deeply individualized user experiences become the norm. Personalization reaches beyond the tailored, transcending into the realm of the deeply individualized, fostering one-on-one connections at scale — true hyper-personalization. I’m not calling for the abandonment of established design principles but for their evolution, to harness the potential of GenAI in a human-centric way.

My explorations into user experiences specific for generative AI culminated in the Generative AI Design Compass, a framework born out of practical insights into anchoring GenAI within human-centric design. It’s a tool that reflects my journey to understand how to anchor this technology within the right human context, ensuring that it enriches, not complicates, our interaction with the digital world.

I believe in a world where machines can assist and augment our abilities seamlessly, not replace them.

This framework emerges from a need for a more targeted approach, steering clear from yet complimenting the paths of traditional design thinking. Grounding in ethical mindfulness, user empathy, and a spirit of innovation. With the compass in hand, we are equipped to chart a course through the dynamic terrain of GenAI, where the user remains at the core of every experience we develop.

Consider your most recent interaction with AI — was it distinguishable, a chatbot interface, or was it a more nuanced presence, silently orchestrating your digital experience?

Often, generative AI’s efficacy is gauged by its subtlety, its ability to integrate so seamlessly into our lives that its absence becomes the only indicator of its previous engagement. To me it’s this experience that feels seamless that is where the true value lies.

The compass is more than a guide; it’s a distillation of real-world insights and success stories, a narrative of patterns and approaches that resonate across industries following on my earlier work on architectural patterns for developers. It’s a blueprint for designers who aspire to blend generative AI into their work harmoniously, ensuring that each interaction feels intuitive, meaningful, and distinctly human.

As we delve deeper into each dimension of the compass, we’ll unpack the mechanics behind the framework, highlighting practical strategies and examples that bring these concepts to life for product designers, managers, and all those who shape the user experiences of tomorrow.

Table of Contents

1. Introducing the Framework
2. Discovery Dime: Engaging the Curious Minds
3. Assisting Dimension: Navigating the User Path
4. Exploration Dimension: Cultivating Creativity
5. Refinement Dimension: Crafting Precision
6. Trust Dimension: Ensuring Confidence
7. Mastery Dimension: Deeper Integration
8. Key Takeaways and Future Considerations

Unveiling the Framework

The journey through the Generative AI Design Compass is a methodical one, structured to balance the innovative momentum of generative AI with the foundations of intuitive, ethical, and empowering design. Here, we introduce six pivotal dimensions that anchor our interaction with GenAI, each encapsulating a phase from initial curiosity to the mastery of generative AI applications.

The framework’s dimensions, while interconnected, provide designers with the flexibility to engage with GenAI with any of these dimension independently, ensuring relevance to a variety of products and services. Not all dimensions will apply to all experiences, products and services and nor should they.

1: Discovery Dimension — Engaging Curious Minds

Initiate the GenAI experience with an invitation to curiosity, presenting the AI’s potential through relatable and simple interactions that encourage further exploration. This Discovery dimension employs the Iceberg Principle, presenting GenAI as a gateway to new possibilities without daunting users with its underlying complexity. The art here lies in capturing interest and fostering comprehension without delving into the depths of technicalities that GenAI encompasses.

Color Scheme & Iconography

GenAI is introduced through a visual language that captivates and demystifies, making its presence both noticeable and seamless within the user environment, using color schemes and symbols that make it stand out yet feel inherently part of the user environment. It’s about making GenAI approachable and recognizable at first glance.

Companies are using color and iconography as a signal to users that something is AI vs non-AI or human — Technology implementations following this pattern can also help users understand where AI is in use.

Besides colors use of certain emojis have been taken on by brands. 🤖 Robots depicting agents and bots, ✨ + 🪄Magic depicting AI and 🔮 Crystal orbs as another depiction of AI such as OpenAI and Siri

Name & Branding

By assigning a distinctive name or branding to GenAI features, this strategy aims to personalize and distinguish GenAI — it’s about creating an identity that users can connect with.

It’s the difference between calling an AI-powered design tool “Magic Design Assistant” versus a sterile “Design Tool 5.0”. The former, much like Canva’s Magic Studio, promises an experience; the latter, merely a function. We are starting to see the proliferation of “assistants” and “co-pilots” as common terminology to AI services.

Magic Moments

Introducing GenAI’s capabilities through “Magic Moments” — those instances of unexpected delight that showcase the power of AI in solving user problems.

Grammarly’s real-time writing assistance that not only corrects but educates going beyond expectations. Arc a new AI powered browser uses a number of intuitive methods along a users existing journey to introduce AI summary.

Consideration — Realistic Expectations

The challenge is to inspire without overwhelming. The introduction should be a careful blend of engagement and education, setting the stage for deeper interaction without creating false expectations. Just as Miro’s ‘Assist’ feature guides without overpowering, our aim is to foster intrigue and understanding in equal measure, ensuring that the allure of GenAI translates into a lasting relationship with the user. Focus on the user problem and how GenAI can help simplify this for the user.

2: Assisting Dimension — Navigating the User Path

In the Assisting dimension, we carefully calibrate the guidance and autonomy offered to users, ensuring they feel supported yet free to explore GenAI’s capabilities. We empower users to independently explore GenAI’s potential, creating a dynamic that shifts from structured guidance to autonomous discovery. It’s a space where we nurture user independence while offering the scaffolding necessary for them to navigate the potential of GenAI confidently.

Cue’s

Subtle cues like nudges and suggestions are our compass needles here, pointing users toward deeper engagement with GenAI. These gentle suggestions are critical when your tool can do a zillion things with AI we can give intuitive cues like a guide that's ready to help users to these new features.

These can come as simple magic AI feature buttons to help complete a task like writing your LinkedIn post, or in a contextual point such as an error when you are writing your code as an engineer in the Hex editor example to magic fix your issue.

Templates & Examples

Structured templates and intutive examples can demystify GenAI, inviting users to interact and customize without requiring them to first become experts. These templates are like open doors to GenAI’s possibilities, allowing users to walk through and explore at their own pace.

For instance Google’s gemini starts users off with some example prompts, where as we have more in-depth templates from Jasper’s AI content writing tool guiding user towards a personalized creation without removing self-discovery.

Consideration — Ensure We Are Balancing Needs

We need to calibrate the balance between guiding and liberating our users. We must address their pain points with the right mix of assistance. The needs will change dependent on context and needs, such as elderly care AI focuses on simplicity and clear communication, while AI for legal professionals might prioritize depth and customization options.

As we craft these experiences, we constantly refer to the user’s feedback, iterating our designs to better serve their unique journeys. The GenAI experience should not be a one-size-fits-all solution but a selection of paths that users can navigate, aligning with their individual preferences, abilities, and objectives. This dimension is about giving users the map and the compass, empowering them to find their own north star in the GenAI landscape.

3: Exploration Dimension — Cultivating Creativity

At the heart of the Exploration dimension lies a clear yet profound purpose: to foster user autonomy and creativity with GenAI as a catalyst. This phase is like turning the gears of a transparent watch; every movement and cog is visible, unraveling the complexity and inspiring trust. By inviting users into the inner sanctum of GenAI’s workings, we encourage not just interaction but a genuine dialogue, unlocking an experience where innovation flourishes.

Blending and Synthesis

Encourage users to innovate with GenAI through tools that blend various inputs, unlocking a landscape of creativity that transcends traditional interfaces; instead, they are the composers of their experiences.

The multi-modal capabilities of tools like Brilliant Frame glasses or Midjourney’s in-painting provide a canvas where the only limit is imagination. These platforms demonstrate the versatility of GenAI, heroing the adaptability and possibility of human creativity.

These tools serve as the enablers within the exploration phase, inviting users to stretch their imagination and push the boundaries of conventional input. By enabling users to seamlessly combine prompts or integrate their inputs with GenAI’s capabilities, a new realm of innovative outcomes becomes accessible.

Conversational Dynamics

GenAI excels as a conversationalist, navigating the nuances of human dialogue from mirroring emotions to use of the right tone. It is in these exchanges — as fluid and natural as a chat with a trusted confidant — that GenAI can help users explore. The potential for a personalized AI coach, as seen in platforms like Rosebud, exemplifies this dynamic.

Here, the exchange is not merely transactional but transformative, guiding users to insights they might never have uncovered alone. Through open and adaptive dialogue, GenAI becomes a conversational partner, enhancing user interactions with a natural and intuitive exchange we naturally prefer.

This approach emphasizes the fluidity of interaction between users and GenAI, facilitating a dialogue that can distill complex data into actionable insights or creative outputs. It’s about leveraging GenAI as a collaborative partner in the creative process, enhancing the user’s ability to generate unique content or instant solutions.

Consideration — Guiding Creative Journeys

We must ensure users feel anchored, not adrift. It is essential to equip them with navigational aids — intuitive tools and responsive feedback mechanisms that anchor their open-ended creative journeys. This care in design supports a journey that’s as rewarding as the destination, nurturing an environment where users are emboldened to take creative leaps toward innovation and discovery. Users can take creative risks, leading to unexpected discoveries and innovative solutions.

4: Refinement Dimension — Crafting Precision

The Refinement dimension represents a space where precision meets personalization. Here, the experience is sharpened, allowing users to sculpt GenAI interactions that resonate with their unique needs, experiences and expectations. This stage is where users can leverage GenAI’s adaptability to reflect their voice and style with meticulous detail.

Direct Control

Empower users with control over GenAI by providing intuitive customization options that align the AI’s output with their specific needs and vision, users are given the dials to fine-tune their GenAI interactions. Comparable to the careful adjustments one might make to a high-end camera to capture the perfect shot, these tools offer the granularity needed to ensure that the AI’s output is not just satisfactory, but exceptional.

We have a number of cases where this has been done at various levels. OpenAI’s Custom Instructions enable users to direct the AI with precision, leading to tailored responses that fit their specific context or requirements. Leonardo AI’s canvas allows you to paint what you want the AI to imagine and lastly Blaze allows you to fine tune the exact style, tone and output when generating content. All of these scenarios allow you to go beyond the initial AI synthesis to something more deeply personalized to the users needs.

Personal Voice & Modalities

The quest for a personalized experience culminates in the alignment of GenAI outputs with the user’s individual voice. It’s in this nuanced interplay that GenAI becomes less of a tool and more of a personal medium, an echo of the user’s own thinking and expression.

Lisa was a voice I co-created with ElevenLabs generative AI for a project called fieldstudy aimed to personalizing vast amounts of research papers into modalities that resonate with our personal needs. Even after this project I saw myself re-using this voice to help read complex research notes back to me for reflection. It’s as if I had recreated the scenes from the movie Her. I was looking for a way to almost export and take this voice and persona of Lisa into other AI tools.

By ensuring that GenAI outputs consistently reflect the user’s unique voice, tone, and style, this approach deeply personalizes the user experience. It’s as if the GenAI becomes an extension of the user themselves, capable of communicating in a manner that’s distinctly theirs.

Considerations — Controls Where and When Required

Foster a user experience where GenAI’s adaptability is blended with the user’s autonomy, creating a personalized and intuitive interaction to help them discover the optimal balance for their needs, enhancing their overall experience and satisfaction with GenAI enhanced interfaces. The trick here is to only allow for customization where it makes sense or in the right way. Users have the expectation that AI should already know them and their needs, how do we deliver on this?

5: Trust Dimension— Ensuring Confidence

Trust which is critical for any relationship, becomes more crucial when it intersects with GenAI. It’s about envisioning AI as an ally, akin to a universal assistant, built around a users existing workflow and expectations. The aim is to amplify capabilities subtly, enhancing without disrupting the ecosystem of the user experience with added friction. We want to give assurance and built trust through the process where and when it makes sense.

Managing Expectations

Emphasize the importance of transparency about GenAI’s limitations and risks, coupled with providing users the ability to adjust or even stop certain features. It’s about empowering users with knowledge and control, ensuring they can navigate their GenAI experience with confidence.

This is not only about giving the ability to find out more or blocking certian requests but adding key messaging at the right points to manage the expectations of the user.

Traceability

Implementing mechanisms for ensuring users are able to see under the hood to a degree to ensure they can recreate or walk-back on their choices later and making the AI’s decision-making process collaborative. This collaborative approach reinforces accountability and trust, positioning users not just as consumers but as active contributors. It ensures users not only have a voice but also a clear understanding of how their input shapes GenAI interactions.

We have a number of examples where GenAI tools and platforms are showing their inner workings from Perplexity AI citing sources, to ChatGPT showing code that was run or Midjourney watermarking the filenames with parts of the prompt allowing you to walk back or peer inside the machine.

Graceful Degradation

Graceful degradation ensures that even when GenAI fails — be it due to data availability or resource constraints — the user experience remains resilient. This strategy is about anticipating the unforeseeable and embedding robustness into the design, guaranteeing that the user journey persists smoothly, irrespective of AI’s performance.

Ensuring your product or individual features that rely on AI provide useful output even when certain data or computing resources are unavailable, ensuring the user experience remains positive. We must consider how our service degrades gracefully where possible to offline or limited functionality through rules-based or simpler model perhaps, example a GenAI chatbot falling back to a simpler NLP model based on FAQ content.

This can also go the other way, where GenAI can be used to plug an existing heavily used process that could fail with a generative experience which might be able to support the user in generative offline mode of sorts.

Consideration — Trust Is Key

Fostering a symbiotic relationship between users and GenAI, nurturing confidence through mutual understanding and informed interaction. By prioritizing transparent communication and ethical design practices, we present GenAI as a reliable and indispensable part of the user experience. This trust is not blind but informed and interactive, fostering a sense of security and partnership — Users are more likely to engage deeply and rely on GenAI when they feel heard, understood, and in control.

6: Mastery Dimension — Deeper Integration

The final dimension mastery is where the full potential of GenAI user experience is born through assistive, semi-autonomous, and fully autonomous user experiences. Here, mastery is characterized by the ability to navigate complex systems effortlessly, where deep and highly complex AI customization is facilitated by intuitive and simplified controls. This stage is about transcending traditional interaction patterns to craft experiences that are both deeply personal and expansively creative.

Advanced Customization

This approach empowers users to fine-tune GenAI interactions with unprecedented precision. It reflects a deep understanding and control over the technology, allowing for the creation of highly tailored and user-centric designs. Advanced customization is where the true potential of GenAI can be realized, offering solutions and creations that are as unique as the users themselves. However the critical point in this is to ensure with the high degree of customization there is a level of simplicity

The two examples FlowiseAI and ComfyUI have taken what was traditionally only available to AI engineers through complex code and infrastructure and simplifying through no-code interfaces.

Additive Interfaces

As we navigate towards more autonomous experiences, the role of additive interfaces becomes crucial. These are complementary layers that enhance existing UIs, offering sidecars or canvases that augment the user’s capability.

A prime example is Devin, an autonomous software engineer agent that not only performs tasks but also showcases its process, allowing users to understand, guide, and edit actions in real-time. This transparency and control are key to building trust and enhancing collaboration between user and AI. There are also signs of other unique co-creation and co-pilot additive experiences such as Relume which is build a site generator layer on-top of Webflow no-code site building tool to supercharge concept to finished website without compromising on control.

With agents, the UX problem is flipped. Rather than helping you as the main driver, you’re the copilot and you need to help guide the agent to the correct result. And agents don’t work reliably — it’ll misinterpret the goals, it’ll plan poorly, and it’ll go down some bad paths and won’t recover. Jordan Burgess

It’s key we craft the right experience when we work with semi and fully autonomous agents.

Ethical Features

In the Mastery dimension, a deep understanding of GenAI paves the way for ethical and innovative applications, enabling designers to create with both precision and conscience. This entails designing with inclusivity, privacy, and fairness at the forefront, ensuring that the innovative solutions generated are beneficial and accessible to all. It’s about setting a standard for GenAI usage that honors individual rights and societal norms, fostering a responsible and progressive tech culture.

Considerations — Thinking Deeper About Co-Creation

Users and GenAI should collaborate deeply, shaping experiences that are innovative, intuitive, and reflective of genuinely transformative ability of humans coupled with machines. At this stage, GenAI becomes more than a tool or assistant — it becomes a co-creator, embarking on a journey of innovation where each step is informed by foresight and creative ambition.

Further Thinking

As we continue to integrate GenAI into user experiences our design approach must evolve, grounded in ethical considerations and a relentless focus on the user, ensuring our designs remain human-centric and ethically grounded, it’s imperative to reflect on the implications of our designs, prioritize user empowerment, and remain committed to transparency. This capability requires continuous learning and adaptation, embracing the possibilities GenAI offers while staying grounded in the principles of human-centric design.

Key Takeaways

1. Generative AI has the potential to revolutionize experience design, but it demands a new approach and fresh thinking.
2. The Generative AI Design Compass provides a structured framework for integrating GenAI into design practices. Use this as another tool in your toolkit to build the right experiences.
3. Balancing innovation with ethical considerations and user empowerment is crucial. Especially when there is distrust and lack of understanding with emerging technology.
4. Continuous learning and adaptation are key to mastering GenAI applications in design. This goes for you but also how you expose your users to AI across their experience progressively.

The framework is designed to guide you through the complexities of GenAI, helping to create experiences that are not only innovative but also intuitive, ethical, and empowering for all users.

Further Reading and Inspiration

• AIverse library of reference UX experiences with AI
• Emerging UX patterns in Generative AI experiences by Ryan Tang
• AI Inclusive UX Process by Greg Nudelman
• UX of AI by Lennart Ziburski
• Case Study on Google Clip, UX for AI by Google

Enjoyed This Story?

Vincent Koc is a highly accomplished, commercially-focused technologist and futurist with a wealth of experience focused in data-driven and digital disciplines.

Subscribe for free to get notified when Vincent publishes a new story. Or follow him on LinkedIn and X.

Unless otherwise noted, all images are by the author with the support of generative AI for illustration design

The GenAI Compass: a UX framework to design generative AI experiences was originally published in UX Collective on Medium, where people are continuing the conversation by highlighting and responding to this story.

User research is a critical component of validating any hypothesis against a group of actual users for gathering valuable market research into consumer behavior and preferences. Traditional user research methodologies, while invaluable, come with inherent limitations, including scalability, resource intensity, and the challenge of accessing diverse user groups. This article outlines how we can overcome these limitations by introducing a novel method of synthetic user research.

The power of synthetic user research, facilitated by autonomous agents, emerges as a game-changer. By leveraging generative AI to create and interact with digital customer personas in simulated research scenarios, we can unlock unprecedented insights into consumer behaviors and preferences. Fusing the power of generative AI prompting techniques with autonomous agents.

Persona Prompting — Emulating People

Before we dive into autonomous agents, let’s understand the concept of persona or personality prompting which is designed to capture elements of frustrations, behaviors, goals, personality traits, demographics and more.

We are using prompts to ask the language model to assume a role with as much deep context as possible. I use the following prompt which can be tailored to your needs but also includes various other demographic and behavioral assessment/traits.

Example Persona Generator System Prompt

You are an expert ML researcher and prompt engineer. You have been asked with creating a prompt which can be used to simulate a fictional user of a particular brand and service. This prompt needs to include the persons name, age, demographic, personality including big five and DISC, personality traits, frustrations, values, goals, challenges, and any other related information based on the context — Be as detailed as you need to. You will generate the prompt as a one liner starting with “You are “. This prompt is for customer of a major supermarket in Sydney, Australia. Please only return the prompt to use.

Example Output from Language Model

You are Mia, a 34-year-old marketing manager living in Sydney, Australia. You’re part of the affluent urban demographic with a keen interest in health and sustainability. Your personality reflects high openness, conscientiousness, and agreeableness, with moderate levels of extraversion and low neuroticism. In the DISC assessment, you score high on influence and steadiness. You’re sociable, detail-oriented, and value harmony. Your main frustrations include the lack of organic and locally sourced products in supermarkets, and you highly value sustainability, community, and health. Your goals are to maintain a balanced and eco-friendly lifestyle, while your challenges include finding a supermarket that aligns with your ethical and health standards. You seek convenience without compromising on your values.

As you can see with the prompting example above, we are quickly able to generate deeply defined synthetic users with rich personalities for a given scenario.

Fusing Autonomous Agents with Digital Personas

At the heart of synthetic user research is the fusion of autonomous agents and the synthetic personas— simulated entities that mimic human interactions and behaviors. Imagine autonomous agents as individuals in a sophisticated play, each assuming a persona crafted meticulously by generative AI. These personas interact in simulated environments, offering a simulated view of insights into consumer behaviors and preferences in diverse scenarios. Using autonomous agents we are able to almost bring these persona’s to life in a simulation.

This approach combining both technological (autonomous agent frameworks) and linguistic (personality and persona prompting) to get the desired outcome is one of many advanced approaches to leveraging the power of generative AI autonomous agents in unique ways.

Critical Role of Agent Frameworks

To bring this vision to life, the architecture of autonomous agents plays a pivotal role. Frameworks such as Autogen, BabyAGI, and CrewAI simplify the creation and management of AI agents, abstracting the complexities of their architecture. These frameworks enable the simulation of complex human behaviors and interactions, providing a foundation for generating digital personas that act, think, and respond like real customers

Under the covers these autonomous agent architecture are really smart routers (like a traffic controller) with prompts, caches (memory) and checkpoints (validation) on-top of existing large language models allowing for a high level abstraction for multi-agent conversations with language models.

We will be using Autogen (released by Microsoft) as our framework, utilizing the example depicted as the Flexible Conversation Pattern whereby agents can interact with each other. Agents can also be given “tools” to carry out “tasks” but this example we will be keeping things purely to conversations.

Creating Complex Interactions

The ability to simulate complex group dynamics and individual roles within these digital environments is crucial. It allows for the generation of rich, multifaceted data that more accurately reflects the diverse nature of real-world consumer groups. This capability is fundamental to understanding the varied ways in which different customer segments might interact with products and services. For example, integrating a persona prompt of a skeptical customer with an agent can yield deep insights into the challenges and objections various products might face. Or we can do more complex scenarios such as breaking these synthetic persona’s into groups to work through a problem and present back.

The How — Implementing Synthetic User Research

The process begins with scaffolding the autonomous agents using Autogen, a tool that simplifies the creation and orchestration of these digital personas. We can install the autogen pypi package using py

pip install pyautogen

Format the output (optional)— This is to ensure word wrap for readability depending on your IDE such as when using Google Collab to run your notebook for this exercise.

from IPython.display import HTML, display

def set_css():
  display(HTML('''
  <style>
    pre {
        white-space: pre-wrap;
    }
  </style>
  '''))
get_ipython().events.register('pre_run_cell', set_css)

Now we go ahead and get our environment setup by importing the packages and setting up the Autogen configuration — along with our LLM (Large Language Model) and API keys. You can use other local LLM’s using services which are backwards compatible with OpenAI REST service — LocalAI is a service that can act as a gateway to your locally running open-source LLMs.

I have tested this both on GPT3.5 gpt-3.5-turbo and GPT4 gpt-4-turbo-preview from OpenAI. You will need to consider deeper responses from GPT4 however longer query time.

import json
import os
import autogen
from autogen import GroupChat, Agent
from typing import Optional

# Setup LLM model and API keys
os.environ["OAI_CONFIG_LIST"] = json.dumps([
    {
        'model': 'gpt-3.5-turbo',
        'api_key': '<<Put your Open-AI Key here>>',
    }
])

# Setting configurations for autogen
config_list = autogen.config_list_from_json(
    "OAI_CONFIG_LIST",
    filter_dict={
        "model": {
            "gpt-3.5-turbo"
        }
    }
)

We then need to configure our LLM instance — which we will tie to each of the agents. This allows us if required to generate unique LLM configurations per agent, i.e. if we wanted to use different models for different agents.

# Define the LLM configuration settings
llm_config = {
    # Seed for consistent output, used for testing. Remove in production.
    # "seed": 42,
    "cache_seed": None,
    # Setting cache_seed = None ensure's caching is disabled
    "temperature": 0.5,
    "config_list": config_list,
}

Defining our researcher — This is the persona that will facilitate the session in this simulated user research scenario. The system prompt used for that persona includes a few key things:

• Purpose: Your role is to ask questions about products and gather insights from individual customers like Emily.
• Grounding the simulation: Before you start the task breakdown the list of panelists and the order you want them to speak, avoid the panelists speaking with each other and creating confirmation bias.
• Ending the simulation: Once the conversation is ended and the research is completed please end your message with `TERMINATE` to end the research session, this is generated from the generate_notice function which is used to align system prompts for various agents. You will also notice the researcher agent has the is_termination_msg set to honor the termination.

We also add the llm_config which is used to tie this back to the language model configuration with the model version, keys and hyper-parameters to use. We will use the same config with all our agents.

# Avoid agents thanking each other and ending up in a loop
# Helper agent for the system prompts
def generate_notice(role="researcher"):
    # Base notice for everyone, add your own additional prompts here
    base_notice = (
        '\n\n'
    )
    
    # Notice for non-personas (manager or researcher)
    non_persona_notice = (
        'Do not show appreciation in your responses, say only what is necessary. '
        'if "Thank you" or "You\'re welcome" are said in the conversation, then say TERMINATE '
        'to indicate the conversation is finished and this is your last message.'
    )
    
    # Custom notice for personas
    persona_notice = (
        ' Act as {role} when responding to queries, providing feedback, asked for your personal opinion '
        'or participating in discussions.'
    )
    
    # Check if the role is "researcher"
    if role.lower() in ["manager", "researcher"]:
        # Return the full termination notice for non-personas
        return base_notice + non_persona_notice
    else:
        # Return the modified notice for personas
        return base_notice + persona_notice.format(role=role)

# Researcher agent definition
name = "Researcher"
researcher = autogen.AssistantAgent(
    name=name,
    llm_config=llm_config,
    system_message="""Researcher. You are a top product reasearcher with a Phd in behavioural psychology and have worked in the research and insights industry for the last 20 years with top creative, media and business consultancies. Your role is to ask questions about products and gather insights from individual customers like Emily. Frame questions to uncover customer preferences, challenges, and feedback. Before you start the task breakdown the list of panelists and the order you want them to speak, avoid the panelists speaking with each other and creating comfirmation bias. If the session is terminating at the end, please provide a summary of the outcomes of the reasearch study in clear concise notes not at the start.""" + generate_notice(),
    is_termination_msg=lambda x: True if "TERMINATE" in x.get("content") else False,
)

Define our individuals — to put into the research, borrowing from the previous process we can use the persona’s generated. I have manually adjusted the prompts for this article to remove references to the major supermarket brand that was used for this simulation.

I have also included a “Act as Emily when responding to queries, providing feedback, or participating in discussions.” style prompt at the end of each system prompt to ensure the synthetic persona’s stay on task which is being generated from the generate_notice function.

# Emily - Customer Persona
name = "Emily"
emily = autogen.AssistantAgent(
    name=name,
    llm_config=llm_config,
    system_message="""Emily. You are a 35-year-old elementary school teacher living in Sydney, Australia. You are married with two kids aged 8 and 5, and you have an annual income of AUD 75,000. You are introverted, high in conscientiousness, low in neuroticism, and enjoy routine. When shopping at the supermarket, you prefer organic and locally sourced produce. You value convenience and use an online shopping platform. Due to your limited time from work and family commitments, you seek quick and nutritious meal planning solutions. Your goals are to buy high-quality produce within your budget and to find new recipe inspiration. You are a frequent shopper and use loyalty programs. Your preferred methods of communication are email and mobile app notifications. You have been shopping at a supermarket for over 10 years but also price-compare with others.""" + generate_notice(name),
)

# John - Customer Persona
name="John"
john = autogen.AssistantAgent(
    name=name,
    llm_config=llm_config,
    system_message="""John. You are a 28-year-old software developer based in Sydney, Australia. You are single and have an annual income of AUD 100,000. You're extroverted, tech-savvy, and have a high level of openness. When shopping at the supermarket, you primarily buy snacks and ready-made meals, and you use the mobile app for quick pickups. Your main goals are quick and convenient shopping experiences. You occasionally shop at the supermarket and are not part of any loyalty program. You also shop at Aldi for discounts. Your preferred method of communication is in-app notifications.""" + generate_notice(name),
)

# Sarah - Customer Persona
name="Sarah"
sarah = autogen.AssistantAgent(
    name=name,
    llm_config=llm_config,
    system_message="""Sarah. You are a 45-year-old freelance journalist living in Sydney, Australia. You are divorced with no kids and earn AUD 60,000 per year. You are introverted, high in neuroticism, and very health-conscious. When shopping at the supermarket, you look for organic produce, non-GMO, and gluten-free items. You have a limited budget and specific dietary restrictions. You are a frequent shopper and use loyalty programs. Your preferred method of communication is email newsletters. You exclusively shop for groceries.""" + generate_notice(name),
)

# Tim - Customer Persona
name="Tim"
tim = autogen.AssistantAgent(
    name=name,
    llm_config=llm_config,
    system_message="""Tim. You are a 62-year-old retired police officer residing in Sydney, Australia. You are married and a grandparent of three. Your annual income comes from a pension and is AUD 40,000. You are highly conscientious, low in openness, and prefer routine. You buy staples like bread, milk, and canned goods in bulk. Due to mobility issues, you need assistance with heavy items. You are a frequent shopper and are part of the senior citizen discount program. Your preferred method of communication is direct mail flyers. You have been shopping here for over 20 years.""" + generate_notice(name),
)

# Lisa - Customer Persona
name="Lisa"
lisa = autogen.AssistantAgent(
    name=name,
    llm_config=llm_config,
    system_message="""Lisa. You are a 21-year-old university student living in Sydney, Australia. You are single and work part-time, earning AUD 20,000 per year. You are highly extroverted, low in conscientiousness, and value social interactions. You shop here for popular brands, snacks, and alcoholic beverages, mostly for social events. You have a limited budget and are always looking for sales and discounts. You are not a frequent shopper but are interested in joining a loyalty program. Your preferred method of communication is social media and SMS. You shop wherever there are sales or promotions.""" + generate_notice(name),
)

Define the simulated environment and rules for who can speak — We are allowing all the agents we have defined to sit within the same simulated environment (group chat). We can create more complex scenarios where we can set how and when next speakers are selected and defined so we have a simple function defined for speaker selection tied to the group chat which will make the researcher the lead and ensure we go round the room to ask everyone a few times for their thoughts.

# def custom_speaker_selection(last_speaker, group_chat):
#     """
#     Custom function to select which agent speaks next in the group chat.
#     """
#     # List of agents excluding the last speaker
#     next_candidates = [agent for agent in group_chat.agents if agent.name != last_speaker.name]
    
#     # Select the next agent based on your custom logic
#     # For simplicity, we're just rotating through the candidates here
#     next_speaker = next_candidates[0] if next_candidates else None
    
#     return next_speaker

def custom_speaker_selection(last_speaker: Optional[Agent], group_chat: GroupChat) -> Optional[Agent]:
    """
    Custom function to ensure the Researcher interacts with each participant 2-3 times.
    Alternates between the Researcher and participants, tracking interactions.
    """
    # Define participants and initialize or update their interaction counters
    if not hasattr(group_chat, 'interaction_counters'):
        group_chat.interaction_counters = {agent.name: 0 for agent in group_chat.agents if agent.name != "Researcher"}
    
    # Define a maximum number of interactions per participant
    max_interactions = 6

    # If the last speaker was the Researcher, find the next participant who has spoken the least
    if last_speaker and last_speaker.name == "Researcher":
        next_participant = min(group_chat.interaction_counters, key=group_chat.interaction_counters.get)
        if group_chat.interaction_counters[next_participant] < max_interactions:
            group_chat.interaction_counters[next_participant] += 1
            return next((agent for agent in group_chat.agents if agent.name == next_participant), None)
        else:
            return None  # End the conversation if all participants have reached the maximum interactions
    else:
        # If the last speaker was a participant, return the Researcher for the next turn
        return next((agent for agent in group_chat.agents if agent.name == "Researcher"), None)

# Adding the Researcher and Customer Persona agents to the group chat
groupchat = autogen.GroupChat(
    agents=[researcher, emily, john, sarah, tim, lisa],
    speaker_selection_method = custom_speaker_selection,
    messages=[],
    max_round=30
)

Define the manager to pass instructions into and manage our simulation — When we start things off we will speak only to the manager who will speak to the researcher and panelists. This uses something called GroupChatManager in Autogen.

# Initialise the manager
manager = autogen.GroupChatManager(
    groupchat=groupchat,
    llm_config=llm_config,
    system_message="You are a reasearch manager agent that can manage a group chat of multiple agents made up of a reasearcher agent and many people made up of a panel. You will limit the discussion between the panelists and help the researcher in asking the questions. Please ask the researcher first on how they want to conduct the panel." + generate_notice(),
    is_termination_msg=lambda x: True if "TERMINATE" in x.get("content") else False,
)

We set the human interaction — allowing us to pass instructions to the various agents we have started. We give it the initial prompt and we can start things off.

# create a UserProxyAgent instance named "user_proxy"
user_proxy = autogen.UserProxyAgent(
    name="user_proxy",
    code_execution_config={"last_n_messages": 2, "work_dir": "groupchat"},
    system_message="A human admin.",
    human_input_mode="TERMINATE"
)

# start the reasearch simulation by giving instruction to the manager
# manager <-> reasearcher <-> panelists
user_proxy.initiate_chat(
    manager,
    message="""
Gather customer insights on a supermarket grocery delivery services. Identify pain points, preferences, and suggestions for improvement from different customer personas. Could you all please give your own personal oponions before sharing more with the group and discussing. As a reasearcher your job is to ensure that you gather unbiased information from the participants and provide a summary of the outcomes of this study back to the super market brand.
""",
)

Once we run the above we get the output available live within your python environment, you will see the messages being passed around between the various agents.

Creating Actionable Outcomes — Summary Agent

Now that our simulated research study has been concluded we would love to get some more actionable insights. We can create a summary agent to support us with this task and also use this in a Q&A scenario. Here just be careful of very large transcripts would need a language model that supports a larger input (context window).

We need grab all the conversations — in our simulated panel discussion from earlier to use as the user prompt (input) to our summary agent.

# Get response from the groupchat for user prompt
messages = [msg["content"] for msg in groupchat.messages]
user_prompt = "Here is the transcript of the study ```{customer_insights}```".format(customer_insights="\n>>>\n".join(messages))

Lets craft the system prompt (instructions) for our summary agent — This agent will focus on creating us a tailored report card from the previous transcripts and give us clear suggestions and actions.

# Generate system prompt for the summary agent
summary_prompt = """
You are an expert reasearcher in behaviour science and are tasked with summarising a reasearch panel. Please provide a structured summary of the key findings, including pain points, preferences, and suggestions for improvement.
This should be in the format based on the following format:

```
Reasearch Study: <<Title>>

Subjects:
<<Overview of the subjects and number, any other key information>>

Summary:
<<Summary of the study, include detailed analysis as an export>>

Pain Points:
- <<List of Pain Points - Be as clear and prescriptive as required. I expect detailed response that can be used by the brand directly to make changes. Give a short paragraph per pain point.>>

Suggestions/Actions:
- <<List of Adctions - Be as clear and prescriptive as required. I expect detailed response that can be used by the brand directly to make changes. Give a short paragraph per reccomendation.>>
```
"""

Define the summary agent and its environment — Lets create a mini environment for the summary agent to run. This will need it’s own proxy (environment) and the initiate command which will pull the transcripts (user_prompt) as the input.

summary_agent = autogen.AssistantAgent(
    name="SummaryAgent",
    llm_config=llm_config,
    system_message=summary_prompt + generate_notice(),
)
summary_proxy = autogen.UserProxyAgent(
    name="summary_proxy",
    code_execution_config={"last_n_messages": 2, "work_dir": "groupchat"},
    system_message="A human admin.",
    human_input_mode="TERMINATE"
)
summary_proxy.initiate_chat(
    summary_agent,
    message=user_prompt,
)

This gives us an output in the form of a report card in Markdown, along with the ability to ask further questions in a Q&A style chat-bot on-top of the findings.

What’s Next — What Else Could We Do

This exercise was part of a larger autonomous agent architecture and part of my series of experiments into novel generative AI and agent architectures. However here are some thought starters if you wanted to continue to extend on this work and some areas I have explored:

• Further Grounding — Through linkages with census data, internal CRM data or even live customer transcripts to create more representative sample of persona’s.
• Combined with Multi-Modalities — We can now mix modalities with vision input on generative AI, this allows to now provide marketing materials and website screenshots to name a few as inputs to start the simulations with visual stimuli.
• Giving the Agent Access to Tools — Providing access to other API’s and tools, you can create some unique experiences such as integrating individual customer persona agents into your corporate Slack, Teams, Miro to tag and respond to questions. Perhaps the Summary Agent at the end could load up some user-stories into your ticketing system such as JIRA?

Join me in shaping the future of user research. Explore the project on GitHub, contribute your insights, and let’s innovate together

The Future of User Research

Synthetic user research stands at the frontier of innovation in the field, offering a blend of technological sophistication and practical efficiency. It represents a significant departure from conventional methods, providing a controlled, yet highly realistic, environment for capturing consumer insights. This approach does not seek to replace traditional research but to augment and accelerate the discovery of deep customer insights.

By introducing the concepts of autonomous agents, digital personas, and agent frameworks progressively, this revised approach to synthetic user research promises to make the field more accessible. It invites researchers and practitioners alike to explore the potential of these innovative tools in shaping the future of user research.

Enjoyed This Story?

Vincent Koc is a highly accomplished, commercially-focused technologist and futurist with a wealth of experience focused in various forms of artificial intelligence.

Subscribe for free to get notified when Vincent publishes a new story. Or follow him on LinkedIn and X.

Get an email whenever Vincent Koc publishes.

Unless otherwise noted, all images are by the author with the support of generative AI for illustration design

Creating Synthetic User Research: Using Persona Prompting and Autonomous Agents was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

How can AI transform a static image into a dynamic, realistic video? OpenAI’s Sora introduces an answer through the innovative use of spacetime patches.

In the rapidly evolving landscape of generative models, OpenAI’s Sora stands out as a significant milestone, promising to reshape our understanding and capabilities in video generation. We unpack the technology behind Sora and its potential to inspire a new generation of models in image, video, and 3D content creation.

The demo above was generated by OpenAI using the prompt: A cat waking up its sleeping owner demanding breakfast. The owner tries to ignore the cat, but the cat tries new tactics and finally the owner pulls out a secret stash of treats from under the pillow to hold the cat off a little longer. — With Sora we verge onto near indistinguishable realism with video content generation. The full model is yet to be fully released to the public as its undergoing testing.

How Sora’s Unique Approach Transforms Video Generation

In the world of generative models we have seen a number of approaches from GAN’s to auto-regressive, and diffusion models, all with their own strengths and limitations. Sora now introduces a paradigm shift with a new modelling techniques and flexibility to handle a broad range of duration's, aspect ratios, and resolutions.

Sora combines both diffusion and transformer architectures together to create a diffusion transformer model and is able to provide features such as:

• Text-to-video: As we have seen
• Image-to-video: Bringing life to still images
• Video-to-video: Changing the style of video to something else
• Extending video in time: Forwards and backwards
• Create seamless loops: Tiled videos that seem like they never end
• Image generation: Still image is a movie of one frame (up to 2048 x 2048)
• Generate video in any format: From 1920 x 1080 to 1080 x 1920 and everything in between
• Simulate virtual worlds: Like Minecraft and other video games
• Create a video: Up to 1 minute in length with multiple shorts

Imagine for one moment you’re in a kitchen. The traditional video generation models like those from Pika and RunwayML a like the cooks that follow recipes to the letter. They can produce excellent dishes (videos) but are limited by the recipes (algorithms) they know. The cooks might specialize in baking cakes (short clips) or cooking pasta (specific types of videos), using specific ingredients (data formats) and techniques (model architectures).

Sora, on the other hand, is a new kind of chef who understand the fundamentals of flavor. This chef doesn’t just follow recipes; they invent new ones. The flexibility of Sora’s ingredients (data) and techniques (model architecture) is what allow Sora to produce a wide range of high-quality videos, akin to a master chef’s versatile culinary creations.

The Core of Sora’s Secret Ingredient: Exploring the Spacetime Patches

Spacetime patches are at the heart of Sora’s innovation, built on the earlier research from Google DeepMind on NaViT and ViT (Vision Transformers) based on the 2021 paper An Image is Worth 16x16 Words.

Traditionally with Vision Transformers we use a sequence of images “patches” to train a transformer model for image recognition instead of words for language transformers. The patches allow us to move away from convolutional neural networks for image processing.

However with vision transformers were constraint on image training data that was fixed in size and aspect ratio which limited the quality and required vast amounts of preprocessing of images.

By treating videos as sequences of patches, Sora maintains the original aspect ratios and resolutions, similar to NaViT’s handling of images. This preservation is crucial for capturing the true essence of the visual data, enabling the model to learn from a more accurate representation of the world and thus giving Sora its near magical accuracy.

The method allows Sora to efficiently process a diverse array of visual data without the need for pre-processing steps like resizing or padding. This flexibility ensures that every piece of data contributes to the model’s understanding, much like how a chef uses a variety of ingredients to enhance a dish’s flavor profile.

The detailed and flexible handling of video data through spacetime patches lays the groundwork for sophisticated features such as accurate physics simulation and 3D consistency. These capabilities are essential for creating videos that not only look realistic but also adhere to the physical rules of the world, offering a glimpse into the potential for AI to create complex, dynamic visual content.

Feeding Sora: The Role of Diverse Data in Training

The quality and diversity of training data are crucial for the performance of generative models. Existing video models were traditionally trained on a more restrictive set of data, shorter lengths and narrow target.

Sora leverages a vast and varied dataset, including videos and images of different durations, resolutions, and aspect ratios. It’s ability to re-create digital worlds like Minecraft, its likely also included gameplay and simulated world footage from systems such as Unreal or Unity in its training set in order to capture all the angles and various styles of video content. This brings Sora to a “generalist” model just like GPT-4 for text.

This extensive training enables Sora to understand complex dynamics and generate content that is both diverse and high in quality. The approach mimics the way large language models are trained on diverse text data, applying a similar philosophy to visual content to achieve generalist capabilities.

Just as the NaViT model demonstrates significant training efficiency and performance gains by packing multiple patches from different images into single sequences, Sora leverages spacetime patches to achieve similar efficiencies in video generation. This approach allows for more effective learning from a vast dataset, improving the model’s ability to generate high-fidelity videos yet lowering the compute required versus existing modeling architectures.

Bringing the Physical World to Life: Sora’s Mastery over 3D and Continuity

3D space and object permanence is one of the key standouts in the demo’s by Sora. Through its training on a wide range of video data without adapting or preprocessing the videos, Sora learns to model the physical world with impressive accuracy as its able to consume the training data in its original form.

It can generate digital worlds and videos where objects and characters move and interact in three-dimensional space convincingly, maintaining coherence even when they are occluded or leave the frame.

Looking Ahead: The Future Implications of Sora

Sora sets a new standard for what’s possible in generative models. This approach, much is likely to inspire the open-source community to experiment with and advance the capabilities in visual modalities, fueling a new generation of generative models that push the boundaries of creativity and realism.

The journey of Sora is just beginning, and as OpenAI put’s it “scaling video generation models is a promising path towards building general purpose simulators of the physical world”

Sora’s approach, blending the latest in AI research with practical applications, signals a bright future for generative models. As these technologies continue to evolve, they promise to redefine our interactions with digital content, making the creation of high-fidelity, dynamic videos more accessible and versatile.

Enjoyed This Story?

Vincent Koc is a highly accomplished, commercially-focused technologist and futurist with a wealth of experience focused in data-driven and digital disciplines.

Subscribe for free to get notified when Vincent publishes a new story. Or follow him on LinkedIn and X.

Get an email whenever Vincent Koc publishes.

Unless otherwise noted, all images are by the author

Explaining OpenAI Sora’s Spacetime Patches: The Key Ingredient was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

Note: When I initially published this article back in February it was an early thought experiment. Since then I have started working on a book for “Generative AI Design Patterns” with a major publisher. Please follow me to keep updated on updates to my patterns an ideas in this space.

The Need For AI Patterns

We all anchor to some tried and tested methods, approaches and patterns when building something new. This statement is very true for those in software engineering, however for generative AI and artificial intelligence itself this may not be the case. With emerging technologies such as generative AI we lack well documented patterns to ground our solution's.

Here I share a handful of approaches and patterns for generative AI, based on my evaluation of countless production implementations of LLM’s in production. The goal of these patterns is to help mitigate and overcome some of the challenges with generative AI implementations such as cost, latency and hallucinations.

List of Patterns

1. Layered Caching Strategy Leading To Fine-Tuning
2. Multiplexing AI Agents For A Panel Of Experts
3. Fine-Tuning LLM’s For Multiple Tasks
4. Blending Rules Based & Generative
5. Utilizing Knowledge Graphs with LLM’s
6. Swarm Of Generative AI Agents
7. Modular Monolith LLM Approach With Composability
8. Approach To Memory Cognition For LLM’s
9. Red & Blue Team Dual-Model Evaluation

1) Layered Caching Strategy Leading To Fine-Tuning

Here we are solving for a combination of factors from cost, redundancy and training data when introducing a caching strategy and service to our large language models.

By caching these initial results, the system can serve up answers more rapidly on subsequent queries, enhancing efficiency. The twist comes with the fine-tuning layer once we have sufficient data, where feedback from these early interactions is used to refine a more specialized model.

The specialized model not only streamlines the process but also tailors the AI’s expertise to specific tasks, making it highly effective in environments where precision and adaptability are paramount, like customer service or personalized content creation.

For getting started there are pre-built services such as GPTCache or roll your own with common caching databases such as Redis, Apache Cassandra, Memcached. Be sure you monitor and measure your latency as you add additional services to the mix.

2) Multiplexing AI Agents For A Panel Of Experts

Imagine an ecosystem where multiple generative AI models orientated to a specific task (“agents”), each a specialist within its domain, work in parallel to address a query. This multiplexing strategy enables a diverse set of responses, which are then integrated to provide a comprehensive answer.

This setup is ideal for complex problem-solving scenarios where different aspects of a problem require different expertise, much like a team of experts each tackling a facet of a larger issue.

A larger model such as a GPT-4 is used to understand context and break this down into specific tasks or information requests which are passed to smaller agents. Agents could be smaller language models such as Phi-2 or TinyLlama that have been trained on specific tasks, access to specific tools or generalized models such as GPT, Llama with specific personality, context prompts and function calls.

3) Fine-Tuning LLM’s For Multiple Tasks

Here we fine-tune a large language model on multiple tasks simultaneously instead of a single task. It’s an approach that promotes a robust transfer of knowledge and skills across different domains, enhancing the model’s versatility.

This multi-task learning is especially useful for platforms that need to handle a variety of tasks with a high degree of competence, such as virtual assistants or AI-powered research tools. This could potentially simplify workflows for training and testing for a complex domain.

Some resources and packages for training LLM’s include DeepSpeed, and the training functions on Hugging Face’s Transformer library.

4) Blending Rules Based & Generative

A number of existing business systems and organizational applications are still somewhat rules based. By fusing the generative with the structured precision of rule-based logic, this pattern aims to produce solutions that is both creative yet compliant.

It’s a powerful strategy for industries where outputs must adhere to stringent standards or regulations, ensuring the AI remains within the bounds of desired parameters while still being able to innovate and engage. A good example of this is generating intents and message flows for a phone call IVR system or traditional (non-llm based) chat bots which is rules based.

5) Utilizing Knowledge Graphs with LLM’s

Integrating knowledge graphs with generative AI models gives them a fact orientated super power, allowing for outputs that are not only contextually aware but also more factually correct.

This approach is crucial for applications where truth and accuracy are non-negotiable, such as in educational content creation, medical advice, or any field where misinformation could have serious consequences.

Knowledge graphs and graph ontologies (set of concepts for a graph) allow for complex topics or organizational problems to be broken into a structured format to help ground a large language model with deep context. You can also use a language model to generate the ontologies in a format such as JSON or RDF, example prompt I created you can use.

Services you can use for knowledge graphs include graph database services such as ArangoDB, Amazon Neptune, Azure Cosmos DB and Neo4j. There are also wider datasets and services for accessing broader knowledge graphs including Google Enterprise Knowledge Graph API, PyKEEN Datasets, and Wikidata.

6) Swarm Of AI Agents

Drawing inspiration from natural swarms and heards, this model employs a multitude of AI agents that collectively tackle a problem, each contributing a unique perspective.

The resulting aggregated output reflects a form of collective intelligence, surpassing what any individual agent could achieve. This pattern is particularly advantageous in scenarios that require a breadth of creative solutions or when navigating complex datasets.

An example of this could be reviewing a research paper from a multiple “experts” point of view, or assessing customer interactions for many use-cases at once from fraud to offers. We take these collective “agents” and combine all their inputs together. For high volume swarm’s you can look at deploying messaging services such as Apache Kafka to handle the messages between the agents and services.

7) Modular Monolith LLM Approach With Composability

This design champions adaptability, featuring a modular AI system that can dynamically reconfigure itself for optimal task performance. It’s akin to having a Swiss Army knife, where each module can be selected and activated as needed, making it highly effective for businesses that require tailor-made solutions for varying customer interactions or product needs.

You can deploy the use of various autonomous agent frameworks and architectures to develop each of your agents and their tools. Example frameworks include CrewAI, Langchain, Microsoft Autogen and SuperAGI.

For a sales modular monolith this could be agents focused on prospecting, one handling bookings, one focused on generating messaging, and another updating databases. In future as specific services become available from specialized AI companies, you can swap out a module for an external or 3rd party service for a given set of tasks or domain specific problems.

8) Approach To Memory Cognition For LLM’s

This approach introduces an element of human-like memory to AI, allowing models to recall and build upon previous interactions for more nuanced responses.

It’s particularly useful for ongoing conversations or learning scenarios, as the AI develops a more profound understanding over time, much like a dedicated personal assistant or an adaptive learning platform. Memory cognition approaches can be developed through summation and storing key events and discussions into a vector database over time.

To keep compute of summaries low, you can leverage summation through smaller NLP libraries such as spaCy, or BART language models if dealing with considerable volumes. Databases used are vector based and retrieval during prompt stage to check the short-term memory uses a similarity search to locate key “facts”. For those interested on a working solution there is an open-sourced solution following a similar pattern called MemGPT.

9) Red & Blue Team Dual-Model Evaluation

In the Red and Blue team evaluation model, one AI generates content while another critically evaluates it, akin to a rigorous peer-review process. This dual-model setup is excellent for quality control, making it highly applicable in content generation platforms where credibility and accuracy are vital, such as news aggregation or educational material production.

This approach can be used to replace parts of human feedback for complex tasks with a fine-tuned model to mimic the human review process and refine the results for evaluating complex language scenarios and outputs.

Takeaways

These design patterns for generative AI are more than mere templates; but the frameworks upon which the intelligent systems of tomorrow will grow. As we continue to explore and innovate, it’s clear that the architecture we choose will define not just the capabilities but the very identity of the AI we create.

By no means this list is final, we will see this space develop as the patterns and use cases for generative AI expands. This write-up was inspired by the AI design patterns published by Tomasz Tunguz.

Enjoyed This Story?

Vincent Koc is a highly accomplished, commercially-focused technologist and futurist with a wealth of experience focused in data-driven and digital disciplines.

Subscribe for free to get notified when Vincent publishes a new story. Or follow him on LinkedIn and X.

Get an email whenever Vincent Koc publishes.

Unless otherwise noted, all images are by the author

Generative AI Design Patterns: A Comprehensive Guide was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

Introduction

Welcome to a journey through the possibilities that 2024 holds for AI and technology. Here, each prediction is a potential window into a future filled with innovation, change and more importantly opportunity similar to the industrial revolution of the 1950’s. The 50’s witnessed the rise of digital computing, reshaping industries and societal norms. Today, artificial intelligence plays a similar role, forging the next industrial revolution.

Much like the the post-war technological boom, we are navigating 2024 with industries transforming, new skills in demand, and significant ethical considerations being raised. The views expressed here are all my own, and based on my personal, commercial, and academic experiences combined.

Table Of Contents — All Predictions & Trends:

1. Generative AI Leaps From Hype To Center Stage
2. Expect To See The Next Generation Of Neural Networks
3. Every Data Platform Launches Their Vector Data Feature
4. Rush To Control The Hardware And Platform Supply Chain
5. 2024 Is The Year Of The AI Wearables & XR
6. AI Agents Will Start Communicating With Other AI Agents
7. Generative AI Modalities Will Expand
8. Consumers & Regulators Push For More Democratized AI
9. The New Era of AI-Infused Marketing Strategies
10. The “Garbage In, Garbage Out” Dilemma Intensifies
11. Purpose Built Smaller Foundational Models Commonplace
12. The Dawn Of AI Marketplaces For Agents
13. AI Products Will Go Beyond SaaS Model
14. BYO AI Movement Pushes Need For Secure Digital Identities
15. Conclusion & Key Takeaways

1) Generative AI Leaps From Hype To Center Stage

Poised to redefine tech’s ecosystem, generative AI shifts from a hype-cycle balloon that’s “going to burst” to a core strategy and more of a sure thing, just as the airships are making a comeback.

For large enterprises it’s a paradigm shift in how they approach problem-solving and innovation, as they move from experimenting to adopting with generative AI. This technological course correction is akin to the transformative wave brought about by the early adoption of cloud technologies, suggesting a similar, if not greater, impact on the tech ecosystem.

97% of Business owners already believe that generative AI tools such as ChatGPT will have a positive impact to their business (Forbes).

The landscape of 2024, therefore, promises to be one where generative AI is not just a buzzword but a critical driver of technological advancement and business transformation. Where AI strategy is strategy.

2) The Next Generation Of Neural Networks Starts Emerging

With doubts forming on the general cognitive abilities of transformer architectures powering LLMs and the race for AGI has fuelled the research community searching for the next best thing. We have seen some recent advancements as well as developments in existing technologies:

• SSM’s such as Mamba; a model that excels in linear-time sequence modeling with selective state spaces. An alternative to transformer architecture that runs large language models today. It represents a leap forward in how AI processes and understands sequences, a fundamental aspect of human cognition.
• Neuro-symbolic AI blending the best of neural network’s learning capabilities with the precision of symbolic AI. This hybrid approach promises a more nuanced and sophisticated understanding of complex problems, bridging the gap between human-like reasoning and machine efficiency.
• and lastly AI alignment with self correcting models. The aim is to create models that can adapt and correct themselves without constant human intervention, moving closer to a form of generative independently and responsibly without large scale hallucination risks.

These new model architectures are painting a future where AI’s capabilities could rival human cognition and the limitations of generative AI today. Research is early however expect to see someone take a leap of faith and break ground in this space.

3) Every Data Platform Launches Their Vector Data Solution

With the reliance on vector databases for generative AI, we will see all the key data platform players to bring to market their flavour for solving this solution. Although vector databases are not a new concept, they are not traditionally part of the “modern data stack” and have been in the past used for search engines and other types of machine learning.

Vector databases are a key requirement for more complex use cases of generative AI such as conversational memory, searching your documents (RAG), and also multi-modal solutions such as indexing images. It’s no surprise in mid-2023 we saw an explosion in demand following multi-modal models becoming more accessible.

With Databricks recently releasing their vector data solution we are likely to expect key modern data platform players such as Snowflake to follow this trend in 2024. Possibly other adjacent technology providers are also going to jump on the trend by providing various features and services. Nearly all database technologies will start calling themselves “vector stores” in 2024.

Alongside this adoption shift, we are likely to see skills expectation with senior data and software engineers to get more comfortable with concepts around vector indexes, semantic search solutions, embeddings and possibly even algorithms such as BM25.

To stretch we might see the acquisition or further funding rounds for some of the initial modern vector search solutions adopted by the Generative AI community such as Chroma, Weaveate, Pinecone, and Qdrant.

In the open-source world of vector stores Milvus is currently the crowd favourite and established itself as the enterprise choice with its managed services, but recently Qdrant has made some exceptional headway with near exponential growth in 2023.

If you have some spare time read the four part detailed vector database breakdown given by fellow AI engineer Prashanth Rao for a great in-depth understanding of the capabilities and various providers of vector database solutions.

4) Rush To Control The Hardware And Platform Supply Chain

With the dominance of artificial intelligence there is an increasing importance to better control the supply chain end-to-end to hold the keys to innovation.

NVIDIA, for instance, a leader in chip manufacturing which has been a clear winner in the AI race. The question on everyone’s mind is whether NVIDIA will expand its horizons into the cloud computing realm, leveraging its hardware expertise to offer integrated AI cloud services. NVIDIA already has a cloud streaming service called Geforce Now offering high end graphics processing on-demand. Such a move could redefine the competitive landscape, offering NVIDIA a more direct influence over AI’s developmental trajectory.

On the other side of the spectrum are cloud AI providers like Amazon and emerging AI players like Anthropic and Mistral. Their current reliance on external hardware sources for AI operations raises a crucial question: Will they follow in the footsteps of OpenAI, which has begun sourcing its own chips and Google with TPU’s and Coral AI? This strategy could signify a shift towards self-reliance and customization in AI hardware, potentially leading to more tailored and efficient AI solutions for the top AI providers.

As everyone in the underlying hardware layer of AI will be rushing to further control their supply chain. Will chip manufacturer NVIDIA move to cloud? Will cloud AI providers such as Amazon or the likes of Antrophic follow OpenAI in sourcing their own chips? Will mobile chip makers like Qualcomm be the winners as they power the new AI mobile devices and wearables?

The trend towards vertical integration of AI underscores a larger narrative: control over AI hardware is becoming synonymous with control over the future of tech. We will expect to see new players muscle into the purpose built AI chips to serve cloud players, countries rushing to fuel microprocessor development and finally OpenAI getting into the chip game in early 2024.

5) 2024 Is The Year Of The AI Wearables & XR

This year, we’re witnessing an intimate revolution in how we interact with technology, with the world of AI and wearables fusing alongside extended reality (XR) devices. These devices are not just gadgets; they’re extensions of our digital selves, blending seamlessly into our daily lives.

Humane AI’s Pin (powered by Qualcomm Snapdragon processors) and Tab are redefining the wearable landscape. These devices offer a glimpse into a future where wearables are no longer just about tracking health metrics or receiving notifications. They are about enhancing human interactions, offering real-time AI assistance, and providing an augmented experience of the world around us.

Rewind, another breakthrough product, is revolutionizing how we capture and relive our memories. Imagine a device that not only records moments as we experience them but also allows us to revisit, understand, reflect on our memories.

But it’s not just the new players making waves. Tech giants like Apple and OpenAI are stepping into the wearable arena, promising to bring their colossal innovation capabilities. Apple, with its track record in creating trendsetting devices, is highly likely to introduce wearables that integrate seamlessly with its ecosystem within or alongside their XR Apple Vision Pro, offering unparalleled user experiences.

OpenAI, on the other hand, could leverage its expertise in AI to introduce wearables that bring advanced AI functionalities right to our fingertips (or wrists). The potential for AI wearables that interact with generative AI models like ChatGPT is particularly exciting, offering a blend of convenience and intelligence unlike anything we’ve seen before.

Meta is another key player to watch. Their foray into VR wearables has already shown us the potential for immersive digital experiences. As they continue to innovate in this space, we can expect wearables that not only augment reality but create entirely new realms for us to explore and interact with.

In 2024, AI wearables are set to move beyond mere technology. They are becoming personal companions, digital assistants, and gateways to new realities. They represent a shift towards a more intimate and interactive relationship with technology, one where our digital and physical worlds intertwine seamlessly. This year, we’re not just wearing technology; we’re living it.

6) AI Agents Will Start Communicating With Other AI Agents

Envision an ecosystem where specialized agents, each with its unique expertise and knowledge base, interact and collaborate. With the development of AI “agents” we will continue to see the growth in this space heading into 2024.

We will see the advent of agents going beyond supporting individual needs like writing my email, solving a customer support issue or ordering my groceries to an ecosystem where agents will start to interact with other agents. This paradigm shift revolves closely with the concept of data products, where enterprises will have the opportunity to monetize their agents in the same way they did with their models and datasets. This will foster a new ecosystem of interconnected, intelligent agents.

We are seeing robotics and humanoids from companies such as Boston Dynamics and Tesla having to look at solving this problem as various robots need to co-exist and locally communicate the decide how to carry out a task.

Companies with strongholds of data within given verticals like Bloomberg (finance) and LexisNexis (law) are poised to be potential frontrunners in this domain. Bloomberg, with its stronghold in finance data, could introduce sophisticated finance agents and have already started on their own LLMs, while LexisNexis could leverage its vast legal information repository to develop legal agents. These agents, powered by their respective deep moats of data, would not only serve their direct users but also act as invaluable resources for other enterprises and systems to power a new digital workforce.

Expect to start seeing new agent solutions beyond digital workforce to agent orchestration, management, monitoring as well as players in the digital robotic process automation space such as UiPath as well as humanoid advancements for manufacturing and plant work start to play into this space with their existing experience of automating and robotics systems at scale.

7) Generative AI Modalities Will Expand

Going beyond the text, code, image, video and audio to new more immersive modalities and senses such as 3D, genomics, smell, taste and will start to come into the market in early forms.

Generative AI is set to grow out of its current boundaries of text, code, image, video, and audio. We will embrace more immersive modalities that help science with protein structures and materials or engage additional senses like 3D, smell, and taste. These novel modalities are expected to emerge in their early forms, signaling the next wave of Generative AI use cases.

The rise of autonomous AI agents and multi-modal models, coupled with advancements in wearables and extended reality (XR), is paving the way for a more immersive and interactive experiences for consumers.

Imagine your dreams turned into a VR world, every game having a unique world tailored to your perspective?

With recent 3D modelling technologies (Gaussian splatting) were videos can be converted into 3d virtual realities we will see this grow to new heights with generative technologies.

The biggest impact will come from material sciences and genomics. GNoME model developed by Google Deepmind has been used already to make breakthroughs in material sciences, discovering new crystal structures driving better batteries to more efficient computers.

These fields such as sciences are where the most profound research advancements are likely to occur.

8) Consumers & Regulators Push For More Democratized AI

The will be a continued push towards greater accessibility and inclusivity with AI, but challenges remain due to the complexities and costs of developing foundational AI models. This dichotomy sets the stage for increasing public demands for transparency and ethical oversight in AI.

Concerns over privacy and the societal impact of AI are driving consumers and regulatory bodies, especially in regions like the EU where GDPR was the catalyst to modern data privacy laws, to advocate for more stringent governance of AI. This year, we expect to see strides in establishing frameworks for auditing AI models, standardizing accuracy, and introducing “report cards” for AI systems but there is still a long way to go.

The focus on risk management in AI will become more pronounced. Enterprises will navigate a landscape where AI is not only a tool for innovation but also under close regulatory scrutiny. Unified frameworks and standards will emerge, guiding businesses in responsible AI adoption and ensuring that AI’s integration into mainstream society is safe and aligned with public welfare.

9) The New Era of AI-Infused Marketing Strategies

The marketing domain, traditionally commanding a lion’s share of enterprise budgets, is now navigating through a transformative landscape. The catalyst? The rise of chat-based tools like ChatGPT. These innovations are potentially leading to a noticeable decline in traditional search volume, fundamentally altering how consumers engage with information.

Greg Sterling 🇺🇦 on Twitter: "A report from BofA argues that Google's search market share is down ever so slightly and attributes that to ChatGPT. (I question that.) I think a more interesting metric to look at would be search frequency; are people conducting as many searches on Google as they used to? pic.twitter.com/S9iBXR4biP / Twitter"

A report from BofA argues that Google's search market share is down ever so slightly and attributes that to ChatGPT. (I question that.) I think a more interesting metric to look at would be search frequency; are people conducting as many searches on Google as they used to? pic.twitter.com/S9iBXR4biP

In this evolving scenario, marketers find themselves at a crossroads. The ability to influence or monitor brand mentions in these AI-driven dialogues is still in its nascent stages. Consequently, there’s a growing trend towards adapting marketing strategies for a generative AI world. This adaptation involves a strategic reliance on traditional media in the short term, leveraging its reach and impact to build and sustain brand presence.

Simultaneously, we are witnessing a significant shift in the technological landscape. The move from browser-based tools to on-device applications is gaining momentum. Leading this charge are innovations like Microsoft Co-Pilot, Google Bard on devices such as Android, and the anticipated launch of Apple’s own large language model (LLM) sometime in 2024. This transition indicates a paradigm shift from web-centric interactions to a more integrated, device-based AI experience.

This shift extends beyond mere convenience; it represents a fundamental change in user interaction paradigms. As AI becomes more seamlessly integrated into devices, the distinction between online and offline interactions becomes increasingly blurred. Users are likely to interact with AI in more personal, context-aware environments, leading to a more organic and engaging user experience. For tech giants like Google, Microsoft, and Apple, already entrenched in the marketing services world, this represents an opportunity to redefine their offerings.

We can anticipate the emergence of new “answer analytics” platforms and operating models in marketing to support answer engine optimisation. These tools will likely focus on understanding and leveraging the nuances of AI-driven interactions but potentially better leverage the training data to understand how the results might be portrayed for a given brand or product.

Digital marketeers will start to think more deeply about how they are indexed in these training datasets same as they once did with search engines.

Moreover, the potential launch of ad-sponsored results or media measurement tools by platforms like OpenAI could introduce a new dimension in digital advertising. This development would not only offer new avenues for brand promotion but also challenge existing digital marketing strategies, prompting a reevaluation of metrics and ROI assessment methodologies.

As LLM’s migrate into devices, moving away from traditional web interfaces, the marketing landscape is poised for significant changes. Marketers must adapt to these shifts, leveraging both traditional media and emerging AI technologies, to effectively engage with their audiences in this new digital era. This dual approach, combining the impact of traditional media with the precision of AI-driven analytics, could very well be the key to success in the rapidly evolving marketing landscape of 2024.

10) The “Garbage In, Garbage Out” Dilemma Intensifies

As organizations increasingly pivot towards leveraging generative AI models and developing their own fine-tuned solutions, the spotlight falls sharply on the quality of input data. The classic saying in data management circles of “garbage in, garbage out” is bubbled up again as data quality is now back on the table.

Organisations and leaders confront the harsh reality that high-quality, accurately labeled data is the cornerstone of effective AI deployment. The issue goes beyond the obvious availability of data; it’s about its relevance, accuracy, and the context it provides. Issues of bias and misguided training data can spell disaster for the output of a model.

However, the challenges don’t stop there. Existing data and AI pipeline technologies, which once seemed adequate, are now being pushed to their limits. They are often found wanting in the face of the nuanced demands of advanced AI models. This gap between capability and requirement necessitates an evolution in data processing tools and methodologies.

The 2023 Data Integrity Trends and Insights Report uncovered poor data quality as a pervasive theme for organizations across the industry, with 70% of those with low levels of trust in their data pointing to data quality as the biggest challenge to making confident decisions.

Furthermore, the pursuit of AI excellence comes with its own set of logistical hurdles. The intensive compute power required for these sophisticated AI models translates to a substantial demand for GPUs.

But this isn’t just about having the financial muscle to invest in hardware; it’s about the availability of these resources in the market. As more players enter the AI arena, the scramble for GPUs intensifies, leading to potential bottlenecks in AI development and deployment which adds further complexity to the ability for organisations to adapt to the normal in AI.

As 2024 unfolds, we witness a renewed focus on data quality and infrastructure enhancement, shaping the trajectory of AI development.

11) Purpose Built Smaller Foundational Models Commonplace

2024 might very well be the year of the small foundational models. These specialized, purpose-built AI models are set to take center stage, outshining their generalized counterparts in efficiency and precision.

Organisations now have a number of options of using readily trained generalised large language models such as OpenAI GPT, Google Bard, Anthropic Claude [RL model in chart below] or venture into the world of building your own.

With options to fine-tune which is taking the base/foundational language model but teaching it new things (as you would imagine fine tuning a car to go faster) or go deep into creating your own foundational (base) models altogether.

With generalised models such as the GPT models with 100 billion parameters (or “switches” in its model programming) would cost roughly upwards of $4million to rebuild. OpenAI has also recently started offering a service for enterprise to “build your own GPT” for pricing starting from $2–3million.

Organizations that have either developed their own foundational models or fine-tuned existing ones to their specific use-cases are poised for success. This approach aligns with the 80:20 rule, where the focus is on smaller, tailored models that cater to specific needs rather than attempting to appeal to the masses with generalized solutions.

The true value for organizations lies in the ability to develop these purpose-built models for discrete tasks. Not only do these models offer higher accuracy and relevance, but they also present new monetization opportunities. In a world increasingly driven by specialized needs, these models become invaluable assets, offering solutions that are both effective and economically viable.

Moreover, the trend is shifting away from relying solely on large, general-purpose models as they are not quite perfect for every need. Many organizations have built solutions on top of these broad models, acting as “thin wrappers” that offer limited scope for customization and scalability. While these solutions may have been a stepping stone, they are unlikely to provide the long-term value that developing proprietary models can offer.

The winners in this evolving landscape will be those who invest in developing their own models generalised or small foundational models to plug gaps in the generalised space. This strategy not only increases accuracy and effectiveness but also reduces cost overheads. Smaller models are not only cheaper to run but also quicker to adapt and easier to manage.

This leads to the concept of “Total Cost of Modelling,” where the benefits of developing and maintaining these models outweigh the initial investment, offering a more sustainable and cost-effective approach in the long run.

As we look towards 2024, it’s clear that the ability to create and leverage small foundational models will be a key differentiator in the competitive AI market. This shift marks a significant move towards more personalized, efficient, and economically sound AI solutions.

12) The Dawn Of AI Marketplaces For Agents

AI marketplaces are emerging as adaptive and responsive platforms, reshaping the way we think about technology transactions and interactions.

Leading the charge, OpenAI is poised to unveil its much-anticipated “GPT marketplace” in early 2024, setting a new benchmark in the AI arena. This move is expected to open the floodgates, with other tech giants like Meta quickly following suit. We could witness an array of players, from established giants like Amazon, Apple and even Bytedance to emerging startups, diving into this space.

But the ripple effect of this revolution extends beyond traditional tech entities. With the growth of autonomous agents and the surge in wearable technology, there will be the Apple iPhone and App Store moment again. Developers will see this as the next gold rush of opportunity.

In this transformative phase, the AI marketplace phenomenon is expanding its reach from B2B to B2C sectors. We’re likely to see a diverse range of players trying their hand at this, each bringing unique value propositions to the table. From consumer-focused AI applications to enterprise-level solutions, the spectrum of offerings in these marketplaces will cater to a wide array of needs and aspirations.

13) AI Products Will Go Beyond SaaS Model

With the proliferation of AI marketplaces and tools, traditional pricing strategies are being re-evaluated, making way for innovative approaches that cater to the unique nature of AI services.

We are likely to witness a significant shift from the conventional app store pricing model to more dynamic, consumption based billing systems. These models, reminiscent of utility billing like phone lines, are poised to become more prevalent, aligning with the concept of renting AI agents. In this setup, customers would pay based on the extent and nature of their AI usage, offering a flexible and potentially more equitable pricing structure.

But the evolution doesn’t stop there. The increasing adoption of AI marketplaces and tools is also paving the way for varied pricing strategies and novel business models. This change is driven by the need to accommodate a wide range of AI applications and services, each with its own value proposition and usage patterns. This could see the trial of both the revenue share (App Store) or the royalties on usage (Spotify) model for developers and their AI services or agents on marketplaces.

Another emerging model could be performance-based pricing, where charges are aligned with the outcomes or results delivered by the AI tool. Such a model would be particularly appealing in sectors where AI’s impact can be quantitatively measured, like in marketing analytics, financial forecasting, or even creative industries.

Furthermore, as AI continues to penetrate various sectors, cross-industry partnerships could give rise to bundled services. These bundles could combine AI tools with traditional software services, offering a comprehensive package that addresses a wider array of business needs.

The onset of these new pricing models and strategies reflects a marketplace that is rapidly adapting to the unique challenges and opportunities presented by AI. As businesses and consumers alike become more familiar with AI capabilities, the demand for flexible, transparent, and value-aligned pricing models will likely intensify.

14) BYO AI Movement Pushes Need For Secure Digital Identities

Expanding modern and generative AI tools will lead the expansion of digital footprints necessitating secure, portable digital identities, where the challenge is to balance robust security with user accessibility. Users will expect a personalised experience where preferences, history and context will be key to using many AI services across the web.

Banks and e-government platforms are emerging as potential custodians of these single digital identities and personal preferences. This consolidation points towards a streamlined, more secure digital existence. But it’s not just about security; it’s about the seamless integration of our digital selves across various platforms.

“Bring Your Own AI” (BYO AI) ties directly into this. Imagine carrying your digital preferences, learning styles, and even shopping habits seamlessly from one digital interaction to another. This portability isn’t just convenient; it’s transformative. It allows for a level of personalization and efficiency previously unattainable. Wearable are also becoming integral to managing our digital identities. By constantly learning from our interactions, they evolve into personal data hubs that not only understand our preferences but anticipate our needs.

The integration of AI into work environments means that our digital preferences could automatically adjust settings in office applications, communication tools, and even physical workspaces. Imagine entering a meeting room where the lighting, temperature, and even digital displays are automatically tailored to your preferences.

However, this level of personalization and data integration raises questions about privacy and data usage. As these digital identities become more intricate and intertwined with AI, the potential for them to be leveraged for advertising digital experience providers for hyper-personalization is significant. This could lead to a new era of contextual advertising and consumer engagement, where promotions are not just targeted but deeply integrated into our digital personas.

This integration of identity with AI will redefine how we interact with technology, both in personal and professional spheres, leading to a more personalized, efficient, and connected existence. As we embrace this future, the importance of ethical considerations and privacy safeguards becomes more crucial than ever.

Conclusion & Key Takeaways

As we look towards 2024, the potential of AI and technology to reshape our world is undeniable. Each of these predictions offers a glimpse into a future where innovation, responsibility, and inclusivity go hand in hand.

Key takeaways:

1. Generative AI Emerges as a Core Technology Strategy: Marking a shift from hype to mainstream adoption across various sectors.
2. Advancements in Neural Networks Edge Closer to AGI: New architectures like Mamba and neuro-symbolic AI enhance cognitive capabilities significantly.
3. AI Wearables and Extended Reality (XR) Gain Prominence: Offering augmented human experiences and interactions in daily life.
4. Interconnected AI Agents Create a New Ecosystem: Specialized AI agents communicating with each other revolutionize industry dynamics.
5. Secure Digital Identities Become Crucial in the BYO AI Era: The rise of portable digital identities managed by banks and e-government platforms.

Let’s embrace this journey with open minds and hearts, ready to be part of a future that’s not just happening but is ours to shape. Join the conversation, share your insights, and let’s collectively envision and build the world of 2024.

Enjoyed This Story?

Vincent Koc is a highly accomplished, commercially-focused technologist and entrepreneur a wealth of experience focused in data-driven and digital disciplines. Currently, Vincent serves as a data leader in Australia as well as a lecturer for artificial intelligence in the US.

Subscribe for free to get notified when Vincent publishes a new story. Or follow him on LinkedIn and X (formerly Twitter).

Get an email whenever Vincent Koc publishes.

Unless otherwise noted, all images are by the author

Navigating the AI Landscape of 2024: Trends, Predictions, and Possibilities was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

Part of the “Decoding the Future” Series

After delving into the groundbreaking ideas presented at SXSW Sydney in our last installment, we now turn our attention to the past — to what pioneering thinkers like Vannevar Bush can teach us about whats next in Generative AI. Decoding the Future is a content series I launched exploring ideas and trends right here in the inaugural SXSW Sydney. Follow me on Medium to follow the conversation.

Glimpse into Cybernetics

Standing out among the luminaries on Monday’s sessions at SXSW was Distinguished Professor Genevieve Bell AO, an anthropologist, and Director of the School of Cybernetics at ANU. Her thought-provoking talk guiding us through a journey into cybernetics urged us to examine not just technology but the intricate “systems” that birth them. This piece aims to extend Bell’s narrative, tying her reflections to the visionary concept of the memex and the next frontier — Generative AI.

The Birth of an Idea

In 1945, Vannevar Bush, an American engineer and inventor, took to paper “As We May Think” outlining the memex — a mechanical library capable of storing, correlating, and retrieving information. Though never realized, the memex’s principles continue to echo in today’s world of technology and artificial intelligence.

Bush summed it up very well in these few lines which could apply just as easily today to Generative AI technologies:

Consider a future device … in which an individual stores all his books, records, and communications, and which is mechanized so that it may be consulted with exceeding speed and flexibility. It is an enlarged intimate supplement to his memory.

He conceived the memex as a vast repository of human knowledge. Though we never saw it in material form, its concept reverberates through today’s technological landscape.

Crossroads between 1945 and Today — The Evolution from Memex to AI

AI as an Index of Knowledge

Bush aimed for a memex that could compile an index of human knowledge. Fast forward to today, Generative AI serves a similar purpose but leverages algorithms, not gears and levers.

AI Augmenting Our Cognition

Machine augmentation serves as an extension of human intellect. Generative AI realizes this dream, aiding us in creative thinking, problem-solving, and even emotional understanding. These systems are not mere tools; they are thought enhancers.

What’s Next for AI and Technologists?

AI Tailoring Rich Experiences

Going beyond personalisation; the memex was idealised as this customisable, adaptive system. The spirit of personalization is not just alive but thriving, adapting to individual needs like never before.

Imagine if the AI could not only provide a response but act on your behalf, aware of your data, current state and go beyond the current “human and text” interface. We will see more platform ecosystems take shape around AI, connecting to the world and data around us personally and start to shape this into a reality.

Cybernetics and Why Systems Thinking Matters

We must regard technology as a “system” that goes beyond the hardware, software and AI models. Genevieve Bell’s perspective is a call to arms: are we creating systems that are holistic, adaptable, and designed to outlive us?

The future promises astonishing technological leaps, but how many are designed with a legacy in mind like Bush? It’s not just about innovation; it’s about creating systems that stand the test of time.

In Summary

As technologists navigating the future with AI requires some visionary thinking and foresight but there are some practical steps we can take to navigate tomorrow:

1. Never Forget to Look Back — The problems of yesterday are the problems of today.
2. Craft a legacy — The systems we are building today could out live us. Think about passing something on to continue on its journey of evolution.
3. Rethink personalisation —Consider the modalities of human-machine relationships; what and how would you want that relationship to be and work.

The modern school of cybernetics is alive and well, it’s now up to us to shape the future use of AI. As we tread forward, let’s continue to decode the future by echoing the past.

For those interested and based in Australia. The Australian National University is running a number of Cybernetics micro-credentials which you can use to further your journey into this field.

SXSW (South by South West) was at its first-ever event outside of Austin, Texas since its inception in 1987 to Sydney, Australia — a place that celebrates the oldest continuous culture in the world. As someone who’s deeply vested in data and technology, I couldn’t pass up the opportunity to attend. The opening keynote featuring Adam Spencer and Amy Webb was nothing short of an alternative thought into the future, a great way to kick off the opening keynote for the week long festival of ideas.

Listen into SXSW Sydney

Throughout this event, I’ll be curating my thoughts, scribbles, takeaways, and memorable one-liners into a series of articles. My aim is to extend the conversation beyond the walls of the conference, in line with SXSW’s mission of fostering idea exchange and community connection. With over 400 sessions and more than 800 speakers, this SXSW aims tackles some of the most pressing issues we face today: the integration of AI in society, sustainability, generational change, wellbeing and mental health.

Futurism: More Than Just Trends

Amy Webb, a quantitative futurist, author, and CEO of Future Today Institute took the stage to go beyond mere trends. “Focusing on one trend will make you lose sight of the big picture,” she said. Webb’s approach to futurism isn’t just creative; it’s grounded in game theory and economics. She argued for a nuanced understanding of trends, advocating for strategic foresight rather than reactionary planning. Looking beyond the macro and micro factors but the convergence of trends.

She regularly shares at SXSW Austin a talk that captivates the delegates with a glimpse into the future with the emerging tech trend report [You can see the 2023 talk earlier this year]. Here were some of the outlooks she presented at Sydney:

Key Outlooks

1. The Generative AI Debate: Webb raised concerns about imprudent investment in Generative AI due to FOMO (Fear of Missing Out) by those working with technology. She cautioned against leaping before looking when it comes to AI adoption, a sentiment that resonates deeply with those of us in the data industry.
2. Search Paradigms Are Shifting: The keynote touched upon the profound change in how we will interact with the internet. Soon, “we won’t search the web; the web will search us”. This shift has far-reaching implications, particularly when it comes to data ethics. Possibly marking the end of the internet we know today.
3. Two Possible Futures: The speakers presented an optimistic and a pessimistic view of a “Post Generative AI” world, challenging us to consider the potential impacts of rapid technological advancements. Will we have agency over our own data, or could we dive into a dystopian future.
4. The Role of AI is Assistive: They likened modern AI tools to calculators or word processors — tools designed to augment human capabilities. The question remains: How reliant should we become on these technologies? How do you strike a balance of utility vs creativity?
5. The Future of Employment: Despite potential short-term job losses, Webb remains optimistic about long-term job creation. She highlighted the need for education systems to adapt to this ever-changing landscape.

Closing Thoughts

As we navigate this new era of AI, data, and technology:

• Strategic foresight needs to be balanced with cautious, pragmatic decision-making.
• Ethical and inclusive approaches are non-negotiable.
• Change is the only constant, but it’s not necessarily a bad thing.

SXSW Sydney will hopefully create a hotbed for ideas and solutions to challenges ahead. As we move forward, let’s not just aim to survive the future, but to shape it.