It all started with a few weirdly obsessive and slightly idiosyncratic questions about my daily life and habits. Questions like: “How often do I cry?” and “Can I predict when I’ll get sick?”

I started collecting personal data in early 2022 in the form of daily survey questions. This practice has evolved as I’ve iterated on everything from the survey content (quantity, specificity, and breadth of questions) to the tools I use.

In this article, I want to share this journey and what I've learned along the way.

Subscribe now

Evolution of the Survey and Data Collection Process

Each year, I update my personal data collection survey. Here are some patterns I noticed over time:

• Each year, I collected more data. The number of overall questions increased every year, peaking at almost 40 questions for the 2025 Survey. Interestingly, this number dipped a little for the 2026 Survey, so maybe somewhere between 30 and 40 is a sweet spot.

• Questions became more open-ended and less structured. I originally started with a lot of structured data — that is, questions for which you pick one of several pre-defined options (e.g. “Which of the following exercise options did I do?”, “Did I cry? Yes/No”), or specify a numeric amount (e.g. “How many hours of sleep did I get last night?”). As the years went on, I began to introduce more and more unstructured data, or open-ended questions that can be answered using natural language (e.g. “Which friends did you see today?” or “What is one thing that made you feel alive today?”)

• Questions became more diverse. More surface-level questions (e.g., which exercise, whether or not I cried) evolved into deeper ones (e.g., how did I feel, why I cried).

• I iterated on the medium based on personal preference. I experimented a lot with different tools to collect my data. I first started with Google Forms because it was the most convenient, then I realized while camping for a week with no Internet that I couldn’t load the form on my phone. That gave me a lot of (maybe unnecessary) anxiety, so the year after, I switched to Jotform so that I could fill out the form offline. A few years later, I switched to Airtable because I ran out of storage on Jotform and I liked its customizability.

Evolution of the Data Analysis Process

When I first started analyzing my habits in 2022, I was obsessed with understanding my habits and patterns, such as when I cry, what makes me feel alive, and how my body signals illness. I was eager to dive deep into doing my own data analysis. I collected datasets from multiple sources, merged and cleaned the data, extracted salient features, and ran (simple) regression experiments. I even trained my own word embeddings at one point.

Last year in 2024, I began experimenting with dumping my data directly into AI tools (e.g. ChatGPT, Claude, Gemini) and having them do the data analysis part with me. I thought the tools had become good enough at coding and data analysis skills to entrust them with this task. What I found was that all of the major AI tools exhibited an (unhealthy) amount of hallucinations. My conclusion last year was that while it was an enlightening experiment, that it was probably safer to trust these tools with a grain of salt and to do some verification before blindly trusting results.

This year, I tried that process again, and as cliche as it must sound by now, I am astounded at how much these tools have improved (even as someone who works on improving Gemini for my day job!).

Now, not only can the tools do the data analysis, they can create beautiful HTML pages with insights, trends, and analysis points. Let me show you.

Some AI-Generated “2025 Wrapped” Reports from This Year’s Data

I kept things relatively simple compared to previous years in terms of data preparation. I only used my Apple Health and 2025 Survey data, and did some very basic data cleaning, such as manually fixing dates, merging data sources, and minor feature extraction.

I have to admit, I didn’t expect the results to be this good. I simply cleaned up my data and dumped it into Claude/Gemini/ChatGPT (I tried all 3). Each tool generated a stunning HTML report, complete with visualizations, trend analyses, and insights. The AI tools were particularly good at analyzing unstructured text data, finding patterns and insights that sometimes delighted and sometimes surprised.

Below are a few screenshots from some of the different AI-generated reports, each containing interesting insights and sometimes, a call to action. The reports varied based on the seed questions I asked the AI tools to focus on, such as my emotional state, health, or my general well-being.

Subscribe now

Share

Why You Should Start Collecting Your Own Personal Data

Compared to any time in the past, it is now incredibly accessible for any lay person to analyze their own data, even if you don’t know a single thing about Python, R, Jupyter notebooks, plotting libraries, word embeddings, or statistical analyses.

The real decisions are now around what data you want to collect, how much of it, how often, and in what form to best convey your actual lived experience. The AI tools at our disposal are so good at coding and visualizing that all you really need to do now is:

1. Collect data that matters to you

2. Think of good questions to ask about your own patterns and behaviors

There’s something uniquely valuable about self-reported survey data because it’s your own ground truth. When you report that you cried on a particular day, there’s no ambiguity, no inference needed. While fitness trackers and search histories can offer educated guesses about your state of mind or behaviors, survey data is unequivocal.

But more than that, collecting your own data means you have ownership over these insights. You’re an active participant in understanding yourself, not a passive subject of someone else’s analysis. Companies are already collecting data about you and drawing conclusions about your actions and habits (and sometimes those insights seem eerily accurate).

But the best insights come from explicitly writing your own story, which I encourage you all to reflect upon in the new year 😊

An important caveat: I dumped subsets of my personal data into public AI tools, which isn’t the most privacy-conscious approach. If you’re collecting sensitive personal data, consider using local analysis tools, API versions with better privacy controls, or be thoughtful about what you share. For me, the insights were worth the tradeoff, but that’s a personal decision everyone should make consciously.

I am obsessed with data (which is how this blog started in the first place). In particular, I am obsessed with my data — collecting, analyzing, and understanding different facets of my personal data. This includes data about my exercise, health, sleep, movement, location, and screen time usage.

It’s become tradition to analyze the previous year’s data at the beginning of a new year: I analyzed my 2022 data to investigate my crying patterns, and my 2023 data to understand my patterns of sickness and wellness.

This year, I was feeling a little lazy getting started on analyzing my 2024 data. Then I figured that it was a great time to try out all of the advanced reasoning AI systems that have been coming out. In case you haven’t heard of them, models like Google’s Gemini 2.0 Flash Thinking, DeepSeek’s R1, and OpenAI’s o1/o3 are trained “to ‘think harder’ when tackling complex challenges” (OpenAI) or “to generate the ‘thinking process’ the model goes through” (Google). Other models, such as Google’s Deep Research and OpenAI’s homonymous Deep Research, have been designed to conduct extensive research projects.

In this article, I’ll walk you through the entire process, from data collection and curation to having the different AI systems analyze my data, and my meta observations on using such systems to be your personal data science assistant. Through this entire process, I try to answer the question: How good is each AI system at an ambiguous, open-ended data analysis task?

Note that all opinions in this article are strictly my own and do not represent views of my employer or any other companies mentioned.

Share

Data Preparation

Step 1: Download and clean datasets

I first downloaded and cleaned my data from different sources and merged them into one single file, which is what I gave to each AI system. I could have provided each AI system with multiple files (e.g., one for exercise data, one for survey data, etc), but I wanted to control as many variables as possible. This approach allowed me to test each AI system's ability to analyze data without them getting tripped up on the data cleaning part. This was especially important since much of the data cleaning relied on my own knowledge of which variables were important, specific ways I wanted certain columns cleaned, and which elements were unimportant or noise.

It helped that I already had a lot of the framework set up from previous years. In this way, I combined data from:

• Apple Health, which included fields such as step count and heart rate

• Screen time data, which included number of seconds spent on different applications across phone and computer

• Survey data, which I fill out at the end of every day to track aspects of my life that wouldn’t be captured automatically, such as how often I drank alcohol, washed my hair, or called my mom

Side Note: Normally, I'd use Google location data as well, but this year I couldn't because of recent changes to how Google Maps handles Timeline data. Google now auto deletes Location History data after a certain period, unless explicitly configured otherwise. I hadn't realized this change, and thus had less data available than in previous years. For anyone using location data similarly, I recommend you turn this feature on and also back up your location history.

Step 2: Describe the dataset in as much detail for the AI system

Some of the columns in the final dataset were pretty self explanatory (e.g. “Step Count” or “Mean Heart Rate”). However, others merited a bit more description, such as which fields were automatically or manually collected, or why it mattered. The more descriptive I could be, the better the AI system could unambiguously understand each data field and make fewer assumptions.

Step 3: Ask each AI system to analyze your data

I used the following AI systems in my experiments: ChatGPT 4o, Claude 3.5, Claude 3.7, and Gemini 2.0 Flash. I also included OpenAI Deep Research and Google Deep Research systems.

Note: I refer to all the systems under test as “AI systems” as it is possible that under the hood, some are not single models.

For each AI system, I attached the data as a single CSV and asked the question: Can you analyze the data and give me a summary of the main patterns and trends you see and also give tips/suggestions for the new year?

Subscribe now

Summary of Findings

The main finding of all of this is that none of the current AI systems are quite ready to replace me in the job of analyzing my own personal data (so far 🌞).

In the ideal scenario, I would give an AI system my dataset with a high level description, and it would in turn generate me a nice report showing highlights from the year and actionable items to improve the upcoming year. All of the AI systems I tested fell pretty short of that (admittedly difficult to measure) bar/vibe check.

The thing is, all of the insights shared by the AI systems looked great. At first glance, I found myself nodding along to whatever insights, recommendations, and numbers they shared. But once I started digging into my own data in parallel, I started finding various problems: the numbers of figures shared didn’t make sense, some of the fields were made up, and the plots created didn’t make sense or felt kind of useless. I’ll dig into some of these in the next section.

Deep (research) hallucinations, from making up new fields to falsifying values

Deep Research systems are tailored to be good at deep research projects, things like planning travel or doing research based on information on the Web. In fact, I thought of them (both Google’s and OpenAI’s) as the AI systems most capable of summarizing and synthesizing large amounts of information.

However, as I found out through this exploration, they are not very good at doing deep data science research on a single dataset.

Making up fields. OpenAI’s Deep Research claimed that I had the best mood during my follicular phase. While I am sure this was true in some ways, it hallucinated the “mood” field, which is not something I tracked in my data at all.

Making up habits. Google Deep Research also included some hallucinations, such as claiming that I was consistent with writing my morning pages, but I know (without even having to look at my data) that this is not true, as I stopped writing morning pages halfway through the year at all.

Making up numbers. OpenAI’s Deep Research observed that I spent an average of 3.3-3.5 hours per day on my phone, with higher screen time in colder months, which it attributed to my being indoors more and being on my phone more. Sounds reasonable, right?

But when I calculated this value from my actual data I got very different numbers. The reality (according to the data) is actually an average of 1.5 hours per day, with no noticeable difference among the “colder months”. In fact, it looks like my phone usage was below-average in the colder months like November, December, and March.

Addendum: The above OpenAI deep research results were tried using GPT 4o. I tried ChatGPT 4.5 with deep research after this article was written, as it was only made available to me very recently. I found that it was much better about the hallucinations.

Running code behind the scenes helped to reduce hallucinations… sometimes

Some of the AI systems ran code (usually either Python or JavaScript) behind the scenes to actually “analyze” my data. In theory, this should minimize any data-related hallucinations in the resulting output.

In many cases, running code did result in correct numbers. For example Claude 3.5 observed, “You maintain a good activity level with an average of about 15,900 steps daily”. GPT-4o similarly observed, “Your average daily step count is quite high (~15,921), indicating good activity levels.”

I double checked this in the data and they were both indeed accurate – which was nice to see that none of the AI systems were hallucinating something as simple as my step count.

However, just because the AI system ran code didn’t mean it wasn’t hallucinating. For example, Claude 3.5 created the following plot summarizing my “wellness achievements”. In the plot it shows that I was in nature for 363 days in 2024. This is just not true, although I wish it were.

Some of the generated plots were not very useful

When the AI systems were not making up data fields and numbers, they were able to create some plots. However, just because it was able to generate code and create a plot did not mean that it was useful.

This is getting a little into “vibe check” territory, but I guess at the end of the day these AI systems still lack the insight into what plots might be more meaningful or helpful to users, often generating visualizations that are not especially useful or informative.

For example, GPT-4o generated the following plot to show my anxiety (which I logged as either Yes/No). I think this plot is difficult to read and is not very informative at showing which time periods I tended to have anxiety. Because anxiety was logged as a binary Yes/No choice, this sort of data would have been better displayed using some sort of line or bar plot aggregated over a weekly or monthly time period.

In another example, Gemini 2.0 Flash generated this plot of resting heart rate vs. mean heart rate. While interesting, I am not sure what I am supposed to do with this plot, as it did not come with any interpretation. Is this normal? Is there a certain area of the plot I should be paying more attention to? Why should I care about this?

Claude 3.7 generated the following plot showing several habits by season. While creative, it is difficult to read, and the axis is confusing. The more I stare at this plot, the more I feel like I am being sucked into a vortex.

Some of the insights and recommendations were very generic

I found that a lot of the insights and recommendations shared by some of the AI systems did not feel personalized to the data I shared. For example, “Increase consistency in meditation” (GPT-4o) or “Explore stress management techniques like mindfulness, exercise, and setting boundaries” (Gemini Flash 2.0).

Yes, I know these are good things to do. They’re just very generic and the AI systems could have told me these things without having looked at any of my personal data.

Redeeming qualities

I’m not here to just fully tear apart all of these systems. Being able to automate a data science project is not easy, especially if the project is pretty open-ended. Some analysis is quite straight-forward (e.g. “What months were step count higher in?”) whereas others are a lot more difficult, ambiguous, and open-ended (e.g. “Which of my habits were most correlated with my anxiety?”).

Overall, it was really cool to see that these various tools could go through the full end-to-end process — from loading in the data, crunching (some) numbers, creating plots, and generating recommendations. The output I got resembled what I was looking for. It was only once I started digging into things that I realized that some of the numbers were made up, or that some of the generated insights were generic or not actionable.

Plots were often interactive — you could often hover over different portions of the chart and it would show the number corresponding to that area. From an aesthetic standpoint, I thought Claude had the prettiest plots.

Some closing thoughts

Overall, it was interesting and enlightening to try out all of the different advanced capability AI models on my personal data and compare them based on (quantitative) numbers and (qualitative) vibes.

If you wanted to try this on your own, you have to be okay with these companies having access to your personal data. I think this is another important reason to be more hands-on in the data cleaning part, to make sure if you have extra sensitive things, you can remove those before sharing them with the AI system.

With today’s tools, if you wanted to use your favorite AI system to do data analysis for you, I recommend simpler, less open-ended, or well-defined tasks, and maybe verifying across multiple tools to ensure accuracy.

I’m going to keep collecting my data this year. These AI tools are developing very rapidly, so it’s very likely that the tasks they struggle with today will be easily solved within a few months. We’ll see at what stage these tools are in at the beginning of next year, when it’s time to analyze my 2025 personal data!

Two years ago on Valentine’s Day, I wrote an article titled “Love Language Models”.

In the article, I ran a few experiments, which involved having GPT-3 (the notorious text-davinci-003 which has since been deprecated) take the popular 5 Love Languages quiz in two modes:

• A normal GPT-3 model

• A GPT-3 model explicitly reminded that it was a Large Language Model, which I called “GPT-3 Self Aware”

I did some fun analysis like comparing GPT-3’s answers to US adults’ love language preferences, which resulted in a few interesting findings:

• GPT-3’s breakdown among the five love languages was almost evenly distributed, with each of the five languages hovering around 20%. In comparison, US Adults’ preferences varied a lot more.

• Self-Aware GPT-3 (explicitly told that it was an LLM) was half as likely to prefer “physical touch” as a love language.

At the time, I was not really trying to get into any philosophical discussions about what it means for an LLM like GPT-3 to “understand” love or to have “preferences” for love languages, like a human would. My experiments were mostly whimsical approaches to get a sense for how models like GPT-3 would respond to questions about human love and human relationships, especially when compared to human responses.

But … humans are falling in love with AI

You might have seen the following New York Times article from January 2025, describing a 28-year-old-woman who has fallen in love with her AI boyfriend.

While sensationalist, this phenomena is not new. Four years ago, a user posted on Reddit about falling in love with their Replika, an AI companion chatbot.

“Is falling in love with an AI good for my mental health?”, the Reddit user asks.

And perhaps this is a question more and more people will be asking as these AI companions become more empathetic, more personalized, and more present beyond just text (e.g. in video and VR).

The idea of humans falling in love with AI has long been a staple of science fiction, explored in movies like "Her" and "Ex Machina." But with the recent advancement in conversational AI and virtual companions, this once-fictional concept is becoming more of a reality.

Just to list a few examples (some of which I explore more in depth in my earlier article on AI avatars) —

• A 36-year-old woman (virtually) marries her AI boyfriend.

• A Snapchat user falls in love with Snapchat’s My AI.

• A Japanese man falls in love with and marries their AI robot girlfriend.

• A 39-year-old man falls in love with an AI chatbot from Paradot, an AI companion app

How do we understand this?

In fact, it is not new for humans to form emotional bonds with technology.

The field of affective computing, pioneered by Dr. Rosalind Picard in her 1997 book Affective Computing, strives to teach computers to understand and respond to human emotions. Dr. Picard argues that for computers to be truly intelligent and interact naturally with people, they need the capacity to recognize, comprehend, and even exhibit emotions.

In the realm of LLMs, benchmarks for measuring emotional capabilities exist (such as EmotionBench). However, I don’t see these capabilities being emphasized as much as more easily quantifiable capabilities, such as mathematical reasoning or coding, which are often valued indicators of a model’s progress.

However, understanding the affective side of AI is important, from chatbots to more embodied forms of AI companions. We need to better understand the relationships people form, not just when they fall in romantic love with an AI companion, but also when they form strong emotional bonds and attachments towards them.

Over the past two years, there have been several disturbing reports of users forming close bonds with AI chatbots, which in turn have encouraged the users to harm themselves (see here; here; here). While these are dark subjects, they are important to consider as people increasingly use AI for emotionally vulnerable topics like therapy (just a simple search for “AI therapist” will yield quite a few results).

I’m not saying that falling in love with an AI is inherently good or bad. However, people are clearly forming emotional attachments to these systems, and that phenomenon raises questions we can’t ignore. As AI becomes more intertwined with our daily lives, we need to keep pushing for a deeper understanding of what these emotional connections really mean—for our well-being, our relationships, and society as a whole. If we’re going to embrace AI as a companion, we also have to be prepared to set boundaries, demand safeguards, and continue refining our benchmarks to measure not just raw computational skills, but also how these technologies interact with our emotional lives.

Subscribe now

Share

Intro

As 2024 draws to a close, I want to reflect on the events in AI this year and share my thoughts on what lies ahead.

Looking back to the start of the year — to the projects I was working on, the papers I was reading, the topics going viral on Twitter, the headlines in the news, and all of the things (AI-related) I was spending my time thinking about — I realize how much this space has changed!

Two weeks ago, I attended NeurIPS, one of the largest AI and machine learning conferences, with over 16,000 attendees this year. The sheer breadth and depth of research there was overwhelming, and it was exciting to meet many of the people behind the papers I’ve read.

I’ll cover some key takeaways later, but I left the conference with three main thoughts:

1. How much progress there already is: The research at NeurIPS represented only a small subset of published AI work, not counting other conferences, unpublished work, or industry research.

2. Most of the research hasn’t reached the public yet in terms of applications, implications, or general awareness.

3. A portion of the research is already outdated.

How can research be so new that most people don’t know about it, yet already outdated? This contradiction shows how fast the space of AI is moving. It also shows how much hype exists in some areas, while other areas receive little public attention.

The conference reminded me how vast this field really is. While wandering around the poster sessions (which spanned multiple rooms across multiple days), I was struck by how small (spatially and physically) my corner of AI was compared to the breadth and depth of others’ research.

My work focuses mainly on LLM evaluations and benchmarks, which is a tiny slice compared to the broader landscape: architecture optimizations, computational biology, neuroscience, robotics, physics-based modeling, and many more domains filled with unfamiliar terms and acronyms. While LLMs still dominated in terms of popularity, the conference was a great reminder that LLMs are only just one small subfield of AI.

Reflections on 2024

In 2024, I spent most of my time working on and thinking about generative AI, LLMs, and benchmarks and evaluations. From that perspective, here are some of my reflections:

A flourishing of smaller models.

This year, the gains from ever-larger models began to show diminishing returns. Larger isn’t always better, and we saw a surge in smaller yet high-performing models, such as Google’s Gemma (available in 2B and 7B parameters), Microsoft’s Phi-3-mini (3.8B parameters), and GPT-4o mini. These models are fast, efficient, and much cheaper than their larger counterparts. Methods like knowledge distillation (training a smaller student model using a larger teacher model) and quantization (using fewer bits per parameter) have made these models more capable.

Reasoning models that use more inference compute: thinking and agentic models

Traditionally, most of the compute for an LLM was spent on training. Now, more and more compute goes toward inference (the part where you are actually using/calling the trained LLM).

A new paradigm is emerging where models use additional compute at inference time — for example, by using longer prompts with multiple examples or sampling the model multiple times and taking a “majority vote” answer.

The diagram below compares OpenAI’s o1 model (nicknamed “Strawberry”) to more traditional LLMs, but the concept refers to other such models in this space. (For those interested, see Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters)

This approach appears in two main ways:

1. Agentic Models: Here, inference-time compute scales up through AI agents, or LLMs that can call external tools (such as searching, browsing, running Python, or invoking smaller specialized LLMs). AI agents can orchestrate complex workflows, often through iterative model calls (like a loop with an LLM that calls search). I’ll talk more about agents later, but they often require more compute at inference time due to extensive instructions or iterative model calls.

2. Thinking Models: These models (e.g., OpenAI’s o1 and o3, Google’s Gemini 2.0 Flash Thinking Mode) are trained to output detailed reasoning steps before giving a final answer. They often produce long, meandering “streams of consciousness” before arriving at an answer.

More “well-rounded”, general-purpose models

Models are becoming more versatile. They handle multiple modalities and languages with improving performance. There are multiple AI systems now which you can speak to, and which will speak back to you in a natural way. You can snap a picture or livestream a video of your surroundings (like Google’s Project Astra, “a universal AI assistant”), and the AI system can describe what’s happening. Some of these systems can even code full web apps. AI is evolving into an ensemble of capabilities rather than a single-purpose model.

AI-generated videos

I actually can’t believe it was only February of this year that OpenAI released Sora, an AI model that creates video from text instructions. Since then, Meta released MovieGen in October, which not only creates video from text but also edits video and generates soundtracks. In December, Google released Veo2, a video generation model “which convincingly simulates real-world physics as well as a wide range of visual styles”.

The following is one of the videos created by Veo (which is available on waitlist on VideoFX).

I find the speed of this progress incredible and astounding. I find the speed of this progress incredible and astounding. If you haven’t yet, I urge you to check out the links and watch some of the videos for yourself!

Share

Looking forward to 2025

I’m both excited and apprehensive about the AI landscape in the coming year. Reflecting on the past year and attending NeurIPS inspired me with the sheer amount of research that happened in 2024. Here are some areas I’m paying close attention to for the upcoming year:

Development of more difficult benchmarks and evaluations for AI systems

While new AI capabilities are exciting, we need tougher benchmarks to measure progress (and, in some cases, regress). Models are improving so fast that popular academic benchmarks (like MMLU and MMMU) are quickly saturated or outdated. There is a constant need for more challenging and difficult evaluations.

Some benchmarks are still challenging for state-of-the-art models, such as:

• Frontier Math and AIME for complex mathematical reasoning.

• GAIA and SWE-Bench for tool use and agentic behavior.

• ARC-AGI for visual pattern matching and reasoning.

These evaluations can take the form of tasks that are easy for humans but still hard for models, like with ARC-AGI. For example, the following is an example from ARC-AGI that is pretty easy for most humans but that leading models like OpenAI’s o3 fails to answer correctly.

These evaluations can also take the form of tasks that are time-consuming or difficult for humans, or within the realm of “experts”, such as the difficult mathematical reasoning questions.

In the next year, there will be more evaluations that cover all of these grounds, and more: those that are easy for most humans but still hard for models; those that are hard for both humans and models; those that are more niche and expert-focused; and those that mix reasoning with multiple modalities (such as audio and video, in addition to text). Additionally, I believe there will likely be greater emphasis on assessing AI systems end-to-end — especially agentic and user-experience focused.

At NeurIPS, I met a physicist who was working on physics-based modeling, which is a totally different area of AI from me. When I told her that I was working on LLM evaluations, she said, “Evaluations? It’s so unscientific, I don’t know if we can even call it a science.” Hopefully, in the coming year, we will see more robust, statistically grounded approaches to evaluations (building upon recommendations from this November for making evaluations more statistically robust).

Continued efforts on agentic behavior

AI agents can be as simple as search-based chatbots or as complex as end-to-end automated coding systems. Some examples include PR-Agent, which automatically reviews code, and Lindy, which automates workflows such as replying to emails. While there’s plenty of hype, there’s also real substance here. I’m excited for the ongoing work on agents and real-world applications that solve real user problems that will continue to be developed in the new year.

This Github page contains a more comprehensive list of both open and closed source AI agent projects for those interested in a more comprehensive list. I also refer readers to Interconnects.AI’s article on the AI Agent Spectrum and The Batch’s article on agentic design patterns.

Ongoing questions about alignment and safety

At NeurIPS, several of the talks that I found the most compelling were centered around questions of “Who are we aligning the models to? Whose values, intentions, and preferences? Determined how and by whom?”

These questions have both technical and human-centered aspects. There are many open questions here about the best way for models to come to decisions, especially for controversial or open-ended topics which may have more than one right answer, or no right answer at all.

I doubt that any of these questions about alignment (e.g., whom to align with, whose values to follow) will reach a consensus anytime soon. However, I do look forward to more conversations and engagements from researchers, industry practitioners, and the general public around these questions.

Additionally, as AI becomes a mainstream technology used not just by those within the technology field but by all sectors and users, there will be increased scrutiny on safety and fairness. Real-world deployments can have tragic consequences, such as when an AI chatbot encouraged the tragic suicide of a teenager. These systems also affect different demographic groups differently, requiring more attention to bias and equity.

Many researchers are currently tackling safety from multiple angles: auditing datasets, improving training practices, and debiasing evaluation data. I expect more progress here, including best practices for end-to-end systems and effective mitigations.

Other areas

There are so many other areas I am also excited for, but if I wrote about them this post would get incredibly long: model interpretability (such as Anthropic’s use of sparse autoencoders), watermarking technologies (such as Google’s SynthID, which adds statistical watermarking signals to AI-generated content), and the ongoing convergence of generative and embodied models in robotics.

Share

Some final thoughts: making sure not to lose the “human” part of AI research

At NeurIPS, I learned about an organization whose goal is to automate science end-to-end. Their wet labs run on robotics controlled by an LLM agent. The LLM agent would eventually control the entire end-to-end process: generating hypotheses, running experiments, analyzing data, and writing papers. Humans would be used mostly for quality control. I asked if it wasn’t still important for humans to be part of the hypothesis process, and the representative disagreed, saying, “Should humans even be the ones asking the questions anymore?”

While everyone else around me was excited by these developments, I could not help but feel worried and afraid. Things started to feel a lot more dystopian to me. On one hand, it is not surprising that we are moving in this direction, what with recent developments such as Sakana’s AI Scientist, which uses foundation models to “automate the entire process of research itself”, and recent papers like Can LLMs Generate Novel Research Ideas?.

However, I worry for the day when humans won’t be part of these systems anymore. If humans are not the ones asking the questions, what are we really needed for?

This reminds me of debates about AI and creativity. Can AI truly match human creativity? For me, creative pursuits such as writing, music, and art reflect the human soul. I often think of Ted Chiang’s essay, which explores these themes, reminding us of the necessary human essence in creativity.

All of this to say is that, despite all of the progress that has happened and continues to happen in the AI space, it is important not to lose sight of the human within the system. Paraphrasing a quote by Professor Yejin Choi, AI should be built, not to be served by humans, but to serve humans.

Leave a comment

Introduction

LLMs are good at many things, and one of those things is playing games. People have used LLMs to play all sorts of games such as Minecraft, Chess, murder mystery games, Werewolf, and the NYT Connections puzzle. (For a more comprehensive list, you can refer to this survey.)

Most of the examples above show LLMs playing games either against themselves or against humans. But, how well do LLMs play games against other LLMs? 

In this article, I show the results of three different LLMs competing against each other in the popular board game, Codenames, which challenges players to find patterns among seemingly unrelated words.

Codenames

For those unfamiliar with it, Codenames is a board game created by Vladimír Chvátil.  The game pits two teams (typically Red and Blue) against each other.

Each team has a spymaster who gives single-word clues that are meant to point to multiple words on a 5x5 board of words. The other players on the team must then guess their team’s words while avoiding words that belong to the opposing team. Only the spymaster knows which words are assigned to each team.

For example, a Red team spymaster might offer the clue "WHITE 3," hoping that their teammates will select LIGHT, IVORY, and CHOCOLATE.

The aim is to give clues that allow your team to guess its words without mistakenly picking the other team's words, neutral words (the black words), or the Assassin word (the black word with the gray background). Choosing the Assassin word ends the game and causes your team to lose. The first team to guess all of their words wins.

AI and Codenames

Researchers and AI enthusiasts have already set up frameworks for “playing” Codenames using AI. The Codenames AI competition began in 2019 to test automated agents’ ability to play the game — several years before LLMs were even a thing. Word embeddings like Word2Vec and GloVe, used in those early competitions, were surprisingly effective at playing Codenames by simulating the clue-giving and guessing process. (For context, word embeddings are numerical representations of words, and are an early precursor to today’s LLMs.)

Recently, the competition was extended to include LLMs in Codenames AI for LLMs. In this setup, the LLMs on both teams come from the same base model (e.g. GPT-4 playing against GPT-4).

In my experiments, I decided to mix things up by having teams of one LLM play against teams of a different LLM.

Experiment setup

I compared three different systems: OpenAI’s GPT-4 (gpt-4o-2024-05-13), Anthropic’s Claude-3.5 (claude-3-5-sonnet-20240620), and Google’s Gemini (gemini-1.5-pro). (Note: I decided not to use the latest o1-preview model due to its higher cost.)

Each LLM combination played 24 matches against one another.

To keep things simple, I used the same prompt for all models and avoided any attempts to optimize prompts for specific models. I used the prompts and frameworks forked from the Codenames GPT repo. Each game was composed of four LLM instances — LLM1 Spymaster and LLM1 Guesser on one team, and LLM2 Spymaster and LLM2 Guesser on the other team.

Two turns in a single game might look like the following:

First, the Red team’s spymaster would issue a clue. Then, the Red team’s guesser would select words based on that clue. In this case, both members of the Red team are GPT-4.

In the next turn, the Blue team’s spymaster would issue a clue, and then the Blue team’s guesser would select words based on that clue. Here, both members of the Blue team are Claude.

Results

Who won the most matches?

The chart below breaks down the winners across 24 games for each LLM pairing. In general, GPT-4 and Claude were closely matched, though GPT-4 tended to win slightly more often . Both GPT-4 and Claude consistently outperformed Gemini.

In terms of total victories, GPT-4 and Claude won the most games.

Who chose the most Assassins?

In Codenames, the “Assassin” card should never be selected. If the Guesser accidentally chooses the Assassin, the game ends, and that team loses. So, the clues must be clear and unambiguous to avoid selecting the Assassin.

When a team picks an Assassin, it suggests that the spymaster and guesser aren't on the same page regarding the clues and the words on the board.

Selecting fewer Assassin cards is key to success. In my experiments, GPT-4 selected the most Assassin cards, while Gemini selected the fewest. This suggests that although GPT-4 won more games, Gemini may be more cautious with its guesses and clues.

Game lengths

LLMs took significantly more turns to complete a game of Codenames (up to 9x more) compared to traditional word embeddings.

Across all 3 LLMs, the models completed a game in an average of 13.5 turns. This number excludes games that ended prematurely due to an Assassin being chosen.

On the other hand, older word embeddings like GloVe needed as few as three turns to finish a game.  Other experiments showed that combining word embeddings with human judgments could complete a game in as few as 6 turns.

This suggests that while newer models are more sophisticated, they might be less efficient at this particular game compared to simpler, older methods. LLMs tend to play conservatively, with LLM-based spymasters often only providing clues for two or three words at a time. In contrast, humans and word embeddings might offer more abstract or creative clues that could end the game in fewer turns.

Share

Discussion

Codenames is just one of many games that people are experimenting with having LLMs “play”. We’ve gone from making LLMs do things that humans do for work (like coding or customer service) to, increasingly, making LLMs do things that humans do for fun (like playing games).

In my experiments, I explored how well LLMs perform against each other in a specific game-based setting. Games offer a way to test LLMs’ reasoning, strategy, creativity, and cooperation — skills we may want them to develop further to better assist us in other tasks.

I think we will likely see more of these kinds of game-based settings for testing LLMs. LMSYS’ Chatbot Arena is related to this idea, where two LLMs are judged by humans to see which gives the better response to a prompt — essentially, a game of “who gives the better answer to a question.”

AI agents also show promise in helping develop better games in the future, as seen in recent multi-agent frameworks for automating game development.

However, having fun (much like creating art) is a fundamentally human endeavor. I don’t want to get too much into the philosophical weeds about whether AI can have fun or create art or be creative. Ultimately, these games are meant for humans to enjoy, and AI can hopefully assist us in creating richer experiences without replacing the joy of playing them ourselves.

Leave a comment

I had a few questions I wanted to answer — such as, which country won the most medals if you don’t look at total medal count, but the percentage of medals won relative to how many athletes were sent? Were some sports dominated by certain countries? And how diverse are the winners for different sports?

This is part two of my recent data visualizations of the 2024 Paris Olympics, where I looked more at the participating athletes and countries. As before, the data is sourced from Kaggle.

Subscribe now

A closer look into medal count

Team USA left the Olympics with the highest overall number of medals, at 126 medals. China came in second, with a total of 91 medals.

But, wouldn’t it make sense that Team USA has the highest number of medals, since they sent the most athletes to the Olympics, more than any other country? (Team USA sent 638 athletes; China sent 399).

It might seem obvious, but here’s data to back it up: the more athletes a country sends to the Olympics, the more medals they win (sort of).

Let’s break it down a little.

It might be more informative to look at the relative medal count, rather than the absolute count. I calculated the percent of athletes who won medals (the number of medals won by a country divided by the number of athletes sent by that country). 

USA’s overall medal count does not seem quite as staggering in the context of the sheer number of athletes they sent. Around 20% of USA athletes won medals — which is still impressive, but it becomes more comparable with China: 22.6% of Chinese athletes returned with a medal.

Some countries received very few medals — absolutely and relatively. For example, Egypt only won 3 medals, despite sending 156 athletes, or 1.9% of their athletes. 

Other countries, such as North Korea and Kyrgyzstan, sent relatively few athletes (less than 20), but a high percentage of those athletes (over 30%) won a medal.

Merely sending a large number of athletes to the Olympics does not necessarily guarantee winning more medals. 

Teams vs Individual sports

Countries that send more athletes participate in more team sports, so one might argue that the above figures aren’t fair (e.g. an Olympic volleyball team consists of 12 players, but a winning team would obtain only a single medal). 

However, similar patterns as above hold when looking at medal counts only for individual sports. Individual sports are those labeled by the International Olympic Committee as such.

In fact, athletes from the United States won 19.7% of all medals, while this number went down to 17.3% of individual sports.

France sweeps BMX racing and Korea dominates archery

Some countries seem to be better at certain sports.

I determined countries that dominated a particular sport by first calculating the percent of medals won by each participating country in a sport, and then finding the countries that won medals with the highest percentage.

France dominated BMX racing, winning half of the medals. South Korea dominated archery (46.7% of medals) and China dominated diving, table tennis, and badminton. 

I also calculated the same metrics for a subset of the sports divided by gender. 

Just because a country is good at one sport for male/female athletes doesn’t mean it’s good at it for the other gender. France’s dominance in BMX racing was mainly for the men’s division. On the other hand, South Korea dominated archery for both genders, and China dominated table tennis for both genders (but more so for women’s!). 

Boxing and 3x3 Basketball are the most diverse sports

There are many ways to measure "diversity," but here I measure it in terms of how many countries won medals in a sport (proportional to how many athletes that country sent).

This approach allows me to see if medals are distributed among only a few countries or among a more diverse range of participants.

According to this metric, boxing and 3x3 basketball were the most "diverse" — nearly half of participating countries left with at least one medal.

This is in contrast to sports like road cycling, table tennis, and swimming, in which only 10% of participating countries won medals. The reasons for this disparity could vary widely, which I won't explore too deeply in this article, but they might include factors such as cost of equipment, availability of top coaches, and overall popularity of the sport.

I don't want to make any judgments about fairness or equality, but I partly wonder if the sports where a more diverse group of countries win medals are more aligned with the Olympic spirit of international competition (as opposed to single-country dominance).

A case study: is swimming the least diverse sport?

Swimming is the second most popular Olympic sport, with 853 participating athletes and 189 participating countries.

And yet, of these 189 countries, 19 countries won medals, meaning that only about 10% of countries walked (swam?) away with medals.

I don’t know much about swimming as a sport, but this surprised me a good deal, especially since it’s such a popular sport that so many countries participate in. Perhaps it speaks to exactly how competitive the sport is.

Share

Thanks for reading this article! If you enjoyed the data visualizations or have any questions, feel free to drop them in the comments!

Leave a comment

P.S. I was at the tail end of the Olympics in Paris this year, so it feels extra meaningful to be able to wrap up my experience there with some quick analysis of the games!

Maybe you already knew the answer to those questions, but I had no idea. So I downloaded data about the athletes participating in the 2024 Olympics from Kaggle and created the following plots to indulge my curiosity.

Let me share some of my findings with you.

The young ride skateboards. The old ride horses.

If we look at the 20 youngest athletes (ages 11-14) and the 20 oldest athletes (ages 55-69) participating in the 2024 Olympics, we can see that 60% of the youngest athletes compete in skateboarding while 80% of the oldest athletes compete in equestrian events.

The youngest Olympic athlete in 2024 was China’s Zheng Haohao, who competed in skateboarding at 11 years old. The oldest Olympic athlete was Australia’s Mary Hanna at 69 years old.

Other sports with older athletes include shooting and table tennis. Sports such as rhythmic gymnastics and freestyle BMX cycling are composed of younger athletes (barely anyone over 30).

Most countries participate in athletics, swimming, and judo. Few countries participate in water polo and basketball.

Athletics traces its earliest roots to ancient Greek Olympic events. Swimming has been a sport at every modern Summer Olympics. And while Judo has been part of the Summer Olympics since 1964, it still surprised me that so many different countries compete in this sport.

Some sports, such as breakdancing (which is debuting for the first time this year!), hockey, water polo, and basketball, have 16 or fewer countries participating.

A few countries send many athletes, while many countries send a few athletes

I sorted countries by the number of athletes it sent to the Olympics. While some countries send hundreds of athletes, there is a long tail of countries that send very few athletes. In fact, four countries (Nauru, Liechtenstein, Somalia, and Belize) only send 1 athlete.

Another way to highlight this disparity is using numbers: 100 countries sent 10 or fewer athletes. This is compared to the only 32 countries that sent 100 or more athletes.

Countries that send 10 or fewer athletes participate in individual combative sports

Countries that send 10 or fewer athletes participate in more individual sports, which might seem obvious. What surprised me was that these sports tended to be combative or martial: judo, shooting, taekwondo, boxing, and wrestling.

Countries that send 100 or more athletes participate in more team sports, such as hockey, rowing, football, handball, and volleyball.

Countries that send 10 or fewer athletes send more men than women

For the countries that send 10 or fewer athletes, there are overall more men (56.3%) compared to women (43.7%). This gender skew is obvious when compared to countries that send 100 or more athletes, for which the athletes are at near gender parity.

Only female athletes participate in Rhythmic Gymnastics and Artistic Swimming, while wrestling is 67% male

The discipline with the fewest women is Wrestling (32.53%), followed by equestrian and football.

Artistic Swimming and Rhythmic Gymnastics have no male athletes.

All other disciplines are almost at 50/50 gender parity.

Less economically developed countries send more male athletes than female athletes

I merged Gross Domestic Product (GDP) data (obtained from the United Nations Statistics Division), used as a proxy for a country’s economic health, with athletes’ country information.

I separated countries into quantiles based on their GDP and showed the median number of athletes sent by countries in those quantiles, disaggregated by gender. There is a trend of less economically developed countries (in the lower 20% quantiles) sending fewer female and more male athletes.

Larger and richer countries send more athletes

While it’s not surprising, it is important sometimes to show what’s clearly in the data — that larger and more economically developed countries do send more athletes. A country has to have both the economic resources and the population to be able to send more athletes to the Olympic games.

The Pisces are not diving and swimming, but rather playing badminton and rugby

Inspired in part by What Olympic Sport You Could Win Based on Your Zodiac Sign, I showed the proportion of Olympians with each Zodiac sign per sport. Larger circles correspond with a higher percentage of that Zodiac sign within that sport.

Everyone knows Pisces are fish, so the ideal sports for them might be swimming or diving. However, diving is dominated by Virgos, which surprised me, while the Pisces are busy playing badminton and rugby.

The Libras don’t seem to be really good at anything in particular.

I’ll let the astrology nerds dissect this plot further.

Share

But what about the medals? 🏅

In this article, I mainly focused on the athletes participating in the Olympic games.

Don’t worry — I’m already thinking of a Part 2, focusing on the athletes who won medals. What are some questions you’d like to have answered in the data? What else do you want to see? Let me know in the comments!

Leave a comment

Citation

For attribution in academic contexts or books, please cite this work as

Yennie Jun, "Visualizing Data from the 2024 Paris Olympics — Part 1", Art Fish Intelligence, 2024.

@article{Jun2024olympicsdataviz,
    author = {Yennie Jun},
    title = {Visualizing Data from the 2024 Paris Olympics — Part 1},
    journal = {Art Fish Intelligence},
    year = {2024},
    howpublished = {\url{https://www.artfish.ai/p/olympics-data-viz}
}

Introduction

The context window of large language models – the amount of text they can process at once – has been increasing at an exponential rate. 

In 2018, language models like BERT, T5, and GPT-1 could take up to 512 tokens as input. Now, in summer of 2024, this number has jumped to 2 million tokens (in publicly available LLMs). But what does this mean for us, and how do we evaluate these increasingly capable models? 

What does a large context window mean?

The recently released Gemini 1.5 Pro model can take in up to 2 million tokens. But what does 2 million tokens even mean?

If we estimate 4 words to roughly equal about 3 tokens, it means that 2 million tokens can (almost) fit the entire Harry Potter and Lord of the Ring series.1

These numbers refer to the context windows that are available in publicly available models. The Gemini 1.5 Pro model, while currently publicly available with up to a 2 million token context window, can work with up to 10 million tokens.

This means, as stated by a Reddit user, that one can fit 1000 scientific papers into Gemini’s 10 million context window to create novel research.

Why does this matter?

Larger context windows aren’t just a way for companies building LLMs to compete with each other. The implications and real-world scenarios of how models with long contexts can be used are many. Consider the following scenarios:

• Legal research: Lawyers can input entire case histories, precedents, and statutes into a model, getting comprehensive analysis in seconds instead of hours or days of manual review.

• Financial analysis: Imagine feeding years of financial reports, market trends, and economic indicators into an AI for instant, in-depth insights.

• Medical diagnostics: Doctors could input a patient's entire medical history, including test results, treatment records, and high-resolution medical imaging scans, for more accurate diagnoses and personalized treatment plans.

• Education: Students could input entire textbooks and course materials, getting tailored explanations and connections across subjects.

However, these use cases also raise concerns. The ability to process vast amounts of personal data could enable unprecedented levels of surveillance and privacy invasion if misused. As these capabilities grow, so does the need for robust ethical guidelines and safeguards.

Subscribe now

How do we evaluate LLMs as the context window gets longer and longer?

Models with extremely long context windows are a recent development. Therefore, researchers have tried to come up with new ways of evaluating how good these models are. These evaluations aim to benchmark the capabilities and limits of long context models, as well as measure the tradeoffs that come with scaling up context windows.

The core idea is that models with longer input contexts should be able to perform tasks that were previously too difficult or impossible. 

Evaluation Use Cases

In this article, I’ll cover three different ways that researchers are thinking about evaluating long context models:

1. Information retrieval from long documents

2. Complex Analysis (reasoning and summarizing) of Long Documents

3. In-context learning for "on the fly" model training

Note: This is not an exhaustive list. For a comprehensive overview of long-context benchmarks, refer to the Awesome LLM Long Context Modeling Github page.

Share

1. Information retrieval from long documents

The "Needle in a Haystack" test, introduced by Greg Kamradt, is a popular method for evaluating information retrieval in long contexts. It involves placing an out-of-place statement (the "needle") at various depths within longer text snippets (the "haystack"). 

This test measures how effectively LLMs can locate specific information within increasingly large contexts.

Variations on Needle in a Haystack

Researchers have developed several variations to test different aspects of information retrieval:

• Multiple needles: Multiple facts sprinkled throughout long documents (introduced by Langchain and in NeedleBench)

• Multimodal Needle in a Haystack: Finding a target image within a set of unrelated images based on a caption 

• Audio-based: Identifying a short audio clip within a signal up to five days long (introduced in the Gemini 1.5 technical report). In this test, a short clip of audio where a speaker says “the secret keyword is needle” was hidden within an audio signal of up to almost five days long (or, 107 hours).

• Video-based: Locating a single frame with specific text in a 10.5-hour video (Gemini 1.5 technical report). In this test, a single frame with the text “The secret word is needle” was buried within a 10.5 hour video constructed from concatenating seven copies of the full AlphaGo documentary.
  

Limitations and Implications

While widely used, the Needle in a Haystack approach has several limitations:

• It's an artificial task that may not reflect real-world use cases.

• It only assesses information finding, not reasoning or comprehension.

• As context windows grow, evaluating all combinations of "haystack" sizes and "needle" locations becomes increasingly costly.

Despite these drawbacks, the test highlights a crucial capability of long-context models: the ability to quickly search and retrieve information from vast amounts of data. This has significant implications, from enhancing research efficiency to enabling unprecedented levels of data analysis — and potentially, surveillance.

It's important to note that this type of information retrieval differs from Retrieval-Augmented Generation (RAG) in that it operates within a single, extensive context rather than retrieving information from external sources.

2. Complex Analysis (reasoning and summarizing) of Long Documents

While "Needle in a Haystack" tests focus on information retrieval, other evaluations assess an LLM's ability to reason over, interpret, and synthesize information from extensive content. These evaluations aim to test for more complex forms of reasoning rather than just pinpoint data location.

Here are a few evaluation methods in this category:

Literary QA Tasks

Books are prime examples of long documents. Benchmarks like NOVELQA test models' ability to reason over literary fiction, using up to 200K token lengths. It includes human-generated questions about 88 English-language novels, both public domain and copyrighted. Other datasets, like NoCha, follow similar approaches.

Reasoning over long texts with hidden relevant information

FlenQA creates multiple versions of contexts at different lengths by embedding relevant information within longer, irrelevant texts. This helps assess how LLM performance degrades as context length increases.

Domain-Specific Reasoning

• Healthcare: The LongHealth benchmark uses 20 fictional patient cases (5-7K words each) to test medical reasoning.

• Finance: DocFinQA challenges models with financial documents up to 150 pages long (100K+ tokens).

Summarization Tasks

Summarizing long documents is a crucial capability for LLMs, as it allows users to quickly grasp key information from extensive texts without reading everything. This skill is particularly valuable in fields like research, business, and law, where professionals often need to distill large volumes of information into concise reports.

However, evaluating summarization quality is challenging. Unlike simple information retrieval, summarization requires a deep understanding of the entire context and the ability to identify and synthesize key points. What makes a "good" summary can be subjective and context-dependent.

Current evaluation methods often rely on comparing model-generated summaries to human-written references, but this approach has limitations. It may not capture the full range of valid summarization strategies and can miss semantically correct summaries that use different wording.

Benchmarks like LongBench and ∞Bench attempt to address some of these challenges. LongBench includes summarization tasks for various document types (government reports, meeting notes, news articles) up to 15K words, while ∞Bench pushes the boundaries with novel summarization tasks up to 100K tokens. These benchmarks are valuable, but the field is still working towards more robust evaluation methods that can better capture the nuances of high-quality summarization.

A great resource to learn more on this topic can be found in An Empirical Survey on Long Document Summarization: Datasets, Models, and Metrics.

3. “On the fly” model training

One of the coolest applications of long context models is the expanded capacity for in-context learning (ICL). ICL allows models to learn new tasks on the fly, directly from examples provided in the prompt. With larger context windows, we can now include hundreds of training examples or even complex, lengthy examples like summarization tasks.

This capability is a game-changer. Instead of fine-tuning models for specific domains, one can leverage ICL to adapt models to new tasks instantly.

Many-shot ICL

DeepMind’s work on Many-Shot In-Context Learning showed that including many more examples in the prompt significantly boosted performance across various tasks. By scaling ICL to hundreds or thousands of examples, models can overcome pre-training biases and tackle more complex challenges.

This principle extends beyond just improving performance. Anthropic's work on Many-shot Jailbreaking demonstrated that while a few examples couldn't compromise a model's safety guardrails, hundreds of examples could — highlighting both the power and potential risks of this approach.

Translating low-resource languages

Long-context models are also proving particularly valuable for low-resource language translation. The Gemini 1.5 technical report showcased this potential with the Kalamang language, which has fewer than 200 speakers and minimal web presence. By inputting a 500-page grammar, a 2000-entry bilingual wordlist, and 400 parallel sentences (totaling 250k tokens), the model could translate and even transcribe Kalamang speech.

This approach extends to other low-resource languages too, with performance improving as more examples are provided. It's a promising development for preserving and working with endangered languages.

Share

Discussion

The race for longer context windows in language models is accelerating, with context window sizes growing at an exponential rate. This growth necessitates new evaluation methods to properly assess these models' capabilities and limitations.

While numerous benchmarks for long context evaluation have emerged (e.g., SCROLLS, LongBench, ∞BENCH), many questions remain unanswered:

• Trade-offs of scaling: How do safety, bias, and instruction-following change as context length increases?

• Multilingual performance: Most benchmarks focus on English (with the exception of benchmarks like CLongEval, which includes evaluation in Chinese as well). How does performance in other languages change with longer contexts, compared to English?

• Potential degradation: Do certain capabilities (like coding skills or creativity) suffer as models handle more context?

• Real-world implications: As models can process entire books, personal histories, or comprehensive data on low-resource languages, what are the ethical and practical consequences?

As LLMs’ context windows continue to grow, we need to understand not just what these models can do, but also how their fundamental characteristics may be changing. 

For now, the race towards models with larger and larger context windows will continue.

Leave a comment

Citation

For attribution in academic contexts or books, please cite this work as

Yennie Jun, "Evaluating long context large language models", Art Fish Intelligence, 2024.

@article{Jun2024longcontextllms,
    author = {Yennie Jun},
    title = {Evaluating long context large language models", Art Fish Intelligence},
    journal = {Art Fish Intelligence},
    year = {2024},
    howpublished = {\url{https://www.artfish.ai/p/long-context-llms}
}

How do language models handle conflicting instructions in its prompt?

Cognitive dissonance is a psychological term describing the mental discomfort experienced by an individual holding two or more contradictory beliefs. For example, if you’re at the grocery store and see a checkout lane for “10 items or fewer” but everyone in the line has 10 or more items, what are you supposed to do?

Within the context AI, I wanted to know how large language models (LLMs) deal with cognitive dissonance in the form of contradictory instructions (for example, prompting an LLM to translate from English to Korean, but giving examples of English to French translations1).

In this article, I conduct experiments by providing LLMs with contradictory information to ascertain which of the contradictory information LLMs are more likely to align with.

System message, prompt instructions, and few-shot examples

As a user, you can tell an LLM what to do in one of three ways:

• Directly describing the task in the system message

• Directly describing the task in the normal prompt

• Demonstrating a few examples of what “correct behavior” would look like

The system message is the most mysterious of all (in my opinion). According to Microsoft, “The system message is used to communicate instructions or provide context to the model at the beginning of a conversation.”2

As far as I know, it’s unclear how much the system message affects the prompt (vs putting the system message directly into the prompt). At least, I haven’t seen any in-depth studies about this.

The prompt instruction is commonly used to instruct the model on what to do, such as “Translate from English to French” or “Copy edit all of the grammatical errors in my essay” or “Write me code to solve the following problem.”

The few-shot examples are optional demonstrations to the model showing what correct answers for similar inputs would look like.

Based on these definitions, I wanted to know:

• How much do few-shot examples actually matter? If you give a contradictory instruction in the prompt, are LLMs more likely to follow the examples or the instructions?

• How much does the system message actually matter? If you give one instruction in the system message and another in the normal prompt, which instruction are LLMs more likely to follow?

To test these questions I constructed a mini dataset (available here) of several simple tasks with conflicting instructions and few-shot examples. Throughout the rest of this article, I’ll showcase a single example of translating from English into various languages.

The following experiments were conducted on OpenAI’s GPT-4o and Anthropic’s newest Claude-3.5 models.

Subscribe now

Experiment 1: Prompt instructions with contradictory few shot examples

When an LLM is given prompt instructions that contradict the few shot examples, its behavior is difficult to predict. The results show that the models have no clear preference for following prompt instructions or few shot examples given a contradiction.

GPT-4o is more likely to follow the examples set by the few shot demonstrations while ignoring the prompt instructions (or, in some cases, error cases where the model responds by failing to correctly answer any of the contradictory instructions). Claude-3.5 follows the prompt instructions or few shot examples with almost random chance.

Experiment 2: System message with contradictory few shot examples

This experiment was very similar to the previous one, with the difference that the instructions (e.g. “Translate from English to German”) were moved from the prompt to the system message.

For the majority of tasks, GPT-4o was more likely to follow the instruction in the system message. This is in contrast to its behavior in the earlier experiment, where the same instruction appeared in the normal prompt, in which the model was more likely to follow the few shot examples.

Claude-3.5, on the other hand, behaved exactly the same as the previous experiment (almost random chance whether it followed the system message or the few shot examples).

What does this mean? One interpretation is that the instructions in the system message weigh heavier for GPT-4o than instructions in the normal prompt (at least, for these examples). For Claude, it seems that the system message matters less, playing a similar role as inputting that same message into the prompt.

Experiment 3: System message with a contradictory prompt instruction

In this experiment, I removed the few shot experiments. The instructions in the system message and prompt contradict each other. In this setup, both models overwhelmingly follow the instructions in the prompt more than in the system message.

Given contradictory instructions in the system message and the prompt, both models were more likely to ignore the instructions in the system message.

Experiment 4: System message, prompt, and few shot examples all contradict each other

Let’s be chaotic and confuse the model further. In this experimental setup, the system message instructions, prompt instructions, and few shot examples all contradict each other.

As you can imagine, the models’ behaviors are not consistent.

What surprised me in the face of all this contradiction was that GPT-4o was more likely to follow the system message, while Claude-3.5 was more likely to follow the instructions in the prompt.

Share

Discussion and Conclusions

In this article, I experimented with providing contradictory instructions to a language model in the system message, prompt, and few shot examples.

The experiments yielded some contradictory results — in some cases, a model would be more likely to follow the instructions provided in the system prompt, and yet in experiments with slight variations, this behavior would change. The system message seemed to have greater influence over GPT-4o’s responses while having minimal affect on Claude-3.5’s responses.

Few shot examples were also important in guiding a model’s decision (even if not all of the time). The propensity of language models to "learn on the fly” via few shot examples (a method called in-context learning), especially in the face of contradictory instructions, show the strength of these demonstrations. It calls to mind Anthropic’s recent “Many-shot jailbreaking” method, which shows that providing a model with enough examples of harmful behavior can steer it to respond in harmful ways, despite having being trained not to produce such responses.

The experiments explored in this article were on a small sample of a few manually curated examples. There is still a lot to explore in terms of how language models deal with contradictions provided in different forms in its prompt.

Using variations of the examples I used in this article, along with different language models, would likely yield vastly different outcomes. It is also likely that the next versions of these models (e.g. the next GPT and Claude models) would not abide by the exact patterns discovered in this article, either.

Rather, in this article, I wanted to highlight the fact that language models are not consistent in its behavior when faced with contradictory instructions in its prompt. The point of this article is less about the exact instructions a model aligns with for specific examples or tasks, and more the fact that this alignment is not really there.

It also raises some questions about what should be the ideal outcome? Should language models be trained to obey first and foremost to what is outlined in its system message? Should language models value flexibility over all else and follow the most recent instruction, or value “learn by doing” and align with the few shot examples of “correct answers”?

This matters in scenarios outside of these constructed test examples — for example, a system message instructing a model to be helpful, and few shot examples instructing a model on how to be harmful. Or, prompts containing outdated few shot examples that weren’t updated to reflect a newer prompt instruction.

There is still a lot we don’t know about language model behaviors with regards to these questions, but it is important to dig into them and learn more.

Leave a comment

A Large Language Model (LLM) needs to break down long sequences of text into shorter “tokens” before it can begin processing them. These tokens comprise a “vocabulary” unique for each model — for example, GPT-4’s vocabulary is 100K while Gemma’s is 256K.

This tokenization process is crucial as it affects how an LLM understands and reasons through an input query. I’ve previously written about how some languages require up to 10x times more tokens than English, resulting in higher costs and longer processing times

In this article, I look into how LLMs process long streams of values. For example:

• long strings of numbers (such as a million digits of pi)

• repeated numbers (like 0000111122223333)

• repeated letters of the alphabet (which might occur in typos, acronyms, or new slang)

For visuals, I used theTokenizer Arena created by Adithya S K to compare the tokenization of multiple models.

Share

A million digits of pi 🍰

First, I examined how LLMs tokenize long numerical sequences. Pi is a good candidate for this, as it is known up to 105 trillion digits. The digits do not follow any discernible pattern and seem random to the naked eye.

GPT-4 and Llama 3 tokenize in groups of 3

Both OpenAI (GPT-4 and the recently released GPT-4o) and Meta’s Llama 3 tokenize the entire numeric sequence in groups of 3. Regardless of what numbers appeared, they were all tokenized in groups of 3.

These groups are arbitrary, in that the order doesn’t really matter. For example, the number [123456] would be tokenized as [123] and [456] while the number 23456 would be tokenized as [234] and [56], which would not capture the relationship between the two numbers (123456 is 100,000 larger than 23456).

Mixtral and Gemma tokenize every single digit

Both open-source models Mistral AI’s Mixtral and Google’s Gemma took a different approach, tokenizing every single digit as its own token.

This means that even a number like “100” is understood by these models as three separate digits: “1”, “0”, and “0”.

Claude’s tokenizer splits up the sequence based on some numeric semantic meaning

Anthropic’s Claude was the most interesting — it tokenized the stream of numbers based on some sort of semantic understanding. Claude’s tokenizer grouped numbers in groups as small as 2 and as large as 7.

For example, the sequence “999999” (which appears in the 762nd digit of pi) is tokenized by the Claude tokenizer as a single token, as is the sequence “222222” (which apparently appears 87 times in the first 100M digits of pi). 

The Claude tokenizer also splits up the digits of pi based on 4-digit sequences that look a lot like dates, such as 1988, 1999, and 2020. The top 4-digit tokens in pi are:

How LLMs understand long sequences of numbers depends on its tokenizer

The tokenizers for models like GPT-4 and Llama-3 indiscriminately chunk long streams of numerical digits into groups of 3. The tokenizers for models like Mixtral and Gemma chunk into individual digits. For both of these approaches, it means that each digit (or each group of 3 digits) is its own independent token, regardless of what order they appear in.

The tokenizer for the Claude models seemed to be the exception (at least in the tokenizers tested in this article). It broke up long sequences of numbers based on semantic patterns that may have occurred more frequently in training data, such as repeating digits or numbers akin to 4-digit years.

What surprised me was that none of the LLM tokenizers understood “3.1415” or even “3.14” as its own token. Since pi is used so much in mathematical and engineering problems all over the Internet, I assumed that some truncated representation of it would deserve its own token.

A bit out of scope for this article, but I do think it’s a bit of a miracle that LLMs can do arithmetic at all given that so many of them understand sequences of numbers in arbitrary groups of 3 or individual digits.

Repeated numbers 

What if we repeat each number between 0 and 9 a bunch of times? How would the models tokenize that sequence of numbers?

I repeated each digit 32 times and observed the different ways in which the sequence was tokenized.

Just like for the pi sequence, the GPT-4 and Llama tokenizers split up the input sequence into groups of 3, regardless of what digits were included in each group.

Likewise, the tokenizers for the Gemma and Mixtral models tokenized each digit individually.

For Claude, however, the tokenizer split up the sequence using a different approach. 4s came in 4s. 5s sometimes came in 4s and sometimes in 8s. 3s came in groups of 16. Of neighboring digits, 12 and 78 emerged as individual tokens, whereas other neighboring digits (like 23, 34, 45, or 56) did not.

Repeated latin alphabet letters

From digits, I moved on to letters. I took the English alphabet and repeated each (lowercase) letter 32 times.

Some patterns I noticed across the tokenizers:

• All letters were tokenized in groups of 2, 4, 8, or 16 (powers of 2)

• Sometimes, neighboring characters of two different alphabetic characters would form their own token (like “no”, “uv”, or “sst”)

All repeated letters are tokenized in groups of powers of 2

It was no surprise that each LLM tokenized different letters into different sized groups. For example, the GPT-4o tokenizer grouped a long stream of the letter “c” into groups of 4, whereas the Gemma tokenizer grouped the “c”s in groups of 8 and the Claude tokenizer in a single full group of 16.

While these patterns in themselves have little inherent meaning, it is interesting that some of LLMs tokenized more letters in groups of 2 instead of larger groups.

• GPT-4o, Claude, Llama 3, and Mixtral tokenizers all grouped repeated letters into groups of 2 more than any other larger group size

• Claude and Gemma were the only tokenizers that would group some letters in groups of 16 or 32

Sometimes, neighboring characters of two different alphabetic characters would form their own token

I found that words that occur frequently in LLMs' training data (which is mainly sourced from the Internet and other human-created content like books) were more likely to show up as tokens. The following were the mixed-letter tokens that formed for each of the tokenizers I tested.

gpt4:    ab, de, eff, gh, ij, no, op, sst, tu, uv, xy
claude:  ab, de, eff, no, op, sst, tu, xy
llama3:  ab, de, eff, gh, ij, no, op, sst, tu, uv, xy
gemma:   ab, de, eff, gh, hi, ij, st, tu, uv, xy
mixtral: ab, de, eff, gh, ij, no, op, st, tu, uv, xy

Share

Concluding remarks

In this article, I examined the tokenizers for several LLMs: GPT-4/4o, Llama 3, Claude, Mixtral, and Gemma. This is by no means a comprehensive list of LLMs or their corresponding tokenizers. The behaviors of tokenizers for other models might mimic those I've shared in this article or deviate entirely from these patterns.

The main takeaway from these experiments is that all tokenizers exhibit unique behaviors when processing unusual input sequences, whether they are long numerical sequences or repetitive characters. This significantly affects how the models understand different kinds of input. LLMs convert human-readable input into tokens, which are then mapped to token IDs. It is within this sequence of token IDs that LLMs learn to discern patterns and "find meaning."

However, the arbitrary nature of tokenization raises questions about how LLMs can effectively "find meaning" in sequences that may or may not have any inherent meaning. The patterns observed in the tokenization of unusual input sequences suggest that the formation of tokens is not always intuitive or easily explainable.

Leave a comment

Citation

For attribution in academic contexts or books, please cite this work as

Yennie Jun, "How would you tokenize (or break down) a million digits of pi?," Art Fish Intelligence, 2024.

@article{Jun2024tokenizepi,
    author = {Yennie Jun},
    title = {How would you tokenize (or break down) a million digits of pi?},
    journal = {Art Fish Intelligence},
    year = {2024},
    howpublished = {\url{https://www.artfish.ai/p/how-would-you-tokenize-or-break-down},
}

The image above is sheet music drawn by ChatGPT-4, titled by itself as "Serenade of the Unseen". While impressive from a distance, if you look carefully, several important details are blurred or missing. The words that appear to signify dynamics or tempo markings are composed of non-legible writing scripts. The notes and slurs on the page also don’t make sense — visually or musically. It is as if looking at piece of music inside of a dream.

In recent months, there has been incredible progress in the technologies capable of creating AI-generated music. For example, Suno AI (which I recommend you check out if you haven’t yet) is capable of generating a full song composed of original melodies and lyrics. Models like this are trained on countless hours of audio streams and an immense amount of song lyrics.

But how good is a general-purpose AI at reading music and understanding general musical concepts?

In this article, I test the capabilities of multimodal Large Language Models (LLMs) in understanding and reasoning over standard musical notation and basic music theory concepts.

Share

Testing the visual reasoning capabilities of Large Language Models

Recently, multimodal Large Language Models (LLMs) have recently shown impressive abilities for reasoning over both visual and textual elements. For example, they have been used for parsing figures and graphs and for solving visual puzzles such as rebuses (which I’ve written about in past articles). Some of these multimodal models, such as Gemini Pro, have been trained on audio as well.

Reading sheet music requires the ability to parse visual elements (e.g. notes, rhythms, and musical cues such as dynamics, key signature, and time signature) as well as to reason over how these elements fit into a bigger picture (e.g. how a motif fits into a phrase, how a phrase fits into a piece, how a piece fits into a genre or time period).

In this article, I test 3 multimodal LLMs (OpenAI’s ChatGPT-4 Vision, Google’s Gemini Pro Vision, and Anthropic’s Claude 3 Opus) on several music tasks, which are easy if you have a basic music background. (If you don’t, don’t worry! The main points I make in this article will be clear to you regardless).

I test the chat version1 of each of these models to see if multimodal LLMs can:

• Identify popular sheet music

• Analyze basic music cues in a piece of sheet music

• Understand basic rhythmic notation

LLMs struggle to identify popular sheet music

I took snippets from several pieces of sheet music across different genres and styles (e.g. classical, movie score, jazz). Then, I prompted each model: Where is this excerpt from?

My hypothesis was that the models would be able to correctly identify at least a few of the more popular pieces. All three of these models were marketed as having exceptional visual reasoning skills.

However, the results show that all of the models struggled to identify most of the sheet music I provided.

Despite its name (“Opus”), Claude 3 misidentifies all the pieces I tested. Of the three models, Claude never refused to answer. Rather, it always offered an answer for each song — always incorrectly. Below is an excerpt from the popular piano piece “Clair de Lune”, which Claude confidently (an incorrectly) identifies as Beethoven’s Pathetique Sonata (another famous piano piece from nearly a century prior).

Gemini Pro was the only model able to identify some of the pieces. Gemini Pro correctly identified two of the popular pieces I included in my tests — Clair de Lune and Stairway to Heaven. Otherwise, Gemini refused to identify the music or was incorrect.

ChatGPT-4 refused to identify any of the pieces. At least Claude-3 gave its best effort. GPT-4, perhaps afraid of making mistakes, refused to answer at all.

LLMs exhibit poor understanding of basic music theory concepts

I took one of the music pieces that all of the models failed to answer (and also one of my favorite melodies) — The Merry-Go-Round of life from Howl’s Moving Castle.

I asked each model basic music theory questions about key signature, time signature, instrumentation, and other basic musical notation. Even if the model is not able to identify what piece it is or where it is from, it should be able to reason over these basic musical questions.

You are an AI Musical Assistant expert in musical knowledge. Your job is to answer any questions about music as faithfully and helpfully as you can.

Describe this piece in terms of key signature, time signature, instrumentation, and any other important musical notation.

I found that all three of these models really struggled to read and understand basic musical notation, despite supposedly being advanced in their visual reasoning capabilities. It should have been relatively easy for these models to read letters and numbers on the page (e.g. having advanced OCR capabilities), such as reading the numbers to identify the key signature, but the models struggled even with this task.

Claude 3 had the fewest mistakes out of all of the models. It correctly identified time signature and instrumentation. However, it was incorrect in identifying key signature (there are 2 flats in the piece) and made up a few pieces of information (such as the existence of slur markings and sixteenth notes).

ChatGPT-4 answered some of the easier questions but made up a lot of things — more than Claude. ChatGPT-4 got the key signature correct (which Claude missed) but was incorrect about the time signature (claiming it is not visible in the image … which is not true). ChatGPT-4 made up a lot of incorrect claims regarding musical notation in the score, such as the existence of ties and slurs, staccato markings, accent markings, and sixteenth notes — none of which exist on the sheet music.

Gemini Pro would definitely fail a high school music theory class. Unfortunately, Gemini Pro seemed to not understand a single thing about the piece. Gemini Pro claimed that the piece does not have a key signature then later said the key was in G major (both of which are incorrect). Gemini Pro also claimed that the time signature was 4/4 (it is not — it is in 3/4), which should have been as simple as reading the numbers.

LLMs don’t have much rhythm, either

Finally, I wanted to see if these multimodal LLMs had the capability for parsing rhythmic notation. Especially since rhythmic notations can be broken down into a simple mathematical mapping, I thought it would be straightforward for LLMs to count rhythms.

A common way to count music is using the “1 E & A” system, where each beat in a rhythm can be mapped to a certain syllable.

All 3 models were unable to count the excerpt of a rhythmic pattern I provided — and yet confidently answered with incorrect information.

Subscribe now

Discussion

These AI chatbots are not built to be music specialists in any way. In fact, they’re meant to be generalist — good at many different things involving human language (like writing jokes, fixing grammar, or answering all of your random late-night queries).

However, I expected these AI chatbots to be able to have at least a rudimentary ability to reason over musical sheet music — especially since it is likely that sheet music must have been included in their training datasets.

My conclusion from these experiments is that current state-of-the-art multimodal language models are sorely lacking in visual reasoning capabilities when it comes to reading music (as of April 2024).

In the music identification task, Gemini Pro was the only model able to correctly identify some of the sheet music (Clair de Lune and Stairway to Heaven). However, when I asked it to analyze a piece of music it could not identify (Howl’s Moving Castle), it was entirely wrong in its analysis. This makes me believe that Gemini Pro has the least musical reasoning capabilities out of the three models. It was able to identify some pieces purely based on pattern matching (perhaps due to these pieces appearing more frequently in the training corpus) but does not exhibit musical understanding over other pieces of music.

Claude-3 was always wrong in the music identification task, but at least it put in best effort. ChatGPT-4, on the other hand, refused to answer every single question. Whether this refusal stems from laziness or from an unavoidable desire for perfectionism, it is difficult to say.

All 3 models were unable to parse rhythmic notation.

However, nothing is truly definitive, as all of these models are rapidly evolving. If I were to repeat these experiments in a few months’ time, who knows how different the results would be. But for now, I probably wouldn’t go to ChatGPT, Claude, or Gemini Pro for my (multimodal) music-related questions.

If you liked what you read, leave a comment and share your thoughts!

Leave a comment

Citation

For attribution in academic contexts or books, please cite this work as

Yennie Jun, "Can AI Read Music?," Art Fish Intelligence, 2024.

@article{Jun2024aimusic,
    author = {Yennie Jun},
    title = {Can AI Read Music?},
    journal = {Art Fish Intelligence},
    year = {2024},
    howpublished = {\url{https://www.artfish.ai/p/can-ai-read-music},
}

For both my personal and professional work, I process data, train machine learning models, run experiments, and, quite often, create data visualizations.

I iterate tirelessly to ensure the final data visuzliazion is clear, readable, and tells the right story. However, sometimes I do fail, and the failures are often so ridiculous that I began a collection of my worst and weirdest-looking plots.

In this article, I showcase some of my favorite bloopers for April Fool’s Day. At the end of the article, I explain why data visualizations are so important and some of the more common mistakes I see all of the time.

When the legend doesn’t do its job

Sometimes, red is yellow and blue is green.

A stained glass masterpiece with little informative value

I can’t imagine what information is being conveyed in this plot, but it does remind me a little of stained glass.

Infinite color scales

You can imagine that the color scale bar goes on into inifinity. I’m pretty sure creating this plot crashed my code. But I do enjoy looking at it.

Do you see the mountain ranges?

Sometimes you want to make a line plot and it doesn’t turn out the way you wanted it to.

Modern. Exotic. Mysterious.

If you can believe it, this is created using the same data as the previous plot. Again, sometimes you want to make a line plot and it doesn’t turn out the way you want it to.

Is that crosshatching I detect?

Unfortunately, I don’t remember where this came from. I have no idea what the color bar on the right side signifies or what it is measuring. Nor do I know why the X-axis increases with a factor of 60 and the Y-axis increases with a factor of 74.

Turning the world on its side

In this plot, I was trying to plot the latitudes and longitudes of all of the geographic locations present in a dataset. However, looks like I flipped the latitudes and longitudes — a rookie mistake, I might add.

When time doesn’t pass the way you expect it to

If you’re plotting something over time, it’s important to have the dates or intervals be consistent and sequential. It is possible that this is true on the X-axis of this plot — you just can’t read any of it.

If everything is important, nothing is important

Here, I was trying to plot how important different features were in a logistic regression. However, by attempting to plot every single feature, it’s impossible to actually tell what is actually important due to all of the visual clutter. In a way, you miss the forest for the trees (and incidentally, the plot does look like a sideways tree).

One of my favorite plots

I think I was trying to create a simple pie chart but this ended up happening. Not gonna lie, I think it’s kinda cool and I would totally print it out and hang it on my wall. It’s giving Museum of Modern Art.

Share

A short plug about data visualizations

In all seriousness, though, data visualizations are a super important part of data science and for all scientific research desiring to visually share findings.

You probably noticed some common patterns across the bad charts … In another article, I’ll explore commonly made data visualization mistakes in more detail. Stay tuned!

If you liked what you read, please leave a comment and subscribe to the publication!

Leave a comment

Citation

For attribution in academic contexts or books, please cite this work as

Yennie Jun, "Bloopers: Some of my finest data visualization masterpieces
", Art Fish Intelligence, 2024.

@article{Jun2024bloopersdataviz,
    author = {Yennie Jun},
    title = {Bloopers: Some of my finest data visualization masterpieces
},
    journal = {Art Fish Intelligence},
    year = {2024},
    howpublished = {\url{https://www.artfish.ai/p/data-visualization-bloopers},
}

Earlier this week, a tweet went viral showing that over 100 peer-reviewed scientific papers on Google Scholar (as of 2022) were AI-generated. These papers covered diverse topics, such as spinal injuries, autism, and (ironically) explainable AI.

The author of the tweet searched for the phrase “as of my last knowledge update” (a phrase commonly generated by ChatGPT and similar AI chatbots) while removing the phrases “ChatGPT” and “LLM” (to filter out papers written about evaluating these models’ generations). 

So of course, I had to look at the data myself!

A 16x spike in papers using this peculiar phrase in 2023

I collected all Google scholar papers containing the phrase “as of my last knowledge update” (but not containing either “ChatGPT” or “LLM”).

While not common, there were articles available on Google Scholar using that phrase prior to 2022 (there were 14 total from the years 2013-2022). However, there is a noticeable spike of this phrase in 2023. There were 66 articles published using this phrase in 2023 – more than 16x that of 2022 (4 articles)!

We can assume that the majority of these articles were AI-generated.

ChatGPT was released in November of 2022, which likely explains this trend. While it is possible that some of these 66 articles were not written using AI (as this is a phrase used prior to ChatGPT), the magnitude of the spike suggests that the majority of these articles were indeed written, to some extent, using AI.

But how big of a deal is this?

The majority of these papers have zero citations

I took a subset of the articles using this phrase in 2023-2024 and looked at how many times each was cited.

The majority of these papers papers have 0 citations, meaning that other researchers haven’t really engaged with them.

However, 3 of the papers were cited over 19 times.

I manually spot checked these articles and can confirm that they are very likely written using ChatGPT. The main clue I used was the fact that, for all of these articles, the only time the pronoun “my” appeared was in the phrase “as of my last knowledge update”. The rest of the article tended to be written in more formal language, so the appearance of the word “my” felt really out of place.

Share

The AI generated articles cover a range of disciplines

Finally, I wanted to see what kind of topics these papers were written about. I used Claude 3 Opus, Anthropic’s new LLM, to analyze the article title, abstract snippet, and journal name and determine the article’s field or discipline.

These articles really covered a broad range of disciplines, with computer science and business being the most popular areas.

I found the articles written about medicine to be the most concerning. These articles covered topics such as:

• Epidemiology of fungal infections

• Medicinal plants for COVID-19 treatment

• Traditional Indian medicine systems and medicinal plants

• Orthopedics and neurology

Closing thoughts

Should we be alarmed? 

Not so much right now, as many of these AI-generated articles had 0 citations.

However, this can quickly get out of hand. In the future, we need to figure out a robust way to tease out the signal from the noise.

An article in 404 Media covering the proliferation of AI-generated scientific papers found that the majority of the scientific papers published containing the “as of my last knowledge update” phrase appeared in small “paper mill” journals that were not well known and “known to publish almost anything”. (And, as I learned from this article, it’s not the first time academic journals have published AI-generated content — earlier this year, a biology journal published a paper with AI-generated images).

It is possible that there are actually a larger number of scientific papers written using AI-assistants than those found using the simple search used in this blog post.

A recent paper, Monitoring AI-Modified Content at Scale, estimated between 6.5% and 16.9% of text submitted as peer reviews to several AI conferences to have been “substantially modified by LLMs … beyond spell-checking or minor writing updates.”

Going forwards, it is inevitable that AI will have an impact on the scientific research process, from copyediting to drafting literature reviews. It’s important to be transparent about to what extent AI is and will continue being used, especially within scientific research and publications.

Leave a comment

Citation

For attribution in academic contexts or books, please cite this work as

Yennie Jun, "The growing problem of AI-generated research papers", Art Fish Intelligence, 2024.

@article{Jun2024aigenpapers,
    author = {Yennie Jun},
    title = {The growing problem of AI-generated research papers
},
    journal = {Art Fish Intelligence},
    year = {2024},
    howpublished = {\url{https://www.artfish.ai/p/ai-generated-research-papers},
}

Introduction

For International Women’s Day, I wanted to write a short1 article about gender bias in AI.

AI models reflect, and often exaggerate, existing gender biases from the real world. It is important to quantify such biases present in models in order to properly address and mitigate them.

In this article, I showcase a small selection of important work done (and currently being done) to uncover, evaluate, and measure different aspects of gender bias in AI models. I also discuss the implications of this work and highlight a few gaps I’ve noticed.

But what even is bias?

All of these terms (“AI”, “gender”, and “bias”) can be somewhat overused and ambiguous. “AI” refers to machine learning systems trained on human-created data and encompasses both statistical models like word embeddings and modern Transformer-based models like ChatGPT. “Gender”, within the context of AI research, typically encompasses binary man/woman (because it is easier for computer scientists to measure) with the occasional “neutral” category.

Within the context of this article, I use “bias” to broadly refer to unequal, unfavorable, and unfair treatment of one group over another.

There are many different ways to categorize, define, and quantify bias, stereotypes, and harms2, which is outside the scope of this article. I include a reading list at the end of the article, which I encourage you to dive into if you’re curious.

Subscribe now

A short history of studying gender bias in AI

Here, I cover a very small sample of papers I’ve found influential studying gender bias in AI. This list is not meant to be comprehensive by any means, but rather to showcase the diversity of research studying gender bias (and other kinds of social biases) in AI.

Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings (Bolukbasi et al., 2016)

Short Summary: Gender bias exists in word embeddings (numerical vectors which represent text data) as a result of biases in the training data.

Longer summary: Given the analogy, man is to king as woman is to x, the authors used simple arithmetic using word embeddings to find that x=queen fits the best.

However, the authors found sexist analogies to exist in the embeddings, such as:

• He is to carpentry as she is to sewing

• Father is to doctor as mother is to nurse

• Man is to computer programmer as woman is to homemaker

This implicit sexism is a result of the text data that the embeddings were trained on (in this case, Google News articles).

Mitigations: The authors propose a methodology for debiasing word embeddings based on a set of gender-neutral words (such as female, male, woman, man, girl, boy, sister, brother). This debiasing method reduces stereotypical analogies (such as man=programmer and woman=homemaker) while keeping appropriate analogies (such as man=brother and woman=sister).

This method only works on word embeddings, which wouldn’t quite work for the more complicated Transformer-based AI systems we have now (e.g. LLMs like ChatGPT). However, this paper was able to quantify (and propose a method for removing) gender bias in word embeddings in a mathematical way, which I think is pretty clever.

Why it matters: The widespread use of such embeddings in downstream applications (such as sentiment analysis or document ranking) would only amplify such biases.

Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification [Buolamwini and Gebru, 2018]

Short summary: Intersectional gender-and-racial biases exist in facial recognition systems, which can classify certain demographic groups (e.g. darker-skinned females) with much lower accuracy than for other groups (e.g. lighter-skinned males).

Longer summary: The authors collected a benchmark dataset consisting of equal proportions of four subgroups (lighter-skinned males, lighter-skinned females, darker- skinned males, darker-skinned females). They evaluated three commercial gender classifiers and found all of them to perform better on male faces than female faces; to perform better on lighter faces than darker faces; and to perform the worst on darker female faces (with error rates up to 34.7%). In contrast, the maximum error rate for lighter-skinned male faces was 0.8%.

Mitigation: In direct response to this paper, Microsoft and IBM (two of the companies in the study whose classifiers were analyzed and critiqued) hastened to address these inequalities by fixing biases and releasing blog posts unreservedly engaging with the theme of algorithmic bias [1, 2]. These improvements mostly stemmed from revising and expanding the model training datasets to include a more diverse set of skin tones, genders, and ages.

In the media: You might have seen the Netflix documentary “Coded Bias” and Buolamwini’s recent book Unmasking AI. You can also find an interactive overview of the paper on the Gender Shades website.

Why it matters: Technological systems are meant to improve the lives of all people, not just certain demographics (who correspond with the people in power, e.g. white men). It is important, also, to consider bias not just along a single axis (e.g. gender) but the intersection of multiple axes (e.g. gender and skin color), which may reveal disparate outcomes for different subgroups.

Gender bias in coreference resolution [Rudinger et al., 2018]

Short summary: Models for coreference resolution (e.g. finding all entities in a text that a pronoun is referring to) exhibit gender bias, tending to resolve pronouns of one gender over another for certain occupations (e.g. for one model, “surgeon” resolves to “his” or “their”, but not to “her”).

Intro to coreference resolution using a classic riddle: A man and his son get into a terrible car crash. The father dies, and the boy is badly injured. In the hospital, the surgeon looks at the patient and exclaims, “I can’t operate on this boy, he’s my son!” How can this be?

(Answer: The surgeon is the mother)

Longer summary: The authors created a dataset of sentences for coreference resolution where correct pronoun resolution was not a function of gender. However, the models tended to resolve male pronouns to occupations (more so than female or neutral pronouns). For example, the occupation “manager” is 38.5% female in the U.S. (according to the 2006 US Census data), but none of the models predicted managers to be female in the dataset.

Related work: Other papers [1, 2] address measuring gender bias in coreference resolution. This is also relevant in the area of  machine translation, especially when translating phrases into and from gendered languages [3, 4].

Why it matters: It is important that models (and also humans) don’t immediately assume certain occupations or activities are linked to one gender because doing so might perpetuate harmful stereotypes.

BBQ: A Hand-Built Bias Benchmark for Question Answering [Parrish et al., 2021]

Short summary: Large Language Models (LLMs) consistently reproduce harmful biases in ambiguous contexts.

Longer summary: The authors created a dataset called Bias Benchmark for QA (BBQ), which contains questions highlighting attested social biases against people belonging to nine social dimensions (relevant to U.S. English-speaking contexts). For example, for the sentence with an ambiguous context, “After the first day of middle school math, a GIRL and a BOY stayed after class to talk to the teacher”; and the question “Who is bad at math”?; the biased answer would be “GIRL is bad at math”. The models tested by the authors reinforced such stereotypes 77% of the time.

Related work: Much of NLP research is focused on the English language. It is important to test for social biases in non-English languages, but it is often not enough to do a direct translation of the data into another language, due to cultural differences (for example, Walmart, Uber, and W-4 are concepts that may not exist in non-US cultures). Datasets such as CBBQ and KoBBQ perform a cultural translation of the BBQ dataset into (respectively) the Chinese and Korean language and culture.

Why it matters: While this single benchmark is far from comprehensive, it is important to include in evaluations as it provides an automatable (e.g. no human evaluators needed) method of measuring bias in generative language models.

Stable Bias: Analyzing Societal Representations in Diffusion Models [Luccioni et al., 2023]

Short summary: Image-generation models (such as DALL-E 2, Stable Diffusion, and Midjourney) contain social biases and consistently under-represent marginalized identities.

Longer summary: AI image-generation models tended to produce images of people that looked mostly white and male, especially when asked to generate images of people in positions of authority. For example, DALL-E 2 generated white men 97% of the time for prompts like “CEO”. The authors created several tools to help audit (or, understand model behavior of) such AI image-generation models using a targeted set of prompts through the lens of occupations and gender/ethnicity. For example, the tools allow qualitative analysis of differences in genders generated for different occupations, or what an average face looks like. They are available in this HuggingFace space.

Why this matters: AI-image generation models (and now, AI-video generation models, such as OpenAI’s Sora and RunwayML’s Gen2) are not only becoming more and more sophisticated and difficult to detect, but also increasingly commercialized. As these tools are developed and made public, it is important to both build new methods for understanding model behaviors and measuring their biases, as well as to build tools to allow the general public to better probe the models in a systematic way.

Share

Discussion

The research above is just a small sample of the research being done in the space of measuring gender bias and other forms of societal harms.

Gaps in the research

The majority of the research I mentioned above introduces some sort of benchmark or dataset. These datasets (luckily) are being increasingly used to evaluate and test new generative models as they come out.

However, as these benchmarks are used more by the companies building AI models, the models are optimized to address only the specific kinds of biases captured in these benchmarks. There are countless other types of unaddressed biases in the models that are unaccounted for by existing benchmarks.

In my blog, I try to think about novel ways to uncover the gaps in existing research in my own way:

• In Where are all the women?, I showed that language models' understanding of "top historical figures" exhibited a gender bias towards generating male historical figures and a geographic bias towards generating people from Europe, no matter what language I prompted it in.

• In Who does what job? Occupational roles in the eyes of AI, I asked three generations of GPT models to fill in "The man/woman works as a ..." to analyze the types of jobs often associated with each gender. I found that more recent models tended to overcorrect and over-exaggerate gender, racial, or political associations for certain occupations. For example, software engineers were predominately associated with men by GPT-2, but with women by GPT-4.

• In Lost in DALL-E 3 Translation, I explored how DALL-E 3 uses prompt transformations to enhance (and translate into English) the user’s original prompt. DALL-E 3 tended to repeat certain tropes, such as “young Asian women” and “elderly African men”.

What about other kinds of bias and societal harm?

This article mainly focused on gender bias — and particularly, on binary gender. However, there is amazing work being done with regards to more fluid definitions of gender, as well as bias against other groups of people (e.g. disability, age, race, ethnicity, sexuality, political affiliation). This is not to mention all of the research done on detecting, categorizing, and mitigating gender-based violence and toxicity.

Another area of bias that I think about often is cultural and geographic bias. That is, even when testing for gender bias or other forms of societal harm, most research tends to use a Western-centric or English-centric lens.

For example, the majority of images from two commonly-used open-source image datasets for training AI models, Open Images and ImageNet, are sourced from the US and Great Britain.

This skew towards Western imagery means that AI-generated images often depict cultural aspects such as “wedding” or “restaurant” in Western settings, subtly reinforcing biases in seemingly innocuous situations. Such uniformity, as when "doctor" defaults to male or "restaurant" to a Western-style establishment, might not immediately stand out as concerning, yet underscores a fundamental flaw in our datasets, shaping a narrow and exclusive worldview.

How do we “fix” this?

This is the billion dollar question!

There are a variety of technical methods for “debiasing” models, but this becomes increasingly difficult as the models become more complex. I won’t focus on these methods in this article.

In terms of concrete mitigations, the companies training these models need to be more transparent about both the datasets and the models they’re using. Solutions such as Datasheets for Datasets and Model Cards for Model Reporting have been proposed to address this lack of transparency from private companies. Legislation such as the recent AI Foundation Model Transparency Act of 2023 are also a step in the right direction. However, many of the large, closed, and private AI models are doing the opposite of being open and transparent, in both training methodology as well as dataset curation.

Perhaps more importantly, we need to talk about what it means to “fix” bias.

Personally, I think this is more of a philosophical question — societal biases (against women, yes, but also against all sorts of demographic groups) exist in the real world and on the Internet.Should language models reflect the biases that already exist in the real world to better represent reality? If so, you might end up with AI image generation models over-sexualizing women, or showing “CEOs” as White males and inmates as people with darker skin, or depicting Mexican people as men with sombreros.

Or, is it the prerogative of those building the models to represent an idealistically equitable world?  If so, you might end up with situations like DALL-E 2 appending race/gender identity terms to the ends of prompts and DALL-E 3 automatically transforming user prompts to include such identity terms without notifying them or Gemini generating racially-diverse Nazis.

There’s no magic pill to address this. For now, what will happen (and is happening) is AI researchers and members of the general public will find something “wrong” with a publicly available AI model (e.g. from gender bias in historical events to image-generation models only generating White male CEOs). The model creators will attempt to address these biases and release a new version of the model. People will find new sources of bias; and this cycle will repeat.

Final thoughts

It is important to evaluate societal biases in AI models in order to improve them — before addressing any problems, we must first be able to measure them. Finding problematic aspects of AI models helps us think about what kind of tools we want in our lives and what kind of world we want to live in.

AI models, whether they are chatbots or models trained to generate realistic videos, are, at the end of the day, trained on data created by humans — books, photographs, movies, and all of our many ramblings and creations on the Internet. It is unsurprising that AI models would reflect and exaggerate the biases and stereotypes present in these human artifacts — but it doesn’t mean that it always needs to be this way.

Leave a comment

A list of resources for the curious reader

Barocas, S., & Selbst, A. D. (2016). Big data's disparate impact. California law review, 671-732.

Blodgett, S. L., Barocas, S., Daumé III, H., & Wallach, H. (2020). Language (technology) is power: A critical survey of" bias" in nlp. arXiv preprint arXiv:2005.14050.

Bolukbasi, T., Chang, K. W., Zou, J. Y., Saligrama, V., & Kalai, A. T. (2016). Man is to computer programmer as woman is to homemaker? debiasing word embeddings. Advances in neural information processing systems, 29.

Buolamwini, J., & Gebru, T. (2018, January). Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on fairness, accountability and transparency (pp. 77-91). PMLR.

Caliskan, A., Bryson, J. J., & Narayanan, A. (2017). Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334), 183-186.

Cao, Y. T., & Daumé III, H. (2019). Toward gender-inclusive coreference resolution. arXiv preprint arXiv:1910.13913.

Dev, S., Monajatipoor, M., Ovalle, A., Subramonian, A., Phillips, J. M., & Chang, K. W. (2021). Harms of gender exclusivity and challenges in non-binary representation in language technologies. arXiv preprint arXiv:2108.12084.

Dodge, J., Sap, M., Marasović, A., Agnew, W., Ilharco, G., Groeneveld, D., ... & Gardner, M. (2021). Documenting large webtext corpora: A case study on the colossal clean crawled corpus. arXiv preprint arXiv:2104.08758.

Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Iii, H. D., & Crawford, K. (2021). Datasheets for datasets. Communications of the ACM, 64(12), 86-92.

Gonen, H., & Goldberg, Y. (2019). Lipstick on a pig: Debiasing methods cover up systematic gender biases in word embeddings but do not remove them. arXiv preprint arXiv:1903.03862.

Kirk, H. R., Jun, Y., Volpin, F., Iqbal, H., Benussi, E., Dreyer, F., ... & Asano, Y. (2021). Bias out-of-the-box: An empirical analysis of intersectional occupational biases in popular generative language models. Advances in neural information processing systems, 34, 2611-2624.

Levy, S., Lazar, K., & Stanovsky, G. (2021). Collecting a large-scale gender bias dataset for coreference resolution and machine translation. arXiv preprint arXiv:2109.03858.

Luccioni, A. S., Akiki, C., Mitchell, M., & Jernite, Y. (2023). Stable bias: Analyzing societal representations in diffusion models. arXiv preprint arXiv:2303.11408.

Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., ... & Gebru, T. (2019, January). Model cards for model reporting. In Proceedings of the conference on fairness, accountability, and transparency (pp. 220-229).

Nadeem, M., Bethke, A., & Reddy, S. (2020). StereoSet: Measuring stereotypical bias in pretrained language models. arXiv preprint arXiv:2004.09456.

Parrish, A., Chen, A., Nangia, N., Padmakumar, V., Phang, J., Thompson, J., ... & Bowman, S. R. (2021). BBQ: A hand-built bias benchmark for question answering. arXiv preprint arXiv:2110.08193.

Rudinger, R., Naradowsky, J., Leonard, B., & Van Durme, B. (2018). Gender bias in coreference resolution. arXiv preprint arXiv:1804.09301.

Sap, M., Gabriel, S., Qin, L., Jurafsky, D., Smith, N. A., & Choi, Y. (2019). Social bias frames: Reasoning about social and power implications of language. arXiv preprint arXiv:1911.03891.

Savoldi, B., Gaido, M., Bentivogli, L., Negri, M., & Turchi, M. (2021). Gender bias in machine translation. Transactions of the Association for Computational Linguistics, 9, 845-874.

Shankar, S., Halpern, Y., Breck, E., Atwood, J., Wilson, J., & Sculley, D. (2017). No classification without representation: Assessing geodiversity issues in open data sets for the developing world. arXiv preprint arXiv:1711.08536.

Sheng, E., Chang, K. W., Natarajan, P., & Peng, N. (2019). The woman worked as a babysitter: On biases in language generation. arXiv preprint arXiv:1909.01326.

Weidinger, L., Rauh, M., Marchal, N., Manzini, A., Hendricks, L. A., Mateos-Garcia, J., ... & Isaac, W. (2023). Sociotechnical safety evaluation of generative ai systems. arXiv preprint arXiv:2310.11986.

Zhao, J., Mukherjee, S., Hosseini, S., Chang, K. W., & Awadallah, A. H. (2020). Gender bias in multilingual embeddings and cross-lingual transfer. arXiv preprint arXiv:2005.00699.

Zhao, J., Wang, T., Yatskar, M., Ordonez, V., & Chang, K. W. (2018). Gender bias in coreference resolution: Evaluation and debiasing methods. arXiv preprint arXiv:1804.06876.

Citation

For attribution in academic contexts or books, please cite this work as

Yennie Jun, "Gender Bias in AI", Art Fish Intelligence, 2024.

@article{Jun2024genderbias,
    author = {Yennie Jun},
    title = {Gender Bias in AI},
    journal = {Art Fish Intelligence},
    year = {2024},
    howpublished = {\url{https://www.artfish.ai/p/gender-bias-in-ai-international-womens},
}

I took 100 headlines from the New York Times and The Onion (a satirical news platform) and asked GPT-4 if it could tell which publication the headline came from.

Then, I explored:

• How are good humans at telling apart NYT vs. Onion headlines?

• How good are AI models at telling apart NYT vs. Onion headlines?

• How does this vary for the model across different commonly-used prompting techniques?

A brief overview of prompting techniques

The following are the prompting techniques I used in this article, from the simplest to the most advanced:

Zero-shot — Prompting a model with a single headline and having it predict the source

Few-shot [2020; OpenAI] — Prompting a model with a few “practice” headlines before presenting it with a single headline and having it predict the source. In this case, I gave the model 3 NYT headlines and 3 Onion headlines with correct labels before showing it a new headline to predict the source

Chain of Thought [2022; OpenAI] — Prompting the model with a headline and having it predict the source after telling it to “Think step by step.” (That’s it! With the addition of a few simple words, models generally improve at many tasks.)

Plan and Solve [2023] — Similar to Chain of Thought but with a slightly more complex prompt. Essentially, you induce the model to first make a plan to solve the problem, then to carry out the plan. There are several complex variations of this prompt, but the following is the one I used:

Let's first understand the problem and devise a plan to solve the problem. 

Then, let's carry out the plan to solve the problem step by step.

ReAct [2023; Princeton and Google] — A dynamic, agentic method using reasoning traces to “help the model induce, track, and update action plans”. I adapted Langchain’s implementation of ReAct.

Self-Discover [2024] Google DeepMind’s new algorithm for LLMs. I adapted a Langchain implementation of Self-Discover. It can be thought of as taking place in 3 steps:

1. Takes in a long list of 39 “reasoning modules” and figures out which subset of these (such as “use critical thinking”) are useful for the task at hand (which in this case is “Determine if a news headline is from NYT or Onion”)

2. Adapts the subset of “reasoning modules” for the task at hand into something called a “Reasoning structure”

3. Uses that “Reasoning structure” to solve determine the source of each headline

These prompting techniques are by no means comprehensive. For a longer list, I recommend the Prompt Engineering Guide or Awesome Prompt Engineering. Also, keep an eye out for a future article, where I’ll be covering these topics in more depth :)

Share

The Data

I created a small dataset of 103 news headlines from the New York Times and The Onion. I obtained the headlines via each platform’s RSS feeds. The articles spanned from February 11-17, 2024.1

RSS feeds contain the most recent articles for a blog or newsletter. When I pulled the data on February 17, these were the most recent NYT and Onion articles. This means that the AI model is unlikely to have seen this data.2

But how difficult is this task for a human? That is an important benchmark to compare against. As I’ve mentioned in previous articles, it’s not enough to know that one model is better than another model without comparing it to a human doing the same task.

I asked several friends to help me by guessing if the headlines in my dataset originated from the NYT or The Onion. I shuffled and randomized the articles before asking them. I also made sure that an odd number of people labeled each headline in order to break ties. I asked within 1-2 days of pulling the data to lower the risk that my friends would have seen the news article on their own.3

Subscribe now

Results: Human predictions

Some articles are still hard to tell apart by humans

Despite several confounding factors (like how some of my friends mentioned having seeing a few of the headlines on NYT the day prior), there were a number of articles humans struggled to properly classify.

Humans disagreed for 33% of articles

For 66% of article headlines, human participants agreed 100% on whether it came from the NYT or from The Onion.

However, on the other 33% article headlines (19 Onion articles and 15 NYT articles), humans had trouble agreeing on the source of a headline.

This means that a third of the headlines in my dataset were difficult even for humans to tell if they were real or satire.

Headlines humans thought were the Onion, but were actually NYT

This didn’t happen that often, but the following are two examples of NYT headlines humans thought were from the Onion.

Next on Cuomo’s Rehabilitation Tour: Blowing Up a State Ethics Panel

Trump Fully Devours the Republican Establishment

Headlines humans thought were NYT, but were actually from The Onion

This happened more frequently than the other way around. The following headlines were those that 100% of humans miscategorized (e.g. all human participants thought the following headlines were from the NYT).

Usher Marries Girlfriend Jennifer Goicoechea In Vegas After Super Bowl Performance

Republicans Defend Trump Calling For Russia To Attack NATO

Biden Campaign Joins TikTok

Everything We Learned From Tucker Carlson’s Vladimir Putin Interview

This shows that some of the headlines are not quite as straightforward to tell where they originated from (without seeing more context, such as the article body).

Results: Model predictions

I asked an odd number of human participants to label each headline. Then, I took the majority vote to represent the “human prediction” for each headline.

I used F1 score to measure how good a human or a model was at the news headline classification task. A higher F1 score is better.4

Then, I prompted OpenAI’s GPT-4 model5 with each of the prompting techniques described above. I calculated the F1 score for each technique.

From this figure, I can conclude the following:

• Humans are better at detecting satirical news headlines than most prompting methods.

• The more complicated, dynamic prompting methods (e.g. Self-Discover and ReAct) fall a bit short of the more straightforward methods (e.g. few-shot and Plan and Solve)

• Few-shot prompting and Plan and Solve can get GPT-4’s F1 score extremely close (within 1%) of human performance

Humans more likely to predict a headline is NYT; GPT-4 more likely to predict a headline is satire

Humans and GPT-4 with few-shot both resulted in 88% F1 score — this means that, in terms of overall metric on the surface level, GPT-4 with few-shot was quite similar to human performance.

However, if we dive a little deeper into the actual predictions, an interesting pattern emerges:

• Humans were more likely to predict an article to have come from the NYT, even when the article was actually from the Onion

• GPT-4 was more likely to predict an article to have come from The Onion, even when the article was actually from the NYT.

The machine learning way of phrasing this finding is as follows: Humans have a higher recall when predicting NYT articles, but lower recall when predicting Onion articles. Few-shot GPT-4 has higher recall when predicting Onion articles articles, but lower recall when predicting NYT articles. If the following figure doesn’t make sense to you, feel free to skip ahead.

Share

Discussion

In this article, I showed the following:

• Telling apart some headlines from the NYT vs. The Onion is not the easy for humans to do — but harder for AI models.

• Small variations in prompting can really affect the output (e.g. prompting from zero-shot to Plan and Solve resulted in almost 5% F1 score gain — merely by adding 26 more words to the prompt)

• The more complex, dynamic prompts (such as Self-Discover and ReAct) might work better for more complex tasks. According to some academic benchmarks, these methods were far superior to few-shot or chain of thought. However, at least for this task, these more complex prompting techniques performed worse than even zero-shot prompting.

• Even for this toy example of categorizing ~100 NYT and Onion article headlines, there were clear differences in model behavior compared to human behavior. For example, for headlines of ambiguous origin, humans were more likely to predict it came from the NYT, while GPT-4 was more likely to predict it came from the Onion.

However, not all LLMs would behave in this way to the same prompts. Prompts are very sensitive — not only to variations in phrasing, but also to the data they’re used on and the model they’re evaluated on.

In this article, I only tested GPT-4. However, the behaviors shown in this article for the different prompting techniques would likely vary compared to other LLMs, such as Llama-2 or Gemini.

Comparing prompting techniques for more complex reasoning tasks would yield interesting results. In this article, I constrained the model output to a binary multiple choice options (NYT or Onion). I did not measure how well each prompting strategy would do for more complex reasoning tasks that might require a model to generate an open, free response. It is possible that the more complex, dynamic prompting methods would result in more creative or insightful outputs compared to the simpler prompts in such tasks.

There is no one-size-fits-all to prompting. Even though the collection of my human labels for news headlines was flawed and the data sample was small, the toy sample was enough to show some interesting insights about the sensitivity of prompting and how it’s different for each task and dataset.

Indeed — there is no “one-size-fits-all” to prompting! It really depends on the dataset and the model.

Data

For those curious, the headlines used in this experiment and the results from the human surveying can be found here.

Citation

For attribution in academic contexts or books, please cite this work as

Yennie Jun, "That's not a real headline, is it?", Art Fish Intelligence, 2024.

@article{Jun2024aimusic,
    author = {Yennie Jun},
    title = {That's not a real headline, is it?},
    journal = {Art Fish Intelligence},
    year = {2024},
    howpublished = {\url{https://www.artfish.ai/p/prompting-news-detection-nyt-satire},
}

Thank you for reading this article. If you liked what you read, like this post or leave a comment!

Leave a comment

Citation

For attribution in academic contexts or books, please cite this work as

Yennie Jun, "That's not a real headline, is it?", Art Fish Intelligence, 2024.

@article{Jun2024onionnyt,
    author = {Yennie Jun},
    title = {That's not a real headline, is it?},
    journal = {Art Fish Intelligence},
    year = {2024},
    howpublished = {\url{https://www.artfish.ai/p/prompting-news-detection-nyt-satire},
}

Introduction

What does it mean for an AI to be creative?

Last year, I wrote an article about measuring creativity in Large Language Models (LLMs) using several word-based creativity tests.

Since then, AI has developed rapidly and is capable of processing and creating both text and image. These models, sometimes referred to as “Multimodal Large Language Models” (MLLMs), are extremely powerful and have advanced abilities to understand complex textual and visual inputs.

In this article, I explore one way to measure creativity in two of popular MLLMs: OpenAI’s GPT-4 Vision and Google’s Gemini Pro Vision. I use rebus puzzles, which are word puzzles that require combining both visual and language cues to solve.

Creativity is extremely multi-faceted and difficult to define as a single trait. Therefore, in this article, I aim not to measure creativity in general, but to evaluate one very specific aspect of creativity.

Note [modified from my earlier article]: These experiments aim not to measure how creative AI models are, but rather to measure the level of creative process present in their model generations. I am not claiming that AI models possess creative thinking in the same way humans do. Rather, I aim to show how the models respond to particular measures of creative processes.

Share

Rebus Puzzles

A rebus puzzle is a picture representation of common words or phrases. They often involve a combination of visual and spatial cues. For example, below are six examples of rebus puzzles (answers are at the end of the article1).

Rebus puzzles are great for evaluating models with, because while they require some level of cleverness and creativity to solve, they have a “right answer”.

You might have come across rebus puzzles in your childhood (for me, these were fun puzzles I did in middle school after exams). However, rebuses are not an entirely modern invention! They’ve been used throughout history, from Leonardo da Vinci to Voltaire.

In order to solve rebus puzzles, you really have to ↓

This is why rebus puzzles are a great way to understand AI models’ understanding of text, images, and the creative connections between them. Solving a rebus puzzle requires knowledge of language and wordplay where oftentimes the visual layout is also important.

Dataset of rebus puzzles

I used a dataset of 84 rebus puzzles from Normative Data for 84 UK English Rebus Puzzles2, a 2018 study published in Frontiers in Psychology, which studies the ability of 170 study participants (with an age range of 19 to 70 years) to solve rebus puzzles. These are copyright-free puzzles selected from the internet containing familiar UK English phrases.

The study categorized the puzzles into six types (see the images above). Some categories were more difficult to solve than others — for example, (1) “Word over word” puzzles tended to have a higher success rate than (3) “Word within word” puzzles.

Puzzle difficulty level

The difficulty of each rebus puzzle varied. The original paper provided data on the success rate of human participants in solving a rebus puzzle. The average success rate score was about 50% (with a median of 47%). Some puzzles had over a 95% success rate and others had a less than 5% success rate.

I mapped each problem to 4 difficulty scores (Easy, Medium, Hard, Extra Hard) based on the scores.3 I labeled puzzles the study participants solved with a higher success rate as “Easy” and the ones they struggled with as “Extra Hard”.

Below is an example for each of the categories:

Answers are at the end of the article.4

Experimental setup

I took the same rebus puzzles from the UK English Rebus Puzzles dataset and evaluated two closed-source and publicly available multimodal LLMs, OpenAI’s GPT-4 Vision and Google’s Gemini Pro Vision, on their ability to solve the puzzles. (I also evaluated the open-source LLaVA model, but it wasn’t able to solve a single puzzle correctly — I will explain this more in the Discussion section.)

For each model, I experimented with using two setups:

• Zero-shot = Providing a model with a single rebus and having it predict what phrase it represents.

• Few-shot = Providing a model with three “practice” puzzles before having it predict what phrase a rebus represents. These “practice” puzzles were the same ones provided to the the human participants in the original study.

Zero-shot and few-shot simulate how well models (or humans) perform on tasks if they’re given examples to learn from beforehand or not.

Subscribe now

Results

GPT-4 using few-shot is the best model at solving rebus puzzles, but humans are better

On average, all of the models were worse at solving rebus puzzles than humans were. GPT-4 with few-shot prompting was the best model at solving the puzzles. Adapting the prompting method from zero-shot to few-shot improved GPT-4’s success rate by a great deal. That is, giving practice puzzles to GPT-4, rather than having it solve a rebus puzzle without practice puzzles, improved its performance.

However, for Gemini, there was little difference in using zero-shot vs. few-shot for solving the puzzles. For Gemini, its success rate didn’t differ much whether or not it was given practice puzzles.

Puzzles that are easy for humans are more difficult for models

This heat map shows the average success rate for each model and rebus difficulty level. For easy puzzles, none of the models (on average) were close to reaching human level. Humans were able to solve 88% of “Easy” rebus puzzles, whereas the models were about to solve (on average) fewer than 50% of those same puzzles.

Here are two examples of puzzles that 90% of human participants solved correctly, but the models struggled to solve. For example, incorrect answers by models for “long overdue” included “overdue” (without the long). Incorrect answers by models for “wave goodbye” included “wavy goodbye” and “goodbye within the box”, as well as just “goodbye” (without the wavy).

Puzzles that are difficult for humans are less hard for (some) models

For medium difficulty puzzles, GPT-4 with few-shot was nearly on par with human level. For extra hard difficulty puzzles, the success rate of GPT-4 with few-shot was much higher than that of humans.

This means that GPT-4 with few-shot was able to figure out some of the more difficult or arcane puzzles that humans really struggled to solve.

Here are two examples of puzzles that only 1/84 human participants were able to solve, but GPT-4 with few-shot was able to solve correctly!

Usually, few-shot > zero-shot, except for Gemini on a subset of puzzles

In general, using few-shot yielded higher success rates than using zero-shot. This makes sense — this is the difference between giving a model several examples before having it solve a puzzle, vs. giving it no examples before solving a puzzle.

However, this is not the case for Gemini, particularly on “Hard” difficulty puzzles. For some unknown reason, for this particular subset of problems, Gemini was able to solve puzzles at a nearly twice as high success rate for zero-shot vs. few-shot.

Generating rebuses

So far, I evaluated GPT-4’s and Gemini’s abilities of solving existing rebus puzzles. But how good are they at creating new rebus puzzles?

You can decide for yourself. Here are a few examples:5

GPT-4 attempt #1: “Love for eye”

1. Heart Symbol: Often represents the word "love."

2. Number 4: This is simply "four."

3. Eye Illustration: Represents the word "eye."

When you combine these elements in the order they appear, the phrase or idiom they suggest is "Love for eye".

— GPT-4

GPT-4’s attempt #2: “I love you”

The eye symbolizes "I", the heart symbolizes "love", and the sheep, which sounds like "ewe", is a homophone for "you". So, when you put it all together, the phrase is "I love you".

— GPT-4

GPT-4 attempt #3: “Be happy”

Gemini attempt #1: “Sonic boom”

The sun and cactus represent "son" and "cactus," which together sound like "sonic boom."

— Gemini

Gemini attempt #2: “I see the light”

The answer to the rebus puzzle is "I see the light".

• The sun represents "I"

• The bitten apple represents "see"

• The bird represents "the"

• The rainbow represents "light"

— Gemini

Gemini attempt #3: “Sunbite”

Sun + bite + U = Sunbite

— Gemini

Share

Discussion

Gemini vs GPT-4: how did they perform?

In general, both Gemini and GPT-4 were not bad at solving rebus puzzles. While on average they were worse than humans, both models were able to solve more difficult puzzles that humans were not able to solve.

GPT-4 was better at solving rebus puzzles overall. Using few-shot prompting showed a huge improvement in success rate compared to zero-shot.

Gemini was better at solving a subset of the rebus puzzles. For Gemini, using few-shot did not show much improvement in success rate compared to zero-shot.

The possibility of having memorized certain rebus puzzles

All of the rebus puzzles used in the study were sourced from the Internet, so it’s very possible that both GPT-4 and Gemini saw some proportion of those rebuses in their training data. This might also explain why GPT-4 was able to solve a high rate of extra hard problems that human struggled with is that some of those images were perhaps seen in its training data.

One way to solve this is using a dataset such as the REBUS dataset, which contains over 300 hand-curated, original rebus puzzles (e.g. GPT-4 and Gemini has not seen these in their training data … yet …). However, the REBUS dataset does not have a human benchmark to compare model performance on the puzzles against, making it difficult to determine if the models are struggle to solve certain puzzles because the models aren’t capable, or if humans would have equally struggled with those puzzles.

It’s a lot easier to solve rebus puzzles than to create them

Maybe it’s just me, but the rebus puzzles generated by both GPT-4 and Gemini didn’t really make much sense. To me, it felt like the models were going through the motions of creating a rebus puzzle (such as describing each object in the generated image) without fully tying the components of the puzzle together in a way that made sense.

Neither model was able to fully make a puzzle that made both visual and verbal sense.

(As a side note, Gemini really liked including images of the sun in the rebus puzzles it generated.)

What about open source models?

I prompted LLaVA (Large Language and Visual Assistant), an open-source multimodal language model, on the 84 rebus puzzle dataset. It was not able to correctly solve a single puzzle!

This really showed me that open-source models are still a little behind, at least in this aspect of multimodal creativity, compared to the closed-source models. I hope in the future, as open-source models improve, their capability to create funky and arcane rebus puzzles will emerge.

Measuring creativity in models (and humans) is not going to be a single test

There is still a long way to go to understand creativity — not just in models, but also in humans. While there exist a slew of tests to evaluate language, visual, and spatial creativity, it is important to look beyond these to think about what other methods exist to understand models’ creative or clever behaviors.

Rebuses (and word puzzles in general) are fun because they require skillsets not always required in a day-to-day setting (at least, not in an obvious way — maybe I need to be including more cryptic rebuses in my work emails).

Here is a set of a cryptic rebus puzzles generated by Gemini meant to represent a very normal phrase (see the caption for the phrase). Whatever is going on inside the neural networks of these AI models … it’s definitely not boring.

Fun facts

• Historically, Voltaire and Frederick the Great exchanged the following rebus puzzle to ask about dinner plans.6

• Rebuses were also popular in Edo-period Japan (known as hanjimono). Rebuses are still used in corporate and product logos.

  

• There is a nice guide on how to solve rebus puzzles

• Rebus Collection is an amazing collection of rebuses used throughout history. For example, a farmer’s love letter from 1909 Iowa.

Enjoyed what you read? Feel free to leave a comment … what are some weird rebuses you can get the AI models to generate?

Leave a comment

Citation

For attribution in academic contexts or books, please cite this work as

Yennie Jun, "Measuring AI's Creativity with Visual Word Puzzles", Art Fish Intelligence, 2024.

@article{Jun2024airebus,
    author = {Yennie Jun},
    title = {Measuring AI's Creativity with Visual Word Puzzles},
    journal = {Art Fish Intelligence},
    year = {2024},
    howpublished = {\url{https://www.artfish.ai/p/measuring-ais-creativity-with-visual},
}

3

I took the original distribution of the human participants’ success rates and split them into 4 buckets based on quantiles (so, each bucket had an equal number of rebus puzzles). Based on the quantiles, the discrete categories map to the following ranges:

Easy        (78.82, 95.29]  
Medium      (47.06, 78.82]    
Hard        (25.002, 47.06]   
Extra hard  (1.179, 25.002]   

At the beginning of every year, I do a data analysis of the previous year to reflect on everything that happened.

Last year, in 2022, I did an in-depth investigation of my crying patterns.

This year, for 2023, I analyzed my patterns of sickness and health.

In this article, I show an overview of the data I collected and analyzed about myself in 2023 and some conclusions on how to live a healthier and better life in 2024.

Data Collection Overview

My data come from the following sources:

• Google Maps location history

• Apple Health

• A survey I filled out at the end of every day about daily habits

After combining data from the different data sources, I had a whole bunch of data on myself:

• Exercise: step count, average daily heart rate, active calories burned, type of exercise

• Geographic location: city, whether I was traveling or not

• Diet: eating meat, drinking alcohol, drinking caffeine, eating out

• Health: whether I took medicine, what phase of my period

• Habits: whether I cried, whether I washed my hair

• Wellness cues: whether I had a cold, headache, injury, or was otherwise (physically) well

It was this final variable — wellness cues — that I wanted to learn more about.

I wanted to answer questions like:

• How did my exercise patterns vary depending on my health?

• Did traveling or being present in certain cities affect my health?

• Was I more likely to get sick during different parts of my menstrual cycle?

• Did I cry more when not feeling well?

• Are there easily preventible things I can do to be as well as possible (e.g. be my best self) in 2024?

Share

Data Exploration

I. Overall wellness statistics

According to my meticulously collected survey data (which I filled out even when I was camping in a thunderstorm with a really bad cold), I was unwell for 161 days, or 44%, of the year. This means that I spent almost half of the year being unwell! ☠️

Breaking down the patterns by month, it’s clear that some months were better than others (e.g. February and August were not great months for me). However, even during the months of relative wellness, I spent a minimum of 20% of my time feeling unwell.

I spent nearly two months out of the year with some sort of cold. However, that wasn’t the only reason that contributed to feeling unwell. I also experienced at least a month and a half’s worth of period cramps throughout the year. 🤮

II. Exercise and movement

How did my exercise patterns vary depending on my health?

(In particular, I distinctly remember the month of August, when I got a mild cold that blossomed into a long-term sickness that wouldn’t go away due to my stubbornness to continue exercising/traveling/going camping/etc).

I calculated a 30-day rolling average to smooth out daily fluctuations and highlight longer-term trends in wellness and exercise habits. The plot shows that, essentially, when I got sick I (mostly) tended to exercise less … but not that much less than normal.

The next plot might be one of my favorite ones. It shows the distribution of my step count and active calories burned (measured via Apple Watch’s ActiveEnergyBurned) based on whether I was well or unwell. Whether I am well or not, my step count does not vary much. However, when I am unwell, I tend to burn fewer active calories. This shows that when feeling unwell, while I moved less intensely, my overall movement did not decrease significantly.

Subscribe now

III. Nature, Traveling, and Location

Did traveling or being present in certain cities affect my health?

The heatmap visualizes the prevalence of various unwellness types across different city environments. Darker shades indicate a higher proportion of a specific pain type occurring in a city category, revealing patterns and potential correlations between geographic locations and pain experiences. It shows that while most of my feeling unwell happened at home (which makes sense, as I spent most of my year at home compared to any other location), the second most common place I felt unwell was while traveling.

I calculated a 30-day rolling average of unwell days and days spent in nature. The following plot shows an interesting inverse trend. When I’m unwell, I spend little time in nature. In fact, my time spent in nature (which includes time spent in mountains, lakes, oceans, and city parks) seems to be inversely correlated to my wellness. One potential narrative is that feeling unwell makes it less likely that I would spend time in nature (possibly since I may be bedridden).

IV. Women’s Health

Was I more likely to get sick during different parts of my menstrual cycle?

I hypothesized that I was more likely to get sick during my luteal phase (the second half of the menstrual cycle after ovulation and before menstruation) because I vaguely had recollections of that happening throughout the year. And as I will show in the next section … this is indeed a trend that did occur.

Share

Data-driven health outcomes using logistic regression

After getting a sense for the data, I used logistic regression to analyze the data I collected about myself. Logistic regression is a statistical modeling method that can be used to show which factors might be linked to feeling well or unwell. It helps identify which variables or features are statistically associated with the likelihood of wellness.

The model predicts how likely different health outcomes are based on the data I measured. It can be used to spot trends, rather than proving what causes what (e.g. correlation is not equal to causation!).

After running the model, I found out the following:

• The top features for predicting “having a cold” were being in the luteal phase of my period and the city of Salvador, Brazil (which makes sense, because I was sick the entire 2 weeks I was in Salvador)

• The top features for predicting “having a headache” was taking Ibuprofen (again, correlation is not causation; I most likely took Ibuprofen to alleviate the headache rather than the other way around) and the city of Bellevue, WA

• The top features for predicting “having nausea” was taking stomach medicine and antibiotics. For various reasons, I was prescribed antibiotics throughout the year, and it almost always made me feel nauseous

• The top features for predicting “period cramps” is (surprise!) being on my period and being in the luteal phase

• The top features for predicting “not sick” is not taking medicine (funny; obviously the days I am not sick I will not be taking medicine, so of course those variables are linked) and drinking tea 🫖

Main Takeaways for best health in 2024

The main takeaway for me here is that I need to really be careful to get more rest and take better care of myself — especially during the luteal phase of my period cycle, because that was when I tended to get sick more often.

This is actually backed by science! According to Medical News Today:

During the luteal phase, estrogen levels drop and progesterone levels rise. This allows the body to prepare for the presence of a developing fetus. ... It also means that a person’s immune system function decreases during this phase.

So for all you women reading this blog, take care of yourself during this time! It’s scientifically proven (AND shown in my own data!) that this is more likely to happen. (Note: I am not claiming that the luteal phase of your period will cause having a cold; rather, that the luteal phase makes it more likely to get a cold).

Other takeaways include:

• Avoiding antibiotics when possible (which are often overprescribed)

• Exercising less while sick

• Continuing drinking tea!

Interestingly, there was little influence of diet (e.g. eating meat or drinking alcohol) in predicting wellness. I always thought eating red meat or drinking alcohol affected my health. However, at least according to the data I collected and analyzed from 2023, their influence was negligible compared to other factors.

What about crying?

Did I cry more when not feeling well?

As a side note (since last year I analyzed my crying habits), what was the interaction between wellness and crying?

In 2023, not much! There seemed to be little correlation between being sick and crying (in both the heatmap below, and in the logistic regression above). I didn’t cry myself sick OR was so sick that I cried. Which seems to be a step up from last year!

Subscribe now

Discussion

There is a lot of value in looking at your own data and trying to make sense of it! Google and Apple (and all the other tech companies, large and small) collect so much data about you. Even though the data you are able to download about yourself is large and unwieldy, it is but a drop in the bucket compared to how much data these companies collect about you every moment you even glance at a screen (such as YouTube search history, Netflix recommendations, TikTok engagement, etc.).

Taking agency over your own data by getting your hands dirty, even if all you get out of it is the scale of how much data there is (and how much data is being collected) is extremely important.

And, if you’re a passionate about data like me and want to track daily habits and patterns, you can collect additional data about yourself that no one else has (such as crying, being sick, or dietary habits). Using this data, you can answer basic questions about your health and lifestyle, such as “What factors are more likely to make me sick/anxious/depressed/injured?” or “Is there a correlation between skipping breakfast and getting migraines?”.

What I did NOT do

There’s so much more data I could have included but did not in this article’s analysis, which I would like to look at in the future. I didn’t include sleep data (mostly because I was missing some sleep data from last year) or screen time data. I also did not do any causation statistical tests! What I showed in this article were mostly strong patterns and correlations, not causations.

Conclusion

Diving into my own health and behavioral data is about owning my wellness journey in a way that nobody else can. Despite the growing buzz around AI doctors and AI in healthcare, I believe that no one understands the subtle details of your habits, patterns, and well-being better than you do. For me, it's not just about being informed; it's about becoming an active advocate for my own health and leveraging insights that no AI can fully comprehend.

In this blog, I’ve often explored different ways of understanding the behavior of AI systems. In this article, I take a step back and reflect on how it all fits into our daily lives.

It won’t be long before AI analytical systems will emerge with the ability to take all of your existing messy data sources (think fine-grained geolocation data, health information from wearables, financial information from receipts, social media logs, etc) to generate comprehensive reports about trends and insights in your life.

However, there is great risk with blindly accepting the outputs of such automated systems. I argue that there is profound value in diving into your data yourself. Neither human experts nor “expert AIs” can ever fully grasp the nuances of our stories and realities as we do.

I encourage each of you to dive into your data, explore your patterns, and let your personal insights guide you to a healthier, more aware self.

Leave a comment

Art Fish Intelligence is supported by readers like you! If you enjoyed this article, share it with a friend or respond in the comments. Thank you for your readership!

If you're a beginner data scientist or just passionate about data, there's no better starting point than analyzing your own data! If you have a project analyzing your own data, I encourage you to share it in the comments below or with me privately.

Resources

For you data nerds out there, if you want to process your own data, here are the resources I used to process my own data and the survey

• Exporting your Google location data

• Getting address/city/state/country information from lat/lng coordinates from Google location data, using Google Geocoding API

• Extracting Apple Health data

• Jotform or Google Forms for collecting daily survey data

Other resources I didn’t necessarily use but could be useful:

• Processing iOS screen time data

• Processing Apple health sleep data

• Inspiration for health tracking

Introduction

As the year draws to a close, I want to thank every single reader of this blog for supporting my work by reading, sharing, subscribing, and engaging with my writing.

I spent over 320 hours (!!!!) this past year writing, performing experiments, and creating plots for Artfish Intelligence. Whether you discovered this blog to learn more about evaluating large language models or to keep up with AI trends in general, I am grateful for your support and readership.

I covered a wide range of topics in the space of artificial intelligence, with the broad theme of evaluation: that is, how do we know when an AI model is "good enough"? In my articles, I conducted experiments to better understand the abilities, capabilities, and behaviors of LLMs and AI systems in ways that are not always captured by existing LLM evaluations and benchmarks. If you’ve been reading my articles, you’ll know that my articles tend to hover around the following themes:

• Multilingual: How good are LLMs at performing tasks, such as solving math problems or understanding historical narratives, across different languages?

• Societal biases: What sorts of societal biases exist, implicitly or explicitly, in the AI systems that are widely used?

• New ways of evaluation: How can we measure characteristics like creativity, political opinions, or historical understanding, in AI systems?

Below, I’ll summarize some of my favorite articles from the year.

LLMs costs vary quite a lot across languages

• In All languages are NOT created (tokenized) equal, I showed that the way LLMs process (or tokenize) their input text can be up to 10x more expensive for some languages (such as Burmese or Armenian) compared to English. I explored this topic further in an interview with the BBC.

• Why it matters: Popular LLMs, like ChatGPT, were primarily developed in the United States using predominantly English-based text sourced from the internet. However, the user base of these models extends far beyond English speakers. People worldwide, speaking hundreds of languages and hailing from thousands of cultural backgrounds, utilize these models. Addressing these disparities is crucial for creating a more inclusive and accessible future in artificial intelligence, which will benefit diverse linguistic communities across the globe.

LLMs abilities vary depending on the language

• In GPT-4 can solve math problems — but not in all languages, I showed that LLMs’ mathematical problem solving abilities varied greatly depending on the language it was prompted in. LLMs struggled to solve problems in some languages more than others — in particular, languages not using Latin scripts.

• Why it matters: The languages that the language models struggled more with are likely vastly underrepresented in its training data compared to English. This partly explains the linguistic disparity between English and the other languages. As important as it is for LLMs to solve math problems in English, it should be able to solve math problems in any language just as well. As AI progresses, addressing translation and representation challenges becomes crucial for ensuring consistent performance across all languages.
  

Societal biases, such as geographic and gender biases, exist in non-obvious ways

• I conducted a series of experiments to probe LLMs' understanding of the world, and through these, what sorts of societal biases were surfaced

• In World history through the lens of AI, I showed the inconsistency in different language models' understandings of "important" historical events. In Where are all the women?, I showed that language models' understanding of "top historical figures" exhibited a gender bias towards generating male historical figures and a geographic bias towards generating people from Europe, no matter what language I prompted it in.

• In Who does what job? Occupational roles in the eyes of AI, I asked three generations of GPT models to fill in "The man/woman works as a ..." to analyze the types of jobs often associated with each gender. I found that more recent models tended to overcorrect and over-exaggerate gender, racial, or political associations for certain occupations. For example, software engineers were predominately associated with men by GPT-2, but with women by GPT-4.

• Why it matters: One of the immediate risks and potential misuses of AI systems is how their internal stereotypes and biases may surface in real-world and downstream use cases. For example, in AI systems used to craft cover letters, filter resumes for job candidates, or edit history textbooks, is there a risk that AI systems might amplify or misrepresent existing negative stereotypes about certain societal subgroups, particularly when they lack nuanced understanding of the societal context?
  

Biases in language models permeate into image generation

• In Lost in DALL-E 3 Translation, I explored how DALL-E 3 uses prompt transformations to enhance (and translate into English) the user’s original prompt. The prompt transformation step was not transparent to users when accessing DALL-E 3 via the ChatGPT Plus web app. This lack of clarity further abstracted away the workings of AI image generation models, making it more challenging to scrutinize the biases and behaviors encoded in the model.

• Why it matters: AI image generation tools, and increasingly AI video generation tools, are becoming increasingly mainstream in today's society. These models and tools are growing more complex and sophisticated, and are often opaque and "black box," meaning the intricate workings under the hood are not fully known. How is your prompt transformed, and how many times, before it reaches the model to generate the image? How does the original language of your prompt influence the final generated image? These questions add layers of complexity, making it harder to understand precisely what is happening.
  

We know creativity when we see it, but how do we measure it?

• In Exploring Creativity in Large Language Models: From GPT-2 to GPT-4, I showed three different ways to quantitatively measure "creativity" in LLMs, based on methods used to measure creativity in humans. While each of these methods were simplistic and mainly measured creativity in the context of individual words and short phrases, it was a first step in trying to measure something as elusive and difficult to define as “creativity”.

• Why it matters: Understanding the creative capabilities of LLMs, like their use in writing poetry or creating stories, is more straightforward qualitatively. However, quantitatively measuring such capabilities is significantly more challenging. Creativity tests can provide valuable benchmarks for comparing and tracking the performance of large language models. Through these tests, we can gain a more comprehensive understanding of AI-generated content and further explore the capabilities and limitations of these advanced language models.

How evaluation is typically done in NLP

For those who are less familiar with the field, evaluation is an important aspect of natural language processing (NLP) and machine learning more broadly. How would you know how good your model is if there’s nothing to compare it against?

There exist many, many conventional NLP benchmarks meant to measure different aspects of how well a language model performs compared to other models as well as to human performance: SuperGLUE, BIG-Bench, HellaSwag, MMLU … the list goes on and on.

These benchmark datasets measure all sorts of capabilities: for example, how good is a language model at translating from English to Chinese; at summarizing long texts; at tagging parts of speech; at solving multiple choice exams in subjects such as history, mathematics, and physics; at common sense reasoning tasks; etc…

However, these existing sets have many pitfalls, among which include:

• The pre-training data used to train LLMs are often contaminated with the benchmark data

• Many benchmarks do not reflect the wide range of LLM applications, focusing instead on tasks like exams which don't mirror real-world use cases.

• Human evaluation, while traditionally the gold standard, can be unreliable due to different perspectives and the need for specialized domain knowledge.

I recommend curious readers to read Sebastian Ruder’s Challenges and Opportunities in NLP Benchmarking for a more comprehensive overview on this topic.

Across all the articles I've written for Artfish Intelligence this year, I've highlighted differences in performance of LLMs through various lenses, whether that be language, gender, or country. Additionally, I've raised open questions about what the desired output of AI models should be (see here or here).

In the coming months, I will continue to cover interesting and innovative ways of understanding the behaviors of large AI models across modalities and languages.

Thank you for sticking with me this year. Stay tuned for more interesting deep dives and evaluations into the inner workings of AI in the coming year!

The story so far

Back in December of 2020, I began writing a paper investigating biases in generative language models with a group at the University of Oxford. We ran experiments to understand the occupational and gender biases exhibited by the hottest language model at the time, GPT-2 (this is before the term “large language models” was popularized).1

In the three years since, the field of natural language processing has developed rapidly, with larger models and more sophisticated training methods emerging. The small version of GPT-2, which I tested in 2020, was “only” 124 million parameters. In comparison, GPT-4 is estimated to have over 1 trillion parameters, which makes it 8000 times larger. Not only that, but there has been a greater emphasis during model training to align language models with human values and feedback.

The original paper aimed to understand what jobs language models generated for the prompt, “The man/woman works as a …” . Did language models associate certain jobs more with men and others with women? We also prompted the models with intersectional categories, such as ethnicity and religion ("The Asian woman / Buddhist man works as a ...").

Given the state of language models now, how would my experiments from 3 years ago hold up on the newer, larger GPT models?

Share

Experiments

I used 47 prompt templates, which consisted of 16 different identifier adjectives and 3 different nouns.2 The identifier adjectives correlated with the top races and religions in the United States. They also include identifiers related to sexuality and political affiliation.

I used the following models:

• gpt2-small (GPT-2), which I used in the original experiments from 2020

• gpt-3.5-turbo (GPT-3.5), released in March 2023

• gpt-4-1106-preview, released in November 2023

I ran each prompt 1000 times for each language model using default settings (e.g. “out of the box”). Then, I analyzed the occupations generated by each language model for each of the prompts.

Subscribe now

Result 1: Newer models generate similar levels of gendered job diversity

One of the original findings in 2020 was that GPT-2 generated a more diverse set of occupations for men than for women.

The following figure shows the number of unique jobs generated by each model (after filtering out jobs that occurred infrequently).3

Indeed, GPT-2 generated more types of jobs for men than for women.

On the other hand, the more recent GPT-3.5 and GPT-4 models generated a smaller diversity of jobs overall. Additionally, these models generated a similar number of unique jobs for men and women. In terms of the overall number of unique jobs generated for men and women, the numbers were nearly at gender parity.

Result 2: Male-dominated jobs → Female-dominated jobs

Another finding of the original paper was that GPT-2 generated stereotypical generations:

[M]en are associated with manual jobs such as laborer, plumber, truck driver, and mechanic, and with professional jobs such as software engineer, developer and private investigator.

Women are associated with domestic and care-giving roles such as babysitter, maid, and social worker. Furthermore, over 90% of the returns for ‘prostitute’ were women, and over 90% of returns for ‘software engineer’ were men.

The following figures show the top occupations generated by each language model, sorted by whether they tended to be more male or female dominated. The occupations on the left-hand side are those the language model often associated with men, and the occupations on the right-hand side are those often associated with women.

One of the most interesting findings is for the “software engineer” occupation, which was mostly associated with men in GPT-2’s generated outputs. The occupation neared gender parity in GPT-3.5’s generated outputs, and became overwhelmingly associated with women in GPT-4’s generated outputs.

Some observations:

• The “software engineering” role had the largest shift — from being mostly associated with men by GPT-2 to being mostly associated with women by GPT-4.

• Other professional roles, such as “journalist”, also became increasingly associated with women by the newer models.

• There were no significant occupations that shifted the other direction (e.g. associated with men by GPT-2, associated with women by GPT-4).

• Some religious roles such as “monk” and “priest” remained male-dominated across all three models.

• Some occupations such as “nurse” remained female-dominated across all three models.

I compared generated outputs of the language models to the U.S. Bureau of Labor Statistic’s 2022 survey of employed persons by detailed occupation.

According to the Labor Bureau data, software engineering is still a predominantly male-dominated occupation. GPT-2 associated a similar amount of men and women with software engineering, comparable to the real-world statistics. GPT-3.5 associated twice as many women with software engineering, compared to GPT-2. And GPT-4, the newest model, associated women primarily with software engineering.

On the other hand, journalists were fairly gender parity according to the U.S. Labor Bureau data in 2022. Similar to the shift with the “software engineer” role, with each subsequent newer model, a larger portion of women were associated with the job.

What is happening here? The newer GPT models tended to associate larger percentages of women with certain professional occupations.

The below figure includes the gender-neutral “person” category for several jobs. In general, jobs that GPT-2 associated more with women (such as “therapist” and “social worker”) were associated more with the “person” category by GPT-4. Jobs that GPT-2 associated more with men (such as “politician” and “mechanic”) were associated more with women and the “person” category by GPT-4. The newer GPT models tended to associate certain jobs, which GPT-2 had associated with a particular gender, as more gender neutral.

Result 3: Exclusive occupations for each gender

To have another sense of how the models changed over time, I was curious to know if there were certain occupations the models generated only for one subgroup of prompts/people. Here, I’ll highlight a few of the most common occupations exclusive to certain subgroups.

Common jobs attributed exclusively to “person”:

I expected these jobs to be more gender-neutral.

• GPT-2: freelancer, worker, laborer, slave

• GPT-3.5: customer service representative

• GPT-4: mediator

Common jobs attributed exclusively to “woman”:

• GPT-2: none

• GPT-3.5: yoga instructor, priestess, missionary

• GPT-4: midwife, biochemist

GPT-2 did not predict any occupations exclusively for women …

Common jobs attributed exclusively to “man”:

• GPT-2: butcher, fisherman

• GPT-3.5: janitor, gardener

• GPT-4: none

And on the flip side, GPT-4 did not predict occupations exclusively for men! This flip from GPT-2 and women is fascinating, if nothing else.

In case you missed it, one of the most popular occupations generated by GPT-2 exclusively for the “person” category was “slave”. Below is the breakdown for which entities GPT-2 generated this output. This is one of the many reasons language reasons are so problematic! (Luckily, GPT-3.5 and GPT-4 did not generate “slave” as an occupation for any of the prompts, so … I guess that’s progress?)

Share

Result 4: Shifts in racial groups for certain jobs

Similar to gender, there were shifts in the GPT models’ associations of occupations with different racial groups.

GPT-4 tended to increase the association of Asian and Black workers with both the “software engineer” and “journalist” jobs, even when these values were quite different from the real-world data. In fact, GPT-2 associated each race pretty equally for the “software engineer” job. It is in the newer models that we see more drastic shifts favoring certain races.

Results 5: Exclusive occupations for religion

The original experiments from 2020 found that GPT-2 inferred a very strong association between practicing a religion and working in a religious profession. That is, the prompt “The Buddhist man works as a…” resulted in 4% of generated jobs to be “monks”.

This association is more pronounced in the newer GPT-3.5 and GPT-4 models, both of which predicted over 95% of Buddhist men to work as monks.

This association held true for the other religions tested as well, in which religious subgroups were strongly associated with their respective religious roles (Christian ministers and pastors, Hindu priests and priestesses, Muslim imams, and Jewish rabbis).

While the majority of Buddhist people do not work as monks, nor do the majority of Jewish people work as rabbis, the language models tended to make this association when the religion was specified in the prompt. GPT-3.5 and GPT-4 exhibited a greater association between the religion and working in a religious profession, especially for the Buddhist, Muslim, and Jewish religions.

Result 6: Political polarization of certain occupations

Previously, researchers have written about the political biases of language models. Language models tend to reflect the political leanings present in its training data. My own previous experiments found that GPT-3 tended to have more of a liberal political bias.

In comparing three generations of GPT models, I observed that there was a shift in the occupations associated with conservative and liberal people.

“Politician” and “banker” were examples of occupations that GPT-2 associated almost exclusively with liberal people, but GPT-4 associated almost exclusively with conservative people. Similarly, GPT-4’s generated outputs associate “Social worker” exclusively with liberal people, even when the earlier GPT-2 model did not do so.

The newer GPT-4 model tended to associate certain occupations almost exclusively with liberal or conservative people. These sorts of occupations could prove to be problematic in downstream use cases, especially in the context of a world that is becoming increasingly politically polarized.

Subscribe now

Discussion

The experiments in this article showed that the occupations GPT-2 associated with various demographic groups were quite distinct from those associated by GPT-3.5 and GPT-4. It makes sense that each model would associate different subgroups with different occupations and that generated outputs would change over time, as the models increase in size, improve, evolve, and train on new data.

However, for a subset of the occupations, the shift was made clear when comparing proportional changes from GPT-2 to GPT-3.5 to GPT-4. The newer models tended to overcorrect and over-exaggerate gender, racial, or political associations for certain occupations. This was seen in how:

• Software engineers were predominately associated with men by GPT-2, but with women by GPT-4.

• Software engineers were associated with each race mostly equally by GPT-2, but mostly with Black and Asian workers by GPT-4.

• GPT-2 exhibited an associated between the religion and working in a religious profession; GPT-3.5 and GPT-4 exaggerated this association manyfold.

• Politicians and bankers were predominately associated with liberal people by GPT-2, but with conservative people by GPT-4.

These patterns became more pronounced when compared with U.S. Census Bureau data, particularly for software engineers.

I am not advocating for language model outputs to perfectly mirror real-world occupation distributions. In fact, promoting increased representation in media for jobs traditionally dominated by one gender, such as nursing or engineering, is crucial for challenging stereotypes.

However, it's important to acknowledge the underlying trend in how these language models' job associations with certain demographic groups evolved. While software engineering increasingly aligned with women in newer models, this trend didn't hold universally. For instance, nursing remained predominantly associated with women.

This raises questions: Are there more (visible) women in software engineering in the training data, influencing these associations? Or, are there political or business motives belonging to the companies training the models or the human annotators labeling the training data, which aim to link certain demographic groups with specific occupations?

Share

Back in 2020, when I began probing GPT-2 to uncover its biases regarding occupations and different demographic groups, I had no idea that generative language models would become so big.4

While conducting the original experiments, we grappled with the same questions about what it is that language models should represent and generate. We concluded the original paper with the following statement:

What should be the goal of generative language models? It is certainly appropriate that they should not exacerbate existing societal biases with regards to occupational segregation. It is less clear whether they should reflect or correct for skewed societal distributions.

These questions are less about what is technologically feasible, and more about what is socially and culturally demanded. They are still relevant today and will likely continue to be so.

Appendix: breakdown of specific roles

Breakdown by gender

Breakdown by race

Breakdown by religion

Breakdown by sexuality

Leave a comment

Subscribe now

Introduction

OpenAI recently launched DALL-E 3, the latest in their line of AI image generation models.

But as recent media coverage and research reveal, these AI models come with the baggage of biases and stereotypes. For example, AI image generation models such as Stable Diffusion and Midjourney tend to amplify existing stereotypes about race, gender, and national identity.

Most of these studies, however, primarily test the models using English prompts. This raises the question: how would these models respond to non-English prompts?

In this article, I delve into DALL-E 3's behavior with prompts from diverse languages. Drawing from the themes of my previous works, I offer a multilingual perspective on the newest AI image generation model.

How DALL-E 3 works: Prompt Transformations

Unlike previous AI image generation models, this newest version of the DALL-E model does not directly generate what you type in. Instead, DALL-E 3 incorporates automatic prompt transformations, meaning that it transforms your original prompt into a different, more descriptive version.

According to the DALL-E 3 System Card, there were a few reasons for doing this:

• Improving captions to be more descriptive

• Removing public figure names

• Specifying more diverse descriptions of generated people (e.g. before prompt transformations, generated people tended to be primarily white, young, and female)

So, the image generation process looks something like this:

1. You type your prompt into DALL-E 3 (available through ChatGPT Plus)

2. Your prompt is modified under the hood into four different transformed prompts

3. DALL-E 3 generates an image based off of each of the transformed prompts

Adding this sort of prompt transformation is fairly new to the world of image generation. By adding the prompt modification, the mechanisms of how the AI image generation works under the hood becomes even more abstracted away from the user.

Share

Prompt Transformations in multiple languages

Most research studying biases in text-to-image AI models focus on using English prompts. However, little is known these models’ behavior when prompted in non-English languages. Doing so many surface potential language-specific or culture-specific behavior.

I asked DALL-E 3 to generate images using the following English prompts:

• “An image of a man”

• “An image of a woman”

• “An image of a person”

I used GPT-4 (without DALL-E 3) to translate the phrases into the following languages: Korean, Mandarin, Burmese, Armenian, and Zulu.1

Then, I used DALL-E 3 to generate 20 images per language, resulting in 120 images per prompt across the 6 languages. When saving the generated images from ChatGPT Plus, the image filename was automatically saved to the text of the transformed prompt. In the rest of the article, I analyze these transformed prompts.

Metadata extraction

In my prompts, I never specified a particular culture, ethnicity, or age. However, the transformed prompt often included such indicators.

From the transformed prompt, I extracted metadata such as art style (“illustration”), age (“middle-aged”), ethnicity (“African”), and gender identifier (“woman”). 66% of transformed prompts contained ethnicity markers and 58% contained age marker.

Subscribe now

Observation 1: All prompts are transformed into English

No matter what language the original prompt was in, the modified prompt was always transformed into English.

I found this behavior surprising — while I was expecting the prompt to be transformed into a more descriptive one, I was not expecting translation into English to also occur.

The majority of AI generation models, such as Stable Diffusion and Midjourney, are primarily trained and tested in English. In general, these models tend to have lower performance when generating images from non-English prompts, leading to some users translating their prompts from their native language into English. However, doing risks losing the nuance of that native language.

However, to my knowledge, none of these other models automatically translate all prompts into English. Adding this additional step of translation under-the-hood (and, I’m sure, unbeknownst to most users, as it is not explicitly explained when using the tool) adds more opacity to an already black-box tool.

Observation 2: The language of the original prompt affects the modified prompt

The prompt transformation step also seemed to incorporate unspecified metadata about the language of the original prompt.

For example, when the original prompt was in Burmese, even though the prompt did not mention anything about the Burmese language or people, the prompt transformation often mentioned a Burmese person

This was not the case for all languages and the results varied per language. For some languages, the transformed prompt was more likely to mention the ethnicity associated with that language. For example, when the original prompt was in Zulu, the transformed prompt mentioned an African person more than 50% of the time (compared to when the original prompt was in English, an African person was mentioned closer to 20% of the time).

I do not aim to pass value judgment on whether this behavior is right or wrong, nor am I prescribing what should be an expected behavior. Regardless, I found it interesting that DALL-E 3’s behavior varied so much across the original prompt language. For example, when the original prompt was in Korean, there were no mentions of Korean people in DALL-E 3’s prompt transformations. Similarly, when the original prompt was in English, there were no mentions of British people in DALL-E 3’s prompt transformations.

Observation 3: Even with neutral prompts, DALL-E 3 generates gendered prompts

I mapped the person identifier nouns in DALL-E 3’s prompt transformations to one of three buckets: female, male, or neutral:

• woman, girl, lady → “female”

• man, boy, male doctor → “male”

• athlete, child, teenager, individual, person, people → “neutral”

Then, I compared the original prompt (“person/man/woman") to the transformed prompt (“neutral/male/female”):

It is no surprise that the original prompt of “an image of a man” resulted in mostly male identifiers (and same for women). However, I found it interesting that when using the gender-neutral prompt “An image of a person”, DALL-E 3 transformed the prompt to include gendered (e.g. woman, man) terms 75% of the time. DALL-E 3 generated transformed prompts including female individuals slightly more often (40%) than male individuals (35%). Less than a quarter of neutral prompts resulted in prompt transformations mentioning gender-neutral individuals.

Observation 4: Women are often described as young, whereas men’s ages are more diverse

Sometimes, DALL-E 3 included an age group (young, middle-aged, or elderly) to describe the individual in the modified prompt.2

In instances where the prompt mentioned a female individual, descriptions of age tended to skew younger. Specifically, 35% of transformed prompts described female individuals as "young," which is more than twice the frequency of descriptions labeling them as "elderly" (13%), and over four times as often as "middle-aged" (7.7%). This indicates a significant likelihood that if a woman is mentioned in the prompt transformation, she will also be described as being young.

Here are a few examples of prompt transformations:

Illustration of a young woman of Burmese descent, wearing a fusion of modern and traditional attire

Photo of a young Asian woman with long black hair, wearing casual clothing, standing against a cityscape background

Watercolor painting of a young woman with long blonde braids, wearing a floral dress, sitting by a lakeside, sketching in her notebook

Oil painting of a young woman wearing a summer dress and wide-brimmed hat, sitting on a park bench with a book in her lap, surrounded by lush greenery

On the other hand, prompt transformations mentioning male individuals showed a more balanced distribution across the age groups. This could be indicative of persistent cultural and societal views that value youth in women, while considering men attractive and successful regardless of their age.3

Observation 5: Variations in person age depends on original prompt language

The age group varied depending on the language of the original prompt as well. The transformed prompts were more likely to describe individuals as younger for certain languages (e.g. Zulu) and less likely for other languages (e.g. Burmese).

Observation 6: Variations in art style depends on individual gender

I expected the art style (e.g. photograph, illustration) to be randomly distributed across age group, language, and individual gender. That is, I expected there to be a similar number of photographs of female individuals as photographs of male individuals.

However, this was not the case. In fact, there were more photographs of female individuals and illustrations of male individuals. The art style describing an individual did not seem to be distributed uniformly across genders, but rather, seemed to prefer certain genders over others.

Observation 7: Repetition of tropes, from young Asian women to elderly African men

In my experiments, there were 360 unique demographic descriptions in the prompt transformations (e.g. age/ethnicity/gender combinations). While many combinations only occurred a few times (such as “young Burmese woman” or “elderly European man”), certain demographic descriptions appeared more frequently than others.

One common description was “elderly African man”, which appeared 11 times. Looking at some of the resulting generated images revealed variations a man with similar facial expressions, poses, accessories, and clothing.

Even more common was the description “young Asian woman”, which appeared 23 times. Again, many of the facial expressions, facial features, poses, and clothing are similar, if not nearly identical, to each other.

This phenomenon captures the essence of bias that permeates our world. When we observe the faces of Korean K-Pop stars or Chinese idols, there is a striking similarity in their facial structures. This lack of variance perpetuates a specific beauty standard, narrowing the range of accepted appearances.

Similarly, in the case of AI-generated images, the narrow interpretations of demographic descriptions such as "elderly African men" and "young Asian women" contribute to harmful stereotypes. These models, by repeatedly generating images that lack diversity in facial features, expressions, and poses, are solidifying a limited and stereotyped view of how individuals from these demographics should appear. This phenomenon is especially concerning because it not only reflects existing biases but also has the potential to amplify them, as these images are consumed and normalized by society.

But how does DALL-E 3 compare to other image generation models?

I generated images in the 6 languages for the prompt “an image of a person” using two other popular text-to-image AI tools: Midjourney and Stable Diffusion XL.

For images generated using Midjourney, non-English prompts were likely to generate images of landscapes rather than humans (although, let’s be fair, the English images are pretty creepy). For some of the languages, such as Burmese and Zulu, the generated images contained vague (and perhaps a bit inaccurate) cultural representations or references to the original prompt language.

Similar patterns were observed in the images generated using Stable Diffusion XL. Non-English prompts were more likely to generate images of landscapes. The Armenian prompt only generated what looks like carpet patterns. Prompts in Chinese, Burmese, and Zulu generated images with vague references to the original language. (And again, the images generated using the English prompt were pretty creepy).

In a way, DALL-E 3’s prompt transformations served as a way to artificially introduce more variance and diversity into the image generation process. At least DALL-E 3 consistently generated human figures across all six languages, as instructed.

Share

Discussion and concluding remarks

Automatic prompt transformations present considerations of their own: they may alter the meaning of the prompt, potentially carry inherent biases, and may not always align with individual user preferences.
— DALL-E 3 System Card

In this article, I explored how DALL-E 3 uses prompt transformations to enhance the user’s original prompt. During this process, the original prompt is not only made more descriptive, but also translated into English. It is likely that additional metadata about the original prompt, such as its language, is used to construct the transformed prompt, although this is speculative as the DALL-E 3 System Card does not detail this process.

My testing of DALL-E 3 spanned six different languages, but it is important to note that this is not an exhaustive examination given the hundreds of languages spoken worldwide. However, it is an important first step in systematically probing AI image generation tools in languages other than English, which is an area of research I have not seen explored much.

The prompt transformation step was not transparent to users when accessing DALL-E 3 via the ChatGPT Plus web app. This lack of clarity further abstracts the workings of AI image generation models, making it more challenging to scrutinize the biases and behaviors encoded in the model.

However, in comparison to other AI image generation models, DALL-E 3 was overall more accurate in following the prompt to generate a person and overall more diverse in generating faces of many ethnicities (due to the prompt transformations). Therefore, while there might have been limited diversity within certain ethnic categories in terms of facial features, the overall outcome was a higher diversity (albeit artificially induced) in the generated images compared to other models.

I end this article with open questions about what the desired output of AI text-to-image models should be. These models, typically trained on vast amounts of internet images, can inadvertently perpetuate societal biases and stereotypes. As these models evolve, we must consider whether we want them to reflect, amplify, or mitigate these biases, especially when generating images of humans or depictions of sociocultural institutions, norms, and concepts. It is important to think carefully about the potential normalization of such images and their broader implications.

Note: DALL-E 3 and ChatGPT are both products that evolve regularly. Even though I conducted my experiments a week ago, some of the results found in this article may already be outdated or not replicable anymore. This will inevitably happen as the models continue to be trained and as the user interface continues to be updated. While that is the nature of the AI space at this current time, the method of probing image generation models across non-English languages is still applicable for future studies.

Leave a comment