How to use generative AI in copywriting for an A/B testing program

AI fans like to rhapsodize about all the great things they can do with ChatGPT and other AI assistants. Here’s a glimpse into how my agency uses an advanced form of ChatGPT to help us with copywriting for an email A/B test.

Although I can’t reveal everything — I need to protect my client’s privacy and competitive information — I’ll share as much as possible, including the steps we used to build the test and how ChatGPT helped us get there.

Say hello to Chad

When we started working together, I told our private instance of ChatGPT that his name is Chad, and my name is Katie. Since then, Chad has become a valuable staff member. At first, we needed extra time to train him in how we do things at my agency. But it was time well spent because he now delivers what we’re looking for with just a few refinements.

Chad takes away the terror of the blank white page. He helps us launch our work faster, reducing the time we spend spinning our wheels trying to find a good starting point for writing copy. Then, he steps aside, waiting for us to edit and refine, do the actual work or give him another task. 

We use a paid version of ChatGPT (currently it uses GPT-4o) because it keeps both ours and our client’s intellectual property out of the training pool and gives us access to GPT-4o’s memory feature, among other benefits. This saves us hours of update time, streamlines our operations and focuses our prompt writing on other tasks so that we’re always working with our most updated information. 

Also, because we use the paid service that has been trained with our own information, everything Chad produces is inspired by our work, not by whatever he picks up from outside sources.

Dig deeper: How to use AI in email marketing without losing the human touch

Using genAI in copywriting for testing

Chad meshes with our advanced testing methodology because it is far more extensive than just comparing a variant against a control. We develop the hypothesis and bespoke strategy for the testing program and turn Chad loose on creating those variants once we give him his prompts.

What is holistic testing?

Holistic testing looks at all of the factors that go into optimizing an email campaign. We test entire emails against each other according to the hypothesis we created for the test. This goes beyond simply testing subject lines or calls to action. 

I created this method to derive more reliable insights about our customers — not just which subject lines get more opens in a single test but what motivates them to act. We can apply these insights throughout the company because our email subscribers represent our customer base.

The campaign goal

We wanted to persuade more people to view and order their personalized products. We used orders as our success metric. We tracked open and clicks, but they were not the primary measurements.

The test hypothesis

A well-formed hypothesis is the foundation for any testing program, whether simple or complex. It gives you the “why” for the test, what you expect will happen and the parameters for setting up and running the test.

This was our hypothesis, which we wrote using what we learned in previous testing for this client and for others:

“We believe the email body copy that incorporates at least one element specifically tailored to appeal to each of the four personality types — spontaneous, methodical, humanistic and competitive — will generate more web visits compared to the control version of the email, which lacks this personalized approach. 

“This increase in web visits is anticipated because the personalized content is likely to resonate more effectively with a broader range of recipients, engaging their specific interests and motivations, thereby enhancing the likelihood of them visiting the website and converting.”

Testing set-up

The control is the email the client had been sending, with no copy personalized to reflect any of the four shopping motivations mentioned in the hypothesis. The variant, which we composed with Chad’s assistance, includes copy that would appeal to each of those four motivations. We split our testing audience into two random sections and sent the control to one and the variant to the other.

Process

Here is the prompt-and-response process that we used to create the copy for the variant email: 

Prompt 1: We ask Chad to familiarize himself with Bryan Eisenberg’s Four Buyer Modalities.

• Response: Chad summarizes the key characteristics of each of the four modalities (personalities).

Prompt 2: I ask Chad to familiarize himself with my client’s company, its products and brand tone of voice. I also asked Chad to list the benefits of my client’s products.

• Response: Chad provides a detailed summary and an accurate list of benefits. Now Chad is in a good place to begin to help me.

Prompt 3: I tell Chad what we want. I asked him to update and improve the client’s version of the campaign email to reflect our hypothesis and create the copy for Variant B. I added the hypothesis to the prompt, uploaded an image of the email and asked Chad to extract the copy from it. He returned the copy in text form so we could copy and adjust it. We also asked for some refinements.

• Response: Chad presents updated copy. It reflects the copy changes I requested. His comment:
  • “Here is the revised email copy, incorporating elements tailored to the four personality types: spontaneous, methodical, humanistic and competitive. … This version should appeal to a broader range of recipients, engaging their specific interests and motivations. Let me know if you need any further adjustments!”

Prompt 4: I provide more specific direction, asking him to blend the copy into one paragraph addressing all four personalities, not one paragraph per personality as he had provided.

• Response: Chad incorporated those changes in a revision, remarking, “This should now appeal to a broader range of recipients from the start. Let me know if there are any more adjustments needed!”

Done! I now have the first draft of an opening paragraph that incorporates the elements I outlined in our hypothesis. The copy presents the benefits clearly and concisely while including elements targeted toward the four shopper personalities I mentioned in the hypothesis. 

Edit and improve. Now that Chad had helped to create the first draft of Variant B, it was time for me to roll up my sleeves and refine it, so that it became the final version. Using the single paragraph plus the four paragraphs previously provided, I created two versions and asked the team to vote on which version they preferred and to rationalize why. This was a beneficial exercise, as it meant they began to understand each of the personalities.

Dig deeper: 3 ways email marketers should actually use AI

4 tips for creating a good relationship with your Chad

Here are my tips for learning and using genAI to streamline your workload and improve your email marketing and testing:

1. Take the time to get it right

The process I outlined here uses higher-level genAI. If you have not worked with genAI, large language models like GPT or chatbots like ChatGPT, don’t try this yet. Instead, spend time learning the process.

As with any skill, you need time to become proficient at using genAI. Your first attempts likely will not be as productive as ours, but neither were ours when we began working with chatbots. But we kept working away and eventually learned how to speak to a non-sentient being without relying on unspoken cues and clues.

Experiment with your prompt writing. This is an essential part of using genAI effectively. If you don’t put time and effort into writing prompts that your chatbot will understand, you won’t get good results and you’ll probably spend just as much time going back and forth with your chatbot as you would have if you had gone it alone.

I have never used a published prompt. I consider Chad to be a member of our team (a young, eager junior) and so I treat briefing him as just that. I give him a brief — not a prompt — of what I want him to do and achieve. 

When you think like this, you innately provide your bot with everything it needs to deliver the request. About 70% of the time, I will ask him whether he has everything he needs and tell him to ask me questions to ensure he has all the necessary information. 

As the saying goes, “Garbage in, garbage out.”

2. Pay for a specialized service

Here’s another saying: “You get what you pay for.” Yes, you can use free AI assistant services like ChatGPT. ChatGPT’s free service is a good training ground for learning prompt writing and working effectively with the assistant. 

However, your work is not necessarily protected from being used to train other users. When you are ready to incorporate your own competitive or sensitive information, use a paid service that you can configure to keep your inputs and results private.

3. Always work from your hypothesis

Your prompts should reflect elements of your hypothesis so that you are testing for reliable results. 

4. Be nice to your bot

Chad’s not human, but we treat him as if he were. We gave Chad a name because it helps us frame questions better as if we were briefing a human coworker. Plus, we get better results when we frame our prompts in full sentences and frame our responses with “please” and “thank you” — just as if you were briefing a team member.

I focused my attention here on using generative AI specifically for copy testing. If you would like advice on using chatbots for other parts of your email program, such as writing campaign briefs and message copy, read two of my previous MarTech articles:

I hope this backstage look shows you how working with Chad has eliminated many steps from the creative process without short-circuiting it or compromising our or our clients’ competitive information. Try it for yourself, and let me know how it works for you!