LLM content design for UX writers: a day in the life

This project started with a nice surprise from an ML engineer.

He reached out to UX with a content problem, the kind of task that gets a product writer’s wheels spinning.

“OK, so how do I do that now?”

Introducing the context

Companies like OpenAI and Anthropic develop, host and provide access to LLMs. Agents use those LLMs as their reasoning engine to complete tasks, among other cool things.

When a user chats with an agent, the agent sends requests to a chosen LLM via a provider’s API. The LLM generates an output, and the agent uses that output to produce the final response for the user.

Every now and then, though, something goes wrong with an LLM provider:

• An API key expires – error 401 (Unauthorized)
• You’ve sent too many requests in a short window or hit your rate limit – error 429 (Too many requests)
• The service isn’t available due to server issues, maintenance or other reasons – error 503 (Service unavailable)

Until recently, the conversation design solution would be heavily manual. We’d map the most frequent edge cases (the 2-3 main error codes) and prepare a fallback response to handle the rest. Rotating responses from a pool of available options was possible too.

Then we started bringing LLMs into the process.

Understanding the use case

The ML engineer explained that we’d have a specialized LLM generate dynamic messages based on the error’s status code and description.

This way, we would no longer need to re-work static microcopy on a case-by-case basis. Error messages would become context-sensitive, thus more relevant and timely.

But the LLM started producing outputs that were a little too obscure for the target audience. The unedited output didn’t look like something the average business user would be able to understand or act on.

That’s when the engineer came to the UX team for support. He was looking for specific instructions and examples on what the messages should and shouldn’t say, the level of detail and any style choices to follow.

So I started putting together a simple, lightweight framework that writers would be able to reuse.

Reviewing the base output

To start with, the engineer showed me an example of a system-generated error message.

If an API key isn’t valid, the client returns something like:

Client error '401 Unauthorized' for [URL]. For more information check [status code]

When asked to take that error and turn it into a user-facing message, here’s what the LLM generated in three different calls:

The error occurred because your request to the [LLM provider] API is missing valid authentication credentials (API key). Please verify your API key is correctly configured in the request headers.

The error occurred because the request to [LLM provider]'s API lacks valid authentication credentials (API key missing/incorrect).

The error "401 Unauthorized" indicates your request to the [LLM provider] API lacks valid authentication credentials (e.g., an API key).

After seeing this, I had a clearer idea of how to make the output more effective. A lot has to do with context, clarity and flow.

First of all, this is meant to be a conversational message. But the LLM jumps straight into describing why an error occurred, without even signaling to the reader that something didn’t go as expected. It skips that step altogether.

Then, the LLM mentions the provider name. This is something business users may have never seen before. And they’re not supposed to: they’re just having a conversation with an agent. How would they know the question they asked got routed to a service that isn’t working right now?

However, an admin will get what this is about, and they have the access rights needed to look into the issue. The LLM’s message needs to address both audiences.

But even if someone is able to follow up on the message, they may receive instructions only in one out of the three calls above. This should be standardized as well.

And, to wrap up this quick overview: familiar, common words would help the average user understand what needs to be done. The error isn’t only for developers and engineers; business users should have the tools to take action too.

Writing instructions

It was now time to decide what to do next. 

I sketched the framework to follow when reviewing LLM-generated content. Then, I started thinking about what would go in the “Instructions” container in this specific case.

Language

First, I read through our style guide one-pager. The LLM already follows most of the internal editorial guidelines: language variant, grammar, syntax, spelling, punctuation, capitalization, acronyms and so on.

In this case, I wanted to strengthen the prompt with a bit about vocabulary and terminology. We should avoid overly technical terms that can confuse, alarm or break immersion for average business users.

Instead, we can state what’s wrong or missing (e.g., “API key”, “authentication details”, or the provider’s name) to guide an admin-level user in case the error is passed on to them.

However, the message shouldn’t mention the code and description string as-is (e.g., “401 Unauthorized”), as it doesn’t help non-technical users.

Structure and content

The error message should consist of two or three sentences at most. It should contain the following:

• what went wrong (the first sentence or first part of the message)
• why (if relevant to understanding the error);
• how to fix it (the second sentence or last part of the message)

Context and expectations

This is the most important portion of the instructions as it’s specific to each business and customer scenario. I’d argue this is the part that will influence the output the most.

In our case, who’s on the receiving end of the LLM-generated error message is likely a read-only user. This means they don’t have access to the product area where they can check and fix the issue.

The read-only user may not be aware of this underlying structure, but they should be able to notify an admin who can actually go and check.

Examples

Now for some few-shot prompting. Based on the instructions above, I provided a set of examples that the LLM can use as a reference.

It looks like I can't get to the information you need right now. This usually means the [LLM provider] API key needs updating. Ask a [role name] to check it, or try a different question in the meantime.

I couldn't complete your request. The [LLM provider] authentication details on my side seem to be out of date. Have a [role name] take a look, or feel free to ask something else while this gets sorted out.

It seems I'm unable to access what you're asking for at the moment. The [LLM provider] API key is likely missing or incorrect. Check with a [role name] to get it updated, or try a different question for now.

Testing and evaluating

The ML team ran a first test applying the instructions to the 401 (Unauthorized) status code. Here are some examples of how the LLM responded in three different calls:

It looks like I can't complete your request right now because the [LLM provider] API key isn't working. Ask a [role name] to update it, or try a different question in the meantime.

I wasn't able to sign in to [LLM provider] to answer your question. Check if a [role name] can update the API key, or try asking something else for now.

I couldn't get through to [LLM provider] just now. This usually means the API key needs updating. Ask a [role name] to check it, or try a different question while this gets resolved.

Overall, the LLM followed the instructions quite closely, resulting in better output quality.

Here’s a breakdown of what worked and what to refine.

What worked:

• Style: the editorial guidelines are consistent
• Value: overall, the messages are clearer and more actionable than before
• Focus: the messages focus on the solution and on users who can’t access admin-only areas

What can be improved:

• Content: one of the outputs seems to frame the issue as information missing on the model side, while the real cause is an authentication problem.
• Structure: in the first sentence, the LLM still introduces technical concepts that a business user might find obscure or confusing (e.g., named providers or APIs). Since this is a conversational message, it could lead with a “what went wrong” introduction, then explain the solution in the following 1-2 sentences for progressive disclosure.
• Length: LLM inference can produce longer outputs at times, but we can still highlight that they should be more concise.

Refining the output

I strengthened my instructions with the points above and additional examples.

It looks like I can't reach [LLM provider] to answer your question right now. The API key probably needs updating. Ask a [role name] to check it, or try a different question in the meantime.

Sorry, I wasn't able to complete your request. The [LLM provider] key I'm using doesn't seem to be working anymore. Have a [role name] update it, or feel free to ask something else while this gets sorted out.

I couldn't sign in to [LLM provider] just now, so I can't get you that answer. This usually means the API key is out of date. Check with a [role name] to get it updated, or try a different question for now.

Here’s how the LLM responded after this round.

I can't answer that right now. There seems to be a problem with the connection to the service. Ask a [role name] to update the [LLM provider] API key, or try again later.

Something's not working on my end, so I can't get you that answer. The [LLM provider] API key likely needs updating. Check with a [role name], or try a different question in the meantime.

I'm unable to complete your request at the moment. This usually means the [LLM provider] API key is out of date. Ask a [role name] to update it, or try again later.

Scaling the approach

The guidelines were originally tested with the 401 error, but they’re designed to apply to similar scenarios as well.

Now that we have more solid instructions, we can extend them to the other error codes, 429 (Too many requests) and 503 (Service unavailable).

Too many requests:

I can't answer that right now. A lot of requests are coming in at once, so things are running slowly. Please wait a moment and try again.

I couldn't get you an answer just now. The service is handling a high volume of requests at the moment. Give it a minute, or try a different question in the meantime.

Something's holding things up right now, so I can't complete your request. This usually clears up on its own pretty quickly. Try again in a little while.

Service unavailable:

I can't answer that right now. The service is temporarily unavailable, possibly for maintenance. Please try again in a little while.

I'm unable to complete your request at the moment. The service seems to be down for now. Try a different question in the meantime, and let a [role name] know if it keeps happening.

Something's not working right now, so I can't get you that answer. This is usually temporary. Try again later, or let a [role name] know if it continues.

Wrapping up

We’ve seen that it’s possible to quickly improve outputs in a limited number of iterations, then incorporate any relevant feedback until you get the desired result.

With a robust UX framework, writers can generate dynamic, contextual messages that apply to a variety of scenarios.

You already know how to maintain clear, well-structured guidelines over time. And you also understand the specific user context you’re writing for. Add close collaboration with ML engineers to the mix, and you’ll have a reusable process that scales beyond error messages.

Let me know what you think. Do you write for LLMs? How do you collaborate with ML engineers?

READ MORE FROM ME

Ciao 👋 I’m Elisa, an Italian product writer who believes good design is service. This is where I document my work in UX.

Go to the blog