The AI assistant stumbles a bit...
Not every day is Skittles and rainbows
In my last post I talked about how I’m experimenting with creating custom GPTs that I configure using uploaded Word documents. Today we’ll talk about that a bit deeper, I’ll dive a bit into the structure of one of these GPTs, and also acknowledge that the technology isn’t perfect.
But first…
As many of you know my role at Microsoft was eliminated as part of a company-wide layoff back in May that affected more than 6,000 people. Fortunately an old friend reached out quickly about my possibly joining her firm and I’m pleased to announce, that’s just what I did. So as of Monday, July 14th, I’m now an Innovation Strategist at Affinity Consulting. A return to legal tech, full time.
I’ll be helping law firms and legal departments be successful with Microsoft 365 and AI. In the recent past I’ve had a handful of folks ask if I’m available to consult with their firm on AI or Microsoft 365 and now the answer is YES!
Microsoft Legal Technology Expert Ben Schorr Joins Affinity Consulting Group | LawSites
Now let’s go deeper on creating assistants
I’ve long said that what I really want is Jarvis from Ironman. That AI assistant, hovering, knowledgeable, accessible, focused on what I need, ready to lend a hand. The new crop of LLMs like ChatGPT and Copilot get us a long way towards that, but out of the box they’re still just like the information desk at the mall - they’re eager, but they’re strangers. They don’t really know what you’re looking for, they don’t really know you, so their answers are generic (and occasionally wrong, but we’ll come back to that).
What we want is that assistant that knows everything it needs to know to help you with whatever you’ve asked for. If you’re creating a GPT to help you plan a business trip it can be helpful if the GPT knows your travel preferences (airlines, hotels, do you like to sit in the aisle or window, what airports do you like or refuse to fly through, etc.).
One way to address that is in your prompts. You can (and should) give the AI specific instructions in your prompt. But just as with a human assistant it would be a little annoying if you had to tell it every single time all the things you want it to know about you in order to give you the assistance you need.
Hence the custom GPT, which lets you set up a set of instructions that it’s going to work with every time. You could give it all of that information in the Instructions box, but…
It’s not the best interface for large amounts of text. It’s a smallish box and you’re typing plain text. And there’s an 8,000 character limit.
It doesn’t let you use formatting to delineate headers, though you could use XML tags.
If you have a lot of text, it may not be easy to find and make updates to the things you want to tweak.
Does it work? Yes, but it’s not elegant and for regular users it may not be very comfortable.
So, as I mentioned in my earlier post, I’m trying to improve on that by putting context and instructions into Word documents and then uploading those Word documents to my custom GPT. At least in theory, the assistant will then have a rich set of context and instructions to use when it responds, which should significantly improve the quality and usefulness of its responses.
❗Important: I have a paid license for ChatGPT which provides additional privacy protections. I would not be uploading any personal information into a free AI platform.
Additionally I’ve gone into Additional Settings and unchecked the box that lets OpenAI train their model on my conversations with my assistant.
The structure of the document
For this example I’m going to use a custom GPT that will be helping with house hunting. My wife and I are looking for a new house and I thought it would be helpful to have an AI assistant. So I created a custom GPT for this, and named it “Steven”. In the instructions I gave it some direction on how I wanted it to respond, voice and tone, and that sort of thing.
Then I created a Word doc called “House hunt” that I uploaded to Steven. Here’s what the structure of the document looks like:
Importantly you’ll note that I’ve used Word heading styles to give the document some structure and help the assistant parse it more accurately. If this were a plain text (TXT) file instead I could do the same thing by using XML tags (<must haves> <nice to haves> etc.) to delineate the various sections.
When I want to update the document I just open a saved copy in Word, make the changes, then replace the version ChatGPT has with the new version.
I’m still experimenting with the best ways to lay the document out, but let me talk you through how it’s currently organized.
Note: The example I’m giving is a house hunt, but you could use basically this same technique for any kind of assistant you wanted to create.
The “zones”
The first part of the document breaks down the geographic areas that we’re interested in. We’re looking at sets of neighborhoods in distinct places and want to be able to refer to them collectively. In the document these take the form of a bulleted list of towns/neighborhoods for example:
ZONE 3 (Tier 1)
Camas, WA
Vancouver, WA
Washougal, WA
We can have as many zones as we want, though at some point it gets a bit unwieldy. Grouping them as zones like this lets me have conversations with the GPT about one or more zones without having to specify them each time.
The “Tier x” tag is an attempt to convey priority. The Tier 1 zones are our preference, the Tier 2 zone is someplace we’re considering but we’re less excited about. I’m not sure that’s really working so far, though, but it’s a recent addition so I’m experimenting.
As a concept, though, grouping neighborhoods into zones has worked fine. The GPT has been able to identify that a property is in Zone 1 or Zone 2, and I can ask questions about the various zones such as about local regulations around ADUs or who the local utilities are.
This technique can be helpful to group like things together so you can discuss them with your assistant as a group. You don’t have to call them “zones” of course, use whatever terminology makes sense to you.
Must haves
The must haves section is a bulleted list of things we require in our next house. For example we want at least 4 bedrooms, it needs to have a room that can be a good home office, it needs to have good schools, etc.
Here’s where you’re telling your assistant what’s essential to you in this use case. If it’s a deal it’s helping you put together you might specify what the absolute minimum amount of $ you require is, or what dates or deadlines are essential to be met.
Nice to haves
This is a section that lists things we want, but that are less important than the Must Haves section. For example, a room with a fireplace that can be used as a library. It can be helpful to give your assistant a sense of priority if it has to choose between two things it should know which are more important to you.
This section is more aspirational. You’ve told your assistant what you must have, now you’re telling it what you’d like to have on top of that. If the deal could also include the buyer covering closing costs, for example, or a 30 day transition period.
Absolute dealbreakers
Now that we’ve told the assistant what we want, we also want to tell it what we DON’T want. These are the qualities that should disqualify any property.
For you these are the lines the assistant should not cross. Under no circumstances will we accept payment in currency other than dollars, for example.
More details
For some of the must haves and nice to haves, such as the playroom or the library, I’ve added a paragraph or two of added detail that I think might be useful for the assistant to know when it’s reviewing.
Examples
AI does better when you can give it examples to refer to. The examples we have here are addresses of properties that are very close to what we want.
If you were asking it to create content you could paste in links to similar content that you wanted it to model on. If you were asking it to review contracts you might paste in links to successful contracts to emulate.
Even the best fall down sometimes
So how has this been going? Generally it’s been going pretty well. The assistant does generally refer to my document and the conversations I’ve had with it have been detailed and specific. For the most part it’s identifying properties that suit our needs, and when it doesn’t I’m able to gently correct it (and update the document) to refine the results.
Is it perfect? No. Is it helpful? Yes, quite helpful.
But…
This morning I grabbed my morning beverage, fired up Steven, and asked it to identify any new listings in Zone 2. It proceeded to list a few properties in London (wrong continent) that didn’t really seem to align to any of our requirements. It’s almost as if Steven couldn’t see our document at all and was just falling back to basic GPT.
A couple of my other AI assistants are exhibiting similar behavior, seeming to have lost awareness of the uploaded instructions. And ChatGPT in general just seems a bit slow and disoriented today. I can’t help but wonder if something is happening on the back-end that is resulting in service degradation. Hopefully soon it will be back to normal.
I guess just like your human assistant can have a sick day, maybe today is a sick day for ChatGPT. 🤧
It’s an important reminder, though, that AI is not “set it and forget it”. You still need to review the output to make sure the AI is doing what you want it to do.
Never show the output of an AI to a client or the court without reviewing it first.
Summary
Giving the AI deeper context and instructions is key to getting better results. You can do that with direct instructions or by uploading a file that contains that information. In my early experiments the direct instructions may be a little more reliable, but the uploaded file gives you a lot more flexibility, easier updating, and allows for more information to be shared.
❓Have you tried using uploaded files to give the AI more context or instruction and, if so how did it go? Any lessons you learned that would improve my assistants?