Chatty Chat Bot, Spinner of Tales

When ChatGPT first came out, I initially was quite excited! After gaining access to the simple web chat page, and playing around with GPT 3.5 for a while, I started to probe about its limitations (which is critical when considering what applications the tech may have). Some things were obvious -- there was a ~4000 token size limit (1 token ~ 1 word, but is closer to 4 characters), so any 'conversation' needed to come in under that limit. Otherwise, the chat bot simply could not provide any more completions. Other things were more difficult to solve: how do you ensure that a chatbot never ends a story? They really, really like to do that on each completion.

The token limit I find similar to how embedded devices operate today -- the program itself takes up a sizable portion of the storage on its own; any leftover storage can typically be used for working memory. While databases can store many orders of magnitude more data than can be held in working memory, if the program's computation doesn't fit in working memory the problem suddenly gets more cumbersome. This too, seems to hold true for the chatbots of today (at least for now). The token size limit is actually a ceiling on the sum of 3 things (sans API overhead) -- the system message, the prompt from the chat user, and the completion provided by the chatbot.

The first app that I created to experiment with the OpenAI Chat API was a rudimentary storytelling narrator. The narrator would write anyone who joined the chat into the existing storyline, and then allow them to continue the story from their perspective. I must admit, the initial results of this experiment were pretty cool. The narrator bot (most of the time) would pick up the names of any new participants (via updating the system message I was sending to the API) and bring them into the story in a fairly generic way. Stories have a habit of being far more than 4000 words however... If I wanted to have the narrator elaborate at all I had to be really careful about prompting it to only return a set amount of words (which it didn't always obey).

Well if I can't have more than 4000 tokens, I figured, I'll just use that as a limiter that makes the storytelling more exciting.