I wish I saw this point earlier in reference to my recent post.
I tried creating a “GPT” using their wizard form, but I found that it performed poorly because it would not reference the user doc and reference manual docs that I had uploaded. I also couldn’t simply paste the documentation into the GPT’s description (I think that’s where you insert it) due to the character limit in the form.
However, as of April 2024, I believe ChatGPT uses GPT-4 Turbo by default, which has a much longer context window. I had decent results pasting both documents, giving few-shot complex examples (thanks to the forum contributors), and then explicitly asking the GPT to only use the provided context to answer my query for a new script.
I think as LLMs progress (context window gets larger) and if we can craft a lot of examples for the context window, we could probably get even better results!
Now, I wish this was all automated since it’s a lot of work to collect all the answers + queries found on this forum to “train” the base model (insert examples into the context window) and update the prompt whenever the official documentation/reference/API gets updated.
Perhaps we could build a script that automates the creation of the prompt by fetching the documentation from GitHub, and another script to help us build a database of Q/A prompts to generate the initial prompt.