A pretrained model only becomes a useful chatbot after fine-tuning on question-answer data and RLHF, where human raters' judgments define what counts as a 'good' response.

46:31the model so that its output actually gives us more of a chatbot-like response rather than just a fancy autocomplete. And then finally, that fine-tuned model goes through one more step of tuning known as RLHF or reinforcement learning from human feedback. Essentially, real humans are hired to sit down and rate the responses of the model. Good chatbot-like responses get ranked high, bad responses get ranked low. Now, what defines good and bad is

46:59entirely up to the company that trains the model, and also up to the judgment of those human rankers. Now, the result of all of this is just a model that behaves like a chatbot, right? It answers questions. It's able to have back-and-forth conversations, but it's still just a standalone model, right? It's It's a weights file. It just lives on a server somewhere. It doesn't know any more than it's actually been trained

47:19on, and it can't reach outside of itself. It It can only produce next tokens. So, this is where the idea of tools come in. You can essentially include a list of available tools and how to call them in the context of your request. Things like get the current weather or search the web for the specific term or run this code file or search the files on your file system.

47:38So, these tool descriptions essentially tell the model what they can perform, and then the models are further fine-tuned to output tool calls. Or instead of just outputting a response like a helpful chatbot, it will actually output structured code that says, "Please call this tool." So, the model itself isn't reaching out over the web or anything like that. It's literally producing a JSON object that says, "Call this tool." And then that's paired with

47:59a harness, like a code editor or a desktop app, that can see that the output of the model was a tool call, and then the harness can in turn actually do what the tool call said to do, like execute code or search the web. The harness will then take the result of that cool tool call and append it to the conversation history. So now the next call to the model has all of that

48:18relevant output information in the context, so it can produce a more accurate answer. Now, there's many other techniques that are used in modern LLMs to increase the quality of the types of answers that it outputs, like mixture of experts and thinking modes, but that's all we're going to cover at a high level in this video. So if you're interested in any of those, let me know down in the