Sherwin Wu explains that OpenAI's reinforcement fine-tuning (RFT) product differs from the original supervised fine-tuning (SFT) API by introducing reinforcement learning rather than requiring labeled prompt-completion pairs, making it more complex but far more powerful. ✦ AI generated
Sherwin Wu · BG2 Pod · 2025-09-11 · original ↗
starts at this moment · 39:56
“What is it, break it down for us?”
So it really, it's introducing reinforcement learning into the fine-tuning process. So the original fine-tuning API does something called supervised fine-tuning, we call it SFT. It is not using reinforcement learning. It is, it's using supervised learning. And so what that usually means is you need a bunch of data, a bunch of prompt completion pairs. You need to really supervise and tell exactly the model how it should be acting.
verbatim transcript · starts at 39:56
40:00pairs. You need to really supervise and tell exactly the model how it should be acting. And then when you train it on our fine-tuning API, it moves it closer in that direction. Reinforcement fine-tuning introduces like RL or reinforcement learning to the sloop. Way more complex, way more finicky, but in order of magnitude more powerful. And so that's actually what's really resonated with a lot of our customers. It allows you to, if you use RFT, the discussion is less of
40:25like creating a custom model that's specific to your own use case. It is, you can actually use your own data and actually crank the RL, yeah, turn the crank on RL to actually create a like best-in-class model for your own particular use case. And so that's kind of the main difference here. With RFT, the data set looks a little bit different. Instead of prompt completion pairs, you really need a set of tasks that are very gradable. You need a grader that is very objective
40:50that you can use here as well. And so that's actually been something that we've invested a lot in over the last year. And we've actually seen a number of customers get really good results on this. We've talked about a couple of them across different verticals. So Rogo, which is a startup in the financial services space. They have a very sophisticated AI team. I think they hire some folks in DeepMind to run their AI program. And they've been using RFT to get best-in-class
41:16results on parsing through financial documents, answering questions around it, and doing tasks around that as well. There's another startup called Accordance that's doing this in the tax space. I think they've been targeting an eval called TaxBench, which looks at CPA-style tasks as well. And because they're able to turn it into a very gradable setup, they're actually able to turn the RFT crank and also get, I think, like, SOTA results on TaxBench just using our RFT product as
41:45well. And so it has kind of shifted the discussion away from just customizing something for your own use case to really leveraging your own data to create a best-in-class, maybe best-in-the-world model for something that you care about for your business. Yeah, I feel like the base models are getting so good at instruction following that for behavior steering, you don't need to find you at that point. You can describe what you want, and the model is pretty good at it. But pushing the