Mercor CPO on Revenue Concentration from Frontier Labs

Oswald Nitski: I wouldn't say that open source model improvements cannibalize our core business because data is most valuable on the frontier of model performance. So each of our customers has their own unique goals and is purchasing eval training data sets to fill gaps in current model capabilities. Open models just raise the floor of what people are interested in. As long as customers still have new capabilities that they want to get better at, our business still continues to grow. Open source models just mean that nobody's buying anything that can already be done.

Oswald Nitski: I don't think there's an ROI problem right now. I think we're in a period of exploration and experimentation where there's more tolerance, more patience to get that ROI calculation right now. There's a lot of different projections around where token prices will go, where performance will go, and right now we're starting to see some amount of tightening of the screws on spend here and there. But I think the paradigm we're in is still let's see what happens because things are moving so quickly that the ROI calculation might shift too dramatically still.

Oswald Nitski: So, I can answer this from a how it affects the product team. We would love to move like our biggest challenge is moving down market so that every single enterprise can efficiently run human data projects for eval and training. And that'll diversify our revenue for sure because there's many more enterprises than there are labs. And that's a harder product to build. And that's the direction that we are taking our products, taking the company — to be able to self-serve projects very efficiently, have like AI project managers so that it's a lot easier to do this work for smaller customers because running a human data project for a lab is incredibly hard. It's a white glove service that requires a lot of people on the operations team. As we make that more efficient with better products, better processes, we can do smaller projects that are more heterogeneous for more customers. It's the direction we have been heading which has reduced concentration and it's the direction that we'll continue to head as every enterprise begins to have human data work for their proprietary use cases.

Oswald Nitski: We're basically like a tech-enabled services company. Our services are incredibly valuable in driving revenue gains for our customers primarily through better model capabilities. Evals and training data are the primary bottlenecks in model performance right now. If every enterprise needs to have specialized proprietary models, even if the capabilities start to saturate, the eval serves as the PRD for exactly what you want, but also the optimization objective for better performance. As long as better models are valuable to the economy, there will be demand for eval sets and training sets. If we can make that process faster and faster, we can serve a growing demand for human data eval training. And then we also have a growing agent deployment enterprise arm as well.