$20k is what it would cost you or me, but it’s just free for them.
No it isn’t. This is not regular software where the bulk of the price is the licensing. With slope-as-a-service, the bulk of the price is the data center operation cost - which Anthropic is certainly not getting for free.
99% pass rate? Maybe that’s super impressive because it’s a stress test, but if 1% of my code fails to compile I think I’d be in deep shit.
Also - one of the main arguments of vibe coding advocators is that you just need to check the result several times and tell the AI assistant what needs fixing. Isn’t a compiler test suite ideal for such workflow? Why couldn’t they just feed the test failures back to the model and tell it to fix them, iterating again and again until they get it to work 100%?
Considering how these companies are losing money because they subsidize these tokens - I doubt that cost is really absorbed.