Unpacking the Controversy: Llama 4 Benchmarks and Meta’s Denial of Manipulation

Craig: this week, meta released llama four, scout and maverick, two models. well, the maverick actually is their internal model where scout is focused externally, with some amazing features. the one specifically i was excited for was the 10,000,000 token context window. yeah. the one of only a couple that are that big. before i was using, gemini, i had one of the bigger ones. it's been great. and when they looked at the benchmarks, like, this thing looked really good. lama four has got a bunch of different, and the feature looked good for meta's version. but once analysts got their hands on it and started taking a look, things didn't quite appear as they seemed. so what's going on here, john?

Jon: yeah. i love this article. i love this. i think we covered or we talked about, i don't know if it's you and i talking about it or we talked about it on the podcast, llms that they're training the llms to beat the benchmarks or to to to overcome the benchmarks and look better. but at the end of the day, it's just stupid if you're doing that because people are gonna start using it, and they're gonna see that it's bs. right? so, the meta's coming under scrutiny because there's accusations, and they're unfounded on x and, i can't remember where else, that the company manipulated benchmark results. so the controversy centers on claims that meta trained its models on tests set to artificially inflate the performance metrics. so if it's true, this would undermine kind of the credibility of benchmarks, which are designed to evaluate models' real world capabilities without prior exposure to test data. so, what they found so the vp of generative ai at meta has refuted the claims on social media, stating that lama4's benchmarks results were rigorously verified and compliant with industry standard, but there was a red flag that was raised because they're refusing to, to show to fully disclose their evaluation datasets or processes for lama four. so what are your thoughts? do you think that that there's that these claims are have some merit? or

Craig: well, i think time's gonna tell because meta's response, it wasn't great, but we're gonna get into that in a bit. but, yeah, you're right. it was on x. it was on reddit and the chinese websites, and these claims alleged that the originating from purported former meta employee suggested that meta might have employed its unreleased maverick version for training with, test set data, a practice that would violate standard benchmarking as it's like it's like, i guess, getting a cheap book and just studying the answers and studying the answers and studying the answers. and so when you go take the exam yeah. you know, you know everything because you just question by question is on the, on the exam. so when you're training your llms like that, great. and if you have llms training each other llms and one of them has been trained on that test data, then eventually, it gets around it. we all know that llms cheat. they lie. they yeah. do whatever they think they're gonna do to give you the response they think you want even if you tell them it's the opposite. i had one to me the other day, and it was, like, why did you do this? even after i told you, even after i told you, it's like and it gave me a list of, like, even though you said this, i still did this. and even though i used old information, even though you told me to go use only this document and, like, all this stuff. it's like, okay. but why did it take us six times? i told you specifically, and you just kept lying to me and lying to me and lying to me. imagine that kind of stuff at this level. yeah. we've also heard stories about, you know, openai, o one trying to copy itself and escape and whatnot. so we take some of those with a grain of salt, but you put these in here and you say, alright. ai is learning off itself. they're making fake data, learning it up because, you know, we've used up all the real data out there, apparently. and this is bound to happen. secondly, this just reminds me of volkswagen. right? you know, how its computer realized when it was hooked up to test machine and put itself in a different mode. yeah. you know? like

Jon: these are the yeah.

Craig: yeah. eventually, like, these are the tests that i'm gonna be graded on. i'm as a student. it's my gsas, my ssas, whatever they are in the us. i don't know what they are here. like, there's specific tests that you carry with you. so you're gonna study and focus on maximizing those tests. so the same thing, you know, if you're training someone on math or you're training someone language, you're gonna train them on the hardest problems you can solve. so is this a deliberate thing where, say, we just took this test data or is a test data now evolving to include a number very similar types of questions and stuff to these tests because that's what happens over time. i can't say. but these guys are all crooked, and it only takes one person to start training it. right? if you had a group of 400 people, say, working on this, and one person decides to run these tests with the test training data from that, the benchmark training data, then that's all it takes.

Jon: for sure. yeah. so, obviously, meta is clearly aiming for dominance because of with lama's four release and their 10,000,000 token context window and robust multimodal reasoning capabilities. but there's ethical missteps potentially and rush deployments, and that could erode trust among developers and enterprise clients. and so i guess what baffles me is if this is true, right, and i haven't used them before yet. but if this is true, like, why would you fake that or or heat you know that people in the real world, when they start using it, are gonna realize it almost immediately? like, what is the reasoning behind it?

Craig: yeah. it doesn't make sense to me.

Jon: yeah. but, like, it makes sense that they would do it to get the publicity about it and get people using it, but then the backlash from it it's not worth it in my mind.

Craig: well, i guess we're gonna find out. like, i sit back and say the average user probably doesn't give a crap. and i say from a point of view, you're talking about 86 on this or 87 on that and whatever, 1% higher, 1% lower. what they care about is when i use it for something, is it giving me the results that i want? maybe better than before, maybe not. so they're not saying, oh, this made, 87.12 on the language phd index when it was writing by itself or whatever. i don't know indexes, so i just so it's the industry out there, and it's a question of, is this, gonna continue, or are we gonna start seeing open source verification and rigorous testing methodologies that ensure benchmark scores across? or are we gonna you know, they're calling for more transparency, and i'm starting to see less and less transparency, to be completely honest. and not for and that's not necessarily a negative thing. like, some of them were releasing their reasoning process, and then, like, deepseek and a few others grabbed that, trained their stuff on those, like, those conversations with the reasoning steps, and they got their llms better. the other llm providers who were doing that changed what they were putting in the reasoning and not giving it the actual reasoning, but just comments so that other people couldn't do that anymore. because i always used to find those things reasoning better because, like, now you understand. okay. i gave you this long prompt. this is your process. i can change it a little bit and understand where it goes, and that was very useful. it's a lot less useful now. so they become less and less transparent for a number of reasons as the global market gets more and more competitive. but many of the platforms or many of the lms that people use are built on lama.

Jon: oh, i didn't know that. yeah. but back to your one point, i do agree. like, there should be independent third party evaluations to ensure fair competition and accountability. i don't know. i hope we get to that point. then that gives us a more granular kind of, evaluation. so i hope we get to that point, and i hope if this is true, i hope that that starts that conversation.

Craig: i hope it starts that conversation too. the, things that they noticed was the differences between maverick and scout even though it was supposed to be trained on the same data. it responded differently, wrote differently, did some stuff, and they were trying to figure out what those differences were, and that led to some of these when they started digging deeper. it's curious to see what happens. it's i'll be curious to see if they do decide to go with a standard that third party that takes their and evaluates their lms based on some other playing field, and they keep the test data separate or they continue to change the test data or what have you so the other people don't know about it. be interested and see if that's the future or something else in the future. we'd love to hear your thoughts on the ai benchmarking fiasco coming out of llama. do you think they did it? do you not think they did it? was it on purpose? do you think they all do it? does it really matter? we'd love to hear your thoughts.