One AP class's exam pass rate went from 20% to 60% after the teacher switched to retrieval practice

In this episode of The Nick Stanley Show, we're joined by the authors of Make It Stick, The Science of Successful Learning, who share powerful new strategies on how to learn more effectively. There's no good experiment that I know of that shows that matching its function to the learning style produces better learning. In some cases, miss matching instruction to the learning style actually produces better learning. We dive into the science behind effective learning, revealing why traditional methods like rereading and highlighting may be holding you back. Students are rereading or rereading and as they read the study, I know this, I know this, I know this.

Without the people's practice, they have no idea whether they know it or not. Earlier, a student, professional, or lifelong learner, this conversation will transform how you approach learning. If you want long-term retention, use the strategy that might hurt you initially, but then it helps you in the long run. Are you ready to learn more effectively? Let's dive in. Are students studying as effectively as they can be or could be? No. The typical students choose to read material, highlight it, underline it, then reread it before the exam. And they don't do, most of them don't do anything much fancier than that.

And there's so many better ways they can be spooning their time. Well, let's start with the problem then. So what is the problem with rereading things, underlining things, highlighting things, and then checking all that important stuff that you underlined and highlighted? Nothing per se is wrong with that. You need to do it to start, but then to make things stick, you need other techniques that are more elaborate. And I'll talk about one of them right now, Marc. I can talk about several others. One is called retrieval practice, which is just simply either the teacher gives you a test or you test yourself and you start practicing generating information out of memory.

We tend to think of, in fact, teachers talk about building your storehouse of knowledge as of getting something into memory, looking at it, having it there is all you need to do. But really, what you need to do is also practice and getting it out, not just getting it in, because after all, that's what you need to do, use the information you have stored. And so one way to do that is simply, I mean, a lot of people do it say when you're learning the multiplication tables, you know, practice six times eight and four times nine and all the others over and over, you have a flash card or something that says six times nine on one side and you try to know what it is.

And if you don't, or even to make sure you're right, you can't look all over. Well, that simple kind of flash card use we're suggesting can be used on much random scales that you could, you know, push yourself on. As a question to increase yourself on she curbs in the back of the chapter to college student. There are all kinds of ways to use being sure you have practice. And we talk about the narrow book, make it sick, the science, the successful learning. So that's one strategy. I'll turn it over to Mark to talk about others. One and real quick, real quick, before we jump to to Mark, yeah, and I am a huge fan of the book that you gentlemen worked on together, make it stick.

If if more students read it on our expose to the incredible insights that are contained in that book, they are going to learn to study more effectively. They're going to do better in school. But more importantly, they're going to, they're going to be more effective in life because learning is a superpower if we can do it effectively. And so you mentioned there, Roddy, before we move on to the next technique, you mentioned a using forms of self testing. And that is a theme that runs throughout the book. And it sounds really simple, right? Like using flash cards when we tried to memorize our multiplication tables.

But as it turns out, self testing is one of the keys to figuring out what you know, what you actually don't know. Like you said, you're practicing getting it out of your memory, not just stopping at the part where we read something. And it's it's rambling around on there somewhere. But there's no practice for the pulling it back out. And I just wanted you to expand on that just a little bit more because it's such the self testing is such an important strategy. Yes. Well, one thing you could do say when reading a text book, I mean, it's find a highlight and circle a key term or something like that that you need to know.

But say you read a section of the chapter, and it would be easy enough just to kind of close your book and say, what did I just read? What can I remember from what I just read, say the last five pages? And then if you do that, that will help you a lot later. And if you can't do that right after you read it, which answer you have later. And so then you can go back and look at the sections where you had trouble. And so that would be one good way to study. It takes more time, of course, but most things that better take more time or more afterful. Okay. Yeah, well, and those I had a student come in to my office the other day who she's in college now, and she was struggling with a micro biology class.

And I asked her about how she was studying for the test because she said, I feel like I know this information. And then I keep getting C's, D's and F's on these exams. I don't know what's going wrong. And as we dug into what's happening during the test and the during her studying, she said, I'm studying so much that when I see the question, I can visualize the color of the highlighter I used on the text passage. So I can remember that that was pink, but I can't remember the information. And I thought it was a really insightful, extreme example where this was a student who was rereading, rereading, rereading, I mean, she's spending all her free time.

She couldn't have tried any harder. And yet there was no self testing in the process. So she didn't know what she needed to reread or what she knew, what she didn't. And she has now started down this path of self testing and we're staying in touch to see how it goes. But on the very first exam, there was a market improvement. And I think as she continues to practice these strategies, we're just going to see more and more improvement going forward. That's a great story. Nick, I'd like to return to your initial statement, which was that rereading does seem to be effective, maybe highlight and so on.

And I think we need to emphasize why that's the case. Yeah, it's illustrated in your example, with this student you're working with. The problem with rereading is that often it doesn't invite deep engagement. Often it's kind of a superficial surface processing. So the reason that when you reread the content, it's familiar. And familiarity is a cue that the brain uses to evaluate whether you know something or not. But it turns out familiarity is a very misleading cue. It gives you the illusion of knowing. So students are rereading or rereading. And as I read this, I know this, I know this, I know this.

And it's just kind of that familiarity gives them the impression they really know it. Without retrieval practice, they have no idea whether they know it or not. And other aspect of rereading is that it's done with a lot of our fluency. So in cognitive science, we know at every level when you reread your brain processes the effort, rational or flow, but at the word level, at the syntactic level, at the level of extracting ideas, all of it's more fluid. Fluencies are more peculiar than the brain uses evaluate how well you know something. And it's also a mistake. It can be very much weak. So the student may experience and will experience fluency rereading at low levels, fluency at the word identification level with syntactic processing level levels that don't contribute directly to deep understanding.

So they think and the signals are telling them they understand the material. What is that is an illusion. So rereading is it gives the student a misleading sense of confidence about the material. And because of that, they don't engage or a deep rereading. Retreatable practice is very helpful as Ronnie suggested before you reread. See how much you can remember from the passage. Now, the student knows exactly what they don't know what they do know. When they can retrieve something that not really know when they can't, then they're having a little difficulty with it. Like the students, you might remember the color of the highlighting, but if you can't retrieve the content, it tells her exactly that that material is not in memory well enough to retrieve it.

And the neat thing is we do basic studies on it, Ronnie and I, that after you do retrieval practice, when you reread, now you're rereading section, you don't know, spending much more time on those in sections you do know. If you don't have retrieval practice, your study policy is driven by low accuracy meta cognition, and the cognition is what you think you know what you think you don't know. So if you reread without retrieval practice, you're reading everything equally, or you're reading things you already know more, and all of that makes for inefficient study and efficient learning. So I'm not suggesting students should reread, but what they need to do is they had need to incorporate these other techniques I can talk about a few more into rereading, doing these techniques prior to rereading, and now the rereading is going to be much more effective and much more active.

So I teach a course on learning strategies, and the student reflections once students said, when my free recall technique, which is to try to recall that information before she reread, she said, it helped me more actively engage in the material when I was rereading, and that's exactly the point. So you want your rereading to be effective, you want to always know what you don't know for table practice, how to do that, and you also want to process the material done more deeply. And there's a term for that. Roddy just studied this a lot, which is test-detached learning. When you try to retrieve and fail, it turns out that reviewing the material after failing to retrieve leads to more effective learning, and if you just give a student that material to reread.

So when you can't get things out of memory and look at it again, now you're figuring out where are my obstacles? Why couldn't I remember it? And you start to think about the material way that allows you better access later on. You connected to prioriology, connected to other things that you are rereading in order to use that as a bootstrap to what you can't retrieve. So there are so many benefits to retrieval practice. Retrieve will itself, produces better long-term memory. Well, I just got a concentration like that. I've got to get it, it makes it increases your accuracy of knowing what you know about now.

That increases your efficiency of directing your study, and having to try to retrieve also increases your efficiency of learning material you couldn't retrieve. All of those things are just dynamite or whatever. So just to summarize here, we've got rereading by itself leads to familiarity with essentially the writing and the information as it's constructed in whatever it is you're reading. You become more familiar with how someone else has written it, but you're not necessarily internalizing the deep concepts in it. And then retrieval practice or self-testing, however we want to refer to that, which can come in the form of flashcards, it could come in the form of just on asking yourself questions and then elaborating on what you read.

Can you answer those questions? Those are ways to self-test to see if you actually are familiar with the material. And then if you're not, then we can go back and spend more time even on very complex things, right? Because it's easy to understand these concepts if we talk about learning vocabulary, learning multiplication tables. But even if it's a complex bodily system for med school, the same strategy still work, they're just a little bit more complicated in their application. Is that an accurate summary so far of what we discussed? Yes. Yeah. The other thing is that what my students tell me is that when they incorporate more retrieval practice, they spend less time rereading.

So I say to them, you might need to spend more time, but by the end of the semester they're saying, it's kind of a watch. I don't do as much rereading and rereading notes, rewriting notes, because I'm using more effective strategies like retrieval practice. So in fact, I'm not really spending more time. I'm spending my study time in a smarter way, for effective way. So that's another lesson is that we're not trying to suggest students need to study more or suggest they study with more effective techniques and replace techniques that aren't, are disaffected. Yes. Now you mentioned in there that it is more efficient if a student fails at a test on a certain some sort of information they're trying to digest.

They fail in trying to retrieve that information. But that leads to better long term results. Can you elaborate on that? Because I think that's a counterintuitive idea, but a very important one. They're long term resorts when they have to go back and review the material or review the feedback. I agree with you. A lot of us are counterintuitive. Just the fact that testing does better than re-study. I had a drill that told me once that that was deeply counterintuitive. Because I think the whole study is the way you learn testing just shows what you know. But no, testing is a means of learning better.

But test-potentiated learning is when you take a test and the surprising thing to me is even if you say an answer and you get it wrong, a lot of people thought earlier in psychology that the worst thing you could do was get a wrong answer because that would stamp it in. It would make you think, oh, that's what I'm going to remember is my wrong answer, even if it's connected. But what the findings are is that as long as you correct that answer, you still get just a potentiated learning. That was better to have answered the question wrong than not to have gotten the question wrong answer, even if you get a wrong answer.

So I find that very counterintuitive, but that's what's been shown in the results over and over in experiments. When it seems that maybe the reason a lot of this is counterintuitive is that in school from five years old until however far you go into your schooling until you're 22 or 28, you're taught that the process is learn, study, test as an evaluation. And then it's the end, we move on to the next thing, learn, study, test. I mean, it's like a straight line. Whereas the way that you're talking about testing is how really how most professionals think about testing out in the real world, whether we're talking coders or scientists or marketers, they see it as a feedback loop, right?

We're going to learn something, study it. The testing is now just feedback in this loop. And then we're going to adjust. So I mean, if it's a a marketer, you would never see that marketer put out a campaign and go, Oh, you know what, we got a B minus on that because we got the color of the logo wrong. And it didn't quite resonate with the customer. Let's just move on to the next thing. They would take that as feedback. It wouldn't be it wouldn't be a great it's just that's just information. Oh, we got a few things wrong. We adjust it and we go again until we get it right. In this increasing spiral of efficiency.

And I that's one of the things I really like about the work you two are doing is pushing this idea forward to students that in school, it can work the same way just because you earned a C minus on the test on ratio problems doesn't mean you are bad at ratio problems. It's just feedback that you need to do something different. We've got to adjust somehow and then go again. Thoughts thoughts on that idea? I agree with you entirely. Every time you do you know, you shoot a basketball, that's a little test major. You didn't make it. And then if you think about what can I do better? So it's always practice, practice, practice, and sports.

Nobody would think about learning ice skating from watching ice skaters or reading about ice skating. You got to go do it. It's the same thing in memory. You got to retrieve from memory. That's what you're trying to do is to retrieve that knowledge and use it when you're asked to use it. One thing that surprised us about the book was we got so many emails from sports coaches, sports figures asking how can we do this better. We also got inquiries from musicians. We got a letter from the head trainer of the Navy Seals. And said, we read your book and we think we need help in how to explain Navy Seals and marking our work out and spent two days with them talking about what they were doing now and how they might modify it according to principles in the book.

What were those modifications that you suggested to the Navy Seals? We haven't talked about some of the principles yet, but one of the main ones was a lot of times in learning even in sports, people use what's called block practice. You just do the same thing over and over again. So if your mark has talked to people and coaches in hockey, and you ask them, well, how do people practice? Well, you know, you have 10 slap shots from the right and 10 from the left and 10 from the middle. Well, do you ever do that in the game? No. So why do you do that in practice? Because you never have to do that. And same thing in baseball, you get practicing batting, you get 10 fast balls and 10 curve balls and 10 sliders.

It never will happen in the game. The whole idea is you need to see what the picture is doing and try to anticipate what's coming. Because it doesn't help in practice, you know what's coming. So if the picture tells you, but that never happens in the game. So why do you practice like that? So the idea is to try to make your practice be as close to what you really want the game situation to be. We've called this transfer appropriate practice or transfer appropriate learning that think of what the criteria task is, think of what the ultimate goal is, and try to make your practice correspond to that as well as you can.

And some of it, they can't help. For example, if you want to, the Navy SEALs have to practice jumping offshore from the helicopter and getting to shore pretty far offshore. So, you know, plan to start operation or something. Well, they can only get the helicopters from the air force on certain days. And so they have to practice over and over on those days, they get the helicopters. So that's one example, maybe Mark will have others. Yeah, Mark, are there other thoughts you had on whether applying this work to the Navy SEAL environment or to athletics? Well, I can expand on the examples of Roddy Gaves.

So you gave examples of practicing individual skills like hitting a particular kind of pitch. But we can also think of this as practicing a constellation of skills that go into particular aspects of the game. So on hockey, for example, which I was talking to, I think he ran a pretty difficult practice, which was for a third of the practice, say they have a three hour practice for an hour, an hour straight, the practicing breakouts on the defensive, out of the defensive zone, which is enough if you're hockey fanning on it's really super important. They'll practice the offensive sets for an hour and maybe getting the puck through the neutral zone for an hour.

So that's what we would call mass or block practice, your practice and that over and over and over again. And it's kind of interesting, as much as people, the players or the coaches think that that's effective, because the players are getting more fluid as they proceed through the hour of the breaking out of the defensive zone, but again, that fluency is not necessarily indicative, robust, long term learning. So in the moment, they're very simple, but the prior week later is maybe not going to stick. The alternative, which produces much better learning, is to mix up that massing. So now, you're doing small segments of breakouts, maybe for 10 or 15 minutes, and then you switch to your offensive zone for 10 or 15 minutes, then you switch to getting through the neutral zone for 10 or 15 minutes.

And then you circle back and do another one of those for 10 or 15 minutes. Now, it's writing says, now you're better simulating the game situation. When in a matter of seconds, you've got to make decisions, now that you're out of the neutral zone, how are you going to get through the new, out of the into the offensive step? And you have to do quickly, and you have to you have to be able to retrieve that stuff for long term memory, muscle memory, if you want to call very quickly. Well, mixing up those practice segments allows you to practice that. And if you're not practicing that in a real game, all you're going to do is know after I practice my defensive clear outs for a little while, I'm really smooth at, but what happens if the team brings down the puck very quickly, you have a chance for defensive clear out, and you're in transition, what are you going to do?

Well, if you practice quickly, switching and making that transition, you can do it. If all you've done is mass practice, you're going to be sluggish. That is part of that not recognizing the cues for different stages of the game, because you've been in this mass practice, where it's very repetitive. You already know the section of the game you're in, so you're not practicing recognizing the cues. And I believe you guys call this interleaving, where we have multiple concepts. Well, when we call it mixing it up, mixing it up is another cue round came up with that phrase is to avoid the jargon, the same you're leaving.

So by mixing it up, it helps to recognize the cues of different environments and when certain strategies are going to be applied. And my understanding is this applies to athletics and to schoolwork. Is that accurate? Yes, certain types of schoolwork, there are studies of things that where you have to learn to solve problems, like solving the volume of a solid. And you get formulas for different solids, and typically the right edge to start in school, you know, you can problem the one type to the other to the next. And that's great. You get better and better at them. But if you already know what kind of salary it is, you're given the formula.

So you just plug in some numbers and if you find it, but then on the test, you're given a solid and say, okay, here it is, what's the volume? So you don't know what kind of solid is, you have to retrieve that from the mean, you've not practiced that. And you've got to retrieve unless it's given to you the formula from memory. And so if you haven't practiced that either, because it's always been given to you. And so that's why people will think they really know the stuff because in the practice they've done perfectly at the end, but then on the test, they can flub it because they those two stages are so critical, recognizing what kind of salary it is.

And the formula remembering the formula, you've never practiced those at all. You've been given those, but you're not given those on the test. And so that's why people can think they really know something. The teacher can think she did a great job in teaching it because the students did so well. And then they're all disappointed by the test results. But it's perfectly predictable if you use block practice. And so the idea is you probably need some life in the beginning to get the hang of it. But then you should make things up. So you have to practice those other two stages of recognizing what type of salary it is.

And then retrieving the formula for that solid. Yeah, that strikes me as interesting because I think I mentioned to you guys before off air, I started an SAT preparation company many years ago, 17 years ago, and maybe 20 now. And this was something that we stumbled on to just by trial and error. But we still have strong students who have a hard time with this idea. If they have certain concepts that they are struggling with, and there probably aren't very many of them, right? But they've got a few things they're having a hard time with. We strongly encourage them to study based on an array of problems.

So it's like, okay, there are three problem types that are giving you trouble in the math section. Well, we still want you to take entire math sections so that you will recognize the cues that you're up against one of those problems and can learn to quickly apply the strategies that apply to that particular problem type. But a strong willed kid who usually they're used to being called smart, they're used to being one of the best in the class. They feel like it's a little bit of a waste of their time. And they just want to focus on the thing that's giving them trouble that mass practice. And we have found just through hundreds of students in these situations, the key to actually solving those little problem areas is the varied practice mixing it up, as you guys said.

Yeah, and the research is showing that the work Brody talked about with learning to compute volumes of different solids. That was done in the laboratory on that height of that, what kind of problem. But now they don't work in physics classes. And physics is notorious for students not being able to do where they get problems on exam, not being able to identify what kind of problem this is. The students know the formula as well. But they found out the wrong but the number of problems they don't recognize what kind of problem is work problem for energy, whatever. So in some current studies, I don't know, they're really intriguing.

One group of students is given a full work, busy homework, where the problems are mixed up over the units. And that's not standard. That's interleaved next up. The other group is given the standard homework problems work for the first week. They only get problems from that week for the second week they get problems for the second week. As Brody says, the students don't exactly type of problem. So there's no problem once they learn the formulas. The other students don't know exactly what type of problem is. So as you said, and if they've got to figure out the cues that identify it as a sort of type of problem, and then they're golden.

And the really neat thing was the classes were two groups. One group got there, leave the first half the semester. The other group got masked. The second half of the semester, they switched to the masked group now got in a cleave. The earlier group got masked. And the scores on the first exam were better for the mixed group than the masked group. The second exam, it flopped. Now the masked group got interleaved and they did better than the group had got that relieved. So it wasn't now smart. The students were, it was exactly how they were given the practice. Brody, that produced the benefits of physics.

I found that to be very college study because it's done in the context of real world teaching, real world problems. And it worked. But teachers are going to have a hard time doing this because they're going to see their students struggle more. The students aren't as fluid when they're doing mass practice. And the teacher said, I'm not a very good teacher. My students are getting it. But it turns out they are getting the students are getting what they have to know, which is how to figure out what kind of problems these are. So it sounds like with these strategies, there's some short term pain. It leads to long term gains.

And on the on the blocked practice, we've got the short term gains, but we lose the long term gains. That's exactly right. If you're a student, let's say you're in a physics class or a trigonometry class, and your teacher is not doing any of these things, but you want to put those into effect in order to improve your own learning and your own results. How can that student go about doing that? Well, they have to find a way to get problems presented in a mixed up way. If it's a great question, Nick, but I can give you an answer to a situation that Roddy and I are very close to. And that is that our department chair at the time had a couple daughters in math class, and they come home with their homework, all the same type of problem.

Jim will move, who made up a mix from 88. And then she said, no, you're not going to do that way. And she cut up the homework sheets and she mixed up the problems they gave it to. And her daughters did very well math after that. But the problem kind of a personal issue here, Nick, is that Roddy and I invited their Christmas dinner to their house. And the daughters wouldn't come to say hi to us. And she said, well, I told them that you were the ones who told me and makes up all those problems. They didn't like the daughters didn't like us very much, because it was kind of the short term pain kind of thing.

Yeah, the parents got the best of both worlds with daughters did well and that's going to be a good point. Right. That's a true story. Anyway, if you I think you need a confederate, you need a partner who will know the problems you have to solve over the course of five or six seven weeks and mix them up for you to present. Or maybe you can just randomly select them out of a homework assignment. And you know, not know which is what you just start randomly selecting and then you try to do. But you're exactly right. That's what the students do want to learn more that that's what they need to do. That's precisely the case.

And interesting anecdote, the student that I mentioned before that's in college and that was having a hard time with microbiology, she was trying to do this and found an app. I think it's called Quizlet where you can create your own quizzes and it mixes things up. And she didn't feel like it was really helping and she sent an email about it and said, what do you think's going on here? And what we figured out was because Quizlet, it's a really great idea in that it randomizes the questions. And then if you miss them, they come up again. But it doesn't it likes you to create multiple choice answers.

And it doesn't randomize the answers. And she figured out that what she was doing was memorizing, oh, yeah, that question is answer B rather than what the actual answer was. And so it was not advancing her learning. And I say all this to just say, man, the devil is in the details with how this stuff is employed and executed. That's true. So I need to say something that I can say something. I want to emphasize that that in an interleap practice schedule, students, their confession will be they are struggling and not learning as much. And that's a real barrier that some students and instructors have to overcome.

So a really nice study that one of our colleagues did looked at learning the painting styles of different artists. And so you studied, any from one artist and another in interleap fashion, or you studied all the paintings in the sample from one artist, then the next, then the next, which is the way the suicide was studied. And in study after study, experiment after experiment, students did better with the interleap approach to learning the painting style of different painters than the master approach. But uniformly, students thought they did better. They learned better with the master approach.

It was easier felt more comfortable. That translates into judgment of high learning. So that's the tension that students have to overcome. Their judgment is the mass presentation is better of art. In fact, objectively, they somehow have to have a test and see that you're performing better after interleap. That's the only way students were convinced that interleapism was better. Some others were after interleap versus mass. But it may be, with a physical example, certainly what you say is right to you just memorizing the answer where it was. But it could be that microbiologists didn't put it, if it were interleapism was not appropriately properly.

She still could have felt it wasn't going as well. Until she took an exam form, then that would give her that information, the accurate information. So that's a real tension that we have to recognize students, they do, teachers do as well. And that objective feedback is the key to convincing that student that it's working. Yeah, I think you have objective. That's right. You have to have many, many instructors who want to help students adopt more effective learning strategies, miss that step. The students need objective feedback. Yeah. Okay, everybody. I'm sorry, go ahead. No, I was going to switch to another anecdote if you because I think it brings all these points together that we had heard Martin Peter Brown for a while soon.

I heard that Harvard Medical School was revamping the medical curriculum and we heard that Measure Sec was playing a part of that. So we asked if we should interview them. And we did this terrific interview where the dean of education at the medical school started out by saying it's amazing. We get students who get into Harvard Medical School who have no idea how to say this. And medical school, especially the first two years, you just get tons of information through the act that you've got to memorize. You've got to know like the palm of your hand. And so they decided to rebuild the curriculum around Measure Sec.

He said that when students asked, they have a meeting of the admitted students at the beginning of the summer, they're not going to strike for the fall. And so one of the students asked them what books should we read this summer? And he was expecting a physiology type book or something. And they said, it's a built-life curriculum around this. You all know how to study, read this book over the summer. It was interesting to me, my daughter went to medical school and said the same thing, but it was amazing how some of them just didn't know how to study. All they did was reread, reread, and then they take the test and the stuff just didn't stick.

So I couldn't really retrieve it when they needed to on the test and very frustrated. So I think that goes to show again how effective these things are and how in these most demanding professions, like becoming a neurosurgeon or something, that they will really work. And I went into in a physical one time with my doctor and he had a resident in there just chattering him around like residents do at teaching hospitals. And he said, by the way, do the same road, make it stick. And I said, yep, that's me. And I said, well, I read that book before I worked at medical school. And he said, he went to Duke Medical School.

So I recommended to everybody, every income issue that you do, you're finding more and more people reading your book. So we're very gratified to hear these things. And we're in fact starting another book that we'll talk about how people have used matrix check in many of these ways. We've got lots of good examples of the people have written us about it. We've discovered one way or another. So stay tuned, there'll be another book. We're just starting right now. We've only got two chapters written. Well, great. I'm looking forward to that. Because I think the anecdotes really bring these ideas to life in the make it stick book.

And so a whole book full of those would be great. We discussed a lot about the short term pain that's necessary to reach these long term gains. Can you tell us more about what you mean by desirable difficulties and the importance of struggle in studying? Yes, term desirable difficulties was developed by Robert Elizabeth B. Roch at UCLA. But what the finding is, so retrieval practice is a good example. In fact, so you can either repeatedly study something or repeatedly after you've studied it once or twice repeatedly test. And the finding is if you test people immediately after this, the people who study repeatedly do better.

But if you wait a couple days or a week, it's the reverse. The people who didn't do as well on the first test, they took all this retrieval practice and didn't get feedback. This is people who didn't get feedback on the retrieval practice. They do worse initially, but they don't lose much. The forgetting curve is very slight for those people. Whereas the people who had the mass study, they did really well initially, but then the forgetting curve is very steep. And so the idea is here's something that is a difficulty and it hurts you initially on an immediate test, but it helps you on a later test.

And that's what's meant. That if you want long term retention, use the strategy that might it's like interleaving does the same thing. Hurts you initially, but then it helps you in the long run. And what we're hoping students want to do is no stuff for the long run, not just to get through the test and coming up in an hour or something. Another point that addresses this is you ask well why the students said you inefficiently? And the answer is if you cram block practice, cram right before the test, it will get you through an immediate test. You might get a B, especially if it's a multiple choice or a true false test, where you just have to recognize something as familiar.

But if you've got to retrieve something, call it with efforts of yourself on your own without having these powerful recognition cues, then you're much better off using retrieval practice and things like that. Again, you're practicing which you'll eventually be asked to do. I would like to offer an extension to that. Sometimes you don't think about the kind of test students are going to take. And oftentimes students are trained to expect tests where memorization really will get them far. You would call those tests of recall, but more and more schools are going to test that also require understanding.

So they require application or synthesis integration. And what we found in the professional schools at Washview at PT is that there's a negative relation between the extent of which students are using a rereading strategies and memorization and the performance of these deep questions, application questions, questions that require understanding. They're fine on the question of the factual questions. So as schools start moving more and more and schools are now, we've talked to the Massachusetts General Hospital School of Health Sciences are moving more and more toward tests. The exams are required understanding synthesis and integration application for PT to Washview is the schools move toward that.

Now these strategies that students are so used to using to be successful are increasingly unsuccessful. It's not just that they don't produce retention, but they also don't produce understanding. And that's a real key failing to these strategies. So when the instructors are requiring understanding through a summative cap or pharmacist, these strategies rereading the memorization become even less effective and desirable. And you can see it in the data. You can see the students are using more memorization. They're fine on the fact questions. They fall down on the application. And their retention also falls down.

So if you look at what you're calling in our professional departments here, learning checks and you get the students the same questions four months later to see how much they've retained, students know they're going to get them, but they're told these are not a station. They don't have to worry about study for these. We find that the incidence, the frequency with which students who are memorization and rereading producers, a draw upon from retention on these learning checks relative to students are relied so much in fact. So if they're catching intervals, what he says is really important for these successful strategies.

That's where they really show off. And also the kind of test application, understanding questions, really important to do these difficult learning activities. I can't believe we're doing that. But I think that's a good idea. It sounds like what you're getting at there is the way school is structured. I mean, if we look at the history of the invention of school, it was originally designed to help people come off the farms and be good factory workers. And so a lot of schooling still rests on that bedrock of that sort of high simplification of memorization, obedience, conformity. And what you're talking about is re prioritizing learning over those original objectives, which were great for the Industrial Revolution.

But don't line up at all with our economy that we exist in today where we need a deep understanding of complicated tasks and ideas. Because I mean, AI can do a lot of this repetitive stuff for us. It's only going to get stronger. And we need students who can really learn effectively so they can add value to whatever it is that they are doing along those lines. How would you revamp school, reinvent school, change school, however you want to put it in order to prioritize learning more in the process of going to school? That's a huge challenge. Yeah. And lots of people working on it. You've got in public schools, you've got so many students have such different abilities in every room.

I think that makes it even more challenging. But I think these strategies will work as far as we can tell for everyone. That they would make it better now to change the whole school system. That's really difficult. There's some people that we worked for maybe five years with our middle school, near us in St. Louis in Columbia, Illinois, across the Mississippi River. And we had three four teachers who were very enthusiastic and they led us to rural experiments in their classrooms. We were wondering if the techniques once teachers saw they were effective, which they did, they would talk to their colleagues and maybe this would spread to the school.

And it didn't seem to happen that way. And so even with the success teachers kept doing the same things they'd been doing. And maybe they built in a little more testing around the edges or something, but it wasn't like they totally revamped their classes to be based on these principles. So I think because they are harder, sometimes harder for the teachers to, especially if you grade quizzes or something, yeah, it's just too big a change right off the bat. So I think I think things aren't getting better, but it's going to be a slow process. Well, I think if we could do it, we need to implement more just over longer retention intervals.

So right now students are going to use the cramming, the K to test. And then many teachers don't even give us cumulative exams anymore. Broady has been very vocal about saying you don't give cumulative exams, you're really already learning because he's got studies that show if a student thinks that's all the tests we're going to take. They'll remember as well as if they believe they're going to take a cumulative test. They should run. I don't know if you got that detail right, but only you're going to take a cumulative test somehow keeps that information active. But more the cumulative tests, I think you're going to test four to five months later after that, if we can do it after the term score, because students studying techniques and the teaching techniques are directed at this immediate retention.

And four months later, the students forget it. So the Dean of the Business School was used to, can you come into office because our staff force everybody's getting days to the staff force, but by the time they get into the courses where they have to apply the statistics, we feel we have to teach it again. So how can we fix up so that that knowledge that we're trying to have the students acquire is retained into the next year. And those techniques are available, but they're not going to be adopted until we start to measure students retention directly. And that becomes a criterion that we shoot for achieving.

So that I don't care if the student gets in there, it's that I do care right away. I want to know how much they know four months later, how much they retain, how much they can use. If we start to major for that, I guarantee us techniques that teachers use as students use are going to be shaped up to help produce better performance there. And we know what those techniques are. Right now, those techniques are not reinforced because they're not necessarily needed to excel on major media tests. So we're not even reinforcing these techniques in our school structure right now. Another issue is they're not taught in education schools.

I mean, we've been doing this work for 20, 25 years and many other people besides us of course. And but if you go into most education schools, they might have a course on educational psychology, but it might not be that our work is represented in their very heavily at all. And so until we start exposing teachers to it in education schools, how will they go about it? So not everybody's going to happen by making it sick, unfortunately. Nick, I might follow up by saying one of the stories is not a story. It's really a development of a new program in teaching math algebra. We're going to talk about our book.

Instructor used lots of energy, very little massive lots of energy leading. And then this instructor gave a test of students when they came back to the summer. And he said typically we all expect there to be the summer. The summer, what do we call it, Roddy? No, summer no. Yeah, summer kept. Yeah. And sure enough, in parallel forces that didn't use their leaving, they dropped in what students did. And the interleave course, very little forgiving over the summer. So those are the kinds of things I think that we're going to push adoption of interleave and we'll be care about what students remember when they come back at the end of the summer, not what they do in May when they left school for the semester.

Anyway, we can our next book, we're going to go into that story. And many others like it that really show the advantages of these techniques and real educational situation with real content and real students. We didn't add much of that. The first book wasn't available. Now we're starting to see these examples as people are adopting the techniques. And they're very inspirational, they're super inspirational. Yeah, I'm getting the sense that medical schools and law schools are more receptive to these ideas because they have a vested interest in producing more effective doctors and lawyers. And it's a harder challenge to transform high school and primary schools because there is so much of it is about grading kids and then ranking them and then getting them onto various tracks.

And it's really there where the biggest shift could happen if perhaps if these law schools and medical schools can show how well these techniques work when the goal becomes produce a better doctor rather than produce more medical students getting A's. The goal is produce a better doctor. Then maybe we could have a trickle down effect to the high schools to the elementary schools or say, what exactly are we doing this for is ranking these kids really the purpose of school or is it producing better learners and long term retention of this information? Point I mean, in the example I just gave about the high school algebra class, the instructor is extremely excited because it's exactly what he says back that the interleaving produced better learners.

So what we're going to point out and the chapter of written is that his learners, he said became adept at making connections across different aspects of math. He said, I've never seen students make these connections. Not only were they making their connections, they were excited to push to make new connections. And he said they became like active mathematicians. And when we talk to him, the excitement is absolutely palpable in the way he sees his students develop into these learners who are excited about learning and reaching for connections that he'd never saw students make before in this mass construction.

We found the same and we didn't. And they structured out in Mississippi and the AP econ. The usual pass rate was 20% of AP exam. And by using lots of retrieval practice, by allowing students to generate wrong answers, alerting but failure, he increased the pass rate to 60%. And again, they cite, and he said he had never had a better day in school. This one that students took the practice exam and saw how well they were doing it. He said they totally bought in. This is one that just pointed out the objective of this. They bought in and said, the whole passion of change. Everybody couldn't wait to do retrieval practice and answer questions and write on the whiteboard.

But the thing he emphasized, we've said before, is that he got to set up a culture where students understand wrong answers, they're okay. We are allowed to make our answer. You're allowed to make yeses. You're allowed to struggle a little bit to get the concept. And in doing that, you start to learn. And this instructor also just became so enthusiastic about teaching and the way the students were learning and how angry they were to learn. So it's what you say, these techniques aren't just creating students or getting a work creating lifelong art, who are really understanding the material and making connections and excited to be in the classroom.

And that's what I mean. Partly what we're writing in the Saka book is that today those actual examples, real students and challenging educational situations and real instructors who are just trying to create that kind of atmosphere and classroom. And it's amazing that work work. What you see, it's just amazing. We interact with a lot of teachers through the business. And I find 99 out of 100 teachers do it because they want kids to learn. They want that AP econ teacher wants kids to be excited about economics. That's why he's doing what he's doing. And I bet when push comes to shove, if the AP exam didn't necessarily matter towards that kid getting into some famous college down the line, he would say, yeah, and whether they pass the AP class or exam or not, isn't really the point.

It's that they got excited about economics. They really learned the material in a way that they can apply out in the real world beyond the classroom. Because who wants to be a teacher where people study really hard, get a grade and then forget everything that they did in that class. I mean, then you had no real discernible impact on that student's life. So I'm sure teachers can get behind this. But it sounds like we need some structural change in order to give teachers the tools to make this stuff happen. But it's scary. It's frightening. It is. I'm gonna imagine, I can't even imagine how these instructors said to themselves, I'm gonna try to interleave these topics in algebra.

Each one day fractions, he moves on to another topic the next day. Imagine the pushback from the students, imagine that parents in the parents, is it gonna work? If it doesn't work the whole semester, it looks like, you know, it's been a complete waste of time. So I think I think a lot of teachers can do this in their classrooms, they have the latitude, but it just is so tough to believe that these difficulties are going to work that it's going to create a positive outcome. And it's tough to change what's comfortable to do as a student and as a teacher. So I think the interesting thing is this particular instructor found that once people saw how well the students were doing, then others in the school came and said, how can I do that?

And now they do have a cultural change in that school, but it started with one classroom change. I'm not, it's a good point. You do need to make some structural changes, but does it come from the bottom up or top down? I think sometimes it asks out from the bottom up with individual teachers, demonstrating it really works. And here's how I did it. And yes, it's frightening to try it, but half a, but if an administrator says to all teachers, you're going to mix everything up, I would be afraid that, you know, lots and lots of pushback and less individual instructors, pride in and a twerking and they can, they can be kind of a mentor to other teachers.

So our, our view is people need to make, we make a stick, the follow up to make it stick, bright these things to their classrooms. There are going to be lots of concrete examples for how to do it. And you kind of create transformations from these teachers and these students showing how much it works. And then I think it might be this, that in those papers take notice and they say, I want, how do I get this? Because it's so successful. Well, Mark, now you're really getting me excited about this new book, by sharing all those examples, it's the proof of concept, right? And a whole bunch of them. And now, just like you said, then you get teachers on board with it, you get students on board with it.

I love the idea of the bottom up revolution, because especially public schools, I mean, they exist based on the taxpayers. And if enough students and parents want to see schools shift to prioritizing learning and see new techniques, however scary they might be that are effective, they can demand that change. And administrators, just like you said, will take notice of that. And then man, then you've got a cascading effect where the balls rolling downhill is gathering steam, and we could have a real revolution in education and how students are taught to learn. And that's exciting to me. That would be exciting.

I think things are getting better, but very slow pace. And some systems are moving and others haven't done anything. Right. But some teachers within those are learning about from teachers conferences or somewhere else. Do you have any other stories or ideas in mind that that are a good proof of concept of employing some of these ideas just to help illustrate how it works? Well, one example, Peter Brown, Florida International University of Law School, he was invited to go down and get the talk because our co-author, it turns out they were already trying things there that they have a new, have a new Dean, this is some years ago now, new Dean of Law School.

And he looked around and his law school was kind of the bottom rank one in Law School. And things like shoots passing the bar and shoots being admitted. They started trying, changing the curriculum, more quizzing. I mean, many law schools, every course is one test at the very end of the course, that's it. So they certainly don't believe in tests. But so they started implementing and the grades went up every year for the students. And then they started also using this for practice on the bar exam, where his school had the lowest pass rate of the Florida bar exam. And then suddenly after three years of this for the students for doing all these new things, even though they hadn't changed their admission criteria, suddenly the students were passing the bar at the highest rate of the bar exam in Florida.

It showed that you can take, you know, it's just a kind of a miracle that students who had been passing the bar at the lowest rate of all the, I forget how many law schools in Florida, now suddenly they were at the top rate, fun using all these different techniques now in learning rather than the standard ones. And this was down to self testing and mixing it up. And doing it in class, not just doing it, or having more testing in the class. After all, the bar exams are test. If you want people to pass the bar exam, you want to test them testing themselves and you testing them. And if you think about every time a lawyer works on a chase, that's a test.

Just like every time a doctor sees a patient, that's a test. Right. What's wrong with this person, they've got these, you know, mixed symptoms, it could be this, could be that, although I figured out this patient, you think of it that way. Yeah. Having people have the ability to achieve more information from memory is what you need to be able to do these kinds of things. And yeah, AI is going to help, I mean, it's changed the law quite a bit already. They used to pay, you know, beginning lawyers to sit through books and look for similar cases. Well, AI can handle that now. So I think these techniques will, you know, we know of these cases where they've collected data, we've heard from lots of other people where they think things really help, because they said I used to see grades of this story in my class, I changed strategies, and now they've risen to hear.

So that's a nice way, it's not exactly a controlled design, but it's still good. And the one place in our experiments we did in middle school didn't seem to have much effect, or as much effect as we expected, was in math, because in math, they do give lots of tests. I mean, every time you do a homework set, I mean, again, the problem in math is block practice and so mix up practice, but they do have lots of practice on the problems. I remember doing lots of problems, more of those in math major, so probably more vivid memories than I did. Let's turn to test anxiety for a moment, because I know you mentioned that in the book, and a lot of students express, they think they're bad testers, quote unquote, because they're thinking of testing as an evaluation, and that they suffer from test anxiety.

Well, test anxiety is a real phenomenon, and one criticism we got of using retrieval practice using more testing, wouldn't this increase testing anxiety? And it's a reasonable point, but when, in second, our middle school, our quizzes we gave to students were zero states, they were just doing this in class to learn, and they knew that. So we did surveys at the end, Pooja Agarwal, who was a colleague of ours who did, was a lead in this study, that we had questions after they took these classes with lots of retrieval practice, and others of the other classes you're taking. In this class, you have more anxiety about tests, the same amount of anxiety, or less anxiety, and 80% of the students said they either had less anxiety, or the same anxiety.

And so it seems like if you're taking quizzes and tests all the time, you're anxiety, and also, especially in many classes, you've got this same in Japan. There's one test that determines your entire future. You take a two-hour test, and it determines whether you go to college, what life of college you go to, all those kinds of things. And Japanese psychology told us all the students who just paralyzed, who's, you know, this is going to determine the rest of their lives. I think that's a bad system, obviously, but one way to help reduce the anxiety is to get students quite used to having tests all the time.

Well, I would think that with repeated quizzing and testing and self-testing, if you can shift the definition of what a test is for a student into that it is feedback rather than a evaluation, a judgment, a permanent verdict on them, then you reduce anxiety. And I've seen this firsthand time and time again with students preparing for the SAT, which is a high-stakes test. But the nice thing about the SAT versus say the test that they use in the Japanese model is that you can take it numerous times. And so it's easy for us to say, okay, so that starting score is, it's in the middle of the pack, it's nowhere near what you want it to be.

That's just a piece of feedback. That is a starting point. We've got to make adjustments to get you where you want to go. And man, when they take that framework into school, suddenly anxiety goes down and performance goes up. They simultaneously can see the, still see the stakes of any given test in any given class, but the edge is taken off of it because it doesn't feel like a personal judgment anymore. It's just feedback to where you're currently at. It's a snapshot. So some tests like the SAT is still pretty high-stage test. And college, where Mark and I are, there are lots of pre-med students, and they worry about the MCAT test.

I forget what it's saying for the medical exam and lawyers worry about the law exam. But the one nice thing is, apparently, unlike when I took the SAT, apparently, you can get lots of old exams now. And what they do, your practice is practice taking all those exams. And it's probably a great thing to do, giving them self-feedback when they don't know it. It's probably what you're doing when you're testing company. That is one of the techniques for sure. Yes, yes, because you become familiar with the format, you have an idea of what to expect. And just like you said, we want to practice like we're going to play, whether it's hockey or the SAT.

Let's set up that same environment. I think it's great. The students asked me, will this help me? I say, yes. I don't know of any strong evidence, but yes, I feel absolutely sure taking those old tests are the best thing you can do. Yeah, it still strikes me as interesting that we see students that will go through the prep program for the SAT and doing the things that you're talking about with. Mixing it up and then a lot of self testing throughout this process and trying to redefine that idea of testing. And suddenly their grades will improve without any tutoring in any specific classes. And what they report anecdotally back to us is they're spending less time studying because they have a better sense of how to figure out what they need to study and what they don't.

I mean, there was a young lady in Massachusetts who on a post program interview, she said, yeah, I was getting a B in my history class and I was studying for days and days and days before every test. And now I find myself studying a couple of hours and I'm getting A's and I kind of feel guilty. I feel like I'm doing something wrong. And I just yeah, it was all right there in that one that one quote. I mean, this really can be transformative when you put this stuff into practice. One other thing I wanted to ask you guys about is learning styles because that's been around in education for a long time.

There are mixed schools of thought on it and I know you mentioned it in the book. And so I would just like to hear your thoughts on that. You can't deny that students will happily take the surveys and self describe themselves as an auditory learner or a visual or any other other 80 or times of learning styles that are fine. Students are quite willing to do that. So it is the case that students believe that maybe they have a learning style that maybe they do have a preferred way of taking an information. But there's no good experiment that I know of that shows that matching instruction to the learning style produces better learning.

And that's the that's the nature of the learning styles orientation is that well, why would you hear about a study style because if you can match instruction to that learning style, you're going to improve learning. No evidence at all. That's the case. In fact, we just completed a study that we published last year, started that in some cases, missed matching instruction to the learning style actually produces better learning. You could all that may be desirable difficulty. The students think a little more difficult and they learn more. But that was just one study at any rate. There's no evidence for it.

That matching instruction to learning style is going to improve learning. However, teachers will believe it. I mean, I've had teachers. I've seen their comments saying yes, I saw the research that the data was published. But I've been in the classroom 20 years and believe me, I know that the matching instruction works. But so, but the teacher has no objective evidence. One of the problems is that good instructional techniques often incorporate multimedia instruction, diagram, visual stuff, verbal stuff, people read or they listen, even the four in the studies things where they're doing experiments, manual experiments, and even virtual reality experiments.

All of those things are going to improve learning. Period. It doesn't matter with the learning styles, but teachers say, oh, this is really good because it matches each individual student's study style. So the problem is that there are some teachers who still are very adamant that it is important to match learning styles. Do instruction and they don't have any objective evidence and parents have mod into that. So, Ronnie and I have a talk at one of the local high schools and we opened it up for questions. And so the teacher said, could you help us with this? Parents continuously call in and say, you're not meeting my students learning stuff.

And as a consequence, my students not doing, but my son, my daughter's not doing it very well. Broading my bread, I think for the first time saw first and how much teachers have to contend with these demands to create instructional environments that the literature shows are worth the time without worth the effort and then out worth the expense. So, despite the evidence to the contrary that you're not worried about learning styles, Nick, there's still a lot of schools and parents and administrators. You think that if I have to pay attention to students learning style, they can't be made good effective instructional techniques in general.

There are lots of eligible practices. So, it's a tough thing. It's a tough thing because again, people's subjective intuitions lead them to effect. They need to match those students learning stuff. Another thing, my example points out that often it's a useful scapegoat for parents and students to say, well, of course, I'm struggling a class because you're not meeting my learning style. So, it's an insidious kind of thing. There was a study done at Indiana University in which they can query the students learning styles and then they looked at how students were studying. And they evaluated the study techniques according to a different orientation, visual, verbal, that kind of thing.

The students preferred study techniques didn't even match their own learning style. So, students themselves aren't even matching, even though they might demand that they can't have to match that in the classroom. It's just an unfortunate situation where it's not kind of a method. It isn't a method and it's sucking up a lot of resources in that if you want it, it may not be. And schools are spending literally buildings with dollars across the country. It's just a shame. You can take it to summer institutes to get certified and being able to identify kids learning style so they can come back and meet those learning style needs.

And as I said before, it's money could be much better stuff. Also, if you're a teacher, you've got students with seven different learning styles. How are you students? That's right. Right. It's an obvious contradiction. And I think what this gets at is the importance of experimentation, of testing and seeing what the objective feedback from different experiments tells us works and doesn't work. I mean, another one is that in math, it is known, period, just like learning styles. It's just known you have to build one concept on top of the next and you can't deviate from that structure or students will get lost.

And yet you've got that great story about the math teacher who is interleaving with different math concepts. And suddenly, you've got kids who are excited about math. And that's what we're really going for anyways. And so it just seems that it's really important to keep doing experiments to see what works, use that objective data to then go back and share best practices. You know, I'll emphasize that's what our book is based on. It's based on science of Florida, the scientific evidence. And we're obviously we advocate that strongly. Well, I am really excited that all of this is catching on. I mean, the book has been popular since its inception, but it seems like it's really picking up some steam lately.

I don't know if you guys saw, but Andrew Huberman did a piece on effective learning, a solo episode. And it had more than a million views in a couple of days. And I would say 80 to 90% of what he talked about was concepts that are covered in making it stick. And I think there's some real momentum right now where it's coming from the bottom up. People know they know something's not quite right. And they want strategies to make learning more effective. And so thank you for all the work you have done that you are doing the new book that's going to be coming out when that gets finished. I think this is really good work that has a huge positive impact on students, teachers, and just people everywhere.

Thanks so much. If people want to learn more about you, Mark and Roddy in the work you're doing, obviously they should check out Make It Stick, the original book. Are there any other places they can go to get more information? Oh, there's lots of good books on learning. And for teachers, Pooja Agilwal, who we mentioned a few moments ago, she and Patrice Bain, who is a teacher, wrote a book called Powerful Teaching. And based on all these techniques and so far from the future point of view, how to go in. Scott Lassie, a good practical advice in it. Daniel Willingham has several books who really quite good.

One was why children don't like school. That one sold very well. Do that. Carrie has a book on how we learn, which is like good. Obviously there's all kinds of papers on our website. Any other closing thoughts or anything that we didn't touch upon? No, we didn't touch on a lot. But no, I think we chapped almost all the major points that we've talked about in the book. Well, great. Great. And maybe we can follow up again when the new book is being released. I learned from my new friend Seth Godin, who has, he's released, I think now 20 bestselling books, all nonfiction that we've had a book called The Timing Podcast release when the books are released can be really effective to just help spread awareness of that new book.

That's his name, Seth. Seth Godin. And you know, he actually wrote a book that might be of interest to you guys as well. I believe it has now been downloaded four million times and it's called Stop Stealing Dreams. And it's just, it's a manifesto on how school needs to be reinvented around learning rather than what we currently have, which basically came out of the industrial revolution. It's a very interesting take on school. Okay, great. Thanks. Heck, I may email over a story about how we're using all this stuff at Tess Prep Guru. That would be great. We'd like to include that in your book, actually.

I put it in the second week. We could possibly use a job product. I have so many stories that I wouldn't even hardly know where the best ones to chew. What type of story are you looking for? If you have ones with data, those are great. With data, all used to do. Now, right. That's right. We don't have that for every story or book, but we're trying to focus on ones that are like just, oh, I think I'm doing better. Well, what's nice about the kids going through the SAT program is that we always have a data point at the beginning and a data point at the end. And just like you guys said, a lot of the time they don't, they feel like it's not working until they see that objective feedback and then they go, oh, and then they really buy in and suddenly that then they're doing all the stuff on their own and then the results just come.

Well, maybe it's time to interview you. Okay. Okay, sure. Yeah. The chapter and then we're going to transmit them. We'd like to tell the story in the person's words. So we would use a lot of your words and of course, thank you to the chapter before it. Yeah, that'd be great. Oh, yeah, I would, I would absolutely be up for that. I mean, I'm, as you can guess, I'm taking this work really seriously and putting it into practice and I just love what you guys are doing. So any, any way I can help. I'm absolutely game for it. Listen, name of your company. Test, test prep gurus. Test prep gurus. To wait.

Okay, everybody until next time, ask questions. Don't accept the status quo and be curious.