Beyond Solitaire Podcast 191: Michelle Strayer on Wargame Analysis
Transcript
It's ad break time. I'm proud to announce that the Beyond Solitire podcast is sponsored by Central Michigan University Center for Learning through Games and Simulations. They just successfully funded their most recent game, Greg Luring Albbright's Keep the Faith, which is a light RPG about the growth of a religion. If you miss the campaign, do keep an eye out for a chance to late pledge in the coming weeks. Also, if you'd like to support Beyond, please head to patreon.com/beyondsolitire to help me keep the lights on. For now, though, enjoy the show. Hey gamers, this is Liz Davidson from Beyond Solitire. And this week on the pod, I'm very excited to have a special guest. This is Michelle Strayer and she is an analyst at CNA. In other words, she's a professional wargamer. How you do today, Michelle? I'm doing great. How are you, Liz? I'm doing awesome. I'm super happy to have you on. I always like to talk to the pro war gamer set. Um so for people who haven't heard previous people from CNA, uh what is it as an organization and what is your role there? Sure. Um CNA stands for the center for naval analyses. We have two branches. One is our FFRDC branch, our federally funded research and development center. We do a lot of contracts for uh DoD in general, but also some affiliated agencies. Um and that was founded in World War II. uh we originally founded to hunt for German Ubot and we have another arm which is our uh policy research arm uh and they do contracts for everyone from uh FEMA to uh the FAA things like that. So we have two separate arms. We co-mingle a lot but we do a lot of research. Um and
amongst our different divisions we do have a wargaming team under our operations research division. So, one of my favorite things about talking to pro war gamers is that some of y'all never gamed in the past and then just kind of got into war gaming by accident and then others of y'all are hardcore gamers. So, are you somebody who gamed all your life and then became a professional war gamer or did you get into gaming because of the war side of it? Sure. Um, so let me start out with a disclaimer that none of the opinions expressed here reflect CNA as an organization or our clients within the DoD. They're all my own. Um, I have been a lifelong hobby gamer, which is how I got into this. Uh, funnily enough, I have been a statistician for a while now. And I got into wargaming through a chance conversation with one of our head designers about gaming and game design and some of the independent things that I've created and run. And we sort of fell into conversations about what wargaming was. And I didn't realize this field existed. And now I've been doing it for about two years. Oh, that's awesome. So, what kind of a gamer are you outside of work? Uh, I do a lot of tabletop gaming. Um, I do some immersive theater stuff and I've consulted for some immersive theater companies around DC, but for the most part, I do tabletop gaming. Excellent.
So, do we mean tabletop as in role playing? Yes, I mean everything. I do board games, too, but a lot of TTRPGs. Yeah. Fantastic. And then what would you say that your current favorite is? How are we really or favorite board game? How about one of each? Okay. Uh favorite board game right now I'd have to give to Betrayal at the House in the Hill. Um I just got the Legacy Edition and I'm really really excited to crack it open. Um it's got great theming, really simple, straightforward mechanics. I'm a big fan. Um and the idea of getting to add to and create a board is really unique and I love legacy games for that reason. Uh, favorite TTRPG. Gosh. Uh, so I just played a couple rounds of Big Damn Heroes, which is the Firefly TTRPG that came out years ago. And I find it's it's more like D&D 35 if I had to compare it to anything.
It's very straightforward, sometimes to a fault, but it's got very neat storylines. Uh, and it the mechanics kind of fit in in a way that doesn't hinder the story and kind of builds on it instead. And I always find that unique in a TTRPG. Nice. All right. So, hobby credentials. Love it. No kidding. Um, so when you I guess this leads to, have you also started to play hobby war games or is your gaming life different from your professional life in that regard? Sure. Uh, before I answer that, can I turn the question around and ask you what's your favorite TTRPG? Um, you know, I don't play a lot of role playing games, so I have like really boring like background. I've played D and D a couple times. What I want to play at some point is Vampire the Masquerade because I just think it looks fun. Um, uh, and I do like solo RPGs. So, I of course like things like Thousand-Year Vampire. Um, I just got a copy. I'm so excited to start it. You're going to love it. It's so interesting.
So, I really like those little journal games. Um, and then I think it's like is it Quill? The one where you're like trying to fit certain words into a letter. Um, I get a big kick out of stuff like that just because, you know, if you are prone to flights of fancy, even if you're by yourself, like those things are are highly entertaining. Definitely. Yeah. But I also I really love that you're like a hobby gamer and not necessarily somebody who is like, "Yes, I love to play six-hour Battle of the Bulge games. Now I'm going to apply it at work." So, have you also acquired a taste for those games? Yes and no. Um, can I play a long form game? Absolutely. Do I have an appreciation for mechanics? Yes. But I think by trade I'm a research statistician and I think the way that I like to separate between my my work life and my for myself life is I will absolutely dissect uh large gaming systems for work but in my free time I prefer story based games. Nice. Okay. So this leads to the next question which is what is your role on the wargaming team?
So you've mentioned that you're a statistician. Are you also doing design and what kinds of roles can you play on a team like that? Sure. So, um, as I mentioned, I'm a statistician, so I do design games. Um, the way that our internal structure kind of works out, I for the most part have been a deputy designer or assisting with design. So, I haven't designed any games by myself or not without effective supervision. Um, just to make sure that everything's kind of going according to plan and we're meeting objectives. But, um, I do design. I also do some data collection and analysis. So, there are games that I've worked on where we've had pretty rigid data collection systems where the participants are filling out a sheet or completing a survey or checking boxes and part of my job is inputting all that data, figuring out how to best analyze it, running statistical tests and things like that.
Okay. So, this is really interesting to me. I've always wondered about these things. So, I'm super happy to have you on. So, how do you decide what data you're going to collect for a war game? And because so much of what happens is kind of interaction between people maybe that's not like I feel like a learning experience, right? There's like kind of intangible things and tangible things. So, what tangible things can you collect about a war game that are meaningful later? And then what do you do about the stuff that's just kind of human interaction and maybe hard to define? That's a really good question. Um, so let me preface by saying a lot of my background is study design. Um, so a big part of the role that I personally play on the team is helping to define who's involved in a game and the way that the game is designed to get the best results from the participants.
Um, and part of that is determining when we can collect solid quantitative data and when we need to collect qualitative data and what kind of data and what it should look like. So what kind of data we collect and this is a disappointing answer I'm sure uh is completely contingent on what the game wants to achieve. So wargaming is most effective when it studies human decision-m and there's variations thereof but that's effectively the best niche for gaming in general. So when we work on games that focus on really specific objectives, I want to know how to move asset A from point one to point 2 that we can usually collect data on, right? It's a what we call in stat, it's a defined decision space. Um so there's only so many factors at play within this very contained parameter set. Um maybe you need to know geography or you need to know political situation or you need to know what vehicles you have, things like that. Um and those things can have data collected on them. We can present players with a series of options and they can choose from among them and we can collect data based on the decisions that they make. Um, if what we're looking for is something much more nebulous, we want to know how these groups of people feel about a specific policy decision or how they think it'll roll out in the future. That's a much more open-ended question and we can't really collect quantitative data based on that. So, it deeply depends upon the kind of game and
depending upon who we have and how we design the game, we can pinpoint exactly what data we'd like to collect and how we can collect it based on who we have there. if that makes sense. Did that answer your question? Yeah, it just makes me have more questions though. Sorry. Okay, so when it comes to something like let's start with quantitative data. How do you make sure like what sorts of things do you collect? Is it only things about the people who are playing and what they do? or are you doing like research where you're also trying to get at some sort of numbers about like what would actually happen in the real world with weaponry or with geography or with any of those things? Good question. So, I can't speak for other organizations.
I've only done wargaming at CNA. Um, we see wargaming as part of a a cycle that also includes exercises and research. So ideally, a war game is run when there's a problem that focuses on human decision-m and decisions that people have to make. We identify a couple lines that need to be researched. We recommend those for research or we pursue those avenues of research and then we in our research conclusions identify areas that should be exercised. Exercises are run. They identify areas for research or wargaming and the cycle continues. Um, so we sort of have envision wargaming as a step in a larger process, but we also engage research ourselves sort of every step of the way. So we do a lot of research beforehand on whatever the topic is. If I'm running a game on country A, then I will do a ton of research on country A. Um, but additionally, while I'm doing research to create the game, I'm also consulting subject matter experts on what that research should look like and also what the current situation is and what should be included. And then throughout the running and adjudication of the game itself, we still engage those experts or new experts or create panels of experts.
Um and then at the end we typically hand off our data and our compiled uh game findings to analysts who specialize in that subject area and they continue forward with the research uh project or analysis. Okay. So basically there's research every step of the way but y'all are doing research before you do the game. You're doing the game in response to previous findings. you're collecting data about the game and then someone like helps interpret that data for the client is that basically yes. So when I again as someone who does not have a classical academic background in wargaming um when I visualize wargaming I like to think of it as a method of analysis because effectively what you're doing is almost like designing a study and you're using a lot of study principles. Again statistitians this is how I frame it.
Um, so you are taking a population of people who have the knowledge that you're looking for or the expertise that you're looking for and you are putting them through a series of scenarios or through a mechanical system designed to pinpoint your specific objectives or target the specific things that you want to get at. And then you're collecting all the data you possibly can um in whatever format you deem best and you're handing it off to the analysts who can best tell you what it means in the scope of the problem that you're trying to solve. Interesting. Okay. So, and this is something I ask every professional war game because the answer is always different, but it's always I just always wonder how do you do it? um you know because no matter how much research you put into a war game, you're still creating something that's theoretical and you are running it in an environment that's not a real environment. How do you make sure that the data that you're collecting is meaningful? Um or I mean I I don't even know is accurate even the right word and accurate is the right word. Um, so also a good question.
Um, it depends upon the game. So sometimes sometimes we're asking people questions about things that they currently deal with. Um, and that's a lot more straightforward because we're asking them questions about a system that they currently interact with and it is a realworld real life scenario or one that very directly mirrors the real world. Um, sometimes we have to extrapolate. Um, and when we do that, it's kind of built in with our levels of assumptions. The same way that when uh a study is run, um, it's run on only the participants who volunteer for it. And so there's a level of assumptions in there that the volunteers reflect the world as a whole, um, or reflect the population that you're trying to study. Um, in the same way that when we do medical studies on people in a lab, we are very strictly controlling the conditions. We have to, but that also means that they don't actually necessarily mirror the conditions out there in the real world.
And we do everything that we can to make sure that we're mirroring those conditions to the best of our ability, but there's always going to be some gap for error. In statistics, we always, sorry, call it the error term. So, there's always going to be an error term. There will always be something that you have to control for. The question is just how much you handle it um beforehand, how much you try to mitigate it and then how much you address it at the end. Do you have analysts who are capable of sitting down and looking at every single way that you decided to design the game and pulling out here's the pieces that might not align with reality and here's how it might play out differently. Or sometimes you ask the players here's the pieces that didn't align with reality. Can you help us understand how it might play out differently? So, it depends. Um, a lot of it has to do with incorporating gaps or error into the analysis afterwards, but because games are all about feeding analysis for us, that's an inherent part of our process.
So, it it flows pretty naturally. Interesting. So, we talked a bit about quantitative data collection. What about qualitative? Like how I mean as an interviewer I know how to ask questions that get a conversation out of somebody but how do you ask the right questions to learn what you need to know when what you need to know isn't a set of facts necessarily? Great question. Um would you mind rephrasing? No. Okay. So when we talk about qualitative data, what kinds of data can you extract from these games? How do you do it? And I guess how do you choose to process it given that people are not necessarily reliable narrators?
Um, great question. So, there's an entire field of study statistics and psychology devoted to this. Uh, which I have not personally studied in depth. However, um, part of it from my stance, and this is a big part of what I contribute to the team at CNA, is how do we minimize how do we minimize bias and how do we minimize the gaps that we leave by not being able to have every single person possible at our war games? Um, so part of how do we get at the root of whatever question we're trying to address is how do you be a good war game designer? A lot of it is how do you create mechanics that are targeted but intuitive enough? How do you simulate an environment that is precise but not too tight? Um, and how do you put your players in an environment where they can answer with the degree of cander but also understand the role that they're able to play within the rank structure that the game imposes on them. So, some of it's a balancing act that just has to do with as you slowly go through the process of design, you learn where you have to draw those barriers. And there's a ton of game design theory that is out to guide you and there's a lot of courses that run on on this. Um I personally took a
course at the Naval War College that was extremely helpful and guiding a lot of my personal design practices. Um, but a step kind of further than that, when we look at those questions in studies in general, there's specific ways that psych tells us that we can frame questions, but also ways that we can approach questions that will be more or less open-ended or more or less structured or help people um help people think in a like a specific direction as opposed to answer in a much more open-ended fashion. Um, I had to read a book on it in college, but effectively there's a lot of kind of structuring that we can do, and a lot of it has to do with game design and mechanics, but there's lots of other fun elements that we can add in.
Interesting. How do you collect more open-ended data? Do you record somebody talking? Do you have somebody who takes notes? And then how do you coate that? Like, so we have there's different shops do it different ways. Um, I've seen shops, uh, wargaming shops that have, um, there's a a term for them, but basically, uh, dictation software, uh, that will just record. We don't do that. We have, uh, data collectors. So, we have folks who are trained to note and we'll have them sitting with the players and they'll jot down important points, discussion points, um, things that come up repeatedly or sometimes if someone makes a really good point, they'll write down word for word. Um I have watched them work many times and it is not a skill that I personally possess. I'm very impressed by them. Um but we do train them to understand what to listen for. We run through our data collection plans with them beforehand so they know what to target and then we have trained facilitators who also guide the conversation so that when it gets off topic and it always gets off topic um they can rein it in a little bit and bring it back to where we want to focus.
So to some extent we do collect um written notes and I know that shops that do more dictation style have um NLP or uh natural language processing software that they run so they can look for uh patterns and trends in words and verbiage. They can look for um different kinds of uh trend and pattern analysis using machine learning techniques. Um, I'm trained in a lot of that, but we don't use it at CNA. Generally speaking, for the most part, we take all of our kind of collated notes. Um, and any data that we've additionally collected. So, if the players uh filled out data sheets or uh plays or if they uh turned in formal actions, we take all of that and we turn it over to the analysts. And generally speaking, what they'll do is they'll deconlict, form a single coherent narrative of what different teams did, how it impacted uh elements throughout the game. If the game is run more than once, they'll look at those separately. Uh and repetition is a big part of the process. And I've seen some shops really dive into the idea of of repetition. Um so we'll take all of that data and the analysts will comb through it to determine exactly what needs to be followed up on and where the research needs to continue. and they'll isolate research design from there. I hope that long-winded answer actually answered your question. No, that's really interesting. So, it sounds like there's this is partially, you know, math. It's
also there's a huge human element to this. You have to have notetakers who are synced up and going to collect the right things, but you're also relying on their judgment to pick out particularly good points or moments in their notes. Um, and you've got a human again who's kind of keeping the conversation on the rails, which means that you've got people who have to be able to hear and like direct, but you also are collecting qualitative data from humans who may not always give like quality answers. I'm sure that you probably have it easier than me because I was a high school teacher. So, I have done extensive prep to ask a question and get like a one-word answer. How do you make sure that the people who are I guess both participating the game and providing your data to you, how do you prime them to give the best possible quality answers?
Uh the phrase going to hear from me over and over again today. It it depends. Uh, so sometimes sometimes we will send out briefs beforehand to make sure they have all of the relevant information. Oftentimes they have more information than we do because they just have specific or niche subject matter expertise that we don't. Um, and sometimes we will include briefs as part of the game day. If there's a specific area that the game is touching on or a specific set of policies, we have them briefed out beforehand just so that the the participants or the players understand exactly what it is that they're doing. But when it comes down to it, a big part of the game's job and the game designer's job is figuring out how to design the game that best gets at exactly the questions that we want to ask. How do we get that good answer from them? Um, and who do we call in to participate that likely has those answers? And part of that is on us as the designer. And part of that is also on the sponsor. Um, whoever is um, whoever sponsoring the the game or the contract, they often have a pretty solid idea who can answer some of their questions. And whenever there's a lack of information about who can answer something, we have a pretty extensive network that we can pull on and see who we can ask that might know. Okay, this is really interesting. So, what I'm understanding from this, this is something that I've not got to discuss with other pro war gamers, is that you
have a sponsor who has a question that you're trying to answer, choosing the players for your game, just as much as you're designing the game to find people with the knowledge and possible responses to elicit like the right kind of data from them is that so like what who I know you can't tell me exactly but like what kinds of people are on your rolodex do people just are they just on call to come play war games as experts um so for the most part we pull it depends upon the kind of game uh for most of the games that I've personally worked on uh we chat with folks who are active duty um and so we're looking for uh folks who fill a specific set of roles or do a specific set of jobs. And so we pull people from within that sphere. Um but for example, uh there was a game that we ran recently on, um trying to think of an example. Uh illegal and unregulated fishing. Um and that's been run a bunch of times. when it was run originally uh for I believe the Pew Charitable Trust, it was run for policy folks because it was inherently a policy game.
And so we pulled from a wide audience that included folks who understood kind of a the policy background who had um country expertise who had subject matter expertise in fishing or economic impact. And that's generally how we isolate the expertise that we need is we take the subject matter and then we pull on every possible thread from all different angles until we tease out a list of the expertise that we need and we hand that to our sponsor typically and we say here's generally speaking what we think we need and they'll give us feedback and it becomes a back and forth of of what we think could answer the question most effectively. So, it sounds like too that you're bringing in people who don't necessarily have the same question as a sponsor. They're just people who are experts who can maybe get at it. So, when you're trying to put people in a situation where they're primed to give you the information you want, how to how much to what degree do you tell them what you want? Because I feel like part of it too is that, you know, h do they when you're trying to answer a question, how much do you give people to get the right kinds of responses but without accidentally telling them too much where they're telling you things to try to please you instead of telling you things that are actually coming from the game naturally? It depends. Uh there are some games where the question is very intuitive or is a question that everyone
in the sphere has. So um one of the games that I designed when I was doing training uh with the naval war college we designed these little mini games and one of the games that my team designed was uh basically how do ships avoid icebergs? And when we daydreamed our list of ideal participants for this, right, we wanted everyone from um mechanical uh engineers to um uh ship builders to welders to captains to anyone who has any familiarity with uh thermodynamics. So we had this huge wish list, right? And part of part of the fun of trainings like that and exercises like that is trying to understand just how many different angles you can approach a problem from.
And all of those people, right, if they sit down and look at an iceberg are going to think, how could you navigate around that? Um, so sometimes you can get people in the room and they while they may not naturally approach that problem within the scope of their everyday lives, they're interested by it. Um, and sometimes it's better for the game as a whole to disclose what the objective is beforehand. And that's generally the kind of games that we work on. We say upfront, here is the subject that we're looking at, that we're looking for your input on. Um, there is something in stats called voluntary response bias. And it's a subset of voluntary response bias that basically says you because you only have volunteers, you will only get specific answers. Um, and a subset of voluntary response bias effectively says if you ask questions that make folks uncomfortable, that are incredibly difficult, if you engage mechanics that are not intuitive, you will only get answers from a very specific subset of people, that won't give you an accurate understanding of what the larger population has to say. Um, and that will bias your research. And so part of what we do when we pull folks in is we sit down as a team and try to figure out is it most helpful to disclose beforehand when everybody sits down to play the game. Here is our research question. Is it better to give them a topic or is it best to say here is the game we are playing and not give a lot of context.
Um and it it depends upon the game. For the most part, every game that I've worked on at CNA we have disclosed beforehand, but I'm sure there are shops that don't. Interesting. Uh oh. Say, okay. So, once you've collected all of this data, how do you how do you choose to interpret it for the client who has has asked for the game? Like, what kinds of reports do they get back from you? And how do you make sure that you're not accidentally overreading into things in a small subset of people who play the game? Um, so what do we turn around for them? Uh, reports can look like anything. Um, I've seen games where after the fact the sponsor didn't want a full report. They wanted a one-page summary. We can do that. Um, I've seen 80 to 200page reports. Um, I've seen reports that were more like game summaries that just sort of summarized what the game what the the game ran through, provided the data, and then recommended uh other shops that might be better suited to analyze it.
So, it it depends. Um, one of the benefits of being an FFRDC is that if we don't have the expertise in house, we're always happy to help the sponsor find someone that does. Um, but we also have a really rich and vast network of expertise within CNA because we just have a lot of very talented folks. Um, so oftent times because I am a statistician, I don't look at that data. Um, I help collect it. I will do a pass on the report afterwards or sometimes I'll help with the analysis, but because I'm not a subject matter expert in a lot of this stuff, um, I'm not involved in a lot of the report writing. So exactly what how much our report writing can vary I can tell you based on the final product but in the process I don't have a lot of context on. Um but it genuinely depends upon what the sponsor is looking for. Um I'll also kind of throw out there.
So every everyone who works in math sort of has their own niche. My niche has mostly been counterterrorism and dradicalization and there's very little gaming in that right now. So as you do you branch out. So I have done a lot of work with data that is not easily collected or has a lot of caveats because terrorists generally speaking don't want to be studied. So anytime you get data on terrorist organizations, it's kind of a question of where did this data come from? How reliable actually is it? What can I use it for? Um and how far does that go? So, one thing that I specifically get tapped for a lot is, "Hey, Michelle, here's the data that we collected. What can we actually do with this?" And based on uh who the players were, what the game looked like, what the questions looked like, I will work to break down sometimes with the study team what can reasonably be done with it. But there's also a lot of other very talented folks with similar backgrounds at CNA who do similar work with the team.
What is the afterlife of the data that you collect? Because I know that like specific sponsors want certain things, but then what happens to the data if they just wanted a one-page summary? What happens to all the things that you collected in support of that? How long do you keep it? And then can are you authorized to dig into the results of other war games you've conducted to answer similar questions that come from other sponsors in the future? Um the long and short of that question is I don't know the specifics offhand. We have a policy a set of policy guidelines that that governs kind of what we can keep and what we can't. Um the games that I have personally been involved with for the most part that data gets wholesale turned over to the sponsor. Um but we have sets of guidelines depending upon what our sponsor chooses that dictates kind of where the data is stored, what we do with it, what we can do with it afterwards, can we use it again? And it it just depends upon what the sponsor needs. Interesting. Yeah. I guess I always wonder like, you know, because a lot of the sponsors are government agencies, everybody's theoretically on the same team. So, does that I I I take it then that that doesn't necessarily mean that everybody's sharing the results of these war games in a collective way. It depends. Um there's there is a an archive there's a a gaming archive um that I know exists but I haven't unfortunately had the time to explore uh that does detail some reports but things
are limited by a whole bunch of different factors including classification level but also um sometimes if a subject is touchy it gets put in a separate corner or um sometimes if uh something is intended for public distribution it gets put into a different corner so that the sponsor can comb through it and then publicly release it. So, it just depends upon kind of what the sponsor is looking for and also what they intend to do with it. Interesting. So, we've talked a lot about collecting and assessing information from these games. But, I mean, obviously a lot of statistical information goes into the design as well because you're trying to assess the probabilities that certain things will happen. So, how will someone respond to aggression in this situation? Or, you know, what is the likelihood that a weapon will malfunction? Or, you know, there's like you there's so much uncertainty.
So, how granular does a professional war game tend to get in that regard? Or is that something that y'all I know it depends, but um the kinds of questions that y'all are typically asked to answer, how granular do they tend to be? Um, I'll spare you saying it again. Uh, some games are very very high level um, and don't get into granularity at all. Some games are extremely granular. Um, so we do integrate some models into our game design and into our game execution. Um, and we integrate some tools, some of which we build ourselves, some of which we we use that other gamers have designed, uh, or other institutions have designed.
Um, but it again it depends. There are some questions that are just innately very strategic. If you want to know um, what does policy need to look like, that's a strategic question. If you want to know how do I get again this thing from point one to point 2 that's a tactical question. So depending upon what the question is we can integrate more or less data or we'll engage modeling but there's also kind of a wider debate within the wargaming community about how and when to engage modeling and how skeptical to be of models. Um, and I think that's a larger conversation that has more to do with the role of modeling and wargaming and and model and mathematical literacy and data literacy overall. Um, that may not be relevant to your question. No, I I think it actually is relevant. Honestly, it's interesting to talk to you because I think data literacy is something that we as a society are low on. It's very interesting to think about like, you know, I was actually going to ask too, you know, you mentioned trying to control the environment and also be ready for bias and responses. How do y'all catch your own potential biases when you're designing these games? I think that that kind of gets the heart of the problem, right? That you can't assign values to things without making decisions about what they are worth. either, you know, in a nebulous sense or in a numerical one.
Um, different shops do this different ways. Um, CNA has a multi-step design process where we have about a 15 to 20 person team who has a variety of backgrounds. Um, so obviously mine is is stat and and math. Um, but we have physicists, we have chemists, we have social scientists and polyai and gov folks. We have um we have a couple astrophysicists, interestingly enough, but we have people from all different backgrounds. Um and what we'll do when we design a game is we have multiple meetings along the way where the design team will pitch the game to the rest of the team and we effectively throw spears at it. Um, so we have multiple kind of internal review meetings where we comb through every aspect of the game and walk through ways that it could break down or it might not work or opportunities for bias. So we rely on um a lot of self-criticism and our colleagues give us a lot of very helpful um feedback and and constructive criticism on where to move forward and and what to abandon. Um, and we constantly engage our sponsor as well.
Wargaming is kind of an iterative process. So we we'll develop a basic concept, go to our sponsor and go to our team, get feedback and the cycle repeats over and over again until we come to a product that we are analytically satisfied with. Interesting. So we mentioned earlier in the show that you are a hobby gamer uh as well and one thing that we all like to complain slash discuss um in complain about in the in our situations as as gamers is is this game balanced? Does this math work out? This game is very mathy. When you are thinking about this in a war game context, obviously fairness is not the point. Um, but I mean, do you is there any sort of craving for any sort of balance in the game play at all? Like, how do you get people to engage with situations that are inherently possibly unfair or kind of wonky?
Interestingly enough, um I don't think I've ever encountered a scenario where the players didn't want to engage. And I think part of that is because the problems that we tend to the problems that we tend to to write games on, the problems that we tend to focus on are usually deeply complex and important problems. Um, and so by the time that we get folks in the room, they're subject matter experts in something directly or tangentially related to the topic and they're excited to address it. And even in cases where they've been skeptical at first, by the time that we really get into it, usually they get very invested. Um, we're asking people about the most difficult or dire or important or in some cases mundane um, aspects of their job. Um, and most of these folks are people who really do love their jobs and want to talk about it. Um, that being said, engaging and keeping interest is also part of the design process. Um, and there's definitely been times when reviewing people's games when folks on the team have said, "You will not keep player interest if that is the way that you frame it. You need to redo this."
Um, or that mechanic is just way too complex. No one's going to understand it. Uh, and there are wargaming shops who designate a specific role of kind of a a developer or like a player advocate whose job it is to sit there and look at everything from a player perspective and throw spears and say that's not going to work. The players will not be able to make sense of it based on the information that you've provided. So, we try to engage that throughout the entirety of our process. Um, but ultimately it's also a question of of bias. At the end of the day, have you developed a game that is engaging enough with mechanics that are straightforward but targeted enough that it'll get out what you want without alienating anybody? Do you find that some of the mechanisms that you enjoy in your hobby life make it into games at work? And like what how much crossover do you typically see?
Um, there's absolutely crossover. Uh, it's one of the great things about being a hobby gamer. Uh, so there are definitely times where I have uh played a game or tried a game and thought, I'm using that mechanic. Uh, or that's really clever. I wonder why that works. And then I'll go down a rabbit hole trying to figure out what it is about a specific mechanical system or specific device that works so well. Um, and I'll try to pull that. A lot of it is a lot of where the crossover happens is finding things that you love or feel really intuitive in a hobby game and then trying to figure out how you can mimic that in a war game when it's needed. Especially when it comes to making mechan especially really complex mechanics intuitive. How are they explained? How are they broken down? How are they graphically broken out?
Absolutely. We use that. Interesting. So you mentioned that um you mentioned that you are a co-designer on a lot of these games. So what what does that co-design relationship tend to look like? Not the research part, but mechanically, you know, how are y'all divvying things up and testing together? Different for every game. Uh some games I am very much kind of the the co-pilot with the lead designer. Excuse me. Um, some games I don't always have that opportunity for a variety of reasons. Um, or they need help or support in a very different way. So, I will kind of be designated tasks and I'll go off and do my task and come back and present the results. Um, sometimes, uh, the lead designer knows what they're looking for and what they need is for me to apply the stat and study design background and be very critical, um, and poke holes.
So, varies game to game. There's something of a debate currently in the wargaming community about what mentorship does look like and how you develop a a junior designer and kind of get them up to speed. And that's I think a process that we're all figuring out. Yeah. I also just think that co-design in board games is so common. Like the expectation that you design on your own. I mean I all my stuff is code designs because I just want to work with someone else because I find that the push and pull of having somebody to talk to is really helpful. Yes. So you know it's not even just junior and senior. I think it's like how do you negotiate a partnership? Yes. And sometimes um more than anything it helps to be able to walk into someone's office and like red string across a whiteboard of like here's where we're starting and here's where we want to end up. how do we create all the diverging paths? Um, and yes, having someone to bounce ideas off of is incredibly helpful. Even better when that person's an expert in whatever it is that you're doing, especially if we're doing modeling and I have a modeling expert that I'm working with. It's fantastic.
Excellent. Um, so the other thing that's sort of interesting is, you know, people kind of fall into professional boarding. I don't know that there is a professional pathway for it still. I know that Sebastian teaches a class. It's there. But in your perception, what is the current state of wargaming, I guess, within the consultation world? Because I talked to a lot of professional war games. It seems like it's hot right now as a way to answer questions. Um, but is that perception correct? And do you feel like wargaming is on the rise? So with the preface that I think my perception is a little bit skewed because CNA isn't quite a consultant and we're not quite a contractor. We fall into this niche as an FFRDC where we do elements of both and elements of neither. So from my perspective it certainly seems as though it's been on the rise.
Um all of the major wargaming conferences have seen a lot more participation um not just domestically but abroad as well. So, it it seems like as a method of analysis, it's certainly up and coming and it's been getting a lot more attention both in the news and from from consultancies and and wargaming shops. And there's been a lot more contracts again from my perception. Um, and there's been a lot more interest in what wargaming can do in difficult analytic corners. um because it it's a neat method of analysis that is can be very very useful when nothing else works. So finally kind of having the verbiage to put to that problem I think has led to a lot of a lot of interest.
Interesting. So if somebody wanted to follow your career path um such as these things go uh what would you recommend to an aspiring future professional war gamer? Um there are now in a way that hasn't really existed before classes on wargaming. So you can go take a class at Georgetown or uh there's now uh more than one master's program in wargaming. I believe most of them are in the UK um with a couple others scattered through Europe. So there's there's larger wargaming methods of study. Um, and you can absolutely go get a degree in wargaming. Alternately, all of the folks on my team personally have varied backgrounds.
So I would certainly say you can go for the academic route or you can become an expert in your chosen field and then see if you can pursue wargaming from there in a combined shop with a varied expertise. Interesting. Yeah, it sounds like it's still as hard a question to answer as how do you become a board game designer period, which is there are there are programs for it now, but uh a lot of people just do what they do and then they move over into it. My again very biased advice uh would be hobby game. If you're interested in this, get a repertoire of mechanics and designs that you can kind of run off of and then from there determine where you want to go with it because I do think that having a background in hobby gaming for me personally has been very useful although there's certainly a lot of talented folks in the field who do not have that.
Excellent. Um, and then I don't know, maybe this is a good question, maybe not, but I just kind of want to know if you have, since you're a statistician and we're in a world where numbers are getting thrown at us all the time, if you are reading a newspaper article, watching the news, looking at a Tik Tok, and you're given some numbers, what is the first thing that you should ask yourself if you're trying to assess how seriously to take them? Um, I love this question. So, uh, number one, I want to know what is it actually measuring? And I'll break that down in a second. Um, I want to know how did the how was the study designed? So, where so what is it measuring? Where did it come from? Um, and what is it a part of? And I I'll explain that more in a second. Um, so what is it actually measuring? It is really common for uh Tik Tok in particular to say 73% of this thing are doing this thing and it sounds like a big number but they don't tell you 70% compared to what um they'll tell you 70% of this thing and then imply that it's large or imply that it's small but they don't tell you what the number was last year or the number is projected to be next year and so it doesn't actually tell you whether that 70% is a big deal or not. So, what is it measuring? Um, and compared to what? Additionally, are you looking at 70% of the entire population? Are you looking at 70% of
women under 13? Are you looking at 70% of Gatorade bottles consumed in the United States? Like, it's very important to understand exactly what you're talking about and the context it exists in. Um, number two, uh, what was the study design and where did it come from? Nine times out of 10, if I see a statistic, uh, I will stop and Google the study and then typically discount it because a lot of kind of pop science studies just don't have rigorous methods. Um, which is part of the role of peer reviewing. Although academia can be very exclusionist and that has its own problems. Um, I say this as an academic, uh, it is still extremely important to go through a peer review process. So I tend to look at the methods and try to make sure the study design is as it presents itself. Um and then finally the what is it in the broader scheme of things. I think that we touched on this earlier. There's kind of a lack of data literacy and a lack of mathematical literacy. And ultimately we like to rely on really complicated study methods. We like to rely on machine learning and AI and they can be very useful tools.
But when it comes to actually answering questions about people, um I had a professor in grad school who used to say this and it stuck with me and I say it all the time now. Uh modeling, statistics, AI, machine learning are very rigid tools. They're kind of like rulers. So trying to use them to solve these very squishy human problems is like trying to measure like a a blackberry with a a yard stick. You you have this very angular thing and a very squishy human problem and it'll get you an answer, but there's going to be edges left over and you won't exactly get at what you need. And as long as we remember that they're very rigid things for very squishy problem, I think it's a lot easier to understand the flexibility and where there's boundaries that we need to keep in mind when reading stats.
Interesting. Yes. So, when someone tells you the numbers don't lie, maybe not, but how did they get there? Yeah. Yeah. All right. So, uh, this is actually, I think, a wonderful place to kind of leave this conversation because this has given me a lot to think about. Uh, I normally ask people what they're what game they've been playing recently that they really like, but I've asked you that already. So, um, I'll ask you, you got a lot of books behind you. What have you read recently that that you've enjoyed? What have I read recently that I've enjoyed? Um, so I just reread To Kill a Mockingbird. That's a classic. Yeah. I try to do rereads of classics um every few years and I hadn't read it since high school, so I reread it recently and I loved it just as much in the second read as I did on the first. That's awesome. Yeah. What about you? Um I actually read a lot this year. It's It couldn't possibly be that I'm stressed out. Um but one of the books I just finished that I really like about to return to library was um Our Share of Night by Mariana Enriquez. It is set in Argentina. It's about it turns into a larger family drama, but it starts with a father who is a medium.
His son who may have the gift and he wants to suppress it because his wife belonged to a creepy family that like worships an entity called the darkness and they've been using him to summon it and they want to use his son and he's trying to like get them out of there. Who did you say this was by? Uh Mariana Enriquez. But it was it took me like a little bit to totally get into it and then it just got its hooks in me and I could not stop. It was a great book. It was really fun. It sounds incredible. Yeah, it was creepy. It was interesting. It was dramatic. Like they should make a miniseries or something out of it. It was really good. I love when you find a book that's so perfectly laid out in terms of the plot and the staging and the scenery uh that you read it and you can't help but picture a film in your head. Oh yeah.
Yeah, definitely. Uh, and then if people want to follow your work, um, where can you be found online or are you just not at all an online person? Um, I am on LinkedIn occasionally as Michelle A. Strayer. Um, and occasionally I will post updates from uh my LinkedIn mostly about CNA stuff. Uh, but honestly I'd say if you're looking to follow Wargaming as a career, follow Sebastian Bay. He's got some really great updates and he posts a lot about the work that CNA does as well as innovations in the field overall. Excellent. Michelle, thank you so much for coming on. This has been a wonderful conversation. I've just really enjoyed it. Thank you for having me. This has been delightful. All right, everybody out there, you know the drill. Please like, subscribe, comment, ask questions, and most of all, happy gaming.