Adrianne: This is Underunderstood.

Billy: Hi friends.

Adrianne: Hello.

John: Hey Billy.

Billy: So today I have a mystery about something that is designed to be mysterious.

Adrianne: Ooh.

John: Is it an escape room?

Regina: Is it clue?

Billy: No. The questions that I have are not necessarily the questions that they want me to ask.

John: Okay…

Regina: Is this a conspiracy theory?

Billy: No, I’m talking of course, about The Masked Singer.

Regina: Oh.

Billy: Sorry, I was just imagining like the music comes in there or something.

Regina: Oh okay good.

Billy: So I actually asked all of you to watch the first episode of the most recent season of The Masked Singer. You all did that, right?

Adrianne: My pleasure.

John: All of the episodes, Billy. Come on.

Billy: Okay. Well that was going to be my question. Uh, how familiar are all of you with this show?

Regina: I watched one episode an hour ago.

Adrianne: I watched all of the first season at the beginning of COVID and then a little bit of the second season.

And then just this episode that you asked us to watch.

John: I am borderline fanatical.

Regina: Wow.

Adrianne: It’s like Goldilocks and the three masked singer fans.

Billy: All right, could one of you sort of explain what the show is?

Adrianne: All right. So the concept of The Masked Singer, first of all, I feel like whenever I tell people about it, I note that it started as a Korean show, which explains some of the zaniness that is like, it’s like up a notch from American television zaniness.

They get celebrities to wear these giant elaborate costumes, and then they sing in costume and you can’t tell who they are. And they have a panel of judges speculate about who they are the whole time. And then there is an audience vote. And they vote someone off the Island and that person has to take off their mask and then everyone in the audience goes, “take it off, take it off.”

And that’s kind of disturbing.

Billy: Exactly.

John: And then you see who the celebrity is at the end of every… every episode, one celebrity gets voted off and then it reveals who they are.

Billy: Right. So, yes, like you mentioned, it’s based on a Korean show called King of Mask Singer which has been around since 2015. By 2019, there was a U.S. version of the show.

Advertisement: TV’s wildest singing show is here. Can you recognize your favorite celebrity just by their voice? You’re about to find out.

Billy: The U.S. one is designed to be like a certain type of competitive reality show. That is, a reality singing competition that takes place over the course of a season. Like The Voice, like American Idol.

Advertisement: Each week, the celebrities will sing to impress the panel in studio audience.

Fan: ” I love him so much!”

Billy: Because the U.S. one is more of a season long competition, it’s also way more secretive than the Korean one, because they don’t want the reveals to be spoiled. And so they go to pretty great lengths to hide the celebrity identities and swear everyone involved to secrecy. And because of that, there are entire communities online, you know, dedicated to trying to decode who these people are and, and figure out who’s behind each mask.

But that’s not what I’m interested in. So The Masked Singer was a big hit in the U.S. It is a big hit in the U.S. They did two seasons in 2019, and in 2020 season three premiered in February, at the beginning of the year. So that was before… it was February 2nd. So that was before COVID was really affecting most people’s daily lives in the United States.

John: When was that?

Billy: Uh, well, when was…?

John: No that was me singing “when was that?”

Billy: Oh.

Adrianne: When was that?

Billy: February 2nd.

John: Yeah. Thanks.

Billy: Um, I don’t think most people, I don’t think mask wearing and, and schools and stuff closing happened until like mid-March. So they film this stuff ahead of time. So there were still full audiences and everything.

And Fox tried to do some additional sort of weird virtual studio stuff that were like after shows where they would have video calls from people who are just eliminated celebrities who are just eliminated.

John: These were the worst things I’ve seen. Like, this was the worst of the pandemic TV.

Billy: This was really terrible.

John: It was so rough. It was on this awful virtual set that looked like they had bought it from like, the cheapest stock source that you could find. They had canned laughter and applause during these things, it was awful looking.

Nick Cannon: Did you ever get a crush on any of our masked singers?

Jeff Dye: Well, every time I hear real beautiful music or something like that, like I was like, “ooh, this kittie’s a hot one” or I go, “who’s this purple faced lady?”

Billy: Those apparently did not perform well. Um, so after that season ended, they were faced with a challenge. You know, the normal show is a huge hit and it’s designed to happen twice a year and they already did the first half. And they have to make a new season post COVID, but the virtual studio experiments, they did with the after show stuff went terribly. So they were left with this question of like, what do we do? So this is what they did.

I’m going to play a bit of the first episode of season four… which premiered on September 23rd, 2020. Feel free to describe what you’re saying.

John: Well, here’s Nick Cannon wearing a fancy little handheld mask.

Laughter

Billy: This part you’re laughing at is this huge virtual set. There’s a UFO, there’s a dinosaur. It looks like the normal set, but it looks like he’s elevated up on a skyscraper.

Nick Cannon: Nick Cannon here. And finally we have something fun involving math. Yeah, there’s a dinosaur. There’s a billboard, some cranes.

Billy: But then an interesting thing happens and he’s like, “check out the set.”

Nick Cannon: Really stepping it up for season four.

Billy: And then he snaps his finger…

Nick Cannon: Through the magic of television it can all go away like this.

Billy: And then it looks pretty normal.

John: I wanted to know why that happened, cause it never happens again. The rest of it. It appears to be on the normal Masked Singer set.

Adrianne: Except without an audience.

John: No, there’s an audience.

Adrianne: No, that’s cuts from previous Masked Singers.

John: But there are these wide shots, you see?

We’re seeing a shot from behind Nick cannon right now, and he’s on the set and then you can see audience surrounding the stage and up in front of him.

And there’s also the panelists.

Billy: Yeah. And then there’s additional people behind them,

John: behind the judges, yeah.

Billy: Yeah, and these… so there’s cutaways. So you’re saying those cutaways look recycled, right?

Adrianne: I believe those cutaways are recycled. I believe when Nick Cannon said “through the magic of television” what he meant was all of these people are fake. And there are also these shots, these shots from farther back where it looks like they have fake heads and they’re shooting from behind the fake heads. So that they’re silhoutted against the stage.

Billy: Right, you’re talking about these jib shots they do that sweep across the stage.

Regina: Right and it’s just like a very nice cut-out.

John: If all of that is fake though, this is like the best fake I’ve seen on TV.

Adrianne: I know they’re going to win an Emmy for it.

Billy: Right?

Adrianne: Best COVID pivot.

John: This looks very, very, very convincing. If it’s all fake.

Billy: The Masked Singer in general has this sort of hyper real quality, right? Where it’s like the cutaways are always kind of too perfect and seemed staged. Um, you know, it’s very like… tight. However, this season does not feel that different than previous seasons.

Whereas like if you watch a baseball game with virtual fans, it feels extremely different, right? So people were very confused by what was going on in the audience of season four of The Masked Singer. Here’s some tweets…

First tweet: Watching The Masked Singer and someone please explain how so many people are in the audience with no masks. Is this prerecorded?

Another tweet: @MaskedSingerFox what’s with the audience? No masks. No distancing. Fake audience? Virus non-believers? I don’t get it.

John: Virus non-believers is really good. Pack ’em in there, we’ll use ’em.

Billy: Like they’re just like… they’re like standing outside a Trump rally with like handouts.

Here’s another tweet: Is The Masked Singer filmed in New Zealand or are they faking the audience? Feels irresponsible.

And then one last tweet: seriously, what drugs do they give the @MaskedSingerFox audience? Or are these all been locked up in COVID isolation with their kids since March?

So it’s safe to say that a lot of people had reactions like this.

And the simple answer to the question here as we’ve kind of guessed, is that all of this is fake.

John: Huh?

Billy: So if you look closely at the end of each episode, there is a disclaimer that goes by very quickly. Here’s what it says: due to health restrictions, visuals of audience featured in this episode included virtual shots as well as shots from past seasons. The use of these visuals was for creative purposes only and did not impact the actual audience voting and outcome.

John: So they said virtual reality.

Billy: Yes.

John: Okay, Caitlin and I were also curious, watching the show, how they were doing the… if they were doing a virtual audience ’cause we saw that opening thing too.

And we noticed that in the end credits, in addition to this thing about there being, uh, that the voting is real or whatever they’re saying, there’s like a special thanks to unreal engine or something like that in there that I don’t think had been in other seasons that made me think, “Oh, they’re doing something with like, With unreal engine to generate a set or generate the audience or something like that in real time,” because that’s a game engine that could be used presumably to make 3D stuff happen in real time.

Billy: Presumably, but we don’t know what they’re using that for.

John: It also looks really good.

Billy: Like, yeah. That’s the thing, like, again, this show has always felt kind of fake. It has a hyper real quality, for lack of a better term. And you see that in other shows like ‘America’s Got Talent’ or whatever, they cut away to people and it’s like that they couldn’t have done that cut live to tape, but The Masked Singer, it sort of exists in its own universe. You know, it’s, it’s, uh, it’s the perfect world for a celebrity. There’s no controversies, you know, they’re not going to make Nick Cannon address his antisemitism scandal, for example, or they’re not going to explain why during a pandemic, they brought back Jenny McCarthy, probably America’s most famous anti-vaxxer. The Masked Singer is an idyllic celebrity safe-haven, where the worst thing that can happen to you is that you get voted off while wearing a banana suit. But reading that disclaimer at the end about the audience being fake, sort of snapped me out of that, when I read it.

And I think it presents a lot of new questions about what is real and what is fake. So I should probably break this down into a list.

So, John, do you remember those sorts of musical notes? We use to break up the questions in the nine 11 episode.

John: Yeah. Yeah.

Billy: Could we bring those back?

John: If I can find the project.

Billy: Okay, great.

Regina: This is why archiving is important.

Billy: If you hear them, John did something, right.

Number one, what is it like to be in the audience of this show? Do these people actually exist?

John: Were they even watching a performance when they were screaming?

Billy: Right? Or, have they been hypnotized somehow? And they have no memory of any of this. So that’s question one.

Question two, what do they mean that “the fake audience doesn’t affect the voting.” Like, is the voting even real?

And number three, what do they mean when they say they’re using innovation and virtual reality? Are there virtual fans, you know, what’s real and what’s not real?

And more importantly, how does it work?

The show is extremely secretive, so it might be hard to talk to real people who were there, who have attended the taping.

Regina: If there are any real people.

Billy: If there are any,

Regina: it could all be virtual.

Billy: Right, maybe the whole crew is virtual too. We don’t know.

Regina: Everything.

Adrianne: It’s all done in unreal engine.

John: Nick Cannon has been dead for years.

Billy: The whole thing is a simulation. So anyway, like I said, there’s a lot of secrecy. This might be a challenge, but I guess that’s kind of the concept of this podcast

Adrianne: Coming up, Billy answers the questions he just asked. I think, you know the drill by now.

Billy: Hello?

John: Hello.

Adrianne: Hi.

Regina: Hey.

Billy: We’re back, and I have answers.

Adrianne: Who is that?

John: What are they?

Billy: It’s me, Billy. And the answers are…

Adrianne: John, you’re going to auto tune all of these when, when you edit right?

John: Absolutely.

Adrianne: Good.

Billy: Okay. Auto-tune this. First up: question one, what is it like to be in the audience of the show?

Adrianne: You can stop.

Billy: Okay.

So question one, what is it like to be in the audience? What’s the deal with the audience?

Kyle Montplaisir: Hi, this is Kyle.

Billy: Hello, Kyle, hi, this is Billy.

Adrianne: Is this an audience member?

Billy: This is Kyle Montplaisir.

(Recording)

Mel: Hi, I’m Mel.

Kyle Montplaisir: And I’m Kyle.

Mel: And welcome to the talent recap show.

Billy: He actually is a host and reporter for a website called Talent Recap, which covers talent shows, and they’ve become sort of well known for their coverage of The Masked Singer. He’s kind of like a prediction guru.

Kyle Montplaisir: You know, when I’m looking online about who people are guessing the popcorn might be, I see Taylor Dane brought up a lot,

and Peter Turner.

Billy: So he has like a deep understanding of the show, but he’s also been there for a taping during season two.

John: Like in the audience?

Billy: Like in the audience.

John: Nice.

Kyle Montplaisir: It was honestly a crazy experience, going to a taping, and I think part of that is because it’s just so surreal.

Billy: And basically as expected, he explained that the whole process is kind of secretive.

Kyle Montplaisir: You go in, they take your cell phone, you sign like a non-disclosure agreement and do all of that, just to say that you’re not going to go and tell everyone who was revealed on the episode you watched.

Billy: So the experience of being there is, as he explains it, really like close to what you see on TV, actually. Like in a normal season, not in this season, obviously. However, you know how I was saying, like, the audience cutaways always do feel staged and fake?

John: Yeah.

Billy: Well it turns out they do a lot of capturing of audience reactions.

John: Like not during performances.

Billy: Exactly.

Kyle Montplaisir: They have an audience coordinator who’s going through in the commercial breaks was doing fun games and things to keep everyone entertained.

John: Oh my God.

Billy: So there’s this audience coordinator. They’re always trying to get these different reactions out of people. All of this stuff is being taped. And then also, after a performance happens, they go back through and get even more reactions from the audience.

Kyle Montplaisir: The Masked Singer leaves the stage and they played their performance over again and just had all of the audience members dance to the song, react to the song, clap. They had us do a bunch of reaction shots. The stage manager would be on stage and say, “okay, let’s, you know, make a face as if Ken just gave some ridiculous guess.”

Ken Jeong: Kerri Washington in the house!

Kyle Montplaisir: And you know, we would react accordingly and they would be taping all of that. Or they would say, okay, now someone made a guess and you think they might be right.

So we did a lot of those shots, which again, talking about this, make sense how they can put together audience shots just based on what they have from previous season, because a lot of that… they have a lot of extra footage there. I’ll say that.

John: I love this.

Billy: Yeah, it really seems like the staging of reaction shots is part of the show’s DNA. You know, part of the charm, I guess. Kyle actually used to work on two shows with live audiences, ‘Wheel of Fortune’ and ‘Jeopardy!’ And for those shows, he says they would shoot three episodes at a time, often six episodes in a day, but for this show, ‘The Masked Singer,’ even though they shot two episodes, the day that he was there, he says they brought in a new audience for each episode. And he says that the process for his group, just for one episode, all in, was probably like five hours. So presumably, they’ve archived a lot of footage of audience reaction shots.

And I asked Kyle. Have you seen shots of yourself in the latest season?

Kyle Montplaisir: I haven’t seen any from the taping I was at this season, but the season that I went to a taping of, they used like clips of my wife and I, and the audience they used in, I think three different episodes. And we only went to one episode taping.

So even like within a season they would recycle footage. And so, you know, on this season, from what I can see,

I think it’s mostly from last season.

Adrianne: The magic of television.

Billy: The magic of television, which brings us to the second question, which was about voting.

Nick Cannon: Enough talk, time to vote.

Billy: The voting on this show is very strange, I would say. It feels more staged to me than the cutaways. What they’ll do is they’ll cut to someone in the audience making a choice on their phone. From an app

John: Like an over the shoulder.

Yeah, yeah, yeah.

Billy: Kyle told me that all of that is staged. They’ll give that person a device and they’ll have a camera person go into the audience and get that shot.

But there does seem to be real voting.

Kyle Montplaisir: They had essentially like a little remote and, you know, you press like A or B, who you wanted to vote for. And then they stage some shots with like a phone.

Billy: But I was still highly suspicious. And here’s why:

Nick Cannon: It’s time to release the dragon.

Billy: One of the performances on the first episode of this season is a character called the dragon. And as soon as they open their mouth, it’s clear that that person is Busta Rhymes.

Regina: Yeah. Like immediately so obvious.

Billy: He has an extremely unique tone. If you’re familiar with Busta Rhymes, it’s like, “Oh, that’s Busta Rhymes.”

Regina: I don’t even think you really need to be that familiar with Busta Rhymes. Like, it was just very obvious.

Billy: Sure, yeah, I mean, I would think if it’s obvious to you that that performer is Busta Rhymes… First of all, I think it was the best performance in the episode.

Adrianne: Yeah.

Busta Rhymes: Rapping.

Billy: I think if it’s clear to you, that person is Busta Rhymes, you want to keep them around. The reveal is worth nothing. So you don’t want to eliminate them. But you would want to keep Busta Rhymes around to see what he does next. Is he going to sing a song? Is he going to do one of his more acrobatic rap performances?

He did a pretty like straightforward rap song on this one, but you would want to see like, yeah, does he do something like more complicated or does he try to sing? What is he going to do? At least if it was me, that would be my thought process.

Adrianne: I think some celebrities, especially some of the bigger celebrities may not want to commit to an entire season of being jerked around and may say, Oh, it’ll be fun to do one or two episodes and perform and get back in the mix a little bit, but not do the whole entire competition.

Billy: This was exactly my thinking and then I looked closely. At the credits and it says this at the end of every episode, it says portions of the program not affecting the outcome have been edited and or recreated. All voting was administered by an independent vote tabulation company and monitored by Fox standards and practices.

And there was actually an article in Cosmopolitan about this titled “Out of Curiosity, How Does The Masked Singer Voting Work This Season?” and in it, it says “this time around The Masked Singer is inviting super fans to become part of an elite group of voters who can influence the outcome of the show by watching via zoom from home. And yes, confidentiality agreements are involved.” So it seems like they’re trying to do this on the up and up. There are all these legal disclaimers, I think because it is, uh, a contest that is on broadcast television. So there’s certain FCC rules and regulations that they are subject to, but I was still, again, kind of suspicious of it.

The results don’t necessarily seem to make sense all the time. So I just asked Kyle straight up.

And do you think the voting’s real, does it actually affect who stays and who goes?

Adrianne: The million dollar question.

Kyle Montplaisir: No, I don’t want to get into any conspiracy theories, but the truth about all of these types of reality competition shows days it’s up to the producer’s discretion, you know, I’m sure especially when you’re dealing with really big celebrities and things like that.

I’m sure there are some who are more eager to leave than others. I’ll say that.

Adrianne: Kyle was validating, again, my theory, that the big names they want to reveal early in order to make it seem like there’s going to be really juicy stuff happening in the season.

John: There was an episode on this season where that happened, Adrianne, and they didn’t vote because the participant took off their mask at the end of the episode.

Nick Cannon: Yeah, right, Gremlin… What are you doing?

Judge: What’s happening?

Nick Cannon: The Gremlin’s telling me he wants to take this off. It’s too damn hot.

John: I don’t think. Maybe that was like a planned thing… like it’s a fun stunt, but it’s like, there is precedent for ‘this celebrity didn’t want to do the whole thing and decided that it was too hot in the suit’ and left.

Billy: They go out of their way to claim that they’re doing things by the book. And again, because it’s a contest show on public airwaves, I’m like, I sort of believe them, but, here is another interesting tidbit from that Cosmopolitan article.

It says, quote, “the judges votes are worth 50% of the total score and the audience votes are also worth 50%.” So if you think about that in practice, yeah, the judges can basically collectively control who gets eliminated, more or less.

John: So it’s, it’s the average of the judges is 50% or the winner of the judges is 50%

What’s the rank?

Billy: So each judge has 12.5% of the vote. Okay. But if they are all on a unified front, they basically control the vote because they’re 50%. And I don’t think there’s ever been a tie, like, the entire audience votes one way, the judges all vote another way, but I would kind of guess like if there was a tie, they would leave it up to the judges.

So that’s how the voting works. It’s the voting is real, but it does seem maybe tilted. Yeah. They have their own weird Masked Singer electoral college going on. Which takes us to the third question, what’s the deal with the audience? The supposedly virtual audience.

Kyle Montplaisir: I was wondering about that too. And so that’s something I’ve looked into and there aren’t really a lot of answers about that.

They say that there is like a virtual reality aspect to this season, but I haven’t seen any specifics as to what that is.

Billy: It seems like this is what the big mystery is. Cause there’s no reporting on this. I’ve only spoken about it in vague terms, and Kyle also told me that he thought that opening shot with the dinosaur and the UFO is weird. But he couldn’t really, he couldn’t really speak directly to that or how it might work. So essentially it seemed like I would need to talk to someone who actually worked on this season.

Adrianne: Nick Cannon.

Billy: So I tried Nick Cannon.

John: Nick’s doing all the virtual reality himself.

Billy: That’s true. No, I didn’t try Nick Canon, but what I did try was I got the information for the PR people that I would need to reach out to, and I never heard back. So I figured I would try reaching out to someone from the crew directly. I don’t know if any of you have actually tried to read the credits on their show.

John: They’re so crazy.

Billy: They go by so fast.

Regina: I actually noticed that. It seems like… I know that happens often, but it seems worse with this show.

Billy: It’s really intense. So I literally had to go through frame by frame and I eventually found a role in the credits titled Pixotope/Unreal programmer and operator.

John: Pixotope?

Billy: Yeah. And it was actually  written as Pixotop, but I quickly found out that that must be a typo because Pixotop is not a thing, but Pixotope is a virtual production system that uses unreal engine. Basically provides a tool set optimized for things like AR. Utilizing unreal engine and the person listed under the role was someone named Nikcon Mirsepassi.

And I figured if anyone would know the real details of what is going on in this show, it would be the Pixotop / Pixotope / unreal programmer and operator.

John: Maybe pixotop is like a knockoff.

Billy: I don’t, I don’t think it’s a knockoff, but I found him and I talked to him and I asked him how he got involved in the show.

Nickon Mirsepassi: During the pandemic, you know, a lot of movie and TV studios had to shutdown. So there wasn’t much progress made with any sort of like production.

Billy: The studios more or less shut down completely for a while. They were on hold, but they came back pretty quickly with new restrictions and guidelines and Nickon says that those new rules led to a pretty big shift.

Nickon Mirsepassi: What happened is a lot of studios decided they want to switch completely over to virtual production… do as much of it in virtual production as possible, not only to save monthly, but because it would make it easier to abide by COVID restrictions. Honestly, that’s when it really blew up, like the concept of fake crowds, the concept of fake sets. You don’t have to hire a bunch of prop people to come in and haul things on stage.

All that kind of stuff.

Billy: Nickon said that this is the first that to his knowledge, The Masked Singer has used virtual production on the show. And specifically what he was there to do was augmented reality.

Adrianne: This feels like a really big shift.

Billy: Yeah, it is. Totally.

Nickon Mirsepassi: You have a broadcast camera. That’s recording everything that you normally would during the TV show and then superimposed onto that are computer graphic elements that essentially exist within that set.

So the camera is being tracked in 3D space and based on location of that camera, you can tel where the 3d graphics elements should be and how they look like inside of the camera. And this all happens in real time. You don’t have to composite this afterwards.

Adrianne: So you’re looking into the camera and you see what the audience will eventually see or something closer to it.

Billy: Yeah, it’s all composited in real time. This isn’t something that’s done in post-production. They can see these effects happening as they’re shooting it, which made me wonder, is this set even real at all? Like, there’s that opening shot… like what’s going on with the set?

Nickon Mirsepassi: The set itself…

Nick Cannon: Can all go away like this.

Nickon Mirsepassi: There is a giant X in the middle of the stage, just like on the show, right? And that X has an LED screen on the floor and behind them, there is a, there’s a sort of like a curved led screen behind them between those two big masks on the right and left. All that stuff is practical. Those are all real.

And then everything that’s not real are things like the floating props on stage, the props that exist, like maybe, around, off the stage. And also oftentimes you’ll see these like videos that seem to be like layered and projected above the stage and on the left and right of the stage, that’s all also AR none of that’s real. There aren’t screens there in real life.

John: Those props on the sides of the stage… we were watching last night and it looks like they’re being lit by the lights that are on the set. It’s amazing looking how good, the match is between lighting on the character and lighting on these props, like is the lighting system also being simulated in this system?

Billy: Yes. I actually want to dig deep into this. Cause I think this is a great technical example of like how this stuff works and what’s so kind of fascinating about it. So this is a performance by giraffe.

John: Oh my God.

Adrianne: Oh yeah, yeah, yeah. I remember this.

Billy: See, if you can spot, what is, what you think is AR in this.

Adrianne: They’re like these gold picture frames hanging above the stage. I’m guessing the gold picture frames are AR.

Regina: Yeah. I think the frames are AR for sure.

John: But it’s flying out really convincingly.

Billy: No, the light hits it pretty naturally. I have to admit, when I watched this, I didn’t realize they were using all of these kind of AR props. I think once you realize and start looking for it, you start noticing like, oh, this isn’t actually on the stage and that isn’t actually on the stage.

John: That’s crazy.

Billy: So, John, you were asking about how the lighting works. Yeah. Uh, you’ve been involved in both theater and television production, you know, about the like protocol that lighting boards use, right?

John: Yes.

Billy: It’s called DMX. Yes. One of the first things Nickon did when he got on the actual set is work with the lighting person to figure out how he can tap into that data and use it.

John: Oh my God.

Nickon Mirsepassi: It’s always like someone on a lighting console with a bunch of like little levers and like sliders and stuff, and they can control different lights in the studio, real physical lights, but that data is just sent across like a local network. So technically, you can forward it to wherever you want. It doesn’t have to go to the lights.

What I was asked. And what I, uh, thought would be a good idea also, would be to link up the lighting controls into the game engine. So essentially we would send those lighting controls for certain like universes of data to our local network, and then have it like broadcast out to each computer.

John: This is so crazy.

Adrianne: Light operators are operating the lights live and instead of operating real live lights, they’re operating lights in unreal engine.

John: They’re operating both.

Billy: Both.

John: And this guy has to have some kind of, he’s got to put virtual lights in a virtual set that matched the real lights in the room and respond to those commands the same way that the ones in the room do.

Billy: Exactly.

John: That’s crazy.

Nickon Mirsepassi: Essentially, we get these DMX controls, which are like a, an order of numbers, right? And using those numbers, you can control different sliders on how it might act. Like how much does it tilt? How much is it zoom? Like how much does it change the cone of its light? And what I would do is then just, you know, code in C++ and unreal blueprints and digest that data through the network, bring it to like a front facing area inside the game engine where you can use game programming to essentially access those numbers and be like, okay, now that I have these numbers in front of me, I can apply these to a fake, uh, like a virtual light.

Billy: So, in the game engine, you need virtual lights to be able to see stuff like, you know, in the real world. Without a virtual light, those picture frames would just render as black. And you can kind of just use a predesigned light in unreal engine to light them, but it wouldn’t look quite right. It wouldn’t match the lights in the studio.

Nickon Mirsepassi: But what you can do is if you give the lighting programmer control, he knows what lights he’s turning on in real life. So he can use the virtual equivalent lights to light the props and the set to the virtual props. So that way you get it almost as if it’s sitting in there and then in the real light.

Billy: What I, what I love about hearing about this stuff is that this is the stuff that people at home don’t notice for the most part, you know, no one was tweeting about the lighting on the picture frames in the Giraffe performance.

But there is some interesting stuff to get into that is noticeable. Like that scene we talked about with the UFO and the dinosaur, it was in the first episode of this season, there was like a really wild opening shot.

Nickon Mirsepassi: Yeah, I know what you’re talking about. Everyone I talked to that wasn’t there for the taping of that episode was like, Oh, we thought that was just done in post.

You’re talking about the one with the city and the dinosaur right?

Nick Cannon: Season four of The Masked Singer.

Nickon Mirsepassi: We had a type of system called a holdout, right? And a holdout is essentially a 3D geometry that we make or laser scan. In this case, we used a Lidar scan to scan the entire set, convert it to a 3D model and then place it in the location where the actual real set is supposed to be.

So as you know, we’re tracking the camera. So based on where the camera is, we know where it’s looking in 3d space so we can place whatever we need to, right?

Billy: I see.

Nickon Mirsepassi: Yeah.

Billy: So it’s just like, it’s like a more dynamic matte basically?

Nickon Mirsepassi: Yeah. Yeah. It’s pretty much just like a fancy matte, honestly.

Adrianne: Will The Masked Singer ever need to shoot another audience again, it sounds like, they have the reaction clips…

John: Wait, is the audience also fake?

Billy: This is…

Adrianne: Oh, did we address this?

Billy: Okay. No, this is the remaining big question. Question three, what’s the deal with the virtual audience?

Nickon Mirsepassi: That was actually a different team called silver spoon.

Adrianne: Whaaaaat?!

Billy: The virtual audience was added using the same software, in the same studio, but in a different isolated room by a different team. So Nickon can’t really speak exactly to how all of that works. But Laura Herzing can.

Laura Herzing: Hello?

Billy: Laura is the executive producer at Silver Spoon in Brooklyn.

Adrianne: Oh my gosh.

Laura Herzing: Silver Spoon is a performance capture studio. That’s kind of, at the root of what we do. So for creating the animations, we started by just capturing of range of human reactions.

Billy: One thing that’s interesting about Silver Spoon is… remember how I mentioned that I thought that Fox baseball stuff wasn’t that great?

Adrianne: Uh-oh.

Billy: Yeah, they did that baseball stuff.

John: Yeah. But that’s like a less controlled thing, right?

Billy: Yeah. It happened very quickly. She explained that, uh, like Nickon said, this was all something they had to get together very quickly because the industry blew up because of COVID.

John: Yeah.

Laura Herzing: Literally just was a conversation that we were having internally about all of these stadiums for professional sports that were now going empty and seasons being postponed because of the pandemic.

News Announcer: Major League Baseball to delay the 2020 opening day by at least two weeks.

Laura Herzing: Somebody said, wouldn’t it be funny if we could put virtual people in these stadiums? And it just kind of… the idea stuck. And we started pitching it to various networks and Fox Sports bit on it and was willing to try it with us. So, uh, in a matter of just a few weeks, we developed an entire virtual crowd platform to allow us to place, you know, 40,000 people in a venue.

Billy: That, essentially, was the foundation for what they used on The Masked Singer. And since this was developed to be able to work live, they were able to add in virtual fans again, all in real time, as they were taping the show.

Laura Herzing: For all of our virtual audience applications, we have an actual operator there onsite who’s watching the game or the show and triggering what the audience should be doing in real time as well.

So we have built in animations for clapping, cheering, booing any kind of reaction that a real audience would have.

Billy: I went through all of the shots we talked about with Laura.

Alright, are you seeing this? The people on the floor were virtual, the backs of people’s heads in those jib shots in silhouette, those are virtual.

Laura Herzing: Yeah, this is virtual.

Billy: Basically all of the audience members, except for those closeups that we talked about, were added virtually with one other exception, which is, there is a wall of fans behind the judges. And that, apparently, is from a separate shoot with real people that was added in during post-production, after the fact. But essentially, everything else, all of the complicated stuff, the people on the floor, all of that is added virtually on the fly.

Adrianne: This has got to be more expensive than a real audience.

John: Well, you only do it once though. Right? They could continue to do this.

Adrianne: It scales.

John: Yeah.

Billy: Yeah, I mean, according to everyone that I talked to it, it definitely ends up being cheaper. But the answer to our question here is… yes, that most of the stuff on The Masked Singer is faked and virtual production and AR production specifically, is what’s making it so believable.

And yes, they will probably have real fans return when they can. But both Laura and Nickon told me that now that COVID has accelerated virtual production to where it is, that a lot of this will become the new normal.

Laura Herzing: Once people do come back to these games and these studio shows, I think that integrating augmented reality with practical set builds is just going to be kind of the way things are done.

That’s the way that broadcasting is heading… is just, it’s all going to be kind of mixed reality.

Nickon Mirsepassi: I’m pretty sure that this is going to stick. Like, it’s hard for not to stick, just because of how much money, you know, production studios are saving. And also how many things they’re able to do that just weren’t even physically possible before.

So I think as the technology gets more polished and it’s like, just become super streamlined, like plug and play type situation, it’ll stay. For sure. Like it’s not going anywhere. It’ll be around pretty much forever.

Regina: It’s really unfortunate that Trump didn’t have access to this for his inauguration.

Billy: Just add in some crowds.

Yeah, I  should give him Silver Spoon’s number, probably.

Regina: Underunderstood is Billy Disney, John Lagomarsino, Adrianne Jeffries, and me, Regina Dellea.

Billy: Special thanks this week to Shane from our discord, Shane does AR and VR stuff, and kindly helped me prep for the interviews so that I sounded smart when talking to our guests.

Adrianne: If you want to be cool like Shane, head over to patreon.com/underunderstood.

Nickon Mirsepassi: You’ll get access to our bonus podcast, Overunderstood. And you can hang out in our discord with the rest of the community. Maybe we’ll even play Among Us or, you know, Overwatch or whatever the kids are playing these days.

John: Thanks to everyone who supported us so far in whatever way you’re supporting us. We will be back with more Underunderstood next week.