The Motley Fool sent a team of analysts down to Austin, Texas for South by Southwest Interactive this week. And in this episode of Industry Focus: Consumer Goods, Vincent Shen is joined by Dylan Lewis to share insights from industry favorites: Netflix (NFLX -1.80%) and Walt Disney (DIS -0.89%).
For the leading streaming service and global entertainment giant, product testing is always happening behind the scenes. Find out how both companies leverage various tools at their disposal to put forward the “perfect” customer experience.
A full transcript follows the video.
This episode was recorded on March 14, 2017.
Vincent Shen: Welcome to Industry Focus, the podcast that dives into a different sector of the stock market every day. I am your host Vincent Shen and it's Tuesday, March 14th. We're continuing with our special coverage of South by Southwest interactive this week. The Motley Fool sent a team of analysts down to Austin, Texas to get a firsthand look at everything coming out of the conference. Calling in today from Austin is Dylan Lewis. How are you, Dylan?
Dylan Lewis: Doing OK. Taking a little break from checking things out at the convention center to give you a call on the hotel phone.
Shen: Sure. Our overarching topic for today's episode really comes down to product testing. We will be looking at this in the context of two very well known consumer companies, those being Netflix and Walt Disney. Starting with the leading streaming service, Dylan, what did Netflix share at their talk at South by Southwest this year?
Lewis: The session hosted by a Netflix rep was probably one of the ones I was most excited about when I was looking at the schedule. I went to a session called Design Like A Scientist: A/B Testing UX at Netflix. There's a lot of letters in there, some acronyms, but the idea here was, there was a designer from Netflix basically giving a chat about: How does Netflix make changes to the platform, keep their existing users happy, and then bring in new ones, as well? That's kind of the big-ticket here, and that's what they walked through.
Shen: Sure. I have heard previously about Netflix and their focus on data. I'm really interested to hear what they talked about at the conference. But the way I think about it sometimes, to put it in context for listeners, it's kind of an old Hollywood versus new Hollywood. If you think about the old model of how movies and TV shows are made, and how Netflix can apply a more modern, data-focused approach, it comes out very superior to the old way, where as recently as even ten years ago, you have industry insiders, people writing scripts, directors, agents, they're making these incredibly important decisions, often with millions of dollars in production budgets in the balance, and it's based on somewhat unquantifiable things like star power or intuition. So, a leading actor or director may have a strong track record at the box office, but even the biggest stars have their flops.
The closest things I've been able to think of, in terms of a more standardized approach to creating content, it might be sequels and franchises where you can count on the previous success and an established audience of an earlier film. You also have things like remakes that bank on nostalgia. And then there's trends, you often see trends at the box office, like, dystopian young adult books often get turned into movies. That was a trend recently. Then zombie films. They will become extremely popular, but eventually they peter out after oversaturating the market. But Netflix, as we'll get to, takes a very different approach, and they're really seeing what will work in terms of the content they'll produce, and also how they present that to their customers, right?
Lewis: Yeah. They're in kind of a unique position. They control the delivery of their content. They know exactly how much time people are spending watching certain series, which ones are incredibly popular, when people start to tune out, all that stuff. So, they have the analytics on that side with their platform. What this conversation really focused on was, how do they update the way that they deliver that content in a way that's super user friendly and keeps people satisfied? And really, this comes down to A/B testing. This is something that a lot of online businesses do. It's actually something we do a little bit at the Fool here and there. What it means is, every day, Navin, who is one of the guys who led this conversation, Navin Iyengar, his team of designers are testing out these little changes to Netflix's platform. And they're doing it on a very small subsection of customers. And basically, they're looking to see how those users, that small section of users, interacts with the changes compared to a control group that's just getting the standard experience. And very often, what they're doing here is trying to drive that improving customer satisfaction, like I talked about. They'll usually tie that to some core metric. Maybe it's log-ins per month, maybe streaming hours or customer retention, something like that. Ultimately, if what they applied to that very small subsection, the test, beats the control, the standard experience, they'll roll out that new design element, or that new functionality to the larger whole Netflix userbase.
Shen: OK. And I'm going to ask you, did they give you any examples of how they use this data to change things that people who are on Netflix might be familiar with, that they have seen or noticed? One thing that really fascinated me when I was doing my research on this, the level of detail they have for every subscriber, they know exactly what you were watching to a shocking level of detail. Time and day of the week, check. Where you're located, check. Device you're watching on, whether it's a tablet, a TV, PC, check. What you search for and how you rate content, check. How you browse the library, even, or your scroll-through, rewind, pause, all of those things are included. It's interesting, the idea that this is kind of real-time feedback they receive, because they can recommend content to a user, and after that subscriber watches that content, they'll often rate it, and voila, you know right then and there if their algorithms are accurate for that specific subscriber, because in the end, obviously, only the subscribers who really discover more interesting content that they like on Netflix will want to stay with the service. This will play into loyalty and retention, if they ever want to raise their rates again, for example.
An article I found in Wired delves into this as well, how the company's really delivering it even in the imagery it uses for TV shows and movies in its Netflix library, and how they'll do a very detailed analysis of the colors and the cover art that they use, and how that can tell the company about user preferences and behavior. But, did the speakers at the conference share details of how that really played through in terms of A/B testing?
Lewis: Yeah, they used a couple high-profile examples that I think a lot of regular users of Netflix might see and actually really like. I think one of the main ones, one of the most recognizable versions of this testing format being eventually rolled out to the larger audience, was Netflix's post-play function. This is basically the mechanism that gets you the next episode rolling as soon as you finish a show. That is what leads so many viewers into that binge-watch cycle. "Oh, the next episode is going to start in 15 seconds." That is something that they tested into because it helps keep people on the platform longer, and it boosted retention, it boosted satisfaction, people were getting more out of the service, they were watching more. I think that's a staple of the online offering right now.
Shen: OK. Yeah, I noticed that. I don't remember exactly when that rolled in, but now, for me, especially as I've been discovering some new series, I've also been trying to push my wife to watch stuff like The OA, and I have to say it is very conducive -- as you mentioned, you play that first episode, you like it, and once it automatically goes through, you don't have to touch the remote on a Roku specifically at all, it can definitely push you into binging those three or four more episodes, and before you know it, Netflix is a bigger and bigger part of your entertainment experience and options.
Lewis: Yeah. If you're looking for how the internal team thinks about testing, and what it looks like on the design side, and the considerations they have, the speaker did walk through their framework a little bit, and talked about some of the keys they focused on. One of the big things was this idea of, test before you invest. Do you remember a couple years ago, they totally redesigned the Netflix site? It went from being an older-tech look to a very fresh, current modern look that we see today. But, when you're talking about overhauling the design of an entire site, that's pretty daunting and it involves a ton of design work. Before they went wholesale with it, what they decided to do was test a redesign mainpage. That's it. All the other pages were in the old skin. Once they saw that the newly designed page was better, and was performing better for the metric they were looking at, they then applied it to all of the subpages, all the show-specific things, all the subcategories, all that type of stuff. So, you see, even within the testing environment, they can get very granular before they decide to apply it broadly to the whole platform.
Shen: OK. What else do they look for, in terms of guiding principles for them in this testing process?
Lewis: We talked about the amount of data they can collect on their users, and I think that the notion of the metric being their compass is really what underlies all of the work that they do. They might be looking at a problem like, "How do we get people to watch more?" And all the design elements are going to be aimed at a specific metric, it's going to be episode starts, or hours streamed, or something like that. But the idea of having a very specific number that they're focused on, that keeps the design work organized, it keeps it fairly unified, and generally aligned toward that goal. If you want to talk about another example of some of this work being successful and later being rolled out broadly on the platform, any time you're looking at a title on Netflix, and you put your cursor over it, there's a play button when you hover over it. That's something they tested into. Ultimately, they want people to be watching a ton on the platform, and the less information they show -- they show interesting graphics, and maybe the rating, and a title, and a very brief description, but people want to get to playing the episodes. And they realize that very early on in a control group, and then rolled it out to the rest of the platform. So, that's another example.
Shen: OK. Something that you had mentioned to me before the show that I thought was really interesting was some of the things that Netflix has spoken of of how people say they prefer certain things, and how the company discovers that, when they observe their actions -- because they have this very granular data, and observing how people use the service -- those things don't always match up, and how the company needs to focus on what their customers actually do, rather than what they say. How did that play out?
Lewis: Yeah, I think that's probably one of the funniest ones. We think, as consumers, we know what we want. That isn't always the case, though. So, they asked this survey question and said, "What's one thing that you would like to know more about before signing up for Netflix?" And 46% of respondents said they wanted to know all the movies and TV shows that are available. You'll notice, if you go to the sign-on page, they don't make a full catalogue available. You can't find that in some of the on-boarding they have for new users. And why? That's something the company tested into to see. It turns out, when you're getting to these sign up pages for Netflix, what really drives people to take the plunge and give it a shot is the free trial. When they offered up people a look at the catalogue, even a small part of it, they got so bogged down in hunting for specific titles, what they found worked best in terms of bringing people on to the system and keeping them was getting them in, and having them sign up and actually use the service. And so, that's a case where, there might have been a little bit of a disconnect between what we thought we wanted and how to find that out, and how we actually acted and were interested in as consumers.
Shen: Wow. I can't say that I'm too surprised. I have personally experienced sitting in a group with my friends, trying to figure out what we're going to watch together, we're on Netflix. And I can see, with that specific example, how you can get bogged down going through the different choices, and I can see how the company wants to get people into that content viewing as quickly as possible. But for investors, turning the roles around a little bit for the investors who are listening, this level of granularity and insight into the customer base, it really allows the company to do things like pay $300 million a year for exclusive rights to Disney content, for example, or spend $4.5 million per episode for House of Cards, or $9 million per episode for their original series Sense8 as well. And they have a lot more confidence here to make these kinds of investments, and ultimately have more confidence that they will pay off. They can take all of this information they have at their disposal, and you combine that with the estimates that Netflix is spending, content budget for 2017 is, I think, about $6 billion. And you realize why the company is able to grow so quickly. Its subscriber count was up 25% in 2016, hit 94 million total as of their most recent report. Any other takeaways here, Dylan, for you on this company or their process, or what you heard at the conference, before we move on?
Lewis: One thing I do think was pretty interesting, and it underscored the importance of this testing approach, was the speaker said, "What we found over the years of testing is that our intuition is generally wrong." And so, these are guys that are professional designers, and people who work in the business of online media, and even they don't have the greatest finger on the pulse of what people want, and the best way to deliver it to them. That is why they have this very rigorous testing environment setup. You may have this hunch about how people behave, but until you have the data to back it up -- or, you might see that you made one assumption that was totally unfounded, and it turns out that's not a priority for users, but, it's nice for them to have these numbers, because otherwise, they would be flying blind.
Shen: Sure. Now, we turn our attention to Walt Disney. From what you described to me before the show, Dylan, it seems like a lot of the testing the company discussed at the conference here was around experiences and their theme parks, and how they can push the envelope with new technology, potentially. Can you give us some more detail?
Lewis: Yeah, the session that I attended that was relevant to Disney was called Using AI and Machine Learning to Extend the Disney Magic. We had a couple different folks from Disney Corporate there. And as the title implies, this was really a talk all about what Disney is doing in the space of artificial intelligence and machine learning. This is obviously a nascent field, and one that I think we're probably not going to know that we're seeing for quite some time. But one thing that I was really struck by was the emphasis on consistency with everything Disney does. They think of their characters, their stories, all these different ways that people interact as touch points they have with the emotions and the characters that bring out those emotions, and how important it is for all of them to be consistent, in line with the character and the Disney brand. So, they're investing heavily on the tech side to see what they can do, and how they might be able to continue to bring these characters to life, but I think they're doing it very carefully. I think a lot of this is going to be behind closed doors for a very long time.
Shen: Sure. I have to ask you, you mentioned artificial intelligence and machine learning, I'm really curious, for me, in the context of a theme park, I'm thinking, maybe the typical person in a costume representing famous Disney characters like Mickey Mouse get replaced, potentially, in the future, by a robot in the costume instead. Did they give you any examples, or showcase any of these things they're working on?
Lewis: They talked a little bit about what different ways AI might be Incorporated by Disney in the future. They talked about chat bots, or some kind of kids chat apps, maybe some quiz apps, things like that. They talked about the idea of having Disney characters at the park powered by AI or machine learning as well. But one thing that I think you have to keep in mind here is that emphasis on consistency, and the experience always being so positive and so seamless. There are a lot of major challenges with AI and machine learning for as far as we've come. In the R&D lab, when they are working on this, and they have some very small groups of people interacting with whatever they're creating, they have total control. In the theme park, they have slightly less control, but I think they're still dictating a decent amount of the interactions and the experiences that their technology will run into. If you're thinking about the consumer products side, and maybe, eventually, being able to buy something that is AI-powered off of supermarket shelves or something like that, this is a Disney-branded product, I think we're way away from that, because it's a matter of control, and understanding the environment that the technology is going to be used in.
Shen: Sure. I'm not surprised that they're taking a more measured pace and approach to this, because a lot of different things could happen in a park with a kid, whereas if you have somebody right now who's in the custom, they can react dynamically as necessary. But, if you think now about certain toy companies, they're kind of entering this with the AI. You see Barbie dolls now that kids can talk to and have a conversation with, and the Barbie will have some pre-programmed responses, but it's learning, and connected to a cloud-like infrastructure that allows them to be a little bit more dynamic when interacting with the kids that are playing with the dolls. But, what else were they able to share, in terms of some developments in the space?
Lewis: Well, they showed a couple different R&D projects that they're working on. One was an AI machine learning-fueled version of Pascal from the movie Tangled, and another one was this project they called Jake. Unfortunately, the videos for these aren't available anywhere. But I think really, what we saw with what they were willing to show us was Disney studying how people interact with these renderings of the characters, and what they do with them, and if they're able to still build that emotional connection, because I think that's what's so important for them as a company -- maintaining that, and not having any of that lost, if they ever decide to go over to the machine side.
Shen: OK. I'm really surprised, I was looking into other ways they've adopted this kind of technology in their theme parks, across the company. In terms of the history of the company, it was really interesting to find that as early as the 1960s, they adopted animatronics at their Enchanted Tiki Room attraction, which has singing and dancing birds. From what I was reading, in terms of news at the time, this bird that would sit there and beckon park guests into the attraction was so popular, it was so groundbreaking at the time, there would be crowds that would gather outside to see this happen. Think about it, in the sixties, there was nothing like that before. It reminds me of other things that the company has done, pushing the envelope a little bit, or at least, be willing to invest to push the envelope. Think about Pixar leading Hollywood at the time with computer animated films, Disney was willing to pay over $7 billion to acquire the expertise and computer animation, which, obviously, their films are focused on now, and intellectual property there. And also, they have things, I haven't been to a park in some time, but my brother told me recently in his trip that they have their MagicBands now. So, available to guests, this is RFID technology that really makes the guest's experience very seamless. You can enter parks, enter hotel rooms, with this band. Just some examples of how the company is definitely not inexperienced in terms of pushing technology and finding ways to innovate and make experiences at the park, and with their products, that much more rich and fun to play with.
Lewis: Yeah, one of the panelists said, "We're not a tech company, but everything we do seems to be under piles and piles of tech as we continue to move along as a company." So, I think users and consumers that go to the park and things like that, they're going to continue to see the experience that they currently have at the park for quite some time. And they might eventually be interacting with AI and machine learning at the park, and not even realize it. I think if Disney had its druthers, that's exactly what would happen. They are so much about capturing the magic and having you not see the strings that are pulling everything in the background, and cultivating this really incredible experience. I think once the technology gets there, you're definitely going to be interacting with it quite a bit more at the theme parks in particular.
Shen: OK. Very cool. That is a wrap on our discussion for today. Just as a preview, is there anything else coming up at South by Southwest this week that you'll be covering on the other shows, or that you're really excited to see?
Lewis: Simon Erickson is going to give me a little break, and he's going to talk on the Healthcare show on Wednesday, and I believe he's also going to talk with Sean O'Reilly on the Energy and Industrials show on Thursday. Some very fun stuff lined up there. Simon is going to be talking health and med-tech with Kristine tomorrow, and I'm hoping to land a couple interviews, we'll see what happens. Friday's episode is still in the air. I have an interview that I booked and locked down with a self-driving car expert. So, we might wind up airing that and double-dipping on self-driving car technology. But if anything else interesting comes up, there might be a change of plans. Either way, we'll publish the episode as a bonus episode, and make sure that all of our listeners are getting all of the content that we put out.
Shen: Sounds good. Thank you, Dylan!
Lewis: It's a pleasure, Vince.
Shen: That wraps up our discussion. You can reach out to us and the rest of the Industry Focus crew via Twitter @MFIndustryFocus, or shoot an email to [email protected]. Don't forget to check out podcasts.fool.com for more Foolish content.
People on the program may own companies discussed on the show, and The Motley Fool may have formal recommendations for or against stocks mentioned, so don't buy or sell anything based solely on what you hear during the program. Thanks for listening and Fool on!
