Blog Series, Article 3: Text Mining Movie Scripts
This is the third of four articles in the series Text Mining Movie Scripts, released weekly. If you missed the first post, you can find them here:
- Introduction to state of the art script analysis (released: July 18, 2019)
- Turning Stories into Data (released: July 25, 2019)
Creating Meaningful Outputs from the Data
Visualization is supposedly one of the most important skills of a Data Scientist. This isn’t only so that we can create useful reports for business stakeholders to convey what is it that we’re actually doing, but there is an equally important role of data viz – it serves to summarize our analyses and outputs and provides another layer of checks on top of it. If the output doesn’t make sense, it’s obvious something went wrong. Let’s check it out.
Our outputs can be classified into three main categories:
- Structured table of script data
- A dashboard visualizing individual scripts
- Character archetype segments
We already took a look at the data in the previous blog post, so let’s go through the remaining deliverables.
Dashboard for Creative Team
Let’s take a look at the dashboard and its contents. I won’t go over all of the charts, just the main ones, so you get the idea. We’ve used Plotly to build these dashboards.
Data about the movies were fetched using the TMDB API. While this isn’t a particularly exciting task, this way we obtain vital information about the data, such as budget, actors, directors and a short plot summary.
Character importance/overall sentiment
Since movie scripts contain a lot of characters, our dashboards only display characters with at least 5% of the dialogue out of all dialogue. This chart shows the number of lines broken down by sentiment for each of such characters:
The plotted data makes a lot of sense. Finn has been a bit of a coward and complainer, while Han was an undeniable cynic. Rey was actually more positive than negative, which makes sense given she starts out as a scavenger dealing with many unpleasant characters and ends up a Jedi prodigy. We’ve tested various approaches to extracting sentiment from the dialogue, mainly comparing the FastText model with Open AI’s Sentiment Neuron. We ended up going with the latter as it worked reasonably well, despite being a bit slow.
Relationships between characters
In addition to overall character sentiment, we’ve analyzed the sentiment of dialogue between characters. Sentiment Neuron doesn’t give us a 0/1 classification, but rather a continuous variable with lower sentiment scores meaning more negative sentiment relative to the rest of the text. We see Rey having a very good relationship with Finn which makes sense given that they fight together and grow together as characters. What is a bit surprising is that out of these three relationships, this is the ‘coldest one’ since they’ve grown quite attached. Keep in mind that this probably occurs because the topics these two talk about are more often unpleasant than not.
You might have noticed that the chart has an on hover effect which displays common topics between characters. I want to show you a very interesting example of this, but first, let’s talk about how we extracted the common topics and keywords:
- Extract all the dialogue for characters in a movie.
- Perform topic modeling on dialogues.
- Note: It didn’t work very well. The most prominent topic words turned out to be character names since in dialogue they tend to address each other by name quite often, so we got a lot of Rey’s, Finn’s, nothing meaningful other than who they are speaking to.
- We removed proper nouns via entity recognition (pulling character names from the TMDB API and finding similar words).
- This got us further, but since scripts are action-oriented, we got a lot of verbs such as go, see, and run. It’s good to know that our characters like to keep in shape, but not useful for extracting additional information.
- Finally, we removed all verbs and proper nouns to obtain a reasonable output.
- We then picked only the keywords from overlapping topics between characters that were not present in other topics.
This shows that, in addition to, the usual text data standardization and cleaning (such as stopwords, lemmatization/stemming, removing special characters, etc.) some text data need additional thought in order to extract meaningful information. Was the output meaningful? Let’s cherry-pick a very interesting result – the keywords for the relationship between Han Solo and Kylo Ren:
I hate spoilers, but I would say that this is a pretty good result. Kylo is Han’s kid and the only time they meet in the movie is in a corridor. Han asked Kylo to stay in the light, and Kylo thanks Han for . . . we’re not going to spoil the scene.
Character Sentiment Timelines
As mentioned in the initial design, we wanted to extract the sentiment of characters throughout the movie. The chart below looks at this exact notion:
Since all of our characters on the chart are the good guys, we can see a very strong trend of initial sentiment growth when meeting each other followed by stagnation and even dips when the plot thickens (and Han’s sentiment plummeting in the previously mentioned scene with Kylo Ren). Sentiment for Finn and Rey reaches a low point when losing in the fight against Kylo. As the movie ends and we move towards a happy ending, sentiment increases for all characters. While this looks to make less sense for Han, it’s mostly due to him being mentioned in the follow-up scenes.
While there’s more inside the dashboard (mostly on scenes) we’re going to cut the dashboard overview short and move to character archetypes.
Our search for character archetypes involved collecting all dialogue for individual characters and running it through Receptivity personality classification API which is designed to analyze people in a workplace setting, not movie characters. Via this API, we get a ranking for each character on 60 different personality dimensions such as cooperative, aggressive, liberal, melancholic, sexual, etc. The good news is that these dimensions are uncorrelated, which is great for clustering. The bad news is that these dimensions are uncorrelated, which isn’t great for visualization (as projecting them onto a 2D space would result in a considerable loss of information).
We have to be smart about visualizations if we want to look at even one character in regard to these dimensions. One trick to reduce the number of dimensions is to come to terms with the fact that not all of them are distinguishing characteristics. For example, when it comes to aggressiveness, we have to keep in mind that the Receptivity API was built for HR purposes and I bet people in the office are much less aggressive, or action-oriented if you will, compared to movie characters that are usually confronted with some kind of conflict. This means some character dimensions will be high/low for all movie characters and we can throw them away as non-distinguishing. One such look we can take is Han Solo from Star Wars: Force Awakens as we’re keeping the same character in multiple movies separate to that movie. We will be using a bullet chart, designed to visualize various goals across various dimensions. We’ll repurpose this chart for benchmarking purposes.
We have picked only the traits that make the Han Solo cluster in stand out. The black bar is Han Solo’s rating on these traits. The further the black bar reaches, the stronger the performance. The red marker represents the cluster centroid values. The green bar represents all characters across the whole movie dataset distinguishing the individual quartiles by the shade of green (beginning with lower quartile, median, top quartile, and maximum). This data reveals Han Solo is more cooperative than most movie characters but slightly less than his cluster average.
An interesting litmus test is looking at the cluster affiliation of characters across movies. Let’s not give those Star Wars characters a rest and take a look at the characters from Star Wars: Force Awakens and Star Wars: Imperium Strikes Back:
- Han Solo is in the same cluster in both movies
- Leia is in the same cluster as C3PO, Finn, and Rey
- Finn and Rey are in the same cluster
- Kylo Ren is in the same cluster as Darth Vader (being his admirer)
- Poe is in the same cluster as Kylo Ren and Darth Vader, while we know no machine learning approach is perfect, this may be due to a number of reasons, most notably to the traits shared by both characters – driven, duty-oriented, etc.
There are so many interesting things that could be extracted from the data we had available at this point. However, because it is a research project, we had to temporarily shift our focus. As the H.P. Lovecraft so eloquently stated, “That is not dead which can eternal lie. And with strange even death may die.”
Curious on how/when script analysis resurfaced? Read the next, and last, iteration of this blog series next week.