Tyrell Tests

Tyrell: Demonstrate it. I want to see it work.
Deckard: Where’s the subject?
Tyrell: I want to see it work on a person. I want to see a negative before I provide you with a positive.
Deckard: What’s that going to prove?

Emphasis mine.

(I don’t think this post contains spoilers for Blade Runner, but you’ve been warned. Also, the movie is almost 40 years old.)

Blade Runner follows a detective named Deckard searching for a group of renegade synthetic humans called replicants in the far-flung future dystopia of 2019. Replicants, created by the extremely wealthy and successful Tyrell, CEO of the massive Tyrell corporation, are indistinguishable at a glance from real humans. When Tyrell meets Deckard for the first time, he asks Deckard to demonstrate the testing device that Deckard uses when interrogating a person detect whether they are replicant or human.

In their exchange, Tyrell wants to apply the test where he knows the answer is negative, and Deckard does not think this is useful. But in general, whenever performing an experiment, testing a new method, or even debugging your code, it’s a great idea to see how the machinery works in the null case. ¹

So, I like to call this practice Tyrell Tests. ²

Cats, Dogs, Mass Hysteria

A student comes to me with their new cat-dog machine learning classifier: “Look, it’s correct 98% of the time!”

“Fantastic, send it some random data. What does it do?”

“Hmm, correct 99% of the time.”

“Interesting… What exactly do you mean by ‘correct’?”

“…”

Time passes. The student returns, “It turns out the training data is 98% cats, so the classifier always says ‘cat’. Then, when I shuffled the image labels—randomly assigning labels to images ³—almost every image was still a cat.

Tyrell Tests are especially handy when using machine learning, where it’s devastating to fail to notice data problems such as class imbalance. But Tyrell Tests can help identify issues in the data, the method or model, or in your code, in all sorts of domains:

• In network science, you have a new inference method for learning statistically significant structures in the network. Send it a random graph and see what it finds.

• In time series analysis, you have a method to forecast future values. Send it a randomized time series and see what it finds.

• In Bayesian inference, send some random data through your likelihood and see how (or if) the posterior changed.

Tyrell Tests are a cutesy name for a simple and rather obvious idea. Still, this cutesy name brings me endless joy!

But the moral of the story is to stress your methods and interrogate your results. It’s easy to be fooled. Tyrell Tests, using known negatives for confirmation, are one tool worth keeping at the ready.