Janelle Shane+ Your Authors @JanelleCShane Research Scientist in optics. Plays with neural networks. Avid reader, writer, and player of Irish flute. she/her. wandering.shop/@janellecshane Jul. 25, 2019 1 min read + Your Authors

I used @runwayml to reconstruct this scene from The Last Jedi by having DeepLab recognize all the objects in each frame, and then having SPADE try to paint the original frame from the objects.

More on the reconstruction process here:  https://aiweirdness.com/post/186539822077/the-world-through-the-eyes-of-a-neural-net 
This is the first step, where DeepLab segments an image into objects. "Lightsaber" is not a category it has, so it improvises.

Next, SPADE has to paint a picture just from DeepLab's labeled image. It sees person with a baseball bat, so it looks a lot like it tried to paint Luke in a baseball jersey.

Here's another scene from TLJ as segmented by DeepLab (which sees people with teddy bears hiding behind the rock) and SPADE (which dutifully has to draw the teddy bears)

I like this sequence. DeepLab didn't detect the lightsabers & decided the ground was snow. (And also a surfboard). Trying to recover the original image, SPADE looked at all the snow & decided to put Luke and Kylo in snowsuits.

I'm really excited to see what these will look like as segmentation & reconstruction get better. Could one segment a scene from Pride and Prejudice and replace all the "human" labels with "bear"? Or "refrigerator"?

Here's a cool variation where @jonathanfly gives SPADE not a DeepLab segmentation but frames from cartoons or 8-bit video games.

flawless reconstruction by our robot overlords

i love this!

You can follow @JanelleCShane.


Tip: mention @threader_app on a Twitter thread with the keyword “compile” to get a link to it.

Enjoy Threader? Sign up.

Since you’re here...

... we’re asking visitors like you to make a contribution to support this independent project. In these uncertain times, access to information is vital. Threader gets 1,000,000+ visits a month and our iOS Twitter client was featured as an App of the Day by Apple. Your financial support will help two developers to keep working on this app. Everyone’s contribution, big or small, is so valuable. Support Threader by becoming premium or by donating on PayPal. Thank you.

Follow Threader