Search form

One-of-a-Kind AI-Assisted Rotoscope Helps Resurrect Life-Loving Legacy of Jackie Shane

Banger Film’s documentary, ‘Any Other Way: The Jackie Shane Story,’ captures the highs and lows of the pioneering transgender soul and R&B artist’s life through an innovative, groundbreaking mix of live-action and vividly animated sequences.

Jackie Shane’s film, a documentary about a soul and rhythm and blues singer, begins as one might expect – with a deep, warm, and heavenly hum from Jackie herself. The singer drawls the notes from her throat in time with rhythmic claps and a distant tambourine. The dark screen then lights up with a sea of watercolor, Jackie’s house suddenly transformed into a dream-like setting that exists somewhere in the back of more than one person’s distant memory. The singer’s silhouette crosses the window, and the viewer is suddenly inside this watercolor house, the animated blushes of pinks and yellows in constant movement, as Jackie’s hand – bejeweled in rings – reaches for an old record. A track is set on the record player, the humming ceases, and the sound of crowds and clinking glasses takes its place.

Then, this experimental rotoscoped animation really steps into the spotlight… literally. Jackie swaggers to the microphone on stage and viewers are twirled around by her music and the camera work that dizzies up an already busy screen. It’s exciting, it’s intoxicating, and it’s likely exactly how it felt to be at one of Jackie’s many energized performances. 

“I've always been a fan of animation that's constantly in motion because every frame is different,” says Luca Tarantini, co-director of animation with Jared Raab on Any Other Way: The Jackie Shane Story, now on the film festival circuit. “There is a balance between taking that too far and taking it to the right point where it doesn't feel abrasive on the eyes, but it still feels stimulating. And of course, the character of Jackie, she's got a lot of energy. And she's a hard person to pin down in a lot of ways. I think the animation portrays that.”

Titled after Jackie’s famous 1962 single of the same name, Banger Film’s Any Other Way, an animation and live-action hybrid directed by Michael Mabbot and Lucah Rosenberg-Lee, journeys through the life of American soul and rhythm and blues singer Jackie Shane, most prominent in Toronto, Canada’s local music scene in the 1960s. Jackie is considered a pioneering transgender performer and the film catalogues the highs and lows of the singer’s time in and out of the spotlight, including her early days, traveling with her family and the circus, before becoming a nationally-recognized star and creating her live album legacy.

The film, executive produced by The Umbrella Academy’s Elliot Page, plays more solemn notes in the last half, as it shares the lonely years Jackie spent in isolation, suffering from the exhaustion of persecution for being transgender. But Jackie’s eventual passing in 2019 is not where the film ends. In true Jackie form, the story wraps with a long-awaited family reunion and a block party to celebrate the life of this singer and the hope she inspired across the U.S. and Canada, and not just in the transgender community.  

Jackie was a multi-faceted musician whose love for people was unconditional and whose influence was unrestricted. It only made sense that the animated portions of the film, which offer a look into parts of Jackie’s life where there are no photos or videos, would be just as vibrant and full of light. 

It also seems poetic that Jackie’s single, “Any Other Way” was first released in Toronto, and the animation for the documentary was largely influenced by the National Film Board of Canada (NFB), which served as the film’s theatrical distributor at SXSW. 

“We obsessively watched the early NFB animated projects from the 50s, through the early 70s, where it was all hand-painted and hand-drawn,” shares Raab. “And there was a period where traditional animation got very experimental at the NFB, especially in the 60s. The aesthetic was really loose and painterly. The shackles of perfection came off and, in some ways, watching that stuff made us fall in love with the idea of creating something that would be almost ethereal and hard to pin down, which was perfect.”

Tarantini adds, “Honestly, I don’t think we even had animation references that weren’t NFB. They really nailed down this beautiful imperfection that came from these independent, maybe even self-taught, filmmakers. And that's kind of the space where we come from.”

The rotoscope animation follows the color trends of media at the time, sticking with black and white during shots from the 50s, moving to sepia for the 60s, and richly saturated color for the 70s. The style is ever moving and looks like a cross between watercolor and splotches of acrylic. 

Like with most rotoscoping methods, the team filmed in live-action first, using trans actors Makayla Walker and Sandra Caldwell to play a young and old Jackie. Tarantini and Raab had planned to animate over that live footage. But their plans changed. 

“When Luca first came on board the project in 2021, he was looking into different modern rotoscoping techniques based on new technology that was just coming out, which could take a hand-painted frame, and a number of keyframes on a moving image, and then track the animation overtop of it,” explains Raab. “I came on board at that point, more to help plan out the initial sequences that we were going to shoot that were then going to be rotoscoped. I wasn't going to be as involved in the actual animating. But as the project evolved, and the filmmakers came back to us, needing and desiring more and more animation, that meant the techniques Luca was initially researching weren’t going to work. So, Luca went back to the drawing board. And we started experimenting a lot with new AI rotoscoping techniques, which could take an image and, through Stable Diffusion, give it a different look.”

The original plan with Any Other Way was that Jackie's voice would only be presented as audio, with maybe the visual of a bouncing wavelength or overtop of one of the few pictures the family had uncovered. But the directors and producers, rightfully, felt this didn’t make Jackie feel as alive as she should be in the documentary. Raab says Mabbott and Rosenberg-Lee were adamant that viewers had to feel Jackie's presence in the film. So, phone conversations were expanded, as was the amount of music played. This meant the role of the animation changed drastically. A method that was originally needed for a minimal amount of runtime was now required for almost a third of the film, and Raab and Tarantini were the production’s only two animators other than VFX artist Patrick Cederberg.

They had their work cut out for them, even with AI tools, which weren’t a walk in the park and were anything but an automatic fix. “At the time I started using Stable Diffusion, it was not used as a rotoscoping tool,” notes Raab. “It was used to generate cartoon characters and cyberpunk cityscapes, and stuff like that. I saw the potential in it, for taking a clean image of an actor and ending up with something that looked like it was painted by a person, but still maintain the elements of the original image. But it wasn’t just one keyframe on which this needed to be done. It was for about 100,000 frames.”

And AI, despite its name implying “intelligence,” wasn’t going to do this on its own. A clean image couldn’t just be shoved into the process. The software needed the right parameters, the right model, and the right prompt.

“We found a very special model for Stable Diffusion that had not been trained on any images of people or animals,” shares Tarantini. “It was exclusively for abstract art.”

Raab adds, “We had a big problem where the technology is built to try to recognize objects and understand what they are and then try to represent them and we didn't want that. We wanted it to look at the tones, the colors, the lines, and turn that into a different kind of rendering. But we didn't want it to add anything.”

The challenge became how to take this technology, which was designed to recreate cleaner representations of images, and use it instead to abstract the Jackie Shane source material, which was HD footage shot on a soundstage. The animators started by shooting their clear footage. They then removed backgrounds with greenscreen and had Tarantini create entirely new 3D environments in Cinema 4D. Using the camera tracking system, they also created entirely animated sequences as animatics, treated it, then started the process of bringing these shots into the hand-painted animation look they were aiming for in the final film. They also used fish-eye lenses at different angles to up the drama and emotion during the scenes of Jackie talking on the phone in her home, narrating her life story. 

“There was a very important pre-treatment process, where we did things like smart blur, selective blur, even adding a bit of motion blur in places,” notes Tarantini. “We had to get the tones right, crushing down the image so that the darks have no detail where you don't want them. Any detail that's there is going to show later, and you can’t get rid of it later. The pre-process treatment for color and look was really important.”

Raab goes so far as to describe AI as being “one of the worst employees you could possibly have.”

“Because you can’t direct it,” explains Raab. “People think of these new AI tools and software as taking some of the decision-making out of the process as if it’s a one-click solution. We found that, in fact, it was the opposite. We had to be much more specific about what we were after before we even got to the level of using the AI tools. We had to narrow down the information when it came to the textures, the colors, the style, the flow, the camerawork, and the animation. All of that had to be basically perfect before we even started to engage with AI.”

There were also moments that involved the animators tricking the AI software into doing what they wanted. For this, Tarantini taught himself how to code in Python using Chat GPT. 

“You'd find this beautiful look, or very hand-painted frame, which would look great on one frame, but as soon as you tried to do that over multiple frames, we found that things like a brushstroke on a face would stick,” said Raab. “And it would move in space with Jackie, almost this like nightmarish track-painted face. So, we had to trick the software into thinking every frame was new so what it did for one wouldn’t carry over into the other. One of those ways was to take a sequence of images – one upright, the next flipped, the other flopped – so the AI couldn’t follow and trace the same objects.”

Tarantini adds, “We called it the Flip-Flop script.”

But each frame still had to remain consisted with the others. The pre-development trails seemed never-ending, but Raab and Tarantini never doubted their resolve to use AI as a collaborator. 

“If there wasn’t just two or three of us making roughly 40 minutes of finished animation in only a few months, then we might have just decided to do it all ourselves.”

“We would have needed three years,” Tarantini interjects. “The reason I always saw this as possible is because I'm a process worker. I like to develop a process that will work for an infinite amount of content in the future. Once you tune a machine like this just right, you can make way more art than you ever could alone, because you've designed a system for it. And so, whenever we were having problems, we both knew it was just a matter of tweaking the right things. I never lost faith in it. Eventually, inevitably, we would get it.”

Tarantini was right because, by the end of development, he and Raab were creating huge five-minute scenes in only a couple of days. In total, pre-development and crafting their well-tuned AI machine took Raab and Tarantini about a year. After that, it took them roughly six months to finish their animation. 

“AI didn’t solve any creative problems for us,” says Raab. “All of that was human decision-making. AI just sped up the process once we spent a year teaching it again and again what we wanted it to do.”

The war on AI is industry-wide, with actors rightfully afraid of their likenesses being stolen, artists fighting for their jobs, and even writers having to prove they still serve a purpose outside of just making prompts to feed into a machine that will then write a desired episode or treatment in moments. And a lot of the chaos is due to no one truly understanding what AI is, or taking the time to learn how AI can be used as a tool to assist, rather than an overlord to dictate. Because, let’s be honest, if Raab and Tarantini have proved anything, it’s that AI isn’t at all helpful if it’s given control. 

“This happens every time there's a new technology that comes up that's going to revolutionize and completely change the way that art is made,” says Raab. “You could see it in music with the introduction of the synthesizer, which created pure panic throughout the industry. The same was true during the transition from film to digital. Then there was the integration of computers into the animation workflow. Time and time again, the invention of any new tool is going to completely change the process and change the way that the labor is done. And maybe shift some of the backbreaking tedious work to more creative problem-solving work. We’re learning to realize that these advancements are just tools. They aren’t replacements for people.”

And these tools can be very helpful to artists if they know how to use the tools to their own advantages. 

“Jared and I aren’t afraid of technology,” says Tarantini. “We love it. But we’re just as skeptical of it. And that’s good, to have people who are deeply skeptical and allergic to inhuman AI ideas to be working with the technology, because those are the people that are going to do something really powerful and cool.”

He continues, “Tension is good for art. And I think that's why we enjoyed the process of working with the AI and finding ways to break the tools a little bit to get a better look. That's extremely important.”

And anyone can try these tools out for themselves, as every software Raab and Tarantini used on Any Other Way’s animation sequences is open source and available to the public to download on their computers. 

“If anyone wants to work on films like this, we're hiring for more films in the future,” says Tarantini. “And I'm so willing to just talk with people over email or phone and just explain all the things I said here and more. If anyone is interested, they can reach out to us, and we’ll teach them about what we did on Any Other Way.”

To contact Tarantini, reach out on Instagram at @lucatarantini or @createdbyaok. 

More screenings for Any Other Way are being scheduled and will be announced on the film’s website and social, links below:

https://collection.nfb.ca/film/any-other-way

https://www.instagram.com/thejackieshanestory/

Victoria Davis's picture

Victoria Davis is a full-time, freelance journalist and part-time Otaku with an affinity for all things anime. She's reported on numerous stories from activist news to entertainment. Find more about her work at victoriadavisdepiction.com.