From Stop-Motion to Surgical Precision: How Cross-Industry Collaboration Drives Medical 3D Printing Innovation

While Hollywood often portrays lone geniuses toiling in isolation to achieve breakthroughs, the truth is that truly transformative projects often emerge from unexpected collaborations. The world of advanced manufacturing, particularly 3D printing, thrives on such interdisciplinary synergy. A compelling testament to this fact is the partnership forged between two keynote speakers at AMUG 2023: Nicholas Jacobson, a distinguished professor at the University of Colorado’s Anschutz Medical Campus, and Rob Ducey, a technical supervisor at the acclaimed animation studio, Laika Studios. Their serendipitous meeting at AMUG 2019 sparked a groundbreaking project that harnesses a specialized 3D modeling and printing process, known as bitmap printing, to revolutionize medical applications for pediatric epilepsy, cardiology, and cleft palates. We had the privilege of sitting down with these innovators to explore their unique journey and the exciting potential of their work.

3DN: Could you please introduce yourselves and your backgrounds?

Nick: Hello, I’m Nick Jacobson, and my involvement with 3D printing spans nearly two decades, starting in 2004. My foundational training is in architecture, where I first encountered 3D printing during my architecture school days. We utilized binder jetting technology with refined corn starch on a Z-corp printer to create intricate landscape and architectural models. This period coincided with the nascent stages of the computational design boom in architecture, where students were just beginning to explore coding and programming. This emerging digital frontier fueled our fascination with translating complex computational ideas into tangible, unusual forms through digital fabrication.

This passion for sophisticated form generation became a consistent thread throughout my architectural career, affording me the opportunity to work alongside renowned architects such as Renzo Piano, Zaha Hadid, and Harry Teague. After several years dedicated to architectural practice, I pursued further education, gaining admission to both the Harvard Graduate School of Design and the Harvard Business School. It was at Harvard that I became an integral part of an early research program focused on Voxel Printing, utilizing what was then known as Connex, and is now Stratasys technology. Our team developed pioneering software to model and 3D print using this advanced Voxel technology, initially concentrating on structural engineering and design applications.

Currently, my work is situated at the University of Colorado, specifically at the Anschutz Medical Campus, where I lead a laboratory dedicated to computational methods and 3D printing for diverse medical applications. The software originally developed at Harvard has proven remarkably adaptable and valuable for the medical field, simply by modifying the input data. While it might seem like a significant leap from architecture to medicine, the underlying principles are surprisingly similar to working with structural analysis data. Our current research is primarily concentrated on three critical areas: surgical guidance and educational models, the development of advanced prosthetics, and innovative implant designs—all with a particular emphasis on soft tissue applications.

Rob: Hi, I’m Rob Ducey, and I hold the position of technical supervisor for what we refer to as rapid prototype services at LAIKA. In essence, my team is responsible for the intricate modeling, facial animation, and precise fabrication of the highly complex puppets that become the beloved stars of our feature films. The term “rapid prototype” is a proud legacy, stemming from the groundbreaking innovations we introduced on our very first film, “Coraline,” back in 2009. For “Coraline,” we pioneered the use of 3D printing for facial animation, a revolutionary approach that later earned us a prestigious sci-tech Oscar in 2016. My personal journey with 3D printing began on that transformative project in 2006, when Brian Mclean and Martin Munier arrived at the studio with an audacious vision: to employ Objet polyjet printers to generate the thousands of unique faces required to convey the nuanced facial performances of the characters in the film. My primary task was to construct an efficient pipeline for modeling, animating, and ultimately printing these intricate faces. Since that initial production, we have continuously pushed the boundaries, innovating our processes to leverage and, indeed, redefine the limits of multi-material 3D printing. Despite my humble beginnings as a fine art major, I now find myself deeply immersed in tinkering with some of the most cutting-edge technologies in computer-aided fabrication, bridging the gap between artistic vision and technical execution.

3DN: You both come from vastly different disciplines – architecture/medicine and animation. How did such an unlikely collaboration begin?

Nick: Our paths crossed at AMUG several years ago, specifically at AMUG 2019, when LAIKA delivered their captivating keynote presentation. Their talk was truly astounding, powerfully illustrating how they leveraged bitmap printing technology to realize a significant portion of their groundbreaking work. For me, it was incredibly cathartic to hear that others were not only using, but also mastering, this same esoteric technology on such a grand scale. I was genuinely in awe of their achievements and, initially, lacked the courage to approach them immediately. However, a fortunate turn of events occurred later when Brian and Rob Ducey attended my own presentation, which focused on voxel modeling and bitmap printing for surgical applications. Afterwards, they were the very first two individuals to come up and engage with me. We instantly bonded over our mutual admiration for bitmap printing and the immense possibilities it presented. Following this initial connection, we continued our discussions virtually, delving into our individual approaches and challenges with the technology. It became immediately apparent that we each held simple, elegant solutions to problems the other was grappling with. For instance, the special effects world had long developed sophisticated solutions for working with intricate, thin, hair-like structures. This expertise made working with tractography data—a form of medical imaging that visualizes neural pathways as fine, fibrous lines—a breeze for them, whereas it had been a significant challenge for my team. In our very first meetings, and still to this day, we continually brainstorm and identify countless ideas and techniques that the special effects industry has perfected over years, which can directly advance medical applications. These are concepts that no one in medicine had previously considered, precisely because there is so little overlap between our professions. All it took was that single, fortuitous connection to open the floodgates for these transformative conversations.

Rob: Here at LAIKA, our creative ethos drives us to constantly seek out new ideas and technical developments, no matter their origin, that can expand our capabilities and redefine what is achievable in stop-motion animation. In our early explorations of multi-material 3D printing, we discovered a small, dedicated community of like-minded innovators and researchers who had access to machine features still considered “in development.” Through persistent effort and a bit of pestering, we managed to initiate conversations and arrange visits with university researchers at esteemed institutions like MIT and Virginia Tech. These academics were naturally curious about what a group of artists and filmmakers, operating from a warehouse in the rain-soaked Pacific Northwest, were accomplishing with these sophisticated machines. These valuable interactions eventually introduced us to AMUG, which we first attended in 2015 and then returned to in 2019 to deliver a keynote. Whenever we attend AMUG, our focus is always sharply tuned towards finding others from whom we can learn, or simply sharing our profound excitement for the potential of multi-material 3D printing. When I saw the description of Nick’s presentation, I made sure we were there. We eagerly approached Nick, thrilled to discover another individual in what was, at the time, a very exclusive club of “voxel printers.” Part of our motivation was to showcase what we had learned and achieved, but equally, we were keen to uncover any secrets or advanced techniques he might have discovered. It truly felt like encountering a long-lost tribe member with whom we shared a unique understanding and passion.

3DN: Could you elaborate on the specifics of your joint project, and for our readers, please explain what bitmap printing entails?

Nick: Our collaborative projects are primarily concentrated in three interconnected areas. Firstly, we are focused on advancing existing bitmap printing techniques and developing more efficient workflows. Secondly, we are deeply involved in multimodal data fusion specifically for medical applications, with a strong emphasis on soft tissue modeling. And thirdly, we are applying parametric design principles to the creation of highly customized prosthetics.

To explain bitmap printing, imagine how a standard inkjet printer works: it applies ink pixel by pixel onto a page. Bitmap printing operates on a remarkably similar principle, but instead of pages, it works with layers. Essentially, it’s the ability to print directly from a stack of bitmap images, where each image represents a distinct layer of the final 3D object. This method directly leverages how a polyjet printer natively operates, allowing us to bypass the printer’s internal slicer and feed it our own pre-defined slices. Each of these custom slices precisely controls the placement of every single material droplet. However, this level of control also presents a significant challenge: we must individually define each droplet, which is far more complex than traditional methods. Most, if not all, conventional modeling software used for 3D printing relies on a surface mesh-based paradigm. This represents objects by their external boundaries, largely ignoring the crucial internal volumetric data. In numerous critical applications, particularly within medical and structural fields, this volumetric information is incredibly valuable. Therefore, to facilitate bitmap printing, we employ a fundamentally different modeling paradigm known as ‘voxel modeling.’ Voxels, analogous to 3D pixels, allow us to model and manipulate volumetric data directly. The profound advantage of this approach is its ability to seamlessly translate image-based data, such as DICOM data from medical scans, directly into a 3D printed object with minimal to no loss of critical information. Another major benefit inherent to voxel modeling and bitmap printing is the unparalleled capability to fabricate material gradients. By precisely controlling the mixing ratios of different material droplets through dithering techniques, we can achieve objects with continuously varying mechanical and material properties throughout their volume.

1. Currently, a significant hurdle for widespread adoption of bitmap printing is the lack of readily available, off-the-shelf software and workflows. To engage in bitmap printing today, one essentially needs to develop the specialized software and custom pipelines to create the necessary files. Recognizing this, we have dedicated considerable effort to combining our distinct workflows and developing new ones. This ongoing work helps us to better understand, refine, and optimize our ability to produce sophisticated bitmap prints. There are numerous complex factors that still require extensive research and development. For instance, predicting the precise outcome of material mixing, especially concerning mechanical and color properties, remains challenging, often leading to unexpected results. This complexity stems from the sheer volume of calculations a computer must perform to accurately digitally preview and predict the final physical print. Our primary focus in this area is geared towards rigorously quantifying color-based mixing ratios, particularly in relation to translucency, and thoroughly qualifying how mechanical properties vary across intricate material gradients within a single print.
2. Multimodal data fusion is an intrinsically streamlined process within voxel-based modeling. Unlike traditional methods that often rely on computationally intensive boolean operations, voxel modeling utilizes a compositing approach. This makes the combination of numerous complex data sources, whether bitmap or vector-based, simple and remarkably computationally inexpensive. Furthermore, voxel modeling provides the unique capability to volumetrically alter color and material properties without concurrently modifying the object’s form. These advantages enable us to seamlessly integrate diverse medical data sources into a single, comprehensive 3D model. This includes data from MRI, fMRI, PET, MEG, Tractography, and even models of implanted medical devices. Our work in this domain has focused on developing precise workflows that allow us to individually control and manipulate each data source, while simultaneously combining all sources into a cohesive, holistic model that provides an unparalleled understanding of patient anatomy and pathology.
3. Parametric design is a widely adopted and highly effective modeling technique, fundamental in both architecture and, crucially, the “bread and butter” of stop-motion animation. Building on this expertise, we have been working to create advanced ‘rigged’ modeling software. This software enables the facile and dynamic manipulation of complex forms over time, which is essential for animation. Our specific focus with this endeavor has been geared towards designing patient-specific prosthetics that require precise modification and adaptation throughout a patient’s treatment or growth. For this innovative application, we have integrated medical data directly with standard stop-motion animation software. This powerful combination allows us to essentially create a “stop-motion series” of models, detailing the progressive changes and adjustments needed for human prosthetics, offering unprecedented customization and adaptability.

Rob: Our initial meeting quickly illuminated a vast landscape of mutual learning, particularly concerning the art and science of designing volumetrically for 3D printing. We immediately recognized that many inherent qualities of medical imaging processes naturally lend themselves to volumetric data capture and analysis. At LAIKA, we had already been developing sophisticated methods for extracting volumetric data from visual effects workflows and then converting that data into formats suitable for multi-material 3D printing. The cornerstone of that intricate process was indeed “bitmap printing,” or as it is now more commonly known, “voxel printing.” Up until our collaboration with Nick, our primary concern had been the external appearance of a printed object and the outermost layers that defined its aesthetic. Our critical partnership with Fraunhofer and their innovative product, Cuttlefish, was instrumental in mastering this aspect of printing from slices—which is precisely what bitmap or voxel printing fundamentally is. We engineered a precise process to author these slices directly from our volumetric data, enabling us to print them with exceptional fidelity. Nick’s work was particularly captivating due to the diverse modalities of medical imaging he was utilizing, all presented in the DICOM format. These DICOM images were directly analogous to the slices we worked with, and seeing them translated into a printed physical form was truly fascinating. We immediately envisioned clear pathways to integrate this rich medical data into our existing VFX pipelines, leading to the generation of incredibly detailed and visually compelling prints. From this point, our discussions naturally evolved into exploring ways to feed these newfound capabilities back into the research areas that Nick and his team at the University of Colorado were actively investigating. This led to further conversations about leveraging well-established techniques and methodologies from the animation and visual effects industry, which could potentially be applied to ongoing medical research. A unique and powerful crossover for LAIKA lies in the fact that we demand similar levels of precision, specific scales, and intricate details in the parts we engineer and generate for our films, comparable to what one might expect in demanding medical applications. Furthermore, we bring valuable insights into how to make complex structures not only functional but also visually compelling and intuitively understandable.

3DN: What specific applications do you envision for this groundbreaking technology?

Nick: We truly envision a multitude of applications for this technology; frankly, we feel we have only just begun to scratch the surface of its potential! For the immediate future, however, our efforts are concentrated on two profoundly impactful areas: Pediatric Epilepsy Surgery and Cleft Lip and Palate treatment. Pediatric Epilepsy Surgery represents perhaps the most powerful and immediate application of bitmap printing for both presurgical planning and intraoperative guidance. This is due to the fact that epilepsy is largely physiologically based, meaning surgeons must meticulously review numerous 2D-based volumetric data sources—individually and sequentially—to construct a comprehensive mental model of the patient’s unique anatomy and pathology before planning treatment. Our current focus is on combining these numerous, and ever-growing, available datasets into a single, cohesive, holistic 3D model. This integrated model can then be used for more precise surgical planning and real-time guidance during operations. We are excited to announce plans to initiate a prospective clinical trial before the end of the year, specifically designed to rigorously evaluate the effectiveness of having such a sophisticated 3D model in improving treatment planning and enhancing post-operative outcomes for pediatric epilepsy patients.

Our work addressing cleft lip and palate is centered around a critical presurgical technique known as Naso Alveolar Molding (NAM). This is a common and highly effective treatment for infants born with cleft lip and palate, typically initiated almost immediately after birth. The treatment involves the precise placement of a custom acrylic device onto the infant’s palate. This device gently directs and encourages proper growth, with the ultimate goal of gradually closing the cleft. Successful NAM treatment can often eliminate the need for an initial surgery and significantly improve the success rates of all subsequent surgical interventions. However, the current protocol requires weekly adjustments to the device, necessitating that families visit the clinic every week for an arduous period of 3 to 9 months. This demanding schedule presents immense logistical and financial challenges, even for families living in close proximity to a clinic. Furthermore, there are relatively few specialized providers for NAM, rendering access to this crucial treatment prohibitive for the vast majority of patients globally. Our groundbreaking work is therefore focused on providing a 3D printed solution that would revolutionize this process: patients could receive a tailored set of models conveniently delivered by mail, thereby eliminating the burdensome need for weekly office visits. This innovation would dramatically expand access, empowering anyone with the ability to take a simple dental impression to receive this life-changing treatment, regardless of their geographical location or access to specialized clinics.

Rob: Nick and his dedicated team are the driving force behind the crucial research on how to effectively apply the knowledge we’re collaboratively gaining in the field of medicine. For LAIKA, this partnership has provided invaluable clarity, not only on the technical intricacies of designing using voxels but also on a deeper understanding of how biological forms and functions are designed volumetrically, viewed through the expert lens of medical professionals. Applying these profound biological concepts to design is an area that additive manufacturing processes are uniquely positioned to exploit and excel in, pushing the boundaries of what’s possible.

3DN: Do you have any final words for our readers, perhaps a call to action?

Both: If you are eager to delve deeper into our collaborative project, learn more about our individual work, or simply connect with us, we warmly invite you to attend AMUG 2023. We are honored to be delivering the keynote speech titled, “Collaborations Between an Animator, an Architect, and a Surgeon: The Keys to Impactful Innovation in Medicine” on Tuesday, March 21st. This will be a unique opportunity to hear firsthand about our journey and the exciting future of interdisciplinary innovation in medical 3D printing. If you haven’t already, you can secure your spot and register for the conference HERE.

What are your thoughts on the pioneering work that Nick Jacobson and Rob Ducey are undertaking? Do you believe that dedicated 3D printing conferences like AMUG play an instrumental role in fostering essential collaboration across various industries? We’d love to hear your insights! Let us know in a comment below or engage with us on ourLinkedIn,Facebook, andTwitter pages! Don’t miss out on the latest advancements and sign up for our free weeklyNewsletter here, delivering cutting-edge 3D printing news directly to your inbox! You can also explore all our informative videos on our officialYouTube channel.

*Cover Photo Credits: Nicholas Jacobson and Robert Ducey