It all started with a message from Niels containing a link to a fascinating GitHub project. The repository featured a virtual fruit fly living on your desktop whose movements, such as walking, resting, grooming, and taking off, were entirely driven by a live emulation of a real fruit fly brain. I was curious to understand how the application worked under the hood and how realistic the emulation actually was in practice, so I decided to take it on as a side project.

To appreciate why this is so remarkable, one has to look back at the history of fruit fly neuroscience. For decades, mapping the complete wiring diagram of an insect brain, known as a connectome, was considered an almost impossible feat. While scientists managed to map the simple 302-neuron nervous system of a roundworm back in the 1980s, an adult fruit fly brain consists of over 139,000 neurons and more than 50 million synaptic connections.

The major breakthrough came through the international FlyWire Consortium, alongside researchers at HHMI’s Janelia Research Campus and Google Research. To capture this complex architecture, an adult female fruit fly brain was sliced into thousands of ultra-thin sections, each measuring just 40 nanometers thick. Every slice was scanned using high-resolution electron microscopy. Computer vision and artificial intelligence algorithms stitched millions of images into a three-dimensional reconstruction, which was then painstakingly proofread by neuroscientists around the world. The resulting FAFB dataset allowed researchers for the first time to trace exactly how sensory inputs flow through internal networks to trigger specific motor outputs, such as the fly’s rapid escape reflex.

The original project that Niels shared used this connectome dataset to run a live Leaky Integrate-and-Fire neural emulation on the desktop. The virtual fly reacted dynamically to screen events because visual stimuli caused emulated neurons to fire up to an escape threshold. The fly also treated open application window borders as physical ledges to walk along.

For my own experiment, I wanted to replicate this behavior inside the Unity game engine. To bridge Unity with the neural emulation, I built a Python socket link. The Python server runs the spiking neuron calculations and streams active firing data continuously to Unity. In Unity, this stream is mapped directly into animations, forces, and character movement. Although the Python server also supports sending environmental feedback back into the brain model, I kept my initial version focused on executing the incoming motor commands.

To make the experiment more entertaining, I decided to give the fly’s internal physiological states a voice. By monitoring firing activity across specific neuron clusters, I could infer whether the fly was startled by the cursor, hungry, or simply relaxing. I connected these emotional states to the ElevenLabs API to let the flies speak out loud. To keep API usage efficient, generated audio clips are saved to a local cache so that repeated phrases can play instantly without making new cloud requests.

It remains incredible to watch a complex neurological dataset travel through a socket link and voice synthesis to become an interactive, talking digital organism on screen.