This project was developed during the course Computational Creativity in 2021. 

The aim of the project was to explore, reproduce and study different types of synesthesia. In order to do so, I implemented a generator which tries to visualize and recreate 3 types of synesthesia: chromesthesia, pitch-size and pitch-location synesthesia.

For an explanation of the technical specifications of the generator refer to files and papers at the end of this post. Briefly, the audio for the model is generated through a Markov Chain probability model, which is learnt in VCVRack. The model is then triggered from a pseudo-random clock, and sends back notes according to the probability distribution. Since the inspiring file was made out of 3 MIDI channel, the model learns 3 different probability distributions. The system is thought to be used in real-time. The file I used as inspiring set is the song Everything you do is a Balloon from Boards of Canada, but every MIDI file would work fine.

In order to study the 3 different types of synesthesia, I have implemented 4 different ways of visualizing the music. In the following video you can see them. 

The top-left corner of the 4 videos tries to visualize the pitch-to-color synesthesia (chromesthesia) and the pitch-to-altitude synesthesia.

The top-right quadrant is trying to visualize chromesthesia and pitch-to-size synesthesia.

The bottom-right quadrant is visualizing both pitch-size and pitch-to-color synesthesia.

The bottom-left quadrant is the most complete among the other ones, trying to visualize all the 3 types of synesthesia described.

Here you can see a demonstration video

For more information about the project, implementation and my research behind synesthesia you can click on the following links: