COMPUTING SYNESTHESIA

snapshot.jpg

Speaking of visual and con­cep­tual rela­tion­ships, I’ve been enjoy­ing some strange trips from using a unique soft­ware called RGB MusicLab. It was clev­erly designed by Kenji Kojima who explains it quite well by saying,

RGB MusicLab con­verts RGB (Red, Green and Blue) value of an image to chro­matic scale sounds. The pro­gram reads RGB value of pix­els from the top left to the bot­tom right of an image. One pixel makes a har­mony of three note of RGB value, and the length of note is deter­mined by bright­ness of the pixel. RGB value 120 or 121 is the cen­ter C, and RGB value 122 or 123 is added a half steps of the scale that is C#. Pure black that is R=0, G=0, B=0 is no sounds.

It is not an impres­sion of paint­ings or pho­tographs of a com­poser. It reads a score from an image data directly.

So, as l recall one of the most influ­en­tial exhibits I’ve ever expe­ri­enced, here a few pieces I’ve come up with.

kit ans saw wave.jpg

Duration: 0:30
Tempo: 183
Instrumentation: 2 elec­tronic kits (right and left chan­nels), 1 saw wave (center)

hmm…an inter­est­ing start

dark.jpg

Duration: 2:09
Tempo: 440
Instrumentation: Choir (left), atmos­phere (cen­ter), orches­tral harp (right)

notice how dark the image is in rela­tion to the pitch

whoa.jpg

Duration: 1:05
Tempo: 470
Instrumentation: celesta (left), pizzi­cato strings (cen­ter), slap bass (right)

this one moves quite quick but is full of color

bigsurdriftwoodshack.jpg

Duration: 1:15
Tempo: 190
Instrumentation: music box (left), gui­tar har­mon­ics (cen­ter), wood­block (right)

sim­i­lar color through­out, crazy ending(?)

bachseal.jpg

Duration: 2:02
Tempo: 600
Instrumentation: harpsichord

Keep in mind, this is Bach’s seal in which he signed his music

dontfuckwithmechopin.jpg

Duration: 3:51
Tempo: 137
Instrumentation: Electric Piano (left), Car Engine (cen­ter), Electric Piano (right)

Spastic, yet rich and full

flush.jpg

Duration: 0:29
Tempo: 80
Instrumentation: Tubular Bells (left), Rain (cen­ter), Tubular Bells (right)

The flu­id­ity of this pic­ture seems to bring out an actual “song”

crazyassorange.jpg

Duration: 0:31
Tempo: 600
Instrumentation: Celesta (left), Dulcimer (cen­ter), Celesta (right)

I absolutely love this inter­pre­ta­tion, it is mad and fits the mood completely

princeofpeace.jpg

Duration: 1:27
Tempo: 309
Instrumentation: Timpani (left), Reverse Cymbal (cen­ter), Electric Piano (right)

Totally rad and con­ve­niently weird — a true stare.

marriageloft.jpg

Duration: 1:20
Tempo: 151
Instrumentation: Percussive Organ (left), Contrabass (cen­ter), Bottle Blow (right)

Reminds me of the moth and it’s journey

9 Responses to “COMPUTING SYNESTHESIA”


  • WHOA! This is per­haps the coolest thing I have ever seen / heard.

    Weird ass imag­i­nary sci­ence fic­tion music!

    Thank you for shar­ing this dude!

    I am inter­ested in how this works. Can you do it in reverse? Also, can you cre­ate a pic­ture with the colors/notes in mind. Fuck music nota­tion. Let’s paint in pho­to­shop. OH Dang. What if you took one of those col­or­ful pic­tures of a quasar or something?

  • Writing through color is some­thing I thought about, but didn’t get deep enough into (yet) to orga­nize anything.

    But yeah, the reverse pic­ture was intrigu­ing as well. I want to input a pic­ture that is already a mosaic lead­ing to even smaller read­able pix­els with less work. If it was col­or­ful, it could lead to some insane rain­bow crys­tals type of shit.

  • this is so cool. this was made for you. you could arrange lots of mono­chro­matic images and play them simul­ta­ne­ously, like mem­bers of an orches­tra (sort of)…i’m just think­ing of how this could be uti­lized in an installation…you should take it big. could you score a film this way? i mean, using the same sort of idea? also, look up an event called unsi­lent night, i bet they’re doing it in port­land. god, this is so cool, i can’t get over how much it fits your per­son­al­ity. thank you!

  • sweet jesus that was sick. I really REALLY love the idea of orches­trat­ing a piece com­pletely by cre­at­ing images. So good man, so good.

  • This is way cool. I down­loaded it and have been tin­ker­ing around with it.

  • yeah, jor­dan, try it with a cir­cu­lar fractal.

  • Jordan! This is amaz­ing. Can this work back­wards? I haven’t lis­tened to any yet because I’m at school, but I’m very inter­ested. Also, how detailed an image can you derive sound from? What are the vari­ables there?

  • back­wards, yes, i really want to try that, but i have a feel­ing it could end up sound­ing very dry, if it is too planned. you could always use the mosiac gen­er­a­tor in the pro­gram, but that takes away most rea­sons for plan­ning it out.

    recently, my comp teacher lec­tured about a very sim­i­lar pro­gram that her hus­band is devel­op­ing. she has been one of the first peo­ple to really uti­lize it and has done some pretty incred­i­ble things with it. recorded breath­ing, stacked chords from library record­ings, over­tone har­mon­ics, etc. it’s sort of like a grand­fa­ther to this one. more on that later, or sooner than later, i sup­pose. last final is tonight, then it’s on!

  • Woah, did you take a look at Kenji Kojima’s paintings?

Leave a Reply