In a world that is increasingly being made into capitalist societies operating on digital fronts, it is an increasing sense of anxiety towards the future of performing arts that has motivated me to attempt this project. By embedding technology onto an apparel that we, for the most part, only use when we go outdoors, I seek to redefine its purpose.
I got the inspiration to try to make this music gloves project based on a blog post put up by Miss Amanda Swann. A British musician, Imogen Heap, has managed to assemble a team to successfully make gloves with the arduino components stitched onto it and make music through gestures. The technology involves transmitting movement and bend data through wi-fi to a system nearby which has custom –made software that can translate the numeric values received into musical notes, which will then be played through speakers. Here is a link to one of her performances :
In an interview, Miss Heap says, “We really feel that they are going to change the way we make music.” She seeks to make herself independent of the need to learn the technology behind traditional synthesizers and getting the more human actions of gesturing, to make music.
In the initial research phase of this project, I found that the programming was predominantly done using an Integrated Development Environment(IDE) called Processing that was different from the Arduino environment we had been introduced to in class. Taking into consideration the time period given to produce the project this semester, I sought to simplify the product. The original uses conductive textiles and expensive electronic components. I tried to make mine using woolen gloves, interfacing fabric, conductive thread and components from the protosnap board. My initial testing was on the code and logic. I worked on the arduino lilypad protosnap board and tried to access the speaker on the board to make selective noises or tones when users entered input, either through button and switch or through the serial port.
Button melody : http://youtu.be/0GbwfgrvIdg
Switch controlled button melody : http://youtu.be/nSiErME1iLU
In both cases, I tried building on the exampled provided by adding the controls and the RGB LED. Once I had established the basic control and code, I proceeded with working with serial input generating tones. I used the example of the board printing the serial value it receives from user and combined it with the tone generation example, with initial input set restricted to the ten digits (0-9). This was part of my need to simplify the whole project by keeping the input from user to the number of fingers. This was a premise I had failed to look through fully. Initially, I had problems with loose contact of the serial cable with the board, leading to incessant and incomplete noise coming from the speaker which kept restarting due to the variant power source. Later, I faced frowns from classmates as I failed to put in the control switch and did not write the code to halt the speaker after the input was done. It was a bit tricky as well as the user input could come at any time and the board could not afford to turn off. I eventually came up with these :
Serial Input tone generation : http://youtu.be/m4rhyZG9Adg
Pi as musical notes (Delayed reading/ouput) : http://youtu.be/MsgmSw-uvio
Being able to listen to the value of Pi, a mathematical constant, was epiphanic in a way. I was suddenly confronted with the idea of taking the concept of visualization of mathematical formulae and constants and applying them to the sphere of music. I decided to keep it as a parallel prototype for the project as the main idea relied on the concept of bend sensors placed under fingers sending data to the Arduino in order to generate music from the system or board. I found that I had missed the necessary Velostat and conducting fabric required in online tutorials to make cheap versions of bend sensors and had to place orders for them. In the meantime, I tested the idea of having data transfer done through bluetooth. I used an Arduino Leonardo, given by Mr.Chris Lewis, and a bluetooth module available for purchase separately.
The connectivity was new to me but, I managed to send and receive data from the board eventually. I noticed that a lot of junk values were being passed as well and recalled a friend’s advise about filtering the extra values coming from an Arduino.
Test 1 : http://youtu.be/ojFKFH31tf0
Test 2 : http://youtu.be/doqv3G8_k3s
I had now established that I can send values to and receive values from the Arduino. I tried to establish a multi-device connection so that I could send data from a mobile phone through the Arduino to the PC as an encrypted message of musical notes, which the system can later decode. It presented a new means of cryptographic transmission. Unfortunately, the board would not accept more than one connection but, it is definitely something that can be explored even further in terms of data security during communication.
I received my materials for making the bend sensors and proceed with making one based on a Youtube tutorial (https://www.youtube.com/watch?v=CkPekPiA-sc ). However, the tutorial involved using some predefined sketches of code in the Processing IDE. I tried connecting the setup to the Lilypad and observing the changes in analog input from the board when I tested the bending.
Analog bend read test : http://youtu.be/nV6TVQ6OoK0
The bend sensor was a failure. However, the fact that I could manipulate the speaker based on the change in analog readings when the circuit was completed by a contact made of conducting fabric, gave me an idea for a design where I could keep contact points on each finger tip such that one finger would be connected to the ground or negative terminal and the other contacts would be connected to the positives. When a particular analog pin gave a change in reading below the threshold, the speaker would play the note mapped to it. I tested the same with a maximum of two analog pins. I also looked at the possibility of having the design made in such a way that I could make use of the maximum possible number of terminals on the lilypad as well my fingers, including combining the glove with a full arm shirt or t-shirt with the wires running down the sleeve to behind the shoulder or in the small of the back.
The design looked very appealing visually, however, during the testing with two leads, I found that if the conducting fabric scrapes against the conducting thread too fast, a spark comes up. There was also a recurring case of random noise coming out of the speaker due to the analog pins catching something from the atmosphere. I had to keep the idea aside for some time considering the safety aspect. There was also the consideration to be made for a t-shirt stitch design that I would have to stitch enough interfacing fabric to avoid skin contact with the conducting wire.
In the run up to the deadline, I went ahead with the parallel idea of sending serial input to create music on the glove. I had added on my earlier code to include the full range of characters that a user can enter through a standard english keyboard.
Character test : http://youtu.be/gYuii4NJ62Q
The output was a success with space set to a pause and arbitrary durations and LEDs selected based on a mathematical operation performed on each entered character’s ASCII value after offsetting for the space’s character value. The serial value is defaulted to zero or space after user input is terminated, hence resolving the issue of the last played note repeating in an irritating manner. Looking through the materials I had, aided by the charity of Mr.Bailey Browder just before the weekend, I drew up sketches to help me with my sewing for the gloves.
Since I was using a glove, I could not afford to use the sewing machine and risk stitching the front and back together. The hand sewing took a total of almost 10 hours.
I turned the glove inside out and sewed the interfacing on first. I faced a problem midway when my attempt to cut off the fabric that was jutting out ended up cutting a stitch off and it had a domino effect. I modified my initial design so that the final connection back to the arduino could come around the thumb rather than the middle finger. This was done as a convenience as the shape of the cut fabric matched the thumb. Unfortunately, during the cross-stitching, I did what I had tried to avoid and found that when I was flipping the glove back out, I had sewn some parts of the front and back together. However, with the number of cross-stitches done, the fabric remained intact despite the broken bonds.
I fixed the lilypad on the back of the wrist. The base design was set and I did not deviate from it. I attached the lilypad first and then fixed one speaker. Given that I had the luxury of an extra speaker, I decided to use it and make changes in code so that a single input produced two sounds in combination rather than just one.
Using the extra LEDs, I decided to work up a quirky design of LEDs in series so that the lights do not just blink in one particular spot for each tone. I also placed the RGB LEDs on the thumb and little finger spots.
As a final add-on, I attached the vibration motor in series with one of the speakers to provide some haptic feedback to the wearer. The sound of the vibration would also contribute to the hearing experience. It was extremely back-bending to work on this stitching. The conducting thread had a tendency to curl up and get entangled (much like my hair) and was difficult working with. I had to be careful with the stitches so that they did not go through the interfacing as well.
Finally, I had my project ready and did a test. Two of the fingers’ connections did not work and only one out of three LEDs lit up on the third finger. The two RGB LEDs were working perfectly, as did the speakers and vibration motor. In attempting to troubleshoot and try to get the remaining arms to work, I eventually lost the one white LED that did work and also one of the RGB LEDs. I could not properly figure out which connection I had gone wrong in but, the visual part of the project has gone to the dumps. I added the code to randomly assign tones to each ASCII character and also used a similar logic to set a LED branch to each character, instead of the very orderly multiples of five calculation I had kept earlier. The new code ensured that the duration and lights associated with each character are unique and enhance the experience of using it.
The project might have been only partially successful but, at the core of it, I have brought out a sense of maker culture that can be disseminated to the public. Music is something all of us enjoy but the art of composition has been restricted mostly to those who have had a formal education in it. This project tosses the authority of formal music education and enables people to find music in their everyday activities and words. Considering that people work with computers too much nowadays, this definitely allows for some creative production during breaks. The nature of the tones generated is also unique as compared to traditional instruments. This project represents a change in the way we perceive an object we work with everyday, the keyboard, and adds the new dimension of musicality into it. It is something that makes me recount a short tale I had read as a child where a mysterious man with a tuning fork visits a family at their home and tunes each object. After he leaves, whenever the family used any of those objects, a particular note or instrument would play, thus making their home and lives all the more vibrant. This project definitely had me feeling like I had become that man with the tuning fork.