Fish in a Pond Documentation

Dave Final Project – Fish in a Pond

title

A fish which can learn what kind of melody the user likes via machine learning and plays them.

I was inspired by my research professor’s project “Simstudent”, in which a human student walks a computer Simstudent through the steps of algebra problems, from which the Simstudent will learn via machine learning. While I was testing it, I found I greatly enjoyed teaching and watching my Simstudent succeed on the problems that once baffled it. Thus, I started off looking for ways to use machine learning as the backend of my project. Music seemed like a good idea, so I went for it, despite having zero experience.

I used Weka’s implementation of the ADTree algorithm as my backbone. I represented a melody as an array of 10 notes, which is limited to 7 pitches, as recommended by Professor Richard Randall. The user can either rate a melody played by either the fish or the user as favorable or unfavorable, from which the fish learns. My thoughts on what the frontend looks like revolved around a fish, because they don’t seem like creatures that will likely be playing music, so I implemented it as such.

In hindsight, music is probably not the best thing for me to do;. I do not have the skills to hear the musical structures in the melodies that were created, and along with the fact that whether melodies are good or bad to a user is highly subjective, caused me to be unable to statistically confirm whether the algorithm is robust. I did however have a musically gifted friend play around with it, and after around 15 trials, he claimed that the fish picked up on a structure that he had played. I also managed to train my fish to know that it must play the last note at a low pitch, which if that means good melodies in my heart, then I am successful.

Source code can be found in this repository:

Dave – Looking Outwards Final Project

Inspirations

1. Simstudent

http://www.hcii.cmu.edu/research/simstudent-computational-model-student-learning-advance-sciences-learning

Simstudent is a research project at HCII at CMU. Humans learn by teaching, so we can take advantage of that fact by having real students teach algebra to a computer program, the Simstudent, so in turn the students themselves can learn. It uses machine learning algorithms to find patterns in algebra problems to solve them in logical steps. At each phase of the problem, the Simstudent will perform a step in an algebra problem, and the real student has the option to accept it as right, or correct the computer, which will proceed to remember the operations taken so it will not make the same mistakes next time. This is not an art project, but I love the feeling of accomplishment I get when my once-derpy Simstudent became a master at algebra and starts mowing problems down with ease. Thus, I wanted to create my final project based on a machine learning algorithm, so that my audiences can feel what I had felt.

2. Falling Stars

“Falling Stars” is an iPad app that lets the user draw objects on screen, which will produce sounds when drops of falling stars collide with them. The ability to easily create a musical composition that sounds decently coherent with seemly random drawings impresses me. The visuals themes are also great looking and create a calming mood. As an interactive program, it allows the user to create something that not only looks cool but also sounds great. I have always wanted to create a project that uses music and sounds, but I myself do not have enough understanding of music theory to create compositions myself. However, if all I do is set up the environment, and let the audiences and computer take care of generation of music instead, just as Falling Stars did, that might just be possible.

3. MIDI keyboard in Processing

I originally wanted all the music creation to be taken care of by the computer. However, if the user’s only interactions are to rate a composition as good or bad, then he or she might have to brute force through a lot of bad music to find one that is good, which is not exactly a rewarding experience that I had been hoping for. In order to make the experience more interactive, I decided to use a Processing library which can receive input from MIDI keyboards known as RWMidi. I also found an example project made from this library, which displays the robustness that I need. I am currently waiting for my MIDI keyboard to be shipped to me so that I can test it as soon as possible.

 

Ideas:

1. There is a fish tank filled with fish. When each of them flails up out of water, it sings a note. When these fishes do this over a sequence of time, they will produce a composition. The user will then vote whether it is good or not. With a machine learning algorithm, the fishes will remember the label and modify their future compositions based on the user’s taste. This effectively lets the user train the fish to produce “music”. The user can also choose to play music with his/her MIDI keyboard, thus giving more teaching power to the user.

WP_000735

2. Something with music information visualization. I invested money into a MIDI keyboard so I will be using it no matter what. It would be nice to create a visual story with music. This will be something similar to the reverse of “Falling Stars”, so it will probably grow things via pattern of music.

WP_000737

Update: 11/18
I have finished the machine learning backend, and it can play music correctly. All that is left to do is to implement the frontend and sync its timing with the backend, which will not be trivial, but it is not going to be planning intensive.

Dave-Looking Outwards-Shields

1. CMUcam

 

https://www.sparkfun.com/products/10032

Made by CMU, this can be used to collect visual information from the surrounding, allowing the Arduino to detect changes in light, color, or motion.

 

2. VoiceBox Shield

https://www.sparkfun.com/products/10661

With a sound synthesizer, we can generate voice, thus allowing the Arduino to speak. This can be very useful to communicate messages en masse.

 

3. TFT Shield

https://www.adafruit.com/products/802

We can display images via this shield, and with some modifications we can display text also. This allows for a wide range of outputs.

Dave-LookingOutwards

1. Carbon Monoxide Sensor

https://www.sparkfun.com/products/9403#images-modal

The possibility of such a tiny thing being able to be used to save many lives struck me. Of course, smoke detectors serve a similar purpose, but these can be used to alert even more people to come to others’ aids.

2. Heating pad

https://www.sparkfun.com/products/11289

Besides heated clothing in the winter, we can use these from keeping food warm to making the toilet seat less painful to sit on during cold mornings.

3. Nunchucky

https://www.adafruit.com/products/345

The Wii Nunchuck takes in many different types of inputs from various kinds of physical interactions, and we can send them easily to Arduinos with this.

Never Give Up (round 2)

The creature works hard to collect bubbles into a grid, and the user can either mess up its work by moving individual bubbles to anger it, or completely obliterate its creation, causing it to break down into panic.

Changes: the creature collects the bubbles into a grid, the creature moves nonlinearly, and the user can click on individual bubbles to move them and cause a slightly less intense reaction from the creature, who will just grunt and continue its duty.

Never Give Up

The creature collects bubbles into the bottom right corner. The user, by holding down the mouse button, can push the bubbles around and thus mess up the setup the creature made, who will then proceed to panic and stream in agony. Eventually the creature calms back down and continue its job.

WP_000674WP_000676

I decided to use Box2D for this part of the assignment also. I also started off with crabs collecting fruits but found germ thing collecting bubbles to be cooler.

My project turned out to be similar to Swetha’s. I find that I spend more time looking at her project though, because I just really want the bird to succeed, so I just sat there waiting for the moment that will never come. On the other hand, I know my creature will succeed unless I bully it, so I just go back to mess with it every once in a while.

Also, I found out that if I increase the bubble count and restitution, the bubbles will become greatly accelerated and will start beating up the creature, who will march on despite the onslaught. I feel like a bad person.

Sex Ed 60210

The two bathroom sign people start rolling into each other, and upon collision they form an explosion and their baby transcends above (gender of baby is randomly determined).

I found a wall with the 2 gendered bathroom signs close to each other, and this just popped into my head. However, the problem was, even though the signs were close together, they were not close enough, as the hallway I was in was too narrow to project correctly. In the end I decided to keep my idea, and instead I used a chalkboard to be projected onto, which I felt to be more fitting into the theme of education (albeit incorrectly). Because I was projecting directly onto a chalkboard with a classroom’s projector, I did not run into any Keystone issues.

In the end, I feel like the chalkboard turned out to be better than the original idea I had with the bathroom signs. I do feel that I cheated a little by modifying the environment before projection and if I can make this somehow more public and put it less under my control, it would be more fun.

WP_000672

Laser Cut – Fallen Leaves

http://cmuems.com/2013/a/wp-content/uploads/sites/2/2013/10/screen_output.pdf

This is my first attempt at data visualization. Each leaf represents the amount of deaths per 100,000 people caused by a category of cancer in 2009. The size of each leaf is proportional to square root of 5 to those numbers.

The tree is also generated by Processing by consecutively drawing smaller rectangles. The leaves are made from Bezier curves.

I was inspired by “Epilogue” by The Antlers in regards to cancer.

Thanks to CDC for their data.

http://apps.nccd.cdc.gov/uscs/cancersbyraceandethnicity.aspx

int square_width = 15;
import processing.pdf.*;


PVector[] leaves;

// these represent cancer death rate per 100,000 people in 2009 US
// data from CDC http://apps.nccd.cdc.gov/uscs/cancersbyraceandethnicity.aspx
float[] rates = {3.7, 54.6, 64.3, 1.5, 5.7, 0.3, 22.4, 13.7,
				 0.1, 5.3, 0.8, 8.6, 4.2, 9.5, 1.5, 14.9};

void setup() {
  randomSeed(15251);
  smooth();
  noLoop();
  size(640, 640 , PDF, "output.pdf");
  //size(640, 640);
}

void draw() {
	background(255);
	fill(0);
	
	pushMatrix();
	translate(width/2-50, 0);
	scale(0.75, -0.75);
	draw_trunk(0, 0, 30.0, 200, 30);
	popMatrix();
	noFill();

	leaves = new PVector[rates.length];
	for(int i = 0; i < rates.length; i++)
	{
		pushMatrix();
		float r = rates[i];
		float x = random(30, width - 30);
		float y = 250+4*r+random(-30, 200);

		// loop through old leaves
		for(int j = 0; j < i; j++)
		{
			float old_x = leaves[j].x;
			float old_y = leaves[j].y;

			// if similar position, set new
			if((old_x + 50.0 > x && old_x - 50.0 < x) ||
			   (old_y + 50.0 > y && old_y - 50.0 < y))
			{
				x = random(30, width - 30);
				y = 250+4*r+random(-30, 200);
			}
		}

		leaves[i] = new PVector(x, y);

		translate(x, y);
		rotate(radians(random(-170, -10)));
		scale(pow(r, 0.2), pow(r, 0.2));
		draw_leaf(0.0,0.0);
		popMatrix();
	}
	exit();
}

  void draw_leaf(float x, float y)
  {
    bezier(x+10, y-3, 
           x+5, y-8, 
           x-5, y-8, 
           x-10, y);
    bezier(x+10, y+3, 
           x+5, y+8, 
           x-5, y+8, 
           x-10, y);

    strokeWeight(0.7);
    bezier(x+15, y, 
           x+12, y-2, 
           x-2, y+2, 
           x-5, y);

    strokeWeight(0.3);
    line(x+7, y, x+5, y-3);
    line(x+7, y, x+5, y+4);

    line(x+3, y+0.2, x+1, y-3);
    line(x+3, y+0.2, x+1, y+4);

    line(x-1, y+0.2, x-3, y-3);
    line(x-1, y+0.2, x-3, y+3.5);
    strokeWeight(1);
    
  }

void draw_trunk(int x, int y, float tree_thickness, int tree_height, int segments)
{
  pushMatrix();
  float current_tree_thickness = tree_thickness;
  float segment_height = tree_height/segments;

  translate(x,y);

  for(int i = 0; i < segments; i++)
  {
    translate(0, -segment_height);
    rect(0, 0, current_tree_thickness, segment_height*2.0);
    rotate(random(-0.1,0.1));
    current_tree_thickness *= 0.97;
  }

  for(int i = 0; i < 10; i++)
  {
    float random_scale = random(0.3, 3.0);
    draw_branch(0,0,radians(random(-30,40)),random_scale, random_direction());
  }

  popMatrix();
}

void draw_branch(int x, int y, float theta, float curve_scale, int direction)
{

  float len = 20.0;
  float branch_width = 5.0;

  pushMatrix();
  translate(x,y);
  rotate(theta*-1*direction);
  scale(direction, 1);
  
  if(direction < 0.0)
    translate(-25,0);
  else
    translate(-20,0);

  int branch_blocks = (int)random(2, 6);


  for(int i = 0; i < 80; i++)
  {
    translate(len, 0);
    rotate(random(-0.15, 0.05));
    rect(0,0,len,branch_width);
    branch_width *= 0.9;
    len = len*0.9;
  }
  popMatrix();
}

int random_direction()
{
  if(random(-1,1) > 0)
    return 1;
  else
    return -1;
}

DAVE-LOOKINGOUTWARDS-3

Printer Orchestra is made by isthis.gd from old computer hardware which are synced to play MIDI arrangements. The music they chose is lively, and I instantly became happy after seeing how cute these old parts are being as they clicked and clanked to make music. This reminded me of the MIDI arrangements played on floppy drives I had seen in the past. The only thing I wished that the makers would do is for them to upload a making of video, so we can see their “orchestra” play other musical arrangements, and how they configured and set up such a whimsical piece.

 

Fire Hero is by oswaldonfire as a version of Guitar Hero which instead of the original computer graphics effects, uses flamethrowers to represent notes being hit. When I first saw this project, I was struck by how insane the creators were; fire is scary, and I would never choose to use it as a medium. But seeing the flames sprout out in beat with the music, I am deeply impressed. The closest thing I have seen resembling this is equalizers, but this takes that to an extreme and dangerous territory. If only the video quality is more professional and they arranged it more dramatically, then that would bring out that edginess of this project even more.

 

It turns out all the projects I found are somehow related to music. Playa is made by rubendhers, and it uses motors, cables and cloth to play a chillingly ambient arrangement on 14 guitars. This provides a striking contrast to the previous 2 projects, as this generates a mysterious atmosphere, while printer Orchestra is whimsical and Fire Hero is hardcore extreme. The autonomous spinners, cables, and the brownish background gave me a steampunk impression, which contributes even further to that otherworldliness of the music and the mysterious atmosphere. This reminded me of my first visit to CMU: I walked into CFA, and I heard an ambient sound. Someone explained that it was Beethoven’s 9th played over the period of 24 hours. It created a feeling that one is constantly discovering and unraveling the secrets of the universe.