Category: Processing

Twitter + Processing + Arduino

The journey from sensor to Twitter is a little complex but we are providing you with examples and code so you only get to do the fun, conceptual part. The system works like this:

1- Sensor input is read by Arduino
2- Arduino talks to Processing through serial
3- Processing uses a library called Twitter4j to interface with the Twitter API (Application Programming Interface)
4- The Twitter API tweets on the selected account

And vice versa.
Please note that if Processing crashes you’ll probably need to restart and unplug the board. And you can’t have
We’ll break this down into simple steps.

Twitter
In order to do this project, you’ll need to have a Twitter account, and then create an authorized “App”. Basically, Twitter wants you to have fun, but they also want to keep track of what you’re doing, so that you don’t send lots of spam. (Computer programs are good at that.) You’ll need to get credentials by doing the following:

  • Get a Twitter account
  • Create a Twitter application
  • Fill out the description of your application (e.g. “Caroline’sEMS2App”)
  • Agree to the developer’s license, fill out the CAPTCHA, etc…
  • Change your app settings (Settings tab) to “Read and Write”. Don’t forget to SAVE the settings
  • Click on the button to request AccessToken and AccessTokenSecret. If you cange the “read and write” settings you have to create new tokens
  • Copy and paste your credentials: OAuthConsumerKey and OAuthConsumerSecret, AccessToken and AccessTokenSecret in the Processing sketch
  • Twitter4j will require those credentials in order for you to do stuff.

Tweeting from Processing

Here’s a simple demo that Tweets from Processing:
processing_to_twitter.zip

Important: You have to change OAuthConsumerKey, OAuthConsumerSecret, AccessToken, AccessTokenSecret or you will tweet with my credentials @ems2tweet.

The documentation for the library (included in the zip) is here:
http://twitter4j.org/en/javadoc/index.html

Arduino
If you successfully sent your first tweet on your account, then you should be able to integrate it with Arduino and Firmata.

Exercise
Try to send something with a switch (possibly de-bounced) or another sensor. Please make sure your application doesn’t send too many tweets (test it offline first) or you account may be suspended.

Reading tweets from processing
This is an example that reads new tweets from your account. In this way you can control a local system (Processing/Arduino) remotely from everywhere.
tweetReceiver.zip

Exercise
Turn an LED on and off by sending certain tweets.

Twitter + Arduino

Here’s a complete example of bi-directional communication between Twitter and Arduino via processing. Tweet anything with the word “sensor” in it and the script will read and tweet the value of the sensor and turn on an LED.
twitterArduino.zip

Various Snippets

Serial issues with Firmata
It’s rare but sometimes you need to manually change the constructor for the Arduino in processing

println(Arduino.list());  
arduino = new Arduino(this, Arduino.list()[0], 57600);

Read the output of the “println(Arduino.list())” and look for the slot corresponding to /dev/tty.usbmodem… That’s the number you have to put between the brackets instead of 0 in Arduino.list()[0].

Comparing strings
You can’t compare strings in processing using the symbol ==. You have to use “equals”:

String p = "potato";
if (p.equals("potato")) {
  println("Yes, the contents of p and potato are the same.");
}

Parsing
Here are some basic snippets you should use to parse a tweet.
Looking for the word “love” in the tweet:

println ("Tweet = " + mostRecentTweetString) ;

    String keyword = "love";
    //indexOf returns the char position OR -1 if no mathces are found
    int locationOfTestStringInTweet = mostRecentTweetString.indexOf(keyword);
    if (locationOfTestStringInTweet != -1) {
      println ("Test string exists! (At character index: " + locationOfTestStringInTweet + ")");
    }

Looking for a number and converting it to integer:

//first I split my string into single words in an array of strings
    //separating with " " space
    String tweetWords[] = split(mostRecentTweetString, " ");

    //check all the words
    for (int i=0; i<tweetWords.length; i++) {      
      //try to see if there is a number
      try {
        int valueFetchedFromTweet = Integer.parseInt(tweetWords[i]);
        //if the number is parseable do something
        println("number found = " + valueFetchedFromTweet);
      }
      catch(Exception e) {
        //if it's not a number it throws an error
        println("No numbers in "+tweetWords[i]);
      }
    }

Parsing a formatted “command” from a tweet. A command in this example is a string followed by a space and a number like “servo 180”

//splitting the tweet into words
    String arrayOfWords[] = split (mostRecentTweetString, " ");

    if (arrayOfWords[0].indexOf("servo") != -1)
    {
      //if the first word is "servo" I look for an angle in the second word
      int servoAngle = Integer.parseInt(tweetWords[1]);
      
      println("moving servo to " + servoAngle);
      //careful if the command is not formatted it will throw an error
      arduino.analogWrite(servoPin1, servoAngle);
    }

    if (arrayOfWords[0].indexOf("LED") != -1)
    {
      println("BLINK BLINK");
      //...and so on
    }

Controlling Servos from Processing (and Twitter)
Unfortunately the Arduino/Firmata library for processing doesn’t allow you to use servo. In order to do that you have you create your own Arduino to Processing communication interface. You can find some examples online:
Controlling a servo motor with processing

Single servo example

Reading tweets from specific users
The code above reads from your account. But if you change this line:

statuses = twitter.getUserTimeline(); 

With something like this:

statuses = twitter.getUserTimeline (17454769);

You’ll get the last tweet of the user associated to that number.
To find the id of a certain user insert the twitter handle here:
http://www.idfromuser.com/

For example putting together this code with the one we saw on monday I can make an application that
1. Reads from @foxnews (Id: 1367531)
2. Looks for the string “Occupy Wall Street”
3. Makes an led blink
4. And “responds” with a tweet such as “@foxnews bullshit!” (not that you need a parser to detect bullshit on fox news)

Reading tweets from all users
You can check when a word is posted from all the users on twitter. Twitter is huge so pick an uncommon word or an hashtag such as #art.
tweetChecker.zip

Uploading a webcam snapshot to Flickr
I haven’t tested this one recently.
http://frontiernerds.com/upload-to-flickr-from-processing

Uploading image + status on Twitpic
I haven’t tested this one recently.
The integration with Twitpic is better than Flickr so if you want to quickly upload images and make them visible on Twitter try this example:
processingTwitpic.zip

It uses another library (included in the “code” folder). It uploads an image and post a twitter update with just a line of code. In this case the image is randomly generated and is uploaded after a keystroke. Use your Twitter account username and password (no API developer keys are needed).

IF THIS THEN THAT

ITTT is a social network glue that allows you to link and trigger events across web 2.0 services
Some examples here

For this assignment you can use your dedicated twitter account connected to processing as intermediary for all the other networks. es:

Facebook -> Twitter -> Processing -> Arduino -> Circuit

So many things can go wrong here :-P

Assignment: Networked Object

Make an artifact that creates a meaningful relation between the Internet (Twitter) and the “physical” world.
It can be a tweeting sensor or an object that responds to the flows of information on Twitter. Or there can be a two way communication. Ideas are more important than technical complexity.

*The point is to connect the physical world with the digital world: Twitter to Processing or Processing to Twitter is not enough.

*If you are reading data from accounts that you are not controlling, first make sure that the whole system works and then try to run it with the actual data.

*Don’t think about Twitter as just a personal microblogging platform but as a scale model of the Internet, a model that is extremely easy to parse and analyze. Pretty much every institution, group or media has a Twitter account, the possibilities are endless.

*If you are not familiar with Twitter find out how #hashtags and @replies and mentions work.

*Recap of the interactions you can easily implement:
– Tweeting from Processing
– Looking for tweets from Processing by account name or basic parsing (somebody tweets a specific word).
– Read sensors and control actuators via Processing/Arduino
– Capture images from the webcam
– Post images on Flickr from processing
– Everything you already know about Processing
– External Processing libraries from the internet

Due dates
Ideas by Tuesday 27
Projects by Thursday 29

Assignment: Alternative Interface

Make a processing sketch (screen based work) that employs a non-standard Arduino-powered interface.

– Think about how these sensors can be integrated into an object (es. flex sensor to detect the opening of a box) and within a specific context (es. pressure sensor to detect somebody sitting in front of the screen).

– Think about a screen or sound output that is meaningful and related to the interaction. Abstract visuals are fine but go beyond randomish flickering shapes this time.

– More is not necessarily better but consider using more than one sensor.

Advanced: make a simple game (single player or multiplayer) built around the interface. Don’t spend too much time making graphics please.

Document interaction and screen output with a video and post the link here. Phone cameras are fine but screen recordings may work better.
Quicktime for Mac can record what happens on the screen and can be used in combination with a webcam feed from photo booth.

Arduino + Processing

The instructions to make arduino talk to Processing through Firmata are here.

1- Unzip the Processing library and copy the “arduino” folder into the “libraries” sub-folder of your Processing Sketchbook.
(You can find the location of your Sketchbook by opening the Processing Preferences. If you haven’t made a “libraries” sub-folder, create one.)

2- Run Arduino, open the Examples > Firmata > StandardFirmata sketch,
and upload it to the Arduino board.

3- Make this circuit:

4- Run the processing sketch below.

//import the libraries for serial communication
//and for arduino/firmata specifically
import processing.serial.*;
import cc.arduino.*;

//this object class Arduino
//represents guess what? Your board
Arduino arduino;

//like in an arduino sketch it's good to
//use variables for pin numbers 
int buttonPin = 13;
int sensorPin = 0;

void setup() {
  size(500, 500);
  smooth();
  noStroke();
  ellipseMode(CENTER);
  
  //the arduino object needs to be created at the beginning
  arduino = new Arduino(this, Arduino.list()[0], 57600);
  
  //once it's created I set the pin modes  
  arduino.pinMode(buttonPin, Arduino.INPUT);
  arduino.pinMode(sensorPin, Arduino.INPUT); 
}

//we are in processing so the main function is draw (not loop)
void draw()
{
  //read a digital value from the button pin
  int digitalValue = arduino.digitalRead(buttonPin);
  
  //change the background accordingly
  if (digitalValue == Arduino.HIGH)
    background(255);
  else
    background(0);
    
  //read an analog value from the sensor pin
  int analogValue =  arduino.analogRead(sensorPin);
  println(analogValue); //print it for testing purposes
  
  //draw a red circle ellipse the size of the analog value
  fill(255,0,0);
  ellipse(width/2, height/2, analogValue, analogValue);
}

Some exercises:
– Threshold switch, if the pressure sensor reaches a certain value a sound is triggered.

– Combine this example with your order and chaos processing sketch, replacing the pressure sensor with the slider.

Try with a photoresistor:

Or a magnet+Hall Effect

– Team up with a friend and use two knobs potentiometers to make a digital etch-a-sketch. One pot controls x, controls y, the red buttons clear the drawing.

Processing to arduino without Firmata

You can write your own serial communication protocol without using firmata in case you have issues with servo and stuff.
Here’s an example with two analog inputs to processing.

Arduino Code:


int sensorValue0;
int sensorValue1;
 
void setup() {
  // initialize the serial communication:
  Serial.begin(9600);
}
 
void loop() {
  sensorValue0 = analogRead(A0);
  sensorValue1 = analogRead(A1);
  Serial.print(sensorValue0);
  Serial.print(",");
  Serial.println(sensorValue1);
 
  delay(10);
}

Processing code:

import processing.serial.*;
 
int sensor0 = 0;
int sensor1 = 0;
 
Serial myPort;
 
void setup() {
  size(200, 200);
 
  // List all the available serial ports
  println(Serial.list());
  // I know that the first port in the serial list on my mac
  // is always my  Arduino, so I open Serial.list()[0].
  // Open whatever port is the one you're using.
  myPort = new Serial(this, Serial.list()[0], 9600);
  // don't generate a serialEvent() unless you get a newline character:
  myPort.bufferUntil('\n');
}
 
void draw() {
  // do your stuff here!
}
 
void serialEvent(Serial myPort) {
  // get the ASCII string:
  String inString = myPort.readStringUntil('\n');
 
  if (inString != null) {
    // trim off any whitespace:
    inString = trim(inString);
    // split the string on the commas and convert the
    // resulting substrings into an integer array:
    int[] sensors = int(split(inString, ","));
    // if the array has at least two elements, you know
    // you got the whole thing.  Put the numbers in the
    // sensor variables:
    if (sensors.length >=2) {
      sensor0 = sensors[0];
      sensor1 = sensors[1];
    }
  }
}

Assignment: Particle System

This is a suggested starting point for the assignment. It’s a slightly different particle system than the one we programmed in class. The “things” move according to a angular direction instead of speedX and speedY.

int NUM_THINGS = 10;

// creating an (empty) global array with a
// capacity of 10 elements of type Ball
Thing[] things = new Thing[NUM_THINGS];

void setup() 
{
  size(500, 500); 
  smooth();


  for (int i = 0; i < NUM_THINGS; i++) 
  {
    things[i] = new Thing();   // populating the array

    things[i].x = random(0, width);
    things[i].y = random(0, height);
    things[i].direction = random(0, 360);
  }
}

void draw() 
{ 
  //bg is not conceptually part of the class
  background( 0, 0, 0);

  //call the method update everytime I draw
  for (int i = 0; i < NUM_THINGS; i++)
  {
    things[i].update();
  }
}


class Thing
{
  //parameters of my thing
  float myDiameter = 10;
  float x;
  float y;
  color col = #00FFFF;
  float speed = 1;
  float direction = 0;
  
  //Constructor function - same name as the class
  Thing()
  {
  /*
  like the setup in the main sketch the constructor function 
  is executed once as soon as the object is created
  */
  }

  //method(s) for my thing
  void update()
  {
    /*
    convert direction into radians
     break it down to the sinus and cosinus
     which are the vertical and horizontal component 
     of the angle then multiply for speed 
     (because in trigonometry you always work with a radius of 1)
     */
    float dX = cos(radians(direction))*speed;
    float dY = sin(radians(direction))*speed;

    //the result are the steps in the vertical and horizontal direction
    x = x + dX;
    y = y + dY;

    //direction++;

    //managing borders
    if (x > width)
    {
      x = 0;
    }

    if (x < 0)
    {
      x = width;
    }

    if (y > height)
    {
      y = 0;
    }

    if (y < 0)
    {
      y = height;
    }

    //draw the thing  
    fill(col);
    noStroke();
    ellipse(x, y, myDiameter, myDiameter);

    /*
    //example of interaction among things//
    //every object scans the other objects using the global array
    for (int i = 0; i < NUM_THINGS; i++)
    {
      //temporary variable to store the other
      Thing other = things[i];

      //it's not a good idea to make theinteract with itself
      //this points to the specific instance
      if (this != other)
      {
        //after I determine it's another thing I check the distance
        if (dist(x, y, other.x, other.y)<100)
        {
          //and do something
          stroke(255);
          line(x, y, other.x, other.y);
        }
      }
    }
    */
    ////////////
  }
}

Assignment

Start from this code and produce a “more interesting” “flock of things” using interaction and/or random (to change behavior every so often for example) and by interrelating the behavior of the various things with each other (uncomment the distance part).

Advanced: create one or more additional classes and produce an artificial ecosystem in wich certain objects interact with others.

Objects

Back to the circle animation:

int myDiameter = 10;
float posX = 250;
float posY = 250;

void setup() 
{
  size(500, 500); 
  smooth();
  noStroke();  
}
 
void draw() 
{ 
  background( 0, 0, 0);
  ellipse(posX, posY, myDiameter, myDiameter);
  posX++;
}

If we want to animate a circle we need to save coordinates.
But what if we want to move many circles, each one independently?
We can make more variables and arrays but when we have a whole lot of different entities with different characteristics it would become a nightmare.

The solution to this an many other problems are objects
and Object Oriented Programming OOP

How to write a class

A class is the code to defines a certain kind of object.
Class is the abstract idea, the blueprint. Objects are “instances” of a class.
Think about the idea of a cat and a cat specimen.

The code can go anywhere outside setup and draw or on another file:

//class name capitalized
class Ball
{
  //we move the variables here because
  //they belong only to this class
  //they are "properties" of this class
  int myDiameter = 10;
  float posX = 250;
  float posY = 250;
  
  //we move here all the functions that pertain to the class
  //they are called "methods"
  void update()
  {
  ellipse(posX, posY, myDiameter, myDiameter);
  posX++;
  }
}

Using Objects

Objects need to be declared like variables and created with the command “new”.

//declare an object myBall of the class Ball
Ball myBall;

void setup() 
{
  size(500, 500); 
  smooth();
  noStroke();
  
  //create the ball
  myBall = new Ball();
  
  //this is how I access the property of the object I just 
  //created, the default value of posX is immediately overwritten 
  myBall.posX = 100;

}

void draw() 
{ 
  //bg is not conceptually part of the class so it's in the draw function
  background( 0, 0, 0);
  
  //this is how I call the method update
  myBall.update();
}


class Ball
{
  //properties with default value
  int myDiameter = 10;
  float posX = 250;
  float posY = 250;
  
  //method that draws and moves the ball
  void update()
  {
  ellipse(posX, posY, myDiameter, myDiameter);
  posX++;
  }
}

Exercises:
Add a color property
Make the ball wrap around the screen
Create another instance of the ball

Arrays of objects

Objects are particularly useful when used in combination with arrays.
This program manages n balls “stored” in a global array.

                               
int NUM_BALLS = 10;

// creating an (empty) global array with a
// capacity of 10 elements of type Ball
Ball[] myBalls = new Ball[NUM_BALLS];
           
void setup() 
{
  size(500, 500); 
  smooth();
  noStroke();
  
  // populating the array
  // with new Ball objects
 for (int i = 0; i < NUM_BALLS; i++) 
  {
    myBalls[i] = new Ball(); 

    //assigning random positions  
    myBalls[i].posX = random(0, width);
    myBalls[i].posY = random(0, height);
    myBalls[i].speed = 2;
  } 
}

void draw() 
{ 
  background( 0, 0, 0);
  
  //call the method update every time I draw
  for (int i = 0; i < NUM_BALLS; i++) 
  {
    myBalls[i].update();
  }
  
  
}


class Ball
{
  //properties
  int myDiameter = 10;
  float posX = 250;
  float posY = 250;
  color col = #00FFFF;
  float speed = 0;
  
  //methods
  void update()
  {
  fill(col);
  ellipse(posX, posY, myDiameter, myDiameter);
  posX = posX + speed;
  
  //bouncing back and forth
  if(posX > width)
    {
    //it's better to instantly update the ball position to the max
    posX = width;
    speed *= -1;
    }
    
  if(posX < 0)
    {
    //it's better to instantly update the ball position to the max
    posX = 0;
    speed *= -1;
    }
  }
  
}

Complexity and Emergence

Game of life by John Conway (1970)

Cellular Automata

Inspired by Go, originally implemented as analog system.

Two steps (generations) of the game of life

Try it in motion online or offline.
Reminds of fire burning forests, bacteria, molds, etc…
(SimCity is based on cellular automata as well)

Flocks

Flock algorithm (Boids) implementation in processing

Birds, fish, bugs, buffalos…

1) Don’t get too crowded.
2) Don’t get too isolated.
3) Swim with neighbors.
(An optional 4th rule is that you should do what it takes to flee predators)

Recursion


In order to understand recursion you must first understand recursion.
Ok, imagine a function that calls itself.

Soulwire recursion toy (click)

Fractals


Self similar patterns based on mathematical sets. Formulas are relatively simple to define but generate a great level of complexity.

Cauliflower, self-similar patterns are also found in nature

Mandelbrot Fractal Set

Some resources about fractals here.

Artificial Life


Levitated – Walking things


Levitated – Legged Creatures


Cindermedusae by Marcin Ignac

Evolutionary algorithms


Darwin Hill (arbitrary selection)


Karl Sims – Evolved Virtual Creatures, Evolution Simulation, 1994

Process:

  • Generate the initial population of individuals randomly – first Generation
  • Evaluate the fitness of each individual in that population
  • Repeat on this generation until termination (time limit, sufficient fitness achieved, etc.):
  • Select the best-fit individuals for reproduction – parents
  • Breed new individuals through crossover and mutation operations to give birth to offspring
  • Evaluate the individual fitness of new individuals
  • Replace least-fit population with new individuals


Communion by Matt Pyke (Universal Everything) and Field.io – Marcus Wendt and Vera-Maria Glahn. Video here.

The installation is in a rectangular room with 6 separate projection on the walls creating almost a 360º environment. The final piece includes an array or evolving creatures going through stages of development – evolution from simple to complex with human like properties with generative behaviours


Want more of this? Read Out of control by Kelvin Kelly. Not extremely new, a little too Californian Ideology (pre dot com boom) but still good.

Patterns, patterns, patterns…

Assignment: Breaking Breakout

Create a digital artwork by modifying the code of a breakout clone.
It doesn’t have to playable or winnable, but it has to work and to be interesting to look at for at least 10 seconds.

Download source from here

-Don’t try to make another, completely different game starting from that code. Try to maintain some kind of breakout “feel”.

-Avoid importing external graphic resources. Keep it abstract.

-Try to start by tweaking the code, not with a specific idea.

Some works from the previous classes.

Clock: Snippets

How to make something happen every second (once).

int s;
int lastSecond;

void setup()
{
  size(200, 200);
}

//this function is called 60 times per second
void draw()
{
  s = second();

  //if the second was updated then do something once
  if (s != lastSecond)
  {
    println("A second elapsed");
    //update the last second
    lastSecond = s;
    //this variables will be the same until
    //the next second change in the clock
  }
}

Common problem: second() is related to the system clock but millis() count starts as you run the applet.
Sometimes you need them to be in sync, for example if you want an analog clock with a second arm moving smoothly:


int millisRolloverTime;
int prevSec;

void setup()
{
  size(100, 100);
  millisRolloverTime = 0;
}

void draw()
{
  background(0);
  
  int H= hour ();
  int M= minute ();
  int S= second ();

  if (prevSec != S) {
    millisRolloverTime = millis();
  }
  
  prevSec = S;
  int mils = millis() - millisRolloverTime;
  text(second() + " " + mils, 10, 20);
  
}

Here’s a classic analog clock:

// All Examples Written by Casey Reas and Ben Fry
// unless otherwise stated.
void setup() {
  size(200, 200);
  stroke(255);
  smooth();
}
void draw() {
  background(0);
  fill(80);
  noStroke();
  // Angles for sin() and cos() start at 3 o'clock;
  // subtract HALF_PI to make them start at the top
  ellipse(100, 100, 160, 160);
  float s = map(second(), 0, 60, 0, TWO_PI) - HALF_PI;
  float m = map(minute(), 0, 60, 0, TWO_PI) - HALF_PI;
  float h = map(hour() % 12, 0, 12, 0, TWO_PI) - HALF_PI;
  stroke(255);
  strokeWeight(1);
  line(100, 100, cos(s) * 72 + 100, sin(s) * 72 + 100);
  strokeWeight(2);
  line(100, 100, cos(m) * 60 + 100, sin(m) * 60 + 100);
  strokeWeight(4);
  line(100, 100, cos(h) * 50 + 100, sin(h) * 50 + 100);
}

Sounds

We didn’t cover this in class but playing sound files in processing is very easy:

import ddf.minim.*;
 
Minim minim;
AudioPlayer song;
 
void setup()
{
  size(100, 100);
 
  minim = new Minim(this);
 
  // this loads mysong.wav from the data folder
  song = minim.loadFile("mysong.wav");
  song.play();
}
 
void draw()
{
  background(0);
}
 
void stop()
{
  song.close();
  minim.stop();
 
  super.stop();
}

More info here:
http://code.compartmental.net/tools/minim/quickstart/

Like when using external images, to submit on processing you have to export as applet and include all the external files in the zip.