Robotic Painting with a Line of Lights

This post documents a method of generating light paintings using a robot moving a linear array of lights through space. The robot moves a simple stick tool which contains 60 lights stretched along its 1 meter length. Each light can be individually controlled, any one of 16 million colors, and that color can vary over time.

The image is recorded using a camera which keeps the shutter open during the entire motion of the robot. Everything happens in a very dark room so the great majority of the light collected comes from the lights only. The result is a single image showing the entire path of the line of lights.

Grasshopper Definition

The robot motion is controlled by a simple Grasshopper definition using the plug-in Kuka|prc. The motion is driven by an input curve and a few parameters described below.

There are two parts to the definition. One simulates and generates the robot code. You work with this part of the definition to assure that the robot can reach the entire curve without hitting joint rotation limits and that the tool does not collide with the robot table.

The other part of the definition is used to simulate the light effect prior to running on the robot. This is important because it is impossible to visualize the path of the lights as they rotate traveling along the input curve.

See the post Working with Planes in Kuka | prc for information on various methods of dividing curves and driving the robot.

Parameters

The adjustable parameters are as follows:

• Curve: The curve the robot follows. The robot face plate and tool rotate so the line of lights is always perpendicular to the curve at every point. It's this twisting motion of the line which generates the visual interest. 
• Curve Divisions: The entire length of the curve is divided into the specified number of segments. By varying this parameter you can get a great variety of outputs. In general, the most interesting results are obtained using low values. 
• Tool Length: This is fixed for the Taubman College tool which is 1 Meter long. 
• Lights on Tool: To simulate accurately you need to tell Grasshopper how many lights will be illuminated on the stick. This can be either 15, 30 or 60. The actual number of lights used is set on the tool itself. 

The Curve obviously has a major impact. But it's surprising that using different curves with low division counts you can get some amazing effects. Here are a few sample curves drawn in Rhino:

It's of course also possible to wire in a parametric curve generated by Grasshopper.

The strip of LEDs has 60 lights per meter which is the length of the stick. The Arduino code allows you to set the number of LEDs which are illuminated. By using every every 1/12th you can have 5 lights show along the length. Using 1/4th you can have 15 as shown below:

Every other LED produces 30 and every LED is 60:

 

Here's are examples using the same curve with a different division count:

Here's a different curve with two different division counts:

 

The End Effector

The tool attached to the robot is basically a stick made of 1/2" Baltic Birch plywood. There's a small box to house the electronics. Milled into the stick are some grooves for the light strip and the wires.

The lights are NeoPixel strips made by Adafruit. Everything is driven by an Arduino.
There are a few simple controls - two push buttons, and three knobs:

Here's a video demo of the controls (description below):

Push-Button 1

Use this to change between the 5 different light effects. Press the button to change to the next mode - the sequence is as follows:

1. Constant color for all lights - same over time. Use Knob 1 to set the fixed color. 
2. Unique color for all lights - same over time. Use Knob 1 to shift the colors along the line of lights. 
3. Unique color for all lights - rotates along the strip over time. Use Knob 1 to shift the colors along the lights. Use Knob 2 to set the rate of change. 
4. Random colors for all lights - same over time. 
5. Random colors for all lights, rotates along the strip over time. Use Knob 2 to set the rate of change. 

Push-Button 2

Use this to control the number of lights which are illuminated on the stick. As you press the button the number of lights lit will switch as follows:

• 5 LEDs 
• 15 LEDs 
• 30 LEDs 
• 60 LEDs

Knob 1

Changes the color of each light. Rotating the knob moves the lights through hue space. All the way counter-clockwise is 0, fully clockwise is 360, where:

0=Red, 60=Yellow, 120=Green, 180=Cyan, 240=Blue, 300=Magenta, 360=0=Red.

Image source

How the change effects each light depends on push button 1 (see above).

Knob 2

Use this to change the intensity (brightness) of the lights. All the way to the left is dimmest, all the way to the right - don sunglasses.

Knob 3

Use this knob to specify the rate of change of the light colors over time. Slowest is all the way counter-clockwise. Fastest is fully clockwise.

The Robot

These examples were generated on a small Kuka Agilus robot. Like all robots it has a limited reach and range of motion. There are a few controls in Grasshopper you can use to allow the robot to reach every point on the curve.

Initial Position: This is key. You need to pose the robot in a neutral position for the start of the motion. None of the joints should be near their limits and it should be roughly centered on your path. You set this in the Kuka | prc Settings, on the Advanced Page.

Middle Point Location: This is the most important tool for solving joint limit problems. If you encounter a position in the path where a joint turns red, leave the simulation slider where that joint is red, and move the middle point location a bit. Usually you can find a spot where the joint is no longer in a limit. This can be up or down, or closer or farther from the robot. Note: There are curves which are impossible to fully reach. But if your curves are not larger than the samples provided you'll probably be able to find a position which works.

Arduino Hardware and Software Details

This section describes the wiring and software setup.

Below is the prototype I built using components I had on hand to test with. A slide potentiometer for the hue selection, a cheap pot for knob 2, one nice push button and one piece of junk. These were replaced with better components in the final tool.

The NeoPixels need current. 60 pixels at full brightness need almost 4 amps. An Arduino supplies only 0.5 amps as does a 9V battery. So an external power supply is necessary. This is the one I prefer: 5V 4A (4000mA) switching power supply.

The color is computed in Hue space but needs to be provided to the LEDs as RGB values. I used some great code I found online to do this: Why every LED light should be using HSI colorspace.

The prototype used an Arduino Uno. The final version needed a smaller footprint micro-controller so I used an Arduino Micro.

The face plates are laser cut acrylic. I got the buttons here and the pots here.

My research assistant Celine Schlueter did all the soldering:

Conclusion

The process needs some more development but I like the initial tests of this new tool.

For some other image painting tests see this post.

Update

I've done more work on this project. The results are here: More Robot Driven Light Paintings

Read More

Robotic Painting with Light

I got interested in trying to "paint" with light. That is, use long exposure photography with a moving light source to generate images. The concept is simple: if you put a camera on a tripod, turn off all the lights, open the camera shutter and keep it open, move a light source around in front of the camera, then close the shutter, you'll wind up with a single image of the light moving in one continuous stream.

If you carefully choreograph the color changes, motion, and turning on and off of lights via software you can generate some interesting "light paintings".

What started me on this idea was some work we do in my Robotics course in Taubman College at the University of Michigan. Here's a post on that: Robot Motion Analysis Using Light.

Hardware

My plan for this required a robot to move the lights around, and some hardware to control the light. I'm using an 8 x 8 array of lights: NeoPixels from Adafruit. This is a really nice unit and only requires 3 pins from the Arduino.

The lights are really bright (understatement) and the programming of them is very easy. The micro-controller hardware which controls the lights is an Arduino Mega (an Uno would be fine as well) with the WiFi Shield. The image below shows the beginning of the prototype. As you can see the wiring is very simple (one resistor, one capacitor and the light array):

The robot used is a Kuka Agilus KR-6. I used the back robot in this picture.

The design of the tool which mounts to the robot looks like this, in model form (pixels, prototyping board, and Arduinos from top to bottom):

The finished tool ready to mount: 

And mounted to the robot: 

Software

This works using Grasshopper and Kuka|prc to control the robot, sample the image, and send the code to the Arduino to drive the pixels.

My first idea was to take any image (photograph, picture of a painting, etc), sample it at points which exactly match the LED light panel pixel spacing, then use that sampled data to illuminate the light grid. Those lights form one small part of the overall image. The robot moves the light array and shows new colors each time.

Single LED light panel (yellow) with 6x6 grid of robot moves (blue):

Sampled points in the image (these are the lower left corner of the LED light panel).

Robot moves in a 6x6 grid with different colors each time:

All the coordination of motion and data is managed through Grasshopper. Here's the definition - as you can see it's pretty simple - not many components. The light blue is the robot control. The light cyan generates the Arduino code:

The Grasshopper definition lets you generate a preview of what the pixelated image will look like with baked Rhino geometry. Here are a few examples:

Van Gogh's Starry Night:

A Self-Portrait:

The robot sequence is this:

1. Move to a new position. 
2. Turn on the lights. 
3. Pause for a bit to expose the image. 
4. Turn off the lights. 
5. Repeat until the entire image is covered. 

My first pass at this loaded all the color data, for every move, into the Arduino memory at the start. The very low memory capacity of an Arduino, even a Mega, made only 30 or so moves possible. I wanted more than that, so I changed to using a Arduino WiFi Shield and transmit the color data to the tool at each motion stop point. This places no memory limit on the size of the area that's covered.

Interfacing with the PLC

A robot has something called a Programmable Logic Controller (PLC). This is what allows the robot to interface to external inputs and outputs. Some example inputs are things like limit switches, and cameras. Outputs are things like servo motors and warning lights.

A program can also use the PLC to find out about the state of the robot - the current position and orientation of the tool, joint angles, etc.

For this project the PLC is used to track when the robot is moving. This is done by having the robot program set a bit of memory in the PLC to indicate the robot has arrived at a new position. Then it waits for a short amount of time (as the lights are turned on). Then that memory bit is turned off.

The Grasshopper definition monitors the PLC and informs the Arduino when to turn on and off. It also sends the new color data over WiFi.

First Test Images

The first attempts were interesting. It proves the concept works - which was very exciting to see for the first time. But they also clearly show there is room for improvement!

Here's the gradient example. I cropped this image wide so you can see the context - robot workcell, robot in the background, window behind, etc. The robot moved the light array 36 times to produce this image. So that's 36 moves times 64 pixels per move or 2304 pixels total. The tool was slightly rotated which results in the grid varying a bit. That's an easy fix - but it's interesting in the image below because it makes it very clear where each move was.

Here's a self portrait. You can see a variation in the color intensity between the moves of the robot. I realized this is because the robot is not staying in each location for the same time. That's because the robot checks if it should move every 200ms. And the exposure time is only 500ms. So when the robot is triggered to move early the exposure can be nearly 40% different. This is also an easy fix - I'll just sample every 10ms. The PLC also supports interrupt driven notifications - which would be even better.

Here's a portion of Vermeer's Girl with a Pearl Earring. This image again shows the variation in intensity. It also shows where colors get very dark the variation in RGB intensity of the pixels can become an issue. Amusingly she has red-eye! Obviously in all the images more pixels are needed to make them look good. That's accomplished with more robot moves but also with moving between the pixels. It's possible to quadruple the resolution of each of these images by simply moving into the space between the pixels and update the color values.

Van Gogh's Starry Night. With many more pixels this could be a nice image. It would also be interesting to experiment with moving the robot back a foot or so, reducing the intensity, and sweeping the robot with the lights on but the colors changing. So you'd have an overlay of the two. You could get a layered look, and a depth, and a sense of motion.

Next Steps

I'll be doing more with this in the future. One goal is to see if I can make Chuck Close style images. Here's an example of one of his self portraits:

His work is much, much more interesting (more examples are available here). Each cell in the image is multi-colored. Also the rectangular array of cells is rotated 45 degrees rather than vertical and horizontal. His cells are also not all circular - but elliptical and triangular. My plan is to put the correct color in the center of each cell. Then generate a new color and move that in a circle around the center point. His cells have 3 or 4 colors each, and also cross the cell boundaries as he sees fit. That I cannot do. But it'll be interesting to see how far I can get.

It's also possible to leave the LEDs on and move the robot. Or move the robot through space as to generate 2D images of painting in 3D. I'll be experimenting with many techniques (software changes) and I'll have another updated post in the future.

Other Methods

See this post for some other experiments I've done: Robotic Painting with a Line of Lights.

Read More

Robot Motion Analysis Using Light

One of the assignments in my robotics course at Taubman College is to analyze different types of robotic motion using long exposure photography. This project allows students to visualize and understand the movement types the robot has available, and how various motion interpolation settings affect that motion.

    

 

Students draw a curve in 3D space made up of linear segments. They then take long exposure photographs of the robot moving through the control points of the curve using the different motion types and approximation (interpolation) settings. They also time each run to understand the effects on execution time. They then systematically compare and contrast the movements types and settings.

Students learn the differences between point-to-point (PTP), linear (LIN), circular (CIR), and spline (SPL) moves. They also learn about the C_PTP, C_DIS, C_VEL, and C_ORI approximation settings.

Motion Analysis Results

The following images and timings are the work of Taubman students Marshall Hebert, Alex Waga, Yinying Chen, and Kati Albee.

Linear Motion was tracked using no interpolation, C_DIS of 50mm, C_DIS of 100mm, C_VEL of 50% and C_VEL of 100%

Spline motion was tracked using no interpolation, C_DIS of 50mm, and C_DIS of 100mm.

Point-to-point motion was tracked using no interpolation, C_PTP of 50% and C_PTP of 100%.

Execution Times

The following table lists the execution times for each run above.

Light Tool Details

The images were recorded using a simple tool made from an Arduino Micro - a popular small micro controller, and a RGB LED.

First I prototyped the setup using a regular Arduino Uno and a prototyping board.

Once I had it working I switched over the Arduino Micro and a Adafruit Perma-Prototype board:

The final step was to install the board into a fixture that could be bolted to the robot:

A push button on the tool lets the user cycle through 12 different standard colors: Red => Yellow => Green => Cyan => Blue => Magenta with half-steps in between each of those.

The simple Arduino code used on the tool follows:
/* 
 * Cycle thru 12 standard RGB colors at the press of a button
 */
const int switchPin = 5;  // pin the push button is attached to
const int ledPinR   = 9;  // pwm pin with red led
const int ledPinG   = 10; // pwm pin with green led
const int ledPinB   = 11; // pwm pin with blue led

int hue = 0; // Incremented to cycle 0-11
int r = 255; // Start with red
int g = 0;
int b = 0;

void setup() { 
  pinMode(ledPinR, OUTPUT);
  pinMode(ledPinG, OUTPUT);
  pinMode(ledPinB, OUTPUT);
  pinMode(switchPin, INPUT_PULLUP);
}

void loop()  { 
   // Switch colors on a button press
   if (digitalRead(switchPin) == LOW) {
     hue = (++hue > 11) ? 0 : hue;
     switch (hue) {
       case 0: // Red
         r = 255; g = 0; b = 0; break;
       case 1: // Orange
         r = 255; g = 128; b = 0; break;
       case 2: // Yellow
         r = 255; g = 255; b = 0; break;
       case 3: // Yellow green
         r = 128; g = 255; b = 0; break;
       case 4: // Green
         r = 0; g = 255; b = 0; break;
       case 5: // Green blue
         r = 0; g = 255; b = 128; break;
       case 6: // Cyan
         r = 0; g = 255; b = 255; break;
       case 7: // Blue green
         r = 0; g = 128; b = 255; break;
       case 8: // Blue
         r = 0; g = 0; b = 255; break;
       case 9: // Light Magenta
         r = 128; g = 0; b = 255; break;
       case 10: // Magenta
         r = 255; g = 0; b = 255; break;
       case 11: // Light red
         r = 255; g = 0; b = 128; break;   
    }
    // Wait for the button release
    while (digitalRead(switchPin) == LOW)
    {
      delay(10);
    }
  }
  
  // Update the LED
  analogWrite(ledPinR, r);  
  analogWrite(ledPinG, g); 
  analogWrite(ledPinB, b); 
  delay(20);
}

For other Arduino based images in light see Robotic Painting with Light.

Read More

Robot Motion Analysis Using Light

One of the assignments in my robotics course at Taubman College is to analyze different types of robotic motion using long exposure photography. This project allows students to visualize and understand the movement types the robot has available, and how various motion interpolation settings affect that motion.

    

 

Students draw a curve in 3D space made up of linear segments. They then take long exposure photographs of the robot moving through the control points of the curve using the different motion types and approximation (interpolation) settings. They also time each run to understand the effects on execution time. They then systematically compare and contrast the movements types and settings.

Students learn the differences between point-to-point (PTP), linear (LIN), circular (CIR), and spline (SPL) moves. They also learn about the C_PTP, C_DIS, C_VEL, and C_ORI approximation settings.

Motion Analysis Results

The following images and timings are the work of Taubman students Marshall Hebert, Alex Waga, Yinying Chen, and Kati Albee.

Linear Motion was tracked using no interpolation, C_DIS of 50mm, C_DIS of 100mm, C_VEL of 50% and C_VEL of 100%

Spline motion was tracked using no interpolation, C_DIS of 50mm, and C_DIS of 100mm.

Point-to-point motion was tracked using no interpolation, C_PTP of 50% and C_PTP of 100%.

Execution Times

The following table lists the execution times for each run above.

Light Tool Details

The images were recorded using a simple tool made from an Arduino Micro - a popular small micro controller, and a RGB LED.

First I prototyped the setup using a regular Arduino Uno and a prototyping board.

Once I had it working I switched over the Arduino Micro and a Adafruit Perma-Prototype board:

The final step was to install the board into a fixture that could be bolted to the robot:

A push button on the tool lets the user cycle through 12 different standard colors: Red => Yellow => Green => Cyan => Blue => Magenta with half-steps in between each of those.

The simple Arduino code used on the tool follows:
/* 
 * Cycle thru 12 standard RGB colors at the press of a button
 */
const int switchPin = 5;  // pin the push button is attached to
const int ledPinR   = 9;  // pwm pin with red led
const int ledPinG   = 10; // pwm pin with green led
const int ledPinB   = 11; // pwm pin with blue led

int hue = 0; // Incremented to cycle 0-11
int r = 255; // Start with red
int g = 0;
int b = 0;

void setup() { 
  pinMode(ledPinR, OUTPUT);
  pinMode(ledPinG, OUTPUT);
  pinMode(ledPinB, OUTPUT);
  pinMode(switchPin, INPUT_PULLUP);
}

void loop()  { 
   // Switch colors on a button press
   if (digitalRead(switchPin) == LOW) {
     hue = (++hue > 11) ? 0 : hue;
     switch (hue) {
       case 0: // Red
         r = 255; g = 0; b = 0; break;
       case 1: // Orange
         r = 255; g = 128; b = 0; break;
       case 2: // Yellow
         r = 255; g = 255; b = 0; break;
       case 3: // Yellow green
         r = 128; g = 255; b = 0; break;
       case 4: // Green
         r = 0; g = 255; b = 0; break;
       case 5: // Green blue
         r = 0; g = 255; b = 128; break;
       case 6: // Cyan
         r = 0; g = 255; b = 255; break;
       case 7: // Blue green
         r = 0; g = 128; b = 255; break;
       case 8: // Blue
         r = 0; g = 0; b = 255; break;
       case 9: // Light Magenta
         r = 128; g = 0; b = 255; break;
       case 10: // Magenta
         r = 255; g = 0; b = 255; break;
       case 11: // Light red
         r = 255; g = 0; b = 128; break;   
    }
    // Wait for the button release
    while (digitalRead(switchPin) == LOW)
    {
      delay(10);
    }
  }
  
  // Update the LED
  analogWrite(ledPinR, r);  
  analogWrite(ledPinG, g); 
  analogWrite(ledPinB, b); 
  delay(20);
}

Read More