3D ball tracker

This page contains information on how to build, set up and run pf3dTracker, the particle-filter-based 3D ball tracker. The page is in the process of being written, so it's incomplete and the information you find on it might be inaccurate. Should you have any question or complaint, please write me an email at: mtaiana at isr*ist*utl*pt.

Get and build the source code

The source code of the tracker is part of the iCub repository, this page explains how to get it and build it. The tracker at the moment depends on the packages: YARP, iCub, OpenCV and iKin. In some occasions on this page you will encounter the variable $ICUB_ROOT. This variable is supposed to hold the path to the directory where you install the iCub software. The source code of the tracker is contained in the directory: $ICUB_ROOT/src/pf3dTracker, the binaries, after the building process, are stored in the directory: $ICUB_ROOT/bin. You should be able to invoke them from any directory.

Set the tracker up

An example configuration comes with the iCub software, so you can test the tracker even without creating the models which are presented hereafter. Beware that the tracker will not work well without customized models.

For the tracker to work properly, you need to create a colour model for the specific ball you want to track. This is done by grabbing images with the camera you want to use, cutting out the parts of the images where the ball is seen and pasting them all together in one file. The background of this image should be white, as white pixels are discarded when building the model histogram. The robustness of the tracker will depend on this model: you should include images in which the ball is seen under different lighting conditions. The more images you cut out, the better. If you can change the colour/brightness parameters of the camera, please do so before creating the colour model and use the same settings every time you use the tracker. Good setting include high saturation and a brightness that never makes parts of the ball appear as white or black.

Two examples of colour template images, for a yellow and a red ball, respectively:

You need to create a shape model for the specific ball you want to track. This is done using the Matlab script $ICUB_ROOT/src/pf3dTracker/matlab_files/write_initial_ball_points.m. You should set three parameters inside the script: R, R1 and R2. R is the radius of the ball you want to track, in millimetres. R1 and R2 are the radii that are used to project the inner and outer contour (see [[1]] for more details). If you want a precise estimate of the 3D position of the ball, you should set R1 and R2 close to the value of R (e.g. 10% difference). If you want the tracker to be able to withstand high accelerations of the ball, maintaining the number of particles used low, you should increase the difference up to 30% (this is the value I typically use). This script will create a file called something like: initial_ball_points_31mm_30percent.csv.

You need to create a dynamic model for the ball. Basically you have to fill in the dynamic matrix. I use a constant velocity model, with random acceleration. The data for this is stored in: models/motion_model_matrix.csv. I'm not sure that the tracker will work properly with other configurations of the motion model. For the dynamic model it is also quite important the parameter AccelStDev, that is set in the initialization file (see below).

You need to calibrate the camera you use i.e. estimate the intrinsic camera parameters. You can do that using camCalibConf, for example.

You need to customize the file that sets the tracker up on start up. The default initialization file is $ICUB_ROOT/app/pf3dTracker/pf3dTracker.ini. Here is an example:

 ####################################
 #configuration file for pf3dTracker#
 ####################################
 
 
 #############
 #module name#
 #############
 name                        /pf3dTracker
 
 #############################
 #parameters of the algorithm#
 #############################
 nParticles                  900
 #nParticles                 number of particles used
 accelStDev                  30
 #accelStDev                 standard deviation of the acceleration noise
 insideOutsideDiffWeight     1.5
 #insideOutsideDiffWeight    inside-outside difference weight for the likelihood function
 colorTransfPolicy           1
 #colorTransfPolicy          [0=transform the whole image | 1=only transform the pixels you need]
 
 
 #########################
 #port names and function#
 #########################
 inputVideoPort              /pf3dTracker/video:i
 #inputVideoPort             receives images from the grabber or the rectifying program.
 outputVideoPort             /pf3dTracker/video:o
 #outputVideoPort            produces images in which the contour of the estimated ball is highlighted
 outputDataPort              /pf3dTracker/data:o
 #outputDataPort             produces a stream of data in the format: X, Y, Z, likelihood, U, V, seeing_object
 outputParticlePort          /pf3dTracker/particle:o
 #outputParticlePort         produces data for the plotter. it is usually not active for performance reasons.
 outputAttentionPort         /pf3dTracker/attention:o
 #outputAttentionPort        produces data for the attention system, in terms of a peak of saliency.
 
 
 #################################
 #projection model and parameters#
 #################################
 #projectionModel [perspective|equidistance|unified]
 projectionModel             perspective
 
 #iCubLisboaLeftEye_Zoom_Lens_2009_05_19
 w 320
 h 240
 perspectiveFx 445.202
 perspectiveFy 445.664
 perspectiveCx 188.297
 perspectiveCy 138.496
 
 
 #######################
 #tracked object models#
 #######################
 #trackedObjectType [sphere|parallelogram]
 trackedObjectType           sphere
 trackedObjectColorTemplate  models/red_smiley_2009_07_02.bmp
 trackedObjectShapeTemplate  models/initial_ball_points_smiley_31mm_20percent.csv
 
 motionModelMatrix           models/motion_model_matrix.csv
 trackedObjectTemp           current_histogram.csv
 
 
 #######################
 #initialization method#
 #######################
 #initialization method [search|3dEstimate|2dEstimate]
 initializationMethod        3dEstimate
 #initial position [meters]
 initialX                       0
 initialY                       0
 initialZ                       0.5  
 
 
 ####################
 #visualization mode#
 ####################
 #circleVisualizationMode	[0=inner and outer cirlce | 1=one circle with the correct radious]
 #default 0. only applies to the sphere.
 circleVisualizationMode	1
 
 
 #########################
 #attention-related stuff#
 #########################
 #the tracker produces a value of likelihood at each time step.
 #this value can be used to infer if the object it is tracking is the correct one.
 #this is not a very robust way of doing so.
 #if likelihood>this value, then probably I'm tracking the object.
 likelihoodThreshold         0.005
 attentionOutputMax          300
 attentionOutputDecrease     0.99
 
 
 ##########################
 #image saving preferences#
 ##########################
 #save images with OpenCV?
 saveImagesWithOpencv        false
 #always use the trailing slash here.
 saveImagesWithOpencvDir     ./graphical_results/

Run the tracker

To run the tracker you need to:

  #run an image rectifier, in case you need it (cameras with a non-negligible distortion)
  camCalib --file iCubLisboaLeftEye320x240_2009_03_04.ini --name /icub/camcalib/left
  
  #run the tracker itself
  ./PF3DTrackerMain --file initialization.ini
  
  #start a viewer
  yarpview /viewer
  
  #connect all the ports
  yarp connect /icub/cam/left /icub/camcalib/left/in
  yarp connect /icub/camcalib/left/out /icub/PF3DTracker/videoIn
  yarp connect /icub/PF3DTracker/videoOut /viewer

Theoretical foundations of the tracker

If you want to know more on the theoretical ideas behind the tracker, please have a look at the papers on this page: [2].

Demo videos

If you want to watch videos and evaluate the performance of the tracker, please have a look at this page: [3].

ToDo

transform in C++ the piece of code that computes the initial ball points and read the value of the radius from the initialization file

read the color/illumination parameters from the camera before starting the tracker; set the desired parameters of the camera, then start the tracker; restore the original parameters when quitting

Make the tracker compute the histogram with Gaussian kernels instead of Dirac's

Make the tracker quit gracefully when asked to, instead of requiring multiple ctrl-c's

Make sure the tracker builds and works well on Windows

Document the code with doxygen

Make the tracker adaptive to different image sizes

Turn the number of particles into a parameter loaded at start time

Get rid of IPP dependency, using OpenCV

Start writing a wiki-based tutorial

(NOT NEEDED ANY MORE) Fix $PF3DTracker/conf/Find_IPP.cmake