Kinect RGB Demo v0.4.0

Research.KinectRgbDemoV4 History

Hide minor edits - Show changes to output

July 19, 2011, at 09:41 PM by 87.217.161.246 -
Changed lines 3-6 from:
! [[KinectRgbDemoV5|%red%NEW: version 0.5]]%%

(:redirect KinectRgbDemoV5:)
to:
! [[KinectRgbDemoV6|%red%NEW: version 0.6]]%%

(:redirect KinectRgbDemoV6:)
May 30, 2011, at 02:44 PM by 163.117.150.79 -
Added lines 3-4:
! [[KinectRgbDemoV5|%red%NEW: version 0.5]]%%
March 23, 2011, at 07:46 PM by 163.117.150.79 -
Added lines 3-4:
(:redirect KinectRgbDemoV5:)
February 28, 2011, at 01:04 PM by 163.117.150.243 -
Changed line 53 from:
sudo apt-get install libusb-1.0-0-dev libqt4-dev libgtk2.0-dev cmake libglew1.5-dev libcv-dev libhighgui-dev libcvaux-dev libgsl0-dev libglut3-dev
to:
sudo apt-get install libusb-1.0-0-dev libqt4-dev libgtk2.0-dev cmake libglew1.5-dev libcv-dev libhighgui-dev libcvaux-dev libgsl0-dev libglut3-dev libxmu-dev
February 21, 2011, at 11:34 PM by 95.18.193.233 -
Changed lines 108-109 from:
You will first need to install the libfreenect drivers. They are shipped with the archive, in the @@drivers@@ directory. When you plug your Kinect, specify the drivers location manually to the @@drivers\xbox nui motor@@ directory. If you missed it, then go into the device manager and update the drivers giving this location. More details there [[http://openkinect.org/wiki/Getting_Started_Windows|OpenKinect Windows]].
to:
You will first need to install the libfreenect drivers. They are shipped with the archive, in the @@drivers@@ directory. When you plug your Kinect, specify the drivers location manually to the @@drivers\xbox nui motor@@ directory. If you missed it, then go into the device manager and update the drivers giving this location. More details there [[http://openkinect.org/wiki/Getting_Started#Driver_installation|OpenKinect Windows]].
Added line 41:
Added line 45:
Changed lines 40-41 from:
You can have a look at the people detection feature on the following video:
to:
You can have a look at the 3D freehand reconstruction on the following video:
(:youtube 2ml8GiUPTao:)

And at the people detection feature on the following video:
Deleted lines 212-213:
(:youtube 2ml8GiUPTao:)
February 07, 2011, at 01:42 AM by 83.40.137.122 -
Changed lines 90-91 from:
You can get MacOSX universal (Intel only) binaries from there: [[http://sourceforge.net/projects/roboticslab/files/RGBDemo-0.4.0-Darwin.zip/download|RGBDemo-0.4.0-Darwin.zip]] (LGPL License).
to:
You can get MacOSX universal (Intel only) binaries from there: [[http://sourceforge.net/projects/roboticslab/files/RGBDemo-0.4.0-Darwin.dmg/download|RGBDemo-0.4.0-Darwin.dmg]] (LGPL License).
February 07, 2011, at 01:41 AM by 83.40.137.122 -
Changed lines 90-91 from:
You can get MacOSX universal (Intel only) binaries from there: [[http://sourceforge.net/projects/roboticslab/files/RGBDemo-0.4.0-Win32.zip/download|RGBDemo-0.4.0-Win32.zip]] (LGPL License).
to:
You can get MacOSX universal (Intel only) binaries from there: [[http://sourceforge.net/projects/roboticslab/files/RGBDemo-0.4.0-Darwin.zip/download|RGBDemo-0.4.0-Darwin.zip]] (LGPL License).
February 07, 2011, at 01:40 AM by 83.40.137.122 -
Changed lines 221-222 from:
to:
Calibration and config files will be loaded automatically is they are in the current directory.
February 07, 2011, at 01:39 AM by 83.40.137.122 -
Changed lines 151-152 from:
'''Not with Mac binaries''': if there is a @@grab1@@ directory in the current directory, it will be loaded automatically.
to:
'''Note with Mac binaries''': if there is a @@grab1@@ directory in the current directory, it will be loaded automatically.
February 07, 2011, at 01:39 AM by 83.40.137.122 -
Added lines 151-152:
'''Not with Mac binaries''': if there is a @@grab1@@ directory in the current directory, it will be loaded automatically.
February 07, 2011, at 01:37 AM by 83.40.137.122 -
Changed line 25 from:
* MacOSX Intel binaries [[http://sourceforge.net/projects/roboticslab/files/RGBDemo-0.4.0-Darwin.dmg/download|RGBDemo-0.4.0-Darwin.dmg]]
to:
* [[#macbin|MacOSX Intel binaries]]
February 07, 2011, at 01:36 AM by 83.40.137.122 -
Changed line 25 from:
* MacOSX Intel binaries (soon) [[http://sourceforge.net/projects/roboticslab/files/RGBDemo-0.4.0-Darwin.dmg/download|RGBDemo-0.4.0-Darwin.dmg]]
to:
* MacOSX Intel binaries [[http://sourceforge.net/projects/roboticslab/files/RGBDemo-0.4.0-Darwin.dmg/download|RGBDemo-0.4.0-Darwin.dmg]]
Added lines 88-91:
!!! Mac binaries [[#macbin]]

You can get MacOSX universal (Intel only) binaries from there: [[http://sourceforge.net/projects/roboticslab/files/RGBDemo-0.4.0-Win32.zip/download|RGBDemo-0.4.0-Win32.zip]] (LGPL License).

February 06, 2011, at 11:24 PM by 83.40.137.122 -
Changed line 25 from:
* MacOSX Intel binaries [[http://sourceforge.net/projects/roboticslab/files/RGBDemo-0.4.0-Darwin.dmg/download|RGBDemo-0.4.0-Darwin.dmg]]
to:
* MacOSX Intel binaries (soon) [[http://sourceforge.net/projects/roboticslab/files/RGBDemo-0.4.0-Darwin.dmg/download|RGBDemo-0.4.0-Darwin.dmg]]
February 06, 2011, at 11:20 PM by 83.40.137.122 -
Changed line 25 from:
* [[http://sourceforge.net/projects/roboticslab/files/RGBDemo-0.4.0-Darwin.dmg/download|RGBDemo-0.4.0-Darwin.dmg|MacOSX Intel binaries]]
to:
* MacOSX Intel binaries [[http://sourceforge.net/projects/roboticslab/files/RGBDemo-0.4.0-Darwin.dmg/download|RGBDemo-0.4.0-Darwin.dmg]]
February 06, 2011, at 11:19 PM by 83.40.137.122 -
Added lines 21-27:
!!! Download

* Source code as an archive [[http://sourceforge.net/projects/roboticslab/files/RGBDemo-0.4.0-Source.tar.gz/download|RGBDemo-0.4.0-Source.tar.gz]] (LGPL License)
* Source code on [[https://github.com/nburrus/rgbdemo|github]]
* [[http://sourceforge.net/projects/roboticslab/files/RGBDemo-0.4.0-Darwin.dmg/download|RGBDemo-0.4.0-Darwin.dmg|MacOSX Intel binaries]]
* [[#winbin|Windows binaries]]

Deleted lines 43-49:
!!! Download

* Source code as an archive [[http://sourceforge.net/projects/roboticslab/files/RGBDemo-0.4.0-Source.tar.gz/download|RGBDemo-0.4.0-Source.tar.gz]] (LGPL License)
* Source code on [[https://github.com/nburrus/rgbdemo|github]]
* [[http://sourceforge.net/projects/roboticslab/files/RGBDemo-0.4.0-Darwin.dmg/download|RGBDemo-0.4.0-Darwin.dmg|MacOSX Intel binaries]]
* [[#winbin|Windows binaries]]

February 06, 2011, at 11:18 PM by 83.40.137.122 -
Changed lines 41-42 from:
* [[#winbin|Windows binaries|]]
to:
* [[http://sourceforge.net/projects/roboticslab/files/RGBDemo-0.4.0-Darwin.dmg/download|RGBDemo-0.4.0-Darwin.dmg|MacOSX Intel binaries]]
* [[#winbin|Windows binaries
]]
February 06, 2011, at 11:16 PM by 83.40.137.122 -
Changed lines 41-42 from:
to:
* [[#winbin|Windows binaries|]]
Changed lines 95-96 from:
!!! Windows binaries
to:
!!! Windows binaries [[#winbin]]
February 05, 2011, at 09:55 PM by 83.40.137.122 -
Changed line 39 from:
* Source code as an archive [[http://sourceforge.net/projects/roboticslab/files/RGBDemo-0.4.0rc1-Source.tar.gz/download|RGBDemo-0.4.0rc1-Source.tar.gz]] (LGPL License)
to:
* Source code as an archive [[http://sourceforge.net/projects/roboticslab/files/RGBDemo-0.4.0-Source.tar.gz/download|RGBDemo-0.4.0-Source.tar.gz]] (LGPL License)
February 05, 2011, at 02:25 AM by 83.40.137.122 -
Added lines 203-204:
(:youtube 2ml8GiUPTao:)
Changed lines 209-210 from:
* Launch @@rgbd-people-tracker@@. More documentation to come.
to:
* Launch @@rgbd-people-tracker@@. You need to specify a configuration file. Here an example of full command line:
[@
build/bin/rgbd-people-tracker --calibration kinect_calibration.yml --config data/tracker_config.yml
@]

February 05, 2011, at 02:21 AM by 83.40.137.122 -
Changed lines 1-2 from:
(:title [=Kinect RGBDemo v0.4=]:)
to:
(:title [=Kinect RGBDemo v0.4.0=]:)
Added lines 21-25:
!!! New features since v0.4.0rc1
* Improved Windows event loop. Performance are now similar to Linux.
* Automatically load a calibration file if present in the current directory.
* Minor fixes.

Changed lines 98-99 from:
You will first need to install the libfreenect drivers. They are shipped with the archive, in the @@drivers@@ directory. When you plug your Kinect, specify the drivers location manually to the @@drivers\xbox@@ nui motor directory. If you missed it, then go into the device manager and update the drivers giving this location. More details there [[http://openkinect.org/wiki/Getting_Started_Windows|OpenKinect Windows]].
to:
You will first need to install the libfreenect drivers. They are shipped with the archive, in the @@drivers@@ directory. When you plug your Kinect, specify the drivers location manually to the @@drivers\xbox nui motor@@ directory. If you missed it, then go into the device manager and update the drivers giving this location. More details there [[http://openkinect.org/wiki/Getting_Started_Windows|OpenKinect Windows]].
Changed lines 151-152 from:
to:
'''New since RGBDemo v0.4.0''': if there is a @@kinect_calibration.yml@@ file in the current directory, it will be loaded automatically.
February 05, 2011, at 01:45 AM by 83.40.137.122 -
Changed line 34 from:
* Source code as an archive [[http://sourceforge.net/projects/roboticslab/files/rgbdemo-0.4.0rc1-Source.tar.gz/download|rgbdemo-0.4.0rc1-Source.tar.gz]] (LGPL License)
to:
* Source code as an archive [[http://sourceforge.net/projects/roboticslab/files/RGBDemo-0.4.0rc1-Source.tar.gz/download|RGBDemo-0.4.0rc1-Source.tar.gz]] (LGPL License)
Added lines 89-94:
!!! Windows binaries

You can get Win32 binaries from there: [[http://sourceforge.net/projects/roboticslab/files/RGBDemo-0.4.0-Win32.zip/download|RGBDemo-0.4.0-Win32.zip]] (LGPL License).

You will first need to install the libfreenect drivers. They are shipped with the archive, in the @@drivers@@ directory. When you plug your Kinect, specify the drivers location manually to the @@drivers\xbox@@ nui motor directory. If you missed it, then go into the device manager and update the drivers giving this location. More details there [[http://openkinect.org/wiki/Getting_Started_Windows|OpenKinect Windows]].

January 30, 2011, at 09:21 PM by 83.40.137.122 -
Changed line 39 from:
* The source includes a copy of OpenCV since Ubuntu packages are buggy. If you want to use an external OpenCV installation, disable the USE_EXTERNAL_OPENCV flag in CMake.
to:
* The source includes a copy of OpenCV since Ubuntu packages are buggy. If you want to use an external OpenCV installation (>= 2.2), disable the USE_EXTERNAL_OPENCV flag in CMake or directly use the @@./linux_configure_external_opencv.sh@@ script.
January 27, 2011, at 07:21 PM by 163.117.150.79 -
Added lines 192-196:

!!! People detection

* Launch @@rgbd-people-tracker@@. More documentation to come.

January 27, 2011, at 07:15 PM by 163.117.150.79 -
Added line 24:
* Better MacOSX support
January 27, 2011, at 06:43 PM by 163.117.150.79 -
Changed lines 19-20 from:
The project is divided into a library called @@nestk@@ and some demo programs using it. The library itself is easy to integrate to an existing project using @@cmake@@: just copy the nestk folder as a subfolder of your project and you should be able to start working with Kinect data.
to:
The project is divided into a library called @@nestk@@ and some demo programs using it. The library itself is easy to integrate to an existing project using @@cmake@@: just copy the nestk folder as a subfolder of your project and you should be able to start working with Kinect data. You can get more information on the [[KinectUseNestk|nestk page]].
January 27, 2011, at 04:49 PM by 163.117.150.79 -
Changed lines 36-37 from:
!!! Compilation
to:
!!! Compilation on Linux (Ubuntu)
Changed lines 61-63 from:
You will need an install of QT and libusb > 1.0.

You might experience some issues with
libusb. If the configure script cannot find it, supposing you installed it in your @@$HOME@@ directory, you can try:
to:
You will need:
* An
install of [[http://qt.nokia.com/downloads/qt-for-open-source-cpp-development-on-mac-os-x|QT]]
* An install of
libusb. You can install it manually from [[http://www.as3kinect.org/distribution/osx/libusb_1.0.pkg|AS3 Download]] or using the provided @@macosx_configure.sh@@ script below.

Then run the following commands
:
Changed lines 67-70 from:
cmake -DLIBUSB_1_INCLUDE_DIR=$HOME/libusb/include -DLIBUSB_1_LIBRARY=$HOME/libusb/lib/libusb-1.0.dylib build
to:
tar xvfz rgbdemo-0.4.0rc1-Source.tar.gz
cd rgbdemo-0.4.0rc1-Source
./macosx_configure.sh
./macosx_build.sh
Changed lines 72-79 from:
to:
The configure script might ask for libusb installation. Say yes if you don't have it installed.

If you still experience some issues with libusb, or have a custom install, you can try:
[@
cmake -DLIBUSB_1_INCLUDE_DIR=$HOME/libusb/include -DLIBUSB_1_LIBRARY=$HOME/libusb/lib/libusb-1.0.dylib build
@]
supposing that you have it installed in @@$HOME/libusb@@.

Changed lines 29-30 from:
(:youtube 2ml8GiUPTao:)
to:
(:youtube nnCDOKLuu0g:)
Added lines 1-178:
(:title [=Kinect RGBDemo v0.4=]:)

(:htoc:)

[[<<]]

!! Demo software to visualize, calibrate and process Kinect cameras output

This software was partly developed in the [[http://roboticslab.uc3m.es | RoboticsLab]] and aims at providing a simple toolkit to start playing with Kinect data. Features include:
* Grab kinect images and visualize / replay them
* Calibrate the camera to get point clouds in metric space
* Export to meshlab/blender using .ply files
* Demo of 3D scene reconstruction using a freehand Kinect
* Demo of people detection and localization
* Linux, MacOSX and (partial) Windows support

The code is based on the [[https://github.com/OpenKinect/libfreenect/|freenect library]] and [[http://opencv.willowgarage.com|OpenCV]].

The project is divided into a library called @@nestk@@ and some demo programs using it. The library itself is easy to integrate to an existing project using @@cmake@@: just copy the nestk folder as a subfolder of your project and you should be able to start working with Kinect data.

!!! New features since v0.3.1
* New demo: experimental people detection and height estimation
* Windows support greatly improved (thanks to Armin Zürcher)
* Optional calibration without distortions for the depth image (--no-depth-undistort)
* Various bug fixes

You can have a look at the people detection feature on the following video:

(:youtube 2ml8GiUPTao:)

!!! Download

* Source code as an archive [[http://sourceforge.net/projects/roboticslab/files/rgbdemo-0.4.0rc1-Source.tar.gz/download|rgbdemo-0.4.0rc1-Source.tar.gz]] (LGPL License)
* Source code on [[https://github.com/nburrus/rgbdemo|github]]

!!! Compilation

* The source includes a copy of OpenCV since Ubuntu packages are buggy. If you want to use an external OpenCV installation, disable the USE_EXTERNAL_OPENCV flag in CMake.
* Install required packages, e.g. on Ubuntu 10.10:
[@
sudo apt-get install libusb-1.0-0-dev libqt4-dev libgtk2.0-dev cmake libglew1.5-dev libcv-dev libhighgui-dev libcvaux-dev libgsl0-dev libglut3-dev
@]

* Untar the source, use the provided scripts to launch cmake and compile:
[@
tar xvfz rgbdemo-0.4.0rc1-Source.tar.gz
cd rgbdemo-0.4.0rc1-Source
./linux_configure.sh
./linux_build.sh
@]

* Note from Stéphane Magnenat about compilation on Ubuntu 10.04 :
[@
> For information, to compile KinectRgbDemo from [1] under Ubuntu 10.04, I had
> to add png12 to all target_link_libraries(...) as well as to had GLU to
> target_link_libraries(...) of rgbd-viewer.
@]

!!! Compilation on Mac

You will need an install of QT and libusb > 1.0.

You might experience some issues with libusb. If the configure script cannot find it, supposing you installed it in your @@$HOME@@ directory, you can try:
[@
cmake -DLIBUSB_1_INCLUDE_DIR=$HOME/libusb/include -DLIBUSB_1_LIBRARY=$HOME/libusb/lib/libusb-1.0.dylib build
@]

!!! Compilation on Windows

It has been tested with MinGW and Visual Studio 10 so far. Here is a step-by-step procedure for MinGW:
* Install QT opensource for Windows. This will also install MinGW.
* Add C:\Qt\2010.05\MinGW\bin to the Path environment variable
* Install and run @@cmake@@ on rgbdemo
* Open the CMakeLists.txt in @@QtCreator@@ or compile manually using mingw-make.

!!! Running the viewer without calibration

* Binaries are in the @@build/bin/@@ directory, you can give it a try without calibration using:
[@
build/bin/rgbd-viewer
@]

If you get an error such as:

[@
libusb couldn't open USB device /dev/bus/usb/001/087: Permission denied.
libusb requires write access to USB device nodes.
FATAL failure: freenect_open_device() failed
@]

Give access rights to your user with:
[@
sudo chmod 666 /dev/bus/usb/001/087
@]
Or install the udev rules provided by libfreenect.

!!! Calibrating your Kinect

A sample calibration file is provided in @@data/kinect_calibration.yml@@. However, you should be able to get a more accurate mapping by estimating new parameters for each Kinect. Below is the procedure I follow.

'''1.''' Build a calibration pattern as shown in [[KinectCalibration]]. You can use the @@Chessboard_A4.pdf@@ or @@Chessboard_A3.pdf@@ file in the @@data/@@ directory for this. I recommend printing the chessboard on a sheet of paper and glue it on a peace of carton. It is not necessary anymore to cut the carton around the paper.

'''2.''' Grab some images of your chessboard using the viewer (File / Grab frame or Ctrl-G). WARNING: you need to grab images in Dual IR/RGB more (enable it in the Capture menu). By default it will save them into directories @@grab1/view????@@. These directories contain the raw files, @@raw/color.png@@, @@raw/depth.yml@@, @@raw/intensity.png@@ that corresponds to the color image, the depth image (in meters), and the IR image normalized to grayscale. You will also get an additional @@raw/depth.png@@ which is the depth image normalized to grayscale.

To get an optimal calibration, grabbed images should ensure the following:
* Cover as most image area as possible. Especially check for coverage of the image corners.
* Try to get the chessboard as close as possible to the camera to get better precision.
* For depth calibration, you will need some images with IR and depth. But for stereo calibration, the depth information is not required, so feel free to cover the IR projector and get very close to the camera to better estimate IR intrinsics and stereo parameters. The calibration algorithm will automatically determine which grabbed images can be used for depth calibration.
* Move the chessboard with various angles.
* I usually grab a set of 30 images to average the errors.
* Typical reprojection error is < 1 pixel. If you get significantly higher values, it means the calibration failed.

'''3.''' Run the calibration program:
[@
build/bin/calibrate_kinect_ir --pattern-size 0.025 grab1
@]
The pattern size correspond to the size in meters of one chessboard square. It should be 0.025 (25mm) for the A4 model.

This will generate the @@kinect_calibration.yml@@ file storing the parameters for the viewer, and two files @@calibration_rgb.yaml@@ and @@calibration_depth.yaml@@ for ROS compatibility.

!!! Running the viewer with calibration

* Just give it the path to the calibration file:
[@
build/bin/rgbd-viewer --calibration kinect_calibration.yml
@]

* You should get a window similar to this:
%height=240px% Attach:viewer_output_main_v2.png

* The main frame is the color-encoded depth image. By moving the mouse, you can see the distance in meters towards a particular pixel. Images are now undistorted.

* You can filter out some value and normalize the depth color range with the filter window (Show / Filters). The Edge filter is recommended.
%height=240px% Attach:viewer_output_filters.png

* You can get a very simple depth-threshold based segmentation with Show / Object Detector
%height=240px% Attach:viewer_output_detection.png

* You can get a 3D view in Show / 3D Window.
%height=240px% Attach:viewer_output_view3d_cloud.png

* By default you get a grayscale point cloud. You can activate color:
%height=240px% Attach:viewer_output_view3d_cloud_color.png

* And finally textured triangles :
%height=240px% Attach:viewer_output_view3d_triangles.png

* You can also save the mesh using the @@Save current mesh@@ button, it will store in into a @@current_mesh.ply@@ file that you can open with Meshlab [[http://meshlab.sourceforge.net/|Meshlab]]:
%height=320px% Attach:viewer_output_meshlab.png

* Or import into [[http://www.blender.org/|Blender]]:
%height=320px% Attach:viewer_output_blender.png

* The associated texture is written into a @@current_mesh.ply.texture.png@@ file and can be loaded into the UV editor in Blender.

!!! Getting Infrared Images

* You can activate the IR mode in the capture menu. There is also a dual RGB/IR mode alternating between the two modes.
%height=320px% Attach:viewer_output_ir.png

!!! Replay mode

* You can grab RGBDImages using the @@File/Grab Frame@@ command. This stores the files into @@viewXXXX@@ directories (see the Calibration section), that can be replayed later using the fake image grabber. This can be activated using the @@--image@@ option:
[@
build/bin/rgbd-viewer --calibration kinect_calibration.yml --image grab1/view0000
@]

* You can also replay a sequence of images stored in a directory with the @@--directory@@ option:
[@
build/bin/rgbd-viewer --calibration kinect_calibration.yml --directory grab1
@]
This will cycle through the set of viewXXXX images inside the @@grab1@@ directory.

!!! Interactive scene reconstruction

* You can try an experimental interactive scene reconstruction mode using the @@build/bin/rgbd-reconstructor@@ program. This is similar to the interative mapping of [[http://ils.intel-research.net/projects/rgbd|Intel RGBD]] but still in a very preliminar stage. The relative pose between image captures is determined using feature points matching and mean squares minimization.

In this mode, point clouds will progressively be aggregated in a single reference frame using a Surfel representation.