Friday, June 02, 2017

Today's Paper: Learning to Generate Long-term Future via Hierarchical Prediction

Great demonstrations of powerful combination of recent deep neural networks applications - stacked hourglass networks for human pose estimation [Newell 2016] and visual analogy learning [Reed 2015], enabled amazing results in future video prediction.

Learning to Generate Long-term Future via Hierarchical Prediction, ICML 2017 (to appear) [Arxiv]

Great demo videos are available online.

We propose a hierarchical approach for making long-term predictions of future frames. To avoid inherent compounding errors in recursive pixel-level prediction, we propose to first estimate high-level structure in the input frames, then predict how that structure evolves in the future, and finally by observing a single frame from the past and the predicted high-level structure, we construct the future frames without having to observe any of the pixel-level predictions. Long-term video prediction is difficult to perform by recurrently observing the predicted frames because the small errors in pixel space exponentially amplify as predictions are made deeper into the future. Our approach prevents pixel-level error propagation from happening by removing the need to observe the predicted frames. Our model is built with a combination of LSTM and analogy based encoder-decoder convolutional neural networks, which independently predict the video structure and generate the future frames, respectively. In experiments, our model is evaluated on the Human3.6M and Penn Action datasets on the task of long-term pixel-level video prediction of humans performing actions and demonstrate significantly better results than the state-of-the-art.
A one shot illustration of their approach can be:

(A picture in the paper, with my annotation of leveraged existing work references.)

Thursday, April 27, 2017

ROS Subscriber may be immediately unsubscribed

A caveat for programming in C++ for ROS (Robot Operating System).  The core ROS architecture is of the pub-sub messaging framework, where you can publish on a topic to send messages and subscribe on a topic to receive messages, among ROS nodes.

Both publishing:
ros::NodeHandle nh;
ros::Publisher pub = nh.advertise("some_topic", ..);pub.publish(message);
and subscribing:
ros::Subscriber sub = nh.subscribe("some_topic", .., callback);
are fairly simple to write.  (The callback can be either a function or method, to handle the message received.)

You must hold the returned subscriber object till the end of running your application.  It is a pitfall you can simply ignore it as follows:
nh.subscribe("some_topic", .., callback);
This might result in the deletion of the subscriber object, calling its destructor to unsubscribe from the topic...  You won't have any callback, then.  ;-<

Tuesday, April 11, 2017

Set up ROS Kinetic in macOS


The official wiki] gives you an extensive (it shouldn't be ideally, though) explanation in installing ROS Kinetic in macOS (f.a.k.a. MacOS X).  Its Trouble Shooting section is mandatory (but not sufficient) to read through.

One of the largest pitfalls in the process is with Qt5 and its python plugin.  Qt is a popular cross-platform UI development framework, on which many applications depend.  Installing `qt` with Homebrew provides Qt4 but we need Qt5 for ROS, which may conflict with Qt4 at least in the Homebrew level.

In my environment to have `Desktop-Full Install` (`Desktop Install (recommended)` might be a lot simpler), at least following was necessary to be set up:
$ brew install --force qt5
$ brew link --force qt5
$ brew install pyqt5 --with-python
$ ln -s /usr/local/Cellar/qt/5.8.0_2/mkspecs /usr/local/mkspecs
$ ln -s /usr/local/Cellar/qt/5.8.0_2/plugins /usr/local/plugins
$ ln -s /usr/local/share/sip/Qt5 /usr/local/share/sip/PyQt5
as well as:
$ brew tap homebrew/science
$ brew install python ; brew linkapps python
$ brew install --force opencv3 --with-contrib ...
$ echo /usr/local/opt/opencv3/lib/python2.7/site-packages >> /usr/local/lib/python2.7/site-packages/opencv3.pth
$ brew install cmake sip poco eigen qhull pcl libogg theora gazebo8
See the Trouble Shooting section and an issue raised in Homebrew GitHub for more details.  Once prepared these prerequisites, you may go with the `rosdep` step in the wiki but with a `--skip-keys` option:
$ rosdep install --from-paths src --ignore-src --rosdistro kinetic -y \
        --skip-keys "libqt5-core libqt5-gui libqt5-opengl libqt5-opengl-dev libqt5-widgets qt5-qmake qtbase5-dev pyqt5"
to allow the installer to skip checking them as you are sure they are there.  See a Q&A entry for more details.

You need to avoid the `string.h` issue in the system, by specifying `-DCMAKE_FIND_FRAMEWORK=LAST` for `catkin_make_isolated` to use it from the standard library.  See an issue in the `ros/rosdistro` GibHub for the solution.  Also, according to a workaround for Homebrew Python Segfault issues] it might be better to specify PYTHON_INCLUDE_DIR and PYTHON_LIBRARY variables to point to Homebrew's.
$ ./src/catkin/bin/catkin_make_isolated --install \
    -DPYTHON_INCLUDE_DIR="/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Headers" \
Some fix was needed for me.  For `src/image_pipeline/image_view/CMakeLists.txt`:
#find_package(OpenCV REQUIRED)
find_package(OpenCV HINTS /usr/local/Cellar/opencv3/3.2.0/ REQUIRED)
Maybe optional but for `src/class_loader/CMakeLists.txt`:
#find_package(Poco REQUIRED COMPONENTS Foundation)
find_package(Poco HINTS /usr/local/Cellar/poco/1.7.8_1 REQUIRED COMPONENTS Foundation)
Do not forget to purge components under `*_isolated/` if you have accidentally failed to build.

To have it consistent with the Ubuntu convention, instead:
$ ./src/catkin/bin/catkin_make_isolated --install \
    --install-space /opt/ros/kinetic \

Fix rviz with a workaround

With Qt5, rviz causes Segmentation Fault due to an improper parameter passed to Ogre's createRenderWindow().  This is due to rviz's dependency on `QT_MAC_USE_COCOA`, used to be defined in Qt4, but no more in Qt5 probably because Qt5 only works with Cocoa but not with Carbon.  A workaround can be found in an issue raised in rviz.

Edit `src/rviz/src/rviz/ogre_helpers/render_system.cpp` to have:
// This is required for QT_MAC_USE_COCOA to be set
// workaround as Qt5 doesn't set it properly
Then rebuild rviz:
$ rm -rf devel_isolated/rviz build_isolated/rviz
$ ./src/catkin/bin/catkin_make_isolated --install --from-pkg rviz -DCMAKE_BUILD_TYPE=Release -DCMAKE_FIND_FRAMEWORK=LAST

Tips in Trial and Errors

To resume from intermediate, where you failed to build and fixed (such as installing dependencies):

$ ./src/catkin/bin/catkin_make_isolated --install \

Other References

# Running roscore and rviz

    $ pip install defusedxml
    $ brew install ogre1.9

First thing is to run `roscore`, which is a collection of nodes and programs that are pre-requisites of a ROS-based system.

    $ source /opt/ros/kinetic/setup.bash
    $ export ROS_HOSTNAME=localhost
    $ export ROS_MASTER_URI=http://localhost:11311
    $ roscore

For setting up networks, see an official Wiki.
You may see something like following:

    $ roscore
    ... logging to /Users/mich/.ros/log/c1234e21-1eb7-11e7-a096-c4b301ccf3ff/roslaunch-mich-mbp2.local-62976.log
    Checking log directory for disk usage. This may take awhile.
    Press Ctrl-C to interrupt
    started roslaunch server http://localhost:55042/
    ros_comm version 1.12.7
     * /rosdistro: kinetic
     * /rosversion: 1.12.7
    auto-starting new master
    process[master]: started with pid [62983]
    setting /run_id to c1234e21-1eb7-11e7-a096-c4b301ccf3ff
    process[rosout-1]: started with pid [62986]
    started core service [/rosout]

In another Terminal shell (tab/window),

    $ source /opt/ros/kinetic/setup.bash
    $ rviz rviz

![Screen Capture of rviz running](

It was supposed to be:

    $ rosrun rviz rviz

but `rosrun` won't work for now.

See an official Wiki page for more details.