Projects

We study the problem of learning the pushing dynamics of arbitrary 2D rigid bodies, developing neural network models trained on simulated data collected with a Franka Panda robot. By comparing a shallow MLP to a deeper point-cloud-inspired network, we show that the deeper model better captures the complex motion dynamics of different 2D shapes. Using the learned models in a Model Predictive Path Integral (MPPI) controller, we successfully achieve closed-loop pushing and obstacle avoidance across diverse 2D rigid bodies.

We introduce VTMo, a modular Vision-Touch Transformer encoder that unifies dual-encoder flexibility with fusion-encoder accuracy through a shared self-attention mechanism and modality-specific or cross-modal experts. VTMo supports image-only, touch-only, and vision-touch fusion tasks, offering versatility for speed or accuracy. Our method achieves competitive performance on the Image-to-Touch Retrieval task while reducing training time and computational complexity.

We introduce the One-Step Tree-of-Thoughts framework, a simplified prompting method that distills multi-step reasoning into a single structured prompt, and demonstrates how knowledge distillation can transfer this reasoning capability from Large Language Models to Small Language Models with much less parameters, enabling significant improvements reasoning performance, beating GPT-4o and GPT-4, as shown on the model's performance on Game of 24.