Oliver Limoyo

I'm a Ph.D. student at the STARS Lab, University of Toronto. My research focuses on the application of generative and self-supervised learning to robotics problems. I am also interested in multimodal sensing for manipulation. I'm supervised by Prof. Jonathan Kelly.

Selected Publications

Photobot: Reference-guided Interactive Photography via Natural Language

O. Limoyo*, J. Li*, D. Rivkin, J. Kelly, and G. Dudek,

Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2024

arXiv

Working backwards: Learning to Place by Picking

O. Limoyo, A. Konar, T. Ablett, J. Kelly, F. Hogan, and G. Dudek,

Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2024

arXivCode

Multimodal and Force-Matched Imitation Learning with a See-Through Visuotactile Sensor

T. Ablett, O. Limoyo, A. Sigal, A. Jilani, J. Kelly, K. Siddiqi, F. Hogan, G. Dudek

Submitted to IEEE Transactions on Robotics (T-RO): Special Section on Tactile Robotics, 2024

arxivCode

Generative Graphical Inverse Kinematics

O. Limoyo*, F. Marić*, M. Giamou, P. Alexson, I. Petrović, J. Kelly

To be published in IEEE Transactions on Robotics (T-RO), 2023

arXivCode

Euclidean Equivariant Models for Generative Graphical Inverse Kinematics

O. Limoyo*, F. Marić*, M. Giamou, P. Alexson, I. Petrović, J. Kelly

Proceedings of the Robotics: Science and Systems (RSS) Workshop on Symmetries in Robot Learning, 2023

arXiv

Learning Sequential Latent Variable Models from Multimodal Time Series Data

O. Limoyo, T. Ablett, J. Kelly

Intelligent Autonomous Systems 17 (IAS'17), 2023

DOIarXivCode

Best Paper Finalist

Heteroscedastic Uncertainty for Robust Generative Latent Dynamics

O. Limoyo, B. Chan, F. Marić, B. Wagstaff, AR. Mahmood, J. Kelly

IEEE Robotics and Automation Letters (RA-L), 2020

DOIarXivVideoCode

The Canadian Planetary Emulation Terrain Energy-Aware Rover Navigation Dataset

O. Lamarre, O. Limoyo, F. Maric, and J. Kelly

The International Journal of Robotics Research (IJRR), 2020

DOICode

Fast Manipulability Maximization Using Continuous-Time Trajectory Optimization

F. Maric, O. Limoyo, L. Petrovic, T. Ablett, I. Petrovic, and J. Kelly

Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2019

DOIarXivVideo

Self-Calibration of Mobile Manipulator Kinematic and Sensor Extrinsic Parameters Through Contact-Based Interaction

O. Limoyo, T. Ablett, F. Maric, L. Volpatti, and J. Kelly

Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), 2018

DOIarXivVideo

Work

Robotics & AI Research Scientist Intern

Samsung AI Centre Montreal, Jan. 2023 - Sept. 2023

Robot Photography via Foundation Models.

Robotics & AI Research Scientist Intern

Samsung AI Centre Montreal, May 2022 - Dec. 2022

Visuotactile Manipulation.

AI Research Scientist Intern

Kindred AI (Part of OCADO Group), Apr. 2019 - Aug. 2019

Learning to Scan and Sort.

Projects

Reproducing Reinforcement Learning Results on Real-World Robots

I alpha-tested Kindred AI's SenseAct reinforcement learning framework. This involved reproducing policy learning results presented on a UR5 in the original paper on my lab's UR10.

Mobile Manipulator Line Drawing with Redundant Inverse Kinematics and Impedance Controller

A mobile manipulator simultaneously moves its base while drawing a straight line with its end-effector using Redundant Inverse Kinematics. The marker maintains contact with the whiteboard using an impedance controller.

Manipulator Teleoperation with Camera-based Hand Tracking

Teleoperate a manipulator using hand tracking with a Kinect camera. The gripper closes and opens based on the output of a neural network that detects if the user's hand is open or closed in the image.

Full Body Admittance Control of Mobile Manipulator

Push or pull a mobile manipulator around based on an admittance controller and force torque sensor. Depending on the amount of force applied the manipulator moves alone or the mobile manipulator moves together.