Oliver Limoyo

I'm a Ph.D. student at the STARS Lab, University of Toronto. My research focuses on the application of self-supervised learning to robotics and multimodal sensing for manipulation. I'm supervised by Prof. Jonathan Kelly.

Selected Publications

Photobot: Reference-guided Interactive Photography via Natural Language

O. Limoyo*, J. Li*, D. Rivkin, J. Kelly, and G. Dudek,

Submitted to IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS'24),

arXiv

Working backwards: Learning to Place by Picking

O. Limoyo, A. Konar, T. Ablett, J. Kelly, F. Hogan, and G. Dudek,

Submitted to IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS'24),

arXivCode

Multimodal and Force-Matched Imitation Learning with a See-Through Visuotactile Sensor

T. Ablett, O. Limoyo, A. Sigal, A. Jilani, J. Kelly, K. Siddiqi, F. Hogan, G. Dudek

Submitted to IEEE Transactions on Robotics (T-RO): Special Section on Tactile Robotics, February 2024

arxivCode

Generative Graphical Inverse Kinematics

O. Limoyo*, F. Marić*, M. Giamou, P. Alexson, I. Petrović, J. Kelly

Submitted to IEEE Transactions on Robotics (T-RO), June 2023

arXivCode

Euclidean Equivariant Models for Generative Graphical Inverse Kinematics

O. Limoyo*, F. Marić*, M. Giamou, P. Alexson, I. Petrović, J. Kelly

Proceedings of the Robotics: Science and Systems (RSS) Workshop on Symmetries in Robot Learning, Daegu, Republic of Korea, July 10 2023

arXiv

Learning Sequential Latent Variable Models from Multimodal Time Series Data

O. Limoyo, T. Ablett, J. Kelly

Intelligent Autonomous Systems 17, Cham, 2023, pp. 511-529,

DOIarXivCode

Best Paper Finalist

Heteroscedastic Uncertainty for Robust Generative Latent Dynamics

O. Limoyo, B. Chan, F. Marić, B. Wagstaff, AR. Mahmood, J. Kelly

IEEE Robotics and Automation Letters, vol. 5, iss. 4, pp. 6654-6661, 2020.,

DOIarXivVideoCode

The Canadian Planetary Emulation Terrain Energy-Aware Rover Navigation Dataset

O. Lamarre, O. Limoyo, F. Maric, and J. Kelly

The International Journal of Robotics Research, vol. 39, iss. 6, pp. 641-650, 2020.,

DOICode

Fast Manipulability Maximization Using Continuous-Time Trajectory Optimization

F. Maric, O. Limoyo, L. Petrovic, T. Ablett, I. Petrovic, and J. Kelly

Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Macau, China, Nov. 4–8, 2019, pp. 8258-8264.,

DOIarXivVideo

Self-Calibration of Mobile Manipulator Kinematic and Sensor Extrinsic Parameters Through Contact-Based Interaction

O. Limoyo, T. Ablett, F. Maric, L. Volpatti, and J. Kelly

Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), Brisbane, Queensland, Australia, May 21–25, 2018.,

DOIarXivVideo

Work

Robotics & AI Research intern

Samsung AI Centre Montreal, May 2022 - Dec. 2023

Visuotactile Manipulation and Robot Photography via Foundation Models. Supervisors: Dr. Francois Hogan and Prof. Gregory Dudek

AI Research intern

Kindred AI (Part of OCADO Group), Apr. 2019 - Aug. 2019

Learning to Scan and Sort. Supervisors: Dr. James Bergstra and Prof. Rupam Mahmood

Projects

Reproducing Reinforcement Learning Results on Real-World Robots

I alpha-tested Kindred AI's SenseAct reinforcement learning framework. This involved reproducing policy learning results presented on a UR5 in the original paper on my lab's UR10.

Mobile Manipulator Line Drawing with Redundant Inverse Kinematics and Impedance Controller

A mobile manipulator simultaneously moves its base while drawing a straight line with its end-effector using Redundant Inverse Kinematics. The marker maintains contact with the whiteboard using an impedance controller.

Manipulator Teleoperation with Camera-based Hand Tracking

Teleoperate a manipulator using hand tracking with a Kinect camera. The gripper closes and opens based on the output of a neural network that detects if the user's hand is open or closed in the image.

Full Body Admittance Control of Mobile Manipulator

Push or pull a mobile manipulator around based on an admittance controller and force torque sensor. Depending on the amount of force applied the manipulator moves alone or the mobile manipulator moves together.