DINO+CDP Tomato Harvesting Dataset
Description
This repository contains a high-fidelity teleoperated robotic manipulation dataset designed to support imitation learning, multi-view perception, and autonomous agricultural manipulation tasks—specifically tomato plant pruning and harvesting. The data captures real-world robot-plant interactions collected in a controlled biocell greenhouse chamber under consistent lighting conditions. Data Collection & Hardware Setup The dataset was generated using a dual-manipulator setup in a leader–follower configuration consisting of two MyCobot M5-280 robotic manipulators. As an operator teleoperated the leader robot (actor), the follower robot (imitator) mirrored its movements. Dataset Structure & Specifications The dataset comprises 100 total demonstrations sampled at a temporal frequency of 20 Hz (0.05-second intervals). Each individual demonstration consists of 300 temporally aligned timesteps containing synchronized visual and kinematic data streams: • Kinematic Data: 6 synchronized joint angles from the actor (leader) robot and 6 synchronized joint angles from the imitator (follower) robot. • Visual Data: Multi-view RGB camera streams captured simultaneously from three distinct viewpoints focusing on the agricultural interaction area. • Cultivar Variations: The dataset spans multiple tomato cultivars, including Celebrity, Beefsteak, and Big Boy Hybrid. Target Applications This data is structured to train and evaluate learning-based robotic frameworks, including self-supervised representation learning models (such as DINO) and policy learning frameworks (such as Conditional Diffusion Policies) for agricultural automation.
Files
Steps to reproduce
1. Experimental Environment & Data Acquisition Setup: The data collection environment consists of a dual-manipulator workstation arranged in a leader–follower configuration using two MyCobot M5-280 robotic manipulators. Procedure: An operator teleoperates the actor (leader) robot, while the imitator (follower) robot dynamically mirrors the trajectory. All tasks are performed inside a controlled biocell greenhouse chamber containing live tomato plants under uniform, stabilized lighting conditions. Sampling: Visual and kinematic data streams are recorded simultaneously at a temporal frequency of 20 Hz (0.05-second intervals). Each complete demonstration runs for exactly 300 synchronized time steps (15 seconds per file). 2. Spatial Calibration & Geometric Validation Camera Setup: Multi-view imagery is captured using three distinct RGB cameras positioned to isolate the plant-manipulator interaction workspace. Calibration Protocol: Intrinsic and extrinsic camera parameters are computed via a camera self-calibration routine. 3. Software Compatibility & Data Parsing File Access: All recorded trajectories, multi-view frames, and joint states are compiled into structured Hierarchical Data Format (.hdf5) files. These files can be read natively using Python (via the h5py package) or MATLAB (using h5read). HDF5 Internal Data Structure: /images: A multi-dimensional tensor containing the synchronized multi-view RGB image arrays from all three camera perspectives. /joint_angles_actor: A 300 × 6 array representing the 6-degree-of-freedom joint configuration vectors of the teleoperated leader manipulator over time. /joint_angles_imitator: A 300 × 6 array representing the matching 6-degree-of-freedom joint configuration vectors of the follower manipulator.
Institutions
- Texas A&M University – Corpus ChristiTexas, Corpus Christi