Moto: Latent Motion Token as the Bridging Language for Robot Manipulation
🚀Introduction
Recent developments in Large Language Models (LLMs) pre-trained on extensive corpora have shown significant success in various natural language processing (NLP) tasks with minimal fine-tuning.
This success offers new promise for robotics, which has long been constrained by the high cost of action-labeled data. We ask: given the abundant video data containing interaction-related knowledge available as a rich "corpus",
can a similar generative pre-training approach be effectively applied to enhance robot learning?
The key challenge is to identify an effective representation for autoregressive pre-training that benefits robot manipulation tasks.
Inspired by the way humans learn new skills through observing dynamic environments, we propose that effective robotic learning should emphasize motion-related knowledge, which is closely tied to low-level actions and is hardware-agnostic, facilitating the transfer of learned motions to actual robot actions.
To this end, we introduce
Moto
, which converts video content into latent
Mo
tion
To
ken sequences by a Latent Motion Tokenizer, learning a bridging "language" of motion from videos in an unsupervised manner.
We pre-train Moto-GPT through motion token autoregression, enabling it to capture diverse visual motion knowledge. After pre-training, Moto-GPT demonstrates the promising ability to produce semantically interpretable motion tokens, predict plausible motion trajectories, and assess trajectory rationality through output likelihood.
To transfer learned motion priors to real robot actions, we implement a co-fine-tuning strategy that seamlessly bridges latent motion token prediction and real robot control. Extensive experiments show that the fine-tuned Moto-GPT exhibits superior robustness and efficiency on robot manipulation benchmarks, underscoring its effectiveness in transferring knowledge from video data to downstream visual manipulations.
⚙️Quick Start
Installation
Clone the repo:
git clone https://github.com/TencentARC/Moto.git
Install minimal requirements for Moto training and inference:
conda create -n moto python=3.8
conda activate moto
cd Moto
pip install -r requirements.txt
cd ..
[Optional] Setup the conda environment for evaluating Moto-GPT on the
CALVIN
benchmark:
conda create -n moto_for_calvin python=3.8
conda activate moto_for_calvin
git clone --recurse-submodules https://github.com/mees/calvin.git
pip install setuptools==57.5.0
cd calvin
cd calvin_env; git checkout main
cd ../calvin_models
sed -i 's/pytorch-lightning==1.8.6/pytorch-lightning/g' requirements.txt
sed -i 's/torch==1.13.1/torch/g' requirements.txt
cd ..
sh ./install.sh
cd ..
sudo apt-get install -y libegl1-mesa libegl1
sudo apt-get install -y libgl1
sudo apt-get install -y libosmesa6-dev
sudo apt-get install -y patchelf
cd Moto
pip install -r requirements.txt
cd ..
[Optional] Setup the conda environment for evaluating Moto-GPT on the
SIMPLER
benchmark:
We release the Latent Motion Tokenizer, the pre-traiend Moto-GPT, and the fine-tuned Moto-GPT in
Moto Hugging Face
.
You can download them separately and save them in corresponding directories (
latent_motion_tokenizer/checkpoints/
and
moto_gpt/checkpoints/
).
💻Inference
Generate latent motion trajectories with the pre-trained Moto-GPT
Moto huggingface.co is an AI model on huggingface.co that provides Moto's model effect (), which can be used instantly with this TencentARC Moto model. huggingface.co supports a free trial of the Moto model, and also provides paid use of the Moto. Support call Moto model through api, including Node.js, Python, http.
Moto huggingface.co is an online trial and call api platform, which integrates Moto's modeling effects, including api services, and provides a free online trial of Moto, you can try Moto online for free by clicking the link below.
TencentARC Moto online free url in huggingface.co:
Moto is an open source model from GitHub that offers a free installation service, and any user can find Moto on GitHub to install. At the same time, huggingface.co provides the effect of Moto install, users can directly use Moto installed effect in huggingface.co for debugging and trial. It also supports api for free installation.