Upper-Body Teleoperation

This page is intended for customer-side deployment of ELF3 upper-body teleoperation. It focuses on Pico 4 Ultra setup, video streaming, simulation validation, real-robot startup, calibration, and operation. For node implementation, controller extension, and development details, refer to the example repository:

• Upper-body teleoperation example: bxi_teleop_v2

Function Overview

The upper-body teleoperation example includes:

• ELF3 reinforcement-learning motion-control example;
• Mujoco simulation and real-robot launch entries;
• Gamepad or keyboard remote control;
• Pico 4 Ultra video streaming;
• Pico controller pose to ELF3 dual-arm IK teleoperation;
• Optional teleoperation recording.

The included BxiPicoApp-release.apk has currently been tested on Pico 4 Ultra. The default deployment uses elf3_arm_bringup_nohand.launch.py, which does not start any external dexterous-hand driver.

Environment

Quick Deployment

1. Get The Example Project

Run the following on the robot controller or a ROS 2 host:

cd ~/bxi_ws
git clone https://github.com/konodoki/bxi_teleop_v2.git
cd bxi_teleop_v2

If GitHub is unavailable on site, download the repository in advance and copy it to ~/bxi_ws/bxi_teleop_v2.

2. Install Dependencies

sudo apt update
sudo apt install -y \
  python3-colcon-common-extensions \
  ffmpeg \
  libglfw3-dev \
  libyaml-cpp-dev \
  python3-pip \
  python3-pyqt5

python3 -m pip install numpy scipy matplotlib ikpy onnx onnxruntime PyYAML

3. Build The Project

cd ~/bxi_ws/bxi_teleop_v2
source /opt/ros/humble/setup.bash
source /opt/bxi/bxi_ros2_pkg/setup.bash
bash build.sh
source install/setup.bash

Pico Video Check

1. Prepare Pico And Camera

1. Install BxiPicoApp-release.apk from the repository root onto Pico 4 Ultra;
2. Make sure Pico and the computer running this project are on the same LAN;
3. Connect the camera to a USB 3.0 port;
4. The current code reads /dev/video4 by default.

2. Start Pico Access And Video Streaming

cd ~/bxi_ws/bxi_teleop_v2
source /opt/ros/humble/setup.bash
source /opt/bxi/bxi_ros2_pkg/setup.bash
source install/setup.bash
ros2 launch elf3_arm_bringup elf3_arm_bringup_nohand.launch.py

If the following log appears, the camera stream is being published:

[pico_bxi_server-1] INF [RTSP] [session ...] is publishing to path 'video', 1 track (H264)

The Pico app scans devices on the LAN that expose port 2212 and lists possible RTSP server IPs. Select the corresponding IP with the Pico controller to view the stream. To close the stream view, press A on the right controller to enter passthrough mode.

You can also test streaming separately:

cd ~/bxi_ws/bxi_teleop_v2
bash push_rtsp.sh

Simulation Validation

Validate in simulation before running on the real robot. Open 3 terminals and source the environment in each:

cd ~/bxi_ws/bxi_teleop_v2
source /opt/ros/humble/setup.bash
source /opt/bxi/bxi_ros2_pkg/setup.bash
source install/setup.bash

Terminal 1: start the remote controller. Gamepad input is used by default:

ros2 launch remote_controller remote_controller.launch.py

Without a gamepad, use keyboard input:

ros2 launch remote_controller remote_controller_keyboard.launch.py

Terminal 2: start Pico, dual-arm IK, and video streaming:

ros2 launch elf3_arm_bringup elf3_arm_bringup_nohand.launch.py

Terminal 3: start Mujoco simulation and the control policy:

ros2 launch bxi_example_py_elf3 example_demo.launch.py

Real-Robot Startup

Real-robot operation is higher risk. Before startup, confirm emergency stop, power, network, workspace, and surrounding safety.

Enter root first:

sudo su
cd /home/bxi/bxi_ws/bxi_teleop_v2
source /opt/ros/humble/setup.bash
source /opt/bxi/bxi_ros2_pkg/setup.bash
source install/setup.bash

Open 3 root terminals.

Terminal 1: start the remote controller:

ros2 launch remote_controller remote_controller.launch.py

Terminal 2: start Pico, dual-arm IK, and video streaming:

ros2 launch elf3_arm_bringup elf3_arm_bringup_nohand.launch.py

Terminal 3: start the real-robot hardware node and control policy:

ros2 launch bxi_example_py_elf3 example_demo_hw.launch.py

If the launch reports that another hardware-control instance is already running, confirm and stop the old instance first. example_demo_hw.launch.py uses /tmp/bxi_example_hw.lock to avoid duplicate hardware-control startup.

Calibration And Operation

1. After elf3_arm_ikpy_control_pico starts, raise both hands above the head;
2. When calibration starts, keep both arms as straight as possible and sweep a full spherical range around the shoulders;
3. Continue after the terminal reports successful calibration;
4. Use the remote controller to put the robot into the normal standing state;
5. Press RT + A to enter teleoperation mode;
6. Hold the Pico grip button. The corresponding robot arm starts following the Pico controller;
7. Press both triggers to start teleoperation recording. Release either trigger to stop recording.

Recording files are saved to:

install/bxi_example_py_elf3/share/bxi_example_py_elf3/data/teleop_records

Acceptance Check

After basic deployment, verify:

1. The Pico app can discover the video service IP on port 2212 and display the camera stream;
2. The elf3_arm_bringup_nohand.launch.py terminal continuously prints Pico connection or RTSP streaming logs;
3. The robot can enter standing and teleoperation flow in simulation;
4. When the Pico grip button is held, both arms follow the Pico controllers in simulation or on the real robot;
5. During real-robot operation, there are no continuous packet losses, protection triggers, or unexpected node exits.

Troubleshooting

communication, mujoco, Or hardware_elf3 Not Found

The BXI ROS 2 base package environment is usually missing:

source /opt/bxi/bxi_ros2_pkg/setup.bash

Then rebuild or restart.

Pico Cannot Find The Video IP

Check:

1. Pico and the computer are on the same LAN;
2. The computer firewall is not blocking port 2212;
3. MediaMTX is listening:

ss -lntup | grep 2212

Camera Cannot Be Opened

The current code uses /dev/video4 by default. If your camera uses another device number, modify:

src/pico_bxi_server/src/pico_bxi_server.cpp
push_rtsp.sh

Also confirm that the camera is connected to a USB 3.0 port and the current user has camera access. For temporary debugging:

sudo chmod 777 /dev/video4

aero_hand_open Not Found When Starting Full Bringup

The current repository does not include the aero_hand_open implementation. If no external dexterous-hand driver is installed, use:

ros2 launch elf3_arm_bringup elf3_arm_bringup_nohand.launch.py

Real-Robot Nodes Cannot Discover Each Other

Make sure the robot and the terminal running the teleoperation project use the same ROS_DOMAIN_ID. Restart related ROS 2 nodes after changing it.

Development Entry

Customer deployment usually does not require changes to the low-level video streaming, Pico data bridge, or IK node. To extend controller input, modify the state machine, replace the model, or connect a dexterous hand, refer to the repository README:

https://github.com/konodoki/bxi_teleop_v2