Using the NPU on RK3588/RK3566 Orange Pis: RKNN-Toolkit2 on the PC, RKNN-Toolkit-Lite2 or the C runtime on the board

The NPU is the main reason to pick an RK3588 Orange Pi over a Raspberry Pi for camera or AI workloads. It isn't used through PyTorch or TensorFlow directly, but through Rockchip's RKNN stack.

The two-step workflow

1. On your PC (x86 Linux): convert the trained model (e.g. ONNX, TFLite, PyTorch) to the .rknn format with RKNN-Toolkit2. The same toolkit can quantize the model, simulate inference and evaluate performance, also against a connected board.
2. On the board: run the .rknn model with
   • RKNN-Toolkit-Lite2 (Python API), or
   • the RKNN Runtime (C/C++ API) for lowest overhead.

The README notes that the NPU kernel driver (RKNPU) is open source in Rockchip's kernel code, so use a board image whose kernel includes it, and check before switching to a different kernel.

Which toolkit for which chip

• Not compatible: RKNN-Toolkit2 and the old RKNN-Toolkit are separate; models and code don't carry over.
• Python: RKNN-Toolkit2 supports Python 3.6 to 3.12; match the wheel to your Python version.
• Performance class: the RK3588/RK3588S NPU is rated at up to 6 TOPS; the Orange Pi 3B's RK3566 NPU at 0.8 TOPS (INT8).

Where to start

• rknn_model_zoo (github.com/airockchip/rknn_model_zoo) has ready conversion and deployment examples for common models; start from one close to yours instead of converting from scratch.
• LLMs use a separate SDK, RKNN-LLM (github.com/airockchip/rknn-llm).
• orangepi-build's RK3588 board support package ships a demo script (test_rknn_demo.sh in /usr/local/bin), useful to check that the NPU works before debugging your own model.

Pitfalls

• Not every model converts cleanly; read the conversion log, and start from a similar model in the model zoo.
• Quantization (INT8) needs a small calibration dataset that resembles real inputs; accuracy can drop noticeably without it.
• Keep the toolkit version on the PC and the runtime version on the board in step.