Rdzleo/CogletESP-CogletESP
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

1、第一次提交项目，项目初始化；
Some checks failed
Build Boards / Determine variants to build (push) Successful in 6m26s
Details
Build Boards / Build ${{ matrix.name }} (push) Failing after 6m43s
Details

Browse Source 
2、修改了RP2040的代码，使其在没有安装摄像头的情况下也可以左右转动眼球和左右转动身体；
3、增加了一些中文注释的说明；
This commit is contained in:
Rdzleo 2026-04-09 14:22:24 +08:00
commit c63b35a0e7
1374 changed files with 130599 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

103
.github/ISSUE_TEMPLATE/01_build_install_bug.yml vendored Normal file
View File

@ -0,0 +1,103 @@
name: Installation or build bug report
description: Report installation or build bugs
labels: ['bug']
body:
- type: checkboxes
id: checklist
attributes:
label: Answers checklist.
description: Before submitting a new issue, please follow the checklist and try to find the answer.
options:
- label: I have read the documentation [XiaoZhi AI Programming Guide](https://ccnphfhqs21z.feishu.cn/wiki/F5krwD16viZoF0kKkvDcrZNYnhb) and the issue is not addressed there.
required: true
- label: I have updated my branch (master or release) to the latest version and checked that the issue is present there.
required: true
- label: I have searched the issue tracker for a similar issue and not found a similar issue.
required: true
- type: input
id: xiaozhi_ai_version
attributes:
label: XiaoZhi AI version.
description: On which XiaoZhi AI version does this issue occur on? Run `git describe --tags` to find it.
placeholder: ex. v1.1.0-44-g140aab8
validations:
required: true
- type: dropdown
id: operating_system
attributes:
label: Operating System used.
multiple: false
options:
- Windows
- Linux
- macOS
validations:
required: true
- type: dropdown
id: build
attributes:
label: How did you build your project?
multiple: false
options:
- Command line with CMake
- Command line with idf.py
- CLion IDE
- VS Code IDE/Cursor
- Other (please specify in More Information)
validations:
required: true
- type: dropdown
id: windows_comand_line
attributes:
label: If you are using Windows, please specify command line type.
multiple: false
options:
- PowerShell
- CMD
validations:
required: false
- type: textarea
id: expected
attributes:
label: What is the expected behavior?
description: Please provide a clear and concise description of the expected behavior.
placeholder: I expected it to...
validations:
required: true
- type: textarea
id: actual
attributes:
label: What is the actual behavior?
description: Please describe actual behavior.
placeholder: Instead it...
validations:
required: true
- type: textarea
id: steps
attributes:
label: Steps to reproduce.
description: 'How do you trigger this bug? Please walk us through it step by step. If this is build bug, please attach sdkconfig file (from your project folder). Please attach your code here.'
value: |
1. Step
2. Step
3. Step
...
validations:
required: true
- type: textarea
id: debug_logs
attributes:
label: Build or installation Logs.
description: Build or installation log goes here, should contain the backtrace, as well as the reset source if it is a crash.
placeholder: Your log goes here.
render: plain
validations:
required: false
- type: textarea
id: more-info
attributes:
label: More Information.
description: Do you have any other information from investigating this?
placeholder: ex. Any more.
validations:
required: false

115
.github/ISSUE_TEMPLATE/02_runtime_bug.yml vendored Normal file
View File

@ -0,0 +1,115 @@
name: Runtime bug report
description: Report runtime bugs
labels: ['bug']
body:
- type: checkboxes
id: checklist
attributes:
label: Answers checklist.
description: Before submitting a new issue, please follow the checklist and try to find the answer.
options:
- label: I have read the documentation [XiaoZhi AI Programming Guide](https://ccnphfhqs21z.feishu.cn/wiki/F5krwD16viZoF0kKkvDcrZNYnhb) and the issue is not addressed there.
required: true
- label: I have updated my firmware to the latest version and checked that the issue is present there.
required: true
- label: I have searched the issue tracker for a similar issue and not found a similar issue.
required: true
- type: input
id: xiaozhi_ai_firmware_version
attributes:
label: XiaoZhi AI firmware version.
description: On which firmware version does this issue occur on?
placeholder: ex. v1.2.1_bread-compact-wifi
validations:
required: true
- type: dropdown
id: operating_system
attributes:
label: Operating System used.
multiple: false
options:
- Windows
- Linux
- macOS
validations:
required: true
- type: dropdown
id: build
attributes:
label: How did you build your project?
multiple: false
options:
- Command line with CMake
- Command line with idf.py
- CLion IDE
- VS Code IDE/Cursor
- Other (please specify in More Information)
validations:
required: true
- type: dropdown
id: windows_comand_line
attributes:
label: If you are using Windows, please specify command line type.
multiple: false
options:
- PowerShell
- CMD
validations:
required: false
- type: dropdown
id: power_supply
attributes:
label: Power Supply used.
multiple: false
options:
- USB
- External 5V
- External 3.3V
- Battery
validations:
required: true
- type: textarea
id: expected
attributes:
label: What is the expected behavior?
description: Please provide a clear and concise description of the expected behavior.
placeholder: I expected it to...
validations:
required: true
- type: textarea
id: actual
attributes:
label: What is the actual behavior?
description: Please describe actual behavior.
placeholder: Instead it...
validations:
required: true
- type: textarea
id: steps
attributes:
label: Steps to reproduce.
description: 'How do you trigger this bug? Please walk us through it step by step. Please attach your code here.'
value: |
1. Step
2. Step
3. Step
...
validations:
required: true
- type: textarea
id: debug_logs
attributes:
label: Debug Logs.
description: Debug log goes here, should contain the backtrace, as well as the reset source if it is a crash.
placeholder: Your log goes here.
render: plain
validations:
required: false
- type: textarea
id: more-info
attributes:
label: More Information.
description: Do you have any other information from investigating this?
placeholder: ex. Any more.
validations:
required: false

34
.github/ISSUE_TEMPLATE/03_feature_request.yml vendored Normal file
View File

@ -0,0 +1,34 @@
name: Feature request
description: Suggest an idea for this project.
labels: ['enhancement']
body:
- type: markdown
attributes:
value: |
* We welcome any ideas or feature requests! It’s helpful if you can explain exactly why the feature would be useful.
* There are usually some outstanding feature requests in the [existing issues list](https://github.com/78/xiaozhi-esp32/labels/enhancement), feel free to add comments to them.
* If you would like to contribute, please read the [contributions guide](https://ccnphfhqs21z.feishu.cn/wiki/F5krwD16viZoF0kKkvDcrZNYnhb).
- type: textarea
id: problem-related
attributes:
label: Is your feature request related to a problem?
description: Please provide a clear and concise description of what the problem is.
placeholder: ex. I'm always frustrated when ...
- type: textarea
id: solution
attributes:
label: Describe the solution you'd like.
description: Please provide a clear and concise description of what you want to happen.
placeholder: ex. When using XiaoZhi ...
- type: textarea
id: alternatives
attributes:
label: Describe alternatives you've considered.
description: Please provide a clear and concise description of any alternative solutions or features you've considered.
placeholder: ex. Choosing other approach wouldn't work, because ...
- type: textarea
id: context
attributes:
label: Additional context.
description: Please add any other context or screenshots about the feature request here.
placeholder: ex. This would work only when ...

8
.github/ISSUE_TEMPLATE/config.yml vendored Normal file
View File

@ -0,0 +1,8 @@
blank_issues_enabled: true
contact_links:
- name: 小智 AI 官方网站
url: https://xiaozhi.me/
about: 激活设备、配置 AI、声纹识别、声音克隆等应有尽有，DIY 属于你自己的小智
- name: 小智 AI 聊天机器人百科全书
url: https://ccnphfhqs21z.feishu.cn/wiki/F5krwD16viZoF0kKkvDcrZNYnhb
about: 开发文档、硬件制作、烧录教程、FAQ尽在小智百科

111
.github/workflows/build.yml vendored Normal file
View File

@ -0,0 +1,111 @@
name: Build Boards
on:
push:
branches:
- main
- ci/* # for ci test
pull_request:
branches:
- main
permissions:
contents: read
jobs:
prepare:
name: Determine variants to build
runs-on: ubuntu-latest
outputs:
variants: ${{ steps.select.outputs.variants }}
steps:
- name: Checkout
uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Install jq
run: sudo apt-get update && sudo apt-get install -y jq
- id: list
name: Get all variant list
run: |
echo "all_variants=$(python scripts/release.py --list-boards --json)" >> $GITHUB_OUTPUT
- id: select
name: Select variants based on changes
env:
ALL_VARIANTS: ${{ steps.list.outputs.all_variants }}
run: |
EVENT_NAME="${{ github.event_name }}"
# push 到 main 分支，编译全部变体
if [[ "$EVENT_NAME" == "push" ]]; then
echo "variants=$ALL_VARIANTS" >> $GITHUB_OUTPUT
exit 0
fi
# pull_request 场景
BASE_SHA="${{ github.event.pull_request.base.sha }}"
HEAD_SHA="${{ github.event.pull_request.head.sha }}"
echo "Base: $BASE_SHA, Head: $HEAD_SHA"
CHANGED=$(git diff --name-only $BASE_SHA $HEAD_SHA || true)
echo -e "Changed files:\n$CHANGED"
NEED_ALL=0
declare -A AFFECTED
while IFS= read -r file; do
if [[ "$file" == main/* && "$file" != main/boards/* ]]; then
NEED_ALL=1
fi
if [[ "$file" == main/boards/common/* ]]; then
NEED_ALL=1
fi
if [[ "$file" == main/boards/* ]]; then
board=$(echo "$file" | cut -d '/' -f3)
AFFECTED[$board]=1
fi
done <<< "$CHANGED"
if [[ "$NEED_ALL" -eq 1 ]]; then
echo "variants=$ALL_VARIANTS" >> $GITHUB_OUTPUT
else
if [[ ${#AFFECTED[@]} -eq 0 ]]; then
echo "variants=[]" >> $GITHUB_OUTPUT
else
BOARDS_JSON=$(printf '%s\n' "${!AFFECTED[@]}" | sort -u | jq -R -s -c 'split("\n")[:-1]')
FILTERED=$(echo "$ALL_VARIANTS" | jq -c --argjson boards "$BOARDS_JSON" 'map(select(.board as $b | $boards | index($b)))')
echo "variants=$FILTERED" >> $GITHUB_OUTPUT
fi
fi
build:
name: Build ${{ matrix.name }}
needs: prepare
if: ${{ needs.prepare.outputs.variants != '[]' }}
strategy:
fail-fast: false # 单个变体失败不影响其它变体
matrix:
include: ${{ fromJson(needs.prepare.outputs.variants) }}
runs-on: ubuntu-latest
container:
image: espressif/idf:release-v5.5
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Build current variant
shell: bash
run: |
source $IDF_PATH/export.sh
python scripts/release.py ${{ matrix.board }} --name ${{ matrix.name }}
- name: Upload artifacts
uses: actions/upload-artifact@v4
with:
name: xiaozhi_${{ matrix.name }}_${{ github.sha }}.bin
path: build/merged-binary.bin
if-no-files-found: error

19
.gitignore vendored Normal file
View File

@ -0,0 +1,19 @@
tmp/
components/
managed_components/
build/
.vscode/
.devcontainer/
sdkconfig.old
sdkconfig
dependencies.lock
.env
releases/
main/assets/lang_config.h
main/mmap_generate_emoji.h
.DS_Store
.cache
*.pyc
*.bin
mmap_generate_*.h
.clangd

14
CMakeLists.txt Executable file
View File

@ -0,0 +1,14 @@
# For more information about build system see
# https://docs.espressif.com/projects/esp-idf/en/latest/api-guides/build-system.html
# The following five lines of boilerplate have to be in your project's
# CMakeLists in this exact order for cmake to work correctly
cmake_minimum_required(VERSION 3.16)
set(PROJECT_VER "2.0.5")
# Add this line to disable the specific warning
add_compile_options(-Wno-missing-field-initializers)
include($ENV{IDF_PATH}/tools/cmake/project.cmake)
project(xiaozhi)

602
Coglet项目分析与开发指南.md Normal file
View File

@ -0,0 +1,602 @@
# Coglet 项目分析与开发指南
## 一、项目简介
**Coglet** 是 YouTube 创作者 Will Cogley（原 Nilheim Mechatronics）开发的 **DIY 桌面 AI 伴侣机器人**，目前在 Kickstarter 众筹中。项目包含 3D 打印外壳、PCB 电路板和双 MCU 软件系统。
### 系统架构
采用**双 MCU 架构**，三个核心模块通过 UART 串联：
```
ESP32-S3 ←── UART 115200 ──→ RP2040 (Pico) ←── UART 921600 ──→ Grove Vision AI V2
(AI/语音/网络) (舵机/动画) (摄像头/人脸检测)
```
| 芯片 | 职责 |
|------|------|
| **ESP32-S3** | AI 对话、语音识别/合成、WiFi 网络、MCP 协议（基于小智 XiaoZhi 固件） |
| **RP2040 (Raspberry Pi Pico)** | 9 个舵机控制、动画表情、人脸追踪（MicroPython） |
| **Grove Vision AI V2** | 摄像头人脸检测 |
### ESP32 固件核心功能
- 连接方式：WiFi / ML307 Cat.1 4G
- 语音交互：流式 ASR + LLM（Qwen/DeepSeek）+ TTS
- 离线唤醒：ESP-SR 本地语音唤醒
- 通信协议：WebSocket 或 MQTT+UDP
- 音频编码：OPUS
- 显示：OLED/LCD + 表情动画
- 多语言：中文/英文/日文
- MCP 协议：设备端 MCP（控制扬声器、LED、舵机、GPIO）+ 云端 MCP（智能家居）
### RP2040 控制的舵机（共 9 个）
| 舵机代号 | 功能 |
|---------|------|
| EYL / EYR | 左右眼水平转动 |
| PIT | 头部俯仰 |
| YAW | 头部偏航 |
| MOU | 嘴巴开合 |
| LID | 眼睑眨眼 |
| 其他 | 耳朵舵机等 |
### RP2040 状态机
`idle` → `speaking` → `listening` → `thinking` → `happy` → `neutral` → `calibration`
---
## 二、GitHub 仓库总览
### 核心仓库
| 仓库 | 地址 | Stars | 说明 |
|------|------|-------|------|
| **Coglet 主仓库** | https://github.com/will-cogley/Coglet | 135 | 3D 打印件、PCB 原理图、文档 |
| **CogletESP 固件** | https://github.com/will-cogley/CogletESP | 5 | ESP32 固件 + RP2040 代码（fork 自小智） |
### 相关参考仓库
| 仓库 | 地址 | 说明 |
|------|------|------|
| 小智上游原版 | https://github.com/78/xiaozhi-esp32 | CogletESP 的上游项目 |
| EyeMech_Epsilon | https://github.com/will-cogley/EyeMech_Epsilon | Coglet 眼球机构的技术前身，RP2040 + MicroPython |
### CogletESP 分支列表
| 分支名 | Commits | 说明 |
|--------|---------|------|
| `main` | 789 | 上游同步的主分支 |
| **`CogletESP`** | 758 | **Coglet 专用分支（无 ESP32 摄像头版本，保留 LCD 显示）** |
| **`camera-version`** | 755 | **ESP32 OV3660 摄像头版本（需 PCB 飞线，禁用 LCD）** |
| `esp32_camera` | - | ESP32 摄像头分支 |
| `v1` | 669 | v1 稳定版 |
| `bed-operator` | - | 功能分支 |
| `fix_resampler` | - | 重采样器修复 |
| `fix_rndis` | - | RNDIS 修复 |
| `listening_wakeword` | - | 唤醒词监听 |
| `nt26_board` | - | NT26 开发板适配 |
### CogletESP 分支 vs camera-version 分支（关键区别）
> **重要区分**：这里的"摄像头"指 **ESP32-S3 上直连的 OV3660 DVP 摄像头**（小智 AI 视觉功能），**不是** Grove Vision AI V2。Grove Vision AI V2 是独立模块接在 RP2040 上，两个分支都支持它。
两个分支仅 **2 个文件**不同，都在 `main/boards/bread-compact-wifi-s3cam/` 目录下：
| 对比项 | `CogletESP` 分支 | `camera-version` 分支 |
|--------|------------------|----------------------|
| ESP32 摄像头(OV3660) | 默认引脚，与 PSRAM 冲突 | 飞线后的新引脚映射 |
| LCD 显示 | 正常支持 | **禁用**（引脚被摄像头占用，用 DummyDisplay 替代） |
| PCB 改动 | 无需飞线 | **需要 3 根飞线**（35→14, 36→41, 37→42） |
| 摄像头画面 | - | 设置 HMirror + VFlip 翻转 |
| JTAG 引脚(GPIO 39-42) | 默认用途 | 释放给摄像头使用 |
| Grove Vision AI V2 | 支持 | 支持 |
#### GPIO 引脚映射差异（部分关键引脚）
| 功能 | `CogletESP` 分支 | `camera-version` 分支 |
|------|------------------|----------------------|
| MIC_WS | GPIO 1 | GPIO 4 |
| MIC_SCK | GPIO 2 | GPIO 5 |
| MIC_DIN | GPIO 42 | GPIO 6 |
| SPK_DOUT | GPIO 39 | GPIO 7 |
| SPK_BCLK | GPIO 40 | GPIO 15 |
| SPK_LRCK | GPIO 41 | GPIO 16 |
| LED | GPIO 48 | GPIO 3 |
| CAMERA D7 | GPIO 16 | GPIO 14（飞线 35→14） |
| CAMERA VSYNC | GPIO 6 | GPIO 41（飞线 36→41） |
| CAMERA HREF | GPIO 7 | GPIO 42（飞线 37→42） |
#### 如何选择分支？
- **带 ESP32 OV3660 摄像头** → 用 `camera-version` 分支 + PCB 做 3 根飞线，但**没有 LCD 屏**
- **不带 ESP32 摄像头 / 想保留 LCD 屏** → 用 `CogletESP` 分支
- **仅使用 Grove Vision AI V2 人脸追踪** → 两个分支都可以，这是 RP2040 端的功能，与 ESP32 分支无关
### Coglet 主仓库目录结构
```
Coglet/
├── 3D Printing Files/
│ └── CogletB34Parts.3mf # 3D 打印零件
├── PCB/
│ ├── CogNogV1_0.pdf # 原理图 PDF
│ └── CogletESP_2026-02-24.epro # EasyEDA 工程文件
├── CogletESP/ # Git 子模块 → CogletESP 仓库
├── Translated Docs for XiaoZhi AI/ # 14 份小智文档英文翻译
│ ├── 固件烧录指南
│ ├── WiFi 配置
│ ├── 面包板接线
│ ├── ESP-IDF 开发环境搭建
│ └── 摄像头接线 ...
└── README.md
```
### 已知 PCB 问题（CogNog V1.0）
- USB-C 接口过紧
- **摄像头引脚与 PSRAM 冲突**（GPIO 35/36/37 被 PSRAM Octal SPI 占用，详见下方飞线说明）
- 体积电容不足
- 摄像头画面水平翻转（camera-version 分支已通过 `SetHMirror(true)` 修正）
### 翻译文档索引（Translated Docs for XiaoZhi AI）
主仓库中 `Translated Docs for XiaoZhi AI/` 包含 13 份小智文档英文翻译，其中与搭建相关的：
| 文件 | 内容 |
|------|------|
| **DIY Breadboard for Xiaozhi AI...** | 完整硬件清单和接线教程（最详细，18MB） |
| **[Latest] Wiring Tutorial...Camera** | **带摄像头的最新接线教程** |
| Setting up ESP IDF 5.5.2... | ESP-IDF 开发环境搭建（Windows） |
| Flash Tool_Web Terminal... | 网页烧录工具（无需 IDF 环境） |
| Configure Device Wi-Fi... | WiFi 配网和设备添加 |
| Steps for Wake Word Change... | 唤醒词修改 |
---
## 三、项目状态
- **活跃度**：积极开发中（最后更新 2026 年 3 月）
- **成熟度**：早期阶段 / 工作进行中
- Releases 页面为空，**无预编译固件**
- PCB 存在多个已知问题
- 没有专用的 Coglet 板级配置（使用通用 bread-compact 系列）
- **众筹**：Kickstarter 进行中
- **社区**：Discord + QQ 群
---
## 四、摄像头版本完整搭建流程（camera-version）
> 你使用的是**带摄像头**的版本，以下是从零到运行的完整操作流程。
### 4.1 为什么需要飞线？
**这是 CogNog V1.0 PCB 的设计缺陷，不是故意的设计。**
ESP32-S3 的 PSRAM 使用 Octal SPI 接口，占用了 GPIO 35/36/37。但 V1.0 PCB 把摄像头的三个信号线（D7、VSYNC、HREF）也接到了这三个引脚上。当摄像头和 PSRAM 同时工作时，引脚冲突导致摄像头无法使用。
**所有使用 CogNog V1.0 PCB + 摄像头的用户都必须飞线。** 目前只有 V1.0 版本的 PCB。
### 4.2 硬件准备清单
| 部件 | 说明 |
|------|------|
| CogNog V1.0 PCB | 需要飞线修改 |
| ESP32-S3 模组 | 带 PSRAM（如 N16R8） |
| OV3660 摄像头模组 | DVP 接口 + 排线 + 转接板 |
| Grove Vision AI V2 | 人脸追踪模块 + 15pin 排线适配器 |
| INMP441 麦克风 | I2S 数字麦克风 |
| 扬声器 | I2S 输出 |
| 9x 舵机 | **必须 180° 标准舵机**（如 KPower M0090 或 MG90S 180° 金属齿轮版），详见下方舵机选型说明 |
| Raspberry Pi Pico (RP2040) | 舵机控制 |
| 6 Pin PicoBlade 连接线 | RP2040 ↔ ESP32 UART |
### 4.3 PCB 飞线操作（3 根线）
在 CogNog V1.0 PCB 上，**切断原走线并飞线到新引脚**：
| 原始引脚 | 飞线到 | 摄像头信号 | 说明 |
|----------|--------|-----------|------|
| GPIO 35 | **GPIO 14** | CAMERA_PIN_D7 | 数据位 7 |
| GPIO 36 | **GPIO 41** | CAMERA_PIN_VSYNC | 垂直同步 |
| GPIO 37 | **GPIO 42** | CAMERA_PIN_HREF | 行参考信号 |
> **注意**：camera-version 分支还释放了 JTAG 引脚（GPIO 39-42）给摄像头使用，代码中通过 `gpio_reset_pin()` 实现。
### 4.4 代码下载
```bash
# 第 1 步：克隆 camera-version 分支（ESP32 固件 + RP2040 代码都在里面）
git clone -b camera-version https://github.com/will-cogley/CogletESP.git
# 第 2 步：克隆主仓库（文档、PCB 原理图、3D 打印件）
git clone https://github.com/will-cogley/Coglet.git
```
克隆后的目录结构：
```
CogletESP/ # camera-version 分支
├── main/
│ └── boards/
│ └── bread-compact-wifi-s3cam/
│ ├── config.h # GPIO 引脚映射（飞线后的版本）
│ └── compact_wifi_board_s3cam.cc # 板级初始化（禁用 LCD，启用摄像头）
├── components/
├── RP2040/ # RP2040 MicroPython 代码
│ ├── main.py # 主控程序、舵机状态机、UART 通信（ESP32 + Grove Vision AI V2）
│ ├── servoclass.py # 舵机驱动（PWM 50Hz，平滑运动）
│ └── animation.py # 动画定义（思考、开心、热身等）
├── CMakeLists.txt
└── sdkconfig
```
### 4.5 编译烧录 ESP32 固件（macOS）
```bash
# 1. 进入项目目录
cd CogletESP
# 2. 激活 IDF 环境
source ~/esp/esp-idf/export.sh
# 3. 设置目标芯片
idf.py set-target esp32s3
# 4. 配置板级选项
idf.py menuconfig
# → 找到板级配置，选择 bread-compact-wifi-s3cam
# 5. 编译
idf.py build
# 6. 查看可用串口
ls /dev/cu.usb*
# 7. 烧录并监控
idf.py -p /dev/cu.usbmodem14101 flash monitor
# 串口名根据实际替换（/dev/cu.usbmodem* 或 /dev/cu.usbserial*）
# 8. 退出监视器：Ctrl + ]
```
### 4.6 烧录 RP2040（MicroPython）
**第 1 步：刷入 MicroPython 固件**
下载地址：https://micropython.org/download/RPI_PICO/
**版本选择**：下载 **v1.24.x ~ v1.25.x 稳定版** `.uf2` 文件。代码只用基础 API（Pin、PWM、UART），v1.20+ 稳定版都兼容。**不要下载 preview/nightly 预览版。**
**硬件说明**：CogNog V1.0 PCB 上 RP2040 是直接焊接的（不是独立的 Raspberry Pi Pico 开发板），PCB 上有两个相关按键：
- **SW1（Boot Switch）**：位于 Flash Memory 旁，即 BOOTSEL 功能，用于进入 UF2 烧录模式
- **Run Switch**：位于 RP2040 旁，连接 RUN 引脚，即**复位按钮**。使用场景：
- 上传新的 .py 文件后，按一下重启 RP2040 使新代码生效（等同于 `mpremote reset`）
- 舵机行为异常时，按一下重启恢复到初始状态
- 配合 SW1 进入烧录模式（见下方方式 B）
操作（macOS / Windows 通用）：
**方式 A — USB 未连接时：**
1. **按住 SW1（Boot Switch）** → 通过 RP2040 的 USB-C 口连接电脑 → 松开 SW1
2. 电脑上出现 `RPI-RP2` U 盘（macOS 在 Finder，Windows 在资源管理器）
3. 将 `.uf2` 文件拖入 U 盘
4. RP2040 自动重启
**方式 B — USB 已连接时（更常用）：**
1. **按住 SW1（Boot Switch）不松手**
2. **按一下 Run Switch（复位）然后松开**（只按一下，不用保持）
3. **松开 SW1** — 整个过程约 1-2 秒，关键是按 Run Switch 时 SW1 必须处于按下状态
4. 电脑上出现 `RPI-RP2` U 盘
5. 将 `.uf2` 文件拖入 U 盘
6. RP2040 自动重启
> **原理**：Run Switch 让 RP2040 复位重启，重启瞬间 SW1 被按着 → RP2040 检测到 BOOTSEL 为低电平 → 进入 UF2 bootloader 模式。
>
> **提示**：如果 RP2040 已经刷过 MicroPython 且只需要更新 .py 文件，不需要重新进入 bootloader 模式，直接用 `mpremote` 上传即可（见第 2 步）。
**第 2 步：上传 .py 文件**
> **注意**：`CogletESP` 分支有 4 个文件（含 `coms.py`），`camera-version` 分支有 3 个文件（无 `coms.py`）。请使用对应分支的文件。
#### macOS
方式 A — mpremote（推荐）：
```bash
pip install mpremote
cd CogletESP/RP2040
mpremote cp main.py :main.py
mpremote cp servoclass.py :servoclass.py
mpremote cp coms.py :coms.py # 仅 CogletESP 分支需要
mpremote cp animation.py :animation.py
mpremote reset # 重启运行
mpremote repl # 查看串口输出（调试用）
```
mpremote 本质就是一个串口文件传输工具，把 .py 文件拷贝到 Pico 的文件系统里。Pico 上电后自动运行 main.py，不需要搭建任何 MicroPython 编译环境。
方式 B — Thonny（图形界面）：
```bash
brew install --cask thonny
```
1. 打开 Thonny → 右下角选择 `MicroPython (Raspberry Pi Pico)`
2. 逐个打开 .py 文件 → 另存为 → 选择 `Raspberry Pi Pico` → 保存同名
3. 保存完成后 Pico 重启自动运行 `main.py`
#### Windows
方式 A — mpremote（推荐）：
```powershell
# 1. 安装 mpremote（需要 Python 3.x，从 python.org 下载安装，勾选 Add to PATH）
pip install mpremote
# 2. 进入 RP2040 目录
cd CogletESP\RP2040
# 3. 上传文件（Pico 连接 USB 后 Windows 会分配 COM 口，mpremote 自动识别）
mpremote cp main.py :main.py
mpremote cp servoclass.py :servoclass.py
mpremote cp coms.py :coms.py # 仅 CogletESP 分支需要
mpremote cp animation.py :animation.py
mpremote reset # 重启运行
mpremote repl # 查看串口输出（调试用，Ctrl+] 退出）
```
> **Windows 注意事项**：
> - 如果 `mpremote` 找不到设备，打开**设备管理器 → 端口(COM 和 LPT)**，确认 Pico 对应的 COM 口（如 COM3），然后用 `mpremote connect COM3 cp main.py :main.py` 指定端口
> - 如果设备管理器中看不到 COM 口，需要安装驱动：MicroPython 固件刷入后 Pico 通常免驱，若未识别可尝试拔插 USB 或换 USB 口
方式 B — Thonny（图形界面，适合新手）：
1. 下载安装 Thonny：https://thonny.org/ （选择 Windows 版本）
2. 打开 Thonny → 右下角选择 `MicroPython (Raspberry Pi Pico)`
3. 逐个打开 .py 文件 → 文件 → 另存为 → 选择 `Raspberry Pi Pico` → 保存同名
4. 保存完成后 Pico 重启自动运行 `main.py`
### 4.7 配置 Grove Vision AI V2（人脸追踪）
**不需要手动刷底层固件**，但需要部署一个人脸检测模型。
**操作步骤：**
1. 用 USB-C 线将 Grove Vision AI V2 连接到电脑
2. 用 Chrome 浏览器访问 **SenseCraft AI 平台**：https://sensecraft.seeed.cc/
3. 模型下载地址：https://sensecraft.seeed.cc/ai/model/deploy?id=60094&uniform_type=36&name=%E4%BA%BA%E8%84%B8%E6%A3%80%E6%B5%8B&adapteds=11&adapteds=12&adapteds=14&task=1
4. 选择一个**人脸检测模型**（推荐 YOLO Face Detection，输入尺寸 **224x224**）
5. 点击部署，等待上传完成
6. 断开 USB，将模块接回 RP2040 的 UART
**为什么是 224x224？** 代码中 `pixel_centre = 112` 即 224/2，表明模型输入分辨率为 224x224。
**工作原理：**
```
RP2040 发送 AT+INVOKE=1,0,1 (SSCMA AT 协议)
↓
Grove Vision AI V2 运行人脸检测推理
↓
返回 JSON，包含 "boxes" 字段 → [x, y, w, h, ...]
↓
RP2040 计算人脸偏移量：x_offset = boxes[0] - 112, y_offset = boxes[1] - 112
↓
驱动舵机追踪：
- EYL/EYR（眼球左右） → 跟随 x_offset
- PIT（头部俯仰） → 跟随 y_offset
- YAW（底座旋转） → 眼球偏离中心 >20° 时延迟跟随
↓
deadzone = 20 像素（避免微小抖动触发舵机）
```
### 4.8 硬件连接
```
┌─────────────┐ UART 115200 ┌──────────────┐ UART 921600 ┌───────────────────┐
│ ESP32-S3 │◄──────────────────►│ RP2040 │◄──────────────────►│ Grove Vision AI V2│
│ │ TX→RX(GP5) │ (Pico) │ TX(GP0)→RX │ │
│ AI/语音 │ RX←TX(GP4) │ 舵机控制 │ RX(GP1)←TX │ 人脸检测 │
│ WiFi/网络 │ GND──GND │ 动画状态机 │ │ YOLO 模型 │
└─────────────┘ └──────────────┘ └───────────────────┘
│
PWM 引脚 x9
│
┌──────┴──────┐
│ 9 个舵机 │
│ EYL/EYR/PIT │
│ YAW/MOU/LID │
│ + 耳朵等 │
└─────────────┘
```
ESP32 通过 UART 发送状态字符串给 RP2040：
`neutral` / `idle` / `listening` / `speaking` / `thinking` / `happy`
→ RP2040 根据状态执行对应舵机动画
### 4.9 舵机选型说明（重要）
> **实测踩坑记录**：使用 MG90S 360° 连续旋转版舵机后，耳朵舵机转到目标角度后无法停止，持续堵转导致齿轮发出刺耳声音、舵机严重发烫，有烧毁风险。更换为 180° 标准舵机后问题解决。
#### 必须使用 180° 标准舵机的原因
Coglet 的 9 个舵机全部采用**角度控制**模式（代码中 `set_target(角度)` 命令舵机转到指定角度并保持），这只有 180° 标准舵机才能实现。
**180° 标准舵机 vs 360° 连续旋转舵机的核心区别：**
| 对比项 | 180° 标准舵机（如 KPower M0090） | 360° 连续旋转舵机（如 MG90S 360° 改装版） |
|--------|-------------------------------|-------------------------------------|
| **内部结构** | 有电位器（可变电阻），实时反馈当前角度 | 拆除/固定了电位器，失去角度反馈能力 |
| **PWM 信号含义** | 对应**目标角度**（1ms=0°, 1.5ms=90°, 2ms=180°） | 对应**旋转速度和方向**（1.5ms=停止, <1.5ms=反转, >1.5ms=正转） |
| **控制逻辑** | 闭环控制：收到 PWM → 对比目标与当前角度 → 自动转到位停住 | 开环控制：收到 PWM → 按速度持续旋转，永不停止 |
| **能否锁定位置** | 能，到达角度后施力保持 | 不能，停转后无保持力，外力可推动 |
| **旋转范围** | 0°~180° 精确定位 | 360° 无限旋转，无法精确定位 |
#### 为什么 360° 舵机无法通过代码适配？
网上有"定时转动"方案（全速旋转 N 毫秒 → 发停止信号），原理是 `转动时间 = (500ms / 360°) × 目标角度`。但这对 Coglet **完全不可行**：
1. **误差累积无法修正**：没有位置反馈，每次转动误差 ±5°~10°。Coglet 的动画是高频反复运动（说话时嘴巴每 250ms 开合一次、耳朵每秒摆动），运行几分钟后累积误差可达几百度，位置完全跑飞
2. **个体差异大**：每台舵机电机特性、齿轮间隙不同，无法统一校准"时间-角度"关系。换一台舵机就要重新标定
3. **无法保持位置**：代码中头部俯仰、眼球等需要停在某个角度并持续施力保持。360° 舵机停转后没有保持力，手一碰就偏了
4. **实时性失效**：Coglet 需要实时响应 AI 状态变化动态调整角度，定时方案无法做到实时同步
5. **负载和温度影响转速**：电压波动、温度变化都会改变旋转速度，使时间估算更不准确
#### 推荐舵机型号
| 型号 | 说明 | 参考价格 |
|------|------|---------|
| **KPower M0090**（官方推荐） | 9g 模拟金属齿轮 180° 舵机 | ~¥8-12/个 |
| **MG90S 180° 金属齿轮版** | 注意必须是 **180° 版**，不带 "continuous/360" 字样 | ~¥5-8/个 |
> **购买注意**：MG90S 有 180° 和 360° 两个版本，外观完全一样。购买时务必确认商品标题或参数中标注 "180°"，避免买到 360° 连续旋转版。
#### 9 个舵机角度范围详解
代码中所有舵机的 PWM 映射为 0°~180°（脉宽 500μs~2500μs），各舵机通过 `min_angle` / `max_angle` 限定实际运动范围（源码：`animation.py` 第 14-23 行）：
| 舵机代号 | 功能 | GPIO 引脚 | 角度范围 | 最大跨度 | 运动说明 |
|---------|------|----------|---------|---------|---------|
| **YAW** | 底座旋转 | GP6 | 10°~170° | 160° | 头部左右转动，人脸追踪时由 Grove Vision AI V2 驱动 |
| **ROL** | 颈部侧倾 | GP7 | 30°~120° | 90° | 头部左右歪头，happy 状态下 70°↔110° 摆动 |
| **PIT** | 头部俯仰 | GP8 | 1°~80° | 79° | 头部上下点头，thinking 状态下 50°↔80° 摆动 |
| **MOU** | 嘴巴开合 | GP19 | 5°~150° | 145° | speaking 时 70°↔130° 每 250ms 切换（张嘴/闭嘴） |
| **EYL** | 左眼球 | GP12 | 30°~150° | 120° | 左右看，人脸追踪时跟随 x 偏移量 |
| **EYR** | 右眼球 | GP13 | 30°~150° | 120° | 与 EYL 同步运动 |
| **LID** | 眼睑 | GP14 | 30°~160° | 130° | 30°=闭眼，110°~160°=睁眼，随机眨眼动画 |
| **EAL** | 左耳 | GP15 | 60°~150° | 90° | happy 状态下 100°↔160° 摆动 |
| **EAR** | 右耳 | GP16 | 30°~120° | 90° | happy 状态下 60°↔120° 摆动 |
> **注意**：以上角度范围是代码中的软件限位，基于官方 3D 打印件和 KPower M0090 舵机校准。如果使用其他舵机或自制外壳，可能需要调整 `animation.py` 中的 `min_angle` / `max_angle` 避免堵转。
> **MOU 引脚差异**：`CogletESP` 分支 MOU 使用 **GP19**，`camera-version` 分支 MOU 使用 **GP9**。请根据实际 PCB 版本确认。
#### 校准模式使用方法
CogNog V1.0 PCB 上有一个 **SW11（Calibration Switch）** 拨动开关，连接 RP2040 的 GPIO20。代码中 GPIO20 配置了内部上拉电阻：
```python
mode = Pin(20, Pin.IN, Pin.PULL_UP)
is_calibrating = (mode.value() == 1) # 高电平 = 校准模式
```
**SW11 逻辑：**
| SW11 状态 | GPIO20 电平 | 模式 | 表现 |
|----------|-----------|------|------|
| 断开（GPIO20 悬空） | 高（内部上拉到 1） | **校准模式** | 所有舵机归 90° 中位，LED 闪烁 |
| 接通（GPIO20 接 GND） | 低 | **正常运行模式** | 响应 ESP32 状态指令，执行动画 |
> **提示**：如果不确定 SW11 哪个位置对应哪种模式，两个位置都试一下，**LED 闪烁的那个就是校准模式**。
#### 校准与组装流程
**前提**：RP2040 已刷入 MicroPython 固件并上传 .py 文件。
```
第 1 步：进入校准模式
└→ 将 SW11 拨到校准位置（LED 闪烁）
第 2 步：通电
└→ 给 RP2040 和舵机供电
└→ 所有 9 个舵机自动转到 90° 中位（staggered startup，逐个上电避免电流冲击）
第 3 步：安装舵机臂
└→ 在 90° 中位状态下，将舵机臂安装到需要的朝向
└→ 拧紧舵机臂固定螺丝
└→ ⚠️ 关键：必须在通电校准状态下安装，否则角度对不上
第 4 步：组装机械结构
└→ 将舵机装入 3D 打印外壳
└→ 连接所有机械连杆
第 5 步：验证
└→ 将 SW11 拨回正常运行模式
└→ 观察舵机是否能正常运动，无堵转
└→ 如有堵转，调整 animation.py 中对应舵机的 min_angle / max_angle
第 6 步：连接 ESP32
└→ 接上 ESP32 UART（TX→GP5, RX→GP4）
└→ ESP32 发送状态指令，舵机开始响应 AI 对话动画
```
> **校准模式不需要断开 ESP32**：校准模式在主循环中每帧都强制覆盖舵机状态（`main.py` 第 153-155 行），即使 ESP32 发送了指令也会被忽略。
### 4.10 camera-version 的代价与限制
| 项目 | 说明 |
|------|------|
| **LCD 屏被禁用** | 摄像头占用了原 LCD 引脚，代码用 DummyDisplay 空实现替代 |
| **音频引脚全部重映射** | MIC 从 GPIO 1/2/42 移到 4/5/6，SPK 从 GPIO 39/40/41 移到 7/15/16 |
| **内存压力** | 摄像头 + PSRAM 同时运行可能偶发内存不足（作者提到 "appears to run out of memory sometimes"） |
| **摄像头画面翻转** | 已通过代码设置 `SetHMirror(true)` + `SetVFlip(true)` 修正 |
---
## 五、无摄像头版本搭建流程（CogletESP 分支）
> 如果不使用 ESP32 OV3660 摄像头，想保留 LCD 显示，使用此方案。
### 5.1 代码下载
```bash
git clone -b CogletESP https://github.com/will-cogley/CogletESP.git
git clone https://github.com/will-cogley/Coglet.git
```
### 5.2 ESP32 编译烧录
与摄像头版本相同（参见 4.5），但 menuconfig 中的板级配置可能不同，PCB **无需飞线**。
### 5.3 RP2040 烧录
MicroPython 固件刷入方式与摄像头版本相同（参见 4.6），但 **RP2040 代码两个分支不同**：
- `CogletESP` 分支有 **4 个文件**：`main.py`、`servoclass.py`、`coms.py`、`animation.py`
- `camera-version` 分支只有 **3 个文件**（无 `coms.py`，UART 通信集成在 `main.py` 中）
请使用对应分支的 RP2040 目录文件上传。
### 5.4 Grove Vision AI V2
与摄像头版本完全相同（参见 4.7），Grove Vision AI V2 的人脸追踪功能由 RP2040 端控制，与 ESP32 分支选择无关。
---
## 六、常见问题
| 问题 | 解决方案 |
|------|---------|
| Python 环境冲突 | `idf.py fullclean` 后重新编译 |
| Bootloader CMake 缓存不匹配 | `rm -rf build/bootloader build/bootloader-prefix` |
| 找不到串口设备 | 安装 CP2102/CH340 驱动，检查 USB 线是否支持数据传输 |
| menuconfig 中找不到板级配置 | 确认在正确的分支（`camera-version` 或 `CogletESP`） |
| 摄像头不工作 | 确认已完成 3 根飞线（35→14, 36→41, 37→42）并使用 `camera-version` 分支 |
| 偶发内存不足 | camera-version 已知问题，摄像头 + PSRAM 同时运行时可能出现 |
| Grove Vision AI V2 无响应 | 确认已通过 SenseCraft 部署人脸检测模型，UART 波特率 921600 |
| 舵机不动 | 检查 RP2040 是否正确上传 .py 文件（CogletESP 分支 4 个 / camera-version 分支 3 个），UART 连接是否正常 |
| 舵机堵转、发烫、齿轮刺耳声 | **使用了 360° 连续旋转舵机**，必须更换为 180° 标准舵机（详见 4.9 舵机选型说明） |
| 舵机持续朝一个方向旋转不停 | 同上，360° 舵机收到非 90° 信号会持续旋转。180° 舵机会转到目标角度后自动停住 |
---
## 七、参考资源
| 资源 | 地址 |
|------|------|
| Coglet GitHub 主仓库 | https://github.com/will-cogley/Coglet |
| CogletESP 固件仓库 | https://github.com/will-cogley/CogletESP |
| 小智 ESP32 原版 | https://github.com/78/xiaozhi-esp32 |
| 眼球机构参考 | https://github.com/will-cogley/EyeMech_Epsilon |
| Coglet 组装视频 | https://www.youtube.com/watch?v=-7I-jFSNP2E |
| MicroPython 固件下载 | https://micropython.org/download/RPI_PICO/ |
| ESP-IDF 官方文档 | https://docs.espressif.com/projects/esp-idf/zh_CN/v5.4.2/ |
| MicroPython 官方文档 | https://docs.micropython.org/en/latest/ |
| mpremote 工具文档 | https://docs.micropython.org/en/latest/reference/mpremote.html |
| SenseCraft AI 平台（Grove Vision AI V2 模型部署） | https://sensecraft.seeed.cc/ |
| Grove Vision AI V2 Wiki | https://wiki.seeedstudio.com/grove_vision_ai_v2/ |

22
LICENSE Normal file
View File

@ -0,0 +1,22 @@
MIT License
Copyright (c) 2025 Shenzhen Xinzhi Future Technology Co., Ltd.
Copyright (c) 2025 Project Contributors
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

172
README.md Normal file
View File

@ -0,0 +1,172 @@
# An MCP-based Chatbot
(English | [中文](README_zh.md) | [日本語](README_ja.md))
## Introduction
👉 [Human: Give AI a camera vs AI: Instantly finds out the owner hasn't washed hair for three days【bilibili】](https://www.bilibili.com/video/BV1bpjgzKEhd/)
👉 [Handcraft your AI girlfriend, beginner's guide【bilibili】](https://www.bilibili.com/video/BV1XnmFYLEJN/)
As a voice interaction entry, the XiaoZhi AI chatbot leverages the AI capabilities of large models like Qwen / DeepSeek, and achieves multi-terminal control via the MCP protocol.
<img src="docs/mcp-based-graph.jpg" alt="Control everything via MCP" width="320">
## Version Notes
The current v2 version is incompatible with the v1 partition table, so it is not possible to upgrade from v1 to v2 via OTA. For partition table details, see [partitions/v2/README.md](partitions/v2/README.md).
All hardware running v1 can be upgraded to v2 by manually flashing the firmware.
The stable version of v1 is 1.9.2. You can switch to v1 by running `git checkout v1`. The v1 branch will be maintained until February 2026.
### Features Implemented
- Wi-Fi / ML307 Cat.1 4G
- Offline voice wake-up [ESP-SR](https://github.com/espressif/esp-sr)
- Supports two communication protocols ([Websocket](docs/websocket.md) or MQTT+UDP)
- Uses OPUS audio codec
- Voice interaction based on streaming ASR + LLM + TTS architecture
- Speaker recognition, identifies the current speaker [3D Speaker](https://github.com/modelscope/3D-Speaker)
- OLED / LCD display, supports emoji display
- Battery display and power management
- Multi-language support (Chinese, English, Japanese)
- Supports ESP32-C3, ESP32-S3, ESP32-P4 chip platforms
- Device-side MCP for device control (Speaker, LED, Servo, GPIO, etc.)
- Cloud-side MCP to extend large model capabilities (smart home control, PC desktop operation, knowledge search, email, etc.)
- Customizable wake words, fonts, emojis, and chat backgrounds with online web-based editing ([Custom Assets Generator](https://github.com/78/xiaozhi-assets-generator))
## Hardware
### Breadboard DIY Practice
See the Feishu document tutorial:
👉 ["XiaoZhi AI Chatbot Encyclopedia"](https://ccnphfhqs21z.feishu.cn/wiki/F5krwD16viZoF0kKkvDcrZNYnhb?from=from_copylink)
Breadboard demo:
![Breadboard Demo](docs/v1/wiring2.jpg)
### Supports 70+ Open Source Hardware (Partial List)
- <a href="https://oshwhub.com/li-chuang-kai-fa-ban/li-chuang-shi-zhan-pai-esp32-s3-kai-fa-ban" target="_blank" title="LiChuang ESP32-S3 Development Board">LiChuang ESP32-S3 Development Board</a>
- <a href="https://github.com/espressif/esp-box" target="_blank" title="Espressif ESP32-S3-BOX3">Espressif ESP32-S3-BOX3</a>
- <a href="https://docs.m5stack.com/zh_CN/core/CoreS3" target="_blank" title="M5Stack CoreS3">M5Stack CoreS3</a>
- <a href="https://docs.m5stack.com/en/atom/Atomic%20Echo%20Base" target="_blank" title="AtomS3R + Echo Base">M5Stack AtomS3R + Echo Base</a>
- <a href="https://gf.bilibili.com/item/detail/1108782064" target="_blank" title="Magic Button 2.4">Magic Button 2.4</a>
- <a href="https://www.waveshare.net/shop/ESP32-S3-Touch-AMOLED-1.8.htm" target="_blank" title="Waveshare ESP32-S3-Touch-AMOLED-1.8">Waveshare ESP32-S3-Touch-AMOLED-1.8</a>
- <a href="https://github.com/Xinyuan-LilyGO/T-Circle-S3" target="_blank" title="LILYGO T-Circle-S3">LILYGO T-Circle-S3</a>
- <a href="https://oshwhub.com/tenclass01/xmini_c3" target="_blank" title="XiaGe Mini C3">XiaGe Mini C3</a>
- <a href="https://oshwhub.com/movecall/cuican-ai-pendant-lights-up-y" target="_blank" title="Movecall CuiCan ESP32S3">CuiCan AI Pendant</a>
- <a href="https://github.com/WMnologo/xingzhi-ai" target="_blank" title="WMnologo-Xingzhi-1.54">WMnologo-Xingzhi-1.54TFT</a>
- <a href="https://www.seeedstudio.com/SenseCAP-Watcher-W1-A-p-5979.html" target="_blank" title="SenseCAP Watcher">SenseCAP Watcher</a>
- <a href="https://www.bilibili.com/video/BV1BHJtz6E2S/" target="_blank" title="ESP-HI Low Cost Robot Dog">ESP-HI Low Cost Robot Dog</a>
<div style="display: flex; justify-content: space-between;">
<a href="docs/v1/lichuang-s3.jpg" target="_blank" title="LiChuang ESP32-S3 Development Board">
<img src="docs/v1/lichuang-s3.jpg" width="240" />
</a>
<a href="docs/v1/espbox3.jpg" target="_blank" title="Espressif ESP32-S3-BOX3">
<img src="docs/v1/espbox3.jpg" width="240" />
</a>
<a href="docs/v1/m5cores3.jpg" target="_blank" title="M5Stack CoreS3">
<img src="docs/v1/m5cores3.jpg" width="240" />
</a>
<a href="docs/v1/atoms3r.jpg" target="_blank" title="AtomS3R + Echo Base">
<img src="docs/v1/atoms3r.jpg" width="240" />
</a>
<a href="docs/v1/magiclick.jpg" target="_blank" title="Magic Button 2.4">
<img src="docs/v1/magiclick.jpg" width="240" />
</a>
<a href="docs/v1/waveshare.jpg" target="_blank" title="Waveshare ESP32-S3-Touch-AMOLED-1.8">
<img src="docs/v1/waveshare.jpg" width="240" />
</a>
<a href="docs/v1/lilygo-t-circle-s3.jpg" target="_blank" title="LILYGO T-Circle-S3">
<img src="docs/v1/lilygo-t-circle-s3.jpg" width="240" />
</a>
<a href="docs/v1/xmini-c3.jpg" target="_blank" title="XiaGe Mini C3">
<img src="docs/v1/xmini-c3.jpg" width="240" />
</a>
<a href="docs/v1/movecall-cuican-esp32s3.jpg" target="_blank" title="CuiCan">
<img src="docs/v1/movecall-cuican-esp32s3.jpg" width="240" />
</a>
<a href="docs/v1/wmnologo_xingzhi_1.54.jpg" target="_blank" title="WMnologo-Xingzhi-1.54">
<img src="docs/v1/wmnologo_xingzhi_1.54.jpg" width="240" />
</a>
<a href="docs/v1/sensecap_watcher.jpg" target="_blank" title="SenseCAP Watcher">
<img src="docs/v1/sensecap_watcher.jpg" width="240" />
</a>
<a href="docs/v1/esp-hi.jpg" target="_blank" title="ESP-HI Low Cost Robot Dog">
<img src="docs/v1/esp-hi.jpg" width="240" />
</a>
</div>
## Software
### Firmware Flashing
For beginners, it is recommended to use the firmware that can be flashed without setting up a development environment.
The firmware connects to the official [xiaozhi.me](https://xiaozhi.me) server by default. Personal users can register an account to use the Qwen real-time model for free.
👉 [Beginner's Firmware Flashing Guide](https://ccnphfhqs21z.feishu.cn/wiki/Zpz4wXBtdimBrLk25WdcXzxcnNS)
### Development Environment
- Cursor or VSCode
- Install ESP-IDF plugin, select SDK version 5.4 or above
- Linux is better than Windows for faster compilation and fewer driver issues
- This project uses Google C++ code style, please ensure compliance when submitting code
### Developer Documentation
- [Custom Board Guide](docs/custom-board.md) - Learn how to create custom boards for XiaoZhi AI
- [MCP Protocol IoT Control Usage](docs/mcp-usage.md) - Learn how to control IoT devices via MCP protocol
- [MCP Protocol Interaction Flow](docs/mcp-protocol.md) - Device-side MCP protocol implementation
- [MQTT + UDP Hybrid Communication Protocol Document](docs/mqtt-udp.md)
- [A detailed WebSocket communication protocol document](docs/websocket.md)
## Large Model Configuration
If you already have a XiaoZhi AI chatbot device and have connected to the official server, you can log in to the [xiaozhi.me](https://xiaozhi.me) console for configuration.
👉 [Backend Operation Video Tutorial (Old Interface)](https://www.bilibili.com/video/BV1jUCUY2EKM/)
## Related Open Source Projects
For server deployment on personal computers, refer to the following open-source projects:
- [xinnan-tech/xiaozhi-esp32-server](https://github.com/xinnan-tech/xiaozhi-esp32-server) Python server
- [joey-zhou/xiaozhi-esp32-server-java](https://github.com/joey-zhou/xiaozhi-esp32-server-java) Java server
- [AnimeAIChat/xiaozhi-server-go](https://github.com/AnimeAIChat/xiaozhi-server-go) Golang server
Other client projects using the XiaoZhi communication protocol:
- [huangjunsen0406/py-xiaozhi](https://github.com/huangjunsen0406/py-xiaozhi) Python client
- [TOM88812/xiaozhi-android-client](https://github.com/TOM88812/xiaozhi-android-client) Android client
- [100askTeam/xiaozhi-linux](http://github.com/100askTeam/xiaozhi-linux) Linux client by 100ask
- [78/xiaozhi-sf32](https://github.com/78/xiaozhi-sf32) Bluetooth chip firmware by Sichuan
- [QuecPython/solution-xiaozhiAI](https://github.com/QuecPython/solution-xiaozhiAI) QuecPython firmware by Quectel
Custom Assets Tools:
- [78/xiaozhi-assets-generator](https://github.com/78/xiaozhi-assets-generator) Custom Assets Generator (Wake words, fonts, emojis, backgrounds)
## About the Project
This is an open-source ESP32 project, released under the MIT license, allowing anyone to use it for free, including for commercial purposes.
We hope this project helps everyone understand AI hardware development and apply rapidly evolving large language models to real hardware devices.
If you have any ideas or suggestions, please feel free to raise Issues or join the QQ group: 1011329060
## Star History
<a href="https://star-history.com/#78/xiaozhi-esp32&Date">
<picture>
<source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/svg?repos=78/xiaozhi-esp32&type=Date&theme=dark" />
<source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/svg?repos=78/xiaozhi-esp32&type=Date" />
<img alt="Star History Chart" src="https://api.star-history.com/svg?repos=78/xiaozhi-esp32&type=Date" />
</picture>
</a>

168
README_ja.md Normal file
View File

@ -0,0 +1,168 @@
# MCP ベースのチャットボット
（日本語 | [中文](README_zh.md) | [English](README.md)）
## はじめに
👉 [人間：AIにカメラを装着 vs AI：その場で飼い主が3日間髪を洗っていないことを発見【bilibili】](https://www.bilibili.com/video/BV1bpjgzKEhd/)
👉 [手作りでAIガールフレンドを作る、初心者入門チュートリアル【bilibili】](https://www.bilibili.com/video/BV1XnmFYLEJN/)
シャオジーAIチャットボットは音声インタラクションの入口として、Qwen / DeepSeekなどの大規模モデルのAI能力を活用し、MCPプロトコルを通じてマルチエンド制御を実現します。
<img src="docs/mcp-based-graph.jpg" alt="MCPであらゆるものを制御" width="320">
## バージョンノート
現在のv2バージョンはv1パーティションテーブルと互換性がないため、v1からv2へOTAでアップグレードすることはできません。パーティションテーブルの詳細については、[partitions/v2/README.md](partitions/v2/README.md)をご参照ください。
v1を実行しているすべてのハードウェアは、ファームウェアを手動で書き込むことでv2にアップグレードできます。
v1の安定版は1.9.2です。`git checkout v1`でv1に切り替えることができます。v1ブランチは2026年2月まで継続的にメンテナンスされます。
### 実装済み機能
- Wi-Fi / ML307 Cat.1 4G
- オフライン音声ウェイクアップ [ESP-SR](https://github.com/espressif/esp-sr)
- 2種類の通信プロトコルに対応（[Websocket](docs/websocket.md) または MQTT+UDP）
- OPUSオーディオコーデックを採用
- ストリーミングASR + LLM + TTSアーキテクチャに基づく音声インタラクション
- 話者認識、現在話している人を識別 [3D Speaker](https://github.com/modelscope/3D-Speaker)
- OLED / LCDディスプレイ、表情表示対応
- バッテリー表示と電源管理
- 多言語対応（中国語、英語、日本語）
- ESP32-C3、ESP32-S3、ESP32-P4チッププラットフォーム対応
- デバイス側MCPによるデバイス制御（音量・明るさ調整、アクション制御など）
- クラウド側MCPで大規模モデル能力を拡張（スマートホーム制御、PCデスクトップ操作、知識検索、メール送受信など）
- カスタマイズ可能なウェイクワード、フォント、絵文字、チャット背景、オンラインWeb編集に対応 ([カスタムアセットジェネレーター](https://github.com/78/xiaozhi-assets-generator))
## ハードウェア
### ブレッドボード手作り実践
Feishuドキュメントチュートリアルをご覧ください：
👉 [「シャオジーAIチャットボット百科事典」](https://ccnphfhqs21z.feishu.cn/wiki/F5krwD16viZoF0kKkvDcrZNYnhb?from=from_copylink)
ブレッドボードのデモ：
![ブレッドボードデモ](docs/v1/wiring2.jpg)
### 70種類以上のオープンソースハードウェアに対応（一部のみ表示）
- <a href="https://oshwhub.com/li-chuang-kai-fa-ban/li-chuang-shi-zhan-pai-esp32-s3-kai-fa-ban" target="_blank" title="立創・実戦派 ESP32-S3 開発ボード">立創・実戦派 ESP32-S3 開発ボード</a>
- <a href="https://github.com/espressif/esp-box" target="_blank" title="楽鑫 ESP32-S3-BOX3">楽鑫 ESP32-S3-BOX3</a>
- <a href="https://docs.m5stack.com/zh_CN/core/CoreS3" target="_blank" title="M5Stack CoreS3">M5Stack CoreS3</a>
- <a href="https://docs.m5stack.com/en/atom/Atomic%20Echo%20Base" target="_blank" title="AtomS3R + Echo Base">M5Stack AtomS3R + Echo Base</a>
- <a href="https://gf.bilibili.com/item/detail/1108782064" target="_blank" title="マジックボタン2.4">マジックボタン2.4</a>
- <a href="https://www.waveshare.net/shop/ESP32-S3-Touch-AMOLED-1.8.htm" target="_blank" title="微雪電子 ESP32-S3-Touch-AMOLED-1.8">微雪電子 ESP32-S3-Touch-AMOLED-1.8</a>
- <a href="https://github.com/Xinyuan-LilyGO/T-Circle-S3" target="_blank" title="LILYGO T-Circle-S3">LILYGO T-Circle-S3</a>
- <a href="https://oshwhub.com/tenclass01/xmini_c3" target="_blank" title="エビ兄さん Mini C3">エビ兄さん Mini C3</a>
- <a href="https://oshwhub.com/movecall/cuican-ai-pendant-lights-up-y" target="_blank" title="Movecall CuiCan ESP32S3">CuiCan AIペンダント</a>
- <a href="https://github.com/WMnologo/xingzhi-ai" target="_blank" title="無名科技Nologo-星智-1.54">無名科技Nologo-星智-1.54TFT</a>
- <a href="https://www.seeedstudio.com/SenseCAP-Watcher-W1-A-p-5979.html" target="_blank" title="SenseCAP Watcher">SenseCAP Watcher</a>
- <a href="https://www.bilibili.com/video/BV1BHJtz6E2S/" target="_blank" title="ESP-HI 超低コストロボット犬">ESP-HI 超低コストロボット犬</a>
<div style="display: flex; justify-content: space-between;">
<a href="docs/v1/lichuang-s3.jpg" target="_blank" title="立創・実戦派 ESP32-S3 開発ボード">
<img src="docs/v1/lichuang-s3.jpg" width="240" />
</a>
<a href="docs/v1/espbox3.jpg" target="_blank" title="楽鑫 ESP32-S3-BOX3">
<img src="docs/v1/espbox3.jpg" width="240" />
</a>
<a href="docs/v1/m5cores3.jpg" target="_blank" title="M5Stack CoreS3">
<img src="docs/v1/m5cores3.jpg" width="240" />
</a>
<a href="docs/v1/atoms3r.jpg" target="_blank" title="AtomS3R + Echo Base">
<img src="docs/v1/atoms3r.jpg" width="240" />
</a>
<a href="docs/v1/magiclick.jpg" target="_blank" title="マジックボタン2.4">
<img src="docs/v1/magiclick.jpg" width="240" />
</a>
<a href="docs/v1/waveshare.jpg" target="_blank" title="微雪電子 ESP32-S3-Touch-AMOLED-1.8">
<img src="docs/v1/waveshare.jpg" width="240" />
</a>
<a href="docs/v1/lilygo-t-circle-s3.jpg" target="_blank" title="LILYGO T-Circle-S3">
<img src="docs/v1/lilygo-t-circle-s3.jpg" width="240" />
</a>
<a href="docs/v1/xmini-c3.jpg" target="_blank" title="エビ兄さん Mini C3">
<img src="docs/v1/xmini-c3.jpg" width="240" />
</a>
<a href="docs/v1/movecall-cuican-esp32s3.jpg" target="_blank" title="CuiCan">
<img src="docs/v1/movecall-cuican-esp32s3.jpg" width="240" />
</a>
<a href="docs/v1/wmnologo_xingzhi_1.54.jpg" target="_blank" title="無名科技Nologo-星智-1.54">
<img src="docs/v1/wmnologo_xingzhi_1.54.jpg" width="240" />
</a>
<a href="docs/v1/sensecap_watcher.jpg" target="_blank" title="SenseCAP Watcher">
<img src="docs/v1/sensecap_watcher.jpg" width="240" />
</a>
<a href="docs/v1/esp-hi.jpg" target="_blank" title="ESP-HI 超低コストロボット犬">
<img src="docs/v1/esp-hi.jpg" width="240" />
</a>
</div>
## ソフトウェア
### ファームウェア書き込み
初心者の方は、まず開発環境を構築せずに書き込み可能なファームウェアを使用することをおすすめします。
ファームウェアはデフォルトで公式 [xiaozhi.me](https://xiaozhi.me) サーバーに接続します。個人ユーザーはアカウント登録でQwenリアルタイムモデルを無料で利用できます。
👉 [初心者向けファームウェア書き込みガイド](https://ccnphfhqs21z.feishu.cn/wiki/Zpz4wXBtdimBrLk25WdcXzxcnNS)
### 開発環境
- Cursor または VSCode
- ESP-IDFプラグインをインストールし、SDKバージョン5.4以上を選択
- LinuxはWindowsよりも優れており、コンパイルが速く、ドライバの問題も少ない
- 本プロジェクトはGoogle C++コードスタイルを採用、コード提出時は準拠を確認してください
### 開発者ドキュメント
- [カスタム開発ボードガイド](docs/custom-board.md) - シャオジーAI用のカスタム開発ボード作成方法
- [MCPプロトコルIoT制御使用法](docs/mcp-usage.md) - MCPプロトコルでIoTデバイスを制御する方法
- [MCPプロトコルインタラクションフロー](docs/mcp-protocol.md) - デバイス側MCPプロトコルの実装方法
- [MQTT + UDP ハイブリッド通信プロトコルドキュメント](docs/mqtt-udp.md)
- [詳細なWebSocket通信プロトコルドキュメント](docs/websocket.md)
## 大規模モデル設定
すでにシャオジーAIチャットボットデバイスをお持ちで、公式サーバーに接続済みの場合は、[xiaozhi.me](https://xiaozhi.me) コンソールで設定できます。
👉 [バックエンド操作ビデオチュートリアル（旧インターフェース）](https://www.bilibili.com/video/BV1jUCUY2EKM/)
## 関連オープンソースプロジェクト
個人PCでサーバーをデプロイする場合は、以下のオープンソースプロジェクトを参照してください：
- [xinnan-tech/xiaozhi-esp32-server](https://github.com/xinnan-tech/xiaozhi-esp32-server) Pythonサーバー
- [joey-zhou/xiaozhi-esp32-server-java](https://github.com/joey-zhou/xiaozhi-esp32-server-java) Javaサーバー
- [AnimeAIChat/xiaozhi-server-go](https://github.com/AnimeAIChat/xiaozhi-server-go) Golangサーバー
シャオジー通信プロトコルを利用した他のクライアントプロジェクト：
- [huangjunsen0406/py-xiaozhi](https://github.com/huangjunsen0406/py-xiaozhi) Pythonクライアント
- [TOM88812/xiaozhi-android-client](https://github.com/TOM88812/xiaozhi-android-client) Androidクライアント
- [100askTeam/xiaozhi-linux](http://github.com/100askTeam/xiaozhi-linux) 百問科技提供のLinuxクライアント
- [78/xiaozhi-sf32](https://github.com/78/xiaozhi-sf32) 思澈科技のBluetoothチップファームウェア
- [QuecPython/solution-xiaozhiAI](https://github.com/QuecPython/solution-xiaozhiAI) 移遠提供のQuecPythonファームウェア
## プロジェクトについて
これはエビ兄さんがオープンソースで公開しているESP32プロジェクトで、MITライセンスのもと、誰でも無料で、商用利用も可能です。
このプロジェクトを通じて、AIハードウェア開発を理解し、急速に進化する大規模言語モデルを実際のハードウェアデバイスに応用できるようになることを目指しています。
ご意見やご提案があれば、いつでもIssueを提出するか、QQグループ：1011329060 にご参加ください。
## スター履歴
<a href="https://star-history.com/#78/xiaozhi-esp32&Date">
<picture>
<source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/svg?repos=78/xiaozhi-esp32&type=Date&theme=dark" />
<source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/svg?repos=78/xiaozhi-esp32&type=Date" />
<img alt="Star History Chart" src="https://api.star-history.com/svg?repos=78/xiaozhi-esp32&type=Date" />
</picture>
</a>

168
README_zh.md Normal file
View File

@ -0,0 +1,168 @@
# An MCP-based Chatbot
（中文 | [English](README.md) | [日本語](README_ja.md)）
## 介绍
👉 [人类：给 AI 装摄像头 vs AI：当场发现主人三天没洗头【bilibili】](https://www.bilibili.com/video/BV1bpjgzKEhd/)
👉 [手工打造你的 AI 女友，新手入门教程【bilibili】](https://www.bilibili.com/video/BV1XnmFYLEJN/)
小智 AI 聊天机器人作为一个语音交互入口，利用 Qwen / DeepSeek 等大模型的 AI 能力，通过 MCP 协议实现多端控制。
<img src="docs/mcp-based-graph.jpg" alt="通过MCP控制万物" width="320">
### 版本说明
当前 v2 版本与 v1 版本分区表不兼容，所以无法从 v1 版本通过 OTA 升级到 v2 版本。分区表说明参见 [partitions/v2/README.md](partitions/v2/README.md)。
使用 v1 版本的所有硬件，可以通过手动烧录固件来升级到 v2 版本。
v1 的稳定版本为 1.9.2，可以通过 `git checkout v1` 来切换到 v1 版本，该分支会持续维护到 2026 年 2 月。
### 已实现功能
- Wi-Fi / ML307 Cat.1 4G
- 离线语音唤醒 [ESP-SR](https://github.com/espressif/esp-sr)
- 支持两种通信协议（[Websocket](docs/websocket.md) 或 MQTT+UDP）
- 采用 OPUS 音频编解码
- 基于流式 ASR + LLM + TTS 架构的语音交互
- 声纹识别，识别当前说话人的身份 [3D Speaker](https://github.com/modelscope/3D-Speaker)
- OLED / LCD 显示屏，支持表情显示
- 电量显示与电源管理
- 支持多语言（中文、英文、日文）
- 支持 ESP32-C3、ESP32-S3、ESP32-P4 芯片平台
- 通过设备端 MCP 实现设备控制（音量、灯光、电机、GPIO 等）
- 通过云端 MCP 扩展大模型能力（智能家居控制、PC桌面操作、知识搜索、邮件收发等）
- 自定义唤醒词、字体、表情与聊天背景，支持网页端在线修改 ([自定义Assets生成器](https://github.com/78/xiaozhi-assets-generator))
## 硬件
### 面包板手工制作实践
详见飞书文档教程：
👉 [《小智 AI 聊天机器人百科全书》](https://ccnphfhqs21z.feishu.cn/wiki/F5krwD16viZoF0kKkvDcrZNYnhb?from=from_copylink)
面包板效果图如下：
![面包板效果图](docs/v1/wiring2.jpg)
### 支持 70 多个开源硬件（仅展示部分）
- <a href="https://oshwhub.com/li-chuang-kai-fa-ban/li-chuang-shi-zhan-pai-esp32-s3-kai-fa-ban" target="_blank" title="立创·实战派 ESP32-S3 开发板">立创·实战派 ESP32-S3 开发板</a>
- <a href="https://github.com/espressif/esp-box" target="_blank" title="乐鑫 ESP32-S3-BOX3">乐鑫 ESP32-S3-BOX3</a>
- <a href="https://docs.m5stack.com/zh_CN/core/CoreS3" target="_blank" title="M5Stack CoreS3">M5Stack CoreS3</a>
- <a href="https://docs.m5stack.com/en/atom/Atomic%20Echo%20Base" target="_blank" title="AtomS3R + Echo Base">M5Stack AtomS3R + Echo Base</a>
- <a href="https://gf.bilibili.com/item/detail/1108782064" target="_blank" title="神奇按钮 2.4">神奇按钮 2.4</a>
- <a href="https://www.waveshare.net/shop/ESP32-S3-Touch-AMOLED-1.8.htm" target="_blank" title="微雪电子 ESP32-S3-Touch-AMOLED-1.8">微雪电子 ESP32-S3-Touch-AMOLED-1.8</a>
- <a href="https://github.com/Xinyuan-LilyGO/T-Circle-S3" target="_blank" title="LILYGO T-Circle-S3">LILYGO T-Circle-S3</a>
- <a href="https://oshwhub.com/tenclass01/xmini_c3" target="_blank" title="虾哥 Mini C3">虾哥 Mini C3</a>
- <a href="https://oshwhub.com/movecall/cuican-ai-pendant-lights-up-y" target="_blank" title="Movecall CuiCan ESP32S3">璀璨·AI 吊坠</a>
- <a href="https://github.com/WMnologo/xingzhi-ai" target="_blank" title="无名科技Nologo-星智-1.54">无名科技 Nologo-星智-1.54TFT</a>
- <a href="https://www.seeedstudio.com/SenseCAP-Watcher-W1-A-p-5979.html" target="_blank" title="SenseCAP Watcher">SenseCAP Watcher</a>
- <a href="https://www.bilibili.com/video/BV1BHJtz6E2S/" target="_blank" title="ESP-HI 超低成本机器狗">ESP-HI 超低成本机器狗</a>
<div style="display: flex; justify-content: space-between;">
<a href="docs/v1/lichuang-s3.jpg" target="_blank" title="立创·实战派 ESP32-S3 开发板">
<img src="docs/v1/lichuang-s3.jpg" width="240" />
</a>
<a href="docs/v1/espbox3.jpg" target="_blank" title="乐鑫 ESP32-S3-BOX3">
<img src="docs/v1/espbox3.jpg" width="240" />
</a>
<a href="docs/v1/m5cores3.jpg" target="_blank" title="M5Stack CoreS3">
<img src="docs/v1/m5cores3.jpg" width="240" />
</a>
<a href="docs/v1/atoms3r.jpg" target="_blank" title="AtomS3R + Echo Base">
<img src="docs/v1/atoms3r.jpg" width="240" />
</a>
<a href="docs/v1/magiclick.jpg" target="_blank" title="神奇按钮 2.4">
<img src="docs/v1/magiclick.jpg" width="240" />
</a>
<a href="docs/v1/waveshare.jpg" target="_blank" title="微雪电子 ESP32-S3-Touch-AMOLED-1.8">
<img src="docs/v1/waveshare.jpg" width="240" />
</a>
<a href="docs/v1/lilygo-t-circle-s3.jpg" target="_blank" title="LILYGO T-Circle-S3">
<img src="docs/v1/lilygo-t-circle-s3.jpg" width="240" />
</a>
<a href="docs/v1/xmini-c3.jpg" target="_blank" title="虾哥 Mini C3">
<img src="docs/v1/xmini-c3.jpg" width="240" />
</a>
<a href="docs/v1/movecall-cuican-esp32s3.jpg" target="_blank" title="CuiCan">
<img src="docs/v1/movecall-cuican-esp32s3.jpg" width="240" />
</a>
<a href="docs/v1/wmnologo_xingzhi_1.54.jpg" target="_blank" title="无名科技Nologo-星智-1.54">
<img src="docs/v1/wmnologo_xingzhi_1.54.jpg" width="240" />
</a>
<a href="docs/v1/sensecap_watcher.jpg" target="_blank" title="SenseCAP Watcher">
<img src="docs/v1/sensecap_watcher.jpg" width="240" />
</a>
<a href="docs/v1/esp-hi.jpg" target="_blank" title="ESP-HI 超低成本机器狗">
<img src="docs/v1/esp-hi.jpg" width="240" />
</a>
</div>
## 软件
### 固件烧录
新手第一次操作建议先不要搭建开发环境，直接使用免开发环境烧录的固件。
固件默认接入 [xiaozhi.me](https://xiaozhi.me) 官方服务器，个人用户注册账号可以免费使用 Qwen 实时模型。
👉 [新手烧录固件教程](https://ccnphfhqs21z.feishu.cn/wiki/Zpz4wXBtdimBrLk25WdcXzxcnNS)
### 开发环境
- Cursor 或 VSCode
- 安装 ESP-IDF 插件，选择 SDK 版本 5.4 或以上
- Linux 比 Windows 更好，编译速度快，也免去驱动问题的困扰
- 本项目使用 Google C++ 代码风格，提交代码时请确保符合规范
### 开发者文档
- [自定义开发板指南](docs/custom-board.md) - 学习如何为小智 AI 创建自定义开发板
- [MCP 协议物联网控制用法说明](docs/mcp-usage.md) - 了解如何通过 MCP 协议控制物联网设备
- [MCP 协议交互流程](docs/mcp-protocol.md) - 设备端 MCP 协议的实现方式
- [MQTT + UDP 混合通信协议文档](docs/mqtt-udp.md)
- [一份详细的 WebSocket 通信协议文档](docs/websocket.md)
## 大模型配置
如果你已经拥有一个小智 AI 聊天机器人设备，并且已接入官方服务器，可以登录 [xiaozhi.me](https://xiaozhi.me) 控制台进行配置。
👉 [后台操作视频教程（旧版界面）](https://www.bilibili.com/video/BV1jUCUY2EKM/)
## 相关开源项目
在个人电脑上部署服务器，可以参考以下第三方开源的项目：
- [xinnan-tech/xiaozhi-esp32-server](https://github.com/xinnan-tech/xiaozhi-esp32-server) Python 服务器
- [joey-zhou/xiaozhi-esp32-server-java](https://github.com/joey-zhou/xiaozhi-esp32-server-java) Java 服务器
- [AnimeAIChat/xiaozhi-server-go](https://github.com/AnimeAIChat/xiaozhi-server-go) Golang 服务器
使用小智通信协议的第三方客户端项目：
- [huangjunsen0406/py-xiaozhi](https://github.com/huangjunsen0406/py-xiaozhi) Python 客户端
- [TOM88812/xiaozhi-android-client](https://github.com/TOM88812/xiaozhi-android-client) Android 客户端
- [100askTeam/xiaozhi-linux](http://github.com/100askTeam/xiaozhi-linux) 百问科技提供的 Linux 客户端
- [78/xiaozhi-sf32](https://github.com/78/xiaozhi-sf32) 思澈科技的蓝牙芯片固件
- [QuecPython/solution-xiaozhiAI](https://github.com/QuecPython/solution-xiaozhiAI) 移远提供的 QuecPython 固件
## 关于项目
这是一个由虾哥开源的 ESP32 项目，以 MIT 许可证发布，允许任何人免费使用，修改或用于商业用途。
我们希望通过这个项目，能够帮助大家了解 AI 硬件开发，将当下飞速发展的大语言模型应用到实际的硬件设备中。
如果你有任何想法或建议，请随时提出 Issues 或加入 QQ 群：1011329060
## Star History
<a href="https://star-history.com/#78/xiaozhi-esp32&Date">
<picture>
<source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/svg?repos=78/xiaozhi-esp32&type=Date&theme=dark" />
<source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/svg?repos=78/xiaozhi-esp32&type=Date" />
<img alt="Star History Chart" src="https://api.star-history.com/svg?repos=78/xiaozhi-esp32&type=Date" />
</picture>
</a>

286
RP2040/Backup/animation.py Normal file
View File

@ -0,0 +1,286 @@
from servoclass import Servo
import time
current_pose = "pose_base"
current_state = "state_startup"
previous_state = None
animation_bool_a = False
animation_bool_b = False
last_toggle_a = time.ticks_ms()
last_toggle_b = time.ticks_ms()
new_state_flag = False
servos = {
"YAW": Servo(pin_num=6, max_speed=400, max_accel=100, min_angle=10, max_angle=170), #Base Yaw Rotation
"ROL": Servo(pin_num=7, max_speed=600, max_accel=400, min_angle=30, max_angle=120), #Neck Roll
"PIT": Servo(pin_num=8, max_speed=600, max_accel=400, min_angle=1, max_angle=80), #Neck Pitch
"MOU": Servo(pin_num=19, max_speed=100000, max_accel=3500, min_angle=5, max_angle=150), #Mouth
"EYL": Servo(pin_num=12, max_speed=200, max_accel=500, min_angle=30, max_angle=150), #Left Eyeball
"EYR": Servo(pin_num=13, max_speed=250, max_accel=500, min_angle=30, max_angle=150), #Right Eyeball
"LID": Servo(pin_num=14, max_speed=100000, max_accel=7000, min_angle=30, max_angle=160), #EyeLid
"EAL": Servo(pin_num=15, max_speed=250, max_accel=200, min_angle=60, max_angle=150), #Left Ear
"EAR": Servo(pin_num=16, max_speed=500, max_accel=200, min_angle=30, max_angle=120), #Right Ear
}
def apply_pose(pose):
pose = pose_map.get(pose, pose_calibrate)
for name, angle in pose.items():
if name in servos:
servos[name].set_target(angle)
def apply_state(state_name):
state_func = state_map.get(state_name)
if state_func:
state_func() # Call the function
else:
print(f"Unknown state: {state_name}")
#_________________# Poses (static servo positions used in states) #_________________#
pose_calibrate = { # Each dictionary key = servo name, value = angle
"YAW": 90,
"ROL": 90,
"PIT": 80,
"MOU": 170,
"LID": 110,
"EYL": 90,
"EYR": 90,
"EAL": 90,
"EAR": 90,
}
pose_sleep = { # Each dictionary key = servo name, value = angle
# "YAW": 88,
# "RWH": 89,
"ROL": 90,
"PIT": 80,
"MOU": 170,
"LID": 30,
# "EYL": 90,
# "EYR": 90,
"EAL": 150,
"EAR": 30,
}
pose_base = { # Each dictionary key = servo name, value = angle
# "YAW": 88,
# "RWH": 89,
"ROL": 90,
# "PIT": 20,
# "MOU": 170,
# "LID": 130,
# "EYL": 90,
# "EYR": 90,
"EAL": 130,
"EAR": 70,
}
pose_speaking = { # Each dictionary key = servo name, value = angle
# "YAW": 88,
# "RWH": 89,
# "ROL": 90,
# "PIT": 20,
"MOU": 10,
# "LID": 130,
# "EYL": 90,
# "EYR": 90,
"EAL": 130,
"EAR": 70,
}
pose_stop_speaking = { # Each dictionary key = servo name, value = angle
# "YAW": 88,
# "RWH": 89,
# "ROL": 90,
# "PIT": 20,
"MOU": 150,
# "LID": 130,
# "EYL": 90,
# "EYR": 90,
"EAL": 130,
"EAR": 70,
}
pose_thinking_1 = { # Each dictionary key = servo name, value = angle
# "YAW": 90,
# "RWH": 90,
"ROL": 130,
# "PIT": 50,
"MOU": 150,
# "LID": 70,
# "EYL": 90,
# "EYR": 90,
"EAL": 150,
"EAR": 120,
}
pose_curious_2 = { # Each dictionary key = servo name, value = angle
# "YAW": 90,
# "RWH": 90,
"ROL": 40,
# "PIT": 10,
"MOU": 160,
# "LID": 130,
# "EYL": 90,
# "EYR": 90,
"EAL": 60,
"EAR": 60,
}
pose_map={
"pose_calibrate": pose_calibrate,
"pose_base": pose_base,
"pose_thinking_1": pose_thinking_1,
"pose_curious_2": pose_curious_2,
"pose_sleep": pose_sleep,
"pose_speaking": pose_speaking,
"pose_stop_speaking": pose_stop_speaking,
}
#_________________# #_________________#
#_________________# States (Mix of poses + animations) #_________________#
def state_startup():
global new_state_flag
if new_state_flag == True:
apply_pose("pose_calibrate")
new_state_flag = False
def state_calibrate():
global new_state_flag
# if new_state_flag == True:
apply_pose("pose_calibrate")
new_state_flag = False
def state_sleep():
global new_state_flag
if new_state_flag == True:
apply_pose("pose_sleep")
new_state_flag = False
def state_speaking():
global new_state_flag
if new_state_flag == True:
apply_pose("pose_speaking")
new_state_flag = False
# if animation_bool_a == False:
# servos["MOU"].set_target(130)
# if animation_bool_a == True:
# servos["MOU"].set_target(10)
# if time.ticks_diff(now, last_toggle_a) >= 300:
# animation_bool_a = not animation_bool_a # flip the boolean
# last_toggle_a = now
def state_thinking():
now=time.ticks_ms()
global animation_bool_a, last_toggle_a, animation_bool_b, last_toggle_b, new_state_flag
if new_state_flag == True:
apply_pose("pose_base")
new_state_flag = False
if animation_bool_a == False:
servos["PIT"].set_target(50)
if animation_bool_a == True:
servos["PIT"].set_target(120)
if time.ticks_diff(now, last_toggle_a) >= 700:
animation_bool_a = not animation_bool_a # flip the boolean
last_toggle_a = now
def state_listen():
global new_state_flag
if new_state_flag == True:
apply_pose("pose_curious_2")
new_state_flag = False
def state_neutral():
global new_state_flag
if new_state_flag == True:
apply_pose("pose_stop_speaking")
new_state_flag = False
def state_happy():
now=time.ticks_ms()
global animation_bool_a, last_toggle_a, animation_bool_b, last_toggle_b, new_state_flag
if new_state_flag == True:
apply_pose("pose_base")
new_state_flag = False
# print("idle")
if animation_bool_a == False:
servos["ROL"].set_target(70)
if animation_bool_a == True:
servos["ROL"].set_target(110)
if time.ticks_diff(now, last_toggle_a) >= 2000:
animation_bool_a = not animation_bool_a # flip the boolean
last_toggle_a = now
if animation_bool_b == False:
servos["EAL"].set_target(100)
servos["EAR"].set_target(60)
if animation_bool_b == True:
servos["EAL"].set_target(160)
servos["EAR"].set_target(120)
if time.ticks_diff(now, last_toggle_b) >= 1000:
animation_bool_b = not animation_bool_b # flip the boolean
last_toggle_b = now
# print("Toggled:", animation_bool_a)
def state_limber_up():
now=time.ticks_ms()
global animation_bool_a, last_toggle_a, animation_bool_b, last_toggle_b, new_state_flag
if new_state_flag == True:
apply_pose("pose_base")
new_state_flag = False
# print("idle")
if animation_bool_a == False:
# servos["YAW"].set_target(50)
servos["ROL"].set_target(70)
servos["PIT"].set_target(70)
servos["EAL"].set_target(100)
servos["EYL"].set_target(60)
servos["MOU"].set_target(20)
servos["LID"].set_target(20)
servos["EYR"].set_target(60)
servos["EAL"].set_target(60)
servos["EAR"].set_target(60)
if animation_bool_a == True:
# servos["YAW"].set_target(130)
servos["ROL"].set_target(110)
servos["PIT"].set_target(5)
servos["EAL"].set_target(160)
servos["EYL"].set_target(120)
servos["MOU"].set_target(160)
servos["LID"].set_target(160)
servos["EYR"].set_target(120)
servos["EAL"].set_target(120)
servos["EAR"].set_target(120)
if time.ticks_diff(now, last_toggle_a) >= 1000:
animation_bool_a = not animation_bool_a # flip the boolean
last_toggle_a = now
# Note: system states:
# - idle: sleeping, wakeword not initiated listening mode
# - neutral: between finished speaking and begun listening. potentially use as marker for end of speech
# - listening
# - speaking
state_map={
"neutral": state_neutral,
"idle": state_sleep,
"listening": state_listen,
"state_startup": state_startup,
"thinking": state_thinking,
"speaking": state_speaking,
"state_limber_up": state_limber_up,
"happy": state_happy,
"state_calibrate": state_calibrate
}
#_________________# #_________________#

106
RP2040/Backup/coms.py Normal file
View File

@ -0,0 +1,106 @@
from machine import Pin, UART
import time
import animation # Assuming this handles your state_map
class Comms:
def __init__(self):
# Hardware Initialization
self.grove = UART(0, baudrate=921600, tx=Pin(0), rx=Pin(1))
self.esp = UART(1, baudrate=115200, tx=Pin(4), rx=Pin(5))
# Constants
self.INVOKE_CMD = b"AT+INVOKE=1,0,1\r"
self.pixel_centre = 112
self.deadzone = 20
self.x_adj_factor = 10
self.y_adj_factor = 10
self.staticflag = False
# Buffers and State
self.cbuf = b""
self.rx_buffer = b""
self.readflag = True
self.last_boxes = None
self.grove.write(self.INVOKE_CMD)
def map_value(self, value, in_min, in_max, out_min, out_max):
return (value - in_min) * (out_max - out_min) / (in_max - in_min) + out_min
def esp_read(self):
# 1. Read new data if there is any
if self.esp.any():
self.rx_buffer += self.esp.read()
# 2. Extract ALL complete lines currently in the buffer
# Notice this is OUTSIDE the `if self.esp.any():` block!
commands = []
while b"\n" in self.rx_buffer:
line, self.rx_buffer = self.rx_buffer.split(b"\n", 1)
rcvstate = line.decode('utf-8').strip()
# Add to our list instead of returning immediately
if rcvstate: # Just to ignore empty blank lines
commands.append(rcvstate)
# Return the list of commands (might be empty, might have 1, might have 3!)
return commands
def grove_read(self):
# State 1: Requesting data
if self.readflag:
# Flush existing buffer quickly
while self.grove.any():
self.grove.read()
self.grove.write(self.INVOKE_CMD)
self.cbuf = b""
self.readflag = False
return None # Return early, data isn't ready yet
# State 2: Receiving data
if self.grove.any():
# PERFORMANCE: Read chunks directly instead of looping char by char
self.cbuf += self.grove.read()
# Wait until the end of the JSON packet arrives
if b'"resolution"' in self.cbuf:
key = b'"boxes":'
i = self.cbuf.find(key)
if i != -1:
boxes_part = self.cbuf[i + len(key):]
# Ensure we have the closing bracket before slicing
end_idx = boxes_part.find(b']')
if end_idx != -1:
boxes_part = boxes_part[:end_idx + 1].strip()
# Process if we have valid, new box data
if boxes_part != b'[]' and boxes_part != self.last_boxes:
self.staticflag = False
self.last_boxes = boxes_part
boxes_str = boxes_part.decode('utf-8').strip('[]')
try:
numbers = [int(n) for n in boxes_str.split(',')]
x_offset = numbers[0] - self.pixel_centre
y_offset = numbers[1] - self.pixel_centre
# Reset for next read
self.cbuf = b""
self.readflag = True
return x_offset, y_offset
except ValueError:
# Handle cases where split data isn't a perfect integer
pass
else:
# Reset for next read (static or empty data)
self.cbuf = b""
self.readflag = True
self.staticflag = True
return None
return None

198
RP2040/Backup/main.py Normal file
View File

@ -0,0 +1,198 @@
# motion_servo.py (MicroPython for RP2040)
from machine import Pin, PWM, UART
import time, random
import urandom
from servoclass import Servo
import sys, select, uselect
import math
import animation
import coms
from animation import servos
mode = Pin(20, Pin.IN, Pin.PULL_UP)
LED = Pin(25, Pin.OUT)
yaw_target = 100
yaw_countdown = yaw_target
LED_oscillator = 25
LED_countdown = LED_oscillator
blink_time = 500
blinking = False
speaking_flag = False
speaking_timer = 0
bool_a = False
last_toggle_a= time.ticks_ms()
external = coms.Comms()
# track direction for each servo (1 = going to max, -1 = going to min)
directions = {name: 1 for name in servos.keys()}
last_time = time.ticks_ms()
last_switch = last_time
# animation.servos["YAW"].set_target(90)
on_time = time.ticks_ms()
startup_sleep = True
def facetrack():
global yaw_countdown, yaw_target
# ALWAYS read the sensor to prevent serial buffer overflows!
offset = external.grove_read()
# Only move the servos if the robot is awake
if animation.current_state != "idle":
eyl = animation.servos["EYL"]
eyr = animation.servos["EYR"]
pit = animation.servos["PIT"]
yaw = animation.servos["YAW"]
if offset:
dead = external.deadzone
static = external.staticflag
x0, y0 = offset
x_scale = external.x_adj_factor / 110
y_scale = external.y_adj_factor / 110
if not static:
if abs(x0) > dead:
x = eyl.target + x0 * x_scale
eyl.set_target(x)
eyr.set_target(x)
if abs(y0) > dead:
y = pit.target + y0 * y_scale
pit.set_target(y)
if abs(90 - eyl.target) >= 20:
yaw_countdown -= 1
if yaw_countdown <= 0:
yaw.set_target(90 + ((eyl.target-90)/2))
yaw_countdown = yaw_target
# --- STAGGERED STARTUP SEQUENCE ---
time.sleep(0.5) # Allow board power to stabilize after boot
# Determine the correct initial pose based on the calibration pin
is_calibrating = (mode.value() == 1)
initial_pose_dict = animation.pose_calibrate if is_calibrating else animation.pose_sleep
print("Initiating staggered servo wakeup...")
for name, servo_obj in animation.servos.items():
# Fetch the target angle for this specific pose (default to 90 if missing)
start_angle = initial_pose_dict.get(name, 90)
# Pre-set the internal tracking positions so the servo math doesn't think it's moving from 90
servo_obj.pos = start_angle
servo_obj.target = start_angle
# Send the first pulse. If physically elsewhere, it will snap, but ALONE.
servo_obj._write_pwm(start_angle)
# Stagger the wakeups. Fast if calibrating, slow and gentle if sleeping.
delay_ms = 50 if is_calibrating else 250
time.sleep_ms(delay_ms)
# Set initial states to match the startup logic
animation.current_state = "state_calibrate" if is_calibrating else "idle"
animation.previous_state = animation.current_state
print("Startup complete.")
# ----------------------------------
# external.grove_invoke()
FTDebug = False # setting to True isolates the face tracking code
while True:
# Update time
now = time.ticks_ms()
dt = time.ticks_diff(now, last_time) / 1000.0
last_time = now
if FTDebug == False:
# Grab all pending commands from the ESP
incoming_commands = external.esp_read()
for data in incoming_commands:
if data in animation.state_map:
animation.new_state_flag = True
animation.current_state = data
# 2. Check if the state changed this frame
if animation.current_state != animation.previous_state:
if animation.current_state == "speaking":
speaking_flag = True
elif animation.current_state in ["neutral", "idle", "listening"]:
if speaking_flag:
speaking_flag = False
# Snap the mouth closed when speech ends!
animation.servos["MOU"].set_target(130)
# 3. Speaking timer / animation
if speaking_flag:
# Set the mouth position based on the current toggle state
if bool_a == False:
animation.servos["MOU"].set_target(130)
elif bool_a == True:
animation.servos["MOU"].set_target(70)
# Flip the boolean every 250ms to flap the mouth
if time.ticks_diff(now, last_toggle_a) >= 250:
bool_a = not bool_a # flip the boolean
last_toggle_a = now
# Blink mode enabler
if not blinking and random.randrange(500) == 0:
blinking = True
blink_counter = blink_time
# Calibration mode
if (mode.value() == 1):
animation.current_state = "state_calibrate"
animation.apply_state("state_calibrate") # change this back to calibrate to keep calibration mode/base for testing
LED_countdown -= 1
if LED_countdown <= 0:
LED.toggle()
LED_countdown = LED_oscillator
else:
if animation.current_state == "idle":
animation.apply_state("idle")
animation.servos["LID"]._write_pwm(30)
else:
if animation.previous_state == "idle":
animation.servos["LID"]._write_pwm(110)
animation.servos["PIT"].set_target(10)
# Blink mode
if blinking and animation.current_state != "idle":
if blink_counter > blink_time - 50:
# closed
animation.servos["LID"]._write_pwm(30)
elif blink_counter > 0:
# reopen
animation.servos["LID"]._write_pwm(110)
blink_counter -= 1
if blink_counter == 0:
blinking = False
# Update with whatever state the ESP says
animation.apply_state(animation.current_state)
animation.previous_state = animation.current_state
else:
animation.servos["LID"]._write_pwm(110)
animation.current_state = "neutral"
facetrack()
# Update all servos except eyelids
for name, s in servos.items():
if name == "LID":
continue
s.update(dt)

94
RP2040/Backup/servoclass.py Normal file
View File

@ -0,0 +1,94 @@
from machine import Pin, PWM
import time
class Servo:
def __init__(self, pin_num, max_speed=120.0, max_accel=360.0, min_angle=0.0, max_angle=180.0, enabled=True):
self.pwm = PWM(Pin(pin_num))
self.pwm.freq(50)
# motion state
self.pos = 90.0
self.vel = 0.0
self.target = 90.0
# tuning
self.max_speed = float(max_speed)
self.max_accel = float(max_accel)
# PERFORMANCE: Precompute inverse of 2*accel to use multiplication instead of division later
self._inv_2_accel = 1.0 / (2.0 * self.max_accel)
# limits
self.min_angle = float(min_angle)
self.max_angle = float(max_angle)
# tolerances
self.pos_tolerance = 0.6
self.vel_tolerance = 1.0
# PERFORMANCE: Precompute PWM conversion constants to avoid heavy math in the update loop
# Mapping 0..180 to 500..2500us on a 20000us period, scaled to 65535
self._pwm_offset = 500.0 * (65535.0 / 20000.0)
self._pwm_slope = (2000.0 / 180.0) * (65535.0 / 20000.0)
# debug flag
self.enabled = enabled
# Start limp to prevent power spike on startup
self.pwm.duty_u16(0)
def set_target(self, angle):
# Inline clamp to save function call overhead
angle = float(angle)
self.target = self.min_angle if angle < self.min_angle else (self.max_angle if angle > self.max_angle else angle)
def update(self, dt):
if not self.enabled or dt <= 0:
return
error = self.target - self.pos
dist = abs(error)
desired_dir = 0.0 if dist < 1e-9 else (1.0 if error > 0 else -1.0)
vel_dir = 0.0 if abs(self.vel) < 1e-9 else (1.0 if self.vel > 0 else -1.0)
# small-target case
if dist <= self.pos_tolerance and abs(self.vel) <= self.vel_tolerance:
self.pos = self.target
self.vel = 0.0
self._write_pwm(self.pos)
return
# braking-first rule
if vel_dir != 0 and desired_dir != 0 and vel_dir != desired_dir:
accel = -vel_dir * self.max_accel
else:
# PERFORMANCE: Using precomputed multiplication instead of division
stopping_dist = (self.vel * self.vel) * self._inv_2_accel
if dist > stopping_dist:
accel = desired_dir * self.max_accel
else:
accel = -desired_dir * self.max_accel
# integrate velocity
new_vel = self.vel + accel * dt
# avoid sign flip jitter
if (self.vel > 0 and new_vel < 0) or (self.vel < 0 and new_vel > 0):
new_vel = 0.0
# Inline clamp speed
self.vel = -self.max_speed if new_vel < -self.max_speed else (self.max_speed if new_vel > self.max_speed else new_vel)
# integrate position
self.pos += self.vel * dt
# Inline clamp position to valid servo range
self.pos = 0.0 if self.pos < 0.0 else (180.0 if self.pos > 180.0 else self.pos)
self._write_pwm(self.pos)
def _write_pwm(self, angle):
# PERFORMANCE: Single multiplication and addition
duty = int(self._pwm_offset + self._pwm_slope * angle)
self.pwm.duty_u16(duty)

327
RP2040/animation.py Normal file
View File

@ -0,0 +1,327 @@
from servoclass import Servo
import time
import urandom
current_pose = "pose_base"
current_state = "state_startup"
previous_state = None
animation_bool_a = False
animation_bool_b = False
last_toggle_a = time.ticks_ms()
last_toggle_b = time.ticks_ms()
new_state_flag = False
# 眼球随机看 + 身体摆动的计时器和状态
eye_last_change = time.ticks_ms()
eye_interval = 2000 # 眼球每 2 秒换一个随机目标
yaw_bool = False
yaw_last_toggle = time.ticks_ms()
# Grove 人脸追踪状态（由 main.py 更新）
grove_active = False # Grove 是否在提供有效数据
grove_last_seen = 0 # 上次收到 Grove 数据的时间戳
servos = {
"YAW": Servo(pin_num=6, max_speed=400, max_accel=100, min_angle=10, max_angle=170), #Base Yaw Rotation
"ROL": Servo(pin_num=7, max_speed=600, max_accel=400, min_angle=30, max_angle=120), #Neck Roll
"PIT": Servo(pin_num=8, max_speed=600, max_accel=400, min_angle=1, max_angle=80), #Neck Pitch
"MOU": Servo(pin_num=19, max_speed=100000, max_accel=3500, min_angle=5, max_angle=150), #Mouth
"EYL": Servo(pin_num=12, max_speed=200, max_accel=500, min_angle=30, max_angle=150), #Left Eyeball
"EYR": Servo(pin_num=13, max_speed=250, max_accel=500, min_angle=30, max_angle=150), #Right Eyeball
"LID": Servo(pin_num=14, max_speed=100000, max_accel=7000, min_angle=30, max_angle=160), #EyeLid
"EAL": Servo(pin_num=15, max_speed=250, max_accel=200, min_angle=60, max_angle=150), #Left Ear
"EAR": Servo(pin_num=16, max_speed=500, max_accel=200, min_angle=30, max_angle=120), #Right Ear
}
def apply_pose(pose):
pose = pose_map.get(pose, pose_calibrate)
for name, angle in pose.items():
if name in servos:
servos[name].set_target(angle)
def apply_state(state_name):
state_func = state_map.get(state_name)
if state_func:
state_func() # Call the function
else:
print(f"Unknown state: {state_name}")
#_________________# Poses (static servo positions used in states) #_________________#
pose_calibrate = { # Each dictionary key = servo name, value = angle
"YAW": 90,
"ROL": 90,
"PIT": 80,
"MOU": 170,
"LID": 110,
"EYL": 90,
"EYR": 90,
"EAL": 90,
"EAR": 90,
}
pose_sleep = { # Each dictionary key = servo name, value = angle
# "YAW": 88,
# "RWH": 89,
"ROL": 90,
"PIT": 80,
"MOU": 170,
"LID": 30,
# "EYL": 90,
# "EYR": 90,
"EAL": 150,
"EAR": 30,
}
pose_base = { # Each dictionary key = servo name, value = angle
# "YAW": 88,
# "RWH": 89,
"ROL": 90,
# "PIT": 20,
# "MOU": 170,
# "LID": 130,
# "EYL": 90,
# "EYR": 90,
"EAL": 130,
"EAR": 70,
}
pose_speaking = { # Each dictionary key = servo name, value = angle
# "YAW": 88,
# "RWH": 89,
# "ROL": 90,
# "PIT": 20,
"MOU": 10,
# "LID": 130,
# "EYL": 90,
# "EYR": 90,
"EAL": 130,
"EAR": 70,
}
pose_stop_speaking = { # Each dictionary key = servo name, value = angle
# "YAW": 88,
# "RWH": 89,
# "ROL": 90,
# "PIT": 20,
"MOU": 150,
# "LID": 130,
# "EYL": 90,
# "EYR": 90,
"EAL": 130,
"EAR": 70,
}
pose_thinking_1 = { # Each dictionary key = servo name, value = angle
# "YAW": 90,
# "RWH": 90,
"ROL": 130,
# "PIT": 50,
"MOU": 150,
# "LID": 70,
# "EYL": 90,
# "EYR": 90,
"EAL": 150,
"EAR": 120,
}
pose_curious_2 = { # Each dictionary key = servo name, value = angle
# "YAW": 90,
# "RWH": 90,
"ROL": 40,
# "PIT": 10,
"MOU": 160,
# "LID": 130,
# "EYL": 90,
# "EYR": 90,
"EAL": 60,
"EAR": 60,
}
pose_map={
"pose_calibrate": pose_calibrate,
"pose_base": pose_base,
"pose_thinking_1": pose_thinking_1,
"pose_curious_2": pose_curious_2,
"pose_sleep": pose_sleep,
"pose_speaking": pose_speaking,
"pose_stop_speaking": pose_stop_speaking,
}
#_________________# #_________________#
#_________________# 眼球随机看（无 Grove 时的自动动画） #_________________#
def eye_random_look():
"""眼球随机左右看。Grove 有数据时跳过，由人脸追踪接管"""
if grove_active:
return
global eye_last_change, eye_interval
now = time.ticks_ms()
if time.ticks_diff(now, eye_last_change) >= eye_interval:
# 随机眼球目标角度：60°~120° 范围内（中位 90°，左右各偏 30°）
target = 60 + urandom.getrandbits(6) % 61 # 60~120
servos["EYL"].set_target(target)
servos["EYR"].set_target(target)
# 随机下次间隔 1.5~3.5 秒，看起来更自然
eye_interval = 1500 + urandom.getrandbits(11) % 2001
eye_last_change = now
def yaw_gentle_sway():
"""身体轻微摆动。Grove 有数据时跳过，由人脸追踪接管"""
if grove_active:
return
global yaw_bool, yaw_last_toggle
now = time.ticks_ms()
if yaw_bool:
servos["YAW"].set_target(80) # 轻微偏左
else:
servos["YAW"].set_target(100) # 轻微偏右
if time.ticks_diff(now, yaw_last_toggle) >= 3000: # 每 3 秒换方向
yaw_bool = not yaw_bool
yaw_last_toggle = now
#_________________# States (Mix of poses + animations) #_________________#
def state_startup():
global new_state_flag
if new_state_flag == True:
apply_pose("pose_calibrate")
new_state_flag = False
def state_calibrate():
global new_state_flag
# if new_state_flag == True:
apply_pose("pose_calibrate")
new_state_flag = False
def state_sleep():
global new_state_flag
if new_state_flag == True:
apply_pose("pose_sleep")
new_state_flag = False
def state_speaking():
global new_state_flag
if new_state_flag == True:
apply_pose("pose_speaking")
new_state_flag = False
eye_random_look()
yaw_gentle_sway()
def state_thinking():
now=time.ticks_ms()
global animation_bool_a, last_toggle_a, animation_bool_b, last_toggle_b, new_state_flag
if new_state_flag == True:
apply_pose("pose_base")
new_state_flag = False
if animation_bool_a == False:
servos["PIT"].set_target(50)
if animation_bool_a == True:
servos["PIT"].set_target(120)
if time.ticks_diff(now, last_toggle_a) >= 700:
animation_bool_a = not animation_bool_a # flip the boolean
last_toggle_a = now
eye_random_look()
def state_listen():
global new_state_flag
if new_state_flag == True:
apply_pose("pose_curious_2")
new_state_flag = False
eye_random_look()
yaw_gentle_sway()
def state_neutral():
global new_state_flag
if new_state_flag == True:
apply_pose("pose_stop_speaking")
new_state_flag = False
eye_random_look()
def state_happy():
now=time.ticks_ms()
global animation_bool_a, last_toggle_a, animation_bool_b, last_toggle_b, new_state_flag
if new_state_flag == True:
apply_pose("pose_base")
new_state_flag = False
eye_random_look()
if animation_bool_a == False:
servos["ROL"].set_target(70)
if animation_bool_a == True:
servos["ROL"].set_target(110)
if time.ticks_diff(now, last_toggle_a) >= 2000:
animation_bool_a = not animation_bool_a # flip the boolean
last_toggle_a = now
if animation_bool_b == False:
servos["EAL"].set_target(100)
servos["EAR"].set_target(60)
if animation_bool_b == True:
servos["EAL"].set_target(160)
servos["EAR"].set_target(120)
if time.ticks_diff(now, last_toggle_b) >= 1000:
animation_bool_b = not animation_bool_b # flip the boolean
last_toggle_b = now
# print("Toggled:", animation_bool_a)
def state_limber_up():
now=time.ticks_ms()
global animation_bool_a, last_toggle_a, animation_bool_b, last_toggle_b, new_state_flag
if new_state_flag == True:
apply_pose("pose_base")
new_state_flag = False
# print("idle")
if animation_bool_a == False:
# servos["YAW"].set_target(50)
servos["ROL"].set_target(70)
servos["PIT"].set_target(70)
servos["EAL"].set_target(100)
servos["EYL"].set_target(60)
servos["MOU"].set_target(20)
servos["LID"].set_target(20)
servos["EYR"].set_target(60)
servos["EAL"].set_target(60)
servos["EAR"].set_target(60)
if animation_bool_a == True:
# servos["YAW"].set_target(130)
servos["ROL"].set_target(110)
servos["PIT"].set_target(5)
servos["EAL"].set_target(160)
servos["EYL"].set_target(120)
servos["MOU"].set_target(160)
servos["LID"].set_target(160)
servos["EYR"].set_target(120)
servos["EAL"].set_target(120)
servos["EAR"].set_target(120)
if time.ticks_diff(now, last_toggle_a) >= 1000:
animation_bool_a = not animation_bool_a # flip the boolean
last_toggle_a = now
# Note: system states:
# - idle: sleeping, wakeword not initiated listening mode
# - neutral: between finished speaking and begun listening. potentially use as marker for end of speech
# - listening
# - speaking
state_map={
"neutral": state_neutral,
"idle": state_sleep,
"listening": state_listen,
"state_startup": state_startup,
"thinking": state_thinking,
"speaking": state_speaking,
"state_limber_up": state_limber_up,
"happy": state_happy,
"state_calibrate": state_calibrate
}
#_________________# #_________________#

106
RP2040/coms.py Normal file
View File

@ -0,0 +1,106 @@
from machine import Pin, UART
import time
import animation # Assuming this handles your state_map
class Comms:
def __init__(self):
# Hardware Initialization
self.grove = UART(0, baudrate=921600, tx=Pin(0), rx=Pin(1))
self.esp = UART(1, baudrate=115200, tx=Pin(4), rx=Pin(5))
# Constants
self.INVOKE_CMD = b"AT+INVOKE=1,0,1\r"
self.pixel_centre = 112
self.deadzone = 20
self.x_adj_factor = 10
self.y_adj_factor = 10
self.staticflag = False
# Buffers and State
self.cbuf = b""
self.rx_buffer = b""
self.readflag = True
self.last_boxes = None
self.grove.write(self.INVOKE_CMD)
def map_value(self, value, in_min, in_max, out_min, out_max):
return (value - in_min) * (out_max - out_min) / (in_max - in_min) + out_min
def esp_read(self):
# 1. Read new data if there is any
if self.esp.any():
self.rx_buffer += self.esp.read()
# 2. Extract ALL complete lines currently in the buffer
# Notice this is OUTSIDE the `if self.esp.any():` block!
commands = []
while b"\n" in self.rx_buffer:
line, self.rx_buffer = self.rx_buffer.split(b"\n", 1)
rcvstate = line.decode('utf-8').strip()
# Add to our list instead of returning immediately
if rcvstate: # Just to ignore empty blank lines
commands.append(rcvstate)
# Return the list of commands (might be empty, might have 1, might have 3!)
return commands
def grove_read(self):
# State 1: Requesting data
if self.readflag:
# Flush existing buffer quickly
while self.grove.any():
self.grove.read()
self.grove.write(self.INVOKE_CMD)
self.cbuf = b""
self.readflag = False
return None # Return early, data isn't ready yet
# State 2: Receiving data
if self.grove.any():
# PERFORMANCE: Read chunks directly instead of looping char by char
self.cbuf += self.grove.read()
# Wait until the end of the JSON packet arrives
if b'"resolution"' in self.cbuf:
key = b'"boxes":'
i = self.cbuf.find(key)
if i != -1:
boxes_part = self.cbuf[i + len(key):]
# Ensure we have the closing bracket before slicing
end_idx = boxes_part.find(b']')
if end_idx != -1:
boxes_part = boxes_part[:end_idx + 1].strip()
# Process if we have valid, new box data
if boxes_part != b'[]' and boxes_part != self.last_boxes:
self.staticflag = False
self.last_boxes = boxes_part
boxes_str = boxes_part.decode('utf-8').strip('[]')
try:
numbers = [int(n) for n in boxes_str.split(',')]
x_offset = numbers[0] - self.pixel_centre
y_offset = numbers[1] - self.pixel_centre
# Reset for next read
self.cbuf = b""
self.readflag = True
return x_offset, y_offset
except ValueError:
# Handle cases where split data isn't a perfect integer
pass
else:
# Reset for next read (static or empty data)
self.cbuf = b""
self.readflag = True
self.staticflag = True
return None
return None

206
RP2040/main.py Normal file
View File

@ -0,0 +1,206 @@
# motion_servo.py (MicroPython for RP2040)
from machine import Pin, PWM, UART
import time, random
import urandom
from servoclass import Servo
import sys, select, uselect
import math
import animation
import coms
from animation import servos
mode = Pin(20, Pin.IN, Pin.PULL_UP)
LED = Pin(25, Pin.OUT)
yaw_target = 100
yaw_countdown = yaw_target
LED_oscillator = 25
LED_countdown = LED_oscillator
blink_time = 500
blinking = False
speaking_flag = False
speaking_timer = 0
bool_a = False
last_toggle_a= time.ticks_ms()
external = coms.Comms()
# track direction for each servo (1 = going to max, -1 = going to min)
directions = {name: 1 for name in servos.keys()}
last_time = time.ticks_ms()
last_switch = last_time
# animation.servos["YAW"].set_target(90)
on_time = time.ticks_ms()
startup_sleep = True
def facetrack():
global yaw_countdown, yaw_target
# ALWAYS read the sensor to prevent serial buffer overflows!
offset = external.grove_read()
# 更新 Grove 活跃状态：有数据则标记活跃，超过 3 秒无数据则标记不活跃
now = time.ticks_ms()
if offset:
animation.grove_active = True
animation.grove_last_seen = now
elif animation.grove_active and time.ticks_diff(now, animation.grove_last_seen) > 3000:
animation.grove_active = False
# Only move the servos if the robot is awake
if animation.current_state != "idle":
eyl = animation.servos["EYL"]
eyr = animation.servos["EYR"]
pit = animation.servos["PIT"]
yaw = animation.servos["YAW"]
if offset:
dead = external.deadzone
static = external.staticflag
x0, y0 = offset
x_scale = external.x_adj_factor / 110
y_scale = external.y_adj_factor / 110
if not static:
if abs(x0) > dead:
x = eyl.target + x0 * x_scale
eyl.set_target(x)
eyr.set_target(x)
if abs(y0) > dead:
y = pit.target + y0 * y_scale
pit.set_target(y)
if abs(90 - eyl.target) >= 20:
yaw_countdown -= 1
if yaw_countdown <= 0:
yaw.set_target(90 + ((eyl.target-90)/2))
yaw_countdown = yaw_target
# --- STAGGERED STARTUP SEQUENCE ---
time.sleep(0.5) # Allow board power to stabilize after boot
# Determine the correct initial pose based on the calibration pin
is_calibrating = (mode.value() == 1)
initial_pose_dict = animation.pose_calibrate if is_calibrating else animation.pose_sleep
print("Initiating staggered servo wakeup...")
for name, servo_obj in animation.servos.items():
# Fetch the target angle for this specific pose (default to 90 if missing)
start_angle = initial_pose_dict.get(name, 90)
# Pre-set the internal tracking positions so the servo math doesn't think it's moving from 90
servo_obj.pos = start_angle
servo_obj.target = start_angle
# Send the first pulse. If physically elsewhere, it will snap, but ALONE.
servo_obj._write_pwm(start_angle)
# Stagger the wakeups. Fast if calibrating, slow and gentle if sleeping.
delay_ms = 50 if is_calibrating else 250
time.sleep_ms(delay_ms)
# Set initial states to match the startup logic
animation.current_state = "state_calibrate" if is_calibrating else "idle"
animation.previous_state = animation.current_state
print("Startup complete.")
# ----------------------------------
# external.grove_invoke()
FTDebug = False # setting to True isolates the face tracking code
while True:
# Update time
now = time.ticks_ms()
dt = time.ticks_diff(now, last_time) / 1000.0
last_time = now
if FTDebug == False:
# Grab all pending commands from the ESP
incoming_commands = external.esp_read()
for data in incoming_commands:
if data in animation.state_map:
animation.new_state_flag = True
animation.current_state = data
# 2. Check if the state changed this frame
if animation.current_state != animation.previous_state:
if animation.current_state == "speaking":
speaking_flag = True
elif animation.current_state in ["neutral", "idle", "listening"]:
if speaking_flag:
speaking_flag = False
# Snap the mouth closed when speech ends!
animation.servos["MOU"].set_target(130)
# 3. Speaking timer / animation
if speaking_flag:
# Set the mouth position based on the current toggle state
if bool_a == False:
animation.servos["MOU"].set_target(130)
elif bool_a == True:
animation.servos["MOU"].set_target(70)
# Flip the boolean every 250ms to flap the mouth
if time.ticks_diff(now, last_toggle_a) >= 250:
bool_a = not bool_a # flip the boolean
last_toggle_a = now
# Blink mode enabler
if not blinking and random.randrange(500) == 0:
blinking = True
blink_counter = blink_time
# Calibration mode
if (mode.value() == 1):
animation.current_state = "state_calibrate"
animation.apply_state("state_calibrate") # change this back to calibrate to keep calibration mode/base for testing
LED_countdown -= 1
if LED_countdown <= 0:
LED.toggle()
LED_countdown = LED_oscillator
else:
if animation.current_state == "idle":
animation.apply_state("idle")
animation.servos["LID"]._write_pwm(30)
else:
if animation.previous_state == "idle":
animation.servos["LID"]._write_pwm(110)
animation.servos["PIT"].set_target(10)
# Blink mode
if blinking and animation.current_state != "idle":
if blink_counter > blink_time - 50:
# closed
animation.servos["LID"]._write_pwm(30)
elif blink_counter > 0:
# reopen
animation.servos["LID"]._write_pwm(110)
blink_counter -= 1
if blink_counter == 0:
blinking = False
# Update with whatever state the ESP says
animation.apply_state(animation.current_state)
animation.previous_state = animation.current_state
else:
animation.servos["LID"]._write_pwm(110)
animation.current_state = "neutral"
facetrack()
# Update all servos except eyelids
for name, s in servos.items():
if name == "LID":
continue
s.update(dt)

94
RP2040/servoclass.py Normal file
View File

@ -0,0 +1,94 @@
from machine import Pin, PWM
import time
class Servo:
def __init__(self, pin_num, max_speed=120.0, max_accel=360.0, min_angle=0.0, max_angle=180.0, enabled=True):
self.pwm = PWM(Pin(pin_num))
self.pwm.freq(50)
# motion state
self.pos = 90.0
self.vel = 0.0
self.target = 90.0
# tuning
self.max_speed = float(max_speed)
self.max_accel = float(max_accel)
# PERFORMANCE: Precompute inverse of 2*accel to use multiplication instead of division later
self._inv_2_accel = 1.0 / (2.0 * self.max_accel)
# limits
self.min_angle = float(min_angle)
self.max_angle = float(max_angle)
# tolerances
self.pos_tolerance = 0.6
self.vel_tolerance = 1.0
# PERFORMANCE: Precompute PWM conversion constants to avoid heavy math in the update loop
# Mapping 0..180 to 500..2500us on a 20000us period, scaled to 65535
self._pwm_offset = 500.0 * (65535.0 / 20000.0)
self._pwm_slope = (2000.0 / 180.0) * (65535.0 / 20000.0)
# debug flag
self.enabled = enabled
# Start limp to prevent power spike on startup
self.pwm.duty_u16(0)
def set_target(self, angle):
# Inline clamp to save function call overhead
angle = float(angle)
self.target = self.min_angle if angle < self.min_angle else (self.max_angle if angle > self.max_angle else angle)
def update(self, dt):
if not self.enabled or dt <= 0:
return
error = self.target - self.pos
dist = abs(error)
desired_dir = 0.0 if dist < 1e-9 else (1.0 if error > 0 else -1.0)
vel_dir = 0.0 if abs(self.vel) < 1e-9 else (1.0 if self.vel > 0 else -1.0)
# small-target case
if dist <= self.pos_tolerance and abs(self.vel) <= self.vel_tolerance:
self.pos = self.target
self.vel = 0.0
self._write_pwm(self.pos)
return
# braking-first rule
if vel_dir != 0 and desired_dir != 0 and vel_dir != desired_dir:
accel = -vel_dir * self.max_accel
else:
# PERFORMANCE: Using precomputed multiplication instead of division
stopping_dist = (self.vel * self.vel) * self._inv_2_accel
if dist > stopping_dist:
accel = desired_dir * self.max_accel
else:
accel = -desired_dir * self.max_accel
# integrate velocity
new_vel = self.vel + accel * dt
# avoid sign flip jitter
if (self.vel > 0 and new_vel < 0) or (self.vel < 0 and new_vel > 0):
new_vel = 0.0
# Inline clamp speed
self.vel = -self.max_speed if new_vel < -self.max_speed else (self.max_speed if new_vel > self.max_speed else new_vel)
# integrate position
self.pos += self.vel * dt
# Inline clamp position to valid servo range
self.pos = 0.0 if self.pos < 0.0 else (180.0 if self.pos > 180.0 else self.pos)
self._write_pwm(self.pos)
def _write_pwm(self, angle):
# PERFORMANCE: Single multiplication and addition
duty = int(self._pwm_offset + self._pwm_slope * angle)
self.pwm.duty_u16(duty)

37225
docs/CogNogV1_0.pdf Normal file
View File

File diff suppressed because it is too large Load Diff

452
docs/custom-board.md Normal file
View File

@ -0,0 +1,452 @@
# 自定义开发板指南
本指南介绍如何为小智AI语音聊天机器人项目定制一个新的开发板初始化程序。小智AI支持70多种ESP32系列开发板，每个开发板的初始化代码都放在对应的目录下。
## 重要提示
> **警告**: 对于自定义开发板，当IO配置与原有开发板不同时，切勿直接覆盖原有开发板的配置编译固件。必须创建新的开发板类型，或者通过config.json文件中的builds配置不同的name和sdkconfig宏定义来区分。使用 `python scripts/release.py [开发板目录名字]` 来编译打包固件。
>
> 如果直接覆盖原有配置，将来OTA升级时，您的自定义固件可能会被原有开发板的标准固件覆盖，导致您的设备无法正常工作。每个开发板有唯一的标识和对应的固件升级通道，保持开发板标识的唯一性非常重要。
## 目录结构
每个开发板的目录结构通常包含以下文件：
- `xxx_board.cc` - 主要的板级初始化代码，实现了板子相关的初始化和功能
- `config.h` - 板级配置文件，定义了硬件管脚映射和其他配置项
- `config.json` - 编译配置，指定目标芯片和特殊的编译选项
- `README.md` - 开发板相关的说明文档
## 定制开发板步骤
### 1. 创建新的开发板目录
首先在`boards/`目录下创建一个新的目录，命名方式应使用 `[品牌名]-[开发板类型]` 的形式，例如 `m5stack-tab5`：
```bash
mkdir main/boards/my-custom-board
```
### 2. 创建配置文件
#### config.h
在`config.h`中定义所有的硬件配置，包括:
- 音频采样率和I2S引脚配置
- 音频编解码芯片地址和I2C引脚配置
- 按钮和LED引脚配置
- 显示屏参数和引脚配置
参考示例（来自lichuang-c3-dev）：
```c
#ifndef _BOARD_CONFIG_H_
#define _BOARD_CONFIG_H_
#include <driver/gpio.h>
// 音频配置
#define AUDIO_INPUT_SAMPLE_RATE 24000
#define AUDIO_OUTPUT_SAMPLE_RATE 24000
#define AUDIO_I2S_GPIO_MCLK GPIO_NUM_10
#define AUDIO_I2S_GPIO_WS GPIO_NUM_12
#define AUDIO_I2S_GPIO_BCLK GPIO_NUM_8
#define AUDIO_I2S_GPIO_DIN GPIO_NUM_7
#define AUDIO_I2S_GPIO_DOUT GPIO_NUM_11
#define AUDIO_CODEC_PA_PIN GPIO_NUM_13
#define AUDIO_CODEC_I2C_SDA_PIN GPIO_NUM_0
#define AUDIO_CODEC_I2C_SCL_PIN GPIO_NUM_1
#define AUDIO_CODEC_ES8311_ADDR ES8311_CODEC_DEFAULT_ADDR
// 按钮配置
#define BOOT_BUTTON_GPIO GPIO_NUM_9
// 显示屏配置
#define DISPLAY_SPI_SCK_PIN GPIO_NUM_3
#define DISPLAY_SPI_MOSI_PIN GPIO_NUM_5
#define DISPLAY_DC_PIN GPIO_NUM_6
#define DISPLAY_SPI_CS_PIN GPIO_NUM_4
#define DISPLAY_WIDTH 320
#define DISPLAY_HEIGHT 240
#define DISPLAY_MIRROR_X true
#define DISPLAY_MIRROR_Y false
#define DISPLAY_SWAP_XY true
#define DISPLAY_OFFSET_X 0
#define DISPLAY_OFFSET_Y 0
#define DISPLAY_BACKLIGHT_PIN GPIO_NUM_2
#define DISPLAY_BACKLIGHT_OUTPUT_INVERT true
#endif // _BOARD_CONFIG_H_
```
#### config.json
在`config.json`中定义编译配置，这个文件用于 `scripts/release.py` 脚本自动化编译：
```json
{
"target": "esp32s3", // 目标芯片型号: esp32, esp32s3, esp32c3, esp32c6, esp32p4等
"builds": [
{
"name": "my-custom-board", // 开发板名称，用于生成固件包
"sdkconfig_append": [
// 特别 Flash 大小配置
"CONFIG_ESPTOOLPY_FLASHSIZE_8MB=y",
// 特别分区表配置
"CONFIG_PARTITION_TABLE_CUSTOM_FILENAME=\"partitions/v2/8m.csv\""
]
}
]
}
```
**配置项说明：**
- `target`: 目标芯片型号，必须与硬件匹配
- `name`: 编译输出的固件包名称，建议与目录名一致
- `sdkconfig_append`: 额外的 sdkconfig 配置项数组，会追加到默认配置中
**常用的 sdkconfig_append 配置：**
```json
// Flash 大小
"CONFIG_ESPTOOLPY_FLASHSIZE_4MB=y" // 4MB Flash
"CONFIG_ESPTOOLPY_FLASHSIZE_8MB=y" // 8MB Flash
"CONFIG_ESPTOOLPY_FLASHSIZE_16MB=y" // 16MB Flash
// 分区表
"CONFIG_PARTITION_TABLE_CUSTOM_FILENAME=\"partitions/v2/4m.csv\"" // 4MB 分区表
"CONFIG_PARTITION_TABLE_CUSTOM_FILENAME=\"partitions/v2/8m.csv\"" // 8MB 分区表
"CONFIG_PARTITION_TABLE_CUSTOM_FILENAME=\"partitions/v2/16m.csv\"" // 16MB 分区表
// 语言配置
"CONFIG_LANGUAGE_EN_US=y" // 英语
"CONFIG_LANGUAGE_ZH_CN=y" // 简体中文
// 唤醒词配置
"CONFIG_USE_DEVICE_AEC=y" // 启用设备端 AEC
"CONFIG_WAKE_WORD_DISABLED=y" // 禁用唤醒词
```
### 3. 编写板级初始化代码
创建一个`my_custom_board.cc`文件，实现开发板的所有初始化逻辑。
一个基本的开发板类定义包含以下几个部分：
1. **类定义**：继承自`WifiBoard`或`Ml307Board`
2. **初始化函数**：包括I2C、显示屏、按钮、IoT等组件的初始化
3. **虚函数重写**：如`GetAudioCodec()`、`GetDisplay()`、`GetBacklight()`等
4. **注册开发板**：使用`DECLARE_BOARD`宏注册开发板
```cpp
#include "wifi_board.h"
#include "codecs/es8311_audio_codec.h"
#include "display/lcd_display.h"
#include "application.h"
#include "button.h"
#include "config.h"
#include "mcp_server.h"
#include <esp_log.h>
#include <driver/i2c_master.h>
#include <driver/spi_common.h>
#define TAG "MyCustomBoard"
class MyCustomBoard : public WifiBoard {
private:
i2c_master_bus_handle_t codec_i2c_bus_;
Button boot_button_;
LcdDisplay* display_;
// I2C初始化
void InitializeI2c() {
i2c_master_bus_config_t i2c_bus_cfg = {
.i2c_port = I2C_NUM_0,
.sda_io_num = AUDIO_CODEC_I2C_SDA_PIN,
.scl_io_num = AUDIO_CODEC_I2C_SCL_PIN,
.clk_source = I2C_CLK_SRC_DEFAULT,
.glitch_ignore_cnt = 7,
.intr_priority = 0,
.trans_queue_depth = 0,
.flags = {
.enable_internal_pullup = 1,
},
};
ESP_ERROR_CHECK(i2c_new_master_bus(&i2c_bus_cfg, &codec_i2c_bus_));
}
// SPI初始化（用于显示屏）
void InitializeSpi() {
spi_bus_config_t buscfg = {};
buscfg.mosi_io_num = DISPLAY_SPI_MOSI_PIN;
buscfg.miso_io_num = GPIO_NUM_NC;
buscfg.sclk_io_num = DISPLAY_SPI_SCK_PIN;
buscfg.quadwp_io_num = GPIO_NUM_NC;
buscfg.quadhd_io_num = GPIO_NUM_NC;
buscfg.max_transfer_sz = DISPLAY_WIDTH * DISPLAY_HEIGHT * sizeof(uint16_t);
ESP_ERROR_CHECK(spi_bus_initialize(SPI2_HOST, &buscfg, SPI_DMA_CH_AUTO));
}
// 按钮初始化
void InitializeButtons() {
boot_button_.OnClick([this]() {
auto& app = Application::GetInstance();
if (app.GetDeviceState() == kDeviceStateStarting && !WifiStation::GetInstance().IsConnected()) {
ResetWifiConfiguration();
}
app.ToggleChatState();
});
}
// 显示屏初始化（以ST7789为例）
void InitializeDisplay() {
esp_lcd_panel_io_handle_t panel_io = nullptr;
esp_lcd_panel_handle_t panel = nullptr;
esp_lcd_panel_io_spi_config_t io_config = {};
io_config.cs_gpio_num = DISPLAY_SPI_CS_PIN;
io_config.dc_gpio_num = DISPLAY_DC_PIN;
io_config.spi_mode = 2;
io_config.pclk_hz = 80 * 1000 * 1000;
io_config.trans_queue_depth = 10;
io_config.lcd_cmd_bits = 8;
io_config.lcd_param_bits = 8;
ESP_ERROR_CHECK(esp_lcd_new_panel_io_spi(SPI2_HOST, &io_config, &panel_io));
esp_lcd_panel_dev_config_t panel_config = {};
panel_config.reset_gpio_num = GPIO_NUM_NC;
panel_config.rgb_ele_order = LCD_RGB_ELEMENT_ORDER_RGB;
panel_config.bits_per_pixel = 16;
ESP_ERROR_CHECK(esp_lcd_new_panel_st7789(panel_io, &panel_config, &panel));
esp_lcd_panel_reset(panel);
esp_lcd_panel_init(panel);
esp_lcd_panel_invert_color(panel, true);
esp_lcd_panel_swap_xy(panel, DISPLAY_SWAP_XY);
esp_lcd_panel_mirror(panel, DISPLAY_MIRROR_X, DISPLAY_MIRROR_Y);
// 创建显示屏对象
display_ = new SpiLcdDisplay(panel_io, panel,
DISPLAY_WIDTH, DISPLAY_HEIGHT,
DISPLAY_OFFSET_X, DISPLAY_OFFSET_Y,
DISPLAY_MIRROR_X, DISPLAY_MIRROR_Y, DISPLAY_SWAP_XY);
}
// MCP Tools 初始化
void InitializeTools() {
// 参考 MCP 文档
}
public:
// 构造函数
MyCustomBoard() : boot_button_(BOOT_BUTTON_GPIO) {
InitializeI2c();
InitializeSpi();
InitializeDisplay();
InitializeButtons();
InitializeTools();
GetBacklight()->SetBrightness(100);
}
// 获取音频编解码器
virtual AudioCodec* GetAudioCodec() override {
static Es8311AudioCodec audio_codec(
codec_i2c_bus_,
I2C_NUM_0,
AUDIO_INPUT_SAMPLE_RATE,
AUDIO_OUTPUT_SAMPLE_RATE,
AUDIO_I2S_GPIO_MCLK,
AUDIO_I2S_GPIO_BCLK,
AUDIO_I2S_GPIO_WS,
AUDIO_I2S_GPIO_DOUT,
AUDIO_I2S_GPIO_DIN,
AUDIO_CODEC_PA_PIN,
AUDIO_CODEC_ES8311_ADDR);
return &audio_codec;
}
// 获取显示屏
virtual Display* GetDisplay() override {
return display_;
}
// 获取背光控制
virtual Backlight* GetBacklight() override {
static PwmBacklight backlight(DISPLAY_BACKLIGHT_PIN, DISPLAY_BACKLIGHT_OUTPUT_INVERT);
return &backlight;
}
};
// 注册开发板
DECLARE_BOARD(MyCustomBoard);
```
### 4. 添加构建系统配置
#### 在 Kconfig.projbuild 中添加开发板选项
打开 `main/Kconfig.projbuild` 文件，在 `choice BOARD_TYPE` 部分添加新的开发板配置项：
```kconfig
choice BOARD_TYPE
prompt "Board Type"
default BOARD_TYPE_BREAD_COMPACT_WIFI
help
Board type. 开发板类型
# ... 其他开发板选项 ...
config BOARD_TYPE_MY_CUSTOM_BOARD
bool "My Custom Board (我的自定义开发板)"
depends on IDF_TARGET_ESP32S3 # 根据你的目标芯片修改
endchoice
```
**注意事项：**
- `BOARD_TYPE_MY_CUSTOM_BOARD` 是配置项名称，需要全大写，使用下划线分隔
- `depends on` 指定了目标芯片类型（如 `IDF_TARGET_ESP32S3`、`IDF_TARGET_ESP32C3` 等）
- 描述文字可以使用中英文
#### 在 CMakeLists.txt 中添加开发板配置
打开 `main/CMakeLists.txt` 文件，在开发板类型判断部分添加新的配置：
```cmake
# 在 elseif 链中添加你的开发板配置
elseif(CONFIG_BOARD_TYPE_MY_CUSTOM_BOARD)
set(BOARD_TYPE "my-custom-board") # 与目录名一致
set(BUILTIN_TEXT_FONT font_puhui_basic_20_4) # 根据屏幕大小选择合适的字体
set(BUILTIN_ICON_FONT font_awesome_20_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_64) # 可选，如果需要表情显示
endif()
```
**字体和表情配置说明：**
根据屏幕分辨率选择合适的字体大小：
- 小屏幕（128x64 OLED）：`font_puhui_basic_14_1` / `font_awesome_14_1`
- 中小屏幕（240x240）：`font_puhui_basic_16_4` / `font_awesome_16_4`
- 中等屏幕（240x320）：`font_puhui_basic_20_4` / `font_awesome_20_4`
- 大屏幕（480x320+）：`font_puhui_basic_30_4` / `font_awesome_30_4`
表情集合选项：
- `twemoji_32` - 32x32 像素表情（小屏幕）
- `twemoji_64` - 64x64 像素表情（大屏幕）
### 5. 配置和编译
#### 方法一：使用 idf.py 手动配置
1. **设置目标芯片**（首次配置或更换芯片时）：
```bash
# 对于 ESP32-S3
idf.py set-target esp32s3
# 对于 ESP32-C3
idf.py set-target esp32c3
# 对于 ESP32
idf.py set-target esp32
```
2. **清理旧配置**：
```bash
idf.py fullclean
```
3. **进入配置菜单**：
```bash
idf.py menuconfig
```
在菜单中导航到：`Xiaozhi Assistant` -> `Board Type`，选择你的自定义开发板。
4. **编译和烧录**：
```bash
idf.py build
idf.py flash monitor
```
#### 方法二：使用 release.py 脚本（推荐）
如果你的开发板目录下有 `config.json` 文件，可以使用此脚本自动完成配置和编译：
```bash
python scripts/release.py my-custom-board
```
此脚本会自动：
- 读取 `config.json` 中的 `target` 配置并设置目标芯片
- 应用 `sdkconfig_append` 中的编译选项
- 完成编译并打包固件
### 6. 创建README.md
在README.md中说明开发板的特性、硬件要求、编译和烧录步骤：
## 常见开发板组件
### 1. 显示屏
项目支持多种显示屏驱动，包括:
- ST7789 (SPI)
- ILI9341 (SPI)
- SH8601 (QSPI)
- 等...
### 2. 音频编解码器
支持的编解码器包括:
- ES8311 (常用)
- ES7210 (麦克风阵列)
- AW88298 (功放)
- 等...
### 3. 电源管理
一些开发板使用电源管理芯片:
- AXP2101
- 其他可用的PMIC
### 4. MCP设备控制
可以添加各种MCP工具，让AI能够使用:
- Speaker (扬声器控制)
- Screen (屏幕亮度调节)
- Battery (电池电量读取)
- Light (灯光控制)
- 等...
## 开发板类继承关系
- `Board` - 基础板级类
- `WifiBoard` - Wi-Fi连接的开发板
- `Ml307Board` - 使用4G模块的开发板
- `DualNetworkBoard` - 支持Wi-Fi与4G网络切换的开发板
## 开发技巧
1. **参考相似的开发板**：如果您的新开发板与现有开发板有相似之处，可以参考现有实现
2. **分步调试**：先实现基础功能（如显示），再添加更复杂的功能（如音频）
3. **管脚映射**：确保在config.h中正确配置所有管脚映射
4. **检查硬件兼容性**：确认所有芯片和驱动程序的兼容性
## 可能遇到的问题
1. **显示屏不正常**：检查SPI配置、镜像设置和颜色反转设置
2. **音频无输出**：检查I2S配置、PA使能引脚和编解码器地址
3. **无法连接网络**：检查Wi-Fi凭据和网络配置
4. **无法与服务器通信**：检查MQTT或WebSocket配置
## 参考资料
- ESP-IDF 文档: https://docs.espressif.com/projects/esp-idf/
- LVGL 文档: https://docs.lvgl.io/
- ESP-SR 文档: https://github.com/espressif/esp-sr

BIN
docs/mcp-based-graph.jpg Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 96 KiB

269
docs/mcp-protocol.md Normal file
View File

@ -0,0 +1,269 @@
# MCP (Model Context Protocol) 交互流程
NOTICE: AI 辅助生成, 在实现后台服务时, 请参照代码确认细节!!
本项目中的 MCP 协议用于后台 API（MCP 客户端）与 ESP32 设备（MCP 服务器）之间的通信，以便后台能够发现和调用设备提供的功能（工具）。
## 协议格式
根据代码 (`main/protocols/protocol.cc`, `main/mcp_server.cc`)，MCP 消息是封装在基础通信协议（如 WebSocket 或 MQTT）的消息体中的。其内部结构遵循 [JSON-RPC 2.0](https://www.jsonrpc.org/specification) 规范。
整体消息结构示例：
```json
{
"session_id": "...", // 会话 ID
"type": "mcp", // 消息类型，固定为 "mcp"
"payload": { // JSON-RPC 2.0 负载
"jsonrpc": "2.0",
"method": "...", // 方法名 (如 "initialize", "tools/list", "tools/call")
"params": { ... }, // 方法参数 (对于 request)
"id": ..., // 请求 ID (对于 request 和 response)
"result": { ... }, // 方法执行结果 (对于 success response)
"error": { ... } // 错误信息 (对于 error response)
}
}
```
其中，`payload` 部分是标准的 JSON-RPC 2.0 消息：
- `jsonrpc`: 固定的字符串 "2.0"。
- `method`: 要调用的方法名称 (对于 Request)。
- `params`: 方法的参数，一个结构化值，通常为对象 (对于 Request)。
- `id`: 请求的标识符，客户端发送请求时提供，服务器响应时原样返回。用于匹配请求和响应。
- `result`: 方法成功执行时的结果 (对于 Success Response)。
- `error`: 方法执行失败时的错误信息 (对于 Error Response)。
## 交互流程及发送时机
MCP 的交互主要围绕客户端（后台 API）发现和调用设备上的“工具”（Tool）进行。
1. **连接建立与能力通告**
- **时机：** 设备启动并成功连接到后台 API 后。
- **发送方：** 设备。
- **消息：** 设备发送基础协议的 "hello" 消息给后台 API，消息中包含设备支持的能力列表，例如通过支持 MCP 协议 (`"mcp": true`)。
- **示例 (非 MCP 负载，而是基础协议消息):**
```json
{
"type": "hello",
"version": ...,
"features": {
"mcp": true,
...
},
"transport": "websocket", // 或 "mqtt"
"audio_params": { ... },
"session_id": "..." // 设备收到服务器hello后可能设置
}
```
2. **初始化 MCP 会话**
- **时机：** 后台 API 收到设备 "hello" 消息，确认设备支持 MCP 后，通常作为 MCP 会话的第一个请求发送。
- **发送方：** 后台 API (客户端)。
- **方法：** `initialize`
- **消息 (MCP payload):**
```json
{
"jsonrpc": "2.0",
"method": "initialize",
"params": {
"capabilities": {
// 客户端能力，可选
// 摄像头视觉相关
"vision": {
"url": "...", //摄像头: 图片处理地址(必须是http地址, 不是websocket地址)
"token": "..." // url token
}
// ... 其他客户端能力
}
},
"id": 1 // 请求 ID
}
```
- **设备响应时机：** 设备收到 `initialize` 请求并处理后。
- **设备响应消息 (MCP payload):**
```json
{
"jsonrpc": "2.0",
"id": 1, // 匹配请求 ID
"result": {
"protocolVersion": "2024-11-05",
"capabilities": {
"tools": {} // 这里的 tools 似乎不列出详细信息，需要 tools/list
},
"serverInfo": {
"name": "...", // 设备名称 (BOARD_NAME)
"version": "..." // 设备固件版本
}
}
}
```
3. **发现设备工具列表**
- **时机：** 后台 API 需要获取设备当前支持的具体功能（工具）列表及其调用方式时。
- **发送方：** 后台 API (客户端)。
- **方法：** `tools/list`
- **消息 (MCP payload):**
```json
{
"jsonrpc": "2.0",
"method": "tools/list",
"params": {
"cursor": "" // 用于分页，首次请求为空字符串
},
"id": 2 // 请求 ID
}
```
- **设备响应时机：** 设备收到 `tools/list` 请求并生成工具列表后。
- **设备响应消息 (MCP payload):**
```json
{
"jsonrpc": "2.0",
"id": 2, // 匹配请求 ID
"result": {
"tools": [ // 工具对象列表
{
"name": "self.get_device_status",
"description": "...",
"inputSchema": { ... } // 参数 schema
},
{
"name": "self.audio_speaker.set_volume",
"description": "...",
"inputSchema": { ... } // 参数 schema
}
// ... 更多工具
],
"nextCursor": "..." // 如果列表很大需要分页，这里会包含下一个请求的 cursor 值
}
}
```
- **分页处理：** 如果 `nextCursor` 字段非空，客户端需要再次发送 `tools/list` 请求，并在 `params` 中带上这个 `cursor` 值以获取下一页工具。
4. **调用设备工具**
- **时机：** 后台 API 需要执行设备上的某个具体功能时。
- **发送方：** 后台 API (客户端)。
- **方法：** `tools/call`
- **消息 (MCP payload):**
```json
{
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"name": "self.audio_speaker.set_volume", // 要调用的工具名称
"arguments": {
// 工具参数，对象格式
"volume": 50 // 参数名及其值
}
},
"id": 3 // 请求 ID
}
```
- **设备响应时机：** 设备收到 `tools/call` 请求，执行相应的工具函数后。
- **设备成功响应消息 (MCP payload):**
```json
{
"jsonrpc": "2.0",
"id": 3, // 匹配请求 ID
"result": {
"content": [
// 工具执行结果内容
{ "type": "text", "text": "true" } // 示例：set_volume 返回 bool
],
"isError": false // 表示成功
}
}
```
- **设备失败响应消息 (MCP payload):**
```json
{
"jsonrpc": "2.0",
"id": 3, // 匹配请求 ID
"error": {
"code": -32601, // JSON-RPC 错误码，例如 Method not found (-32601)
"message": "Unknown tool: self.non_existent_tool" // 错误描述
}
}
```
5. **设备主动发送消息 (Notifications)**
- **时机：** 设备内部发生需要通知后台 API 的事件时（例如，状态变化，虽然代码示例中没有明确的工具发送此类消息，但 `Application::SendMcpMessage` 的存在暗示了设备可能主动发送 MCP 消息）。
- **发送方：** 设备 (服务器)。
- **方法：** 可能是以 `notifications/` 开头的方法名，或者其他自定义方法。
- **消息 (MCP payload):** 遵循 JSON-RPC Notification 格式，没有 `id` 字段。
```json
{
"jsonrpc": "2.0",
"method": "notifications/state_changed", // 示例方法名
"params": {
"newState": "idle",
"oldState": "connecting"
}
// 没有 id 字段
}
```
- **后台 API 处理：** 接收到 Notification 后，后台 API 进行相应的处理，但不回复。
## 交互图
下面是一个简化的交互序列图，展示了主要的 MCP 消息流程：
```mermaid
sequenceDiagram
participant Device as ESP32 Device
participant BackendAPI as 后台 API (Client)
Note over Device, BackendAPI: 建立 WebSocket / MQTT 连接
Device->>BackendAPI: Hello Message (包含 "mcp": true)
BackendAPI->>Device: MCP Initialize Request
Note over BackendAPI: method: initialize
Note over BackendAPI: params: { capabilities: ... }
Device->>BackendAPI: MCP Initialize Response
Note over Device: result: { protocolVersion: ..., serverInfo: ... }
BackendAPI->>Device: MCP Get Tools List Request
Note over BackendAPI: method: tools/list
Note over BackendAPI: params: { cursor: "" }
Device->>BackendAPI: MCP Get Tools List Response
Note over Device: result: { tools: [...], nextCursor: ... }
loop Optional Pagination
BackendAPI->>Device: MCP Get Tools List Request
Note over BackendAPI: method: tools/list
Note over BackendAPI: params: { cursor: "..." }
Device->>BackendAPI: MCP Get Tools List Response
Note over Device: result: { tools: [...], nextCursor: "" }
end
BackendAPI->>Device: MCP Call Tool Request
Note over BackendAPI: method: tools/call
Note over BackendAPI: params: { name: "...", arguments: { ... } }
alt Tool Call Successful
Device->>BackendAPI: MCP Tool Call Success Response
Note over Device: result: { content: [...], isError: false }
else Tool Call Failed
Device->>BackendAPI: MCP Tool Call Error Response
Note over Device: error: { code: ..., message: ... }
end
opt Device Notification
Device->>BackendAPI: MCP Notification
Note over Device: method: notifications/...
Note over Device: params: { ... }
end
```
这份文档概述了该项目中 MCP 协议的主要交互流程。具体的参数细节和工具功能需要参考 `main/mcp_server.cc` 中 `McpServer::AddCommonTools` 以及各个工具的实现。

115
docs/mcp-usage.md Normal file
View File

@ -0,0 +1,115 @@
# MCP 协议物联网控制用法说明
> 本文档介绍如何基于 MCP 协议实现 ESP32 设备的物联网控制。详细协议流程请参考 [`mcp-protocol.md`](./mcp-protocol.md)。
## 简介
MCP（Model Context Protocol）是新一代推荐用于物联网控制的协议，通过标准 JSON-RPC 2.0 格式在后台与设备间发现和调用"工具"（Tool），实现灵活的设备控制。
## 典型使用流程
1. 设备启动后通过基础协议（如 WebSocket/MQTT）与后台建立连接。
2. 后台通过 MCP 协议的 `initialize` 方法初始化会话。
3. 后台通过 `tools/list` 获取设备支持的所有工具（功能）及参数说明。
4. 后台通过 `tools/call` 调用具体工具，实现对设备的控制。
详细协议格式与交互请见 [`mcp-protocol.md`](./mcp-protocol.md)。
## 设备端工具注册方法说明
设备通过 `McpServer::AddTool` 方法注册可被后台调用的"工具"。其常用函数签名如下：
```cpp
void AddTool(
const std::string& name, // 工具名称，建议唯一且有层次感，如 self.dog.forward
const std::string& description, // 工具描述，简明说明功能，便于大模型理解
const PropertyList& properties, // 输入参数列表（可为空），支持类型：布尔、整数、字符串
std::function<ReturnValue(const PropertyList&)> callback // 工具被调用时的回调实现
);
```
- name：工具唯一标识，建议用"模块.功能"命名风格。
- description：自然语言描述，便于 AI/用户理解。
- properties：参数列表，支持类型有布尔、整数、字符串，可指定范围和默认值。
- callback：收到调用请求时的实际执行逻辑，返回值可为 bool/int/string。
## 典型注册示例（以 ESP-Hi 为例）
```cpp
void InitializeTools() {
auto& mcp_server = McpServer::GetInstance();
// 例1：无参数，控制机器人前进
mcp_server.AddTool("self.dog.forward", "机器人向前移动", PropertyList(), [this](const PropertyList&) -> ReturnValue {
servo_dog_ctrl_send(DOG_STATE_FORWARD, NULL);
return true;
});
// 例2：带参数，设置灯光 RGB 颜色
mcp_server.AddTool("self.light.set_rgb", "设置RGB颜色", PropertyList({
Property("r", kPropertyTypeInteger, 0, 255),
Property("g", kPropertyTypeInteger, 0, 255),
Property("b", kPropertyTypeInteger, 0, 255)
}), [this](const PropertyList& properties) -> ReturnValue {
int r = properties["r"].value<int>();
int g = properties["g"].value<int>();
int b = properties["b"].value<int>();
led_on_ = true;
SetLedColor(r, g, b);
return true;
});
}
```
## 常见工具调用 JSON-RPC 示例
### 1. 获取工具列表
```json
{
"jsonrpc": "2.0",
"method": "tools/list",
"params": { "cursor": "" },
"id": 1
}
```
### 2. 控制底盘前进
```json
{
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"name": "self.chassis.go_forward",
"arguments": {}
},
"id": 2
}
```
### 3. 切换灯光模式
```json
{
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"name": "self.chassis.switch_light_mode",
"arguments": { "light_mode": 3 }
},
"id": 3
}
```
### 4. 摄像头翻转
```json
{
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"name": "self.camera.set_camera_flipped",
"arguments": {}
},
"id": 4
}
```
## 备注
- 工具名称、参数及返回值请以设备端 `AddTool` 注册为准。
- 推荐所有新项目统一采用 MCP 协议进行物联网控制。
- 详细协议与进阶用法请查阅 [`mcp-protocol.md`](./mcp-protocol.md)。

393
docs/mqtt-udp.md Normal file
View File

@ -0,0 +1,393 @@
# MQTT + UDP 混合通信协议文档
基于代码实现整理的 MQTT + UDP 混合通信协议文档，概述设备端与服务器之间如何通过 MQTT 进行控制消息传输，通过 UDP 进行音频数据传输的交互方式。
---
## 1. 协议概览
本协议采用混合传输方式：
- **MQTT**：用于控制消息、状态同步、JSON 数据交换
- **UDP**：用于实时音频数据传输，支持加密
### 1.1 协议特点
- **双通道设计**：控制与数据分离，确保实时性
- **加密传输**：UDP 音频数据使用 AES-CTR 加密
- **序列号保护**：防止数据包重放和乱序
- **自动重连**：MQTT 连接断开时自动重连
---
## 2. 总体流程概览
```mermaid
sequenceDiagram
participant Device as ESP32 设备
participant MQTT as MQTT 服务器
participant UDP as UDP 服务器
Note over Device, UDP: 1. 建立 MQTT 连接
Device->>MQTT: MQTT Connect
MQTT->>Device: Connected
Note over Device, UDP: 2. 请求音频通道
Device->>MQTT: Hello Message (type: "hello", transport: "udp")
MQTT->>Device: Hello Response (UDP 连接信息 + 加密密钥)
Note over Device, UDP: 3. 建立 UDP 连接
Device->>UDP: UDP Connect
UDP->>Device: Connected
Note over Device, UDP: 4. 音频数据传输
loop 音频流传输
Device->>UDP: 加密音频数据 (Opus)
UDP->>Device: 加密音频数据 (Opus)
end
Note over Device, UDP: 5. 控制消息交换
par 控制消息
Device->>MQTT: Listen/TTS/MCP 消息
MQTT->>Device: STT/TTS/MCP 响应
end
Note over Device, UDP: 6. 关闭连接
Device->>MQTT: Goodbye Message
Device->>UDP: Disconnect
```
---
## 3. MQTT 控制通道
### 3.1 连接建立
设备通过 MQTT 连接到服务器，连接参数包括：
- **Endpoint**：MQTT 服务器地址和端口
- **Client ID**：设备唯一标识符
- **Username/Password**：认证凭据
- **Keep Alive**：心跳间隔（默认240秒）
### 3.2 Hello 消息交换
#### 3.2.1 设备端发送 Hello
```json
{
"type": "hello",
"version": 3,
"transport": "udp",
"features": {
"mcp": true
},
"audio_params": {
"format": "opus",
"sample_rate": 16000,
"channels": 1,
"frame_duration": 60
}
}
```
#### 3.2.2 服务器响应 Hello
```json
{
"type": "hello",
"transport": "udp",
"session_id": "xxx",
"audio_params": {
"format": "opus",
"sample_rate": 24000,
"channels": 1,
"frame_duration": 60
},
"udp": {
"server": "192.168.1.100",
"port": 8888,
"key": "0123456789ABCDEF0123456789ABCDEF",
"nonce": "0123456789ABCDEF0123456789ABCDEF"
}
}
```
**字段说明：**
- `udp.server`：UDP 服务器地址
- `udp.port`：UDP 服务器端口
- `udp.key`：AES 加密密钥（十六进制字符串）
- `udp.nonce`：AES 加密随机数（十六进制字符串）
### 3.3 JSON 消息类型
#### 3.3.1 设备端→服务器
1. **Listen 消息**
```json
{
"session_id": "xxx",
"type": "listen",
"state": "start",
"mode": "manual"
}
```
2. **Abort 消息**
```json
{
"session_id": "xxx",
"type": "abort",
"reason": "wake_word_detected"
}
```
3. **MCP 消息**
```json
{
"session_id": "xxx",
"type": "mcp",
"payload": {
"jsonrpc": "2.0",
"id": 1,
"result": {...}
}
}
```
4. **Goodbye 消息**
```json
{
"session_id": "xxx",
"type": "goodbye"
}
```
#### 3.3.2 服务器→设备端
支持的消息类型与 WebSocket 协议一致，包括：
- **STT**：语音识别结果
- **TTS**：语音合成控制
- **LLM**：情感表达控制
- **MCP**：物联网控制
- **System**：系统控制
- **Custom**：自定义消息（可选）
---
## 4. UDP 音频通道
### 4.1 连接建立
设备收到 MQTT Hello 响应后，使用其中的 UDP 连接信息建立音频通道：
1. 解析 UDP 服务器地址和端口
2. 解析加密密钥和随机数
3. 初始化 AES-CTR 加密上下文
4. 建立 UDP 连接
### 4.2 音频数据格式
#### 4.2.1 加密音频包结构
```
|type 1byte|flags 1byte|payload_len 2bytes|ssrc 4bytes|timestamp 4bytes|sequence 4bytes|
|payload payload_len bytes|
```
**字段说明：**
- `type`：数据包类型，固定为 0x01
- `flags`：标志位，当前未使用
- `payload_len`：负载长度（网络字节序）
- `ssrc`：同步源标识符
- `timestamp`：时间戳（网络字节序）
- `sequence`：序列号（网络字节序）
- `payload`：加密的 Opus 音频数据
#### 4.2.2 加密算法
使用 **AES-CTR** 模式加密：
- **密钥**：128位，由服务器提供
- **随机数**：128位，由服务器提供
- **计数器**：包含时间戳和序列号信息
### 4.3 序列号管理
- **发送端**：`local_sequence_` 单调递增
- **接收端**：`remote_sequence_` 验证连续性
- **防重放**：拒绝序列号小于期望值的数据包
- **容错处理**：允许轻微的序列号跳跃，记录警告
### 4.4 错误处理
1. **解密失败**：记录错误，丢弃数据包
2. **序列号异常**：记录警告，但仍处理数据包
3. **数据包格式错误**：记录错误，丢弃数据包
---
## 5. 状态管理
### 5.1 连接状态
```mermaid
stateDiagram
direction TB
[*] --> Disconnected
Disconnected --> MqttConnecting: StartMqttClient()
MqttConnecting --> MqttConnected: MQTT Connected
MqttConnecting --> Disconnected: Connect Failed
MqttConnected --> RequestingChannel: OpenAudioChannel()
RequestingChannel --> ChannelOpened: Hello Exchange Success
RequestingChannel --> MqttConnected: Hello Timeout/Failed
ChannelOpened --> UdpConnected: UDP Connect Success
UdpConnected --> AudioStreaming: Start Audio Transfer
AudioStreaming --> UdpConnected: Stop Audio Transfer
UdpConnected --> ChannelOpened: UDP Disconnect
ChannelOpened --> MqttConnected: CloseAudioChannel()
MqttConnected --> Disconnected: MQTT Disconnect
```
### 5.2 状态检查
设备通过以下条件判断音频通道是否可用：
```cpp
bool IsAudioChannelOpened() const {
return udp_ != nullptr && !error_occurred_ && !IsTimeout();
}
```
---
## 6. 配置参数
### 6.1 MQTT 配置
从设置中读取的配置项：
- `endpoint`：MQTT 服务器地址
- `client_id`：客户端标识符
- `username`：用户名
- `password`：密码
- `keepalive`：心跳间隔（默认240秒）
- `publish_topic`：发布主题
### 6.2 音频参数
- **格式**：Opus
- **采样率**：16000 Hz（设备端）/ 24000 Hz（服务器端）
- **声道数**：1（单声道）
- **帧时长**：60ms
---
## 7. 错误处理与重连
### 7.1 MQTT 重连机制
- 连接失败时自动重试
- 支持错误上报控制
- 断线时触发清理流程
### 7.2 UDP 连接管理
- 连接失败时不自动重试
- 依赖 MQTT 通道重新协商
- 支持连接状态查询
### 7.3 超时处理
基类 `Protocol` 提供超时检测：
- 默认超时时间：120 秒
- 基于最后接收时间计算
- 超时时自动标记为不可用
---
## 8. 安全考虑
### 8.1 传输加密
- **MQTT**：支持 TLS/SSL 加密（端口8883）
- **UDP**：使用 AES-CTR 加密音频数据
### 8.2 认证机制
- **MQTT**：用户名/密码认证
- **UDP**：通过 MQTT 通道分发密钥
### 8.3 防重放攻击
- 序列号单调递增
- 拒绝过期数据包
- 时间戳验证
---
## 9. 性能优化
### 9.1 并发控制
使用互斥锁保护 UDP 连接：
```cpp
std::lock_guard<std::mutex> lock(channel_mutex_);
```
### 9.2 内存管理
- 动态创建/销毁网络对象
- 智能指针管理音频数据包
- 及时释放加密上下文
### 9.3 网络优化
- UDP 连接复用
- 数据包大小优化
- 序列号连续性检查
---
## 10. 与 WebSocket 协议的比较
| 特性 | MQTT + UDP | WebSocket |
|------|------------|-----------|
| 控制通道 | MQTT | WebSocket |
| 音频通道 | UDP (加密) | WebSocket (二进制) |
| 实时性 | 高 (UDP) | 中等 |
| 可靠性 | 中等 | 高 |
| 复杂度 | 高 | 低 |
| 加密 | AES-CTR | TLS |
| 防火墙友好度 | 低 | 高 |
---
## 11. 部署建议
### 11.1 网络环境
- 确保 UDP 端口可达
- 配置防火墙规则
- 考虑 NAT 穿透
### 11.2 服务器配置
- MQTT Broker 配置
- UDP 服务器部署
- 密钥管理系统
### 11.3 监控指标
- 连接成功率
- 音频传输延迟
- 数据包丢失率
- 解密失败率
---
## 12. 总结
MQTT + UDP 混合协议通过以下设计实现高效的音视频通信：
- **分离式架构**：控制与数据通道分离，各司其职
- **加密保护**：AES-CTR 确保音频数据安全传输
- **序列化管理**：防止重放攻击和数据乱序
- **自动恢复**：支持连接断开后的自动重连
- **性能优化**：UDP 传输保证音频数据的实时性
该协议适用于对实时性要求较高的语音交互场景，但需要在网络复杂度和传输性能之间做出权衡。

BIN
docs/v0/AtomMatrix-echo-base.jpg Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 36 KiB

BIN
docs/v0/ESP32-BreadBoard.jpg Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 94 KiB

BIN
docs/v0/atoms3r-echo-base.jpg Executable file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 25 KiB

BIN
docs/v0/esp32s3-box3.jpg Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 17 KiB

BIN
docs/v0/lichuang-s3.jpg Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 20 KiB

BIN
docs/v0/m5stack-cores3.jpg Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 19 KiB

BIN
docs/v0/magiclick-2p4.jpg Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 12 KiB

BIN
docs/v0/waveshare-esp32-s3-touch-amoled-1.8.jpg Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 46 KiB

BIN
docs/v0/wiring.jpg Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 121 KiB

BIN
docs/v1/atoms3r.jpg Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 28 KiB

BIN
docs/v1/electron-bot.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 21 KiB

BIN
docs/v1/esp-hi.jpg Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 18 KiB

BIN
docs/v1/esp-sparkbot.jpg Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 35 KiB

BIN
docs/v1/espbox3.jpg Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 31 KiB

BIN
docs/v1/lichuang-s3.jpg Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 39 KiB

BIN
docs/v1/lilygo-t-circle-s3.jpg Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 47 KiB

BIN
docs/v1/m5cores3.jpg Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 23 KiB

BIN
docs/v1/magiclick.jpg Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 44 KiB

BIN
docs/v1/movecall-cuican-esp32s3.jpg Executable file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 50 KiB

BIN
docs/v1/movecall-moji-esp32s3.jpg Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 45 KiB

BIN
docs/v1/otto-robot.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 27 KiB

BIN
docs/v1/sensecap_watcher.jpg Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 38 KiB

BIN
docs/v1/waveshare.jpg Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 22 KiB

BIN
docs/v1/wiring2.jpg Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 57 KiB

BIN
docs/v1/wmnologo_xingzhi_0.96.jpg Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 24 KiB

BIN
docs/v1/wmnologo_xingzhi_1.54.jpg Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 51 KiB

BIN
docs/v1/xmini-c3.jpg Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 30 KiB

495
docs/websocket.md Normal file
View File

@ -0,0 +1,495 @@
以下是一份基于代码实现整理的 WebSocket 通信协议文档，概述设备端与服务器之间如何通过 WebSocket 进行交互。
该文档仅基于所提供的代码推断，实际部署时可能需要结合服务器端实现进行进一步确认或补充。
---
## 1. 总体流程概览
1. **设备端初始化**
- 设备上电、初始化 `Application`：
- 初始化音频编解码器、显示屏、LED 等
- 连接网络
- 创建并初始化实现 `Protocol` 接口的 WebSocket 协议实例（`WebsocketProtocol`）
- 进入主循环等待事件（音频输入、音频输出、调度任务等）。
2. **建立 WebSocket 连接**
- 当设备需要开始语音会话时（例如用户唤醒、手动按键触发等），调用 `OpenAudioChannel()`：
- 根据配置获取 WebSocket URL
- 设置若干请求头（`Authorization`, `Protocol-Version`, `Device-Id`, `Client-Id`）
- 调用 `Connect()` 与服务器建立 WebSocket 连接
3. **设备端发送 "hello" 消息**
- 连接成功后，设备会发送一条 JSON 消息，示例结构如下：
```json
{
"type": "hello",
"version": 1,
"features": {
"mcp": true
},
"transport": "websocket",
"audio_params": {
"format": "opus",
"sample_rate": 16000,
"channels": 1,
"frame_duration": 60
}
}
```
- 其中 `features` 字段为可选，内容根据设备编译配置自动生成。例如：`"mcp": true` 表示支持 MCP 协议。
- `frame_duration` 的值对应 `OPUS_FRAME_DURATION_MS`（例如 60ms）。
4. **服务器回复 "hello"**
- 设备等待服务器返回一条包含 `"type": "hello"` 的 JSON 消息，并检查 `"transport": "websocket"` 是否匹配。
- 服务器可选下发 `session_id` 字段，设备端收到后会自动记录。
- 示例：
```json
{
"type": "hello",
"transport": "websocket",
"session_id": "xxx",
"audio_params": {
"format": "opus",
"sample_rate": 24000,
"channels": 1,
"frame_duration": 60
}
}
```
- 如果匹配，则认为服务器已就绪，标记音频通道打开成功。
- 如果在超时时间（默认 10 秒）内未收到正确回复，认为连接失败并触发网络错误回调。
5. **后续消息交互**
- 设备端和服务器端之间可发送两种主要类型的数据：
1. **二进制音频数据**（Opus 编码）
2. **文本 JSON 消息**（用于传输聊天状态、TTS/STT 事件、MCP 协议消息等）
- 在代码里，接收回调主要分为：
- `OnData(...)`:
- 当 `binary` 为 `true` 时，认为是音频帧；设备会将其当作 Opus 数据进行解码。
- 当 `binary` 为 `false` 时，认为是 JSON 文本，需要在设备端用 cJSON 进行解析并做相应业务逻辑处理（如聊天、TTS、MCP 协议消息等）。
- 当服务器或网络出现断连，回调 `OnDisconnected()` 被触发：
- 设备会调用 `on_audio_channel_closed_()`，并最终回到空闲状态。
6. **关闭 WebSocket 连接**
- 设备在需要结束语音会话时，会调用 `CloseAudioChannel()` 主动断开连接，并回到空闲状态。
- 或者如果服务器端主动断开，也会引发同样的回调流程。
---
## 2. 通用请求头
在建立 WebSocket 连接时，代码示例中设置了以下请求头：
- `Authorization`: 用于存放访问令牌，形如 `"Bearer <token>"`
- `Protocol-Version`: 协议版本号，与 hello 消息体内的 `version` 字段保持一致
- `Device-Id`: 设备物理网卡 MAC 地址
- `Client-Id`: 软件生成的 UUID（擦除 NVS 或重新烧录完整固件会重置）
这些头会随着 WebSocket 握手一起发送到服务器，服务器可根据需求进行校验、认证等。
---
## 3. 二进制协议版本
设备支持多种二进制协议版本，通过配置中的 `version` 字段指定：
### 3.1 版本1（默认）
直接发送 Opus 音频数据，无额外元数据。Websocket 协议会区分 text 与 binary。
### 3.2 版本2
使用 `BinaryProtocol2` 结构：
```c
struct BinaryProtocol2 {
uint16_t version; // 协议版本
uint16_t type; // 消息类型 (0: OPUS, 1: JSON)
uint32_t reserved; // 保留字段
uint32_t timestamp; // 时间戳（毫秒，用于服务器端AEC）
uint32_t payload_size; // 负载大小（字节）
uint8_t payload[]; // 负载数据
} __attribute__((packed));
```
### 3.3 版本3
使用 `BinaryProtocol3` 结构：
```c
struct BinaryProtocol3 {
uint8_t type; // 消息类型
uint8_t reserved; // 保留字段
uint16_t payload_size; // 负载大小
uint8_t payload[]; // 负载数据
} __attribute__((packed));
```
---
## 4. JSON 消息结构
WebSocket 文本帧以 JSON 方式传输，以下为常见的 `"type"` 字段及其对应业务逻辑。若消息里包含未列出的字段，可能为可选或特定实现细节。
### 4.1 设备端→服务器
1. **Hello**
- 连接成功后，由设备端发送，告知服务器基本参数。
- 例：
```json
{
"type": "hello",
"version": 1,
"features": {
"mcp": true
},
"transport": "websocket",
"audio_params": {
"format": "opus",
"sample_rate": 16000,
"channels": 1,
"frame_duration": 60
}
}
```
2. **Listen**
- 表示设备端开始或停止录音监听。
- 常见字段：
- `"session_id"`：会话标识
- `"type": "listen"`
- `"state"`：`"start"`, `"stop"`, `"detect"`（唤醒检测已触发）
- `"mode"`：`"auto"`, `"manual"` 或 `"realtime"`，表示识别模式。
- 例：开始监听
```json
{
"session_id": "xxx",
"type": "listen",
"state": "start",
"mode": "manual"
}
```
3. **Abort**
- 终止当前说话（TTS 播放）或语音通道。
- 例：
```json
{
"session_id": "xxx",
"type": "abort",
"reason": "wake_word_detected"
}
```
- `reason` 值可为 `"wake_word_detected"` 或其他。
4. **Wake Word Detected**
- 用于设备端向服务器告知检测到唤醒词。
- 在发送该消息之前，可提前发送唤醒词的 Opus 音频数据，用于服务器进行声纹检测。
- 例：
```json
{
"session_id": "xxx",
"type": "listen",
"state": "detect",
"text": "你好小明"
}
```
5. **MCP**
- 推荐用于物联网控制的新一代协议。所有设备能力发现、工具调用等均通过 type: "mcp" 的消息进行，payload 内部为标准 JSON-RPC 2.0（详见 [MCP 协议文档](./mcp-protocol.md)）。
- **设备端到服务器发送 result 的例子：**
```json
{
"session_id": "xxx",
"type": "mcp",
"payload": {
"jsonrpc": "2.0",
"id": 1,
"result": {
"content": [
{ "type": "text", "text": "true" }
],
"isError": false
}
}
}
```
---
### 4.2 服务器→设备端
1. **Hello**
- 服务器端返回的握手确认消息。
- 必须包含 `"type": "hello"` 和 `"transport": "websocket"`。
- 可能会带有 `audio_params`，表示服务器期望的音频参数，或与设备端对齐的配置。
- 服务器可选下发 `session_id` 字段，设备端收到后会自动记录。
- 成功接收后设备端会设置事件标志，表示 WebSocket 通道就绪。
2. **STT**
- `{"session_id": "xxx", "type": "stt", "text": "..."}`
- 表示服务器端识别到了用户语音。（例如语音转文本结果）
- 设备可能将此文本显示到屏幕上，后续再进入回答等流程。
3. **LLM**
- `{"session_id": "xxx", "type": "llm", "emotion": "happy", "text": "😀"}`
- 服务器指示设备调整表情动画 / UI 表达。
4. **TTS**
- `{"session_id": "xxx", "type": "tts", "state": "start"}`：服务器准备下发 TTS 音频，设备端进入 "speaking" 播放状态。
- `{"session_id": "xxx", "type": "tts", "state": "stop"}`：表示本次 TTS 结束。
- `{"session_id": "xxx", "type": "tts", "state": "sentence_start", "text": "..."}`
- 让设备在界面上显示当前要播放或朗读的文本片段（例如用于显示给用户）。
5. **MCP**
- 服务器通过 type: "mcp" 的消息下发物联网相关的控制指令或返回调用结果，payload 结构同上。
- **服务器到设备端发送 tools/call 的例子：**
```json
{
"session_id": "xxx",
"type": "mcp",
"payload": {
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"name": "self.light.set_rgb",
"arguments": { "r": 255, "g": 0, "b": 0 }
},
"id": 1
}
}
```
6. **System**
- 系统控制命令，常用于远程升级更新。
- 例：
```json
{
"session_id": "xxx",
"type": "system",
"command": "reboot"
}
```
- 支持的命令：
- `"reboot"`：重启设备
7. **Custom**（可选）
- 自定义消息，当 `CONFIG_RECEIVE_CUSTOM_MESSAGE` 启用时支持。
- 例：
```json
{
"session_id": "xxx",
"type": "custom",
"payload": {
"message": "自定义内容"
}
}
```
8. **音频数据：二进制帧**
- 当服务器发送音频二进制帧（Opus 编码）时，设备端解码并播放。
- 若设备端正在处于 "listening" （录音）状态，收到的音频帧会被忽略或清空以防冲突。
---
## 5. 音频编解码
1. **设备端发送录音数据**
- 音频输入经过可能的回声消除、降噪或音量增益后，通过 Opus 编码打包为二进制帧发送给服务器。
- 根据协议版本，可能直接发送 Opus 数据（版本1）或使用带元数据的二进制协议（版本2/3）。
2. **设备端播放收到的音频**
- 收到服务器的二进制帧时，同样认定是 Opus 数据。
- 设备端会进行解码，然后交由音频输出接口播放。
- 如果服务器的音频采样率与设备不一致，会在解码后再进行重采样。
---
## 6. 常见状态流转
以下为常见设备端关键状态流转，与 WebSocket 消息对应：
1. **Idle** → **Connecting**
- 用户触发或唤醒后，设备调用 `OpenAudioChannel()` → 建立 WebSocket 连接 → 发送 `"type":"hello"`。
2. **Connecting** → **Listening**
- 成功建立连接后，若继续执行 `SendStartListening(...)`，则进入录音状态。此时设备会持续编码麦克风数据并发送到服务器。
3. **Listening** → **Speaking**
- 收到服务器 TTS Start 消息 (`{"type":"tts","state":"start"}`) → 停止录音并播放接收到的音频。
4. **Speaking** → **Idle**
- 服务器 TTS Stop (`{"type":"tts","state":"stop"}`) → 音频播放结束。若未继续进入自动监听，则返回 Idle；如果配置了自动循环，则再度进入 Listening。
5. **Listening** / **Speaking** → **Idle**（遇到异常或主动中断）
- 调用 `SendAbortSpeaking(...)` 或 `CloseAudioChannel()` → 中断会话 → 关闭 WebSocket → 状态回到 Idle。
### 自动模式状态流转图
```mermaid
stateDiagram
direction TB
[*] --> kDeviceStateUnknown
kDeviceStateUnknown --> kDeviceStateStarting:初始化
kDeviceStateStarting --> kDeviceStateWifiConfiguring:配置WiFi
kDeviceStateStarting --> kDeviceStateActivating:激活设备
kDeviceStateActivating --> kDeviceStateUpgrading:检测到新版本
kDeviceStateActivating --> kDeviceStateIdle:激活完成
kDeviceStateIdle --> kDeviceStateConnecting:开始连接
kDeviceStateConnecting --> kDeviceStateIdle:连接失败
kDeviceStateConnecting --> kDeviceStateListening:连接成功
kDeviceStateListening --> kDeviceStateSpeaking:开始说话
kDeviceStateSpeaking --> kDeviceStateListening:结束说话
kDeviceStateListening --> kDeviceStateIdle:手动终止
kDeviceStateSpeaking --> kDeviceStateIdle:自动终止
```
### 手动模式状态流转图
```mermaid
stateDiagram
direction TB
[*] --> kDeviceStateUnknown
kDeviceStateUnknown --> kDeviceStateStarting:初始化
kDeviceStateStarting --> kDeviceStateWifiConfiguring:配置WiFi
kDeviceStateStarting --> kDeviceStateActivating:激活设备
kDeviceStateActivating --> kDeviceStateUpgrading:检测到新版本
kDeviceStateActivating --> kDeviceStateIdle:激活完成
kDeviceStateIdle --> kDeviceStateConnecting:开始连接
kDeviceStateConnecting --> kDeviceStateIdle:连接失败
kDeviceStateConnecting --> kDeviceStateListening:连接成功
kDeviceStateIdle --> kDeviceStateListening:开始监听
kDeviceStateListening --> kDeviceStateIdle:停止监听
kDeviceStateIdle --> kDeviceStateSpeaking:开始说话
kDeviceStateSpeaking --> kDeviceStateIdle:结束说话
```
---
## 7. 错误处理
1. **连接失败**
- 如果 `Connect(url)` 返回失败或在等待服务器 "hello" 消息时超时，触发 `on_network_error_()` 回调。设备会提示"无法连接到服务"或类似错误信息。
2. **服务器断开**
- 如果 WebSocket 异常断开，回调 `OnDisconnected()`：
- 设备回调 `on_audio_channel_closed_()`
- 切换到 Idle 或其他重试逻辑。
---
## 8. 其它注意事项
1. **鉴权**
- 设备通过设置 `Authorization: Bearer <token>` 提供鉴权，服务器端需验证是否有效。
- 如果令牌过期或无效，服务器可拒绝握手或在后续断开。
2. **会话控制**
- 代码中部分消息包含 `session_id`，用于区分独立的对话或操作。服务端可根据需要对不同会话做分离处理。
3. **音频负载**
- 代码里默认使用 Opus 格式，并设置 `sample_rate = 16000`，单声道。帧时长由 `OPUS_FRAME_DURATION_MS` 控制，一般为 60ms。可根据带宽或性能做适当调整。为了获得更好的音乐播放效果，服务器下行音频可能使用 24000 采样率。
4. **协议版本配置**
- 通过设置中的 `version` 字段配置二进制协议版本（1、2 或 3）
- 版本1：直接发送 Opus 数据
- 版本2：使用带时间戳的二进制协议，适用于服务器端 AEC
- 版本3：使用简化的二进制协议
5. **物联网控制推荐 MCP 协议**
- 设备与服务器之间的物联网能力发现、状态同步、控制指令等，建议全部通过 MCP 协议（type: "mcp"）实现。原有的 type: "iot" 方案已废弃。
- MCP 协议可在 WebSocket、MQTT 等多种底层协议上传输，具备更好的扩展性和标准化能力。
- 详细用法请参考 [MCP 协议文档](./mcp-protocol.md) 及 [MCP 物联网控制用法](./mcp-usage.md)。
6. **错误或异常 JSON**
- 当 JSON 中缺少必要字段，例如 `{"type": ...}`，设备端会记录错误日志（`ESP_LOGE(TAG, "Missing message type, data: %s", data);`），不会执行任何业务。
---
## 9. 消息示例
下面给出一个典型的双向消息示例（流程简化示意）：
1. **设备端 → 服务器**（握手）
```json
{
"type": "hello",
"version": 1,
"features": {
"mcp": true
},
"transport": "websocket",
"audio_params": {
"format": "opus",
"sample_rate": 16000,
"channels": 1,
"frame_duration": 60
}
}
```
2. **服务器 → 设备端**（握手应答）
```json
{
"type": "hello",
"transport": "websocket",
"session_id": "xxx",
"audio_params": {
"format": "opus",
"sample_rate": 16000
}
}
```
3. **设备端 → 服务器**（开始监听）
```json
{
"session_id": "xxx",
"type": "listen",
"state": "start",
"mode": "auto"
}
```
同时设备端开始发送二进制帧（Opus 数据）。
4. **服务器 → 设备端**（ASR 结果）
```json
{
"session_id": "xxx",
"type": "stt",
"text": "用户说的话"
}
```
5. **服务器 → 设备端**（TTS开始）
```json
{
"session_id": "xxx",
"type": "tts",
"state": "start"
}
```
接着服务器发送二进制音频帧给设备端播放。
6. **服务器 → 设备端**（TTS结束）
```json
{
"session_id": "xxx",
"type": "tts",
"state": "stop"
}
```
设备端停止播放音频，若无更多指令，则回到空闲状态。
---
## 10. 总结
本协议通过在 WebSocket 上层传输 JSON 文本与二进制音频帧，完成功能包括音频流上传、TTS 音频播放、语音识别与状态管理、MCP 指令下发等。其核心特征：
- **握手阶段**：发送 `"type":"hello"`，等待服务器返回。
- **音频通道**：采用 Opus 编码的二进制帧双向传输语音流，支持多种协议版本。
- **JSON 消息**：使用 `"type"` 为核心字段标识不同业务逻辑，包括 TTS、STT、MCP、WakeWord、System、Custom 等。
- **扩展性**：可根据实际需求在 JSON 消息中添加字段，或在 headers 里进行额外鉴权。
服务器与设备端需提前约定各类消息的字段含义、时序逻辑以及错误处理规则，方能保证通信顺畅。上述信息可作为基础文档，便于后续对接、开发或扩展。

918
main/CMakeLists.txt Normal file
View File

@ -0,0 +1,918 @@
# Define source files
set(SOURCES "audio/audio_codec.cc"
"audio/audio_service.cc"
"audio/codecs/no_audio_codec.cc"
"audio/codecs/box_audio_codec.cc"
"audio/codecs/es8311_audio_codec.cc"
"audio/codecs/es8374_audio_codec.cc"
"audio/codecs/es8388_audio_codec.cc"
"audio/codecs/es8389_audio_codec.cc"
"audio/codecs/dummy_audio_codec.cc"
"audio/processors/audio_debugger.cc"
"led/single_led.cc"
"led/circular_strip.cc"
"led/gpio_led.cc"
"display/display.cc"
"display/lcd_display.cc"
"display/oled_display.cc"
"display/lvgl_display/lvgl_display.cc"
"display/emote_display.cc"
"display/lvgl_display/emoji_collection.cc"
"display/lvgl_display/lvgl_theme.cc"
"display/lvgl_display/lvgl_font.cc"
"display/lvgl_display/lvgl_image.cc"
"display/lvgl_display/gif/lvgl_gif.cc"
"display/lvgl_display/gif/gifdec.c"
"display/lvgl_display/jpg/image_to_jpeg.cpp"
"display/lvgl_display/jpg/jpeg_to_image.c"
"protocols/protocol.cc"
"protocols/mqtt_protocol.cc"
"protocols/websocket_protocol.cc"
"mcp_server.cc"
"system_info.cc"
"application.cc"
"ota.cc"
"settings.cc"
"device_state_event.cc"
"assets.cc"
"main.cc"
"uart_component.cc"
)
set(INCLUDE_DIRS "." "display" "display/lvgl_display" "display/lvgl_display/jpg" "audio" "protocols")
# Add board common files
file(GLOB BOARD_COMMON_SOURCES ${CMAKE_CURRENT_SOURCE_DIR}/boards/common/*.cc)
list(APPEND SOURCES ${BOARD_COMMON_SOURCES})
list(APPEND INCLUDE_DIRS ${CMAKE_CURRENT_SOURCE_DIR}/boards/common)
idf_build_get_property(build_components BUILD_COMPONENTS)
# Function to find component dynamically by pattern
function(find_component_by_pattern PATTERN COMPONENT_VAR PATH_VAR)
foreach(COMPONENT ${build_components})
if(COMPONENT MATCHES "${PATTERN}")
set(${COMPONENT_VAR} ${COMPONENT} PARENT_SCOPE)
idf_component_get_property(COMPONENT_PATH ${COMPONENT} COMPONENT_DIR)
set(${PATH_VAR} "${COMPONENT_PATH}" PARENT_SCOPE)
break()
endif()
endforeach()
endfunction()
# Set default BUILTIN_TEXT_FONT and BUILTIN_ICON_FONT
set(BUILTIN_TEXT_FONT font_puhui_14_1)
set(BUILTIN_ICON_FONT font_awesome_14_1)
# Add board files according to BOARD_TYPE
# Set default assets if the board uses partition table V2
if(CONFIG_BOARD_TYPE_BREAD_COMPACT_WIFI)
set(BOARD_TYPE "bread-compact-wifi")
set(BUILTIN_TEXT_FONT font_puhui_basic_14_1)
set(BUILTIN_ICON_FONT font_awesome_14_1)
elseif(CONFIG_BOARD_TYPE_BREAD_COMPACT_ML307)
set(BOARD_TYPE "bread-compact-ml307")
set(BUILTIN_TEXT_FONT font_puhui_basic_14_1)
set(BUILTIN_ICON_FONT font_awesome_14_1)
elseif(CONFIG_BOARD_TYPE_BREAD_COMPACT_ESP32)
set(BOARD_TYPE "bread-compact-esp32")
elseif(CONFIG_BOARD_TYPE_BREAD_COMPACT_ESP32_LCD)
set(BOARD_TYPE "bread-compact-esp32-lcd")
set(BUILTIN_TEXT_FONT font_puhui_basic_14_1)
set(BUILTIN_ICON_FONT font_awesome_14_1)
elseif(CONFIG_BOARD_TYPE_DF_K10)
set(BOARD_TYPE "df-k10")
set(BUILTIN_TEXT_FONT font_puhui_basic_20_4)
set(BUILTIN_ICON_FONT font_awesome_20_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_64)
elseif(CONFIG_BOARD_TYPE_DF_S3_AI_CAM)
set(BOARD_TYPE "df-s3-ai-cam")
elseif(CONFIG_BOARD_TYPE_ESP_BOX_3)
set(BOARD_TYPE "esp-box-3")
set(BUILTIN_TEXT_FONT font_puhui_basic_20_4)
set(BUILTIN_ICON_FONT font_awesome_20_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_64)
elseif(CONFIG_BOARD_TYPE_ESP_BOX)
set(BOARD_TYPE "esp-box")
set(BUILTIN_TEXT_FONT font_puhui_basic_20_4)
set(BUILTIN_ICON_FONT font_awesome_20_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_64)
elseif(CONFIG_BOARD_TYPE_ESP_BOX_LITE)
set(BOARD_TYPE "esp-box-lite")
set(BUILTIN_TEXT_FONT font_puhui_basic_20_4)
set(BUILTIN_ICON_FONT font_awesome_20_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_64)
elseif(CONFIG_BOARD_TYPE_KEVIN_BOX_2)
set(BOARD_TYPE "kevin-box-2")
set(BUILTIN_TEXT_FONT font_puhui_basic_14_1)
set(BUILTIN_ICON_FONT font_awesome_14_1)
elseif(CONFIG_BOARD_TYPE_KEVIN_C3)
set(BOARD_TYPE "kevin-c3")
elseif(CONFIG_BOARD_TYPE_KEVIN_SP_V3_DEV)
set(BOARD_TYPE "kevin-sp-v3-dev")
set(BUILTIN_TEXT_FONT font_puhui_basic_20_4)
set(BUILTIN_ICON_FONT font_awesome_20_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_64)
elseif(CONFIG_BOARD_TYPE_KEVIN_SP_V4_DEV)
set(BOARD_TYPE "kevin-sp-v4-dev")
set(BUILTIN_TEXT_FONT font_puhui_basic_20_4)
set(BUILTIN_ICON_FONT font_awesome_20_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_64)
elseif(CONFIG_BOARD_TYPE_KEVIN_YUYING_313LCD)
set(BOARD_TYPE "kevin-yuying-313lcd")
set(BUILTIN_TEXT_FONT font_puhui_basic_30_4)
set(BUILTIN_ICON_FONT font_awesome_30_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_64)
elseif(CONFIG_BOARD_TYPE_LICHUANG_DEV_S3)
set(BOARD_TYPE "lichuang-dev")
set(BUILTIN_TEXT_FONT font_puhui_basic_20_4)
set(BUILTIN_ICON_FONT font_awesome_20_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_64)
elseif(CONFIG_BOARD_TYPE_LICHUANG_DEV_C3)
set(BOARD_TYPE "lichuang-c3-dev")
set(BUILTIN_TEXT_FONT font_puhui_basic_20_4)
set(BUILTIN_ICON_FONT font_awesome_20_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_32)
elseif(CONFIG_BOARD_TYPE_MAGICLICK_S3_2P4)
set(BOARD_TYPE "magiclick-2p4")
set(BUILTIN_TEXT_FONT font_puhui_basic_16_4)
set(BUILTIN_ICON_FONT font_awesome_16_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_32)
elseif(CONFIG_BOARD_TYPE_MAGICLICK_S3_2P5)
set(BOARD_TYPE "magiclick-2p5")
set(BUILTIN_TEXT_FONT font_puhui_basic_16_4)
set(BUILTIN_ICON_FONT font_awesome_16_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_32)
elseif(CONFIG_BOARD_TYPE_MAGICLICK_C3)
set(BOARD_TYPE "magiclick-c3")
set(BUILTIN_TEXT_FONT font_puhui_basic_16_4)
set(BUILTIN_ICON_FONT font_awesome_16_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_32)
elseif(CONFIG_BOARD_TYPE_MAGICLICK_C3_V2)
set(BOARD_TYPE "magiclick-c3-v2")
set(BUILTIN_TEXT_FONT font_puhui_basic_16_4)
set(BUILTIN_ICON_FONT font_awesome_16_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_32)
elseif(CONFIG_BOARD_TYPE_M5STACK_CORE_S3)
set(BOARD_TYPE "m5stack-core-s3")
set(BUILTIN_TEXT_FONT font_puhui_basic_20_4)
set(BUILTIN_ICON_FONT font_awesome_20_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_64)
elseif(CONFIG_BOARD_TYPE_M5STACK_CORE_TAB5)
set(BOARD_TYPE "m5stack-tab5")
set(BUILTIN_TEXT_FONT font_puhui_basic_30_4)
set(BUILTIN_ICON_FONT font_awesome_30_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_64)
elseif(CONFIG_BOARD_TYPE_M5STACK_ATOM_S3_ECHO_BASE)
set(BOARD_TYPE "atoms3-echo-base")
set(BUILTIN_TEXT_FONT font_puhui_basic_16_4)
set(BUILTIN_ICON_FONT font_awesome_16_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_32)
elseif(CONFIG_BOARD_TYPE_M5STACK_ATOM_S3R_ECHO_BASE)
set(BOARD_TYPE "atoms3r-echo-base")
set(BUILTIN_TEXT_FONT font_puhui_basic_16_4)
set(BUILTIN_ICON_FONT font_awesome_16_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_32)
elseif(CONFIG_BOARD_TYPE_M5STACK_ATOM_S3R_CAM_M12_ECHO_BASE)
set(BOARD_TYPE "atoms3r-cam-m12-echo-base")
elseif(CONFIG_BOARD_TYPE_M5STACK_ATOM_ECHOS3R)
set(BOARD_TYPE "atom-echos3r")
elseif(CONFIG_BOARD_TYPE_M5STACK_ATOM_MATRIX_ECHO_BASE)
set(BOARD_TYPE "atommatrix-echo-base")
elseif(CONFIG_BOARD_TYPE_XMINI_C3_V3)
set(BOARD_TYPE "xmini-c3-v3")
set(BUILTIN_TEXT_FONT font_puhui_basic_14_1)
set(BUILTIN_ICON_FONT font_awesome_14_1)
elseif(CONFIG_BOARD_TYPE_XMINI_C3_4G)
set(BOARD_TYPE "xmini-c3-4g")
set(BUILTIN_TEXT_FONT font_puhui_basic_14_1)
set(BUILTIN_ICON_FONT font_awesome_14_1)
elseif(CONFIG_BOARD_TYPE_XMINI_C3)
set(BOARD_TYPE "xmini-c3")
set(BUILTIN_TEXT_FONT font_puhui_basic_14_1)
set(BUILTIN_ICON_FONT font_awesome_14_1)
elseif(CONFIG_BOARD_TYPE_ESP_KORVO2_V3)
set(BOARD_TYPE "esp32s3-korvo2-v3")
set(BUILTIN_TEXT_FONT font_puhui_basic_20_4)
set(BUILTIN_ICON_FONT font_awesome_20_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_64)
elseif(CONFIG_BOARD_TYPE_ESP_SPARKBOT)
set(BOARD_TYPE "esp-sparkbot")
set(BUILTIN_TEXT_FONT font_puhui_basic_20_4)
set(BUILTIN_ICON_FONT font_awesome_20_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_64)
elseif(CONFIG_BOARD_TYPE_ESP_SPOT_S3)
set(BOARD_TYPE "esp-spot")
elseif(CONFIG_BOARD_TYPE_ESP_SPOT_C5)
set(BOARD_TYPE "esp-spot")
elseif(CONFIG_BOARD_TYPE_ESP_HI)
set(BOARD_TYPE "esp-hi")
# Set ESP_HI emoji directory for DEFAULT_ASSETS_EXTRA_FILES
set(DEFAULT_ASSETS_EXTRA_FILES "${CMAKE_BINARY_DIR}/emoji")
elseif(CONFIG_BOARD_TYPE_ECHOEAR)
set(BOARD_TYPE "echoear")
set(BUILTIN_TEXT_FONT font_puhui_20_4)
set(BUILTIN_ICON_FONT font_awesome_20_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_64)
elseif(CONFIG_BOARD_TYPE_WAVESHARE_S3_AUDIO_BOARD)
set(BOARD_TYPE "waveshare-s3-audio-board")
set(BUILTIN_TEXT_FONT font_puhui_basic_16_4)
set(BUILTIN_ICON_FONT font_awesome_16_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_32)
elseif(CONFIG_BOARD_TYPE_WAVESHARE_S3_TOUCH_AMOLED_1_8)
set(BOARD_TYPE "esp32-s3-touch-amoled-1.8")
set(BUILTIN_TEXT_FONT font_puhui_basic_30_4)
set(BUILTIN_ICON_FONT font_awesome_30_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_64)
elseif(CONFIG_BOARD_TYPE_WAVESHARE_S3_TOUCH_AMOLED_2_06)
set(BOARD_TYPE "waveshare-s3-touch-amoled-2.06")
set(BUILTIN_TEXT_FONT font_puhui_basic_30_4)
set(BUILTIN_ICON_FONT font_awesome_30_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_64)
elseif(CONFIG_BOARD_TYPE_WAVESHARE_C6_TOUCH_AMOLED_2_06)
set(BOARD_TYPE "waveshare-c6-touch-amoled-2.06")
set(BUILTIN_TEXT_FONT font_puhui_basic_30_4)
set(BUILTIN_ICON_FONT font_awesome_30_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_64)
elseif(CONFIG_BOARD_TYPE_WAVESHARE_S3_TOUCH_LCD_4B)
set(BOARD_TYPE "waveshare-s3-touch-lcd-4b")
set(BUILTIN_TEXT_FONT font_puhui_basic_30_4)
set(BUILTIN_ICON_FONT font_awesome_30_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_64)
elseif(CONFIG_BOARD_TYPE_WAVESHARE_S3_TOUCH_AMOLED_1_75)
set(BOARD_TYPE "waveshare-s3-touch-amoled-1.75")
set(BUILTIN_TEXT_FONT font_puhui_basic_30_4)
set(BUILTIN_ICON_FONT font_awesome_30_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_64)
elseif(CONFIG_BOARD_TYPE_WAVESHARE_S3_TOUCH_LCD_1_83)
set(BOARD_TYPE "waveshare-s3-touch-lcd-1.83")
set(BUILTIN_TEXT_FONT font_puhui_basic_16_4)
set(BUILTIN_ICON_FONT font_awesome_16_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_64)
elseif(CONFIG_BOARD_TYPE_WAVESHARE_S3_TOUCH_LCD_1_85C)
set(BOARD_TYPE "esp32-s3-touch-lcd-1.85c")
set(BUILTIN_TEXT_FONT font_puhui_basic_16_4)
set(BUILTIN_ICON_FONT font_awesome_16_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_64)
elseif(CONFIG_BOARD_TYPE_WAVESHARE_S3_TOUCH_LCD_1_85)
set(BOARD_TYPE "esp32-s3-touch-lcd-1.85")
set(BUILTIN_TEXT_FONT font_puhui_basic_16_4)
set(BUILTIN_ICON_FONT font_awesome_16_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_64)
elseif(CONFIG_BOARD_TYPE_WAVESHARE_S3_TOUCH_LCD_1_46)
set(BOARD_TYPE "esp32-s3-touch-lcd-1.46")
set(BUILTIN_TEXT_FONT font_puhui_basic_16_4)
set(BUILTIN_ICON_FONT font_awesome_16_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_64)
elseif(CONFIG_BOARD_TYPE_WAVESHARE_S3_TOUCH_LCD_3_5)
set(BOARD_TYPE "esp32-s3-touch-lcd-3.5")
set(BUILTIN_TEXT_FONT font_puhui_basic_20_4)
set(BUILTIN_ICON_FONT font_awesome_20_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_64)
elseif(CONFIG_BOARD_TYPE_WAVESHARE_S3_TOUCH_LCD_3_5B)
set(BOARD_TYPE "waveshare-s3-touch-lcd-3.5b")
set(BUILTIN_TEXT_FONT font_puhui_basic_16_4)
set(BUILTIN_ICON_FONT font_awesome_16_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_32)
elseif(CONFIG_BOARD_TYPE_WAVESHARE_S3_ePaper_1_54)
set(BOARD_TYPE "waveshare-s3-epaper-1.54")
set(BUILTIN_TEXT_FONT font_puhui_basic_20_4)
set(BUILTIN_ICON_FONT font_awesome_20_4)
elseif(CONFIG_BOARD_TYPE_WAVESHARE_S3_TOUCH_LCD_3_49)
set(BOARD_TYPE "waveshare-s3-touch-lcd-3.49")
set(LVGL_TEXT_FONT font_puhui_basic_30_4)
set(LVGL_ICON_FONT font_awesome_30_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_64)
elseif(CONFIG_BOARD_TYPE_WAVESHARE_C6_LCD_1_69)
set(BOARD_TYPE "waveshare-c6-lcd-1.69")
set(BUILTIN_TEXT_FONT font_puhui_basic_20_4)
set(BUILTIN_ICON_FONT font_awesome_20_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_64)
elseif(CONFIG_BOARD_TYPE_WAVESHARE_C6_TOUCH_AMOLED_1_43)
set(BOARD_TYPE "waveshare-c6-touch-amoled-1.43")
set(BUILTIN_TEXT_FONT font_puhui_basic_30_4)
set(BUILTIN_ICON_FONT font_awesome_30_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_64)
elseif(CONFIG_BOARD_TYPE_WAVESHARE_C6_TOUCH_AMOLED_1_32)
set(BOARD_TYPE "waveshare-c6-touch-amoled-1.32")
set(BUILTIN_TEXT_FONT font_puhui_basic_20_4)
set(BUILTIN_ICON_FONT font_awesome_20_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_32)
elseif(CONFIG_BOARD_TYPE_WAVESHARE_S3_TOUCH_AMOLED_1_32)
set(BOARD_TYPE "waveshare-s3-touch-amoled-1.32")
set(BUILTIN_TEXT_FONT font_puhui_basic_20_4)
set(BUILTIN_ICON_FONT font_awesome_20_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_32)
elseif(CONFIG_BOARD_TYPE_WAVESHARE_P4_NANO)
set(BOARD_TYPE "waveshare-p4-nano")
set(BUILTIN_TEXT_FONT font_puhui_basic_30_4)
set(BUILTIN_ICON_FONT font_awesome_30_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_64)
elseif(CONFIG_BOARD_TYPE_WAVESHARE_P4_WIFI6_TOUCH_LCD_4B)
set(BOARD_TYPE "waveshare-p4-wifi6-touch-lcd-4b")
set(BUILTIN_TEXT_FONT font_puhui_basic_30_4)
set(BUILTIN_ICON_FONT font_awesome_30_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_64)
elseif(CONFIG_BOARD_TYPE_WAVESHARE_P4_WIFI6_TOUCH_LCD_7B)
set(BOARD_TYPE "waveshare-p4-wifi6-touch-lcd-7b")
set(BUILTIN_TEXT_FONT font_puhui_basic_30_4)
set(BUILTIN_ICON_FONT font_awesome_30_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_64)
elseif(CONFIG_BOARD_TYPE_WAVESHARE_P4_WIFI6_TOUCH_LCD_XC)
set(BOARD_TYPE "waveshare-p4-wifi6-touch-lcd-xc")
set(BUILTIN_TEXT_FONT font_puhui_basic_30_4)
set(BUILTIN_ICON_FONT font_awesome_30_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_64)
elseif(CONFIG_BOARD_TYPE_ESP_P4_FUNCTION_EV_BOARD)
set(BOARD_TYPE "esp-p4-function-ev-board")
elseif(CONFIG_BOARD_TYPE_BREAD_COMPACT_WIFI_LCD)
set(BOARD_TYPE "bread-compact-wifi-lcd")
set(BUILTIN_TEXT_FONT font_puhui_basic_16_4)
set(BUILTIN_ICON_FONT font_awesome_16_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_32)
elseif(CONFIG_BOARD_TYPE_TUDOUZI)
set(BOARD_TYPE "tudouzi")
set(BUILTIN_TEXT_FONT font_puhui_basic_14_1)
set(BUILTIN_ICON_FONT font_awesome_14_1)
elseif(CONFIG_BOARD_TYPE_LILYGO_T_CIRCLE_S3)
set(BOARD_TYPE "lilygo-t-circle-s3")
set(BUILTIN_TEXT_FONT font_puhui_basic_16_4)
set(BUILTIN_ICON_FONT font_awesome_16_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_32)
elseif(CONFIG_BOARD_TYPE_LILYGO_T_CAMERAPLUS_S3_V1_0_V1_1)
set(BOARD_TYPE "lilygo-t-cameraplus-s3")
set(BUILTIN_TEXT_FONT font_puhui_basic_20_4)
set(BUILTIN_ICON_FONT font_awesome_20_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_64)
elseif(CONFIG_BOARD_TYPE_LILYGO_T_CAMERAPLUS_S3_V1_2)
set(BOARD_TYPE "lilygo-t-cameraplus-s3")
set(BUILTIN_TEXT_FONT font_puhui_basic_20_4)
set(BUILTIN_ICON_FONT font_awesome_20_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_64)
elseif(CONFIG_BOARD_TYPE_LILYGO_T_DISPLAY_S3_PRO_MVSRLORA)
set(BOARD_TYPE "lilygo-t-display-s3-pro-mvsrlora")
set(BUILTIN_TEXT_FONT font_puhui_basic_20_4)
set(BUILTIN_ICON_FONT font_awesome_20_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_64)
elseif(CONFIG_BOARD_TYPE_LILYGO_T_DISPLAY_S3_PRO_MVSRLORA_NO_BATTERY)
set(BOARD_TYPE "lilygo-t-display-s3-pro-mvsrlora")
set(BUILTIN_TEXT_FONT font_puhui_basic_20_4)
set(BUILTIN_ICON_FONT font_awesome_20_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_64)
elseif(CONFIG_BOARD_TYPE_LILYGO_T_DISPLAY_P4)
set(BOARD_TYPE "lilygo-t-display-p4")
set(BUILTIN_TEXT_FONT font_puhui_basic_30_4)
set(BUILTIN_ICON_FONT font_awesome_30_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_64)
elseif(CONFIG_BOARD_TYPE_MOVECALL_MOJI_ESP32S3)
set(BOARD_TYPE "movecall-moji-esp32s3")
set(BUILTIN_TEXT_FONT font_puhui_basic_20_4)
set(BUILTIN_ICON_FONT font_awesome_20_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_64)
elseif(CONFIG_BOARD_TYPE_MOVECALL_CUICAN_ESP32S3)
set(BOARD_TYPE "movecall-cuican-esp32s3")
set(BUILTIN_TEXT_FONT font_puhui_basic_20_4)
set(BUILTIN_ICON_FONT font_awesome_20_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_64)
elseif(CONFIG_BOARD_TYPE_ATK_DNESP32S3)
set(BOARD_TYPE "atk-dnesp32s3")
set(BUILTIN_TEXT_FONT font_puhui_basic_20_4)
set(BUILTIN_ICON_FONT font_awesome_20_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_64)
elseif(CONFIG_BOARD_TYPE_ATK_DNESP32S3_BOX)
set(BOARD_TYPE "atk-dnesp32s3-box")
set(BUILTIN_TEXT_FONT font_puhui_basic_20_4)
set(BUILTIN_ICON_FONT font_awesome_20_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_64)
elseif(CONFIG_BOARD_TYPE_ATK_DNESP32S3_BOX0)
set(BOARD_TYPE "atk-dnesp32s3-box0")
set(BUILTIN_TEXT_FONT font_puhui_basic_20_4)
set(BUILTIN_ICON_FONT font_awesome_20_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_64)
elseif(CONFIG_BOARD_TYPE_ATK_DNESP32S3_BOX2_WIFI)
set(BOARD_TYPE "atk-dnesp32s3-box2-wifi")
set(BUILTIN_TEXT_FONT font_puhui_basic_20_4)
set(BUILTIN_ICON_FONT font_awesome_20_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_64)
elseif(CONFIG_BOARD_TYPE_ATK_DNESP32S3_BOX2_4G)
set(BOARD_TYPE "atk-dnesp32s3-box2-4g")
set(BUILTIN_TEXT_FONT font_puhui_basic_20_4)
set(BUILTIN_ICON_FONT font_awesome_20_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_64)
elseif(CONFIG_BOARD_TYPE_ATK_DNESP32S3M_WIFI)
set(BOARD_TYPE "atk-dnesp32s3m-wifi")
set(BUILTIN_TEXT_FONT font_puhui_basic_16_4)
set(BUILTIN_ICON_FONT font_awesome_16_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_32)
elseif(CONFIG_BOARD_TYPE_ATK_DNESP32S3M_4G)
set(BOARD_TYPE "atk-dnesp32s3m-4g")
set(BUILTIN_TEXT_FONT font_puhui_basic_16_4)
set(BUILTIN_ICON_FONT font_awesome_16_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_32)
elseif(CONFIG_BOARD_TYPE_DU_CHATX)
set(BOARD_TYPE "du-chatx")
set(BUILTIN_TEXT_FONT font_puhui_basic_16_4)
set(BUILTIN_ICON_FONT font_awesome_16_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_32)
elseif(CONFIG_BOARD_TYPE_TAIJI_PI_S3)
set(BOARD_TYPE "taiji-pi-s3")
set(BUILTIN_TEXT_FONT font_puhui_basic_20_4)
set(BUILTIN_ICON_FONT font_awesome_20_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_64)
elseif(CONFIG_BOARD_TYPE_XINGZHI_CUBE_0_85TFT_WIFI)
set(BOARD_TYPE "xingzhi-cube-0.85tft-wifi")
set(BUILTIN_TEXT_FONT font_puhui_basic_16_4)
set(BUILTIN_ICON_FONT font_awesome_16_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_32)
elseif(CONFIG_BOARD_TYPE_XINGZHI_CUBE_0_85TFT_ML307)
set(BOARD_TYPE "xingzhi-cube-0.85tft-ml307")
set(BUILTIN_TEXT_FONT font_puhui_basic_16_4)
set(BUILTIN_ICON_FONT font_awesome_16_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_32)
elseif(CONFIG_BOARD_TYPE_XINGZHI_CUBE_0_96OLED_WIFI)
set(BOARD_TYPE "xingzhi-cube-0.96oled-wifi")
set(BUILTIN_TEXT_FONT font_puhui_basic_14_1)
set(BUILTIN_ICON_FONT font_awesome_14_1)
elseif(CONFIG_BOARD_TYPE_XINGZHI_CUBE_0_96OLED_ML307)
set(BOARD_TYPE "xingzhi-cube-0.96oled-ml307")
set(BUILTIN_TEXT_FONT font_puhui_basic_14_1)
set(BUILTIN_ICON_FONT font_awesome_14_1)
elseif(CONFIG_BOARD_TYPE_XINGZHI_CUBE_1_54TFT_WIFI)
set(BOARD_TYPE "xingzhi-cube-1.54tft-wifi")
set(BUILTIN_TEXT_FONT font_puhui_basic_20_4)
set(BUILTIN_ICON_FONT font_awesome_20_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_64)
elseif(CONFIG_BOARD_TYPE_XINGZHI_CUBE_1_54TFT_ML307)
set(BOARD_TYPE "xingzhi-cube-1.54tft-ml307")
set(BUILTIN_TEXT_FONT font_puhui_basic_20_4)
set(BUILTIN_ICON_FONT font_awesome_20_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_64)
elseif(CONFIG_BOARD_TYPE_XINGZHI_METAL_1_54_WIFI)
set(BOARD_TYPE "xingzhi-metal-1.54-wifi")
set(BUILTIN_TEXT_FONT font_puhui_basic_20_4)
set(BUILTIN_ICON_FONT font_awesome_20_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_64)
elseif(CONFIG_BOARD_TYPE_SEEED_STUDIO_SENSECAP_WATCHER)
set(BOARD_TYPE "sensecap-watcher")
set(BUILTIN_TEXT_FONT font_puhui_basic_30_4)
set(BUILTIN_ICON_FONT font_awesome_20_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_64)
elseif(CONFIG_BOARD_TYPE_DOIT_S3_AIBOX)
set(BOARD_TYPE "doit-s3-aibox")
elseif(CONFIG_BOARD_TYPE_MIXGO_NOVA)
set(BOARD_TYPE "mixgo-nova")
set(BUILTIN_TEXT_FONT font_puhui_basic_16_4)
set(BUILTIN_ICON_FONT font_awesome_16_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_32)
elseif(CONFIG_BOARD_TYPE_GENJUTECH_S3_1_54TFT)
set(BOARD_TYPE "genjutech-s3-1.54tft")
set(BUILTIN_TEXT_FONT font_puhui_basic_20_4)
set(BUILTIN_ICON_FONT font_awesome_20_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_64)
elseif(CONFIG_BOARD_TYPE_CGC)
set(BOARD_TYPE "esp32-cgc")
set(BUILTIN_TEXT_FONT font_puhui_basic_14_1)
set(BUILTIN_ICON_FONT font_awesome_14_1)
elseif(CONFIG_BOARD_TYPE_CGC_144)
set(BOARD_TYPE "esp32-cgc-144")
set(BUILTIN_TEXT_FONT font_puhui_basic_14_1)
set(BUILTIN_ICON_FONT font_awesome_14_1)
elseif(CONFIG_BOARD_TYPE_ESP_S3_LCD_EV_Board)
set(BOARD_TYPE "esp-s3-lcd-ev-board")
set(BUILTIN_TEXT_FONT font_puhui_basic_30_4)
set(BUILTIN_ICON_FONT font_awesome_30_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_64)
elseif(CONFIG_BOARD_TYPE_ESP_S3_LCD_EV_Board_2)
set(BOARD_TYPE "esp-s3-lcd-ev-board-2")
set(BUILTIN_TEXT_FONT font_puhui_basic_30_4)
set(BUILTIN_ICON_FONT font_awesome_30_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_64)
elseif(CONFIG_BOARD_TYPE_ZHENGCHEN_1_54TFT_WIFI)
set(BOARD_TYPE "zhengchen-1.54tft-wifi")
set(BUILTIN_TEXT_FONT font_puhui_basic_20_4)
set(BUILTIN_ICON_FONT font_awesome_20_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_64)
elseif(CONFIG_BOARD_TYPE_MINSI_K08_DUAL)
set(BOARD_TYPE "minsi-k08-dual")
set(BUILTIN_TEXT_FONT font_puhui_basic_20_4)
set(BUILTIN_ICON_FONT font_awesome_20_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_64)
elseif(CONFIG_BOARD_TYPE_ZHENGCHEN_1_54TFT_ML307)
set(BOARD_TYPE "zhengchen-1.54tft-ml307")
set(BUILTIN_TEXT_FONT font_puhui_basic_20_4)
set(BUILTIN_ICON_FONT font_awesome_20_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_64)
elseif(CONFIG_BOARD_TYPE_SPOTPEAR_ESP32_S3_1_54_MUMA)
set(BOARD_TYPE "sp-esp32-s3-1.54-muma")
set(BUILTIN_TEXT_FONT font_puhui_basic_16_4)
set(BUILTIN_ICON_FONT font_awesome_16_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_32)
elseif(CONFIG_BOARD_TYPE_SPOTPEAR_ESP32_S3_1_28_BOX)
set(BOARD_TYPE "sp-esp32-s3-1.28-box")
set(BUILTIN_TEXT_FONT font_puhui_basic_16_4)
set(BUILTIN_ICON_FONT font_awesome_16_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_64)
elseif(CONFIG_BOARD_TYPE_OTTO_ROBOT)
set(BOARD_TYPE "otto-robot")
set(BUILTIN_TEXT_FONT font_puhui_16_4)
set(BUILTIN_ICON_FONT font_awesome_16_4)
elseif(CONFIG_BOARD_TYPE_ELECTRON_BOT)
set(BOARD_TYPE "electron-bot")
set(BUILTIN_TEXT_FONT font_puhui_20_4)
set(BUILTIN_ICON_FONT font_awesome_20_4)
elseif(CONFIG_BOARD_TYPE_BREAD_COMPACT_WIFI_CAM)
set(BOARD_TYPE "bread-compact-wifi-s3cam")
set(BUILTIN_TEXT_FONT font_puhui_basic_16_4)
set(BUILTIN_ICON_FONT font_awesome_16_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_64)
elseif(CONFIG_BOARD_TYPE_JIUCHUAN)
set(BOARD_TYPE "jiuchuan-s3")
set(BUILTIN_TEXT_FONT font_puhui_basic_20_4)
set(BUILTIN_ICON_FONT font_awesome_20_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_64)
elseif(CONFIG_BOARD_TYPE_LABPLUS_MPYTHON_V3)
set(BOARD_TYPE "labplus-mpython-v3")
set(BUILTIN_TEXT_FONT font_puhui_basic_20_4)
set(BUILTIN_ICON_FONT font_awesome_20_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_64)
elseif(CONFIG_BOARD_TYPE_LABPLUS_LEDONG_V2)
set(BOARD_TYPE "labplus-ledong-v2")
set(BUILTIN_TEXT_FONT font_puhui_basic_20_4)
set(BUILTIN_ICON_FONT font_awesome_20_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_64)
elseif(CONFIG_BOARD_TYPE_SURFER_C3_1_14TFT)
set(BOARD_TYPE "surfer-c3-1.14tft")
set(BUILTIN_TEXT_FONT font_puhui_basic_20_4)
set(BUILTIN_ICON_FONT font_awesome_20_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_32)
elseif(CONFIG_BOARD_TYPE_YUNLIAO_S3)
set(BOARD_TYPE "yunliao-s3")
set(BUILTIN_TEXT_FONT font_puhui_basic_20_4)
set(BUILTIN_ICON_FONT font_awesome_20_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_64)
elseif(CONFIG_BOARD_TYPE_WIRELESS_TAG_WTP4C5MP07S)
set(BOARD_TYPE "wireless-tag-wtp4c5mp07s")
set(BUILTIN_TEXT_FONT font_puhui_basic_30_4)
set(BUILTIN_ICON_FONT font_awesome_30_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_64)
elseif(CONFIG_BOARD_TYPE_AIPI_LITE)
set(BOARD_TYPE "aipi-lite")
set(BUILTIN_TEXT_FONT font_puhui_basic_14_1)
set(BUILTIN_ICON_FONT font_awesome_14_1)
set(DEFAULT_EMOJI_COLLECTION twemoji_32)
elseif(CONFIG_BOARD_TYPE_HU_087)
set(BOARD_TYPE "hu-087")
set(BUILTIN_TEXT_FONT font_puhui_basic_14_1)
set(BUILTIN_ICON_FONT font_awesome_14_1)
endif()
file(GLOB BOARD_SOURCES
${CMAKE_CURRENT_SOURCE_DIR}/boards/${BOARD_TYPE}/*.cc
${CMAKE_CURRENT_SOURCE_DIR}/boards/${BOARD_TYPE}/*.c
)
list(APPEND SOURCES ${BOARD_SOURCES})
# Select audio processor according to Kconfig
if(CONFIG_USE_AUDIO_PROCESSOR)
list(APPEND SOURCES "audio/processors/afe_audio_processor.cc")
else()
list(APPEND SOURCES "audio/processors/no_audio_processor.cc")
endif()
if(CONFIG_IDF_TARGET_ESP32S3 OR CONFIG_IDF_TARGET_ESP32P4)
list(APPEND SOURCES "audio/wake_words/afe_wake_word.cc")
list(APPEND SOURCES "audio/wake_words/custom_wake_word.cc")
else()
list(APPEND SOURCES "audio/wake_words/esp_wake_word.cc")
endif()
# Select language directory according to Kconfig
if(CONFIG_LANGUAGE_ZH_CN)
set(LANG_DIR "zh-CN")
elseif(CONFIG_LANGUAGE_ZH_TW)
set(LANG_DIR "zh-TW")
elseif(CONFIG_LANGUAGE_EN_US)
set(LANG_DIR "en-US")
elseif(CONFIG_LANGUAGE_JA_JP)
set(LANG_DIR "ja-JP")
elseif(CONFIG_LANGUAGE_KO_KR)
set(LANG_DIR "ko-KR")
elseif(CONFIG_LANGUAGE_VI_VN)
set(LANG_DIR "vi-VN")
elseif(CONFIG_LANGUAGE_TH_TH)
set(LANG_DIR "th-TH")
elseif(CONFIG_LANGUAGE_DE_DE)
set(LANG_DIR "de-DE")
elseif(CONFIG_LANGUAGE_FR_FR)
set(LANG_DIR "fr-FR")
elseif(CONFIG_LANGUAGE_ES_ES)
set(LANG_DIR "es-ES")
elseif(CONFIG_LANGUAGE_IT_IT)
set(LANG_DIR "it-IT")
elseif(CONFIG_LANGUAGE_RU_RU)
set(LANG_DIR "ru-RU")
elseif(CONFIG_LANGUAGE_AR_SA)
set(LANG_DIR "ar-SA")
elseif(CONFIG_LANGUAGE_HI_IN)
set(LANG_DIR "hi-IN")
elseif(CONFIG_LANGUAGE_PT_PT)
set(LANG_DIR "pt-PT")
elseif(CONFIG_LANGUAGE_PL_PL)
set(LANG_DIR "pl-PL")
elseif(CONFIG_LANGUAGE_CS_CZ)
set(LANG_DIR "cs-CZ")
elseif(CONFIG_LANGUAGE_FI_FI)
set(LANG_DIR "fi-FI")
elseif(CONFIG_LANGUAGE_TR_TR)
set(LANG_DIR "tr-TR")
elseif(CONFIG_LANGUAGE_ID_ID)
set(LANG_DIR "id-ID")
elseif(CONFIG_LANGUAGE_UK_UA)
set(LANG_DIR "uk-UA")
elseif(CONFIG_LANGUAGE_RO_RO)
set(LANG_DIR "ro-RO")
elseif(CONFIG_LANGUAGE_BG_BG)
set(LANG_DIR "bg-BG")
elseif(CONFIG_LANGUAGE_CA_ES)
set(LANG_DIR "ca-ES")
elseif(CONFIG_LANGUAGE_DA_DK)
set(LANG_DIR "da-DK")
elseif(CONFIG_LANGUAGE_EL_GR)
set(LANG_DIR "el-GR")
elseif(CONFIG_LANGUAGE_FA_IR)
set(LANG_DIR "fa-IR")
elseif(CONFIG_LANGUAGE_FIL_PH)
set(LANG_DIR "fil-PH")
elseif(CONFIG_LANGUAGE_HE_IL)
set(LANG_DIR "he-IL")
elseif(CONFIG_LANGUAGE_HR_HR)
set(LANG_DIR "hr-HR")
elseif(CONFIG_LANGUAGE_HU_HU)
set(LANG_DIR "hu-HU")
elseif(CONFIG_LANGUAGE_MS_MY)
set(LANG_DIR "ms-MY")
elseif(CONFIG_LANGUAGE_NB_NO)
set(LANG_DIR "nb-NO")
elseif(CONFIG_LANGUAGE_NL_NL)
set(LANG_DIR "nl-NL")
elseif(CONFIG_LANGUAGE_SK_SK)
set(LANG_DIR "sk-SK")
elseif(CONFIG_LANGUAGE_SL_SI)
set(LANG_DIR "sl-SI")
elseif(CONFIG_LANGUAGE_SV_SE)
set(LANG_DIR "sv-SE")
elseif(CONFIG_LANGUAGE_SR_RS)
set(LANG_DIR "sr-RS")
endif()
# Define generation path
set(LANG_JSON "${CMAKE_CURRENT_SOURCE_DIR}/assets/locales/${LANG_DIR}/language.json")
set(LANG_HEADER "${CMAKE_CURRENT_SOURCE_DIR}/assets/lang_config.h")
# Collect current language audio files
file(GLOB LANG_SOUNDS ${CMAKE_CURRENT_SOURCE_DIR}/assets/locales/${LANG_DIR}/*.ogg)
# If not en-US, collect en-US audio files as fallback for missing files
if(NOT LANG_DIR STREQUAL "en-US")
file(GLOB EN_US_SOUNDS ${CMAKE_CURRENT_SOURCE_DIR}/assets/locales/en-US/*.ogg)
# Extract filenames (without path) from current language
set(EXISTING_NAMES "")
foreach(SOUND_FILE ${LANG_SOUNDS})
get_filename_component(FILENAME ${SOUND_FILE} NAME)
list(APPEND EXISTING_NAMES ${FILENAME})
endforeach()
# Only add en-US audio files that are missing in current language
foreach(EN_SOUND ${EN_US_SOUNDS})
get_filename_component(FILENAME ${EN_SOUND} NAME)
if(NOT ${FILENAME} IN_LIST EXISTING_NAMES)
list(APPEND LANG_SOUNDS ${EN_SOUND})
message(STATUS "Using en-US fallback for missing audio: ${FILENAME}")
endif()
endforeach()
endif()
file(GLOB COMMON_SOUNDS ${CMAKE_CURRENT_SOURCE_DIR}/assets/common/*.ogg)
# If target chip is ESP32, exclude specific files to avoid build errors
if(CONFIG_IDF_TARGET_ESP32)
list(REMOVE_ITEM SOURCES "audio/codecs/box_audio_codec.cc"
"audio/codecs/es8388_audio_codec.cc"
"audio/codecs/es8389_audio_codec.cc"
"led/gpio_led.cc"
"${CMAKE_CURRENT_SOURCE_DIR}/boards/common/esp32_camera.cc"
"display/lvgl_display/jpg/image_to_jpeg.cpp"
"display/lvgl_display/jpg/jpeg_to_image.c"
)
endif()
idf_component_register(SRCS ${SOURCES}
EMBED_FILES ${LANG_SOUNDS} ${COMMON_SOUNDS}
INCLUDE_DIRS ${INCLUDE_DIRS}
WHOLE_ARCHIVE
)
# Use target_compile_definitions to define BOARD_TYPE, BOARD_NAME
# If BOARD_NAME is empty, use BOARD_TYPE
if(NOT BOARD_NAME)
set(BOARD_NAME ${BOARD_TYPE})
endif()
target_compile_definitions(${COMPONENT_LIB}
PRIVATE BOARD_TYPE=\"${BOARD_TYPE}\" BOARD_NAME=\"${BOARD_NAME}\"
PRIVATE BUILTIN_TEXT_FONT=${BUILTIN_TEXT_FONT} BUILTIN_ICON_FONT=${BUILTIN_ICON_FONT}
)
# Add generation rules
add_custom_command(
OUTPUT ${LANG_HEADER}
COMMAND python ${PROJECT_DIR}/scripts/gen_lang.py
--language "${LANG_DIR}"
--output "${LANG_HEADER}"
DEPENDS
${LANG_JSON}
${PROJECT_DIR}/scripts/gen_lang.py
COMMENT "Generating ${LANG_DIR} language config"
)
# Force build generation dependencies
add_custom_target(lang_header ALL
DEPENDS ${LANG_HEADER}
)
# Find ESP-SR component dynamically
find_component_by_pattern("espressif__esp-sr" ESP_SR_COMPONENT ESP_SR_COMPONENT_PATH)
if(ESP_SR_COMPONENT_PATH)
set(ESP_SR_MODEL_PATH "${ESP_SR_COMPONENT_PATH}/model")
endif()
# Find xiaozhi-fonts component dynamically
find_component_by_pattern("xiaozhi-fonts" XIAOZHI_FONTS_COMPONENT XIAOZHI_FONTS_COMPONENT_PATH)
if(XIAOZHI_FONTS_COMPONENT_PATH)
set(XIAOZHI_FONTS_PATH "${XIAOZHI_FONTS_COMPONENT_PATH}")
endif()
if(CONFIG_BOARD_TYPE_ESP_HI)
set(URL "https://github.com/espressif2022/image_player/raw/main/test_apps/test_8bit")
set(EMOJI_DIR "${CMAKE_BINARY_DIR}/emoji")
file(MAKE_DIRECTORY ${EMOJI_DIR})
# List all files to download
set(FILES_TO_DOWNLOAD "")
list(APPEND FILES_TO_DOWNLOAD "Anger_enter.aaf" "Anger_loop.aaf" "Anger_return.aaf")
list(APPEND FILES_TO_DOWNLOAD "happy_enter.aaf" "happy_loop.aaf" "happ_return.aaf")
list(APPEND FILES_TO_DOWNLOAD "sad_enter.aaf" "sad_loop.aaf" "sad_return.aaf")
list(APPEND FILES_TO_DOWNLOAD "scorn_enter.aaf" "scorn_loop.aaf" "scorn_return.aaf")
list(APPEND FILES_TO_DOWNLOAD "left_enter.aaf" "left_loop.aaf" "left_return.aaf")
list(APPEND FILES_TO_DOWNLOAD "right_enter.aaf" "right_loop.aaf" "right_return.aaf")
list(APPEND FILES_TO_DOWNLOAD "asking.aaf" "blink_once.aaf" "blink_quick.aaf")
list(APPEND FILES_TO_DOWNLOAD "connecting.aaf" "panic_enter.aaf" "panic_loop.aaf")
list(APPEND FILES_TO_DOWNLOAD "panic_return.aaf" "wake.aaf")
foreach(FILENAME IN LISTS FILES_TO_DOWNLOAD)
set(REMOTE_FILE "${URL}/${FILENAME}")
set(LOCAL_FILE "${EMOJI_DIR}/${FILENAME}")
# Check if local file exists
if(EXISTS ${LOCAL_FILE})
message(STATUS "File ${FILENAME} already exists, skipping download")
else()
message(STATUS "Downloading ${FILENAME}")
file(DOWNLOAD ${REMOTE_FILE} ${LOCAL_FILE}
STATUS DOWNLOAD_STATUS)
list(GET DOWNLOAD_STATUS 0 STATUS_CODE)
if(NOT STATUS_CODE EQUAL 0)
message(FATAL_ERROR "Failed to download ${FILENAME} from ${URL}")
endif()
endif()
endforeach()
endif()
# Function to build default assets based on configuration
function(build_default_assets_bin)
# Set output path for generated assets.bin
set(GENERATED_ASSETS_BIN "${CMAKE_BINARY_DIR}/generated_assets.bin")
# Prepare arguments for build script
set(BUILD_ARGS
"--sdkconfig" "${SDKCONFIG}"
"--output" "${GENERATED_ASSETS_BIN}"
)
# Add builtin text font if defined
if(BUILTIN_TEXT_FONT)
list(APPEND BUILD_ARGS "--builtin_text_font" "${BUILTIN_TEXT_FONT}")
endif()
# Add default emoji collection if defined
if(DEFAULT_EMOJI_COLLECTION)
list(APPEND BUILD_ARGS "--emoji_collection" "${DEFAULT_EMOJI_COLLECTION}")
endif()
# Add default assets extra files if defined
if(DEFAULT_ASSETS_EXTRA_FILES)
list(APPEND BUILD_ARGS "--extra_files" "${DEFAULT_ASSETS_EXTRA_FILES}")
endif()
list(APPEND BUILD_ARGS "--esp_sr_model_path" "${ESP_SR_MODEL_PATH}")
list(APPEND BUILD_ARGS "--xiaozhi_fonts_path" "${XIAOZHI_FONTS_PATH}")
# Create custom command to build assets
add_custom_command(
OUTPUT ${GENERATED_ASSETS_BIN}
COMMAND python ${PROJECT_DIR}/scripts/build_default_assets.py ${BUILD_ARGS}
DEPENDS
${SDKCONFIG}
${PROJECT_DIR}/scripts/build_default_assets.py
COMMENT "Building default assets.bin based on configuration"
VERBATIM
)
# Create target for generated assets
add_custom_target(generated_default_assets ALL
DEPENDS ${GENERATED_ASSETS_BIN}
)
# Set the generated file path in parent scope
set(GENERATED_ASSETS_LOCAL_FILE ${GENERATED_ASSETS_BIN} PARENT_SCOPE)
message(STATUS "Default assets build configured: ${GENERATED_ASSETS_BIN}")
endfunction()
# Function to get local assets file path (handles both URL and local file)
function(get_assets_local_file assets_source assets_local_file_var)
# Check if it's a URL (starts with http:// or https://)
if(assets_source MATCHES "^https?://")
# It's a URL, download it
get_filename_component(ASSETS_FILENAME "${assets_source}" NAME)
set(ASSETS_LOCAL_FILE "${CMAKE_BINARY_DIR}/${ASSETS_FILENAME}")
set(ASSETS_TEMP_FILE "${CMAKE_BINARY_DIR}/${ASSETS_FILENAME}.tmp")
# Check if local file exists
if(EXISTS ${ASSETS_LOCAL_FILE})
message(STATUS "Assets file ${ASSETS_FILENAME} already exists, skipping download")
else()
message(STATUS "Downloading ${ASSETS_FILENAME}")
# Clean up any existing temp file
if(EXISTS ${ASSETS_TEMP_FILE})
file(REMOVE ${ASSETS_TEMP_FILE})
endif()
# Download to temporary file first
file(DOWNLOAD ${assets_source} ${ASSETS_TEMP_FILE}
STATUS DOWNLOAD_STATUS)
list(GET DOWNLOAD_STATUS 0 STATUS_CODE)
if(NOT STATUS_CODE EQUAL 0)
# Clean up temp file on failure
if(EXISTS ${ASSETS_TEMP_FILE})
file(REMOVE ${ASSETS_TEMP_FILE})
endif()
message(FATAL_ERROR "Failed to download ${ASSETS_FILENAME} from ${assets_source}")
endif()
# Move temp file to final location (atomic operation)
file(RENAME ${ASSETS_TEMP_FILE} ${ASSETS_LOCAL_FILE})
message(STATUS "Successfully downloaded ${ASSETS_FILENAME}")
endif()
else()
# It's a local file path
if(IS_ABSOLUTE "${assets_source}")
set(ASSETS_LOCAL_FILE "${assets_source}")
else()
set(ASSETS_LOCAL_FILE "${CMAKE_CURRENT_SOURCE_DIR}/${assets_source}")
endif()
# Check if local file exists
if(NOT EXISTS ${ASSETS_LOCAL_FILE})
message(FATAL_ERROR "Assets file not found: ${ASSETS_LOCAL_FILE}")
endif()
message(STATUS "Using assets file: ${ASSETS_LOCAL_FILE}")
endif()
set(${assets_local_file_var} ${ASSETS_LOCAL_FILE} PARENT_SCOPE)
endfunction()
partition_table_get_partition_info(size "--partition-name assets" "size")
partition_table_get_partition_info(offset "--partition-name assets" "offset")
if ("${size}" AND "${offset}")
# Flash assets based on configuration
if(CONFIG_FLASH_DEFAULT_ASSETS)
# Build default assets based on configuration
build_default_assets_bin()
esptool_py_flash_to_partition(flash "assets" "${GENERATED_ASSETS_LOCAL_FILE}")
message(STATUS "Generated default assets flash configured: ${GENERATED_ASSETS_LOCAL_FILE} -> assets partition")
elseif(CONFIG_FLASH_CUSTOM_ASSETS)
# Flash custom assets
get_assets_local_file("${CONFIG_CUSTOM_ASSETS_FILE}" ASSETS_LOCAL_FILE)
esptool_py_flash_to_partition(flash "assets" "${ASSETS_LOCAL_FILE}")
message(STATUS "Custom assets flash configured: ${ASSETS_LOCAL_FILE} -> assets partition")
elseif(CONFIG_FLASH_NONE_ASSETS)
message(STATUS "Assets flashing disabled (FLASH_NONE_ASSETS)")
endif()
else()
message(STATUS "Assets partition not found, using v1 partition table")
endif()

791
main/Kconfig.projbuild Normal file
View File

@ -0,0 +1,791 @@
menu "Xiaozhi Assistant"
config OTA_URL
string "Default OTA URL"
default "https://api.tenclass.net/xiaozhi/ota/"
help
The application will access this URL to check for new firmwares and server address.
choice
prompt "Flash Assets"
default FLASH_DEFAULT_ASSETS
help
Select the assets to flash.
config FLASH_NONE_ASSETS
bool "Do not flash assets"
config FLASH_DEFAULT_ASSETS
bool "Flash Default Assets"
config FLASH_CUSTOM_ASSETS
bool "Flash Custom Assets"
endchoice
config CUSTOM_ASSETS_FILE
depends on FLASH_CUSTOM_ASSETS
string "Custom Assets File"
default "assets.bin"
help
The custom assets file to flash.
It can be a local file relative to the project directory or a remote url.
choice
prompt "Default Language"
default LANGUAGE_ZH_CN
help
Select device display language
config LANGUAGE_ZH_CN
bool "Chinese"
config LANGUAGE_ZH_TW
bool "Chinese Traditional"
config LANGUAGE_EN_US
bool "English"
config LANGUAGE_JA_JP
bool "Japanese"
config LANGUAGE_KO_KR
bool "Korean"
config LANGUAGE_VI_VN
bool "Vietnamese"
config LANGUAGE_TH_TH
bool "Thai"
config LANGUAGE_DE_DE
bool "German"
config LANGUAGE_FR_FR
bool "French"
config LANGUAGE_ES_ES
bool "Spanish"
config LANGUAGE_IT_IT
bool "Italian"
config LANGUAGE_RU_RU
bool "Russian"
config LANGUAGE_AR_SA
bool "Arabic"
config LANGUAGE_HI_IN
bool "Hindi"
config LANGUAGE_PT_PT
bool "Portuguese"
config LANGUAGE_PL_PL
bool "Polish"
config LANGUAGE_CS_CZ
bool "Czech"
config LANGUAGE_FI_FI
bool "Finnish"
config LANGUAGE_TR_TR
bool "Turkish"
config LANGUAGE_ID_ID
bool "Indonesian"
config LANGUAGE_UK_UA
bool "Ukrainian"
config LANGUAGE_RO_RO
bool "Romanian"
config LANGUAGE_BG_BG
bool "Bulgarian"
config LANGUAGE_CA_ES
bool "Catalan"
config LANGUAGE_DA_DK
bool "Danish"
config LANGUAGE_EL_GR
bool "Greek"
config LANGUAGE_FA_IR
bool "Persian"
config LANGUAGE_FIL_PH
bool "Filipino"
config LANGUAGE_HE_IL
bool "Hebrew"
config LANGUAGE_HR_HR
bool "Croatian"
config LANGUAGE_HU_HU
bool "Hungarian"
config LANGUAGE_MS_MY
bool "Malay"
config LANGUAGE_NB_NO
bool "Norwegian"
config LANGUAGE_NL_NL
bool "Dutch"
config LANGUAGE_SK_SK
bool "Slovak"
config LANGUAGE_SL_SI
bool "Slovenian"
config LANGUAGE_SV_SE
bool "Swedish"
config LANGUAGE_SR_RS
bool "Serbian"
endchoice
choice BOARD_TYPE
prompt "Board Type"
default BOARD_TYPE_BREAD_COMPACT_WIFI
help
Board type. 开发板类型
config BOARD_TYPE_BREAD_COMPACT_WIFI
bool "Bread Compact WiFi (面包板)"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_BREAD_COMPACT_WIFI_LCD
bool "Bread Compact WiFi + LCD (面包板)"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_BREAD_COMPACT_WIFI_CAM
bool "Bread Compact WiFi + LCD + Camera (面包板)"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_BREAD_COMPACT_ML307
bool "Bread Compact ML307/EC801E (面包板 4G)"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_BREAD_COMPACT_ESP32
bool "Bread Compact ESP32 DevKit (面包板)"
depends on IDF_TARGET_ESP32
config BOARD_TYPE_BREAD_COMPACT_ESP32_LCD
bool "Bread Compact ESP32 DevKit + LCD (面包板)"
depends on IDF_TARGET_ESP32
config BOARD_TYPE_XMINI_C3_V3
bool "Xmini C3 V3"
depends on IDF_TARGET_ESP32C3
config BOARD_TYPE_XMINI_C3_4G
bool "Xmini C3 4G"
depends on IDF_TARGET_ESP32C3
config BOARD_TYPE_XMINI_C3
bool "Xmini C3"
depends on IDF_TARGET_ESP32C3
config BOARD_TYPE_ESP_KORVO2_V3
bool "Espressif Korvo2 V3"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_ESP_SPARKBOT
bool "Espressif SparkBot"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_ESP_SPOT_S3
bool "Espressif Spot-S3"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_ESP_SPOT_C5
bool "Espressif Spot-C5"
depends on IDF_TARGET_ESP32C5
config BOARD_TYPE_ESP_HI
bool "Espressif ESP-HI"
depends on IDF_TARGET_ESP32C3
config BOARD_TYPE_ESP_BOX_3
bool "Espressif ESP-BOX-3"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_ESP_BOX
bool "Espressif ESP-BOX"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_ESP_BOX_LITE
bool "Espressif ESP-BOX-Lite"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_ESP_P4_FUNCTION_EV_BOARD
bool "Espressif ESP-P4-Function-EV-Board"
depends on IDF_TARGET_ESP32P4
config BOARD_TYPE_ECHOEAR
bool "EchoEar"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_KEVIN_BOX_2
bool "Kevin Box 2"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_KEVIN_C3
bool "Kevin C3"
depends on IDF_TARGET_ESP32C3
config BOARD_TYPE_KEVIN_SP_V3_DEV
bool "Kevin SP V3"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_KEVIN_SP_V4_DEV
bool "Kevin SP V4"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_KEVIN_YUYING_313LCD
bool "鱼鹰科技 3.13LCD"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_CGC
bool "CGC"
depends on IDF_TARGET_ESP32
config BOARD_TYPE_CGC_144
bool "CGC 144"
depends on IDF_TARGET_ESP32
config BOARD_TYPE_LICHUANG_DEV_S3
bool "立创·实战派 ESP32-S3"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_LICHUANG_DEV_C3
bool "立创·实战派 ESP32-C3"
depends on IDF_TARGET_ESP32C3
config BOARD_TYPE_DF_K10
bool "DFRobot 行空板 k10"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_DF_S3_AI_CAM
bool "DFRobot ESP32-S3 AI智能摄像头模块"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_MAGICLICK_S3_2P4
bool "神奇按钮 Magiclick_2.4"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_MAGICLICK_S3_2P5
bool "神奇按钮 Magiclick_2.5"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_MAGICLICK_C3
bool "神奇按钮 Magiclick_C3"
depends on IDF_TARGET_ESP32C3
config BOARD_TYPE_MAGICLICK_C3_V2
bool "神奇按钮 Magiclick_C3_v2"
depends on IDF_TARGET_ESP32C3
config BOARD_TYPE_M5STACK_CORE_S3
bool "M5Stack CoreS3"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_M5STACK_CORE_TAB5
bool "M5Stack Tab5"
depends on IDF_TARGET_ESP32P4
config BOARD_TYPE_M5STACK_ATOM_S3_ECHO_BASE
bool "M5Stack AtomS3 + Echo Base"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_M5STACK_ATOM_S3R_ECHO_BASE
bool "M5Stack AtomS3R + Echo Base"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_M5STACK_ATOM_S3R_CAM_M12_ECHO_BASE
bool "M5Stack AtomS3R CAM/M12 + Echo Base"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_M5STACK_ATOM_ECHOS3R
bool "M5Stack AtomEchoS3R"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_M5STACK_ATOM_MATRIX_ECHO_BASE
bool "M5Stack AtomMatrix + Echo Base"
depends on IDF_TARGET_ESP32
config BOARD_TYPE_WAVESHARE_S3_AUDIO_BOARD
bool "Waveshare ESP32-S3-Audio-Board"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_WAVESHARE_S3_TOUCH_AMOLED_1_8
bool "Waveshare ESP32-S3-Touch-AMOLED-1.8"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_WAVESHARE_S3_TOUCH_AMOLED_2_06
bool "Waveshare ESP32-S3-Touch-AMOLED-2.06"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_WAVESHARE_C6_TOUCH_AMOLED_2_06
bool "Waveshare ESP32-C6-Touch-AMOLED-2.06"
depends on IDF_TARGET_ESP32C6
config BOARD_TYPE_WAVESHARE_S3_TOUCH_AMOLED_1_75
bool "Waveshare ESP32-S3-Touch-AMOLED-1.75"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_WAVESHARE_S3_TOUCH_LCD_1_83
bool "Waveshare ESP32-S3-Touch-LCD-1.83"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_WAVESHARE_S3_TOUCH_LCD_4B
bool "Waveshare ESP32-S3-Touch-LCD-4B"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_WAVESHARE_S3_TOUCH_LCD_1_85C
bool "Waveshare ESP32-S3-Touch-LCD-1.85C"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_WAVESHARE_S3_TOUCH_LCD_1_85
bool "Waveshare ESP32-S3-Touch-LCD-1.85"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_WAVESHARE_S3_TOUCH_LCD_1_46
bool "Waveshare ESP32-S3-Touch-LCD-1.46"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_WAVESHARE_C6_LCD_1_69
bool "Waveshare ESP32-C6-LCD-1.69"
depends on IDF_TARGET_ESP32C6
config BOARD_TYPE_WAVESHARE_C6_TOUCH_AMOLED_1_43
bool "Waveshare ESP32-C6-Touch-AMOLOED-1.43"
depends on IDF_TARGET_ESP32C6
config BOARD_TYPE_WAVESHARE_C6_TOUCH_AMOLED_1_32
bool "Waveshare ESP32-C6-Touch-AMOLOED-1.32"
depends on IDF_TARGET_ESP32C6
config BOARD_TYPE_WAVESHARE_S3_TOUCH_AMOLED_1_32
bool "Waveshare ESP32-S3-Touch-AMOLOED-1.32"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_WAVESHARE_S3_TOUCH_LCD_3_49
bool "Waveshare ESP32-S3-Touch-LCD-3.49"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_WAVESHARE_S3_TOUCH_LCD_3_5
bool "Waveshare ESP32-S3-Touch-LCD-3.5"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_WAVESHARE_S3_ePaper_1_54
bool "Waveshare ESP32-S3-ePaper-1.54"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_WAVESHARE_S3_TOUCH_LCD_3_5B
bool "Waveshare ESP32-S3-Touch-LCD-3.5B"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_WAVESHARE_P4_NANO
bool "Waveshare ESP32-P4-NANO"
depends on IDF_TARGET_ESP32P4
config BOARD_TYPE_WAVESHARE_P4_WIFI6_TOUCH_LCD_4B
bool "Waveshare ESP32-P4-WIFI6-Touch-LCD-4B"
depends on IDF_TARGET_ESP32P4
config BOARD_TYPE_WAVESHARE_P4_WIFI6_TOUCH_LCD_7B
bool "Waveshare ESP32-P4-WIFI6-Touch-LCD-7B"
depends on IDF_TARGET_ESP32P4
config BOARD_TYPE_WAVESHARE_P4_WIFI6_TOUCH_LCD_XC
bool "Waveshare ESP32-P4-WIFI6-Touch-LCD-3.4C or ESP32-P4-WIFI6-Touch-LCD-4C"
depends on IDF_TARGET_ESP32P4
config BOARD_TYPE_TUDOUZI
bool "土豆子"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_LILYGO_T_CIRCLE_S3
bool "LILYGO T-Circle-S3"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_LILYGO_T_CAMERAPLUS_S3_V1_0_V1_1
bool "LILYGO T-CameraPlus-S3_V1_0_V1_1"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_LILYGO_T_CAMERAPLUS_S3_V1_2
bool "LILYGO T-CameraPlus-S3_V1_2"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_LILYGO_T_DISPLAY_S3_PRO_MVSRLORA
bool "LILYGO T-Display-S3-Pro-MVSRLora"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_LILYGO_T_DISPLAY_S3_PRO_MVSRLORA_NO_BATTERY
bool "LILYGO T-Display-S3-Pro-MVSRLora_No_Battery"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_LILYGO_T_DISPLAY_P4
bool "LILYGO T-Display-P4"
depends on IDF_TARGET_ESP32P4
config BOARD_TYPE_MOVECALL_MOJI_ESP32S3
bool "Movecall Moji 小智AI衍生版"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_MOVECALL_CUICAN_ESP32S3
bool "Movecall CuiCan 璀璨·AI吊坠"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_ATK_DNESP32S3
bool "正点原子DNESP32S3开发板"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_ATK_DNESP32S3_BOX
bool "正点原子DNESP32S3-BOX"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_ATK_DNESP32S3_BOX0
bool "正点原子DNESP32S3-BOX0"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_ATK_DNESP32S3_BOX2_WIFI
bool "正点原子DNESP32S3-BOX2-WIFI"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_ATK_DNESP32S3_BOX2_4G
bool "正点原子DNESP32S3-BOX2-4G"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_ATK_DNESP32S3M_WIFI
bool "正点原子DNESP32S3M-WIFI"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_ATK_DNESP32S3M_4G
bool "正点原子DNESP32S3M-4G"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_DU_CHATX
bool "嘟嘟开发板CHATX(wifi)"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_TAIJI_PI_S3
bool "太极小派esp32s3"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_XINGZHI_CUBE_0_85TFT_WIFI
bool "无名科技星智0.85(WIFI)"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_XINGZHI_CUBE_0_85TFT_ML307
bool "无名科技星智0.85(ML307)"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_XINGZHI_CUBE_0_96OLED_WIFI
bool "无名科技星智0.96(WIFI)"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_XINGZHI_CUBE_0_96OLED_ML307
bool "无名科技星智0.96(ML307)"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_XINGZHI_CUBE_1_54TFT_WIFI
bool "无名科技星智1.54(WIFI)"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_XINGZHI_CUBE_1_54TFT_ML307
bool "无名科技星智1.54(ML307)"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_XINGZHI_METAL_1_54_WIFI
bool "无名科技星智1.54 METAL(wifi)"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_SEEED_STUDIO_SENSECAP_WATCHER
bool "Seeed Studio SenseCAP Watcher"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_DOIT_S3_AIBOX
bool "四博智联AI陪伴盒子"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_MIXGO_NOVA
bool "元控·青春"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_GENJUTECH_S3_1_54TFT
bool "亘具科技1.54(s3)"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_ESP_S3_LCD_EV_Board
bool "乐鑫ESP S3 LCD EV Board开发板"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_ESP_S3_LCD_EV_Board_2
bool "乐鑫ESP S3 LCD EV Board 2开发板"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_ZHENGCHEN_1_54TFT_WIFI
bool "征辰科技1.54(WIFI)"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_ZHENGCHEN_1_54TFT_ML307
bool "征辰科技1.54(ML307)"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_MINSI_K08_DUAL
bool "敏思科技K08(DUAL)"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_SPOTPEAR_ESP32_S3_1_54_MUMA
bool "Spotpear ESP32-S3-1.54-MUMA"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_SPOTPEAR_ESP32_S3_1_28_BOX
bool "Spotpear ESP32-S3-1.28-BOX"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_OTTO_ROBOT
bool "ottoRobot"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_ELECTRON_BOT
bool "electronBot"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_JIUCHUAN
bool "九川智能"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_LABPLUS_MPYTHON_V3
bool "labplus mpython_v3 board"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_LABPLUS_LEDONG_V2
bool "labplus ledong_v2 board"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_SURFER_C3_1_14TFT
bool "Surfer-C3-1.14TFT"
depends on IDF_TARGET_ESP32C3
config BOARD_TYPE_YUNLIAO_S3
bool "小智云聊-S3"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_WIRELESS_TAG_WTP4C5MP07S
bool "Wireless-Tag WTP4C5MP07S"
depends on IDF_TARGET_ESP32P4
config BOARD_TYPE_AIPI_LITE
bool "AIPI-Lite"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_HU_087
bool "HU-087"
depends on IDF_TARGET_ESP32S3
endchoice
choice
depends on BOARD_TYPE_LILYGO_T_DISPLAY_P4
prompt "Select the screen type"
default SCREEN_TYPE_HI8561
config SCREEN_TYPE_HI8561
bool "HI8561"
config SCREEN_TYPE_RM69A10
bool "RM69A10"
endchoice
choice
depends on BOARD_TYPE_LILYGO_T_DISPLAY_P4
prompt "Select the color format of the screen"
default SCREEN_PIXEL_FORMAT_RGB565
config SCREEN_PIXEL_FORMAT_RGB565
bool "RGB565"
config SCREEN_PIXEL_FORMAT_RGB888
bool "RGB888"
endchoice
choice ESP_S3_LCD_EV_Board_Version_TYPE
depends on BOARD_TYPE_ESP_S3_LCD_EV_Board
prompt "EV_BOARD Type"
default ESP_S3_LCD_EV_Board_1p4
config ESP_S3_LCD_EV_Board_1p4
bool "乐鑫ESP32_S3_LCD_EV_Board-MB_V1.4"
config ESP_S3_LCD_EV_Board_1p5
bool "乐鑫ESP32_S3_LCD_EV_Board-MB_V1.5"
endchoice
choice DISPLAY_OLED_TYPE
depends on BOARD_TYPE_BREAD_COMPACT_WIFI || BOARD_TYPE_BREAD_COMPACT_ML307 || BOARD_TYPE_BREAD_COMPACT_ESP32 || BOARD_TYPE_HU_087
prompt "OLED Type"
default OLED_SSD1306_128X32
help
OLED Monochrome Display Type
config OLED_SSD1306_128X32
bool "SSD1306 128*32"
config OLED_SSD1306_128X64
bool "SSD1306 128*64"
config OLED_SH1106_128X64
bool "SH1106 128*64"
endchoice
choice DISPLAY_LCD_TYPE
depends on BOARD_TYPE_BREAD_COMPACT_WIFI_LCD || BOARD_TYPE_BREAD_COMPACT_ESP32_LCD || BOARD_TYPE_CGC || BOARD_TYPE_WAVESHARE_P4_NANO || BOARD_TYPE_WAVESHARE_P4_WIFI6_TOUCH_LCD_XC || BOARD_TYPE_BREAD_COMPACT_WIFI_CAM
prompt "LCD Type"
default LCD_ST7789_240X320
help
LCD Display Type
config LCD_ST7789_240X320
bool "ST7789 240*320, IPS"
config LCD_ST7789_240X320_NO_IPS
bool "ST7789 240*320, Non-IPS"
config LCD_ST7789_170X320
bool "ST7789 170*320"
config LCD_ST7789_172X320
bool "ST7789 172*320"
config LCD_ST7789_240X280
bool "ST7789 240*280"
config LCD_ST7789_240X240
bool "ST7789 240*240"
config LCD_ST7789_240X240_7PIN
bool "ST7789 240*240, 7PIN"
config LCD_ST7789_240X135
bool "ST7789 240*135"
config LCD_ST7735_128X160
bool "ST7735 128*160"
config LCD_ST7735_128X128
bool "ST7735 128*128"
config LCD_ST7796_320X480
bool "ST7796 320*480 IPS"
config LCD_ST7796_320X480_NO_IPS
bool "ST7796 320*480, Non-IPS"
config LCD_ILI9341_240X320
bool "ILI9341 240*320"
config LCD_ILI9341_240X320_NO_IPS
bool "ILI9341 240*320, Non-IPS"
config LCD_GC9A01_240X240
bool "GC9A01 240*240 Circle"
config LCD_TYPE_800_1280_10_1_INCH
bool "Waveshare 101M-8001280-IPS-CT-K Display"
config LCD_TYPE_800_1280_10_1_INCH_A
bool "Waveshare 10.1-DSI-TOUCH-A Display"
config LCD_TYPE_800_800_3_4_INCH
bool "Waveshare ESP32-P4-WIFI6-Touch-LCD-3.4C with 800*800 3.4inch round display"
config LCD_TYPE_720_720_4_INCH
bool "Waveshare ESP32-P4-WIFI6-Touch-LCD-4C with 720*720 4inch round display"
config LCD_CUSTOM
bool "Custom LCD (自定义屏幕参数)"
endchoice
choice DISPLAY_ESP32S3_KORVO2_V3
depends on BOARD_TYPE_ESP_KORVO2_V3
prompt "ESP32S3_KORVO2_V3 LCD Type"
default ESP32S3_KORVO2_V3_LCD_ST7789
help
LCD Display Type
config ESP32S3_KORVO2_V3_LCD_ST7789
bool "ST7789 240*280"
config ESP32S3_KORVO2_V3_LCD_ILI9341
bool "ILI9341 240*320"
endchoice
choice DISPLAY_ESP32S3_AUDIO_BOARD
depends on BOARD_TYPE_WAVESHARE_S3_AUDIO_BOARD
prompt "ESP32S3_AUDIO_BOARD LCD Type"
default AUDIO_BOARD_LCD_JD9853
help
LCD Display Type
config AUDIO_BOARD_LCD_JD9853
bool "JD9853 320*172"
config AUDIO_BOARD_LCD_ST7789
bool "ST7789 240*320"
endchoice
choice DISPLAY_STYLE
prompt "Select display style"
default USE_DEFAULT_MESSAGE_STYLE
help
Select display style for Xiaozhi device
config USE_DEFAULT_MESSAGE_STYLE
bool "Enable default message style"
config USE_WECHAT_MESSAGE_STYLE
bool "Enable WeChat Message Style"
config USE_EMOTE_MESSAGE_STYLE
bool "Emote animation style"
depends on BOARD_TYPE_ESP_BOX_3 || BOARD_TYPE_ECHOEAR || BOARD_TYPE_LICHUANG_DEV_S3
endchoice
choice WAKE_WORD_TYPE
prompt "Wake Word Implementation Type"
default USE_AFE_WAKE_WORD if (IDF_TARGET_ESP32S3 || IDF_TARGET_ESP32P4) && SPIRAM
default WAKE_WORD_DISABLED
help
Choose the type of wake word implementation to use
config WAKE_WORD_DISABLED
bool "Disabled"
help
Disable wake word detection
config USE_ESP_WAKE_WORD
bool "Wakenet model without AFE"
depends on IDF_TARGET_ESP32C3 || IDF_TARGET_ESP32C5 || IDF_TARGET_ESP32C6 || (IDF_TARGET_ESP32 && SPIRAM)
help
Support ESP32 C3、ESP32 C5 与 ESP32 C6, and (ESP32 with PSRAM)
config USE_AFE_WAKE_WORD
bool "Wakenet model with AFE"
depends on (IDF_TARGET_ESP32S3 || IDF_TARGET_ESP32P4) && SPIRAM
help
Support AEC if available, requires ESP32 S3 and PSRAM
config USE_CUSTOM_WAKE_WORD
bool "Multinet model (Custom Wake Word)"
depends on (IDF_TARGET_ESP32S3 || IDF_TARGET_ESP32P4) && SPIRAM
help
Requires ESP32 S3 and PSRAM
endchoice
config CUSTOM_WAKE_WORD
string "Custom Wake Word"
default "xiao tu dou"
depends on USE_CUSTOM_WAKE_WORD
help
Custom Wake Word, use pinyin for Chinese, separated by spaces
config CUSTOM_WAKE_WORD_DISPLAY
string "Custom Wake Word Display"
default "小土豆"
depends on USE_CUSTOM_WAKE_WORD
help
Greeting sent to the server after wake word detection
config CUSTOM_WAKE_WORD_THRESHOLD
int "Custom Wake Word Threshold (%)"
default 20
range 1 99
depends on USE_CUSTOM_WAKE_WORD
help
Custom Wake Word Threshold, range 1-99, the smaller the more sensitive, default 20
config SEND_WAKE_WORD_DATA
bool "Send Wake Word Data"
default y
depends on USE_AFE_WAKE_WORD || USE_CUSTOM_WAKE_WORD
help
Send wake word data to the server as the first message of the conversation and wait for response
config USE_AUDIO_PROCESSOR
bool "Enable Audio Noise Reduction"
default y
depends on (IDF_TARGET_ESP32S3 || IDF_TARGET_ESP32P4) && SPIRAM
help
Requires ESP32 S3 and PSRAM
config USE_DEVICE_AEC
bool "Enable Device-Side AEC"
default n
depends on USE_AUDIO_PROCESSOR && (BOARD_TYPE_ESP_BOX_3 || BOARD_TYPE_ESP_BOX || BOARD_TYPE_ESP_BOX_LITE \
|| BOARD_TYPE_LICHUANG_DEV_S3 || BOARD_TYPE_ESP_KORVO2_V3 || BOARD_TYPE_WAVESHARE_S3_TOUCH_AMOLED_1_75 || BOARD_TYPE_WAVESHARE_S3_TOUCH_LCD_1_83\
|| BOARD_TYPE_WAVESHARE_S3_TOUCH_AMOLED_2_06 || BOARD_TYPE_WAVESHARE_S3_TOUCH_LCD_4B || BOARD_TYPE_WAVESHARE_P4_WIFI6_TOUCH_LCD_4B || BOARD_TYPE_WAVESHARE_P4_WIFI6_TOUCH_LCD_7B \
|| BOARD_TYPE_WAVESHARE_P4_WIFI6_TOUCH_LCD_XC || BOARD_TYPE_ESP_S3_LCD_EV_Board_2 || BOARD_TYPE_YUNLIAO_S3 \
|| BOARD_TYPE_ECHOEAR || BOARD_TYPE_WAVESHARE_S3_TOUCH_LCD_3_49)
help
To work properly, device-side AEC requires a clean output reference path from the speaker signal and physical acoustic isolation between the microphone and speaker.
config USE_SERVER_AEC
bool "Enable Server-Side AEC (Unstable)"
default n
depends on USE_AUDIO_PROCESSOR
help
To work perperly, server-side AEC requires server support
config USE_AUDIO_DEBUGGER
bool "Enable Audio Debugger"
default n
help
Enable audio debugger, send audio data through UDP to the host machine
config AUDIO_DEBUG_UDP_SERVER
string "Audio Debug UDP Server Address"
default "192.168.2.100:8000"
depends on USE_AUDIO_DEBUGGER
help
UDP server address, format: IP:PORT, used to receive audio debugging data
config USE_ACOUSTIC_WIFI_PROVISIONING
bool "Enable Acoustic WiFi Provisioning"
default n
help
Enable acoustic WiFi provisioning, use audio signal to transmit WiFi configuration data
config RECEIVE_CUSTOM_MESSAGE
bool "Enable Custom Message Reception"
default n
help
Enable custom message reception, allow the device to receive custom messages from the server (preferably through the MQTT protocol)
menu "Camera Configuration"
depends on !IDF_TARGET_ESP32
comment "Warning: Please read the help text before modifying these settings."
config XIAOZHI_CAMERA_ALLOW_JPEG_INPUT
bool "Allow JPEG Input"
default n
help
Allow JPEG Input format for the camera.
This option may need to be enabled when using a USB camera.
Not currently supported when used simultaneously with XIAOZHI_ENABLE_ROTATE_CAMERA_IMAGE.
config XIAOZHI_ENABLE_HARDWARE_JPEG_ENCODER
bool "Enable Hardware JPEG Encoder"
default y
depends on SOC_JPEG_ENCODE_SUPPORTED
help
Use hardware JPEG encoder on ESP32-P4 to encode image to JPEG.
See https://docs.espressif.com/projects/esp-idf/en/stable/esp32p4/api-reference/peripherals/jpeg.html for more details.
config XIAOZHI_ENABLE_HARDWARE_JPEG_DECODER
bool "Enable Hardware JPEG Decoder"
default n
depends on SOC_JPEG_DECODE_SUPPORTED && XIAOZHI_CAMERA_ALLOW_JPEG_INPUT
help
Use hardware JPEG decoder on ESP32-P4 to decode JPEG to image.
See https://docs.espressif.com/projects/esp-idf/en/stable/esp32p4/api-reference/peripherals/jpeg.html for more details.
config XIAOZHI_ENABLE_CAMERA_DEBUG_MODE
bool "Enable Camera Debug Mode"
default n
help
Enable camera debug mode, print camera debug information to the console.
Only works on boards that support camera.
config XIAOZHI_ENABLE_CAMERA_ENDIANNESS_SWAP
bool "Enable software camera buffer endianness swapping"
default n
depends on !CAMERA_SENSOR_SWAP_PIXEL_BYTE_ORDER
help
This option treats the camera buffer as a uint16_t[] array and performs byte-swapping (endianness conversion) on each element.
Should only be modified by development board integration engineers.
**Incorrect usage may result in incorrect image colors!**
ATTENTION: If the option CAMERA_SENSOR_SWAP_PIXEL_BYTE_ORDER is available for your sensor, please use that instead.
menuconfig XIAOZHI_ENABLE_ROTATE_CAMERA_IMAGE
bool "Enable Camera Image Rotation"
default n
depends on !XIAOZHI_CAMERA_ALLOW_JPEG_INPUT
help
Enable camera image rotation, rotate the camera image to the correct orientation.
- On ESP32-P4, rotation is handled by PPA hardware.
- On other chips, rotation is done in software with performance cost.
- For 180° rotation, use HFlip + VFlip instead of this option.
Not currently supported when used simultaneously with XIAOZHI_CAMERA_ALLOW_JPEG_INPUT.
if XIAOZHI_ENABLE_ROTATE_CAMERA_IMAGE
choice XIAOZHI_CAMERA_IMAGE_ROTATION_ANGLE
prompt "Camera Image Rotation Angle (clockwise)"
default XIAOZHI_CAMERA_IMAGE_ROTATION_ANGLE_90
help
Camera image rotation angle.
config XIAOZHI_CAMERA_IMAGE_ROTATION_ANGLE_90
bool "90°"
config XIAOZHI_CAMERA_IMAGE_ROTATION_ANGLE_270
bool "270°"
comment "For 180° rotation, use HFlip + VFlip instead of this option"
endchoice
endif
endmenu
menu "TAIJIPAI_S3_CONFIG"
depends on BOARD_TYPE_TAIJI_PI_S3
choice I2S_TYPE_TAIJIPI_S3
prompt "taiji-pi-S3 I2S Type"
default TAIJIPAI_I2S_TYPE_STD
help
I2S 类型选择
config TAIJIPAI_I2S_TYPE_STD
bool "I2S Type STD"
config TAIJIPAI_I2S_TYPE_PDM
bool "I2S Type PDM"
endchoice
config I2S_USE_2SLOT
bool "Enable I2S 2 Slot"
default y
help
启动双声道
endmenu
endmenu

910
main/application.cc Normal file
View File

@ -0,0 +1,910 @@
#include "application.h"
#include "board.h"
#include "display.h"
#include "system_info.h"
#include "audio_codec.h"
#include "mqtt_protocol.h"
#include "websocket_protocol.h"
#include "assets/lang_config.h"
#include "mcp_server.h"
#include "assets.h"
#include "settings.h"
#include <cstring>
#include <esp_log.h>
#include <cJSON.h>
#include <driver/gpio.h>
#include <arpa/inet.h>
#include <font_awesome.h>
#include <uart_component.h>
#define TAG "Application"
// ESP32 设备状态字符串，用于日志输出
// 其中 "idle"、"listening"、"speaking" 会通过 UART 发送给 RP2040 驱动舵机动画
static const char* const STATE_STRINGS[] = {
"unknown",
"starting",
"configuring",
"idle", // → RP2040 执行 state_sleep（闭眼、耳朵收起）
"connecting",
"listening", // → RP2040 执行 state_listen（好奇姿态、眼球随机看）
"speaking", // → RP2040 执行 state_speaking（嘴巴开合、眼球随机看、身体摆动）
"upgrading",
"activating",
"audio_testing",
"fatal_error",
"invalid_state"
};
Application::Application() {
event_group_ = xEventGroupCreate();
#if CONFIG_USE_DEVICE_AEC && CONFIG_USE_SERVER_AEC
#error "CONFIG_USE_DEVICE_AEC and CONFIG_USE_SERVER_AEC cannot be enabled at the same time"
#elif CONFIG_USE_DEVICE_AEC
aec_mode_ = kAecOnDeviceSide;
#elif CONFIG_USE_SERVER_AEC
aec_mode_ = kAecOnServerSide;
#else
aec_mode_ = kAecOff;
#endif
esp_timer_create_args_t clock_timer_args = {
.callback = [](void* arg) {
Application* app = (Application*)arg;
xEventGroupSetBits(app->event_group_, MAIN_EVENT_CLOCK_TICK);
},
.arg = this,
.dispatch_method = ESP_TIMER_TASK,
.name = "clock_timer",
.skip_unhandled_events = true
};
esp_timer_create(&clock_timer_args, &clock_timer_handle_);
}
Application::~Application() {
if (clock_timer_handle_ != nullptr) {
esp_timer_stop(clock_timer_handle_);
esp_timer_delete(clock_timer_handle_);
}
vEventGroupDelete(event_group_);
}
void Application::CheckAssetsVersion() {
auto& board = Board::GetInstance();
auto display = board.GetDisplay();
auto& assets = Assets::GetInstance();
if (!assets.partition_valid()) {
ESP_LOGW(TAG, "Assets partition is disabled for board %s", BOARD_NAME);
return;
}
Settings settings("assets", true);
// Check if there is a new assets need to be downloaded
std::string download_url = settings.GetString("download_url");
if (!download_url.empty()) {
settings.EraseKey("download_url");
char message[256];
snprintf(message, sizeof(message), Lang::Strings::FOUND_NEW_ASSETS, download_url.c_str());
Alert(Lang::Strings::LOADING_ASSETS, message, "cloud_arrow_down", Lang::Sounds::OGG_UPGRADE);
// Wait for the audio service to be idle for 3 seconds
vTaskDelay(pdMS_TO_TICKS(3000));
SetDeviceState(kDeviceStateUpgrading);
board.SetPowerSaveMode(false);
display->SetChatMessage("system", Lang::Strings::PLEASE_WAIT);
bool success = assets.Download(download_url, [display](int progress, size_t speed) -> void {
std::thread([display, progress, speed]() {
char buffer[32];
snprintf(buffer, sizeof(buffer), "%d%% %uKB/s", progress, speed / 1024);
display->SetChatMessage("system", buffer);
}).detach();
});
board.SetPowerSaveMode(true);
vTaskDelay(pdMS_TO_TICKS(1000));
if (!success) {
Alert(Lang::Strings::ERROR, Lang::Strings::DOWNLOAD_ASSETS_FAILED, "circle_xmark", Lang::Sounds::OGG_EXCLAMATION);
vTaskDelay(pdMS_TO_TICKS(2000));
return;
}
}
// Apply assets
assets.Apply();
display->SetChatMessage("system", "");
display->SetEmotion("microchip_ai");
uart_send_string("configuring");
}
void Application::CheckNewVersion(Ota& ota) {
const int MAX_RETRY = 10;
int retry_count = 0;
int retry_delay = 10; // 初始重试延迟为10秒
auto& board = Board::GetInstance();
while (true) {
SetDeviceState(kDeviceStateActivating);
auto display = board.GetDisplay();
display->SetStatus(Lang::Strings::CHECKING_NEW_VERSION);
esp_err_t err = ota.CheckVersion();
if (err != ESP_OK) {
retry_count++;
if (retry_count >= MAX_RETRY) {
ESP_LOGE(TAG, "Too many retries, exit version check");
return;
}
char error_message[128];
snprintf(error_message, sizeof(error_message), "code=%d, url=%s", err, ota.GetCheckVersionUrl().c_str());
char buffer[256];
snprintf(buffer, sizeof(buffer), Lang::Strings::CHECK_NEW_VERSION_FAILED, retry_delay, error_message);
Alert(Lang::Strings::ERROR, buffer, "cloud_slash", Lang::Sounds::OGG_EXCLAMATION);
ESP_LOGW(TAG, "Check new version failed, retry in %d seconds (%d/%d)", retry_delay, retry_count, MAX_RETRY);
for (int i = 0; i < retry_delay; i++) {
vTaskDelay(pdMS_TO_TICKS(1000));
if (device_state_ == kDeviceStateIdle) {
break;
}
}
retry_delay *= 2; // 每次重试后延迟时间翻倍
continue;
}
retry_count = 0;
retry_delay = 10; // 重置重试延迟时间
if (ota.HasNewVersion()) {
if (UpgradeFirmware(ota)) {
return; // This line will never be reached after reboot
}
// If upgrade failed, continue to normal operation (don't break, just fall through)
}
// No new version, mark the current version as valid
ota.MarkCurrentVersionValid();
if (!ota.HasActivationCode() && !ota.HasActivationChallenge()) {
xEventGroupSetBits(event_group_, MAIN_EVENT_CHECK_NEW_VERSION_DONE);
// Exit the loop if done checking new version
break;
}
display->SetStatus(Lang::Strings::ACTIVATION);
// Activation code is shown to the user and waiting for the user to input
if (ota.HasActivationCode()) {
ShowActivationCode(ota.GetActivationCode(), ota.GetActivationMessage());
}
// This will block the loop until the activation is done or timeout
for (int i = 0; i < 10; ++i) {
ESP_LOGI(TAG, "Activating... %d/%d", i + 1, 10);
esp_err_t err = ota.Activate();
if (err == ESP_OK) {
xEventGroupSetBits(event_group_, MAIN_EVENT_CHECK_NEW_VERSION_DONE);
break;
} else if (err == ESP_ERR_TIMEOUT) {
vTaskDelay(pdMS_TO_TICKS(3000));
} else {
vTaskDelay(pdMS_TO_TICKS(10000));
}
if (device_state_ == kDeviceStateIdle) {
break;
}
}
}
}
void Application::ShowActivationCode(const std::string& code, const std::string& message) {
struct digit_sound {
char digit;
const std::string_view& sound;
};
static const std::array<digit_sound, 10> digit_sounds{{
digit_sound{'0', Lang::Sounds::OGG_0},
digit_sound{'1', Lang::Sounds::OGG_1},
digit_sound{'2', Lang::Sounds::OGG_2},
digit_sound{'3', Lang::Sounds::OGG_3},
digit_sound{'4', Lang::Sounds::OGG_4},
digit_sound{'5', Lang::Sounds::OGG_5},
digit_sound{'6', Lang::Sounds::OGG_6},
digit_sound{'7', Lang::Sounds::OGG_7},
digit_sound{'8', Lang::Sounds::OGG_8},
digit_sound{'9', Lang::Sounds::OGG_9}
}};
// This sentence uses 9KB of SRAM, so we need to wait for it to finish
Alert(Lang::Strings::ACTIVATION, message.c_str(), "link", Lang::Sounds::OGG_ACTIVATION);
for (const auto& digit : code) {
auto it = std::find_if(digit_sounds.begin(), digit_sounds.end(),
[digit](const digit_sound& ds) { return ds.digit == digit; });
if (it != digit_sounds.end()) {
audio_service_.PlaySound(it->sound);
}
}
}
void Application::Alert(const char* status, const char* message, const char* emotion, const std::string_view& sound) {
ESP_LOGW(TAG, "Alert [%s] %s: %s", emotion, status, message);
auto display = Board::GetInstance().GetDisplay();
display->SetStatus(status);
display->SetEmotion(emotion);
// Alert 时将情绪字符串（如 "happy"、"neutral"）发送给 RP2040 触发对应舵机动画
uart_send_string(emotion);
display->SetChatMessage("system", message);
if (!sound.empty()) {
audio_service_.PlaySound(sound);
}
}
void Application::DismissAlert() {
if (device_state_ == kDeviceStateIdle) {
auto display = Board::GetInstance().GetDisplay();
display->SetStatus(Lang::Strings::STANDBY);
display->SetEmotion("neutral");
uart_send_string("idle");
display->SetChatMessage("system", "");
}
}
void Application::ToggleChatState() {
if (device_state_ == kDeviceStateActivating) {
SetDeviceState(kDeviceStateIdle);
return;
} else if (device_state_ == kDeviceStateWifiConfiguring) {
audio_service_.EnableAudioTesting(true);
SetDeviceState(kDeviceStateAudioTesting);
return;
} else if (device_state_ == kDeviceStateAudioTesting) {
audio_service_.EnableAudioTesting(false);
SetDeviceState(kDeviceStateWifiConfiguring);
return;
}
if (!protocol_) {
ESP_LOGE(TAG, "Protocol not initialized");
return;
}
if (device_state_ == kDeviceStateIdle) {
Schedule([this]() {
if (!protocol_->IsAudioChannelOpened()) {
SetDeviceState(kDeviceStateConnecting);
if (!protocol_->OpenAudioChannel()) {
return;
}
}
SetListeningMode(aec_mode_ == kAecOff ? kListeningModeAutoStop : kListeningModeRealtime);
});
} else if (device_state_ == kDeviceStateSpeaking) {
Schedule([this]() {
AbortSpeaking(kAbortReasonNone);
});
} else if (device_state_ == kDeviceStateListening) {
Schedule([this]() {
protocol_->CloseAudioChannel();
});
}
}
void Application::StartListening() {
if (device_state_ == kDeviceStateActivating) {
SetDeviceState(kDeviceStateIdle);
return;
} else if (device_state_ == kDeviceStateWifiConfiguring) {
audio_service_.EnableAudioTesting(true);
SetDeviceState(kDeviceStateAudioTesting);
return;
}
if (!protocol_) {
ESP_LOGE(TAG, "Protocol not initialized");
return;
}
if (device_state_ == kDeviceStateIdle) {
Schedule([this]() {
if (!protocol_->IsAudioChannelOpened()) {
SetDeviceState(kDeviceStateConnecting);
if (!protocol_->OpenAudioChannel()) {
return;
}
}
SetListeningMode(kListeningModeManualStop);
});
} else if (device_state_ == kDeviceStateSpeaking) {
Schedule([this]() {
AbortSpeaking(kAbortReasonNone);
SetListeningMode(kListeningModeManualStop);
});
}
}
void Application::StopListening() {
if (device_state_ == kDeviceStateAudioTesting) {
audio_service_.EnableAudioTesting(false);
SetDeviceState(kDeviceStateWifiConfiguring);
return;
}
const std::array<int, 3> valid_states = {
kDeviceStateListening,
kDeviceStateSpeaking,
kDeviceStateIdle,
};
// If not valid, do nothing
if (std::find(valid_states.begin(), valid_states.end(), device_state_) == valid_states.end()) {
return;
}
Schedule([this]() {
if (device_state_ == kDeviceStateListening) {
protocol_->SendStopListening();
SetDeviceState(kDeviceStateIdle);
}
});
}
void Application::Start() {
auto& board = Board::GetInstance();
SetDeviceState(kDeviceStateStarting);
/* Setup the display */
auto display = board.GetDisplay();
// Print board name/version info
display->SetChatMessage("system", SystemInfo::GetUserAgent().c_str());
/* Setup the audio service */
auto codec = board.GetAudioCodec();
audio_service_.Initialize(codec);
audio_service_.Start();
AudioServiceCallbacks callbacks;
callbacks.on_send_queue_available = [this]() {
xEventGroupSetBits(event_group_, MAIN_EVENT_SEND_AUDIO);
};
callbacks.on_wake_word_detected = [this](const std::string& wake_word) {
xEventGroupSetBits(event_group_, MAIN_EVENT_WAKE_WORD_DETECTED);
};
callbacks.on_vad_change = [this](bool speaking) {
xEventGroupSetBits(event_group_, MAIN_EVENT_VAD_CHANGE);
};
audio_service_.SetCallbacks(callbacks);
// Start the main event loop task with priority 3
xTaskCreate([](void* arg) {
((Application*)arg)->MainEventLoop();
vTaskDelete(NULL);
}, "main_event_loop", 2048 * 4, this, 3, &main_event_loop_task_handle_);
/* Start the clock timer to update the status bar */
esp_timer_start_periodic(clock_timer_handle_, 1000000);
/* Wait for the network to be ready */
board.StartNetwork();
// Update the status bar immediately to show the network state
display->UpdateStatusBar(true);
// Check for new assets version
CheckAssetsVersion();
// Check for new firmware version or get the MQTT broker address
Ota ota;
CheckNewVersion(ota);
// Initialize the protocol
display->SetStatus(Lang::Strings::LOADING_PROTOCOL);
// Add MCP common tools before initializing the protocol
auto& mcp_server = McpServer::GetInstance();
mcp_server.AddCommonTools();
mcp_server.AddUserOnlyTools();
if (ota.HasMqttConfig()) {
protocol_ = std::make_unique<MqttProtocol>();
} else if (ota.HasWebsocketConfig()) {
protocol_ = std::make_unique<WebsocketProtocol>();
} else {
ESP_LOGW(TAG, "No protocol specified in the OTA config, using MQTT");
protocol_ = std::make_unique<MqttProtocol>();
}
protocol_->OnConnected([this]() {
DismissAlert();
});
protocol_->OnNetworkError([this](const std::string& message) {
last_error_message_ = message;
xEventGroupSetBits(event_group_, MAIN_EVENT_ERROR);
});
protocol_->OnIncomingAudio([this](std::unique_ptr<AudioStreamPacket> packet) {
if (device_state_ == kDeviceStateSpeaking) {
audio_service_.PushPacketToDecodeQueue(std::move(packet));
}
});
protocol_->OnAudioChannelOpened([this, codec, &board]() {
board.SetPowerSaveMode(false);
if (protocol_->server_sample_rate() != codec->output_sample_rate()) {
ESP_LOGW(TAG, "Server sample rate %d does not match device output sample rate %d, resampling may cause distortion",
protocol_->server_sample_rate(), codec->output_sample_rate());
}
});
protocol_->OnAudioChannelClosed([this, &board]() {
board.SetPowerSaveMode(true);
Schedule([this]() {
auto display = Board::GetInstance().GetDisplay();
display->SetChatMessage("system", "");
SetDeviceState(kDeviceStateIdle);
});
});
protocol_->OnIncomingJson([this, display](const cJSON* root) {
// Parse JSON data
auto type = cJSON_GetObjectItem(root, "type");
if (strcmp(type->valuestring, "tts") == 0) {
auto state = cJSON_GetObjectItem(root, "state");
if (strcmp(state->valuestring, "start") == 0) {
Schedule([this]() {
aborted_ = false;
if (device_state_ == kDeviceStateIdle || device_state_ == kDeviceStateListening) {
SetDeviceState(kDeviceStateSpeaking);
}
});
} else if (strcmp(state->valuestring, "stop") == 0) {
Schedule([this]() {
if (device_state_ == kDeviceStateSpeaking) {
if (listening_mode_ == kListeningModeManualStop) {
SetDeviceState(kDeviceStateIdle);
} else {
SetDeviceState(kDeviceStateListening);
}
}
});
} else if (strcmp(state->valuestring, "sentence_start") == 0) {
auto text = cJSON_GetObjectItem(root, "text");
if (cJSON_IsString(text)) {
ESP_LOGI(TAG, "<< %s", text->valuestring);
Schedule([this, display, message = std::string(text->valuestring)]() {
display->SetChatMessage("assistant", message.c_str());
});
}
}
} else if (strcmp(type->valuestring, "stt") == 0) {
auto text = cJSON_GetObjectItem(root, "text");
if (cJSON_IsString(text)) {
ESP_LOGI(TAG, ">> %s", text->valuestring);
Schedule([this, display, message = std::string(text->valuestring)]() {
display->SetChatMessage("user", message.c_str());
});
}
} else if (strcmp(type->valuestring, "llm") == 0) {
// LLM 回复中携带的情绪标签（如 "happy"、"thinking"）
auto emotion = cJSON_GetObjectItem(root, "emotion");
if (cJSON_IsString(emotion)) {
Schedule([this, display, emotion_str = std::string(emotion->valuestring)]() {
display->SetEmotion(emotion_str.c_str());
// 将 AI 返回的情绪标签发送给 RP2040，实时切换舵机动画表情
uart_send_string(emotion_str.c_str());
});
}
} else if (strcmp(type->valuestring, "mcp") == 0) {
auto payload = cJSON_GetObjectItem(root, "payload");
if (cJSON_IsObject(payload)) {
McpServer::GetInstance().ParseMessage(payload);
}
} else if (strcmp(type->valuestring, "system") == 0) {
auto command = cJSON_GetObjectItem(root, "command");
if (cJSON_IsString(command)) {
ESP_LOGI(TAG, "System command: %s", command->valuestring);
if (strcmp(command->valuestring, "reboot") == 0) {
// Do a reboot if user requests a OTA update
Schedule([this]() {
Reboot();
});
} else {
ESP_LOGW(TAG, "Unknown system command: %s", command->valuestring);
}
}
} else if (strcmp(type->valuestring, "alert") == 0) {
auto status = cJSON_GetObjectItem(root, "status");
auto message = cJSON_GetObjectItem(root, "message");
auto emotion = cJSON_GetObjectItem(root, "emotion");
if (cJSON_IsString(status) && cJSON_IsString(message) && cJSON_IsString(emotion)) {
Alert(status->valuestring, message->valuestring, emotion->valuestring, Lang::Sounds::OGG_VIBRATION);
} else {
ESP_LOGW(TAG, "Alert command requires status, message and emotion");
}
#if CONFIG_RECEIVE_CUSTOM_MESSAGE
} else if (strcmp(type->valuestring, "custom") == 0) {
auto payload = cJSON_GetObjectItem(root, "payload");
ESP_LOGI(TAG, "Received custom message: %s", cJSON_PrintUnformatted(root));
if (cJSON_IsObject(payload)) {
Schedule([this, display, payload_str = std::string(cJSON_PrintUnformatted(payload))]() {
display->SetChatMessage("system", payload_str.c_str());
});
} else {
ESP_LOGW(TAG, "Invalid custom message format: missing payload");
}
#endif
} else {
ESP_LOGW(TAG, "Unknown message type: %s", type->valuestring);
}
});
bool protocol_started = protocol_->Start();
SystemInfo::PrintHeapStats();
SetDeviceState(kDeviceStateIdle);
has_server_time_ = ota.HasServerTime();
if (protocol_started) {
std::string message = std::string(Lang::Strings::VERSION) + ota.GetCurrentVersion();
display->ShowNotification(message.c_str());
display->SetChatMessage("system", "");
// Play the success sound to indicate the device is ready
audio_service_.PlaySound(Lang::Sounds::OGG_SUCCESS);
}
}
// Add a async task to MainLoop
void Application::Schedule(std::function<void()> callback) {
{
std::lock_guard<std::mutex> lock(mutex_);
main_tasks_.push_back(std::move(callback));
}
xEventGroupSetBits(event_group_, MAIN_EVENT_SCHEDULE);
}
// The Main Event Loop controls the chat state and websocket connection
// If other tasks need to access the websocket or chat state,
// they should use Schedule to call this function
void Application::MainEventLoop() {
while (true) {
auto bits = xEventGroupWaitBits(event_group_, MAIN_EVENT_SCHEDULE |
MAIN_EVENT_SEND_AUDIO |
MAIN_EVENT_WAKE_WORD_DETECTED |
MAIN_EVENT_VAD_CHANGE |
MAIN_EVENT_CLOCK_TICK |
MAIN_EVENT_ERROR, pdTRUE, pdFALSE, portMAX_DELAY);
if (bits & MAIN_EVENT_ERROR) {
SetDeviceState(kDeviceStateIdle);
Alert(Lang::Strings::ERROR, last_error_message_.c_str(), "circle_xmark", Lang::Sounds::OGG_EXCLAMATION);
}
if (bits & MAIN_EVENT_SEND_AUDIO) {
while (auto packet = audio_service_.PopPacketFromSendQueue()) {
if (protocol_ && !protocol_->SendAudio(std::move(packet))) {
break;
}
}
}
if (bits & MAIN_EVENT_WAKE_WORD_DETECTED) {
OnWakeWordDetected();
}
if (bits & MAIN_EVENT_VAD_CHANGE) {
if (device_state_ == kDeviceStateListening) {
auto led = Board::GetInstance().GetLed();
led->OnStateChanged();
}
}
if (bits & MAIN_EVENT_SCHEDULE) {
std::unique_lock<std::mutex> lock(mutex_);
auto tasks = std::move(main_tasks_);
lock.unlock();
for (auto& task : tasks) {
task();
}
}
if (bits & MAIN_EVENT_CLOCK_TICK) {
clock_ticks_++;
auto display = Board::GetInstance().GetDisplay();
display->UpdateStatusBar();
// Print the debug info every 10 seconds
if (clock_ticks_ % 10 == 0) {
// SystemInfo::PrintTaskCpuUsage(pdMS_TO_TICKS(1000));
// SystemInfo::PrintTaskList();
SystemInfo::PrintHeapStats();
}
}
}
}
void Application::OnWakeWordDetected() {
if (!protocol_) {
return;
}
if (device_state_ == kDeviceStateIdle) {
audio_service_.EncodeWakeWord();
if (!protocol_->IsAudioChannelOpened()) {
SetDeviceState(kDeviceStateConnecting);
if (!protocol_->OpenAudioChannel()) {
audio_service_.EnableWakeWordDetection(true);
return;
}
}
auto wake_word = audio_service_.GetLastWakeWord();
ESP_LOGI(TAG, "Wake word detected: %s", wake_word.c_str());
#if CONFIG_SEND_WAKE_WORD_DATA
// Encode and send the wake word data to the server
while (auto packet = audio_service_.PopWakeWordPacket()) {
protocol_->SendAudio(std::move(packet));
}
// Set the chat state to wake word detected
protocol_->SendWakeWordDetected(wake_word);
SetListeningMode(aec_mode_ == kAecOff ? kListeningModeAutoStop : kListeningModeRealtime);
#else
SetListeningMode(aec_mode_ == kAecOff ? kListeningModeAutoStop : kListeningModeRealtime);
// Play the pop up sound to indicate the wake word is detected
audio_service_.PlaySound(Lang::Sounds::OGG_POPUP);
#endif
} else if (device_state_ == kDeviceStateSpeaking) {
AbortSpeaking(kAbortReasonWakeWordDetected);
} else if (device_state_ == kDeviceStateActivating) {
SetDeviceState(kDeviceStateIdle);
}
}
void Application::AbortSpeaking(AbortReason reason) {
ESP_LOGI(TAG, "Abort speaking");
aborted_ = true;
if (protocol_) {
protocol_->SendAbortSpeaking(reason);
}
}
void Application::SetListeningMode(ListeningMode mode) {
listening_mode_ = mode;
SetDeviceState(kDeviceStateListening);
}
void Application::SetDeviceState(DeviceState state) {
if (device_state_ == state) {
return;
}
clock_ticks_ = 0;
auto previous_state = device_state_;
device_state_ = state;
ESP_LOGI(TAG, "STATE: %s", STATE_STRINGS[device_state_]);
// Send the state change event
DeviceStateEventManager::GetInstance().PostStateChangeEvent(previous_state, state);
auto& board = Board::GetInstance();
auto display = board.GetDisplay();
auto led = board.GetLed();
led->OnStateChanged();
// 根据设备状态通过 UART 发送指令给 RP2040，驱动舵机执行对应动画
switch (state) {
case kDeviceStateUnknown:
case kDeviceStateIdle:
display->SetStatus(Lang::Strings::STANDBY);
display->SetEmotion("neutral");
audio_service_.EnableVoiceProcessing(false);
audio_service_.EnableWakeWordDetection(true);
uart_send_string("idle"); // → RP2040: 进入睡眠姿态（闭眼、耳朵收起）
break;
case kDeviceStateConnecting:
display->SetStatus(Lang::Strings::CONNECTING);
display->SetEmotion("neutral");
display->SetChatMessage("system", "");
break;
case kDeviceStateListening:
display->SetStatus(Lang::Strings::LISTENING);
display->SetEmotion("neutral");
uart_send_string("listening"); // → RP2040: 进入聆听姿态（好奇歪头、眼球随机看）
// Make sure the audio processor is running
if (!audio_service_.IsAudioProcessorRunning()) {
// Send the start listening command
protocol_->SendStartListening(listening_mode_);
audio_service_.EnableVoiceProcessing(true);
audio_service_.EnableWakeWordDetection(false);
}
break;
case kDeviceStateSpeaking:
display->SetStatus(Lang::Strings::SPEAKING);
uart_send_string("speaking"); // → RP2040: 进入说话姿态（嘴巴开合、眼球随机看、身体摆动）
if (listening_mode_ != kListeningModeRealtime) {
audio_service_.EnableVoiceProcessing(false);
// Only AFE wake word can be detected in speaking mode
audio_service_.EnableWakeWordDetection(audio_service_.IsAfeWakeWord());
}
audio_service_.ResetDecoder();
break;
default:
// Do nothing
break;
}
}
void Application::Reboot() {
ESP_LOGI(TAG, "Rebooting...");
// Disconnect the audio channel
if (protocol_ && protocol_->IsAudioChannelOpened()) {
protocol_->CloseAudioChannel();
}
protocol_.reset();
audio_service_.Stop();
vTaskDelay(pdMS_TO_TICKS(1000));
esp_restart();
}
bool Application::UpgradeFirmware(Ota& ota, const std::string& url) {
auto& board = Board::GetInstance();
auto display = board.GetDisplay();
// Use provided URL or get from OTA object
std::string upgrade_url = url.empty() ? ota.GetFirmwareUrl() : url;
std::string version_info = url.empty() ? ota.GetFirmwareVersion() : "(Manual upgrade)";
// Close audio channel if it's open
if (protocol_ && protocol_->IsAudioChannelOpened()) {
ESP_LOGI(TAG, "Closing audio channel before firmware upgrade");
protocol_->CloseAudioChannel();
}
ESP_LOGI(TAG, "Starting firmware upgrade from URL: %s", upgrade_url.c_str());
Alert(Lang::Strings::OTA_UPGRADE, Lang::Strings::UPGRADING, "download", Lang::Sounds::OGG_UPGRADE);
vTaskDelay(pdMS_TO_TICKS(3000));
SetDeviceState(kDeviceStateUpgrading);
std::string message = std::string(Lang::Strings::NEW_VERSION) + version_info;
display->SetChatMessage("system", message.c_str());
board.SetPowerSaveMode(false);
audio_service_.Stop();
vTaskDelay(pdMS_TO_TICKS(1000));
bool upgrade_success = ota.StartUpgradeFromUrl(upgrade_url, [display](int progress, size_t speed) {
std::thread([display, progress, speed]() {
char buffer[32];
snprintf(buffer, sizeof(buffer), "%d%% %uKB/s", progress, speed / 1024);
display->SetChatMessage("system", buffer);
}).detach();
});
if (!upgrade_success) {
// Upgrade failed, restart audio service and continue running
ESP_LOGE(TAG, "Firmware upgrade failed, restarting audio service and continuing operation...");
audio_service_.Start(); // Restart audio service
board.SetPowerSaveMode(true); // Restore power save mode
Alert(Lang::Strings::ERROR, Lang::Strings::UPGRADE_FAILED, "circle_xmark", Lang::Sounds::OGG_EXCLAMATION);
vTaskDelay(pdMS_TO_TICKS(3000));
return false;
} else {
// Upgrade success, reboot immediately
ESP_LOGI(TAG, "Firmware upgrade successful, rebooting...");
display->SetChatMessage("system", "Upgrade successful, rebooting...");
vTaskDelay(pdMS_TO_TICKS(1000)); // Brief pause to show message
Reboot();
return true;
}
}
void Application::WakeWordInvoke(const std::string& wake_word) {
if (!protocol_) {
return;
}
if (device_state_ == kDeviceStateIdle) {
audio_service_.EncodeWakeWord();
if (!protocol_->IsAudioChannelOpened()) {
SetDeviceState(kDeviceStateConnecting);
if (!protocol_->OpenAudioChannel()) {
audio_service_.EnableWakeWordDetection(true);
return;
}
}
ESP_LOGI(TAG, "Wake word detected: %s", wake_word.c_str());
#if CONFIG_USE_AFE_WAKE_WORD || CONFIG_USE_CUSTOM_WAKE_WORD
// Encode and send the wake word data to the server
while (auto packet = audio_service_.PopWakeWordPacket()) {
protocol_->SendAudio(std::move(packet));
}
// Set the chat state to wake word detected
protocol_->SendWakeWordDetected(wake_word);
SetListeningMode(aec_mode_ == kAecOff ? kListeningModeAutoStop : kListeningModeRealtime);
#else
SetListeningMode(aec_mode_ == kAecOff ? kListeningModeAutoStop : kListeningModeRealtime);
// Play the pop up sound to indicate the wake word is detected
audio_service_.PlaySound(Lang::Sounds::OGG_POPUP);
#endif
} else if (device_state_ == kDeviceStateSpeaking) {
Schedule([this]() {
AbortSpeaking(kAbortReasonNone);
});
} else if (device_state_ == kDeviceStateListening) {
Schedule([this]() {
if (protocol_) {
protocol_->CloseAudioChannel();
}
});
}
}
bool Application::CanEnterSleepMode() {
if (device_state_ != kDeviceStateIdle) {
return false;
}
if (protocol_ && protocol_->IsAudioChannelOpened()) {
return false;
}
if (!audio_service_.IsIdle()) {
return false;
}
// Now it is safe to enter sleep mode
return true;
}
void Application::SendMcpMessage(const std::string& payload) {
if (protocol_ == nullptr) {
return;
}
// Make sure you are using main thread to send MCP message
if (xTaskGetCurrentTaskHandle() == main_event_loop_task_handle_) {
protocol_->SendMcpMessage(payload);
} else {
Schedule([this, payload = std::move(payload)]() {
protocol_->SendMcpMessage(payload);
});
}
}
void Application::SetAecMode(AecMode mode) {
aec_mode_ = mode;
Schedule([this]() {
auto& board = Board::GetInstance();
auto display = board.GetDisplay();
switch (aec_mode_) {
case kAecOff:
audio_service_.EnableDeviceAec(false);
display->ShowNotification(Lang::Strings::RTC_MODE_OFF);
break;
case kAecOnServerSide:
audio_service_.EnableDeviceAec(false);
display->ShowNotification(Lang::Strings::RTC_MODE_ON);
break;
case kAecOnDeviceSide:
audio_service_.EnableDeviceAec(true);
display->ShowNotification(Lang::Strings::RTC_MODE_ON);
break;
}
// If the AEC mode is changed, close the audio channel
if (protocol_ && protocol_->IsAudioChannelOpened()) {
protocol_->CloseAudioChannel();
}
});
}
void Application::PlaySound(const std::string_view& sound) {
audio_service_.PlaySound(sound);
}

110
main/application.h Normal file
View File

@ -0,0 +1,110 @@
#ifndef _APPLICATION_H_
#define _APPLICATION_H_
#include <freertos/FreeRTOS.h>
#include <freertos/event_groups.h>
#include <freertos/task.h>
#include <esp_timer.h>
#include <string>
#include <mutex>
#include <deque>
#include <memory>
#include "protocol.h"
#include "ota.h"
#include "audio_service.h"
#include "device_state_event.h"
#define MAIN_EVENT_SCHEDULE (1 << 0)
#define MAIN_EVENT_SEND_AUDIO (1 << 1)
#define MAIN_EVENT_WAKE_WORD_DETECTED (1 << 2)
#define MAIN_EVENT_VAD_CHANGE (1 << 3)
#define MAIN_EVENT_ERROR (1 << 4)
#define MAIN_EVENT_CHECK_NEW_VERSION_DONE (1 << 5)
#define MAIN_EVENT_CLOCK_TICK (1 << 6)
enum AecMode {
kAecOff,
kAecOnDeviceSide,
kAecOnServerSide,
};
class Application {
public:
static Application& GetInstance() {
static Application instance;
return instance;
}
// 删除拷贝构造函数和赋值运算符
Application(const Application&) = delete;
Application& operator=(const Application&) = delete;
void Start();
void MainEventLoop();
DeviceState GetDeviceState() const { return device_state_; }
bool IsVoiceDetected() const { return audio_service_.IsVoiceDetected(); }
void Schedule(std::function<void()> callback);
void SetDeviceState(DeviceState state);
void Alert(const char* status, const char* message, const char* emotion = "", const std::string_view& sound = "");
void DismissAlert();
void AbortSpeaking(AbortReason reason);
void ToggleChatState();
void StartListening();
void StopListening();
void Reboot();
void WakeWordInvoke(const std::string& wake_word);
bool UpgradeFirmware(Ota& ota, const std::string& url = "");
bool CanEnterSleepMode();
void SendMcpMessage(const std::string& payload);
void SetAecMode(AecMode mode);
AecMode GetAecMode() const { return aec_mode_; }
void PlaySound(const std::string_view& sound);
AudioService& GetAudioService() { return audio_service_; }
private:
Application();
~Application();
std::mutex mutex_;
std::deque<std::function<void()>> main_tasks_;
std::unique_ptr<Protocol> protocol_;
EventGroupHandle_t event_group_ = nullptr;
esp_timer_handle_t clock_timer_handle_ = nullptr;
volatile DeviceState device_state_ = kDeviceStateUnknown;
ListeningMode listening_mode_ = kListeningModeAutoStop;
AecMode aec_mode_ = kAecOff;
std::string last_error_message_;
AudioService audio_service_;
bool has_server_time_ = false;
bool aborted_ = false;
int clock_ticks_ = 0;
TaskHandle_t check_new_version_task_handle_ = nullptr;
TaskHandle_t main_event_loop_task_handle_ = nullptr;
void OnWakeWordDetected();
void CheckNewVersion(Ota& ota);
void CheckAssetsVersion();
void ShowActivationCode(const std::string& code, const std::string& message);
void SetListeningMode(ListeningMode mode);
};
class TaskPriorityReset {
public:
TaskPriorityReset(BaseType_t priority) {
original_priority_ = uxTaskPriorityGet(NULL);
vTaskPrioritySet(NULL, priority);
}
~TaskPriorityReset() {
vTaskPrioritySet(NULL, original_priority_);
}
private:
BaseType_t original_priority_;
};
#endif // _APPLICATION_H_

532
main/assets.cc Normal file
View File

@ -0,0 +1,532 @@
#include "assets.h"
#include "board.h"
#include "display.h"
#include "application.h"
#include "lvgl_theme.h"
#include "emote_display.h"
#ifdef HAVE_LVGL
#include "display/lcd_display.h"
#endif
#include <esp_log.h>
#include <spi_flash_mmap.h>
#include <esp_timer.h>
#include <cbin_font.h>
#define TAG "Assets"
struct mmap_assets_table {
char asset_name[32]; /*!< Name of the asset */
uint32_t asset_size; /*!< Size of the asset */
uint32_t asset_offset; /*!< Offset of the asset */
uint16_t asset_width; /*!< Width of the asset */
uint16_t asset_height; /*!< Height of the asset */
};
Assets::Assets() {
// Initialize the partition
InitializePartition();
}
Assets::~Assets() {
if (mmap_handle_ != 0) {
esp_partition_munmap(mmap_handle_);
}
}
uint32_t Assets::CalculateChecksum(const char* data, uint32_t length) {
uint32_t checksum = 0;
for (uint32_t i = 0; i < length; i++) {
checksum += data[i];
}
return checksum & 0xFFFF;
}
bool Assets::InitializePartition() {
partition_valid_ = false;
checksum_valid_ = false;
assets_.clear();
partition_ = esp_partition_find_first(ESP_PARTITION_TYPE_ANY, ESP_PARTITION_SUBTYPE_ANY, "assets");
if (partition_ == nullptr) {
ESP_LOGI(TAG, "No assets partition found");
return false;
}
int free_pages = spi_flash_mmap_get_free_pages(SPI_FLASH_MMAP_DATA);
uint32_t storage_size = free_pages * 64 * 1024;
ESP_LOGI(TAG, "The storage free size is %ld KB", storage_size / 1024);
ESP_LOGI(TAG, "The partition size is %ld KB", partition_->size / 1024);
if (storage_size < partition_->size) {
ESP_LOGE(TAG, "The free size %ld KB is less than assets partition required %ld KB", storage_size / 1024, partition_->size / 1024);
return false;
}
esp_err_t err = esp_partition_mmap(partition_, 0, partition_->size, ESP_PARTITION_MMAP_DATA, (const void**)&mmap_root_, &mmap_handle_);
if (err != ESP_OK) {
ESP_LOGE(TAG, "Failed to mmap assets partition: %s", esp_err_to_name(err));
return false;
}
partition_valid_ = true;
uint32_t stored_files = *(uint32_t*)(mmap_root_ + 0);
uint32_t stored_chksum = *(uint32_t*)(mmap_root_ + 4);
uint32_t stored_len = *(uint32_t*)(mmap_root_ + 8);
if (stored_len > partition_->size - 12) {
ESP_LOGD(TAG, "The stored_len (0x%lx) is greater than the partition size (0x%lx) - 12", stored_len, partition_->size);
return false;
}
auto start_time = esp_timer_get_time();
uint32_t calculated_checksum = CalculateChecksum(mmap_root_ + 12, stored_len);
auto end_time = esp_timer_get_time();
ESP_LOGI(TAG, "The checksum calculation time is %d ms", int((end_time - start_time) / 1000));
if (calculated_checksum != stored_chksum) {
ESP_LOGE(TAG, "The calculated checksum (0x%lx) does not match the stored checksum (0x%lx)", calculated_checksum, stored_chksum);
return false;
}
checksum_valid_ = true;
for (uint32_t i = 0; i < stored_files; i++) {
auto item = (const mmap_assets_table*)(mmap_root_ + 12 + i * sizeof(mmap_assets_table));
auto asset = Asset{
.size = static_cast<size_t>(item->asset_size),
.offset = static_cast<size_t>(12 + sizeof(mmap_assets_table) * stored_files + item->asset_offset)
};
assets_[item->asset_name] = asset;
}
return checksum_valid_;
}
bool Assets::Apply() {
void* ptr = nullptr;
size_t size = 0;
if (!GetAssetData("index.json", ptr, size)) {
ESP_LOGE(TAG, "The index.json file is not found");
return false;
}
cJSON* root = cJSON_ParseWithLength(static_cast<char*>(ptr), size);
if (root == nullptr) {
ESP_LOGE(TAG, "The index.json file is not valid");
return false;
}
cJSON* version = cJSON_GetObjectItem(root, "version");
if (cJSON_IsNumber(version)) {
if (version->valuedouble > 1) {
ESP_LOGE(TAG, "The assets version %d is not supported, please upgrade the firmware", version->valueint);
return false;
}
}
cJSON* srmodels = cJSON_GetObjectItem(root, "srmodels");
if (cJSON_IsString(srmodels)) {
std::string srmodels_file = srmodels->valuestring;
if (GetAssetData(srmodels_file, ptr, size)) {
if (models_list_ != nullptr) {
esp_srmodel_deinit(models_list_);
models_list_ = nullptr;
}
models_list_ = srmodel_load(static_cast<uint8_t*>(ptr));
if (models_list_ != nullptr) {
auto& app = Application::GetInstance();
app.GetAudioService().SetModelsList(models_list_);
} else {
ESP_LOGE(TAG, "Failed to load srmodels.bin");
}
} else {
ESP_LOGE(TAG, "The srmodels file %s is not found", srmodels_file.c_str());
}
}
#ifdef HAVE_LVGL
auto& theme_manager = LvglThemeManager::GetInstance();
auto light_theme = theme_manager.GetTheme("light");
auto dark_theme = theme_manager.GetTheme("dark");
cJSON* font = cJSON_GetObjectItem(root, "text_font");
if (cJSON_IsString(font)) {
std::string fonts_text_file = font->valuestring;
if (GetAssetData(fonts_text_file, ptr, size)) {
auto text_font = std::make_shared<LvglCBinFont>(ptr);
if (text_font->font() == nullptr) {
ESP_LOGE(TAG, "Failed to load fonts.bin");
return false;
}
if (light_theme != nullptr) {
light_theme->set_text_font(text_font);
}
if (dark_theme != nullptr) {
dark_theme->set_text_font(text_font);
}
} else {
ESP_LOGE(TAG, "The font file %s is not found", fonts_text_file.c_str());
}
}
cJSON* emoji_collection = cJSON_GetObjectItem(root, "emoji_collection");
if (cJSON_IsArray(emoji_collection)) {
auto custom_emoji_collection = std::make_shared<EmojiCollection>();
int emoji_count = cJSON_GetArraySize(emoji_collection);
for (int i = 0; i < emoji_count; i++) {
cJSON* emoji = cJSON_GetArrayItem(emoji_collection, i);
if (cJSON_IsObject(emoji)) {
cJSON* name = cJSON_GetObjectItem(emoji, "name");
cJSON* file = cJSON_GetObjectItem(emoji, "file");
cJSON* eaf = cJSON_GetObjectItem(emoji, "eaf");
if (cJSON_IsString(name) && cJSON_IsString(file) && (NULL== eaf)) {
if (!GetAssetData(file->valuestring, ptr, size)) {
ESP_LOGE(TAG, "Emoji %s image file %s is not found", name->valuestring, file->valuestring);
continue;
}
custom_emoji_collection->AddEmoji(name->valuestring, new LvglRawImage(ptr, size));
}
}
}
if (light_theme != nullptr) {
light_theme->set_emoji_collection(custom_emoji_collection);
}
if (dark_theme != nullptr) {
dark_theme->set_emoji_collection(custom_emoji_collection);
}
}
cJSON* skin = cJSON_GetObjectItem(root, "skin");
if (cJSON_IsObject(skin)) {
cJSON* light_skin = cJSON_GetObjectItem(skin, "light");
if (cJSON_IsObject(light_skin) && light_theme != nullptr) {
cJSON* text_color = cJSON_GetObjectItem(light_skin, "text_color");
cJSON* background_color = cJSON_GetObjectItem(light_skin, "background_color");
cJSON* background_image = cJSON_GetObjectItem(light_skin, "background_image");
if (cJSON_IsString(text_color)) {
light_theme->set_text_color(LvglTheme::ParseColor(text_color->valuestring));
}
if (cJSON_IsString(background_color)) {
light_theme->set_background_color(LvglTheme::ParseColor(background_color->valuestring));
light_theme->set_chat_background_color(LvglTheme::ParseColor(background_color->valuestring));
}
if (cJSON_IsString(background_image)) {
if (!GetAssetData(background_image->valuestring, ptr, size)) {
ESP_LOGE(TAG, "The background image file %s is not found", background_image->valuestring);
return false;
}
auto background_image = std::make_shared<LvglCBinImage>(ptr);
light_theme->set_background_image(background_image);
}
}
cJSON* dark_skin = cJSON_GetObjectItem(skin, "dark");
if (cJSON_IsObject(dark_skin) && dark_theme != nullptr) {
cJSON* text_color = cJSON_GetObjectItem(dark_skin, "text_color");
cJSON* background_color = cJSON_GetObjectItem(dark_skin, "background_color");
cJSON* background_image = cJSON_GetObjectItem(dark_skin, "background_image");
if (cJSON_IsString(text_color)) {
dark_theme->set_text_color(LvglTheme::ParseColor(text_color->valuestring));
}
if (cJSON_IsString(background_color)) {
dark_theme->set_background_color(LvglTheme::ParseColor(background_color->valuestring));
dark_theme->set_chat_background_color(LvglTheme::ParseColor(background_color->valuestring));
}
if (cJSON_IsString(background_image)) {
if (!GetAssetData(background_image->valuestring, ptr, size)) {
ESP_LOGE(TAG, "The background image file %s is not found", background_image->valuestring);
return false;
}
auto background_image = std::make_shared<LvglCBinImage>(ptr);
dark_theme->set_background_image(background_image);
}
}
}
auto display = Board::GetInstance().GetDisplay();
ESP_LOGI(TAG, "Refreshing display theme...");
auto current_theme = display->GetTheme();
if (current_theme != nullptr) {
display->SetTheme(current_theme);
}
// Parse hide_subtitle configuration
cJSON* hide_subtitle = cJSON_GetObjectItem(root, "hide_subtitle");
if (cJSON_IsBool(hide_subtitle)) {
bool hide = cJSON_IsTrue(hide_subtitle);
auto lcd_display = dynamic_cast<LcdDisplay*>(display);
if (lcd_display != nullptr) {
lcd_display->SetHideSubtitle(hide);
ESP_LOGI(TAG, "Set hide_subtitle to %s", hide ? "true" : "false");
}
}
#elif defined(CONFIG_USE_EMOTE_MESSAGE_STYLE)
auto &board = Board::GetInstance();
auto display = board.GetDisplay();
auto emote_display = dynamic_cast<emote::EmoteDisplay*>(display);
cJSON* font = cJSON_GetObjectItem(root, "text_font");
if (cJSON_IsString(font)) {
std::string fonts_text_file = font->valuestring;
if (GetAssetData(fonts_text_file, ptr, size)) {
auto text_font = std::make_shared<LvglCBinFont>(ptr);
if (text_font->font() == nullptr) {
ESP_LOGE(TAG, "Failed to load fonts.bin");
return false;
}
if (emote_display) {
emote_display->AddTextFont(text_font);
}
} else {
ESP_LOGE(TAG, "The font file %s is not found", fonts_text_file.c_str());
}
}
cJSON* emoji_collection = cJSON_GetObjectItem(root, "emoji_collection");
if (cJSON_IsArray(emoji_collection)) {
int emoji_count = cJSON_GetArraySize(emoji_collection);
if (emote_display) {
for (int i = 0; i < emoji_count; i++) {
cJSON* icon = cJSON_GetArrayItem(emoji_collection, i);
if (cJSON_IsObject(icon)) {
cJSON* name = cJSON_GetObjectItem(icon, "name");
cJSON* file = cJSON_GetObjectItem(icon, "file");
if (cJSON_IsString(name) && cJSON_IsString(file)) {
if (GetAssetData(file->valuestring, ptr, size)) {
cJSON* eaf = cJSON_GetObjectItem(icon, "eaf");
bool lack_value = false;
bool loop_value = false;
int fps_value = 0;
if (cJSON_IsObject(eaf)) {
cJSON* lack = cJSON_GetObjectItem(eaf, "lack");
cJSON* loop = cJSON_GetObjectItem(eaf, "loop");
cJSON* fps = cJSON_GetObjectItem(eaf, "fps");
lack_value = lack ? cJSON_IsTrue(lack) : false;
loop_value = loop ? cJSON_IsTrue(loop) : false;
fps_value = fps ? fps->valueint : 0;
emote_display->AddEmojiData(name->valuestring, ptr, size,
static_cast<uint8_t>(fps_value),
loop_value, lack_value);
}
} else {
ESP_LOGE(TAG, "Emoji \"%10s\" image file %s is not found", name->valuestring, file->valuestring);
}
}
}
}
}
}
cJSON* icon_collection = cJSON_GetObjectItem(root, "icon_collection");
if (cJSON_IsArray(icon_collection)) {
if (emote_display) {
int icon_count = cJSON_GetArraySize(icon_collection);
for (int i = 0; i < icon_count; i++) {
cJSON* icon = cJSON_GetArrayItem(icon_collection, i);
if (cJSON_IsObject(icon)) {
cJSON* name = cJSON_GetObjectItem(icon, "name");
cJSON* file = cJSON_GetObjectItem(icon, "file");
if (cJSON_IsString(name) && cJSON_IsString(file)) {
if (GetAssetData(file->valuestring, ptr, size)) {
emote_display->AddIconData(name->valuestring, ptr, size);
} else {
ESP_LOGE(TAG, "Icon \"%10s\" image file %s is not found", name->valuestring, file->valuestring);
}
}
}
}
}
}
cJSON* layout_json = cJSON_GetObjectItem(root, "layout");
if (cJSON_IsArray(layout_json)) {
int layout_count = cJSON_GetArraySize(layout_json);
for (int i = 0; i < layout_count; i++) {
cJSON* layout_item = cJSON_GetArrayItem(layout_json, i);
if (cJSON_IsObject(layout_item)) {
cJSON* name = cJSON_GetObjectItem(layout_item, "name");
cJSON* align = cJSON_GetObjectItem(layout_item, "align");
cJSON* x = cJSON_GetObjectItem(layout_item, "x");
cJSON* y = cJSON_GetObjectItem(layout_item, "y");
cJSON* width = cJSON_GetObjectItem(layout_item, "width");
cJSON* height = cJSON_GetObjectItem(layout_item, "height");
if (cJSON_IsString(name) && cJSON_IsString(align) && cJSON_IsNumber(x) && cJSON_IsNumber(y)) {
int width_val = cJSON_IsNumber(width) ? width->valueint : 0;
int height_val = cJSON_IsNumber(height) ? height->valueint : 0;
if (emote_display) {
emote_display->AddLayoutData(name->valuestring, align->valuestring,
x->valueint, y->valueint, width_val, height_val);
}
} else {
ESP_LOGW(TAG, "Invalid layout item %d: missing required fields", i);
}
}
}
}
#endif
cJSON_Delete(root);
return true;
}
bool Assets::Download(std::string url, std::function<void(int progress, size_t speed)> progress_callback) {
ESP_LOGI(TAG, "Downloading new version of assets from %s", url.c_str());
// 取消当前资源分区的内存映射
if (mmap_handle_ != 0) {
esp_partition_munmap(mmap_handle_);
mmap_handle_ = 0;
mmap_root_ = nullptr;
}
checksum_valid_ = false;
assets_.clear();
// 下载新的资源文件
auto network = Board::GetInstance().GetNetwork();
auto http = network->CreateHttp(0);
if (!http->Open("GET", url)) {
ESP_LOGE(TAG, "Failed to open HTTP connection");
return false;
}
if (http->GetStatusCode() != 200) {
ESP_LOGE(TAG, "Failed to get assets, status code: %d", http->GetStatusCode());
return false;
}
size_t content_length = http->GetBodyLength();
if (content_length == 0) {
ESP_LOGE(TAG, "Failed to get content length");
return false;
}
if (content_length > partition_->size) {
ESP_LOGE(TAG, "Assets file size (%u) is larger than partition size (%lu)", content_length, partition_->size);
return false;
}
// 定义扇区大小为4KB（ESP32的标准扇区大小）
const size_t SECTOR_SIZE = esp_partition_get_main_flash_sector_size();
// 计算需要擦除的扇区数量
size_t sectors_to_erase = (content_length + SECTOR_SIZE - 1) / SECTOR_SIZE; // 向上取整
size_t total_erase_size = sectors_to_erase * SECTOR_SIZE;
ESP_LOGI(TAG, "Sector size: %u, content length: %u, sectors to erase: %u, total erase size: %u",
SECTOR_SIZE, content_length, sectors_to_erase, total_erase_size);
// 写入新的资源文件到分区，一边erase一边写入
char buffer[512];
size_t total_written = 0;
size_t recent_written = 0;
size_t current_sector = 0;
auto last_calc_time = esp_timer_get_time();
while (true) {
int ret = http->Read(buffer, sizeof(buffer));
if (ret < 0) {
ESP_LOGE(TAG, "Failed to read HTTP data: %s", esp_err_to_name(ret));
return false;
}
if (ret == 0) {
break;
}
// 检查是否需要擦除新的扇区
size_t write_end_offset = total_written + ret;
size_t needed_sectors = (write_end_offset + SECTOR_SIZE - 1) / SECTOR_SIZE;
// 擦除需要的新扇区
while (current_sector < needed_sectors) {
size_t sector_start = current_sector * SECTOR_SIZE;
size_t sector_end = (current_sector + 1) * SECTOR_SIZE;
// 确保擦除范围不超过分区大小
if (sector_end > partition_->size) {
ESP_LOGE(TAG, "Sector end (%u) exceeds partition size (%lu)", sector_end, partition_->size);
return false;
}
ESP_LOGD(TAG, "Erasing sector %u (offset: %u, size: %u)", current_sector, sector_start, SECTOR_SIZE);
esp_err_t err = esp_partition_erase_range(partition_, sector_start, SECTOR_SIZE);
if (err != ESP_OK) {
ESP_LOGE(TAG, "Failed to erase sector %u at offset %u: %s", current_sector, sector_start, esp_err_to_name(err));
return false;
}
current_sector++;
}
// 写入数据到分区
esp_err_t err = esp_partition_write(partition_, total_written, buffer, ret);
if (err != ESP_OK) {
ESP_LOGE(TAG, "Failed to write to assets partition at offset %u: %s", total_written, esp_err_to_name(err));
return false;
}
total_written += ret;
recent_written += ret;
// 计算进度和速度
if (esp_timer_get_time() - last_calc_time >= 1000000 || total_written == content_length || ret == 0) {
size_t progress = total_written * 100 / content_length;
size_t speed = recent_written; // 每秒的字节数
ESP_LOGI(TAG, "Progress: %u%% (%u/%u), Speed: %u B/s, Sectors erased: %u",
progress, total_written, content_length, speed, current_sector);
if (progress_callback) {
progress_callback(progress, speed);
}
last_calc_time = esp_timer_get_time();
recent_written = 0; // 重置最近写入的字节数
}
}
http->Close();
if (total_written != content_length) {
ESP_LOGE(TAG, "Downloaded size (%u) does not match expected size (%u)", total_written, content_length);
return false;
}
ESP_LOGI(TAG, "Assets download completed, total written: %u bytes, total sectors erased: %u",
total_written, current_sector);
// 重新初始化资源分区
if (!InitializePartition()) {
ESP_LOGE(TAG, "Failed to re-initialize assets partition");
return false;
}
return true;
}
bool Assets::GetAssetData(const std::string& name, void*& ptr, size_t& size) {
auto asset = assets_.find(name);
if (asset == assets_.end()) {
return false;
}
auto data = (const char*)(mmap_root_ + asset->second.offset);
if (data[0] != 'Z' || data[1] != 'Z') {
ESP_LOGE(TAG, "The asset %s is not valid with magic %02x%02x", name.c_str(), data[0], data[1]);
return false;
}
ptr = static_cast<void*>(const_cast<char*>(data + 2));
size = asset->second.size;
return true;
}

52
main/assets.h Normal file
View File

@ -0,0 +1,52 @@
#ifndef ASSETS_H
#define ASSETS_H
#include <map>
#include <string>
#include <functional>
#include <cJSON.h>
#include <esp_partition.h>
#include <model_path.h>
struct Asset {
size_t size;
size_t offset;
};
class Assets {
public:
static Assets& GetInstance() {
static Assets instance;
return instance;
}
~Assets();
bool Download(std::string url, std::function<void(int progress, size_t speed)> progress_callback);
bool Apply();
bool GetAssetData(const std::string& name, void*& ptr, size_t& size);
inline bool partition_valid() const { return partition_valid_; }
inline bool checksum_valid() const { return checksum_valid_; }
inline std::string default_assets_url() const { return default_assets_url_; }
private:
Assets();
Assets(const Assets&) = delete;
Assets& operator=(const Assets&) = delete;
bool InitializePartition();
uint32_t CalculateChecksum(const char* data, uint32_t length);
const esp_partition_t* partition_ = nullptr;
esp_partition_mmap_handle_t mmap_handle_ = 0;
const char* mmap_root_ = nullptr;
bool partition_valid_ = false;
bool checksum_valid_ = false;
std::string default_assets_url_;
srmodel_list_t* models_list_ = nullptr;
std::map<std::string, Asset> assets_;
};
#endif

BIN
main/assets/common/exclamation.ogg Normal file
View File

Binary file not shown.

BIN
main/assets/common/low_battery.ogg Normal file
View File

Binary file not shown.

BIN
main/assets/common/popup.ogg Normal file
View File

Binary file not shown.

BIN
main/assets/common/success.ogg Normal file
View File

Binary file not shown.

BIN
main/assets/common/vibration.ogg Normal file
View File

Binary file not shown.

BIN
main/assets/locales/ar-SA/0.ogg Normal file
View File

Binary file not shown.

BIN
main/assets/locales/ar-SA/1.ogg Normal file
View File

Binary file not shown.

BIN
main/assets/locales/ar-SA/2.ogg Normal file
View File

Binary file not shown.

BIN
main/assets/locales/ar-SA/3.ogg Normal file
View File

Binary file not shown.

BIN
main/assets/locales/ar-SA/4.ogg Normal file
View File

Binary file not shown.

BIN
main/assets/locales/ar-SA/5.ogg Normal file
View File

Binary file not shown.

BIN
main/assets/locales/ar-SA/6.ogg Normal file
View File

Binary file not shown.

BIN
main/assets/locales/ar-SA/7.ogg Normal file
View File

Binary file not shown.

BIN
main/assets/locales/ar-SA/8.ogg Normal file
View File

Binary file not shown.

BIN
main/assets/locales/ar-SA/9.ogg Normal file
View File

Binary file not shown.

BIN
main/assets/locales/ar-SA/activation.ogg Normal file
View File

Binary file not shown.

BIN
main/assets/locales/ar-SA/err_pin.ogg Normal file
View File

Binary file not shown.

BIN
main/assets/locales/ar-SA/err_reg.ogg Normal file
View File

Binary file not shown.

55
main/assets/locales/ar-SA/language.json Normal file
View File

@ -0,0 +1,55 @@
{
"language": {
"type": "ar-SA"
},
"strings": {
"WARNING": "تحذير",
"INFO": "معلومات",
"ERROR": "خطأ",
"VERSION": "الإصدار ",
"LOADING_PROTOCOL": "الاتصال بالخادم...",
"INITIALIZING": "التهيئة...",
"PIN_ERROR": "يرجى إدخال بطاقة SIM",
"REG_ERROR": "لا يمكن الوصول إلى الشبكة، يرجى التحقق من حالة بطاقة البيانات",
"DETECTING_MODULE": "اكتشاف الوحدة...",
"REGISTERING_NETWORK": "انتظار الشبكة...",
"CHECKING_NEW_VERSION": "فحص الإصدار الجديد...",
"CHECK_NEW_VERSION_FAILED": "فشل فحص الإصدار الجديد، سيتم المحاولة خلال %d ثانية: %s",
"SWITCH_TO_WIFI_NETWORK": "التبديل إلى Wi-Fi...",
"SWITCH_TO_4G_NETWORK": "التبديل إلى 4G...",
"STANDBY": "في الانتظار",
"CONNECT_TO": "الاتصال بـ ",
"CONNECTING": "جاري الاتصال...",
"CONNECTED_TO": "متصل بـ ",
"LISTENING": "الاستماع...",
"SPEAKING": "التحدث...",
"SERVER_NOT_FOUND": "البحث عن خدمة متاحة",
"SERVER_NOT_CONNECTED": "لا يمكن الاتصال بالخدمة، يرجى المحاولة لاحقاً",
"SERVER_TIMEOUT": "انتهت مهلة الاستجابة",
"SERVER_ERROR": "فشل الإرسال، يرجى التحقق من الشبكة",
"CONNECT_TO_HOTSPOT": "اتصل الهاتف بنقطة الاتصال ",
"ACCESS_VIA_BROWSER": "،الوصول عبر المتصفح ",
"WIFI_CONFIG_MODE": "وضع تكوين الشبكة",
"ENTERING_WIFI_CONFIG_MODE": "الدخول في وضع تكوين الشبكة...",
"SCANNING_WIFI": "فحص Wi-Fi...",
"NEW_VERSION": "إصدار جديد ",
"OTA_UPGRADE": "تحديث OTA",
"UPGRADING": "تحديث النظام...",
"UPGRADE_FAILED": "فشل التحديث",
"ACTIVATION": "تفعيل الجهاز",
"BATTERY_LOW": "البطارية منخفضة",
"BATTERY_CHARGING": "جاري الشحن",
"BATTERY_FULL": "البطارية ممتلئة",
"BATTERY_NEED_CHARGE": "البطارية منخفضة، يرجى الشحن",
"VOLUME": "الصوت ",
"MUTED": "صامت",
"MAX_VOLUME": "أقصى صوت",
"RTC_MODE_OFF": "AEC مُوقف",
"RTC_MODE_ON": "AEC مُشغل",
"DOWNLOAD_ASSETS_FAILED": "فشل في تنزيل الموارد",
"LOADING_ASSETS": "جاري تحميل الموارد...",
"PLEASE_WAIT": "يرجى الانتظار...",
"FOUND_NEW_ASSETS": "تم العثور على موارد جديدة: %s",
"HELLO_MY_FRIEND": "مرحباً، صديقي!"
}
}

BIN
main/assets/locales/ar-SA/upgrade.ogg Normal file
View File

Binary file not shown.

BIN
main/assets/locales/ar-SA/welcome.ogg Normal file
View File

Binary file not shown.

BIN
main/assets/locales/ar-SA/wificonfig.ogg Normal file
View File

Binary file not shown.

BIN
main/assets/locales/bg-BG/0.ogg Normal file
View File

Binary file not shown.

BIN
main/assets/locales/bg-BG/1.ogg Normal file
View File

Binary file not shown.

BIN
main/assets/locales/bg-BG/2.ogg Normal file
View File

Binary file not shown.

BIN
main/assets/locales/bg-BG/3.ogg Normal file
View File

Binary file not shown.

BIN
main/assets/locales/bg-BG/4.ogg Normal file
View File

Binary file not shown.

BIN
main/assets/locales/bg-BG/5.ogg Normal file
View File

Binary file not shown.

BIN
main/assets/locales/bg-BG/6.ogg Normal file
View File

Binary file not shown.

BIN
main/assets/locales/bg-BG/7.ogg Normal file
View File

Binary file not shown.

BIN
main/assets/locales/bg-BG/8.ogg Normal file
View File

Binary file not shown.

BIN
main/assets/locales/bg-BG/9.ogg Normal file
View File

Binary file not shown.

BIN
main/assets/locales/bg-BG/activation.ogg Normal file
View File

Binary file not shown.

BIN
main/assets/locales/bg-BG/err_pin.ogg Normal file
View File

Binary file not shown.

BIN
main/assets/locales/bg-BG/err_reg.ogg Normal file
View File

Binary file not shown.

57
main/assets/locales/bg-BG/language.json Normal file
View File

@ -0,0 +1,57 @@
{
"language": {
"type": "bg-BG"
},
"strings": {
"WARNING": "Предупреждение",
"INFO": "Информация",
"ERROR": "Грешка",
"VERSION": "Версия ",
"LOADING_PROTOCOL": "Влизане в системата...",
"INITIALIZING": "Инициализация...",
"PIN_ERROR": "Моля, поставете SIM карта",
"REG_ERROR": "Не може да се осъществи достъп до мрежата, моля проверете статуса на SIM картата",
"DETECTING_MODULE": "Откриване на модул...",
"REGISTERING_NETWORK": "Изчакване на мрежата...",
"CHECKING_NEW_VERSION": "Проверка за нова версия...",
"CHECK_NEW_VERSION_FAILED": "Проверката за нова версия е неуспешна, ще се опита отново след %d секунди: %s",
"SWITCH_TO_WIFI_NETWORK": "Превключване към Wi-Fi...",
"SWITCH_TO_4G_NETWORK": "Превключване към 4G...",
"STANDBY": "Режим на готовност",
"CONNECT_TO": "Свързване към ",
"CONNECTING": "Свързване...",
"CONNECTION_SUCCESSFUL": "Успешно свързване",
"CONNECTED_TO": "Свързан към ",
"LISTENING": "Слушане...",
"SPEAKING": "Говорене...",
"SERVER_NOT_FOUND": "Търсене на налична услуга",
"SERVER_NOT_CONNECTED": "Не може да се свърже с услугата, моля опитайте по-късно",
"SERVER_TIMEOUT": "Времето за изчакване на отговор изтече",
"SERVER_ERROR": "Неуспешно изпращане, моля проверете мрежата",
"CONNECT_TO_HOTSPOT": "Горещa точка: ",
"ACCESS_VIA_BROWSER": " Конфигурационен URL: ",
"WIFI_CONFIG_MODE": "Режим на конфигуриране на Wi-Fi",
"ENTERING_WIFI_CONFIG_MODE": "Влизане в режим на конфигуриране на Wi-Fi...",
"SCANNING_WIFI": "Сканиране на Wi-Fi...",
"NEW_VERSION": "Нова версия ",
"OTA_UPGRADE": "OTA надстройка",
"UPGRADING": "Системата се надстройва...",
"UPGRADE_FAILED": "Надстройката е неуспешна",
"ACTIVATION": "Активация",
"BATTERY_LOW": "Слаба батерия",
"BATTERY_CHARGING": "Зарядна",
"BATTERY_FULL": "Батерията е пълна",
"BATTERY_NEED_CHARGE": "Слаба батерия, моля заредете",
"VOLUME": "Сила на звука ",
"MUTED": "Заглушено",
"MAX_VOLUME": "Максимална сила на звука",
"RTC_MODE_OFF": "AEC изключен",
"RTC_MODE_ON": "AEC включен",
"PLEASE_WAIT": "Моля, изчакайте...",
"FOUND_NEW_ASSETS": "Намерени нови ресурси: %s",
"DOWNLOAD_ASSETS_FAILED": "Неуспешно изтегляне на ресурси",
"LOADING_ASSETS": "Зареждане на ресурси...",
"HELLO_MY_FRIEND": "Здравей, мой приятел!"
}
}

BIN
main/assets/locales/bg-BG/upgrade.ogg Normal file
View File

Binary file not shown.

BIN
main/assets/locales/bg-BG/welcome.ogg Normal file
View File

Binary file not shown.

BIN
main/assets/locales/bg-BG/wificonfig.ogg Normal file
View File

Binary file not shown.

BIN
main/assets/locales/ca-ES/0.ogg Normal file
View File

Binary file not shown.

Some files were not shown because too many files have changed in this diff Show More