toy-Kapi_Rtc/main/CMakeLists.txt
Rdzleo a369796eb6 feat(audio): 设备端软件 loopback ref AEC (路径 D'') 完整实施
N16R8 模组无法跑硬件 ADC 回采 (32-bit STEREO codec + 火山 RTC + 80MHz PSRAM 三者
不可共存, 详见 commit fb4b607 探索教训)。改走软件 loopback ref 方案:
codec 保持 baseline 1ch 16-bit (RTC 链路 100% 稳定), DAC 输出 PCM 软件复制一份作
AEC ref 信号, 用 esp_aec.h 底层同步 API (不启后台任务, 不抢 RTC 调度) 处理。

实测验证有效:
- AI 说话: mic=187 ref=8929 clean=30 → 回声消除 84%
- 用户说话: mic=456 ref=8 clean=456 → passthrough 100% 保留
- 服务端 ASR 正常识别用户语音, AI 正常响应 (📝 USER: + 📝 AI: 字幕完整)
- 无 WiFi pm_coex panic, idle 倒计时稳定

主要变动:

1. main/CMakeLists.txt (4 行)
   - REQUIRES 加 esp-sr (引入 esp_aec.h 底层同步 API)

2. main/application.h (23 行)
   - aec_handle_ / aec_chunk_size_ / ref_ring_buf_ / ref_ring_capacity_ /
     ref_ring_write_idx_ / ref_ring_filled_ / aec_ref_delay_samples_ /
     ref_ring_mutex_ 成员
   - InitAec / DeinitAec / AppendRefSamples / GetDelayedRef / ApplyAEC 函数声明

3. main/application.cc (242 行)
   - include esp_aec.h + esp_heap_caps.h
   - InitAec: lazy 初始化 (Application 构造时不调, ReadAudio 首次走 AEC 路径触发),
     避免开机占内部 SRAM 影响 WiFi 启动; ref_ring_buf 优先 PSRAM 分配 200ms 容量
   - DeinitAec: 析构时清理 aec_handle / ref_ring_buf / ref_ring_mutex
   - AppendRefSamples: DAC PCM 推入 ref ring buffer (mutex 互斥)
   - GetDelayedRef: 从 ref ring buffer 取延迟后 ref (mic 同步用)
   - ApplyAEC: 按 chunk_size 处理, 加 ref 静音检测 (RMS<50 时 passthrough),
     RMS 诊断日志每 2 秒打印一次 (mic/ref/clean)
   - OnAudioOutput 两个分支 (player_pipeline_write / codec->OutputData) 都加
     AppendRefSamples hook, 复制 PCM 到 ref ring buffer
   - ReadAudio: recorder_pipeline 路径加 lazy InitAec + ApplyAEC, target_samples
     取 max(caller_samples, chunk_size) 保持 baseline 20ms PCM 帧大小
   - 析构调 DeinitAec

实施 4 大踩坑 (详见 ~/.claude/projects/.../memory/project_software_aec_implementation.md):

a) portMUX (spinlock) 禁中断与 WiFi pm_coex 模块冲突 → IllegalInstruction panic
   修复: 用 SemaphoreHandle_t (FreeRTOS mutex, 2ms 超时) 替代, 不禁中断

b) AI 静音后 AEC 滤波器维持 echo 模式错误压制用户语音 → ASR 不识别
   修复: ApplyAEC 加 ref 静音检测, ref RMS<50 时 passthrough 不调 aec_process

c) chunk_size (256, 16ms) ≠ caller_samples (320, 20ms) 让上行 PCM 帧大小变 →
   服务端 ASR 不识别非标准帧
   修复: target_samples = max(samples, aec_chunk_size_), 保持 baseline 20ms 帧

d) aec_create 占内部 SRAM (~30-50KB) 影响 WiFi RX buffer 分配 → panic 重启
   修复: lazy init, ReadAudio 首次需要时才创建实例

资源占用 (实测):
- Flash: +59 KB (esp-sr libaec.a)
- Internal SRAM: +35-50 KB (aec_handle_t 工作 buffer)
- PSRAM: +10-15 KB (ref_ring_buf 200ms + 临时 buffer)
- Core 1 CPU: +6-12% (chunk=256, 每 16ms 一次 aec_process)
- 整体评估: 适中, 不影响 RTC/WiFi 等其他功能

自言自语根因辨析 (重要认知更正):
- 火山控制台 "AI 降噪 OFF" 是 NS 不是 AEC, 服务端 AEC 默认 ON 不显示在 UI
- baseline 不自言自语 = 云端 AEC 在兜底
- 自言自语真因常是上行 PCM 数据异常 (如嘟嘟嘟阶段 channel_mask 错位) 触发服务端
  VAD 误判, 不是 echo 太大
- 设备端软件 AEC 是减轻云端负载 + 极端场景兜底, 非必需但工程价值显著

调优指南: aec_ref_delay_samples_ 当前 800 (50ms), 根据 mic 离扬声器距离调
30-80ms, 监听 RMS 中 AI 说话期间 clean 最小为最优

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-19 15:30:20 +08:00

242 lines
9.5 KiB
CMake
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

set(SOURCES "audio_codecs/audio_codec.cc"
"audio_codecs/no_audio_codec.cc"
"audio_codecs/box_audio_codec.cc"
"audio_codecs/es8311_audio_codec.cc"
"audio_codecs/es8388_audio_codec.cc"
"audio/simple_pipeline.cc"
"led/single_led.cc"
"led/circular_strip.cc"
"led/gpio_led.cc"
"display/display.cc"
#"display/lcd_display.cc" # 移除LCD显示器支持
#"display/oled_display.cc" # 移除OLED显示器支持
"protocols/protocol.cc"
"iot/thing.cc"
"iot/thing_manager.cc"
"system_info.cc"
"application.cc"
"ota.cc"
"settings.cc"
"background_task.cc"
"bluetooth_provisioning.cc" # 蓝牙配网实现
#"ble_service.cc" # BLE JSON 通讯服务(暂不使用,保留代码)
"weather_api.cc"
"main.cc"
)
set(INCLUDE_DIRS "." "display" "audio_codecs" "protocols" "audio_processing" "audio")
# 添加 IOT 相关文件
file(GLOB IOT_SOURCES ${CMAKE_CURRENT_SOURCE_DIR}/iot/things/*.cc)
# 排除 screen.cc 文件,因为这个板子没有显示器
list(REMOVE_ITEM IOT_SOURCES ${CMAKE_CURRENT_SOURCE_DIR}/iot/things/screen.cc)
list(APPEND SOURCES ${IOT_SOURCES})
# 添加板级公共文件
file(GLOB BOARD_COMMON_SOURCES ${CMAKE_CURRENT_SOURCE_DIR}/boards/common/*.cc)
# Exclude ml307_board.cc by default, only include it when ML307 board is selected
list(FILTER BOARD_COMMON_SOURCES EXCLUDE REGEX ".*ml307_board\.cc$")
list(APPEND SOURCES ${BOARD_COMMON_SOURCES})
list(APPEND INCLUDE_DIRS ${CMAKE_CURRENT_SOURCE_DIR}/boards/common)
# Include ml307_board.cc only when ML307 board is selected
if(CONFIG_BOARD_TYPE_BREAD_COMPACT_ML307 OR CONFIG_BOARD_TYPE_XINGZHI_Cube_0_85TFT_ML307 OR CONFIG_BOARD_TYPE_XINGZHI_Cube_0_96OLED_ML307 OR CONFIG_BOARD_TYPE_XINGZHI_Cube_1_54TFT_ML307)
list(APPEND SOURCES ${CMAKE_CURRENT_SOURCE_DIR}/boards/common/ml307_board.cc)
endif()
# 根据 BOARD_TYPE 配置添加对应的板级文件
if(CONFIG_BOARD_TYPE_BREAD_COMPACT_WIFI)
set(BOARD_TYPE "bread-compact-wifi")
elseif(CONFIG_BOARD_TYPE_BREAD_COMPACT_ML307)
set(BOARD_TYPE "bread-compact-ml307")
elseif(CONFIG_BOARD_TYPE_BREAD_COMPACT_ESP32)
set(BOARD_TYPE "bread-compact-esp32")
elseif(CONFIG_BOARD_TYPE_BREAD_COMPACT_ESP32_LCD)
set(BOARD_TYPE "bread-compact-esp32-lcd")
elseif(CONFIG_BOARD_TYPE_DF_K10)
set(BOARD_TYPE "df-k10")
elseif(CONFIG_BOARD_TYPE_ESP_BOX_3)
set(BOARD_TYPE "esp-box-3")
elseif(CONFIG_BOARD_TYPE_ESP_BOX)
set(BOARD_TYPE "esp-box")
elseif(CONFIG_BOARD_TYPE_ESP_BOX_LITE)
set(BOARD_TYPE "esp-box-lite")
elseif(CONFIG_BOARD_TYPE_KEVIN_BOX_1)
set(BOARD_TYPE "kevin-box-1")
elseif(CONFIG_BOARD_TYPE_KEVIN_BOX_2)
set(BOARD_TYPE "kevin-box-2")
elseif(CONFIG_BOARD_TYPE_KEVIN_C3)
set(BOARD_TYPE "kevin-c3")
elseif(CONFIG_BOARD_TYPE_KEVIN_SP_V3_DEV)
set(BOARD_TYPE "kevin-sp-v3-dev")
elseif(CONFIG_BOARD_TYPE_KEVIN_SP_V4_DEV)
set(BOARD_TYPE "kevin-sp-v4-dev")
elseif(CONFIG_BOARD_TYPE_KEVIN_YUYING_313LCD)
set(BOARD_TYPE "kevin-yuying-313lcd")
elseif(CONFIG_BOARD_TYPE_LICHUANG_DEV)
set(BOARD_TYPE "lichuang-dev")
elseif(CONFIG_BOARD_TYPE_LICHUANG_C3_DEV)
set(BOARD_TYPE "lichuang-c3-dev")
elseif(CONFIG_BOARD_TYPE_MAGICLICK_2P4)
set(BOARD_TYPE "magiclick-2p4")
elseif(CONFIG_BOARD_TYPE_MAGICLICK_2P5)
set(BOARD_TYPE "magiclick-2p5")
elseif(CONFIG_BOARD_TYPE_MAGICLICK_C3)
set(BOARD_TYPE "magiclick-c3")
elseif(CONFIG_BOARD_TYPE_MAGICLICK_C3_V2)
set(BOARD_TYPE "magiclick-c3-v2")
elseif(CONFIG_BOARD_TYPE_M5STACK_CORE_S3)
set(BOARD_TYPE "m5stack-core-s3")
elseif(CONFIG_BOARD_TYPE_ATOMS3_ECHO_BASE)
set(BOARD_TYPE "atoms3-echo-base")
elseif(CONFIG_BOARD_TYPE_ATOMS3R_ECHO_BASE)
set(BOARD_TYPE "atoms3r-echo-base")
elseif(CONFIG_BOARD_TYPE_ATOMS3R_CAM_M12_ECHO_BASE)
set(BOARD_TYPE "atoms3r-cam-m12-echo-base")
elseif(CONFIG_BOARD_TYPE_ATOMMATRIX_ECHO_BASE)
set(BOARD_TYPE "atommatrix-echo-base")
elseif(CONFIG_BOARD_TYPE_XMINI_C3)
set(BOARD_TYPE "xmini-c3")
elseif(CONFIG_BOARD_TYPE_ESP32S3_KORVO2_V3)
set(BOARD_TYPE "esp32s3-korvo2-v3")
elseif(CONFIG_BOARD_TYPE_ESP_SPARKBOT)
set(BOARD_TYPE "esp-sparkbot")
elseif(CONFIG_BOARD_TYPE_ESP32S3_Touch_AMOLED_1_8)
set(BOARD_TYPE "esp32-s3-touch-amoled-1.8")
elseif(CONFIG_BOARD_TYPE_ESP32S3_Touch_LCD_1_85C)
set(BOARD_TYPE "esp32-s3-touch-lcd-1.85c")
elseif(CONFIG_BOARD_TYPE_ESP32S3_Touch_LCD_1_85)
set(BOARD_TYPE "esp32-s3-touch-lcd-1.85")
elseif(CONFIG_BOARD_TYPE_ESP32S3_Touch_LCD_1_46)
set(BOARD_TYPE "esp32-s3-touch-lcd-1.46")
elseif(CONFIG_BOARD_TYPE_ESP32S3_Touch_LCD_3_5)
set(BOARD_TYPE "esp32-s3-touch-lcd-3.5")
elseif(CONFIG_BOARD_TYPE_BREAD_COMPACT_WIFI_LCD)
set(BOARD_TYPE "bread-compact-wifi-lcd")
elseif(CONFIG_BOARD_TYPE_TUDOUZI)
set(BOARD_TYPE "tudouzi")
elseif(CONFIG_BOARD_TYPE_LILYGO_T_CIRCLE_S3)
set(BOARD_TYPE "lilygo-t-circle-s3")
elseif(CONFIG_BOARD_TYPE_LILYGO_T_CAMERAPLUS_S3)
set(BOARD_TYPE "lilygo-t-cameraplus-s3")
elseif(CONFIG_BOARD_TYPE_MOVECALL_MOJI_ESP32S3)
set(BOARD_TYPE "movecall-moji-esp32s3")
elseif(CONFIG_BOARD_TYPE_MOVECALL_CUICAN_ESP32S3)
set(BOARD_TYPE "movecall-cuican-esp32s3")
elseif(CONFIG_BOARD_TYPE_ATK_DNESP32S3)
set(BOARD_TYPE "atk-dnesp32s3")
elseif(CONFIG_BOARD_TYPE_ATK_DNESP32S3_BOX)
set(BOARD_TYPE "atk-dnesp32s3-box")
elseif(CONFIG_BOARD_TYPE_DU_CHATX)
set(BOARD_TYPE "du-chatx")
elseif(CONFIG_BOARD_TYPE_ESP32S3_Taiji_Pi)
set(BOARD_TYPE "taiji-pi-s3")
elseif(CONFIG_BOARD_TYPE_XINGZHI_Cube_0_85TFT_WIFI)
set(BOARD_TYPE "xingzhi-cube-0.85tft-wifi")
elseif(CONFIG_BOARD_TYPE_XINGZHI_Cube_0_85TFT_ML307)
set(BOARD_TYPE "xingzhi-cube-0.85tft-ml307")
elseif(CONFIG_BOARD_TYPE_XINGZHI_Cube_0_96OLED_WIFI)
set(BOARD_TYPE "xingzhi-cube-0.96oled-wifi")
elseif(CONFIG_BOARD_TYPE_XINGZHI_Cube_0_96OLED_ML307)
set(BOARD_TYPE "xingzhi-cube-0.96oled-ml307")
elseif(CONFIG_BOARD_TYPE_XINGZHI_Cube_1_54TFT_WIFI)
set(BOARD_TYPE "xingzhi-cube-1.54tft-wifi")
elseif(CONFIG_BOARD_TYPE_XINGZHI_Cube_1_54TFT_ML307)
set(BOARD_TYPE "xingzhi-cube-1.54tft-ml307")
elseif(CONFIG_BOARD_TYPE_SENSECAP_WATCHER)
set(BOARD_TYPE "sensecap-watcher")
elseif(CONFIG_BOARD_TYPE_ESP32_CGC)
set(BOARD_TYPE "esp32-cgc")
endif()
file(GLOB BOARD_SOURCES
${CMAKE_CURRENT_SOURCE_DIR}/boards/${BOARD_TYPE}/*.cc
${CMAKE_CURRENT_SOURCE_DIR}/boards/${BOARD_TYPE}/*.c
)
list(APPEND SOURCES ${BOARD_SOURCES})
if(CONFIG_CONNECTION_TYPE_MQTT_UDP)
list(APPEND SOURCES "protocols/mqtt_protocol.cc")
endif()
if(CONFIG_CONNECTION_TYPE_WEBSOCKET)
list(APPEND SOURCES "protocols/websocket_protocol.cc")
endif()
if(CONFIG_CONNECTION_TYPE_VOLC_RTC)
list(APPEND SOURCES "protocols/volc_rtc_protocol.cc")
endif()
if(CONFIG_USE_AUDIO_PROCESSOR)
list(APPEND SOURCES "audio_processing/audio_processor.cc")
endif()
if(CONFIG_USE_WAKE_WORD_DETECT)
list(APPEND SOURCES "audio_processing/wake_word_detect.cc")
elseif(CONFIG_USE_CUSTOM_WAKE_WORD)
list(APPEND SOURCES "audio_processing/custom_wake_word.cc")
endif()
# 根据Kconfig选择语言目录
if(CONFIG_LANGUAGE_ZH_CN)
set(LANG_DIR "zh-CN")
elseif(CONFIG_LANGUAGE_ZH_TW)
set(LANG_DIR "zh-TW")
elseif(CONFIG_LANGUAGE_EN_US)
set(LANG_DIR "en-US")
elseif(CONFIG_LANGUAGE_JA_JP)
set(LANG_DIR "ja-JP")
endif()
# 定义生成路径
set(LANG_JSON "${CMAKE_CURRENT_SOURCE_DIR}/assets/${LANG_DIR}/language.json")
set(LANG_HEADER "${CMAKE_CURRENT_SOURCE_DIR}/assets/lang_config.h")
file(GLOB LANG_SOUNDS ${CMAKE_CURRENT_SOURCE_DIR}/assets/${LANG_DIR}/*.p3)
file(GLOB COMMON_SOUNDS ${CMAKE_CURRENT_SOURCE_DIR}/assets/common/*.p3)
# 如果目标芯片是 ESP32则排除特定文件
if(CONFIG_IDF_TARGET_ESP32)
list(REMOVE_ITEM SOURCES "audio_codecs/box_audio_codec.cc"
"audio_codecs/es8388_audio_codec.cc"
"led/gpio_led.cc"
)
endif()
idf_component_register(SRCS ${SOURCES}
EMBED_FILES ${LANG_SOUNDS} ${COMMON_SOUNDS}
INCLUDE_DIRS ${INCLUDE_DIRS}
# 路径 D'' AEC: esp-sr 提供 esp_aec.h 底层同步 API (aec_create/aec_process/aec_destroy)
# 配合软件 loopback ref (DAC PCM copy 到 ring buffer) 实现设备端 AEC
REQUIRES esp_wifi esp_netif esp_event nvs_flash bt spi_flash app_update efuse volc_engine_rtc_lite common zlib esp-sr
WHOLE_ARCHIVE
)
# 使用 target_compile_definitions 来定义 BOARD_TYPE, BOARD_NAME
# 如果 BOARD_NAME 为空,则使用 BOARD_TYPE
if(NOT BOARD_NAME)
set(BOARD_NAME ${BOARD_TYPE})
endif()
target_compile_definitions(${COMPONENT_LIB}
PRIVATE BOARD_TYPE=\"${BOARD_TYPE}\" BOARD_NAME=\"${BOARD_NAME}\"
)
# 添加生成规则
add_custom_command(
OUTPUT ${LANG_HEADER}
COMMAND python ${PROJECT_DIR}/scripts/gen_lang.py
--input "${LANG_JSON}"
--output "${LANG_HEADER}"
DEPENDS
${LANG_JSON}
${PROJECT_DIR}/scripts/gen_lang.py
COMMENT "Generating ${LANG_DIR} language config"
)
# 强制建立生成依赖
add_custom_target(lang_header ALL
DEPENDS ${LANG_HEADER}
)
# Add ENABLE_RTC_MODE definition if VOLC_RTC connection type is selected
if(CONFIG_CONNECTION_TYPE_VOLC_RTC)
target_compile_definitions(${COMPONENT_LIB} PRIVATE ENABLE_RTC_MODE)
# Link against zlib library directly
target_link_libraries(${COMPONENT_LIB} PRIVATE ${CMAKE_BINARY_DIR}/esp-idf/zlib/libzlib.a)
endif()