commit 128f7ca02ef75350f5338aa09a431ef7d839bc89 Author: Rdzleo Date: Thu Apr 30 09:55:49 2026 +0800 init: KWS 软件唤醒 APK 工程骨架（基于 sherpa-onnx 改造） 工程结构: - com.lila.wakeup 包名,基于 sherpa-onnx v1.13.0 SherpaOnnxKws demo 改造 - 9 个 Kotlin 模块: WakeupForegroundService / KwsStateMachine / AudioCapture / KwsEngine / BootReceiver / ProtocolReceiver / BroadcastSender / Config / MainActivity - 5 个 Lila 中文唤醒词(你好Lila/hello Lila/Lila同学/Lila你好/小Lila) - 仅 arm64-v8a(OrangePi CM5 / RK3588) - 协议 v2.1: 双向 setPackage / silence_ms 3000 / 2min 兜底 - 数字人 APP 包名: com.qy.lila 构建通过项: - Gradle Sync OK(腾讯云 Gradle 镜像 + 阿里云 Maven 镜像) - APK 编译成功 55MB,含 25MB onnxruntime + 24MB 模型 + 9 个 Kotlin .dex - adb install OK,Service onCreate 被调用,sherpa-onnx 加载模型 已知问题(下一 commit 修复): - KwsEngine.kt 的 OnlineModelConfig modelType 设为 'zipformer' 实际应为 'zipformer2',native 加载在 InitEncoder 后崩溃 - 修复方法: 用 sherpa-onnx 官方 getKwsModelConfig(0) 工厂函数 参考文档: - 协议: ~/OrangePi_CM5_Project/docs/.../KWS唤醒协议-V2.md (v2.1) - 设计: ~/OrangePi_CM5_Project/docs/.../KWS服务设计.md - 评估: ~/OrangePi_CM5_Project/docs/.../KWS唤醒方案适配评估_Unity.md diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..e2151f9 --- /dev/null +++ b/.gitignore @@ -0,0 +1,17 @@ +*.iml +.gradle +/local.properties +/.idea/caches +/.idea/libraries +/.idea/modules.xml +/.idea/workspace.xml +/.idea/navEditor.xml +/.idea/assetWizardSettings.xml +.DS_Store +/build +/captures +.externalNativeBuild +.cxx +local.properties +*.bak +*.original diff --git a/.idea/.gitignore b/.idea/.gitignore new file mode 100644 index 0000000..26d3352 --- /dev/null +++ b/.idea/.gitignore @@ -0,0 +1,3 @@ +# Default ignored files +/shelf/ +/workspace.xml diff --git a/.idea/AndroidProjectSystem.xml b/.idea/AndroidProjectSystem.xml new file mode 100644 index 0000000..4a53bee --- /dev/null +++ b/.idea/AndroidProjectSystem.xml @@ -0,0 +1,6 @@ + + + + + \ No newline at end of file diff --git a/.idea/codeStyles/Project.xml b/.idea/codeStyles/Project.xml new file mode 100644 index 0000000..7643783 --- /dev/null +++ b/.idea/codeStyles/Project.xml @@ -0,0 +1,123 @@ + + + + + + + + + + \ No newline at end of file diff --git a/.idea/codeStyles/codeStyleConfig.xml b/.idea/codeStyles/codeStyleConfig.xml new file mode 100644 index 0000000..79ee123 --- /dev/null +++ b/.idea/codeStyles/codeStyleConfig.xml @@ -0,0 +1,5 @@ + + + + \ No newline at end of file diff --git a/.idea/compiler.xml b/.idea/compiler.xml new file mode 100644 index 0000000..b589d56 --- /dev/null +++ b/.idea/compiler.xml @@ -0,0 +1,6 @@ + + + + + + \ No newline at end of file diff --git a/.idea/deploymentTargetSelector.xml b/.idea/deploymentTargetSelector.xml new file mode 100644 index 0000000..ca16a99 --- /dev/null +++ b/.idea/deploymentTargetSelector.xml @@ -0,0 +1,11 @@ + + + + + + + + + \ No newline at end of file diff --git a/.idea/gradle.xml b/.idea/gradle.xml new file mode 100644 index 0000000..639c779 --- /dev/null +++ b/.idea/gradle.xml @@ -0,0 +1,19 @@ + + + + + + + \ No newline at end of file diff --git a/.idea/migrations.xml b/.idea/migrations.xml new file mode 100644 index 0000000..f8051a6 --- /dev/null +++ b/.idea/migrations.xml @@ -0,0 +1,10 @@ + + + + + + \ No newline at end of file diff --git a/.idea/misc.xml b/.idea/misc.xml new file mode 100644 index 0000000..8978d23 --- /dev/null +++ b/.idea/misc.xml @@ -0,0 +1,9 @@ + + + + + + + + \ No newline at end of file diff --git a/.idea/runConfigurations.xml b/.idea/runConfigurations.xml new file mode 100644 index 0000000..16660f1 --- /dev/null +++ b/.idea/runConfigurations.xml @@ -0,0 +1,17 @@ + + + + + + \ No newline at end of file diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..f8383e3 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,224 @@ +# Lila Wakeup APK 工程上下文（com.lila.wakeup） + +> **本文件用于 Claude Code 插件加载，自动获取项目背景。** +> 本工程是 OrangePi CM5 数字人产品的 KWS 软件唤醒服务（独立 APK），与第三方 Unity 数字人 APP（包名 `com.qy.lila`）通过 Android Broadcast 协议联动。 + +--- + +## 一、项目本质（一句话） + +基于 **sherpa-onnx** 引擎的 Android 13 KWS（Keyword Spotting / 关键词唤醒）APK，监听用户喊"你好 Lila"等唤醒词后通过 Broadcast 通知数字人 APP 显示数字人 + 接通火山 RTC。 + +替代原方案的"ESP32 + 离线语音芯片 + USB 串口 SO_WAKEUP1"硬件方案，省 BOM 成本 + 更灵活。 + +--- + +## 二、核心元信息（写代码前必读） + +| 项 | 值 | +|---|---| +| 本 APK 包名 | `com.lila.wakeup` | +| 数字人 APP 包名 | `com.qy.lila`（已确认，来源 LTY_Project ProjectSettings.asset） | +| 目标系统 | Android 13 (API 33) | +| 目标架构 | arm64-v8a（仅打包此架构，节省 APK 体积） | +| 目标设备 | OrangePi CM5（Rockchip RK3588，rk3588s_s） | +| 协议版本 | **v2.1**（详见上游 `docs/OrangePi_CM5/MD_Document/KWS唤醒协议-V2.md`） | +| KWS 引擎 | sherpa-onnx Zipformer wenetspeech 3.3M（中文，已预编译 native）| +| 集成方式 | **基于 sherpa-onnx 官方 SherpaOnnxKws demo 改造**（不是 Maven 依赖，因 Android arm64-v8a .so + Kotlin Wrapper 在 Maven 上没发布完整包） | +| 当前阶段 | **方案 A（普通 APK）**（量产前升级到方案 B：系统签名 + /system/priv-app/）| + +--- + +## 三、协议规约（v2.1 核心摘要） + +### 三个 Broadcast Action（双向 setPackage 强制要求） + +| 方向 | Action | 字段 | +|---|---|---| +| **WAKEUP**（本 → APP）| `com.lila.intent.action.WAKEUP` | `keyword: String` / `timestamp: long` / `confidence: float` | +| **KWS_PAUSE**（APP → 本）| `com.lila.intent.action.KWS_PAUSE` | `silence_ms: long`（默认 **3000**，Networking 场景 60000）/ `reason: String` | +| **KWS_RESUME**（APP → 本）| `com.lila.intent.action.KWS_RESUME` | `reason: String` | + +### 关键设计约束（不要破坏） + +1. **silence_ms 二次保险**：收到 PAUSE 后即使收到 RESUME 也要等到 silence 满才真正 resume —— 防 AI 自我唤醒 +2. **2 分钟兜底超时**：APP 漏发 RESUME 时 KWS 自动恢复（防永久哑火） +3. **后验平滑**：连续 N 帧（默认 2）置信度过阈值才发 WAKEUP —— 降误唤醒 +4. **AudioRecord 暂停时完全 release**：把麦克风让给 RTC SDK，避免资源冲突 +5. **WAKEUP 命中后不自动 PAUSE 自己**：必须由 APP 主动发 PAUSE —— 防漏命中导致的死锁 + +--- + +## 四、工程结构（基于 sherpa-onnx demo 改造后） + +``` +工程根/（原 sherpa-onnx/android/SherpaOnnxKws） +├── CLAUDE.md ← 本文件 +├── app/ +│ ├── build.gradle.kts ← 已改：包名 + arm64-v8a only + targetSdk 33 +│ └── src/main/ +│ ├── AndroidManifest.xml ← 已改：完整权限 + 4 个组件声明 +│ ├── assets/ +│ │ └── sherpa-onnx-kws-zipformer-wenetspeech-3.3M-2024-01-01/ +│ │ ├── *.onnx ← 原 demo 自带（不动） +│ │ ├── tokens.txt ← 原 demo 自带（不动） +│ │ └── keywords.txt ← 已改：5 个 Lila 唤醒词 +│ └── java/com/lila/wakeup/ ← 已改：包名 + 9 个新文件 +│ ├── Config.kt ← 协议常量、阈值、模型路径 +│ ├── MainActivity.kt ← 状态查看 UI + 权限申请（开发期辅助） +│ ├── BootReceiver.kt ← 开机自启 +│ ├── WakeupForegroundService.kt ← 前台 Service 主体 +│ ├── kws/ +│ │ ├── AudioCapture.kt ← AudioRecord 16kHz mono PCM16 +│ │ ├── KwsEngine.kt ← sherpa-onnx KWS 封装 +│ │ └── KwsStateMachine.kt ← LISTENING/PAUSED 状态机 + silence_ms +│ ├── protocol/ +│ │ ├── BroadcastSender.kt ← 发 WAKEUP 给 com.qy.lila +│ │ └── ProtocolReceiver.kt ← 收 PAUSE/RESUME +│ └── （sherpa-onnx 自带的 Wrapper 类保留，包名 com.k2fsa.sherpa.onnx 不动） +└── build.gradle.kts (项目级) ← 不动 +``` + +### 9 个我们写的 Kotlin 文件代码量 + +| 文件 | 行数 | 职责 | +|---|---|---| +| Config.kt | 108 | 全局常量 | +| MainActivity.kt | 117 | 状态 UI + 权限申请 | +| BootReceiver.kt | 50 | 开机自启 | +| WakeupForegroundService.kt | 142 | Service 主体 + 通知栏 | +| AudioCapture.kt | 112 | AudioRecord 封装 | +| KwsEngine.kt | 119 | sherpa-onnx 封装 | +| KwsStateMachine.kt | 163 | 状态机 + silence_ms + 兜底 | +| BroadcastSender.kt | 36 | 发 WAKEUP | +| ProtocolReceiver.kt | 63 | 收 PAUSE/RESUME | + +--- + +## 五、关键参考文档（在主仓库 OrangePi_CM5_Project 中） + +> **这些文档是工程的"上游契约"**，本工程实现必须与之一致。改协议要先去主仓库改文档。 + +| 文档 | 路径 | 作用 | +|---|---|---| +| 协议规约 v2.1 | `/Volumes/LinuxDev/OrangePi_CM5_Project/docs/OrangePi_CM5/MD_Document/KWS唤醒协议-V2.md` | 双方对接契约（最权威）| +| 服务设计 | `/Volumes/LinuxDev/OrangePi_CM5_Project/docs/OrangePi_CM5/MD_Document/KWS服务设计.md` | 我侧实现指南 | +| 第三方评估 | `/Volumes/LinuxDev/OrangePi_CM5_Project/docs/OrangePi_CM5/MD_Document/KWS唤醒方案适配评估_Unity.md` | Unity APP 团队的对接评估（含 v2 微调建议）| +| 工程骨架 README | `/Volumes/LinuxDev/OrangePi_CM5_Project/docs/OrangePi_CM5/MD_Document/KWS-APK-工程骨架/README.md` | 9 步实施指南（含路线图、坑提示）| + +--- + +## 六、用户偏好与代码风格（继承自主项目） + +- **语言**：所有沟通用中文，代码注释用中文 +- **代码风格**：snake_case 变量、UPPER_SNAKE_CASE 常量、_t 后缀类型；Kotlin 这边按 Kotlin 标准（CamelCase 类名 / camelCase 变量），但**注释一律中文** +- **嵌入式原则**：节省 RAM/Flash > 时序优化 > 模块化（虽然这是 Android 但 OrangePi 资源仍宝贵） +- **不主动加冗余校验**（CLAUDE.md 全局规则） +- **破坏性操作先确认**（git push --force / rm 等） +- **优先静态分配**：避免频繁 malloc/free，字符串处理用固定缓冲区 + +--- + +## 七、当前任务路线图 + +| 阶段 | 内容 | 状态 | +|---|---|---| +| 0. 算法验证 | sherpa-onnx 预编译 KWS APK 装到板子，测识别率 | ✅ 已完成（5/8 命中）| +| 1. 工程骨架 | 9 个 Kotlin 类 + Manifest + gradle | ✅ 已完成（在主仓库 KWS-APK-工程骨架/）| +| 2. AS 装齐组件 | API 33 / Build-Tools 33 / NDK 25 / CMake 3.22 | 🔄 进行中 | +| 3. 克隆 sherpa-onnx + 改造 | git clone + Refactor 包名 + 拷骨架文件 | 📋 待做 | +| 4. Build → adb install 到 OrangePi CM5 | 验证唤醒效果 | 📋 待做 | +| 5. 协议联调（与 Unity APP）| 按协议 v2.1 §八 联调清单 | 📋 待 APP 团队 | +| 6. 方案 B 升级 | 系统签名 + 集成进 update.img | 📋 联调通过后 | +| 7. 长稳测试 | 72h 连续运行 | 📋 量产前 | + +--- + +## 八、关键技术决策记录（不要重新讨论） + +| 决策 | 结论 | 理由（一句话）| +|---|---|---| +| 用 Maven 依赖还是克隆 demo？ | **克隆 demo** | Maven 上 sherpa-onnx 的 Android arm64-v8a .so + Kotlin Wrapper 不全 | +| 方案 A / B / C 怎么走？ | **A 跑通 → B 升级**（不走 C）| C 改 AOSP system_server 收益不抵成本 | +| silence_ms 默认值 | **3000ms** | 第三方实测 RTC 远端音频回灌 800-1500ms + AI TTS 触发，2000ms 不够 | +| Networking 场景是否唤醒？ | **不允许**（APP 进入发 PAUSE）| 配网中被唤醒会要求 APP 新增响应逻辑 + 决策跳哪里 | +| BroadcastReceiver 与 Service 通信方式 | **Locator 模式** | 比 startService / AIDL 简单，性能好 | +| AudioRecord 暂停时是 stop 还是 release？ | **完全 release** | 释放麦克风给 RTC SDK，避免资源冲突 | + +--- + +## 九、绝对不要做的事 + +1. ❌ **不要改包名 `com.qy.lila`**：APP 团队已确认，改了协议失效 +2. ❌ **不要改 Action 名 `com.lila.intent.action.*`**：双方协议契约 +3. ❌ **不要把 sherpa-onnx 源码包名 `com.k2fsa.sherpa.onnx` 改掉**：JNI native 库 binding 的硬编码包名 +4. ❌ **不要在前台 Service 里调耗时操作（vTaskDelay 类）阻塞 main looper**：StateMachine 的 timeout 用 Handler.postDelayed +5. ❌ **不要在 AudioCapture 线程里同步调网络 / 文件 IO**：会丢音频帧 +6. ❌ **不要禁用 START_STICKY**：Service 异常重启依赖此标志 +7. ❌ **不要把 RECORD_AUDIO 申请放在 Service**：必须在 Activity 里申请（Android 限制） + +--- + +## 十、与主项目（OrangePi_CM5_Project）的协作 + +主项目的 Claude Code 会话负责： +- 协议规约文档（KWS唤醒协议-V2.md）演进 +- 服务设计文档（KWS服务设计.md）演进 +- 与第三方 APP 团队对接 +- 系统侧（kernel / dts / Audio HAL）相关修改 +- 整机集成（方案 B 升级） + +本工程（sherpa-onnx 改造）的 Claude Code 会话负责： +- 9 个 Kotlin 文件的具体实现 +- Gradle / Manifest 配置 +- Android Studio Build / Run 调试 +- KWS 引擎参数调优（threshold、N 帧平滑） +- 真机测试（与 OrangePi CM5 联调） + +**协议改动必须先在主项目改文档，再同步到本工程。** + +--- + +## 十一、典型工作请求示例（让 Claude Code 上手） + +新来 Claude Code 插件可以处理的典型任务： + +- "帮我改 [`KwsStateMachine.kt`](app/src/main/java/com/lila/wakeup/kws/KwsStateMachine.kt) 的后验平滑窗口从 2 帧改为 3 帧" +- "把 [`build.gradle.kts`](app/build.gradle.kts) 的 minSdk 从 26 改为 24" +- "新增一个 keywords 重载接口，让 [`KwsEngine.kt`](app/src/main/java/com/lila/wakeup/kws/KwsEngine.kt) 支持热更新唤醒词" +- "[`MainActivity.kt`](app/src/main/java/com/lila/wakeup/MainActivity.kt) 状态展示太丑，帮我用 Compose 重写" +- "Build 报错 `Unresolved reference: KeywordSpotter`，怎么改？" +- "帮我加一个 logcat 过滤器，只看 KWS 命中日志" + +需要主项目协调的请求（**不要自己改本工程**）： + +- "改协议字段名" → 让用户回主项目讨论 +- "升级方案 B" → 涉及 SDK 集成，回主项目 +- "对接第三方新需求" → 协议要先改 + +--- + +## 十二、运行时验证 + +```bash +# 装到板子 +adb install -r -g ./app/build/outputs/apk/debug/app-debug.apk + +# 看日志 +adb logcat -s KwsService.Svc:* KwsService.Engine:* KwsService.State:* KwsService.Audio:* KwsService.Send:* KwsService.Proto:* KwsService.Boot:* + +# 关键日志关键字 +[Engine] init done ← 引擎加载成功 +[State] -> LISTENING ← 进入监听 +[KWS] HIT confirmed ← 命中 +[Broadcast] WAKEUP -> com.qy.lila ← 发广播给 APP + +# 测试 APP 端 PAUSE/RESUME（不依赖 Unity APP，用 adb 模拟） +adb shell am broadcast -a com.lila.intent.action.KWS_PAUSE \ + -n com.lila.wakeup/.protocol.ProtocolReceiver \ + --el silence_ms 3000 --es reason "test_pause" + +adb shell am broadcast -a com.lila.intent.action.KWS_RESUME \ + -n com.lila.wakeup/.protocol.ProtocolReceiver \ + --es reason "test_resume" +``` diff --git a/app/.gitignore b/app/.gitignore new file mode 100644 index 0000000..42afabf --- /dev/null +++ b/app/.gitignore @@ -0,0 +1 @@ +/build \ No newline at end of file diff --git a/app/build.gradle b/app/build.gradle new file mode 100644 index 0000000..1b8b214 --- /dev/null +++ b/app/build.gradle @@ -0,0 +1,45 @@ +plugins { + id 'com.android.application' + id 'org.jetbrains.kotlin.android' +} + +android { + namespace 'com.lila.wakeup' + compileSdk 33 + + defaultConfig { + applicationId "com.lila.wakeup" + minSdk 26 + targetSdk 33 + versionCode 1 + versionName "0.1.0-alpha" + + testInstrumentationRunner "androidx.test.runner.AndroidJUnitRunner" + ndk { abiFilters "arm64-v8a" } + } + + buildTypes { + release { + minifyEnabled false + proguardFiles getDefaultProguardFile('proguard-android-optimize.txt'), 'proguard-rules.pro' + } + } + compileOptions { + sourceCompatibility JavaVersion.VERSION_1_8 + targetCompatibility JavaVersion.VERSION_1_8 + } + kotlinOptions { + jvmTarget = '1.8' + } +} + +dependencies { + + implementation 'androidx.core:core-ktx:1.7.0' + implementation 'androidx.appcompat:appcompat:1.5.1' + implementation 'com.google.android.material:material:1.7.0' + implementation 'androidx.constraintlayout:constraintlayout:2.1.4' + testImplementation 'junit:junit:4.13.2' + androidTestImplementation 'androidx.test.ext:junit:1.1.4' + androidTestImplementation 'androidx.test.espresso:espresso-core:3.5.0' +} \ No newline at end of file diff --git a/app/proguard-rules.pro b/app/proguard-rules.pro new file mode 100644 index 0000000..481bb43 --- /dev/null +++ b/app/proguard-rules.pro @@ -0,0 +1,21 @@ +# Add project specific ProGuard rules here. +# You can control the set of applied configuration files using the +# proguardFiles setting in build.gradle. +# +# For more details, see +# http://developer.android.com/guide/developing/tools/proguard.html + +# If your project uses WebView with JS, uncomment the following +# and specify the fully qualified class name to the JavaScript interface +# class: +#-keepclassmembers class fqcn.of.javascript.interface.for.webview { +# public *; +#} + +# Uncomment this to preserve the line number information for +# debugging stack traces. +#-keepattributes SourceFile,LineNumberTable + +# If you keep the line number information, uncomment this to +# hide the original source file name. +#-renamesourcefileattribute SourceFile \ No newline at end of file diff --git a/app/src/androidTest/java/com/k2fsa/sherpa/onnx/ExampleInstrumentedTest.kt b/app/src/androidTest/java/com/k2fsa/sherpa/onnx/ExampleInstrumentedTest.kt new file mode 100644 index 0000000..1833832 --- /dev/null +++ b/app/src/androidTest/java/com/k2fsa/sherpa/onnx/ExampleInstrumentedTest.kt @@ -0,0 +1,24 @@ +package com.k2fsa.sherpa.onnx + +import androidx.test.platform.app.InstrumentationRegistry +import androidx.test.ext.junit.runners.AndroidJUnit4 + +import org.junit.Test +import org.junit.runner.RunWith + +import org.junit.Assert.* + +/** + * Instrumented test, which will execute on an Android device. + * + * See [testing documentation](http://d.android.com/tools/testing). + */ +@RunWith(AndroidJUnit4::class) +class ExampleInstrumentedTest { + @Test + fun useAppContext() { + // Context of the app under test. + val appContext = InstrumentationRegistry.getInstrumentation().targetContext + assertEquals("com.k2fsa.sherpa.onnx", appContext.packageName) + } +} \ No newline at end of file diff --git a/app/src/main/AndroidManifest.xml b/app/src/main/AndroidManifest.xml new file mode 100644 index 0000000..e779919 --- /dev/null +++ b/app/src/main/AndroidManifest.xml @@ -0,0 +1,85 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/app/src/main/assets/.gitkeep b/app/src/main/assets/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/app/src/main/assets/sherpa-onnx-kws-zipformer-wenetspeech-3.3M-2024-01-01/decoder-epoch-12-avg-2-chunk-16-left-64.onnx b/app/src/main/assets/sherpa-onnx-kws-zipformer-wenetspeech-3.3M-2024-01-01/decoder-epoch-12-avg-2-chunk-16-left-64.onnx new file mode 100644 index 0000000..1d78310 Binary files /dev/null and b/app/src/main/assets/sherpa-onnx-kws-zipformer-wenetspeech-3.3M-2024-01-01/decoder-epoch-12-avg-2-chunk-16-left-64.onnx differ diff --git a/app/src/main/assets/sherpa-onnx-kws-zipformer-wenetspeech-3.3M-2024-01-01/decoder-epoch-99-avg-1-chunk-16-left-64.onnx b/app/src/main/assets/sherpa-onnx-kws-zipformer-wenetspeech-3.3M-2024-01-01/decoder-epoch-99-avg-1-chunk-16-left-64.onnx new file mode 100644 index 0000000..1d78310 Binary files /dev/null and b/app/src/main/assets/sherpa-onnx-kws-zipformer-wenetspeech-3.3M-2024-01-01/decoder-epoch-99-avg-1-chunk-16-left-64.onnx differ diff --git a/app/src/main/assets/sherpa-onnx-kws-zipformer-wenetspeech-3.3M-2024-01-01/encoder-epoch-12-avg-2-chunk-16-left-64.onnx b/app/src/main/assets/sherpa-onnx-kws-zipformer-wenetspeech-3.3M-2024-01-01/encoder-epoch-12-avg-2-chunk-16-left-64.onnx new file mode 100644 index 0000000..9d0beb3 Binary files /dev/null and b/app/src/main/assets/sherpa-onnx-kws-zipformer-wenetspeech-3.3M-2024-01-01/encoder-epoch-12-avg-2-chunk-16-left-64.onnx differ diff --git a/app/src/main/assets/sherpa-onnx-kws-zipformer-wenetspeech-3.3M-2024-01-01/encoder-epoch-99-avg-1-chunk-16-left-64.onnx b/app/src/main/assets/sherpa-onnx-kws-zipformer-wenetspeech-3.3M-2024-01-01/encoder-epoch-99-avg-1-chunk-16-left-64.onnx new file mode 100644 index 0000000..9d0beb3 Binary files /dev/null and b/app/src/main/assets/sherpa-onnx-kws-zipformer-wenetspeech-3.3M-2024-01-01/encoder-epoch-99-avg-1-chunk-16-left-64.onnx differ diff --git a/app/src/main/assets/sherpa-onnx-kws-zipformer-wenetspeech-3.3M-2024-01-01/joiner-epoch-12-avg-2-chunk-16-left-64.onnx b/app/src/main/assets/sherpa-onnx-kws-zipformer-wenetspeech-3.3M-2024-01-01/joiner-epoch-12-avg-2-chunk-16-left-64.onnx new file mode 100644 index 0000000..5687aef Binary files /dev/null and b/app/src/main/assets/sherpa-onnx-kws-zipformer-wenetspeech-3.3M-2024-01-01/joiner-epoch-12-avg-2-chunk-16-left-64.onnx differ diff --git a/app/src/main/assets/sherpa-onnx-kws-zipformer-wenetspeech-3.3M-2024-01-01/joiner-epoch-99-avg-1-chunk-16-left-64.onnx b/app/src/main/assets/sherpa-onnx-kws-zipformer-wenetspeech-3.3M-2024-01-01/joiner-epoch-99-avg-1-chunk-16-left-64.onnx new file mode 100644 index 0000000..5687aef Binary files /dev/null and b/app/src/main/assets/sherpa-onnx-kws-zipformer-wenetspeech-3.3M-2024-01-01/joiner-epoch-99-avg-1-chunk-16-left-64.onnx differ diff --git a/app/src/main/assets/sherpa-onnx-kws-zipformer-wenetspeech-3.3M-2024-01-01/keywords.txt b/app/src/main/assets/sherpa-onnx-kws-zipformer-wenetspeech-3.3M-2024-01-01/keywords.txt new file mode 100644 index 0000000..d296696 --- /dev/null +++ b/app/src/main/assets/sherpa-onnx-kws-zipformer-wenetspeech-3.3M-2024-01-01/keywords.txt @@ -0,0 +1,5 @@ +n ǐ h ǎo l ì l ā @你好Lila +h è l ōu l ì l ā @hello Lila +l ì l ā t óng x ué @Lila同学 +l ì l ā n ǐ h ǎo @Lila你好 +x i ǎo l ì l ā @小Lila diff --git a/app/src/main/assets/sherpa-onnx-kws-zipformer-wenetspeech-3.3M-2024-01-01/tokens.txt b/app/src/main/assets/sherpa-onnx-kws-zipformer-wenetspeech-3.3M-2024-01-01/tokens.txt new file mode 100644 index 0000000..5880fbf --- /dev/null +++ b/app/src/main/assets/sherpa-onnx-kws-zipformer-wenetspeech-3.3M-2024-01-01/tokens.txt @@ -0,0 +1,227 @@ + 0 + 1 + 2 +A 3 +B 4 +S 5 +I 6 +f 7 +ù 8 +ǔ 9 +zh 10 +y 11 +ī 12 +sh 13 +ēng 14 +uè 15 +p 16 +iàn 17 +K 18 +L 19 +P 20 +R 21 +T 22 +M 23 +j 24 +W 25 +q 26 +s 27 +ān 28 +b 29 +ā 30 +l 31 +íng 32 +iǔ 33 +ch 34 +ǐ 35 +ì 36 +èr 37 +w 38 +r 39 +án 40 +én 41 +ó 42 +g 43 +uǎn 44 +c 45 +āng 46 +ǎn 47 +x 48 +iān 49 +ōng 50 +àng 51 +òng 52 +d 53 +ài 54 +iù 55 +ìng 56 +ū 57 +m 58 +àn 59 +ǎ 60 +í 61 +z 62 +k 63 +ǒu 64 +ūn 65 +uó 66 +èng 67 +òu 68 +ú 69 +éng 70 +à 71 +iào 72 +āo 73 +ào 74 +h 75 +uáng 76 +t 77 +iáo 78 +è 79 +iǎn 80 +er 81 +n 82 +èi 83 +ǐng 84 +uā 85 +ēi 86 +ǎo 87 +iú 88 +uàng 89 +uí 90 +ǜ 91 +āi 92 +óu 93 +ià 94 +ǎng 95 +īn 96 +uà 97 +uǐ 98 +ái 99 +ò 100 +ē 101 +óng 102 +uàn 103 +é 104 +ìn 105 +ùn 106 +uò 107 +ún 108 +uì 109 +uān 110 +ǐn 111 +iāo 112 +ōu 113 +uán 114 +iǎo 115 +á 116 +ér 117 +èn 118 +C 119 +D 120 +E 121 +O 122 +N 123 +Z 124 +ěi 125 +ián 126 +ēn 127 +iā 128 +ǚ 129 +áng 130 +iě 131 +ǒng 132 +éi 133 +iāng 134 +J 135 +V 136 +ín 137 +ié 138 +īng 139 +ō 140 +ěr 141 +F 142 +ué 143 +iǎng 144 +uō 145 +G 146 +ǎi 147 +ěn 148 +H 149 +en 150 +X 151 +i 152 +ǒ 153 +uǒ 154 +iè 155 +iū 156 +U 157 +iàng 158 +uāng 159 +uá 160 +Q 161 +ě 162 +e 163 +iǎ 164 +iáng 165 +Y 166 +○ 167 +iē 168 +uī 169 +ěng 170 +uài 171 +áo 172 +a 173 +ou 174 +ǔn 175 +uái 176 +uāi 177 +ióng 178 +ei 179 +ń 180 +iá 181 +iōng 182 +üè 183 +uē 184 +uǎng 185 +uǎ 186 +o 187 +uǎi 188 +uě 189 +ǘ 190 +uo 191 +iǒng 192 +ang 193 +ia 194 +u 195 +üě 196 +#0 197 +#1 198 +#2 199 +#3 200 +#4 201 +#5 202 +#6 203 +#7 204 +#8 205 +#9 206 +#10 207 +#11 208 +#12 209 +#13 210 +#14 211 +#15 212 +#16 213 +#17 214 +#18 215 +#19 216 +#20 217 +#21 218 +#22 219 +#23 220 +#24 221 +#25 222 +#26 223 +#27 224 +#28 225 +#29 226 diff --git a/app/src/main/java/com/k2fsa/sherpa/onnx/FeatureConfig.kt b/app/src/main/java/com/k2fsa/sherpa/onnx/FeatureConfig.kt new file mode 100644 index 0000000..2d2eba4 --- /dev/null +++ b/app/src/main/java/com/k2fsa/sherpa/onnx/FeatureConfig.kt @@ -0,0 +1,11 @@ +package com.k2fsa.sherpa.onnx + +data class FeatureConfig( + var sampleRate: Int = 16000, + var featureDim: Int = 80, + var dither: Float = 0.0f +) + +fun getFeatureConfig(sampleRate: Int, featureDim: Int): FeatureConfig { + return FeatureConfig(sampleRate = sampleRate, featureDim = featureDim) +} diff --git a/app/src/main/java/com/k2fsa/sherpa/onnx/HomophoneReplacerConfig.kt b/app/src/main/java/com/k2fsa/sherpa/onnx/HomophoneReplacerConfig.kt new file mode 100644 index 0000000..8baa3ff --- /dev/null +++ b/app/src/main/java/com/k2fsa/sherpa/onnx/HomophoneReplacerConfig.kt @@ -0,0 +1,7 @@ +package com.k2fsa.sherpa.onnx + +data class HomophoneReplacerConfig( + var dictDir: String = "", // unused + var lexicon: String = "", + var ruleFsts: String = "", +) diff --git a/app/src/main/java/com/k2fsa/sherpa/onnx/KeywordSpotter.kt b/app/src/main/java/com/k2fsa/sherpa/onnx/KeywordSpotter.kt new file mode 100644 index 0000000..c0452ea --- /dev/null +++ b/app/src/main/java/com/k2fsa/sherpa/onnx/KeywordSpotter.kt @@ -0,0 +1,156 @@ +// Copyright (c) 2024 Xiaomi Corporation +package com.k2fsa.sherpa.onnx + +import android.content.res.AssetManager + +data class KeywordSpotterConfig( + var featConfig: FeatureConfig = FeatureConfig(), + var modelConfig: OnlineModelConfig = OnlineModelConfig(), + var maxActivePaths: Int = 4, + var keywordsFile: String = "keywords.txt", + var keywordsScore: Float = 1.5f, + var keywordsThreshold: Float = 0.25f, + var numTrailingBlanks: Int = 2, +) + +data class KeywordSpotterResult( + val keyword: String, + val tokens: Array, + val timestamps: FloatArray, + // TODO(fangjun): Add more fields +) { + override fun toString(): String { + val tokensStr = tokens.joinToString(", ") + val timestampsStr = timestamps.joinToString(", ") { "%.2f".format(it) } + return "Keyword: $keyword\nTokens: [$tokensStr]\nTimestamps: [$timestampsStr]" + } +} + +class KeywordSpotter( + assetManager: AssetManager? = null, + val config: KeywordSpotterConfig, +) { + private var ptr: Long + + init { + ptr = if (assetManager != null) { + newFromAsset(assetManager, config) + } else { + newFromFile(config) + } + } + + protected fun finalize() { + if (ptr != 0L) { + delete(ptr) + ptr = 0 + } + } + + fun release() = finalize() + + fun createStream(keywords: String = ""): OnlineStream { + val p = createStream(ptr, keywords) + return OnlineStream(p) + } + + fun decode(stream: OnlineStream) = decode(ptr, stream.ptr) + fun reset(stream: OnlineStream) = reset(ptr, stream.ptr) + fun isReady(stream: OnlineStream) = isReady(ptr, stream.ptr) + fun getResult(stream: OnlineStream): KeywordSpotterResult { + return getResult(ptr, stream.ptr) + } + + private external fun delete(ptr: Long) + + private external fun newFromAsset( + assetManager: AssetManager, + config: KeywordSpotterConfig, + ): Long + + private external fun newFromFile( + config: KeywordSpotterConfig, + ): Long + + private external fun createStream(ptr: Long, keywords: String): Long + private external fun isReady(ptr: Long, streamPtr: Long): Boolean + private external fun decode(ptr: Long, streamPtr: Long) + private external fun reset(ptr: Long, streamPtr: Long) + private external fun getResult(ptr: Long, streamPtr: Long): KeywordSpotterResult + + companion object { + init { + System.loadLibrary("sherpa-onnx-jni") + } + } +} + +/* +Please see +https://k2-fsa.github.io/sherpa/onnx/kws/pretrained_models/index.html +for a list of pre-trained models. + +We only add a few here. Please change the following code +to add your own. (It should be straightforward to add a new model +by following the code) + +@param type +0 - sherpa-onnx-kws-zipformer-wenetspeech-3.3M-2024-01-01 (Chinese) + https://www.modelscope.cn/models/pkufool/sherpa-onnx-kws-zipformer-wenetspeech-3.3M-2024-01-01/summary + +1 - sherpa-onnx-kws-zipformer-gigaspeech-3.3M-2024-01-01 (English) + https://www.modelscope.cn/models/pkufool/sherpa-onnx-kws-zipformer-gigaspeech-3.3M-2024-01-01/summary + + */ +fun getKwsModelConfig(type: Int): OnlineModelConfig? { + when (type) { + 0 -> { + val modelDir = "sherpa-onnx-kws-zipformer-wenetspeech-3.3M-2024-01-01" + return OnlineModelConfig( + transducer = OnlineTransducerModelConfig( + encoder = "$modelDir/encoder-epoch-12-avg-2-chunk-16-left-64.onnx", + decoder = "$modelDir/decoder-epoch-12-avg-2-chunk-16-left-64.onnx", + joiner = "$modelDir/joiner-epoch-12-avg-2-chunk-16-left-64.onnx", + ), + tokens = "$modelDir/tokens.txt", + modelType = "zipformer2", + ) + } + + 1 -> { + val modelDir = "sherpa-onnx-kws-zipformer-gigaspeech-3.3M-2024-01-01" + return OnlineModelConfig( + transducer = OnlineTransducerModelConfig( + encoder = "$modelDir/encoder-epoch-12-avg-2-chunk-16-left-64.onnx", + decoder = "$modelDir/decoder-epoch-12-avg-2-chunk-16-left-64.onnx", + joiner = "$modelDir/joiner-epoch-12-avg-2-chunk-16-left-64.onnx", + ), + tokens = "$modelDir/tokens.txt", + modelType = "zipformer2", + ) + } + + } + return null +} + +/* + * Get the default keywords for each model. + * Caution: The types and modelDir should be the same as those in getModelConfig + * function above. + */ +fun getKeywordsFile(type: Int): String { + when (type) { + 0 -> { + val modelDir = "sherpa-onnx-kws-zipformer-wenetspeech-3.3M-2024-01-01" + return "$modelDir/keywords.txt" + } + + 1 -> { + val modelDir = "sherpa-onnx-kws-zipformer-gigaspeech-3.3M-2024-01-01" + return "$modelDir/keywords.txt" + } + + } + return "" +} diff --git a/app/src/main/java/com/k2fsa/sherpa/onnx/OnlineRecognizer.kt b/app/src/main/java/com/k2fsa/sherpa/onnx/OnlineRecognizer.kt new file mode 100644 index 0000000..c0a1873 --- /dev/null +++ b/app/src/main/java/com/k2fsa/sherpa/onnx/OnlineRecognizer.kt @@ -0,0 +1,669 @@ +package com.k2fsa.sherpa.onnx + +import android.content.res.AssetManager + +data class EndpointRule( + var mustContainNonSilence: Boolean, + var minTrailingSilence: Float, + var minUtteranceLength: Float, +) + +data class EndpointConfig( + var rule1: EndpointRule = EndpointRule(false, 2.4f, 0.0f), + var rule2: EndpointRule = EndpointRule(true, 1.4f, 0.0f), + var rule3: EndpointRule = EndpointRule(false, 0.0f, 20.0f) +) + +data class OnlineTransducerModelConfig( + var encoder: String = "", + var decoder: String = "", + var joiner: String = "", +) + +data class OnlineParaformerModelConfig( + var encoder: String = "", + var decoder: String = "", +) + +data class OnlineZipformer2CtcModelConfig( + var model: String = "", +) + +data class OnlineNeMoCtcModelConfig( + var model: String = "", +) + +data class OnlineToneCtcModelConfig( + var model: String = "", +) + +data class OnlineModelConfig( + var transducer: OnlineTransducerModelConfig = OnlineTransducerModelConfig(), + var paraformer: OnlineParaformerModelConfig = OnlineParaformerModelConfig(), + var zipformer2Ctc: OnlineZipformer2CtcModelConfig = OnlineZipformer2CtcModelConfig(), + var neMoCtc: OnlineNeMoCtcModelConfig = OnlineNeMoCtcModelConfig(), + var toneCtc: OnlineToneCtcModelConfig = OnlineToneCtcModelConfig(), + var tokens: String = "", + var numThreads: Int = 1, + var debug: Boolean = false, + var provider: String = "cpu", + var modelType: String = "", + var modelingUnit: String = "", + var bpeVocab: String = "", +) + +data class OnlineLMConfig( + var model: String = "", + var scale: Float = 0.5f, +) + +data class OnlineCtcFstDecoderConfig( + var graph: String = "", + var maxActive: Int = 3000, +) + +data class OnlineRecognizerConfig( + var featConfig: FeatureConfig = FeatureConfig(), + var modelConfig: OnlineModelConfig = OnlineModelConfig(), + var lmConfig: OnlineLMConfig = OnlineLMConfig(), + var ctcFstDecoderConfig: OnlineCtcFstDecoderConfig = OnlineCtcFstDecoderConfig(), + var hr: HomophoneReplacerConfig = HomophoneReplacerConfig(), + var endpointConfig: EndpointConfig = EndpointConfig(), + var enableEndpoint: Boolean = true, + var decodingMethod: String = "greedy_search", + var maxActivePaths: Int = 4, + var hotwordsFile: String = "", + var hotwordsScore: Float = 1.5f, + var ruleFsts: String = "", + var ruleFars: String = "", + var blankPenalty: Float = 0.0f, +) + +data class OnlineRecognizerResult( + val text: String, + val tokens: Array, + val timestamps: FloatArray, + val ysProbs: FloatArray, + // TODO(fangjun): Add more fields +) + +class OnlineRecognizer( + assetManager: AssetManager? = null, + val config: OnlineRecognizerConfig, +) { + private var ptr: Long + + init { + ptr = if (assetManager != null) { + newFromAsset(assetManager, config) + } else { + newFromFile(config) + } + } + + protected fun finalize() { + if (ptr != 0L) { + delete(ptr) + ptr = 0 + } + } + + fun release() = finalize() + + fun createStream(hotwords: String = ""): OnlineStream { + val p = createStream(ptr, hotwords) + return OnlineStream(p) + } + + fun reset(stream: OnlineStream) = reset(ptr, stream.ptr) + fun decode(stream: OnlineStream) = decode(ptr, stream.ptr) + fun isEndpoint(stream: OnlineStream) = isEndpoint(ptr, stream.ptr) + fun isReady(stream: OnlineStream) = isReady(ptr, stream.ptr) + fun getResult(stream: OnlineStream): OnlineRecognizerResult { + return getResult(ptr, stream.ptr) + } + + private external fun delete(ptr: Long) + + private external fun newFromAsset( + assetManager: AssetManager, + config: OnlineRecognizerConfig, + ): Long + + private external fun newFromFile( + config: OnlineRecognizerConfig, + ): Long + + private external fun createStream(ptr: Long, hotwords: String): Long + private external fun reset(ptr: Long, streamPtr: Long) + private external fun decode(ptr: Long, streamPtr: Long) + private external fun isEndpoint(ptr: Long, streamPtr: Long): Boolean + private external fun isReady(ptr: Long, streamPtr: Long): Boolean + private external fun getResult(ptr: Long, streamPtr: Long): OnlineRecognizerResult + + companion object { + init { + System.loadLibrary("sherpa-onnx-jni") + } + } +} + + +/* +Please see +https://k2-fsa.github.io/sherpa/onnx/pretrained_models/index.html +for a list of pre-trained models. + +We only add a few here. Please change the following code +to add your own. (It should be straightforward to add a new model +by following the code) + +@param type +0 - sherpa-onnx-streaming-zipformer-bilingual-zh-en-2023-02-20 (Bilingual, Chinese + English) + https://k2-fsa.github.io/sherpa/onnx/pretrained_models/zipformer-transducer-models.html#sherpa-onnx-streaming-zipformer-bilingual-zh-en-2023-02-20-bilingual-chinese-english + +1 - csukuangfj/sherpa-onnx-lstm-zh-2023-02-20 (Chinese) + + https://k2-fsa.github.io/sherpa/onnx/pretrained_models/lstm-transducer-models.html#csukuangfj-sherpa-onnx-lstm-zh-2023-02-20-chinese + +2 - csukuangfj/sherpa-onnx-lstm-en-2023-02-17 (English) + https://k2-fsa.github.io/sherpa/onnx/pretrained_models/lstm-transducer-models.html#csukuangfj-sherpa-onnx-lstm-en-2023-02-17-english + +3,4 - pkufool/icefall-asr-zipformer-streaming-wenetspeech-20230615 + https://huggingface.co/pkufool/icefall-asr-zipformer-streaming-wenetspeech-20230615 + 3 - int8 encoder + 4 - float32 encoder + +5 - csukuangfj/sherpa-onnx-streaming-paraformer-bilingual-zh-en + https://huggingface.co/csukuangfj/sherpa-onnx-streaming-paraformer-bilingual-zh-en + +6 - sherpa-onnx-streaming-zipformer-en-2023-06-26 + https://huggingface.co/csukuangfj/sherpa-onnx-streaming-zipformer-en-2023-06-26 + +7 - shaojieli/sherpa-onnx-streaming-zipformer-fr-2023-04-14 (French) + https://huggingface.co/shaojieli/sherpa-onnx-streaming-zipformer-fr-2023-04-14 + +8 - csukuangfj/sherpa-onnx-streaming-zipformer-bilingual-zh-en-2023-02-20 (Bilingual, Chinese + English) + https://huggingface.co/csukuangfj/sherpa-onnx-streaming-zipformer-bilingual-zh-en-2023-02-20 + encoder int8, decoder/joiner float32 + + */ +fun getModelConfig(type: Int): OnlineModelConfig? { + when (type) { + 0 -> { + val modelDir = "sherpa-onnx-streaming-zipformer-bilingual-zh-en-2023-02-20" + return OnlineModelConfig( + transducer = OnlineTransducerModelConfig( + encoder = "$modelDir/encoder-epoch-99-avg-1.onnx", + decoder = "$modelDir/decoder-epoch-99-avg-1.onnx", + joiner = "$modelDir/joiner-epoch-99-avg-1.onnx", + ), + tokens = "$modelDir/tokens.txt", + modelType = "zipformer", + ) + } + + 1 -> { + val modelDir = "sherpa-onnx-lstm-zh-2023-02-20" + return OnlineModelConfig( + transducer = OnlineTransducerModelConfig( + encoder = "$modelDir/encoder-epoch-11-avg-1.onnx", + decoder = "$modelDir/decoder-epoch-11-avg-1.onnx", + joiner = "$modelDir/joiner-epoch-11-avg-1.onnx", + ), + tokens = "$modelDir/tokens.txt", + modelType = "lstm", + ) + } + + 2 -> { + val modelDir = "sherpa-onnx-lstm-en-2023-02-17" + return OnlineModelConfig( + transducer = OnlineTransducerModelConfig( + encoder = "$modelDir/encoder-epoch-99-avg-1.onnx", + decoder = "$modelDir/decoder-epoch-99-avg-1.onnx", + joiner = "$modelDir/joiner-epoch-99-avg-1.onnx", + ), + tokens = "$modelDir/tokens.txt", + modelType = "lstm", + ) + } + + 3 -> { + val modelDir = "icefall-asr-zipformer-streaming-wenetspeech-20230615" + return OnlineModelConfig( + transducer = OnlineTransducerModelConfig( + encoder = "$modelDir/exp/encoder-epoch-12-avg-4-chunk-16-left-128.int8.onnx", + decoder = "$modelDir/exp/decoder-epoch-12-avg-4-chunk-16-left-128.onnx", + joiner = "$modelDir/exp/joiner-epoch-12-avg-4-chunk-16-left-128.onnx", + ), + tokens = "$modelDir/data/lang_char/tokens.txt", + modelType = "zipformer2", + ) + } + + 4 -> { + val modelDir = "icefall-asr-zipformer-streaming-wenetspeech-20230615" + return OnlineModelConfig( + transducer = OnlineTransducerModelConfig( + encoder = "$modelDir/exp/encoder-epoch-12-avg-4-chunk-16-left-128.onnx", + decoder = "$modelDir/exp/decoder-epoch-12-avg-4-chunk-16-left-128.onnx", + joiner = "$modelDir/exp/joiner-epoch-12-avg-4-chunk-16-left-128.onnx", + ), + tokens = "$modelDir/data/lang_char/tokens.txt", + modelType = "zipformer2", + ) + } + + 5 -> { + val modelDir = "sherpa-onnx-streaming-paraformer-bilingual-zh-en" + return OnlineModelConfig( + paraformer = OnlineParaformerModelConfig( + encoder = "$modelDir/encoder.int8.onnx", + decoder = "$modelDir/decoder.int8.onnx", + ), + tokens = "$modelDir/tokens.txt", + modelType = "paraformer", + ) + } + + 6 -> { + val modelDir = "sherpa-onnx-streaming-zipformer-en-2023-06-26" + return OnlineModelConfig( + transducer = OnlineTransducerModelConfig( + encoder = "$modelDir/encoder-epoch-99-avg-1-chunk-16-left-128.int8.onnx", + decoder = "$modelDir/decoder-epoch-99-avg-1-chunk-16-left-128.onnx", + joiner = "$modelDir/joiner-epoch-99-avg-1-chunk-16-left-128.onnx", + ), + tokens = "$modelDir/tokens.txt", + modelType = "zipformer2", + ) + } + + 7 -> { + val modelDir = "sherpa-onnx-streaming-zipformer-fr-2023-04-14" + return OnlineModelConfig( + transducer = OnlineTransducerModelConfig( + encoder = "$modelDir/encoder-epoch-29-avg-9-with-averaged-model.int8.onnx", + decoder = "$modelDir/decoder-epoch-29-avg-9-with-averaged-model.onnx", + joiner = "$modelDir/joiner-epoch-29-avg-9-with-averaged-model.onnx", + ), + tokens = "$modelDir/tokens.txt", + modelType = "zipformer", + ) + } + + 8 -> { + val modelDir = "sherpa-onnx-streaming-zipformer-bilingual-zh-en-2023-02-20" + return OnlineModelConfig( + transducer = OnlineTransducerModelConfig( + encoder = "$modelDir/encoder-epoch-99-avg-1.int8.onnx", + decoder = "$modelDir/decoder-epoch-99-avg-1.onnx", + joiner = "$modelDir/joiner-epoch-99-avg-1.int8.onnx", + ), + tokens = "$modelDir/tokens.txt", + modelType = "zipformer", + ) + } + + 9 -> { + val modelDir = "sherpa-onnx-streaming-zipformer-zh-14M-2023-02-23" + return OnlineModelConfig( + transducer = OnlineTransducerModelConfig( + encoder = "$modelDir/encoder-epoch-99-avg-1.int8.onnx", + decoder = "$modelDir/decoder-epoch-99-avg-1.onnx", + joiner = "$modelDir/joiner-epoch-99-avg-1.int8.onnx", + ), + tokens = "$modelDir/tokens.txt", + modelType = "zipformer", + ) + } + + 10 -> { + val modelDir = "sherpa-onnx-streaming-zipformer-en-20M-2023-02-17" + return OnlineModelConfig( + transducer = OnlineTransducerModelConfig( + encoder = "$modelDir/encoder-epoch-99-avg-1.int8.onnx", + decoder = "$modelDir/decoder-epoch-99-avg-1.onnx", + joiner = "$modelDir/joiner-epoch-99-avg-1.int8.onnx", + ), + tokens = "$modelDir/tokens.txt", + modelType = "zipformer", + ) + } + + 11 -> { + val modelDir = "sherpa-onnx-nemo-streaming-fast-conformer-ctc-en-80ms" + return OnlineModelConfig( + neMoCtc = OnlineNeMoCtcModelConfig( + model = "$modelDir/model.onnx", + ), + tokens = "$modelDir/tokens.txt", + ) + } + + 12 -> { + val modelDir = "sherpa-onnx-nemo-streaming-fast-conformer-ctc-en-480ms" + return OnlineModelConfig( + neMoCtc = OnlineNeMoCtcModelConfig( + model = "$modelDir/model.onnx", + ), + tokens = "$modelDir/tokens.txt", + ) + } + + 13 -> { + val modelDir = "sherpa-onnx-nemo-streaming-fast-conformer-ctc-en-1040ms" + return OnlineModelConfig( + neMoCtc = OnlineNeMoCtcModelConfig( + model = "$modelDir/model.onnx", + ), + tokens = "$modelDir/tokens.txt", + ) + } + + 14 -> { + val modelDir = "sherpa-onnx-streaming-zipformer-korean-2024-06-16" + return OnlineModelConfig( + transducer = OnlineTransducerModelConfig( + encoder = "$modelDir/encoder-epoch-99-avg-1.int8.onnx", + decoder = "$modelDir/decoder-epoch-99-avg-1.onnx", + joiner = "$modelDir/joiner-epoch-99-avg-1.int8.onnx", + ), + tokens = "$modelDir/tokens.txt", + modelType = "zipformer", + ) + } + + 15 -> { + val modelDir = "sherpa-onnx-streaming-zipformer-small-ctc-zh-int8-2025-04-01" + return OnlineModelConfig( + zipformer2Ctc = OnlineZipformer2CtcModelConfig( + model = "$modelDir/model.int8.onnx", + ), + tokens = "$modelDir/tokens.txt", + ) + } + + 16 -> { + val modelDir = "sherpa-onnx-streaming-zipformer-small-ctc-zh-2025-04-01" + return OnlineModelConfig( + zipformer2Ctc = OnlineZipformer2CtcModelConfig( + model = "$modelDir/model.onnx", + ), + tokens = "$modelDir/tokens.txt", + ) + } + + 17 -> { + val modelDir = "sherpa-onnx-streaming-zipformer-ctc-zh-int8-2025-06-30" + return OnlineModelConfig( + zipformer2Ctc = OnlineZipformer2CtcModelConfig( + model = "$modelDir/model.int8.onnx", + ), + tokens = "$modelDir/tokens.txt", + ) + } + + 18 -> { + val modelDir = "sherpa-onnx-streaming-zipformer-ctc-zh-2025-06-30" + return OnlineModelConfig( + zipformer2Ctc = OnlineZipformer2CtcModelConfig( + model = "$modelDir/model.onnx", + ), + tokens = "$modelDir/tokens.txt", + modelType = "zipformer2", + ) + } + + 19 -> { + val modelDir = "sherpa-onnx-streaming-zipformer-ctc-zh-fp16-2025-06-30" + return OnlineModelConfig( + zipformer2Ctc = OnlineZipformer2CtcModelConfig( + model = "$modelDir/model.fp16.onnx", + ), + tokens = "$modelDir/tokens.txt", + modelType = "zipformer2", + ) + } + + 20 -> { + val modelDir = "sherpa-onnx-streaming-zipformer-zh-int8-2025-06-30" + return OnlineModelConfig( + transducer = OnlineTransducerModelConfig( + encoder = "$modelDir/encoder.int8.onnx", + decoder = "$modelDir/decoder.onnx", + joiner = "$modelDir/joiner.int8.onnx", + ), + tokens = "$modelDir/tokens.txt", + modelType = "zipformer2", + ) + } + + 21 -> { + val modelDir = "sherpa-onnx-streaming-zipformer-en-kroko-2025-08-06" + return OnlineModelConfig( + transducer = OnlineTransducerModelConfig( + encoder = "$modelDir/encoder.onnx", + decoder = "$modelDir/decoder.onnx", + joiner = "$modelDir/joiner.onnx", + ), + tokens = "$modelDir/tokens.txt", + modelType = "zipformer2", + ) + } + + 22 -> { + val modelDir = "sherpa-onnx-streaming-zipformer-es-kroko-2025-08-06" + return OnlineModelConfig( + transducer = OnlineTransducerModelConfig( + encoder = "$modelDir/encoder.onnx", + decoder = "$modelDir/decoder.onnx", + joiner = "$modelDir/joiner.onnx", + ), + tokens = "$modelDir/tokens.txt", + modelType = "zipformer2", + ) + } + + 23 -> { + val modelDir = "sherpa-onnx-streaming-zipformer-fr-kroko-2025-08-06" + return OnlineModelConfig( + transducer = OnlineTransducerModelConfig( + encoder = "$modelDir/encoder.onnx", + decoder = "$modelDir/decoder.onnx", + joiner = "$modelDir/joiner.onnx", + ), + tokens = "$modelDir/tokens.txt", + modelType = "zipformer2", + ) + } + + 24 -> { + val modelDir = "sherpa-onnx-streaming-zipformer-de-kroko-2025-08-06" + return OnlineModelConfig( + transducer = OnlineTransducerModelConfig( + encoder = "$modelDir/encoder.onnx", + decoder = "$modelDir/decoder.onnx", + joiner = "$modelDir/joiner.onnx", + ), + tokens = "$modelDir/tokens.txt", + modelType = "zipformer2", + ) + } + + 25 -> { + val modelDir = "sherpa-onnx-streaming-zipformer-small-ru-vosk-int8-2025-08-16" + return OnlineModelConfig( + transducer = OnlineTransducerModelConfig( + encoder = "$modelDir/encoder.int8.onnx", + decoder = "$modelDir/decoder.onnx", + joiner = "$modelDir/joiner.int8.onnx", + ), + tokens = "$modelDir/tokens.txt", + modelType = "zipformer2", + ) + } + + 26 -> { + val modelDir = "sherpa-onnx-streaming-zipformer-small-ru-vosk-2025-08-16" + return OnlineModelConfig( + transducer = OnlineTransducerModelConfig( + encoder = "$modelDir/encoder.onnx", + decoder = "$modelDir/decoder.onnx", + joiner = "$modelDir/joiner.onnx", + ), + tokens = "$modelDir/tokens.txt", + modelType = "zipformer2", + ) + } + + 27 -> { + val modelDir = "sherpa-onnx-streaming-t-one-russian-2025-09-08" + return OnlineModelConfig( + toneCtc = OnlineToneCtcModelConfig( + model = "$modelDir/model.onnx", + ), + tokens = "$modelDir/tokens.txt", + ) + } + + 28 -> { + val modelDir = "sherpa-onnx-nemotron-speech-streaming-en-0.6b-int8-2026-01-14" + return OnlineModelConfig( + transducer = OnlineTransducerModelConfig( + encoder = "$modelDir/encoder.int8.onnx", + decoder = "$modelDir/decoder.int8.onnx", + joiner = "$modelDir/joiner.int8.onnx", + ), + tokens = "$modelDir/tokens.txt", + ) + } + + 29 -> { + val modelDir = "sherpa-onnx-streaming-zipformer-bn-vosk-2026-02-09" + return OnlineModelConfig( + transducer = OnlineTransducerModelConfig( + encoder = "$modelDir/encoder.onnx", + decoder = "$modelDir/decoder.onnx", + joiner = "$modelDir/joiner.onnx", + ), + tokens = "$modelDir/tokens.txt", + modelType = "zipformer2", + ) + } + + 30 -> { + val modelDir = "sherpa-onnx-nemotron-speech-streaming-en-0.6b-80ms-int8-2026-04-25" + return OnlineModelConfig( + transducer = OnlineTransducerModelConfig( + encoder = "$modelDir/encoder.int8.onnx", + decoder = "$modelDir/decoder.int8.onnx", + joiner = "$modelDir/joiner.int8.onnx", + ), + tokens = "$modelDir/tokens.txt", + ) + } + + 31 -> { + val modelDir = "sherpa-onnx-nemotron-speech-streaming-en-0.6b-160ms-int8-2026-04-25" + return OnlineModelConfig( + transducer = OnlineTransducerModelConfig( + encoder = "$modelDir/encoder.int8.onnx", + decoder = "$modelDir/decoder.int8.onnx", + joiner = "$modelDir/joiner.int8.onnx", + ), + tokens = "$modelDir/tokens.txt", + ) + } + + 32 -> { + val modelDir = "sherpa-onnx-nemotron-speech-streaming-en-0.6b-560ms-int8-2026-04-25" + return OnlineModelConfig( + transducer = OnlineTransducerModelConfig( + encoder = "$modelDir/encoder.int8.onnx", + decoder = "$modelDir/decoder.int8.onnx", + joiner = "$modelDir/joiner.int8.onnx", + ), + tokens = "$modelDir/tokens.txt", + ) + } + + 33 -> { + val modelDir = "sherpa-onnx-nemotron-speech-streaming-en-0.6b-1120ms-int8-2026-04-25" + return OnlineModelConfig( + transducer = OnlineTransducerModelConfig( + encoder = "$modelDir/encoder.int8.onnx", + decoder = "$modelDir/decoder.int8.onnx", + joiner = "$modelDir/joiner.int8.onnx", + ), + tokens = "$modelDir/tokens.txt", + ) + } + + 1000 -> { + val modelDir = "sherpa-onnx-rk3588-streaming-zipformer-bilingual-zh-en-2023-02-20" + return OnlineModelConfig( + transducer = OnlineTransducerModelConfig( + encoder = "$modelDir/encoder.rknn", + decoder = "$modelDir/decoder.rknn", + joiner = "$modelDir/joiner.rknn", + ), + tokens = "$modelDir/tokens.txt", + modelType = "zipformer", + provider = "rknn", + ) + } + + 1001 -> { + val modelDir = "sherpa-onnx-rk3588-streaming-zipformer-small-bilingual-zh-en-2023-02-16" + return OnlineModelConfig( + transducer = OnlineTransducerModelConfig( + encoder = "$modelDir/encoder.rknn", + decoder = "$modelDir/decoder.rknn", + joiner = "$modelDir/joiner.rknn", + ), + tokens = "$modelDir/tokens.txt", + modelType = "zipformer", + provider = "rknn", + ) + } + + } + return null +} + +/* +Please see +https://k2-fsa.github.io/sherpa/onnx/pretrained_models/index.html +for a list of pre-trained models. + +We only add a few here. Please change the following code +to add your own LM model. (It should be straightforward to train a new NN LM model +by following the code, https://github.com/k2-fsa/icefall/blob/master/icefall/rnn_lm/train.py) + +@param type +0 - sherpa-onnx-streaming-zipformer-bilingual-zh-en-2023-02-20 (Bilingual, Chinese + English) + https://k2-fsa.github.io/sherpa/onnx/pretrained_models/zipformer-transducer-models.html#sherpa-onnx-streaming-zipformer-bilingual-zh-en-2023-02-20-bilingual-chinese-english + */ +fun getOnlineLMConfig(type: Int): OnlineLMConfig { + when (type) { + 0 -> { + val modelDir = "sherpa-onnx-streaming-zipformer-bilingual-zh-en-2023-02-20" + return OnlineLMConfig( + model = "$modelDir/with-state-epoch-99-avg-1.int8.onnx", + scale = 0.5f, + ) + } + } + return OnlineLMConfig() +} + +fun getEndpointConfig(): EndpointConfig { + return EndpointConfig( + rule1 = EndpointRule(false, 2.4f, 0.0f), + rule2 = EndpointRule(true, 1.4f, 0.0f), + rule3 = EndpointRule(false, 0.0f, 20.0f) + ) +} + diff --git a/app/src/main/java/com/k2fsa/sherpa/onnx/OnlineStream.kt b/app/src/main/java/com/k2fsa/sherpa/onnx/OnlineStream.kt new file mode 100644 index 0000000..ace8623 --- /dev/null +++ b/app/src/main/java/com/k2fsa/sherpa/onnx/OnlineStream.kt @@ -0,0 +1,42 @@ +package com.k2fsa.sherpa.onnx + +class OnlineStream(var ptr: Long = 0) { + fun acceptWaveform(samples: FloatArray, sampleRate: Int) = + acceptWaveform(ptr, samples, sampleRate) + + fun inputFinished() = inputFinished(ptr) + + fun setOption(key: String, value: String) = setOption(ptr, key, value) + + fun getOption(key: String): String = getOption(ptr, key) + + protected fun finalize() { + if (ptr != 0L) { + delete(ptr) + ptr = 0 + } + } + + fun release() = finalize() + + fun use(block: (OnlineStream) -> Unit) { + try { + block(this) + } finally { + release() + } + } + + private external fun acceptWaveform(ptr: Long, samples: FloatArray, sampleRate: Int) + private external fun inputFinished(ptr: Long) + private external fun setOption(ptr: Long, key: String, value: String) + private external fun getOption(ptr: Long, key: String): String + private external fun delete(ptr: Long) + + + companion object { + init { + System.loadLibrary("sherpa-onnx-jni") + } + } +} diff --git a/app/src/main/java/com/lila/wakeup/BootReceiver.kt b/app/src/main/java/com/lila/wakeup/BootReceiver.kt new file mode 100644 index 0000000..ad0f7e4 --- /dev/null +++ b/app/src/main/java/com/lila/wakeup/BootReceiver.kt @@ -0,0 +1,50 @@ +package com.lila.wakeup + +import android.content.BroadcastReceiver +import android.content.Context +import android.content.Intent +import android.os.Build +import android.util.Log + +/** + * 开机自启接收器。 + * + * 监听 BOOT_COMPLETED / LOCKED_BOOT_COMPLETED / QUICKBOOT_POWERON 三种开机广播， + * 收到后启动 [WakeupForegroundService] 进入监听状态。 + * + * Android 限制： + * - 普通 APK 必须用户手动启动过一次 APP 后，BootReceiver 才会生效（系统安全限制）。 + * - 升级到方案 B（系统签名 + /system/priv-app/）后此限制取消。 + */ +class BootReceiver : BroadcastReceiver() { + + companion object { + private const val TAG = "KwsService.Boot" + } + + override fun onReceive(context: Context, intent: Intent) { + val action = intent.action + Log.i(TAG, "Received boot broadcast: $action") + + when (action) { + Intent.ACTION_BOOT_COMPLETED, + Intent.ACTION_LOCKED_BOOT_COMPLETED, + "android.intent.action.QUICKBOOT_POWERON" -> { + startWakeupService(context) + } + else -> { + Log.w(TAG, "Ignored unknown action: $action") + } + } + } + + private fun startWakeupService(context: Context) { + val serviceIntent = Intent(context, WakeupForegroundService::class.java) + if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.O) { + context.startForegroundService(serviceIntent) + } else { + context.startService(serviceIntent) + } + Log.i(TAG, "WakeupForegroundService started after boot") + } +} diff --git a/app/src/main/java/com/lila/wakeup/Config.kt b/app/src/main/java/com/lila/wakeup/Config.kt new file mode 100644 index 0000000..9bca245 --- /dev/null +++ b/app/src/main/java/com/lila/wakeup/Config.kt @@ -0,0 +1,108 @@ +package com.lila.wakeup + +import android.media.AudioFormat + +/** + * KWS APK 全局配置常量。 + * + * 协议版本：v2.1（详见 KWS唤醒协议-V2.md） + * 涵盖三类常量： + * - 协议常量（Action 名、对端包名、字段键） + * - 引擎参数（阈值、平滑、超时、采样率） + * - 模型路径（assets 内相对路径） + */ +object Config { + + // ============================================================ + // 一、协议常量 + // ============================================================ + + /** 本 APK 包名（自身） */ + const val SELF_PACKAGE = "com.lila.wakeup" + + /** 数字人 APP 包名（已确认 v2.1） */ + const val APP_PACKAGE = "com.qy.lila" + + /** 唤醒事件广播 Action（本 → APP） */ + const val ACTION_WAKEUP = "com.lila.intent.action.WAKEUP" + + /** 暂停 KWS 广播 Action（APP → 本） */ + const val ACTION_KWS_PAUSE = "com.lila.intent.action.KWS_PAUSE" + + /** 恢复 KWS 广播 Action（APP → 本） */ + const val ACTION_KWS_RESUME = "com.lila.intent.action.KWS_RESUME" + + /** WAKEUP Extras: 命中的唤醒词文本 */ + const val EXTRA_KEYWORD = "keyword" + + /** WAKEUP Extras: 命中时间戳（毫秒） */ + const val EXTRA_TIMESTAMP = "timestamp" + + /** WAKEUP Extras: 置信度 [0,1] */ + const val EXTRA_CONFIDENCE = "confidence" + + /** PAUSE Extras: 静默期毫秒数 */ + const val EXTRA_SILENCE_MS = "silence_ms" + + /** PAUSE/RESUME Extras: 调用原因（透传日志） */ + const val EXTRA_REASON = "reason" + + // ============================================================ + // 二、引擎参数 + // ============================================================ + + /** PAUSE 默认静默期（v2.1 微调：3000ms 覆盖 RTC 远端音频回灌 + AI TTS 触发延迟） */ + const val DEFAULT_SILENCE_MS = 3000L + + /** APP 漏发 RESUME 时的兜底超时（2 分钟） */ + const val PAUSE_TIMEOUT_MS = 2 * 60 * 1000L + + /** KWS 主阈值（"你好Lila" 主词） */ + const val KWS_THRESHOLD_PRIMARY = 0.85f + + /** KWS 次阈值（其他变体词） */ + const val KWS_THRESHOLD_SECONDARY = 0.80f + + /** 后验平滑：连续 N 帧 confidence > 阈值才算命中 */ + const val SMOOTH_FRAMES = 2 + + // ============================================================ + // 三、AudioRecord 配置 + // ============================================================ + + /** 采样率（sherpa-onnx 模型固定 16kHz） */ + const val SAMPLE_RATE = 16_000 + + /** 单声道 */ + const val CHANNEL_CONFIG = AudioFormat.CHANNEL_IN_MONO + + /** 16-bit PCM */ + const val ENCODING = AudioFormat.ENCODING_PCM_16BIT + + /** 单帧采样点数（10ms 帧 @ 16kHz = 160 samples） */ + const val FRAME_SAMPLES = SAMPLE_RATE / 100 + + // ============================================================ + // 四、模型路径（assets 内） + // ============================================================ + + /** + * 模型目录，与 sherpa-onnx 官方 demo 一致。 + * 切换模型时改这一行 + 对应的 keywords.txt。 + */ + const val MODEL_DIR = "sherpa-onnx-kws-zipformer-wenetspeech-3.3M-2024-01-01" + + const val MODEL_ENCODER = "$MODEL_DIR/encoder-epoch-12-avg-2-chunk-16-left-64.onnx" + const val MODEL_DECODER = "$MODEL_DIR/decoder-epoch-12-avg-2-chunk-16-left-64.onnx" + const val MODEL_JOINER = "$MODEL_DIR/joiner-epoch-12-avg-2-chunk-16-left-64.onnx" + const val MODEL_TOKENS = "$MODEL_DIR/tokens.txt" + const val MODEL_KEYWORDS = "$MODEL_DIR/keywords.txt" + + // ============================================================ + // 五、通知栏 + // ============================================================ + + const val NOTIF_CHANNEL_ID = "lila_wakeup_kws" + const val NOTIF_CHANNEL_NAME = "Lila 语音唤醒" + const val NOTIF_ID = 1 +} diff --git a/app/src/main/java/com/lila/wakeup/MainActivity.kt b/app/src/main/java/com/lila/wakeup/MainActivity.kt new file mode 100644 index 0000000..02773b1 --- /dev/null +++ b/app/src/main/java/com/lila/wakeup/MainActivity.kt @@ -0,0 +1,117 @@ +package com.lila.wakeup + +import android.Manifest +import android.content.Intent +import android.content.pm.PackageManager +import android.os.Build +import android.os.Bundle +import android.util.Log +import android.widget.TextView +import androidx.appcompat.app.AppCompatActivity +import androidx.core.app.ActivityCompat +import androidx.core.content.ContextCompat + +/** + * 状态查看 Activity（开发期辅助 + 首次启动权限申请入口）。 + * + * 职责： + * 1. 首次启动时弹麦克风权限申请 + * 2. 启动 [WakeupForegroundService] + * 3. 显示当前状态（开发期可选） + * + * 量产时（方案 B）改为 launcher 隐藏（删 LAUNCHER intent-filter）， + * 用户不用进 APP 也能开机自启。 + */ +class MainActivity : AppCompatActivity() { + + companion object { + private const val TAG = "KwsService.Main" + private const val PERMISSION_REQUEST_CODE = 1001 + } + + private lateinit var statusText: TextView + + override fun onCreate(savedInstanceState: Bundle?) { + super.onCreate(savedInstanceState) + + // 简化：用一个 TextView 直接显示状态，不用 layout xml + statusText = TextView(this).apply { + text = "Lila 语音唤醒服务\n\n初始化中..." + textSize = 18f + setPadding(60, 60, 60, 60) + } + setContentView(statusText) + + checkAndRequestPermissions() + } + + private fun checkAndRequestPermissions() { + val needRequestList = mutableListOf() + + // 麦克风权限（核心） + if (ContextCompat.checkSelfPermission(this, Manifest.permission.RECORD_AUDIO) + != PackageManager.PERMISSION_GRANTED + ) { + needRequestList.add(Manifest.permission.RECORD_AUDIO) + } + + // 通知权限（Android 13+） + if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.TIRAMISU) { + if (ContextCompat.checkSelfPermission(this, Manifest.permission.POST_NOTIFICATIONS) + != PackageManager.PERMISSION_GRANTED + ) { + needRequestList.add(Manifest.permission.POST_NOTIFICATIONS) + } + } + + if (needRequestList.isNotEmpty()) { + ActivityCompat.requestPermissions( + this, + needRequestList.toTypedArray(), + PERMISSION_REQUEST_CODE + ) + } else { + startWakeupService() + } + } + + override fun onRequestPermissionsResult( + requestCode: Int, + permissions: Array, + grantResults: IntArray + ) { + super.onRequestPermissionsResult(requestCode, permissions, grantResults) + + if (requestCode != PERMISSION_REQUEST_CODE) return + + val recordIdx = permissions.indexOf(Manifest.permission.RECORD_AUDIO) + if (recordIdx >= 0 && grantResults[recordIdx] == PackageManager.PERMISSION_GRANTED) { + Log.i(TAG, "RECORD_AUDIO granted") + startWakeupService() + } else { + statusText.text = "Lila 语音唤醒服务\n\n❌ 麦克风权限被拒绝，无法启动唤醒" + Log.e(TAG, "RECORD_AUDIO denied, service NOT started") + } + } + + private fun startWakeupService() { + val intent = Intent(this, WakeupForegroundService::class.java) + if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.O) { + startForegroundService(intent) + } else { + startService(intent) + } + statusText.text = """ + Lila 语音唤醒服务 + + ✅ 服务已启动 + ✅ 监听唤醒词中 + + 可关闭本界面，服务会在后台持续运行。 + 通知栏可看到运行状态。 + + 日志：adb logcat -s KwsService + """.trimIndent() + Log.i(TAG, "WakeupForegroundService start requested") + } +} diff --git a/app/src/main/java/com/lila/wakeup/WakeupForegroundService.kt b/app/src/main/java/com/lila/wakeup/WakeupForegroundService.kt new file mode 100644 index 0000000..93d1b61 --- /dev/null +++ b/app/src/main/java/com/lila/wakeup/WakeupForegroundService.kt @@ -0,0 +1,140 @@ +package com.lila.wakeup + +import android.app.Notification +import android.app.NotificationChannel +import android.app.NotificationManager +import android.app.PendingIntent +import android.app.Service +import android.content.Context +import android.content.Intent +import android.content.pm.ServiceInfo +import android.os.Build +import android.os.IBinder +import android.util.Log +import androidx.core.app.NotificationCompat +import com.lila.wakeup.kws.AudioCapture +import com.lila.wakeup.kws.KwsEngine +import com.lila.wakeup.kws.KwsStateMachine +import com.lila.wakeup.protocol.BroadcastSender +import com.lila.wakeup.protocol.WakeupServiceLocator + +/** + * KWS 唤醒前台 Service —— 整个 APK 的运行时核心。 + * + * 生命周期： + * - onCreate: 初始化引擎、状态机、采集器、通知栏 + * - onStartCommand: startForeground + 启动音频采集 + 进入 LISTENING + * - onDestroy: 停止采集、释放引擎 + * + * START_STICKY: 被系统杀死后会自动重启（无音频帧丢失影响业务）。 + */ +class WakeupForegroundService : Service() { + + companion object { + private const val TAG = "KwsService.Svc" + } + + private lateinit var engine: KwsEngine + private lateinit var stateMachine: KwsStateMachine + private lateinit var audioCapture: AudioCapture + private lateinit var sender: BroadcastSender + + override fun onCreate() { + super.onCreate() + Log.i(TAG, "onCreate") + + // 通知栏 channel 必须在 startForeground 之前创建 + createNotificationChannel() + + // 初始化引擎栈（顺序：sender → engine → state → audio） + sender = BroadcastSender(applicationContext) + engine = KwsEngine(applicationContext).also { it.init() } + stateMachine = KwsStateMachine( + engine = engine, + sender = sender, + onTimeoutResume = { Log.w(TAG, "[Tmout] PAUSE 超时自动 RESUME") } + ) + audioCapture = AudioCapture(onFrame = { pcm -> stateMachine.onAudioFrame(pcm) }) + + // 注册到全局定位器，使 ProtocolReceiver 能投递 PAUSE/RESUME 事件 + WakeupServiceLocator.stateMachine = stateMachine + } + + override fun onStartCommand(intent: Intent?, flags: Int, startId: Int): Int { + Log.i(TAG, "onStartCommand") + startInForeground() + + // 进入监听态 + stateMachine.transitionToListening() + audioCapture.start() + + // 被杀重启时尝试恢复（不带 Intent） + return START_STICKY + } + + override fun onDestroy() { + Log.i(TAG, "onDestroy") + // 先解绑 locator，避免 ProtocolReceiver 投递到正在销毁的 stateMachine + WakeupServiceLocator.stateMachine = null + audioCapture.stop() + engine.release() + stateMachine.shutdown() + super.onDestroy() + } + + override fun onBind(intent: Intent?): IBinder? = null + + // ============================================================ + // 通知栏（前台 Service 强制要求） + // ============================================================ + + private fun startInForeground() { + val notification = buildNotification() + if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.Q) { + startForeground( + Config.NOTIF_ID, + notification, + ServiceInfo.FOREGROUND_SERVICE_TYPE_MICROPHONE + ) + } else { + startForeground(Config.NOTIF_ID, notification) + } + } + + private fun createNotificationChannel() { + if (Build.VERSION.SDK_INT < Build.VERSION_CODES.O) return + + val mgr = getSystemService(Context.NOTIFICATION_SERVICE) as NotificationManager + if (mgr.getNotificationChannel(Config.NOTIF_CHANNEL_ID) != null) return + + val channel = NotificationChannel( + Config.NOTIF_CHANNEL_ID, + Config.NOTIF_CHANNEL_NAME, + NotificationManager.IMPORTANCE_MIN // 最低优先级，不打扰用户 + ).apply { + description = "语音唤醒服务运行状态" + setShowBadge(false) + } + mgr.createNotificationChannel(channel) + } + + private fun buildNotification(): Notification { + // 点击通知打开 MainActivity（状态查看） + val pendingIntent = PendingIntent.getActivity( + this, + 0, + Intent(this, MainActivity::class.java), + PendingIntent.FLAG_IMMUTABLE + ) + + return NotificationCompat.Builder(this, Config.NOTIF_CHANNEL_ID) + .setContentTitle("Lila 语音助手") + .setContentText("正在监听唤醒词") + .setSmallIcon(android.R.drawable.ic_btn_speak_now) // 用系统默认麦克风图标，方案 B 时换自家 + .setOngoing(true) + .setPriority(NotificationCompat.PRIORITY_MIN) + .setContentIntent(pendingIntent) + .setForegroundServiceBehavior(NotificationCompat.FOREGROUND_SERVICE_IMMEDIATE) + .build() + } +} diff --git a/app/src/main/java/com/lila/wakeup/kws/AudioCapture.kt b/app/src/main/java/com/lila/wakeup/kws/AudioCapture.kt new file mode 100644 index 0000000..0efd567 --- /dev/null +++ b/app/src/main/java/com/lila/wakeup/kws/AudioCapture.kt @@ -0,0 +1,112 @@ +package com.lila.wakeup.kws + +import android.annotation.SuppressLint +import android.media.AudioFormat +import android.media.AudioRecord +import android.media.MediaRecorder +import android.util.Log +import com.lila.wakeup.Config + +/** + * 麦克风音频采集封装。 + * + * 设计要点： + * - AudioSource 用 VOICE_RECOGNITION（自动启用 NS/AGC，省去自己降噪） + * - 16kHz / mono / 16-bit PCM（与 sherpa-onnx 模型严格对齐） + * - 10ms 帧（160 sample）便于和 VAD 对齐 + * - 暂停时完全 release AudioRecord，释放麦克风给 RTC SDK + * + * 注：这是常驻读取线程，不是 ForegroundService 主线程。 + */ +class AudioCapture( + /** 每读到一帧 PCM 数据时回调，参数为 16-bit PCM short[] */ + private val onFrame: (ShortArray) -> Unit +) { + companion object { + private const val TAG = "KwsService.Audio" + } + + @Volatile private var record: AudioRecord? = null + @Volatile private var thread: Thread? = null + @Volatile private var running = false + + @SuppressLint("MissingPermission") // 权限由 MainActivity 运行时申请 + fun start() { + if (running) { + Log.w(TAG, "start() called while already running, skip") + return + } + + val minBuf = AudioRecord.getMinBufferSize( + Config.SAMPLE_RATE, Config.CHANNEL_CONFIG, Config.ENCODING + ) + // 至少 100ms ring buffer，多于 minBuf 防止音频丢帧 + val bufSize = maxOf(minBuf, Config.SAMPLE_RATE * 2 * 100 / 1000) + + record = AudioRecord.Builder() + .setAudioSource(MediaRecorder.AudioSource.VOICE_RECOGNITION) + .setAudioFormat( + AudioFormat.Builder() + .setSampleRate(Config.SAMPLE_RATE) + .setChannelMask(Config.CHANNEL_CONFIG) + .setEncoding(Config.ENCODING) + .build() + ) + .setBufferSizeInBytes(bufSize) + .build() + + if (record?.state != AudioRecord.STATE_INITIALIZED) { + Log.e(TAG, "AudioRecord init failed, state=${record?.state}") + release() + return + } + + record?.startRecording() + running = true + Log.i(TAG, "AudioRecord started, bufSize=$bufSize bytes") + + thread = Thread { + val frame = ShortArray(Config.FRAME_SAMPLES) + while (running) { + val n = record?.read(frame, 0, frame.size) ?: 0 + if (n > 0) { + onFrame(frame) + } else if (n < 0) { + Log.e(TAG, "AudioRecord.read error code=$n, abort") + break + } + } + Log.i(TAG, "AudioCapture thread exit") + }.apply { + name = "KwsAudioCapture" + priority = Thread.NORM_PRIORITY + 2 // 略高于普通线程 + isDaemon = true + start() + } + } + + fun stop() { + if (!running) return + running = false + thread?.join(500) + thread = null + release() + Log.i(TAG, "AudioRecord stopped") + } + + private fun release() { + record?.let { + try { + if (it.recordingState == AudioRecord.RECORDSTATE_RECORDING) { + it.stop() + } + it.release() + } catch (e: Exception) { + Log.e(TAG, "release failed: ${e.message}") + } + } + record = null + } + + fun isRunning(): Boolean = running +} diff --git a/app/src/main/java/com/lila/wakeup/kws/KwsEngine.kt b/app/src/main/java/com/lila/wakeup/kws/KwsEngine.kt new file mode 100644 index 0000000..9f2f86d --- /dev/null +++ b/app/src/main/java/com/lila/wakeup/kws/KwsEngine.kt @@ -0,0 +1,119 @@ +package com.lila.wakeup.kws + +import android.content.Context +import android.util.Log +import com.k2fsa.sherpa.onnx.KeywordSpotter +import com.k2fsa.sherpa.onnx.KeywordSpotterConfig +import com.k2fsa.sherpa.onnx.OnlineModelConfig +import com.k2fsa.sherpa.onnx.OnlineStream +import com.k2fsa.sherpa.onnx.OnlineTransducerModelConfig +import com.lila.wakeup.Config + +/** + * sherpa-onnx KWS 引擎封装。 + * + * 该类是对 sherpa-onnx 官方 [KeywordSpotter] Kotlin Wrapper 的薄封装， + * 隐藏底层 API 细节，对外暴露简洁的 [process] 接口。 + * + * 模型路径：见 [Config.MODEL_DIR]，从 APK assets 加载。 + * + * ⚠️ 版本兼容：本骨架基于 sherpa-onnx v1.13.0+ 的 Kotlin API。 + * 若克隆的 sherpa-onnx 仓库 API 有差异（如类名变更），仅需调整本文件。 + * + * 业务结果数据类。 + * + * @param keyword 命中的唤醒词文本（如 "你好Lila"），未命中为空字符串 + * @param confidence 置信度（[0,1]） + */ +data class KwsResult( + val keyword: String, + val confidence: Float +) { + val isHit: Boolean get() = keyword.isNotEmpty() +} + +class KwsEngine(private val context: Context) { + + companion object { + private const val TAG = "KwsService.Engine" + } + + private var spotter: KeywordSpotter? = null + private var stream: OnlineStream? = null + + /** + * 加载模型 + keywords.txt，初始化推理引擎。 + * 必须在 [process] 之前调用一次。 + */ + fun init() { + val cfg = KeywordSpotterConfig( + featConfig = com.k2fsa.sherpa.onnx.FeatureConfig( + sampleRate = Config.SAMPLE_RATE, + featureDim = 80 + ), + modelConfig = OnlineModelConfig( + transducer = OnlineTransducerModelConfig( + encoder = Config.MODEL_ENCODER, + decoder = Config.MODEL_DECODER, + joiner = Config.MODEL_JOINER + ), + tokens = Config.MODEL_TOKENS, + modelType = "zipformer" + ), + keywordsFile = Config.MODEL_KEYWORDS, + keywordsThreshold = Config.KWS_THRESHOLD_PRIMARY, + keywordsScore = 1.5f, + numTrailingBlanks = 2 + ) + + spotter = KeywordSpotter(assetManager = context.assets, config = cfg) + stream = spotter?.createStream() + Log.i(TAG, "[Engine] init done, keywordsFile=${Config.MODEL_KEYWORDS}") + } + + /** + * 喂一帧 PCM 数据进引擎，返回是否命中。 + * + * @param pcm 16-bit PCM short[] 数据，长度任意（一般 10ms 帧 = 160 samples） + * @return 命中则 [KwsResult.isHit] 为 true + */ + fun process(pcm: ShortArray): KwsResult { + val sp = spotter ?: return KwsResult("", 0f) + val st = stream ?: return KwsResult("", 0f) + + // sherpa-onnx 输入要求 [-1, 1] 范围 float + val floatBuf = FloatArray(pcm.size) { pcm[it] / 32768f } + st.acceptWaveform(floatBuf, Config.SAMPLE_RATE) + + while (sp.isReady(st)) { + sp.decode(st) + } + + val result = sp.getResult(st) + val keyword = result.keyword + + return if (keyword.isNotEmpty()) { + // 命中后必须 reset，否则下次不会再触发 + sp.reset(st) + Log.i(TAG, "[Engine] hit keyword=$keyword") + KwsResult(keyword, 1f) // sherpa-onnx 不直接给 confidence，通过 keyword_score 间接控制 + } else { + KwsResult("", 0f) + } + } + + /** + * 释放 native 资源。在 Service.onDestroy 中调用。 + */ + fun release() { + try { + stream?.release() + spotter?.release() + } catch (e: Exception) { + Log.e(TAG, "release failed: ${e.message}") + } + stream = null + spotter = null + Log.i(TAG, "[Engine] released") + } +} diff --git a/app/src/main/java/com/lila/wakeup/kws/KwsStateMachine.kt b/app/src/main/java/com/lila/wakeup/kws/KwsStateMachine.kt new file mode 100644 index 0000000..d0564b2 --- /dev/null +++ b/app/src/main/java/com/lila/wakeup/kws/KwsStateMachine.kt @@ -0,0 +1,163 @@ +package com.lila.wakeup.kws + +import android.os.Handler +import android.os.Looper +import android.util.Log +import com.lila.wakeup.Config +import com.lila.wakeup.protocol.BroadcastSender +import java.util.concurrent.atomic.AtomicReference + +/** + * KWS 状态机 —— 协议 v2.1 规则的中央控制器。 + * + * 三个状态： + * - [State.Idle]: 服务启动中（极短） + * - [State.Listening]: 正在监听唤醒词，命中即发 WAKEUP 广播 + * - [State.Paused]: APP 主动暂停（含 silence_ms 二次保险窗口） + * + * silence_ms 实现： + * - 收到 PAUSE → 立即进入 Paused，记录 silenceUntilMs + * - silenceUntilMs 期间收到 RESUME → 等到 silenceUntilMs 过后才真正 resume（防 AI 自我唤醒） + * - 命中事件在 Paused 期间忽略 + * + * 兜底超时：APP 漏发 RESUME 时，2 分钟自动 resume（详见 Config.PAUSE_TIMEOUT_MS）。 + */ +class KwsStateMachine( + private val engine: KwsEngine, + private val sender: BroadcastSender, + private val onTimeoutResume: () -> Unit = {} +) { + companion object { + private const val TAG = "KwsService.State" + } + + sealed class State { + object Idle : State() + object Listening : State() + data class Paused(val silenceUntilMs: Long, val reason: String) : State() + } + + /** 用 Atomic 做无锁状态切换，避免读写竞态 */ + private val state = AtomicReference(State.Idle) + + private val mainHandler = Handler(Looper.getMainLooper()) + private var timeoutRunnable: Runnable? = null + private var deferredResumeRunnable: Runnable? = null + + /** 平滑窗口：连续 N 帧都命中才真正发 WAKEUP（v2.1 后验平滑） */ + private var consecutiveHits = 0 + + fun transitionToListening() { + state.set(State.Listening) + consecutiveHits = 0 + Log.i(TAG, "[State] -> LISTENING") + } + + /** + * 收到 KWS_PAUSE 广播时调用。 + * + * @param silenceMs 静默期（毫秒），默认 [Config.DEFAULT_SILENCE_MS] + * @param reason 调用原因（仅日志） + */ + fun onPauseReceived(silenceMs: Long, reason: String) { + val until = System.currentTimeMillis() + silenceMs + state.set(State.Paused(until, reason)) + consecutiveHits = 0 + Log.i(TAG, "[State] -> PAUSED reason=$reason silence_ms=$silenceMs until=$until") + + // 取消上一次的 deferred resume（如果有） + cancelDeferredResume() + + // 启动 2 分钟兜底超时 + scheduleTimeoutResume() + } + + /** + * 收到 KWS_RESUME 广播时调用。 + */ + fun onResumeReceived(reason: String) { + val current = state.get() + when (current) { + is State.Paused -> { + val now = System.currentTimeMillis() + if (now < current.silenceUntilMs) { + // silence 期间，延迟到 silence 满才真正 resume + val delay = current.silenceUntilMs - now + Log.i(TAG, "[State] RESUME deferred ${delay}ms (still in silence)") + deferredResumeRunnable = Runnable { + if (state.get() is State.Paused) { + transitionToListening() + Log.i(TAG, "[State] RESUME (silence elapsed) reason=$reason") + } + }.also { mainHandler.postDelayed(it, delay) } + } else { + transitionToListening() + Log.i(TAG, "[State] RESUME immediate reason=$reason") + } + } + is State.Listening -> { + // 幂等：本就在监听态，无副作用 + Log.d(TAG, "[State] RESUME ignored, already LISTENING") + } + is State.Idle -> { + transitionToListening() + Log.i(TAG, "[State] Idle -> LISTENING via RESUME reason=$reason") + } + } + cancelTimeoutResume() + } + + /** + * AudioCapture 每帧调用一次。在 Listening 态喂引擎并检查命中。 + */ + fun onAudioFrame(pcm: ShortArray) { + if (state.get() !is State.Listening) return + + val result = engine.process(pcm) + if (result.isHit) { + consecutiveHits++ + if (consecutiveHits >= Config.SMOOTH_FRAMES) { + // 命中且过平滑 → 发 WAKEUP 广播 + Log.i(TAG, "[KWS] HIT confirmed keyword=${result.keyword} smooth=$consecutiveHits") + sender.sendWakeup(result.keyword, result.confidence) + consecutiveHits = 0 + } else { + Log.d(TAG, "[KWS] hit pending smooth=$consecutiveHits/${Config.SMOOTH_FRAMES}") + } + } else { + // 未命中重置计数 + if (consecutiveHits > 0) consecutiveHits = 0 + } + } + + fun shutdown() { + cancelTimeoutResume() + cancelDeferredResume() + state.set(State.Idle) + } + + // ============================================================ + // 内部：兜底定时器 + // ============================================================ + + private fun scheduleTimeoutResume() { + cancelTimeoutResume() + timeoutRunnable = Runnable { + Log.w(TAG, "[Tmout] PAUSE 超时（${Config.PAUSE_TIMEOUT_MS / 1000}s），强制 RESUME") + transitionToListening() + onTimeoutResume() + }.also { + mainHandler.postDelayed(it, Config.PAUSE_TIMEOUT_MS) + } + } + + private fun cancelTimeoutResume() { + timeoutRunnable?.let { mainHandler.removeCallbacks(it) } + timeoutRunnable = null + } + + private fun cancelDeferredResume() { + deferredResumeRunnable?.let { mainHandler.removeCallbacks(it) } + deferredResumeRunnable = null + } +} diff --git a/app/src/main/java/com/lila/wakeup/protocol/BroadcastSender.kt b/app/src/main/java/com/lila/wakeup/protocol/BroadcastSender.kt new file mode 100644 index 0000000..6dee129 --- /dev/null +++ b/app/src/main/java/com/lila/wakeup/protocol/BroadcastSender.kt @@ -0,0 +1,36 @@ +package com.lila.wakeup.protocol + +import android.content.Context +import android.content.Intent +import android.util.Log +import com.lila.wakeup.Config + +/** + * 向数字人 APP 发送 WAKEUP 广播。 + * + * 协议 v2.1 要求： + * - 必须显式 setPackage(Config.APP_PACKAGE) ——> "com.qy.lila" + * - 必须带 keyword / timestamp / confidence 三个 Extras + * + * Android 13 安全规范：隐式广播会被丢弃，setPackage 是强制要求。 + */ +class BroadcastSender(private val context: Context) { + + companion object { + private const val TAG = "KwsService.Send" + } + + fun sendWakeup(keyword: String, confidence: Float) { + val timestamp = System.currentTimeMillis() + + val intent = Intent(Config.ACTION_WAKEUP).apply { + setPackage(Config.APP_PACKAGE) // v2.1 双向 setPackage 强制要求 + putExtra(Config.EXTRA_KEYWORD, keyword) + putExtra(Config.EXTRA_TIMESTAMP, timestamp) + putExtra(Config.EXTRA_CONFIDENCE, confidence) + } + + context.sendBroadcast(intent) + Log.i(TAG, "WAKEUP -> ${Config.APP_PACKAGE} keyword=$keyword conf=$confidence ts=$timestamp") + } +} diff --git a/app/src/main/java/com/lila/wakeup/protocol/ProtocolReceiver.kt b/app/src/main/java/com/lila/wakeup/protocol/ProtocolReceiver.kt new file mode 100644 index 0000000..d238771 --- /dev/null +++ b/app/src/main/java/com/lila/wakeup/protocol/ProtocolReceiver.kt @@ -0,0 +1,63 @@ +package com.lila.wakeup.protocol + +import android.content.BroadcastReceiver +import android.content.Context +import android.content.Intent +import android.util.Log +import com.lila.wakeup.Config + +/** + * 接收 APP 端发来的 KWS_PAUSE / KWS_RESUME 广播。 + * + * 协议 v2.1 字段： + * - silence_ms (long): PAUSE 默认 3000，Networking 60000 + * - reason (String): 透传日志，便于排查 + * + * 实现方式：通过 [WakeupServiceLocator] 静态引用 KwsStateMachine 投递事件， + * 避免 BroadcastReceiver 与 Service 之间频繁 startService 启动开销。 + */ +class ProtocolReceiver : BroadcastReceiver() { + + companion object { + private const val TAG = "KwsService.Proto" + } + + override fun onReceive(context: Context, intent: Intent) { + val action = intent.action ?: return + + // 安全检查：仅处理协议白名单 Action + when (action) { + Config.ACTION_KWS_PAUSE -> handlePause(intent) + Config.ACTION_KWS_RESUME -> handleResume(intent) + else -> Log.w(TAG, "Ignored unknown action: $action") + } + } + + private fun handlePause(intent: Intent) { + val silenceMs = intent.getLongExtra(Config.EXTRA_SILENCE_MS, Config.DEFAULT_SILENCE_MS) + val reason = intent.getStringExtra(Config.EXTRA_REASON) ?: "unknown" + Log.i(TAG, "Recv PAUSE silence_ms=$silenceMs reason=$reason") + + WakeupServiceLocator.stateMachine?.onPauseReceived(silenceMs, reason) + ?: Log.e(TAG, "stateMachine is null! Service not running?") + } + + private fun handleResume(intent: Intent) { + val reason = intent.getStringExtra(Config.EXTRA_REASON) ?: "unknown" + Log.i(TAG, "Recv RESUME reason=$reason") + + WakeupServiceLocator.stateMachine?.onResumeReceived(reason) + ?: Log.e(TAG, "stateMachine is null! Service not running?") + } +} + +/** + * 服务定位器：BroadcastReceiver 与 Service 之间的桥接。 + * + * Service.onCreate 中赋值，onDestroy 中置 null。 + * 该方式避免每次广播都 bindService（开销大），且不引入 AIDL 复杂度。 + */ +object WakeupServiceLocator { + @Volatile + var stateMachine: com.lila.wakeup.kws.KwsStateMachine? = null +} diff --git a/app/src/main/jniLibs/.gitignore b/app/src/main/jniLibs/.gitignore new file mode 100644 index 0000000..949c039 --- /dev/null +++ b/app/src/main/jniLibs/.gitignore @@ -0,0 +1,4 @@ +*.so +*.txt +*.onnx +*.wav diff --git a/app/src/main/jniLibs/arm64-v8a/.gitkeep b/app/src/main/jniLibs/arm64-v8a/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/app/src/main/res/drawable-v24/ic_launcher_foreground.xml b/app/src/main/res/drawable-v24/ic_launcher_foreground.xml new file mode 100644 index 0000000..2b068d1 --- /dev/null +++ b/app/src/main/res/drawable-v24/ic_launcher_foreground.xml @@ -0,0 +1,30 @@ + + + + + + + + + + + \ No newline at end of file diff --git a/app/src/main/res/drawable/ic_launcher_background.xml b/app/src/main/res/drawable/ic_launcher_background.xml new file mode 100644 index 0000000..07d5da9 --- /dev/null +++ b/app/src/main/res/drawable/ic_launcher_background.xml @@ -0,0 +1,170 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/app/src/main/res/mipmap-anydpi-v26/ic_launcher.xml b/app/src/main/res/mipmap-anydpi-v26/ic_launcher.xml new file mode 100644 index 0000000..eca70cf --- /dev/null +++ b/app/src/main/res/mipmap-anydpi-v26/ic_launcher.xml @@ -0,0 +1,5 @@ + + + + + \ No newline at end of file diff --git a/app/src/main/res/mipmap-anydpi-v26/ic_launcher_round.xml b/app/src/main/res/mipmap-anydpi-v26/ic_launcher_round.xml new file mode 100644 index 0000000..eca70cf --- /dev/null +++ b/app/src/main/res/mipmap-anydpi-v26/ic_launcher_round.xml @@ -0,0 +1,5 @@ + + + + + \ No newline at end of file diff --git a/app/src/main/res/mipmap-hdpi/ic_launcher.webp b/app/src/main/res/mipmap-hdpi/ic_launcher.webp new file mode 100644 index 0000000..c209e78 Binary files /dev/null and b/app/src/main/res/mipmap-hdpi/ic_launcher.webp differ diff --git a/app/src/main/res/mipmap-hdpi/ic_launcher_round.webp b/app/src/main/res/mipmap-hdpi/ic_launcher_round.webp new file mode 100644 index 0000000..b2dfe3d Binary files /dev/null and b/app/src/main/res/mipmap-hdpi/ic_launcher_round.webp differ diff --git a/app/src/main/res/mipmap-mdpi/ic_launcher.webp b/app/src/main/res/mipmap-mdpi/ic_launcher.webp new file mode 100644 index 0000000..4f0f1d6 Binary files /dev/null and b/app/src/main/res/mipmap-mdpi/ic_launcher.webp differ diff --git a/app/src/main/res/mipmap-mdpi/ic_launcher_round.webp b/app/src/main/res/mipmap-mdpi/ic_launcher_round.webp new file mode 100644 index 0000000..62b611d Binary files /dev/null and b/app/src/main/res/mipmap-mdpi/ic_launcher_round.webp differ diff --git a/app/src/main/res/mipmap-xhdpi/ic_launcher.webp b/app/src/main/res/mipmap-xhdpi/ic_launcher.webp new file mode 100644 index 0000000..948a307 Binary files /dev/null and b/app/src/main/res/mipmap-xhdpi/ic_launcher.webp differ diff --git a/app/src/main/res/mipmap-xhdpi/ic_launcher_round.webp b/app/src/main/res/mipmap-xhdpi/ic_launcher_round.webp new file mode 100644 index 0000000..1b9a695 Binary files /dev/null and b/app/src/main/res/mipmap-xhdpi/ic_launcher_round.webp differ diff --git a/app/src/main/res/mipmap-xxhdpi/ic_launcher.webp b/app/src/main/res/mipmap-xxhdpi/ic_launcher.webp new file mode 100644 index 0000000..28d4b77 Binary files /dev/null and b/app/src/main/res/mipmap-xxhdpi/ic_launcher.webp differ diff --git a/app/src/main/res/mipmap-xxhdpi/ic_launcher_round.webp b/app/src/main/res/mipmap-xxhdpi/ic_launcher_round.webp new file mode 100644 index 0000000..9287f50 Binary files /dev/null and b/app/src/main/res/mipmap-xxhdpi/ic_launcher_round.webp differ diff --git a/app/src/main/res/mipmap-xxxhdpi/ic_launcher.webp b/app/src/main/res/mipmap-xxxhdpi/ic_launcher.webp new file mode 100644 index 0000000..aa7d642 Binary files /dev/null and b/app/src/main/res/mipmap-xxxhdpi/ic_launcher.webp differ diff --git a/app/src/main/res/mipmap-xxxhdpi/ic_launcher_round.webp b/app/src/main/res/mipmap-xxxhdpi/ic_launcher_round.webp new file mode 100644 index 0000000..9126ae3 Binary files /dev/null and b/app/src/main/res/mipmap-xxxhdpi/ic_launcher_round.webp differ diff --git a/app/src/main/res/values-night/themes.xml b/app/src/main/res/values-night/themes.xml new file mode 100644 index 0000000..e346c65 --- /dev/null +++ b/app/src/main/res/values-night/themes.xml @@ -0,0 +1,16 @@ + + + + \ No newline at end of file diff --git a/app/src/main/res/values/colors.xml b/app/src/main/res/values/colors.xml new file mode 100644 index 0000000..f8c6127 --- /dev/null +++ b/app/src/main/res/values/colors.xml @@ -0,0 +1,10 @@ + + + #FFBB86FC + #FF6200EE + #FF3700B3 + #FF03DAC5 + #FF018786 + #FF000000 + #FFFFFFFF + \ No newline at end of file diff --git a/app/src/main/res/values/strings.xml b/app/src/main/res/values/strings.xml new file mode 100644 index 0000000..f002a81 --- /dev/null +++ b/app/src/main/res/values/strings.xml @@ -0,0 +1,3 @@ + + Lila 语音唤醒 + diff --git a/app/src/main/res/values/themes.xml b/app/src/main/res/values/themes.xml new file mode 100644 index 0000000..cd5c777 --- /dev/null +++ b/app/src/main/res/values/themes.xml @@ -0,0 +1,16 @@ + + + + \ No newline at end of file diff --git a/app/src/main/res/xml/backup_rules.xml b/app/src/main/res/xml/backup_rules.xml new file mode 100644 index 0000000..fa0f996 --- /dev/null +++ b/app/src/main/res/xml/backup_rules.xml @@ -0,0 +1,13 @@ + + + + \ No newline at end of file diff --git a/app/src/main/res/xml/data_extraction_rules.xml b/app/src/main/res/xml/data_extraction_rules.xml new file mode 100644 index 0000000..9ee9997 --- /dev/null +++ b/app/src/main/res/xml/data_extraction_rules.xml @@ -0,0 +1,19 @@ + + + + + + + \ No newline at end of file diff --git a/app/src/test/java/com/k2fsa/sherpa/onnx/ExampleUnitTest.kt b/app/src/test/java/com/k2fsa/sherpa/onnx/ExampleUnitTest.kt new file mode 100644 index 0000000..05dfcd6 --- /dev/null +++ b/app/src/test/java/com/k2fsa/sherpa/onnx/ExampleUnitTest.kt @@ -0,0 +1,17 @@ +package com.k2fsa.sherpa.onnx + +import org.junit.Test + +import org.junit.Assert.* + +/** + * Example local unit test, which will execute on the development machine (host). + * + * See [testing documentation](http://d.android.com/tools/testing). + */ +class ExampleUnitTest { + @Test + fun addition_isCorrect() { + assertEquals(4, 2 + 2) + } +} \ No newline at end of file diff --git a/build.gradle b/build.gradle new file mode 100644 index 0000000..2536974 --- /dev/null +++ b/build.gradle @@ -0,0 +1,6 @@ +// Top-level build file where you can add configuration options common to all sub-projects/modules. +plugins { + id 'com.android.application' version '7.3.1' apply false + id 'com.android.library' version '7.3.1' apply false + id 'org.jetbrains.kotlin.android' version '1.7.20' apply false +} \ No newline at end of file diff --git a/gradle.properties b/gradle.properties new file mode 100644 index 0000000..3c5031e --- /dev/null +++ b/gradle.properties @@ -0,0 +1,23 @@ +# Project-wide Gradle settings. +# IDE (e.g. Android Studio) users: +# Gradle settings configured through the IDE *will override* +# any settings specified in this file. +# For more details on how to configure your build environment visit +# http://www.gradle.org/docs/current/userguide/build_environment.html +# Specifies the JVM arguments used for the daemon process. +# The setting is particularly useful for tweaking memory settings. +org.gradle.jvmargs=-Xmx2048m -Dfile.encoding=UTF-8 +# When configured, Gradle will run in incubating parallel mode. +# This option should only be used with decoupled projects. More details, visit +# http://www.gradle.org/docs/current/userguide/multi_project_builds.html#sec:decoupled_projects +# org.gradle.parallel=true +# AndroidX package structure to make it clearer which packages are bundled with the +# Android operating system, and which are packaged with your app's APK +# https://developer.android.com/topic/libraries/support-library/androidx-rn +android.useAndroidX=true +# Kotlin code style for this project: "official" or "obsolete": +kotlin.code.style=official +# Enables namespacing of each library's R class so that its R class includes only the +# resources declared in the library itself and none from the library's dependencies, +# thereby reducing the size of the R class for that library +android.nonTransitiveRClass=true \ No newline at end of file diff --git a/gradle/wrapper/gradle-wrapper.jar b/gradle/wrapper/gradle-wrapper.jar new file mode 100644 index 0000000..e708b1c Binary files /dev/null and b/gradle/wrapper/gradle-wrapper.jar differ diff --git a/gradle/wrapper/gradle-wrapper.properties b/gradle/wrapper/gradle-wrapper.properties new file mode 100644 index 0000000..2431ea2 --- /dev/null +++ b/gradle/wrapper/gradle-wrapper.properties @@ -0,0 +1,6 @@ +#Thu Feb 23 11:09:06 CST 2023 +distributionBase=GRADLE_USER_HOME +distributionUrl=https\://mirrors.cloud.tencent.com/gradle/gradle-8.2-bin.zip +distributionPath=wrapper/dists +zipStorePath=wrapper/dists +zipStoreBase=GRADLE_USER_HOME diff --git a/gradlew b/gradlew new file mode 100755 index 0000000..4f906e0 --- /dev/null +++ b/gradlew @@ -0,0 +1,185 @@ +#!/usr/bin/env sh + +# +# Copyright 2015 the original author or authors. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# https://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +############################################################################## +## +## Gradle start up script for UN*X +## +############################################################################## + +# Attempt to set APP_HOME +# Resolve links: $0 may be a link +PRG="$0" +# Need this for relative symlinks. +while [ -h "$PRG" ] ; do + ls=`ls -ld "$PRG"` + link=`expr "$ls" : '.*-> \(.*\)$'` + if expr "$link" : '/.*' > /dev/null; then + PRG="$link" + else + PRG=`dirname "$PRG"`"/$link" + fi +done +SAVED="`pwd`" +cd "`dirname \"$PRG\"`/" >/dev/null +APP_HOME="`pwd -P`" +cd "$SAVED" >/dev/null + +APP_NAME="Gradle" +APP_BASE_NAME=`basename "$0"` + +# Add default JVM options here. You can also use JAVA_OPTS and GRADLE_OPTS to pass JVM options to this script. +DEFAULT_JVM_OPTS='"-Xmx64m" "-Xms64m"' + +# Use the maximum available, or set MAX_FD != -1 to use that value. +MAX_FD="maximum" + +warn () { + echo "$*" +} + +die () { + echo + echo "$*" + echo + exit 1 +} + +# OS specific support (must be 'true' or 'false'). +cygwin=false +msys=false +darwin=false +nonstop=false +case "`uname`" in + CYGWIN* ) + cygwin=true + ;; + Darwin* ) + darwin=true + ;; + MINGW* ) + msys=true + ;; + NONSTOP* ) + nonstop=true + ;; +esac + +CLASSPATH=$APP_HOME/gradle/wrapper/gradle-wrapper.jar + + +# Determine the Java command to use to start the JVM. +if [ -n "$JAVA_HOME" ] ; then + if [ -x "$JAVA_HOME/jre/sh/java" ] ; then + # IBM's JDK on AIX uses strange locations for the executables + JAVACMD="$JAVA_HOME/jre/sh/java" + else + JAVACMD="$JAVA_HOME/bin/java" + fi + if [ ! -x "$JAVACMD" ] ; then + die "ERROR: JAVA_HOME is set to an invalid directory: $JAVA_HOME + +Please set the JAVA_HOME variable in your environment to match the +location of your Java installation." + fi +else + JAVACMD="java" + which java >/dev/null 2>&1 || die "ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH. + +Please set the JAVA_HOME variable in your environment to match the +location of your Java installation." +fi + +# Increase the maximum file descriptors if we can. +if [ "$cygwin" = "false" -a "$darwin" = "false" -a "$nonstop" = "false" ] ; then + MAX_FD_LIMIT=`ulimit -H -n` + if [ $? -eq 0 ] ; then + if [ "$MAX_FD" = "maximum" -o "$MAX_FD" = "max" ] ; then + MAX_FD="$MAX_FD_LIMIT" + fi + ulimit -n $MAX_FD + if [ $? -ne 0 ] ; then + warn "Could not set maximum file descriptor limit: $MAX_FD" + fi + else + warn "Could not query maximum file descriptor limit: $MAX_FD_LIMIT" + fi +fi + +# For Darwin, add options to specify how the application appears in the dock +if $darwin; then + GRADLE_OPTS="$GRADLE_OPTS \"-Xdock:name=$APP_NAME\" \"-Xdock:icon=$APP_HOME/media/gradle.icns\"" +fi + +# For Cygwin or MSYS, switch paths to Windows format before running java +if [ "$cygwin" = "true" -o "$msys" = "true" ] ; then + APP_HOME=`cygpath --path --mixed "$APP_HOME"` + CLASSPATH=`cygpath --path --mixed "$CLASSPATH"` + + JAVACMD=`cygpath --unix "$JAVACMD"` + + # We build the pattern for arguments to be converted via cygpath + ROOTDIRSRAW=`find -L / -maxdepth 1 -mindepth 1 -type d 2>/dev/null` + SEP="" + for dir in $ROOTDIRSRAW ; do + ROOTDIRS="$ROOTDIRS$SEP$dir" + SEP="|" + done + OURCYGPATTERN="(^($ROOTDIRS))" + # Add a user-defined pattern to the cygpath arguments + if [ "$GRADLE_CYGPATTERN" != "" ] ; then + OURCYGPATTERN="$OURCYGPATTERN|($GRADLE_CYGPATTERN)" + fi + # Now convert the arguments - kludge to limit ourselves to /bin/sh + i=0 + for arg in "$@" ; do + CHECK=`echo "$arg"|egrep -c "$OURCYGPATTERN" -` + CHECK2=`echo "$arg"|egrep -c "^-"` ### Determine if an option + + if [ $CHECK -ne 0 ] && [ $CHECK2 -eq 0 ] ; then ### Added a condition + eval `echo args$i`=`cygpath --path --ignore --mixed "$arg"` + else + eval `echo args$i`="\"$arg\"" + fi + i=`expr $i + 1` + done + case $i in + 0) set -- ;; + 1) set -- "$args0" ;; + 2) set -- "$args0" "$args1" ;; + 3) set -- "$args0" "$args1" "$args2" ;; + 4) set -- "$args0" "$args1" "$args2" "$args3" ;; + 5) set -- "$args0" "$args1" "$args2" "$args3" "$args4" ;; + 6) set -- "$args0" "$args1" "$args2" "$args3" "$args4" "$args5" ;; + 7) set -- "$args0" "$args1" "$args2" "$args3" "$args4" "$args5" "$args6" ;; + 8) set -- "$args0" "$args1" "$args2" "$args3" "$args4" "$args5" "$args6" "$args7" ;; + 9) set -- "$args0" "$args1" "$args2" "$args3" "$args4" "$args5" "$args6" "$args7" "$args8" ;; + esac +fi + +# Escape application args +save () { + for i do printf %s\\n "$i" | sed "s/'/'\\\\''/g;1s/^/'/;\$s/\$/' \\\\/" ; done + echo " " +} +APP_ARGS=`save "$@"` + +# Collect all arguments for the java command, following the shell quoting and substitution rules +eval set -- $DEFAULT_JVM_OPTS $JAVA_OPTS $GRADLE_OPTS "\"-Dorg.gradle.appname=$APP_BASE_NAME\"" -classpath "\"$CLASSPATH\"" org.gradle.wrapper.GradleWrapperMain "$APP_ARGS" + +exec "$JAVACMD" "$@" diff --git a/gradlew.bat b/gradlew.bat new file mode 100644 index 0000000..ac1b06f --- /dev/null +++ b/gradlew.bat @@ -0,0 +1,89 @@ +@rem +@rem Copyright 2015 the original author or authors. +@rem +@rem Licensed under the Apache License, Version 2.0 (the "License"); +@rem you may not use this file except in compliance with the License. +@rem You may obtain a copy of the License at +@rem +@rem https://www.apache.org/licenses/LICENSE-2.0 +@rem +@rem Unless required by applicable law or agreed to in writing, software +@rem distributed under the License is distributed on an "AS IS" BASIS, +@rem WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +@rem See the License for the specific language governing permissions and +@rem limitations under the License. +@rem + +@if "%DEBUG%" == "" @echo off +@rem ########################################################################## +@rem +@rem Gradle startup script for Windows +@rem +@rem ########################################################################## + +@rem Set local scope for the variables with windows NT shell +if "%OS%"=="Windows_NT" setlocal + +set DIRNAME=%~dp0 +if "%DIRNAME%" == "" set DIRNAME=. +set APP_BASE_NAME=%~n0 +set APP_HOME=%DIRNAME% + +@rem Resolve any "." and ".." in APP_HOME to make it shorter. +for %%i in ("%APP_HOME%") do set APP_HOME=%%~fi + +@rem Add default JVM options here. You can also use JAVA_OPTS and GRADLE_OPTS to pass JVM options to this script. +set DEFAULT_JVM_OPTS="-Xmx64m" "-Xms64m" + +@rem Find java.exe +if defined JAVA_HOME goto findJavaFromJavaHome + +set JAVA_EXE=java.exe +%JAVA_EXE% -version >NUL 2>&1 +if "%ERRORLEVEL%" == "0" goto execute + +echo. +echo ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH. +echo. +echo Please set the JAVA_HOME variable in your environment to match the +echo location of your Java installation. + +goto fail + +:findJavaFromJavaHome +set JAVA_HOME=%JAVA_HOME:"=% +set JAVA_EXE=%JAVA_HOME%/bin/java.exe + +if exist "%JAVA_EXE%" goto execute + +echo. +echo ERROR: JAVA_HOME is set to an invalid directory: %JAVA_HOME% +echo. +echo Please set the JAVA_HOME variable in your environment to match the +echo location of your Java installation. + +goto fail + +:execute +@rem Setup the command line + +set CLASSPATH=%APP_HOME%\gradle\wrapper\gradle-wrapper.jar + + +@rem Execute Gradle +"%JAVA_EXE%" %DEFAULT_JVM_OPTS% %JAVA_OPTS% %GRADLE_OPTS% "-Dorg.gradle.appname=%APP_BASE_NAME%" -classpath "%CLASSPATH%" org.gradle.wrapper.GradleWrapperMain %* + +:end +@rem End local scope for the variables with windows NT shell +if "%ERRORLEVEL%"=="0" goto mainEnd + +:fail +rem Set variable GRADLE_EXIT_CONSOLE if you need the _script_ return code instead of +rem the _cmd.exe /c_ return code! +if not "" == "%GRADLE_EXIT_CONSOLE%" exit 1 +exit /b 1 + +:mainEnd +if "%OS%"=="Windows_NT" endlocal + +:omega diff --git a/settings.gradle b/settings.gradle new file mode 100644 index 0000000..280c95a --- /dev/null +++ b/settings.gradle @@ -0,0 +1,16 @@ +pluginManagement { + repositories { + gradlePluginPortal() + google() + mavenCentral() + } +} +dependencyResolutionManagement { + repositoriesMode.set(RepositoriesMode.FAIL_ON_PROJECT_REPOS) + repositories { + google() + mavenCentral() + } +} +rootProject.name = "LilaWakeup_App" +include ':app'