feat(voice): 新增 picoclaw-voice 语音对话网关
独立模块 cmd/picoclaw-voice，实现端到端语音对话链路（ASR → LLM → TTS）。 协议层基于 xiaozhi WebSocket 协议，做了适当扩展。 - 流式 ASR：豆包 / FunASR（本地） - 流式 TTS：豆包 / Fish Speech（本地） - Docker Compose 一键部署本地 ASR + TTS 服务 - Python demo 客户端用于联调
This commit is contained in:
kkroid
will.zhang
parent
c513ad22d7
commit
350f5a20fd
32 changed files with 4791 additions and 0 deletions
|
|
@ -8,3 +8,6 @@ config/
|
|||
*.md
|
||||
LICENSE
|
||||
assets/
|
||||
# picoclaw-voice binary
|
||||
cmd/picoclaw-voice/picoclaw-voice
|
||||
cmd/picoclaw-voice/.env
|
||||
|
|
|
|||
6
.github/copilot-instructions.md
vendored
Normal file
6
.github/copilot-instructions.md
vendored
Normal file
|
|
@ -0,0 +1,6 @@
|
|||
- 必须用简体中文回答问题，这是铁律！
|
||||
- 需求不明确要先确认再行动
|
||||
- 如果我的要求不合理，你要大胆的提出来，并给出合理的建议
|
||||
- 所有修改以代码质量优先，默认不考虑与旧逻辑兼容
|
||||
- 提交代码时同步更新文档
|
||||
- 注释和代码提交使用中文，日志打印使用英文
|
||||
4
.gitignore
vendored
4
.gitignore
vendored
|
|
@ -10,6 +10,9 @@ build/
|
|||
*.out
|
||||
/picoclaw
|
||||
/picoclaw-test
|
||||
# picoclaw-voice binary (cmd/picoclaw-voice/ module)
|
||||
/picoclaw-voice
|
||||
cmd/picoclaw-voice/picoclaw-voice
|
||||
cmd/**/workspace
|
||||
|
||||
# Picoclaw specific
|
||||
|
|
@ -25,6 +28,7 @@ build/
|
|||
# Secrets & Config (keep templates, ignore actual secrets)
|
||||
.env
|
||||
config/config.json
|
||||
config/config.yaml
|
||||
|
||||
# Test
|
||||
coverage.txt
|
||||
|
|
|
|||
58
cmd/picoclaw-voice/.env.example
Normal file
58
cmd/picoclaw-voice/.env.example
Normal file
|
|
@ -0,0 +1,58 @@
|
|||
# picoclaw-voice 配置样例
|
||||
# 启动方式（推荐）：
|
||||
# cp .env.example .env # 编辑 .env 填入实际值
|
||||
# ./picoclaw-voice
|
||||
|
||||
# ─── 监听 ──────────────────────────────────────────────────────────────────────
|
||||
PICOCLAW_VOICE_LISTEN=:8765
|
||||
|
||||
# ─── picoclaw 配置路径（LLM / 记忆 / MCP 工具） ───────────────────────────────
|
||||
# 默认读取 ~/.picoclaw/config.json，与 picoclaw 进程共享同一份配置。
|
||||
# PICOCLAW_CONFIG=/path/to/config.json
|
||||
# PICOCLAW_HOME=/path/to/.picoclaw
|
||||
|
||||
# ─── 多轮记忆：固定 owner_id ──────────────────────────────────────────────────
|
||||
# 设置后所有设备共享同一份对话记忆；留空则每台设备独立记忆。
|
||||
# PICOCLAW_VOICE_OWNER_ID=alice
|
||||
|
||||
# ══════════════════════════════════════════════════════════════════════════════
|
||||
# 方案 A：doubao（火山引擎）供应商（默认）
|
||||
# ══════════════════════════════════════════════════════════════════════════════
|
||||
# ASR + TTS 使用同一套火山引擎账号时，只需填共享凭证：
|
||||
PICOCLAW_VOICE_APPID=your_volcengine_appid
|
||||
PICOCLAW_VOICE_TOKEN=your_volcengine_token
|
||||
|
||||
# ASR（可选覆盖）
|
||||
PICOCLAW_VOICE_ASR_PROVIDER=doubao
|
||||
# PICOCLAW_VOICE_ASR_APPID= # 覆盖共享 AppID
|
||||
# PICOCLAW_VOICE_ASR_TOKEN= # 覆盖共享 Token
|
||||
PICOCLAW_VOICE_ASR_CLUSTER=bigmodel_transcribe
|
||||
PICOCLAW_VOICE_ASR_RESOURCE_ID=volc.bigasr.sauc.duration
|
||||
|
||||
# TTS（可选覆盖）
|
||||
PICOCLAW_VOICE_TTS_PROVIDER=doubao
|
||||
# PICOCLAW_VOICE_TTS_APPID= # 覆盖共享 AppID
|
||||
# PICOCLAW_VOICE_TTS_TOKEN= # 覆盖共享 Token
|
||||
PICOCLAW_VOICE_TTS_CLUSTER=volcano_tts
|
||||
PICOCLAW_VOICE_TTS_VOICE=zh_female_wanwanxiaohe_moon_bigtts
|
||||
|
||||
# ══════════════════════════════════════════════════════════════════════════════
|
||||
# 方案 B：本地免费方案（FunASR + Fish Speech）
|
||||
# 需要在本机先启动 FunASR（端口 10095）和 Fish Speech（端口 8080）服务
|
||||
# 参见：docker/docker-compose.asr-tts.yml
|
||||
# ══════════════════════════════════════════════════════════════════════════════
|
||||
# ASR：FunASR 本地 WebSocket（支持中文，2pass 模式实时转写+高精度）
|
||||
# PICOCLAW_VOICE_ASR_PROVIDER=funasr
|
||||
# PICOCLAW_VOICE_ASR_WS_URL=wss://127.0.0.1:10095 # 内置自签名证书，跳过校验
|
||||
# PICOCLAW_VOICE_ASR_MODE=2pass # 可选：2pass（默认）| online | offline
|
||||
|
||||
# TTS：Fish Speech 本地 HTTP（PCM 输出，中文效果好，推荐 GPU）
|
||||
# PICOCLAW_VOICE_TTS_PROVIDER=fishspeech
|
||||
# PICOCLAW_VOICE_TTS_API_URL=http://127.0.0.1:8080
|
||||
# PICOCLAW_VOICE_TTS_REFERENCE_ID= # 留空用默认音色
|
||||
# PICOCLAW_VOICE_TTS_SAMPLE_RATE=44100
|
||||
|
||||
# ══════════════════════════════════════════════════════════════════════════════
|
||||
# 方案 C：混搭（例如 doubao ASR + Fish Speech TTS）
|
||||
# ══════════════════════════════════════════════════════════════════════════════
|
||||
# 只需分别设置 ASR/TTS 对应的 PROVIDER 及其凭证字段，两套配置互不干扰。
|
||||
26
cmd/picoclaw-voice/Dockerfile
Normal file
26
cmd/picoclaw-voice/Dockerfile
Normal file
|
|
@ -0,0 +1,26 @@
|
|||
# ─── Build stage ─────────────────────────────────────────────────────────────
|
||||
# Build context: picoclaw repo root
|
||||
# go.mod 中有 replace github.com/sipeed/picoclaw => ../../，需要 repo 根目录可见
|
||||
FROM golang:1.25-alpine AS builder
|
||||
|
||||
# 国内 Go 模块代理
|
||||
ENV GOPROXY=https://goproxy.cn,direct
|
||||
ENV GONOSUMCHECK=*
|
||||
|
||||
# 复制整个 repo，保证 replace 指令的相对路径 ../../ 在容器内成立
|
||||
WORKDIR /build
|
||||
COPY . .
|
||||
|
||||
WORKDIR /build/cmd/picoclaw-voice
|
||||
RUN go mod download
|
||||
# 纯 Go 静态构建，无 CGO 依赖
|
||||
RUN CGO_ENABLED=0 go build -ldflags "-s -w" -o /app/picoclaw-voice .
|
||||
|
||||
# ─── Runtime stage ────────────────────────────────────────────────────────────
|
||||
FROM scratch
|
||||
|
||||
COPY --from=golang:1.25-alpine /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
|
||||
COPY --from=builder /app/picoclaw-voice /picoclaw-voice
|
||||
|
||||
EXPOSE 8765
|
||||
CMD ["/picoclaw-voice"]
|
||||
208
cmd/picoclaw-voice/README.md
Normal file
208
cmd/picoclaw-voice/README.md
Normal file
|
|
@ -0,0 +1,208 @@
|
|||
# picoclaw-voice
|
||||
|
||||
xiaozhi WebSocket 语音网关，将 xiaozhi-esp32 协议设备（ESP32、桌面客户端等）接入 PicoClaw AI 引擎，提供完整的 ASR → LLM → TTS 语音对话流水线。
|
||||
|
||||
## 功能特性
|
||||
|
||||
- **WebSocket 服务端**：实现 [xiaozhi-esp32 协议](../../docs/channels/xiaozhi/README.zh.md) v3，路径 `/xiaozhi/v1/`
|
||||
- **音频格式协商**：服务端在 hello 握手阶段通过 `asr_params` / `tts_params` 下发上下行格式（PCM 或 Opus），客户端自动适配
|
||||
- **三段并发流水线**：实时 ASR（接收音频帧即转写）→ LLM 推理（流式断句）→ TTS 合成推流，端到端首字延迟最优
|
||||
- **多 provider 支持**：ASR 支持豆包（云端）和 FunASR（本地），TTS 支持豆包（云端）和 Fish Speech（本地），可自由混搭
|
||||
- **设备管理**：`device_id` 注册表，同一设备重连时自动驱逐旧会话
|
||||
- **多轮记忆**：通过 `PICOCLAW_VOICE_OWNER_ID` 控制跨设备/跨频道共享同一对话记忆
|
||||
- **LLM thinking 通知**：支持推理模型 `<think>` 块，向客户端发送 `llm.thinking_start/end` 事件
|
||||
|
||||
## 快速开始
|
||||
|
||||
### 直接运行（推荐）
|
||||
|
||||
```bash
|
||||
# 1. 构建
|
||||
cd cmd/picoclaw-voice
|
||||
go build -o picoclaw-voice .
|
||||
|
||||
# 2. 配置
|
||||
cp .env.example .env
|
||||
$EDITOR .env # 至少填写 LLM 配置和 ASR/TTS 凭证
|
||||
|
||||
# 3. 启动
|
||||
./picoclaw-voice
|
||||
```
|
||||
|
||||
### Docker
|
||||
|
||||
```bash
|
||||
# 从 repo 根目录构建（Dockerfile context 需要整个 repo）
|
||||
docker build -f cmd/picoclaw-voice/Dockerfile -t picoclaw-voice .
|
||||
|
||||
docker run -d --name picoclaw-voice \
|
||||
--env-file cmd/picoclaw-voice/.env \
|
||||
-v ~/.picoclaw:/root/.picoclaw \
|
||||
-p 8765:8765 \
|
||||
picoclaw-voice
|
||||
```
|
||||
|
||||
### 连接到 picoclaw
|
||||
|
||||
picoclaw-voice 读取 `~/.picoclaw/config.json`（与 picoclaw 主进程共享同一份配置）以获取 LLM provider、记忆后端和 MCP 工具配置。
|
||||
|
||||
确保 picoclaw 已完成 [初始化配置](https://github.com/sipeed/picoclaw#configuration) 后再启动 picoclaw-voice。
|
||||
|
||||
## 环境变量
|
||||
|
||||
> 加载优先级（高→低）：Shell 进程环境 → `.env` 文件 → 代码默认值
|
||||
|
||||
### 通用
|
||||
|
||||
| 变量 | 默认值 | 说明 |
|
||||
|------|--------|------|
|
||||
| `PICOCLAW_VOICE_LISTEN` | `:8765` | WebSocket 监听地址 |
|
||||
| `PICOCLAW_VOICE_OWNER_ID` | *(空)* | 固定 owner_id；空时每台设备独立记忆 |
|
||||
| `PICOCLAW_CONFIG` | `~/.picoclaw/config.json` | picoclaw 配置文件路径 |
|
||||
| `PICOCLAW_HOME` | `~/.picoclaw` | picoclaw home 目录（次优先） |
|
||||
|
||||
### ASR 配置
|
||||
|
||||
| 变量 | 默认值 | 说明 |
|
||||
|------|--------|------|
|
||||
| `PICOCLAW_VOICE_ASR_PROVIDER` | `doubao` | ASR 供应商：`doubao` \| `funasr` |
|
||||
| `PICOCLAW_VOICE_APPID` | | 火山引擎 AppID（`doubao` provider 兜底） |
|
||||
| `PICOCLAW_VOICE_TOKEN` | | 火山引擎 Token（`doubao` provider 兜底） |
|
||||
| `PICOCLAW_VOICE_ASR_APPID` | | 单独覆盖 ASR AppID |
|
||||
| `PICOCLAW_VOICE_ASR_TOKEN` | | 单独覆盖 ASR Token |
|
||||
| `PICOCLAW_VOICE_ASR_CLUSTER` | `bigmodel_transcribe` | doubao ASR 集群 |
|
||||
| `PICOCLAW_VOICE_ASR_RESOURCE_ID` | `volc.bigasr.sauc.duration` | doubao ASR 资源 ID（见下方模型列表） |
|
||||
| `PICOCLAW_VOICE_ASR_WS_URL` | `wss://127.0.0.1:10095` | funasr WebSocket 地址（内置自签名证书，自动跳过校验） |
|
||||
| `PICOCLAW_VOICE_ASR_MODE` | `2pass` | funasr 识别模式：`2pass` \| `online` \| `offline` |
|
||||
|
||||
### TTS 配置
|
||||
|
||||
| 变量 | 默认值 | 说明 |
|
||||
|------|--------|------|
|
||||
| `PICOCLAW_VOICE_TTS_PROVIDER` | `doubao` | TTS 供应商：`doubao` \| `fishspeech` |
|
||||
| `PICOCLAW_VOICE_TTS_APPID` | | 单独覆盖 TTS AppID（doubao） |
|
||||
| `PICOCLAW_VOICE_TTS_TOKEN` | | 单独覆盖 TTS Token（doubao） |
|
||||
| `PICOCLAW_VOICE_TTS_CLUSTER` | `volcano_tts` | doubao TTS 集群 |
|
||||
| `PICOCLAW_VOICE_TTS_VOICE` | | doubao 音色 ID，见[火山引擎音色列表](https://www.volcengine.com/docs/6561/97465) |
|
||||
| `PICOCLAW_VOICE_TTS_API_URL` | `http://127.0.0.1:8080` | fishspeech HTTP 服务地址 |
|
||||
| `PICOCLAW_VOICE_TTS_API_KEY` | | fishspeech Bearer Token（不需要鉴权时留空） |
|
||||
| `PICOCLAW_VOICE_TTS_REFERENCE_ID` | | fishspeech 参考音色 ID（留空用服务默认） |
|
||||
| `PICOCLAW_VOICE_TTS_SAMPLE_RATE` | `0` | fishspeech 输出采样率（0 = provider 默认，通常为 44100 Hz） |
|
||||
|
||||
## ASR Provider 说明
|
||||
|
||||
### doubao（豆包，默认）
|
||||
|
||||
调用[火山引擎大模型语音识别](https://console.volcengine.com/speech/service/10)，流式实时转写。
|
||||
|
||||
**凭证填写方式（两选一）：**
|
||||
- **API Key 模式**（推荐）：只填 `PICOCLAW_VOICE_TOKEN`（UUID 格式），`APPID` 留空
|
||||
- **App Key 模式**：填数字格式 `PICOCLAW_VOICE_APPID` + `PICOCLAW_VOICE_TOKEN`
|
||||
|
||||
**模型选择（`PICOCLAW_VOICE_ASR_RESOURCE_ID`）：**
|
||||
- `volc.bigasr.sauc.duration` — 豆包流式 ASR 1.0（默认，稳定）
|
||||
- `volc.seedasr.sauc.duration` — 豆包流式 ASR 2.0 Seed（更高精度）
|
||||
|
||||
### funasr（本地免费）
|
||||
|
||||
连接本机运行的 [FunASR](https://github.com/modelscope/FunASR) 推理服务，免费且支持中文。
|
||||
|
||||
**快速启动 FunASR 容器：**
|
||||
|
||||
```bash
|
||||
# 从 repo 根目录执行
|
||||
docker compose -f docker/docker-compose.asr-tts.yml up -d funasr
|
||||
```
|
||||
|
||||
模型首次启动时自动从 ModelScope 下载（约 1 GB），后续启动使用缓存（`~/models/funasr/`）。
|
||||
|
||||
## TTS Provider 说明
|
||||
|
||||
### doubao（豆包，默认）
|
||||
|
||||
调用[火山引擎语音合成](https://console.volcengine.com/speech/service/8)，输出 Opus 音频流。凭证配置同 doubao ASR。
|
||||
|
||||
### fishspeech（本地免费，推荐 GPU）
|
||||
|
||||
连接本机运行的 [Fish Speech](https://github.com/fishaudio/fish-speech) v1.5.x 服务，输出 PCM 音频流（44100 Hz），效果自然、中文优秀。
|
||||
|
||||
**快速启动 Fish Speech 容器（需 NVIDIA GPU）：**
|
||||
|
||||
```bash
|
||||
# 安装 NVIDIA Container Toolkit（首次）
|
||||
# 参见：https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html
|
||||
|
||||
# 从 repo 根目录执行
|
||||
docker compose -f docker/docker-compose.asr-tts.yml up -d fishspeech
|
||||
```
|
||||
|
||||
模型已内置于 `fishaudio/fish-speech:v1.5.1` 镜像，无需单独挂载。
|
||||
|
||||
## 本地免费方案（FunASR + Fish Speech）
|
||||
|
||||
一键启动完整本地推理栈：
|
||||
|
||||
```bash
|
||||
docker compose -f docker/docker-compose.asr-tts.yml up -d
|
||||
```
|
||||
|
||||
然后在 `.env` 中切换 provider：
|
||||
|
||||
```dotenv
|
||||
PICOCLAW_VOICE_ASR_PROVIDER=funasr
|
||||
PICOCLAW_VOICE_ASR_WS_URL=wss://127.0.0.1:10095
|
||||
|
||||
PICOCLAW_VOICE_TTS_PROVIDER=fishspeech
|
||||
PICOCLAW_VOICE_TTS_API_URL=http://127.0.0.1:8080
|
||||
```
|
||||
|
||||
也可以混搭，例如云端 ASR + 本地 TTS，只需分别设置对应的 `PROVIDER` 变量即可。
|
||||
|
||||
## 协议说明
|
||||
|
||||
picoclaw-voice 实现 xiaozhi-esp32 协议 v3，并在此基础上做了少量扩展。完整协议文档见 [docs/channels/xiaozhi/README.zh.md](../../docs/channels/xiaozhi/README.zh.md)。
|
||||
|
||||
**音频格式协商流程：**
|
||||
|
||||
1. 客户端发送 `hello`，携带上行 `audio_params`（建议 PCM 16kHz mono）
|
||||
2. 服务端回 `hello`，携带：
|
||||
- `asr_params`：服务端期望的**上行**音频格式（客户端按此格式发送语音帧）
|
||||
- `tts_params`：服务端实际的**下行** TTS 格式（doubao 输出 Opus，fishspeech 输出 PCM）
|
||||
3. 后续二进制帧格式以协商结果为准
|
||||
|
||||
**picoclaw 扩展字段：**
|
||||
|
||||
| 方向 | 字段 | 说明 |
|
||||
|------|------|------|
|
||||
| 客户端→服务端 | `listen.memory_id` | 指定 LLM 多轮记忆 key，相同 key 共享对话上下文（跨设备/跨会话） |
|
||||
| 服务端→客户端 | `hello.session_id` | 连接级 UUID，用于日志关联 |
|
||||
| 服务端→客户端 | `llm`（新消息类型） | LLM 推理通知：`{"type":"llm","text":"..."}` 断句文本；`thinking_start` / `thinking_end` 思考链事件 |
|
||||
|
||||
## Demo
|
||||
|
||||
Python 演示客户端，完整演示握手→录音→流式 ASR→LLM→TTS 播放全流程。
|
||||
|
||||
**系统依赖（仅 doubao TTS/ASR 的 Opus 模式需要）：**
|
||||
|
||||
```bash
|
||||
# Debian/Ubuntu
|
||||
sudo apt install portaudio19-dev libopus-dev
|
||||
|
||||
# macOS
|
||||
brew install portaudio opus
|
||||
```
|
||||
|
||||
**运行：**
|
||||
|
||||
```bash
|
||||
cd cmd/picoclaw-voice/demo
|
||||
pip install -r requirements.txt
|
||||
|
||||
# 麦克风模式：按空格开始/停止录音
|
||||
python client.py --url ws://localhost:8765/xiaozhi/v1/
|
||||
|
||||
# 文件模式：自动发送音频文件后退出
|
||||
python client.py --audio-file /tmp/input.wav
|
||||
```
|
||||
|
||||
详见 [demo/README.md](demo/README.md)。
|
||||
62
cmd/picoclaw-voice/demo/README.md
Normal file
62
cmd/picoclaw-voice/demo/README.md
Normal file
|
|
@ -0,0 +1,62 @@
|
|||
# picoclaw-voice demo
|
||||
|
||||
演示如何通过 xiaozhi 协议连接 picoclaw-voice，完整体验 ASR → LLM → TTS 流水线。
|
||||
|
||||
支持两种 TTS 下行音频格式（由服务端协商决定，客户端自动适配）：
|
||||
- **opus**：doubao TTS 输出，需要 `opuslib` 解码
|
||||
- **pcm**：Fish Speech TTS 输出，直接播放 s16le PCM
|
||||
|
||||
## 系统依赖
|
||||
|
||||
`opuslib` 仅在 doubao TTS（Opus 格式）时需要：
|
||||
|
||||
```bash
|
||||
# Debian/Ubuntu
|
||||
sudo apt install portaudio19-dev libopus-dev
|
||||
|
||||
# macOS
|
||||
brew install portaudio opus
|
||||
```
|
||||
|
||||
使用 Fish Speech TTS（PCM 格式）时无需 opus 系统库。
|
||||
|
||||
## 安装 Python 依赖
|
||||
|
||||
```bash
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
`opuslib` 为可选依赖，仅 doubao Opus 格式时使用。
|
||||
|
||||
## 运行
|
||||
|
||||
```bash
|
||||
# 麦克风模式：按空格开始录音，再按空格停止
|
||||
python client.py
|
||||
|
||||
# 指定服务地址
|
||||
python client.py --url ws://192.168.1.100:8765/xiaozhi/v1/
|
||||
|
||||
# 文件模式：自动发送音频文件，TTS 播放完毕后退出
|
||||
python client.py --audio-file /tmp/input.wav
|
||||
```
|
||||
|
||||
## 输出示例
|
||||
|
||||
```
|
||||
[连接] ws://localhost:8765/xiaozhi/v1/
|
||||
[握手] session_id=a1b2c3d4-...
|
||||
↑ ASR上行: pcm 16000Hz 1ch
|
||||
↓ TTS下行: opus 24000Hz 1ch
|
||||
[就绪] 按空格开始/停止录音，Ctrl+C 退出
|
||||
|
||||
────────────────────────────────────────
|
||||
[●] 音频数据发送中... (再按空格停止)
|
||||
[○] 音频数据发送已停止...
|
||||
[流式识别结果] 你好
|
||||
[你] 你好，今天天气怎么样？
|
||||
[LLM] 您好！今天北京的天气是晴天，气温约 25 度...
|
||||
[合成中]
|
||||
[合成完毕]
|
||||
```
|
||||
|
||||
439
cmd/picoclaw-voice/demo/client.py
Normal file
439
cmd/picoclaw-voice/demo/client.py
Normal file
|
|
@ -0,0 +1,439 @@
|
|||
#!/usr/bin/env python3
|
||||
"""
|
||||
picoclaw-voice 演示客户端
|
||||
|
||||
完整演示 ASR → LLM → TTS 流水线及 picoclaw 协议（音频格式协商、llm thinking 事件）。
|
||||
|
||||
支持两种 TTS 下行格式（由服务端协商）：
|
||||
- opus：接收 Opus 包，用 opuslib 解码后播放（doubao TTS）
|
||||
- pcm：接收原始 PCM s16le 数据，直接播放（Fish Speech TTS）
|
||||
|
||||
协议流程：
|
||||
1. 客户端发 hello
|
||||
2. 服务端回 hello，携带 asr_params / tts_params（协商音频格式）
|
||||
3. 按空格 → 发 listen.start，流式发送麦克风音频帧
|
||||
4. 再按空格 → 发 listen.end，停止发送
|
||||
5. 服务端推送事件：stt / llm / tts
|
||||
6. 服务端在 tts 期间推送二进制 Opus 帧
|
||||
|
||||
用法：
|
||||
python client.py --url ws://HOST:18765/xiaozhi/v1/
|
||||
python client.py --audio-file /tmp/input.wav # 文件模式，自动发送后退出
|
||||
|
||||
依赖：pip install -r requirements.txt
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import platform
|
||||
import queue
|
||||
import signal
|
||||
import sys
|
||||
import threading
|
||||
import time
|
||||
import uuid
|
||||
|
||||
try:
|
||||
import miniaudio
|
||||
import pyaudio
|
||||
import websocket
|
||||
except ImportError as e:
|
||||
print(f"缺少依赖：{e}")
|
||||
print("请先执行：pip install -r requirements.txt")
|
||||
sys.exit(1)
|
||||
|
||||
try:
|
||||
import opuslib
|
||||
_HAS_OPUSLIB = True
|
||||
except ImportError:
|
||||
_HAS_OPUSLIB = False
|
||||
|
||||
SAMPLE_RATE = 16000
|
||||
CHANNELS = 1
|
||||
FRAME_DURATION_MS = 60
|
||||
FRAME_SAMPLES = SAMPLE_RATE * FRAME_DURATION_MS // 1000 # 960 samples = 60ms
|
||||
PYAUDIO_FORMAT = pyaudio.paInt16
|
||||
|
||||
|
||||
class PicoclawVoiceClient:
|
||||
|
||||
def __init__(self, url: str):
|
||||
self.url = url
|
||||
self.device_id = "demo:" + uuid.uuid4().hex[:8]
|
||||
self.ws = None
|
||||
|
||||
# 握手后由服务端 hello 的 audio_params / tts_params 更新
|
||||
self.audio_fmt = "pcm" # 上行 ASR 格式
|
||||
self.tts_sample_rate = SAMPLE_RATE # 下行 TTS 采样率
|
||||
self.tts_channels = CHANNELS # 下行 TTS 声道数
|
||||
self.tts_format = "opus" # 下行 TTS 编码格式
|
||||
self.handshake_done = threading.Event()
|
||||
self._stop = threading.Event() # 通知播放线程退出
|
||||
self._is_listening = False # 是否正在发送音频帧给服务端
|
||||
self._pushing = False # 空格是否正被按住
|
||||
self._file_mode = False # 文件输入模式
|
||||
|
||||
self.pa = pyaudio.PyAudio()
|
||||
self._audio_device = self._find_pulse_device()
|
||||
self.audio_out_q: queue.Queue = queue.Queue()
|
||||
self.dec = None # 据 tts_format 延迟初始化
|
||||
self.enc = None # 据 audio_fmt 延迟初始化
|
||||
|
||||
def _log(self, msg: str = ""):
|
||||
# raw tty 模式（Linux）下 \n 不回行首，必须用 \r\n
|
||||
end = "\n" if platform.system() == "Windows" else "\r\n"
|
||||
sys.stdout.write(msg + end)
|
||||
sys.stdout.flush()
|
||||
|
||||
def _find_pulse_device(self):
|
||||
"""返回 PulseAudio 设备索引，找不到时返回 None（让 PyAudio 用系统默认）。"""
|
||||
for i in range(self.pa.get_device_count()):
|
||||
info = self.pa.get_device_info_by_index(i)
|
||||
if "pulse" in info["name"].lower():
|
||||
return i
|
||||
return None
|
||||
|
||||
# ── WebSocket 回调 ────────────────────────────────────────────────────────
|
||||
|
||||
def on_open(self, ws):
|
||||
self._log(f"[连接] {self.url}")
|
||||
ws.send(json.dumps({
|
||||
"type": "hello",
|
||||
"version": 3,
|
||||
"transport": "websocket",
|
||||
"device_id": self.device_id,
|
||||
"audio_params": {
|
||||
"format": "pcm",
|
||||
"sample_rate": SAMPLE_RATE,
|
||||
"channels": CHANNELS,
|
||||
"frame_duration": FRAME_DURATION_MS,
|
||||
},
|
||||
}))
|
||||
|
||||
def on_message(self, ws, message):
|
||||
if isinstance(message, bytes):
|
||||
# 下行 Opus 音频帧（tts.sentence_start/end 之间）
|
||||
self.audio_out_q.put(message)
|
||||
return
|
||||
try:
|
||||
msg = json.loads(message)
|
||||
except json.JSONDecodeError:
|
||||
return
|
||||
|
||||
mtype = msg.get("type", "")
|
||||
|
||||
if mtype == "hello":
|
||||
# 上行：ASR 期望格式（客户端发送音频给服务端）
|
||||
asr_params = msg.get("asr_params", {})
|
||||
self.audio_fmt = asr_params.get("format", "pcm")
|
||||
# 下行：TTS 输出格式（服务端发送 Opus 音频给客户端）
|
||||
tts_params = msg.get("tts_params", {})
|
||||
self.tts_sample_rate = tts_params.get("sample_rate", SAMPLE_RATE)
|
||||
self.tts_channels = tts_params.get("channels", CHANNELS)
|
||||
self.tts_format = tts_params.get("format", "opus")
|
||||
if self.tts_format == "opus":
|
||||
if not _HAS_OPUSLIB:
|
||||
raise RuntimeError("TTS 格式为 opus，但 opuslib 未安装：pip install opuslib")
|
||||
self.dec = opuslib.Decoder(self.tts_sample_rate, self.tts_channels)
|
||||
if self.audio_fmt == "opus":
|
||||
if not _HAS_OPUSLIB:
|
||||
raise RuntimeError("ASR 上行格式为 opus，但 opuslib 未安装：pip install opuslib")
|
||||
self.enc = opuslib.Encoder(SAMPLE_RATE, CHANNELS, opuslib.APPLICATION_VOIP)
|
||||
self._log(f"[握手] session_id={msg.get('session_id', '')}")
|
||||
self._log(f" ↑ ASR上行: {self.audio_fmt} {asr_params.get('sample_rate')}Hz {asr_params.get('channels')}ch")
|
||||
self._log(f" ↓ TTS下行: {self.tts_format} {self.tts_sample_rate}Hz {self.tts_channels}ch")
|
||||
self.handshake_done.set()
|
||||
if self._file_mode:
|
||||
self._log("[就绪] 文件模式，正在自动发送音频...")
|
||||
else:
|
||||
self._log("[就绪] 按空格开始/停止录音，Ctrl+C 退出")
|
||||
|
||||
elif mtype == "stt":
|
||||
state = msg.get("state", "")
|
||||
text = msg.get("text", "")
|
||||
if state == "recognizing" and text:
|
||||
self._log(f" [流式识别结果] {text}")
|
||||
elif state == "stop":
|
||||
if text:
|
||||
self._log(f" [你] {text}")
|
||||
else:
|
||||
self._log(" [你] （静音/未识别）")
|
||||
|
||||
elif mtype == "llm":
|
||||
state = msg.get("state", "")
|
||||
text = msg.get("text", "")
|
||||
if state == "thinking_start":
|
||||
self._log(" ⏳ 思考中...")
|
||||
elif state == "thinking_end":
|
||||
self._log(f" ✓ 思考完成（{msg.get('duration_ms', 0)}ms）")
|
||||
elif text:
|
||||
self._log(f" [LLM] {text}")
|
||||
|
||||
elif mtype == "tts":
|
||||
state = msg.get("state", "")
|
||||
if state == "start":
|
||||
self._log(" [合成中]")
|
||||
elif state == "stop":
|
||||
self._log(" [合成完毕]")
|
||||
if self._file_mode:
|
||||
threading.Thread(target=lambda: (time.sleep(0.5), self.ws.close()), daemon=True).start()
|
||||
else:
|
||||
self._log("")
|
||||
self._log("[就绪] 按空格开始/停止录音，Ctrl+C 退出")
|
||||
elif state == "abort":
|
||||
self._log(" [合成中断]")
|
||||
if not self._file_mode:
|
||||
self._log("")
|
||||
self._log("[就绪] 按空格开始/停止录音，Ctrl+C 退出")
|
||||
|
||||
def on_error(self, ws, error):
|
||||
self._log(f"[错误] {error}")
|
||||
|
||||
def on_close(self, ws, code, msg):
|
||||
self._log(f"[断开] {code} {msg}")
|
||||
self.handshake_done.set()
|
||||
|
||||
# ── PTT 按键控制 ───────────────────────────────────────────────────────────
|
||||
|
||||
def _key_thread(self):
|
||||
"""跨平台按键读取：空格切换 PTT，Ctrl+C 退出。"""
|
||||
self.handshake_done.wait()
|
||||
if platform.system() == "Windows":
|
||||
import msvcrt
|
||||
while not self._stop.is_set():
|
||||
if msvcrt.kbhit():
|
||||
ch = msvcrt.getwch()
|
||||
if ch == ' ':
|
||||
self._push_start() if not self._pushing else self._push_end()
|
||||
elif ch == '\x03': # Ctrl+C
|
||||
self._stop.set()
|
||||
if self.ws:
|
||||
self.ws.close()
|
||||
os.kill(os.getpid(), signal.SIGINT)
|
||||
break
|
||||
else:
|
||||
threading.Event().wait(0.05)
|
||||
else:
|
||||
import select, termios, tty
|
||||
fd = sys.stdin.fileno()
|
||||
old = termios.tcgetattr(fd)
|
||||
try:
|
||||
tty.setraw(fd)
|
||||
while not self._stop.is_set():
|
||||
r, _, _ = select.select([fd], [], [], 0.1)
|
||||
if not r:
|
||||
continue
|
||||
ch = os.read(fd, 1)
|
||||
if ch == b' ':
|
||||
self._push_start() if not self._pushing else self._push_end()
|
||||
elif ch in (b'\x03', b'\x1c'):
|
||||
self._stop.set()
|
||||
if self.ws:
|
||||
self.ws.close()
|
||||
os.kill(os.getpid(), signal.SIGINT)
|
||||
break
|
||||
finally:
|
||||
termios.tcsetattr(fd, termios.TCSADRAIN, old)
|
||||
|
||||
def _push_start(self):
|
||||
"""空格按下：发 listen.start，开始流式发送音频。"""
|
||||
if self._pushing or self._stop.is_set():
|
||||
return
|
||||
self._pushing = True
|
||||
sid = str(uuid.uuid4())
|
||||
self._is_listening = True
|
||||
try:
|
||||
self.ws.send(json.dumps({"type": "listen", "state": "start", "session_id": sid}))
|
||||
self._log("")
|
||||
self._log("─" * 40)
|
||||
self._log("[●] 音频数据发送中... (再按空格停止)")
|
||||
except Exception:
|
||||
self._is_listening = False
|
||||
self._pushing = False
|
||||
|
||||
def _push_end(self):
|
||||
"""空格松开：停止发送音频，发 listen.end。"""
|
||||
if not self._pushing:
|
||||
return
|
||||
self._pushing = False
|
||||
self._is_listening = False
|
||||
try:
|
||||
self.ws.send(json.dumps({"type": "listen", "state": "end"}))
|
||||
self._log("[○] 音频数据发送已停止...")
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# ── 下行音频播放（兼容 opus / pcm）────────────────────────────────────────
|
||||
|
||||
def _playback_thread(self):
|
||||
self.handshake_done.wait()
|
||||
max_frame_size = self.tts_sample_rate * 120 // 1000
|
||||
bytes_per_frame = self.tts_channels * 2 # s16le
|
||||
|
||||
_buf = bytearray()
|
||||
_lock = threading.Lock()
|
||||
|
||||
def _feed_loop():
|
||||
while not self._stop.is_set():
|
||||
try:
|
||||
frame = self.audio_out_q.get(timeout=0.1)
|
||||
if self.tts_format == "opus":
|
||||
pcm = self.dec.decode(frame, max_frame_size)
|
||||
else:
|
||||
pcm = frame # PCM 直接使用
|
||||
with _lock:
|
||||
_buf.extend(pcm)
|
||||
except queue.Empty:
|
||||
pass
|
||||
except Exception as e:
|
||||
self._log(f'[解码错误] {e}')
|
||||
|
||||
threading.Thread(target=_feed_loop, daemon=True).start()
|
||||
|
||||
def _pcm_stream():
|
||||
num_frames = yield b"" # 预激，接收首次 send(num_frames)
|
||||
while True:
|
||||
if self._stop.is_set():
|
||||
return # 生成器结束，miniaudio 停止播放并退出 dev.start()
|
||||
needed = num_frames * bytes_per_frame
|
||||
with _lock:
|
||||
if len(_buf) >= needed:
|
||||
chunk = bytes(_buf[:needed])
|
||||
del _buf[:needed]
|
||||
else:
|
||||
chunk = bytes(_buf) + b'\x00' * (needed - len(_buf))
|
||||
_buf.clear()
|
||||
num_frames = yield chunk
|
||||
|
||||
gen = _pcm_stream()
|
||||
next(gen)
|
||||
with miniaudio.PlaybackDevice(
|
||||
output_format=miniaudio.SampleFormat.SIGNED16,
|
||||
nchannels=self.tts_channels,
|
||||
sample_rate=self.tts_sample_rate,
|
||||
) as dev:
|
||||
dev.start(gen)
|
||||
self._stop.wait()
|
||||
|
||||
# ── 文件模式：从音频文件读取并发送 ASR 帧 ──────────────────────────────────
|
||||
|
||||
def _file_record_thread(self, audio_file: str):
|
||||
"""读取音频文件（自动重采样到 16kHz mono），以实时速率流式发送给 ASR。"""
|
||||
self.handshake_done.wait()
|
||||
try:
|
||||
decoded = miniaudio.decode_file(
|
||||
audio_file,
|
||||
output_format=miniaudio.SampleFormat.SIGNED16,
|
||||
nchannels=CHANNELS,
|
||||
sample_rate=SAMPLE_RATE,
|
||||
)
|
||||
except Exception as e:
|
||||
self._log(f"[文件读取失败] {e}")
|
||||
self._stop.set()
|
||||
return
|
||||
|
||||
pcm_data = bytes(decoded.samples)
|
||||
frame_bytes = FRAME_SAMPLES * CHANNELS * 2 # s16le，每帧字节数
|
||||
duration_s = len(pcm_data) / 2 / SAMPLE_RATE
|
||||
|
||||
sid = str(uuid.uuid4())
|
||||
self.ws.send(json.dumps({"type": "listen", "state": "start", "session_id": sid}))
|
||||
self._log("─" * 40)
|
||||
self._log(f"[文件模式] 发送: {os.path.basename(audio_file)} ({duration_s:.1f}s)")
|
||||
|
||||
offset = 0
|
||||
while offset < len(pcm_data) and not self._stop.is_set():
|
||||
chunk = pcm_data[offset:offset + frame_bytes]
|
||||
if len(chunk) < frame_bytes:
|
||||
chunk = chunk + b'\x00' * (frame_bytes - len(chunk))
|
||||
if self.audio_fmt == "opus":
|
||||
chunk = bytes(self.enc.encode(chunk, FRAME_SAMPLES))
|
||||
self.ws.send(chunk, opcode=websocket.ABNF.OPCODE_BINARY)
|
||||
offset += frame_bytes
|
||||
time.sleep(FRAME_DURATION_MS / 1000.0) # 模拟实时节奏
|
||||
|
||||
self.ws.send(json.dumps({"type": "listen", "state": "end"}))
|
||||
self._log("[文件模式] 发送完毕，等待识别...")
|
||||
|
||||
# ── 麦克风录音：持续采集，_is_listening 控制是否发送 ─────────────────────
|
||||
|
||||
def _record_thread(self):
|
||||
"""持续录制麦克风音频，仅在 _is_listening=True 时向服务端发送帧。"""
|
||||
self.handshake_done.wait()
|
||||
|
||||
stream = self.pa.open(
|
||||
format=PYAUDIO_FORMAT, channels=CHANNELS,
|
||||
rate=SAMPLE_RATE, input=True,
|
||||
input_device_index=self._audio_device,
|
||||
frames_per_buffer=FRAME_SAMPLES,
|
||||
)
|
||||
try:
|
||||
while not self._stop.is_set():
|
||||
try:
|
||||
pcm_bytes = stream.read(FRAME_SAMPLES, exception_on_overflow=False)
|
||||
except OSError:
|
||||
break
|
||||
if not self._is_listening:
|
||||
continue
|
||||
if self.audio_fmt == "opus":
|
||||
frame = bytes(self.enc.encode(pcm_bytes, FRAME_SAMPLES))
|
||||
else:
|
||||
frame = pcm_bytes
|
||||
try:
|
||||
self.ws.send(frame, opcode=websocket.ABNF.OPCODE_BINARY)
|
||||
except Exception:
|
||||
break
|
||||
finally:
|
||||
stream.stop_stream()
|
||||
stream.close()
|
||||
|
||||
# ── 主入口 ────────────────────────────────────────────────────────────────
|
||||
|
||||
def run(self, audio_file: str = ""):
|
||||
self._file_mode = bool(audio_file)
|
||||
ws = websocket.WebSocketApp(
|
||||
self.url,
|
||||
on_open=self.on_open,
|
||||
on_message=self.on_message,
|
||||
on_error=self.on_error,
|
||||
on_close=self.on_close,
|
||||
)
|
||||
self.ws = ws
|
||||
|
||||
playback_t = threading.Thread(target=self._playback_thread, daemon=True)
|
||||
playback_t.start()
|
||||
if audio_file:
|
||||
threading.Thread(target=self._file_record_thread, args=(audio_file,), daemon=True).start()
|
||||
else:
|
||||
threading.Thread(target=self._record_thread, daemon=True).start()
|
||||
threading.Thread(target=self._key_thread, daemon=True).start()
|
||||
|
||||
try:
|
||||
ws.run_forever()
|
||||
except KeyboardInterrupt:
|
||||
pass
|
||||
self._stop.set()
|
||||
playback_t.join(timeout=1.0)
|
||||
self.pa.terminate()
|
||||
|
||||
|
||||
def main():
|
||||
if platform.system() != "Windows":
|
||||
# 压制 ALSA/PortAudio C 层噪声（不影响 Python stderr）
|
||||
_devnull = os.open(os.devnull, os.O_WRONLY)
|
||||
os.dup2(_devnull, 2)
|
||||
os.close(_devnull)
|
||||
|
||||
parser = argparse.ArgumentParser(description="picoclaw-voice 演示客户端")
|
||||
parser.add_argument("--url", default="ws://127.0.0.1:18765/xiaozhi/v1/",
|
||||
help="服务端 WebSocket 地址")
|
||||
parser.add_argument("--audio-file", default="",
|
||||
help="音频文件路径（代替麦克风，自动发送后退出）")
|
||||
args = parser.parse_args()
|
||||
|
||||
PicoclawVoiceClient(args.url).run(audio_file=args.audio_file)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
4
cmd/picoclaw-voice/demo/requirements.txt
Normal file
4
cmd/picoclaw-voice/demo/requirements.txt
Normal file
|
|
@ -0,0 +1,4 @@
|
|||
websocket-client>=1.6.0
|
||||
pyaudio>=0.2.14
|
||||
opuslib>=3.0.1
|
||||
miniaudio>=1.2
|
||||
39
cmd/picoclaw-voice/go.mod
Normal file
39
cmd/picoclaw-voice/go.mod
Normal file
|
|
@ -0,0 +1,39 @@
|
|||
module github.com/sipeed/picoclaw/picoclaw-voice
|
||||
|
||||
go 1.25.7
|
||||
|
||||
require (
|
||||
github.com/caarlos0/env/v11 v11.3.1
|
||||
github.com/google/uuid v1.6.0
|
||||
github.com/gorilla/websocket v1.5.3
|
||||
github.com/pion/opus v0.0.0-20260219180131-abe26becac00
|
||||
github.com/sipeed/picoclaw v0.0.0
|
||||
)
|
||||
|
||||
require (
|
||||
github.com/adhocore/gronx v1.19.6 // indirect
|
||||
github.com/anthropics/anthropic-sdk-go v1.22.1 // indirect
|
||||
github.com/github/copilot-sdk/go v0.1.23 // indirect
|
||||
github.com/gomarkdown/markdown v0.0.0-20260217112301-37c66b85d6ab // indirect
|
||||
github.com/google/jsonschema-go v0.4.2 // indirect
|
||||
github.com/h2non/filetype v1.1.3 // indirect
|
||||
github.com/mattn/go-colorable v0.1.14 // indirect
|
||||
github.com/mattn/go-isatty v0.0.20 // indirect
|
||||
github.com/modelcontextprotocol/go-sdk v1.3.1 // indirect
|
||||
github.com/openai/openai-go/v3 v3.22.0 // indirect
|
||||
github.com/rs/zerolog v1.34.0 // indirect
|
||||
github.com/segmentio/asm v1.1.3 // indirect
|
||||
github.com/segmentio/encoding v0.5.3 // indirect
|
||||
github.com/tidwall/gjson v1.18.0 // indirect
|
||||
github.com/tidwall/match v1.2.0 // indirect
|
||||
github.com/tidwall/pretty v1.2.1 // indirect
|
||||
github.com/tidwall/sjson v1.2.5 // indirect
|
||||
github.com/yosida95/uritemplate/v3 v3.0.2 // indirect
|
||||
golang.org/x/oauth2 v0.35.0 // indirect
|
||||
golang.org/x/sys v0.41.0 // indirect
|
||||
golang.org/x/time v0.14.0 // indirect
|
||||
gopkg.in/yaml.v3 v3.0.1 // indirect
|
||||
)
|
||||
|
||||
// 指向本地 picoclaw fork 根目录
|
||||
replace github.com/sipeed/picoclaw => ../../
|
||||
79
cmd/picoclaw-voice/go.sum
Normal file
79
cmd/picoclaw-voice/go.sum
Normal file
|
|
@ -0,0 +1,79 @@
|
|||
github.com/adhocore/gronx v1.19.6 h1:5KNVcoR9ACgL9HhEqCm5QXsab/gI4QDIybTAWcXDKDc=
|
||||
github.com/adhocore/gronx v1.19.6/go.mod h1:7oUY1WAU8rEJWmAxXR2DN0JaO4gi9khSgKjiRypqteg=
|
||||
github.com/anthropics/anthropic-sdk-go v1.22.1 h1:xbsc3vJKCX/ELDZSpTNfz9wCgrFsamwFewPb1iI0Xh0=
|
||||
github.com/anthropics/anthropic-sdk-go v1.22.1/go.mod h1:WTz31rIUHUHqai2UslPpw5CwXrQP3geYBioRV4WOLvE=
|
||||
github.com/caarlos0/env/v11 v11.3.1 h1:cArPWC15hWmEt+gWk7YBi7lEXTXCvpaSdCiZE2X5mCA=
|
||||
github.com/caarlos0/env/v11 v11.3.1/go.mod h1:qupehSf/Y0TUTsxKywqRt/vJjN5nz6vauiYEUUr8P4U=
|
||||
github.com/coreos/go-systemd/v22 v22.5.0/go.mod h1:Y58oyj3AT4RCenI/lSvhwexgC+NSVTIJ3seZv2GcEnc=
|
||||
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
|
||||
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||
github.com/github/copilot-sdk/go v0.1.23 h1:uExtO/inZQndCZMiSAA1hvXINiz9tqo/MZgQzFzurxw=
|
||||
github.com/github/copilot-sdk/go v0.1.23/go.mod h1:GdwwBfMbm9AABLEM3x5IZKw4ZfwCYxZ1BgyytmZenQ0=
|
||||
github.com/godbus/dbus/v5 v5.0.4/go.mod h1:xhWf0FNVPg57R7Z0UbKHbJfkEywrmjJnf7w5xrFpKfA=
|
||||
github.com/golang-jwt/jwt/v5 v5.2.2 h1:Rl4B7itRWVtYIHFrSNd7vhTiz9UpLdi6gZhZ3wEeDy8=
|
||||
github.com/golang-jwt/jwt/v5 v5.2.2/go.mod h1:pqrtFR0X4osieyHYxtmOUWsAWrfe1Q5UVIyoH402zdk=
|
||||
github.com/gomarkdown/markdown v0.0.0-20260217112301-37c66b85d6ab h1:VYNivV7P8IRHUam2swVUNkhIdp0LRRFKe4hXNnoZKTc=
|
||||
github.com/gomarkdown/markdown v0.0.0-20260217112301-37c66b85d6ab/go.mod h1:JDGcbDT52eL4fju3sZ4TeHGsQwhG9nbDV21aMyhwPoA=
|
||||
github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
|
||||
github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
|
||||
github.com/google/jsonschema-go v0.4.2 h1:tmrUohrwoLZZS/P3x7ex0WAVknEkBZM46iALbcqoRA8=
|
||||
github.com/google/jsonschema-go v0.4.2/go.mod h1:r5quNTdLOYEz95Ru18zA0ydNbBuYoo9tgaYcxEYhJVE=
|
||||
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
|
||||
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
|
||||
github.com/gorilla/websocket v1.5.3 h1:saDtZ6Pbx/0u+bgYQ3q96pZgCzfhKXGPqt7kZ72aNNg=
|
||||
github.com/gorilla/websocket v1.5.3/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE=
|
||||
github.com/h2non/filetype v1.1.3 h1:FKkx9QbD7HR/zjK1Ia5XiBsq9zdLi5Kf3zGyFTAFkGg=
|
||||
github.com/h2non/filetype v1.1.3/go.mod h1:319b3zT68BvV+WRj7cwy856M2ehB3HqNOt6sy1HndBY=
|
||||
github.com/mattn/go-colorable v0.1.13/go.mod h1:7S9/ev0klgBDR4GtXTXX8a3vIGJpMovkB8vQcUbaXHg=
|
||||
github.com/mattn/go-colorable v0.1.14 h1:9A9LHSqF/7dyVVX6g0U9cwm9pG3kP9gSzcuIPHPsaIE=
|
||||
github.com/mattn/go-colorable v0.1.14/go.mod h1:6LmQG8QLFO4G5z1gPvYEzlUgJ2wF+stgPZH1UqBm1s8=
|
||||
github.com/mattn/go-isatty v0.0.16/go.mod h1:kYGgaQfpe5nmfYZH+SKPsOc2e4SrIfOl2e/yFXSvRLM=
|
||||
github.com/mattn/go-isatty v0.0.19/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y=
|
||||
github.com/mattn/go-isatty v0.0.20 h1:xfD0iDuEKnDkl03q4limB+vH+GxLEtL/jb4xVJSWWEY=
|
||||
github.com/mattn/go-isatty v0.0.20/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y=
|
||||
github.com/modelcontextprotocol/go-sdk v1.3.1 h1:TfqtNKOIWN4Z1oqmPAiWDC2Jq7K9OdJaooe0teoXASI=
|
||||
github.com/modelcontextprotocol/go-sdk v1.3.1/go.mod h1:DgVX498dMD8UJlseK1S5i1T4tFz2fkBk4xogC3D15nw=
|
||||
github.com/openai/openai-go/v3 v3.22.0 h1:6MEoNoV8sbjOVmXdvhmuX3BjVbVdcExbVyGixiyJ8ys=
|
||||
github.com/openai/openai-go/v3 v3.22.0/go.mod h1:cdufnVK14cWcT9qA1rRtrXx4FTRsgbDPW7Ia7SS5cZo=
|
||||
github.com/pion/opus v0.0.0-20260219180131-abe26becac00 h1:+PXsZ5OSLoyzPdzKvr8X/C4WtIWk7GbAAIKZOrFV744=
|
||||
github.com/pion/opus v0.0.0-20260219180131-abe26becac00/go.mod h1:t5Xog2n682JnawoykACE6nKVmupFvmJvkpM7x6bTv6g=
|
||||
github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
|
||||
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
|
||||
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
|
||||
github.com/rs/xid v1.6.0/go.mod h1:7XoLgs4eV+QndskICGsho+ADou8ySMSjJKDIan90Nz0=
|
||||
github.com/rs/zerolog v1.34.0 h1:k43nTLIwcTVQAncfCw4KZ2VY6ukYoZaBPNOE8txlOeY=
|
||||
github.com/rs/zerolog v1.34.0/go.mod h1:bJsvje4Z08ROH4Nhs5iH600c3IkWhwp44iRc54W6wYQ=
|
||||
github.com/segmentio/asm v1.1.3 h1:WM03sfUOENvvKexOLp+pCqgb/WDjsi7EK8gIsICtzhc=
|
||||
github.com/segmentio/asm v1.1.3/go.mod h1:Ld3L4ZXGNcSLRg4JBsZ3//1+f/TjYl0Mzen/DQy1EJg=
|
||||
github.com/segmentio/encoding v0.5.3 h1:OjMgICtcSFuNvQCdwqMCv9Tg7lEOXGwm1J5RPQccx6w=
|
||||
github.com/segmentio/encoding v0.5.3/go.mod h1:HS1ZKa3kSN32ZHVZ7ZLPLXWvOVIiZtyJnO1gPH1sKt0=
|
||||
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
|
||||
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
|
||||
github.com/tidwall/gjson v1.14.2/go.mod h1:/wbyibRr2FHMks5tjHJ5F8dMZh3AcwJEMf5vlfC0lxk=
|
||||
github.com/tidwall/gjson v1.18.0 h1:FIDeeyB800efLX89e5a8Y0BNH+LOngJyGrIWxG2FKQY=
|
||||
github.com/tidwall/gjson v1.18.0/go.mod h1:/wbyibRr2FHMks5tjHJ5F8dMZh3AcwJEMf5vlfC0lxk=
|
||||
github.com/tidwall/match v1.1.1/go.mod h1:eRSPERbgtNPcGhD8UCthc6PmLEQXEWd3PRB5JTxsfmM=
|
||||
github.com/tidwall/match v1.2.0 h1:0pt8FlkOwjN2fPt4bIl4BoNxb98gGHN2ObFEDkrfZnM=
|
||||
github.com/tidwall/match v1.2.0/go.mod h1:eRSPERbgtNPcGhD8UCthc6PmLEQXEWd3PRB5JTxsfmM=
|
||||
github.com/tidwall/pretty v1.2.0/go.mod h1:ITEVvHYasfjBbM0u2Pg8T2nJnzm8xPwvNhhsoaGGjNU=
|
||||
github.com/tidwall/pretty v1.2.1 h1:qjsOFOWWQl+N3RsoF5/ssm1pHmJJwhjlSbZ51I6wMl4=
|
||||
github.com/tidwall/pretty v1.2.1/go.mod h1:ITEVvHYasfjBbM0u2Pg8T2nJnzm8xPwvNhhsoaGGjNU=
|
||||
github.com/tidwall/sjson v1.2.5 h1:kLy8mja+1c9jlljvWTlSazM7cKDRfJuR/bOJhcY5NcY=
|
||||
github.com/tidwall/sjson v1.2.5/go.mod h1:Fvgq9kS/6ociJEDnK0Fk1cpYF4FIW6ZF7LAe+6jwd28=
|
||||
github.com/yosida95/uritemplate/v3 v3.0.2 h1:Ed3Oyj9yrmi9087+NczuL5BwkIc4wvTb5zIM+UJPGz4=
|
||||
github.com/yosida95/uritemplate/v3 v3.0.2/go.mod h1:ILOh0sOhIJR3+L/8afwt/kE++YT040gmv5BQTMR2HP4=
|
||||
golang.org/x/oauth2 v0.35.0 h1:Mv2mzuHuZuY2+bkyWXIHMfhNdJAdwW3FuWeCPYN5GVQ=
|
||||
golang.org/x/oauth2 v0.35.0/go.mod h1:lzm5WQJQwKZ3nwavOZ3IS5Aulzxi68dUSgRHujetwEA=
|
||||
golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
|
||||
golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
|
||||
golang.org/x/sys v0.12.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
|
||||
golang.org/x/sys v0.41.0 h1:Ivj+2Cp/ylzLiEU89QhWblYnOE9zerudt9Ftecq2C6k=
|
||||
golang.org/x/sys v0.41.0/go.mod h1:OgkHotnGiDImocRcuBABYBEXf8A9a87e/uXjp9XT3ks=
|
||||
golang.org/x/time v0.14.0 h1:MRx4UaLrDotUKUdCIqzPC48t1Y9hANFKIRpNx+Te8PI=
|
||||
golang.org/x/time v0.14.0/go.mod h1:eL/Oa2bBBK0TkX57Fyni+NgnyQQN4LitPmob2Hjnqw4=
|
||||
golang.org/x/tools v0.34.0 h1:qIpSLOxeCYGg9TrcJokLBG4KFA6d795g0xkBkiESGlo=
|
||||
golang.org/x/tools v0.34.0/go.mod h1:pAP9OwEaY1CAW3HOmg3hLZC5Z0CCmzjAF2UQMSqNARg=
|
||||
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
|
||||
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
|
||||
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
|
||||
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
||||
744
cmd/picoclaw-voice/handler.go
Normal file
744
cmd/picoclaw-voice/handler.go
Normal file
|
|
@ -0,0 +1,744 @@
|
|||
// PicoClaw - Ultra-lightweight personal AI agent
|
||||
// License: MIT
|
||||
//
|
||||
// Copyright (c) 2026 PicoClaw contributors
|
||||
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"log"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
"unicode/utf8"
|
||||
|
||||
"github.com/google/uuid"
|
||||
"github.com/gorilla/websocket"
|
||||
|
||||
"github.com/sipeed/picoclaw/pkg/agent"
|
||||
"github.com/sipeed/picoclaw/pkg/asr"
|
||||
"github.com/sipeed/picoclaw/pkg/tts"
|
||||
)
|
||||
|
||||
// ---- session ----
|
||||
|
||||
// deviceRegistry 维护 device_id → 活跃 session 的映射。
|
||||
// 同一 device_id 的新连接到来时，旧连接被立即关闭（last-write-wins）。
|
||||
type deviceRegistry struct {
|
||||
mu sync.Mutex
|
||||
active map[string]*session
|
||||
}
|
||||
|
||||
func newDeviceRegistry() *deviceRegistry {
|
||||
return &deviceRegistry{active: make(map[string]*session)}
|
||||
}
|
||||
|
||||
// register 注册新 session，并关闭同设备的旧连接。
|
||||
func (r *deviceRegistry) register(deviceID string, s *session) {
|
||||
r.mu.Lock()
|
||||
old, exists := r.active[deviceID]
|
||||
r.active[deviceID] = s
|
||||
r.mu.Unlock()
|
||||
if exists && old != s {
|
||||
log.Printf("picoclaw-voice: device %s reconnected, evicting old session", deviceID)
|
||||
old.connCancel()
|
||||
old.conn.Close()
|
||||
}
|
||||
}
|
||||
|
||||
// unregister 在 session 退出时清理（仅当仍是活跃 session 时）。
|
||||
func (r *deviceRegistry) unregister(deviceID string, s *session) {
|
||||
r.mu.Lock()
|
||||
if r.active[deviceID] == s {
|
||||
delete(r.active, deviceID)
|
||||
}
|
||||
r.mu.Unlock()
|
||||
}
|
||||
|
||||
// asrChunk 是实时 ASR 音频帧，isLast=true 表示结束帧。
|
||||
// frame 内容格式与协商的 audioFmt 一致（常见为 PCM 字节）。
|
||||
type asrChunk struct {
|
||||
frame []byte
|
||||
isLast bool
|
||||
}
|
||||
|
||||
// audioItem 是 TTS 正在合成的一句话，frameCh 流式传递音频帧。
|
||||
// 阶段 B 合成时边回调边写入 frameCh，close(frameCh) 表示本句合成完毕。
|
||||
type audioItem struct {
|
||||
text string
|
||||
frameCh chan []byte // 流式音频帧，close 表示本句结束
|
||||
}
|
||||
|
||||
type session struct {
|
||||
conn *websocket.Conn
|
||||
writeMu sync.Mutex
|
||||
asr asr.Provider
|
||||
tts tts.Provider
|
||||
agentLoop *agent.AgentLoop
|
||||
registry *deviceRegistry
|
||||
// connCancel 由 deviceRegistry.register 在设备重连时调用，强制关闭本连接。
|
||||
connCancel context.CancelFunc
|
||||
// llmSessOverride：通过 PICOCLAW_VOICE_OWNER_ID 设置的 owner_id（如 "picoclaw"），优先级最高。
|
||||
// 同时作为 connID 基础和强制 memoryID，确保所有渠道共享同一记忆上下文（跨渠道记忆统一）。
|
||||
llmSessOverride string
|
||||
// deviceID：从 hello 消息提取的设备标识（MAC 或 device_id）
|
||||
deviceID string
|
||||
// connID：WebSocket 连接级 ID，服务端在 hello 时生成并回传客户端，客户端不关心。
|
||||
// 用于标识一条长连接，设备重连时更新。
|
||||
connID string
|
||||
// turnID：问题级 ID，由客户端在 VAD start（listen.start）时生成并下发。
|
||||
// 贯穿整个问题生命周期：VAD start → ASR → LLM → TTS stop。
|
||||
turnID string
|
||||
// memoryID：LLM 多轮记忆 key，由客户端在 listen.start 时指定。
|
||||
// 同一 memoryID 的问题共享对话历史；客户端可按场景/用户自由切换。
|
||||
// 未携带时退化为 deviceID，deviceID 也无时退化为 connID。
|
||||
memoryID string
|
||||
|
||||
// audioFmt 是在 hello 阶段由 ASR provider 声明并写入 helloReply 的格式。
|
||||
// handleAudio 按此格式进行轻量校验，provider 直接按此格式使用。
|
||||
audioFmt asr.AudioFormat
|
||||
audioBuf [][]byte
|
||||
|
||||
// asrSess 和 asrFeedCh 仅在使用实时 ASR（RealtimeProvider）时非 nil。
|
||||
// listen start 时建连，handleAudio 推送帧，listen end 时发结束帧。
|
||||
asrSess asr.StreamingSession
|
||||
asrFeedCh chan asrChunk
|
||||
|
||||
// 当前语音交互的总取消函数
|
||||
cancelMu sync.Mutex
|
||||
cancel context.CancelFunc
|
||||
}
|
||||
|
||||
func newSession(conn *websocket.Conn, asrProv asr.Provider, ttsProv tts.Provider, al *agent.AgentLoop, llmSessOverride string, reg *deviceRegistry) *session {
|
||||
_, cancel := context.WithCancel(context.Background())
|
||||
return &session{
|
||||
conn: conn,
|
||||
asr: asrProv,
|
||||
tts: ttsProv,
|
||||
agentLoop: al,
|
||||
registry: reg,
|
||||
connCancel: cancel,
|
||||
llmSessOverride: llmSessOverride,
|
||||
}
|
||||
}
|
||||
|
||||
func (s *session) run() {
|
||||
defer s.connCancel()
|
||||
defer s.conn.Close()
|
||||
defer func() {
|
||||
if s.deviceID != "" {
|
||||
s.registry.unregister(s.deviceID, s)
|
||||
}
|
||||
}()
|
||||
for {
|
||||
msgType, data, err := s.conn.ReadMessage()
|
||||
if err != nil {
|
||||
log.Printf("picoclaw-voice: session %s (%s) read error: %v", s.connID, s.deviceID, err)
|
||||
return
|
||||
}
|
||||
switch msgType {
|
||||
case websocket.TextMessage:
|
||||
s.handleText(data)
|
||||
case websocket.BinaryMessage:
|
||||
s.handleAudio(data)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (s *session) handleText(data []byte) {
|
||||
var base struct {
|
||||
Type string `json:"type"`
|
||||
}
|
||||
if err := json.Unmarshal(data, &base); err != nil {
|
||||
return
|
||||
}
|
||||
|
||||
switch base.Type {
|
||||
case "hello":
|
||||
var msg helloMsg
|
||||
if err := json.Unmarshal(data, &msg); err == nil {
|
||||
s.deviceID = msg.DeviceID
|
||||
}
|
||||
// hello：服务端生成连接级 connID 并回传客户端（客户端不关心，仅用于日志关联）
|
||||
if s.connID == "" {
|
||||
if s.llmSessOverride != "" {
|
||||
s.connID = s.llmSessOverride
|
||||
} else {
|
||||
s.connID = uuid.New().String()
|
||||
}
|
||||
log.Printf("picoclaw-voice: conn_id=%s device=%s", s.connID, s.deviceID)
|
||||
}
|
||||
// 注册设备：驱逐同设备的旧连接（重连场景）
|
||||
if s.deviceID != "" {
|
||||
s.registry.register(s.deviceID, s)
|
||||
}
|
||||
// 从 ASR/TTS provider 获取各自格式，写入 helloReply 协商给客户端
|
||||
s.audioFmt = s.asr.AudioFormat()
|
||||
s.writeText(helloReply(s.connID, s.audioFmt, s.tts.AudioFormat()))
|
||||
|
||||
case "listen":
|
||||
var msg listenMsg
|
||||
if err := json.Unmarshal(data, &msg); err != nil {
|
||||
return
|
||||
}
|
||||
switch msg.State {
|
||||
case "start":
|
||||
s.audioBuf = s.audioBuf[:0] // 重置帧缓冲（保留 slice 底层内存）
|
||||
// turn_id 由客户端在 listen start 时携带（session_id 字段）；
|
||||
// 未携带时服务端生成 UUID，确保每轮可独立追踪。
|
||||
if msg.SessionID != "" {
|
||||
s.turnID = msg.SessionID
|
||||
} else {
|
||||
s.turnID = uuid.New().String()
|
||||
}
|
||||
log.Printf("picoclaw-voice: turn_id=%s", s.turnID)
|
||||
// memoryID 决定本轮 LLM 使用哪份多轮对话历史：
|
||||
// PICOCLAW_VOICE_OWNER_ID（全局覆盖）> listen.memory_id [picoclaw 扩展] > device_id > connection_id
|
||||
if s.llmSessOverride != "" {
|
||||
s.memoryID = s.llmSessOverride
|
||||
} else if msg.MemoryID != "" {
|
||||
s.memoryID = msg.MemoryID
|
||||
} else if s.deviceID != "" {
|
||||
s.memoryID = s.deviceID
|
||||
} else {
|
||||
s.memoryID = s.connID
|
||||
}
|
||||
// 取消旧流水线，关闭旧实时 ASR 会话
|
||||
s.cancelMu.Lock()
|
||||
if s.cancel != nil {
|
||||
s.cancel()
|
||||
s.cancel = nil
|
||||
}
|
||||
s.cancelMu.Unlock()
|
||||
if s.asrSess != nil {
|
||||
s.asrSess.Close()
|
||||
s.asrSess = nil
|
||||
s.asrFeedCh = nil
|
||||
}
|
||||
// 实时 ASR 模式：listen start 时就建连，音频帧实时推送
|
||||
if rp, ok := s.asr.(asr.RealtimeProvider); ok {
|
||||
pipeCtx, pipeCancel := context.WithCancel(context.Background())
|
||||
s.cancelMu.Lock()
|
||||
s.cancel = pipeCancel
|
||||
s.cancelMu.Unlock()
|
||||
asrCtx, asrCancel := context.WithTimeout(pipeCtx, 30*time.Second)
|
||||
sess, err := rp.OpenSession(asrCtx, func(text string, final bool) {
|
||||
state := "recognizing"
|
||||
if final {
|
||||
state = "stop"
|
||||
}
|
||||
s.writeText(newStt(text, state))
|
||||
})
|
||||
if err != nil {
|
||||
asrCancel()
|
||||
pipeCancel()
|
||||
s.cancelMu.Lock()
|
||||
s.cancel = nil
|
||||
s.cancelMu.Unlock()
|
||||
log.Printf("picoclaw-voice: open asr session: %v", err)
|
||||
} else {
|
||||
feedCh := make(chan asrChunk, 64)
|
||||
s.asrSess = sess
|
||||
s.asrFeedCh = feedCh
|
||||
turnID, memoryID := s.turnID, s.memoryID
|
||||
go func() {
|
||||
defer asrCancel()
|
||||
s.processStreamSpeech(pipeCtx, sess, feedCh, time.Now(), turnID, memoryID)
|
||||
// 流水线结束，清理引用。handleAudio 看到 nil 后切换到批量缓冲模式。
|
||||
// 轻微竞争可接受：最多丢个别帧，下一轮 listen.start 会重新赋值。
|
||||
s.asrSess = nil
|
||||
s.asrFeedCh = nil
|
||||
}()
|
||||
}
|
||||
}
|
||||
case "end", "stop":
|
||||
if s.asrSess != nil {
|
||||
// 实时模式：发送结束帧，processStreamSpeech 等待最终 ASR 结果
|
||||
select {
|
||||
case s.asrFeedCh <- asrChunk{isLast: true}:
|
||||
default:
|
||||
log.Printf("picoclaw-voice: asr feed channel full on finalize")
|
||||
}
|
||||
s.asrSess = nil
|
||||
s.asrFeedCh = nil
|
||||
} else {
|
||||
// 批量模式：VAD 结束后一次性 ASR
|
||||
s.cancelMu.Lock()
|
||||
if s.cancel != nil {
|
||||
s.cancel()
|
||||
s.cancel = nil
|
||||
}
|
||||
s.cancelMu.Unlock()
|
||||
buf := make([][]byte, len(s.audioBuf))
|
||||
copy(buf, s.audioBuf)
|
||||
go s.processSpeech(buf)
|
||||
}
|
||||
}
|
||||
|
||||
case "abort":
|
||||
// 关闭实时 ASR 会话（如果有）
|
||||
if s.asrSess != nil {
|
||||
s.asrSess.Close()
|
||||
s.asrSess = nil
|
||||
s.asrFeedCh = nil
|
||||
}
|
||||
s.cancelMu.Lock()
|
||||
if s.cancel != nil {
|
||||
s.cancel()
|
||||
s.cancel = nil
|
||||
}
|
||||
s.cancelMu.Unlock()
|
||||
// 通知客户端立即清空播放缓冲（触发 miniaudio stop+restart）
|
||||
s.writeText(newTts("abort", ""))
|
||||
}
|
||||
}
|
||||
|
||||
// validateAudioFrame 轻量校验帧格式是否与协商结果一致。
|
||||
// 安全防御：客户端应遵守 helloReply 下发的格式，不符则记 warn 并丢弃。
|
||||
func (s *session) validateAudioFrame(data []byte) bool {
|
||||
if len(data) == 0 {
|
||||
log.Printf("picoclaw-voice: empty audio frame, dropping")
|
||||
return false
|
||||
}
|
||||
switch s.audioFmt.Codec {
|
||||
case "pcm":
|
||||
if len(data)%2 != 0 {
|
||||
log.Printf("picoclaw-voice: pcm frame size=%d not 16-bit aligned, dropping", len(data))
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
func (s *session) handleAudio(data []byte) {
|
||||
// 轻量校验帧格式（格式已在 hello 阶段协商，防御客户端实现缺陷）
|
||||
if !s.validateAudioFrame(data) {
|
||||
return
|
||||
}
|
||||
frame := make([]byte, len(data))
|
||||
copy(frame, data)
|
||||
if s.asrFeedCh != nil {
|
||||
// 实时模式：直接推送给 ASR，不缓冲
|
||||
select {
|
||||
case s.asrFeedCh <- asrChunk{frame: frame}:
|
||||
default:
|
||||
log.Printf("picoclaw-voice: asr feed channel full, dropping frame")
|
||||
}
|
||||
} else {
|
||||
// 批量模式：累积到 audioBuf，等 VAD end 后一次性 ASR
|
||||
s.audioBuf = append(s.audioBuf, frame)
|
||||
}
|
||||
}
|
||||
|
||||
// processSpeech: ASR → LLM → TTS 三阶段并发流水线。
|
||||
//
|
||||
// 阶段 A (goroutine): LLM 流式推理，按断句符切句，写入 sentCh。
|
||||
// 阶段 B (goroutine): 从 sentCh 读句子，调用 TTS 合成，写入 readyCh。
|
||||
//
|
||||
// B 与 A 并发：LLM 生下一句时，TTS 已在合成当前句。
|
||||
//
|
||||
// 阶段 C (main): 从 readyCh 按序读音频帧，推给客户端。
|
||||
//
|
||||
// C 发音频时，B 已在合成下一句，消除句间空档。
|
||||
func (s *session) processSpeech(frames [][]byte) {
|
||||
reqStart := time.Now()
|
||||
log.Printf("picoclaw-voice: processSpeech: frames=%d", len(frames))
|
||||
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
s.cancelMu.Lock()
|
||||
s.cancel = cancel
|
||||
s.cancelMu.Unlock()
|
||||
defer cancel()
|
||||
|
||||
// 1. ASR（独立 20s 超时，不影响后续 LLM/TTS 管道）
|
||||
asrCtx, asrCancel := context.WithTimeout(ctx, 20*time.Second)
|
||||
defer asrCancel()
|
||||
|
||||
var finalText string
|
||||
var asrErr error
|
||||
if sp, ok := s.asr.(asr.StreamingProvider); ok {
|
||||
// 流式 ASR：每次文本变化都发给客户端（recognizing），最终结果发 stop
|
||||
asrErr = sp.TranscribeStream(asrCtx, frames, func(text string, final bool) {
|
||||
state := "recognizing"
|
||||
if final {
|
||||
state = "stop"
|
||||
finalText = text
|
||||
}
|
||||
s.writeText(newStt(text, state))
|
||||
})
|
||||
} else {
|
||||
finalText, asrErr = s.asr.Transcribe(asrCtx, frames)
|
||||
if asrErr == nil && strings.TrimSpace(finalText) != "" {
|
||||
s.writeText(newStt(finalText, "stop"))
|
||||
}
|
||||
}
|
||||
|
||||
if asrErr != nil {
|
||||
if ctx.Err() == nil {
|
||||
log.Printf("picoclaw-voice: asr: %v", asrErr)
|
||||
}
|
||||
// 通知客户端复位 FSM（PROCESSING → IDLE）
|
||||
s.writeText(newTts("stop", ""))
|
||||
return
|
||||
}
|
||||
if strings.TrimSpace(finalText) == "" {
|
||||
// 静默或未识别：通知客户端复位 FSM（PROCESSING → IDLE）
|
||||
s.writeText(newTts("stop", ""))
|
||||
return
|
||||
}
|
||||
log.Printf("picoclaw-voice: asr=%q latency=%dms", finalText, time.Since(reqStart).Milliseconds())
|
||||
|
||||
turnID := s.turnID
|
||||
if turnID == "" {
|
||||
turnID = s.connID
|
||||
}
|
||||
memoryID := s.memoryID
|
||||
if memoryID == "" {
|
||||
memoryID = s.connID
|
||||
}
|
||||
s.runPipeline(ctx, finalText, reqStart, turnID, memoryID)
|
||||
}
|
||||
|
||||
// runPipeline 执行 LLM → TTS 三阶段并发流水线。
|
||||
// 由 processSpeech（批量 ASR）和 processStreamSpeech（实时 ASR）共同调用。
|
||||
func (s *session) runPipeline(ctx context.Context, text string, reqStart time.Time, turnID, memoryID string) {
|
||||
sentCh := make(chan string, 8) // 阶段 A → B
|
||||
readyCh := make(chan audioItem, 4) // 阶段 B → C（最多 4 句并发预合成）
|
||||
|
||||
// 阶段 A: LLM 流式 → sentCh
|
||||
go func() {
|
||||
defer close(sentCh)
|
||||
if err := s.streamLLM(ctx, turnID, memoryID, text, reqStart, func(sentence string) {
|
||||
select {
|
||||
case sentCh <- sentence:
|
||||
case <-ctx.Done():
|
||||
}
|
||||
}); err != nil {
|
||||
// ctx.Err() != nil 表示主动 abort/重请求；err == Canceled 是 watchdog 取消（已自行记录）
|
||||
if ctx.Err() == nil && err != context.Canceled && err != context.DeadlineExceeded {
|
||||
log.Printf("picoclaw-voice: llm: %v", err)
|
||||
}
|
||||
}
|
||||
}()
|
||||
|
||||
// 阶段 B: sentCh → TTS 并发合成 → readyCh（流式）
|
||||
// 每句起一个独立 goroutine 合成，Stage B 不阻塞等待当前句合成完成，
|
||||
// 立刻读取下一句并开始其合成——多句 HTTP 请求并发进行。
|
||||
// Stage C 仍按 readyCh 顺序消费，保证播放有序。
|
||||
// readyCh 缓冲控制最大并发预合成句数（buffer=4 表示最多超前合成 4 句）。
|
||||
go func() {
|
||||
defer close(readyCh)
|
||||
for {
|
||||
select {
|
||||
case sentence, ok := <-sentCh:
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
frameCh := make(chan []byte, 32)
|
||||
select {
|
||||
case readyCh <- audioItem{text: sentence, frameCh: frameCh}:
|
||||
case <-ctx.Done():
|
||||
close(frameCh)
|
||||
return
|
||||
}
|
||||
// 异步合成：Stage B 立即处理下一句，勿阻塞
|
||||
go func(sentence string, frameCh chan []byte) {
|
||||
defer close(frameCh)
|
||||
synthErr := s.tts.SynthesizeFrames(ctx, sentence, "", func(frame []byte) {
|
||||
select {
|
||||
case frameCh <- frame:
|
||||
case <-ctx.Done():
|
||||
}
|
||||
})
|
||||
if synthErr != nil && ctx.Err() == nil {
|
||||
log.Printf("picoclaw-voice: tts sentence %q: %v", sentence, synthErr)
|
||||
}
|
||||
}(sentence, frameCh)
|
||||
case <-ctx.Done():
|
||||
return
|
||||
}
|
||||
}
|
||||
}()
|
||||
|
||||
// 阶段 C: readyCh → 音频帧 → WebSocket（保序）
|
||||
// 每句收到 audioItem 后立即发 sentence_start，然后从 frameCh 流式转发帧。
|
||||
firstItem := true
|
||||
for item := range readyCh {
|
||||
if firstItem {
|
||||
// tts.start 触发客户端 FSM: PROCESSING → SPEAKING
|
||||
s.writeText(newTts("start", ""))
|
||||
firstItem = false
|
||||
}
|
||||
s.writeText(newTts("sentence_start", item.text))
|
||||
for frame := range item.frameCh {
|
||||
if err := s.writeBinary(frame); err != nil {
|
||||
log.Printf("picoclaw-voice: send audio frame: %v", err)
|
||||
}
|
||||
}
|
||||
s.writeText(newTts("sentence_end", ""))
|
||||
}
|
||||
|
||||
s.writeText(newTts("stop", ""))
|
||||
}
|
||||
|
||||
// processStreamSpeech 处理实时 ASR 流：并发发送音频帧 + 等待最终识别结果，然后运行 LLM/TTS 流水线。
|
||||
func (s *session) processStreamSpeech(ctx context.Context, sess asr.StreamingSession, feedCh <-chan asrChunk, reqStart time.Time, turnID, memoryID string) {
|
||||
defer sess.Close()
|
||||
|
||||
// 并发发送音频帧给 ASR 服务
|
||||
sendDone := make(chan struct{})
|
||||
go func() {
|
||||
defer close(sendDone)
|
||||
for {
|
||||
select {
|
||||
case chunk, ok := <-feedCh:
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
if err := sess.SendAudio(chunk.frame, chunk.isLast); err != nil {
|
||||
if ctx.Err() == nil {
|
||||
log.Printf("picoclaw-voice: asr send: %v", err)
|
||||
}
|
||||
return
|
||||
}
|
||||
if chunk.isLast {
|
||||
return
|
||||
}
|
||||
case <-ctx.Done():
|
||||
return
|
||||
}
|
||||
}
|
||||
}()
|
||||
|
||||
// 等待 ASR 最终识别结果
|
||||
text, err := sess.Wait(ctx)
|
||||
<-sendDone
|
||||
|
||||
if err != nil {
|
||||
if ctx.Err() == nil && !errors.Is(err, asr.ErrSessionClosed) {
|
||||
log.Printf("picoclaw-voice: asr: %v", err)
|
||||
s.writeText(newTts("stop", ""))
|
||||
}
|
||||
return
|
||||
}
|
||||
if strings.TrimSpace(text) == "" {
|
||||
s.writeText(newTts("stop", ""))
|
||||
return
|
||||
}
|
||||
log.Printf("picoclaw-voice: asr=%q latency=%dms", text, time.Since(reqStart).Milliseconds())
|
||||
|
||||
s.runPipeline(ctx, text, reqStart, turnID, memoryID)
|
||||
}
|
||||
|
||||
// streamLLM 直接调用 AgentLoop 流式推理，按句触发 onSentence 回调。
|
||||
// 同时支持 thinking 模式（<think>...</think>）和普通模式：
|
||||
// - thinking 块内容不送 TTS，仅记录耗时日志
|
||||
// - thinking 块之外的内容按断句符触发 TTS
|
||||
//
|
||||
// reqStart 用于记录 LLM 首 token 延迟（含 ASR 时间，为端到端指标）。
|
||||
func (s *session) streamLLM(ctx context.Context, turnID, memoryID, userText string, reqStart time.Time, onSentence func(string)) error {
|
||||
llmStart := time.Now()
|
||||
log.Printf("picoclaw-voice: llm request: turn=%s memory=%s text=%q", turnID, memoryID, userText)
|
||||
const startTag = "<think>"
|
||||
const endTag = "</think>"
|
||||
|
||||
// 首 token 超时 30s；之后每个 token 间隔超时 8s（每收到 token 自动重置）
|
||||
const firstTokenTimeout = 30 * time.Second
|
||||
const interTokenTimeout = 8 * time.Second
|
||||
llmCtx, llmCancel := context.WithCancel(ctx)
|
||||
defer llmCancel()
|
||||
tokenActivity := make(chan struct{}, 1)
|
||||
go func() {
|
||||
firstToken := true
|
||||
t := time.NewTimer(firstTokenTimeout)
|
||||
defer t.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-llmCtx.Done():
|
||||
return
|
||||
case <-tokenActivity:
|
||||
if firstToken {
|
||||
firstToken = false
|
||||
}
|
||||
if !t.Stop() {
|
||||
select {
|
||||
case <-t.C:
|
||||
default:
|
||||
}
|
||||
}
|
||||
t.Reset(interTokenTimeout)
|
||||
case <-t.C:
|
||||
if firstToken {
|
||||
log.Printf("picoclaw-voice: llm timeout: no first token after %s", firstTokenTimeout)
|
||||
} else {
|
||||
log.Printf("picoclaw-voice: llm timeout: no token for %s", interTokenTimeout)
|
||||
}
|
||||
llmCancel()
|
||||
return
|
||||
}
|
||||
}
|
||||
}()
|
||||
|
||||
var normalBuf strings.Builder
|
||||
// thinkTail 保存 thinking 块末尾最多 len(endTag)-1 个字节，
|
||||
// 用于跨 token 边界检测 </think> 而无需缓存完整 thinking 内容。
|
||||
thinkTail := make([]byte, 0, len(endTag)-1)
|
||||
inThinking := false
|
||||
var thinkStart time.Time
|
||||
firstContent := reqStart // 非零时尚未记录首个实际内容 token 延迟
|
||||
|
||||
flushNormal := func() {
|
||||
text := normalBuf.String()
|
||||
if idx := lastSentenceBreak(text); idx > 0 {
|
||||
sentence := text[:idx]
|
||||
remaining := text[idx:]
|
||||
// 先更新缓冲，再发送（onSentence 可能阻塞）
|
||||
normalBuf.Reset()
|
||||
normalBuf.WriteString(remaining)
|
||||
if trimmed := strings.TrimSpace(sentence); trimmed != "" {
|
||||
log.Printf("picoclaw-voice: llm sentence: %q", trimmed)
|
||||
// [picoclaw 扩展] llm 消息将断句后的 LLM 输出同步推送到客户端（标准 xiaozhi 无此消息类型）
|
||||
s.writeText(newLlmText(trimmed))
|
||||
onSentence(trimmed)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
onToken := func(token string) {
|
||||
// token 到达，重置 watchdog 定时器
|
||||
select {
|
||||
case tokenActivity <- struct{}{}:
|
||||
default:
|
||||
}
|
||||
// token 可能跨多个 tag 边界，用循环处理完
|
||||
for token != "" {
|
||||
if inThinking {
|
||||
// 用滑动窗口检测 </think>，不存储完整 thinking 内容
|
||||
scan := string(thinkTail) + token
|
||||
if idx := strings.Index(scan, endTag); idx >= 0 {
|
||||
durationMs := time.Since(thinkStart).Milliseconds()
|
||||
log.Printf("picoclaw-voice: thinking=%dms", durationMs)
|
||||
s.writeText(newLlmThinkingEnd(durationMs))
|
||||
after := scan[idx+len(endTag):]
|
||||
thinkTail = thinkTail[:0]
|
||||
inThinking = false
|
||||
token = after // 继续处理 </think> 之后的内容
|
||||
} else {
|
||||
// 更新尾部窗口（保留最多 len(endTag)-1 字节）
|
||||
if len(scan) >= len(endTag)-1 {
|
||||
thinkTail = []byte(scan[len(scan)-(len(endTag)-1):])
|
||||
} else {
|
||||
thinkTail = []byte(scan)
|
||||
}
|
||||
token = ""
|
||||
}
|
||||
} else {
|
||||
normalBuf.WriteString(token)
|
||||
token = ""
|
||||
combined := normalBuf.String()
|
||||
if idx := strings.Index(combined, startTag); idx >= 0 {
|
||||
before := combined[:idx]
|
||||
after := combined[idx+len(startTag):]
|
||||
normalBuf.Reset()
|
||||
normalBuf.WriteString(before)
|
||||
// before 非空时记录首内容延迟（thinking 前已有实际输出）
|
||||
if !firstContent.IsZero() && before != "" {
|
||||
log.Printf("picoclaw-voice: llm_first_token latency=%dms", time.Since(firstContent).Milliseconds())
|
||||
firstContent = time.Time{}
|
||||
}
|
||||
flushNormal()
|
||||
inThinking = true
|
||||
thinkStart = time.Now()
|
||||
thinkTail = thinkTail[:0]
|
||||
log.Printf("picoclaw-voice: llm thinking...")
|
||||
s.writeText(newLlmThinkingStart())
|
||||
token = after
|
||||
} else {
|
||||
// 普通内容：记录首 token 延迟并断句
|
||||
if !firstContent.IsZero() && strings.TrimSpace(normalBuf.String()) != "" {
|
||||
log.Printf("picoclaw-voice: llm_first_token latency=%dms", time.Since(firstContent).Milliseconds())
|
||||
firstContent = time.Time{}
|
||||
}
|
||||
flushNormal()
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if err := s.agentLoop.RunStreamAgentLoop(llmCtx, memoryID, userText, "voice", turnID, onToken); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// 处理最后剩余片段（thinking 块未正常关闭时跳过，避免把 thinking 内容送 TTS）
|
||||
if !inThinking && normalBuf.Len() > 0 {
|
||||
if trimmed := strings.TrimSpace(normalBuf.String()); trimmed != "" {
|
||||
log.Printf("picoclaw-voice: llm sentence: %q", trimmed)
|
||||
s.writeText(newLlmText(trimmed))
|
||||
onSentence(trimmed)
|
||||
}
|
||||
}
|
||||
|
||||
log.Printf("picoclaw-voice: llm done: turn=%s latency=%dms", turnID, time.Since(llmStart).Milliseconds())
|
||||
return nil
|
||||
}
|
||||
|
||||
// lastSentenceBreak 返回字符串中最后一个句子边界之后的字节偏移，
|
||||
// 即 sentenceTerminators rune 之后第一个字节的位置。
|
||||
// 找不到时返回 0。
|
||||
//
|
||||
// 规则：
|
||||
// - 中文结束符（。！？）和换行符（\n）直接算句末。
|
||||
// - 英文 !? 后跟空格时才切（末尾不切，防止流式缓冲末尾误判）。
|
||||
// - 英文 . 仅当同时满足以下条件才切：
|
||||
// 1. 前一字符不是数字（排除版本号/小数点 1.25.7）
|
||||
// 2. 后一字符不是数字
|
||||
// 3. 后跟 "空格 + 大写字母"（典型新句子开头），而非末尾等待
|
||||
func lastSentenceBreak(s string) int {
|
||||
idx := -1
|
||||
bytes := []byte(s)
|
||||
bytePos := 0
|
||||
for _, r := range s {
|
||||
rLen := utf8.RuneLen(r)
|
||||
if strings.ContainsRune("。！？\n", r) {
|
||||
// 中文结束符 / 换行：直接切
|
||||
idx = bytePos + rLen
|
||||
} else if strings.ContainsRune("!?", r) {
|
||||
// 英文 !?：后跟空格时才切（末尾不切）
|
||||
nextPos := bytePos + rLen
|
||||
if nextPos < len(bytes) && (bytes[nextPos] == ' ' || bytes[nextPos] == '\t') {
|
||||
idx = nextPos
|
||||
}
|
||||
} else if r == '.' {
|
||||
// 英文句号：区分句末与版本号/小数点
|
||||
// 切分条件：前一字符非数字 AND 后跟空格（不是数字开头的版本号）
|
||||
nextPos := bytePos + rLen
|
||||
prevIsDigit := bytePos > 0 && bytes[bytePos-1] >= '0' && bytes[bytePos-1] <= '9'
|
||||
if !prevIsDigit && nextPos < len(bytes) && !(bytes[nextPos] >= '0' && bytes[nextPos] <= '9') {
|
||||
if bytes[nextPos] == ' ' || bytes[nextPos] == '\t' {
|
||||
idx = nextPos
|
||||
}
|
||||
}
|
||||
// 末尾不切：等待下一个 token 确认是版本号还是真正句末
|
||||
}
|
||||
bytePos += rLen
|
||||
}
|
||||
if idx < 0 {
|
||||
return 0
|
||||
}
|
||||
return idx
|
||||
}
|
||||
|
||||
func (s *session) writeText(data []byte) error {
|
||||
s.writeMu.Lock()
|
||||
defer s.writeMu.Unlock()
|
||||
return s.conn.WriteMessage(websocket.TextMessage, data)
|
||||
}
|
||||
|
||||
func (s *session) writeBinary(data []byte) error {
|
||||
s.writeMu.Lock()
|
||||
defer s.writeMu.Unlock()
|
||||
return s.conn.WriteMessage(websocket.BinaryMessage, data)
|
||||
}
|
||||
234
cmd/picoclaw-voice/handler_test.go
Normal file
234
cmd/picoclaw-voice/handler_test.go
Normal file
|
|
@ -0,0 +1,234 @@
|
|||
// PicoClaw - Ultra-lightweight personal AI agent
|
||||
// License: MIT
|
||||
//
|
||||
// Copyright (c) 2026 PicoClaw contributors
|
||||
|
||||
package main
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestLastSentenceBreak_Empty(t *testing.T) {
|
||||
if got := lastSentenceBreak(""); got != 0 {
|
||||
t.Errorf("got %d, want 0", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLastSentenceBreak_NoTerminator(t *testing.T) {
|
||||
if got := lastSentenceBreak("你好世界 hello world"); got != 0 {
|
||||
t.Errorf("got %d, want 0", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLastSentenceBreak_ChinesePeriod(t *testing.T) {
|
||||
s := "你好。"
|
||||
idx := lastSentenceBreak(s)
|
||||
if idx != len(s) {
|
||||
t.Errorf("got %d, want %d (end of string)", idx, len(s))
|
||||
}
|
||||
}
|
||||
|
||||
func TestLastSentenceBreak_MultipleTerminators(t *testing.T) {
|
||||
// 应返回最后一个句子边界之后
|
||||
s := "你好。再见！"
|
||||
idx := lastSentenceBreak(s)
|
||||
if idx != len(s) {
|
||||
t.Errorf("got %d, want %d", idx, len(s))
|
||||
}
|
||||
}
|
||||
|
||||
func TestLastSentenceBreak_TrailingText(t *testing.T) {
|
||||
// 断句后有剩余内容
|
||||
s := "你好。还有更多内容"
|
||||
// "你好。" = 9 bytes（每个汉字3字节，句号3字节）
|
||||
wantIdx := len("你好。")
|
||||
idx := lastSentenceBreak(s)
|
||||
if idx != wantIdx {
|
||||
t.Errorf("got %d, want %d", idx, wantIdx)
|
||||
}
|
||||
remaining := s[idx:]
|
||||
if remaining != "还有更多内容" {
|
||||
t.Errorf("remaining = %q, want '还有更多内容'", remaining)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLastSentenceBreak_EnglishPeriod(t *testing.T) {
|
||||
s := "Hello. World"
|
||||
idx := lastSentenceBreak(s)
|
||||
if idx != len("Hello.") {
|
||||
t.Errorf("got %d, want %d", idx, len("Hello."))
|
||||
}
|
||||
}
|
||||
|
||||
func TestLastSentenceBreak_Newline(t *testing.T) {
|
||||
s := "line one\nline two"
|
||||
idx := lastSentenceBreak(s)
|
||||
if idx != len("line one\n") {
|
||||
t.Errorf("got %d, want %d", idx, len("line one\n"))
|
||||
}
|
||||
}
|
||||
|
||||
func TestLastSentenceBreak_MixedScript(t *testing.T) {
|
||||
s := "你好！Hello.再见"
|
||||
// '！' 是全角感叹号，直接切；英文 '.' 后跟汉字（非空格），不满足切分条件。
|
||||
// 所以最后一个断点是 '！' 之后，即 "你好！" 的字节长度。
|
||||
want := len("你好！")
|
||||
if got := lastSentenceBreak(s); got != want {
|
||||
t.Errorf("got %d, want %d", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLastSentenceBreak_OnlyTerminator(t *testing.T) {
|
||||
s := "。"
|
||||
if got := lastSentenceBreak(s); got != len(s) {
|
||||
t.Errorf("got %d, want %d", got, len(s))
|
||||
}
|
||||
}
|
||||
|
||||
// ---- thinking 状态机测试 ----
|
||||
// 通过 simulateTokens 模拟 onToken 回调，验证 thinking 块过滤和断句行为。
|
||||
|
||||
func simulateTokens(tokens []string) (sentences []string, thinkingLogged bool) {
|
||||
const startTag = "<think>"
|
||||
const endTag = "</think>"
|
||||
|
||||
var normalBuf strings.Builder
|
||||
thinkTail := make([]byte, 0, len(endTag)-1)
|
||||
inThinking := false
|
||||
gotThinking := false
|
||||
|
||||
flushNormal := func() {
|
||||
text := normalBuf.String()
|
||||
if idx := lastSentenceBreak(text); idx > 0 {
|
||||
sentences = append(sentences, text[:idx])
|
||||
remaining := text[idx:]
|
||||
normalBuf.Reset()
|
||||
normalBuf.WriteString(remaining)
|
||||
}
|
||||
}
|
||||
|
||||
onToken := func(token string) {
|
||||
for token != "" {
|
||||
if inThinking {
|
||||
scan := string(thinkTail) + token
|
||||
if idx := strings.Index(scan, endTag); idx >= 0 {
|
||||
gotThinking = true
|
||||
after := scan[idx+len(endTag):]
|
||||
thinkTail = thinkTail[:0]
|
||||
inThinking = false
|
||||
token = after
|
||||
} else {
|
||||
if len(scan) >= len(endTag)-1 {
|
||||
thinkTail = []byte(scan[len(scan)-(len(endTag)-1):])
|
||||
} else {
|
||||
thinkTail = []byte(scan)
|
||||
}
|
||||
token = ""
|
||||
}
|
||||
} else {
|
||||
normalBuf.WriteString(token)
|
||||
token = ""
|
||||
combined := normalBuf.String()
|
||||
if idx := strings.Index(combined, startTag); idx >= 0 {
|
||||
before := combined[:idx]
|
||||
after := combined[idx+len(startTag):]
|
||||
normalBuf.Reset()
|
||||
normalBuf.WriteString(before)
|
||||
flushNormal()
|
||||
inThinking = true
|
||||
thinkTail = thinkTail[:0]
|
||||
token = after
|
||||
} else {
|
||||
flushNormal()
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
for _, t := range tokens {
|
||||
onToken(t)
|
||||
}
|
||||
// 处理剩余
|
||||
if !inThinking && normalBuf.Len() > 0 {
|
||||
sentences = append(sentences, normalBuf.String())
|
||||
}
|
||||
return sentences, gotThinking
|
||||
}
|
||||
|
||||
func TestThinking_NoThinkTag(t *testing.T) {
|
||||
// 非 thinking 模式：所有内容正常断句
|
||||
tokens := []string{"你好，", "我是 AI。", "有什么可以帮您？"}
|
||||
sentences, thinking := simulateTokens(tokens)
|
||||
if thinking {
|
||||
t.Error("should not detect thinking")
|
||||
}
|
||||
// "我是 AI。" 和 "有什么可以帮您？" 各触发一次断句
|
||||
if len(sentences) != 2 {
|
||||
t.Errorf("want 2 sentences, got %d: %v", len(sentences), sentences)
|
||||
}
|
||||
if sentences[0] != "你好，我是 AI。" {
|
||||
t.Errorf("sentences[0] = %q", sentences[0])
|
||||
}
|
||||
if sentences[1] != "有什么可以帮您？" {
|
||||
t.Errorf("sentences[1] = %q", sentences[1])
|
||||
}
|
||||
}
|
||||
|
||||
func TestThinking_ThinkBlockSkipped(t *testing.T) {
|
||||
// thinking 模式：<think>...</think> 内容不送 TTS
|
||||
tokens := []string{"<think>让我想一想。</think>", "你好！"}
|
||||
sentences, thinking := simulateTokens(tokens)
|
||||
if !thinking {
|
||||
t.Error("should detect thinking")
|
||||
}
|
||||
if len(sentences) != 1 {
|
||||
t.Errorf("want 1 sentence, got %d: %v", len(sentences), sentences)
|
||||
}
|
||||
if sentences[0] != "你好！" {
|
||||
t.Errorf("sentence[0] = %q", sentences[0])
|
||||
}
|
||||
}
|
||||
|
||||
func TestThinking_TagSplitAcrossTokens(t *testing.T) {
|
||||
// </think> 跨 token 边界
|
||||
tokens := []string{"<think>thinking", "</thi", "nk>回答。"}
|
||||
sentences, thinking := simulateTokens(tokens)
|
||||
if !thinking {
|
||||
t.Error("should detect thinking")
|
||||
}
|
||||
if len(sentences) != 1 || sentences[0] != "回答。" {
|
||||
t.Errorf("sentences = %v", sentences)
|
||||
}
|
||||
}
|
||||
|
||||
func TestThinking_ContentBeforeThink(t *testing.T) {
|
||||
// <think> 前已有正常内容
|
||||
tokens := []string{"先说一句。<think>思考中</think>然后继续。"}
|
||||
sentences, thinking := simulateTokens(tokens)
|
||||
if !thinking {
|
||||
t.Error("should detect thinking")
|
||||
}
|
||||
if len(sentences) != 2 {
|
||||
t.Fatalf("want 2 sentences, got %d: %v", len(sentences), sentences)
|
||||
}
|
||||
if sentences[0] != "先说一句。" {
|
||||
t.Errorf("sentences[0] = %q", sentences[0])
|
||||
}
|
||||
if sentences[1] != "然后继续。" {
|
||||
t.Errorf("sentences[1] = %q", sentences[1])
|
||||
}
|
||||
}
|
||||
|
||||
func TestThinking_MultipleNewlinesInsideThink(t *testing.T) {
|
||||
// thinking 块内大量换行不触发断句
|
||||
tokens := strings.Split("<think>\n\n\n很多换行\n\n\n</think>好的。", "")
|
||||
sentences, thinking := simulateTokens(tokens)
|
||||
if !thinking {
|
||||
t.Error("should detect thinking")
|
||||
}
|
||||
if len(sentences) != 1 || sentences[0] != "好的。" {
|
||||
t.Errorf("sentences = %v", sentences)
|
||||
}
|
||||
}
|
||||
231
cmd/picoclaw-voice/main.go
Normal file
231
cmd/picoclaw-voice/main.go
Normal file
|
|
@ -0,0 +1,231 @@
|
|||
// PicoClaw - Ultra-lightweight personal AI agent
|
||||
// License: MIT
|
||||
//
|
||||
// Copyright (c) 2026 PicoClaw contributors
|
||||
|
||||
// picoclaw-voice: xiaozhi WebSocket gateway — Opus↔ASR→LLM→TTS→Opus pipeline.
|
||||
// 通过 xiaozhi-esp32 协议与客户端通信，内部调用 ASR/LLM/TTS 三方服务。
|
||||
//
|
||||
// 配置通过环境变量注入（见 Config 结构体）:
|
||||
// PICOCLAW_VOICE_LISTEN 监听地址，默认 :8765
|
||||
// PICOCLAW_VOICE_OWNER_ID 固定 owner_id，多设备共享同一份记忆，默认空（per-device 独立记忆）
|
||||
// PICOCLAW_CONFIG picoclaw config.json 路径，默认 ~/.picoclaw/config.json
|
||||
// PICOCLAW_HOME picoclaw home 目录，默认 ~/.picoclaw
|
||||
//
|
||||
// 通用供应商选择：
|
||||
// PICOCLAW_VOICE_ASR_PROVIDER ASR 供应商，可选 doubao | funasr，默认 doubao
|
||||
// PICOCLAW_VOICE_TTS_PROVIDER TTS 供应商，可选 doubao | fishspeech，默认 doubao
|
||||
//
|
||||
// FunASR（本地免费 ASR，推荐中文）：
|
||||
// PICOCLAW_VOICE_ASR_WS_URL FunASR WebSocket 地址，默认 wss://127.0.0.1:10095（内置自签名证书，跳过校验）
|
||||
// PICOCLAW_VOICE_ASR_MODE 识别模式：2pass（默认）| online | offline
|
||||
//
|
||||
// Fish Speech（本地免费 TTS，推荐中文，需 GPU）：
|
||||
// PICOCLAW_VOICE_TTS_API_URL Fish Speech HTTP 地址，默认 http://127.0.0.1:8080
|
||||
// PICOCLAW_VOICE_TTS_REFERENCE_ID 参考音色 ID，留空使用服务默认
|
||||
// PICOCLAW_VOICE_TTS_SAMPLE_RATE 输出采样率，默认 44100（Fish Speech v2 默认）
|
||||
//
|
||||
// 共享火山引擎凭证（doubao 供应商 ASR/TTS 使用同一账号时，只需填这两项）：
|
||||
// PICOCLAW_VOICE_APPID 火山引擎 AppID（ASR/TTS 共用兜底）
|
||||
// PICOCLAW_VOICE_TOKEN 火山引擎 Access Token（ASR/TTS 共用兜底）
|
||||
//
|
||||
// doubao ASR 单独覆盖（可选）：
|
||||
// PICOCLAW_VOICE_ASR_APPID 覆盖 PICOCLAW_VOICE_APPID
|
||||
// PICOCLAW_VOICE_ASR_TOKEN 覆盖 PICOCLAW_VOICE_TOKEN
|
||||
// PICOCLAW_VOICE_ASR_CLUSTER 默认 bigmodel_transcribe
|
||||
// PICOCLAW_VOICE_ASR_RESOURCE_ID 默认 volc.bigasr.sauc.duration
|
||||
//
|
||||
// doubao TTS 单独覆盖（可选）：
|
||||
// PICOCLAW_VOICE_TTS_APPID 覆盖 PICOCLAW_VOICE_APPID
|
||||
// PICOCLAW_VOICE_TTS_TOKEN 覆盖 PICOCLAW_VOICE_TOKEN
|
||||
// PICOCLAW_VOICE_TTS_CLUSTER 默认 volcano_tts
|
||||
// PICOCLAW_VOICE_TTS_VOICE 音色
|
||||
package main
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"log"
|
||||
"net/http"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
|
||||
"github.com/caarlos0/env/v11"
|
||||
"github.com/gorilla/websocket"
|
||||
|
||||
"github.com/sipeed/picoclaw/pkg/agent"
|
||||
"github.com/sipeed/picoclaw/pkg/asr"
|
||||
_ "github.com/sipeed/picoclaw/pkg/asr/doubao"
|
||||
_ "github.com/sipeed/picoclaw/pkg/asr/funasr"
|
||||
"github.com/sipeed/picoclaw/pkg/bus"
|
||||
picoconfig "github.com/sipeed/picoclaw/pkg/config"
|
||||
"github.com/sipeed/picoclaw/pkg/providers"
|
||||
"github.com/sipeed/picoclaw/pkg/tts"
|
||||
_ "github.com/sipeed/picoclaw/pkg/tts/doubao"
|
||||
_ "github.com/sipeed/picoclaw/pkg/tts/fishspeech"
|
||||
)
|
||||
|
||||
type config struct {
|
||||
Listen string `env:"PICOCLAW_VOICE_LISTEN" envDefault:":8765"`
|
||||
OwnerID string `env:"PICOCLAW_VOICE_OWNER_ID"`
|
||||
|
||||
// 共享凭证：ASR 和 TTS 可复用同一套火山引擎账号
|
||||
AppID string `env:"PICOCLAW_VOICE_APPID"`
|
||||
Token string `env:"PICOCLAW_VOICE_TOKEN"`
|
||||
|
||||
// ASR 供应商选择 (doubao | funasr)
|
||||
ASRProvider string `env:"PICOCLAW_VOICE_ASR_PROVIDER" envDefault:"doubao"`
|
||||
ASRAppID string `env:"PICOCLAW_VOICE_ASR_APPID"` // doubao：覆盖 PICOCLAW_VOICE_APPID
|
||||
ASRToken string `env:"PICOCLAW_VOICE_ASR_TOKEN"` // doubao：覆盖 PICOCLAW_VOICE_TOKEN
|
||||
ASRCluster string `env:"PICOCLAW_VOICE_ASR_CLUSTER" envDefault:"bigmodel_transcribe"`
|
||||
ASRResourceID string `env:"PICOCLAW_VOICE_ASR_RESOURCE_ID"` // doubao：资源 ID
|
||||
ASRWsURL string `env:"PICOCLAW_VOICE_ASR_WS_URL"` // funasr：WebSocket 地址
|
||||
ASRMode string `env:"PICOCLAW_VOICE_ASR_MODE"` // funasr：2pass | online | offline
|
||||
|
||||
// TTS 供应商选择 (doubao | fishspeech)
|
||||
TTSProvider string `env:"PICOCLAW_VOICE_TTS_PROVIDER" envDefault:"doubao"`
|
||||
TTSAPIURL string `env:"PICOCLAW_VOICE_TTS_API_URL"` // fishspeech：HTTP 服务地址
|
||||
TTSAPIKey string `env:"PICOCLAW_VOICE_TTS_API_KEY"` // fishspeech：Bearer Token
|
||||
TTSReferenceID string `env:"PICOCLAW_VOICE_TTS_REFERENCE_ID"` // fishspeech：参考音色 ID
|
||||
TTSSampleRate int `env:"PICOCLAW_VOICE_TTS_SAMPLE_RATE" envDefault:"0"` // fishspeech：输出采样率，0=provider 默认
|
||||
TTSSeed int `env:"PICOCLAW_VOICE_TTS_SEED" envDefault:"0"` // fishspeech：固定随机种子，0=随机音色
|
||||
TTSAppID string `env:"PICOCLAW_VOICE_TTS_APPID"` // doubao：覆盖 PICOCLAW_VOICE_APPID
|
||||
TTSToken string `env:"PICOCLAW_VOICE_TTS_TOKEN"` // doubao：覆盖 PICOCLAW_VOICE_TOKEN
|
||||
TTSCluster string `env:"PICOCLAW_VOICE_TTS_CLUSTER" envDefault:"volcano_tts"`
|
||||
TTSVoice string `env:"PICOCLAW_VOICE_TTS_VOICE"`
|
||||
}
|
||||
|
||||
func main() {
|
||||
// 优先从可执行文件所在目录加载 .env，回退到当前工作目录。
|
||||
// 已有同名环境变量时跳过（不覆盖），即 Docker environment: 覆盖 > .env。
|
||||
if exe, err := os.Executable(); err == nil {
|
||||
loadDotEnv(filepath.Join(filepath.Dir(exe), ".env"))
|
||||
} else {
|
||||
loadDotEnv(".env")
|
||||
}
|
||||
|
||||
var cfg config
|
||||
if err := env.Parse(&cfg); err != nil {
|
||||
log.Fatalf("picoclaw-voice: parse config: %v", err)
|
||||
}
|
||||
|
||||
// 加载 picoclaw config 并初始化 AgentLoop (LLM)
|
||||
pcCfgPath := picoclawConfigPath()
|
||||
pcCfg, err := picoconfig.LoadConfig(pcCfgPath)
|
||||
if err != nil {
|
||||
log.Fatalf("picoclaw-voice: load picoclaw config from %s: %v", pcCfgPath, err)
|
||||
}
|
||||
llmProvider, modelID, err := providers.CreateProvider(pcCfg)
|
||||
if err != nil {
|
||||
log.Fatalf("picoclaw-voice: create LLM provider: %v", err)
|
||||
}
|
||||
if modelID != "" {
|
||||
pcCfg.Agents.Defaults.ModelName = modelID
|
||||
}
|
||||
msgBus := bus.NewMessageBus()
|
||||
agentLoop := agent.NewAgentLoop(pcCfg, msgBus, llmProvider)
|
||||
|
||||
log.Printf("picoclaw-voice: ASR config: provider=%s ws_url=%s mode=%s", cfg.ASRProvider, cfg.ASRWsURL, cfg.ASRMode)
|
||||
log.Printf("picoclaw-voice: TTS config: provider=%s api_url=%s seed=%d", cfg.TTSProvider, cfg.TTSAPIURL, cfg.TTSSeed)
|
||||
|
||||
asrProvider, err := asr.New(cfg.ASRProvider, map[string]any{
|
||||
// doubao 字段
|
||||
"appid": orStr(cfg.ASRAppID, cfg.AppID),
|
||||
"access_token": orStr(cfg.ASRToken, cfg.Token),
|
||||
"cluster": cfg.ASRCluster,
|
||||
"resource_id": orStr(cfg.ASRResourceID, "volc.bigasr.sauc.duration"),
|
||||
// funasr 字段
|
||||
"ws_url": cfg.ASRWsURL,
|
||||
"mode": cfg.ASRMode,
|
||||
})
|
||||
if err != nil {
|
||||
log.Fatalf("picoclaw-voice: init ASR: %v", err)
|
||||
}
|
||||
|
||||
ttsProvider, err := tts.New(cfg.TTSProvider, map[string]any{
|
||||
// fishspeech 字段
|
||||
"api_url": cfg.TTSAPIURL,
|
||||
"api_key": cfg.TTSAPIKey,
|
||||
"reference_id": cfg.TTSReferenceID,
|
||||
"sample_rate": cfg.TTSSampleRate,
|
||||
"seed": cfg.TTSSeed,
|
||||
// doubao 字段
|
||||
"appid": orStr(cfg.TTSAppID, cfg.AppID),
|
||||
"access_token": orStr(cfg.TTSToken, cfg.Token),
|
||||
"cluster": cfg.TTSCluster,
|
||||
"voice": cfg.TTSVoice,
|
||||
})
|
||||
if err != nil {
|
||||
log.Fatalf("picoclaw-voice: init TTS: %v", err)
|
||||
}
|
||||
upgrader := websocket.Upgrader{
|
||||
CheckOrigin: func(r *http.Request) bool { return true },
|
||||
}
|
||||
|
||||
reg := newDeviceRegistry()
|
||||
|
||||
http.HandleFunc("/xiaozhi/v1/", func(w http.ResponseWriter, r *http.Request) {
|
||||
conn, err := upgrader.Upgrade(w, r, nil)
|
||||
if err != nil {
|
||||
log.Printf("picoclaw-voice: ws upgrade: %v", err)
|
||||
return
|
||||
}
|
||||
s := newSession(conn, asrProvider, ttsProvider, agentLoop, cfg.OwnerID, reg)
|
||||
s.run()
|
||||
})
|
||||
|
||||
log.Printf("picoclaw-voice: listening on %s", cfg.Listen)
|
||||
if err := http.ListenAndServe(cfg.Listen, nil); err != nil {
|
||||
log.Fatalf("picoclaw-voice: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// picoclawConfigPath 返回 picoclaw config.json 路径。
|
||||
// 优先级: $PICOCLAW_CONFIG > $PICOCLAW_HOME/config.json > ~/.picoclaw/config.json
|
||||
func picoclawConfigPath() string {
|
||||
if p := os.Getenv("PICOCLAW_CONFIG"); p != "" {
|
||||
return p
|
||||
}
|
||||
if h := os.Getenv("PICOCLAW_HOME"); h != "" {
|
||||
return filepath.Join(h, "config.json")
|
||||
}
|
||||
home, _ := os.UserHomeDir()
|
||||
return filepath.Join(home, ".picoclaw", "config.json")
|
||||
}
|
||||
|
||||
// orStr 返回 a（若非空），否则返回 b。
|
||||
func orStr(a, b string) string {
|
||||
if a != "" {
|
||||
return a
|
||||
}
|
||||
return b
|
||||
}
|
||||
|
||||
// loadDotEnv 从指定路径加载 .env 文件。
|
||||
// 格式：KEY=VALUE，忽略空行和 # 注释行。
|
||||
// .env 的值总是生效（覆盖 stale shell export），因此本地开发无需手动 unset 旧变量。
|
||||
// Docker 容器内不存在此文件（Dockerfile 未 COPY），故 Docker 环境不受影响。
|
||||
func loadDotEnv(path string) {
|
||||
f, err := os.Open(path)
|
||||
if err != nil {
|
||||
return // 文件不存在时静默跳过（Docker 容器内正常触发此分支）
|
||||
}
|
||||
defer f.Close()
|
||||
scanner := bufio.NewScanner(f)
|
||||
for scanner.Scan() {
|
||||
line := strings.TrimSpace(scanner.Text())
|
||||
if line == "" || strings.HasPrefix(line, "#") {
|
||||
continue
|
||||
}
|
||||
idx := strings.IndexByte(line, '=')
|
||||
if idx < 0 {
|
||||
continue
|
||||
}
|
||||
key := strings.TrimSpace(line[:idx])
|
||||
val := strings.TrimSpace(line[idx+1:])
|
||||
if key == "" {
|
||||
continue
|
||||
}
|
||||
os.Setenv(key, val) // 总是覆盖，防止 stale shell export 干扰本地调试
|
||||
}
|
||||
}
|
||||
197
cmd/picoclaw-voice/protocol.go
Normal file
197
cmd/picoclaw-voice/protocol.go
Normal file
|
|
@ -0,0 +1,197 @@
|
|||
// PicoClaw - Ultra-lightweight personal AI agent
|
||||
// License: MIT
|
||||
//
|
||||
// Copyright (c) 2026 PicoClaw contributors
|
||||
|
||||
package main
|
||||
|
||||
// protocol.go 定义 picoclaw-voice 与客户端之间的 WebSocket 消息协议。
|
||||
//
|
||||
// 基础协议：xiaozhi（https://github.com/78/xiaozhi-esp32）
|
||||
// 扩展协议：picoclaw 在 xiaozhi 基础上新增若干字段和消息类型，统一标注为 [picoclaw 扩展]。
|
||||
//
|
||||
// ---- 客户端 → 服务端 ----
|
||||
//
|
||||
// hello 握手，见 helloMsg
|
||||
// listen VAD 控制，见 listenMsg；state = "start" | "end" | "stop"
|
||||
// abort 打断当前播放，见 abortMsg
|
||||
// ping 心跳（服务端回 pong，本实现暂不处理）
|
||||
//
|
||||
// ---- 服务端 → 客户端 ----
|
||||
//
|
||||
// hello 握手响应，见 helloReplyMsg
|
||||
// stt 语音识别结果，见 sttMsg
|
||||
// tts 语音合成控制，见 ttsMsg；sentence_start/sentence_end 之间是对应句子的二进制 Opus 帧
|
||||
//
|
||||
// [picoclaw 扩展] 服务端 → 客户端：
|
||||
//
|
||||
// llm LLM 推理通知，见 llmMsg
|
||||
//
|
||||
// ---- picoclaw 扩展字段一览 ----
|
||||
//
|
||||
// 客户端 → 服务端：
|
||||
//
|
||||
// listen.memory_id string
|
||||
// LLM 多轮记忆 key。相同 memory_id 的多次对话共享同一上下文（跨设备、跨会话）。
|
||||
// 不传时退化为 connID，即单连接内记忆隔离。
|
||||
// 典型用法：同一用户在 App 和硬件设备上使用相同 memory_id，实现跨渠道记忆统一。
|
||||
//
|
||||
// 服务端 → 客户端：
|
||||
//
|
||||
// hello.session_id string
|
||||
// 连接级 ID，由服务端在每次 WebSocket 握手时生成（UUID）。
|
||||
// 设备重连时刷新；与客户端 listen.session_id（turn_id，轮次标识）语义不同。
|
||||
// 主要用途：服务端日志关联，客户端无需持久化。
|
||||
//
|
||||
// llm（新增消息类型）
|
||||
// LLM 推理过程通知，xiaozhi 标准协议中无此消息类型，见 llmMsg。
|
||||
// 三种形态：
|
||||
// {"type":"llm","text":"..."}
|
||||
// LLM 断句后的回复文字。比对应的 tts.sentence_start 早约 200-500ms 发出，
|
||||
// 可用于在音频播放前在显示屏上呈现打字机效果。
|
||||
// {"type":"llm","state":"thinking_start"}
|
||||
// 检测到 <think> 标记，模型进入思考阶段。
|
||||
// {"type":"llm","state":"thinking_end","duration_ms":N}
|
||||
// 检测到 </think> 标记，思考结束，duration_ms 为本次 thinking 耗时（毫秒）。
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
|
||||
"github.com/sipeed/picoclaw/pkg/asr"
|
||||
"github.com/sipeed/picoclaw/pkg/tts"
|
||||
)
|
||||
|
||||
// ---- 客户端 → 服务端 ----
|
||||
|
||||
// helloMsg 是客户端握手消息。picoclaw 协议仅使用 device_id 作为设备标识。
|
||||
type helloMsg struct {
|
||||
Type string `json:"type"`
|
||||
Version int `json:"version"`
|
||||
Transport string `json:"transport"`
|
||||
DeviceID string `json:"device_id,omitempty"`
|
||||
AudioParams *audioParams `json:"audio_params,omitempty"`
|
||||
}
|
||||
|
||||
type audioParams struct {
|
||||
Format string `json:"format"`
|
||||
SampleRate int `json:"sample_rate"`
|
||||
Channels int `json:"channels"`
|
||||
}
|
||||
|
||||
// listenMsg 是客户端语音控制消息。
|
||||
// SessionID（session_id）由客户端在每轮问答开始时生成，服务端用作 turn_id，贯穿整轮 ASR→LLM→TTS。
|
||||
// MemoryID（memory_id）是 picoclaw 协议扩展字段：指定 LLM 多轮记忆 key，控制跨会话上下文共享。
|
||||
type listenMsg struct {
|
||||
Type string `json:"type"`
|
||||
State string `json:"state"` // "start" | "end" | "stop"
|
||||
Mode string `json:"mode,omitempty"`
|
||||
SessionID string `json:"session_id,omitempty"`
|
||||
MemoryID string `json:"memory_id,omitempty"` // [picoclaw 扩展] LLM 多轮记忆 key
|
||||
}
|
||||
|
||||
type abortMsg struct {
|
||||
Type string `json:"type"`
|
||||
Reason string `json:"reason,omitempty"`
|
||||
}
|
||||
|
||||
// ---- 服务端 → 客户端 ----
|
||||
|
||||
// helloReplyAudioParams 是服务端 hello 响应中的 audio_params 字段。
|
||||
// frame_duration=60 为 xiaozhi 兼容固定值，用于设备端播放节拍控制。
|
||||
type helloReplyAudioParams struct {
|
||||
Format string `json:"format"`
|
||||
SampleRate int `json:"sample_rate"`
|
||||
Channels int `json:"channels"`
|
||||
FrameDuration int `json:"frame_duration"`
|
||||
}
|
||||
|
||||
// helloReplyMsg 是服务端握手响应。
|
||||
// [picoclaw 扩展] session_id 承载服务端生成的连接级 ID，在设备重连时刷新；
|
||||
// 与客户端 listen.session_id（turn_id）是不同语义的字段。
|
||||
//
|
||||
// audio_params：上行音频格式（客户端 → 服务端，用于 ASR），由 ASR provider 声明。
|
||||
// tts_params：下行音频格式（服务端 → 客户端，用于 TTS 播放），固定为 Opus 16kHz mono。
|
||||
type helloReplyMsg struct {
|
||||
Type string `json:"type"`
|
||||
Version int `json:"version"`
|
||||
Transport string `json:"transport"`
|
||||
SessionID string `json:"session_id"`
|
||||
AsrParams helloReplyAudioParams `json:"asr_params"` // 上行：ASR 期望格式
|
||||
TTSParams helloReplyAudioParams `json:"tts_params"` // 下行：TTS 输出格式
|
||||
}
|
||||
|
||||
// sttMsg 是 ASR 识别结果通知。
|
||||
type sttMsg struct {
|
||||
Type string `json:"type"`
|
||||
Text string `json:"text"`
|
||||
State string `json:"state"` // "recognizing"（中间结果）| "stop"（最终结果）
|
||||
}
|
||||
|
||||
// ttsMsg 驱动客户端播放状态机。
|
||||
type ttsMsg struct {
|
||||
Type string `json:"type"`
|
||||
State string `json:"state"` // "start" | "sentence_start" | "sentence_end" | "stop" | "abort"
|
||||
Text string `json:"text,omitempty"` // 仅 sentence_start 携带
|
||||
}
|
||||
|
||||
// llmMsg 是 picoclaw 扩展的 LLM 推理通知，xiaozhi 标准协议无此类型。
|
||||
// State="thinking_start"：检测到 <think>；State="thinking_end"：检测到 </think>，携带 DurationMs。
|
||||
// Text 字段：LLM 断句后的回复文字，早于对应 tts.sentence_start 发出。
|
||||
type llmMsg struct {
|
||||
Type string `json:"type"`
|
||||
Text string `json:"text,omitempty"`
|
||||
State string `json:"state,omitempty"`
|
||||
DurationMs int64 `json:"duration_ms,omitempty"`
|
||||
}
|
||||
|
||||
// ---- 构造函数 ----
|
||||
|
||||
func newStt(text, state string) []byte {
|
||||
b, _ := json.Marshal(sttMsg{Type: "stt", Text: text, State: state})
|
||||
return b
|
||||
}
|
||||
|
||||
func newTts(state, text string) []byte {
|
||||
b, _ := json.Marshal(ttsMsg{Type: "tts", State: state, Text: text})
|
||||
return b
|
||||
}
|
||||
|
||||
func newLlmText(text string) []byte {
|
||||
b, _ := json.Marshal(llmMsg{Type: "llm", Text: text})
|
||||
return b
|
||||
}
|
||||
|
||||
func newLlmThinkingStart() []byte {
|
||||
b, _ := json.Marshal(llmMsg{Type: "llm", State: "thinking_start"})
|
||||
return b
|
||||
}
|
||||
|
||||
func newLlmThinkingEnd(ms int64) []byte {
|
||||
b, _ := json.Marshal(llmMsg{Type: "llm", State: "thinking_end", DurationMs: ms})
|
||||
return b
|
||||
}
|
||||
|
||||
// helloReply 构建服务端 hello 响应。
|
||||
// asrFmt：ASR provider 声明的上行格式，客户端必须按此格式发送音频。
|
||||
// ttsFmt：TTS provider 声明的下行格式，客户端按此初始化解码器。
|
||||
func helloReply(sessID string, asrFmt asr.AudioFormat, ttsFmt tts.AudioFormat) []byte {
|
||||
b, _ := json.Marshal(helloReplyMsg{
|
||||
Type: "hello",
|
||||
Version: 3,
|
||||
Transport: "websocket",
|
||||
SessionID: sessID,
|
||||
AsrParams: helloReplyAudioParams{
|
||||
Format: asrFmt.Codec,
|
||||
SampleRate: asrFmt.SampleRate,
|
||||
Channels: asrFmt.Channels,
|
||||
FrameDuration: 60,
|
||||
},
|
||||
TTSParams: helloReplyAudioParams{
|
||||
Format: ttsFmt.Codec,
|
||||
SampleRate: ttsFmt.SampleRate,
|
||||
Channels: ttsFmt.Channels,
|
||||
FrameDuration: 60,
|
||||
},
|
||||
})
|
||||
return b
|
||||
}
|
||||
83
docker/docker-compose.asr-tts.yml
Normal file
83
docker/docker-compose.asr-tts.yml
Normal file
|
|
@ -0,0 +1,83 @@
|
|||
# 本地 AI 推理服务（FunASR + Fish Speech）
|
||||
#
|
||||
# 用法（在 repo 根目录执行）：
|
||||
# docker compose -f docker/docker-compose.asr-tts.yml up -d
|
||||
# docker compose -f docker/docker-compose.asr-tts.yml logs -f
|
||||
# docker compose -f docker/docker-compose.asr-tts.yml down
|
||||
#
|
||||
# 前置条件：
|
||||
# 1. 安装 NVIDIA Container Toolkit（Docker GPU 支持）：
|
||||
# curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
|
||||
# ... （见 README）
|
||||
# 2. FunASR 模型首次启动自动从 ModelScope 下载到 ~/models/funasr/（约 500MB）
|
||||
# 3. Fish Speech 模型已内置于 v1.5.1 镜像，无需单独下载
|
||||
#
|
||||
# picoclaw-voice 配置：
|
||||
# PICOCLAW_VOICE_ASR_PROVIDER=funasr
|
||||
# PICOCLAW_VOICE_ASR_WS_URL=ws://127.0.0.1:10095
|
||||
# PICOCLAW_VOICE_TTS_PROVIDER=fishspeech
|
||||
# PICOCLAW_VOICE_TTS_API_URL=http://127.0.0.1:8080
|
||||
|
||||
services:
|
||||
|
||||
# ─── FunASR：本地中文语音识别（2pass 模式，免费）──────────────────────────
|
||||
# 首次启动自动下载模型（约 1 GB），需要访问 ModelScope（modelscope.cn）
|
||||
funasr:
|
||||
image: funasr/funasr:funasr-runtime-sdk-cpu-0.1.0
|
||||
container_name: funasr
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "10095:10095"
|
||||
volumes:
|
||||
# 模型缓存，避免每次重启重新下载
|
||||
- ${HOME}/models/funasr:/workspace/models
|
||||
command: >
|
||||
bash -c "cd /workspace/FunASR/funasr/runtime &&
|
||||
bash run_server.sh
|
||||
--download-model-dir /workspace/models
|
||||
--model-dir damo/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-onnx
|
||||
--vad-dir damo/speech_fsmn_vad_zh-cn-16k-common-onnx
|
||||
--punc-dir damo/punc_ct-transformer_zh-cn-common-vocab272727-onnx
|
||||
--port 10095
|
||||
--certfile 0 --keyfile 0"
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "nc -z localhost 10095 || exit 1"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 5
|
||||
start_period: 120s # 首次启动需下载模型，延迟检查
|
||||
|
||||
# ─── Fish Speech：本地中文 TTS（PCM 输出，模型内置于镜像，需 GPU）──────────
|
||||
# 使用 v1.5.1 版本（含 FireflyGAN decoder，模型已内置无需挂载）
|
||||
fishspeech:
|
||||
image: fishaudio/fish-speech:v1.5.1
|
||||
container_name: fishspeech
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "8080:8080"
|
||||
command:
|
||||
- python
|
||||
- tools/api_server.py
|
||||
- --llama-checkpoint-path
|
||||
- checkpoints/fish-speech-1.5
|
||||
- --decoder-checkpoint-path
|
||||
- checkpoints/fish-speech-1.5/firefly-gan-vq-fsq-8x1024-21hz-generator.pth
|
||||
- --decoder-config-name
|
||||
- firefly_gan_vq
|
||||
- --device
|
||||
- cuda
|
||||
- --listen
|
||||
- 0.0.0.0:8080
|
||||
deploy:
|
||||
resources:
|
||||
reservations:
|
||||
devices:
|
||||
- driver: nvidia
|
||||
count: 1
|
||||
capabilities: [gpu]
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "curl -sf http://localhost:8080/ || exit 1"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 5
|
||||
start_period: 60s
|
||||
52
docker/docker-compose.voice.yml
Normal file
52
docker/docker-compose.voice.yml
Normal file
|
|
@ -0,0 +1,52 @@
|
|||
# picoclaw-voice 部署 docker-compose
|
||||
#
|
||||
# 用法：
|
||||
# 首次启动：docker compose -f docker/docker-compose.voice.yml up -d --build
|
||||
# 重启： docker compose -f docker/docker-compose.voice.yml restart
|
||||
# 查看日志：docker compose -f docker/docker-compose.voice.yml logs -f
|
||||
# 停止： docker compose -f docker/docker-compose.voice.yml down
|
||||
#
|
||||
# 注意：picoclaw-voice 端口由 .env 中的 PICOCLAW_VOICE_LISTEN 决定（默认 :8765）。
|
||||
# 若修改 PICOCLAW_VOICE_LISTEN 端口，需同步修改下方 ports 映射。
|
||||
|
||||
services:
|
||||
# picoclaw gateway + Web UI
|
||||
# Web UI：http://localhost:18800
|
||||
picoclaw:
|
||||
image: sipeed/picoclaw:launcher
|
||||
container_name: picoclaw
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
# 允许容器外部（Web UI）访问 gateway
|
||||
- PICOCLAW_GATEWAY_HOST=0.0.0.0
|
||||
ports:
|
||||
# Web UI，仅本机可访问
|
||||
- "127.0.0.1:18800:18800"
|
||||
volumes:
|
||||
# 与 picoclaw-voice 共享同一份配置、workspace、记忆
|
||||
- /home/kkroid/.picoclaw:/root/.picoclaw
|
||||
|
||||
# picoclaw-voice：xiaozhi 协议语音网关（doubao ASR/TTS）
|
||||
# picoclaw-voice 客户端 连接：ws://<WSL_IP>:18765
|
||||
picoclaw-voice:
|
||||
build:
|
||||
# context 必须是 repo 根，因为 go.mod 有 replace => ../../
|
||||
context: ..
|
||||
dockerfile: cmd/picoclaw-voice/Dockerfile
|
||||
image: picoclaw-voice:local
|
||||
container_name: picoclaw-voice
|
||||
restart: unless-stopped
|
||||
env_file: ../cmd/picoclaw-voice/.env
|
||||
environment:
|
||||
# 容器内通过 host-gateway 访问宿主机的 funasr/fishspeech
|
||||
- PICOCLAW_VOICE_ASR_WS_URL=wss://host.docker.internal:10095
|
||||
- PICOCLAW_VOICE_TTS_API_URL=http://host.docker.internal:8080
|
||||
ports:
|
||||
# picoclaw-voice 客户端（UE5，Windows）经 WSL 局域网连接此端口
|
||||
- "18765:18765"
|
||||
extra_hosts:
|
||||
# Linux 下使 host.docker.internal 解析到宿主机（访问 funasr/fishspeech）
|
||||
- "host.docker.internal:host-gateway"
|
||||
volumes:
|
||||
# 读取 picoclaw 配置：LLM API key、记忆、MCP 工具
|
||||
- /home/kkroid/.picoclaw:/root/.picoclaw
|
||||
248
docs/channels/xiaozhi/README.zh.md
Normal file
248
docs/channels/xiaozhi/README.zh.md
Normal file
|
|
@ -0,0 +1,248 @@
|
|||
# xiaozhi 频道（picoclaw-voice）
|
||||
|
||||
`picoclaw-voice` 是实现 xiaozhi-esp32 WebSocket 协议的独立语音网关进程。它将运行 xiaozhi 固件的 ESP32 设备或任意兼容客户端接入 PicoClaw AI 引擎，完成 **Opus 音频 → ASR → LLM → TTS → Opus 音频** 的全双工对话链路。
|
||||
|
||||
## 架构概览
|
||||
|
||||
```
|
||||
xiaozhi 客户端 picoclaw-voice PicoClaw
|
||||
（ESP32 / UE5 / ...） WebSocket :8765 config.json
|
||||
│ │ │
|
||||
│ WebSocket (Opus) │ │
|
||||
│ ─────────────────────> │ │
|
||||
│ │ 火山引擎 ASR (doubao) │
|
||||
│ │ ─────────────────────> │
|
||||
│ │ AgentLoop (LLM) │
|
||||
│ │ ─────────────────────> │
|
||||
│ │ 火山引擎 TTS (doubao) │
|
||||
│ WebSocket (Opus) │ ─────────────────────> │
|
||||
│ <───────────────────── │ │
|
||||
```
|
||||
|
||||
## 连接
|
||||
|
||||
WebSocket 端点：`ws://<host>:8765/xiaozhi/v1/`
|
||||
|
||||
连接建立后，**客户端必须首先发送 `hello` 消息**，完成握手后方可收发音频。
|
||||
|
||||
## 消息格式
|
||||
|
||||
所有文本消息均为 JSON，二进制消息为原始 Opus 帧。
|
||||
|
||||
### 客户端 → 服务端
|
||||
|
||||
#### hello — 握手
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "hello",
|
||||
"version": 3,
|
||||
"transport": "websocket",
|
||||
"device_id": "aa:bb:cc:dd:ee:ff",
|
||||
"audio_params": {
|
||||
"format": "opus",
|
||||
"sample_rate": 16000,
|
||||
"channels": 1,
|
||||
"frame_duration": 60
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
| 字段 | 类型 | 必填 | 说明 |
|
||||
|------|------|------|------|
|
||||
| `version` | int | 是 | 协议版本，当前为 `3` |
|
||||
| `transport` | string | 是 | 传输方式，固定 `"websocket"` |
|
||||
| `device_id` | string | 否 | 设备唯一标识（MAC 地址或自定义 ID）；留空时服务端自动分配 |
|
||||
| `audio_params.format` | string | 是 | 音频格式，固定 `"opus"` |
|
||||
| `audio_params.sample_rate` | int | 是 | 采样率，固定 `16000` |
|
||||
| `audio_params.channels` | int | 是 | 声道数，固定 `1` |
|
||||
| `audio_params.frame_duration` | int | 是 | 帧时长（ms），固定 `60` |
|
||||
|
||||
---
|
||||
|
||||
#### listen — VAD 控制
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "listen",
|
||||
"state": "start",
|
||||
"session_id": "uuid-v4",
|
||||
"memory_id": "user-key"
|
||||
}
|
||||
```
|
||||
|
||||
| 字段 | 类型 | 必填 | 说明 |
|
||||
|------|------|------|------|
|
||||
| `state` | string | 是 | `"start"` 开始录音 / `"end"` 停止录音（VAD 完成）/ `"stop"` 取消当前轮次 |
|
||||
| `session_id` | string | 否 | 本轮对话的唯一 ID（turn_id）；`start` 时建议由客户端生成 UUID，服务端兜底自动生成 |
|
||||
| `memory_id` | string | 否 | 指定 LLM 记忆 key；留空时依次回退：`PICOCLAW_VOICE_OWNER_ID` → `device_id` |
|
||||
|
||||
状态转换：
|
||||
|
||||
```
|
||||
(空闲) ── start ──> (录音中) ── end ──> (推理中) ── (完成) ──> (空闲)
|
||||
└── stop ──> (空闲，丢弃当前音频)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
#### abort — 打断
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "abort",
|
||||
"reason": "wake_word_detected"
|
||||
}
|
||||
```
|
||||
|
||||
立即中止当前 TTS 播放和推理，服务端发送 `tts.state=abort` 并回到空闲状态。
|
||||
|
||||
---
|
||||
|
||||
#### 二进制帧 — 上行音频
|
||||
|
||||
`listen.start` 之后持续发送的 Opus 压缩帧，每帧 60 ms（960 samples @ 16 kHz）。
|
||||
|
||||
---
|
||||
|
||||
### 服务端 → 客户端
|
||||
|
||||
#### hello — 握手响应
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "hello",
|
||||
"version": 3,
|
||||
"transport": "websocket",
|
||||
"session_id": "uuid-v4",
|
||||
"audio_params": {
|
||||
"format": "opus",
|
||||
"sample_rate": 16000,
|
||||
"channels": 1,
|
||||
"frame_duration": 60
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
| 字段 | 说明 |
|
||||
|------|------|
|
||||
| `session_id` | 本次连接的会话 ID |
|
||||
|
||||
---
|
||||
|
||||
#### stt — ASR 识别结果
|
||||
|
||||
```json
|
||||
{ "type": "stt", "text": "你好世界", "state": "recognizing" }
|
||||
{ "type": "stt", "text": "你好世界。", "state": "stop" }
|
||||
```
|
||||
|
||||
| `state` 值 | 说明 |
|
||||
|------------|------|
|
||||
| `"recognizing"` | 流式中间结果（可能更新） |
|
||||
| `"stop"` | 最终识别结果 |
|
||||
|
||||
---
|
||||
|
||||
#### llm — LLM 推理内容
|
||||
|
||||
```json
|
||||
{ "type": "llm", "text": "你好！有什么可以帮你的？", "emotion": "neutral" }
|
||||
```
|
||||
|
||||
| 字段 | 说明 |
|
||||
|------|------|
|
||||
| `text` | 本句 LLM 输出文本（断句后逐句发送） |
|
||||
| `emotion` | 当前情绪标签，固定 `"neutral"`（预留扩展） |
|
||||
|
||||
---
|
||||
|
||||
#### tts — TTS 状态控制
|
||||
|
||||
```json
|
||||
{ "type": "tts", "state": "start" }
|
||||
{ "type": "tts", "state": "sentence_start", "text": "你好！有什么可以帮你的？" }
|
||||
{ "type": "tts", "state": "sentence_end" }
|
||||
{ "type": "tts", "state": "stop" }
|
||||
```
|
||||
|
||||
| `state` 值 | 说明 |
|
||||
|------------|------|
|
||||
| `"start"` | 开始整轮 TTS 播放 |
|
||||
| `"sentence_start"` | 本句开始，附带文本 `text` 字段 |
|
||||
| `"sentence_end"` | 本句结束 |
|
||||
| `"stop"` | 整轮 TTS 播放结束 |
|
||||
| `"abort"` | TTS 被打断（客户端发 `abort` 或会话异常） |
|
||||
|
||||
---
|
||||
|
||||
#### 二进制帧 — 下行音频
|
||||
|
||||
TTS 合成后的 Opus 帧，编码参数与握手一致（16 kHz、单声道、60 ms）。
|
||||
|
||||
---
|
||||
|
||||
## 完整会话时序
|
||||
|
||||
```
|
||||
Client Server
|
||||
│ │
|
||||
│ WS connect │
|
||||
│ ─────────────────────────> │
|
||||
│ hello (device_id, ...) │
|
||||
│ ─────────────────────────> │
|
||||
│ │ hello (session_id)
|
||||
│ <───────────────────────── │
|
||||
│ │
|
||||
│ listen (state=start) │
|
||||
│ ─────────────────────────> │
|
||||
│ [binary Opus frames] │
|
||||
│ ─────────────────────────> │
|
||||
│ listen (state=end) │
|
||||
│ ─────────────────────────> │
|
||||
│ │ stt (recognizing)
|
||||
│ <───────────────────────── │
|
||||
│ │ stt (stop, final text)
|
||||
│ <───────────────────────── │
|
||||
│ │ llm (text=...)
|
||||
│ <───────────────────────── │
|
||||
│ │ tts (start)
|
||||
│ <───────────────────────── │
|
||||
│ │ tts (sentence_start)
|
||||
│ <───────────────────────── │
|
||||
│ │ [binary Opus frames]
|
||||
│ <───────────────────────── │
|
||||
│ │ tts (sentence_end)
|
||||
│ <───────────────────────── │
|
||||
│ │ tts (stop)
|
||||
│ <───────────────────────── │
|
||||
```
|
||||
|
||||
## 音频规格
|
||||
|
||||
| 参数 | 值 |
|
||||
|------|----|
|
||||
| 编码 | Opus (libopus) |
|
||||
| 采样率 | 16000 Hz |
|
||||
| 声道 | 1（单声道） |
|
||||
| 帧时长 | 60 ms（960 samples/帧） |
|
||||
| 最大帧长 | 4000 bytes |
|
||||
| 应用模式 | VoIP (`OPUS_APPLICATION_VOIP`) |
|
||||
|
||||
## 会话与记忆管理
|
||||
|
||||
**device_id 注册**：同一 `device_id` 新连接到来时，旧会话立即被关闭（last-write-wins），确保不存在幽灵会话。
|
||||
|
||||
**记忆 key 优先级**（高→低）：
|
||||
1. `PICOCLAW_VOICE_OWNER_ID` 环境变量（强制全局共享，适合单用户部署）
|
||||
2. 客户端 `listen.memory_id` 字段
|
||||
3. 客户端 `hello.device_id`
|
||||
4. 服务端自动生成的连接 ID
|
||||
|
||||
**turn_id**：每轮对话的唯一标识，由客户端在 `listen.start` 时通过 `session_id` 字段传入；客户端未提供时服务端自动生成。
|
||||
|
||||
## 与 xiaozhi-esp32-server 的关系
|
||||
|
||||
本模块实现了与 [xiaozhi-esp32-server](https://github.com/xinnan-tech/xiaozhi-esp32-server) 兼容的 v3 协议子集，替换其 LLM/ASR/TTS 后端为 PicoClaw 引擎与火山引擎服务。
|
||||
|
||||
未实现的 xiaozhi 协议扩展：OTA 更新、配网相关消息。
|
||||
1
go.mod
1
go.mod
|
|
@ -49,6 +49,7 @@ require (
|
|||
github.com/mattn/go-isatty v0.0.20 // indirect
|
||||
github.com/ncruces/go-strftime v1.0.0 // indirect
|
||||
github.com/petermattis/goid v0.0.0-20260113132338-7c7de50cc741 // indirect
|
||||
github.com/pion/opus v0.0.0-20260219180131-abe26becac00 // indirect
|
||||
github.com/pmezard/go-difflib v1.0.0 // indirect
|
||||
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec // indirect
|
||||
github.com/rivo/uniseg v0.4.7 // indirect
|
||||
|
|
|
|||
4
go.sum
4
go.sum
|
|
@ -156,6 +156,8 @@ github.com/openai/openai-go/v3 v3.22.0 h1:6MEoNoV8sbjOVmXdvhmuX3BjVbVdcExbVyGixi
|
|||
github.com/openai/openai-go/v3 v3.22.0/go.mod h1:cdufnVK14cWcT9qA1rRtrXx4FTRsgbDPW7Ia7SS5cZo=
|
||||
github.com/petermattis/goid v0.0.0-20260113132338-7c7de50cc741 h1:KPpdlQLZcHfTMQRi6bFQ7ogNO0ltFT4PmtwTLW4W+14=
|
||||
github.com/petermattis/goid v0.0.0-20260113132338-7c7de50cc741/go.mod h1:pxMtw7cyUw6B2bRH0ZBANSPg+AoSud1I1iyJHI69jH4=
|
||||
github.com/pion/opus v0.0.0-20260219180131-abe26becac00 h1:+PXsZ5OSLoyzPdzKvr8X/C4WtIWk7GbAAIKZOrFV744=
|
||||
github.com/pion/opus v0.0.0-20260219180131-abe26becac00/go.mod h1:t5Xog2n682JnawoykACE6nKVmupFvmJvkpM7x6bTv6g=
|
||||
github.com/pkg/diff v0.0.0-20210226163009-20ebb0f2a09e/go.mod h1:pJLUxLENpZxwdsKMEsNbx1VGcRFpLqf3715MtcvvzbA=
|
||||
github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
|
||||
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
|
||||
|
|
@ -352,6 +354,8 @@ gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntN
|
|||
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q=
|
||||
gopkg.in/errgo.v2 v2.1.0/go.mod h1:hNsd1EY+bozCKY1Ytp96fpM3vjJbqLJn88ws8XvfDNI=
|
||||
gopkg.in/fsnotify.v1 v1.4.7/go.mod h1:Tz8NjZHkW78fSQdbUxIjBTcgA1z1m8ZHf0WmKUhAMys=
|
||||
gopkg.in/hraban/opus.v2 v2.0.0-20230925203106-0188a62cb302 h1:xeVptzkP8BuJhoIjNizd2bRHfq9KB9HfOLZu90T04XM=
|
||||
gopkg.in/hraban/opus.v2 v2.0.0-20230925203106-0188a62cb302/go.mod h1:/L5E7a21VWl8DeuCPKxQBdVG5cy+L0MRZ08B1wnqt7g=
|
||||
gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7/go.mod h1:dt/ZhP58zS4L8KSrWDmTeBkI65Dw0HsyUHuEVlX15mw=
|
||||
gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
|
||||
gopkg.in/yaml.v2 v2.2.4/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
|
||||
|
|
|
|||
116
pkg/agent/stream.go
Normal file
116
pkg/agent/stream.go
Normal file
|
|
@ -0,0 +1,116 @@
|
|||
// PicoClaw - Ultra-lightweight personal AI agent
|
||||
// License: MIT
|
||||
//
|
||||
// Copyright (c) 2026 PicoClaw contributors
|
||||
|
||||
package agent
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"time"
|
||||
|
||||
"github.com/sipeed/picoclaw/pkg/logger"
|
||||
"github.com/sipeed/picoclaw/pkg/providers"
|
||||
)
|
||||
|
||||
// RunStreamAgentLoop runs a streaming LLM agent loop for voice input.
|
||||
// Text tokens are delivered via onToken as they are generated by the LLM.
|
||||
// Tool calls are executed transparently between iterations; the loop
|
||||
// re-prompts with results and continues until no more tool calls are issued
|
||||
// or MaxIterations is reached.
|
||||
// The full exchange is persisted to session memory on completion.
|
||||
func (al *AgentLoop) RunStreamAgentLoop(
|
||||
ctx context.Context,
|
||||
sessionKey, userText, channel, chatID string,
|
||||
onToken func(string),
|
||||
) error {
|
||||
agentInst := al.registry.GetDefaultAgent()
|
||||
if agentInst == nil {
|
||||
return fmt.Errorf("no default agent available")
|
||||
}
|
||||
|
||||
sp, ok := agentInst.Provider.(providers.StreamingProvider)
|
||||
if !ok {
|
||||
return fmt.Errorf("provider %T does not implement StreamingProvider", agentInst.Provider)
|
||||
}
|
||||
|
||||
start := time.Now()
|
||||
|
||||
history := agentInst.Sessions.GetHistory(sessionKey)
|
||||
summary := agentInst.Sessions.GetSummary(sessionKey)
|
||||
messages := agentInst.ContextBuilder.BuildMessages(
|
||||
history, summary, userText, nil, channel, chatID,
|
||||
)
|
||||
|
||||
agentInst.Sessions.AddMessage(sessionKey, "user", userText)
|
||||
|
||||
llmOpts := map[string]any{
|
||||
"max_tokens": agentInst.MaxTokens,
|
||||
"temperature": agentInst.Temperature,
|
||||
"prompt_cache_key": agentInst.ID,
|
||||
}
|
||||
|
||||
var finalContent string
|
||||
for iteration := 0; iteration < agentInst.MaxIterations; iteration++ {
|
||||
toolDefs := agentInst.Tools.ToProviderDefs()
|
||||
|
||||
// Only stream tokens on the first (user-facing) iteration.
|
||||
// Subsequent iterations are tool-follow-up calls; no need to stream them.
|
||||
var tokenCb func(string)
|
||||
if iteration == 0 {
|
||||
tokenCb = onToken
|
||||
}
|
||||
|
||||
resp, err := sp.ChatStream(ctx, messages, toolDefs, agentInst.Model, llmOpts, tokenCb)
|
||||
if err != nil {
|
||||
return fmt.Errorf("stream error (iteration %d): %w", iteration+1, err)
|
||||
}
|
||||
finalContent = resp.Content
|
||||
|
||||
if len(resp.ToolCalls) == 0 {
|
||||
break
|
||||
}
|
||||
|
||||
// Normalize tool calls so Name/Arguments map are always populated.
|
||||
normalizedCalls := make([]providers.ToolCall, 0, len(resp.ToolCalls))
|
||||
for _, tc := range resp.ToolCalls {
|
||||
normalizedCalls = append(normalizedCalls, providers.NormalizeToolCall(tc))
|
||||
}
|
||||
|
||||
// Append assistant message with tool calls.
|
||||
assistantMsg := providers.Message{
|
||||
Role: "assistant",
|
||||
Content: resp.Content,
|
||||
}
|
||||
for _, tc := range normalizedCalls {
|
||||
assistantMsg.ToolCalls = append(assistantMsg.ToolCalls, tc)
|
||||
}
|
||||
messages = append(messages, assistantMsg)
|
||||
|
||||
// Execute each tool and add results to context.
|
||||
for _, tc := range normalizedCalls {
|
||||
result := agentInst.Tools.Execute(ctx, tc.Name, tc.Arguments)
|
||||
messages = agentInst.ContextBuilder.AddToolResult(
|
||||
messages, tc.ID, tc.Name, result.ForLLM,
|
||||
)
|
||||
logger.DebugCF("voice-stream", "Tool executed",
|
||||
map[string]any{"tool": tc.Name, "result_len": len(result.ForLLM)})
|
||||
}
|
||||
}
|
||||
|
||||
agentInst.Sessions.AddMessage(sessionKey, "assistant", finalContent)
|
||||
if err := agentInst.Sessions.Save(sessionKey); err != nil {
|
||||
logger.WarnCF("voice-stream", "Failed to save session",
|
||||
map[string]any{"session_key": sessionKey, "error": err.Error()})
|
||||
}
|
||||
|
||||
logger.InfoCF("voice-stream", "Stream completed",
|
||||
map[string]any{
|
||||
"session_key": sessionKey,
|
||||
"latency_ms": time.Since(start).Milliseconds(),
|
||||
"content_len": len(finalContent),
|
||||
})
|
||||
|
||||
return nil
|
||||
}
|
||||
500
pkg/asr/doubao/provider.go
Normal file
500
pkg/asr/doubao/provider.go
Normal file
|
|
@ -0,0 +1,500 @@
|
|||
// PicoClaw - Ultra-lightweight personal AI agent
|
||||
// License: MIT
|
||||
//
|
||||
// Copyright (c) 2026 PicoClaw contributors
|
||||
|
||||
// Package doubao implements the Doubao (火山引擎) streaming ASR provider.
|
||||
// Protocol: binary WebSocket with a 4-byte header + gzip-compressed payloads.
|
||||
// Docs: https://www.volcengine.com/docs/6561/80818
|
||||
package doubao
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"compress/gzip"
|
||||
"context"
|
||||
"encoding/binary"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"log"
|
||||
"net"
|
||||
"net/http"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/google/uuid"
|
||||
"github.com/gorilla/websocket"
|
||||
"github.com/sipeed/picoclaw/pkg/asr"
|
||||
)
|
||||
|
||||
func init() {
|
||||
asr.Register("doubao", func(cfg map[string]any) (asr.Provider, error) {
|
||||
return newProvider(cfg)
|
||||
})
|
||||
}
|
||||
|
||||
const defaultASRURL = "wss://openspeech.bytedance.com/api/v3/sauc/bigmodel"
|
||||
|
||||
// 时长包资源 ID 以控制台实际购买的为准：
|
||||
// 模型1.0: volc.bigasr.sauc.duration 模型2.0(Seed): volc.seedasr.sauc.duration
|
||||
const defaultResourceID = "volc.bigasr.sauc.duration"
|
||||
|
||||
// header byte layout (4 bytes):
|
||||
//
|
||||
// [0]: (version=0x01 << 4) | header_size=0x01
|
||||
// [1]: (message_type << 4) | message_type_specific_flags
|
||||
// [2]: (serial_method << 4) | compression_type
|
||||
// [3]: reserved=0x00
|
||||
const (
|
||||
msgTypeFullClientRequest = 0x01 // JSON init frame
|
||||
msgTypeAudioOnly = 0x02 // audio frame
|
||||
msgTypeServerError = 0x0F
|
||||
|
||||
flagNormal = 0x00
|
||||
flagLastFrame = 0x02
|
||||
|
||||
serialJSON = 0x01
|
||||
compressGZP = 0x01
|
||||
)
|
||||
|
||||
type provider struct {
|
||||
appID string
|
||||
token string
|
||||
cluster string
|
||||
resourceID string
|
||||
wsURL string
|
||||
dialer *websocket.Dialer
|
||||
}
|
||||
|
||||
func newProvider(cfg map[string]any) (*provider, error) {
|
||||
appID, _ := cfg["appid"].(string)
|
||||
token, _ := cfg["access_token"].(string)
|
||||
cluster, _ := cfg["cluster"].(string)
|
||||
rid, _ := cfg["resource_id"].(string)
|
||||
wsURL, _ := cfg["ws_url"].(string)
|
||||
|
||||
if rid == "" {
|
||||
rid = defaultResourceID
|
||||
}
|
||||
if wsURL == "" {
|
||||
wsURL = defaultASRURL
|
||||
}
|
||||
// 快捷API接入只需要 token（API Key）和 cluster；传统模式还需要 appid。
|
||||
if token == "" || cluster == "" {
|
||||
return nil, fmt.Errorf("doubao asr: access_token and cluster required")
|
||||
}
|
||||
|
||||
// 不走代理直连火山引擎 ASR，避免本地 http_proxy 拦截 WebSocket 连接。
|
||||
dialer := &websocket.Dialer{
|
||||
NetDialContext: (&net.Dialer{
|
||||
Timeout: 30 * time.Second,
|
||||
KeepAlive: 30 * time.Second,
|
||||
}).DialContext,
|
||||
HandshakeTimeout: 10 * time.Second,
|
||||
}
|
||||
|
||||
return &provider{appID: appID, token: token, cluster: cluster, resourceID: rid, wsURL: wsURL, dialer: dialer}, nil
|
||||
}
|
||||
|
||||
func (p *provider) Name() string { return "doubao" }
|
||||
|
||||
// AudioFormat 声明 doubao ASR 期望收到原始 PCM 字节（网关透传，无需 Opus 解码）。
|
||||
func (p *provider) AudioFormat() asr.AudioFormat {
|
||||
return asr.AudioFormat{Codec: "pcm", SampleRate: 16000, Channels: 1}
|
||||
}
|
||||
|
||||
// Transcribe sends PCM frames to Doubao streaming ASR and returns the final text.
|
||||
func (p *provider) Transcribe(ctx context.Context, frames [][]byte) (string, error) {
|
||||
var final string
|
||||
err := p.transcribeInternal(ctx, asr.MergeFrames(frames), func(text string, isDef bool) {
|
||||
if isDef {
|
||||
final = text
|
||||
}
|
||||
})
|
||||
return final, err
|
||||
}
|
||||
|
||||
// TranscribeStream sends PCM frames to Doubao streaming ASR and calls callback for each
|
||||
// incremental result. callback is invoked only when recognized text changes, and
|
||||
// final=true when recognition is complete.
|
||||
func (p *provider) TranscribeStream(ctx context.Context, frames [][]byte, callback asr.ResultCallback) error {
|
||||
return p.transcribeInternal(ctx, asr.MergeFrames(frames), callback)
|
||||
}
|
||||
|
||||
// connect 建立到豆包 ASR 服务的 WebSocket 连接并完成握手。
|
||||
// 返回已握手的 conn，调用方负责关闭。
|
||||
// ctx 取消时会异步关闭连接，使阻塞的 ReadMessage 立即返回。
|
||||
func (p *provider) connect(ctx context.Context) (*websocket.Conn, error) {
|
||||
var headers http.Header
|
||||
if p.appID == "" {
|
||||
headers = http.Header{
|
||||
"Authorization": {"Bearer " + p.token},
|
||||
"X-Api-Resource-Id": {p.resourceID},
|
||||
"X-Api-Connect-Id": {uuid.New().String()},
|
||||
}
|
||||
} else {
|
||||
headers = http.Header{
|
||||
"X-Api-App-Key": {p.appID},
|
||||
"X-Api-Access-Key": {p.token},
|
||||
"X-Api-Resource-Id": {p.resourceID},
|
||||
"X-Api-Connect-Id": {uuid.New().String()},
|
||||
}
|
||||
}
|
||||
|
||||
conn, resp, err := p.dialer.DialContext(ctx, p.wsURL, headers)
|
||||
if err != nil {
|
||||
if resp != nil {
|
||||
body, _ := io.ReadAll(io.LimitReader(resp.Body, 512))
|
||||
resp.Body.Close()
|
||||
return nil, fmt.Errorf("doubao asr: dial: %w (HTTP %d: %s)", err, resp.StatusCode, bytes.TrimSpace(body))
|
||||
}
|
||||
return nil, fmt.Errorf("doubao asr: dial: %w", err)
|
||||
}
|
||||
|
||||
// ctx 取消时关闭连接，使 ReadMessage 立即解除阻塞。
|
||||
go func() {
|
||||
<-ctx.Done()
|
||||
conn.Close()
|
||||
}()
|
||||
|
||||
// 发送初始化帧（协议握手）
|
||||
// Model 2.0（seedasr）：鉴权纯靠 HTTP Header，body 里不需要 app 节。
|
||||
initReq := map[string]any{
|
||||
"user": map[string]any{"uid": "picoclaw"},
|
||||
"request": map[string]any{
|
||||
"model_name": "bigmodel",
|
||||
"show_utterances": true,
|
||||
"result_type": "stream",
|
||||
"end_window_size": 200,
|
||||
},
|
||||
"audio": map[string]any{
|
||||
"format": "pcm",
|
||||
"codec": "pcm",
|
||||
"rate": 16000,
|
||||
"bits": 16,
|
||||
"channel": 1,
|
||||
"sample_rate": 16000,
|
||||
},
|
||||
}
|
||||
frame, err := buildJSONFrame(msgTypeFullClientRequest, flagNormal, initReq)
|
||||
if err != nil {
|
||||
conn.Close()
|
||||
return nil, fmt.Errorf("doubao asr: build init frame: %w", err)
|
||||
}
|
||||
if err := conn.WriteMessage(websocket.BinaryMessage, frame); err != nil {
|
||||
conn.Close()
|
||||
return nil, fmt.Errorf("doubao asr: send init: %w", err)
|
||||
}
|
||||
_, initResp, err := conn.ReadMessage()
|
||||
if err != nil {
|
||||
conn.Close()
|
||||
return nil, fmt.Errorf("doubao asr: read init response: %w", err)
|
||||
}
|
||||
if err := checkErrorResponse(initResp); err != nil {
|
||||
conn.Close()
|
||||
return nil, err
|
||||
}
|
||||
return conn, nil
|
||||
}
|
||||
|
||||
// transcribeInternal is the core implementation shared by Transcribe and TranscribeStream.
|
||||
// pcmBytes: 16kHz 16-bit mono PCM, little-endian, raw bytes（由 MergeFrames 合并后传入）。
|
||||
func (p *provider) transcribeInternal(ctx context.Context, pcmBytes []byte, callback asr.ResultCallback) error {
|
||||
conn, err := p.connect(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer conn.Close()
|
||||
|
||||
// Send PCM in 100ms chunks（3200 bytes = 1600 samples × 2 bytes），last chunk has flagLastFrame
|
||||
const chunkBytes = 3200
|
||||
for i := 0; i < len(pcmBytes); {
|
||||
end := i + chunkBytes
|
||||
if end > len(pcmBytes) {
|
||||
end = len(pcmBytes)
|
||||
}
|
||||
isLast := end >= len(pcmBytes)
|
||||
flags := byte(flagNormal)
|
||||
if isLast {
|
||||
flags = flagLastFrame
|
||||
}
|
||||
audioFrame, err := buildAudioFrame(flags, pcmBytes[i:end])
|
||||
if err != nil {
|
||||
return fmt.Errorf("doubao asr: build audio frame: %w", err)
|
||||
}
|
||||
if err := conn.WriteMessage(websocket.BinaryMessage, audioFrame); err != nil {
|
||||
return fmt.Errorf("doubao asr: send audio: %w", err)
|
||||
}
|
||||
i = end
|
||||
|
||||
// Check for context cancellation mid-stream
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return ctx.Err()
|
||||
default:
|
||||
}
|
||||
}
|
||||
|
||||
// Read responses until we get a definite utterance or connection closes.
|
||||
var lastText string
|
||||
for {
|
||||
_, msg, err := conn.ReadMessage()
|
||||
if err != nil {
|
||||
if ctx.Err() != nil {
|
||||
return ctx.Err()
|
||||
}
|
||||
break // server closed normally
|
||||
}
|
||||
text, definite, done := parseASRResult(msg)
|
||||
// 只在文本有变化时回调，消除重复片段
|
||||
if text != "" && text != lastText {
|
||||
lastText = text
|
||||
label := "partial"
|
||||
if definite {
|
||||
label = "final"
|
||||
}
|
||||
log.Printf("doubao asr: %s text=%q", label, text)
|
||||
if callback != nil {
|
||||
callback(text, definite)
|
||||
}
|
||||
}
|
||||
if done {
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
if lastText == "" {
|
||||
log.Printf("doubao asr: no final text (silent or unrecognized)")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// asrResult carries the final ASR recognition result or an error.
|
||||
type asrResult struct {
|
||||
text string
|
||||
err error
|
||||
}
|
||||
|
||||
// streamingSession is an active live doubao ASR session.
|
||||
type streamingSession struct {
|
||||
conn *websocket.Conn
|
||||
sendMu sync.Mutex
|
||||
resultCh chan asrResult // exactly one value written by the read goroutine
|
||||
closedCh chan struct{}
|
||||
closeOnce sync.Once
|
||||
}
|
||||
|
||||
// OpenSession establishes a new real-time ASR session.
|
||||
// The passed context controls the session lifetime: cancellation closes the connection.
|
||||
func (p *provider) OpenSession(ctx context.Context, callback asr.ResultCallback) (asr.StreamingSession, error) {
|
||||
conn, err := p.connect(ctx)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
sess := &streamingSession{
|
||||
conn: conn,
|
||||
resultCh: make(chan asrResult, 1),
|
||||
closedCh: make(chan struct{}),
|
||||
}
|
||||
|
||||
// 读取协程：持续接收 ASR 结果，将最终结果写入 resultCh
|
||||
go func() {
|
||||
var lastText string
|
||||
for {
|
||||
_, msg, err := conn.ReadMessage()
|
||||
if err != nil {
|
||||
sess.resultCh <- asrResult{err: err}
|
||||
return
|
||||
}
|
||||
text, definite, done := parseASRResult(msg)
|
||||
if text != "" && text != lastText {
|
||||
lastText = text
|
||||
label := "partial"
|
||||
if definite {
|
||||
label = "final"
|
||||
}
|
||||
log.Printf("doubao asr: %s text=%q", label, text)
|
||||
if callback != nil {
|
||||
callback(text, definite)
|
||||
}
|
||||
}
|
||||
if done {
|
||||
if lastText == "" {
|
||||
log.Printf("doubao asr: no final text (silent or unrecognized)")
|
||||
}
|
||||
sess.resultCh <- asrResult{text: lastText}
|
||||
return
|
||||
}
|
||||
}
|
||||
}()
|
||||
|
||||
return sess, nil
|
||||
}
|
||||
|
||||
// SendAudio pushes a raw PCM frame to the live ASR session.
|
||||
// frame: 16kHz 16-bit mono PCM bytes（与 AudioFormat 声明一致）。
|
||||
func (ss *streamingSession) SendAudio(frame []byte, isLast bool) error {
|
||||
select {
|
||||
case <-ss.closedCh:
|
||||
return asr.ErrSessionClosed
|
||||
default:
|
||||
}
|
||||
flags := byte(flagNormal)
|
||||
if isLast {
|
||||
flags = flagLastFrame
|
||||
}
|
||||
audioFrame, err := buildAudioFrame(flags, frame)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
ss.sendMu.Lock()
|
||||
defer ss.sendMu.Unlock()
|
||||
return ss.conn.WriteMessage(websocket.BinaryMessage, audioFrame)
|
||||
}
|
||||
|
||||
// Wait blocks until the final ASR result is available, ctx is cancelled,
|
||||
// or the session is closed.
|
||||
func (ss *streamingSession) Wait(ctx context.Context) (string, error) {
|
||||
select {
|
||||
case r := <-ss.resultCh:
|
||||
return r.text, r.err
|
||||
case <-ctx.Done():
|
||||
return "", ctx.Err()
|
||||
case <-ss.closedCh:
|
||||
return "", asr.ErrSessionClosed
|
||||
}
|
||||
}
|
||||
|
||||
// Close aborts the session and releases all resources.
|
||||
func (ss *streamingSession) Close() {
|
||||
ss.closeOnce.Do(func() {
|
||||
close(ss.closedCh)
|
||||
ss.conn.Close()
|
||||
})
|
||||
}
|
||||
|
||||
// buildJSONFrame wraps a JSON payload in the doubao binary frame format.
|
||||
func buildJSONFrame(msgType, flags byte, payload any) ([]byte, error) {
|
||||
data, err := json.Marshal(payload)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
compressed, err := gzipBytes(data)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return buildFrame(msgType, flags, serialJSON, compressGZP, compressed), nil
|
||||
}
|
||||
|
||||
// buildAudioFrame wraps raw PCM bytes in the doubao binary frame format.
|
||||
func buildAudioFrame(flags byte, pcmBytes []byte) ([]byte, error) {
|
||||
compressed, err := gzipBytes(pcmBytes)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return buildFrame(msgTypeAudioOnly, flags, serialJSON, compressGZP, compressed), nil
|
||||
}
|
||||
|
||||
func buildFrame(msgType, flags, serial, compress byte, payload []byte) []byte {
|
||||
hdr := [4]byte{
|
||||
(0x01 << 4) | 0x01, // version=1, header_size=1
|
||||
(msgType << 4) | flags,
|
||||
(serial << 4) | compress,
|
||||
0x00,
|
||||
}
|
||||
frame := make([]byte, 0, 8+len(payload))
|
||||
frame = append(frame, hdr[:]...)
|
||||
frame = binary.BigEndian.AppendUint32(frame, uint32(len(payload)))
|
||||
frame = append(frame, payload...)
|
||||
return frame
|
||||
}
|
||||
|
||||
func checkErrorResponse(data []byte) error {
|
||||
if len(data) < 4 {
|
||||
return fmt.Errorf("doubao asr: response too short (%d bytes)", len(data))
|
||||
}
|
||||
if (data[1] >> 4) == msgTypeServerError {
|
||||
if len(data) >= 8 {
|
||||
code := binary.BigEndian.Uint32(data[4:8])
|
||||
return fmt.Errorf("doubao asr: server error code=%d", code)
|
||||
}
|
||||
return fmt.Errorf("doubao asr: server error")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
type asrPayload struct {
|
||||
Code int `json:"code"`
|
||||
Result struct {
|
||||
Utterances []struct {
|
||||
Text string `json:"text"`
|
||||
Definite bool `json:"definite"`
|
||||
} `json:"utterances"`
|
||||
} `json:"result"`
|
||||
}
|
||||
|
||||
// parseASRResult parses a server response frame, handling optional gzip compression.
|
||||
// Returns (text, definite, done).
|
||||
// - definite=false → 中间结果，text 可能非空（继续等待后续帧）
|
||||
// - definite=true → 最终结果，text 非空，done=true
|
||||
func parseASRResult(data []byte) (text string, definite bool, done bool) {
|
||||
if len(data) < 12 {
|
||||
return "", false, false
|
||||
}
|
||||
if (data[1] >> 4) == msgTypeServerError {
|
||||
return "", false, true
|
||||
}
|
||||
// byte 2 lower nibble = compression type: 0x01 = gzip
|
||||
payload := data[12:]
|
||||
if data[2]&0x0F == compressGZP {
|
||||
uncompressed, err := gunzipBytes(payload)
|
||||
if err != nil {
|
||||
log.Printf("doubao asr: decompress response: %v", err)
|
||||
return "", false, false
|
||||
}
|
||||
payload = uncompressed
|
||||
}
|
||||
var p asrPayload
|
||||
if err := json.Unmarshal(payload, &p); err != nil {
|
||||
log.Printf("doubao asr: parse response JSON: %v", err)
|
||||
return "", false, false
|
||||
}
|
||||
if p.Code == 1013 { // no speech detected
|
||||
return "", false, true
|
||||
}
|
||||
if p.Code != 0 && p.Code != 1000 {
|
||||
log.Printf("doubao asr: server code=%d", p.Code)
|
||||
}
|
||||
for _, u := range p.Result.Utterances {
|
||||
if u.Text != "" {
|
||||
if u.Definite {
|
||||
return u.Text, true, true
|
||||
}
|
||||
// 返回中间结果，不打 log（由调用方在文本变化时记录）
|
||||
return u.Text, false, false
|
||||
}
|
||||
}
|
||||
return "", false, false
|
||||
}
|
||||
|
||||
func gunzipBytes(data []byte) ([]byte, error) {
|
||||
r, err := gzip.NewReader(bytes.NewReader(data))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer r.Close()
|
||||
return io.ReadAll(r)
|
||||
}
|
||||
|
||||
func gzipBytes(data []byte) ([]byte, error) {
|
||||
var buf bytes.Buffer
|
||||
w := gzip.NewWriter(&buf)
|
||||
if _, err := w.Write(data); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err := w.Close(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return buf.Bytes(), nil
|
||||
}
|
||||
218
pkg/asr/doubao/provider_test.go
Normal file
218
pkg/asr/doubao/provider_test.go
Normal file
|
|
@ -0,0 +1,218 @@
|
|||
// PicoClaw - Ultra-lightweight personal AI agent
|
||||
// License: MIT
|
||||
//
|
||||
// Copyright (c) 2026 PicoClaw contributors
|
||||
|
||||
package doubao
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"compress/gzip"
|
||||
"encoding/binary"
|
||||
"encoding/json"
|
||||
"io"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// ---- buildFrame ----
|
||||
|
||||
func TestBuildFrame_HeaderLayout(t *testing.T) {
|
||||
frame := buildFrame(msgTypeFullClientRequest, flagNormal, serialJSON, compressGZP, []byte("payload"))
|
||||
|
||||
if len(frame) < 8 {
|
||||
t.Fatalf("frame too short: %d bytes", len(frame))
|
||||
}
|
||||
// byte[0]: (version=0x01 << 4) | header_size=0x01 = 0x11
|
||||
if frame[0] != 0x11 {
|
||||
t.Errorf("byte[0] = 0x%02x, want 0x11", frame[0])
|
||||
}
|
||||
// byte[1]: (msgType << 4) | flags
|
||||
wantByte1 := byte((msgTypeFullClientRequest << 4) | flagNormal)
|
||||
if frame[1] != wantByte1 {
|
||||
t.Errorf("byte[1] = 0x%02x, want 0x%02x", frame[1], wantByte1)
|
||||
}
|
||||
// byte[2]: (serial << 4) | compress
|
||||
wantByte2 := byte((serialJSON << 4) | compressGZP)
|
||||
if frame[2] != wantByte2 {
|
||||
t.Errorf("byte[2] = 0x%02x, want 0x%02x", frame[2], wantByte2)
|
||||
}
|
||||
// byte[3]: reserved = 0x00
|
||||
if frame[3] != 0x00 {
|
||||
t.Errorf("byte[3] = 0x%02x, want 0x00", frame[3])
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildFrame_PayloadLength(t *testing.T) {
|
||||
payload := []byte("hello world")
|
||||
frame := buildFrame(msgTypeAudioOnly, flagLastFrame, serialJSON, compressGZP, payload)
|
||||
|
||||
// bytes[4:8] = big-endian uint32 of len(payload)
|
||||
gotLen := binary.BigEndian.Uint32(frame[4:8])
|
||||
if gotLen != uint32(len(payload)) {
|
||||
t.Errorf("payload length = %d, want %d", gotLen, len(payload))
|
||||
}
|
||||
if !bytes.Equal(frame[8:], payload) {
|
||||
t.Errorf("payload bytes mismatch")
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildFrame_AudioFlags(t *testing.T) {
|
||||
frame := buildFrame(msgTypeAudioOnly, flagLastFrame, serialJSON, compressGZP, []byte{})
|
||||
// byte[1] should carry flagLastFrame in lower nibble
|
||||
if frame[1]&0x0F != flagLastFrame {
|
||||
t.Errorf("flags nibble = 0x%x, want 0x%x", frame[1]&0x0F, flagLastFrame)
|
||||
}
|
||||
}
|
||||
|
||||
// ---- buildJSONFrame ----
|
||||
|
||||
func TestBuildJSONFrame_GzipPayload(t *testing.T) {
|
||||
payload := map[string]any{"key": "value"}
|
||||
frame, err := buildJSONFrame(msgTypeFullClientRequest, flagNormal, payload)
|
||||
if err != nil {
|
||||
t.Fatalf("buildJSONFrame: %v", err)
|
||||
}
|
||||
if len(frame) < 8 {
|
||||
t.Fatalf("frame too short: %d", len(frame))
|
||||
}
|
||||
// bytes[4:8] = payload length
|
||||
compressedLen := binary.BigEndian.Uint32(frame[4:8])
|
||||
if int(compressedLen) != len(frame)-8 {
|
||||
t.Errorf("compressed length field %d != actual %d", compressedLen, len(frame)-8)
|
||||
}
|
||||
// Verify gzip decompresses to valid JSON containing "key"
|
||||
r, err := gzip.NewReader(bytes.NewReader(frame[8:]))
|
||||
if err != nil {
|
||||
t.Fatalf("gzip reader: %v", err)
|
||||
}
|
||||
raw, err := io.ReadAll(r)
|
||||
if err != nil {
|
||||
t.Fatalf("gzip read: %v", err)
|
||||
}
|
||||
var out map[string]any
|
||||
if err := json.Unmarshal(raw, &out); err != nil {
|
||||
t.Fatalf("json unmarshal: %v", err)
|
||||
}
|
||||
if out["key"] != "value" {
|
||||
t.Errorf("got key=%v, want 'value'", out["key"])
|
||||
}
|
||||
}
|
||||
|
||||
// ---- buildAudioFrame ----
|
||||
|
||||
func TestBuildAudioFrame_NormalFlag(t *testing.T) {
|
||||
pcm := make([]byte, 3200) // 100ms of silence
|
||||
frame, err := buildAudioFrame(flagNormal, pcm)
|
||||
if err != nil {
|
||||
t.Fatalf("buildAudioFrame: %v", err)
|
||||
}
|
||||
if (frame[1] >> 4) != msgTypeAudioOnly {
|
||||
t.Errorf("msgType = 0x%x, want 0x%x", frame[1]>>4, msgTypeAudioOnly)
|
||||
}
|
||||
if frame[1]&0x0F != flagNormal {
|
||||
t.Errorf("flags = 0x%x, want 0x%x", frame[1]&0x0F, flagNormal)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildAudioFrame_LastFlag(t *testing.T) {
|
||||
frame, err := buildAudioFrame(flagLastFrame, []byte{})
|
||||
if err != nil {
|
||||
t.Fatalf("buildAudioFrame: %v", err)
|
||||
}
|
||||
if frame[1]&0x0F != flagLastFrame {
|
||||
t.Errorf("flags = 0x%x, want 0x%x", frame[1]&0x0F, flagLastFrame)
|
||||
}
|
||||
}
|
||||
|
||||
// ---- parseASRResult ----
|
||||
|
||||
func buildMockResponse(code int, text string, definite bool) []byte {
|
||||
payload, _ := json.Marshal(map[string]any{
|
||||
"code": code,
|
||||
"result": map[string]any{
|
||||
"utterances": []map[string]any{
|
||||
{"text": text, "definite": definite},
|
||||
},
|
||||
},
|
||||
})
|
||||
// 响应格式：4 字节 header + 8 字节跳过 + JSON
|
||||
var buf []byte
|
||||
buf = append(buf, 0x11, (0x0B<<4)|0x00, 0x00, 0x00) // header
|
||||
buf = append(buf, 0, 0, 0, 0, 0, 0, 0, 0) // 8 bytes skipped
|
||||
buf = append(buf, payload...)
|
||||
return buf
|
||||
}
|
||||
|
||||
func TestParseASRResult_DefiniteUtterance(t *testing.T) {
|
||||
data := buildMockResponse(1000, "你好世界", true)
|
||||
text, definite, done := parseASRResult(data)
|
||||
if text != "你好世界" {
|
||||
t.Errorf("text = %q, want '你好世界'", text)
|
||||
}
|
||||
if !definite {
|
||||
t.Error("definite = false, want true")
|
||||
}
|
||||
if !done {
|
||||
t.Error("done = false, want true")
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseASRResult_IndefiniteUtterance(t *testing.T) {
|
||||
data := buildMockResponse(1000, "你好", false)
|
||||
text, definite, done := parseASRResult(data)
|
||||
if text != "你好" {
|
||||
t.Errorf("text = %q, want '你好' (intermediate result)", text)
|
||||
}
|
||||
if definite {
|
||||
t.Error("definite = true, want false for non-definite")
|
||||
}
|
||||
if done {
|
||||
t.Error("done = true, want false for non-definite")
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseASRResult_NoSpeechCode(t *testing.T) {
|
||||
data := buildMockResponse(1013, "", false)
|
||||
text, _, done := parseASRResult(data)
|
||||
if text != "" || !done {
|
||||
t.Errorf("got text=%q done=%v, want empty/true for code 1013 (silent, session ends)", text, done)
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseASRResult_TooShort(t *testing.T) {
|
||||
_, _, done := parseASRResult([]byte{0x11, 0x00})
|
||||
if done {
|
||||
t.Error("expected done=false for short frame")
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseASRResult_ServerError(t *testing.T) {
|
||||
// msgType 0x0F in byte[1] upper nibble
|
||||
data := []byte{0x11, (msgTypeServerError << 4), 0x00, 0x00, 0, 0, 0, 42, 0, 0, 0, 0}
|
||||
_, _, done := parseASRResult(data)
|
||||
if !done {
|
||||
t.Error("expected done=true for server error frame")
|
||||
}
|
||||
}
|
||||
|
||||
// ---- checkErrorResponse ----
|
||||
|
||||
func TestCheckErrorResponse_OK(t *testing.T) {
|
||||
frame := buildFrame(0x0B, flagNormal, serialJSON, compressGZP, []byte("{}"))
|
||||
if err := checkErrorResponse(frame); err != nil {
|
||||
t.Errorf("unexpected error: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCheckErrorResponse_ServerError(t *testing.T) {
|
||||
frame := []byte{0x11, (msgTypeServerError << 4) | 0x00, 0x00, 0x00, 0, 0, 0, 99, 0, 0, 0, 4}
|
||||
if err := checkErrorResponse(frame); err == nil {
|
||||
t.Error("expected error for server error frame")
|
||||
}
|
||||
}
|
||||
|
||||
func TestCheckErrorResponse_TooShort(t *testing.T) {
|
||||
if err := checkErrorResponse([]byte{0x11}); err == nil {
|
||||
t.Error("expected error for too-short frame")
|
||||
}
|
||||
}
|
||||
237
pkg/asr/funasr/provider.go
Normal file
237
pkg/asr/funasr/provider.go
Normal file
|
|
@ -0,0 +1,237 @@
|
|||
// PicoClaw - Ultra-lightweight personal AI agent
|
||||
// License: MIT
|
||||
//
|
||||
// Copyright (c) 2026 PicoClaw contributors
|
||||
|
||||
// Package funasr 对接 FunASR WebSocket 服务（iic/SenseVoiceSmall 等模型）。
|
||||
// 支持实时流式 ASR（RealtimeProvider），中文识别效果优秀，完全本地部署，无需付费 API。
|
||||
//
|
||||
// 部署：
|
||||
//
|
||||
// docker compose -f docker/docker-compose.asr-tts.yml up -d funasr
|
||||
//
|
||||
// 配置（环境变量）：
|
||||
//
|
||||
// PICOCLAW_VOICE_ASR_WS_URL WebSocket 地址，默认 wss://127.0.0.1:10095
|
||||
// PICOCLAW_VOICE_ASR_MODE 识别模式：2pass（默认）| online | offline
|
||||
package funasr
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/tls"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"log"
|
||||
"net/http"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/gorilla/websocket"
|
||||
"github.com/sipeed/picoclaw/pkg/asr"
|
||||
)
|
||||
|
||||
func init() {
|
||||
asr.Register("funasr", func(cfg map[string]any) (asr.Provider, error) {
|
||||
return newProvider(cfg)
|
||||
})
|
||||
}
|
||||
|
||||
// 默认使用 WSS，FunASR SDK 镜像内置自签名证书
|
||||
const defaultWSURL = "wss://127.0.0.1:10095"
|
||||
|
||||
type provider struct {
|
||||
wsURL string
|
||||
mode string // "2pass" | "online" | "offline"
|
||||
dialer *websocket.Dialer
|
||||
}
|
||||
|
||||
func newProvider(cfg map[string]any) (*provider, error) {
|
||||
wsURL, _ := cfg["ws_url"].(string)
|
||||
mode, _ := cfg["mode"].(string)
|
||||
if wsURL == "" {
|
||||
wsURL = defaultWSURL
|
||||
}
|
||||
if mode == "" {
|
||||
mode = "2pass"
|
||||
}
|
||||
return &provider{
|
||||
wsURL: wsURL,
|
||||
mode: mode,
|
||||
dialer: &websocket.Dialer{
|
||||
HandshakeTimeout: 10 * time.Second,
|
||||
// FunASR 使用自签名证书，跳过校验
|
||||
TLSClientConfig: &tls.Config{InsecureSkipVerify: true}, //nolint:gosec
|
||||
},
|
||||
}, nil
|
||||
}
|
||||
|
||||
func (p *provider) Name() string { return "funasr" }
|
||||
|
||||
func (p *provider) AudioFormat() asr.AudioFormat {
|
||||
return asr.AudioFormat{Codec: "pcm", SampleRate: 16000, Channels: 1}
|
||||
}
|
||||
|
||||
// Transcribe 批量模式：将所有帧合并后一次性送识别。
|
||||
// 使用 OpenSession 实现，以复用实时 ASR 连接逻辑。
|
||||
func (p *provider) Transcribe(ctx context.Context, frames [][]byte) (string, error) {
|
||||
sess, err := p.OpenSession(ctx, func(_ string, _ bool) {})
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
defer sess.Close()
|
||||
|
||||
pcm := asr.MergeFrames(frames)
|
||||
// 每次推送 3200 字节（100ms @ 16kHz 16-bit mono），匹配 FunASR 推荐块大小
|
||||
const chunkSize = 3200
|
||||
for i := 0; i < len(pcm); i += chunkSize {
|
||||
end := i + chunkSize
|
||||
if end > len(pcm) {
|
||||
end = len(pcm)
|
||||
}
|
||||
if err := sess.SendAudio(pcm[i:end], end >= len(pcm)); err != nil {
|
||||
return "", err
|
||||
}
|
||||
}
|
||||
return sess.Wait(ctx)
|
||||
}
|
||||
|
||||
// OpenSession 实现 RealtimeProvider：建立 WebSocket 连接，启动收包 goroutine。
|
||||
func (p *provider) OpenSession(ctx context.Context, callback asr.ResultCallback) (asr.StreamingSession, error) {
|
||||
// FunASR WebSocket 服务要求 Sec-WebSocket-Protocol: binary
|
||||
header := http.Header{"Sec-WebSocket-Protocol": {"binary"}}
|
||||
conn, _, err := p.dialer.DialContext(ctx, p.wsURL, header)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("funasr: dial %s: %w", p.wsURL, err)
|
||||
}
|
||||
|
||||
// ctx 取消时关闭连接，使 ReadMessage 立即返回
|
||||
go func() {
|
||||
<-ctx.Done()
|
||||
conn.Close()
|
||||
}()
|
||||
|
||||
cfg := map[string]any{
|
||||
"mode": p.mode,
|
||||
"chunk_size": []int{5, 10, 5},
|
||||
"chunk_interval": 10,
|
||||
"wav_name": "picoclaw",
|
||||
"is_speaking": true,
|
||||
"wav_format": "pcm",
|
||||
"itn": true,
|
||||
}
|
||||
cfgBytes, _ := json.Marshal(cfg)
|
||||
if err := conn.WriteMessage(websocket.TextMessage, cfgBytes); err != nil {
|
||||
conn.Close()
|
||||
return nil, fmt.Errorf("funasr: send config: %w", err)
|
||||
}
|
||||
|
||||
sess := &streamSession{
|
||||
conn: conn,
|
||||
callback: callback,
|
||||
mode: p.mode,
|
||||
resultCh: make(chan string, 1),
|
||||
errCh: make(chan error, 1),
|
||||
}
|
||||
go sess.readLoop()
|
||||
return sess, nil
|
||||
}
|
||||
|
||||
// ── session ──────────────────────────────────────────────────────────────────
|
||||
|
||||
type streamSession struct {
|
||||
conn *websocket.Conn
|
||||
callback asr.ResultCallback
|
||||
mode string
|
||||
mu sync.Mutex
|
||||
closed bool
|
||||
resultCh chan string
|
||||
errCh chan error
|
||||
}
|
||||
|
||||
type funasrResult struct {
|
||||
Mode string `json:"mode"`
|
||||
Text string `json:"text"`
|
||||
IsFinal bool `json:"is_final"`
|
||||
}
|
||||
|
||||
// readLoop 持续读取服务端推送的 JSON 识别结果。
|
||||
// 2pass 模式：2pass-online 为中间结果，2pass-offline 为最终高质量结果。
|
||||
// online/offline 模式：is_final=true 表示识别完成。
|
||||
func (s *streamSession) readLoop() {
|
||||
for {
|
||||
_, data, err := s.conn.ReadMessage()
|
||||
if err != nil {
|
||||
// ctx 取消时 conn.Close() 触发此处，属于正常退出路径
|
||||
select {
|
||||
case s.errCh <- fmt.Errorf("funasr: read: %w", err):
|
||||
default:
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
var r funasrResult
|
||||
if err := json.Unmarshal(data, &r); err != nil {
|
||||
log.Printf("funasr: parse result: %v (raw: %s)", err, data)
|
||||
continue
|
||||
}
|
||||
|
||||
// 判断是否为最终结果：
|
||||
// - 2pass-offline：FunASR 双路模式的最终离线结果（最高精度）
|
||||
// - offline：纯离线单路模式，收到结果即终态
|
||||
// - is_final=true：在线模式显式标记
|
||||
isFinal := r.Mode == "2pass-offline" || r.Mode == "offline" || (r.Mode == "online" && r.IsFinal)
|
||||
|
||||
s.callback(r.Text, isFinal)
|
||||
|
||||
if isFinal {
|
||||
select {
|
||||
case s.resultCh <- r.Text:
|
||||
default:
|
||||
}
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (s *streamSession) SendAudio(frame []byte, isLast bool) error {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
if s.closed {
|
||||
return asr.ErrSessionClosed
|
||||
}
|
||||
if len(frame) > 0 {
|
||||
if err := s.conn.WriteMessage(websocket.BinaryMessage, frame); err != nil {
|
||||
return fmt.Errorf("funasr: send audio: %w", err)
|
||||
}
|
||||
}
|
||||
if isLast {
|
||||
end, _ := json.Marshal(map[string]any{"is_speaking": false})
|
||||
if err := s.conn.WriteMessage(websocket.TextMessage, end); err != nil {
|
||||
return fmt.Errorf("funasr: send end: %w", err)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (s *streamSession) Wait(ctx context.Context) (string, error) {
|
||||
select {
|
||||
case text := <-s.resultCh:
|
||||
return text, nil
|
||||
case err := <-s.errCh:
|
||||
if ctx.Err() != nil {
|
||||
return "", asr.ErrSessionClosed
|
||||
}
|
||||
return "", err
|
||||
case <-ctx.Done():
|
||||
return "", asr.ErrSessionClosed
|
||||
}
|
||||
}
|
||||
|
||||
func (s *streamSession) Close() {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
if !s.closed {
|
||||
s.closed = true
|
||||
s.conn.Close()
|
||||
}
|
||||
}
|
||||
102
pkg/asr/provider.go
Normal file
102
pkg/asr/provider.go
Normal file
|
|
@ -0,0 +1,102 @@
|
|||
// PicoClaw - Ultra-lightweight personal AI agent
|
||||
// License: MIT
|
||||
//
|
||||
// Copyright (c) 2026 PicoClaw contributors
|
||||
|
||||
package asr
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
)
|
||||
|
||||
// AudioFormat describes the audio wire format a provider expects from the device.
|
||||
// The gateway uses this to populate helloReply.audio_params; devices must send accordingly.
|
||||
type AudioFormat struct {
|
||||
Codec string // "pcm" | "opus"
|
||||
SampleRate int // Hz, e.g. 16000
|
||||
Channels int // 1 = mono
|
||||
}
|
||||
|
||||
// MergeFrames concatenates audio frames into a single byte slice.
|
||||
// Useful for batch-mode providers that process all audio at once.
|
||||
func MergeFrames(frames [][]byte) []byte {
|
||||
total := 0
|
||||
for _, f := range frames {
|
||||
total += len(f)
|
||||
}
|
||||
merged := make([]byte, 0, total)
|
||||
for _, f := range frames {
|
||||
merged = append(merged, f...)
|
||||
}
|
||||
return merged
|
||||
}
|
||||
|
||||
// Provider is the ASR interface.
|
||||
type Provider interface {
|
||||
Name() string
|
||||
// AudioFormat returns the audio format this provider expects from the device.
|
||||
// The gateway advertises this via helloReply so the device sends the correct format.
|
||||
AudioFormat() AudioFormat
|
||||
// Transcribe converts audio frames to text.
|
||||
// frames: raw audio bytes in the format declared by AudioFormat().
|
||||
Transcribe(ctx context.Context, frames [][]byte) (string, error)
|
||||
}
|
||||
|
||||
// ResultCallback receives incremental ASR results.
|
||||
// final=true 表示识别完成（definite），后续不再有回调。
|
||||
type ResultCallback func(text string, final bool)
|
||||
|
||||
// StreamingProvider extends Provider with batch streaming ASR support.
|
||||
// The callback is invoked each time the recognized text changes, and once more
|
||||
// with final=true when recognition is complete.
|
||||
type StreamingProvider interface {
|
||||
Provider
|
||||
TranscribeStream(ctx context.Context, frames [][]byte, callback ResultCallback) error
|
||||
}
|
||||
|
||||
// ErrSessionClosed is returned by StreamingSession.Wait when the session
|
||||
// was closed before a final result was available.
|
||||
var ErrSessionClosed = errors.New("asr: session closed")
|
||||
|
||||
// StreamingSession is an active real-time ASR session.
|
||||
// Open with RealtimeProvider.OpenSession; feed audio with SendAudio;
|
||||
// signal end-of-audio by passing isLast=true; then call Wait for the result.
|
||||
type StreamingSession interface {
|
||||
// SendAudio pushes a raw audio frame. Set isLast=true on the final frame.
|
||||
// frame: raw audio bytes in the format declared by the provider's AudioFormat().
|
||||
SendAudio(frame []byte, isLast bool) error
|
||||
// Wait blocks until the final recognition result is available.
|
||||
Wait(ctx context.Context) (string, error)
|
||||
// Close aborts the session and releases all resources.
|
||||
Close()
|
||||
}
|
||||
|
||||
// RealtimeProvider extends Provider with live frame-by-frame ASR.
|
||||
// Audio is fed incrementally as it arrives, reducing latency.
|
||||
type RealtimeProvider interface {
|
||||
Provider
|
||||
// OpenSession starts a new live ASR session.
|
||||
// callback receives each incremental result (final=true on completion).
|
||||
OpenSession(ctx context.Context, callback ResultCallback) (StreamingSession, error)
|
||||
}
|
||||
|
||||
// Factory creates a Provider from a config map.
|
||||
type Factory func(cfg map[string]any) (Provider, error)
|
||||
|
||||
var factories = map[string]Factory{}
|
||||
|
||||
// Register adds a provider factory. Called from provider init() functions.
|
||||
func Register(name string, f Factory) {
|
||||
factories[name] = f
|
||||
}
|
||||
|
||||
// New creates a Provider by name.
|
||||
func New(name string, cfg map[string]any) (Provider, error) {
|
||||
f, ok := factories[name]
|
||||
if !ok {
|
||||
return nil, fmt.Errorf("asr: provider %q not registered", name)
|
||||
}
|
||||
return f(cfg)
|
||||
}
|
||||
|
|
@ -52,6 +52,17 @@ func (p *HTTPProvider) Chat(
|
|||
return p.delegate.Chat(ctx, messages, tools, model, options)
|
||||
}
|
||||
|
||||
func (p *HTTPProvider) ChatStream(
|
||||
ctx context.Context,
|
||||
messages []Message,
|
||||
tools []ToolDefinition,
|
||||
model string,
|
||||
options map[string]any,
|
||||
onToken func(string),
|
||||
) (*LLMResponse, error) {
|
||||
return p.delegate.ChatStream(ctx, messages, tools, model, options, onToken)
|
||||
}
|
||||
|
||||
func (p *HTTPProvider) GetDefaultModel() string {
|
||||
return ""
|
||||
}
|
||||
|
|
|
|||
198
pkg/providers/openai_compat/streaming.go
Normal file
198
pkg/providers/openai_compat/streaming.go
Normal file
|
|
@ -0,0 +1,198 @@
|
|||
// PicoClaw - Ultra-lightweight personal AI agent
|
||||
// License: MIT
|
||||
//
|
||||
// Copyright (c) 2026 PicoClaw contributors
|
||||
|
||||
package openai_compat
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// ChatStream implements providers.StreamingProvider on *Provider.
|
||||
// Text tokens are delivered via onToken as they arrive. Tool calls are
|
||||
// accumulated across chunks and returned in the final *LLMResponse.
|
||||
func (p *Provider) ChatStream(
|
||||
ctx context.Context,
|
||||
messages []Message,
|
||||
tools []ToolDefinition,
|
||||
model string,
|
||||
options map[string]any,
|
||||
onToken func(string),
|
||||
) (*LLMResponse, error) {
|
||||
if p.apiBase == "" {
|
||||
return nil, fmt.Errorf("API base not configured")
|
||||
}
|
||||
model = normalizeModel(model, p.apiBase)
|
||||
|
||||
requestBody := map[string]any{
|
||||
"model": model,
|
||||
"messages": serializeMessages(messages),
|
||||
"stream": true,
|
||||
}
|
||||
if len(tools) > 0 {
|
||||
requestBody["tools"] = tools
|
||||
requestBody["tool_choice"] = "auto"
|
||||
}
|
||||
if maxTokens, ok := asInt(options["max_tokens"]); ok {
|
||||
fieldName := p.maxTokensField
|
||||
if fieldName == "" {
|
||||
lm := strings.ToLower(model)
|
||||
if strings.Contains(lm, "glm") || strings.Contains(lm, "o1") || strings.Contains(lm, "gpt-5") {
|
||||
fieldName = "max_completion_tokens"
|
||||
} else {
|
||||
fieldName = "max_tokens"
|
||||
}
|
||||
}
|
||||
requestBody[fieldName] = maxTokens
|
||||
}
|
||||
if temperature, ok := asFloat(options["temperature"]); ok {
|
||||
lm := strings.ToLower(model)
|
||||
if strings.Contains(lm, "kimi") && strings.Contains(lm, "k2") {
|
||||
requestBody["temperature"] = 1.0
|
||||
} else {
|
||||
requestBody["temperature"] = temperature
|
||||
}
|
||||
}
|
||||
|
||||
jsonData, err := json.Marshal(requestBody)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to marshal request: %w", err)
|
||||
}
|
||||
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodPost, p.apiBase+"/chat/completions", bytes.NewReader(jsonData))
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to create request: %w", err)
|
||||
}
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
if p.apiKey != "" {
|
||||
req.Header.Set("Authorization", "Bearer "+p.apiKey)
|
||||
}
|
||||
|
||||
// Use a client without a read timeout — context handles cancellation.
|
||||
streamClient := &http.Client{Transport: p.httpClient.Transport}
|
||||
resp, err := streamClient.Do(req)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to send request: %w", err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
body, _ := io.ReadAll(io.LimitReader(resp.Body, 256))
|
||||
return nil, fmt.Errorf("API request failed: status=%d body=%s", resp.StatusCode, responsePreview(body, 128))
|
||||
}
|
||||
|
||||
return parseStreamResponse(resp.Body, onToken)
|
||||
}
|
||||
|
||||
type toolCallChunkBuilder struct {
|
||||
id string
|
||||
callType string
|
||||
name string
|
||||
args strings.Builder
|
||||
}
|
||||
|
||||
func parseStreamResponse(body io.Reader, onToken func(string)) (*LLMResponse, error) {
|
||||
var contentBuf strings.Builder
|
||||
var finishReason string
|
||||
builders := map[int]*toolCallChunkBuilder{}
|
||||
|
||||
scanner := bufio.NewScanner(body)
|
||||
scanner.Buffer(make([]byte, 64*1024), 64*1024)
|
||||
for scanner.Scan() {
|
||||
line := scanner.Text()
|
||||
if !strings.HasPrefix(line, "data: ") {
|
||||
continue
|
||||
}
|
||||
data := strings.TrimPrefix(line, "data: ")
|
||||
if data == "[DONE]" {
|
||||
break
|
||||
}
|
||||
|
||||
var chunk struct {
|
||||
Choices []struct {
|
||||
Delta struct {
|
||||
Content string `json:"content"`
|
||||
ToolCalls []struct {
|
||||
Index int `json:"index"`
|
||||
ID string `json:"id"`
|
||||
Type string `json:"type"`
|
||||
Function *struct {
|
||||
Name string `json:"name"`
|
||||
Arguments string `json:"arguments"`
|
||||
} `json:"function"`
|
||||
} `json:"tool_calls"`
|
||||
} `json:"delta"`
|
||||
FinishReason *string `json:"finish_reason"`
|
||||
} `json:"choices"`
|
||||
}
|
||||
if err := json.Unmarshal([]byte(data), &chunk); err != nil {
|
||||
continue
|
||||
}
|
||||
if len(chunk.Choices) == 0 {
|
||||
continue
|
||||
}
|
||||
choice := chunk.Choices[0]
|
||||
if choice.FinishReason != nil {
|
||||
finishReason = *choice.FinishReason
|
||||
}
|
||||
if choice.Delta.Content != "" {
|
||||
contentBuf.WriteString(choice.Delta.Content)
|
||||
if onToken != nil {
|
||||
onToken(choice.Delta.Content)
|
||||
}
|
||||
}
|
||||
for _, tc := range choice.Delta.ToolCalls {
|
||||
b, ok := builders[tc.Index]
|
||||
if !ok {
|
||||
b = &toolCallChunkBuilder{}
|
||||
builders[tc.Index] = b
|
||||
}
|
||||
if tc.ID != "" {
|
||||
b.id = tc.ID
|
||||
}
|
||||
if tc.Type != "" {
|
||||
b.callType = tc.Type
|
||||
}
|
||||
if tc.Function != nil {
|
||||
if tc.Function.Name != "" {
|
||||
b.name = tc.Function.Name
|
||||
}
|
||||
b.args.WriteString(tc.Function.Arguments)
|
||||
}
|
||||
}
|
||||
}
|
||||
if err := scanner.Err(); err != nil {
|
||||
return nil, fmt.Errorf("stream read error: %w", err)
|
||||
}
|
||||
|
||||
toolCalls := make([]ToolCall, 0, len(builders))
|
||||
for i := 0; i < len(builders); i++ {
|
||||
b, ok := builders[i]
|
||||
if !ok {
|
||||
break
|
||||
}
|
||||
toolCalls = append(toolCalls, ToolCall{
|
||||
ID: b.id,
|
||||
Type: b.callType,
|
||||
Name: b.name,
|
||||
Function: &FunctionCall{
|
||||
Name: b.name,
|
||||
Arguments: b.args.String(),
|
||||
},
|
||||
})
|
||||
}
|
||||
|
||||
return &LLMResponse{
|
||||
Content: contentBuf.String(),
|
||||
ToolCalls: toolCalls,
|
||||
FinishReason: finishReason,
|
||||
}, nil
|
||||
}
|
||||
23
pkg/providers/streaming.go
Normal file
23
pkg/providers/streaming.go
Normal file
|
|
@ -0,0 +1,23 @@
|
|||
// PicoClaw - Ultra-lightweight personal AI agent
|
||||
// License: MIT
|
||||
//
|
||||
// Copyright (c) 2026 PicoClaw contributors
|
||||
|
||||
package providers
|
||||
|
||||
import "context"
|
||||
|
||||
// StreamingProvider extends LLMProvider with streaming capability.
|
||||
// Text tokens are delivered via onToken as they arrive from the API.
|
||||
// Tool calls are accumulated internally and returned in the final *LLMResponse.
|
||||
type StreamingProvider interface {
|
||||
LLMProvider
|
||||
ChatStream(
|
||||
ctx context.Context,
|
||||
messages []Message,
|
||||
tools []ToolDefinition,
|
||||
model string,
|
||||
options map[string]any,
|
||||
onToken func(string),
|
||||
) (*LLMResponse, error)
|
||||
}
|
||||
331
pkg/tts/doubao/provider.go
Normal file
331
pkg/tts/doubao/provider.go
Normal file
|
|
@ -0,0 +1,331 @@
|
|||
// PicoClaw - Ultra-lightweight personal AI agent
|
||||
// License: MIT
|
||||
//
|
||||
// Copyright (c) 2026 PicoClaw contributors
|
||||
|
||||
// Package doubao 通过豆包（火山引擎）TTS WebSocket 流式接口合成语音。
|
||||
// 协议参考：wss://openspeech.bytedance.com/api/v1/tts/ws_binary
|
||||
//
|
||||
// 服务端帧类型（data[1]>>4）：
|
||||
//
|
||||
// 0x0B（音频帧）：4字节头 + 4字节seq + 4字节载荷长度 + Ogg Opus 数据
|
||||
// 0x0C（合成结束）/0x0F（错误）：4字节头 + 4字节载荷长度 + gzip JSON
|
||||
//
|
||||
// 鉴权：Authorization: Bearer;{token}（火山引擎非标准分号格式）
|
||||
// 无 CGO，纯 Go 实现。
|
||||
package doubao
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"compress/gzip"
|
||||
"context"
|
||||
"encoding/binary"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"log"
|
||||
"net"
|
||||
"net/http"
|
||||
"time"
|
||||
|
||||
"github.com/google/uuid"
|
||||
"github.com/gorilla/websocket"
|
||||
"github.com/sipeed/picoclaw/pkg/tts"
|
||||
)
|
||||
|
||||
const (
|
||||
defaultWSURL = "wss://openspeech.bytedance.com/api/v1/tts/ws_binary"
|
||||
defaultCluster = "volcano_tts"
|
||||
defaultVoice = "zh_female_wanwanxiaohe_moon_bigtts"
|
||||
)
|
||||
|
||||
// 二进制帧协议常量
|
||||
const (
|
||||
msgTypeFullClientRequest = byte(0x01)
|
||||
msgTypeServerAudio = byte(0x0B) // 音频帧，含 seq + payload_size
|
||||
msgTypeServerError = byte(0x0F) // 错误帧，含 payload_size（无 seq）
|
||||
|
||||
flagNormal = byte(0x00)
|
||||
serialJSON = byte(0x01)
|
||||
compressGZP = byte(0x01)
|
||||
)
|
||||
|
||||
func init() {
|
||||
tts.Register("doubao", func(cfg map[string]any) (tts.Provider, error) {
|
||||
return newProvider(cfg)
|
||||
})
|
||||
}
|
||||
|
||||
type provider struct {
|
||||
appid string
|
||||
accessToken string
|
||||
cluster string
|
||||
defaultVoice string
|
||||
wsURL string
|
||||
dialer *websocket.Dialer
|
||||
}
|
||||
|
||||
func newProvider(cfg map[string]any) (*provider, error) {
|
||||
get := func(key string) string {
|
||||
v, _ := cfg[key].(string)
|
||||
return v
|
||||
}
|
||||
p := &provider{
|
||||
appid: get("appid"),
|
||||
accessToken: get("access_token"),
|
||||
cluster: get("cluster"),
|
||||
defaultVoice: get("voice"),
|
||||
wsURL: get("ws_url"),
|
||||
}
|
||||
if p.accessToken == "" {
|
||||
return nil, fmt.Errorf("tts/doubao: access_token is required")
|
||||
}
|
||||
if p.cluster == "" {
|
||||
p.cluster = defaultCluster
|
||||
}
|
||||
if p.defaultVoice == "" {
|
||||
p.defaultVoice = defaultVoice
|
||||
}
|
||||
if p.wsURL == "" {
|
||||
p.wsURL = defaultWSURL
|
||||
}
|
||||
// 不走代理直连火山引擎，避免本地 http_proxy 拦截 WebSocket。
|
||||
p.dialer = &websocket.Dialer{
|
||||
NetDialContext: (&net.Dialer{
|
||||
Timeout: 30 * time.Second,
|
||||
KeepAlive: 30 * time.Second,
|
||||
}).DialContext,
|
||||
HandshakeTimeout: 10 * time.Second,
|
||||
}
|
||||
return p, nil
|
||||
}
|
||||
|
||||
func (p *provider) Name() string { return "doubao" }
|
||||
|
||||
func (p *provider) AudioFormat() tts.AudioFormat {
|
||||
return tts.AudioFormat{Codec: "opus", SampleRate: 16000, Channels: 1}
|
||||
}
|
||||
|
||||
// SynthesizeFrames 通过豆包 TTS WebSocket 流式接口合成语音。
|
||||
// 连接建立后立即发送合成请求（含文字），服务端边合成边推送原始 Opus 帧，
|
||||
// 每帧到达即触发 onFrame 回调，无需等待整句合成完成。
|
||||
func (p *provider) SynthesizeFrames(ctx context.Context, text, voice string, onFrame func([]byte)) error {
|
||||
if voice == "" {
|
||||
voice = p.defaultVoice
|
||||
}
|
||||
|
||||
// 火山引擎 TTS WS 接口要求 Authorization: Bearer;{token}（分号分隔，非标准空格）。
|
||||
// appid 在 JSON 请求体 app.appid 里传，Bearer 头仅携带 token。
|
||||
headers := http.Header{
|
||||
"Authorization": {"Bearer;" + p.accessToken},
|
||||
"X-Api-Connect-Id": {uuid.New().String()},
|
||||
}
|
||||
|
||||
conn, resp, err := p.dialer.DialContext(ctx, p.wsURL, headers)
|
||||
if err != nil {
|
||||
if resp != nil {
|
||||
body, _ := io.ReadAll(io.LimitReader(resp.Body, 512))
|
||||
resp.Body.Close()
|
||||
return fmt.Errorf("tts/doubao: dial: %w (HTTP %d: %s)", err, resp.StatusCode, bytes.TrimSpace(body))
|
||||
}
|
||||
return fmt.Errorf("tts/doubao: dial: %w", err)
|
||||
}
|
||||
defer conn.Close()
|
||||
|
||||
// ctx 取消时关闭连接，使 ReadMessage 立即解除阻塞。
|
||||
go func() {
|
||||
<-ctx.Done()
|
||||
conn.Close()
|
||||
}()
|
||||
|
||||
// 发送合成请求帧（包含完整文字；operation="submit" 触发流式推送）
|
||||
// 注意：火山引擎 TTS 二进制 WS 协议中 app.token 需加 "access_" 前缀，
|
||||
// 而 HTTP Authorization Bearer 使用原始 token。
|
||||
reqPayload := map[string]any{
|
||||
"app": map[string]any{
|
||||
"appid": p.appid,
|
||||
"token": "access_" + p.accessToken,
|
||||
"cluster": p.cluster,
|
||||
},
|
||||
"user": map[string]any{"uid": "picoclaw"},
|
||||
"audio": map[string]any{
|
||||
"voice_type": voice,
|
||||
// ogg_opus：服务端推送 Ogg 容器包装的 Opus 数据；
|
||||
// 服务端 API encoding 与内部 AudioFormat codec 解耦，
|
||||
// 此处通过 io.Pipe + ParseOggOpusPackets 流式解包后回调原始 Opus 帧。
|
||||
"encoding": "ogg_opus",
|
||||
"rate": 16000,
|
||||
"channel": 1,
|
||||
},
|
||||
"request": map[string]any{
|
||||
"reqid": uuid.New().String(),
|
||||
"text": text,
|
||||
"text_type": "plain",
|
||||
"operation": "submit", // submit = 流式推送；query = HTTP 一次性返回
|
||||
"with_frontend": 1,
|
||||
"frontend_type": "unitTson",
|
||||
},
|
||||
}
|
||||
initFrame, err := buildTTSFrame(reqPayload)
|
||||
if err != nil {
|
||||
return fmt.Errorf("tts/doubao: build request frame: %w", err)
|
||||
}
|
||||
if err := conn.WriteMessage(websocket.BinaryMessage, initFrame); err != nil {
|
||||
return fmt.Errorf("tts/doubao: send request: %w", err)
|
||||
}
|
||||
|
||||
// 用 io.Pipe 将流式 Ogg 载荷接入 ParseOggOpusPackets：
|
||||
// 主循环写 → 解析协程读，边收 WS 帧边解包 Opus，实现真正流式回调。
|
||||
pr, pw := io.Pipe()
|
||||
parseErrCh := make(chan error, 1)
|
||||
go func() {
|
||||
parseErrCh <- tts.ParseOggOpusPackets(pr, onFrame)
|
||||
}()
|
||||
|
||||
frameCount := 0
|
||||
for {
|
||||
_, msg, err := conn.ReadMessage()
|
||||
if err != nil {
|
||||
if ctx.Err() != nil {
|
||||
pw.CloseWithError(ctx.Err())
|
||||
return ctx.Err()
|
||||
}
|
||||
pw.CloseWithError(err)
|
||||
break
|
||||
}
|
||||
audio, isLast, parseErr := parseTTSFrame(frameCount, msg)
|
||||
if parseErr != nil {
|
||||
pw.CloseWithError(parseErr)
|
||||
return parseErr
|
||||
}
|
||||
if len(audio) > 0 {
|
||||
frameCount++
|
||||
if _, werr := pw.Write(audio); werr != nil {
|
||||
return werr
|
||||
}
|
||||
}
|
||||
if isLast {
|
||||
pw.Close()
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
if err := <-parseErrCh; err != nil && ctx.Err() == nil {
|
||||
return err
|
||||
}
|
||||
log.Printf("tts/doubao: streamed %d ogg pages for %q", frameCount, truncate(text, 20))
|
||||
return nil
|
||||
}
|
||||
|
||||
// parseTTSFrame 解析豆包 TTS WebSocket 服务端响应帧。
|
||||
//
|
||||
// 帧类型由 data[1]>>4 决定，不同类型结构不同：
|
||||
//
|
||||
// msgType=0x0B（音频帧）：[4:8]=seq int32（负=末帧）, [8:12]=载荷长度, [12:]=Ogg数据
|
||||
// msgType=其他（0x0C合成结束/0x0F错误）：[4:8]=载荷长度, [8:]=gzip JSON，无 seq 字段
|
||||
func parseTTSFrame(idx int, data []byte) (audio []byte, isLast bool, err error) {
|
||||
if len(data) < 4 {
|
||||
return nil, false, fmt.Errorf("tts/doubao: frame too short (%d bytes)", len(data))
|
||||
}
|
||||
|
||||
msgType := data[1] >> 4
|
||||
compressType := data[2] & 0x0F
|
||||
|
||||
if msgType != 0x0B {
|
||||
// 非音频帧（0x0C=合成结束，0x0F=服务端错误）：[4:8]=载荷长度，[8:]=gzip JSON
|
||||
if len(data) >= 8 {
|
||||
payloadSize := binary.BigEndian.Uint32(data[4:8])
|
||||
end := 8 + int(payloadSize)
|
||||
if end <= len(data) {
|
||||
payload := data[8:end]
|
||||
if compressType == compressGZP {
|
||||
if decoded, e := gunzipBytes(payload); e == nil {
|
||||
payload = decoded
|
||||
}
|
||||
}
|
||||
if len(payload) > 2 && payload[0] == '{' {
|
||||
var resp struct {
|
||||
Code int `json:"code"`
|
||||
}
|
||||
if json.Unmarshal(payload, &resp) == nil && resp.Code != 0 && resp.Code != 200 {
|
||||
s := string(payload)
|
||||
if len(s) > 256 {
|
||||
s = s[:256]
|
||||
}
|
||||
log.Printf("tts/doubao: frame[%d] server error: %s", idx, s)
|
||||
return nil, true, fmt.Errorf("tts/doubao: server error: %s", s)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
return nil, true, nil // 非音频帧均视为流结束信号
|
||||
}
|
||||
|
||||
// msgType=0x0B：音频帧，[4:8]=seq，[8:12]=载荷长度，[12:]=Ogg数据
|
||||
if len(data) == 8 {
|
||||
seq := int32(binary.BigEndian.Uint32(data[4:8]))
|
||||
return nil, seq < 0, nil
|
||||
}
|
||||
if len(data) < 12 {
|
||||
return nil, false, fmt.Errorf("tts/doubao: audio frame too short (%d bytes)", len(data))
|
||||
}
|
||||
seq := int32(binary.BigEndian.Uint32(data[4:8]))
|
||||
payloadSize := binary.BigEndian.Uint32(data[8:12])
|
||||
end := 12 + int(payloadSize)
|
||||
if end > len(data) {
|
||||
return nil, false, fmt.Errorf("tts/doubao: payload size %d exceeds frame length %d", payloadSize, len(data))
|
||||
}
|
||||
return data[12:end], seq < 0, nil
|
||||
}
|
||||
|
||||
// buildTTSFrame 将 JSON payload gzip 压缩后封装为豆包二进制协议帧（客户端 → 服务端）。
|
||||
func buildTTSFrame(payload any) ([]byte, error) {
|
||||
data, err := json.Marshal(payload)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
compressed, err := gzipBytes(data)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
hdr := [4]byte{
|
||||
(0x01 << 4) | 0x01,
|
||||
(msgTypeFullClientRequest << 4) | flagNormal,
|
||||
(serialJSON << 4) | compressGZP,
|
||||
0x00,
|
||||
}
|
||||
frame := make([]byte, 0, 8+len(compressed))
|
||||
frame = append(frame, hdr[:]...)
|
||||
frame = binary.BigEndian.AppendUint32(frame, uint32(len(compressed)))
|
||||
frame = append(frame, compressed...)
|
||||
return frame, nil
|
||||
}
|
||||
|
||||
func gunzipBytes(data []byte) ([]byte, error) {
|
||||
r, err := gzip.NewReader(bytes.NewReader(data))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer r.Close()
|
||||
return io.ReadAll(r)
|
||||
}
|
||||
|
||||
func gzipBytes(data []byte) ([]byte, error) {
|
||||
var buf bytes.Buffer
|
||||
w := gzip.NewWriter(&buf)
|
||||
if _, err := w.Write(data); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err := w.Close(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return buf.Bytes(), nil
|
||||
}
|
||||
|
||||
func truncate(s string, n int) string {
|
||||
runes := []rune(s)
|
||||
if len(runes) <= n {
|
||||
return s
|
||||
}
|
||||
return string(runes[:n]) + "…"
|
||||
}
|
||||
227
pkg/tts/fishspeech/provider.go
Normal file
227
pkg/tts/fishspeech/provider.go
Normal file
|
|
@ -0,0 +1,227 @@
|
|||
// PicoClaw - Ultra-lightweight personal AI agent
|
||||
// License: MIT
|
||||
//
|
||||
// Copyright (c) 2026 PicoClaw contributors
|
||||
|
||||
// Package fishspeech 对接 Fish Speech v1.5.x HTTP TTS 服务。
|
||||
// POST /v1/tts 输出 WAV（streaming=true），流式读取 PCM 帧后直接下发客户端。
|
||||
// 无需 CGO，完全纯 Go。
|
||||
//
|
||||
// 部署：
|
||||
//
|
||||
// docker compose -f docker/docker-compose.asr-tts.yml up -d fishspeech
|
||||
//
|
||||
// 配置（环境变量）：
|
||||
//
|
||||
// PICOCLAW_VOICE_TTS_API_URL 服务地址，默认 http://127.0.0.1:8080
|
||||
// PICOCLAW_VOICE_TTS_API_KEY Bearer Token，本地部署通常无需填写
|
||||
// PICOCLAW_VOICE_TTS_REFERENCE_ID 参考音色 ID（留空使用服务默认音色）
|
||||
// PICOCLAW_VOICE_TTS_SAMPLE_RATE 输出采样率，默认 44100（Fish Speech v1.5.x 默认值）
|
||||
package fishspeech
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/base64"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"log"
|
||||
"net"
|
||||
"net/http"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/sipeed/picoclaw/pkg/tts"
|
||||
)
|
||||
|
||||
func init() {
|
||||
tts.Register("fishspeech", func(cfg map[string]any) (tts.Provider, error) {
|
||||
return newProvider(cfg)
|
||||
})
|
||||
}
|
||||
|
||||
const (
|
||||
defaultAPIBase = "http://127.0.0.1:8080"
|
||||
defaultSampleRate = 44100
|
||||
// 每次回调约 46ms 的 PCM（@44100Hz mono s16le）
|
||||
pcmChunkBytes = 4096
|
||||
// 用于生成参考音频的固定短句（中性内容，音色稳定）
|
||||
refGenText = "你好，我是你的语音助手。"
|
||||
)
|
||||
|
||||
// refAudio 缓存：首次合成时用 seed 生成一段参考 WAV，后续请求带上它以固定音色。
|
||||
type refAudio struct {
|
||||
once sync.Once
|
||||
wav []byte // RIFF WAV 文件（streaming=false, format=wav）
|
||||
ready bool
|
||||
}
|
||||
|
||||
type provider struct {
|
||||
apiBase string
|
||||
apiKey string
|
||||
referenceID string
|
||||
sampleRate int
|
||||
seed int // 0 = 随机音色；非 0 = 首次合成后锁定参考音频
|
||||
ref refAudio
|
||||
client *http.Client
|
||||
}
|
||||
|
||||
func newProvider(cfg map[string]any) (*provider, error) {
|
||||
apiBase, _ := cfg["api_url"].(string)
|
||||
apiKey, _ := cfg["api_key"].(string)
|
||||
referenceID, _ := cfg["reference_id"].(string)
|
||||
if apiBase == "" {
|
||||
apiBase = defaultAPIBase
|
||||
}
|
||||
sampleRate := defaultSampleRate
|
||||
if sr, ok := cfg["sample_rate"].(int); ok && sr > 0 {
|
||||
sampleRate = sr
|
||||
}
|
||||
seed, _ := cfg["seed"].(int)
|
||||
return &provider{
|
||||
apiBase: apiBase,
|
||||
apiKey: apiKey,
|
||||
referenceID: referenceID,
|
||||
sampleRate: sampleRate,
|
||||
seed: seed,
|
||||
// 仅设置连接建立超时；response body 读取时间由 ctx 控制（streaming=true 时 body 无期限流式输出）
|
||||
client: &http.Client{
|
||||
Transport: &http.Transport{
|
||||
DialContext: (&net.Dialer{Timeout: 10 * time.Second}).DialContext,
|
||||
},
|
||||
},
|
||||
}, nil
|
||||
}
|
||||
|
||||
func (p *provider) Name() string { return "fishspeech" }
|
||||
|
||||
// AudioFormat 声明输出格式为 PCM s16le，采样率由配置指定（默认 44100Hz mono）。
|
||||
// 网关通过 tts_params 下发给客户端，客户端据此初始化播放设备，无需 Opus 解码。
|
||||
func (p *provider) AudioFormat() tts.AudioFormat {
|
||||
return tts.AudioFormat{Codec: "pcm", SampleRate: p.sampleRate, Channels: 1}
|
||||
}
|
||||
|
||||
// SynthesizeFrames 向 Fish Speech HTTP API 发请求，流式读取原始 PCM s16le 数据分块回调。
|
||||
// 若配置了 seed，首次调用会先用固定短句生成参考音频并缓存，后续每次合成都带上该参考，
|
||||
// 从而保证不同句子的音色一致。
|
||||
func (p *provider) SynthesizeFrames(ctx context.Context, text, voice string, onFrame func([]byte)) error {
|
||||
refID := voice
|
||||
if refID == "" {
|
||||
refID = p.referenceID
|
||||
}
|
||||
|
||||
// seed 非 0 且没有外部 reference_id 时，使用内部参考音频固定音色
|
||||
var refWAV []byte
|
||||
if p.seed != 0 && refID == "" {
|
||||
p.ref.once.Do(func() {
|
||||
wav, err := p.generateRefWAV()
|
||||
if err != nil {
|
||||
log.Printf("tts/fishspeech: generate ref audio: %v (voice may vary)", err)
|
||||
return
|
||||
}
|
||||
p.ref.wav = wav
|
||||
p.ref.ready = true
|
||||
})
|
||||
if p.ref.ready {
|
||||
refWAV = p.ref.wav
|
||||
}
|
||||
}
|
||||
|
||||
payload := map[string]any{
|
||||
"text": text,
|
||||
"streaming": true,
|
||||
"chunk_length": 100,
|
||||
}
|
||||
if refID != "" {
|
||||
payload["reference_id"] = refID
|
||||
} else if len(refWAV) > 0 {
|
||||
// 将缓存的参考 WAV 编码为 base64，作为 in-context speaker 固定音色
|
||||
payload["references"] = []map[string]any{
|
||||
{
|
||||
"audio": base64.StdEncoding.EncodeToString(refWAV),
|
||||
"text": refGenText,
|
||||
},
|
||||
}
|
||||
} else if p.seed != 0 {
|
||||
// 参考音频未就绪（生成失败）时退化为 seed 模式
|
||||
payload["seed"] = p.seed
|
||||
}
|
||||
|
||||
body, err := json.Marshal(payload)
|
||||
if err != nil {
|
||||
return fmt.Errorf("tts/fishspeech: marshal request: %w", err)
|
||||
}
|
||||
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodPost, p.apiBase+"/v1/tts", bytes.NewReader(body))
|
||||
if err != nil {
|
||||
return fmt.Errorf("tts/fishspeech: create request: %w", err)
|
||||
}
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
if p.apiKey != "" {
|
||||
req.Header.Set("Authorization", "Bearer "+p.apiKey)
|
||||
}
|
||||
|
||||
resp, err := p.client.Do(req)
|
||||
if err != nil {
|
||||
return fmt.Errorf("tts/fishspeech: http: %w", err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
b, _ := io.ReadAll(io.LimitReader(resp.Body, 512))
|
||||
return fmt.Errorf("tts/fishspeech: http %d: %s", resp.StatusCode, b)
|
||||
}
|
||||
|
||||
// Fish Speech streaming=true 直接返回裸 PCM s16le，无 RIFF 头
|
||||
buf := make([]byte, pcmChunkBytes)
|
||||
for {
|
||||
n, err := io.ReadFull(resp.Body, buf)
|
||||
if n > 0 {
|
||||
chunk := make([]byte, n)
|
||||
copy(chunk, buf[:n])
|
||||
onFrame(chunk)
|
||||
}
|
||||
if err == io.EOF || err == io.ErrUnexpectedEOF {
|
||||
break
|
||||
}
|
||||
if err != nil {
|
||||
if ctx.Err() != nil {
|
||||
return nil
|
||||
}
|
||||
return fmt.Errorf("tts/fishspeech: read: %w", err)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// generateRefWAV 用 refGenText + seed 生成一段参考 WAV，streaming=false 以获取完整文件。
|
||||
func (p *provider) generateRefWAV() ([]byte, error) {
|
||||
payload := map[string]any{
|
||||
"text": refGenText,
|
||||
"streaming": false,
|
||||
"format": "wav",
|
||||
"seed": p.seed,
|
||||
}
|
||||
body, _ := json.Marshal(payload)
|
||||
req, err := http.NewRequest(http.MethodPost, p.apiBase+"/v1/tts", bytes.NewReader(body))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
if p.apiKey != "" {
|
||||
req.Header.Set("Authorization", "Bearer "+p.apiKey)
|
||||
}
|
||||
resp, err := p.client.Do(req)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
b, _ := io.ReadAll(io.LimitReader(resp.Body, 256))
|
||||
return nil, fmt.Errorf("http %d: %s", resp.StatusCode, b)
|
||||
}
|
||||
return io.ReadAll(resp.Body)
|
||||
}
|
||||
|
||||
|
||||
57
pkg/tts/ogg.go
Normal file
57
pkg/tts/ogg.go
Normal file
|
|
@ -0,0 +1,57 @@
|
|||
// PicoClaw - Ultra-lightweight personal AI agent
|
||||
// License: MIT
|
||||
//
|
||||
// Copyright (c) 2026 PicoClaw contributors
|
||||
|
||||
package tts
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"fmt"
|
||||
"io"
|
||||
|
||||
"github.com/pion/opus/pkg/oggreader"
|
||||
)
|
||||
|
||||
// ParseOggOpusPackets 从 Ogg Opus 流中解析完整 Opus 包并逐个回调。
|
||||
//
|
||||
// 正确处理 Ogg lacing：pion/oggreader 的 ParseNextPage 按原始 segment（最多 255 字节）
|
||||
// 切割返回，一个 Opus 包若 > 255 字节会横跨多个 segment。本函数按 Ogg lacing 规则
|
||||
// 拼接这些 segment，确保每次 onPacket 回调都是一个完整可解码的 Opus 包。
|
||||
//
|
||||
// oggreader.NewWith 内部会消耗第一个 Ogg 页（OpusHead），因此本函数从第一个
|
||||
// ParseNextPage 开始，跳过 OpusTags 注释页后直接处理音频数据页。
|
||||
func ParseOggOpusPackets(r io.Reader, onPacket func([]byte)) error {
|
||||
ogg, _, err := oggreader.NewWith(r)
|
||||
if err != nil {
|
||||
return fmt.Errorf("tts/ogg: init reader: %w", err)
|
||||
}
|
||||
var buf []byte
|
||||
for {
|
||||
segments, _, pageErr := ogg.ParseNextPage()
|
||||
if pageErr == io.EOF {
|
||||
break
|
||||
}
|
||||
if pageErr != nil {
|
||||
return fmt.Errorf("tts/ogg: parse page: %w", pageErr)
|
||||
}
|
||||
// 跳过 OpusTags 注释头页
|
||||
if len(segments) > 0 && bytes.HasPrefix(segments[0], []byte("OpusTags")) {
|
||||
continue
|
||||
}
|
||||
// 按 Ogg lacing 规则拼接 segments 为完整 Opus 包：
|
||||
// segment 长度 == 255 表示当前包未结束；< 255 表示当前包的最后一个 segment。
|
||||
for _, seg := range segments {
|
||||
buf = append(buf, seg...)
|
||||
if len(seg) < 255 {
|
||||
if len(buf) > 0 {
|
||||
pkt := make([]byte, len(buf))
|
||||
copy(pkt, buf)
|
||||
onPacket(pkt)
|
||||
buf = buf[:0]
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
49
pkg/tts/provider.go
Normal file
49
pkg/tts/provider.go
Normal file
|
|
@ -0,0 +1,49 @@
|
|||
// PicoClaw - Ultra-lightweight personal AI agent
|
||||
// License: MIT
|
||||
//
|
||||
// Copyright (c) 2026 PicoClaw contributors
|
||||
|
||||
package tts
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
)
|
||||
|
||||
// AudioFormat 描述 TTS provider 的输出音频格式，由 provider 自身声明。
|
||||
// 网关通过 helloReply.tts_params 下发给客户端，客户端据此初始化解码器。
|
||||
type AudioFormat struct {
|
||||
Codec string // 编码格式，如 "opus"、"pcm"
|
||||
SampleRate int // 采样率，如 16000
|
||||
Channels int // 声道数，1 = 单声道
|
||||
}
|
||||
|
||||
// Provider is the TTS interface.
|
||||
// AudioFormat 由 provider 自身声明；SynthesizeFrames 输出与之对应格式的帧。
|
||||
type Provider interface {
|
||||
Name() string
|
||||
// AudioFormat 返回本 provider 的输出格式，用于向客户端协商解码参数。
|
||||
AudioFormat() AudioFormat
|
||||
// SynthesizeFrames 将文本合成为音频，每帧通过 onFrame 回调返回。
|
||||
// 帧格式由 AudioFormat() 声明；voice 为空时使用 provider 默认音色。
|
||||
SynthesizeFrames(ctx context.Context, text, voice string, onFrame func([]byte)) error
|
||||
}
|
||||
|
||||
// Factory creates a Provider from a config map.
|
||||
type Factory func(cfg map[string]any) (Provider, error)
|
||||
|
||||
var factories = map[string]Factory{}
|
||||
|
||||
// Register adds a provider factory. Called from provider init() functions.
|
||||
func Register(name string, f Factory) {
|
||||
factories[name] = f
|
||||
}
|
||||
|
||||
// New creates a Provider by name.
|
||||
func New(name string, cfg map[string]any) (Provider, error) {
|
||||
f, ok := factories[name]
|
||||
if !ok {
|
||||
return nil, fmt.Errorf("tts: provider %q not registered", name)
|
||||
}
|
||||
return f(cfg)
|
||||
}
|
||||
Loading…
Add table
Reference in a new issue