update documents
This commit is contained in:
Huaaudio
parent
c3b85c7d55
commit
3c50ab3a06
7 changed files with 621 additions and 52 deletions
|
|
@ -1,27 +1,85 @@
|
||||||
# ASR (Automatic Speech Recognition)
|
# ASR (Automatic Speech Recognition)
|
||||||
|
|
||||||
This package handles Automatic Speech Recognition (speech-to-text) capabilities.
|
This package handles speech-to-text for PicoClaw voice input.
|
||||||
|
|
||||||
## Configuration
|
If you are new to ASR setup, the simplest mental model is:
|
||||||
|
|
||||||
PicoClaw uses the unified and secure `ModelList` configuration for ASR. Instead of plain-text API keys in the `voice` configuration, you should define your ASR providers in the global `model_list` and reference them by name in the `voice` configuration section.
|
1. Add one or more ASR-capable entries to `model_list`.
|
||||||
|
2. Point `voice.model_name` at the one you want to use.
|
||||||
|
3. Put the API key in `.security.yml`.
|
||||||
|
|
||||||
To configure an ASR model, set the `model_name` under the `voice` configuration to match a defined model in your `model_list`.
|
## Quick Recommendation
|
||||||
|
|
||||||
### Example `config.json`
|
For most new users, start with one of these:
|
||||||
|
|
||||||
|
| Provider | Example model | Why start here |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| [Groq](https://console.groq.com/keys) | `groq/whisper-large-v3-turbo` | Fast Whisper-style transcription and a straightforward OpenAI-compatible API. Groq currently advertises a free tier plan for 2000 reqs/day. |
|
||||||
|
| [ElevenLabs](https://elevenlabs.io/pricing) | `elevenlabs/scribe_v1` | Easy setup and strong speech-to-text quality. ElevenLabs currently advertises a free plan that includes speech-to-text usage. |
|
||||||
|
|
||||||
|
Pricing and free-plan limits can change, so check the linked pricing pages before depending on them in production.
|
||||||
|
|
||||||
|
## How ASR Configuration Works
|
||||||
|
|
||||||
|
PicoClaw does not keep ASR API keys inside the `voice` section.
|
||||||
|
|
||||||
|
Instead:
|
||||||
|
|
||||||
|
- `voice.model_name` chooses a named entry from `model_list`.
|
||||||
|
- The matching `model_list` entry describes the actual provider and model.
|
||||||
|
- `.security.yml` stores the API key for that named model entry.
|
||||||
|
|
||||||
|
This is the recommended pattern because it is explicit, reusable, and consistent with the rest of PicoClaw's model configuration.
|
||||||
|
|
||||||
|
## Recommended Setup
|
||||||
|
|
||||||
|
### Option A: Groq Whisper
|
||||||
|
|
||||||
|
`config.json`
|
||||||
|
|
||||||
```json
|
```json
|
||||||
{
|
{
|
||||||
"voice": {
|
"voice": {
|
||||||
"model_name": "my-asr-model",
|
"model_name": "groq-asr",
|
||||||
"echo_transcription": true
|
"echo_transcription": true
|
||||||
},
|
},
|
||||||
"model_list": [
|
"model_list": [
|
||||||
{
|
{
|
||||||
"model_name": "my-asr-model",
|
"model_name": "groq-asr",
|
||||||
"model": "openai/whisper-1",
|
"model": "groq/whisper-large-v3-turbo"
|
||||||
"api_base": "https://api.openai.com/v1"
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
`.security.yml`
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
model_list:
|
||||||
|
groq-asr:
|
||||||
|
api_keys:
|
||||||
|
- "gsk_your_groq_key"
|
||||||
|
```
|
||||||
|
|
||||||
|
Notes:
|
||||||
|
|
||||||
|
- You can omit `api_base` and PicoClaw will use Groq's default API base automatically.
|
||||||
|
- If you set `api_base` manually for Groq Whisper, both of these forms work:
|
||||||
|
- `https://api.groq.com/openai/v1`
|
||||||
|
- `https://api.groq.com/openai/v1/audio/transcriptions`
|
||||||
|
- Any OpenAI-compatible Whisper model name containing `whisper` can use the Whisper transcription path, not only `whisper-large-v3-turbo`.
|
||||||
|
|
||||||
|
### Option B: ElevenLabs
|
||||||
|
|
||||||
|
`config.json`
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"voice": {
|
||||||
|
"model_name": "elevenlabs-asr",
|
||||||
|
"echo_transcription": true
|
||||||
},
|
},
|
||||||
|
"model_list": [
|
||||||
{
|
{
|
||||||
"model_name": "elevenlabs-asr",
|
"model_name": "elevenlabs-asr",
|
||||||
"model": "elevenlabs/scribe_v1"
|
"model": "elevenlabs/scribe_v1"
|
||||||
|
|
@ -30,29 +88,79 @@ To configure an ASR model, set the `model_name` under the `voice` configuration
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
### Security Configuration
|
`.security.yml`
|
||||||
|
|
||||||
API keys for the ASR model should be supplied in your `.security.yml` file matching the respective `model_name`:
|
|
||||||
|
|
||||||
```yaml
|
```yaml
|
||||||
model_list:
|
model_list:
|
||||||
my-asr-model:
|
|
||||||
api_keys:
|
|
||||||
- "sk-openai-your-key-here"
|
|
||||||
elevenlabs-asr:
|
elevenlabs-asr:
|
||||||
api_keys:
|
api_keys:
|
||||||
- "sk-elevenlabs-your-key"
|
- "sk-elevenlabs-your-key"
|
||||||
```
|
```
|
||||||
|
|
||||||
## How It Works
|
### Option C: OpenAI Whisper
|
||||||
|
|
||||||
PicoClaw's `DetectTranscriber` function will attempt to detect the appropriate Transcriber in the following order:
|
`config.json`
|
||||||
|
|
||||||
1. **Targeted Selection**: Resolve `cfg.Voice.ModelName` against `model_list`, then create the transcriber from that resolved model entry.
|
```json
|
||||||
- This means aliases such as `my-asr-model` are the primary ASR contract.
|
{
|
||||||
- If the resolved model uses `elevenlabs/...`, the ElevenLabs transcriber is initiated.
|
"voice": {
|
||||||
- If the resolved model uses an OpenAI-compatible Whisper model name such as `openai/whisper-1` or `groq/whisper-large-v3`, the Whisper transcriber is initiated.
|
"model_name": "openai-asr"
|
||||||
- If the resolved model uses an OpenAI-compatible audio-capable provider (for example `openai`, `azure`, `gemini`, `deepseek`), `AudioModelTranscriber` is leveraged.
|
},
|
||||||
2. **Fallback Scanning**: If no `model_name` is selected, PicoClaw performs a compatibility scan through `model_list` for legacy auto-detected ASR providers such as `elevenlabs/...` entries and OpenAI-compatible Whisper models.
|
"model_list": [
|
||||||
|
{
|
||||||
|
"model_name": "openai-asr",
|
||||||
|
"model": "openai/whisper-1"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
Fallback scanning exists for compatibility, but the recommended configuration is to set `voice.model_name` to a named `model_list` entry such as `my-asr-model`.
|
`.security.yml`
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
model_list:
|
||||||
|
openai-asr:
|
||||||
|
api_keys:
|
||||||
|
- "sk-openai-your-key"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Other ASR-Capable Model Types
|
||||||
|
|
||||||
|
PicoClaw currently supports three main ASR routes:
|
||||||
|
|
||||||
|
| Route | Example models | Behavior |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| ElevenLabs ASR | `elevenlabs/scribe_v1` | Uses the ElevenLabs transcription API. |
|
||||||
|
| Whisper endpoint models | `openai/whisper-1`, `groq/whisper-large-v3` | Uses an OpenAI-compatible `/audio/transcriptions` endpoint. |
|
||||||
|
| Audio-capable chat models **(Under construction)** | `openai/gpt-4o-audio-preview`, `gemini/gemini-2.5-flash` | Sends audio to a multimodal chat model and asks it to transcribe. |
|
||||||
|
|
||||||
|
If you are unsure which one to pick, choose Groq Whisper or ElevenLabs first.
|
||||||
|
|
||||||
|
## How PicoClaw Chooses a Transcriber
|
||||||
|
|
||||||
|
`DetectTranscriber` resolves ASR in this order:
|
||||||
|
|
||||||
|
1. **Preferred path**: resolve `voice.model_name` against `model_list`.
|
||||||
|
2. If that resolved model is:
|
||||||
|
- `elevenlabs/...`, PicoClaw uses the ElevenLabs transcriber.
|
||||||
|
- an OpenAI-compatible Whisper model, PicoClaw uses the Whisper transcriber.
|
||||||
|
- an audio-capable chat model, PicoClaw uses `AudioModelTranscriber`.
|
||||||
|
3. **Fallback path**: if `voice.model_name` is not set, PicoClaw performs a compatibility scan through `model_list` for legacy auto-detected ASR entries.
|
||||||
|
|
||||||
|
Fallback scanning exists for backward compatibility. New configurations should set `voice.model_name` explicitly.
|
||||||
|
|
||||||
|
## Common Mistakes
|
||||||
|
|
||||||
|
- Defining an ASR model in `model_list` but forgetting to set `voice.model_name`.
|
||||||
|
- Putting the API key in `voice` instead of `.security.yml`.
|
||||||
|
- Using a non-ASR model and expecting Whisper-style transcription behavior.
|
||||||
|
- Setting a custom `api_base` that points to the wrong provider endpoint.
|
||||||
|
|
||||||
|
## Minimal Checklist
|
||||||
|
|
||||||
|
Before testing voice input, make sure:
|
||||||
|
|
||||||
|
- `voice.model_name` matches a `model_list[].model_name`.
|
||||||
|
- The matching `.security.yml` entry contains a valid API key.
|
||||||
|
- The selected model is actually ASR-capable.
|
||||||
|
- Voice input is enabled for the channel you are using.
|
||||||
|
|
|
||||||
166
pkg/audio/asr/README_zh.md
Normal file
166
pkg/audio/asr/README_zh.md
Normal file
|
|
@ -0,0 +1,166 @@
|
||||||
|
# ASR（自动语音识别）
|
||||||
|
|
||||||
|
这个目录负责 PicoClaw 的语音转文字能力。
|
||||||
|
|
||||||
|
如果你是第一次配置 ASR，可以参考如下步骤：
|
||||||
|
|
||||||
|
1. 在 `model_list` 里添加一个或多个支持 ASR 的模型条目。
|
||||||
|
2. 用 `voice.model_name` 指向你想使用的那个条目。
|
||||||
|
3. 在 `.security.yml` 里配置对应的 API Key。
|
||||||
|
|
||||||
|
## 快速推荐
|
||||||
|
|
||||||
|
对于大多数新用户，建议先从下面两种开始：
|
||||||
|
|
||||||
|
| 提供商 | 示例模型 | 推荐理由 |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| [Groq](https://console.groq.com/keys) | `groq/whisper-large-v3-turbo` | Whisper 风格转录速度快，并且提供 OpenAI 兼容接口，配置比较直接。Groq 目前官方提供2000请求每日的免费套餐。 |
|
||||||
|
| [ElevenLabs](https://elevenlabs.io/pricing) | `elevenlabs/scribe_v1` | 上手简单，语音转文字质量也不错。ElevenLabs 目前官方免费套餐包含 STT 用量。 |
|
||||||
|
|
||||||
|
价格和免费额度可能会变化，正式使用前请以官网定价页为准。
|
||||||
|
|
||||||
|
## ASR 配置是如何工作的
|
||||||
|
|
||||||
|
PicoClaw 不会把 ASR 的 API Key 放在 `voice` 配置里。
|
||||||
|
|
||||||
|
推荐的方式是：
|
||||||
|
|
||||||
|
- `voice.model_name` 用来选择 `model_list` 里的某个命名模型。
|
||||||
|
- `model_list` 条目描述真实的提供商和模型。
|
||||||
|
- `.security.yml` 负责保存该模型条目的 API Key。
|
||||||
|
|
||||||
|
这种方式更明确、更安全，也和 PicoClaw 其他模型配置方式保持一致。
|
||||||
|
|
||||||
|
## 推荐配置方式
|
||||||
|
|
||||||
|
### 方案 A：Groq Whisper
|
||||||
|
|
||||||
|
`config.json`
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"voice": {
|
||||||
|
"model_name": "groq-asr",
|
||||||
|
"echo_transcription": true
|
||||||
|
},
|
||||||
|
"model_list": [
|
||||||
|
{
|
||||||
|
"model_name": "groq-asr",
|
||||||
|
"model": "groq/whisper-large-v3-turbo"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
`.security.yml`
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
model_list:
|
||||||
|
groq-asr:
|
||||||
|
api_keys:
|
||||||
|
- "gsk_your_groq_key"
|
||||||
|
```
|
||||||
|
|
||||||
|
说明：
|
||||||
|
|
||||||
|
- 你可以不写 `api_base`，PicoClaw 会自动使用 Groq 默认接口地址。
|
||||||
|
- 如果你手动设置 Groq Whisper 的 `api_base`，下面两种写法都可以：
|
||||||
|
- `https://api.groq.com/openai/v1`
|
||||||
|
- `https://api.groq.com/openai/v1/audio/transcriptions`
|
||||||
|
- 只要是 OpenAI 兼容、并且模型名里包含 `whisper` 的模型，都可以走 Whisper 转录路径，不仅限于 `whisper-large-v3-turbo`。
|
||||||
|
|
||||||
|
### 方案 B：ElevenLabs
|
||||||
|
|
||||||
|
`config.json`
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"voice": {
|
||||||
|
"model_name": "elevenlabs-asr",
|
||||||
|
"echo_transcription": true
|
||||||
|
},
|
||||||
|
"model_list": [
|
||||||
|
{
|
||||||
|
"model_name": "elevenlabs-asr",
|
||||||
|
"model": "elevenlabs/scribe_v1"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
`.security.yml`
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
model_list:
|
||||||
|
elevenlabs-asr:
|
||||||
|
api_keys:
|
||||||
|
- "sk-elevenlabs-your-key"
|
||||||
|
```
|
||||||
|
|
||||||
|
### 方案 C：OpenAI Whisper
|
||||||
|
|
||||||
|
`config.json`
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"voice": {
|
||||||
|
"model_name": "openai-asr"
|
||||||
|
},
|
||||||
|
"model_list": [
|
||||||
|
{
|
||||||
|
"model_name": "openai-asr",
|
||||||
|
"model": "openai/whisper-1"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
`.security.yml`
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
model_list:
|
||||||
|
openai-asr:
|
||||||
|
api_keys:
|
||||||
|
- "sk-openai-your-key"
|
||||||
|
```
|
||||||
|
|
||||||
|
## 其他支持 ASR 的模型类型
|
||||||
|
|
||||||
|
PicoClaw 目前主要支持三种 ASR 路径：
|
||||||
|
|
||||||
|
| 路径 | 示例模型 | 行为说明 |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| ElevenLabs ASR | `elevenlabs/scribe_v1` | 使用 ElevenLabs 的语音转录接口。 |
|
||||||
|
| Whisper 接口模型 | `openai/whisper-1`、`groq/whisper-large-v3` | 使用 OpenAI 兼容的 `/audio/transcriptions` 接口。 |
|
||||||
|
| 支持音频的聊天模型 **（重构中）** | `openai/gpt-4o-audio-preview`、`gemini/gemini-2.5-flash` | 把音频发给多模态聊天模型，并要求它返回转录结果。 |
|
||||||
|
|
||||||
|
如果你不确定该选哪种，建议优先使用 Groq Whisper 或 ElevenLabs。
|
||||||
|
|
||||||
|
## PicoClaw 如何选择转录器
|
||||||
|
|
||||||
|
`DetectTranscriber` 会按下面顺序选择 ASR：
|
||||||
|
|
||||||
|
1. **首选路径**：根据 `voice.model_name` 在 `model_list` 中找到对应模型。
|
||||||
|
2. 如果找到的模型属于以下类型：
|
||||||
|
- `elevenlabs/...`，则使用 ElevenLabs transcriber。
|
||||||
|
- OpenAI 兼容的 Whisper 模型，则使用 Whisper transcriber。
|
||||||
|
- 支持音频输入的聊天模型，则使用 `AudioModelTranscriber`。
|
||||||
|
3. **回退路径**：如果没有设置 `voice.model_name`，PicoClaw 会为了兼容旧配置，扫描 `model_list` 中可自动识别的 ASR 条目。
|
||||||
|
|
||||||
|
回退扫描只是为了兼容旧行为。新配置建议始终显式设置 `voice.model_name`。
|
||||||
|
|
||||||
|
## 常见错误
|
||||||
|
|
||||||
|
- 在 `model_list` 里定义了 ASR 模型，但忘了设置 `voice.model_name`。
|
||||||
|
- 把 API Key 写进了 `voice`，而不是 `.security.yml`。
|
||||||
|
- 选择了不支持 ASR 的模型，却期望得到 Whisper 风格的转录结果。
|
||||||
|
- 自定义了错误的 `api_base`，导致请求打到错误的接口地址。
|
||||||
|
|
||||||
|
## 最小检查清单
|
||||||
|
|
||||||
|
在测试语音输入前，请确认：
|
||||||
|
|
||||||
|
- `voice.model_name` 能正确匹配某个 `model_list[].model_name`。
|
||||||
|
- `.security.yml` 中对应条目已经配置了有效 API Key。
|
||||||
|
- 你选择的模型确实支持 ASR。
|
||||||
|
- 你当前使用的频道已经启用了语音输入能力。
|
||||||
|
|
@ -1,46 +1,137 @@
|
||||||
# TTS (Text-to-Speech)
|
# TTS (Text-to-Speech)
|
||||||
|
|
||||||
This package handles Text-to-Speech (speech synthesis) capabilities.
|
This package handles speech synthesis for PicoClaw.
|
||||||
|
|
||||||
## Configuration
|
If you are new to TTS setup, the simplest workflow is:
|
||||||
|
|
||||||
PicoClaw uses the unified and secure `ModelList` configuration for TTS. Plain-text API keys are no longer tolerated in the `voice` config block directly.
|
1. Add a TTS-capable entry to `model_list`.
|
||||||
|
2. Point `voice.tts_model_name` at that entry.
|
||||||
|
3. Put the API key in `.security.yml`.
|
||||||
|
|
||||||
To configure a TTS model, define it in your `model_list`, and set it in your `voice` configuration block using the `tts_model_name` field.
|
## Quick Recommendation
|
||||||
|
|
||||||
### Example `config.json`
|
For most users, these are the best starting points:
|
||||||
|
|
||||||
|
| Provider | Why start here |
|
||||||
|
| --- | --- |
|
||||||
|
| [OpenAI](https://platform.openai.com/docs/guides/text-to-speech) | Best-supported path in PicoClaw today. The current TTS implementation is built around the OpenAI-compatible `/audio/speech` API shape, and OpenAI is the safest default. |
|
||||||
|
| [Xiaomi MiMo](https://platform.xiaomimimo.com) | A good second option if you want an OpenAI-compatible provider endpoint and are already using MiMo models in the rest of your stack. |
|
||||||
|
|
||||||
|
## How TTS Configuration Works
|
||||||
|
|
||||||
|
PicoClaw does not keep TTS API keys inside `voice`.
|
||||||
|
|
||||||
|
Instead:
|
||||||
|
|
||||||
|
- `voice.tts_model_name` selects a named entry from `model_list`.
|
||||||
|
- That `model_list` entry provides the provider, model ID, API base, and proxy settings.
|
||||||
|
- `.security.yml` stores the API key for the same named model entry.
|
||||||
|
|
||||||
|
This is the recommended and supported configuration pattern.
|
||||||
|
|
||||||
|
## Recommended Setup
|
||||||
|
|
||||||
|
### Option A: OpenAI
|
||||||
|
|
||||||
|
`config.json`
|
||||||
|
|
||||||
```json
|
```json
|
||||||
{
|
{
|
||||||
"voice": {
|
"voice": {
|
||||||
"tts_model_name": "my-tts-model"
|
"tts_model_name": "openai-tts"
|
||||||
},
|
},
|
||||||
"model_list": [
|
"model_list": [
|
||||||
{
|
{
|
||||||
"model_name": "my-tts-model",
|
"model_name": "openai-tts",
|
||||||
"model": "openai/tts-1",
|
"model": "openai/tts-1"
|
||||||
"api_base": "https://api.openai.com/v1"
|
|
||||||
}
|
}
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
### Security Configuration
|
`.security.yml`
|
||||||
|
|
||||||
API keys for your TTS model are managed securely with standard `model_list` entries in `.security.yml`:
|
|
||||||
|
|
||||||
```yaml
|
```yaml
|
||||||
model_list:
|
model_list:
|
||||||
my-tts-model:
|
openai-tts:
|
||||||
api_keys:
|
api_keys:
|
||||||
- "sk-openai-your-key-here"
|
- "sk-openai-your-key"
|
||||||
```
|
```
|
||||||
|
|
||||||
## How It Works
|
### Option B: Xiaomi MiMo
|
||||||
|
|
||||||
PicoClaw's `DetectTTS` function resolves the TTS Provider efficiently using the secure definitions:
|
`config.json`
|
||||||
|
|
||||||
1. **Targeted Selection**: It will resolve the TTS Provider strictly via the `tts_model_name` configured in the `voice` block to pluck the respective model instance, base URL, keys, and proxy details.
|
```json
|
||||||
2. **Fallback Scanning**: If no explicit `tts_model_name` is set (or missing), PicoClaw will scan the `model_list` for any entry whose model structure explicitly contains the word `tts` and possesses a valid API key.
|
{
|
||||||
|
"voice": {
|
||||||
|
"tts_model_name": "mimo-tts"
|
||||||
|
},
|
||||||
|
"model_list": [
|
||||||
|
{
|
||||||
|
"model_name": "mimo-tts",
|
||||||
|
"model": "mimo/mimo-v2-tts"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
Most standard TTS routing passes through `OpenAITTSProvider`, which acts universally for OpenAI-compatible audio speech synthesis API formats.
|
`.security.yml`
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
model_list:
|
||||||
|
mimo-tts:
|
||||||
|
api_keys:
|
||||||
|
- "your-mimo-key"
|
||||||
|
```
|
||||||
|
|
||||||
|
If you use a custom MiMo endpoint, you can also set `api_base` explicitly. Otherwise PicoClaw will use the provider default.
|
||||||
|
|
||||||
|
## What PicoClaw Sends Today
|
||||||
|
|
||||||
|
The current TTS runtime uses an OpenAI-compatible speech request with these defaults:
|
||||||
|
|
||||||
|
- Endpoint: `/audio/speech`
|
||||||
|
- Response format: `opus`
|
||||||
|
- Voice: `alloy`
|
||||||
|
- Model: taken from the selected `model_list` entry
|
||||||
|
|
||||||
|
That means:
|
||||||
|
|
||||||
|
- `openai/tts-1` works naturally.
|
||||||
|
- Other OpenAI-compatible providers can work if they accept the same request format.
|
||||||
|
- PicoClaw currently does not expose a user-facing config field for changing the TTS voice from `alloy`.
|
||||||
|
|
||||||
|
## How PicoClaw Chooses a TTS Provider
|
||||||
|
|
||||||
|
`DetectTTS` resolves TTS in this order:
|
||||||
|
|
||||||
|
1. **Preferred path**: resolve `voice.tts_model_name` against `model_list`.
|
||||||
|
2. If a matching model entry exists and has an API key, PicoClaw creates an OpenAI-compatible TTS provider using that model's settings.
|
||||||
|
3. **Fallback path**: if `voice.tts_model_name` is not set or cannot be resolved, PicoClaw scans `model_list` for the first entry whose model string contains `tts` and has an API key.
|
||||||
|
|
||||||
|
Fallback scanning exists for compatibility. New configs should set `voice.tts_model_name` explicitly.
|
||||||
|
|
||||||
|
## Notes About API Base Handling
|
||||||
|
|
||||||
|
PicoClaw normalizes the configured base URL for TTS:
|
||||||
|
|
||||||
|
- For OpenAI, a base like `https://api.openai.com` or `https://api.openai.com/v1` becomes `https://api.openai.com/v1/audio/speech`.
|
||||||
|
- For other OpenAI-compatible providers, PicoClaw preserves the configured base path and ensures it ends with `/audio/speech`.
|
||||||
|
- If `api_base` is omitted, PicoClaw uses the provider default base when the model prefix is known.
|
||||||
|
|
||||||
|
## Common Mistakes
|
||||||
|
|
||||||
|
- Setting `voice.tts_model_name` to a name that does not exist in `model_list`.
|
||||||
|
- Adding a TTS model but forgetting to put its API key in `.security.yml`.
|
||||||
|
- Assuming PicoClaw will automatically use provider-specific custom voices.
|
||||||
|
- Using a provider endpoint that is not compatible with the OpenAI `/audio/speech` request format.
|
||||||
|
|
||||||
|
## Minimal Checklist
|
||||||
|
|
||||||
|
Before testing `send_tts`, make sure:
|
||||||
|
|
||||||
|
- `voice.tts_model_name` matches a `model_list[].model_name`.
|
||||||
|
- The matching `.security.yml` entry contains a valid API key.
|
||||||
|
- The chosen provider supports an OpenAI-compatible speech synthesis endpoint.
|
||||||
|
- Your selected model is actually a TTS-capable model.
|
||||||
|
|
|
||||||
137
pkg/audio/tts/README_zh.md
Normal file
137
pkg/audio/tts/README_zh.md
Normal file
|
|
@ -0,0 +1,137 @@
|
||||||
|
# TTS（文本转语音）
|
||||||
|
|
||||||
|
这个目录负责 PicoClaw 的语音合成能力。
|
||||||
|
|
||||||
|
如果你是第一次配置 TTS，可以参照下面这个流程：
|
||||||
|
|
||||||
|
1. 在 `model_list` 里添加一个支持 TTS 的模型。
|
||||||
|
2. 用 `voice.tts_model_name` 指向这个模型。
|
||||||
|
3. 在 `.security.yml` 里配置对应的 API Key。
|
||||||
|
|
||||||
|
## 快速推荐
|
||||||
|
|
||||||
|
对于大多数用户，建议优先从下面两种开始：
|
||||||
|
|
||||||
|
| 提供商 | 推荐理由 |
|
||||||
|
| --- | --- |
|
||||||
|
| [OpenAI](https://platform.openai.com/docs/guides/text-to-speech) | 这是 PicoClaw 当前最稳定、最直接的 TTS 路径。当前实现就是围绕 OpenAI 兼容的 `/audio/speech` 接口格式构建的，所以 OpenAI 是最稳妥的默认选择。 |
|
||||||
|
| [Xiaomi MiMo](https://platform.xiaomimimo.com) | 由于响应速度和语音音色对于中国用户更友好，MiMo 是一个不错的第二选择。 |
|
||||||
|
|
||||||
|
## TTS 配置是如何工作的
|
||||||
|
|
||||||
|
PicoClaw 不会把 TTS 的 API Key 放在 `voice` 配置里。
|
||||||
|
|
||||||
|
推荐方式是：
|
||||||
|
|
||||||
|
- `voice.tts_model_name` 用来选择 `model_list` 里的某个命名模型。
|
||||||
|
- 对应的 `model_list` 条目提供真实的 provider、model ID、`api_base` 和代理配置。
|
||||||
|
- `.security.yml` 负责保存该模型条目的 API Key。
|
||||||
|
|
||||||
|
这是当前推荐且受支持的配置方式。
|
||||||
|
|
||||||
|
## 推荐配置方式
|
||||||
|
|
||||||
|
### 方案 A：OpenAI
|
||||||
|
|
||||||
|
`config.json`
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"voice": {
|
||||||
|
"tts_model_name": "openai-tts"
|
||||||
|
},
|
||||||
|
"model_list": [
|
||||||
|
{
|
||||||
|
"model_name": "openai-tts",
|
||||||
|
"model": "openai/tts-1"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
`.security.yml`
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
model_list:
|
||||||
|
openai-tts:
|
||||||
|
api_keys:
|
||||||
|
- "sk-openai-your-key"
|
||||||
|
```
|
||||||
|
|
||||||
|
### 方案 B：Xiaomi MiMo
|
||||||
|
|
||||||
|
`config.json`
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"voice": {
|
||||||
|
"tts_model_name": "mimo-tts"
|
||||||
|
},
|
||||||
|
"model_list": [
|
||||||
|
{
|
||||||
|
"model_name": "mimo-tts",
|
||||||
|
"model": "mimo/mimo-v2-tts"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
`.security.yml`
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
model_list:
|
||||||
|
mimo-tts:
|
||||||
|
api_keys:
|
||||||
|
- "your-mimo-key"
|
||||||
|
```
|
||||||
|
|
||||||
|
如果你使用自定义的 MiMo 接口地址，也可以显式设置 `api_base`。如果不设置，PicoClaw 会自动使用该 provider 的默认地址。
|
||||||
|
|
||||||
|
## PicoClaw 当前实际发送的 TTS 请求
|
||||||
|
|
||||||
|
当前 TTS 运行时使用的是 OpenAI 兼容的语音合成请求，并带有以下默认值：
|
||||||
|
|
||||||
|
- Endpoint：`/audio/speech`
|
||||||
|
- 返回格式：`opus`
|
||||||
|
- Voice：`alloy`
|
||||||
|
- Model：来自你所选中的 `model_list` 条目
|
||||||
|
|
||||||
|
这意味着：
|
||||||
|
|
||||||
|
- `openai/tts-1` 可以自然工作。
|
||||||
|
- 其他 OpenAI 兼容 provider 也可能可用，前提是它们接受相同的请求格式。
|
||||||
|
- PicoClaw 目前还没有对用户暴露一个配置项来修改 TTS voice，当前固定为 `alloy`。
|
||||||
|
|
||||||
|
## PicoClaw 如何选择 TTS Provider
|
||||||
|
|
||||||
|
`DetectTTS` 会按下面顺序选择 TTS：
|
||||||
|
|
||||||
|
1. **首选路径**：根据 `voice.tts_model_name` 在 `model_list` 中找到对应模型。
|
||||||
|
2. 如果找到了匹配条目，并且它有 API Key，PicoClaw 就会使用这个模型条目的配置创建一个 OpenAI 兼容的 TTS provider。
|
||||||
|
3. **回退路径**：如果没有设置 `voice.tts_model_name`，或者该名字无法解析，PicoClaw 会扫描 `model_list`，选中第一个模型字符串里包含 `tts` 且带有 API Key 的条目。
|
||||||
|
|
||||||
|
回退扫描只是为了兼容旧行为。新配置建议始终显式设置 `voice.tts_model_name`。
|
||||||
|
|
||||||
|
## 关于 API Base 的处理方式
|
||||||
|
|
||||||
|
PicoClaw 会对 TTS 的 `api_base` 做规范化处理：
|
||||||
|
|
||||||
|
- 对 OpenAI 来说，像 `https://api.openai.com` 或 `https://api.openai.com/v1` 这样的地址，会自动变成 `https://api.openai.com/v1/audio/speech`。
|
||||||
|
- 对其他 OpenAI 兼容 provider，PicoClaw 会尽量保留你提供的基础路径，只确保它最终以 `/audio/speech` 结尾。
|
||||||
|
- 如果没有设置 `api_base`，并且模型前缀是已知 provider，PicoClaw 会自动使用该 provider 的默认地址。
|
||||||
|
|
||||||
|
## 常见错误
|
||||||
|
|
||||||
|
- `voice.tts_model_name` 指向了一个不存在的 `model_list` 名称。
|
||||||
|
- 在 `model_list` 里定义了 TTS 模型，但忘了在 `.security.yml` 中配置对应 API Key。
|
||||||
|
- 误以为 PicoClaw 会自动支持 provider 自定义 voice 参数。
|
||||||
|
- 使用了不兼容 OpenAI `/audio/speech` 请求格式的接口地址。
|
||||||
|
|
||||||
|
## 最小检查清单
|
||||||
|
|
||||||
|
在测试 `send_tts` 之前，请确认：
|
||||||
|
|
||||||
|
- `voice.tts_model_name` 能正确匹配某个 `model_list[].model_name`。
|
||||||
|
- `.security.yml` 中对应条目已经配置了有效 API Key。
|
||||||
|
- 你所选的 provider 支持 OpenAI 兼容的语音合成接口。
|
||||||
|
- 你选择的模型本身确实支持 TTS。
|
||||||
|
|
@ -23,7 +23,7 @@ type OpenAITTSProvider struct {
|
||||||
httpClient *http.Client
|
httpClient *http.Client
|
||||||
}
|
}
|
||||||
|
|
||||||
func NewOpenAITTSProvider(apiKey string, apiBase string, proxyURL string) *OpenAITTSProvider {
|
func NewOpenAITTSProvider(apiKey string, apiBase string, proxyURL string, model string) *OpenAITTSProvider {
|
||||||
// Normalize apiBase to avoid malformed endpoints like
|
// Normalize apiBase to avoid malformed endpoints like
|
||||||
// "https://api.openai.com/audio/speech" when "/v1" is required.
|
// "https://api.openai.com/audio/speech" when "/v1" is required.
|
||||||
if apiBase == "" {
|
if apiBase == "" {
|
||||||
|
|
@ -70,11 +70,16 @@ func NewOpenAITTSProvider(apiKey string, apiBase string, proxyURL string) *OpenA
|
||||||
client := common.NewHTTPClient(proxyURL)
|
client := common.NewHTTPClient(proxyURL)
|
||||||
client.Timeout = 60 * time.Second
|
client.Timeout = 60 * time.Second
|
||||||
|
|
||||||
|
model = strings.TrimSpace(model)
|
||||||
|
if model == "" {
|
||||||
|
model = "tts-1"
|
||||||
|
}
|
||||||
|
|
||||||
return &OpenAITTSProvider{
|
return &OpenAITTSProvider{
|
||||||
apiKey: apiKey,
|
apiKey: apiKey,
|
||||||
apiBase: apiBase,
|
apiBase: apiBase,
|
||||||
voice: "alloy",
|
voice: "alloy",
|
||||||
model: "tts-1",
|
model: model,
|
||||||
httpClient: client,
|
httpClient: client,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -11,6 +11,7 @@ import (
|
||||||
|
|
||||||
"github.com/sipeed/picoclaw/pkg/config"
|
"github.com/sipeed/picoclaw/pkg/config"
|
||||||
"github.com/sipeed/picoclaw/pkg/media"
|
"github.com/sipeed/picoclaw/pkg/media"
|
||||||
|
"github.com/sipeed/picoclaw/pkg/providers"
|
||||||
)
|
)
|
||||||
|
|
||||||
type TTSProvider interface {
|
type TTSProvider interface {
|
||||||
|
|
@ -18,16 +19,37 @@ type TTSProvider interface {
|
||||||
Synthesize(ctx context.Context, text string) (io.ReadCloser, error)
|
Synthesize(ctx context.Context, text string) (io.ReadCloser, error)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func providerFromModelConfig(mc *config.ModelConfig) TTSProvider {
|
||||||
|
if mc == nil || mc.APIKey() == "" {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
_, modelID := providers.ExtractProtocol(mc.Model)
|
||||||
|
if modelID == "" {
|
||||||
|
modelID = strings.TrimSpace(mc.Model)
|
||||||
|
}
|
||||||
|
|
||||||
|
return NewOpenAITTSProvider(mc.APIKey(), providers.ResolveAPIBase(mc), mc.Proxy, modelID)
|
||||||
|
}
|
||||||
|
|
||||||
func DetectTTS(cfg *config.Config) TTSProvider {
|
func DetectTTS(cfg *config.Config) TTSProvider {
|
||||||
|
if cfg == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
if modelName := strings.TrimSpace(cfg.Voice.TTSModelName); modelName != "" {
|
if modelName := strings.TrimSpace(cfg.Voice.TTSModelName); modelName != "" {
|
||||||
if mc, err := cfg.GetModelConfig(modelName); err == nil && mc.APIKey() != "" {
|
if mc, err := cfg.GetModelConfig(modelName); err == nil {
|
||||||
return NewOpenAITTSProvider(mc.APIKey(), mc.APIBase, mc.Proxy)
|
if provider := providerFromModelConfig(mc); provider != nil {
|
||||||
|
return provider
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
for _, mc := range cfg.ModelList {
|
for _, mc := range cfg.ModelList {
|
||||||
if strings.Contains(strings.ToLower(mc.Model), "tts") && mc.APIKey() != "" {
|
if strings.Contains(strings.ToLower(mc.Model), "tts") && mc.APIKey() != "" {
|
||||||
return NewOpenAITTSProvider(mc.APIKey(), mc.APIBase, mc.Proxy)
|
if provider := providerFromModelConfig(mc); provider != nil {
|
||||||
|
return provider
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
return nil
|
return nil
|
||||||
|
|
|
||||||
|
|
@ -8,6 +8,8 @@ import (
|
||||||
"net/http/httptest"
|
"net/http/httptest"
|
||||||
"strings"
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
|
|
||||||
|
"github.com/sipeed/picoclaw/pkg/config"
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestNewOpenAITTSProvider_APIBaseNormalization(t *testing.T) {
|
func TestNewOpenAITTSProvider_APIBaseNormalization(t *testing.T) {
|
||||||
|
|
@ -48,7 +50,7 @@ func TestNewOpenAITTSProvider_APIBaseNormalization(t *testing.T) {
|
||||||
for _, tc := range cases {
|
for _, tc := range cases {
|
||||||
t.Run(tc.name, func(t *testing.T) {
|
t.Run(tc.name, func(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
provider := NewOpenAITTSProvider("key", tc.input, "")
|
provider := NewOpenAITTSProvider("key", tc.input, "", "")
|
||||||
if provider.apiBase != tc.expect {
|
if provider.apiBase != tc.expect {
|
||||||
t.Fatalf("apiBase mismatch: got %q, want %q", provider.apiBase, tc.expect)
|
t.Fatalf("apiBase mismatch: got %q, want %q", provider.apiBase, tc.expect)
|
||||||
}
|
}
|
||||||
|
|
@ -78,7 +80,7 @@ func TestOpenAITTSProvider_SynthesizeSuccess(t *testing.T) {
|
||||||
}))
|
}))
|
||||||
defer server.Close()
|
defer server.Close()
|
||||||
|
|
||||||
provider := NewOpenAITTSProvider("k123", server.URL, "")
|
provider := NewOpenAITTSProvider("k123", server.URL, "", "")
|
||||||
stream, err := provider.Synthesize(context.Background(), "hello")
|
stream, err := provider.Synthesize(context.Background(), "hello")
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("Synthesize failed: %v", err)
|
t.Fatalf("Synthesize failed: %v", err)
|
||||||
|
|
@ -118,7 +120,7 @@ func TestOpenAITTSProvider_SynthesizeNon200(t *testing.T) {
|
||||||
}))
|
}))
|
||||||
defer server.Close()
|
defer server.Close()
|
||||||
|
|
||||||
provider := NewOpenAITTSProvider("k123", server.URL, "")
|
provider := NewOpenAITTSProvider("k123", server.URL, "", "")
|
||||||
_, err := provider.Synthesize(context.Background(), "hello")
|
_, err := provider.Synthesize(context.Background(), "hello")
|
||||||
if err == nil {
|
if err == nil {
|
||||||
t.Fatal("expected error")
|
t.Fatal("expected error")
|
||||||
|
|
@ -127,3 +129,41 @@ func TestOpenAITTSProvider_SynthesizeNon200(t *testing.T) {
|
||||||
t.Fatalf("unexpected error: %v", err)
|
t.Fatalf("unexpected error: %v", err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestNewOpenAITTSProvider_UsesConfiguredModel(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
provider := NewOpenAITTSProvider("key", "https://api.xiaomimimo.com/v1", "", "mimo-v2-tts")
|
||||||
|
if provider.model != "mimo-v2-tts" {
|
||||||
|
t.Fatalf("model mismatch: got %q, want %q", provider.model, "mimo-v2-tts")
|
||||||
|
}
|
||||||
|
if provider.apiBase != "https://api.xiaomimimo.com/v1/audio/speech" {
|
||||||
|
t.Fatalf("apiBase mismatch: got %q", provider.apiBase)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestDetectTTS_UsesConfiguredModelAndProviderBase(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
provider := DetectTTS(&config.Config{
|
||||||
|
Voice: config.VoiceConfig{TTSModelName: "mimo-tts"},
|
||||||
|
ModelList: []*config.ModelConfig{
|
||||||
|
{
|
||||||
|
ModelName: "mimo-tts",
|
||||||
|
Model: "mimo/mimo-v2-tts",
|
||||||
|
APIKeys: config.SimpleSecureStrings("sk-mimo"),
|
||||||
|
},
|
||||||
|
},
|
||||||
|
})
|
||||||
|
|
||||||
|
ttsProvider, ok := provider.(*OpenAITTSProvider)
|
||||||
|
if !ok {
|
||||||
|
t.Fatalf("DetectTTS() type = %T, want *OpenAITTSProvider", provider)
|
||||||
|
}
|
||||||
|
if ttsProvider.model != "mimo-v2-tts" {
|
||||||
|
t.Fatalf("model mismatch: got %q, want %q", ttsProvider.model, "mimo-v2-tts")
|
||||||
|
}
|
||||||
|
if ttsProvider.apiBase != "https://api.xiaomimimo.com/v1/audio/speech" {
|
||||||
|
t.Fatalf("apiBase mismatch: got %q", ttsProvider.apiBase)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
|
||||||
Loading…
Add table
Reference in a new issue