zyc/video-flow-toon
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge branch '108' of https://github.com/HBAI-Ltd/Toonflow-app into 108

Browse Source 
# Conflicts:
#	src/lib/initDB.ts
This commit is contained in:
zhishi 2026-03-19 20:05:58 +08:00
commit 06b6569c8e
16 changed files with 661 additions and 154 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

36
data/skills/production-agent/SKILL.md Normal file
View File

@ -0,0 +1,36 @@
---
name: production-agent
description: 短剧漫剧制作助手。协助用户进行剧本创作、分镜设计、角色设定、场景描述、对白润色及制作流程规划。当用户提及剧本、分镜、角色、场景、对白、短剧、漫剧等关键词时使用此技能。
---
# Production Agent
短剧漫剧制作专用技能，提供剧本创作、分镜设计、角色设定等全流程指导。
## 何时使用
当用户需要以下帮助时激活此技能：
- 剧本创作与优化
- 分镜脚本设计
- 角色设定与描述
- 场景构建与描绘
- 对白润色与调整
- 制作流程规划
## 工作指引
1. 理解用户的创作意图，根据项目类型（短剧/漫剧）调整输出风格
2. 遵循标准的剧本格式，包含场景描述、角色动作、对白等要素
3. 保持角色一致性，关注剧情连贯性
4. 输出使用中文
## 参考资料
本技能附带以下参考资料，根据任务需要使用 `read_skill_file` 工具按需加载：
- [剧本格式规范](references/script-format.md) — 场景描述、对白、旁白、转场等标准写法
- [角色设定模板](references/character-template.md) — 基础信息、性格特点、背景设定、角色弧光
- [分镜设计指南](references/storyboard-guide.md) — 景别分类、常用构图、镜头运动、分镜描述模板
**注意**：根据用户当前任务选择性加载对应参考资料，不要一次性全部加载。

67
data/skills/production-agent/references/character-template.md Normal file
View File

@ -0,0 +1,67 @@
# 角色设定模板
## 基础信息
- **姓名**：
- **年龄**：
- **性别**：
- **职业**：
- **外貌特征**：身高、体型、发型、标志性特征
## 性格特点
- **核心性格**：用 3-5 个关键词概括
- **优点**：
- **缺点**：
- **口头禅**：
- **说话风格**：正式/随意/文艺/粗犷
## 人物背景
- **家庭背景**：
- **成长经历**：
- **关键转折事件**：
## 人物关系
| 角色 | 关系 | 互动方式 |
|------|------|----------|
| | | |
## 角色弧光
- **起点状态**：故事开始时的状态
- **核心冲突**：角色面临的主要矛盾
- **转变契机**：什么事件触发角色转变
- **终点状态**：故事结束时的状态
## 填写示例
### 基础信息
- **姓名**：林晓
- **年龄**：26 岁
- **性别**：女
- **职业**：自由插画师
- **外貌特征**：165cm，偏瘦，黑色短发微卷，左耳戴银色耳钉
### 性格特点
- **核心性格**：敏感、倔强、善良
- **优点**：对艺术有极强的感知力，重情义
- **缺点**：逃避冲突，容易钻牛角尖
- **口头禅**："算了吧"
- **说话风格**：随意，偶尔会冒出文艺的比喻
### 人物背景
- **家庭背景**：单亲家庭，母亲独自经营花店
- **成长经历**：从小在花店长大，高中获美术竞赛一等奖后决定走艺术道路
- **关键转折事件**：大学毕业作品展遭导师否定后退学
### 角色弧光
- **起点状态**：对自己的能力缺乏信心，靠接零散商稿维生
- **核心冲突**：内心的艺术追求与现实生存压力
- **转变契机**：一位老画家看中她的画作并邀请合作
- **终点状态**：重新找回创作的勇气，举办个人画展

61
data/skills/production-agent/references/script-format.md Normal file
View File

@ -0,0 +1,61 @@
# 标准剧本格式规范
## 场景描述
```
场景 1 - 内景/咖啡厅/日
```
- 以"场景 + 编号"开头
- 标注内景/外景
- 标注地点
- 标注时间（日/夜/黄昏/清晨）
## 角色动作
用括号标注角色动作和表情：
```
小明（皱眉，放下咖啡杯）
```
## 对白格式
```
小明：你确定这个计划可行？
小红：（犹豫片刻）我……我不确定。
```
## 旁白/画外音
```
【旁白】三年前的那个夏天，一切都还没有开始。
```
## 转场标注
- 切至：硬切
- 淡入/淡出：渐变过渡
- 叠化：两个画面重叠过渡
## 完整示例
```
场景 1 - 内景/咖啡厅/日
（阳光透过落地窗洒进咖啡厅，背景音乐轻柔）
小明独自坐在靠窗的位置，低头搅动着咖啡。
小红推门走入，环顾四周后看到小明。
小红：（微笑着走过来）好久不见。
小明：（抬头，愣了一下）你……怎么来了？
小红：（坐下，放下手提包）路过这里，想起你说过喜欢这家店。
【旁白】他们已经三年没有见面了。
—— 切至 ——
场景 2 - 外景/街道/夜
```

61
data/skills/production-agent/references/storyboard-guide.md Normal file
View File

@ -0,0 +1,61 @@
# 分镜设计指南
## 景别分类
| 景别 | 范围 | 用途 |
|------|------|------|
| 远景 | 环境全貌 | 交代环境、气氛渲染 |
| 全景 | 人物全身 | 展示人物与环境关系 |
| 中景 | 膝盖以上 | 日常对话、叙事推进 |
| 近景 | 胸部以上 | 表情细节、情绪传达 |
| 特写 | 面部/物件 | 强调情绪或关键道具 |
## 常用构图
- **三分法**：主体置于三分线交叉点
- **对称构图**：营造庄重、对峙感
- **引导线构图**：利用线条引导视线
- **框中框**：通过门窗等框住主体
- **前景遮挡**：增加画面层次感
## 镜头运动
- **推**：由远及近，聚焦主体
- **拉**：由近及远，展示全貌
- **摇**：固定位置旋转，扫视场景
- **移**：跟随角色移动
- **升降**：垂直运动，营造压迫或释放感
- **手持**：模拟真实视角，增加临场感
## 分镜描述模板
```
镜号：001
景别：近景
角度：平视
构图：三分法，人物偏左
动作：小明缓缓抬起头
对白：小明："原来是你。"
音效：雨声渐弱
时长：3s
```
## 完整分镜示例
### 场景：咖啡厅重逢
| 镜号 | 景别 | 构图 | 内容描述 | 对白/音效 | 时长 |
|------|------|------|----------|-----------|------|
| 001 | 远景 | 对称 | 咖啡厅外观，暖黄灯光 | 轻柔钢琴BGM | 2s |
| 002 | 中景 | 三分法 | 小明独坐窗边，搅动咖啡 | 咖啡杯碰撞声 | 3s |
| 003 | 全景 | 引导线 | 小红推门进入，逆光 | 门铃声 | 2s |
| 004 | 近景 | 居中 | 小明抬头，表情从平淡到惊讶 | 小红："好久不见。" | 3s |
| 005 | 特写 | 居中 | 小明的眼睛，瞳孔微微放大 | BGM渐弱 | 1.5s |
| 006 | 中景 | 对称 | 两人面对面坐下 | 小明："你怎么来了？" | 3s |
### 情绪节奏说明
- 001-002：平静、孤独（慢节奏）
- 003：转折信号（节奏变化）
- 004-005：情绪高点（短促镜头）
- 006：回归对话（节奏恢复）

1
src/app.ts
View File

@ -9,7 +9,6 @@ import logger from "morgan";
import cors from "cors"; import cors from "cors";
import buildRoute from "@/core"; import buildRoute from "@/core";
import fs from "fs"; import fs from "fs";
import path from "path";
import u from "@/utils"; import u from "@/utils";
import jwt from "jsonwebtoken"; import jwt from "jsonwebtoken";
import socketInit from "@/socket/index"; import socketInit from "@/socket/index";

141
src/lib/initDB.ts
View File

@ -50,18 +50,6 @@ export default async (knex: Knex, forceInit: boolean = false): Promise<void> =>
}, },
initData: async (knex) => {}, initData: async (knex) => {},
}, },
//技能表
{
name: "o_skills",
builder: (table) => {
table.integer("id").notNullable();
table.string("name");
table.integer("startTime");
table.primary(["id"]);
table.unique(["id"]);
},
initData: async (knex) => {},
},
//Agent配置表 //Agent配置表
{ {
name: "o_agentDeploy", name: "o_agentDeploy",
@ -186,64 +174,9 @@ export default async (knex: Knex, forceInit: boolean = false): Promise<void> =>
]); ]);
}, },
}, },
//模型表
{
name: "o_model",
builder: (table) => {
table.integer("id").notNullable();
table.text("type");
table.text("model");
table.text("modelType");
table.text("apiKey");
table.text("baseUrl");
table.text("manufacturer");
table.integer("createTime");
table.integer("index");
table.primary(["id"]);
table.unique(["id"]);
},
initData: async (knex) => {},
},
//提示词表
{
name: "o_prompts",
builder: (table) => {
table.integer("id").notNullable();
table.text("code"); // 代号，唯一标识
table.text("name"); // 名称/描述
table.text("type"); // 类型：mainAgent/subAgent/system
table.text("parentCode"); // 父级代号（subAgent关联主agent）
table.text("defaultValue"); // 默认提示词
table.text("customValue"); // 自定义修改值
table.primary(["id"]);
table.unique(["id"]);
table.unique(["code"]); // 代号唯一
},
initData: async (knex) => {},
},
//资产表
{
name: "o_assets",
builder: (table) => {
table.integer("id").notNullable();
table.text("name");
table.text("prompt");
table.text("remark");
table.text("type");
table.text("describe");
table.integer("imageId").unsigned().references("id").inTable("o_image");
table.integer("sonId");
table.integer("projectId");
table.integer("startTime");
table.text("state");
table.primary(["id"]);
table.unique(["id"]);
},
initData: async (knex) => {},
},
//任务中心表 //任务中心表
{ {
name: "o_myTasks", name: "o_tasks",
builder: (table) => { builder: (table) => {
table.integer("id").notNullable(); table.integer("id").notNullable();
table.integer("projectId"); table.integer("projectId");
@ -333,24 +266,38 @@ export default async (knex: Knex, forceInit: boolean = false): Promise<void> =>
table.unique(["id"]); table.unique(["id"]);
}, },
}, },
//剧本-大纲 //资产表
{ {
name: "o_scriptOutline", name: "o_assets",
builder: (table) => { builder: (table) => {
table.integer("id").notNullable(); table.integer("id").notNullable();
table.integer("scriptId").unsigned().references("id").inTable("o_script"); table.text("name");
table.integer("outlineId").unsigned().references("id").inTable("o_outline"); table.text("prompt");
table.text("remark");
table.text("type");
table.text("describe");
table.integer("imageId").unsigned().references("id").inTable("o_image");
table.integer("sonId");
table.integer("projectId");
table.integer("startTime");
table.text("state");
table.primary(["id"]); table.primary(["id"]);
table.unique(["id"]); table.unique(["id"]);
}, },
initData: async (knex) => {},
}, },
//剧本-资产 //生成图片表
{ {
name: "o_scriptAssets", name: "o_image",
builder: (table) => { builder: (table) => {
table.integer("id").notNullable(); table.integer("id").notNullable();
table.integer("assetsId").unsigned().references("id").inTable("o_assets"); table.text("filePath");
table.integer("scriptId").unsigned().references("id").inTable("o_script"); table.text("type");
table.integer("assetsId");
table.integer("scriptId");
table.integer("projectId");
table.integer("videoId");
table.text("state");
table.primary(["id"]); table.primary(["id"]);
table.unique(["id"]); table.unique(["id"]);
}, },
@ -366,13 +313,13 @@ export default async (knex: Knex, forceInit: boolean = false): Promise<void> =>
table.unique(["id"]); table.unique(["id"]);
}, },
}, },
//分镜-剧本 //flowData-剧本
{ {
name: "o_storyboardScript", name: "o_flowData",
builder: (table) => { builder: (table) => {
table.integer("id").notNullable(); table.integer("id").notNullable();
table.integer("storyboardId").unsigned().references("id").inTable("o_storyboard"); table.string("name");
table.integer("scriptId").unsigned().references("id").inTable("o_script"); table.integer("createTime");
table.primary(["id"]); table.primary(["id"]);
table.unique(["id"]); table.unique(["id"]);
}, },
@ -397,19 +344,6 @@ export default async (knex: Knex, forceInit: boolean = false): Promise<void> =>
table.unique(["id"]); table.unique(["id"]);
}, },
}, },
//聊天记录
{
name: "o_chatHistory",
builder: (table) => {
table.integer("id").notNullable();
table.text("type");
table.text("data");
table.text("novel");
table.integer("projectId");
table.primary(["id"]);
table.unique(["id"]);
},
},
//视频配置 //视频配置
{ {
name: "o_videoConfig", name: "o_videoConfig",
@ -434,18 +368,19 @@ export default async (knex: Knex, forceInit: boolean = false): Promise<void> =>
table.unique(["id"]); table.unique(["id"]);
}, },
}, },
//生成图片表 //供应商配置表
{ {
name: "o_image", name: "o_vendorConfig",
builder: (table) => { builder: (table) => {
table.integer("id").notNullable(); table.integer("id").notNullable();
table.text("filePath"); table.text("name");
table.text("type"); table.text("version");
table.integer("assetsId"); table.text("icon");
table.integer("scriptId"); table.text("inputs"); // 输入项配置 JSON
table.integer("projectId"); table.text("inputValues"); // 输入项值 JSON
table.integer("videoId"); table.text("models"); // 模型配置 JSON
table.text("state"); table.text("code"); // 模型配置 JSON
table.integer("createTime");
table.primary(["id"]); table.primary(["id"]);
table.unique(["id"]); table.unique(["id"]);
}, },
@ -479,7 +414,7 @@ export default async (knex: Knex, forceInit: boolean = false): Promise<void> =>
table.text("embedding"); // 向量嵌入 JSON table.text("embedding"); // 向量嵌入 JSON
table.text("relatedMessageIds"); // summary关联的message id列表 JSON table.text("relatedMessageIds"); // summary关联的message id列表 JSON
table.integer("summarized").defaultTo(0); // message是否已被总结 0/1 table.integer("summarized").defaultTo(0); // message是否已被总结 0/1
table.integer("createdAt").notNullable(); table.integer("createTime").notNullable();
table.primary(["id"]); table.primary(["id"]);
table.index(["isolationKey", "type"]); table.index(["isolationKey", "type"]);
table.index(["isolationKey", "summarized"]); table.index(["isolationKey", "summarized"]);

6
src/routes/agents/getMemory.ts
View File

@ -18,14 +18,14 @@ export default router.post(
const rows = await u const rows = await u
.db("memories") .db("memories")
.where({ isolationKey, type: "message" }) .where({ isolationKey, type: "message" })
.orderBy("createdAt", "asc") .orderBy("createTime", "asc")
.select("id", "content", "createdAt"); .select("id", "content", "createTime");
const history = rows.map((row) => ({ const history = rows.map((row) => ({
id: row.id, id: row.id,
role: "user", role: "user",
content: [{ type: "text", status: "complete", data: row.content }], content: [{ type: "text", status: "complete", data: row.content }],
createdAt: row.createdAt, createTime: row.createTime,
})); }));
res.status(200).send(success({ history })); res.status(200).send(success({ history }));

28
src/routes/agents/productionAgent.ts
View File

@ -1,9 +1,8 @@
import { tool } from "ai";
import { z } from "zod";
import express from "express"; import express from "express";
import { createAGUIStream } from "@/utils/agent/aguiTools"; import { createAGUIStream } from "@/utils/agent/aguiTools";
import u from "@/utils"; import u from "@/utils";
import Memory from "@/utils/agent/memory"; import Memory from "@/utils/agent/memory";
import { useSkill } from "@/utils/agent/skillsTools";
const router = express.Router(); const router = express.Router();
@ -14,7 +13,11 @@ function delay(ms: number) {
export default router.post("/", async (req, res) => { export default router.post("/", async (req, res) => {
const { prompt: text, projectId, episodesId } = req.body; const { prompt: text, projectId, episodesId } = req.body;
const isolationKey = `${projectId}:${episodesId}`; const isolationKey = `${projectId}:${episodesId}`;
//记忆
const memory = new Memory("productionAgent", isolationKey); const memory = new Memory("productionAgent", isolationKey);
//skill
const skill = await useSkill("production-agent");
const agui = createAGUIStream(res); const agui = createAGUIStream(res);
agui.runStarted(); agui.runStarted();
@ -24,9 +27,6 @@ export default router.post("/", async (req, res) => {
// 获取记忆上下文 // 获取记忆上下文
const mem = await memory.get(text); const mem = await memory.get(text);
console.log("======================================================");
// 构建记忆上下文文本（顺序：历史摘要 → 相关记忆 → 近期对话）
const memoryContext = [ const memoryContext = [
mem.rag.length > 0 && `[相关记忆]\n${mem.rag.map((r) => r.content).join("\n")}`, mem.rag.length > 0 && `[相关记忆]\n${mem.rag.map((r) => r.content).join("\n")}`,
mem.summaries.length > 0 && `[历史摘要]\n${mem.summaries.map((s, i) => `${i + 1}. ${s.content}`).join("\n")}`, mem.summaries.length > 0 && `[历史摘要]\n${mem.summaries.map((s, i) => `${i + 1}. ${s.content}`).join("\n")}`,
@ -35,9 +35,10 @@ export default router.post("/", async (req, res) => {
.filter(Boolean) .filter(Boolean)
.join("\n\n"); .join("\n\n");
console.log("%c Line:27 🍏 memoryContext", "background:#3f7cff", memoryContext);
const systemPrompt = `You are a helpful assistant.${memoryContext ? `\n\n以下是你对用户的记忆，可作为参考：\n${memoryContext}` : ""}`; const systemPrompt = [skill.prompt, memoryContext && `## Memory\n以下是你对用户的记忆，可作为参考但不要主动提及：\n${memoryContext}`]
.filter(Boolean)
.join("\n\n");
const messages = [ const messages = [
{ {
@ -50,17 +51,8 @@ export default router.post("/", async (req, res) => {
system: systemPrompt, system: systemPrompt,
messages, messages,
tools: { tools: {
deepRetrieve: tool({ ...skill.tools,
description: "深度检索记忆：当你需要回忆与某个关键词相关的详细历史信息时使用此工具", ...memory.getTools(),
inputSchema: z.object({
keyword: z.string().describe("要检索的关键词"),
}),
execute: async ({ keyword }) => {
const results = await memory.deepRetrieve(keyword);
if (results.length === 0) return { found: false, message: "未找到相关记忆" };
return { found: true, memories: results.map((r) => r.content) };
},
}),
}, },
onFinish: async (completion) => { onFinish: async (completion) => {
// 存入助手回复 // 存入助手回复

4
src/routes/project/delProject.ts
View File

@ -26,7 +26,7 @@ export default router.post(
await u.db("o_project").where("id", id).delete(); await u.db("o_project").where("id", id).delete();
await u.db("o_novel").where("projectId", id).delete(); await u.db("o_novel").where("projectId", id).delete();
await u.db("o_outline").where("projectId", id).delete(); await u.db("o_outline").where("projectId", id).delete();
await u.db("o_myTasks").where("projectId", id).delete(); await u.db("o_tasks").where("projectId", id).delete();
await u.db("o_script").where("projectId", id).delete(); await u.db("o_script").where("projectId", id).delete();
await u.db("o_assets").where("projectId", id).delete(); await u.db("o_assets").where("projectId", id).delete();
@ -45,8 +45,6 @@ export default router.post(
await u.db("o_video").whereIn("scriptId", scriptIds).delete(); await u.db("o_video").whereIn("scriptId", scriptIds).delete();
await u.db("o_chatHistory").where("projectId", id).delete();
try { try {
await u.oss.deleteDirectory(`${id}/`); await u.oss.deleteDirectory(`${id}/`);
console.log(`项目 ${id} 的OSS文件夹删除成功`); console.log(`项目 ${id} 的OSS文件夹删除成功`);

16
src/routes/task/getMyTaskApi.ts
View File

@ -16,27 +16,27 @@ export default router.post(
const { taskClass, state, page = 1, limit = 10 }: any = req.body; const { taskClass, state, page = 1, limit = 10 }: any = req.body;
const offset = (page - 1) * limit; const offset = (page - 1) * limit;
const data = await u const data = await u
.db("o_myTasks") .db("o_tasks")
.leftJoin("o_project", "o_project.id", "o_myTasks.projectId") .leftJoin("o_project", "o_project.id", "o_tasks.projectId")
.andWhere((qb) => { .andWhere((qb) => {
if (taskClass) { if (taskClass) {
qb.andWhere("o_myTasks.taskClass", taskClass); qb.andWhere("o_tasks.taskClass", taskClass);
} }
if (state) { if (state) {
qb.andWhere("o_myTasks.state", state); qb.andWhere("o_tasks.state", state);
} }
}) })
.select("o_myTasks.*", "o_project.* ") .select("o_tasks.*", "o_project.* ")
.offset(offset) .offset(offset)
.limit(limit); .limit(limit);
const totalQuery = (await u const totalQuery = (await u
.db("o_myTasks") .db("o_tasks")
.andWhere((qb) => { .andWhere((qb) => {
if (taskClass) { if (taskClass) {
qb.andWhere("o_myTasks.taskClass", taskClass); qb.andWhere("o_tasks.taskClass", taskClass);
} }
if (state) { if (state) {
qb.andWhere("o_myTasks.state", state); qb.andWhere("o_tasks.state", state);
} }
}) })
.count("* as total") .count("* as total")

2
src/routes/task/getTaskCategories.ts
View File

@ -11,7 +11,7 @@ export default router.post(
projectId: z.number(), projectId: z.number(),
}), }),
async (req, res) => { async (req, res) => {
const data = await u.db("o_myTasks").where("projectId", req.body.projectId).select("taskClass").groupBy("taskClass"); const data = await u.db("o_tasks").where("projectId", req.body.projectId).select("taskClass").groupBy("taskClass");
res.status(200).send(success(data)); res.status(200).send(success(data));
}, },
); );

2
src/routes/task/taskDetails.ts
View File

@ -12,7 +12,7 @@ export default router.post(
}), }),
async (req, res) => { async (req, res) => {
const { taskId } = req.body; const { taskId } = req.body;
const data = await u.db("o_myTasks").where("id", taskId).select("*").first(); const data = await u.db("o_tasks").where("id", taskId).select("*").first();
res.status(200).send(success(data)); res.status(200).send(success(data));
} }
); );

20
src/types/database.d.ts vendored
View File

@ -1,4 +1,4 @@
// @db-hash ad26ece2cf8002d48518ba2b8cd908f8 // @db-hash 2f9e6a9e9145cead00652858cafb9159
//该文件由脚本自动生成，请勿手动修改 //该文件由脚本自动生成，请勿手动修改
export interface memories { export interface memories {
@ -58,6 +58,11 @@ export interface o_eventChapter {
'id'?: number; 'id'?: number;
'novelId'?: number | null; 'novelId'?: number | null;
} }
export interface o_flowData {
'createTime'?: number | null;
'id'?: number;
'name'?: string | null;
}
export interface o_image { export interface o_image {
'assetsId'?: number | null; 'assetsId'?: number | null;
'filePath'?: string | null; 'filePath'?: string | null;
@ -166,6 +171,17 @@ export interface o_storyboardScript {
'scriptId'?: number | null; 'scriptId'?: number | null;
'storyboardId'?: number | null; 'storyboardId'?: number | null;
} }
export interface o_tasks {
'describe'?: string | null;
'id'?: number;
'model'?: string | null;
'projectId'?: number | null;
'reason'?: string | null;
'relatedObjects'?: string | null;
'startTime'?: number | null;
'state'?: string | null;
'taskClass'?: string | null;
}
export interface o_user { export interface o_user {
'id'?: number; 'id'?: number;
'name'?: string | null; 'name'?: string | null;
@ -223,6 +239,7 @@ export interface DB {
"o_chatHistory": o_chatHistory; "o_chatHistory": o_chatHistory;
"o_event": o_event; "o_event": o_event;
"o_eventChapter": o_eventChapter; "o_eventChapter": o_eventChapter;
"o_flowData": o_flowData;
"o_image": o_image; "o_image": o_image;
"o_model": o_model; "o_model": o_model;
"o_myTasks": o_myTasks; "o_myTasks": o_myTasks;
@ -238,6 +255,7 @@ export interface DB {
"o_skills": o_skills; "o_skills": o_skills;
"o_storyboard": o_storyboard; "o_storyboard": o_storyboard;
"o_storyboardScript": o_storyboardScript; "o_storyboardScript": o_storyboardScript;
"o_tasks": o_tasks;
"o_user": o_user; "o_user": o_user;
"o_vendorConfig": o_vendorConfig; "o_vendorConfig": o_vendorConfig;
"o_video": o_video; "o_video": o_video;

37
src/utils/agent/memory.ts
View File

@ -2,6 +2,8 @@ import u from "@/utils";
import { v4 as uuidv4 } from "uuid"; import { v4 as uuidv4 } from "uuid";
import { getEmbedding, cosineSimilarity } from "./embedding"; import { getEmbedding, cosineSimilarity } from "./embedding";
import type { memories as MemoryRow } from "@/types/database"; import type { memories as MemoryRow } from "@/types/database";
import { tool } from "ai";
import { z } from "zod";
// ── 可调配置 ── // ── 可调配置 ──
const messagesPerSummary = 3; // 每累积多少条message触发一次summary生成 const messagesPerSummary = 3; // 每累积多少条message触发一次summary生成
@ -56,7 +58,6 @@ class Memory {
async add( role: string = "user",content: string) { async add( role: string = "user",content: string) {
const id = uuidv4(); const id = uuidv4();
const embedding = await getEmbedding(content); const embedding = await getEmbedding(content);
const now = Date.now();
const isolationKey = this.isolationKey; const isolationKey = this.isolationKey;
await u.db("memories").insert({ await u.db("memories").insert({
@ -68,11 +69,11 @@ class Memory {
embedding: JSON.stringify(embedding), embedding: JSON.stringify(embedding),
relatedMessageIds: null, relatedMessageIds: null,
summarized: 0, summarized: 0,
createdAt: now, createTime: Date.now(),
} as any); } as any);
// 检查未总结消息数量 // 检查未总结消息数量
const unsummarized = await u.db("memories").where({ isolationKey, type: "message", summarized: 0 }).orderBy("createdAt", "asc"); const unsummarized = await u.db("memories").where({ isolationKey, type: "message", summarized: 0 }).orderBy("createTime", "asc");
if (unsummarized.length >= messagesPerSummary) { if (unsummarized.length >= messagesPerSummary) {
const batch = unsummarized.slice(0, messagesPerSummary); const batch = unsummarized.slice(0, messagesPerSummary);
@ -91,7 +92,7 @@ class Memory {
embedding: JSON.stringify(summaryEmbedding), embedding: JSON.stringify(summaryEmbedding),
relatedMessageIds: JSON.stringify(batchIds), relatedMessageIds: JSON.stringify(batchIds),
summarized: 0, summarized: 0,
createdAt: Date.now(), createTime: Date.now(),
}); });
// 标记已总结 // 标记已总结
@ -105,12 +106,12 @@ class Memory {
const shortTerm = await u const shortTerm = await u
.db("memories") .db("memories")
.where({ isolationKey, type: "message", summarized: 0 }) .where({ isolationKey, type: "message", summarized: 0 })
.orderBy("createdAt", "desc") .orderBy("createTime", "desc")
.limit(shortTermLimit); .limit(shortTermLimit);
shortTerm.reverse(); // 最旧在前 shortTerm.reverse(); // 最旧在前
// summaries: 最近的 summary // summaries: 最近的 summary
const summaries = await u.db("memories").where({ isolationKey, type: "summary" }).orderBy("createdAt", "desc").limit(summaryLimit); const summaries = await u.db("memories").where({ isolationKey, type: "summary" }).orderBy("createTime", "desc").limit(summaryLimit);
summaries.reverse(); summaries.reverse();
// rag: 向量搜索所有 messages // rag: 向量搜索所有 messages
@ -119,12 +120,12 @@ class Memory {
const ragResults = vectorSearch(allMessages, queryEmbedding, ragLimit); const ragResults = vectorSearch(allMessages, queryEmbedding, ragLimit);
return { return {
shortTerm: shortTerm.map((m: any) => ({ id: m.id, role: m.role, content: m.content, createdAt: m.createdAt })), shortTerm: shortTerm.map((m: any) => ({ id: m.id, role: m.role, content: m.content, createTime: m.createTime })),
summaries: summaries.map((s) => ({ summaries: summaries.map((s) => ({
id: s.id, id: s.id,
content: s.content, content: s.content,
relatedMessageIds: JSON.parse(s.relatedMessageIds || "[]"), relatedMessageIds: JSON.parse(s.relatedMessageIds || "[]"),
createdAt: s.createdAt, createTime: s.createTime,
})), })),
rag: ragResults.map((r) => ({ id: r.id, content: r.content, similarity: r.similarity })), rag: ragResults.map((r) => ({ id: r.id, content: r.content, similarity: r.similarity })),
}; };
@ -153,9 +154,25 @@ class Memory {
if (messageIds.length === 0) return []; if (messageIds.length === 0) return [];
const messages = await u.db("memories").whereIn("id", messageIds).orderBy("createdAt", "asc"); const messages = await u.db("memories").whereIn("id", messageIds).orderBy("createTime", "asc");
return messages.map((m) => ({ id: m.id, content: m.content, createdAt: m.createdAt })); return messages.map((m) => ({ id: m.id, content: m.content, createTime: m.createTime }));
}
getTools() {
return {
deepRetrieve: tool({
description: "深度检索记忆：当你需要回忆与某个关键词相关的详细历史信息时使用此工具",
inputSchema: z.object({
keyword: z.string().describe("要检索的关键词"),
}),
execute: async ({ keyword }) => {
const results = await this.deepRetrieve(keyword);
if (results.length === 0) return { found: false, message: "未找到相关记忆" };
return { found: true, memories: results.map((r) => r.content) };
},
}),
};
} }
} }

323
src/utils/agent/skillsTools.ts Normal file
View File

@ -0,0 +1,323 @@
import { tool } from "ai";
import { z } from "zod";
import path from "path";
import fs from "fs/promises";
import isPathInside from "is-path-inside";
import getPath from "@/utils/getPath";
// ==================== 类型 ====================
interface SkillRecord {
name: string;
description: string;
location: string; // SKILL.md 绝对路径
baseDir: string; // skill 目录绝对路径
}
// ==================== Step 2: 解析 SKILL.md ====================
/**
* 解析 SKILL.md frontmatter
* 支持 YAML 单行值及多行块标量（>、>-、|、|-）
*/
function parseFrontmatter(content: string): { name: string; description: string } {
const match = content.match(/^---\r?\n([\s\S]*?)\r?\n---/);
if (!match?.[1]) throw new Error("No frontmatter found");
const result: Record<string, string> = {};
const lines = match[1].split("\n");
let i = 0;
while (i < lines.length) {
const line = lines[i];
const colonIndex = line.indexOf(":");
if (colonIndex === -1) { i++; continue; }
const key = line.slice(0, colonIndex).trim();
if (!key) { i++; continue; }
let value = line.slice(colonIndex + 1).trim();
// 检测 YAML 块标量指示符 (>, >-, |, |-)
if (/^[>|]-?$/.test(value)) {
const fold = value.startsWith(">");
const parts: string[] = [];
i++;
while (i < lines.length) {
const next = lines[i];
// 缩进行属于当前块
if (/^\s+/.test(next)) {
parts.push(next.trim());
i++;
} else {
break;
}
}
value = fold ? parts.join(" ") : parts.join("\n");
} else {
i++;
}
result[key] = value;
}
if (!result.name) throw new Error("Frontmatter missing required field: name");
if (!result.description) throw new Error("Frontmatter missing required field: description");
return { name: result.name, description: result.description };
}
/**
* 去除 frontmatter，返回正文（body）
*/
function stripFrontmatter(content: string): string {
const match = content.match(/^---\r?\n[\s\S]*?\r?\n---\r?\n?/);
return match ? content.slice(match[0].length).trim() : content.trim();
}
// ==================== 资源枚举 ====================
/**
* 递归扫描目录，返回相对路径列表（排除 SKILL.md）
*/
async function listResources(dir: string, base: string = ""): Promise<string[]> {
const files: string[] = [];
let entries;
try {
entries = await fs.readdir(dir, { withFileTypes: true });
} catch {
return files;
}
for (const entry of entries) {
const rel = base ? `${base}/${entry.name}` : entry.name;
if (entry.isDirectory()) {
files.push(...(await listResources(path.join(dir, entry.name), rel)));
} else if (entry.name !== "SKILL.md") {
files.push(rel);
}
}
return files;
}
// ==================== Step 1: 发现 skills ====================
/**
* 扫描指定目录，发现所有包含 SKILL.md 的子目录
*/
async function discoverSkills(directories: string[]): Promise<SkillRecord[]> {
const skills: SkillRecord[] = [];
const seenNames = new Set<string>();
for (const dir of directories) {
let entries;
try {
entries = await fs.readdir(dir, { withFileTypes: true });
} catch {
continue;
}
for (const entry of entries) {
if (!entry.isDirectory()) continue;
const baseDir = path.join(dir, entry.name);
const location = path.join(baseDir, "SKILL.md");
let content: string;
try {
content = await fs.readFile(location, "utf-8");
} catch {
continue;
}
let metadata: { name: string; description: string };
try {
metadata = parseFrontmatter(content);
} catch (e) {
console.log(`[Skill] ⚠️ 跳过 "${entry.name}"：${(e as Error).message}`);
continue;
}
// 宽松校验：name 与目录名不匹配时仅告警
if (metadata.name !== entry.name) {
console.log(`[Skill] ⚠️ 技能名 "${metadata.name}" 与目录名 "${entry.name}" 不一致，仍加载`);
}
if (metadata.name.length > 64) {
console.log(`[Skill] ⚠️ 技能名 "${metadata.name}" 超过 64 字符，仍加载`);
}
// 先发现的同名 skill 优先（项目级覆盖用户级）
if (seenNames.has(metadata.name)) {
console.log(`[Skill] ⚠️ 技能 "${metadata.name}" 名称冲突，已被先前发现的同名技能覆盖`);
continue;
}
seenNames.add(metadata.name);
skills.push({
name: metadata.name,
description: metadata.description,
location,
baseDir,
});
console.log(`[Skill] ✅ 发现技能：${metadata.name} — ${metadata.description}`);
}
}
return skills;
}
// ==================== Step 3: 构建技能目录 ====================
/**
* 构建 XML 格式的技能目录 + 行为指令，注入到 system prompt
*/
function buildCatalog(skills: SkillRecord[]): string {
if (skills.length === 0) return "";
const skillsXml = skills
.map(
(s) =>
` <skill>\n <name>${s.name}</name>\n <description>${s.description}</description>\n </skill>`
)
.join("\n");
return [
"## Skills",
"以下技能提供了专业任务的专用指令。",
"当任务与某个技能的描述匹配时，调用 activate_skill 工具并传入技能名称来加载完整指令。",
"加载后遵循技能指令执行任务，需要时调用 read_skill_file 读取资源文件内容。",
"",
"<available_skills>",
skillsXml,
"</available_skills>",
].join("\n");
}
// ==================== Step 4 & 5: 激活 + 执行工具 ====================
/**
* 创建 activate_skill 和 read_skill_file 工具
*/
function createSkillTools(skills: SkillRecord[]) {
// 激活去重：记录当前会话已激活的 skill
const activated = new Set<string>();
const validNames = skills.map((s) => s.name);
return {
activate_skill: tool({
description: `激活一个技能，加载其完整指令和捆绑资源列表到上下文。可用技能：${validNames.join(", ")}`,
inputSchema: z.object({
name: z.enum(validNames as [string, ...string[]]).describe("要激活的技能名称"),
}),
execute: async ({ name }) => {
const skill = skills.find((s) => s.name === name);
if (!skill) {
console.log(`[Skill] ❌ 激活失败：未找到技能 "${name}"`);
return { error: `Skill '${name}' not found` };
}
// Step 5: 去重检查
if (activated.has(name)) {
console.log(`[Skill] ℹ️ 技能 "${name}" 已在当前会话中激活，跳过重复注入`);
return { already_active: true, message: `技能 "${name}" 已激活，无需重复加载` };
}
let content: string;
try {
content = await fs.readFile(skill.location, "utf-8");
} catch {
console.log(`[Skill] ❌ 激活失败：无法读取 ${skill.location}`);
return { error: `Failed to read SKILL.md for '${name}'` };
}
const body = stripFrontmatter(content);
const resources = await listResources(skill.baseDir);
activated.add(name);
const resourcesXml =
resources.length > 0
? `\n<skill_resources>\n${resources.map((f) => ` <file>${f}</file>`).join("\n")}\n</skill_resources>`
: "";
const wrapped = [
`<skill_content name="${skill.name}">`,
body,
"",
`Skill directory: ${skill.baseDir}`,
`相对路径基于此技能目录解析，使用 read_skill_file 工具读取资源文件。`,
resourcesXml,
`</skill_content>`,
].join("\n");
console.log(
`[Skill] 📖 已激活技能：${skill.name}（${body.length} 字符，${resources.length} 个资源文件）`
);
return { content: wrapped };
},
}),
read_skill_file: tool({
description: "读取已激活技能目录下的资源文件。传入 activate_skill 返回的 skill_resources 中的文件路径。",
inputSchema: z.object({
skillName: z.string().describe("技能名称"),
filePath: z.string().describe("资源文件的相对路径，来自 activate_skill 返回的 skill_resources"),
}),
execute: async ({ skillName, filePath: relPath }) => {
const skill = skills.find((s) => s.name === skillName);
if (!skill) {
console.log(`[Skill] ❌ 读取失败：未找到技能 "${skillName}"`);
return { error: `Skill '${skillName}' not found` };
}
const fullPath = path.resolve(path.join(skill.baseDir, relPath));
if (!isPathInside(fullPath, skill.baseDir)) {
console.log(`[Skill] 🚫 路径越界已拦截："${relPath}" 超出技能目录范围`);
return { error: "Access denied: path is outside skill directory" };
}
try {
const fileContent = await fs.readFile(fullPath, "utf-8");
console.log(`[Skill] 📄 已读取文件：${skillName}/${relPath}（${fileContent.length} 字符）`);
return { content: fileContent };
} catch {
console.log(`[Skill] ❌ 读取失败：未找到文件 "${relPath}"`);
return { error: `File not found: ${relPath}` };
}
},
}),
};
}
// ==================== 对外接口 ====================
/**
* 使用指定 skill（渐进式披露）
*
* 遵循 agentskills.io 规范：
* Step 1 — Discovery: 扫描 data/skills/{name} 目录
* Step 2 — Parse: 提取 frontmatter 元数据
* Step 3 — Disclose: 构建 XML 目录注入 system prompt
* Step 4 — Activate: activate_skill 工具加载完整指令 + 结构化包装 + 资源列表
* Step 5 — Manage: read_skill_file 读取资源 + 激活去重
*
* @param name skill 名称，对应 data/skills/{name} 目录
*/
export async function useSkill(name: string) {
const skills = await discoverSkills([getPath("skills")]);
// 过滤出指定 skill
const matched = skills.filter((s) => s.name === name);
if (matched.length === 0) {
console.log(`[Skill] ⚠️ 未发现名为 "${name}" 的技能`);
return { prompt: "", tools: {} };
}
return {
prompt: buildCatalog(matched),
tools: createSkillTools(matched),
};
}

4
src/utils/taskRecord.ts
View File

@ -38,7 +38,7 @@ export default async function taskRecord(
} }
} }
const [id] = await db("o_myTasks").insert({ const [id] = await db("o_tasks").insert({
projectId, projectId,
taskClass, taskClass,
relatedObjects: opteorContent, relatedObjects: opteorContent,
@ -50,7 +50,7 @@ export default async function taskRecord(
/** 任务成功时调用 done(1)，失败时调用 done(-1, '原因') */ /** 任务成功时调用 done(1)，失败时调用 done(-1, '原因') */
return async function done(state: 1 | -1, reason?: string) { return async function done(state: 1 | -1, reason?: string) {
await db("o_myTasks") await db("o_tasks")
.where("id", id) .where("id", id)
.update({ .update({
state: taskStateMap[state], state: taskStateMap[state],