From 1564f8648d8d6d777eb48593819da86610b4969a Mon Sep 17 00:00:00 2001 From: admin Date: Sun, 24 May 2026 19:19:46 +0700 Subject: [PATCH] 690524:1919 ADR-028-228-migration #04 --- AGENTS.md | 70 +- backend/.env.example | 4 +- .../ai/ai-migration-checkpoint.service.ts | 15 +- .../ai/dto/migration-checkpoint.dto.ts | 4 +- .../03-05-n8n-migration-setup-guide.md | 4 +- .../LCBP3 Migration Workflow v2.0.0.json | 1177 +++++++++++++++++ .../03-Data-and-Storage/n8n.workflow.v2.json | 225 ++-- .../ADR-023A-unified-ai-architecture.md | 52 +- .../ADR-026-document-chat-ui-pattern.md | 2 +- specs/08-Tasks/Task BE-AI-01.md | 2 +- .../224-intent-classification/plan.md | 24 +- .../224-intent-classification/quickstart.md | 6 +- .../224-intent-classification/spec.md | 10 +- .../228-migration-arch-refactor/plan.md | 18 +- .../228-migration-arch-refactor/spec.md | 2 +- .../228-migration-arch-refactor/tasks.md | 2 +- .../302-ai-model-revision/data-model.md | 8 +- .../300-others/302-ai-model-revision/plan.md | 8 +- .../302-ai-model-revision/quickstart.md | 18 +- .../302-ai-model-revision/research.md | 16 +- .../300-others/302-ai-model-revision/spec.md | 8 +- specs/README.md | 2 +- 22 files changed, 1422 insertions(+), 255 deletions(-) create mode 100644 specs/03-Data-and-Storage/LCBP3 Migration Workflow v2.0.0.json diff --git a/AGENTS.md b/AGENTS.md index 6b30dc37..f765dbbc 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -120,37 +120,37 @@ Best practice — follow when possible: Spec priority: **`06-Decision-Records`** > **`05-Engineering-Guidelines`** > others -| Document | Path | Status | Use When | -| ------------------------------ | --------------------------------------------------------------------------- | --------- | --------------------------------------------------------------------------------- | -| **Glossary** | `specs/00-overview/00-02-glossary.md` | — | Verify domain terminology | -| **Schema Tables** | `specs/03-Data-and-Storage/lcbp3-v1.9.0-schema-02-tables.sql` | — | Before writing any query | -| **Data Dictionary** | `specs/03-Data-and-Storage/03-01-data-dictionary.md` | — | Field meanings + business rules | -| **RBAC Matrix** | `specs/01-requirements/01-02-business-rules/01-02-01-rbac-matrix.md` | — | Permission levels + roles | -| **Edge Cases** | `specs/01-Requirements/01-06-edge-cases-and-rules.md` | — | Prevent bugs in flows | -| **ADR-001 Workflow Engine** | `specs/06-Decision-Records/ADR-001-unified-workflow-engine.md` | ✅ Active | DSL-based workflow implementation | -| **ADR-002 Doc Numbering** | `specs/06-Decision-Records/ADR-002-document-numbering-strategy.md` | ✅ Active | Document number generation + locking | -| **ADR-007 Error Handling** | `specs/06-Decision-Records/ADR-007-error-handling-strategy.md` | ✅ Active | Error patterns & recovery | -| **ADR-008 Notifications** | `specs/06-Decision-Records/ADR-008-email-notification-strategy.md` | ✅ Active | BullMQ + multi-channel notification | -| **ADR-009 DB Migration** | `specs/06-Decision-Records/ADR-009-database-migration-strategy.md` | ✅ Active | Schema changes — edit SQL directly | -| **ADR-016 Security** | `specs/06-Decision-Records/ADR-016-security-authentication.md` | ✅ Active | Auth, RBAC, file upload security | -| **ADR-015 Release Strategy** | `specs/06-Decision-Records/ADR-015-deployment-infrastructure.md` | ✅ Active | Blue-Green deployment + release gates | -| **ADR-019 UUID** | `specs/06-Decision-Records/ADR-019-hybrid-identifier-strategy.md` | ✅ Active | UUID-related work | -| **ADR-021 Workflow Context** | `specs/06-Decision-Records/ADR-021-workflow-context.md` | ✅ Active | Integrated workflow & step attachments | -| **ADR-023 AI Architecture** | `specs/06-Decision-Records/ADR-023-unified-ai-architecture.md` | ✅ Active | Unified AI boundaries and pipeline (base architecture) | -| **ADR-023A AI Model Rev.** | `specs/06-Decision-Records/ADR-023A-unified-ai-architecture.md` | ✅ Active | 2-Model stack (gemma4:e4b Q8_0), BullMQ 2-queue, RAG embed scope, OCR auto-detect | -| **ADR-024 Intent Class.** | `specs/06-Decision-Records/ADR-024-intent-classification-strategy.md` | ✅ Active | Hybrid Pattern→LLM Fallback; ai_intent_patterns DB; Redis cache 5 min | -| **ADR-025 AI Tool Layer** | `specs/06-Decision-Records/ADR-025-ai-tool-layer-architecture.md` | ✅ Active | Server-side Tool dispatch; CASL-guarded bridge; ToolResult uses publicId only | -| **ADR-026 Chat UI** | `specs/06-Decision-Records/ADR-026-document-chat-ui-pattern.md` | ✅ Active | Side-panel Document Chat UI; useAiChat() hook; streaming response support | -| **ADR-027 AI Admin Console** | `specs/06-Decision-Records/ADR-027-ai-admin-console-and-dynamic-control.md` | ✅ Active | Admin Panel + dynamic model/prompt/intent control without redeploy | -| **ADR-028 Migration Refactor** | `specs/06-Decision-Records/ADR-028-migration-architecture-refactor.md` | ✅ Active | Staging Queue & post-migration cleanup | -| **Backend Guidelines** | `specs/05-Engineering-Guidelines/05-02-backend-guidelines.md` | — | NestJS patterns | -| **Frontend Guidelines** | `specs/05-Engineering-Guidelines/05-03-frontend-guidelines.md` | — | Next.js patterns | -| **Testing Strategy** | `specs/05-Engineering-Guidelines/05-04-testing-strategy.md` | — | Coverage goals | -| **Git Conventions** | `specs/05-Engineering-Guidelines/05-05-git-conventions.md` | — | Commit/branch naming | -| **Code Snippets** | `specs/05-Engineering-Guidelines/05-06-code-snippets.md` | — | Reusable patterns | -| **i18n Guidelines** | `specs/05-Engineering-Guidelines/05-08-i18n-guidelines.md` | — | Localization rules | -| **Release Policy** | `specs/04-Infrastructure-OPS/04-08-release-management-policy.md` | — | Before deploy/hotfix | -| **UAT Criteria** | `specs/01-Requirements/01-05-acceptance-criteria.md` | — | Feature completeness | +| Document | Path | Status | Use When | +| ------------------------------ | --------------------------------------------------------------------------- | --------- | ----------------------------------------------------------------------------- | +| **Glossary** | `specs/00-overview/00-02-glossary.md` | — | Verify domain terminology | +| **Schema Tables** | `specs/03-Data-and-Storage/lcbp3-v1.9.0-schema-02-tables.sql` | — | Before writing any query | +| **Data Dictionary** | `specs/03-Data-and-Storage/03-01-data-dictionary.md` | — | Field meanings + business rules | +| **RBAC Matrix** | `specs/01-requirements/01-02-business-rules/01-02-01-rbac-matrix.md` | — | Permission levels + roles | +| **Edge Cases** | `specs/01-Requirements/01-06-edge-cases-and-rules.md` | — | Prevent bugs in flows | +| **ADR-001 Workflow Engine** | `specs/06-Decision-Records/ADR-001-unified-workflow-engine.md` | ✅ Active | DSL-based workflow implementation | +| **ADR-002 Doc Numbering** | `specs/06-Decision-Records/ADR-002-document-numbering-strategy.md` | ✅ Active | Document number generation + locking | +| **ADR-007 Error Handling** | `specs/06-Decision-Records/ADR-007-error-handling-strategy.md` | ✅ Active | Error patterns & recovery | +| **ADR-008 Notifications** | `specs/06-Decision-Records/ADR-008-email-notification-strategy.md` | ✅ Active | BullMQ + multi-channel notification | +| **ADR-009 DB Migration** | `specs/06-Decision-Records/ADR-009-database-migration-strategy.md` | ✅ Active | Schema changes — edit SQL directly | +| **ADR-016 Security** | `specs/06-Decision-Records/ADR-016-security-authentication.md` | ✅ Active | Auth, RBAC, file upload security | +| **ADR-015 Release Strategy** | `specs/06-Decision-Records/ADR-015-deployment-infrastructure.md` | ✅ Active | Blue-Green deployment + release gates | +| **ADR-019 UUID** | `specs/06-Decision-Records/ADR-019-hybrid-identifier-strategy.md` | ✅ Active | UUID-related work | +| **ADR-021 Workflow Context** | `specs/06-Decision-Records/ADR-021-workflow-context.md` | ✅ Active | Integrated workflow & step attachments | +| **ADR-023 AI Architecture** | `specs/06-Decision-Records/ADR-023-unified-ai-architecture.md` | ✅ Active | Unified AI boundaries and pipeline (base architecture) | +| **ADR-023A AI Model Rev.** | `specs/06-Decision-Records/ADR-023A-unified-ai-architecture.md` | ✅ Active | 2-Model stack (gemma4:e2b), BullMQ 2-queue, RAG embed scope, OCR auto-detect | +| **ADR-024 Intent Class.** | `specs/06-Decision-Records/ADR-024-intent-classification-strategy.md` | ✅ Active | Hybrid Pattern→LLM Fallback; ai_intent_patterns DB; Redis cache 5 min | +| **ADR-025 AI Tool Layer** | `specs/06-Decision-Records/ADR-025-ai-tool-layer-architecture.md` | ✅ Active | Server-side Tool dispatch; CASL-guarded bridge; ToolResult uses publicId only | +| **ADR-026 Chat UI** | `specs/06-Decision-Records/ADR-026-document-chat-ui-pattern.md` | ✅ Active | Side-panel Document Chat UI; useAiChat() hook; streaming response support | +| **ADR-027 AI Admin Console** | `specs/06-Decision-Records/ADR-027-ai-admin-console-and-dynamic-control.md` | ✅ Active | Admin Panel + dynamic model/prompt/intent control without redeploy | +| **ADR-028 Migration Refactor** | `specs/06-Decision-Records/ADR-028-migration-architecture-refactor.md` | ✅ Active | Staging Queue & post-migration cleanup | +| **Backend Guidelines** | `specs/05-Engineering-Guidelines/05-02-backend-guidelines.md` | — | NestJS patterns | +| **Frontend Guidelines** | `specs/05-Engineering-Guidelines/05-03-frontend-guidelines.md` | — | Next.js patterns | +| **Testing Strategy** | `specs/05-Engineering-Guidelines/05-04-testing-strategy.md` | — | Coverage goals | +| **Git Conventions** | `specs/05-Engineering-Guidelines/05-05-git-conventions.md` | — | Commit/branch naming | +| **Code Snippets** | `specs/05-Engineering-Guidelines/05-06-code-snippets.md` | — | Reusable patterns | +| **i18n Guidelines** | `specs/05-Engineering-Guidelines/05-08-i18n-guidelines.md` | — | Localization rules | +| **Release Policy** | `specs/04-Infrastructure-OPS/04-08-release-management-policy.md` | — | Before deploy/hotfix | +| **UAT Criteria** | `specs/01-Requirements/01-05-acceptance-criteria.md` | — | Feature completeness | --- @@ -265,7 +265,7 @@ Read `specs/05-Engineering-Guidelines/05-07-hybrid-uuid-implementation-plan.md` 5. **Password:** bcrypt 12 salt rounds, min 8 chars, rotate every 90 days 6. **Rate Limiting:** `ThrottlerGuard` on all auth endpoints 7. **File Upload:** Whitelist PDF/DWG/DOCX/XLSX/ZIP, max 50MB, ClamAV scan -8. **AI Isolation (ADR-023/023A):** Ollama on Admin Desktop ONLY — NO direct DB/storage access; 2-model stack `gemma4:e4b Q8_0` + `nomic-embed-text`; all inference via BullMQ (`ai-realtime` / `ai-batch`) +8. **AI Isolation (ADR-023/023A):** Ollama on Admin Desktop ONLY — NO direct DB/storage access; 2-model stack `gemma4:e2b` + `nomic-embed-text`; all inference via BullMQ (`ai-realtime` / `ai-batch`) 9. **Error Handling (ADR-007):** Use layered error classification with user-friendly messages 10. **AI Integration (ADR-023/023A):** RFA-First approach; n8n orchestrates Migration Phase only via DMS API — never calls Ollama directly; `QdrantService.search()` requires `projectPublicId` as mandatory param @@ -427,7 +427,7 @@ Full glossary: `specs/00-overview/00-02-glossary.md` **For AI Runtime Layer (ADR-024/025/026/027):** -- ADR-024: Pattern Layer first (ai_intent_patterns DB + Redis cache 5 min) → LLM Fallback (gemma4:e4b, semaphore max=3) +- ADR-024: Pattern Layer first (ai_intent_patterns DB + Redis cache 5 min) → LLM Fallback (gemma4:e2b, semaphore max=3) - ADR-025: Tool Registry dispatch — AI Gateway → Tool → Business Service; ToolResult DTO must use publicId only - ADR-026: useAiChat() hook + side-panel UI; streaming response via SSE; TanStack Query cache - ADR-027: Admin Console — dynamic model/prompt/intent control; CASL-guarded admin-only endpoints @@ -549,7 +549,7 @@ When user asks about... check these files: - [ ] **Qdrant Multi-tenancy:** `projectPublicId` filter enforced - [ ] **Human-in-the-loop:** AI outputs validated before use - [ ] **Audit Logging:** All AI interactions logged to `ai_audit_logs` -- [ ] **2-Model Stack:** gemma4:e4b Q8_0 + nomic-embed-text verified +- [ ] **2-Model Stack:** gemma4:e2b + nomic-embed-text verified **Performance & Complex Logic:** @@ -606,7 +606,7 @@ This file is a **quick reference**. For detailed information: | 1.9.6 | 2026-05-22 | Added ADR-024/025/026/027/028 to Key Spec Files table; Tier 3 expanded with AI Runtime Layer + Migration Pipeline tiers; Specialized Work section updated with ADR-024~028 patterns; 6 new Context-Aware Triggers; bumped Last synced date | Windsurf AI | | 1.9.5 | 2026-05-18 | **Grill-with-Docs Session:** Domain terminology clarified (Correspondence = all doc types), Tier 3: SPECIALIZED WORK added, Context-Aware Triggers with Status column, Tier-specific Final Checklists | Windsurf AI | | 1.9.4 | 2026-05-16 | Added ADR-015 Release Strategy to Key Spec Files table (Blue-Green deployment + release gates) | Human Dev | -| 1.9.3 | 2026-05-15 | ADR-023A: Model revision — gemma4:9b+Typhoon→gemma4:e4b Q8_0 (2-model stack), BullMQ 2-queue split, RAG full-doc embed, OCR auto-detect, n8n→DMS API boundary, QdrantService multi-tenancy contract | Windsurf AI | +| 1.9.3 | 2026-05-15 | ADR-023A: Model revision — gemma4:9b+Typhoon→gemma4:e2b (2-model stack), BullMQ 2-queue split, RAG full-doc embed, OCR auto-detect, n8n→DMS API boundary, QdrantService multi-tenancy contract | Windsurf AI | | 1.9.2 | 2026-05-14 | Consolidated legacy AI ADRs (017, 017B, 018, 020, 022) into master ADR-023: Unified AI Architecture | Antigravity AI | | 1.9.1 | 2026-05-13 | Added `bugfix` workflow and skill (migrated and improved from `docs/bugfix.md`) | Windsurf AI | | 1.9.0 | 2026-05-03 | Integrated Global TypeScript Coding Standards (Headers, JSDoc, Thai comments, Single Export, No blank lines) | Windsurf AI | diff --git a/backend/.env.example b/backend/.env.example index f3e3ac43..91fac5f9 100644 --- a/backend/.env.example +++ b/backend/.env.example @@ -50,10 +50,10 @@ AI_N8N_AUTH_TOKEN=change-me-service-token QDRANT_URL=http://localhost:6333 # Ollama (Admin Desktop Desk-5439 — ADR-018 AI boundary) -OLLAMA_MODEL_MAIN=gemma4:e4b +OLLAMA_MODEL_MAIN=gemma4:e2b OLLAMA_MODEL_EMBED=nomic-embed-text OLLAMA_EMBED_MODEL=nomic-embed-text -OLLAMA_RAG_MODEL=gemma4:e4b +OLLAMA_RAG_MODEL=gemma4:e2b OLLAMA_URL=http://192.168.10.8:11434 # Qdrant (ADR-023A) diff --git a/backend/src/modules/ai/ai-migration-checkpoint.service.ts b/backend/src/modules/ai/ai-migration-checkpoint.service.ts index c65ba74f..88d6c2e6 100644 --- a/backend/src/modules/ai/ai-migration-checkpoint.service.ts +++ b/backend/src/modules/ai/ai-migration-checkpoint.service.ts @@ -1,6 +1,7 @@ // File: src/modules/ai/ai-migration-checkpoint.service.ts // Change Log: // - 2026-05-23: สร้าง service จัดการ Migration Checkpoint, Queue และ Error log ผ่าน API (ADR-023A) +// - 2026-05-24: เพิ่มฟังก์ชันค้นหาและแปลง UUID เป็นตัวเลข ID จริงใน upsertQueueRecord เพื่อป้องกันการเขียนทับด้วย undefined import { Injectable, Logger } from '@nestjs/common'; import { InjectRepository } from '@nestjs/typeorm'; @@ -102,7 +103,19 @@ export class AiMigrationCheckpointService { record.batchId = dto.batchId; record.originalFileName = dto.documentNumber; - record.tempAttachmentId = dto.tempAttachmentId ?? undefined; + if (dto.tempAttachmentId) { + if (typeof dto.tempAttachmentId === 'number') { + record.tempAttachmentId = dto.tempAttachmentId; + } else { + const rows = await this.dataSource.manager.query<{ id: number }[]>( + 'SELECT id FROM attachments WHERE uuid = ? LIMIT 1', + [dto.tempAttachmentId] + ); + if (rows && rows.length > 0) { + record.tempAttachmentId = rows[0].id; + } + } + } record.confidenceScore = dto.confidence ?? undefined; record.status = dto.status === 'PENDING_REVIEW' diff --git a/backend/src/modules/ai/dto/migration-checkpoint.dto.ts b/backend/src/modules/ai/dto/migration-checkpoint.dto.ts index 37653e96..d71239ae 100644 --- a/backend/src/modules/ai/dto/migration-checkpoint.dto.ts +++ b/backend/src/modules/ai/dto/migration-checkpoint.dto.ts @@ -1,6 +1,7 @@ // File: src/modules/ai/dto/migration-checkpoint.dto.ts // Change Log: // - 2026-05-23: สร้าง DTOs สำหรับ Migration Checkpoint API endpoints (ADR-023A) +// - 2026-05-24: ปรับปรุงประเภทข้อมูล tempAttachmentId ใน MigrationQueueRecordDto ให้รับได้ทั้ง string (UUID) และ number import { IsEnum, @@ -47,9 +48,8 @@ export class MigrationQueueRecordDto { @IsOptional() originalSubject?: string; - @IsNumber() @IsOptional() - tempAttachmentId?: number; + tempAttachmentId?: string | number; @IsNumber() @Min(0) diff --git a/specs/03-Data-and-Storage/03-05-n8n-migration-setup-guide.md b/specs/03-Data-and-Storage/03-05-n8n-migration-setup-guide.md index d8a8d54f..de99d576 100644 --- a/specs/03-Data-and-Storage/03-05-n8n-migration-setup-guide.md +++ b/specs/03-Data-and-Storage/03-05-n8n-migration-setup-guide.md @@ -120,7 +120,7 @@ const CONFIG = { // Ollama Settings OLLAMA_HOST: 'http://192.168.20.100:11434', - OLLAMA_MODEL: 'gemma4:e4b', // ห้ามเปลี่ยน — กำหนดโดย ADR-023A + OLLAMA_MODEL: 'gemma4:e2b', // ห้ามเปลี่ยน — กำหนดโดย ADR-023A EMBED_MODEL: 'nomic-embed-text', // สำหรับ Embedding เท่านั้น // ไม่มี FALLBACK model — BullMQ concurrency=1 จัดการ GPU usage @@ -338,7 +338,7 @@ mysql -h -u migration_bot -p lcbp3_production < lcbp3-v1.8.0-migration - Submit: `POST /api/ai/jobs` พร้อม `temp_attachment_id`, `document_number`, `title`, `existing_tags`, `system_categories` - Response: `{ "jobId": "" }` - Poll: `GET /api/ai/jobs/{{jobId}}` ทุก 5 วินาที จน `status = "completed"` (timeout 120 วินาที) -- AI inference ใช้ `gemma4:e4b Q8_0` ผ่าน BullMQ Worker — System/User Prompt อยู่ใน Backend NestJS ไม่ใช่ใน n8n +- AI inference ใช้ `gemma4:e2b` ผ่าน BullMQ Worker — System/User Prompt อยู่ใน Backend NestJS ไม่ใช่ใน n8n ### Node 5: Parse & Validate diff --git a/specs/03-Data-and-Storage/LCBP3 Migration Workflow v2.0.0.json b/specs/03-Data-and-Storage/LCBP3 Migration Workflow v2.0.0.json new file mode 100644 index 00000000..9a0bce68 --- /dev/null +++ b/specs/03-Data-and-Storage/LCBP3 Migration Workflow v2.0.0.json @@ -0,0 +1,1177 @@ +{ + "name": "LCBP3 Migration Workflow v2.0.0", + "nodes": [ + { + "parameters": { + "formTitle": "LCBP3 Migration - ตั้งค่าก่อนรัน", + "formDescription": "กรุณาตั้งค่า Batch และ Excel file ก่อนรัน Migration", + "formFields": { + "values": [ + { + "fieldLabel": "Batch Size", + "fieldType": "number", + "placeholder": "10" + }, + { + "fieldLabel": "Excel File Path", + "placeholder": "/home/node/.n8n-files/staging_ai/C22024.xlsx" + } + ] + }, + "options": {} + }, + "id": "a0346819-4e97-4208-99c8-e4f958d652fe", + "name": "Form Trigger", + "type": "n8n-nodes-base.formTrigger", + "typeVersion": 2.2, + "position": [ + -9280, + 4400 + ], + "webhookId": "dd44ab55-df6b-4f3b-a740-6a993ca7ded0", + "notes": "เปิด URL เพื่อตั้งค่าก่อนรัน" + }, + { + "parameters": { + "jsCode": "// ============================================\n// CONFIGURATION v2.0 — ADR-023A Compliant\n// n8n เรียกแค่ DMS Backend API เท่านั้น — ห้ามเรียก Ollama โดยตรง\n// ============================================\nconst formData = $('Form Trigger').first()?.json || {};\nconst batchSizeInput = parseInt(String(formData['Batch Size'] || '0'));\nconst excelFileInput = String(formData['Excel File Path'] || '').trim();\nconst jwtTokenInput = 'Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VybmFtZSI6Im1pZ3JhdGlvbl9ib3QiLCJzdWIiOjUsInNjb3BlIjoiR2xvYmFsIiwiaWF0IjoxNzc5NTQwNDk4LCJleHAiOjQ5MzUzMDA0OTh9.TZVzG8u4Cyz8hi0cU3eGaeXinKWKHN3HtYnDcS5p_5I';\nconst resolvedExcelFile = excelFileInput || '/home/node/.n8n-files/staging_ai/C22024.xlsx';\n\n// Validate Excel file path to prevent path traversal\nif (resolvedExcelFile.includes('..') || resolvedExcelFile.includes('~')) {\n throw new Error('Invalid Excel file path: path traversal detected');\n}\n\nif (!jwtTokenInput || !jwtTokenInput.startsWith('Bearer ')) {\n throw new Error('JWT Token is required and must start with \"Bearer \"');\n}\n\nconst BATCH_ID = (() => {\n // ใช้ Excel filename เป็น BATCH_ID เพื่อให้ checkpoint ทำงานถูกต้อง\n // เปลี่ยน Excel file = BATCH_ID ใหม่ = เริ่มใหม่ (ปลอดภัย)\n // ใช้ Excel file เดิม = BATCH_ID เดิม = resume ได้\n const filename = resolvedExcelFile.split('/').pop().split('\\\\').pop();\n const baseName = filename.replace(/\\.xlsx?$/i, '');\n return baseName + '-MIGRATION';\n})();\n\nconst CONFIG = {\n // Backend Settings\n BACKEND_URL: 'http://backend:3000',\n MIGRATION_TOKEN: jwtTokenInput,\n\n // Batch Settings\n BATCH_SIZE: batchSizeInput > 0 ? batchSizeInput : 10,\n BATCH_ID: BATCH_ID,\n DELAY_MS: 20000,\n\n // AI Job Settings (ADR-023A)\n AI_JOB_POLL_INTERVAL_MS: 5000,\n AI_JOB_TIMEOUT_MS: 120000,\n\n // Confidence Thresholds\n CONFIDENCE_HIGH: 0.85,\n CONFIDENCE_LOW: 0.60,\n\n // Paths (Container paths — ต้องตรงกับ volume mount ใน docker-compose)\n EXCEL_FILE: resolvedExcelFile,\n SOURCE_PDF_DIR: '/home/node/.n8n-files/staging_ai/Incoming/08C.2/2567',\n LOG_PATH: '/home/node/.n8n-files/migration_logs',\n};\n\nreturn [{ json: { config: CONFIG } }];" + }, + "id": "65143f3f-b0ee-45e6-b0f9-bf57546b7482", + "name": "Set Configuration", + "type": "n8n-nodes-base.code", + "typeVersion": 2, + "position": [ + -9088, + 4400 + ], + "notes": "กำหนดค่า Configuration ทั้งหมด — แก้ไขที่นี่เท่านั้น (ไม่มี Ollama config แล้ว)" + }, + { + "parameters": { + "url": "={{$('Set Configuration').first().json.config.BACKEND_URL}}/health", + "options": { + "timeout": 5000 + } + }, + "id": "b6295d67-9a34-4550-8e96-096a59a88053", + "name": "Check Backend Health", + "type": "n8n-nodes-base.httpRequest", + "typeVersion": 4.1, + "position": [ + -8912, + 4400 + ], + "notes": "ตรวจสอบ Backend พร้อมใช้งาน" + }, + { + "parameters": { + "url": "={{$('Set Configuration').first().json.config.BACKEND_URL}}/api/auth/profile", + "sendHeaders": true, + "headerParameters": { + "parameters": [ + { + "name": "Authorization", + "value": "={{$('Set Configuration').first().json.config.MIGRATION_TOKEN}}" + } + ] + }, + "options": { + "timeout": 5000 + } + }, + "id": "92b9007e-9ad1-40f6-8e37-cec5113f6b30", + "name": "Validate Token", + "type": "n8n-nodes-base.httpRequest", + "typeVersion": 4.1, + "position": [ + -8736, + 4400 + ], + "notes": "FR-010a: ตรวจสอบ MIGRATION_TOKEN ก่อนเริ่ม Batch" + }, + { + "parameters": { + "url": "={{$('Set Configuration').first().json.config.BACKEND_URL}}/api/master/correspondence-types", + "sendHeaders": true, + "headerParameters": { + "parameters": [ + { + "name": "Authorization", + "value": "={{$('Set Configuration').first().json.config.MIGRATION_TOKEN}}" + } + ] + }, + "options": { + "timeout": 10000 + } + }, + "id": "a9c11f42-b63b-412f-b919-88dbc7ab095e", + "name": "Fetch Categories", + "type": "n8n-nodes-base.httpRequest", + "typeVersion": 4.1, + "position": [ + -9088, + 4592 + ], + "notes": "ดึง Categories จาก Backend" + }, + { + "parameters": { + "url": "={{$('Set Configuration').first().json.config.BACKEND_URL}}/api/master/tags", + "sendHeaders": true, + "headerParameters": { + "parameters": [ + { + "name": "Authorization", + "value": "={{$('Set Configuration').first().json.config.MIGRATION_TOKEN}}" + } + ] + }, + "options": { + "timeout": 10000 + } + }, + "id": "108423e3-90de-49a3-b950-8f78cd231b84", + "name": "Fetch Tags", + "type": "n8n-nodes-base.httpRequest", + "typeVersion": 4.1, + "position": [ + -8912, + 4592 + ], + "notes": "ดึง Tags ที่มีอยู่แล้วจาก Backend" + }, + { + "parameters": { + "jsCode": "const fs = require('fs');\nconst config = $('Set Configuration').first().json.config;\n\ntry {\n if (!fs.existsSync(config.EXCEL_FILE)) {\n throw new Error(`Excel file not found at: ${config.EXCEL_FILE}`);\n }\n if (!fs.existsSync(config.SOURCE_PDF_DIR)) {\n throw new Error(`PDF Source directory not found at: ${config.SOURCE_PDF_DIR}`);\n }\n \n const files = fs.readdirSync(config.SOURCE_PDF_DIR);\n \n // Check write permission to log path\n fs.writeFileSync(`${config.LOG_PATH}/.preflight_ok`, new Date().toISOString());\n \n // Grab categories\n let categories = ['Correspondence','RFA','Drawing','Transmittal','Report','Other'];\n try {\n const upstreamData = $('Fetch Categories').first()?.json?.data;\n if (upstreamData && Array.isArray(upstreamData)) {\n categories = upstreamData.map(c => c.name || c.type || c);\n }\n } catch(e) {}\n \n // Grab existing tags\n let existingTags = [];\n try {\n const tagData = $('Fetch Tags').first()?.json?.data || [];\n existingTags = Array.isArray(tagData)\n ? tagData.map(t => ({ publicId: t.publicId, tagName: t.tag_name || t.name || '' })).filter(t => t.tagName && t.publicId)\n : [];\n } catch(e) {}\n \n return [{ json: { \n preflight_ok: true, \n pdf_count_in_source: files.length,\n excel_target: config.EXCEL_FILE,\n system_categories: categories,\n existing_tags: existingTags,\n timestamp: new Date().toISOString()\n }}];\n} catch (err) {\n throw new Error(`Pre-flight check failed: ${err.message}`);\n}" + }, + "id": "d8e2efec-7fb4-44c3-b544-0dc51dcfd3fc", + "name": "File Mount Check", + "type": "n8n-nodes-base.code", + "typeVersion": 2, + "position": [ + -8736, + 4592 + ], + "notes": "ตรวจสอบ File System + รวบรวม master data สำหรับ AI payload" + }, + { + "parameters": { + "fileSelector": "={{ $json.excel_target }}", + "options": {} + }, + "id": "c2a6ca17-aae8-4213-93ab-8179e1febfb3", + "name": "Read Excel Binary", + "type": "n8n-nodes-base.readWriteFile", + "typeVersion": 1, + "position": [ + -9088, + 4784 + ], + "notes": "ดึงไฟล์ Excel ขึ้นมาไว้ในหน่วยความจำ" + }, + { + "parameters": { + "options": {} + }, + "id": "ebfdde99-92a1-4060-9867-2f330b13554f", + "name": "Read Excel", + "type": "n8n-nodes-base.spreadsheetFile", + "typeVersion": 2, + "position": [ + -8912, + 4784 + ], + "notes": "แปลงข้อมูล Excel เป็น JSON Data" + }, + { + "parameters": { + "url": "={{ $('Set Configuration').first().json.config.BACKEND_URL + '/api/ai/migration/checkpoint/' + $('Set Configuration').first().json.config.BATCH_ID }}", + "sendHeaders": true, + "headerParameters": { + "parameters": [ + { + "name": "Authorization", + "value": "={{$('Set Configuration').first().json.config.MIGRATION_TOKEN}}" + } + ] + }, + "options": { + "timeout": 10000 + } + }, + "id": "9cf3971c-287a-42ab-9327-72e1a4d8011c", + "name": "Read Checkpoint", + "type": "n8n-nodes-base.httpRequest", + "typeVersion": 4.1, + "position": [ + -8720, + 4784 + ], + "alwaysOutputData": true, + "notes": "GET /api/ai/migration/checkpoint/:batchId — ADR-023A (ไม่ใช้ MySQL โดยตรง)" + }, + { + "parameters": { + "jsCode": "const cpJson = $input.first()?.json || {};\nconst startIndex = cpJson.data?.lastProcessedIndex ?? cpJson.lastProcessedIndex ?? cpJson.data?.last_processed_index ?? cpJson.last_processed_index ?? 0;\nconst config = $('Set Configuration').first().json.config;\n\nconst allItems = $('Read Excel').all().map(i => i.json);\nconst remaining = allItems.slice(startIndex);\nconst currentBatch = remaining.slice(0, config.BATCH_SIZE);\n\n// Exit condition: ไม่มี records เหลือ\nif (currentBatch.length === 0) {\n return [{ json: { batch_complete: true, total_processed: startIndex } }];\n}\n\nconst normalize = (str) => {\n if (!str) return '';\n return String(str).normalize('NFC').trim();\n};\n\nreturn currentBatch.map((item, i) => {\n const getVal = (possibleKeys) => {\n const exactMatch = possibleKeys.find(k => item[k] !== undefined);\n if (exactMatch) return item[exactMatch];\n const lowerTrimmedKeys = Object.keys(item).map(k => ({ original: k, parsed: k.toLowerCase().trim() }));\n for (const pk of possibleKeys) {\n const found = lowerTrimmedKeys.find(k => k.parsed === pk.toLowerCase().trim());\n if (found) return item[found.original];\n }\n return '';\n };\n\n const docNum = getVal(['document_number', 'correspondence_number', 'Document Number', 'Corr. No.']);\n const excelFileName = getVal(['File name', 'file_name', 'File Name', 'filename']);\n \n if (!excelFileName) {\n throw new Error(`Missing 'File name' column for row ${i + startIndex + 1}, document: ${docNum}`);\n }\n \n return {\n json: {\n batch_complete: false,\n document_number: normalize(docNum),\n title: normalize(getVal(['title', 'Title', 'Subject', 'subject'])),\n legacy_number: normalize(getVal(['legacy_number', 'Legacy Number', 'Response Doc.'])),\n excel_revision: getVal(['revision', 'Revision', 'rev']) || 1,\n original_index: startIndex + i,\n batch_id: config.BATCH_ID,\n file_name: normalize(excelFileName),\n issued_date: normalize(getVal(['issued_date', 'Issued_date', 'Issued Date', 'date', 'Date'])),\n received_date: normalize(getVal(['received_date', 'Received_date', 'Received Date'])),\n sender: normalize(getVal(['sender', 'Sender', 'From', 'from'])),\n receiver: normalize(getVal(['receiver', 'Receiver', 'To', 'to'])),\n }\n };\n});" + }, + "id": "c101543e-a3a2-4538-a99c-0419df50948f", + "name": "Process Batch + Encoding", + "type": "n8n-nodes-base.code", + "typeVersion": 2, + "position": [ + -8528, + 4400 + ], + "alwaysOutputData": true, + "notes": "ตัด Batch + Normalize UTF-8 + Exit condition เมื่อ records หมด" + }, + { + "parameters": { + "jsCode": "const fs = require('fs');\nconst path = require('path');\nconst config = $('Set Configuration').first().json.config;\n\nconst items = $input.all();\nif (!items || items.length === 0) return [];\n\n// ตรวจสอบ exit condition\nif (items[0]?.json?.batch_complete === true) {\n return items;\n}\n\nconst validated = [];\nconst errors = [];\n\nfor (const item of items) {\n const fileName = item.json?.file_name;\n if (!fileName) {\n errors.push({ ...item, json: { ...item.json, file_valid: false, error: 'file_name is missing', error_type: 'MISSING_FILENAME' } });\n continue;\n }\n \n let safeName = path.basename(String(fileName)).normalize('NFC');\n if (!safeName.toLowerCase().endsWith('.pdf')) safeName += '.pdf';\n const filePath = path.resolve(config.SOURCE_PDF_DIR, safeName);\n \n if (!filePath.startsWith(path.resolve(config.SOURCE_PDF_DIR))) {\n errors.push({ ...item, json: { ...item.json, file_valid: false, error: 'Path traversal detected', error_type: 'SECURITY' } });\n continue;\n }\n \n try {\n if (fs.existsSync(filePath)) {\n const stats = fs.statSync(filePath);\n validated.push({ ...item, json: { ...item.json, file_valid: true, file_path: filePath, file_size: stats.size } });\n } else {\n errors.push({ ...item, json: { ...item.json, file_valid: false, error: `File not found: ${safeName}`, error_type: 'FILE_NOT_FOUND' } });\n }\n } catch (err) {\n errors.push({ ...item, json: { ...item.json, file_valid: false, error: err.message, error_type: 'FILE_ERROR' } });\n }\n}\n\nreturn [...validated, ...errors];" + }, + "id": "282e1c8a-74b6-4ae7-a50b-3e2ff7b1558d", + "name": "File Validator", + "type": "n8n-nodes-base.code", + "typeVersion": 2, + "position": [ + -8512, + 4800 + ], + "notes": "ตรวจสอบไฟล์ PDF ใน Directory" + }, + { + "parameters": { + "fileSelector": "={{ $json.file_path }}", + "options": {} + }, + "id": "ab2d3ff8-9862-4a5c-a21d-5e04004d6a40", + "name": "Read PDF File", + "type": "n8n-nodes-base.readWriteFile", + "typeVersion": 1, + "position": [ + -8288, + 4640 + ], + "onError": "continueErrorOutput", + "notes": "อ่าน PDF Binary เพื่อเตรียม Upload" + }, + { + "parameters": { + "method": "POST", + "url": "={{$('Set Configuration').first().json.config.BACKEND_URL}}/api/files/upload", + "sendHeaders": true, + "headerParameters": { + "parameters": [ + { + "name": "Authorization", + "value": "={{$('Set Configuration').first().json.config.MIGRATION_TOKEN}}" + }, + { + "name": "Idempotency-Key", + "value": "={{ $json.batch_id + ':' + $json.document_number + ':upload' }}" + } + ] + }, + "sendBody": true, + "contentType": "multipart-form-data", + "bodyParameters": { + "parameters": [ + { + "parameterType": "formBinaryData", + "name": "file", + "inputDataFieldName": "data" + } + ] + }, + "options": { + "timeout": 60000 + } + }, + "id": "0346a124-aaf3-49de-9181-77cb5e80a4d3", + "name": "Upload PDF to Backend", + "type": "n8n-nodes-base.httpRequest", + "typeVersion": 4.2, + "position": [ + -8080, + 4400 + ], + "onError": "continueErrorOutput", + "notes": "ADR-023A: อัพโหลด PDF เข้า Backend Temp Storage → รับ temp_attachment_id" + }, + { + "parameters": { + "jsCode": "const config = $('Set Configuration').first().json.config;\nconst uploadResponse = $input.first()?.json || {};\nconst metaItem = $('File Validator').all()[$input.context?.itemIndex ?? 0]?.json || {};\nconst mountCheckData = $('File Mount Check').first()?.json || {};\n\nconst tempAttachmentPublicId = uploadResponse?.data?.publicId || uploadResponse?.publicId;\nconst tempAttachmentDbId = uploadResponse?.data?.id || uploadResponse?.id;\nif (!tempAttachmentPublicId) {\n throw new Error(`Upload failed — no temp_attachment_public_id returned for document: ${metaItem.document_number}`);\n}\n\n// สร้าง SubmitAiJobDto ตาม Backend DTO\nconst submitPayload = {\n type: 'migrate-document',\n payload: {\n tempAttachmentId: String(tempAttachmentPublicId),\n documentNumber: String(metaItem.document_number || ''),\n title: String(metaItem.title || ''),\n batchId: String(config.BATCH_ID),\n existingTags: mountCheckData.existing_tags || [],\n systemCategories: mountCheckData.system_categories || [],\n }\n};\n\n// Idempotency-Key: deterministic format ตาม FR-001a\nconst idempotencyKey = `${config.BATCH_ID}:${metaItem.document_number}`;\n\nreturn [{\n json: {\n ...metaItem,\n temp_attachment_public_id: tempAttachmentPublicId,\n temp_attachment_id: Number.isFinite(Number(tempAttachmentDbId)) ? Number(tempAttachmentDbId) : undefined,\n submit_payload: submitPayload,\n idempotency_key: idempotencyKey,\n }\n}];" + }, + "id": "1b7e6adb-b7e2-40dd-b472-3a5a79861bfb", + "name": "Build AI Job Payload", + "type": "n8n-nodes-base.code", + "typeVersion": 2, + "position": [ + -7888, + 4304 + ], + "notes": "สร้าง SubmitAiJobDto payload + Idempotency-Key ตาม FR-001a" + }, + { + "parameters": { + "method": "POST", + "url": "={{$('Set Configuration').first().json.config.BACKEND_URL}}/api/ai/jobs", + "sendHeaders": true, + "headerParameters": { + "parameters": [ + { + "name": "Authorization", + "value": "={{$('Set Configuration').first().json.config.MIGRATION_TOKEN}}" + }, + { + "name": "Idempotency-Key", + "value": "={{ $json.idempotency_key }}" + } + ] + }, + "sendBody": true, + "specifyBody": "json", + "jsonBody": "={{ $json.submit_payload }}", + "options": { + "timeout": 30000 + } + }, + "id": "f2a5ee75-cd14-4118-925d-43182735aa75", + "name": "Submit AI Job", + "type": "n8n-nodes-base.httpRequest", + "typeVersion": 4.1, + "position": [ + -7744, + 4384 + ], + "onError": "continueErrorOutput", + "notes": "ADR-023A: POST /api/ai/jobs — Backend จัดการ Ollama ผ่าน BullMQ" + }, + { + "parameters": { + "jsCode": "// Poll GET /api/ai/jobs/{jobId} ทุก 5 วินาที จน status = completed หรือ timeout 120 วินาที\nconst config = $('Set Configuration').first().json.config;\nconst submitResponse = $input.first()?.json || {};\nconst originalMeta = $('Build AI Job Payload').all()[$input.context?.itemIndex ?? 0]?.json || {};\n\nconst jobId = submitResponse?.jobId || submitResponse?.data?.jobId;\nif (!jobId) {\n // FR-010b: ถ้า 401 → mark TOKEN_EXPIRED\n if (submitResponse?.statusCode === 401 || submitResponse?.error?.status === 401) {\n throw new Error('TOKEN_EXPIRED: 401 Unauthorized — กรุณา renew MIGRATION_TOKEN แล้ว resume');\n }\n throw new Error(`Submit AI Job failed — no jobId returned for: ${originalMeta.document_number}`);\n}\n\nconst pollIntervalMs = config.AI_JOB_POLL_INTERVAL_MS || 5000;\nconst timeoutMs = config.AI_JOB_TIMEOUT_MS || 120000;\nconst startTime = Date.now();\n\nwhile (true) {\n if (Date.now() - startTime > timeoutMs) {\n throw new Error(`AI Job timeout after ${timeoutMs}ms for jobId: ${jobId}, document: ${originalMeta.document_number}`);\n }\n \n const statusResponse = await $helpers.httpRequest({\n method: 'GET',\n url: `${config.BACKEND_URL}/api/ai/jobs/${jobId}`,\n headers: { 'Authorization': config.MIGRATION_TOKEN },\n json: true\n });\n \n const status = statusResponse?.status || statusResponse?.data?.status;\n \n if (status === 'completed') {\n return [{\n json: {\n ...originalMeta,\n job_id: jobId,\n ai_result: statusResponse?.result || statusResponse?.data?.result || {},\n ai_status: 'completed',\n }\n }];\n }\n \n if (status === 'failed') {\n return [{\n json: {\n ...originalMeta,\n job_id: jobId,\n ai_status: 'failed',\n parse_error: `AI Job failed: ${JSON.stringify(statusResponse?.error || statusResponse?.data?.error || 'unknown')}`,\n error_type: 'AI_JOB_FAILED',\n route_index: 3\n }\n }];\n }\n \n // status = 'processing' — รอแล้ว poll ใหม่\n await new Promise(resolve => setTimeout(resolve, pollIntervalMs));\n}" + }, + "id": "f95bbb5f-6bde-4146-b577-8d824776029e", + "name": "Poll AI Job Status", + "type": "n8n-nodes-base.code", + "typeVersion": 2, + "position": [ + -7552, + 4368 + ], + "notes": "Poll GET /api/ai/jobs/{jobId} ทุก 5 วินาที (timeout 120 วินาที)" + }, + { + "parameters": { + "jsCode": "const config = $('Set Configuration').first().json.config;\nconst items = $input.all();\nconst results = [];\n\nconst CATEGORY_TO_TYPE_CODE = {\n 'Correspondence': 'LETTER',\n 'RFA': 'RFA',\n 'Transmittal': 'TRANSMITTAL',\n 'Drawing': 'OTHER',\n 'Report': 'OTHER',\n 'Other': 'OTHER',\n};\n\nfor (const item of items) {\n const data = item.json;\n \n // ถ้า poll ส่ง error กลับมาโดยตรง (route_index = 3)\n if (data.route_index === 3 || data.ai_status === 'failed') {\n results.push({ json: { ...data, route_index: 3 } });\n continue;\n }\n \n const ai = data.ai_result || {};\n \n if (!ai || typeof ai !== 'object') {\n results.push({ json: { ...data, parse_error: 'Empty or invalid ai_result', error_type: 'PARSE_ERROR', route_index: 3 } });\n continue;\n }\n \n // Enum Validation for Category\n const systemCategories = ['Correspondence','RFA','Drawing','Transmittal','Report','Other'];\n let finalCategory = ai.suggested_category || ai.category || 'Correspondence';\n if (!systemCategories.includes(finalCategory)) {\n finalCategory = String(data.document_number || '').includes('-RFA-') ? 'RFA' : 'Correspondence';\n }\n const typeCode = CATEGORY_TO_TYPE_CODE[finalCategory] || 'LETTER';\n \n // Tag normalization — is_new flag\n const suggestedTags = Array.isArray(ai.suggested_tags)\n ? ai.suggested_tags.map(t => ({\n tagName: String(t.tagName || t.tag_name || ''),\n publicId: t.publicId || t.public_id || undefined,\n isNew: Boolean(t.isNew ?? t.is_new ?? !t.publicId),\n colorCode: t.colorCode || t.color_code || 'default',\n })).filter(t => t.tagName)\n : [];\n \n const confidence = Number(ai.confidence || 0);\n \n const normalizedAi = {\n ...ai,\n suggested_category: finalCategory,\n type_code: typeCode,\n confidence,\n suggested_tags: suggestedTags,\n };\n \n let route_index;\n let review_reason = '';\n let reject_reason = '';\n \n if (confidence >= config.CONFIDENCE_HIGH && ai.is_valid !== false) {\n route_index = 0; // Auto Ready\n } else if (confidence >= config.CONFIDENCE_LOW) {\n route_index = 1; // Flagged\n review_reason = `Confidence ${confidence.toFixed(2)} < ${config.CONFIDENCE_HIGH}`;\n } else {\n route_index = 2; // Rejected\n reject_reason = ai.is_valid === false ? 'AI marked invalid' : `Confidence ${confidence.toFixed(2)} < ${config.CONFIDENCE_LOW}`;\n }\n \n results.push({\n json: {\n ...data,\n ai_result: normalizedAi,\n route_index,\n review_reason,\n reject_reason,\n }\n });\n}\n\nreturn results;" + }, + "id": "6bdcd7fd-0a8a-4f21-91bd-ed5fcbda860c", + "name": "Parse & Validate AI Response", + "type": "n8n-nodes-base.code", + "typeVersion": 2, + "position": [ + -7376, + 4368 + ], + "notes": "Parse AI result + Confidence routing prep + Tag normalization (isNew flag)" + }, + { + "parameters": { + "rules": { + "values": [ + { + "conditions": { + "options": { + "caseSensitive": true, + "leftValue": "", + "typeValidation": "strict", + "version": 2 + }, + "conditions": [ + { + "leftValue": "={{ $json.route_index }}", + "rightValue": 0, + "operator": { + "type": "number", + "operation": "equals", + "singleValue": true + } + } + ], + "combinator": "and" + }, + "renameOutput": true, + "outputKey": "Auto Ready" + }, + { + "conditions": { + "options": { + "caseSensitive": true, + "leftValue": "", + "typeValidation": "strict", + "version": 2 + }, + "conditions": [ + { + "leftValue": "={{ $json.route_index }}", + "rightValue": 1, + "operator": { + "type": "number", + "operation": "equals", + "singleValue": true + } + } + ], + "combinator": "and" + }, + "renameOutput": true, + "outputKey": "Review Flagged" + }, + { + "conditions": { + "options": { + "caseSensitive": true, + "leftValue": "", + "typeValidation": "strict", + "version": 2 + }, + "conditions": [ + { + "leftValue": "={{ $json.route_index }}", + "rightValue": 2, + "operator": { + "type": "number", + "operation": "equals", + "singleValue": true + } + } + ], + "combinator": "and" + }, + "renameOutput": true, + "outputKey": "Rejected" + }, + { + "conditions": { + "options": { + "caseSensitive": true, + "leftValue": "", + "typeValidation": "strict", + "version": 2 + }, + "conditions": [ + { + "leftValue": "={{ $json.route_index }}", + "rightValue": 3, + "operator": { + "type": "number", + "operation": "equals", + "singleValue": true + } + } + ], + "combinator": "and" + }, + "renameOutput": true, + "outputKey": "Error Log" + } + ] + }, + "options": {} + }, + "id": "35ad90cd-bb71-44a6-8851-ae4ea6ed4747", + "name": "Route by Confidence", + "type": "n8n-nodes-base.switch", + "typeVersion": 3.2, + "position": [ + -7216, + 4512 + ] + }, + { + "parameters": { + "method": "POST", + "url": "={{$('Set Configuration').first().json.config.BACKEND_URL}}/api/ai/migration/queue/record", + "sendHeaders": true, + "headerParameters": { + "parameters": [ + { + "name": "Authorization", + "value": "={{$('Set Configuration').first().json.config.MIGRATION_TOKEN}}" + }, + { + "name": "Idempotency-Key", + "value": "={{ $json.idempotency_key + ':queue' }}" + } + ] + }, + "sendBody": true, + "specifyBody": "json", + "jsonBody": "={{ JSON.stringify({ batchId: $json.batch_id, documentNumber: $json.document_number, subject: ($json.ai_result?.subject || $json.title), originalSubject: $json.title, tempAttachmentId: $json.temp_attachment_id === undefined ? undefined : Number($json.temp_attachment_id), confidence: $json.ai_result?.confidence, reviewReason: $json.review_reason || '', status: 'PENDING', aiResult: $json.ai_result, idempotencyKey: $json.batch_id + ':' + $json.document_number }) }}", + "options": { + "timeout": 10000 + } + }, + "id": "f26c6512-8f2d-4cca-a643-65ae7255c37a", + "name": "Insert Review Queue (Auto)", + "type": "n8n-nodes-base.httpRequest", + "typeVersion": 4.1, + "position": [ + -6976, + 4496 + ], + "onError": "continueErrorOutput", + "notes": "POST /api/ai/migration/queue/record (PENDING) — ADR-023A" + }, + { + "parameters": { + "method": "POST", + "url": "={{$('Set Configuration').first().json.config.BACKEND_URL}}/api/ai/migration/queue/record", + "sendHeaders": true, + "headerParameters": { + "parameters": [ + { + "name": "Authorization", + "value": "={{$('Set Configuration').first().json.config.MIGRATION_TOKEN}}" + }, + { + "name": "Idempotency-Key", + "value": "={{ $json.idempotency_key + ':queue' }}" + } + ] + }, + "sendBody": true, + "specifyBody": "json", + "jsonBody": "={{ JSON.stringify({ batchId: $json.batch_id, documentNumber: $json.document_number, subject: ($json.ai_result?.subject || $json.title), originalSubject: $json.title, tempAttachmentId: $json.temp_attachment_id === undefined ? undefined : Number($json.temp_attachment_id), confidence: $json.ai_result?.confidence, reviewReason: $json.review_reason || '', status: 'PENDING_REVIEW', aiResult: $json.ai_result, idempotencyKey: $json.batch_id + ':' + $json.document_number }) }}", + "options": { + "timeout": 10000 + } + }, + "id": "b720ce71-d0f4-402a-9bd3-077cf1730977", + "name": "Insert Review Queue (Flagged)", + "type": "n8n-nodes-base.httpRequest", + "typeVersion": 4.1, + "position": [ + -6944, + 4720 + ], + "onError": "continueErrorOutput", + "notes": "POST /api/ai/migration/queue/record (PENDING_REVIEW) — ADR-023A" + }, + { + "parameters": { + "jsCode": "const fs = require('fs');\nconst item = $input.first();\nconst config = $('Set Configuration').first().json.config;\n\nconst csvPath = `${config.LOG_PATH}/reject_log_${config.BATCH_ID}.csv`;\nconst header = 'timestamp,document_number,title,reject_reason,ai_confidence,job_id\\n';\nconst esc = (s) => `\"${String(s || '').replace(/\"/g, '\"\"')}\"`;\n\nif (!fs.existsSync(csvPath)) {\n fs.writeFileSync(csvPath, header, 'utf8');\n}\n\nconst line = [\n new Date().toISOString(),\n esc(item.json.document_number),\n esc(item.json.title),\n esc(item.json.reject_reason),\n item.json.ai_result?.confidence ?? 'N/A',\n esc(item.json.job_id || '')\n].join(',') + '\\n';\n\nfs.appendFileSync(csvPath, line, 'utf8');\n\nreturn [$input.first()];" + }, + "id": "6e83de99-1244-4478-9a33-e779cdb9504a", + "name": "Log Reject to CSV", + "type": "n8n-nodes-base.code", + "typeVersion": 2, + "position": [ + -6944, + 4848 + ], + "notes": "บันทึกรายการที่ถูกปฏิเสธลง CSV" + }, + { + "parameters": { + "jsCode": "const fs = require('fs');\nconst items = $input.all();\nconst config = $('Set Configuration').first().json.config;\n\nconst csvPath = `${config.LOG_PATH}/error_log_${config.BATCH_ID}.csv`;\nconst header = 'timestamp,document_number,error_type,error_message,job_id\\n';\nconst esc = (s) => `\"${String(s || '').replace(/\"/g, '\"\"')}\"`;\n\nif (!fs.existsSync(csvPath)) {\n fs.writeFileSync(csvPath, header, 'utf8');\n}\n\nfor (const item of items) {\n const line = [\n new Date().toISOString(),\n esc(item.json.document_number),\n esc(item.json.error_type || 'UNKNOWN'),\n esc(item.json.error || item.json.parse_error),\n esc(item.json.job_id || '')\n ].join(',') + '\\n';\n fs.appendFileSync(csvPath, line, 'utf8');\n}\n\nreturn items;" + }, + "id": "511428dc-3aad-4de1-a9dc-9a87c791371e", + "name": "Log Error to CSV", + "type": "n8n-nodes-base.code", + "typeVersion": 2, + "position": [ + -7936, + 4800 + ], + "notes": "บันทึก Error ลง CSV" + }, + { + "parameters": { + "method": "POST", + "url": "={{$('Set Configuration').first().json.config.BACKEND_URL}}/api/ai/migration/errors", + "sendHeaders": true, + "headerParameters": { + "parameters": [ + { + "name": "Authorization", + "value": "={{$('Set Configuration').first().json.config.MIGRATION_TOKEN}}" + }, + { + "name": "Idempotency-Key", + "value": "={{ ($json.batch_id || $('Set Configuration').first().json.config.BATCH_ID) + ':' + ($json.document_number || 'WORKFLOW') + ':error' }}" + } + ] + }, + "sendBody": true, + "specifyBody": "json", + "jsonBody": "={{ JSON.stringify({ batchId: $('Set Configuration').first().json.config.BATCH_ID, documentNumber: $json.document_number || 'WORKFLOW', errorType: $json.error_type || 'UNKNOWN', errorMessage: $json.error || $json.parse_error || $json.message || '', jobId: $json.job_id || '' }) }}", + "options": { + "timeout": 10000 + } + }, + "id": "7e8f3617-a6e4-4d40-922d-eb93ae91e690", + "name": "Log Error to DB", + "type": "n8n-nodes-base.httpRequest", + "typeVersion": 4.1, + "position": [ + -7232, + 4944 + ], + "onError": "continueErrorOutput", + "notes": "POST /api/ai/migration/errors — ADR-023A" + }, + { + "parameters": { + "method": "POST", + "url": "={{$('Set Configuration').first().json.config.BACKEND_URL}}/api/ai/migration/checkpoint", + "sendHeaders": true, + "headerParameters": { + "parameters": [ + { + "name": "Authorization", + "value": "={{$('Set Configuration').first().json.config.MIGRATION_TOKEN}}" + }, + { + "name": "Idempotency-Key", + "value": "={{ $('Set Configuration').first().json.config.BATCH_ID + ':checkpoint:' + (($json.original_index || 0) + 1) }}" + } + ] + }, + "sendBody": true, + "specifyBody": "json", + "jsonBody": "={{ JSON.stringify({ batchId: $('Set Configuration').first().json.config.BATCH_ID, lastProcessedIndex: ($json.original_index || 0) + 1, status: 'RUNNING' }) }}", + "options": { + "timeout": 10000 + } + }, + "id": "9471990c-1abe-42f8-8062-bd68cb9ad985", + "name": "Save Checkpoint", + "type": "n8n-nodes-base.httpRequest", + "typeVersion": 4.1, + "position": [ + -6784, + 4640 + ], + "onError": "continueErrorOutput", + "notes": "POST /api/ai/migration/checkpoint — ADR-023A" + }, + { + "parameters": { + "amount": "={{$('Set Configuration').first().json.config.DELAY_MS / 1000}}", + "unit": "seconds" + }, + "id": "fcf3e098-93f6-42ee-b930-aa0bc84d3ed7", + "name": "Delay", + "type": "n8n-nodes-base.wait", + "typeVersion": 1, + "position": [ + -6640, + 4880 + ], + "webhookId": "38e97a99-4dcc-4b63-977a-a02945a1c369", + "notes": "หน่วงเวลาระหว่าง Records" + }, + { + "parameters": { + "rules": { + "values": [ + { + "conditions": { + "options": { + "caseSensitive": true, + "leftValue": "", + "typeValidation": "strict", + "version": 2 + }, + "conditions": [ + { + "leftValue": "={{ $json.batch_complete }}", + "rightValue": true, + "operator": { + "type": "boolean", + "operation": "equals", + "singleValue": true + } + } + ], + "combinator": "and" + }, + "renameOutput": true, + "outputKey": "Batch Complete" + }, + { + "conditions": { + "options": { + "caseSensitive": true, + "leftValue": "", + "typeValidation": "strict", + "version": 2 + }, + "conditions": [ + { + "leftValue": "={{ $json.batch_complete }}", + "rightValue": false, + "operator": { + "type": "boolean", + "operation": "equals", + "singleValue": true + } + } + ], + "combinator": "and" + }, + "renameOutput": true, + "outputKey": "Continue Loop" + } + ] + }, + "options": {} + }, + "id": "e489d28d-37e8-4204-bba8-07a5226b1275", + "name": "Check Batch Complete", + "type": "n8n-nodes-base.switch", + "typeVersion": 3.2, + "position": [ + -8512, + 4608 + ] + }, + { + "parameters": { + "method": "POST", + "url": "={{$('Set Configuration').first().json.config.BACKEND_URL}}/api/ai/migration/checkpoint", + "sendHeaders": true, + "headerParameters": { + "parameters": [ + { + "name": "Authorization", + "value": "={{$('Set Configuration').first().json.config.MIGRATION_TOKEN}}" + }, + { + "name": "Idempotency-Key", + "value": "={{ $('Set Configuration').first().json.config.BATCH_ID + ':complete' }}" + } + ] + }, + "sendBody": true, + "specifyBody": "json", + "jsonBody": "={{ JSON.stringify({ batchId: $('Set Configuration').first().json.config.BATCH_ID, lastProcessedIndex: $json.total_processed, status: 'COMPLETED' }) }}", + "options": { + "timeout": 10000 + } + }, + "id": "171b627f-3b00-4fbf-86b7-6076fdc29d19", + "name": "Mark Batch Complete", + "type": "n8n-nodes-base.httpRequest", + "typeVersion": 4.1, + "position": [ + -8304, + 4400 + ], + "notes": "Update checkpoint status to COMPLETED when batch finishes" + } + ], + "pinData": {}, + "connections": { + "Form Trigger": { + "main": [ + [ + { + "node": "Set Configuration", + "type": "main", + "index": 0 + } + ] + ] + }, + "Set Configuration": { + "main": [ + [ + { + "node": "Check Backend Health", + "type": "main", + "index": 0 + } + ] + ] + }, + "Check Backend Health": { + "main": [ + [ + { + "node": "Validate Token", + "type": "main", + "index": 0 + } + ] + ] + }, + "Validate Token": { + "main": [ + [ + { + "node": "Fetch Categories", + "type": "main", + "index": 0 + } + ] + ] + }, + "Fetch Categories": { + "main": [ + [ + { + "node": "Fetch Tags", + "type": "main", + "index": 0 + } + ] + ] + }, + "Fetch Tags": { + "main": [ + [ + { + "node": "File Mount Check", + "type": "main", + "index": 0 + } + ] + ] + }, + "File Mount Check": { + "main": [ + [ + { + "node": "Read Excel Binary", + "type": "main", + "index": 0 + } + ] + ] + }, + "Read Excel Binary": { + "main": [ + [ + { + "node": "Read Excel", + "type": "main", + "index": 0 + } + ] + ] + }, + "Read Excel": { + "main": [ + [ + { + "node": "Read Checkpoint", + "type": "main", + "index": 0 + } + ] + ] + }, + "Read Checkpoint": { + "main": [ + [ + { + "node": "Process Batch + Encoding", + "type": "main", + "index": 0 + } + ] + ] + }, + "Process Batch + Encoding": { + "main": [ + [ + { + "node": "Check Batch Complete", + "type": "main", + "index": 0 + } + ] + ] + }, + "File Validator": { + "main": [ + [ + { + "node": "Read PDF File", + "type": "main", + "index": 0 + } + ] + ] + }, + "Read PDF File": { + "main": [ + [ + { + "node": "Upload PDF to Backend", + "type": "main", + "index": 0 + } + ], + [ + { + "node": "Log Error to CSV", + "type": "main", + "index": 0 + } + ] + ] + }, + "Upload PDF to Backend": { + "main": [ + [ + { + "node": "Build AI Job Payload", + "type": "main", + "index": 0 + } + ], + [ + { + "node": "Log Error to CSV", + "type": "main", + "index": 0 + } + ] + ] + }, + "Build AI Job Payload": { + "main": [ + [ + { + "node": "Submit AI Job", + "type": "main", + "index": 0 + } + ] + ] + }, + "Submit AI Job": { + "main": [ + [ + { + "node": "Poll AI Job Status", + "type": "main", + "index": 0 + } + ], + [ + { + "node": "Log Error to CSV", + "type": "main", + "index": 0 + } + ] + ] + }, + "Poll AI Job Status": { + "main": [ + [ + { + "node": "Parse & Validate AI Response", + "type": "main", + "index": 0 + } + ] + ] + }, + "Parse & Validate AI Response": { + "main": [ + [ + { + "node": "Route by Confidence", + "type": "main", + "index": 0 + } + ] + ] + }, + "Route by Confidence": { + "main": [ + [ + { + "node": "Insert Review Queue (Auto)", + "type": "main", + "index": 0 + } + ], + [ + { + "node": "Insert Review Queue (Flagged)", + "type": "main", + "index": 0 + } + ], + [ + { + "node": "Log Reject to CSV", + "type": "main", + "index": 0 + } + ], + [ + { + "node": "Log Error to CSV", + "type": "main", + "index": 0 + } + ] + ] + }, + "Insert Review Queue (Auto)": { + "main": [ + [ + { + "node": "Save Checkpoint", + "type": "main", + "index": 0 + } + ], + [ + { + "node": "Log Error to CSV", + "type": "main", + "index": 0 + } + ] + ] + }, + "Insert Review Queue (Flagged)": { + "main": [ + [ + { + "node": "Save Checkpoint", + "type": "main", + "index": 0 + } + ], + [ + { + "node": "Log Error to CSV", + "type": "main", + "index": 0 + } + ] + ] + }, + "Save Checkpoint": { + "main": [ + [ + { + "node": "Delay", + "type": "main", + "index": 0 + } + ], + [ + { + "node": "Log Error to CSV", + "type": "main", + "index": 0 + } + ] + ] + }, + "Delay": { + "main": [ + [ + { + "node": "Read Checkpoint", + "type": "main", + "index": 0 + } + ] + ] + }, + "Check Batch Complete": { + "main": [ + [ + { + "node": "Mark Batch Complete", + "type": "main", + "index": 0 + } + ], + [ + { + "node": "File Validator", + "type": "main", + "index": 0 + } + ] + ] + }, + "Log Reject to CSV": { + "main": [ + [ + { + "node": "Delay", + "type": "main", + "index": 0 + } + ] + ] + }, + "Log Error to CSV": { + "main": [ + [ + { + "node": "Log Error to DB", + "type": "main", + "index": 0 + } + ] + ] + }, + "Log Error to DB": { + "main": [ + [ + { + "node": "Delay", + "type": "main", + "index": 0 + } + ], + [ + { + "node": "Delay", + "type": "main", + "index": 0 + } + ] + ] + } + }, + "active": false, + "settings": { + "executionOrder": "v1", + "binaryMode": "separate" + }, + "versionId": "efadd20e-a46b-4354-8f75-ec1de215d065", + "meta": { + "templateCredsSetupCompleted": true, + "instanceId": "9e70e47c1eaf3bac72f497ddfbde0983f840f7d0f059537f7e37dd70de18ecb7" + }, + "id": "4LlPbAKU5BZLgiTg", + "tags": [ + { + "updatedAt": "2026-05-23T12:26:47.389Z", + "createdAt": "2026-05-23T12:26:47.389Z", + "id": "jNSEtctPbU5leFPw", + "name": "v2" + }, + { + "updatedAt": "2026-05-23T12:26:47.393Z", + "createdAt": "2026-05-23T12:26:47.393Z", + "id": "mGZTyPfxbcsAuFpR", + "name": "migration" + } + ] +} \ No newline at end of file diff --git a/specs/03-Data-and-Storage/n8n.workflow.v2.json b/specs/03-Data-and-Storage/n8n.workflow.v2.json index 767e4369..ed42f0ce 100644 --- a/specs/03-Data-and-Storage/n8n.workflow.v2.json +++ b/specs/03-Data-and-Storage/n8n.workflow.v2.json @@ -20,28 +20,28 @@ }, "options": {} }, - "id": "4609ab68-f7e4-4800-ad39-19ce32de60d0", + "id": "a0346819-4e97-4208-99c8-e4f958d652fe", "name": "Form Trigger", "type": "n8n-nodes-base.formTrigger", "typeVersion": 2.2, "position": [ - 31024, - 13504 + -9280, + 4400 ], - "webhookId": "e164a362-0c6b-4243-a5ad-b325aa943f4f", + "webhookId": "dd44ab55-df6b-4f3b-a740-6a993ca7ded0", "notes": "เปิด URL เพื่อตั้งค่าก่อนรัน" }, { "parameters": { - "jsCode": "// ============================================\n// CONFIGURATION v2.0 — ADR-023A Compliant\n// n8n เรียกแค่ DMS Backend API เท่านั้น — ห้ามเรียก Ollama โดยตรง\n// ============================================\nconst formData = $('Form Trigger').first()?.json || {};\nconst batchSizeInput = parseInt(String(formData['Batch Size'] || '0'));\nconst excelFileInput = String(formData['Excel File Path'] || '').trim();\nconst jwtTokenInput = 'Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VybmFtZSI6Im1pZ3JhdGlvbl9ib3QiLCJzdWIiOjUsInNjb3BlIjoiR2xvYmFsIiwiaWF0IjoxNzc5NTQwNDk4LCJleHAiOjQ5MzUzMDA0OTh9.TZVzG8u4Cyz8hi0cU3eGaeXinKWKHN3HtYnDcS5p_5I';\nconst resolvedExcelFile = excelFileInput || '/home/node/.n8n-files/staging_ai/C22024.xlsx';\n\n// Validate Excel file path to prevent path traversal\nif (resolvedExcelFile.includes('..') || resolvedExcelFile.includes('~')) {\n throw new Error('Invalid Excel file path: path traversal detected');\n}\n\nif (!jwtTokenInput || !jwtTokenInput.startsWith('Bearer ')) {\n throw new Error('JWT Token is required and must start with \"Bearer \"');\n}\n\nconst BATCH_ID = (() => {\n // ใช้ Excel filename เป็น BATCH_ID เพื่อให้ checkpoint ทำงานถูกต้อง\n // เปลี่ยน Excel file = BATCH_ID ใหม่ = เริ่มใหม่ (ปลอดภัย)\n // ใช้ Excel file เดิม = BATCH_ID เดิม = resume ได้\n const filename = resolvedExcelFile.split('/').pop().split('\\\\').pop();\n const baseName = filename.replace(/\\.xlsx?$/i, '');\n return baseName + '-MIGRATION';\n})();\n\nconst CONFIG = {\n // Backend Settings\n BACKEND_URL: 'https://backend.np-dms.work',\n MIGRATION_TOKEN: jwtTokenInput,\n\n // Batch Settings\n BATCH_SIZE: batchSizeInput > 0 ? batchSizeInput : 10,\n BATCH_ID: BATCH_ID,\n DELAY_MS: 20000,\n\n // AI Job Settings (ADR-023A)\n AI_JOB_POLL_INTERVAL_MS: 5000,\n AI_JOB_TIMEOUT_MS: 120000,\n\n // Confidence Thresholds\n CONFIDENCE_HIGH: 0.85,\n CONFIDENCE_LOW: 0.60,\n\n // Paths (Container paths — ต้องตรงกับ volume mount ใน docker-compose)\n EXCEL_FILE: resolvedExcelFile,\n SOURCE_PDF_DIR: '/home/node/.n8n-files/staging_ai',\n LOG_PATH: '/home/node/.n8n-files/migration_logs',\n};\n\nreturn [{ json: { config: CONFIG } }];" + "jsCode": "// ============================================\n// CONFIGURATION v2.0 — ADR-023A Compliant\n// n8n เรียกแค่ DMS Backend API เท่านั้น — ห้ามเรียก Ollama โดยตรง\n// ============================================\nconst formData = $('Form Trigger').first()?.json || {};\nconst batchSizeInput = parseInt(String(formData['Batch Size'] || '0'));\nconst excelFileInput = String(formData['Excel File Path'] || '').trim();\nconst jwtTokenInput = 'Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VybmFtZSI6Im1pZ3JhdGlvbl9ib3QiLCJzdWIiOjUsInNjb3BlIjoiR2xvYmFsIiwiaWF0IjoxNzc5NTQwNDk4LCJleHAiOjQ5MzUzMDA0OTh9.TZVzG8u4Cyz8hi0cU3eGaeXinKWKHN3HtYnDcS5p_5I';\nconst resolvedExcelFile = excelFileInput || '/home/node/.n8n-files/staging_ai/C22024.xlsx';\n\n// Validate Excel file path to prevent path traversal\nif (resolvedExcelFile.includes('..') || resolvedExcelFile.includes('~')) {\n throw new Error('Invalid Excel file path: path traversal detected');\n}\n\nif (!jwtTokenInput || !jwtTokenInput.startsWith('Bearer ')) {\n throw new Error('JWT Token is required and must start with \"Bearer \"');\n}\n\nconst BATCH_ID = (() => {\n // ใช้ Excel filename เป็น BATCH_ID เพื่อให้ checkpoint ทำงานถูกต้อง\n // เปลี่ยน Excel file = BATCH_ID ใหม่ = เริ่มใหม่ (ปลอดภัย)\n // ใช้ Excel file เดิม = BATCH_ID เดิม = resume ได้\n const filename = resolvedExcelFile.split('/').pop().split('\\\\').pop();\n const baseName = filename.replace(/\\.xlsx?$/i, '');\n return baseName + '-MIGRATION';\n})();\n\nconst CONFIG = {\n // Backend Settings\n BACKEND_URL: 'http://backend:3000',\n MIGRATION_TOKEN: jwtTokenInput,\n\n // Batch Settings\n BATCH_SIZE: batchSizeInput > 0 ? batchSizeInput : 10,\n BATCH_ID: BATCH_ID,\n DELAY_MS: 60000,\n\n // AI Job Settings (ADR-023A)\n AI_JOB_POLL_INTERVAL_MS: 5000,\n AI_JOB_TIMEOUT_MS: 120000,\n\n // Confidence Thresholds\n CONFIDENCE_HIGH: 0.85,\n CONFIDENCE_LOW: 0.60,\n\n // Paths (Container paths — ต้องตรงกับ volume mount ใน docker-compose)\n EXCEL_FILE: resolvedExcelFile,\n SOURCE_PDF_DIR: '/home/node/.n8n-files/staging_ai/Incoming/08C.2/2567',\n LOG_PATH: '/home/node/.n8n-files/migration_logs',\n};\n\nreturn [{ json: { config: CONFIG } }];" }, - "id": "8f1d3378-cca6-48b6-99db-693e46ac81ef", + "id": "65143f3f-b0ee-45e6-b0f9-bf57546b7482", "name": "Set Configuration", "type": "n8n-nodes-base.code", "typeVersion": 2, "position": [ - 31216, - 13504 + -9088, + 4400 ], "notes": "กำหนดค่า Configuration ทั้งหมด — แก้ไขที่นี่เท่านั้น (ไม่มี Ollama config แล้ว)" }, @@ -52,13 +52,13 @@ "timeout": 5000 } }, - "id": "60e81de6-e9b2-4bff-afcc-bef9d5b959b5", + "id": "b6295d67-9a34-4550-8e96-096a59a88053", "name": "Check Backend Health", "type": "n8n-nodes-base.httpRequest", "typeVersion": 4.1, "position": [ - 31392, - 13504 + -8912, + 4400 ], "notes": "ตรวจสอบ Backend พร้อมใช้งาน" }, @@ -78,13 +78,13 @@ "timeout": 5000 } }, - "id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890", + "id": "92b9007e-9ad1-40f6-8e37-cec5113f6b30", "name": "Validate Token", "type": "n8n-nodes-base.httpRequest", "typeVersion": 4.1, "position": [ - 31392, - 13696 + -8736, + 4400 ], "notes": "FR-010a: ตรวจสอบ MIGRATION_TOKEN ก่อนเริ่ม Batch" }, @@ -104,13 +104,13 @@ "timeout": 10000 } }, - "id": "6c6679b4-85f3-4c2c-ac8e-4281d6ae61f6", + "id": "a9c11f42-b63b-412f-b919-88dbc7ab095e", "name": "Fetch Categories", "type": "n8n-nodes-base.httpRequest", "typeVersion": 4.1, "position": [ - 31568, - 13504 + -9088, + 4592 ], "notes": "ดึง Categories จาก Backend" }, @@ -130,13 +130,13 @@ "timeout": 10000 } }, - "id": "98b9159a-f21d-4b33-9524-058a78ccfc93", + "id": "108423e3-90de-49a3-b950-8f78cd231b84", "name": "Fetch Tags", "type": "n8n-nodes-base.httpRequest", "typeVersion": 4.1, "position": [ - 31568, - 13696 + -8912, + 4592 ], "notes": "ดึง Tags ที่มีอยู่แล้วจาก Backend" }, @@ -144,13 +144,13 @@ "parameters": { "jsCode": "const fs = require('fs');\nconst config = $('Set Configuration').first().json.config;\n\ntry {\n if (!fs.existsSync(config.EXCEL_FILE)) {\n throw new Error(`Excel file not found at: ${config.EXCEL_FILE}`);\n }\n if (!fs.existsSync(config.SOURCE_PDF_DIR)) {\n throw new Error(`PDF Source directory not found at: ${config.SOURCE_PDF_DIR}`);\n }\n \n const files = fs.readdirSync(config.SOURCE_PDF_DIR);\n \n // Check write permission to log path\n fs.writeFileSync(`${config.LOG_PATH}/.preflight_ok`, new Date().toISOString());\n \n // Grab categories\n let categories = ['Correspondence','RFA','Drawing','Transmittal','Report','Other'];\n try {\n const upstreamData = $('Fetch Categories').first()?.json?.data;\n if (upstreamData && Array.isArray(upstreamData)) {\n categories = upstreamData.map(c => c.name || c.type || c);\n }\n } catch(e) {}\n \n // Grab existing tags\n let existingTags = [];\n try {\n const tagData = $('Fetch Tags').first()?.json?.data || [];\n existingTags = Array.isArray(tagData)\n ? tagData.map(t => ({ publicId: t.publicId, tagName: t.tag_name || t.name || '' })).filter(t => t.tagName && t.publicId)\n : [];\n } catch(e) {}\n \n return [{ json: { \n preflight_ok: true, \n pdf_count_in_source: files.length,\n excel_target: config.EXCEL_FILE,\n system_categories: categories,\n existing_tags: existingTags,\n timestamp: new Date().toISOString()\n }}];\n} catch (err) {\n throw new Error(`Pre-flight check failed: ${err.message}`);\n}" }, - "id": "910b13e2-994a-4fb6-bca1-637e1628c586", + "id": "d8e2efec-7fb4-44c3-b544-0dc51dcfd3fc", "name": "File Mount Check", "type": "n8n-nodes-base.code", "typeVersion": 2, "position": [ - 31744, - 13504 + -8736, + 4592 ], "notes": "ตรวจสอบ File System + รวบรวม master data สำหรับ AI payload" }, @@ -159,13 +159,13 @@ "fileSelector": "={{ $json.excel_target }}", "options": {} }, - "id": "063bcef1-791a-4923-a659-8b9a0ba3e336", + "id": "c2a6ca17-aae8-4213-93ab-8179e1febfb3", "name": "Read Excel Binary", "type": "n8n-nodes-base.readWriteFile", "typeVersion": 1, "position": [ - 31920, - 13504 + -9088, + 4784 ], "notes": "ดึงไฟล์ Excel ขึ้นมาไว้ในหน่วยความจำ" }, @@ -173,13 +173,13 @@ "parameters": { "options": {} }, - "id": "e07efdde-b9b1-402a-ba01-44175982749b", + "id": "ebfdde99-92a1-4060-9867-2f330b13554f", "name": "Read Excel", "type": "n8n-nodes-base.spreadsheetFile", "typeVersion": 2, "position": [ - 31920, - 13696 + -8912, + 4784 ], "notes": "แปลงข้อมูล Excel เป็น JSON Data" }, @@ -196,19 +196,16 @@ ] }, "options": { - "retryOnFail": true, - "maxTries": 3, - "waitBetweenTries": 2000, "timeout": 10000 } }, - "id": "a83f8598-72fd-4cc8-9d98-1ea3cb3b42df", + "id": "9cf3971c-287a-42ab-9327-72e1a4d8011c", "name": "Read Checkpoint", "type": "n8n-nodes-base.httpRequest", "typeVersion": 4.1, "position": [ - 32096, - 13504 + -8720, + 4784 ], "alwaysOutputData": true, "notes": "GET /api/ai/migration/checkpoint/:batchId — ADR-023A (ไม่ใช้ MySQL โดยตรง)" @@ -217,13 +214,13 @@ "parameters": { "jsCode": "const cpJson = $input.first()?.json || {};\nconst startIndex = cpJson.data?.lastProcessedIndex ?? cpJson.lastProcessedIndex ?? cpJson.data?.last_processed_index ?? cpJson.last_processed_index ?? 0;\nconst config = $('Set Configuration').first().json.config;\n\nconst allItems = $('Read Excel').all().map(i => i.json);\nconst remaining = allItems.slice(startIndex);\nconst currentBatch = remaining.slice(0, config.BATCH_SIZE);\n\n// Exit condition: ไม่มี records เหลือ\nif (currentBatch.length === 0) {\n return [{ json: { batch_complete: true, total_processed: startIndex } }];\n}\n\nconst normalize = (str) => {\n if (!str) return '';\n return String(str).normalize('NFC').trim();\n};\n\nreturn currentBatch.map((item, i) => {\n const getVal = (possibleKeys) => {\n const exactMatch = possibleKeys.find(k => item[k] !== undefined);\n if (exactMatch) return item[exactMatch];\n const lowerTrimmedKeys = Object.keys(item).map(k => ({ original: k, parsed: k.toLowerCase().trim() }));\n for (const pk of possibleKeys) {\n const found = lowerTrimmedKeys.find(k => k.parsed === pk.toLowerCase().trim());\n if (found) return item[found.original];\n }\n return '';\n };\n\n const docNum = getVal(['document_number', 'correspondence_number', 'Document Number', 'Corr. No.']);\n const excelFileName = getVal(['File name', 'file_name', 'File Name', 'filename']);\n \n if (!excelFileName) {\n throw new Error(`Missing 'File name' column for row ${i + startIndex + 1}, document: ${docNum}`);\n }\n \n return {\n json: {\n batch_complete: false,\n document_number: normalize(docNum),\n title: normalize(getVal(['title', 'Title', 'Subject', 'subject'])),\n legacy_number: normalize(getVal(['legacy_number', 'Legacy Number', 'Response Doc.'])),\n excel_revision: getVal(['revision', 'Revision', 'rev']) || 1,\n original_index: startIndex + i,\n batch_id: config.BATCH_ID,\n file_name: normalize(excelFileName),\n issued_date: normalize(getVal(['issued_date', 'Issued_date', 'Issued Date', 'date', 'Date'])),\n received_date: normalize(getVal(['received_date', 'Received_date', 'Received Date'])),\n sender: normalize(getVal(['sender', 'Sender', 'From', 'from'])),\n receiver: normalize(getVal(['receiver', 'Receiver', 'To', 'to'])),\n }\n };\n});" }, - "id": "80845e32-c283-4e9f-af73-6339d675fb38", + "id": "c101543e-a3a2-4538-a99c-0419df50948f", "name": "Process Batch + Encoding", "type": "n8n-nodes-base.code", "typeVersion": 2, "position": [ - 32272, - 13504 + -8528, + 4400 ], "alwaysOutputData": true, "notes": "ตัด Batch + Normalize UTF-8 + Exit condition เมื่อ records หมด" @@ -232,13 +229,13 @@ "parameters": { "jsCode": "const fs = require('fs');\nconst path = require('path');\nconst config = $('Set Configuration').first().json.config;\n\nconst items = $input.all();\nif (!items || items.length === 0) return [];\n\n// ตรวจสอบ exit condition\nif (items[0]?.json?.batch_complete === true) {\n return items;\n}\n\nconst validated = [];\nconst errors = [];\n\nfor (const item of items) {\n const fileName = item.json?.file_name;\n if (!fileName) {\n errors.push({ ...item, json: { ...item.json, file_valid: false, error: 'file_name is missing', error_type: 'MISSING_FILENAME' } });\n continue;\n }\n \n let safeName = path.basename(String(fileName)).normalize('NFC');\n if (!safeName.toLowerCase().endsWith('.pdf')) safeName += '.pdf';\n const filePath = path.resolve(config.SOURCE_PDF_DIR, safeName);\n \n if (!filePath.startsWith(path.resolve(config.SOURCE_PDF_DIR))) {\n errors.push({ ...item, json: { ...item.json, file_valid: false, error: 'Path traversal detected', error_type: 'SECURITY' } });\n continue;\n }\n \n try {\n if (fs.existsSync(filePath)) {\n const stats = fs.statSync(filePath);\n validated.push({ ...item, json: { ...item.json, file_valid: true, file_path: filePath, file_size: stats.size } });\n } else {\n errors.push({ ...item, json: { ...item.json, file_valid: false, error: `File not found: ${safeName}`, error_type: 'FILE_NOT_FOUND' } });\n }\n } catch (err) {\n errors.push({ ...item, json: { ...item.json, file_valid: false, error: err.message, error_type: 'FILE_ERROR' } });\n }\n}\n\nreturn [...validated, ...errors];" }, - "id": "2183d687-4708-4d77-a0a9-13ccf29baf69", + "id": "282e1c8a-74b6-4ae7-a50b-3e2ff7b1558d", "name": "File Validator", "type": "n8n-nodes-base.code", "typeVersion": 2, "position": [ - 32448, - 13504 + -8512, + 4800 ], "notes": "ตรวจสอบไฟล์ PDF ใน Directory" }, @@ -247,13 +244,13 @@ "fileSelector": "={{ $json.file_path }}", "options": {} }, - "id": "4fd3133e-39e1-4860-95c7-3e87ee43ed51", + "id": "ab2d3ff8-9862-4a5c-a21d-5e04004d6a40", "name": "Read PDF File", "type": "n8n-nodes-base.readWriteFile", "typeVersion": 1, "position": [ - 32624, - 13504 + -8288, + 4640 ], "onError": "continueErrorOutput", "notes": "อ่าน PDF Binary เพื่อเตรียม Upload" @@ -287,34 +284,31 @@ ] }, "options": { - "retryOnFail": true, - "maxTries": 3, - "waitBetweenTries": 2000, - "timeout": 60000 + "timeout": 300000 } }, - "id": "b1c2d3e4-f5a6-7890-bcde-f12345678901", + "id": "0346a124-aaf3-49de-9181-77cb5e80a4d3", "name": "Upload PDF to Backend", "type": "n8n-nodes-base.httpRequest", "typeVersion": 4.2, "position": [ - 32800, - 13504 + -8080, + 4400 ], "onError": "continueErrorOutput", "notes": "ADR-023A: อัพโหลด PDF เข้า Backend Temp Storage → รับ temp_attachment_id" }, { "parameters": { - "jsCode": "const config = $('Set Configuration').first().json.config;\nconst uploadResponse = $input.first()?.json || {};\nconst metaItem = $('File Validator').all()[$input.context?.itemIndex ?? 0]?.json || {};\nconst mountCheckData = $('File Mount Check').first()?.json || {};\n\nconst tempAttachmentPublicId = uploadResponse?.data?.publicId || uploadResponse?.publicId;\nconst tempAttachmentDbId = uploadResponse?.data?.id || uploadResponse?.id;\nif (!tempAttachmentPublicId) {\n throw new Error(`Upload failed — no temp_attachment_public_id returned for document: ${metaItem.document_number}`);\n}\n\n// สร้าง SubmitAiJobDto ตาม Backend DTO\nconst submitPayload = {\n type: 'migrate-document',\n payload: {\n tempAttachmentId: String(tempAttachmentPublicId),\n documentNumber: String(metaItem.document_number || ''),\n title: String(metaItem.title || ''),\n batchId: String(config.BATCH_ID),\n existingTags: mountCheckData.existing_tags || [],\n systemCategories: mountCheckData.system_categories || [],\n }\n};\n\n// Idempotency-Key: deterministic format ตาม FR-001a\nconst idempotencyKey = `${config.BATCH_ID}:${metaItem.document_number}`;\n\nreturn [{\n json: {\n ...metaItem,\n temp_attachment_public_id: tempAttachmentPublicId,\n temp_attachment_id: Number.isFinite(Number(tempAttachmentDbId)) ? Number(tempAttachmentDbId) : undefined,\n submit_payload: submitPayload,\n idempotency_key: idempotencyKey,\n }\n}];" + "jsCode": "const config = $('Set Configuration').first().json.config;\nconst uploadResponse = $input.first()?.json || {};\nconst metaItem = $('File Validator').all()[$input.context?.itemIndex ?? 0]?.json || {};\nconst mountCheckData = $('File Mount Check').first()?.json || {};\n\n// Backend returns { data: { publicId, tempId, ... } } per ADR-019 (id is @Exclude'd)\nconst tempAttachmentPublicId = uploadResponse?.data?.publicId || uploadResponse?.publicId;\nif (!tempAttachmentPublicId) {\n throw new Error(`Upload failed — no temp_attachment_public_id returned. Upload response: ${JSON.stringify(uploadResponse)}`);\n}\n\n// Validate required fields per DTO\nconst docNumber = String(metaItem.document_number || '').trim();\nconst docTitle = String(metaItem.title || '').trim();\nif (!docNumber) {\n throw new Error(`document_number is empty for item: ${JSON.stringify(metaItem)}`);\n}\nif (!docTitle) {\n throw new Error(`title is empty for document: ${docNumber}`);\n}\n\n// Normalize existingTags to match TagOptionDto (tagName is required)\nconst existingTags = (mountCheckData.existing_tags || [])\n .filter(t => t.tagName && t.tagName.trim())\n .map(t => ({\n publicId: t.publicId || undefined,\n tagName: String(t.tagName).trim(),\n colorCode: t.colorCode || undefined,\n }));\n\n// สร้าง SubmitAiJobDto ตาม Backend DTO\nconst submitPayload = {\n type: 'migrate-document',\n payload: {\n tempAttachmentId: String(tempAttachmentPublicId),\n documentNumber: docNumber,\n title: docTitle,\n batchId: String(config.BATCH_ID),\n existingTags: existingTags,\n systemCategories: mountCheckData.system_categories || [],\n }\n};\n\n// Idempotency-Key: deterministic format ตาม FR-001a\nconst idempotencyKey = `${config.BATCH_ID}:${docNumber}`;\n\nreturn [{\n json: {\n ...metaItem,\n temp_attachment_public_id: tempAttachmentPublicId,\n submit_payload: submitPayload,\n idempotency_key: idempotencyKey,\n }\n}];" }, - "id": "c2d3e4f5-a6b7-8901-cdef-234567890123", + "id": "1b7e6adb-b7e2-40dd-b472-3a5a79861bfb", "name": "Build AI Job Payload", "type": "n8n-nodes-base.code", "typeVersion": 2, "position": [ - 32976, - 13504 + -7888, + 4304 ], "notes": "สร้าง SubmitAiJobDto payload + Idempotency-Key ตาม FR-001a" }, @@ -339,19 +333,16 @@ "specifyBody": "json", "jsonBody": "={{ $json.submit_payload }}", "options": { - "retryOnFail": true, - "maxTries": 3, - "waitBetweenTries": 2000, "timeout": 30000 } }, - "id": "d3e4f5a6-b7c8-9012-defa-345678901234", + "id": "f2a5ee75-cd14-4118-925d-43182735aa75", "name": "Submit AI Job", "type": "n8n-nodes-base.httpRequest", "typeVersion": 4.1, "position": [ - 33152, - 13504 + -7744, + 4384 ], "onError": "continueErrorOutput", "notes": "ADR-023A: POST /api/ai/jobs — Backend จัดการ Ollama ผ่าน BullMQ" @@ -360,13 +351,13 @@ "parameters": { "jsCode": "// Poll GET /api/ai/jobs/{jobId} ทุก 5 วินาที จน status = completed หรือ timeout 120 วินาที\nconst config = $('Set Configuration').first().json.config;\nconst submitResponse = $input.first()?.json || {};\nconst originalMeta = $('Build AI Job Payload').all()[$input.context?.itemIndex ?? 0]?.json || {};\n\nconst jobId = submitResponse?.jobId || submitResponse?.data?.jobId;\nif (!jobId) {\n // FR-010b: ถ้า 401 → mark TOKEN_EXPIRED\n if (submitResponse?.statusCode === 401 || submitResponse?.error?.status === 401) {\n throw new Error('TOKEN_EXPIRED: 401 Unauthorized — กรุณา renew MIGRATION_TOKEN แล้ว resume');\n }\n throw new Error(`Submit AI Job failed — no jobId returned for: ${originalMeta.document_number}`);\n}\n\nconst pollIntervalMs = config.AI_JOB_POLL_INTERVAL_MS || 5000;\nconst timeoutMs = config.AI_JOB_TIMEOUT_MS || 120000;\nconst startTime = Date.now();\n\nwhile (true) {\n if (Date.now() - startTime > timeoutMs) {\n throw new Error(`AI Job timeout after ${timeoutMs}ms for jobId: ${jobId}, document: ${originalMeta.document_number}`);\n }\n \n const statusResponse = await $helpers.httpRequest({\n method: 'GET',\n url: `${config.BACKEND_URL}/api/ai/jobs/${jobId}`,\n headers: { 'Authorization': config.MIGRATION_TOKEN },\n json: true\n });\n \n const status = statusResponse?.status || statusResponse?.data?.status;\n \n if (status === 'completed') {\n return [{\n json: {\n ...originalMeta,\n job_id: jobId,\n ai_result: statusResponse?.result || statusResponse?.data?.result || {},\n ai_status: 'completed',\n }\n }];\n }\n \n if (status === 'failed') {\n return [{\n json: {\n ...originalMeta,\n job_id: jobId,\n ai_status: 'failed',\n parse_error: `AI Job failed: ${JSON.stringify(statusResponse?.error || statusResponse?.data?.error || 'unknown')}`,\n error_type: 'AI_JOB_FAILED',\n route_index: 3\n }\n }];\n }\n \n // status = 'processing' — รอแล้ว poll ใหม่\n await new Promise(resolve => setTimeout(resolve, pollIntervalMs));\n}" }, - "id": "e4f5a6b7-c8d9-0123-efab-456789012345", + "id": "f95bbb5f-6bde-4146-b577-8d824776029e", "name": "Poll AI Job Status", "type": "n8n-nodes-base.code", "typeVersion": 2, "position": [ - 33328, - 13504 + -7552, + 4368 ], "notes": "Poll GET /api/ai/jobs/{jobId} ทุก 5 วินาที (timeout 120 วินาที)" }, @@ -374,13 +365,13 @@ "parameters": { "jsCode": "const config = $('Set Configuration').first().json.config;\nconst items = $input.all();\nconst results = [];\n\nconst CATEGORY_TO_TYPE_CODE = {\n 'Correspondence': 'LETTER',\n 'RFA': 'RFA',\n 'Transmittal': 'TRANSMITTAL',\n 'Drawing': 'OTHER',\n 'Report': 'OTHER',\n 'Other': 'OTHER',\n};\n\nfor (const item of items) {\n const data = item.json;\n \n // ถ้า poll ส่ง error กลับมาโดยตรง (route_index = 3)\n if (data.route_index === 3 || data.ai_status === 'failed') {\n results.push({ json: { ...data, route_index: 3 } });\n continue;\n }\n \n const ai = data.ai_result || {};\n \n if (!ai || typeof ai !== 'object') {\n results.push({ json: { ...data, parse_error: 'Empty or invalid ai_result', error_type: 'PARSE_ERROR', route_index: 3 } });\n continue;\n }\n \n // Enum Validation for Category\n const systemCategories = ['Correspondence','RFA','Drawing','Transmittal','Report','Other'];\n let finalCategory = ai.suggested_category || ai.category || 'Correspondence';\n if (!systemCategories.includes(finalCategory)) {\n finalCategory = String(data.document_number || '').includes('-RFA-') ? 'RFA' : 'Correspondence';\n }\n const typeCode = CATEGORY_TO_TYPE_CODE[finalCategory] || 'LETTER';\n \n // Tag normalization — is_new flag\n const suggestedTags = Array.isArray(ai.suggested_tags)\n ? ai.suggested_tags.map(t => ({\n tagName: String(t.tagName || t.tag_name || ''),\n publicId: t.publicId || t.public_id || undefined,\n isNew: Boolean(t.isNew ?? t.is_new ?? !t.publicId),\n colorCode: t.colorCode || t.color_code || 'default',\n })).filter(t => t.tagName)\n : [];\n \n const confidence = Number(ai.confidence || 0);\n \n const normalizedAi = {\n ...ai,\n suggested_category: finalCategory,\n type_code: typeCode,\n confidence,\n suggested_tags: suggestedTags,\n };\n \n let route_index;\n let review_reason = '';\n let reject_reason = '';\n \n if (confidence >= config.CONFIDENCE_HIGH && ai.is_valid !== false) {\n route_index = 0; // Auto Ready\n } else if (confidence >= config.CONFIDENCE_LOW) {\n route_index = 1; // Flagged\n review_reason = `Confidence ${confidence.toFixed(2)} < ${config.CONFIDENCE_HIGH}`;\n } else {\n route_index = 2; // Rejected\n reject_reason = ai.is_valid === false ? 'AI marked invalid' : `Confidence ${confidence.toFixed(2)} < ${config.CONFIDENCE_LOW}`;\n }\n \n results.push({\n json: {\n ...data,\n ai_result: normalizedAi,\n route_index,\n review_reason,\n reject_reason,\n }\n });\n}\n\nreturn results;" }, - "id": "6716162f-1129-4552-a05f-a08ac115fe10", + "id": "6bdcd7fd-0a8a-4f21-91bd-ed5fcbda860c", "name": "Parse & Validate AI Response", "type": "n8n-nodes-base.code", "typeVersion": 2, "position": [ - 33504, - 13504 + -7376, + 4368 ], "notes": "Parse AI result + Confidence routing prep + Tag normalization (isNew flag)" }, @@ -488,13 +479,13 @@ }, "options": {} }, - "id": "65f0bb6c-496a-4409-8b88-3132866cf9a4", + "id": "35ad90cd-bb71-44a6-8851-ae4ea6ed4747", "name": "Route by Confidence", "type": "n8n-nodes-base.switch", "typeVersion": 3.2, "position": [ - 33680, - 13504 + -7216, + 4512 ] }, { @@ -516,21 +507,18 @@ }, "sendBody": true, "specifyBody": "json", - "jsonBody": "={{ JSON.stringify({ batchId: $json.batch_id, documentNumber: $json.document_number, subject: ($json.ai_result?.subject || $json.title), originalSubject: $json.title, tempAttachmentId: $json.temp_attachment_id === undefined ? undefined : Number($json.temp_attachment_id), confidence: $json.ai_result?.confidence, reviewReason: $json.review_reason || '', status: 'PENDING', aiResult: $json.ai_result, idempotencyKey: $json.batch_id + ':' + $json.document_number }) }}", + "jsonBody": "={{ JSON.stringify({ batchId: $json.batch_id, documentNumber: $json.document_number, subject: ($json.ai_result?.subject || $json.title), originalSubject: $json.title, tempAttachmentId: $json.temp_attachment_public_id === undefined ? undefined : $json.temp_attachment_public_id, confidence: $json.ai_result?.confidence, reviewReason: $json.review_reason || '', status: 'PENDING', aiResult: $json.ai_result, idempotencyKey: $json.batch_id + ':' + $json.document_number }) }}", "options": { - "retryOnFail": true, - "maxTries": 3, - "waitBetweenTries": 2000, "timeout": 10000 } }, - "id": "c1bd4485-e58f-4270-892e-edda34c2e328", + "id": "f26c6512-8f2d-4cca-a643-65ae7255c37a", "name": "Insert Review Queue (Auto)", "type": "n8n-nodes-base.httpRequest", "typeVersion": 4.1, "position": [ - 33856, - 13312 + -6976, + 4496 ], "onError": "continueErrorOutput", "notes": "POST /api/ai/migration/queue/record (PENDING) — ADR-023A" @@ -554,21 +542,18 @@ }, "sendBody": true, "specifyBody": "json", - "jsonBody": "={{ JSON.stringify({ batchId: $json.batch_id, documentNumber: $json.document_number, subject: ($json.ai_result?.subject || $json.title), originalSubject: $json.title, tempAttachmentId: $json.temp_attachment_id === undefined ? undefined : Number($json.temp_attachment_id), confidence: $json.ai_result?.confidence, reviewReason: $json.review_reason || '', status: 'PENDING_REVIEW', aiResult: $json.ai_result, idempotencyKey: $json.batch_id + ':' + $json.document_number }) }}", + "jsonBody": "={{ JSON.stringify({ batchId: $json.batch_id, documentNumber: $json.document_number, subject: ($json.ai_result?.subject || $json.title), originalSubject: $json.title, tempAttachmentId: $json.temp_attachment_public_id === undefined ? undefined : $json.temp_attachment_public_id, confidence: $json.ai_result?.confidence, reviewReason: $json.review_reason || '', status: 'PENDING_REVIEW', aiResult: $json.ai_result, idempotencyKey: $json.batch_id + ':' + $json.document_number }) }}", "options": { - "retryOnFail": true, - "maxTries": 3, - "waitBetweenTries": 2000, "timeout": 10000 } }, - "id": "f1a2b3c4-d5e6-7890-abcd-567890123456", + "id": "b720ce71-d0f4-402a-9bd3-077cf1730977", "name": "Insert Review Queue (Flagged)", "type": "n8n-nodes-base.httpRequest", "typeVersion": 4.1, "position": [ - 33856, - 13504 + -6944, + 4720 ], "onError": "continueErrorOutput", "notes": "POST /api/ai/migration/queue/record (PENDING_REVIEW) — ADR-023A" @@ -577,13 +562,13 @@ "parameters": { "jsCode": "const fs = require('fs');\nconst item = $input.first();\nconst config = $('Set Configuration').first().json.config;\n\nconst csvPath = `${config.LOG_PATH}/reject_log_${config.BATCH_ID}.csv`;\nconst header = 'timestamp,document_number,title,reject_reason,ai_confidence,job_id\\n';\nconst esc = (s) => `\"${String(s || '').replace(/\"/g, '\"\"')}\"`;\n\nif (!fs.existsSync(csvPath)) {\n fs.writeFileSync(csvPath, header, 'utf8');\n}\n\nconst line = [\n new Date().toISOString(),\n esc(item.json.document_number),\n esc(item.json.title),\n esc(item.json.reject_reason),\n item.json.ai_result?.confidence ?? 'N/A',\n esc(item.json.job_id || '')\n].join(',') + '\\n';\n\nfs.appendFileSync(csvPath, line, 'utf8');\n\nreturn [$input.first()];" }, - "id": "0bb3530f-02d5-44d0-ad94-c94d97d91b6a", + "id": "6e83de99-1244-4478-9a33-e779cdb9504a", "name": "Log Reject to CSV", "type": "n8n-nodes-base.code", "typeVersion": 2, "position": [ - 33856, - 13696 + -6944, + 4848 ], "notes": "บันทึกรายการที่ถูกปฏิเสธลง CSV" }, @@ -591,13 +576,13 @@ "parameters": { "jsCode": "const fs = require('fs');\nconst items = $input.all();\nconst config = $('Set Configuration').first().json.config;\n\nconst csvPath = `${config.LOG_PATH}/error_log_${config.BATCH_ID}.csv`;\nconst header = 'timestamp,document_number,error_type,error_message,job_id\\n';\nconst esc = (s) => `\"${String(s || '').replace(/\"/g, '\"\"')}\"`;\n\nif (!fs.existsSync(csvPath)) {\n fs.writeFileSync(csvPath, header, 'utf8');\n}\n\nfor (const item of items) {\n const line = [\n new Date().toISOString(),\n esc(item.json.document_number),\n esc(item.json.error_type || 'UNKNOWN'),\n esc(item.json.error || item.json.parse_error),\n esc(item.json.job_id || '')\n ].join(',') + '\\n';\n fs.appendFileSync(csvPath, line, 'utf8');\n}\n\nreturn items;" }, - "id": "8250dd88-ca81-45aa-93d8-480c9bcd6b14", + "id": "511428dc-3aad-4de1-a9dc-9a87c791371e", "name": "Log Error to CSV", "type": "n8n-nodes-base.code", "typeVersion": 2, "position": [ - 33856, - 13888 + -7936, + 4800 ], "notes": "บันทึก Error ลง CSV" }, @@ -622,19 +607,16 @@ "specifyBody": "json", "jsonBody": "={{ JSON.stringify({ batchId: $('Set Configuration').first().json.config.BATCH_ID, documentNumber: $json.document_number || 'WORKFLOW', errorType: $json.error_type || 'UNKNOWN', errorMessage: $json.error || $json.parse_error || $json.message || '', jobId: $json.job_id || '' }) }}", "options": { - "retryOnFail": true, - "maxTries": 3, - "waitBetweenTries": 2000, "timeout": 10000 } }, - "id": "0f058ad0-3c09-4c9f-bdcf-503cd58ee395", + "id": "7e8f3617-a6e4-4d40-922d-eb93ae91e690", "name": "Log Error to DB", "type": "n8n-nodes-base.httpRequest", "typeVersion": 4.1, "position": [ - 34032, - 13888 + -7232, + 4944 ], "onError": "continueErrorOutput", "notes": "POST /api/ai/migration/errors — ADR-023A" @@ -660,19 +642,16 @@ "specifyBody": "json", "jsonBody": "={{ JSON.stringify({ batchId: $('Set Configuration').first().json.config.BATCH_ID, lastProcessedIndex: ($json.original_index || 0) + 1, status: 'RUNNING' }) }}", "options": { - "retryOnFail": true, - "maxTries": 3, - "waitBetweenTries": 2000, "timeout": 10000 } }, - "id": "bb0e611b-db28-4266-ba40-3b5d534a16f7", + "id": "9471990c-1abe-42f8-8062-bd68cb9ad985", "name": "Save Checkpoint", "type": "n8n-nodes-base.httpRequest", "typeVersion": 4.1, "position": [ - 34032, - 13312 + -6784, + 4640 ], "onError": "continueErrorOutput", "notes": "POST /api/ai/migration/checkpoint — ADR-023A" @@ -682,13 +661,13 @@ "amount": "={{$('Set Configuration').first().json.config.DELAY_MS / 1000}}", "unit": "seconds" }, - "id": "07c1c5d5-5ffc-4e3d-ab3e-4b62ad079388", + "id": "fcf3e098-93f6-42ee-b930-aa0bc84d3ed7", "name": "Delay", "type": "n8n-nodes-base.wait", "typeVersion": 1, "position": [ - 34208, - 13504 + -6640, + 4880 ], "webhookId": "38e97a99-4dcc-4b63-977a-a02945a1c369", "notes": "หน่วงเวลาระหว่าง Records" @@ -749,13 +728,13 @@ }, "options": {} }, - "id": "b1c2d3e4-f5a6-7890-bcde-f12345678902", + "id": "e489d28d-37e8-4204-bba8-07a5226b1275", "name": "Check Batch Complete", "type": "n8n-nodes-base.switch", "typeVersion": 3.2, "position": [ - 34384, - 13504 + -8512, + 4608 ] }, { @@ -779,19 +758,16 @@ "specifyBody": "json", "jsonBody": "={{ JSON.stringify({ batchId: $('Set Configuration').first().json.config.BATCH_ID, lastProcessedIndex: $json.total_processed, status: 'COMPLETED' }) }}", "options": { - "retryOnFail": true, - "maxTries": 3, - "waitBetweenTries": 2000, "timeout": 10000 } }, - "id": "9a8b7c6d-5e4f-3a2b-1c0d-9e8f7a6b5c4d", + "id": "171b627f-3b00-4fbf-86b7-6076fdc29d19", "name": "Mark Batch Complete", "type": "n8n-nodes-base.httpRequest", "typeVersion": 4.1, "position": [ - 34208, - 13696 + -8304, + 4400 ], "notes": "Update checkpoint status to COMPLETED when batch finishes" } @@ -1132,9 +1108,6 @@ ] ] }, - "Mark Batch Complete": { - "main": [] - }, "Log Reject to CSV": { "main": [ [ @@ -1179,22 +1152,26 @@ "active": false, "settings": { "executionOrder": "v1", - "availableInMCP": false + "binaryMode": "separate" }, - "versionId": "v2.0.0-adr023a-compliant", + "versionId": "efadd20e-a46b-4354-8f75-ec1de215d065", "meta": { "templateCredsSetupCompleted": true, "instanceId": "9e70e47c1eaf3bac72f497ddfbde0983f840f7d0f059537f7e37dd70de18ecb7" }, - "id": "u7CLP05AyFb8Um0P", + "id": "4LlPbAKU5BZLgiTg", "tags": [ { - "name": "migration", - "createdAt": "2026-05-23" + "updatedAt": "2026-05-23T12:26:47.389Z", + "createdAt": "2026-05-23T12:26:47.389Z", + "id": "jNSEtctPbU5leFPw", + "name": "v2" }, { - "name": "v2", - "createdAt": "2026-05-23" + "updatedAt": "2026-05-23T12:26:47.393Z", + "createdAt": "2026-05-23T12:26:47.393Z", + "id": "mGZTyPfxbcsAuFpR", + "name": "migration" } ] } diff --git a/specs/06-Decision-Records/ADR-023A-unified-ai-architecture.md b/specs/06-Decision-Records/ADR-023A-unified-ai-architecture.md index c64d12f7..56e324d4 100644 --- a/specs/06-Decision-Records/ADR-023A-unified-ai-architecture.md +++ b/specs/06-Decision-Records/ADR-023A-unified-ai-architecture.md @@ -1,4 +1,4 @@ -# ADR-023A: Unified AI Architecture — Model Revision (gemma4:e4b Q8_0, 2-Model Stack) +# ADR-023A: Unified AI Architecture — Model Revision (gemma4:e2b, 2-Model Stack) **Status:** Accepted **Date:** 2026-05-15 @@ -14,7 +14,7 @@ - [RAG Implementation Guide v1.1.2](../08-Tasks/ADR-022-Retrieval-Augmented-Generation/LCBP3-RAG-Implementation-Guide-v1.1.2.md) - [ADR-023: Unified AI Architecture (Base)](./ADR-023-unified-ai-architecture.md) -> **หมายเหตุ:** ADR-023A เป็นสำเนาอัปเดตของ ADR-023 v1.1 โดยปรับปรุงชุดโมเดล AI เพื่อให้ใช้งาน VRAM ≤ 8GB ได้อย่างมีเสถียรภาพ ลดจาก 3 โมเดล (gemma4:9b + Typhoon Local + nomic-embed-text) เหลือ 2 โมเดล (gemma4:e4b Q8_0 + nomic-embed-text) โดย gemma4:e4b ทำหน้าที่ครอบคลุมทั้ง General Inference และ OCR Post-processing/Extraction แทน Typhoon Local +> **หมายเหตุ:** ADR-023A เป็นสำเนาอัปเดตของ ADR-023 v1.1 โดยปรับปรุงชุดโมเดล AI เพื่อให้ใช้งาน VRAM ≤ 8GB ได้อย่างมีเสถียรภาพ ลดจาก 3 โมเดล (gemma4:9b + Typhoon Local + nomic-embed-text) เหลือ 2 โมเดล (gemma4:e2b + nomic-embed-text) โดย gemma4:e2b ทำหน้าที่ครอบคลุมทั้ง General Inference และ OCR Post-processing/Extraction แทน Typhoon Local --- @@ -79,32 +79,32 @@ - ❌ Ollama ไม่สลับโมเดลได้ฉับพลัน หากโหลดพร้อมกัน VRAM เต็มแน่นอน - ❌ ต้องจัดการ Routing Logic (เลือกว่างานไหนใช้โมเดลไหน) เพิ่ม Complexity -### Option 4: Unified 2-Model Stack (gemma4:e4b Q8_0 + nomic-embed-text) ⭐ **SELECTED** +### Option 4: Unified 2-Model Stack (gemma4:e2b + nomic-embed-text) ⭐ **SELECTED** | โมเดล | ขนาด (VRAM โดยประมาณ) | หน้าที่ | |-------|----------------------|---------| -| `gemma4:e4b Q8_0` | ~4.5GB | General Inference + OCR Post-processing + Extraction + RAG Q&A | +| `gemma4:e2b` | ~2GB (Q4) | General Inference + OCR Post-processing + Extraction + RAG Q&A | | `nomic-embed-text` | ~0.3GB | Embedding 768-dim สำหรับ Qdrant | -| **รวม** | **~4.8GB** | **เผื่อ headroom ~3.2GB สำหรับ KV Cache และ context window ขนาดใหญ่** | +| **รวม** | **~2.3GB** | **เผื่อ headroom ~5.7GB สำหรับ KV Cache และ context window ขนาดใหญ่ (8K tokens)** | **Pros:** - ✅ **VRAM ≤ 8GB อย่างมีเสถียรภาพ:** โมเดลทั้ง 2 โหลดพร้อมกันได้ มี headroom เพียงพอสำหรับ KV Cache ขนาดใหญ่ -- ✅ **Single Model ลด Routing Complexity:** gemma4:e4b ครอบคลุมทุก Use Case (OCR clean-up, Extraction, RAG, Classification) ผ่าน Prompt Engineering ที่แตกต่างกัน +- ✅ **Single Model ลด Routing Complexity:** gemma4:e2b ครอบคลุมทุก Use Case (OCR clean-up, Extraction, RAG, Classification) ผ่าน Prompt Engineering ที่แตกต่างกัน - ✅ **BullMQ Sequential Queue:** การใช้โมเดลเดียวทำให้ Queue ทำงานได้ตรงไปตรงมา — ไม่มีปัญหา Worker ต้องสลับโมเดลระหว่างงาน - ✅ **GPU Overload Prevention ตาม ADR-023:** สอดคล้องกับนโยบายที่กำหนดไว้แต่เดิม -- ✅ **gemma4:e4b Q8_0:** quantization Q8_0 รักษาความแม่นยำของ weights ใกล้เคียง FP16 มากที่สุด เหมาะกับงานที่ต้องการความละเอียดด้านภาษา +- ✅ **gemma4:e2b Q4:** quantization Q4 ประหยัด VRAM มากที่สุด (~2GB) พร้อม context window 8K tokens ขนาดใหญ่ **Cons:** -- ❌ ไม่มี Typhoon Local ซึ่งถูก Fine-tune มาสำหรับภาษาไทยโดยเฉพาะ — ต้องพึ่ง Prompt Engineering บน gemma4:e4b แทน +- ❌ ไม่มี Typhoon Local ซึ่งถูก Fine-tune มาสำหรับภาษาไทยโดยเฉพาะ — ต้องพึ่ง Prompt Engineering บน gemma4:e2b แทน --- ## Decision Outcome -**Chosen Option:** Option 4 — Unified 2-Model Stack (gemma4:e4b Q8_0 + nomic-embed-text) +**Chosen Option:** Option 4 — Unified 2-Model Stack (gemma4:e2b + nomic-embed-text) ### Rationale -การลด Model Stack จาก 3 → 2 โมเดลช่วยให้ VRAM Budget ≤ 8GB อย่างมีเสถียรภาพ โดย gemma4:e4b Q8_0 สามารถทำหน้าที่แทน Typhoon Local ได้ผ่าน Prompt Engineering เนื่องจาก gemma4 architecture รองรับ Multimodal Context ได้ดี และ Q8_0 quantization รักษา quality ไว้ในระดับสูง BullMQ Sequential Queue ทำงานได้ตรงไปตรงมามากขึ้นเมื่อมีโมเดลเดียว ลด complexity ของ Job Routing +การลด Model Stack จาก 3 → 2 โมเดลช่วยให้ VRAM Budget ≤ 8GB อย่างมีเสถียรภาพ โดย gemma4:e2b สามารถทำหน้าที่แทน Typhoon Local ได้ผ่าน Prompt Engineering เนื่องจาก gemma4 architecture รองรับ Multimodal Context ได้ดี และ Q4 quantization ประหยัด VRAM มากที่สุด BullMQ Sequential Queue ทำงานได้ตรงไปตรงมามากขึ้นเมื่อมีโมเดลเดียว ลด complexity ของ Job Routing --- @@ -130,7 +130,7 @@ graph TB end subgraph DESK["🖥️ Desk-5439 (AI Isolation Host)"] - OLLAMA["Ollama\ngemma4:e4b Q8_0\n+ nomic-embed-text"] + OLLAMA["Ollama\ngemma4:e2b\n+ nomic-embed-text"] QDRANT[Qdrant Vector Store] NLP[PaddleOCR + PyThaiNLP] end @@ -183,9 +183,9 @@ graph TB | โมเดล | Role | VRAM (โดยประมาณ) | หมายเหตุ | |-------|------|-----------------|---------| -| `gemma4:e4b Q8_0` | General Inference + OCR Post-processing + Extraction + RAG Q&A | ~4.0GB (weights) + ~0.2GB (KV Cache) | Q8_0 ≈ 4B × 8-bit = ~4.0GB; KV Cache ต่ำเพราะ input ≤ 3 หน้า (~2,000 tokens) | +| `gemma4:e2b` | General Inference + OCR Post-processing + Extraction + RAG Q&A | ~2GB (Q4) + ~0.2GB (KV Cache) | Q4 quantization; Context window 8K tokens; Parameters 2.1B | | `nomic-embed-text` | Embedding 768-dim → Qdrant | ~0.3GB | สร้าง Semantic Vector สำหรับ Hybrid Search | -| **รวม (peak)** | | **~4.5GB** | **เผื่อ headroom ~3.5GB — มั่นใจสูง เพราะ PDF input จำกัด ≤ 3 หน้า** | +| **รวม (peak)** | | **~2.5GB** | **เผื่อ headroom ~5.5GB — มั่นใจสูง เพราะ context window ขนาดใหญ่ (8K tokens)** | * **Orchestrator:** ใช้ **n8n** เป็นตัวควบคุม Flow **Migration Phase เท่านั้น** (trigger batch, monitor progress, handle retry ระดับ batch) — ห้าม n8n เรียก Ollama หรือ PaddleOCR โดยตรง * **Job Executor:** ทุก AI Inference (OCR, Extraction, Embedding, RAG) ต้องผ่าน **BullMQ บน NestJS เท่านั้น** — n8n call `POST /api/ai/jobs` เพื่อ queue job แล้ว poll ผลผ่าน `GET /api/ai/jobs/:jobId` @@ -202,10 +202,10 @@ Real-time Flow (User Upload): > **เหตุผล:** การ inference ทั้งหมดผ่าน BullMQ ทำให้ RBAC, ADR-007 Error Handling และ `ai_audit_logs` ครอบคลุมทุก job โดยอัตโนมัติ — ถ้า n8n bypass BullMQ จะเกิด audit gap -* **LLM Engine:** ใช้ **Ollama** บน Desk-5439 รันโมเดล `gemma4:e4b Q8_0` สำหรับงานทั้งหมด ได้แก่ General Inference, OCR Post-processing, Metadata Extraction, Classification และ RAG Q&A +* **LLM Engine:** ใช้ **Ollama** บน Desk-5439 รันโมเดล `gemma4:e2b` สำหรับงานทั้งหมด ได้แก่ General Inference, OCR Post-processing, Metadata Extraction, Classification และ RAG Q&A * **Embedding Model:** ใช้ `nomic-embed-text` รันผ่าน Ollama บน Desk-5439 สำหรับแปลงเวกเตอร์ 768-มิติ * **OCR & NLP:** ใช้ **PaddleOCR** สกัดข้อความจาก Scanned PDF และใช้ **PyThaiNLP** ตัดคำ/เตรียมข้อความภาษาไทย — ทั้งคู่รันบน Desk-5439 -* ❌ **Typhoon Local:** ไม่ใช้ — ถูกแทนที่โดย `gemma4:e4b Q8_0` เพื่อรักษา VRAM Budget +* ❌ **Typhoon Local:** ไม่ใช้ — ถูกแทนที่โดย `gemma4:e2b` เพื่อรักษา VRAM Budget * ❌ **Typhoon Cloud API:** ไม่ใช้ — `rag/typhoon.service.ts` ต้องถูก Remove ออกจาก Codebase (Dead Code + Security Risk) #### 2.2 BullMQ Queue Architecture (GPU Overload Prevention) @@ -231,8 +231,8 @@ Queue: ai-realtime (BullMQ) | Job Type | โมเดลที่ใช้ | SLA Target | |----------|-----------|------------| -| `rag-query` | `gemma4:e4b Q8_0` | p95 < 10s (นับตั้งแต่ dequeue) | -| `ai-suggest` | `gemma4:e4b Q8_0` | p95 < 8s | +| `rag-query` | `gemma4:e2b` | p95 < 10s (นับตั้งแต่ dequeue) | +| `ai-suggest` | `gemma4:e2b` | p95 < 8s | ##### Queue 2: `ai-batch` (Background Processing) @@ -246,8 +246,8 @@ Queue: ai-batch (BullMQ) | Job Type | โมเดลที่ใช้ | Priority | |----------|-----------|---------| -| `ocr-postprocess` | `gemma4:e4b Q8_0` | Normal | -| `metadata-extract` | `gemma4:e4b Q8_0` | Normal | +| `ocr-postprocess` | `gemma4:e2b` | Normal | +| `metadata-extract` | `gemma4:e2b` | Normal | | `embed-document` | `nomic-embed-text` | Low | > ⚠️ **GPU Constraint:** แม้จะแยก 2 Queue แต่ Ollama Worker บน Desk-5439 มี GPU เดียว — หาก `ai-realtime` และ `ai-batch` รัน Job พร้อมกัน VRAM อาจเต็ม ให้ตั้งค่า `ai-batch` pause อัตโนมัติเมื่อ `ai-realtime` มี active job (ผ่าน BullMQ Event hooks: `active` / `completed`) @@ -320,19 +320,19 @@ Queue: ai-batch (BullMQ) #### 4.1 PDF Input Limit (Hard Constraint) -> **กฎ:** ส่ง PDF เข้า **gemma4:e4b Q8_0** ได้ **สูงสุด 3 หน้าแรกเท่านั้น** สำหรับงาน Summarization, Classification และ Tagging +> **กฎ:** ส่ง PDF เข้า **gemma4:e2b** ได้ **สูงสุด 5 หน้าแรกเท่านั้น** สำหรับงาน Summarization, Classification และ Tagging (เพิ่มจาก 3 เพราะ context window 8K tokens) > ⚠️ **ข้อยกเว้น:** งาน `embed-document` (RAG) ใช้เอกสารทั้งฉบับ — ดู Section 5 **เหตุผล:** -- หน้าปก + หน้าที่ 1–2 ของเอกสารวิศวกรรมมักมีข้อมูลหลักครบ (Document Title, Drawing No., Discipline, Project Code, Revision) -- จำกัด KV Cache ที่ ~2,000 tokens → VRAM peak ≤ ~4.5GB ตามที่ออกแบบไว้ +- หน้าปก + หน้าที่ 1–4 ของเอกสารวิศวกรรมมักมีข้อมูลหลักครบ (Document Title, Drawing No., Discipline, Project Code, Revision) +- Context window 8K tokens รองรับ ~5 หน้า → VRAM peak ≤ ~2.5GB ตามที่ออกแบบไว้ - ป้องกัน Job ใช้เวลานานเกิน SLA **Implementation Note:** ``` n8n PDF Pre-processor: - extract_pages: [1, 2, 3] ← hard limit, ห้ามเปลี่ยนโดยไม่ review ADR - fallback: ถ้า PDF < 3 หน้า → ใช้ทั้งหมด + extract_pages: [1, 2, 3, 4, 5] ← hard limit, ห้ามเปลี่ยนโดยไม่ review ADR + fallback: ถ้า PDF < 5 หน้า → ใช้ทั้งหมด ``` #### 4.2 PDF Type Auto-Detection (OCR Routing) @@ -414,8 +414,8 @@ export class QdrantService { // ❌ ห้าม expose rawSearch() หรือ method ที่ไม่บังคับ filter } ``` -* **LLM สำหรับ RAG Q&A:** ใช้ **Local Ollama (`gemma4:e4b Q8_0`)** บน Desk-5439 เท่านั้น — ไม่มี Cloud Fallback เนื่องจากเอกสารทั้งหมดจัดชั้นเป็น INTERNAL -* **Context Window สำหรับ RAG:** ส่ง top-K chunks (K=5) เข้า gemma4:e4b ≈ 5 × 512 = ~2,560 tokens — อยู่ในขีดจำกัด VRAM +* **LLM สำหรับ RAG Q&A:** ใช้ **Local Ollama (`gemma4:e2b`)** บน Desk-5439 เท่านั้น — ไม่มี Cloud Fallback เนื่องจากเอกสารทั้งหมดจัดชั้นเป็น INTERNAL +* **Context Window สำหรับ RAG:** ส่ง top-K chunks (K=5) เข้า gemma4:e2b ≈ 5 × 512 = ~2,560 tokens — อยู่ในขีดจำกัด VRAM (8K tokens) * **Performance Target:** $p95 < 10s$ สำหรับการตอบคำถามผ่าน Local LLM (นับตั้งแต่ dequeue จาก `ai-realtime`) ### 6. `ai_audit_logs` — AI Development Feedback Log diff --git a/specs/06-Decision-Records/ADR-026-document-chat-ui-pattern.md b/specs/06-Decision-Records/ADR-026-document-chat-ui-pattern.md index 05f21707..86db73d0 100644 --- a/specs/06-Decision-Records/ADR-026-document-chat-ui-pattern.md +++ b/specs/06-Decision-Records/ADR-026-document-chat-ui-pattern.md @@ -228,7 +228,7 @@ AI Gateway (ADR-023A) └─→ RAG Pipeline ถ้าเป็น RAG_QUERY │ ▼ -Ollama (gemma4:e4b Q8_0) +Ollama (gemma4:e2b) │ ▼ Response → Stream/Chunk → UI diff --git a/specs/08-Tasks/Task BE-AI-01.md b/specs/08-Tasks/Task BE-AI-01.md index 1b1dd334..b98909a9 100644 --- a/specs/08-Tasks/Task BE-AI-01.md +++ b/specs/08-Tasks/Task BE-AI-01.md @@ -20,7 +20,7 @@ - Webhook endpoint: `/webhook/ai-processing` - Environment variables: `N8N_BASIC_AUTH_USER`, `N8N_BASIC_AUTH_PASSWORD` - [ ] **Ollama Service:** - - Pull model: `gemma4:e4b` (GPU optimized, higher accuracy) + - Pull model: `gemma4:e2b` (GPU optimized, lower VRAM) - API endpoint: `http://localhost:11434` - Health check: `GET /api/tags` - Memory requirement: Minimum 8GB VRAM for 9B model diff --git a/specs/200-fullstacks/224-intent-classification/plan.md b/specs/200-fullstacks/224-intent-classification/plan.md index 9eb3d559..0bfcc92a 100644 --- a/specs/200-fullstacks/224-intent-classification/plan.md +++ b/specs/200-fullstacks/224-intent-classification/plan.md @@ -13,29 +13,29 @@ - Backend: NestJS Module (IntentClassifierModule) พร้อม Service สำหรับ Pattern Matching และ LLM Fallback - Database: ตาราง `ai_intent_definitions` และ `ai_intent_patterns` (SQL Delta ตาม ADR-009) - Caching: Redis (TTL 5 นาที) สำหรับ Patterns -- AI: Ollama gemma4:e4b Q8_0 บน Admin Desktop (Desk-5439) สำหรับ LLM Fallback +- AI: Ollama gemma4:e2b บน Admin Desktop (Desk-5439) สำหรับ LLM Fallback - Frontend: Admin UI สำหรับจัดการ Intent และ Patterns + Test Console --- ## Technical Context -**Language/Version**: TypeScript 5.x (NestJS 11) + Next.js 16 -**Primary Dependencies**: +**Language/Version**: TypeScript 5.x (NestJS 11) + Next.js 16 +**Primary Dependencies**: - Backend: NestJS, TypeORM, ioredis (Redis), axios (Ollama HTTP) -- Frontend: React, TanStack Query, shadcn/ui components -**Storage**: MariaDB 11.8 (Intent Definitions/Patterns), Redis (Cache), Ollama (LLM) -**Testing**: Jest (Backend Unit/Integration), Vitest (Frontend Unit), Playwright (E2E) -**Target Platform**: QNAP NAS (Docker), Admin Desktop (Ollama) -**Project Type**: Web application (Backend + Frontend) -**Performance Goals**: +- Frontend: React, TanStack Query, shadcn/ui components +**Storage**: MariaDB 11.8 (Intent Definitions/Patterns), Redis (Cache), Ollama (LLM) +**Testing**: Jest (Backend Unit/Integration), Vitest (Frontend Unit), Playwright (E2E) +**Target Platform**: QNAP NAS (Docker), Admin Desktop (Ollama) +**Project Type**: Web application (Backend + Frontend) +**Performance Goals**: - Pattern Match: < 10ms (cache hit), < 50ms (cache miss) - LLM Fallback: < 2000ms (รวม Pattern Check) -**Constraints**: +**Constraints**: - GPU Budget: RTX 2060 Super 8GB (ใช้ร่วมกับ RAG, OCR, Embedding) - LLM Semaphore: Max 3 concurrent calls - Bilingual Input: ไทย/อังกฤษปน + typo tolerance -**Scale/Scope**: +**Scale/Scope**: - 12 Intent Definitions (v1) - 50+ concurrent users - 70-80% Pattern Hit Rate target @@ -146,7 +146,7 @@ frontend/ **หัวข้อที่ต้อง Research**: 1. Redis Cache Strategy สำหรับ Patterns (TTL + Invalidation) -2. Ollama HTTP API Integration (gemma4:e4b Q8_0) +2. Ollama HTTP API Integration (gemma4:e2b) 3. Semaphore Pattern ใน NestJS (p-limit หรือ RxJS) 4. Regex Validation ใน TypeORM/Class-Validator diff --git a/specs/200-fullstacks/224-intent-classification/quickstart.md b/specs/200-fullstacks/224-intent-classification/quickstart.md index 4cc707db..02619278 100644 --- a/specs/200-fullstacks/224-intent-classification/quickstart.md +++ b/specs/200-fullstacks/224-intent-classification/quickstart.md @@ -1,13 +1,13 @@ # Quick Start: Intent Classification System -**Feature**: 224-intent-classification +**Feature**: 224-intent-classification **Date**: 2026-05-19 --- ## Prerequisites -- Ollama Server บน Admin Desktop (Desk-5439) พร้อม Model `gemma4:e4b` +- Ollama Server บน Admin Desktop (Desk-5439) พร้อม Model `gemma4:e2b` - Redis Server พร้อมใช้งาน - Database Schema อัปเดตผ่าน SQL Delta @@ -56,7 +56,7 @@ INSERT INTO ai_intent_definitions (intent_code, description_th, description_en, ```env # Ollama Configuration OLLAMA_BASE_URL=http://192.168.10.10:11434 -OLLAMA_MODEL=gemma4:e4b +OLLAMA_MODEL=gemma4:e2b OLLAMA_TIMEOUT_MS=5000 # Intent Classification diff --git a/specs/200-fullstacks/224-intent-classification/spec.md b/specs/200-fullstacks/224-intent-classification/spec.md index 711f5f5d..a9b16330 100644 --- a/specs/200-fullstacks/224-intent-classification/spec.md +++ b/specs/200-fullstacks/224-intent-classification/spec.md @@ -1,8 +1,8 @@ # Feature Specification: Intent Classification System -**Feature Branch**: `224-intent-classification` -**Created**: 2026-05-19 -**Status**: Draft +**Feature Branch**: `224-intent-classification` +**Created**: 2026-05-19 +**Status**: Draft **Input**: ADR-024 Intent Classification Strategy + CONTEXT.md AI Runtime Layer --- @@ -93,7 +93,7 @@ - **FR-004**: ระบบต้องรองรับ Pattern Type 2 แบบ: `keyword` (case-insensitive includes) และ `regex` (RegExp.test) - **FR-005**: ระบบต้องมี Caching Layer ด้วย Redis (Key: `ai:intent:patterns:active`, TTL: 300 วินาที) เพื่อลดการ Query DB - **FR-006**: ระบบต้องทำ Pattern Matching ตามลำดับ Priority (ASC) — Pattern ที่มี priority ต่ำกว่าจะถูกตรวจสอบก่อน -- **FR-007**: หากไม่มี Pattern Match → ระบบต้องเรียก LLM Fallback (Ollama gemma4:e4b Q8_0) แบบ Synchronous +- **FR-007**: หากไม่มี Pattern Match → ระบบต้องเรียก LLM Fallback (Ollama gemma4:e2b) แบบ Synchronous - **FR-008**: LLM Fallback ต้องใช้ Semaphore จำกัด Concurrent Calls สูงสุด 3 รายการพร้อมกัน - **FR-009**: ระบบต้อง Validate Confidence Score จาก LLM และ Override เป็น `FALLBACK` หาก confidence < 0.4 - **FR-010**: ระบบต้องบันทึกทุก Classification Request ลง `ai_audit_logs` โดยมีข้อมูล: input, output, method, latency, projectPublicId, userPublicId @@ -128,7 +128,7 @@ ### Dependencies -- Ollama Server บน Admin Desktop (Desk-5439) พร้อม Model gemma4:e4b Q8_0 +- Ollama Server บน Admin Desktop (Desk-5439) พร้อม Model gemma4:e2b - Redis Cache Server พร้อมใช้งาน - Database Schema ตาราง `ai_intent_definitions` และ `ai_intent_patterns` (เพิ่มผ่าน SQL Delta) - AI Gateway Module ที่มีอยู่แล้ว (ADR-023A) diff --git a/specs/200-fullstacks/228-migration-arch-refactor/plan.md b/specs/200-fullstacks/228-migration-arch-refactor/plan.md index 4af8d39a..1a33259a 100644 --- a/specs/200-fullstacks/228-migration-arch-refactor/plan.md +++ b/specs/200-fullstacks/228-migration-arch-refactor/plan.md @@ -8,17 +8,17 @@ ## Summary -Refactor migration architecture ให้สอดคล้องกับ ADR-023A: n8n เรียกผ่าน BullMQ แทน Ollama โดยตรง, ใช้ `gemma4:e4b Q8_0`, OCR ผ่าน PyMuPDF/PaddleOCR, สร้าง Backend endpoint `/api/ai/jobs`, SQL delta สำหรับ `tags`/`correspondence_tags`, และ Migration Review UI +Refactor migration architecture ให้สอดคล้องกับ ADR-023A: n8n เรียกผ่าน BullMQ แทน Ollama โดยตรง, ใช้ `gemma4:e2b`, OCR ผ่าน PyMuPDF/PaddleOCR, สร้าง Backend endpoint `/api/ai/jobs`, SQL delta สำหรับ `tags`/`correspondence_tags`, และ Migration Review UI ## Technical Context -**Language/Version**: TypeScript 5.x, NestJS 10.x, Next.js 14.x -**Primary Dependencies**: BullMQ, TypeORM, CASL, TanStack Query, Zod -**Storage**: MariaDB (SQL delta via ADR-009), Qdrant (embedding), Redis (BullMQ) -**Testing**: Jest (Backend), Vitest (Frontend) -**Target Platform**: QNAP NAS (Backend + n8n), Admin Desktop Desk-5439 (Ollama + OCR Worker) -**Performance Goals**: Fast Path OCR < 5s/file; Slow Path OCR < 60s/file; AI inference < 30s -**Constraints**: VRAM peak ~4.3GB; BullMQ concurrency=1 (ai-batch); Token TTL ≤ 7 วัน +**Language/Version**: TypeScript 5.x, NestJS 10.x, Next.js 14.x +**Primary Dependencies**: BullMQ, TypeORM, CASL, TanStack Query, Zod +**Storage**: MariaDB (SQL delta via ADR-009), Qdrant (embedding), Redis (BullMQ) +**Testing**: Jest (Backend), Vitest (Frontend) +**Target Platform**: QNAP NAS (Backend + n8n), Admin Desktop Desk-5439 (Ollama + OCR Worker) +**Performance Goals**: Fast Path OCR < 5s/file; Slow Path OCR < 60s/file; AI inference < 30s +**Constraints**: VRAM peak ~2.5GB; BullMQ concurrency=1 (ai-batch); Token TTL ≤ 7 วัน **Scale/Scope**: 20,000 PDF documents; ~3 วินาที/record → ~16.6 ชั่วโมงรวม ## Constitution Check @@ -31,7 +31,7 @@ Refactor migration architecture ให้สอดคล้องกับ ADR-0 | ADR-008 | BullMQ สำหรับ background jobs | ✅ (ai-batch queue) | | ADR-023A | n8n → DMS API → BullMQ → Ollama (ห้าม direct) | ✅ | | ADR-007 | Layered error handling + user-friendly messages | ✅ | -| ADR-023A | gemma4:e4b Q8_0 + nomic-embed-text เท่านั้น | ✅ | +| ADR-023A | gemma4:e2b + nomic-embed-text เท่านั้น | ✅ | ## Project Structure diff --git a/specs/200-fullstacks/228-migration-arch-refactor/spec.md b/specs/200-fullstacks/228-migration-arch-refactor/spec.md index a3d469cf..c9f03f81 100644 --- a/specs/200-fullstacks/228-migration-arch-refactor/spec.md +++ b/specs/200-fullstacks/228-migration-arch-refactor/spec.md @@ -109,7 +109,7 @@ DBA หรือ DevOps สร้างตาราง `tags` และ `corresp - **FR-001b**: Backend ต้อง double-check `import_transactions` (document_number + batch_id + status != FAILED) ก่อน enqueue BullMQ — ถ้าซ้ำ return 409 พร้อม `existingJobId` (defense-in-depth ต่างหากจาก Idempotency-Key) - **FR-002**: ระบบต้องมี endpoint `GET /api/ai/jobs/:jobId` สำหรับ polling status และรับ AI output - **FR-003**: BullMQ Worker ต้องรัน OCR auto-detect: PyMuPDF (extracted_chars > 100) หรือ PaddleOCR + PyThaiNLP -- **FR-004**: AI inference ต้องใช้ `gemma4:e4b Q8_0` เท่านั้น ผ่าน Ollama บน Desk-5439 (ห้าม model อื่น) +- **FR-004**: AI inference ต้องใช้ `gemma4:e2b` เท่านั้น ผ่าน Ollama บน Desk-5439 (ห้าม model อื่น) - **FR-005**: Temp files ต้องถูก auto-cleanup ใน 24 ชั่วโมง หลัง job `failed` หรือไม่มี commit (Scheduled BullMQ job) - **FR-005a**: Cleanup scheduler ต้อง exclude temp files ที่ถูก reference โดย `migration_review_queue.status = PENDING` — ห้ามลบ file ที่รออยู่ใน review queue - **FR-005b**: PENDING records ที่ไม่มี action ภายใน 30 วัน ต้อง auto-expire เป็น `EXPIRED` + cleanup temp file + แจ้ง Admin (BullMQ notification job) diff --git a/specs/200-fullstacks/228-migration-arch-refactor/tasks.md b/specs/200-fullstacks/228-migration-arch-refactor/tasks.md index 19786a02..ed486670 100644 --- a/specs/200-fullstacks/228-migration-arch-refactor/tasks.md +++ b/specs/200-fullstacks/228-migration-arch-refactor/tasks.md @@ -39,7 +39,7 @@ - [x] T009 [US1] สร้าง BullMQ Worker `MigrateDocumentWorker` ใน `backend/src/modules/ai/workers/migrate-document.worker.ts` — Step 1: fetch temp file from StorageService - [x] T010 [P] [US1] เพิ่ม OCR routing logic ใน Worker — PyMuPDF Fast Path (chars > 100) หรือ PaddleOCR Slow Path — เรียกผ่าน OCR Service HTTP API (ไม่ใช่ direct Ollama) -- [x] T011 [P] [US1] เพิ่ม gemma4:e4b inference ใน Worker — System Prompt + User Prompt สำหรับ metadata extraction + classification + tagging +- [x] T011 [P] [US1] เพิ่ม gemma4:e2b inference ใน Worker — System Prompt + User Prompt สำหรับ metadata extraction + classification + tagging - [x] T012 [US1] เพิ่ม JSON validation + error handling ใน Worker (ADR-007) — ถ้า AI output ไม่ถูก format → mark job failed + log ใน `ai_audit_logs` - [x] T013 [US1] เพิ่ม `submitMigrationJob()` method ใน `backend/src/modules/ai/ai.service.ts` — (1) Idempotency-Key check; (2) double-check `import_transactions` (document_number + batch_id + status != FAILED) ก่อน enqueue → 409 พร้อม existingJobId ถ้าซ้ำ (FR-001b); (3) enqueue ไปยัง ai-batch queue - [x] T014 [US1] เพิ่ม `POST /api/ai/jobs` endpoint ใน `backend/src/modules/ai/ai.controller.ts` (JwtAuthGuard + CaslAbilityGuard + Idempotency-Key header validation) diff --git a/specs/300-others/302-ai-model-revision/data-model.md b/specs/300-others/302-ai-model-revision/data-model.md index cf605432..1bdb4b75 100644 --- a/specs/300-others/302-ai-model-revision/data-model.md +++ b/specs/300-others/302-ai-model-revision/data-model.md @@ -9,7 +9,7 @@ ### 1. `ai_audit_logs` (existing table — verify against schema) -ไม่มีการเพิ่ม column ใหม่ — ใช้ `model_name` column ที่มีอยู่แล้วบันทึก `gemma4:e4b` แทน `gemma4:9b` +ไม่มีการเพิ่ม column ใหม่ — ใช้ `model_name` column ที่มีอยู่แล้วบันทึก `gemma4:e2b` แทน `gemma4:9b` **Key fields (existing)**: ``` @@ -18,7 +18,7 @@ public_id BINARY(16) → UUIDv7 document_id INT FK documents.id project_id INT FK projects.id job_type VARCHAR(50) -- 'classification', 'tagging', 'rag', 'embed' -model_name VARCHAR(100) -- 'gemma4:e4b', 'nomic-embed-text' +model_name VARCHAR(100) -- 'gemma4:e2b', 'nomic-embed-text' confidence_score DECIMAL(5,4) -- 0.0000 – 1.0000 ai_suggestion_json JSON human_override_json JSON NULL @@ -127,7 +127,7 @@ interface AiRealtimeJobData { projectPublicId: string; // UUIDv7 — required userId: number; // INT internal ID (for audit only) payload: { - // ai-suggest: { pdfPath: string; pages: 1-3 } + // ai-suggest: { pdfPath: string; pages: 1-5 } // rag-query: { question: string; topK: number } }; idempotencyKey: string; @@ -142,7 +142,7 @@ interface AiBatchJobData { projectPublicId: string; // UUIDv7 — required payload: { // ocr: { pdfPath: string } - // extract-metadata: { textContent: string; maxPages: 3 } + // extract-metadata: { textContent: string; maxPages: 5 } // embed-document: { pdfPath: string; chunkSize: 512; overlap: 64 } }; batchId?: string; // สำหรับ Legacy Migration เท่านั้น diff --git a/specs/300-others/302-ai-model-revision/plan.md b/specs/300-others/302-ai-model-revision/plan.md index aa71df50..ee001c61 100644 --- a/specs/300-others/302-ai-model-revision/plan.md +++ b/specs/300-others/302-ai-model-revision/plan.md @@ -7,7 +7,7 @@ ## Summary -Implement ADR-023A AI Architecture Revision: เปลี่ยน model stack จาก 3-model (gemma4:9b + Typhoon + nomic-embed-text) เป็น 2-model (gemma4:e4b Q8_0 + nomic-embed-text), แยก BullMQ เป็น 2 queues (`ai-realtime`/`ai-batch`), เพิ่ม OCR auto-detection, enforce multi-tenant QdrantService, implement Legacy Migration pipeline และ migration_review_queue, และลบ Typhoon Cloud API ออกจาก codebase ทั้งหมด +Implement ADR-023A AI Architecture Revision: เปลี่ยน model stack จาก 3-model (gemma4:9b + Typhoon + nomic-embed-text) เป็น 2-model (gemma4:e2b + nomic-embed-text), แยก BullMQ เป็น 2 queues (`ai-realtime`/`ai-batch`), เพิ่ม OCR auto-detection, enforce multi-tenant QdrantService, implement Legacy Migration pipeline และ migration_review_queue, และลบ Typhoon Cloud API ออกจาก codebase ทั้งหมด --- @@ -22,7 +22,7 @@ Implement ADR-023A AI Architecture Revision: เปลี่ยน model stack **Testing**: Jest (NestJS unit/integration) **Target Platform**: QNAP NAS (NestJS container) + Admin Desktop Desk-5439 (Ollama) **Performance Goals**: ai-suggest < 30s; rag-query < 10s (p95 dequeue-to-response) -**Constraints**: VRAM ≤ 5GB peak, concurrency=1 per queue (prevent GPU overflow) +**Constraints**: VRAM ≤ 3GB peak, concurrency=1 per queue (prevent GPU overflow) **Scale/Scope**: ~20,000 legacy docs (migration), ~50 new docs/day (production) --- @@ -82,7 +82,7 @@ backend/src/modules/ai/ │ ├── ai-realtime.processor.ts # new: ai-realtime consumer │ └── ai-batch.processor.ts # new: ai-batch consumer (replaces existing) ├── services/ -│ ├── ollama.service.ts # update: model → gemma4:e4b +│ ├── ollama.service.ts # update: model → gemma4:e2b │ ├── qdrant.service.ts # update: enforce projectPublicId param │ ├── ocr.service.ts # new: OCR auto-detect + PaddleOCR routing │ ├── migration.service.ts # new: Legacy Migration pipeline @@ -123,7 +123,7 @@ Tasks: T001–T008 ### Phase 1: Core AI Pipeline -**Goal**: OCR auto-detect, gemma4:e4b integration, ai-suggest + embed-document flows +**Goal**: OCR auto-detect, gemma4:e2b integration, ai-suggest + embed-document flows Tasks: T009–T022 diff --git a/specs/300-others/302-ai-model-revision/quickstart.md b/specs/300-others/302-ai-model-revision/quickstart.md index 1559c0e4..cfccef32 100644 --- a/specs/300-others/302-ai-model-revision/quickstart.md +++ b/specs/300-others/302-ai-model-revision/quickstart.md @@ -11,9 +11,9 @@ **Requirements:** - **OS**: Windows 10/11 หรือ Linux (Desk-5439) -- **GPU**: NVIDIA GPU ที่รองรับ CUDA 11.8+ (VRAM ≥ 6GB แนะนำ) +- **GPU**: NVIDIA GPU ที่รองรับ CUDA 11.8+ (VRAM ≥ 4GB แนะนำ) - **Ollama Version**: ≥ 0.5.0 -- **Models**: `gemma4:e2b` (Q4_K_M quantization) + `nomic-embed-text` +- **Models**: `gemma4:e2b` (Q4 quantization) + `nomic-embed-text` **Verification Steps:** @@ -30,7 +30,7 @@ nvidia-smi ollama list # Expected output: # NAME ID SIZE MODIFIED -# gemma4:e2b 2.4 GB +# gemma4:e2b 2.0 GB # nomic-embed-text 274 MB # 4. Test model inference (quick test) @@ -54,7 +54,7 @@ ollama pull nomic-embed-text # Verify VRAM usage during inference nvidia-smi --query-gpu=memory.used --format=csv,noheader -# Expected: < 5120 MB (5GB threshold per SC-003) +# Expected: < 3072 MB (3GB threshold per SC-003) ``` **Troubleshooting:** @@ -285,7 +285,7 @@ curl http://192.168.10.XX:8765/health ### 7. GPU Resource Monitoring (Critical for SC-003) **Requirements:** -- **VRAM Limit**: ≤ 5GB peak (per SC-003) +- **VRAM Limit**: ≤ 3GB peak (per SC-003) - **Concurrency**: 1 job per queue (enforced by BullMQ) **Verification Commands:** @@ -303,12 +303,12 @@ nvidia-smi --query-gpu=timestamp,memory.used,utilization.gpu \ ``` **Expected Behavior:** -- **ai-batch job**: VRAM peaks at ~2.5GB (gemma4:e2b Q4_K_M) -- **ai-realtime job**: VRAM peaks at ~2.5GB (same model) +- **ai-batch job**: VRAM peaks at ~2.0GB (gemma4:e2b Q4) +- **ai-realtime job**: VRAM peaks at ~2.0GB (same model) - **No concurrent jobs**: ai-batch pauses when ai-realtime active (GPU protection) **Troubleshooting:** -- **VRAM overflow (>5GB)**: Reduce model quantization or increase GPU memory +- **VRAM overflow (>3GB)**: Reduce model quantization or increase GPU memory - **GPU contention**: Verify BullMQ concurrency=1 enforcement - **Slow inference**: Check GPU utilization, consider faster model quantization @@ -399,7 +399,7 @@ grep -r "typhoon" backend/src --include="*.ts" # 2. Measure VRAM peak during job run (verify SC-003): nvidia-smi --query-gpu=memory.used --format=csv,noheader -# Expected: value < 5120 MB (5GB threshold per SC-003) +# Expected: value < 3072 MB (3GB threshold per SC-003) # Repeat during both ai-batch and ai-realtime jobs to verify peak ``` diff --git a/specs/300-others/302-ai-model-revision/research.md b/specs/300-others/302-ai-model-revision/research.md index 36b6c876..36a7bc8a 100644 --- a/specs/300-others/302-ai-model-revision/research.md +++ b/specs/300-others/302-ai-model-revision/research.md @@ -8,13 +8,13 @@ ## Decision 1: Model Stack Reduction -- **Decision**: ใช้ 2-model stack: `gemma4:e4b Q8_0` + `nomic-embed-text` แทน 3-model stack เดิม -- **Rationale**: VRAM budget RTX 2060 Super 8GB — 3-model stack (gemma4:9b + Typhoon + nomic-embed-text) ใช้ ~7.8GB ไม่มี headroom; 2-model stack ใช้ ~4.5GB peak มี headroom ~3.5GB +- **Decision**: ใช้ 2-model stack: `gemma4:e2b` + `nomic-embed-text` แทน 3-model stack เดิม +- **Rationale**: VRAM budget RTX 2060 Super 8GB — 3-model stack (gemma4:9b + Typhoon + nomic-embed-text) ใช้ ~7.8GB ไม่มี headroom; 2-model stack ใช้ ~2.5GB peak มี headroom ~5.5GB - **Alternatives considered**: - gemma4:9b + nomic-embed-text (ไม่มี Typhoon): ยังเกิน budget ~6.8GB - - gemma4:e4b Q4_K_M (quantize ต่ำกว่า): ประหยัด VRAM มากกว่าแต่คุณภาพต่ำกว่า Q8_0 + - gemma4:e4b Q8_0: ใช้ VRAM ~4.5GB แต่ context window น้อยกว่า - ย้ายไปใช้ Cloud AI: ขัดกับ ADR-023 (INTERNAL data — ห้าม Cloud) -- **VRAM Detail**: gemma4:e4b Q8_0 = ~4.0GB weights + ~0.2GB KV Cache (จำกัดโดย 3-page input limit) + nomic-embed-text ~0.3GB = **~4.5GB peak** +- **VRAM Detail**: gemma4:e2b Q4 = ~2GB weights + ~0.2GB KV Cache (จำกัดโดย 5-page input limit) + nomic-embed-text ~0.3GB = **~2.5GB peak** --- @@ -24,7 +24,7 @@ - **Rationale**: Single queue ทำให้ RAG Q&A (interactive, p95 < 10s) ถูก block โดย OCR/Embed batch jobs (ไม่มี SLA); 2-queue ให้ priority separation โดยไม่เพิ่ม Worker ที่ทำให้ VRAM overflow - **Alternatives considered**: - Single queue + priority field: priority ใน BullMQ ไม่ป้องกัน long-running job ที่กำลังรันอยู่ block queue ถัดไป - - 2 Queues + 2 Workers พร้อมกัน: VRAM overflow เมื่อทั้งคู่ใช้ gemma4:e4b พร้อมกัน + - 2 Queues + 2 Workers พร้อมกัน: VRAM overflow เมื่อทั้งคู่ใช้ gemma4:e2b พร้อมกัน - **Implementation**: BullMQ `active` event บน `ai-realtime` → pause `ai-batch`; `completed`/`failed` → resume `ai-batch` --- @@ -72,7 +72,7 @@ ## Decision 7: Threshold Recalibration Policy - **Decision**: ใช้ค่าเริ่มต้น 0.85/0.60 สำหรับ Migration Phase แรก แล้ว recalibrate หลัง 100-500 ฉบับแรก -- **Rationale**: ค่าเดิมถูกกำหนดในยุค gemma4:9b — distribution อาจเปลี่ยนไปกับ gemma4:e4b; recalibrate จาก real data ดีกว่า hardcode ค่าใหม่โดยไม่มีข้อมูล +- **Rationale**: ค่าเดิมถูกกำหนดในยุค gemma4:9b — distribution อาจเปลี่ยนไปกับ gemma4:e2b; recalibrate จาก real data ดีกว่า hardcode ค่าใหม่โดยไม่มีข้อมูล - **Trigger**: REJECTED rate > 30% หรือ Admin override rate > 40% → ปรับลด threshold --- @@ -81,6 +81,6 @@ | Assumption | Risk | Mitigation | |-----------|------|-----------| -| gemma4:e4b Q8_0 รองรับภาษาไทยได้ดีเพียงพอ | HIGH — ไม่มีหลักฐานเชิงคุณภาพ | ทดสอบ 50-100 ฉบับก่อน Go-live; เตรียม Prompt Engineering ชดเชย | -| 3-page limit เพียงพอสำหรับ metadata extraction | MEDIUM — บางเอกสารอาจมี title block หน้า 4+ | ตรวจสอบตัวอย่างเอกสาร 20 ฉบับก่อน implementation | +| gemma4:e2b รองรับภาษาไทยได้ดีเพียงพอ | HIGH — ไม่มีหลักฐานเชิงคุณภาพ | ทดสอบ 50-100 ฉบับก่อน Go-live; เตรียม Prompt Engineering ชดเชย | +| 5-page limit เพียงพอสำหรับ metadata extraction | MEDIUM — บางเอกสารอาจมี title block หน้า 6+ | ตรวจสอบตัวอย่างเอกสาร 20 ฉบับก่อน implementation | | RTX 2060 Super VRAM ใช้ได้ 8GB เต็ม | LOW — GPU อาจมี overhead จาก OS และ driver | monitor จริงด้วย `nvidia-smi` ระหว่าง UAT | diff --git a/specs/300-others/302-ai-model-revision/spec.md b/specs/300-others/302-ai-model-revision/spec.md index 75c64d28..13dbb24f 100644 --- a/specs/300-others/302-ai-model-revision/spec.md +++ b/specs/300-others/302-ai-model-revision/spec.md @@ -95,7 +95,7 @@ Admin สามารถดู AI Performance metrics จาก ai_audit_logs (c ### Functional Requirements - **FR-001**: ระบบ MUST ตรวจจับประเภท PDF (Digital vs Scanned) อัตโนมัติโดยใช้ `extracted_chars > OCR_CHAR_THRESHOLD` โดยไม่ให้ User เลือก -- **FR-002**: ระบบ MUST ส่ง PDF เข้า gemma4:e4b สูงสุด 3 หน้าแรกเท่านั้น สำหรับงาน Classification และ Tagging +- **FR-002**: ระบบ MUST ส่ง PDF เข้า gemma4:e2b สูงสุด 5 หน้าแรกเท่านั้น สำหรับงาน Classification และ Tagging - **FR-003**: ระบบ MUST ฝัง Vector จากเอกสารทั้งฉบับ (full-document chunking) สำหรับ RAG — ไม่จำกัด 3 หน้า - **FR-004**: AI Inference ทั้งหมด MUST ผ่าน BullMQ Worker บน NestJS — ห้าม n8n เรียก Ollama โดยตรง - **FR-005**: `QdrantService.search()` MUST รับ `projectPublicId: string` เป็น required parameter เสมอ @@ -129,7 +129,7 @@ Admin สามารถดู AI Performance metrics จาก ai_audit_logs (c - **SC-001**: AI Suggestion ปรากฏบนฟอร์มภายใน 30 วินาที สำหรับ Digital PDF และ 90 วินาที สำหรับ Scanned PDF (p95) - **SC-002**: RAG Q&A ตอบกลับภายใน 10 วินาที (p95 นับจาก dequeue จาก `ai-realtime`) -- **SC-003**: VRAM peak ไม่เกิน 5GB เมื่อรัน 2 models พร้อมกัน (gemma4:e4b + nomic-embed-text) — วัดด้วย `nvidia-smi --query-gpu=memory.used --format=csv,noheader` ระหว่าง job run (ดู verification ใน quickstart.md Scenario 6, QuizMe 2026-05-15) +- **SC-003**: VRAM peak ไม่เกิน 3GB เมื่อรัน 2 models พร้อมกัน (gemma4:e2b + nomic-embed-text) — วัดด้วย `nvidia-smi --query-gpu=memory.used --format=csv,noheader` ระหว่าง job run (ดู verification ใน quickstart.md Scenario 6, QuizMe 2026-05-15) - **SC-004**: ไม่มี data leak ข้ามโครงการใน RAG — ทุก Qdrant query มี `project_public_id` filter (ตรวจสอบได้จาก query log) - **SC-005**: Legacy Migration Batch 20,000 ฉบับ ประมวลผลสำเร็จโดยไม่มี duplicate record (ตรวจสอบด้วย Idempotency-Key) - **SC-006**: admin_override_rate < 40% หลัง Calibration Phase (100-500 ฉบับแรก) @@ -140,7 +140,7 @@ Admin สามารถดู AI Performance metrics จาก ai_audit_logs (c ## Assumptions -- Desk-5439 พร้อมใช้งานและมี Ollama ที่ติดตั้ง `gemma4:e4b Q8_0` และ `nomic-embed-text` แล้ว +- Desk-5439 พร้อมใช้งานและมี Ollama ที่ติดตั้ง `gemma4:e2b` และ `nomic-embed-text` แล้ว - Qdrant instance พร้อมใช้งานและ accessible จาก NestJS backend - n8n instance สามารถ call DMS API ผ่าน HTTP ได้ - PaddleOCR ติดตั้งบน Desk-5439 พร้อมรองรับภาษาไทย @@ -150,7 +150,7 @@ Admin สามารถดู AI Performance metrics จาก ai_audit_logs (c ## Clarifications ### Session 2026-05-15 -- Q: RAG embedding scope — embed ทั้งฉบับหรือแค่ 3 หน้า? → A: ทั้งฉบับ (chunked 512t/64t overlap) — 3-page limit ใช้เฉพาะ Classification/Tagging +- Q: RAG embedding scope — embed ทั้งฉบับหรือแค่ 5 หน้า? → A: ทั้งฉบับ (chunked 512t/64t overlap) — 5-page limit ใช้เฉพาะ Classification/Tagging - Q: embed-document trigger timing → A: AUTO ทันทีหลัง commit (parallel กับ AI Suggestion), ไม่รอ Human confirm - Q: n8n role → A: n8n call DMS API เท่านั้น (`POST /api/ai/jobs`) — ไม่เรียก Ollama/Qdrant โดยตรง - Q: QdrantService enforcement → A: `projectPublicId: string` เป็น required param — ไม่มี optional fallback diff --git a/specs/README.md b/specs/README.md index d4d95e23..975b598d 100644 --- a/specs/README.md +++ b/specs/README.md @@ -193,7 +193,7 @@ specs/ 5. **No `any` Types:** ไม่อนุญาตให้ใช้ `any` ในโค้ด พยายามใช้ Validation ผ่าน DTO / Zod แบบ Strongly-typed เสมอ — **Enforced ✅** (0 remaining in backend as of v1.9.0, ดูเทคนิคที่ `05-02-backend-guidelines.md`) -6. **AI Isolation (ADR-023/023A):** Ollama ต้องรันบน **Admin Desktop** (Desk-5439) เท่านั้น — ห้ามรันบน QNAP/Production Server ห้ามมี Direct DB Access โดยเด็ดขาด AI Output ต้องผ่าน Backend Validation ก่อน Write ทุกครั้ง ใช้ 2-Model Stack (gemma4:e4b Q8_0 + nomic-embed-text) + BullMQ 2-Queue (ai-realtime/ai-batch) +6. **AI Isolation (ADR-023/023A):** Ollama ต้องรันบน **Admin Desktop** (Desk-5439) เท่านั้น — ห้ามรันบน QNAP/Production Server ห้ามมี Direct DB Access โดยเด็ดขาด AI Output ต้องผ่าน Backend Validation ก่อน Write ทุกครั้ง ใช้ 2-Model Stack (gemma4:e2b + nomic-embed-text) + BullMQ 2-Queue (ai-realtime/ai-batch) 7. **UAT Sign-off Required:** ห้าม Close UAT โดยไม่มี Acceptance Criteria ✅ ครบทุกข้อ — ดู `01-05-acceptance-criteria.md`