260227:1640 20260227: add ollama #2
All checks were successful
Build and Deploy / deploy (push) Successful in 2m37s
All checks were successful
Build and Deploy / deploy (push) Successful in 2m37s
This commit is contained in:
admin
@@ -971,4 +971,7 @@
|
|||||||
},
|
},
|
||||||
],
|
],
|
||||||
},
|
},
|
||||||
|
"extensions": {
|
||||||
|
"recommendations": ["jlcodes.antigravity-cockpit"],
|
||||||
|
},
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,15 +1,19 @@
|
|||||||
---
|
|
||||||
description: legacy PDF document migration to system v1.8.0 uses n8n and Ollama
|
|
||||||
version: 1.8.0
|
|
||||||
---
|
|
||||||
|
|
||||||
# 03-04: Legacy Data Migration Plan (PDF 20k Docs)
|
# 03-04: Legacy Data Migration Plan (PDF 20k Docs)
|
||||||
|
|
||||||
|
| description | version |
|
||||||
|
| ------------------------------------------------------------------ | ------- |
|
||||||
|
| legacy PDF document migration to system v1.8.0 uses n8n and Ollama | 1.8.0 |
|
||||||
|
|
||||||
|
> **Note:** Category Enum system-driven, Idempotency Contract, Duplicate Handling Clarification, Storage Enforcement, Audit Log Enhancement, Review Queue Integration, Revision Drift Protection, Execution Time, Encoding Normalization, Security Hardening, AI Physical Isolation (ASUSTOR), Folder Standard (/data/dms)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
## 1. วัตถุประสงค์ (Objectives)
|
## 1. วัตถุประสงค์ (Objectives)
|
||||||
|
|
||||||
* นำเข้าเอกสาร PDF 20,000 ฉบับ พร้อม Metadata จาก Excel (Legacy system export) เข้าสู่ระบบ LCBP3-DMS
|
- นำเข้าเอกสาร PDF 20,000 ฉบับ พร้อม Metadata จาก Excel (Legacy system export) เข้าสู่ระบบ LCBP3-DMS
|
||||||
* ใช้ AI (Ollama Local Model) เพื่อตรวจสอบความถูกต้องของลักษณะข้อมูล (Data format, Title consistency) ก่อนการนำเข้า
|
- ใช้ AI (Ollama Local Model) เพื่อตรวจสอบความถูกต้องของลักษณะข้อมูล (Data format, Title consistency) ก่อนการนำเข้า
|
||||||
* รักษาโครงสร้างความสัมพันธ์ (Project / Contract / Ref No.) และระบบการทำ Revision ตาม Business Rules
|
- รักษาโครงสร้างความสัมพันธ์ (Project / Contract / Ref No.) และระบบการทำ Revision ตาม Business Rules
|
||||||
|
- **Checkpoint Support:** รองรับการหยุดและเริ่มงานต่อ (Resume) จากจุดที่ค้างอยู่ได้กรณีเกิดเหตุขัดข้อง
|
||||||
|
|
||||||
> **Note:** เอกสารนี้ขยายความถึงวิธีปฏิบัติ (Implementation) จากการตัดสินใจทางสถาปัตยกรรมใน [ADR-017: Ollama Data Migration Architecture](../06-Decision-Records/ADR-017-ollama-data-migration.md)
|
> **Note:** เอกสารนี้ขยายความถึงวิธีปฏิบัติ (Implementation) จากการตัดสินใจทางสถาปัตยกรรมใน [ADR-017: Ollama Data Migration Architecture](../06-Decision-Records/ADR-017-ollama-data-migration.md)
|
||||||
|
|
||||||
@@ -17,13 +21,11 @@ version: 1.8.0
|
|||||||
|
|
||||||
## 2. โครงสร้างพื้นฐาน (Migration Infrastructure)
|
## 2. โครงสร้างพื้นฐาน (Migration Infrastructure)
|
||||||
|
|
||||||
การนำเข้าข้อมูลชุดใหญ่นี้จะไม่กระทำผ่าน User Interface แต่จะใช้โครงสร้างสถาปัตยกรรมชั่วคราวและ APIs:
|
- **Migration Orchestrator:** n8n (รันจาก Docker Container บน ASUSTOR NAS)
|
||||||
|
- **AI Validator:** Ollama (รันใน Internal Network บน ASUSTOR NAS)
|
||||||
* **Migration Orchestrator:** n8n (รันจาก Docker Container บน QNAP NAS)
|
- **Target Database:** MariaDB (`correspondences` table) บน QNAP NAS
|
||||||
* **AI Validator:** Ollama Native (รันบน Windows Desktop - i9 + RTX 2060 SUPER)
|
- **Target Storage:** QNAP File System — **ผ่าน Backend StorageService API เท่านั้น** (ห้าม move file โดยตรง)
|
||||||
* **Target Database:** MariaDB (`correspondences` table)
|
- **Connection:** 2.5G LAN + LACP / Internal VLAN
|
||||||
* **Target Storage:** QNAP File System (Mount volumes เข้า Application)
|
|
||||||
* **Connection:** ข้อมูลก้อนใหญ่ถูกโยกย้ายผ่าน 2.5G LAN + LACP เพื่อประสิทธิผลสูงสุด
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -31,71 +33,395 @@ version: 1.8.0
|
|||||||
|
|
||||||
### Phase 1: การเตรียม Infrastructure และ Storage (สัปดาห์ที่ 1)
|
### Phase 1: การเตรียม Infrastructure และ Storage (สัปดาห์ที่ 1)
|
||||||
|
|
||||||
1. **File Migration:**
|
**File Migration:**
|
||||||
ย้ายไฟล์ PDF ทั้งหมดจากแหล่งเก็บ (Desktop/External Drive) ไปยัง Folder ชั่วคราวบน NAS เพื่อรอการประมวลผล แนะนำใช้ `Robocopy` หรือ `Rsync`
|
- ย้ายไฟล์ PDF ทั้งหมดจากแหล่งเก็บไปยัง Folder ชั่วคราวบน NAS (QNAP)
|
||||||
* *Target Path:* `/share/DMS_Storage/migration_temp/`
|
- Target Path: `/data/dms/staging_ai/`
|
||||||
|
|
||||||
2. **Mount Folder:**
|
**Mount Folder:**
|
||||||
ทำการ Bind Mount โฟลเดอร์ `migration_temp/` เข้ากับ Container ของ n8n เพื่อให้ n8n เช็คความมีอยู่ของไฟล์ด้วย Disk I/O speed.
|
- Bind Mount `/data/dms/staging_ai/` เข้ากับ n8n Container แบบ **read-only**
|
||||||
|
- สร้าง `/data/dms/migration_logs/` Volume แยกสำหรับเขียน Log แบบ **read-write**
|
||||||
|
|
||||||
3. **Ollama Config:**
|
**Ollama Config:**
|
||||||
* ติดตั้ง Ollama แบบ Native บน Desktop
|
- ติดตั้ง Ollama บน ASUSTOR NAS
|
||||||
* ตั้งค่า Environment Variable `OLLAMA_HOST=0.0.0.0`
|
- No DB credentials, Internal network only
|
||||||
* Fix IP ให้ Desktop เครื่องโฮสต์ และเปิด Port `11434` ที่ระดับ OS Firewall
|
|
||||||
* รันคำสั่ง `ollama pull llama3.2` (หรือ Model ที่เหมาะสม)
|
```bash
|
||||||
|
# แนะนำ: llama3.2:3b (เร็ว, VRAM ~3GB, เหมาะ Classification)
|
||||||
|
ollama pull llama3.2:3b
|
||||||
|
|
||||||
|
# Fallback: mistral:7b-instruct-q4_K_M (แม่นกว่า, VRAM ~5GB)
|
||||||
|
# ollama pull mistral:7b-instruct-q4_K_M
|
||||||
|
```
|
||||||
|
|
||||||
|
**ทดสอบ Ollama:**
|
||||||
|
```bash
|
||||||
|
curl http://<OLLAMA_HOST>:11434/api/generate \
|
||||||
|
-d '{"model":"llama3.2:3b","prompt":"reply: ok","stream":false}'
|
||||||
|
```
|
||||||
|
|
||||||
|
**Concurrency Configuration:**
|
||||||
|
- Sequential: Batch Size = 1, Delay ≥ 2 วินาที, ปิด Parallel Execution
|
||||||
|
- เพิ่ม Health Check Node ก่อนเริ่ม Batch เพื่อป้องกัน Workflow ค้างหาก Desktop Sleep หรือ Overheat
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
### Phase 2: การเตรียม Target Database และ API (สัปดาห์ที่ 1)
|
### Phase 2: การเตรียม Target Database และ API (สัปดาห์ที่ 1)
|
||||||
|
|
||||||
1. **SQL Indexing:**
|
**SQL Indexing:**
|
||||||
เพื่อให้ API ตรวจจับ Duplicate record ได้อย่างรวดเร็ว (สำหรับ 20,000 แถว) ให้กระทำคำสั่ง SQL เพื่อเพิ่ม Index ลงฐานข้อมูล Production ชั่วคราว (หรือถาวร):
|
|
||||||
```sql
|
```sql
|
||||||
ALTER TABLE correspondences ADD INDEX idx_doc_number (document_number);
|
ALTER TABLE correspondences ADD INDEX idx_doc_number (document_number);
|
||||||
ALTER TABLE correspondences ADD INDEX idx_deleted_at (deleted_at);
|
ALTER TABLE correspondences ADD INDEX idx_deleted_at (deleted_at);
|
||||||
|
ALTER TABLE correspondences ADD INDEX idx_created_by (created_by);
|
||||||
```
|
```
|
||||||
|
|
||||||
2. **API Authentication:**
|
**Checkpoint Table:**
|
||||||
ระบบ LCBP3-DMS ต้องสร้าง Access Token แบบผูกพันกับ Role ระดับสูง (เช่น `SYSTEM_ADMIN` หรือ `MIGRATION_USER`) ซึ่งมีสิทธิ์ Bypass การ Validation บางประการ (ถ้าได้รับการอนุญาต) ส่งมอบให้ n8n
|
```sql
|
||||||
|
CREATE TABLE IF NOT EXISTS migration_progress (
|
||||||
|
batch_id VARCHAR(50) PRIMARY KEY,
|
||||||
|
last_processed_index INT DEFAULT 0,
|
||||||
|
status ENUM('RUNNING','COMPLETED','FAILED') DEFAULT 'RUNNING',
|
||||||
|
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP
|
||||||
|
);
|
||||||
|
```
|
||||||
|
|
||||||
|
**Idempotency Table :**
|
||||||
|
```sql
|
||||||
|
CREATE TABLE IF NOT EXISTS import_transactions (
|
||||||
|
id INT AUTO_INCREMENT PRIMARY KEY,
|
||||||
|
idempotency_key VARCHAR(255) UNIQUE NOT NULL,
|
||||||
|
document_number VARCHAR(100),
|
||||||
|
batch_id VARCHAR(100),
|
||||||
|
status_code INT DEFAULT 201,
|
||||||
|
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||||
|
INDEX idx_idem_key (idempotency_key)
|
||||||
|
);
|
||||||
|
```
|
||||||
|
|
||||||
|
> **Idempotency Logic:** ถ้า `idempotency_key` ซ้ำ → Backend คืน HTTP 200 ทันที (ไม่สร้าง Revision ซ้ำ) ถ้าไม่ซ้ำ → ประมวลผลปกติ
|
||||||
|
|
||||||
|
**API Authentication — Migration Token:**
|
||||||
|
```sql
|
||||||
|
INSERT INTO users (username, email, role, is_active)
|
||||||
|
VALUES ('migration_bot', 'migration@system.internal', 'SYSTEM_ADMIN', true);
|
||||||
|
```
|
||||||
|
|
||||||
|
**Scope ของ Migration Token (Patch — คำนิยามชัดเจน):**
|
||||||
|
|
||||||
|
| สิทธิ์ | ปกติ | Migration Token | หมายเหตุ |
|
||||||
|
| ------------------------------------- | --- | --------------- | --------------------------------- |
|
||||||
|
| Bypass File Virus Scan | ❌ | ✅ | ไฟล์ผ่าน Scan มาแล้วก่อน Import |
|
||||||
|
| Bypass Duplicate **Validation Error** | ❌ | ✅ | **Revision Logic ยัง enforce ปกติ** |
|
||||||
|
| Bypass Created-by User validation | ❌ | ✅ | |
|
||||||
|
| Overwrite existing revision | ❌ | ❌ | **ห้ามโดยเด็ดขาด** |
|
||||||
|
| Delete previous revision | ❌ | ❌ | **ห้ามโดยเด็ดขาด** |
|
||||||
|
| ลบ / แก้ไข Record อื่น | ❌ | ❌ | **ห้ามโดยเด็ดขาด** |
|
||||||
|
|
||||||
|
> ⚠️ **Patch Clarification:** "Bypass Duplicate Number Check" ถูกแทนด้วย "Bypass Duplicate **Validation Error**" — Revision increment logic ยังทำงานตามปกติทุกกรณี
|
||||||
|
|
||||||
|
- **Token Expiry:** ไม่เกิน **7 วัน** ต้อง Revoke ทันทีหลัง Migration เสร็จ
|
||||||
|
- **IP Whitelist:** ใช้ได้เฉพาะจาก `<NAS_IP>` เท่านั้น
|
||||||
|
- **Audit:** ทุก Request บันทึก `created_by = 'SYSTEM_IMPORT'`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
### Phase 3: การออกแบบ n8n Workflow (The Migration Logic)
|
### Phase 3: การออกแบบ n8n Workflow (The Migration Logic)
|
||||||
|
|
||||||
Workflow ควบคุมการไหลของข้อมูลประกอบด้วย 4 ส่วนการทำงาน:
|
#### Node 0: Pre-flight Health Check + Fetch System Categories
|
||||||
|
|
||||||
1. **Data Reader Node:**
|
ตรวจสอบทุก dependency ก่อน Batch:
|
||||||
อ่านไฟล์ Metadata จาก Excel แล้วแยกส่วน (Batching) เพื่อทยอยดำเนินการทีละ 100 แถว
|
|
||||||
2. **File Validator Node:**
|
1. HTTP GET Ollama `/api/tags` → ต้อง HTTP 200
|
||||||
ตรวจจับว่าเส้นทางนามสกุลไฟล์ เช่น `Iyyccnnnn-[doc_number].pdf` มีอยู่จริงในระบบเก็บข้อมูล NAS (บริเวณโฟลเดอร์ temp)
|
2. MariaDB `SELECT 1` → ต้องเชื่อมได้
|
||||||
3. **AI Analysis Node (HTTP Request to Ollama):**
|
3. HTTP GET Backend `/health` → ต้อง HTTP 200
|
||||||
* ส่ง Metadata เช่น (Document Number, Title) ให้ AI ตรวจสอบ
|
4. File Mount Check → `staging_ai` มีไฟล์, `migration_logs` เขียนได้
|
||||||
* *System Prompt Example:* "You are a Document Controller. Verify if the document title [Title] matches the numbering pattern [Pattern]. Categorize this into [Category List]. Output in JSON format only."
|
|
||||||
4. **Data Ingestion Node:**
|
**Fetch System Categories (Patch — ห้าม hardcode):**
|
||||||
* ใช้ HTTP Request ยิง POST เวิร์กโหลดเข้าไปที่ Backend API
|
```http
|
||||||
* Backend API ต้องมีความสามารถในการรองรับ *Idempotency* และจัดการตรวจสอบว่าหาก `document_number` เกิดซ้ำกัน ต้องยกระดับไปสร้างบันทึกใหม่เป็น Version / Revision (+1) ไม่ใช่การทับลง Record เดิม
|
GET /api/meta/categories
|
||||||
|
Authorization: Bearer <MIGRATION_TOKEN>
|
||||||
|
```
|
||||||
|
Response:
|
||||||
|
```json
|
||||||
|
{ "categories": ["Correspondence","RFA","Drawing","Transmittal","Report","Other"] }
|
||||||
|
```
|
||||||
|
n8n ต้องเก็บ categories นี้ไว้ใน Workflow Variable (`system_categories`) และ inject เข้า AI Prompt ทุก Request
|
||||||
|
|
||||||
|
#### Node 1: Data Reader & Checkpoint
|
||||||
|
|
||||||
|
- อ่าน Checkpoint จาก **MariaDB Node แยก** (ไม่ใช่ async call ใน Code Node)
|
||||||
|
- Batch ทีละ **10–20 แถว** ตาม `$env.MIGRATION_BATCH_SIZE`
|
||||||
|
- ติด `original_index` ทุก Item
|
||||||
|
|
||||||
|
**Encoding Normalization:**
|
||||||
|
```javascript
|
||||||
|
// Normalize ข้อมูลจาก Excel เป็น UTF-8 NFC ก่อนประมวลผล
|
||||||
|
const normalize = (str) => {
|
||||||
|
if (!str) return '';
|
||||||
|
return Buffer.from(str, 'utf8').toString('utf8').normalize('NFC');
|
||||||
|
};
|
||||||
|
|
||||||
|
return items.map(item => ({
|
||||||
|
...item,
|
||||||
|
json: {
|
||||||
|
...item.json,
|
||||||
|
document_number: normalize(item.json.document_number),
|
||||||
|
title: normalize(item.json.title)
|
||||||
|
}
|
||||||
|
}));
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Node 2: File Validator & Sanitizer
|
||||||
|
|
||||||
|
- ตรวจสอบไฟล์ PDF มีอยู่จริงบน NAS
|
||||||
|
- Normalize ชื่อไฟล์เป็น **UTF-8 NFC**
|
||||||
|
- Path Traversal Guard: resolved path ต้องอยู่ใน `/data/dms/staging_ai` เท่านั้น
|
||||||
|
- **Output 0** → valid → Node 3
|
||||||
|
- **Output 1** → error → Node 5D (ไม่หายเงียบ)
|
||||||
|
|
||||||
|
#### Node 3: AI Analysis (Sequential เท่านั้น)
|
||||||
|
|
||||||
|
**System Prompt:**
|
||||||
|
```text
|
||||||
|
You are a Document Controller for a large construction project.
|
||||||
|
Your task is to validate document metadata.
|
||||||
|
You MUST respond ONLY with valid JSON. No explanation, no markdown, no extra text.
|
||||||
|
If there are no issues, "detected_issues" must be an empty array [].
|
||||||
|
```
|
||||||
|
|
||||||
|
**User Prompt (Category List มาจาก Backend ไม่ hardcode):**
|
||||||
|
```text
|
||||||
|
Validate this document metadata and respond in JSON:
|
||||||
|
|
||||||
|
Document Number: {{$json.document_number}}
|
||||||
|
Title: {{$json.title}}
|
||||||
|
Expected Pattern: [ORG]-[TYPE]-[SEQ] e.g. "TCC-COR-0001"
|
||||||
|
Category List (MUST match system enum exactly): {{$workflow.variables.system_categories}}
|
||||||
|
|
||||||
|
Respond ONLY with this exact JSON structure:
|
||||||
|
{
|
||||||
|
"is_valid": true | false,
|
||||||
|
"confidence": 0.0 to 1.0,
|
||||||
|
"suggested_category": "<one from Category List>",
|
||||||
|
"detected_issues": ["<issue1>"],
|
||||||
|
"suggested_title": "<corrected title or null>"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**JSON Validation (ตรวจ Category ตรง Enum):**
|
||||||
|
```javascript
|
||||||
|
const systemCategories = $workflow.variables.system_categories;
|
||||||
|
if (!systemCategories.includes(result.suggested_category)) {
|
||||||
|
throw new Error(`Category "${result.suggested_category}" not in system enum: ${systemCategories.join(', ')}`);
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Node 3.5: Fallback Model Manager
|
||||||
|
|
||||||
|
- อัปเดต `migration_fallback_state` ทุกครั้งที่เกิด Parse Error
|
||||||
|
- Auto-switch ไป `OLLAMA_MODEL_FALLBACK` เมื่อ Error ≥ `FALLBACK_ERROR_THRESHOLD`
|
||||||
|
- ส่ง Alert Email เมื่อ Fallback ถูก Activate
|
||||||
|
|
||||||
|
#### Node 4: Confidence Router (4 outputs)
|
||||||
|
|
||||||
|
| เงื่อนไข | การดำเนินการ |
|
||||||
|
| ------------------------------------------ | -------------------------------- |
|
||||||
|
| `confidence >= 0.85` และ `is_valid = true` | **Output 0** → Auto Ingest |
|
||||||
|
| `confidence >= 0.60` และ `< 0.85` | **Output 1** → Review Queue |
|
||||||
|
| `confidence < 0.60` หรือ `is_valid = false` | **Output 2** → Reject Log |
|
||||||
|
| Parse Error / AI ไม่ตอบ | **Output 3** → Error Log |
|
||||||
|
| Fallback: Error > 5 ใน 10 Request | สลับ Model / หยุด Workflow + Alert |
|
||||||
|
|
||||||
|
**Revision Drift Protection:**
|
||||||
|
```javascript
|
||||||
|
// ถ้า Excel มี revision column — ตรวจสอบก่อน route
|
||||||
|
if (item.json.excel_revision !== undefined) {
|
||||||
|
const expectedRevision = (item.json.current_db_revision || 0) + 1;
|
||||||
|
if (parseInt(item.json.excel_revision) !== expectedRevision) {
|
||||||
|
item.json.review_reason = `Revision drift: Excel=${item.json.excel_revision}, Expected=${expectedRevision}`;
|
||||||
|
reviewQueue.push(item);
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Node 5A: Auto Ingest — Backend API
|
||||||
|
|
||||||
|
> ⚠️ **Storage Enforcement:** n8n ส่งแค่ `source_file_path` — Backend จะ generate UUID, enforce path strategy (`/data/dms/uploads/YYYY/MM/{uuid}.pdf`), และ move file atomically ผ่าน StorageService
|
||||||
|
|
||||||
|
```http
|
||||||
|
POST /api/correspondences/import
|
||||||
|
Authorization: Bearer <MIGRATION_TOKEN>
|
||||||
|
Idempotency-Key: <document_number>:<batch_id>
|
||||||
|
Content-Type: application/json
|
||||||
|
```
|
||||||
|
|
||||||
|
Payload:
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"document_number": "{{document_number}}",
|
||||||
|
"title": "{{ai_result.suggested_title || title}}",
|
||||||
|
"category": "{{ai_result.suggested_category}}",
|
||||||
|
"source_file_path": "{{file_path}}",
|
||||||
|
"ai_confidence": "{{ai_result.confidence}}",
|
||||||
|
"ai_issues": "{{ai_result.detected_issues}}",
|
||||||
|
"migrated_by": "SYSTEM_IMPORT",
|
||||||
|
"batch_id": "{{$env.MIGRATION_BATCH_ID}}"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Audit Log ที่ Backend ต้องสร้าง:**
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"action": "IMPORT",
|
||||||
|
"source": "MIGRATION",
|
||||||
|
"batch_id": "migration_20260226",
|
||||||
|
"created_by": "SYSTEM_IMPORT",
|
||||||
|
"metadata": {
|
||||||
|
"migration": true,
|
||||||
|
"batch_id": "migration_20260226",
|
||||||
|
"ai_confidence": 0.91
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Checkpoint Update (ทุก 10 Records — ผ่าน IF Node + MariaDB Node):**
|
||||||
|
```sql
|
||||||
|
INSERT INTO migration_progress (batch_id, last_processed_index, status)
|
||||||
|
VALUES ('{{$env.MIGRATION_BATCH_ID}}', {{checkpoint_index}}, 'RUNNING')
|
||||||
|
ON DUPLICATE KEY UPDATE
|
||||||
|
last_processed_index = {{checkpoint_index}},
|
||||||
|
updated_at = NOW();
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Node 5B: Review Queue
|
||||||
|
|
||||||
|
> ⚠️ **`migration_review_queue` เป็น Temporary Table เท่านั้น** — ห้ามสร้าง Correspondence record จนกว่า Admin จะ Approve
|
||||||
|
|
||||||
|
Approval Flow:
|
||||||
|
```
|
||||||
|
Review → Admin Approve → POST /api/correspondences/import (เหมือน Auto Ingest)
|
||||||
|
Admin Reject → ลบออกจาก queue ไม่สร้าง record
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Node 5C: Reject Log → `/data/dms/migration_logs/reject_log.csv`
|
||||||
|
|
||||||
|
#### Node 5D: Error Log → `/data/dms/migration_logs/error_log.csv` + MariaDB
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
### Phase 4: แผนการทดสอบ (Testing & QA)
|
### Phase 4: แผนการทดสอบ (Testing & QA)
|
||||||
|
|
||||||
1. **Dry Run:** รันเพียง 1 Batch ข้อมูล (ขนาด ~100 แถว) ทะลวงระบบจากต้นน้ำถึงปลายน้ำ
|
**Dry Run Policy (Mandatory):**
|
||||||
2. **Integrity Check:** QA เช็คในหน้า UI และในฐานข้อมูล MariaDB ว่า Metadata โอนถ่ายได้ครบถ้วน และไฟล์ Physical file ย้ายไปสู่โฟลเดอร์ Webroot (Permanent Storage) ของเอกสาร
|
- All migrations MUST run with `--dry-run`
|
||||||
3. **Hardware Tuning:** จับตาดู RAM ของ NAS และ GPU VRAM/Temperatures ของ Desktop ฝั่ง Ollama. ปรับหน่วง Delay ระหว่าง Request ได้ในตัว n8n หากฮาร์ดแวร์ทำงานหนักเกิน
|
- No DB commit until validation approved
|
||||||
|
|
||||||
### Phase 5: การรันงานจริง (Execution & Monitoring)
|
**Dry Run Validation (20–50 แถว):**
|
||||||
|
- JSON Parse Success Rate > 95%
|
||||||
|
- Category ที่ AI ตอบตรงกับ System Enum ทุกรายการ
|
||||||
|
- รัน Batch เดิมซ้ำ 2 รอบ → ต้องไม่สร้าง Duplicate หรือ Revision ซ้ำ (Idempotency Test)
|
||||||
|
- Storage Path ตรงตาม Core Storage Spec v1.8.0
|
||||||
|
- Revision Drift ถูก route ไป Review Queue
|
||||||
|
|
||||||
1. **Scheduling:** เปิดตัวรันอัตโนมัติเฉพาะเวลากลางคืน (Offline hours)
|
**Integrity Check:**
|
||||||
2. **Log Monitoring:** เปิด Execution Logs ของ n8n ควบคู่ไปกับ Docker Logs
|
|
||||||
3. **Post-Migration Audit:** ผู้ดูแลระบบสั่งรัน Query สอบยอดหลังบ้าน:
|
|
||||||
```sql
|
```sql
|
||||||
SELECT count(*) FROM correspondences WHERE created_by = 'SYSTEM_IMPORT';
|
-- ตรวจยอด
|
||||||
|
SELECT COUNT(*) FROM correspondences WHERE created_by = 'SYSTEM_IMPORT';
|
||||||
|
|
||||||
|
-- ตรวจ Revision ซ้ำ
|
||||||
|
SELECT document_number, COUNT(*) as cnt
|
||||||
|
FROM correspondences WHERE created_by = 'SYSTEM_IMPORT'
|
||||||
|
GROUP BY document_number HAVING cnt > 1;
|
||||||
|
|
||||||
|
-- ตรวจ Idempotency Key ไม่ซ้ำ
|
||||||
|
SELECT idempotency_key, COUNT(*) as cnt
|
||||||
|
FROM import_transactions GROUP BY idempotency_key HAVING cnt > 1;
|
||||||
|
|
||||||
|
-- ตรวจ Audit Log ครบ
|
||||||
|
SELECT COUNT(*) FROM audit_logs
|
||||||
|
WHERE created_by = 'SYSTEM_IMPORT' AND action = 'IMPORT';
|
||||||
```
|
```
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 4. แผนรับมือความเสี่ยง (Risk Management)
|
### Phase 5: การรันงานจริง (Execution & Monitoring)
|
||||||
|
|
||||||
| ลำดับที่ | ความเสี่ยง | การจัดการ (Mitigation) |
|
- **Scheduling:** รันอัตโนมัติ 22:00–06:00
|
||||||
| ---- | --------------------------------- | -------------------------------------------------------------------------------------------- |
|
- **Expected Runtime:** ~3 วินาที/record (2 sec delay + ~1 sec inference) → 20,000 records ≈ **60,000 วินาที (~16.6 ชั่วโมง)** → ใช้เวลาประมาณ **3–4 คืน**
|
||||||
| 1 | AI Node หรือ GPU ค้าง (OOT/Timeout) | กำหนดให้ Node มี Node Retry Mechanism (เช่น รอ 1 นาที ทำซ้ำ 3 รอบ) ประกอบกับการจับ Wait node ไว้ดักทิศทาง |
|
- **Daily Check:** Admin ตรวจ Review Queue และ Reject Log ทุกเช้าจาก Night Summary Email
|
||||||
| 2 | หมายเลขเอกสารซ้ำซ้อนกันใน Excel | Backend ต้องมี Controller Logic รับมือ และส่งออกเลข Revision เพื่อลงฐานข้อมูล |
|
- **Progress Tracking:** อัปเดต `migration_progress` ทุก 10 Records
|
||||||
| 3 | ดิสก์ NAS ทรุด หรือ Database บวมชั่วขณะ | ปิดฟีเจอร์ "Save Successful Executions" ใน Options ของ n8n |
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
> **ข้อแนะนำด้าน Physical Storage:** หลังจากนำเข้าข้อมูลเสร็จ ตรวจสอบให้แน่ใจว่าไฟล์ PDF ทั้ง 20,000 ไฟล์ ถูกย้าย (Move) หรือคัดลอก (Copy) ไปเก็บยัง Local Storage Strategy ตามที่ได้ตกลงระบุไว้ใน Specs ฉบับที่เกี่ยวข้องกับ Storage อย่างถูกต้อง ไม่ปล่อยค้างไว้ที่ Temp Folder
|
## 4. Rollback Plan
|
||||||
|
|
||||||
|
**Step 1:** หยุด n8n และ Disable Token
|
||||||
|
```sql
|
||||||
|
UPDATE users SET is_active = false WHERE username = 'migration_bot';
|
||||||
|
```
|
||||||
|
|
||||||
|
**Step 2:** ลบ Records (Transaction)
|
||||||
|
```sql
|
||||||
|
START TRANSACTION;
|
||||||
|
DELETE FROM correspondence_files
|
||||||
|
WHERE correspondence_id IN (SELECT id FROM correspondences WHERE created_by = 'SYSTEM_IMPORT');
|
||||||
|
DELETE FROM correspondences WHERE created_by = 'SYSTEM_IMPORT';
|
||||||
|
DELETE FROM import_transactions WHERE batch_id = 'migration_20260226';
|
||||||
|
SELECT ROW_COUNT();
|
||||||
|
COMMIT;
|
||||||
|
```
|
||||||
|
|
||||||
|
**Step 3:** ย้ายไฟล์กลับ `/data/dms/staging_ai/` ผ่าน Script แยก
|
||||||
|
|
||||||
|
**Step 4:** Reset State
|
||||||
|
```sql
|
||||||
|
UPDATE migration_progress
|
||||||
|
SET status = 'FAILED', last_processed_index = 0
|
||||||
|
WHERE batch_id = 'migration_20260226';
|
||||||
|
|
||||||
|
UPDATE migration_fallback_state
|
||||||
|
SET recent_error_count = 0, is_fallback_active = FALSE
|
||||||
|
WHERE batch_id = 'migration_20260226';
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. แผนรับมือความเสี่ยง (Risk Management)
|
||||||
|
|
||||||
|
| ลำดับที่ | ความเสี่ยง | การจัดการ (Mitigation) |
|
||||||
|
| ---- | -------------------------- | -------------------------------------------------- |
|
||||||
|
| 1 | AI Node หรือ GPU ค้าง | Timeout 30 วินาที, Retry 3 รอบ, Delay 60 วินาที |
|
||||||
|
| 2 | Ollama ตอบไม่ใช่ JSON | JSON Pre-processor + ส่ง Human Review Queue |
|
||||||
|
| 3 | Category ไม่ตรง System Enum | Fetch `/api/meta/categories` ก่อน Batch ทุกครั้ง |
|
||||||
|
| 4 | Idempotency ซ้ำ | `import_transactions` table + Backend คืน HTTP 200 |
|
||||||
|
| 5 | Revision Drift | ตรวจ Excel revision column → Route ไป Review Queue |
|
||||||
|
| 6 | Storage bypass | ห้าม move file โดยตรง — ผ่าน Backend API เท่านั้น |
|
||||||
|
| 7 | GPU VRAM Overflow | ใช้เฉพาะ Quantized Model (q4_K_M) |
|
||||||
|
| 8 | ดิสก์ NAS เต็ม | ปิด "Save Successful Executions" ใน n8n |
|
||||||
|
| 9 | Migration Token ถูกขโมย | Token 7 วัน, IP Whitelist `<NAS_IP>` เท่านั้น |
|
||||||
|
| 10 | ไฟดับ/ล่มกลางคัน | Checkpoint Table → Resume จากจุดที่ค้าง |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. Post-Migration Verification
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- 1. ตรวจยอดครบ 20,000
|
||||||
|
SELECT COUNT(*) FROM correspondences WHERE created_by = 'SYSTEM_IMPORT';
|
||||||
|
|
||||||
|
-- 2. ตรวจ Revision ผิดปกติ
|
||||||
|
SELECT document_number, COUNT(*) FROM correspondences
|
||||||
|
WHERE created_by = 'SYSTEM_IMPORT'
|
||||||
|
GROUP BY document_number HAVING COUNT(*) > 5;
|
||||||
|
|
||||||
|
-- 3. ตรวจ Audit Log ครบ
|
||||||
|
SELECT COUNT(*) FROM audit_logs
|
||||||
|
WHERE created_by = 'SYSTEM_IMPORT' AND action = 'IMPORT';
|
||||||
|
|
||||||
|
-- 4. ตรวจ Idempotency ไม่มีซ้ำ
|
||||||
|
SELECT idempotency_key, COUNT(*) FROM import_transactions
|
||||||
|
GROUP BY idempotency_key HAVING COUNT(*) > 1;
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
> **ข้อแนะนำด้าน Physical Storage:** ไฟล์ PDF ทั้ง 20,000 ไฟล์จะถูก move โดย Backend StorageService ไปยัง path ที่ถูกต้องโดยอัตโนมัติ ไม่ปล่อยค้างไว้ที่ `/data/dms/staging_ai/`
|
||||||
|
|||||||
918
specs/03-Data-and-Storage/03-05-n8n-migration-setup-guide.md
Normal file
918
specs/03-Data-and-Storage/03-05-n8n-migration-setup-guide.md
Normal file
@@ -0,0 +1,918 @@
|
|||||||
|
# 📋 คู่มือการตั้งค่า n8n สำหรับ Legacy Data Migration
|
||||||
|
|
||||||
|
เอกสารนี้จัดทำขึ้นเพื่อรองรับการ Migration เอกสาร PDF 20,000 ฉบับ ตามแผนใน `03-04-legacy-data-migration.md` และ `ADR-017-ollama-data-migration.md`
|
||||||
|
|
||||||
|
> **Note:** Category Enum system-driven, Idempotency-Key Header, Storage Enforcement, Audit Log, Encoding Normalization, Security Hardening, Nginx Rate Limit, Docker Hardening, AI Physical Isolation (ASUSTOR), Folder Standard (/data/dms)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 📌 ส่วนที่ 1: การติดตั้งและตั้งค่าเบื้องต้น
|
||||||
|
|
||||||
|
### 1.1 ติดตั้ง n8n บน ASUSTOR NAS (Docker)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
mkdir -p /data/dms/n8n
|
||||||
|
cd /data/dms/n8n
|
||||||
|
|
||||||
|
cat > docker-compose.yml << 'EOF'
|
||||||
|
version: '3.8'
|
||||||
|
|
||||||
|
services:
|
||||||
|
n8n:
|
||||||
|
image: n8nio/n8n:latest
|
||||||
|
container_name: n8n-migration
|
||||||
|
restart: unless-stopped
|
||||||
|
# Docker Hardening (Patch)
|
||||||
|
mem_limit: 2g
|
||||||
|
logging:
|
||||||
|
driver: json-file
|
||||||
|
options:
|
||||||
|
max-size: "10m"
|
||||||
|
max-file: "3"
|
||||||
|
ports:
|
||||||
|
- "5678:5678"
|
||||||
|
environment:
|
||||||
|
- N8N_HOST=0.0.0.0
|
||||||
|
- N8N_PORT=5678
|
||||||
|
- N8N_PROTOCOL=http
|
||||||
|
- NODE_ENV=production
|
||||||
|
- WEBHOOK_URL=http://<NAS_IP>:5678/
|
||||||
|
- GENERIC_TIMEZONE=Asia/Bangkok
|
||||||
|
- TZ=Asia/Bangkok
|
||||||
|
- N8N_SECURE_COOKIE=false
|
||||||
|
- N8N_USER_FOLDER=/home/node/.n8n
|
||||||
|
- N8N_PUBLIC_API_DISABLED=true
|
||||||
|
- N8N_BASIC_AUTH_ACTIVE=true
|
||||||
|
- N8N_BASIC_AUTH_USER=admin
|
||||||
|
- N8N_BASIC_AUTH_PASSWORD=<strong_password>
|
||||||
|
- N8N_PAYLOAD_SIZE_MAX=10485760
|
||||||
|
- EXECUTIONS_DATA_PRUNE=true
|
||||||
|
- EXECUTIONS_DATA_MAX_AGE=168
|
||||||
|
- EXECUTIONS_DATA_PRUNE_TIMEOUT=60
|
||||||
|
- DB_TYPE=postgresdb
|
||||||
|
- DB_POSTGRESDB_HOST=<DB_IP>
|
||||||
|
- DB_POSTGRESDB_PORT=5432
|
||||||
|
- DB_POSTGRESDB_DATABASE=n8n
|
||||||
|
- DB_POSTGRESDB_USER=n8n
|
||||||
|
- DB_POSTGRESDB_PASSWORD=<password>
|
||||||
|
volumes:
|
||||||
|
- ./n8n_data:/home/node/.n8n
|
||||||
|
# read-only: อ่านไฟล์ PDF ต้นฉบับเท่านั้น
|
||||||
|
- /data/dms/staging_ai:/data/dms/staging_ai:ro
|
||||||
|
# read-write: เขียน Log และ CSV ทั้งหมด
|
||||||
|
- /data/dms/migration_logs:/data/dms/migration_logs:rw
|
||||||
|
networks:
|
||||||
|
- n8n-network
|
||||||
|
|
||||||
|
networks:
|
||||||
|
n8n-network:
|
||||||
|
driver: bridge
|
||||||
|
EOF
|
||||||
|
|
||||||
|
docker-compose up -d
|
||||||
|
```
|
||||||
|
|
||||||
|
> ⚠️ **Volume หมายเหตุ:** `/data/dms/staging_ai` = **read-only** (อ่านไฟล์ต้นฉบับ) และ `/data/dms/migration_logs` = **read-write** (เขียน Log/CSV) — ห้ามเขียน CSV ลง `staging_ai` เพราะจะ Error ทันที
|
||||||
|
|
||||||
|
### 1.2 Nginx Rate Limit
|
||||||
|
|
||||||
|
เพิ่มใน Nginx config สำหรับ Migration API:
|
||||||
|
|
||||||
|
```nginx
|
||||||
|
# nginx.conf หรือ site config
|
||||||
|
limit_req_zone $binary_remote_addr zone=migration:10m rate=1r/s;
|
||||||
|
|
||||||
|
location /api/correspondences/import {
|
||||||
|
limit_req zone=migration burst=5 nodelay;
|
||||||
|
proxy_pass http://backend:3001;
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### 1.3 Environment Variables
|
||||||
|
|
||||||
|
**Settings → Environment Variables ใน n8n UI:**
|
||||||
|
|
||||||
|
| Variable | ค่าที่แนะนำ | คำอธิบาย |
|
||||||
|
| --------------------------- | ---------------------------- | ------------------------------------ |
|
||||||
|
| `OLLAMA_HOST` | `http://<ASUSTOR_IP>:11434` | URL ของ Ollama (ใน internal network) |
|
||||||
|
| `OLLAMA_MODEL_PRIMARY` | `llama3.2:3b` | Model หลัก |
|
||||||
|
| `OLLAMA_MODEL_FALLBACK` | `mistral:7b-instruct-q4_K_M` | Model สำรอง |
|
||||||
|
| `MIGRATION_BATCH_SIZE` | `10` | จำนวน Record ต่อ Batch |
|
||||||
|
| `MIGRATION_DELAY_MS` | `2000` | Delay ระหว่าง Request (ms) |
|
||||||
|
| `CONFIDENCE_THRESHOLD_HIGH` | `0.85` | Threshold Auto Ingest |
|
||||||
|
| `CONFIDENCE_THRESHOLD_LOW` | `0.60` | Threshold Review Queue |
|
||||||
|
| `MAX_RETRY_COUNT` | `3` | จำนวนครั้ง Retry |
|
||||||
|
| `FALLBACK_ERROR_THRESHOLD` | `5` | Error ที่ trigger Fallback |
|
||||||
|
| `BACKEND_URL` | `https://<BACKEND_URL>` | URL ของ LCBP3 Backend |
|
||||||
|
| `MIGRATION_BATCH_ID` | `migration_20260226` | ID ของ Batch |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 📌 ส่วนที่ 2: การเตรียม Database
|
||||||
|
|
||||||
|
รัน SQL นี้บน MariaDB **ก่อน** เริ่ม n8n Workflow:
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- Checkpoint
|
||||||
|
CREATE TABLE IF NOT EXISTS migration_progress (
|
||||||
|
batch_id VARCHAR(50) PRIMARY KEY,
|
||||||
|
last_processed_index INT DEFAULT 0,
|
||||||
|
status ENUM('RUNNING','COMPLETED','FAILED') DEFAULT 'RUNNING',
|
||||||
|
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP
|
||||||
|
);
|
||||||
|
|
||||||
|
-- Review Queue (Temporary — ไม่ใช่ Business Schema)
|
||||||
|
CREATE TABLE IF NOT EXISTS migration_review_queue (
|
||||||
|
id INT AUTO_INCREMENT PRIMARY KEY,
|
||||||
|
document_number VARCHAR(100) NOT NULL,
|
||||||
|
title TEXT,
|
||||||
|
original_title TEXT,
|
||||||
|
ai_suggested_category VARCHAR(50),
|
||||||
|
ai_confidence DECIMAL(4,3),
|
||||||
|
ai_issues JSON,
|
||||||
|
review_reason VARCHAR(255),
|
||||||
|
status ENUM('PENDING','APPROVED','REJECTED') DEFAULT 'PENDING',
|
||||||
|
reviewed_by VARCHAR(100),
|
||||||
|
reviewed_at TIMESTAMP NULL,
|
||||||
|
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||||
|
UNIQUE KEY uq_doc_number (document_number)
|
||||||
|
);
|
||||||
|
|
||||||
|
-- Error Log
|
||||||
|
CREATE TABLE IF NOT EXISTS migration_errors (
|
||||||
|
id INT AUTO_INCREMENT PRIMARY KEY,
|
||||||
|
batch_id VARCHAR(50),
|
||||||
|
document_number VARCHAR(100),
|
||||||
|
error_type ENUM('FILE_NOT_FOUND','AI_PARSE_ERROR','API_ERROR','DB_ERROR','UNKNOWN'),
|
||||||
|
error_message TEXT,
|
||||||
|
raw_ai_response TEXT,
|
||||||
|
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||||
|
INDEX idx_batch_id (batch_id),
|
||||||
|
INDEX idx_error_type (error_type)
|
||||||
|
);
|
||||||
|
|
||||||
|
-- Fallback State
|
||||||
|
CREATE TABLE IF NOT EXISTS migration_fallback_state (
|
||||||
|
id INT AUTO_INCREMENT PRIMARY KEY,
|
||||||
|
batch_id VARCHAR(50) UNIQUE,
|
||||||
|
recent_error_count INT DEFAULT 0,
|
||||||
|
is_fallback_active BOOLEAN DEFAULT FALSE,
|
||||||
|
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP
|
||||||
|
);
|
||||||
|
|
||||||
|
-- Idempotency (Patch)
|
||||||
|
CREATE TABLE IF NOT EXISTS import_transactions (
|
||||||
|
id INT AUTO_INCREMENT PRIMARY KEY,
|
||||||
|
idempotency_key VARCHAR(255) UNIQUE NOT NULL,
|
||||||
|
document_number VARCHAR(100),
|
||||||
|
batch_id VARCHAR(100),
|
||||||
|
status_code INT DEFAULT 201,
|
||||||
|
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||||
|
INDEX idx_idem_key (idempotency_key)
|
||||||
|
);
|
||||||
|
|
||||||
|
-- Daily Summary
|
||||||
|
CREATE TABLE IF NOT EXISTS migration_daily_summary (
|
||||||
|
id INT AUTO_INCREMENT PRIMARY KEY,
|
||||||
|
batch_id VARCHAR(50),
|
||||||
|
summary_date DATE,
|
||||||
|
total_processed INT DEFAULT 0,
|
||||||
|
auto_ingested INT DEFAULT 0,
|
||||||
|
sent_to_review INT DEFAULT 0,
|
||||||
|
rejected INT DEFAULT 0,
|
||||||
|
errors INT DEFAULT 0,
|
||||||
|
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||||
|
UNIQUE KEY uq_batch_date (batch_id, summary_date)
|
||||||
|
);
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 📌 ส่วนที่ 3: Credentials
|
||||||
|
|
||||||
|
**Credentials → Add New:**
|
||||||
|
|
||||||
|
#### 🔐 Ollama API
|
||||||
|
| Field | ค่า |
|
||||||
|
| -------------- | --------------------------- |
|
||||||
|
| Name | `Ollama Local API` |
|
||||||
|
| Type | `HTTP Request` |
|
||||||
|
| Base URL | `http://<ASUSTOR_IP>:11434` |
|
||||||
|
| Authentication | `None` |
|
||||||
|
|
||||||
|
#### 🔐 LCBP3 Backend API
|
||||||
|
| Field | ค่า |
|
||||||
|
| -------------- | --------------------------- |
|
||||||
|
| Name | `LCBP3 Migration Token` |
|
||||||
|
| Type | `HTTP Request` |
|
||||||
|
| Base URL | `https://<BACKEND_URL>/api` |
|
||||||
|
| Authentication | `Header Auth` |
|
||||||
|
| Header Name | `Authorization` |
|
||||||
|
| Header Value | `Bearer <MIGRATION_TOKEN>` |
|
||||||
|
|
||||||
|
#### 🔐 MariaDB
|
||||||
|
| Field | ค่า |
|
||||||
|
| -------- | ------------------ |
|
||||||
|
| Name | `LCBP3 MariaDB` |
|
||||||
|
| Type | `MariaDB` |
|
||||||
|
| Host | `<DB_IP>` |
|
||||||
|
| Port | `3306` |
|
||||||
|
| Database | `lcbp3_production` |
|
||||||
|
| User | `migration_bot` |
|
||||||
|
| Password | `<password>` |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 📌 ส่วนที่ 4: Workflow (Step-by-Step)
|
||||||
|
|
||||||
|
### 4.1 โครงสร้างภาพรวม
|
||||||
|
|
||||||
|
```
|
||||||
|
┌──────────────────────────────────────────────────────────────────────┐
|
||||||
|
│ MIGRATION WORKFLOW v1.8.0 │
|
||||||
|
├──────────────────────────────────────────────────────────────────────┤
|
||||||
|
│ │
|
||||||
|
│ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐ │
|
||||||
|
│ │ Node 0 │──▶│ Node 1 │──▶│ Node 2 │──▶│ Node 3 │ │
|
||||||
|
│ │Pre- │ │ Data │ │ File │ │ AI │ │
|
||||||
|
│ │flight + │ │ Reader │ │ Validat.│ │Analysis │ │
|
||||||
|
│ │Fetch Cat│ │+Encoding│ │+Sanitize│ │+Enum Chk│ │
|
||||||
|
│ └─────────┘ └─────────┘ └──┬──┬───┘ └────┬────┘ │
|
||||||
|
│ │ │ │ │
|
||||||
|
│ valid │ │ error ┌───▼──────────────┐ │
|
||||||
|
│ │ └──────▶ │ Node 3.5 │ │
|
||||||
|
│ │ │ Fallback Manager │ │
|
||||||
|
│ │ └──────────────────┘ │
|
||||||
|
│ ▼ │ │
|
||||||
|
│ ┌─────────┐ ┌────▼────┐ │
|
||||||
|
│ │ Node 5D │ │ Node 4 │ │
|
||||||
|
│ │ Error │ │Confidence│ │
|
||||||
|
│ │ Log │ │+Revision │ │
|
||||||
|
│ └─────────┘ │ Drift │ │
|
||||||
|
│ └┬──┬──┬──┘ │
|
||||||
|
│ │ │ │ │
|
||||||
|
│ ┌────────────┘ │ └──────┐ │
|
||||||
|
│ ▼ ▼ ▼ │
|
||||||
|
│ ┌──────────┐ ┌──────────┐ ┌──────┐ │
|
||||||
|
│ │ Node 5A │ │ Node 5B │ │ 5C │ │
|
||||||
|
│ │ Auto │ │ Review │ │Reject│ │
|
||||||
|
│ │ Ingest │ │ Queue │ │ Log │ │
|
||||||
|
│ │+Idempot. │ │(Temp only│ └──────┘ │
|
||||||
|
│ └────┬─────┘ └──────────┘ │
|
||||||
|
│ │ │
|
||||||
|
│ ┌────▼──────┐ │
|
||||||
|
│ │ Checkpoint│ │
|
||||||
|
│ └───────────┘ │
|
||||||
|
└──────────────────────────────────────────────────────────────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 4.2 Node 0: Pre-flight + Fetch System Categories
|
||||||
|
|
||||||
|
Fetch System Categories ก่อน Batch ทุกครั้ง
|
||||||
|
|
||||||
|
**Sub-flow:**
|
||||||
|
```
|
||||||
|
[Trigger] → [HTTP: Ollama /api/tags] → [MariaDB: SELECT 1]
|
||||||
|
→ [HTTP: Backend /health] → [Code: File Mount Check]
|
||||||
|
→ [HTTP: GET /api/meta/categories] → [Store in Workflow Variable]
|
||||||
|
→ [IF all pass → Node 1] [ELSE → Stop + Alert]
|
||||||
|
```
|
||||||
|
|
||||||
|
**HTTP Node — Fetch Categories:**
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"method": "GET",
|
||||||
|
"url": "={{ $env.BACKEND_URL }}/api/meta/categories",
|
||||||
|
"authentication": "genericCredentialType",
|
||||||
|
"genericAuthType": "lcbp3MigrationToken"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Code Node — Store Categories + File Mount Check:**
|
||||||
|
```javascript
|
||||||
|
const fs = require('fs');
|
||||||
|
|
||||||
|
// เก็บ categories ใน Workflow Variable
|
||||||
|
const categories = $input.first().json.categories;
|
||||||
|
if (!categories || !Array.isArray(categories) || categories.length === 0) {
|
||||||
|
throw new Error('Failed to fetch system categories from backend');
|
||||||
|
}
|
||||||
|
|
||||||
|
// Set Workflow Variable เพื่อใช้ใน Node 3
|
||||||
|
$workflow.variables = $workflow.variables || {};
|
||||||
|
$workflow.variables.system_categories = categories;
|
||||||
|
|
||||||
|
// ตรวจ File Mount
|
||||||
|
try {
|
||||||
|
const files = fs.readdirSync('/data/dms/staging_ai');
|
||||||
|
if (files.length === 0) throw new Error('staging_ai is empty');
|
||||||
|
fs.writeFileSync('/data/dms/migration_logs/.preflight_ok', new Date().toISOString());
|
||||||
|
} catch (err) {
|
||||||
|
throw new Error(`File mount check failed: ${err.message}`);
|
||||||
|
}
|
||||||
|
|
||||||
|
return [{ json: { preflight_ok: true, system_categories: categories } }];
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 4.3 Node 1: Load Checkpoint + Read Excel + Encoding Normalization
|
||||||
|
|
||||||
|
**Step 1 — MariaDB Node (Read Checkpoint):**
|
||||||
|
```sql
|
||||||
|
SELECT last_processed_index, status
|
||||||
|
FROM migration_progress
|
||||||
|
WHERE batch_id = '{{ $env.MIGRATION_BATCH_ID }}'
|
||||||
|
LIMIT 1;
|
||||||
|
```
|
||||||
|
|
||||||
|
**Step 2 — Spreadsheet File Node:**
|
||||||
|
```json
|
||||||
|
{ "operation": "toData", "binaryProperty": "data", "options": { "sheetName": "Sheet1" } }
|
||||||
|
```
|
||||||
|
|
||||||
|
**Step 3 — Code Node (Checkpoint + Batch + Encoding):**
|
||||||
|
```javascript
|
||||||
|
const checkpointResult = $('Read Checkpoint').first();
|
||||||
|
let startIndex = 0;
|
||||||
|
if (checkpointResult && checkpointResult.json.status === 'RUNNING') {
|
||||||
|
startIndex = checkpointResult.json.last_processed_index || 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
const allItems = $('Read Excel').all();
|
||||||
|
const remaining = allItems.slice(startIndex);
|
||||||
|
const batchSize = parseInt($env.MIGRATION_BATCH_SIZE) || 10;
|
||||||
|
const currentBatch = remaining.slice(0, batchSize);
|
||||||
|
|
||||||
|
// Encoding Normalization: Excel → UTF-8 NFC (Patch)
|
||||||
|
const normalize = (str) => {
|
||||||
|
if (!str) return '';
|
||||||
|
return Buffer.from(String(str), 'utf8').toString('utf8').normalize('NFC');
|
||||||
|
};
|
||||||
|
|
||||||
|
return currentBatch.map((item, i) => ({
|
||||||
|
...item,
|
||||||
|
json: {
|
||||||
|
...item.json,
|
||||||
|
document_number: normalize(item.json.document_number),
|
||||||
|
title: normalize(item.json.title),
|
||||||
|
original_index: startIndex + i
|
||||||
|
}
|
||||||
|
}));
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 4.4 Node 2: File Validator & Sanitizer
|
||||||
|
|
||||||
|
**Node Type:** `Code` — **2 Outputs**
|
||||||
|
|
||||||
|
```javascript
|
||||||
|
const fs = require('fs');
|
||||||
|
const path = require('path');
|
||||||
|
const items = $input.all();
|
||||||
|
const validatedItems = [];
|
||||||
|
const errorItems = [];
|
||||||
|
|
||||||
|
for (const item of items) {
|
||||||
|
const docNumber = item.json.document_number;
|
||||||
|
// Sanitize + Normalize Filename (Patch)
|
||||||
|
const safeName = path.basename(
|
||||||
|
String(docNumber).replace(/[^a-zA-Z0-9\-_.]/g, '_')
|
||||||
|
).normalize('NFC');
|
||||||
|
const filePath = path.resolve('/data/dms/staging_ai', `${safeName}.pdf`);
|
||||||
|
|
||||||
|
if (!filePath.startsWith('/data/dms/staging_ai/')) {
|
||||||
|
errorItems.push({ ...item, json: { ...item.json, error: 'Path traversal detected', error_type: 'FILE_NOT_FOUND' } });
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
try {
|
||||||
|
if (fs.existsSync(filePath)) {
|
||||||
|
const stats = fs.statSync(filePath);
|
||||||
|
validatedItems.push({ ...item, json: { ...item.json, file_exists: true, file_size: stats.size, file_path: filePath } });
|
||||||
|
} else {
|
||||||
|
errorItems.push({ ...item, json: { ...item.json, error: `File not found: ${filePath}`, error_type: 'FILE_NOT_FOUND', file_exists: false } });
|
||||||
|
}
|
||||||
|
} catch (err) {
|
||||||
|
errorItems.push({ ...item, json: { ...item.json, error: err.message, error_type: 'FILE_NOT_FOUND', file_exists: false } });
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Output 0 → Node 3 | Output 1 → Node 5D
|
||||||
|
return [validatedItems, errorItems];
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 4.5 Node 3: Build Prompt + AI Analysis
|
||||||
|
|
||||||
|
**Step 1 — MariaDB (Read Fallback State):**
|
||||||
|
```sql
|
||||||
|
SELECT is_fallback_active FROM migration_fallback_state
|
||||||
|
WHERE batch_id = '{{ $env.MIGRATION_BATCH_ID }}' LIMIT 1;
|
||||||
|
```
|
||||||
|
|
||||||
|
**Step 2 — Code Node (Build Prompt: inject system_categories):**
|
||||||
|
```javascript
|
||||||
|
const fallbackState = $('Read Fallback State').first();
|
||||||
|
const isFallback = fallbackState?.json?.is_fallback_active || false;
|
||||||
|
const model = isFallback ? $env.OLLAMA_MODEL_FALLBACK : $env.OLLAMA_MODEL_PRIMARY;
|
||||||
|
|
||||||
|
// ใช้ system_categories จาก Workflow Variable (ไม่ hardcode)
|
||||||
|
const systemCategories = $workflow.variables?.system_categories
|
||||||
|
|| ['Correspondence','RFA','Drawing','Transmittal','Report','Other'];
|
||||||
|
|
||||||
|
const item = $input.first();
|
||||||
|
|
||||||
|
const systemPrompt = `You are a Document Controller for a large construction project.
|
||||||
|
Your task is to validate document metadata.
|
||||||
|
You MUST respond ONLY with valid JSON. No explanation, no markdown, no extra text.
|
||||||
|
If there are no issues, "detected_issues" must be an empty array [].`;
|
||||||
|
|
||||||
|
const userPrompt = `Validate this document metadata and respond in JSON:
|
||||||
|
|
||||||
|
Document Number: ${item.json.document_number}
|
||||||
|
Title: ${item.json.title}
|
||||||
|
Expected Pattern: [ORG]-[TYPE]-[SEQ] e.g. "TCC-COR-0001"
|
||||||
|
Category List (MUST match system enum exactly): ${JSON.stringify(systemCategories)}
|
||||||
|
|
||||||
|
Respond ONLY with this exact JSON structure:
|
||||||
|
{
|
||||||
|
"is_valid": true | false,
|
||||||
|
"confidence": 0.0 to 1.0,
|
||||||
|
"suggested_category": "<one from Category List>",
|
||||||
|
"detected_issues": ["<issue1>"],
|
||||||
|
"suggested_title": "<corrected title or null>"
|
||||||
|
}`;
|
||||||
|
|
||||||
|
return [{
|
||||||
|
json: {
|
||||||
|
...item.json,
|
||||||
|
active_model: model,
|
||||||
|
system_categories: systemCategories,
|
||||||
|
ollama_payload: { model, prompt: `${systemPrompt}\n\n${userPrompt}`, stream: false, format: 'json' }
|
||||||
|
}
|
||||||
|
}];
|
||||||
|
```
|
||||||
|
|
||||||
|
**Step 3 — HTTP Request Node (Ollama):**
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"method": "POST",
|
||||||
|
"url": "={{ $env.OLLAMA_HOST }}/api/generate",
|
||||||
|
"sendBody": true,
|
||||||
|
"specifyBody": "json",
|
||||||
|
"jsonBody": "={{ $json.ollama_payload }}",
|
||||||
|
"options": { "timeout": 30000, "retry": { "count": 3, "delay": 2000, "backoff": "exponential" } }
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Step 4 — Code Node (Parse + Validate: Enum check):**
|
||||||
|
```javascript
|
||||||
|
const items = $input.all();
|
||||||
|
const parsed = [];
|
||||||
|
const parseErrors = [];
|
||||||
|
|
||||||
|
for (const item of items) {
|
||||||
|
try {
|
||||||
|
let raw = item.json.response || '';
|
||||||
|
raw = raw.replace(/```json/gi, '').replace(/```/g, '').trim();
|
||||||
|
const result = JSON.parse(raw);
|
||||||
|
|
||||||
|
// Strict Schema Validation
|
||||||
|
if (typeof result.is_valid !== 'boolean')
|
||||||
|
throw new Error('is_valid must be boolean');
|
||||||
|
if (typeof result.confidence !== 'number' || result.confidence < 0 || result.confidence > 1)
|
||||||
|
throw new Error('confidence must be float 0.0–1.0');
|
||||||
|
if (!Array.isArray(result.detected_issues))
|
||||||
|
throw new Error('detected_issues must be array');
|
||||||
|
|
||||||
|
// Enum Validation ตรง System Categories (Patch)
|
||||||
|
const systemCategories = item.json.system_categories || [];
|
||||||
|
if (!systemCategories.includes(result.suggested_category))
|
||||||
|
throw new Error(`Category "${result.suggested_category}" not in system enum: [${systemCategories.join(', ')}]`);
|
||||||
|
|
||||||
|
parsed.push({ ...item, json: { ...item.json, ai_result: result, parse_error: null } });
|
||||||
|
} catch (err) {
|
||||||
|
parseErrors.push({
|
||||||
|
...item,
|
||||||
|
json: { ...item.json, ai_result: null, parse_error: err.message, raw_ai_response: item.json.response, error_type: 'AI_PARSE_ERROR' }
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Output 0 → Node 4 | Output 1 → Node 3.5 + Node 5D
|
||||||
|
return [parsed, parseErrors];
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 4.6 Node 3.5: Fallback Model Manager
|
||||||
|
|
||||||
|
**MariaDB Node:**
|
||||||
|
```sql
|
||||||
|
INSERT INTO migration_fallback_state (batch_id, recent_error_count, is_fallback_active)
|
||||||
|
VALUES ('{{ $env.MIGRATION_BATCH_ID }}', 1, FALSE)
|
||||||
|
ON DUPLICATE KEY UPDATE
|
||||||
|
recent_error_count = recent_error_count + 1,
|
||||||
|
is_fallback_active = CASE
|
||||||
|
WHEN recent_error_count + 1 >= {{ $env.FALLBACK_ERROR_THRESHOLD }} THEN TRUE
|
||||||
|
ELSE is_fallback_active
|
||||||
|
END,
|
||||||
|
updated_at = NOW();
|
||||||
|
```
|
||||||
|
|
||||||
|
**Code Node (Alert):**
|
||||||
|
```javascript
|
||||||
|
const state = $input.first().json;
|
||||||
|
if (state.is_fallback_active) {
|
||||||
|
return [{ json: {
|
||||||
|
...state, alert: true,
|
||||||
|
alert_message: `⚠️ Fallback model (${$env.OLLAMA_MODEL_FALLBACK}) activated after ${state.recent_error_count} errors`
|
||||||
|
}}];
|
||||||
|
}
|
||||||
|
return [{ json: { ...state, alert: false } }];
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 4.7 Node 4: Confidence Router + Revision Drift Protection
|
||||||
|
|
||||||
|
**Node Type:** `Code` — **4 Outputs**
|
||||||
|
|
||||||
|
```javascript
|
||||||
|
const items = $input.all();
|
||||||
|
const autoIngest = [];
|
||||||
|
const reviewQueue = [];
|
||||||
|
const rejectLog = [];
|
||||||
|
const errorLog = [];
|
||||||
|
|
||||||
|
const HIGH = parseFloat($env.CONFIDENCE_THRESHOLD_HIGH) || 0.85;
|
||||||
|
const LOW = parseFloat($env.CONFIDENCE_THRESHOLD_LOW) || 0.60;
|
||||||
|
|
||||||
|
for (const item of items) {
|
||||||
|
if (item.json.parse_error || !item.json.ai_result) {
|
||||||
|
errorLog.push(item); continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Revision Drift Protection
|
||||||
|
if (item.json.excel_revision !== undefined) {
|
||||||
|
const expectedRev = (item.json.current_db_revision || 0) + 1;
|
||||||
|
if (parseInt(item.json.excel_revision) !== expectedRev) {
|
||||||
|
reviewQueue.push({
|
||||||
|
...item,
|
||||||
|
json: { ...item.json, review_reason: `Revision drift: Excel=${item.json.excel_revision}, Expected=${expectedRev}` }
|
||||||
|
});
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
const ai = item.json.ai_result;
|
||||||
|
if (ai.confidence >= HIGH && ai.is_valid === true) {
|
||||||
|
autoIngest.push(item);
|
||||||
|
} else if (ai.confidence >= LOW) {
|
||||||
|
reviewQueue.push({ ...item, json: { ...item.json, review_reason: `Confidence ${ai.confidence.toFixed(2)} < ${HIGH}` } });
|
||||||
|
} else {
|
||||||
|
rejectLog.push({
|
||||||
|
...item,
|
||||||
|
json: { ...item.json, reject_reason: ai.is_valid === false ? 'AI marked invalid' : `Confidence ${ai.confidence.toFixed(2)} < ${LOW}` }
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Output 0: Auto Ingest | 1: Review Queue | 2: Reject Log | 3: Error Log
|
||||||
|
return [autoIngest, reviewQueue, rejectLog, errorLog];
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 4.8 Node 5A: Auto Ingest + Idempotency + Checkpoint
|
||||||
|
|
||||||
|
**HTTP Request Node (Patch — Idempotency-Key Header + source_file_path):**
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"method": "POST",
|
||||||
|
"url": "={{ $env.BACKEND_URL }}/api/correspondences/import",
|
||||||
|
"authentication": "genericCredentialType",
|
||||||
|
"genericAuthType": "lcbp3MigrationToken",
|
||||||
|
"sendHeaders": true,
|
||||||
|
"headers": {
|
||||||
|
"Idempotency-Key": "={{ $json.document_number }}:{{ $env.MIGRATION_BATCH_ID }}"
|
||||||
|
},
|
||||||
|
"sendBody": true,
|
||||||
|
"specifyBody": "json",
|
||||||
|
"jsonBody": {
|
||||||
|
"document_number": "={{ $json.document_number }}",
|
||||||
|
"title": "={{ $json.ai_result.suggested_title || $json.title }}",
|
||||||
|
"category": "={{ $json.ai_result.suggested_category }}",
|
||||||
|
"source_file_path": "={{ $json.file_path }}",
|
||||||
|
"ai_confidence": "={{ $json.ai_result.confidence }}",
|
||||||
|
"ai_issues": "={{ $json.ai_result.detected_issues }}",
|
||||||
|
"migrated_by": "SYSTEM_IMPORT",
|
||||||
|
"batch_id": "={{ $env.MIGRATION_BATCH_ID }}"
|
||||||
|
},
|
||||||
|
"options": { "timeout": 30000, "retry": { "count": 3, "delay": 5000 } }
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
> Backend จะ generate UUID, enforce Storage path `/storage/{project}/{category}/{year}/{month}/{uuid}.pdf`, move file ผ่าน StorageService และบันทึก Audit Log `action=IMPORT, source=MIGRATION`
|
||||||
|
|
||||||
|
**Checkpoint Code Node (ทุก 10 Records):**
|
||||||
|
```javascript
|
||||||
|
const item = $input.first();
|
||||||
|
return [{ json: {
|
||||||
|
...item.json,
|
||||||
|
should_update_checkpoint: item.json.original_index % 10 === 0,
|
||||||
|
checkpoint_index: item.json.original_index
|
||||||
|
}}];
|
||||||
|
```
|
||||||
|
|
||||||
|
**IF Node → MariaDB Checkpoint:**
|
||||||
|
```sql
|
||||||
|
INSERT INTO migration_progress (batch_id, last_processed_index, status)
|
||||||
|
VALUES ('{{ $env.MIGRATION_BATCH_ID }}', {{ $json.checkpoint_index }}, 'RUNNING')
|
||||||
|
ON DUPLICATE KEY UPDATE last_processed_index = {{ $json.checkpoint_index }}, updated_at = NOW();
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 4.9 Node 5B: Review Queue (Temporary Table)
|
||||||
|
|
||||||
|
> ⚠️ ห้ามสร้าง Correspondence record — รอ Admin Approve แล้วค่อย POST `/api/correspondences/import`
|
||||||
|
|
||||||
|
```sql
|
||||||
|
INSERT INTO migration_review_queue
|
||||||
|
(document_number, title, original_title, ai_suggested_category,
|
||||||
|
ai_confidence, ai_issues, review_reason, status, created_at)
|
||||||
|
VALUES (
|
||||||
|
'{{ $json.document_number }}',
|
||||||
|
'{{ $json.ai_result.suggested_title || $json.title }}',
|
||||||
|
'{{ $json.title }}',
|
||||||
|
'{{ $json.ai_result.suggested_category }}',
|
||||||
|
{{ $json.ai_result.confidence }},
|
||||||
|
'{{ JSON.stringify($json.ai_result.detected_issues) }}',
|
||||||
|
'{{ $json.review_reason }}',
|
||||||
|
'PENDING', NOW()
|
||||||
|
)
|
||||||
|
ON DUPLICATE KEY UPDATE status = 'PENDING', review_reason = '{{ $json.review_reason }}';
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 4.10 Node 5C: Reject Log → `/data/migration_logs/`
|
||||||
|
|
||||||
|
```javascript
|
||||||
|
const fs = require('fs');
|
||||||
|
const item = $input.first();
|
||||||
|
const csvPath = '/data/dms/migration_logs/reject_log.csv';
|
||||||
|
const header = 'timestamp,document_number,title,reject_reason,ai_confidence,ai_issues\n';
|
||||||
|
const esc = (s) => `"${String(s||'').replace(/"/g,'""')}"`;
|
||||||
|
|
||||||
|
if (!fs.existsSync(csvPath)) fs.writeFileSync(csvPath, header, 'utf8');
|
||||||
|
|
||||||
|
const line = [
|
||||||
|
new Date().toISOString(),
|
||||||
|
esc(item.json.document_number), esc(item.json.title),
|
||||||
|
esc(item.json.reject_reason),
|
||||||
|
item.json.ai_result?.confidence ?? 'N/A',
|
||||||
|
esc(JSON.stringify(item.json.ai_result?.detected_issues || []))
|
||||||
|
].join(',') + '\n';
|
||||||
|
|
||||||
|
fs.appendFileSync(csvPath, line, 'utf8');
|
||||||
|
return [$input.first()];
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 4.11 Node 5D: Error Log → `/data/migration_logs/` + MariaDB
|
||||||
|
|
||||||
|
```javascript
|
||||||
|
const fs = require('fs');
|
||||||
|
const item = $input.first();
|
||||||
|
const csvPath = '/data/dms/migration_logs/error_log.csv';
|
||||||
|
const header = 'timestamp,document_number,error_type,error_message,raw_ai_response\n';
|
||||||
|
const esc = (s) => `"${String(s||'').replace(/"/g,'""')}"`;
|
||||||
|
|
||||||
|
if (!fs.existsSync(csvPath)) fs.writeFileSync(csvPath, header, 'utf8');
|
||||||
|
|
||||||
|
const line = [
|
||||||
|
new Date().toISOString(),
|
||||||
|
esc(item.json.document_number),
|
||||||
|
esc(item.json.error_type || 'UNKNOWN'),
|
||||||
|
esc(item.json.error || item.json.parse_error),
|
||||||
|
esc(item.json.raw_ai_response || '')
|
||||||
|
].join(',') + '\n';
|
||||||
|
|
||||||
|
fs.appendFileSync(csvPath, line, 'utf8');
|
||||||
|
return [$input.first()];
|
||||||
|
```
|
||||||
|
|
||||||
|
**MariaDB Node:**
|
||||||
|
```sql
|
||||||
|
INSERT INTO migration_errors
|
||||||
|
(batch_id, document_number, error_type, error_message, raw_ai_response, created_at)
|
||||||
|
VALUES (
|
||||||
|
'{{ $env.MIGRATION_BATCH_ID }}', '{{ $json.document_number }}',
|
||||||
|
'{{ $json.error_type || "UNKNOWN" }}', '{{ $json.error || $json.parse_error }}',
|
||||||
|
'{{ $json.raw_ai_response || "" }}', NOW()
|
||||||
|
);
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 📌 ส่วนที่ 5: Rollback Workflow
|
||||||
|
|
||||||
|
**Workflow: `Migration Rollback`** — Manual Trigger เท่านั้น
|
||||||
|
|
||||||
|
```
|
||||||
|
[Manual Trigger: {confirmation: "CONFIRM_ROLLBACK"}]
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
[Code: Guard — ต้องพิมพ์ "CONFIRM_ROLLBACK"]
|
||||||
|
│ PASS
|
||||||
|
▼
|
||||||
|
[MariaDB: Disable Token]
|
||||||
|
UPDATE users SET is_active = false WHERE username = 'migration_bot';
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
[MariaDB: Delete File Records]
|
||||||
|
DELETE FROM correspondence_files WHERE correspondence_id IN
|
||||||
|
(SELECT id FROM correspondences WHERE created_by = 'SYSTEM_IMPORT');
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
[MariaDB: Delete Correspondence Records]
|
||||||
|
DELETE FROM correspondences WHERE created_by = 'SYSTEM_IMPORT';
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
[MariaDB: Clear Idempotency Records]
|
||||||
|
DELETE FROM import_transactions WHERE batch_id = '{{$env.MIGRATION_BATCH_ID}}';
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
[MariaDB: Reset Checkpoint + Fallback State]
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
[Email: Rollback Report → Admin]
|
||||||
|
```
|
||||||
|
|
||||||
|
**Confirmation Guard:**
|
||||||
|
```javascript
|
||||||
|
if ($input.first().json.confirmation !== 'CONFIRM_ROLLBACK') {
|
||||||
|
throw new Error('Rollback cancelled: type "CONFIRM_ROLLBACK" to proceed.');
|
||||||
|
}
|
||||||
|
return $input.all();
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 📌 ส่วนที่ 6: End-of-Night Summary (06:30 ทุกวัน)
|
||||||
|
|
||||||
|
**MariaDB:**
|
||||||
|
```sql
|
||||||
|
SELECT
|
||||||
|
mp.last_processed_index AS total_progress,
|
||||||
|
(SELECT COUNT(*) FROM correspondences
|
||||||
|
WHERE created_by = 'SYSTEM_IMPORT' AND DATE(created_at) = CURDATE()) AS auto_ingested,
|
||||||
|
(SELECT COUNT(*) FROM migration_review_queue WHERE DATE(created_at) = CURDATE()) AS sent_to_review,
|
||||||
|
(SELECT COUNT(*) FROM migration_errors
|
||||||
|
WHERE batch_id = '{{ $env.MIGRATION_BATCH_ID }}' AND DATE(created_at) = CURDATE()) AS errors
|
||||||
|
FROM migration_progress mp WHERE mp.batch_id = '{{ $env.MIGRATION_BATCH_ID }}';
|
||||||
|
```
|
||||||
|
|
||||||
|
**Code Node (Build Report):**
|
||||||
|
```javascript
|
||||||
|
const s = $input.first().json;
|
||||||
|
const total = 20000;
|
||||||
|
const pct = ((s.total_progress / total) * 100).toFixed(1);
|
||||||
|
const nightsLeft = Math.ceil((total - s.total_progress) / (8 * 3600 / 3));
|
||||||
|
|
||||||
|
const report = `
|
||||||
|
📊 Migration Night Summary — ${new Date().toLocaleDateString('th-TH')}
|
||||||
|
${'─'.repeat(50)}
|
||||||
|
✅ Auto Ingested : ${s.auto_ingested}
|
||||||
|
🔍 Sent to Review : ${s.sent_to_review}
|
||||||
|
❌ Errors : ${s.errors}
|
||||||
|
─────────────────────────────────────────────────
|
||||||
|
📈 Progress : ${s.total_progress} / ${total} (${pct}%)
|
||||||
|
🌙 Est. Nights Left: ~${nightsLeft} คืน
|
||||||
|
${'─'.repeat(50)}
|
||||||
|
${s.errors > 50 ? '⚠️ WARNING: High error count — investigate before next run' : '✅ Error rate OK'}
|
||||||
|
`;
|
||||||
|
return [{ json: { report, stats: s } }];
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 📌 ส่วนที่ 7: Monitoring (Hourly Alert — เฉพาะเมื่อเกิน Threshold)
|
||||||
|
|
||||||
|
**Code Node (Evaluate):**
|
||||||
|
```javascript
|
||||||
|
const s = $input.first().json;
|
||||||
|
const alerts = [];
|
||||||
|
|
||||||
|
if (s.minutes_since_update > 30)
|
||||||
|
alerts.push(`⚠️ No progress for ${s.minutes_since_update} min — may be stuck`);
|
||||||
|
if (s.is_fallback_active)
|
||||||
|
alerts.push(`⚠️ Fallback model active — errors: ${s.recent_error_count}`);
|
||||||
|
if (s.recent_error_count >= 20)
|
||||||
|
alerts.push(`🔴 Critical: ${s.recent_error_count} errors — consider stopping`);
|
||||||
|
|
||||||
|
return [{ json: { ...s, has_alerts: alerts.length > 0, alerts } }];
|
||||||
|
```
|
||||||
|
|
||||||
|
**IF `has_alerts = true` → Email Alert ทันที**
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 📌 ส่วนที่ 8: Pre-Production Checklist
|
||||||
|
|
||||||
|
| ลำดับ | รายการทดสอบ | ผลลัพธ์ที่คาดหวัง | ✅/❌ |
|
||||||
|
| --- | --------------------------------------------- | ---------------------- | --- |
|
||||||
|
| 1 | Pre-flight ผ่านทุก Check | All green | |
|
||||||
|
| 2 | `GET /api/meta/categories` สำเร็จ | categories array ไม่ว่าง | |
|
||||||
|
| 3 | Enum ใน Prompt ไม่ hardcode | ตรงกับ Backend | |
|
||||||
|
| 4 | Idempotency: รัน Batch ซ้ำ | ไม่สร้าง Revision ซ้ำ | |
|
||||||
|
| 5 | Storage path ตาม Spec | UUID + /year/month/ | |
|
||||||
|
| 6 | Audit Log มี `action=IMPORT, source=MIGRATION` | Verified | |
|
||||||
|
| 7 | Review Queue ไม่สร้าง record อัตโนมัติ | Verified | |
|
||||||
|
| 8 | Revision drift → Review Queue | Verified | |
|
||||||
|
| 9 | Error ≥ 5 → Fallback Model สลับ | mistral:7b active | |
|
||||||
|
| 10 | Reject/Error CSV เขียนลง `migration_logs/` | ไม่ใช่ `staging_ai/` | |
|
||||||
|
| 11 | Rollback Guard ต้องพิมพ์ CONFIRM_ROLLBACK | Block ทำงาน | |
|
||||||
|
| 12 | Night Summary 06:30 + Est. nights left | Email ถึง Admin | |
|
||||||
|
| 13 | Monitoring Alert เฉพาะเกิน Threshold | ไม่ spam ทุกชั่วโมง | |
|
||||||
|
| 14 | Nginx Rate Limit `burst=5` | Configured | |
|
||||||
|
| 15 | Docker `mem_limit=2g` + log rotation | Configured | |
|
||||||
|
|
||||||
|
**คำสั่งทดสอบ:**
|
||||||
|
```bash
|
||||||
|
# Ollama
|
||||||
|
docker exec -it n8n-migration curl http://<ASUSTOR_IP>:11434/api/tags
|
||||||
|
|
||||||
|
# RO mount
|
||||||
|
docker exec -it n8n-migration ls /data/dms/staging_ai | head -5
|
||||||
|
|
||||||
|
# RW mount
|
||||||
|
docker exec -it n8n-migration sh -c "echo ok > /data/dms/migration_logs/test.txt && echo '✅ rw OK'"
|
||||||
|
|
||||||
|
# DB
|
||||||
|
docker exec -it n8n-migration mysql -h <DB_IP> -u migration_bot -p -e "SELECT 1"
|
||||||
|
|
||||||
|
# Backend + Category endpoint
|
||||||
|
curl -H "Authorization: Bearer <TOKEN>" https://<BACKEND>/api/meta/categories
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 📌 ส่วนที่ 9: การรันงานจริง
|
||||||
|
|
||||||
|
### 9.1 Daily Operation
|
||||||
|
|
||||||
|
| เวลา | กิจกรรม | ผู้รับผิดชอบ |
|
||||||
|
| ----- | ------------------------------ | ------------------- |
|
||||||
|
| 08:00 | ตรวจสอบ Night Summary Email | Admin |
|
||||||
|
| 09:00 | Approve/Reject ใน Review Queue | Document Controller |
|
||||||
|
| 17:00 | ตรวจ Disk Space + GPU Temp | DevOps |
|
||||||
|
| 22:00 | Workflow เริ่มรันอัตโนมัติ | System |
|
||||||
|
| 06:30 | Night Summary Report ส่ง Email | System |
|
||||||
|
|
||||||
|
### 9.2 Emergency Stop
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 1. หยุด n8n
|
||||||
|
docker stop n8n-migration
|
||||||
|
|
||||||
|
# 2. Disable Token
|
||||||
|
mysql -h <DB_IP> -u root -p \
|
||||||
|
-e "UPDATE users SET is_active = false WHERE username = 'migration_bot';"
|
||||||
|
|
||||||
|
# 3. Progress
|
||||||
|
mysql -h <DB_IP> -u root -p \
|
||||||
|
-e "SELECT * FROM migration_progress WHERE batch_id = 'migration_20260226';"
|
||||||
|
|
||||||
|
# 4. Errors
|
||||||
|
mysql -h <DB_IP> -u root -p \
|
||||||
|
-e "SELECT * FROM migration_errors ORDER BY created_at DESC LIMIT 20;"
|
||||||
|
|
||||||
|
# 5. Rollback ผ่าน Webhook
|
||||||
|
curl -X POST http://<NAS_IP>:5678/webhook/rollback \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{"confirmation":"CONFIRM_ROLLBACK"}'
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 📞 การติดต่อสนับสนุน
|
||||||
|
|
||||||
|
| ปัญหา | ช่องทางติดต่อ |
|
||||||
|
| --------------- | ------------------------------------------- |
|
||||||
|
| Technical Issue | DevOps Team (Slack: #migration-support) |
|
||||||
|
| Data Issue | Document Controller (Email: dc@lcbp3.local) |
|
||||||
|
| Security Issue | Security Team (Email: security@lcbp3.local) |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
**เอกสารฉบับนี้จัดทำขึ้นเพื่อรองรับ Migration ตาม ADR-017 และ 03-04**
|
||||||
|
**Version:** 1.8.0 | **Last Updated:** 2026-02-27 | **Author:** Development Team
|
||||||
2070
specs/03-Data-and-Storage/lcbp3-v1.8.0-schema.sql
Normal file
2070
specs/03-Data-and-Storage/lcbp3-v1.8.0-schema.sql
Normal file
File diff suppressed because it is too large
Load Diff
2203
specs/03-Data-and-Storage/lcbp3-v1.8.0-seed-basic.sql
Normal file
2203
specs/03-Data-and-Storage/lcbp3-v1.8.0-seed-basic.sql
Normal file
File diff suppressed because it is too large
Load Diff
@@ -2,33 +2,40 @@
|
|||||||
|
|
||||||
**Status:** Accepted
|
**Status:** Accepted
|
||||||
**Date:** 2026-02-26
|
**Date:** 2026-02-26
|
||||||
|
**Version:** 1.8.0
|
||||||
**Decision Makers:** Development Team, DevOps Engineer
|
**Decision Makers:** Development Team, DevOps Engineer
|
||||||
**Related Documents:**
|
**Related Documents:**
|
||||||
|
|
||||||
- [Legacy Data Migration Plan](../03-Data-and-Storage/03-04-legacy-data-migration.md)
|
- [Legacy Data Migration Plan](../03-Data-and-Storage/03-04-legacy-data-migration.md)
|
||||||
|
- [n8n Migration Setup Guide](../03-Data-and-Storage/03-05-n8n-migration-setup-guide.md)
|
||||||
- [Software Architecture](../02-Architecture/02-02-software-architecture.md)
|
- [Software Architecture](../02-Architecture/02-02-software-architecture.md)
|
||||||
- [Data Dictionary](../03-Data-and-Storage/03-01-data-dictionary.md)
|
- [Data Dictionary](../03-Data-and-Storage/03-01-data-dictionary.md)
|
||||||
|
|
||||||
|
> **Note:** ADR-017 is clarified and hardened by ADR-018 regarding AI physical isolation. Category Enum system-driven, Idempotency Contract, Duplicate Handling Clarification, Storage Enforcement, Audit Log Enhancement, Review Queue Integration, Revision Drift Protection, Execution Time, Encoding Normalization, Security Hardening, AI Physical Isolation (ASUSTOR).
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Context and Problem Statement
|
## Context and Problem Statement
|
||||||
|
|
||||||
โครงการ LCBP3-DMS มีความจำเป็นต้องนำเข้าเอกสาร (Data Migration) ประเภท PDF เก่าจำนวนกว่า 20,000 ฉบับ ซึ่งมาพร้อมกับ Metadata ในรูปแบบไฟล์ Excel เข้าสู่ระบบใหม่เพื่อให้สามารถเริ่มใช้งานได้อย่างสมบูรณ์
|
โครงการ LCBP3-DMS มีความจำเป็นต้องนำเข้าเอกสาร PDF เก่าจำนวนกว่า 20,000 ฉบับ พร้อม Metadata ใน Excel เข้าสู่ระบบใหม่
|
||||||
|
|
||||||
ความท้าทายหลักของการทำ Migration ชุดนี้คือ **Data Integrity และความถูกต้องของ Metadata** เนื่องจากเป็นข้อมูลเก่าที่มีโอกาสเกิด Human Error ในขั้นตอนการจัดทำ Index (เช่น ชื่อไฟล์ หรือ Document Number พิมพ์ผิด) เราจึงต้องการเครื่องมืออัตโนมัติมาช่วย Validate เอกสารและจำแนกประเภทก่อนการนำเข้า
|
ความท้าทายหลักคือ **Data Integrity และความถูกต้องของ Metadata** เนื่องจากข้อมูลเก่ามีโอกาสเกิด Human Error เราจึงต้องการ AI ช่วย Validate ก่อนนำเข้า
|
||||||
|
|
||||||
ทว่าการส่งข้อมูล 20,000 รายการ ขึ้นไปวิเคราะห์บน Cloud AI Provider (เช่น OpenAI, Anthropic) มีปัญหาใหญ่ 2 ประการ:
|
การส่งข้อมูลขึ้น Cloud AI Provider มีปัญหา 2 ประการ:
|
||||||
1. **Data Privacy / Confidentiality:** เอกสารก่อสร้างท่าเรือเป็นข้อมูลความลับ ไม่ควรส่งขึ้น Public API
|
1. **Data Privacy:** เอกสารก่อสร้างท่าเรือเป็นความลับ ห้ามออกนอกเครือข่าย
|
||||||
2. **Cost:** ค่าใช้จ่ายต่อ Token ในการวิเคราะห์เอกสารจำนวนมากจะสูงเกินความจำเป็น
|
2. **Cost:** ~$0.01–0.03 ต่อ Record = อาจสูงถึง $600 สำหรับ 20,000 records
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Decision Drivers
|
## Decision Drivers
|
||||||
|
|
||||||
- **Security & Privacy:** ต้องเก็บข้อมูลและประมวลผลภายในระบบเครือข่ายภายในองค์กร (On-Premise)
|
- **Security & Privacy:** ประมวลผลภายในเครือข่ายองค์กร (On-Premise) เท่านั้น
|
||||||
- **Cost Effectiveness:** ไม่เสียค่าใช้จ่ายแบบ Pay-per-use (API Costs) ไม่จำกัดจำนวน Request
|
- **Cost Effectiveness:** ไม่เสียค่า Pay-per-use
|
||||||
- **Performance:** ต้องสามารถประมวลผลได้อย่างรวดเร็วในระยะเวลาที่จำกัด
|
- **Performance:** ประมวลผลได้ในระยะเวลาที่จำกัด (~3–4 คืน)
|
||||||
- **Maintainability:** เครื่องมือ Migration ต้องแยก Context ออกจาก Core Application (ไม่นำไปเขียนเป็น Script ฝังใน NestJS เพื่อทำงานชั่วคราว)
|
- **Maintainability:** แยก Migration ออกจาก Core Application
|
||||||
|
- **Recoverability:** Rollback ได้สมบูรณ์
|
||||||
|
- **Resilience:** รองรับ Checkpoint/Resume และ Hardware Failure
|
||||||
|
- **Data Integrity:** Idempotency, Revision Drift Protection, Enum Enforcement
|
||||||
|
- **Storage Governance:** ทุก File Move ต้องผ่าน StorageService
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -36,57 +43,306 @@
|
|||||||
|
|
||||||
### Option 1: NestJS Custom Script + Public AI API
|
### Option 1: NestJS Custom Script + Public AI API
|
||||||
|
|
||||||
**แนวทาง:** เขียน Script ชั่วคราวใน NestJS อ่านไฟล์ Excel และยิง API ไปยัง OpenAI/Anthropic เพื่อตรวจสอบ
|
**Pros:** ไม่ต้องจัดหา Hardware เพิ่ม, AI ฉลาดสูง
|
||||||
|
|
||||||
**Pros:**
|
|
||||||
- ไม่ต้องจัดหา Hardware เพิ่มเติมสำหรับประมวลผล AI
|
|
||||||
- AI มีความฉลาดสูง (GPT-4 / Claude 3)
|
|
||||||
|
|
||||||
**Cons:**
|
**Cons:**
|
||||||
- ❌ ผิดนโยบายเรื่อง Data Privacy
|
- ❌ ผิดนโยบาย Data Privacy
|
||||||
- ❌ มีค่าใช้จ่ายต่อเนื่องตามจำนวน Token ที่ประมวลผล
|
- ❌ ค่าใช้จ่ายสูง (~$600)
|
||||||
- ❌ โค้ดสกปรก: นำ Script การทำงานชั่วคราวไปปะปนกับ Source Code หลักของ Application
|
- ❌ Code สกปรก ปะปนกับ Source Code หลัก
|
||||||
|
|
||||||
### Option 2: Pure Scripting (No AI)
|
### Option 2: Pure Scripting (No AI)
|
||||||
|
|
||||||
**แนวทาง:** เขียน Script ตรวจสอบ Format โดยใช้ Regular Expressions เช็คความยาวหรือ Pattern ของข้อความเท่านั้น
|
**Pros:** เร็ว ไม่มีค่าใช้จ่าย
|
||||||
|
|
||||||
**Pros:**
|
|
||||||
- เร็วมาก และไม่มีค่าใช้จ่าย
|
|
||||||
|
|
||||||
**Cons:**
|
**Cons:**
|
||||||
- ❌ ความแม่นยำต่ำ: ทราบเพียงว่า Format ตรง แต่ไม่ทราบว่าความหมายของชื่อเรื่อง สอดคล้องกับประเภทเอกสารและเอกสารอ้างอิงหรือไม่
|
- ❌ ความแม่นยำต่ำ ตรวจได้แค่ Format
|
||||||
- ❌ ต้องใช้แรงงานคน (Manual Review) กลับมาสุ่มตรวจหรือแก้ไขข้อผิดพลาดจำนวนมากหลังนำเข้า
|
- ❌ ต้องใช้ Manual Review จำนวนมาก
|
||||||
|
|
||||||
### Option 3: Local AI Model (Ollama) + n8n Workflow Automation ⭐ (Selected)
|
### Option 3: Local AI Model (Ollama) + n8n ⭐ (Selected)
|
||||||
|
|
||||||
**แนวทาง:** จำลอง Workflow การ Migration ผ่าน n8n (ซึ่งติดตั้งอยู่บน QNAP NAS ของระบบอยู่แล้ว) และใช้ Ollama รัน Local Language Model (เช่น LLaMA 3.2 หรือ Mistral) โดยประมวลผลบนเครื่อง Desktop PC ที่มี GPU (เช่น RTX 2060 SUPER) ภายในเครือข่าย Local Network เดียวกัน ความเร็วในการส่งไฟล์ผ่าน 2.5G LAN
|
|
||||||
|
|
||||||
**Pros:**
|
**Pros:**
|
||||||
- ✅ **Privacy Guaranteed:** ข้อมูลไม่รั่วไหลออกสู่อินเทอร์เน็ต
|
- ✅ Privacy Guaranteed
|
||||||
- ✅ **Zero Cost:** ใช้ Hardware ที่มีอยู่แล้ว ไม่มีค่าใช้จ่ายด้าน Token
|
- ✅ Zero Cost
|
||||||
- ✅ **Clean Architecture:** กระบวนการทำ Migration ถูกแยกออกจากการพัฒนาซอฟต์แวร์หลักของระบบ (NestJS Backend รับผิดชอบแค่ Ingest API เท่านั้น)
|
- ✅ Clean Architecture
|
||||||
- ✅ **Visual & Debuggable:** n8n ช่วยให้มองเห็น Flow การทำงานแบบเป็นภาพ (Visual Node Editor) จัดการ Batch, Retry และดู Error Logs ได้ง่าย
|
- ✅ Visual & Debuggable
|
||||||
|
- ✅ Resilient (Checkpoint/Resume)
|
||||||
|
- ✅ Structured Output ด้วย JSON Schema
|
||||||
|
|
||||||
**Cons:**
|
**Cons:**
|
||||||
- ❌ จำเป็นต้องเปิดคอมพิวเตอร์ Desktop ทิ้งไว้ และควบคุมอุณหภูมิ GPU ในช่วงที่ทำ Migration
|
- ❌ ต้องเปิด Desktop ทิ้งไว้ดูแล GPU Temperature
|
||||||
|
- ❌ Model เล็กอาจแม่นน้อยกว่า Cloud AI → ต้องมี Human Review Queue
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Decision Outcome
|
## Decision Outcome
|
||||||
|
|
||||||
**Chosen Option:** Option 3 - Local AI Model (Ollama) + n8n Workflow Automation
|
**Chosen Option:** Option 3 — Local AI Model (Ollama) + n8n
|
||||||
|
|
||||||
### Rationale
|
**Rationale:** ประยุกต์ใช้ Hardware ที่มีอยู่ โดยไม่ขัดหลัก Privacy และ Security ของโครงการ n8n ช่วยลด Risk ที่จะกระทบ Core Backend และรองรับ Checkpoint/Resume ได้ดีกว่าการเขียน Script เอง
|
||||||
|
|
||||||
เราเลือกแนวทางนี้เพราะเป็นการประยุกต์ใช้ทรัพยากรที่มีอยู่ให้เกิดประโยชน์สูงสุด โดยไม่ขัดหลักการด้าน Cybersecurity และ Privacy ของโครงการ การนำ Automation Tool (n8n) แยกออกมาเป็น Orchestrator ช่วยลดความเสี่ยงที่การรัน Migration script ขนาดใหญ่จะไปส่งผลกระทบให้ Core Backend (NestJS) ของระบบในฝั่ง Production หยุดชะงัก (Downtime) หรือ Memory รั่ว
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Implementation Summary
|
## Implementation Summary
|
||||||
|
|
||||||
- **Migration Orchestrator:** n8n (Docker container ภายในระบบ Infrastructure เดิม)
|
| Component | รายละเอียด |
|
||||||
- **AI Brain:** Ollama Native (รันนอก Environment หลัก บน Hardware แยกเพื่อรับโหลด AI โดยตรง)
|
| ---------------------- | ------------------------------------------------------------- |
|
||||||
- **Data Ingestion:** ส่งผ่าน RESTful API ของ LCBP3-DMS Backend (พร้อม Token สิทธิพิเศษ)
|
| Migration Orchestrator | n8n (Docker บน ASUSTOR NAS) |
|
||||||
|
| AI Model Primary | Ollama `llama3.2:3b` |
|
||||||
|
| AI Model Fallback | Ollama `mistral:7b-instruct-q4_K_M` |
|
||||||
|
| Hardware | ASUSTOR NAS (AI Processing Only) |
|
||||||
|
| Data Ingestion | RESTful API + Migration Token (7 วัน) + Idempotency-Key Header |
|
||||||
|
| Concurrency | Sequential — 1 Request/ครั้ง, Delay 2 วินาที |
|
||||||
|
| Checkpoint | MariaDB `migration_progress` |
|
||||||
|
| Fallback | Auto-switch Model เมื่อ Error ≥ Threshold |
|
||||||
|
| Storage | Backend StorageService เท่านั้น — ห้าม move file โดยตรง |
|
||||||
|
| Expected Runtime | ~16.6 ชั่วโมง (~3–4 คืน) สำหรับ 20,000 records |
|
||||||
|
|
||||||
*หมายเหตุ: สำหรับขั้นตอนปฏิบัติงานแบบละเอียด โปรดดูที่ไฟล์ `03-04-legacy-data-migration.md`*
|
---
|
||||||
|
|
||||||
|
## AI Output Contract (JSON Schema)
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"is_valid": true,
|
||||||
|
"confidence": 0.92,
|
||||||
|
"suggested_category": "Correspondence",
|
||||||
|
"detected_issues": [],
|
||||||
|
"suggested_title": null
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
| Field | Type | คำอธิบาย |
|
||||||
|
| -------------------- | ------------------------- | --------------------------- |
|
||||||
|
| `is_valid` | boolean | เอกสารผ่านการตรวจสอบหรือไม่ |
|
||||||
|
| `confidence` | float (0.0–1.0) | ความมั่นใจของ AI |
|
||||||
|
| `suggested_category` | string (enum จาก Backend) | หมวดหมู่ที่ AI แนะนำ |
|
||||||
|
| `detected_issues` | string[] | รายการปัญหา (array ว่างถ้าไม่มี) |
|
||||||
|
| `suggested_title` | string \| null | Title ที่แก้ไขแล้ว หรือ null |
|
||||||
|
|
||||||
|
> ⚠️ **Patch:** `suggested_category` ต้องตรงกับ System Enum จาก `GET /api/meta/categories` เท่านั้น — ห้าม hardcode Category List ใน Prompt
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Confidence Threshold Policy
|
||||||
|
|
||||||
|
| ระดับ Confidence | การดำเนินการ |
|
||||||
|
| ------------------------------- | --------------------------------------- |
|
||||||
|
| `>= 0.85` และ `is_valid = true` | Auto Ingest เข้าระบบ |
|
||||||
|
| `0.60–0.84` | ส่งไป Human Review Queue |
|
||||||
|
| `< 0.60` หรือ `is_valid = false` | ส่งไป Reject Log รอ Manual Fix |
|
||||||
|
| AI Parse Error | ส่งไป Error Log + Trigger Fallback Logic |
|
||||||
|
| Revision Drift | ส่งไป Review Queue พร้อม reason |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Idempotency Contract
|
||||||
|
|
||||||
|
**HTTP Header ที่ต้องส่งทุก Request:**
|
||||||
|
```
|
||||||
|
Idempotency-Key: <document_number>:<batch_id>
|
||||||
|
```
|
||||||
|
|
||||||
|
**Backend Logic:**
|
||||||
|
```
|
||||||
|
IF idempotency_key EXISTS in import_transactions → RETURN HTTP 200 (no action)
|
||||||
|
ELSE → Process normally → INSERT import_transactions → RETURN HTTP 201
|
||||||
|
```
|
||||||
|
|
||||||
|
ป้องกัน Revision ซ้ำกรณี n8n Retry หรือ Network Error
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Duplicate Handling Clarification
|
||||||
|
|
||||||
|
Bypass Duplicate **Validation Error**
|
||||||
|
|
||||||
|
Hard Rules:
|
||||||
|
- ❌ Migration Token ไม่สามารถ Overwrite Revision ที่มีอยู่
|
||||||
|
- ❌ Migration Token ไม่สามารถ Delete Revision ก่อนหน้า
|
||||||
|
- ✅ Migration Token trigger Revision increment logic ตามปกติเท่านั้น
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Storage Governance (Patch)
|
||||||
|
|
||||||
|
**ข้อห้าม:**
|
||||||
|
```
|
||||||
|
❌ mv /data/dms/staging_ai/TCC-COR-0001.pdf /final/path/...
|
||||||
|
```
|
||||||
|
|
||||||
|
**ข้อบังคับ:**
|
||||||
|
```
|
||||||
|
✅ POST /api/correspondences/import
|
||||||
|
body: { source_file_path: "/data/dms/staging_ai/TCC-COR-0001.pdf", ... }
|
||||||
|
```
|
||||||
|
|
||||||
|
Backend จะ:
|
||||||
|
1. Generate UUID
|
||||||
|
2. Enforce path strategy: `/data/dms/uploads/YYYY/MM/{uuid}.pdf`
|
||||||
|
3. Move file atomically ผ่าน StorageService
|
||||||
|
4. Create revision folder ถ้าจำเป็น
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Review Queue Contract
|
||||||
|
|
||||||
|
- `migration_review_queue` เป็น **Temporary Table เท่านั้น** — ไม่ใช่ Business Schema
|
||||||
|
- ห้ามสร้าง Correspondence record จนกว่า Admin จะ Approve
|
||||||
|
- Approval Flow: `Review → Admin Approve → POST /api/correspondences/import`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Revision Drift Protection
|
||||||
|
|
||||||
|
ถ้า Excel มี revision column:
|
||||||
|
```
|
||||||
|
IF excel_revision != current_db_revision + 1
|
||||||
|
→ ROUTE ไป Review Queue พร้อม reason: "Revision drift"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Execution Time Estimate
|
||||||
|
|
||||||
|
| Parameter | ค่า |
|
||||||
|
| -------------------- | ---------------------------- |
|
||||||
|
| Delay ระหว่าง Request | 2 วินาที |
|
||||||
|
| Inference Time (avg) | ~1 วินาที |
|
||||||
|
| เวลาต่อ Record | ~3 วินาที |
|
||||||
|
| จำนวน Record | 20,000 |
|
||||||
|
| เวลารวม | ~60,000 วินาที (~16.6 ชั่วโมง) |
|
||||||
|
| **จำนวนคืนที่ต้องใช้** | **~3–4 คืน** (รัน 22:00–06:00) |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Encoding Normalization
|
||||||
|
|
||||||
|
ก่อน Ingestion ทุกครั้ง:
|
||||||
|
- Excel data → Convert เป็น **UTF-8**
|
||||||
|
- Filename → Normalize เป็น **NFC UTF-8** ป้องกันปัญหาภาษาไทยเพี้ยนข้าม OS
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Security Constraints
|
||||||
|
|
||||||
|
1. Migration Token อายุ **≤ 7 วัน** — Revoke ทันทีหลัง Migration
|
||||||
|
2. Token Bypass ได้เฉพาะ: Virus Scan, Duplicate Validation Error, Created-by
|
||||||
|
3. Token **ไม่มีสิทธิ์** ลบหรือ Overwrite Record เดิม
|
||||||
|
4. ทุก Request บันทึก Audit Log: `action=IMPORT, source=MIGRATION, created_by=SYSTEM_IMPORT`
|
||||||
|
5. **IP Whitelist:** ใช้ได้เฉพาะจาก `<NAS_IP>`
|
||||||
|
6. **Nginx Rate Limit:** `limit_req zone=migration burst=5 nodelay`
|
||||||
|
7. **Docker Hardening:** `mem_limit: 2g`, log rotation `max-size: 10m, max-file: 3`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Rollback Strategy
|
||||||
|
|
||||||
|
1. Disable Migration Token ใน DB ทันที
|
||||||
|
2. ลบ Records ทั้งหมด `created_by = 'SYSTEM_IMPORT'` ผ่าน Transaction SQL (รวม `import_transactions`)
|
||||||
|
3. ย้ายไฟล์ PDF กลับ `migration_temp/`
|
||||||
|
4. Reset `migration_progress` และ `migration_fallback_state`
|
||||||
|
5. วิเคราะห์ Root Cause ก่อนรันใหม่
|
||||||
|
|
||||||
|
รายละเอียดดูที่ `03-04-legacy-data-migration.md` หัวข้อ 4
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Architecture Validation Checklist (GO-LIVE GATE)
|
||||||
|
|
||||||
|
### 🟢 A. Infrastructure Validation
|
||||||
|
|
||||||
|
| Check | Expected | ✅ |
|
||||||
|
| ---------------------------- | ------------- | --- |
|
||||||
|
| Ollama `/api/tags` reachable | HTTP 200 | |
|
||||||
|
| Backend `/health` OK | HTTP 200 | |
|
||||||
|
| MariaDB reachable | SELECT 1 | |
|
||||||
|
| `staging_ai` mounted RO | ls works | |
|
||||||
|
| `migration_logs` mounted RW | write test OK | |
|
||||||
|
| GPU VRAM < 70% idle | safe margin | |
|
||||||
|
| Disk space > 30% free | safe | |
|
||||||
|
|
||||||
|
### 🟢 B. Security Validation
|
||||||
|
|
||||||
|
| Check | Expected | ✅ |
|
||||||
|
| -------------------------------------- | -------- | --- |
|
||||||
|
| Migration Token expiry ≤ 7 days | Verified | |
|
||||||
|
| Token IP Whitelist = NAS IP only | Verified | |
|
||||||
|
| Token cannot DELETE records | Verified | |
|
||||||
|
| Token cannot UPDATE non-import records | Verified | |
|
||||||
|
| Audit Log records `source=MIGRATION` | Verified | |
|
||||||
|
| Nginx rate limit configured | Verified | |
|
||||||
|
| Docker mem_limit = 2g | Verified | |
|
||||||
|
|
||||||
|
### 🟢 C. Data Integrity Validation
|
||||||
|
|
||||||
|
| Check | Expected | ✅ |
|
||||||
|
| ---------------------------------------------- | -------------- | --- |
|
||||||
|
| Enum fetched from `/api/meta/categories` | Not hardcoded | |
|
||||||
|
| `Idempotency-Key` header enforced | Verified | |
|
||||||
|
| Duplicate revision test (run same batch twice) | No overwrite | |
|
||||||
|
| Revision drift test | Sent to Review | |
|
||||||
|
| Storage path matches Core Storage Spec v1.8.0 | Verified | |
|
||||||
|
| Encoding normalization NFC UTF-8 | Verified | |
|
||||||
|
|
||||||
|
### 🟢 D. Workflow Validation (Dry Run 20 Records)
|
||||||
|
|
||||||
|
| Check | Expected | ✅ |
|
||||||
|
| ---------------------------------------- | ------------ | --- |
|
||||||
|
| JSON parse success rate | > 95% | |
|
||||||
|
| Confidence distribution reasonable | Mean 0.7–0.9 | |
|
||||||
|
| Checkpoint updates every 10 records | Verified | |
|
||||||
|
| Fallback model not prematurely triggered | Verified | |
|
||||||
|
| Reject log written to `migration_logs/` | Verified | |
|
||||||
|
| Error log written to `migration_logs/` | Verified | |
|
||||||
|
| Review queue inserts to DB | Verified | |
|
||||||
|
|
||||||
|
### 🟢 E. Performance Validation
|
||||||
|
|
||||||
|
| Check | Expected | ✅ |
|
||||||
|
| ------------------------------- | -------- | --- |
|
||||||
|
| 10 records processed < 1 minute | Verified | |
|
||||||
|
| GPU temp < 80°C | Verified | |
|
||||||
|
| No memory leak after 1 hour | Verified | |
|
||||||
|
| No duplicate revision created | Verified | |
|
||||||
|
|
||||||
|
### 🟢 F. Rollback Test (Mandatory)
|
||||||
|
|
||||||
|
| Check | Expected | ✅ |
|
||||||
|
| ------------------------------------ | ----------------- | --- |
|
||||||
|
| Disable token works | is_active = false | |
|
||||||
|
| Delete `SYSTEM_IMPORT` records works | COUNT = 0 | |
|
||||||
|
| `import_transactions` cleared | COUNT = 0 | |
|
||||||
|
| Checkpoint reset to 0 | Verified | |
|
||||||
|
| Fallback state reset | Verified | |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## GO / NO-GO Criteria
|
||||||
|
|
||||||
|
**GO ถ้า:**
|
||||||
|
- A, B, C ทุก Check = PASS
|
||||||
|
- Dry run error rate < 10%
|
||||||
|
- JSON parse failure < 5%
|
||||||
|
- Revision conflict < 3%
|
||||||
|
|
||||||
|
**NO-GO ถ้า:**
|
||||||
|
- Enum mismatch (Category hardcoded)
|
||||||
|
- Idempotency ไม่ได้ implement
|
||||||
|
- Storage bypass (move file โดยตรง)
|
||||||
|
- Audit log ไม่ครบ
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Final Architectural Assessment
|
||||||
|
|
||||||
|
| Area | Status |
|
||||||
|
| ------------------ | ------------------------------------------------ |
|
||||||
|
| ADR Compliance | ✅ Fully aligned |
|
||||||
|
| Security | ✅ Hardened (IP Whitelist, Rate Limit, Docker) |
|
||||||
|
| Data Integrity | ✅ Controlled (Idempotency, Revision Drift, Enum) |
|
||||||
|
| Storage Governance | ✅ Enforced (StorageService only) |
|
||||||
|
| Operational Safety | ✅ Production Grade |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
*สำหรับขั้นตอนปฏิบัติงานแบบละเอียด ดูที่ `03-04-legacy-data-migration.md` และ `03-05-n8n-migration-setup-guide.md`*
|
||||||
|
|||||||
378
specs/06-Decision-Records/Patch 1.8.1.md
Normal file
378
specs/06-Decision-Records/Patch 1.8.1.md
Normal file
@@ -0,0 +1,378 @@
|
|||||||
|
สรุป Patch 1.8.1
|
||||||
|
---
|
||||||
|
|
||||||
|
# 📘 1) Formal Spec — Version 1.8.1
|
||||||
|
|
||||||
|
**Document ID:** DMS-SPEC-1.8.1
|
||||||
|
**Status:** Approved for Implementation
|
||||||
|
**Supersedes:** 1.8.0
|
||||||
|
**Effective Date:** 2026-02-26
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. Purpose
|
||||||
|
|
||||||
|
Spec 1.8.1 แก้ความไม่สอดคล้องระหว่าง:
|
||||||
|
|
||||||
|
* 03-04-legacy-data-migration.md
|
||||||
|
* 03-05-n8n-migration-setup-guide.md
|
||||||
|
* ADR-017-ollama-data-migration.md
|
||||||
|
|
||||||
|
และกำหนด Production Boundary ที่ชัดเจน
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. Authoritative Architecture (Binding)
|
||||||
|
|
||||||
|
### Infrastructure Layout
|
||||||
|
|
||||||
|
| Component | Host | Responsibility |
|
||||||
|
| ------------- | ------- | -------------------- |
|
||||||
|
| DMS App | QNAP | Production system |
|
||||||
|
| MariaDB | QNAP | Authoritative DB |
|
||||||
|
| File Storage | QNAP | Primary file store |
|
||||||
|
| Reverse Proxy | QNAP | Public ingress |
|
||||||
|
| Ollama | ASUSTOR | AI processing only |
|
||||||
|
| n8n | ASUSTOR | Automation engine |
|
||||||
|
| Portainer | ASUSTOR | Container management |
|
||||||
|
|
||||||
|
**Constraint:**
|
||||||
|
|
||||||
|
* Ollama MUST NOT run on QNAP
|
||||||
|
* AI containers MUST NOT access production DB directly
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. Source of Truth Definition
|
||||||
|
|
||||||
|
During Migration:
|
||||||
|
|
||||||
|
| Data Type | Authority |
|
||||||
|
| ------------ | --------------------------------------- |
|
||||||
|
| File content | Legacy file server |
|
||||||
|
| RFA metadata | Gmail notification |
|
||||||
|
| Assignment | Circulation sheet |
|
||||||
|
| DMS DB | NOT authoritative until validation pass |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. Metadata Mapping Contract (Mandatory)
|
||||||
|
|
||||||
|
| Legacy | DMS | Required | Rule |
|
||||||
|
| ------------- | ------------- | -------- | ----------------- |
|
||||||
|
| RFA No | rfa_number | YES | UNIQUE |
|
||||||
|
| Title | title | YES | NOT NULL |
|
||||||
|
| Issue Date | issue_date | YES | Valid date |
|
||||||
|
| Revision | revision_code | YES | Pattern validated |
|
||||||
|
| Assigned User | user_id | YES | FK must exist |
|
||||||
|
| File Path | file_path | YES | File must exist |
|
||||||
|
|
||||||
|
Migration MUST fail if required fields invalid.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. Idempotent Execution Requirement
|
||||||
|
|
||||||
|
Automation must:
|
||||||
|
|
||||||
|
* Check existence by rfa_number
|
||||||
|
* Validate file hash
|
||||||
|
* UPDATE instead of INSERT if exists
|
||||||
|
* Prevent duplicate revision chain
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. Folder Standard
|
||||||
|
|
||||||
|
```
|
||||||
|
/data/dms/
|
||||||
|
├── uploads/YYYY/MM/
|
||||||
|
├── staging_ai/
|
||||||
|
├── migration_logs/
|
||||||
|
└── archive_legacy/
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 7. File Naming Standard
|
||||||
|
|
||||||
|
```
|
||||||
|
{RFA_NO}_{REV}_{YYYYMMDD}.pdf
|
||||||
|
```
|
||||||
|
|
||||||
|
Example:
|
||||||
|
|
||||||
|
```
|
||||||
|
RFA-2026-001_A_20260225.pdf
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 8. Logging Standard
|
||||||
|
|
||||||
|
| System | Required |
|
||||||
|
| ---------------- | -------------- |
|
||||||
|
| Migration script | structured log |
|
||||||
|
| n8n | execution log |
|
||||||
|
| Ollama | inference log |
|
||||||
|
| DMS | audit_log |
|
||||||
|
|
||||||
|
Retention: 90 days minimum.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 9. Dry Run Policy
|
||||||
|
|
||||||
|
All migrations MUST run with:
|
||||||
|
|
||||||
|
```
|
||||||
|
--dry-run
|
||||||
|
```
|
||||||
|
|
||||||
|
No DB commit until validation approved.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 10. Rollback Strategy
|
||||||
|
|
||||||
|
1. Disable n8n
|
||||||
|
2. Restore DB snapshot
|
||||||
|
3. Restore file snapshot
|
||||||
|
4. Clear staging_ai
|
||||||
|
5. Re-run validation
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
# 📄 2) ADR-018 — AI Boundary Hardening
|
||||||
|
|
||||||
|
**Title:** AI Isolation & Production Boundary Enforcement
|
||||||
|
**Status:** Accepted
|
||||||
|
**Date:** 2026-02-26
|
||||||
|
**Supersedes:** Clarifies ADR-017
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
AI-based migration using Ollama introduces:
|
||||||
|
|
||||||
|
* DB corruption risk
|
||||||
|
* Hallucinated metadata
|
||||||
|
* Unauthorized modification
|
||||||
|
* Privilege escalation risk
|
||||||
|
|
||||||
|
Production DMS must remain authoritative.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Decision
|
||||||
|
|
||||||
|
### 1. AI Isolation Model
|
||||||
|
|
||||||
|
Ollama must:
|
||||||
|
|
||||||
|
* Run on ASUSTOR only
|
||||||
|
* Have NO DB credentials
|
||||||
|
* Have NO write access to uploads
|
||||||
|
* Access only `/staging_ai`
|
||||||
|
* Output JSON only
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 2. Data Flow Model
|
||||||
|
|
||||||
|
```
|
||||||
|
Legacy File → staging_ai → Ollama → JSON
|
||||||
|
↓
|
||||||
|
Validation Script
|
||||||
|
↓
|
||||||
|
DMS API (write)
|
||||||
|
```
|
||||||
|
|
||||||
|
AI never writes directly.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 3. API Gatekeeping
|
||||||
|
|
||||||
|
All writes must go through:
|
||||||
|
|
||||||
|
* Authenticated DMS API
|
||||||
|
* RBAC enforced
|
||||||
|
* Audit log recorded
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 4. Hallucination Mitigation
|
||||||
|
|
||||||
|
AI output must:
|
||||||
|
|
||||||
|
* Match schema
|
||||||
|
* Pass validation script
|
||||||
|
* Fail on missing required fields
|
||||||
|
* Reject unknown users
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 5. Security Controls
|
||||||
|
|
||||||
|
| Risk | Control |
|
||||||
|
| ------------------ | ------------------ |
|
||||||
|
| DB corruption | No DB access |
|
||||||
|
| File overwrite | Read-only mount |
|
||||||
|
| Public AI exposure | No exposed port |
|
||||||
|
| Data leak | Internal VLAN only |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Consequences
|
||||||
|
|
||||||
|
Pros:
|
||||||
|
|
||||||
|
* Production safe
|
||||||
|
* Predictable migration
|
||||||
|
* Audit trail preserved
|
||||||
|
|
||||||
|
Cons:
|
||||||
|
|
||||||
|
* Slightly slower pipeline
|
||||||
|
* Requires validation layer
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
# 🧠 3) Full Migration Runbook
|
||||||
|
|
||||||
|
**Production Execution Guide**
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
# PHASE 0 — Pre-Run Validation
|
||||||
|
|
||||||
|
☐ Full DB backup
|
||||||
|
☐ File storage snapshot
|
||||||
|
☐ Restore test verified
|
||||||
|
☐ 10-sample manual compare
|
||||||
|
☐ Dry-run executed
|
||||||
|
☐ Dry-run report approved
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
# PHASE 1 — Environment Preparation
|
||||||
|
|
||||||
|
1. Stop public automation
|
||||||
|
2. Disable n8n production workflows
|
||||||
|
3. Clear `/staging_ai`
|
||||||
|
4. Confirm Ollama healthy
|
||||||
|
5. Confirm DMS API reachable
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
# PHASE 2 — Controlled Batch Migration
|
||||||
|
|
||||||
|
Batch size recommendation:
|
||||||
|
|
||||||
|
* 20–50 RFAs per batch
|
||||||
|
|
||||||
|
Process:
|
||||||
|
|
||||||
|
1. Copy files to `/staging_ai`
|
||||||
|
2. Run AI extraction
|
||||||
|
3. Validate JSON
|
||||||
|
4. Push via DMS API
|
||||||
|
5. Log result
|
||||||
|
6. Manual sample verify (10%)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
# PHASE 3 — Post-Batch Validation
|
||||||
|
|
||||||
|
After each batch:
|
||||||
|
|
||||||
|
☐ Record count match
|
||||||
|
☐ File open test
|
||||||
|
☐ Revision chain correct
|
||||||
|
☐ Assignment correct
|
||||||
|
☐ No duplicate rfa_number
|
||||||
|
|
||||||
|
If fail → STOP and rollback.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
# PHASE 4 — Cutover
|
||||||
|
|
||||||
|
When all batches pass:
|
||||||
|
|
||||||
|
1. Enable n8n automation
|
||||||
|
2. Monitor logs 24h
|
||||||
|
3. Lock legacy system (read-only)
|
||||||
|
4. Final backup snapshot
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
# PHASE 5 — Post-Go-Live Monitoring (72 hours)
|
||||||
|
|
||||||
|
Monitor:
|
||||||
|
|
||||||
|
* DB errors
|
||||||
|
* Duplicate insert
|
||||||
|
* Missing files
|
||||||
|
* AI extraction errors
|
||||||
|
* API error rate
|
||||||
|
|
||||||
|
If anomaly >5% → trigger rollback plan.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
# 🚨 Emergency Rollback Procedure
|
||||||
|
|
||||||
|
1. Disable n8n
|
||||||
|
2. Restore DB snapshot
|
||||||
|
3. Restore file snapshot
|
||||||
|
4. Clear staging_ai
|
||||||
|
5. Reconcile delta manually
|
||||||
|
|
||||||
|
Target RTO: < 2 hours
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
# 📊 Risk Matrix (Condensed)
|
||||||
|
|
||||||
|
| Risk | Probability | Impact | Mitigation |
|
||||||
|
| ---------------- | ----------- | -------- | ----------------- |
|
||||||
|
| Duplicate RFA | Medium | High | Idempotent check |
|
||||||
|
| Wrong assignment | Medium | Medium | Validation rule |
|
||||||
|
| AI hallucination | Medium | High | Schema validation |
|
||||||
|
| Storage mismatch | Low | High | File hash verify |
|
||||||
|
| DB corruption | Low | Critical | No AI DB access |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
# 🏁 Production Readiness Criteria
|
||||||
|
|
||||||
|
System may go live only if:
|
||||||
|
|
||||||
|
* All dry-run tests pass
|
||||||
|
* 100% required fields valid
|
||||||
|
* 0 duplicate RFA
|
||||||
|
* Sample QA pass >95%
|
||||||
|
* Backup verified
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
# 🎯 Final Result
|
||||||
|
|
||||||
|
ตอนนี้เอกสารทั้งชุด:
|
||||||
|
|
||||||
|
✔ Architecture consistent
|
||||||
|
✔ AI boundary hardened
|
||||||
|
✔ Migration deterministic
|
||||||
|
✔ Rollback defined
|
||||||
|
✔ Production-safe
|
||||||
|
|
||||||
|
---
|
||||||
2372
specs/99-archives/lcbp3-v1.7.0-seed-contractdrawing.sql
Normal file
2372
specs/99-archives/lcbp3-v1.7.0-seed-contractdrawing.sql
Normal file
File diff suppressed because it is too large
Load Diff
1067
specs/99-archives/lcbp3-v1.7.0-seed-permissions.sql
Normal file
1067
specs/99-archives/lcbp3-v1.7.0-seed-permissions.sql
Normal file
File diff suppressed because it is too large
Load Diff
5626
specs/99-archives/lcbp3-v1.7.0-seed-shopdrawing.sql
Normal file
5626
specs/99-archives/lcbp3-v1.7.0-seed-shopdrawing.sql
Normal file
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user