260227:1640 20260227: add ollama #2
All checks were successful
Build and Deploy / deploy (push) Successful in 2m37s
All checks were successful
Build and Deploy / deploy (push) Successful in 2m37s
This commit is contained in:
@@ -1,15 +1,19 @@
|
||||
---
|
||||
description: legacy PDF document migration to system v1.8.0 uses n8n and Ollama
|
||||
version: 1.8.0
|
||||
---
|
||||
|
||||
# 03-04: Legacy Data Migration Plan (PDF 20k Docs)
|
||||
|
||||
| description | version |
|
||||
| ------------------------------------------------------------------ | ------- |
|
||||
| legacy PDF document migration to system v1.8.0 uses n8n and Ollama | 1.8.0 |
|
||||
|
||||
> **Note:** Category Enum system-driven, Idempotency Contract, Duplicate Handling Clarification, Storage Enforcement, Audit Log Enhancement, Review Queue Integration, Revision Drift Protection, Execution Time, Encoding Normalization, Security Hardening, AI Physical Isolation (ASUSTOR), Folder Standard (/data/dms)
|
||||
|
||||
---
|
||||
|
||||
## 1. วัตถุประสงค์ (Objectives)
|
||||
|
||||
* นำเข้าเอกสาร PDF 20,000 ฉบับ พร้อม Metadata จาก Excel (Legacy system export) เข้าสู่ระบบ LCBP3-DMS
|
||||
* ใช้ AI (Ollama Local Model) เพื่อตรวจสอบความถูกต้องของลักษณะข้อมูล (Data format, Title consistency) ก่อนการนำเข้า
|
||||
* รักษาโครงสร้างความสัมพันธ์ (Project / Contract / Ref No.) และระบบการทำ Revision ตาม Business Rules
|
||||
- นำเข้าเอกสาร PDF 20,000 ฉบับ พร้อม Metadata จาก Excel (Legacy system export) เข้าสู่ระบบ LCBP3-DMS
|
||||
- ใช้ AI (Ollama Local Model) เพื่อตรวจสอบความถูกต้องของลักษณะข้อมูล (Data format, Title consistency) ก่อนการนำเข้า
|
||||
- รักษาโครงสร้างความสัมพันธ์ (Project / Contract / Ref No.) และระบบการทำ Revision ตาม Business Rules
|
||||
- **Checkpoint Support:** รองรับการหยุดและเริ่มงานต่อ (Resume) จากจุดที่ค้างอยู่ได้กรณีเกิดเหตุขัดข้อง
|
||||
|
||||
> **Note:** เอกสารนี้ขยายความถึงวิธีปฏิบัติ (Implementation) จากการตัดสินใจทางสถาปัตยกรรมใน [ADR-017: Ollama Data Migration Architecture](../06-Decision-Records/ADR-017-ollama-data-migration.md)
|
||||
|
||||
@@ -17,13 +21,11 @@ version: 1.8.0
|
||||
|
||||
## 2. โครงสร้างพื้นฐาน (Migration Infrastructure)
|
||||
|
||||
การนำเข้าข้อมูลชุดใหญ่นี้จะไม่กระทำผ่าน User Interface แต่จะใช้โครงสร้างสถาปัตยกรรมชั่วคราวและ APIs:
|
||||
|
||||
* **Migration Orchestrator:** n8n (รันจาก Docker Container บน QNAP NAS)
|
||||
* **AI Validator:** Ollama Native (รันบน Windows Desktop - i9 + RTX 2060 SUPER)
|
||||
* **Target Database:** MariaDB (`correspondences` table)
|
||||
* **Target Storage:** QNAP File System (Mount volumes เข้า Application)
|
||||
* **Connection:** ข้อมูลก้อนใหญ่ถูกโยกย้ายผ่าน 2.5G LAN + LACP เพื่อประสิทธิผลสูงสุด
|
||||
- **Migration Orchestrator:** n8n (รันจาก Docker Container บน ASUSTOR NAS)
|
||||
- **AI Validator:** Ollama (รันใน Internal Network บน ASUSTOR NAS)
|
||||
- **Target Database:** MariaDB (`correspondences` table) บน QNAP NAS
|
||||
- **Target Storage:** QNAP File System — **ผ่าน Backend StorageService API เท่านั้น** (ห้าม move file โดยตรง)
|
||||
- **Connection:** 2.5G LAN + LACP / Internal VLAN
|
||||
|
||||
---
|
||||
|
||||
@@ -31,71 +33,395 @@ version: 1.8.0
|
||||
|
||||
### Phase 1: การเตรียม Infrastructure และ Storage (สัปดาห์ที่ 1)
|
||||
|
||||
1. **File Migration:**
|
||||
ย้ายไฟล์ PDF ทั้งหมดจากแหล่งเก็บ (Desktop/External Drive) ไปยัง Folder ชั่วคราวบน NAS เพื่อรอการประมวลผล แนะนำใช้ `Robocopy` หรือ `Rsync`
|
||||
* *Target Path:* `/share/DMS_Storage/migration_temp/`
|
||||
**File Migration:**
|
||||
- ย้ายไฟล์ PDF ทั้งหมดจากแหล่งเก็บไปยัง Folder ชั่วคราวบน NAS (QNAP)
|
||||
- Target Path: `/data/dms/staging_ai/`
|
||||
|
||||
2. **Mount Folder:**
|
||||
ทำการ Bind Mount โฟลเดอร์ `migration_temp/` เข้ากับ Container ของ n8n เพื่อให้ n8n เช็คความมีอยู่ของไฟล์ด้วย Disk I/O speed.
|
||||
**Mount Folder:**
|
||||
- Bind Mount `/data/dms/staging_ai/` เข้ากับ n8n Container แบบ **read-only**
|
||||
- สร้าง `/data/dms/migration_logs/` Volume แยกสำหรับเขียน Log แบบ **read-write**
|
||||
|
||||
3. **Ollama Config:**
|
||||
* ติดตั้ง Ollama แบบ Native บน Desktop
|
||||
* ตั้งค่า Environment Variable `OLLAMA_HOST=0.0.0.0`
|
||||
* Fix IP ให้ Desktop เครื่องโฮสต์ และเปิด Port `11434` ที่ระดับ OS Firewall
|
||||
* รันคำสั่ง `ollama pull llama3.2` (หรือ Model ที่เหมาะสม)
|
||||
**Ollama Config:**
|
||||
- ติดตั้ง Ollama บน ASUSTOR NAS
|
||||
- No DB credentials, Internal network only
|
||||
|
||||
```bash
|
||||
# แนะนำ: llama3.2:3b (เร็ว, VRAM ~3GB, เหมาะ Classification)
|
||||
ollama pull llama3.2:3b
|
||||
|
||||
# Fallback: mistral:7b-instruct-q4_K_M (แม่นกว่า, VRAM ~5GB)
|
||||
# ollama pull mistral:7b-instruct-q4_K_M
|
||||
```
|
||||
|
||||
**ทดสอบ Ollama:**
|
||||
```bash
|
||||
curl http://<OLLAMA_HOST>:11434/api/generate \
|
||||
-d '{"model":"llama3.2:3b","prompt":"reply: ok","stream":false}'
|
||||
```
|
||||
|
||||
**Concurrency Configuration:**
|
||||
- Sequential: Batch Size = 1, Delay ≥ 2 วินาที, ปิด Parallel Execution
|
||||
- เพิ่ม Health Check Node ก่อนเริ่ม Batch เพื่อป้องกัน Workflow ค้างหาก Desktop Sleep หรือ Overheat
|
||||
|
||||
---
|
||||
|
||||
### Phase 2: การเตรียม Target Database และ API (สัปดาห์ที่ 1)
|
||||
|
||||
1. **SQL Indexing:**
|
||||
เพื่อให้ API ตรวจจับ Duplicate record ได้อย่างรวดเร็ว (สำหรับ 20,000 แถว) ให้กระทำคำสั่ง SQL เพื่อเพิ่ม Index ลงฐานข้อมูล Production ชั่วคราว (หรือถาวร):
|
||||
```sql
|
||||
ALTER TABLE correspondences ADD INDEX idx_doc_number (document_number);
|
||||
ALTER TABLE correspondences ADD INDEX idx_deleted_at (deleted_at);
|
||||
```
|
||||
**SQL Indexing:**
|
||||
```sql
|
||||
ALTER TABLE correspondences ADD INDEX idx_doc_number (document_number);
|
||||
ALTER TABLE correspondences ADD INDEX idx_deleted_at (deleted_at);
|
||||
ALTER TABLE correspondences ADD INDEX idx_created_by (created_by);
|
||||
```
|
||||
|
||||
2. **API Authentication:**
|
||||
ระบบ LCBP3-DMS ต้องสร้าง Access Token แบบผูกพันกับ Role ระดับสูง (เช่น `SYSTEM_ADMIN` หรือ `MIGRATION_USER`) ซึ่งมีสิทธิ์ Bypass การ Validation บางประการ (ถ้าได้รับการอนุญาต) ส่งมอบให้ n8n
|
||||
**Checkpoint Table:**
|
||||
```sql
|
||||
CREATE TABLE IF NOT EXISTS migration_progress (
|
||||
batch_id VARCHAR(50) PRIMARY KEY,
|
||||
last_processed_index INT DEFAULT 0,
|
||||
status ENUM('RUNNING','COMPLETED','FAILED') DEFAULT 'RUNNING',
|
||||
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP
|
||||
);
|
||||
```
|
||||
|
||||
**Idempotency Table :**
|
||||
```sql
|
||||
CREATE TABLE IF NOT EXISTS import_transactions (
|
||||
id INT AUTO_INCREMENT PRIMARY KEY,
|
||||
idempotency_key VARCHAR(255) UNIQUE NOT NULL,
|
||||
document_number VARCHAR(100),
|
||||
batch_id VARCHAR(100),
|
||||
status_code INT DEFAULT 201,
|
||||
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||
INDEX idx_idem_key (idempotency_key)
|
||||
);
|
||||
```
|
||||
|
||||
> **Idempotency Logic:** ถ้า `idempotency_key` ซ้ำ → Backend คืน HTTP 200 ทันที (ไม่สร้าง Revision ซ้ำ) ถ้าไม่ซ้ำ → ประมวลผลปกติ
|
||||
|
||||
**API Authentication — Migration Token:**
|
||||
```sql
|
||||
INSERT INTO users (username, email, role, is_active)
|
||||
VALUES ('migration_bot', 'migration@system.internal', 'SYSTEM_ADMIN', true);
|
||||
```
|
||||
|
||||
**Scope ของ Migration Token (Patch — คำนิยามชัดเจน):**
|
||||
|
||||
| สิทธิ์ | ปกติ | Migration Token | หมายเหตุ |
|
||||
| ------------------------------------- | --- | --------------- | --------------------------------- |
|
||||
| Bypass File Virus Scan | ❌ | ✅ | ไฟล์ผ่าน Scan มาแล้วก่อน Import |
|
||||
| Bypass Duplicate **Validation Error** | ❌ | ✅ | **Revision Logic ยัง enforce ปกติ** |
|
||||
| Bypass Created-by User validation | ❌ | ✅ | |
|
||||
| Overwrite existing revision | ❌ | ❌ | **ห้ามโดยเด็ดขาด** |
|
||||
| Delete previous revision | ❌ | ❌ | **ห้ามโดยเด็ดขาด** |
|
||||
| ลบ / แก้ไข Record อื่น | ❌ | ❌ | **ห้ามโดยเด็ดขาด** |
|
||||
|
||||
> ⚠️ **Patch Clarification:** "Bypass Duplicate Number Check" ถูกแทนด้วย "Bypass Duplicate **Validation Error**" — Revision increment logic ยังทำงานตามปกติทุกกรณี
|
||||
|
||||
- **Token Expiry:** ไม่เกิน **7 วัน** ต้อง Revoke ทันทีหลัง Migration เสร็จ
|
||||
- **IP Whitelist:** ใช้ได้เฉพาะจาก `<NAS_IP>` เท่านั้น
|
||||
- **Audit:** ทุก Request บันทึก `created_by = 'SYSTEM_IMPORT'`
|
||||
|
||||
---
|
||||
|
||||
### Phase 3: การออกแบบ n8n Workflow (The Migration Logic)
|
||||
|
||||
Workflow ควบคุมการไหลของข้อมูลประกอบด้วย 4 ส่วนการทำงาน:
|
||||
#### Node 0: Pre-flight Health Check + Fetch System Categories
|
||||
|
||||
1. **Data Reader Node:**
|
||||
อ่านไฟล์ Metadata จาก Excel แล้วแยกส่วน (Batching) เพื่อทยอยดำเนินการทีละ 100 แถว
|
||||
2. **File Validator Node:**
|
||||
ตรวจจับว่าเส้นทางนามสกุลไฟล์ เช่น `Iyyccnnnn-[doc_number].pdf` มีอยู่จริงในระบบเก็บข้อมูล NAS (บริเวณโฟลเดอร์ temp)
|
||||
3. **AI Analysis Node (HTTP Request to Ollama):**
|
||||
* ส่ง Metadata เช่น (Document Number, Title) ให้ AI ตรวจสอบ
|
||||
* *System Prompt Example:* "You are a Document Controller. Verify if the document title [Title] matches the numbering pattern [Pattern]. Categorize this into [Category List]. Output in JSON format only."
|
||||
4. **Data Ingestion Node:**
|
||||
* ใช้ HTTP Request ยิง POST เวิร์กโหลดเข้าไปที่ Backend API
|
||||
* Backend API ต้องมีความสามารถในการรองรับ *Idempotency* และจัดการตรวจสอบว่าหาก `document_number` เกิดซ้ำกัน ต้องยกระดับไปสร้างบันทึกใหม่เป็น Version / Revision (+1) ไม่ใช่การทับลง Record เดิม
|
||||
ตรวจสอบทุก dependency ก่อน Batch:
|
||||
|
||||
1. HTTP GET Ollama `/api/tags` → ต้อง HTTP 200
|
||||
2. MariaDB `SELECT 1` → ต้องเชื่อมได้
|
||||
3. HTTP GET Backend `/health` → ต้อง HTTP 200
|
||||
4. File Mount Check → `staging_ai` มีไฟล์, `migration_logs` เขียนได้
|
||||
|
||||
**Fetch System Categories (Patch — ห้าม hardcode):**
|
||||
```http
|
||||
GET /api/meta/categories
|
||||
Authorization: Bearer <MIGRATION_TOKEN>
|
||||
```
|
||||
Response:
|
||||
```json
|
||||
{ "categories": ["Correspondence","RFA","Drawing","Transmittal","Report","Other"] }
|
||||
```
|
||||
n8n ต้องเก็บ categories นี้ไว้ใน Workflow Variable (`system_categories`) และ inject เข้า AI Prompt ทุก Request
|
||||
|
||||
#### Node 1: Data Reader & Checkpoint
|
||||
|
||||
- อ่าน Checkpoint จาก **MariaDB Node แยก** (ไม่ใช่ async call ใน Code Node)
|
||||
- Batch ทีละ **10–20 แถว** ตาม `$env.MIGRATION_BATCH_SIZE`
|
||||
- ติด `original_index` ทุก Item
|
||||
|
||||
**Encoding Normalization:**
|
||||
```javascript
|
||||
// Normalize ข้อมูลจาก Excel เป็น UTF-8 NFC ก่อนประมวลผล
|
||||
const normalize = (str) => {
|
||||
if (!str) return '';
|
||||
return Buffer.from(str, 'utf8').toString('utf8').normalize('NFC');
|
||||
};
|
||||
|
||||
return items.map(item => ({
|
||||
...item,
|
||||
json: {
|
||||
...item.json,
|
||||
document_number: normalize(item.json.document_number),
|
||||
title: normalize(item.json.title)
|
||||
}
|
||||
}));
|
||||
```
|
||||
|
||||
#### Node 2: File Validator & Sanitizer
|
||||
|
||||
- ตรวจสอบไฟล์ PDF มีอยู่จริงบน NAS
|
||||
- Normalize ชื่อไฟล์เป็น **UTF-8 NFC**
|
||||
- Path Traversal Guard: resolved path ต้องอยู่ใน `/data/dms/staging_ai` เท่านั้น
|
||||
- **Output 0** → valid → Node 3
|
||||
- **Output 1** → error → Node 5D (ไม่หายเงียบ)
|
||||
|
||||
#### Node 3: AI Analysis (Sequential เท่านั้น)
|
||||
|
||||
**System Prompt:**
|
||||
```text
|
||||
You are a Document Controller for a large construction project.
|
||||
Your task is to validate document metadata.
|
||||
You MUST respond ONLY with valid JSON. No explanation, no markdown, no extra text.
|
||||
If there are no issues, "detected_issues" must be an empty array [].
|
||||
```
|
||||
|
||||
**User Prompt (Category List มาจาก Backend ไม่ hardcode):**
|
||||
```text
|
||||
Validate this document metadata and respond in JSON:
|
||||
|
||||
Document Number: {{$json.document_number}}
|
||||
Title: {{$json.title}}
|
||||
Expected Pattern: [ORG]-[TYPE]-[SEQ] e.g. "TCC-COR-0001"
|
||||
Category List (MUST match system enum exactly): {{$workflow.variables.system_categories}}
|
||||
|
||||
Respond ONLY with this exact JSON structure:
|
||||
{
|
||||
"is_valid": true | false,
|
||||
"confidence": 0.0 to 1.0,
|
||||
"suggested_category": "<one from Category List>",
|
||||
"detected_issues": ["<issue1>"],
|
||||
"suggested_title": "<corrected title or null>"
|
||||
}
|
||||
```
|
||||
|
||||
**JSON Validation (ตรวจ Category ตรง Enum):**
|
||||
```javascript
|
||||
const systemCategories = $workflow.variables.system_categories;
|
||||
if (!systemCategories.includes(result.suggested_category)) {
|
||||
throw new Error(`Category "${result.suggested_category}" not in system enum: ${systemCategories.join(', ')}`);
|
||||
}
|
||||
```
|
||||
|
||||
#### Node 3.5: Fallback Model Manager
|
||||
|
||||
- อัปเดต `migration_fallback_state` ทุกครั้งที่เกิด Parse Error
|
||||
- Auto-switch ไป `OLLAMA_MODEL_FALLBACK` เมื่อ Error ≥ `FALLBACK_ERROR_THRESHOLD`
|
||||
- ส่ง Alert Email เมื่อ Fallback ถูก Activate
|
||||
|
||||
#### Node 4: Confidence Router (4 outputs)
|
||||
|
||||
| เงื่อนไข | การดำเนินการ |
|
||||
| ------------------------------------------ | -------------------------------- |
|
||||
| `confidence >= 0.85` และ `is_valid = true` | **Output 0** → Auto Ingest |
|
||||
| `confidence >= 0.60` และ `< 0.85` | **Output 1** → Review Queue |
|
||||
| `confidence < 0.60` หรือ `is_valid = false` | **Output 2** → Reject Log |
|
||||
| Parse Error / AI ไม่ตอบ | **Output 3** → Error Log |
|
||||
| Fallback: Error > 5 ใน 10 Request | สลับ Model / หยุด Workflow + Alert |
|
||||
|
||||
**Revision Drift Protection:**
|
||||
```javascript
|
||||
// ถ้า Excel มี revision column — ตรวจสอบก่อน route
|
||||
if (item.json.excel_revision !== undefined) {
|
||||
const expectedRevision = (item.json.current_db_revision || 0) + 1;
|
||||
if (parseInt(item.json.excel_revision) !== expectedRevision) {
|
||||
item.json.review_reason = `Revision drift: Excel=${item.json.excel_revision}, Expected=${expectedRevision}`;
|
||||
reviewQueue.push(item);
|
||||
continue;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### Node 5A: Auto Ingest — Backend API
|
||||
|
||||
> ⚠️ **Storage Enforcement:** n8n ส่งแค่ `source_file_path` — Backend จะ generate UUID, enforce path strategy (`/data/dms/uploads/YYYY/MM/{uuid}.pdf`), และ move file atomically ผ่าน StorageService
|
||||
|
||||
```http
|
||||
POST /api/correspondences/import
|
||||
Authorization: Bearer <MIGRATION_TOKEN>
|
||||
Idempotency-Key: <document_number>:<batch_id>
|
||||
Content-Type: application/json
|
||||
```
|
||||
|
||||
Payload:
|
||||
```json
|
||||
{
|
||||
"document_number": "{{document_number}}",
|
||||
"title": "{{ai_result.suggested_title || title}}",
|
||||
"category": "{{ai_result.suggested_category}}",
|
||||
"source_file_path": "{{file_path}}",
|
||||
"ai_confidence": "{{ai_result.confidence}}",
|
||||
"ai_issues": "{{ai_result.detected_issues}}",
|
||||
"migrated_by": "SYSTEM_IMPORT",
|
||||
"batch_id": "{{$env.MIGRATION_BATCH_ID}}"
|
||||
}
|
||||
```
|
||||
|
||||
**Audit Log ที่ Backend ต้องสร้าง:**
|
||||
```json
|
||||
{
|
||||
"action": "IMPORT",
|
||||
"source": "MIGRATION",
|
||||
"batch_id": "migration_20260226",
|
||||
"created_by": "SYSTEM_IMPORT",
|
||||
"metadata": {
|
||||
"migration": true,
|
||||
"batch_id": "migration_20260226",
|
||||
"ai_confidence": 0.91
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Checkpoint Update (ทุก 10 Records — ผ่าน IF Node + MariaDB Node):**
|
||||
```sql
|
||||
INSERT INTO migration_progress (batch_id, last_processed_index, status)
|
||||
VALUES ('{{$env.MIGRATION_BATCH_ID}}', {{checkpoint_index}}, 'RUNNING')
|
||||
ON DUPLICATE KEY UPDATE
|
||||
last_processed_index = {{checkpoint_index}},
|
||||
updated_at = NOW();
|
||||
```
|
||||
|
||||
#### Node 5B: Review Queue
|
||||
|
||||
> ⚠️ **`migration_review_queue` เป็น Temporary Table เท่านั้น** — ห้ามสร้าง Correspondence record จนกว่า Admin จะ Approve
|
||||
|
||||
Approval Flow:
|
||||
```
|
||||
Review → Admin Approve → POST /api/correspondences/import (เหมือน Auto Ingest)
|
||||
Admin Reject → ลบออกจาก queue ไม่สร้าง record
|
||||
```
|
||||
|
||||
#### Node 5C: Reject Log → `/data/dms/migration_logs/reject_log.csv`
|
||||
|
||||
#### Node 5D: Error Log → `/data/dms/migration_logs/error_log.csv` + MariaDB
|
||||
|
||||
---
|
||||
|
||||
### Phase 4: แผนการทดสอบ (Testing & QA)
|
||||
|
||||
1. **Dry Run:** รันเพียง 1 Batch ข้อมูล (ขนาด ~100 แถว) ทะลวงระบบจากต้นน้ำถึงปลายน้ำ
|
||||
2. **Integrity Check:** QA เช็คในหน้า UI และในฐานข้อมูล MariaDB ว่า Metadata โอนถ่ายได้ครบถ้วน และไฟล์ Physical file ย้ายไปสู่โฟลเดอร์ Webroot (Permanent Storage) ของเอกสาร
|
||||
3. **Hardware Tuning:** จับตาดู RAM ของ NAS และ GPU VRAM/Temperatures ของ Desktop ฝั่ง Ollama. ปรับหน่วง Delay ระหว่าง Request ได้ในตัว n8n หากฮาร์ดแวร์ทำงานหนักเกิน
|
||||
**Dry Run Policy (Mandatory):**
|
||||
- All migrations MUST run with `--dry-run`
|
||||
- No DB commit until validation approved
|
||||
|
||||
**Dry Run Validation (20–50 แถว):**
|
||||
- JSON Parse Success Rate > 95%
|
||||
- Category ที่ AI ตอบตรงกับ System Enum ทุกรายการ
|
||||
- รัน Batch เดิมซ้ำ 2 รอบ → ต้องไม่สร้าง Duplicate หรือ Revision ซ้ำ (Idempotency Test)
|
||||
- Storage Path ตรงตาม Core Storage Spec v1.8.0
|
||||
- Revision Drift ถูก route ไป Review Queue
|
||||
|
||||
**Integrity Check:**
|
||||
```sql
|
||||
-- ตรวจยอด
|
||||
SELECT COUNT(*) FROM correspondences WHERE created_by = 'SYSTEM_IMPORT';
|
||||
|
||||
-- ตรวจ Revision ซ้ำ
|
||||
SELECT document_number, COUNT(*) as cnt
|
||||
FROM correspondences WHERE created_by = 'SYSTEM_IMPORT'
|
||||
GROUP BY document_number HAVING cnt > 1;
|
||||
|
||||
-- ตรวจ Idempotency Key ไม่ซ้ำ
|
||||
SELECT idempotency_key, COUNT(*) as cnt
|
||||
FROM import_transactions GROUP BY idempotency_key HAVING cnt > 1;
|
||||
|
||||
-- ตรวจ Audit Log ครบ
|
||||
SELECT COUNT(*) FROM audit_logs
|
||||
WHERE created_by = 'SYSTEM_IMPORT' AND action = 'IMPORT';
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Phase 5: การรันงานจริง (Execution & Monitoring)
|
||||
|
||||
1. **Scheduling:** เปิดตัวรันอัตโนมัติเฉพาะเวลากลางคืน (Offline hours)
|
||||
2. **Log Monitoring:** เปิด Execution Logs ของ n8n ควบคู่ไปกับ Docker Logs
|
||||
3. **Post-Migration Audit:** ผู้ดูแลระบบสั่งรัน Query สอบยอดหลังบ้าน:
|
||||
```sql
|
||||
SELECT count(*) FROM correspondences WHERE created_by = 'SYSTEM_IMPORT';
|
||||
```
|
||||
- **Scheduling:** รันอัตโนมัติ 22:00–06:00
|
||||
- **Expected Runtime:** ~3 วินาที/record (2 sec delay + ~1 sec inference) → 20,000 records ≈ **60,000 วินาที (~16.6 ชั่วโมง)** → ใช้เวลาประมาณ **3–4 คืน**
|
||||
- **Daily Check:** Admin ตรวจ Review Queue และ Reject Log ทุกเช้าจาก Night Summary Email
|
||||
- **Progress Tracking:** อัปเดต `migration_progress` ทุก 10 Records
|
||||
|
||||
---
|
||||
|
||||
## 4. แผนรับมือความเสี่ยง (Risk Management)
|
||||
## 4. Rollback Plan
|
||||
|
||||
| ลำดับที่ | ความเสี่ยง | การจัดการ (Mitigation) |
|
||||
| ---- | --------------------------------- | -------------------------------------------------------------------------------------------- |
|
||||
| 1 | AI Node หรือ GPU ค้าง (OOT/Timeout) | กำหนดให้ Node มี Node Retry Mechanism (เช่น รอ 1 นาที ทำซ้ำ 3 รอบ) ประกอบกับการจับ Wait node ไว้ดักทิศทาง |
|
||||
| 2 | หมายเลขเอกสารซ้ำซ้อนกันใน Excel | Backend ต้องมี Controller Logic รับมือ และส่งออกเลข Revision เพื่อลงฐานข้อมูล |
|
||||
| 3 | ดิสก์ NAS ทรุด หรือ Database บวมชั่วขณะ | ปิดฟีเจอร์ "Save Successful Executions" ใน Options ของ n8n |
|
||||
**Step 1:** หยุด n8n และ Disable Token
|
||||
```sql
|
||||
UPDATE users SET is_active = false WHERE username = 'migration_bot';
|
||||
```
|
||||
|
||||
**Step 2:** ลบ Records (Transaction)
|
||||
```sql
|
||||
START TRANSACTION;
|
||||
DELETE FROM correspondence_files
|
||||
WHERE correspondence_id IN (SELECT id FROM correspondences WHERE created_by = 'SYSTEM_IMPORT');
|
||||
DELETE FROM correspondences WHERE created_by = 'SYSTEM_IMPORT';
|
||||
DELETE FROM import_transactions WHERE batch_id = 'migration_20260226';
|
||||
SELECT ROW_COUNT();
|
||||
COMMIT;
|
||||
```
|
||||
|
||||
**Step 3:** ย้ายไฟล์กลับ `/data/dms/staging_ai/` ผ่าน Script แยก
|
||||
|
||||
**Step 4:** Reset State
|
||||
```sql
|
||||
UPDATE migration_progress
|
||||
SET status = 'FAILED', last_processed_index = 0
|
||||
WHERE batch_id = 'migration_20260226';
|
||||
|
||||
UPDATE migration_fallback_state
|
||||
SET recent_error_count = 0, is_fallback_active = FALSE
|
||||
WHERE batch_id = 'migration_20260226';
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
> **ข้อแนะนำด้าน Physical Storage:** หลังจากนำเข้าข้อมูลเสร็จ ตรวจสอบให้แน่ใจว่าไฟล์ PDF ทั้ง 20,000 ไฟล์ ถูกย้าย (Move) หรือคัดลอก (Copy) ไปเก็บยัง Local Storage Strategy ตามที่ได้ตกลงระบุไว้ใน Specs ฉบับที่เกี่ยวข้องกับ Storage อย่างถูกต้อง ไม่ปล่อยค้างไว้ที่ Temp Folder
|
||||
## 5. แผนรับมือความเสี่ยง (Risk Management)
|
||||
|
||||
| ลำดับที่ | ความเสี่ยง | การจัดการ (Mitigation) |
|
||||
| ---- | -------------------------- | -------------------------------------------------- |
|
||||
| 1 | AI Node หรือ GPU ค้าง | Timeout 30 วินาที, Retry 3 รอบ, Delay 60 วินาที |
|
||||
| 2 | Ollama ตอบไม่ใช่ JSON | JSON Pre-processor + ส่ง Human Review Queue |
|
||||
| 3 | Category ไม่ตรง System Enum | Fetch `/api/meta/categories` ก่อน Batch ทุกครั้ง |
|
||||
| 4 | Idempotency ซ้ำ | `import_transactions` table + Backend คืน HTTP 200 |
|
||||
| 5 | Revision Drift | ตรวจ Excel revision column → Route ไป Review Queue |
|
||||
| 6 | Storage bypass | ห้าม move file โดยตรง — ผ่าน Backend API เท่านั้น |
|
||||
| 7 | GPU VRAM Overflow | ใช้เฉพาะ Quantized Model (q4_K_M) |
|
||||
| 8 | ดิสก์ NAS เต็ม | ปิด "Save Successful Executions" ใน n8n |
|
||||
| 9 | Migration Token ถูกขโมย | Token 7 วัน, IP Whitelist `<NAS_IP>` เท่านั้น |
|
||||
| 10 | ไฟดับ/ล่มกลางคัน | Checkpoint Table → Resume จากจุดที่ค้าง |
|
||||
|
||||
---
|
||||
|
||||
## 6. Post-Migration Verification
|
||||
|
||||
```sql
|
||||
-- 1. ตรวจยอดครบ 20,000
|
||||
SELECT COUNT(*) FROM correspondences WHERE created_by = 'SYSTEM_IMPORT';
|
||||
|
||||
-- 2. ตรวจ Revision ผิดปกติ
|
||||
SELECT document_number, COUNT(*) FROM correspondences
|
||||
WHERE created_by = 'SYSTEM_IMPORT'
|
||||
GROUP BY document_number HAVING COUNT(*) > 5;
|
||||
|
||||
-- 3. ตรวจ Audit Log ครบ
|
||||
SELECT COUNT(*) FROM audit_logs
|
||||
WHERE created_by = 'SYSTEM_IMPORT' AND action = 'IMPORT';
|
||||
|
||||
-- 4. ตรวจ Idempotency ไม่มีซ้ำ
|
||||
SELECT idempotency_key, COUNT(*) FROM import_transactions
|
||||
GROUP BY idempotency_key HAVING COUNT(*) > 1;
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
> **ข้อแนะนำด้าน Physical Storage:** ไฟล์ PDF ทั้ง 20,000 ไฟล์จะถูก move โดย Backend StorageService ไปยัง path ที่ถูกต้องโดยอัตโนมัติ ไม่ปล่อยค้างไว้ที่ `/data/dms/staging_ai/`
|
||||
|
||||
918
specs/03-Data-and-Storage/03-05-n8n-migration-setup-guide.md
Normal file
918
specs/03-Data-and-Storage/03-05-n8n-migration-setup-guide.md
Normal file
@@ -0,0 +1,918 @@
|
||||
# 📋 คู่มือการตั้งค่า n8n สำหรับ Legacy Data Migration
|
||||
|
||||
เอกสารนี้จัดทำขึ้นเพื่อรองรับการ Migration เอกสาร PDF 20,000 ฉบับ ตามแผนใน `03-04-legacy-data-migration.md` และ `ADR-017-ollama-data-migration.md`
|
||||
|
||||
> **Note:** Category Enum system-driven, Idempotency-Key Header, Storage Enforcement, Audit Log, Encoding Normalization, Security Hardening, Nginx Rate Limit, Docker Hardening, AI Physical Isolation (ASUSTOR), Folder Standard (/data/dms)
|
||||
|
||||
---
|
||||
|
||||
## 📌 ส่วนที่ 1: การติดตั้งและตั้งค่าเบื้องต้น
|
||||
|
||||
### 1.1 ติดตั้ง n8n บน ASUSTOR NAS (Docker)
|
||||
|
||||
```bash
|
||||
mkdir -p /data/dms/n8n
|
||||
cd /data/dms/n8n
|
||||
|
||||
cat > docker-compose.yml << 'EOF'
|
||||
version: '3.8'
|
||||
|
||||
services:
|
||||
n8n:
|
||||
image: n8nio/n8n:latest
|
||||
container_name: n8n-migration
|
||||
restart: unless-stopped
|
||||
# Docker Hardening (Patch)
|
||||
mem_limit: 2g
|
||||
logging:
|
||||
driver: json-file
|
||||
options:
|
||||
max-size: "10m"
|
||||
max-file: "3"
|
||||
ports:
|
||||
- "5678:5678"
|
||||
environment:
|
||||
- N8N_HOST=0.0.0.0
|
||||
- N8N_PORT=5678
|
||||
- N8N_PROTOCOL=http
|
||||
- NODE_ENV=production
|
||||
- WEBHOOK_URL=http://<NAS_IP>:5678/
|
||||
- GENERIC_TIMEZONE=Asia/Bangkok
|
||||
- TZ=Asia/Bangkok
|
||||
- N8N_SECURE_COOKIE=false
|
||||
- N8N_USER_FOLDER=/home/node/.n8n
|
||||
- N8N_PUBLIC_API_DISABLED=true
|
||||
- N8N_BASIC_AUTH_ACTIVE=true
|
||||
- N8N_BASIC_AUTH_USER=admin
|
||||
- N8N_BASIC_AUTH_PASSWORD=<strong_password>
|
||||
- N8N_PAYLOAD_SIZE_MAX=10485760
|
||||
- EXECUTIONS_DATA_PRUNE=true
|
||||
- EXECUTIONS_DATA_MAX_AGE=168
|
||||
- EXECUTIONS_DATA_PRUNE_TIMEOUT=60
|
||||
- DB_TYPE=postgresdb
|
||||
- DB_POSTGRESDB_HOST=<DB_IP>
|
||||
- DB_POSTGRESDB_PORT=5432
|
||||
- DB_POSTGRESDB_DATABASE=n8n
|
||||
- DB_POSTGRESDB_USER=n8n
|
||||
- DB_POSTGRESDB_PASSWORD=<password>
|
||||
volumes:
|
||||
- ./n8n_data:/home/node/.n8n
|
||||
# read-only: อ่านไฟล์ PDF ต้นฉบับเท่านั้น
|
||||
- /data/dms/staging_ai:/data/dms/staging_ai:ro
|
||||
# read-write: เขียน Log และ CSV ทั้งหมด
|
||||
- /data/dms/migration_logs:/data/dms/migration_logs:rw
|
||||
networks:
|
||||
- n8n-network
|
||||
|
||||
networks:
|
||||
n8n-network:
|
||||
driver: bridge
|
||||
EOF
|
||||
|
||||
docker-compose up -d
|
||||
```
|
||||
|
||||
> ⚠️ **Volume หมายเหตุ:** `/data/dms/staging_ai` = **read-only** (อ่านไฟล์ต้นฉบับ) และ `/data/dms/migration_logs` = **read-write** (เขียน Log/CSV) — ห้ามเขียน CSV ลง `staging_ai` เพราะจะ Error ทันที
|
||||
|
||||
### 1.2 Nginx Rate Limit
|
||||
|
||||
เพิ่มใน Nginx config สำหรับ Migration API:
|
||||
|
||||
```nginx
|
||||
# nginx.conf หรือ site config
|
||||
limit_req_zone $binary_remote_addr zone=migration:10m rate=1r/s;
|
||||
|
||||
location /api/correspondences/import {
|
||||
limit_req zone=migration burst=5 nodelay;
|
||||
proxy_pass http://backend:3001;
|
||||
}
|
||||
```
|
||||
|
||||
### 1.3 Environment Variables
|
||||
|
||||
**Settings → Environment Variables ใน n8n UI:**
|
||||
|
||||
| Variable | ค่าที่แนะนำ | คำอธิบาย |
|
||||
| --------------------------- | ---------------------------- | ------------------------------------ |
|
||||
| `OLLAMA_HOST` | `http://<ASUSTOR_IP>:11434` | URL ของ Ollama (ใน internal network) |
|
||||
| `OLLAMA_MODEL_PRIMARY` | `llama3.2:3b` | Model หลัก |
|
||||
| `OLLAMA_MODEL_FALLBACK` | `mistral:7b-instruct-q4_K_M` | Model สำรอง |
|
||||
| `MIGRATION_BATCH_SIZE` | `10` | จำนวน Record ต่อ Batch |
|
||||
| `MIGRATION_DELAY_MS` | `2000` | Delay ระหว่าง Request (ms) |
|
||||
| `CONFIDENCE_THRESHOLD_HIGH` | `0.85` | Threshold Auto Ingest |
|
||||
| `CONFIDENCE_THRESHOLD_LOW` | `0.60` | Threshold Review Queue |
|
||||
| `MAX_RETRY_COUNT` | `3` | จำนวนครั้ง Retry |
|
||||
| `FALLBACK_ERROR_THRESHOLD` | `5` | Error ที่ trigger Fallback |
|
||||
| `BACKEND_URL` | `https://<BACKEND_URL>` | URL ของ LCBP3 Backend |
|
||||
| `MIGRATION_BATCH_ID` | `migration_20260226` | ID ของ Batch |
|
||||
|
||||
---
|
||||
|
||||
## 📌 ส่วนที่ 2: การเตรียม Database
|
||||
|
||||
รัน SQL นี้บน MariaDB **ก่อน** เริ่ม n8n Workflow:
|
||||
|
||||
```sql
|
||||
-- Checkpoint
|
||||
CREATE TABLE IF NOT EXISTS migration_progress (
|
||||
batch_id VARCHAR(50) PRIMARY KEY,
|
||||
last_processed_index INT DEFAULT 0,
|
||||
status ENUM('RUNNING','COMPLETED','FAILED') DEFAULT 'RUNNING',
|
||||
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP
|
||||
);
|
||||
|
||||
-- Review Queue (Temporary — ไม่ใช่ Business Schema)
|
||||
CREATE TABLE IF NOT EXISTS migration_review_queue (
|
||||
id INT AUTO_INCREMENT PRIMARY KEY,
|
||||
document_number VARCHAR(100) NOT NULL,
|
||||
title TEXT,
|
||||
original_title TEXT,
|
||||
ai_suggested_category VARCHAR(50),
|
||||
ai_confidence DECIMAL(4,3),
|
||||
ai_issues JSON,
|
||||
review_reason VARCHAR(255),
|
||||
status ENUM('PENDING','APPROVED','REJECTED') DEFAULT 'PENDING',
|
||||
reviewed_by VARCHAR(100),
|
||||
reviewed_at TIMESTAMP NULL,
|
||||
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||
UNIQUE KEY uq_doc_number (document_number)
|
||||
);
|
||||
|
||||
-- Error Log
|
||||
CREATE TABLE IF NOT EXISTS migration_errors (
|
||||
id INT AUTO_INCREMENT PRIMARY KEY,
|
||||
batch_id VARCHAR(50),
|
||||
document_number VARCHAR(100),
|
||||
error_type ENUM('FILE_NOT_FOUND','AI_PARSE_ERROR','API_ERROR','DB_ERROR','UNKNOWN'),
|
||||
error_message TEXT,
|
||||
raw_ai_response TEXT,
|
||||
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||
INDEX idx_batch_id (batch_id),
|
||||
INDEX idx_error_type (error_type)
|
||||
);
|
||||
|
||||
-- Fallback State
|
||||
CREATE TABLE IF NOT EXISTS migration_fallback_state (
|
||||
id INT AUTO_INCREMENT PRIMARY KEY,
|
||||
batch_id VARCHAR(50) UNIQUE,
|
||||
recent_error_count INT DEFAULT 0,
|
||||
is_fallback_active BOOLEAN DEFAULT FALSE,
|
||||
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP
|
||||
);
|
||||
|
||||
-- Idempotency (Patch)
|
||||
CREATE TABLE IF NOT EXISTS import_transactions (
|
||||
id INT AUTO_INCREMENT PRIMARY KEY,
|
||||
idempotency_key VARCHAR(255) UNIQUE NOT NULL,
|
||||
document_number VARCHAR(100),
|
||||
batch_id VARCHAR(100),
|
||||
status_code INT DEFAULT 201,
|
||||
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||
INDEX idx_idem_key (idempotency_key)
|
||||
);
|
||||
|
||||
-- Daily Summary
|
||||
CREATE TABLE IF NOT EXISTS migration_daily_summary (
|
||||
id INT AUTO_INCREMENT PRIMARY KEY,
|
||||
batch_id VARCHAR(50),
|
||||
summary_date DATE,
|
||||
total_processed INT DEFAULT 0,
|
||||
auto_ingested INT DEFAULT 0,
|
||||
sent_to_review INT DEFAULT 0,
|
||||
rejected INT DEFAULT 0,
|
||||
errors INT DEFAULT 0,
|
||||
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||
UNIQUE KEY uq_batch_date (batch_id, summary_date)
|
||||
);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📌 ส่วนที่ 3: Credentials
|
||||
|
||||
**Credentials → Add New:**
|
||||
|
||||
#### 🔐 Ollama API
|
||||
| Field | ค่า |
|
||||
| -------------- | --------------------------- |
|
||||
| Name | `Ollama Local API` |
|
||||
| Type | `HTTP Request` |
|
||||
| Base URL | `http://<ASUSTOR_IP>:11434` |
|
||||
| Authentication | `None` |
|
||||
|
||||
#### 🔐 LCBP3 Backend API
|
||||
| Field | ค่า |
|
||||
| -------------- | --------------------------- |
|
||||
| Name | `LCBP3 Migration Token` |
|
||||
| Type | `HTTP Request` |
|
||||
| Base URL | `https://<BACKEND_URL>/api` |
|
||||
| Authentication | `Header Auth` |
|
||||
| Header Name | `Authorization` |
|
||||
| Header Value | `Bearer <MIGRATION_TOKEN>` |
|
||||
|
||||
#### 🔐 MariaDB
|
||||
| Field | ค่า |
|
||||
| -------- | ------------------ |
|
||||
| Name | `LCBP3 MariaDB` |
|
||||
| Type | `MariaDB` |
|
||||
| Host | `<DB_IP>` |
|
||||
| Port | `3306` |
|
||||
| Database | `lcbp3_production` |
|
||||
| User | `migration_bot` |
|
||||
| Password | `<password>` |
|
||||
|
||||
---
|
||||
|
||||
## 📌 ส่วนที่ 4: Workflow (Step-by-Step)
|
||||
|
||||
### 4.1 โครงสร้างภาพรวม
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────────────────────────────────────┐
|
||||
│ MIGRATION WORKFLOW v1.8.0 │
|
||||
├──────────────────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐ │
|
||||
│ │ Node 0 │──▶│ Node 1 │──▶│ Node 2 │──▶│ Node 3 │ │
|
||||
│ │Pre- │ │ Data │ │ File │ │ AI │ │
|
||||
│ │flight + │ │ Reader │ │ Validat.│ │Analysis │ │
|
||||
│ │Fetch Cat│ │+Encoding│ │+Sanitize│ │+Enum Chk│ │
|
||||
│ └─────────┘ └─────────┘ └──┬──┬───┘ └────┬────┘ │
|
||||
│ │ │ │ │
|
||||
│ valid │ │ error ┌───▼──────────────┐ │
|
||||
│ │ └──────▶ │ Node 3.5 │ │
|
||||
│ │ │ Fallback Manager │ │
|
||||
│ │ └──────────────────┘ │
|
||||
│ ▼ │ │
|
||||
│ ┌─────────┐ ┌────▼────┐ │
|
||||
│ │ Node 5D │ │ Node 4 │ │
|
||||
│ │ Error │ │Confidence│ │
|
||||
│ │ Log │ │+Revision │ │
|
||||
│ └─────────┘ │ Drift │ │
|
||||
│ └┬──┬──┬──┘ │
|
||||
│ │ │ │ │
|
||||
│ ┌────────────┘ │ └──────┐ │
|
||||
│ ▼ ▼ ▼ │
|
||||
│ ┌──────────┐ ┌──────────┐ ┌──────┐ │
|
||||
│ │ Node 5A │ │ Node 5B │ │ 5C │ │
|
||||
│ │ Auto │ │ Review │ │Reject│ │
|
||||
│ │ Ingest │ │ Queue │ │ Log │ │
|
||||
│ │+Idempot. │ │(Temp only│ └──────┘ │
|
||||
│ └────┬─────┘ └──────────┘ │
|
||||
│ │ │
|
||||
│ ┌────▼──────┐ │
|
||||
│ │ Checkpoint│ │
|
||||
│ └───────────┘ │
|
||||
└──────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 4.2 Node 0: Pre-flight + Fetch System Categories
|
||||
|
||||
Fetch System Categories ก่อน Batch ทุกครั้ง
|
||||
|
||||
**Sub-flow:**
|
||||
```
|
||||
[Trigger] → [HTTP: Ollama /api/tags] → [MariaDB: SELECT 1]
|
||||
→ [HTTP: Backend /health] → [Code: File Mount Check]
|
||||
→ [HTTP: GET /api/meta/categories] → [Store in Workflow Variable]
|
||||
→ [IF all pass → Node 1] [ELSE → Stop + Alert]
|
||||
```
|
||||
|
||||
**HTTP Node — Fetch Categories:**
|
||||
```json
|
||||
{
|
||||
"method": "GET",
|
||||
"url": "={{ $env.BACKEND_URL }}/api/meta/categories",
|
||||
"authentication": "genericCredentialType",
|
||||
"genericAuthType": "lcbp3MigrationToken"
|
||||
}
|
||||
```
|
||||
|
||||
**Code Node — Store Categories + File Mount Check:**
|
||||
```javascript
|
||||
const fs = require('fs');
|
||||
|
||||
// เก็บ categories ใน Workflow Variable
|
||||
const categories = $input.first().json.categories;
|
||||
if (!categories || !Array.isArray(categories) || categories.length === 0) {
|
||||
throw new Error('Failed to fetch system categories from backend');
|
||||
}
|
||||
|
||||
// Set Workflow Variable เพื่อใช้ใน Node 3
|
||||
$workflow.variables = $workflow.variables || {};
|
||||
$workflow.variables.system_categories = categories;
|
||||
|
||||
// ตรวจ File Mount
|
||||
try {
|
||||
const files = fs.readdirSync('/data/dms/staging_ai');
|
||||
if (files.length === 0) throw new Error('staging_ai is empty');
|
||||
fs.writeFileSync('/data/dms/migration_logs/.preflight_ok', new Date().toISOString());
|
||||
} catch (err) {
|
||||
throw new Error(`File mount check failed: ${err.message}`);
|
||||
}
|
||||
|
||||
return [{ json: { preflight_ok: true, system_categories: categories } }];
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 4.3 Node 1: Load Checkpoint + Read Excel + Encoding Normalization
|
||||
|
||||
**Step 1 — MariaDB Node (Read Checkpoint):**
|
||||
```sql
|
||||
SELECT last_processed_index, status
|
||||
FROM migration_progress
|
||||
WHERE batch_id = '{{ $env.MIGRATION_BATCH_ID }}'
|
||||
LIMIT 1;
|
||||
```
|
||||
|
||||
**Step 2 — Spreadsheet File Node:**
|
||||
```json
|
||||
{ "operation": "toData", "binaryProperty": "data", "options": { "sheetName": "Sheet1" } }
|
||||
```
|
||||
|
||||
**Step 3 — Code Node (Checkpoint + Batch + Encoding):**
|
||||
```javascript
|
||||
const checkpointResult = $('Read Checkpoint').first();
|
||||
let startIndex = 0;
|
||||
if (checkpointResult && checkpointResult.json.status === 'RUNNING') {
|
||||
startIndex = checkpointResult.json.last_processed_index || 0;
|
||||
}
|
||||
|
||||
const allItems = $('Read Excel').all();
|
||||
const remaining = allItems.slice(startIndex);
|
||||
const batchSize = parseInt($env.MIGRATION_BATCH_SIZE) || 10;
|
||||
const currentBatch = remaining.slice(0, batchSize);
|
||||
|
||||
// Encoding Normalization: Excel → UTF-8 NFC (Patch)
|
||||
const normalize = (str) => {
|
||||
if (!str) return '';
|
||||
return Buffer.from(String(str), 'utf8').toString('utf8').normalize('NFC');
|
||||
};
|
||||
|
||||
return currentBatch.map((item, i) => ({
|
||||
...item,
|
||||
json: {
|
||||
...item.json,
|
||||
document_number: normalize(item.json.document_number),
|
||||
title: normalize(item.json.title),
|
||||
original_index: startIndex + i
|
||||
}
|
||||
}));
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 4.4 Node 2: File Validator & Sanitizer
|
||||
|
||||
**Node Type:** `Code` — **2 Outputs**
|
||||
|
||||
```javascript
|
||||
const fs = require('fs');
|
||||
const path = require('path');
|
||||
const items = $input.all();
|
||||
const validatedItems = [];
|
||||
const errorItems = [];
|
||||
|
||||
for (const item of items) {
|
||||
const docNumber = item.json.document_number;
|
||||
// Sanitize + Normalize Filename (Patch)
|
||||
const safeName = path.basename(
|
||||
String(docNumber).replace(/[^a-zA-Z0-9\-_.]/g, '_')
|
||||
).normalize('NFC');
|
||||
const filePath = path.resolve('/data/dms/staging_ai', `${safeName}.pdf`);
|
||||
|
||||
if (!filePath.startsWith('/data/dms/staging_ai/')) {
|
||||
errorItems.push({ ...item, json: { ...item.json, error: 'Path traversal detected', error_type: 'FILE_NOT_FOUND' } });
|
||||
continue;
|
||||
}
|
||||
|
||||
try {
|
||||
if (fs.existsSync(filePath)) {
|
||||
const stats = fs.statSync(filePath);
|
||||
validatedItems.push({ ...item, json: { ...item.json, file_exists: true, file_size: stats.size, file_path: filePath } });
|
||||
} else {
|
||||
errorItems.push({ ...item, json: { ...item.json, error: `File not found: ${filePath}`, error_type: 'FILE_NOT_FOUND', file_exists: false } });
|
||||
}
|
||||
} catch (err) {
|
||||
errorItems.push({ ...item, json: { ...item.json, error: err.message, error_type: 'FILE_NOT_FOUND', file_exists: false } });
|
||||
}
|
||||
}
|
||||
|
||||
// Output 0 → Node 3 | Output 1 → Node 5D
|
||||
return [validatedItems, errorItems];
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 4.5 Node 3: Build Prompt + AI Analysis
|
||||
|
||||
**Step 1 — MariaDB (Read Fallback State):**
|
||||
```sql
|
||||
SELECT is_fallback_active FROM migration_fallback_state
|
||||
WHERE batch_id = '{{ $env.MIGRATION_BATCH_ID }}' LIMIT 1;
|
||||
```
|
||||
|
||||
**Step 2 — Code Node (Build Prompt: inject system_categories):**
|
||||
```javascript
|
||||
const fallbackState = $('Read Fallback State').first();
|
||||
const isFallback = fallbackState?.json?.is_fallback_active || false;
|
||||
const model = isFallback ? $env.OLLAMA_MODEL_FALLBACK : $env.OLLAMA_MODEL_PRIMARY;
|
||||
|
||||
// ใช้ system_categories จาก Workflow Variable (ไม่ hardcode)
|
||||
const systemCategories = $workflow.variables?.system_categories
|
||||
|| ['Correspondence','RFA','Drawing','Transmittal','Report','Other'];
|
||||
|
||||
const item = $input.first();
|
||||
|
||||
const systemPrompt = `You are a Document Controller for a large construction project.
|
||||
Your task is to validate document metadata.
|
||||
You MUST respond ONLY with valid JSON. No explanation, no markdown, no extra text.
|
||||
If there are no issues, "detected_issues" must be an empty array [].`;
|
||||
|
||||
const userPrompt = `Validate this document metadata and respond in JSON:
|
||||
|
||||
Document Number: ${item.json.document_number}
|
||||
Title: ${item.json.title}
|
||||
Expected Pattern: [ORG]-[TYPE]-[SEQ] e.g. "TCC-COR-0001"
|
||||
Category List (MUST match system enum exactly): ${JSON.stringify(systemCategories)}
|
||||
|
||||
Respond ONLY with this exact JSON structure:
|
||||
{
|
||||
"is_valid": true | false,
|
||||
"confidence": 0.0 to 1.0,
|
||||
"suggested_category": "<one from Category List>",
|
||||
"detected_issues": ["<issue1>"],
|
||||
"suggested_title": "<corrected title or null>"
|
||||
}`;
|
||||
|
||||
return [{
|
||||
json: {
|
||||
...item.json,
|
||||
active_model: model,
|
||||
system_categories: systemCategories,
|
||||
ollama_payload: { model, prompt: `${systemPrompt}\n\n${userPrompt}`, stream: false, format: 'json' }
|
||||
}
|
||||
}];
|
||||
```
|
||||
|
||||
**Step 3 — HTTP Request Node (Ollama):**
|
||||
```json
|
||||
{
|
||||
"method": "POST",
|
||||
"url": "={{ $env.OLLAMA_HOST }}/api/generate",
|
||||
"sendBody": true,
|
||||
"specifyBody": "json",
|
||||
"jsonBody": "={{ $json.ollama_payload }}",
|
||||
"options": { "timeout": 30000, "retry": { "count": 3, "delay": 2000, "backoff": "exponential" } }
|
||||
}
|
||||
```
|
||||
|
||||
**Step 4 — Code Node (Parse + Validate: Enum check):**
|
||||
```javascript
|
||||
const items = $input.all();
|
||||
const parsed = [];
|
||||
const parseErrors = [];
|
||||
|
||||
for (const item of items) {
|
||||
try {
|
||||
let raw = item.json.response || '';
|
||||
raw = raw.replace(/```json/gi, '').replace(/```/g, '').trim();
|
||||
const result = JSON.parse(raw);
|
||||
|
||||
// Strict Schema Validation
|
||||
if (typeof result.is_valid !== 'boolean')
|
||||
throw new Error('is_valid must be boolean');
|
||||
if (typeof result.confidence !== 'number' || result.confidence < 0 || result.confidence > 1)
|
||||
throw new Error('confidence must be float 0.0–1.0');
|
||||
if (!Array.isArray(result.detected_issues))
|
||||
throw new Error('detected_issues must be array');
|
||||
|
||||
// Enum Validation ตรง System Categories (Patch)
|
||||
const systemCategories = item.json.system_categories || [];
|
||||
if (!systemCategories.includes(result.suggested_category))
|
||||
throw new Error(`Category "${result.suggested_category}" not in system enum: [${systemCategories.join(', ')}]`);
|
||||
|
||||
parsed.push({ ...item, json: { ...item.json, ai_result: result, parse_error: null } });
|
||||
} catch (err) {
|
||||
parseErrors.push({
|
||||
...item,
|
||||
json: { ...item.json, ai_result: null, parse_error: err.message, raw_ai_response: item.json.response, error_type: 'AI_PARSE_ERROR' }
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
// Output 0 → Node 4 | Output 1 → Node 3.5 + Node 5D
|
||||
return [parsed, parseErrors];
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 4.6 Node 3.5: Fallback Model Manager
|
||||
|
||||
**MariaDB Node:**
|
||||
```sql
|
||||
INSERT INTO migration_fallback_state (batch_id, recent_error_count, is_fallback_active)
|
||||
VALUES ('{{ $env.MIGRATION_BATCH_ID }}', 1, FALSE)
|
||||
ON DUPLICATE KEY UPDATE
|
||||
recent_error_count = recent_error_count + 1,
|
||||
is_fallback_active = CASE
|
||||
WHEN recent_error_count + 1 >= {{ $env.FALLBACK_ERROR_THRESHOLD }} THEN TRUE
|
||||
ELSE is_fallback_active
|
||||
END,
|
||||
updated_at = NOW();
|
||||
```
|
||||
|
||||
**Code Node (Alert):**
|
||||
```javascript
|
||||
const state = $input.first().json;
|
||||
if (state.is_fallback_active) {
|
||||
return [{ json: {
|
||||
...state, alert: true,
|
||||
alert_message: `⚠️ Fallback model (${$env.OLLAMA_MODEL_FALLBACK}) activated after ${state.recent_error_count} errors`
|
||||
}}];
|
||||
}
|
||||
return [{ json: { ...state, alert: false } }];
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 4.7 Node 4: Confidence Router + Revision Drift Protection
|
||||
|
||||
**Node Type:** `Code` — **4 Outputs**
|
||||
|
||||
```javascript
|
||||
const items = $input.all();
|
||||
const autoIngest = [];
|
||||
const reviewQueue = [];
|
||||
const rejectLog = [];
|
||||
const errorLog = [];
|
||||
|
||||
const HIGH = parseFloat($env.CONFIDENCE_THRESHOLD_HIGH) || 0.85;
|
||||
const LOW = parseFloat($env.CONFIDENCE_THRESHOLD_LOW) || 0.60;
|
||||
|
||||
for (const item of items) {
|
||||
if (item.json.parse_error || !item.json.ai_result) {
|
||||
errorLog.push(item); continue;
|
||||
}
|
||||
|
||||
// Revision Drift Protection
|
||||
if (item.json.excel_revision !== undefined) {
|
||||
const expectedRev = (item.json.current_db_revision || 0) + 1;
|
||||
if (parseInt(item.json.excel_revision) !== expectedRev) {
|
||||
reviewQueue.push({
|
||||
...item,
|
||||
json: { ...item.json, review_reason: `Revision drift: Excel=${item.json.excel_revision}, Expected=${expectedRev}` }
|
||||
});
|
||||
continue;
|
||||
}
|
||||
}
|
||||
|
||||
const ai = item.json.ai_result;
|
||||
if (ai.confidence >= HIGH && ai.is_valid === true) {
|
||||
autoIngest.push(item);
|
||||
} else if (ai.confidence >= LOW) {
|
||||
reviewQueue.push({ ...item, json: { ...item.json, review_reason: `Confidence ${ai.confidence.toFixed(2)} < ${HIGH}` } });
|
||||
} else {
|
||||
rejectLog.push({
|
||||
...item,
|
||||
json: { ...item.json, reject_reason: ai.is_valid === false ? 'AI marked invalid' : `Confidence ${ai.confidence.toFixed(2)} < ${LOW}` }
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
// Output 0: Auto Ingest | 1: Review Queue | 2: Reject Log | 3: Error Log
|
||||
return [autoIngest, reviewQueue, rejectLog, errorLog];
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 4.8 Node 5A: Auto Ingest + Idempotency + Checkpoint
|
||||
|
||||
**HTTP Request Node (Patch — Idempotency-Key Header + source_file_path):**
|
||||
```json
|
||||
{
|
||||
"method": "POST",
|
||||
"url": "={{ $env.BACKEND_URL }}/api/correspondences/import",
|
||||
"authentication": "genericCredentialType",
|
||||
"genericAuthType": "lcbp3MigrationToken",
|
||||
"sendHeaders": true,
|
||||
"headers": {
|
||||
"Idempotency-Key": "={{ $json.document_number }}:{{ $env.MIGRATION_BATCH_ID }}"
|
||||
},
|
||||
"sendBody": true,
|
||||
"specifyBody": "json",
|
||||
"jsonBody": {
|
||||
"document_number": "={{ $json.document_number }}",
|
||||
"title": "={{ $json.ai_result.suggested_title || $json.title }}",
|
||||
"category": "={{ $json.ai_result.suggested_category }}",
|
||||
"source_file_path": "={{ $json.file_path }}",
|
||||
"ai_confidence": "={{ $json.ai_result.confidence }}",
|
||||
"ai_issues": "={{ $json.ai_result.detected_issues }}",
|
||||
"migrated_by": "SYSTEM_IMPORT",
|
||||
"batch_id": "={{ $env.MIGRATION_BATCH_ID }}"
|
||||
},
|
||||
"options": { "timeout": 30000, "retry": { "count": 3, "delay": 5000 } }
|
||||
}
|
||||
```
|
||||
|
||||
> Backend จะ generate UUID, enforce Storage path `/storage/{project}/{category}/{year}/{month}/{uuid}.pdf`, move file ผ่าน StorageService และบันทึก Audit Log `action=IMPORT, source=MIGRATION`
|
||||
|
||||
**Checkpoint Code Node (ทุก 10 Records):**
|
||||
```javascript
|
||||
const item = $input.first();
|
||||
return [{ json: {
|
||||
...item.json,
|
||||
should_update_checkpoint: item.json.original_index % 10 === 0,
|
||||
checkpoint_index: item.json.original_index
|
||||
}}];
|
||||
```
|
||||
|
||||
**IF Node → MariaDB Checkpoint:**
|
||||
```sql
|
||||
INSERT INTO migration_progress (batch_id, last_processed_index, status)
|
||||
VALUES ('{{ $env.MIGRATION_BATCH_ID }}', {{ $json.checkpoint_index }}, 'RUNNING')
|
||||
ON DUPLICATE KEY UPDATE last_processed_index = {{ $json.checkpoint_index }}, updated_at = NOW();
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 4.9 Node 5B: Review Queue (Temporary Table)
|
||||
|
||||
> ⚠️ ห้ามสร้าง Correspondence record — รอ Admin Approve แล้วค่อย POST `/api/correspondences/import`
|
||||
|
||||
```sql
|
||||
INSERT INTO migration_review_queue
|
||||
(document_number, title, original_title, ai_suggested_category,
|
||||
ai_confidence, ai_issues, review_reason, status, created_at)
|
||||
VALUES (
|
||||
'{{ $json.document_number }}',
|
||||
'{{ $json.ai_result.suggested_title || $json.title }}',
|
||||
'{{ $json.title }}',
|
||||
'{{ $json.ai_result.suggested_category }}',
|
||||
{{ $json.ai_result.confidence }},
|
||||
'{{ JSON.stringify($json.ai_result.detected_issues) }}',
|
||||
'{{ $json.review_reason }}',
|
||||
'PENDING', NOW()
|
||||
)
|
||||
ON DUPLICATE KEY UPDATE status = 'PENDING', review_reason = '{{ $json.review_reason }}';
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 4.10 Node 5C: Reject Log → `/data/migration_logs/`
|
||||
|
||||
```javascript
|
||||
const fs = require('fs');
|
||||
const item = $input.first();
|
||||
const csvPath = '/data/dms/migration_logs/reject_log.csv';
|
||||
const header = 'timestamp,document_number,title,reject_reason,ai_confidence,ai_issues\n';
|
||||
const esc = (s) => `"${String(s||'').replace(/"/g,'""')}"`;
|
||||
|
||||
if (!fs.existsSync(csvPath)) fs.writeFileSync(csvPath, header, 'utf8');
|
||||
|
||||
const line = [
|
||||
new Date().toISOString(),
|
||||
esc(item.json.document_number), esc(item.json.title),
|
||||
esc(item.json.reject_reason),
|
||||
item.json.ai_result?.confidence ?? 'N/A',
|
||||
esc(JSON.stringify(item.json.ai_result?.detected_issues || []))
|
||||
].join(',') + '\n';
|
||||
|
||||
fs.appendFileSync(csvPath, line, 'utf8');
|
||||
return [$input.first()];
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 4.11 Node 5D: Error Log → `/data/migration_logs/` + MariaDB
|
||||
|
||||
```javascript
|
||||
const fs = require('fs');
|
||||
const item = $input.first();
|
||||
const csvPath = '/data/dms/migration_logs/error_log.csv';
|
||||
const header = 'timestamp,document_number,error_type,error_message,raw_ai_response\n';
|
||||
const esc = (s) => `"${String(s||'').replace(/"/g,'""')}"`;
|
||||
|
||||
if (!fs.existsSync(csvPath)) fs.writeFileSync(csvPath, header, 'utf8');
|
||||
|
||||
const line = [
|
||||
new Date().toISOString(),
|
||||
esc(item.json.document_number),
|
||||
esc(item.json.error_type || 'UNKNOWN'),
|
||||
esc(item.json.error || item.json.parse_error),
|
||||
esc(item.json.raw_ai_response || '')
|
||||
].join(',') + '\n';
|
||||
|
||||
fs.appendFileSync(csvPath, line, 'utf8');
|
||||
return [$input.first()];
|
||||
```
|
||||
|
||||
**MariaDB Node:**
|
||||
```sql
|
||||
INSERT INTO migration_errors
|
||||
(batch_id, document_number, error_type, error_message, raw_ai_response, created_at)
|
||||
VALUES (
|
||||
'{{ $env.MIGRATION_BATCH_ID }}', '{{ $json.document_number }}',
|
||||
'{{ $json.error_type || "UNKNOWN" }}', '{{ $json.error || $json.parse_error }}',
|
||||
'{{ $json.raw_ai_response || "" }}', NOW()
|
||||
);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📌 ส่วนที่ 5: Rollback Workflow
|
||||
|
||||
**Workflow: `Migration Rollback`** — Manual Trigger เท่านั้น
|
||||
|
||||
```
|
||||
[Manual Trigger: {confirmation: "CONFIRM_ROLLBACK"}]
|
||||
│
|
||||
▼
|
||||
[Code: Guard — ต้องพิมพ์ "CONFIRM_ROLLBACK"]
|
||||
│ PASS
|
||||
▼
|
||||
[MariaDB: Disable Token]
|
||||
UPDATE users SET is_active = false WHERE username = 'migration_bot';
|
||||
│
|
||||
▼
|
||||
[MariaDB: Delete File Records]
|
||||
DELETE FROM correspondence_files WHERE correspondence_id IN
|
||||
(SELECT id FROM correspondences WHERE created_by = 'SYSTEM_IMPORT');
|
||||
│
|
||||
▼
|
||||
[MariaDB: Delete Correspondence Records]
|
||||
DELETE FROM correspondences WHERE created_by = 'SYSTEM_IMPORT';
|
||||
│
|
||||
▼
|
||||
[MariaDB: Clear Idempotency Records]
|
||||
DELETE FROM import_transactions WHERE batch_id = '{{$env.MIGRATION_BATCH_ID}}';
|
||||
│
|
||||
▼
|
||||
[MariaDB: Reset Checkpoint + Fallback State]
|
||||
│
|
||||
▼
|
||||
[Email: Rollback Report → Admin]
|
||||
```
|
||||
|
||||
**Confirmation Guard:**
|
||||
```javascript
|
||||
if ($input.first().json.confirmation !== 'CONFIRM_ROLLBACK') {
|
||||
throw new Error('Rollback cancelled: type "CONFIRM_ROLLBACK" to proceed.');
|
||||
}
|
||||
return $input.all();
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📌 ส่วนที่ 6: End-of-Night Summary (06:30 ทุกวัน)
|
||||
|
||||
**MariaDB:**
|
||||
```sql
|
||||
SELECT
|
||||
mp.last_processed_index AS total_progress,
|
||||
(SELECT COUNT(*) FROM correspondences
|
||||
WHERE created_by = 'SYSTEM_IMPORT' AND DATE(created_at) = CURDATE()) AS auto_ingested,
|
||||
(SELECT COUNT(*) FROM migration_review_queue WHERE DATE(created_at) = CURDATE()) AS sent_to_review,
|
||||
(SELECT COUNT(*) FROM migration_errors
|
||||
WHERE batch_id = '{{ $env.MIGRATION_BATCH_ID }}' AND DATE(created_at) = CURDATE()) AS errors
|
||||
FROM migration_progress mp WHERE mp.batch_id = '{{ $env.MIGRATION_BATCH_ID }}';
|
||||
```
|
||||
|
||||
**Code Node (Build Report):**
|
||||
```javascript
|
||||
const s = $input.first().json;
|
||||
const total = 20000;
|
||||
const pct = ((s.total_progress / total) * 100).toFixed(1);
|
||||
const nightsLeft = Math.ceil((total - s.total_progress) / (8 * 3600 / 3));
|
||||
|
||||
const report = `
|
||||
📊 Migration Night Summary — ${new Date().toLocaleDateString('th-TH')}
|
||||
${'─'.repeat(50)}
|
||||
✅ Auto Ingested : ${s.auto_ingested}
|
||||
🔍 Sent to Review : ${s.sent_to_review}
|
||||
❌ Errors : ${s.errors}
|
||||
─────────────────────────────────────────────────
|
||||
📈 Progress : ${s.total_progress} / ${total} (${pct}%)
|
||||
🌙 Est. Nights Left: ~${nightsLeft} คืน
|
||||
${'─'.repeat(50)}
|
||||
${s.errors > 50 ? '⚠️ WARNING: High error count — investigate before next run' : '✅ Error rate OK'}
|
||||
`;
|
||||
return [{ json: { report, stats: s } }];
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📌 ส่วนที่ 7: Monitoring (Hourly Alert — เฉพาะเมื่อเกิน Threshold)
|
||||
|
||||
**Code Node (Evaluate):**
|
||||
```javascript
|
||||
const s = $input.first().json;
|
||||
const alerts = [];
|
||||
|
||||
if (s.minutes_since_update > 30)
|
||||
alerts.push(`⚠️ No progress for ${s.minutes_since_update} min — may be stuck`);
|
||||
if (s.is_fallback_active)
|
||||
alerts.push(`⚠️ Fallback model active — errors: ${s.recent_error_count}`);
|
||||
if (s.recent_error_count >= 20)
|
||||
alerts.push(`🔴 Critical: ${s.recent_error_count} errors — consider stopping`);
|
||||
|
||||
return [{ json: { ...s, has_alerts: alerts.length > 0, alerts } }];
|
||||
```
|
||||
|
||||
**IF `has_alerts = true` → Email Alert ทันที**
|
||||
|
||||
---
|
||||
|
||||
## 📌 ส่วนที่ 8: Pre-Production Checklist
|
||||
|
||||
| ลำดับ | รายการทดสอบ | ผลลัพธ์ที่คาดหวัง | ✅/❌ |
|
||||
| --- | --------------------------------------------- | ---------------------- | --- |
|
||||
| 1 | Pre-flight ผ่านทุก Check | All green | |
|
||||
| 2 | `GET /api/meta/categories` สำเร็จ | categories array ไม่ว่าง | |
|
||||
| 3 | Enum ใน Prompt ไม่ hardcode | ตรงกับ Backend | |
|
||||
| 4 | Idempotency: รัน Batch ซ้ำ | ไม่สร้าง Revision ซ้ำ | |
|
||||
| 5 | Storage path ตาม Spec | UUID + /year/month/ | |
|
||||
| 6 | Audit Log มี `action=IMPORT, source=MIGRATION` | Verified | |
|
||||
| 7 | Review Queue ไม่สร้าง record อัตโนมัติ | Verified | |
|
||||
| 8 | Revision drift → Review Queue | Verified | |
|
||||
| 9 | Error ≥ 5 → Fallback Model สลับ | mistral:7b active | |
|
||||
| 10 | Reject/Error CSV เขียนลง `migration_logs/` | ไม่ใช่ `staging_ai/` | |
|
||||
| 11 | Rollback Guard ต้องพิมพ์ CONFIRM_ROLLBACK | Block ทำงาน | |
|
||||
| 12 | Night Summary 06:30 + Est. nights left | Email ถึง Admin | |
|
||||
| 13 | Monitoring Alert เฉพาะเกิน Threshold | ไม่ spam ทุกชั่วโมง | |
|
||||
| 14 | Nginx Rate Limit `burst=5` | Configured | |
|
||||
| 15 | Docker `mem_limit=2g` + log rotation | Configured | |
|
||||
|
||||
**คำสั่งทดสอบ:**
|
||||
```bash
|
||||
# Ollama
|
||||
docker exec -it n8n-migration curl http://<ASUSTOR_IP>:11434/api/tags
|
||||
|
||||
# RO mount
|
||||
docker exec -it n8n-migration ls /data/dms/staging_ai | head -5
|
||||
|
||||
# RW mount
|
||||
docker exec -it n8n-migration sh -c "echo ok > /data/dms/migration_logs/test.txt && echo '✅ rw OK'"
|
||||
|
||||
# DB
|
||||
docker exec -it n8n-migration mysql -h <DB_IP> -u migration_bot -p -e "SELECT 1"
|
||||
|
||||
# Backend + Category endpoint
|
||||
curl -H "Authorization: Bearer <TOKEN>" https://<BACKEND>/api/meta/categories
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📌 ส่วนที่ 9: การรันงานจริง
|
||||
|
||||
### 9.1 Daily Operation
|
||||
|
||||
| เวลา | กิจกรรม | ผู้รับผิดชอบ |
|
||||
| ----- | ------------------------------ | ------------------- |
|
||||
| 08:00 | ตรวจสอบ Night Summary Email | Admin |
|
||||
| 09:00 | Approve/Reject ใน Review Queue | Document Controller |
|
||||
| 17:00 | ตรวจ Disk Space + GPU Temp | DevOps |
|
||||
| 22:00 | Workflow เริ่มรันอัตโนมัติ | System |
|
||||
| 06:30 | Night Summary Report ส่ง Email | System |
|
||||
|
||||
### 9.2 Emergency Stop
|
||||
|
||||
```bash
|
||||
# 1. หยุด n8n
|
||||
docker stop n8n-migration
|
||||
|
||||
# 2. Disable Token
|
||||
mysql -h <DB_IP> -u root -p \
|
||||
-e "UPDATE users SET is_active = false WHERE username = 'migration_bot';"
|
||||
|
||||
# 3. Progress
|
||||
mysql -h <DB_IP> -u root -p \
|
||||
-e "SELECT * FROM migration_progress WHERE batch_id = 'migration_20260226';"
|
||||
|
||||
# 4. Errors
|
||||
mysql -h <DB_IP> -u root -p \
|
||||
-e "SELECT * FROM migration_errors ORDER BY created_at DESC LIMIT 20;"
|
||||
|
||||
# 5. Rollback ผ่าน Webhook
|
||||
curl -X POST http://<NAS_IP>:5678/webhook/rollback \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"confirmation":"CONFIRM_ROLLBACK"}'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📞 การติดต่อสนับสนุน
|
||||
|
||||
| ปัญหา | ช่องทางติดต่อ |
|
||||
| --------------- | ------------------------------------------- |
|
||||
| Technical Issue | DevOps Team (Slack: #migration-support) |
|
||||
| Data Issue | Document Controller (Email: dc@lcbp3.local) |
|
||||
| Security Issue | Security Team (Email: security@lcbp3.local) |
|
||||
|
||||
---
|
||||
|
||||
**เอกสารฉบับนี้จัดทำขึ้นเพื่อรองรับ Migration ตาม ADR-017 และ 03-04**
|
||||
**Version:** 1.8.0 | **Last Updated:** 2026-02-27 | **Author:** Development Team
|
||||
@@ -1,5 +1,5 @@
|
||||
-- ==========================================================
|
||||
-- DMS v1.7.0 Document Management System Database
|
||||
-- DMS v1.8.0 Document Management System Database
|
||||
-- Deploy Script Schema
|
||||
-- Server: Container Station on QNAP TS-473A
|
||||
-- Database service: MariaDB 11.8
|
||||
@@ -10,22 +10,18 @@
|
||||
-- reverse proxy: jc21/nginx-proxy-manager:latest
|
||||
-- cron service: n8n
|
||||
-- ==========================================================
|
||||
-- [v1.7.0 UPDATE] Refactor Schema
|
||||
-- Update: Upgraded from v1.6.0
|
||||
-- Last Updated: 2025-12-18
|
||||
-- [v1.8.0 UPDATE] Prepare migration
|
||||
-- Update: Upgraded from v1.7.0
|
||||
-- Last Updated: 2026-02-27
|
||||
-- Major Changes:
|
||||
-- 1. ปรับปรุง:
|
||||
-- 1.1 TABLE contract_drawings
|
||||
-- 1.2 TABLE contract_drawing_subcat_cat_maps
|
||||
-- 1.3 TABLE shop_drawing_sub_categories
|
||||
-- 1.4 TABLE shop_drawing_main_categories
|
||||
-- 1.5 TABLE shop_drawings
|
||||
-- 1.6 TABLE shop_drawing_revisions
|
||||
-- 1.1 TABLE correspondences
|
||||
-- - INDEX idx_doc_number (document_number),
|
||||
-- - INDEX idx_deleted_at (deleted_at),
|
||||
-- - INDEX idx_created_by (created_by),
|
||||
-- 2. เพิ่ม:
|
||||
-- 2.1 TABLE asbuilt_drawings
|
||||
-- 2.2 TABLE asbuilt_drawing_revisions
|
||||
-- 2.3 TABLE asbuilt_revision_shop_revisions_refs
|
||||
-- 2.4 TABLE asbuilt_drawing_revision_attachments
|
||||
-- 2.1 TABLE migration_progress
|
||||
-- 2.2 TABLE import_transactions
|
||||
-- ==========================================================
|
||||
SET NAMES utf8mb4;
|
||||
|
||||
@@ -54,7 +50,8 @@ DROP VIEW IF EXISTS v_current_correspondences;
|
||||
-- 🗑️ DROP TABLE SCRIPT: LCBP3-DMS v1.4.2
|
||||
-- คำเตือน: ข้อมูลทั้งหมดจะหายไป กรุณา Backup ก่อนรันบน Production
|
||||
SET FOREIGN_KEY_CHECKS = 0;
|
||||
|
||||
DROP TABLE IF EXISTS migration_progress;
|
||||
DROP TABLE IF EXISTS import_transactions;
|
||||
-- ============================================================
|
||||
-- ส่วนที่ 1: ตาราง System, Logs & Preferences (ตารางปลายทาง/ส่วนเสริม)
|
||||
-- ============================================================
|
||||
@@ -475,16 +472,16 @@ CREATE TABLE correspondences (
|
||||
created_at DATETIME DEFAULT CURRENT_TIMESTAMP COMMENT 'วันที่สร้าง',
|
||||
created_by INT COMMENT 'ผู้สร้าง',
|
||||
deleted_at DATETIME NULL COMMENT 'สำหรับ Soft Delete',
|
||||
INDEX idx_doc_number (document_number),
|
||||
INDEX idx_deleted_at (deleted_at),
|
||||
INDEX idx_created_by (created_by),
|
||||
FOREIGN KEY (correspondence_type_id) REFERENCES correspondence_types (id) ON DELETE RESTRICT,
|
||||
FOREIGN KEY (project_id) REFERENCES projects (id) ON DELETE CASCADE,
|
||||
FOREIGN KEY (originator_id) REFERENCES organizations (id) ON DELETE
|
||||
SET NULL,
|
||||
FOREIGN KEY (created_by) REFERENCES users (user_id) ON DELETE
|
||||
SET NULL,
|
||||
FOREIGN KEY (originator_id) REFERENCES organizations (id) ON DELETE SET NULL,
|
||||
FOREIGN KEY (created_by) REFERENCES users (user_id) ON DELETE SET NULL,
|
||||
-- Foreign Key ที่รวมเข้ามาจาก ALTER (ระบุชื่อ Constraint ตามที่ต้องการ)
|
||||
CONSTRAINT fk_corr_discipline FOREIGN KEY (discipline_id) REFERENCES disciplines (id) ON DELETE
|
||||
SET NULL,
|
||||
UNIQUE KEY uq_corr_no_per_project (project_id, correspondence_number)
|
||||
CONSTRAINT fk_corr_discipline FOREIGN KEY (discipline_id) REFERENCES disciplines (id) ON DELETE SET NULL,
|
||||
UNIQUE KEY uq_corr_no_per_project (project_id, correspondence_number)
|
||||
) ENGINE = InnoDB DEFAULT CHARSET = utf8mb4 COLLATE = utf8mb4_general_ci COMMENT = 'ตาราง "แม่" ของเอกสารโต้ตอบ เก็บข้อมูลที่ไม่เปลี่ยนตาม Revision';
|
||||
|
||||
-- ตารางเชื่อมผู้รับ (TO/CC) สำหรับเอกสารแต่ละฉบับ (M:N)
|
||||
@@ -1546,6 +1543,23 @@ CREATE INDEX idx_wf_hist_instance ON workflow_histories (instance_id);
|
||||
|
||||
CREATE INDEX idx_wf_hist_user ON workflow_histories (action_by_user_id);
|
||||
|
||||
-- Checkpoint Table:
|
||||
CREATE TABLE IF NOT EXISTS migration_progress (
|
||||
batch_id VARCHAR(50) PRIMARY KEY,
|
||||
last_processed_index INT DEFAULT 0,
|
||||
status ENUM('RUNNING','COMPLETED','FAILED') DEFAULT 'RUNNING',
|
||||
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP
|
||||
);
|
||||
-- Idempotency Table :
|
||||
CREATE TABLE IF NOT EXISTS import_transactions (
|
||||
id INT AUTO_INCREMENT PRIMARY KEY,
|
||||
idempotency_key VARCHAR(255) UNIQUE NOT NULL,
|
||||
document_number VARCHAR(100),
|
||||
batch_id VARCHAR(100),
|
||||
status_code INT DEFAULT 201,
|
||||
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||
INDEX idx_idem_key (idempotency_key)
|
||||
);
|
||||
-- ============================================================
|
||||
-- 5. PARTITIONING PREPARATION (Advance - Optional)
|
||||
-- ============================================================
|
||||
@@ -1,3 +1,24 @@
|
||||
-- ==========================================================
|
||||
-- DMS v1.8.0 Document Management System Database
|
||||
-- Deploy Script Schema
|
||||
-- Server: Container Station on QNAP TS-473A
|
||||
-- Database service: MariaDB 11.8
|
||||
-- database web ui: phpmyadmin 5-apache
|
||||
-- database development ui: DBeaver
|
||||
-- backend service: NestJS
|
||||
-- frontend service: next.js
|
||||
-- reverse proxy: jc21/nginx-proxy-manager:latest
|
||||
-- cron service: n8n
|
||||
-- ==========================================================
|
||||
-- [v1.8.0 UPDATE] Prepare migration
|
||||
-- Update: Upgraded from v1.7.0
|
||||
-- Last Updated: 2026-02-27
|
||||
-- Major Changes:
|
||||
-- 1. เพิ่ม:
|
||||
-- 2.1 username = migration_bot
|
||||
-- 2.2
|
||||
-- ==========================================================
|
||||
|
||||
INSERT INTO organization_roles (id, role_name)
|
||||
VALUES (1, 'OWNER'),
|
||||
(2, 'DESIGNER'),
|
||||
@@ -235,6 +256,14 @@ VALUES (
|
||||
NULL,
|
||||
10
|
||||
);
|
||||
|
||||
INSERT INTO users (username, email, role, is_active)
|
||||
VALUES (
|
||||
'migration_bot',
|
||||
'migration@system.internal',
|
||||
'SYSTEM_ADMIN',
|
||||
TRUE
|
||||
);
|
||||
-- ==========================================================
|
||||
-- Seed Roles (บทบาทพื้นฐาน 5 บทบาท ตาม Req 4.3)
|
||||
-- ==========================================================
|
||||
@@ -2171,4 +2200,4 @@ VALUES (
|
||||
NULL,
|
||||
'2025-12-16 09:34:10',
|
||||
'2025-12-16 09:34:10'
|
||||
);
|
||||
);
|
||||
Reference in New Issue
Block a user