34 KiB
34 KiB
📋 คู่มือการตั้งค่า n8n สำหรับ Legacy Data Migration
เอกสารนี้จัดทำขึ้นเพื่อรองรับการ Migration เอกสาร PDF 20,000 ฉบับ ตามแผนใน 03-04-legacy-data-migration.md และ ADR-017-ollama-data-migration.md
Note: Category Enum system-driven, Idempotency-Key Header, Storage Enforcement, Audit Log, Encoding Normalization, Security Hardening, Nginx Rate Limit, Docker Hardening, AI Physical Isolation (ASUSTOR), Folder Standard (/data/dms)
📌 ส่วนที่ 1: การติดตั้งและตั้งค่าเบื้องต้น
1.1 ติดตั้ง n8n บน ASUSTOR NAS (Docker)
mkdir -p /data/dms/n8n
cd /data/dms/n8n
cat > docker-compose.yml << 'EOF'
version: '3.8'
services:
n8n:
image: n8nio/n8n:latest
container_name: n8n-migration
restart: unless-stopped
# Docker Hardening (Patch)
mem_limit: 2g
logging:
driver: json-file
options:
max-size: "10m"
max-file: "3"
ports:
- "5678:5678"
environment:
- N8N_HOST=0.0.0.0
- N8N_PORT=5678
- N8N_PROTOCOL=http
- NODE_ENV=production
- WEBHOOK_URL=http://<NAS_IP>:5678/
- GENERIC_TIMEZONE=Asia/Bangkok
- TZ=Asia/Bangkok
- N8N_SECURE_COOKIE=false
- N8N_USER_FOLDER=/home/node/.n8n
- N8N_PUBLIC_API_DISABLED=true
- N8N_BASIC_AUTH_ACTIVE=true
- N8N_BASIC_AUTH_USER=admin
- N8N_BASIC_AUTH_PASSWORD=<strong_password>
- N8N_PAYLOAD_SIZE_MAX=10485760
- EXECUTIONS_DATA_PRUNE=true
- EXECUTIONS_DATA_MAX_AGE=168
- EXECUTIONS_DATA_PRUNE_TIMEOUT=60
- DB_TYPE=postgresdb
- DB_POSTGRESDB_HOST=<DB_IP>
- DB_POSTGRESDB_PORT=5432
- DB_POSTGRESDB_DATABASE=n8n
- DB_POSTGRESDB_USER=n8n
- DB_POSTGRESDB_PASSWORD=<password>
volumes:
- ./n8n_data:/home/node/.n8n
# read-only: อ่านไฟล์ PDF ต้นฉบับเท่านั้น
- /data/dms/staging_ai:/data/dms/staging_ai:ro
# read-write: เขียน Log และ CSV ทั้งหมด
- /data/dms/migration_logs:/data/dms/migration_logs:rw
networks:
- n8n-network
networks:
n8n-network:
driver: bridge
EOF
docker-compose up -d
⚠️ Volume หมายเหตุ:
/data/dms/staging_ai= read-only (อ่านไฟล์ต้นฉบับ) และ/data/dms/migration_logs= read-write (เขียน Log/CSV) — ห้ามเขียน CSV ลงstaging_aiเพราะจะ Error ทันที
1.2 Nginx Rate Limit
เพิ่มใน Nginx config สำหรับ Migration API:
# nginx.conf หรือ site config
limit_req_zone $binary_remote_addr zone=migration:10m rate=1r/s;
location /api/correspondences/import {
limit_req zone=migration burst=5 nodelay;
proxy_pass http://backend:3001;
}
1.3 Environment Variables
Settings → Environment Variables ใน n8n UI:
| Variable | ค่าที่แนะนำ | คำอธิบาย |
|---|---|---|
OLLAMA_HOST |
http://<ASUSTOR_IP>:11434 |
URL ของ Ollama (ใน internal network) |
OLLAMA_MODEL_PRIMARY |
llama3.2:3b |
Model หลัก |
OLLAMA_MODEL_FALLBACK |
mistral:7b-instruct-q4_K_M |
Model สำรอง |
MIGRATION_BATCH_SIZE |
10 |
จำนวน Record ต่อ Batch |
MIGRATION_DELAY_MS |
2000 |
Delay ระหว่าง Request (ms) |
CONFIDENCE_THRESHOLD_HIGH |
0.85 |
Threshold Auto Ingest |
CONFIDENCE_THRESHOLD_LOW |
0.60 |
Threshold Review Queue |
MAX_RETRY_COUNT |
3 |
จำนวนครั้ง Retry |
FALLBACK_ERROR_THRESHOLD |
5 |
Error ที่ trigger Fallback |
BACKEND_URL |
https://<BACKEND_URL> |
URL ของ LCBP3 Backend |
MIGRATION_BATCH_ID |
migration_20260226 |
ID ของ Batch |
📌 ส่วนที่ 2: การเตรียม Database
รัน SQL นี้บน MariaDB ก่อน เริ่ม n8n Workflow:
-- Checkpoint
CREATE TABLE IF NOT EXISTS migration_progress (
batch_id VARCHAR(50) PRIMARY KEY,
last_processed_index INT DEFAULT 0,
status ENUM('RUNNING','COMPLETED','FAILED') DEFAULT 'RUNNING',
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP
);
-- Review Queue (Temporary — ไม่ใช่ Business Schema)
CREATE TABLE IF NOT EXISTS migration_review_queue (
id INT AUTO_INCREMENT PRIMARY KEY,
document_number VARCHAR(100) NOT NULL,
title TEXT,
original_title TEXT,
ai_suggested_category VARCHAR(50),
ai_confidence DECIMAL(4,3),
ai_issues JSON,
review_reason VARCHAR(255),
status ENUM('PENDING','APPROVED','REJECTED') DEFAULT 'PENDING',
reviewed_by VARCHAR(100),
reviewed_at TIMESTAMP NULL,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
UNIQUE KEY uq_doc_number (document_number)
);
-- Error Log
CREATE TABLE IF NOT EXISTS migration_errors (
id INT AUTO_INCREMENT PRIMARY KEY,
batch_id VARCHAR(50),
document_number VARCHAR(100),
error_type ENUM('FILE_NOT_FOUND','AI_PARSE_ERROR','API_ERROR','DB_ERROR','UNKNOWN'),
error_message TEXT,
raw_ai_response TEXT,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
INDEX idx_batch_id (batch_id),
INDEX idx_error_type (error_type)
);
-- Fallback State
CREATE TABLE IF NOT EXISTS migration_fallback_state (
id INT AUTO_INCREMENT PRIMARY KEY,
batch_id VARCHAR(50) UNIQUE,
recent_error_count INT DEFAULT 0,
is_fallback_active BOOLEAN DEFAULT FALSE,
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP
);
-- Idempotency (Patch)
CREATE TABLE IF NOT EXISTS import_transactions (
id INT AUTO_INCREMENT PRIMARY KEY,
idempotency_key VARCHAR(255) UNIQUE NOT NULL,
document_number VARCHAR(100),
batch_id VARCHAR(100),
status_code INT DEFAULT 201,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
INDEX idx_idem_key (idempotency_key)
);
-- Daily Summary
CREATE TABLE IF NOT EXISTS migration_daily_summary (
id INT AUTO_INCREMENT PRIMARY KEY,
batch_id VARCHAR(50),
summary_date DATE,
total_processed INT DEFAULT 0,
auto_ingested INT DEFAULT 0,
sent_to_review INT DEFAULT 0,
rejected INT DEFAULT 0,
errors INT DEFAULT 0,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
UNIQUE KEY uq_batch_date (batch_id, summary_date)
);
📌 ส่วนที่ 3: Credentials
Credentials → Add New:
🔐 Ollama API
| Field | ค่า |
|---|---|
| Name | Ollama Local API |
| Type | HTTP Request |
| Base URL | http://<ASUSTOR_IP>:11434 |
| Authentication | None |
🔐 LCBP3 Backend API
| Field | ค่า |
|---|---|
| Name | LCBP3 Migration Token |
| Type | HTTP Request |
| Base URL | https://<BACKEND_URL>/api |
| Authentication | Header Auth |
| Header Name | Authorization |
| Header Value | Bearer <MIGRATION_TOKEN> |
🔐 MariaDB
| Field | ค่า |
|---|---|
| Name | LCBP3 MariaDB |
| Type | MariaDB |
| Host | <DB_IP> |
| Port | 3306 |
| Database | lcbp3_production |
| User | migration_bot |
| Password | <password> |
📌 ส่วนที่ 4: Workflow (Step-by-Step)
4.1 โครงสร้างภาพรวม
┌──────────────────────────────────────────────────────────────────────┐
│ MIGRATION WORKFLOW v1.8.0 │
├──────────────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐ │
│ │ Node 0 │──▶│ Node 1 │──▶│ Node 2 │──▶│ Node 3 │ │
│ │Pre- │ │ Data │ │ File │ │ AI │ │
│ │flight + │ │ Reader │ │ Validat.│ │Analysis │ │
│ │Fetch Cat│ │+Encoding│ │+Sanitize│ │+Enum Chk│ │
│ └─────────┘ └─────────┘ └──┬──┬───┘ └────┬────┘ │
│ │ │ │ │
│ valid │ │ error ┌───▼──────────────┐ │
│ │ └──────▶ │ Node 3.5 │ │
│ │ │ Fallback Manager │ │
│ │ └──────────────────┘ │
│ ▼ │ │
│ ┌─────────┐ ┌────▼────┐ │
│ │ Node 5D │ │ Node 4 │ │
│ │ Error │ │Confidence│ │
│ │ Log │ │+Revision │ │
│ └─────────┘ │ Drift │ │
│ └┬──┬──┬──┘ │
│ │ │ │ │
│ ┌────────────┘ │ └──────┐ │
│ ▼ ▼ ▼ │
│ ┌──────────┐ ┌──────────┐ ┌──────┐ │
│ │ Node 5A │ │ Node 5B │ │ 5C │ │
│ │ Auto │ │ Review │ │Reject│ │
│ │ Ingest │ │ Queue │ │ Log │ │
│ │+Idempot. │ │(Temp only│ └──────┘ │
│ └────┬─────┘ └──────────┘ │
│ │ │
│ ┌────▼──────┐ │
│ │ Checkpoint│ │
│ └───────────┘ │
└──────────────────────────────────────────────────────────────────────┘
4.2 Node 0: Pre-flight + Fetch System Categories
Fetch System Categories ก่อน Batch ทุกครั้ง
Sub-flow:
[Trigger] → [HTTP: Ollama /api/tags] → [MariaDB: SELECT 1]
→ [HTTP: Backend /health] → [Code: File Mount Check]
→ [HTTP: GET /api/meta/categories] → [Store in Workflow Variable]
→ [IF all pass → Node 1] [ELSE → Stop + Alert]
HTTP Node — Fetch Categories:
{
"method": "GET",
"url": "={{ $env.BACKEND_URL }}/api/meta/categories",
"authentication": "genericCredentialType",
"genericAuthType": "lcbp3MigrationToken"
}
Code Node — Store Categories + File Mount Check:
const fs = require('fs');
// เก็บ categories ใน Workflow Variable
const categories = $input.first().json.categories;
if (!categories || !Array.isArray(categories) || categories.length === 0) {
throw new Error('Failed to fetch system categories from backend');
}
// Set Workflow Variable เพื่อใช้ใน Node 3
$workflow.variables = $workflow.variables || {};
$workflow.variables.system_categories = categories;
// ตรวจ File Mount
try {
const files = fs.readdirSync('/data/dms/staging_ai');
if (files.length === 0) throw new Error('staging_ai is empty');
fs.writeFileSync('/data/dms/migration_logs/.preflight_ok', new Date().toISOString());
} catch (err) {
throw new Error(`File mount check failed: ${err.message}`);
}
return [{ json: { preflight_ok: true, system_categories: categories } }];
4.3 Node 1: Load Checkpoint + Read Excel + Encoding Normalization
Step 1 — MariaDB Node (Read Checkpoint):
SELECT last_processed_index, status
FROM migration_progress
WHERE batch_id = '{{ $env.MIGRATION_BATCH_ID }}'
LIMIT 1;
Step 2 — Spreadsheet File Node:
{ "operation": "toData", "binaryProperty": "data", "options": { "sheetName": "Sheet1" } }
Step 3 — Code Node (Checkpoint + Batch + Encoding):
const checkpointResult = $('Read Checkpoint').first();
let startIndex = 0;
if (checkpointResult && checkpointResult.json.status === 'RUNNING') {
startIndex = checkpointResult.json.last_processed_index || 0;
}
const allItems = $('Read Excel').all();
const remaining = allItems.slice(startIndex);
const batchSize = parseInt($env.MIGRATION_BATCH_SIZE) || 10;
const currentBatch = remaining.slice(0, batchSize);
// Encoding Normalization: Excel → UTF-8 NFC (Patch)
const normalize = (str) => {
if (!str) return '';
return Buffer.from(String(str), 'utf8').toString('utf8').normalize('NFC');
};
return currentBatch.map((item, i) => ({
...item,
json: {
...item.json,
document_number: normalize(item.json.document_number),
title: normalize(item.json.title),
original_index: startIndex + i
}
}));
4.4 Node 2: File Validator & Sanitizer
Node Type: Code — 2 Outputs
const fs = require('fs');
const path = require('path');
const items = $input.all();
const validatedItems = [];
const errorItems = [];
for (const item of items) {
const docNumber = item.json.document_number;
// Sanitize + Normalize Filename (Patch)
const safeName = path.basename(
String(docNumber).replace(/[^a-zA-Z0-9\-_.]/g, '_')
).normalize('NFC');
const filePath = path.resolve('/data/dms/staging_ai', `${safeName}.pdf`);
if (!filePath.startsWith('/data/dms/staging_ai/')) {
errorItems.push({ ...item, json: { ...item.json, error: 'Path traversal detected', error_type: 'FILE_NOT_FOUND' } });
continue;
}
try {
if (fs.existsSync(filePath)) {
const stats = fs.statSync(filePath);
validatedItems.push({ ...item, json: { ...item.json, file_exists: true, file_size: stats.size, file_path: filePath } });
} else {
errorItems.push({ ...item, json: { ...item.json, error: `File not found: ${filePath}`, error_type: 'FILE_NOT_FOUND', file_exists: false } });
}
} catch (err) {
errorItems.push({ ...item, json: { ...item.json, error: err.message, error_type: 'FILE_NOT_FOUND', file_exists: false } });
}
}
// Output 0 → Node 3 | Output 1 → Node 5D
return [validatedItems, errorItems];
4.5 Node 3: Build Prompt + AI Analysis
Step 1 — MariaDB (Read Fallback State):
SELECT is_fallback_active FROM migration_fallback_state
WHERE batch_id = '{{ $env.MIGRATION_BATCH_ID }}' LIMIT 1;
Step 2 — Code Node (Build Prompt: inject system_categories):
const fallbackState = $('Read Fallback State').first();
const isFallback = fallbackState?.json?.is_fallback_active || false;
const model = isFallback ? $env.OLLAMA_MODEL_FALLBACK : $env.OLLAMA_MODEL_PRIMARY;
// ใช้ system_categories จาก Workflow Variable (ไม่ hardcode)
const systemCategories = $workflow.variables?.system_categories
|| ['Correspondence','RFA','Drawing','Transmittal','Report','Other'];
const item = $input.first();
const systemPrompt = `You are a Document Controller for a large construction project.
Your task is to validate document metadata.
You MUST respond ONLY with valid JSON. No explanation, no markdown, no extra text.
If there are no issues, "detected_issues" must be an empty array [].`;
const userPrompt = `Validate this document metadata and respond in JSON:
Document Number: ${item.json.document_number}
Title: ${item.json.title}
Expected Pattern: [ORG]-[TYPE]-[SEQ] e.g. "TCC-COR-0001"
Category List (MUST match system enum exactly): ${JSON.stringify(systemCategories)}
Respond ONLY with this exact JSON structure:
{
"is_valid": true | false,
"confidence": 0.0 to 1.0,
"suggested_category": "<one from Category List>",
"detected_issues": ["<issue1>"],
"suggested_title": "<corrected title or null>"
}`;
return [{
json: {
...item.json,
active_model: model,
system_categories: systemCategories,
ollama_payload: { model, prompt: `${systemPrompt}\n\n${userPrompt}`, stream: false, format: 'json' }
}
}];
Step 3 — HTTP Request Node (Ollama):
{
"method": "POST",
"url": "={{ $env.OLLAMA_HOST }}/api/generate",
"sendBody": true,
"specifyBody": "json",
"jsonBody": "={{ $json.ollama_payload }}",
"options": { "timeout": 30000, "retry": { "count": 3, "delay": 2000, "backoff": "exponential" } }
}
Step 4 — Code Node (Parse + Validate: Enum check):
const items = $input.all();
const parsed = [];
const parseErrors = [];
for (const item of items) {
try {
let raw = item.json.response || '';
raw = raw.replace(/```json/gi, '').replace(/```/g, '').trim();
const result = JSON.parse(raw);
// Strict Schema Validation
if (typeof result.is_valid !== 'boolean')
throw new Error('is_valid must be boolean');
if (typeof result.confidence !== 'number' || result.confidence < 0 || result.confidence > 1)
throw new Error('confidence must be float 0.0–1.0');
if (!Array.isArray(result.detected_issues))
throw new Error('detected_issues must be array');
// Enum Validation ตรง System Categories (Patch)
const systemCategories = item.json.system_categories || [];
if (!systemCategories.includes(result.suggested_category))
throw new Error(`Category "${result.suggested_category}" not in system enum: [${systemCategories.join(', ')}]`);
parsed.push({ ...item, json: { ...item.json, ai_result: result, parse_error: null } });
} catch (err) {
parseErrors.push({
...item,
json: { ...item.json, ai_result: null, parse_error: err.message, raw_ai_response: item.json.response, error_type: 'AI_PARSE_ERROR' }
});
}
}
// Output 0 → Node 4 | Output 1 → Node 3.5 + Node 5D
return [parsed, parseErrors];
4.6 Node 3.5: Fallback Model Manager
MariaDB Node:
INSERT INTO migration_fallback_state (batch_id, recent_error_count, is_fallback_active)
VALUES ('{{ $env.MIGRATION_BATCH_ID }}', 1, FALSE)
ON DUPLICATE KEY UPDATE
recent_error_count = recent_error_count + 1,
is_fallback_active = CASE
WHEN recent_error_count + 1 >= {{ $env.FALLBACK_ERROR_THRESHOLD }} THEN TRUE
ELSE is_fallback_active
END,
updated_at = NOW();
Code Node (Alert):
const state = $input.first().json;
if (state.is_fallback_active) {
return [{ json: {
...state, alert: true,
alert_message: `⚠️ Fallback model (${$env.OLLAMA_MODEL_FALLBACK}) activated after ${state.recent_error_count} errors`
}}];
}
return [{ json: { ...state, alert: false } }];
4.7 Node 4: Confidence Router + Revision Drift Protection
Node Type: Code — 4 Outputs
const items = $input.all();
const autoIngest = [];
const reviewQueue = [];
const rejectLog = [];
const errorLog = [];
const HIGH = parseFloat($env.CONFIDENCE_THRESHOLD_HIGH) || 0.85;
const LOW = parseFloat($env.CONFIDENCE_THRESHOLD_LOW) || 0.60;
for (const item of items) {
if (item.json.parse_error || !item.json.ai_result) {
errorLog.push(item); continue;
}
// Revision Drift Protection
if (item.json.excel_revision !== undefined) {
const expectedRev = (item.json.current_db_revision || 0) + 1;
if (parseInt(item.json.excel_revision) !== expectedRev) {
reviewQueue.push({
...item,
json: { ...item.json, review_reason: `Revision drift: Excel=${item.json.excel_revision}, Expected=${expectedRev}` }
});
continue;
}
}
const ai = item.json.ai_result;
if (ai.confidence >= HIGH && ai.is_valid === true) {
autoIngest.push(item);
} else if (ai.confidence >= LOW) {
reviewQueue.push({ ...item, json: { ...item.json, review_reason: `Confidence ${ai.confidence.toFixed(2)} < ${HIGH}` } });
} else {
rejectLog.push({
...item,
json: { ...item.json, reject_reason: ai.is_valid === false ? 'AI marked invalid' : `Confidence ${ai.confidence.toFixed(2)} < ${LOW}` }
});
}
}
// Output 0: Auto Ingest | 1: Review Queue | 2: Reject Log | 3: Error Log
return [autoIngest, reviewQueue, rejectLog, errorLog];
4.8 Node 5A: Auto Ingest + Idempotency + Checkpoint
HTTP Request Node (Patch — Idempotency-Key Header + source_file_path):
{
"method": "POST",
"url": "={{ $env.BACKEND_URL }}/api/correspondences/import",
"authentication": "genericCredentialType",
"genericAuthType": "lcbp3MigrationToken",
"sendHeaders": true,
"headers": {
"Idempotency-Key": "={{ $json.document_number }}:{{ $env.MIGRATION_BATCH_ID }}"
},
"sendBody": true,
"specifyBody": "json",
"jsonBody": {
"document_number": "={{ $json.document_number }}",
"title": "={{ $json.ai_result.suggested_title || $json.title }}",
"category": "={{ $json.ai_result.suggested_category }}",
"source_file_path": "={{ $json.file_path }}",
"ai_confidence": "={{ $json.ai_result.confidence }}",
"ai_issues": "={{ $json.ai_result.detected_issues }}",
"migrated_by": "SYSTEM_IMPORT",
"batch_id": "={{ $env.MIGRATION_BATCH_ID }}"
},
"options": { "timeout": 30000, "retry": { "count": 3, "delay": 5000 } }
}
Backend จะ generate UUID, enforce Storage path
/storage/{project}/{category}/{year}/{month}/{uuid}.pdf, move file ผ่าน StorageService และบันทึก Audit Logaction=IMPORT, source=MIGRATION
Checkpoint Code Node (ทุก 10 Records):
const item = $input.first();
return [{ json: {
...item.json,
should_update_checkpoint: item.json.original_index % 10 === 0,
checkpoint_index: item.json.original_index
}}];
IF Node → MariaDB Checkpoint:
INSERT INTO migration_progress (batch_id, last_processed_index, status)
VALUES ('{{ $env.MIGRATION_BATCH_ID }}', {{ $json.checkpoint_index }}, 'RUNNING')
ON DUPLICATE KEY UPDATE last_processed_index = {{ $json.checkpoint_index }}, updated_at = NOW();
4.9 Node 5B: Review Queue (Temporary Table)
⚠️ ห้ามสร้าง Correspondence record — รอ Admin Approve แล้วค่อย POST
/api/correspondences/import
INSERT INTO migration_review_queue
(document_number, title, original_title, ai_suggested_category,
ai_confidence, ai_issues, review_reason, status, created_at)
VALUES (
'{{ $json.document_number }}',
'{{ $json.ai_result.suggested_title || $json.title }}',
'{{ $json.title }}',
'{{ $json.ai_result.suggested_category }}',
{{ $json.ai_result.confidence }},
'{{ JSON.stringify($json.ai_result.detected_issues) }}',
'{{ $json.review_reason }}',
'PENDING', NOW()
)
ON DUPLICATE KEY UPDATE status = 'PENDING', review_reason = '{{ $json.review_reason }}';
4.10 Node 5C: Reject Log → /data/migration_logs/
const fs = require('fs');
const item = $input.first();
const csvPath = '/data/dms/migration_logs/reject_log.csv';
const header = 'timestamp,document_number,title,reject_reason,ai_confidence,ai_issues\n';
const esc = (s) => `"${String(s||'').replace(/"/g,'""')}"`;
if (!fs.existsSync(csvPath)) fs.writeFileSync(csvPath, header, 'utf8');
const line = [
new Date().toISOString(),
esc(item.json.document_number), esc(item.json.title),
esc(item.json.reject_reason),
item.json.ai_result?.confidence ?? 'N/A',
esc(JSON.stringify(item.json.ai_result?.detected_issues || []))
].join(',') + '\n';
fs.appendFileSync(csvPath, line, 'utf8');
return [$input.first()];
4.11 Node 5D: Error Log → /data/migration_logs/ + MariaDB
const fs = require('fs');
const item = $input.first();
const csvPath = '/data/dms/migration_logs/error_log.csv';
const header = 'timestamp,document_number,error_type,error_message,raw_ai_response\n';
const esc = (s) => `"${String(s||'').replace(/"/g,'""')}"`;
if (!fs.existsSync(csvPath)) fs.writeFileSync(csvPath, header, 'utf8');
const line = [
new Date().toISOString(),
esc(item.json.document_number),
esc(item.json.error_type || 'UNKNOWN'),
esc(item.json.error || item.json.parse_error),
esc(item.json.raw_ai_response || '')
].join(',') + '\n';
fs.appendFileSync(csvPath, line, 'utf8');
return [$input.first()];
MariaDB Node:
INSERT INTO migration_errors
(batch_id, document_number, error_type, error_message, raw_ai_response, created_at)
VALUES (
'{{ $env.MIGRATION_BATCH_ID }}', '{{ $json.document_number }}',
'{{ $json.error_type || "UNKNOWN" }}', '{{ $json.error || $json.parse_error }}',
'{{ $json.raw_ai_response || "" }}', NOW()
);
📌 ส่วนที่ 5: Rollback Workflow
Workflow: Migration Rollback — Manual Trigger เท่านั้น
[Manual Trigger: {confirmation: "CONFIRM_ROLLBACK"}]
│
▼
[Code: Guard — ต้องพิมพ์ "CONFIRM_ROLLBACK"]
│ PASS
▼
[MariaDB: Disable Token]
UPDATE users SET is_active = false WHERE username = 'migration_bot';
│
▼
[MariaDB: Delete File Records]
DELETE FROM correspondence_files WHERE correspondence_id IN
(SELECT id FROM correspondences WHERE created_by = 'SYSTEM_IMPORT');
│
▼
[MariaDB: Delete Correspondence Records]
DELETE FROM correspondences WHERE created_by = 'SYSTEM_IMPORT';
│
▼
[MariaDB: Clear Idempotency Records]
DELETE FROM import_transactions WHERE batch_id = '{{$env.MIGRATION_BATCH_ID}}';
│
▼
[MariaDB: Reset Checkpoint + Fallback State]
│
▼
[Email: Rollback Report → Admin]
Confirmation Guard:
if ($input.first().json.confirmation !== 'CONFIRM_ROLLBACK') {
throw new Error('Rollback cancelled: type "CONFIRM_ROLLBACK" to proceed.');
}
return $input.all();
📌 ส่วนที่ 6: End-of-Night Summary (06:30 ทุกวัน)
MariaDB:
SELECT
mp.last_processed_index AS total_progress,
(SELECT COUNT(*) FROM correspondences
WHERE created_by = 'SYSTEM_IMPORT' AND DATE(created_at) = CURDATE()) AS auto_ingested,
(SELECT COUNT(*) FROM migration_review_queue WHERE DATE(created_at) = CURDATE()) AS sent_to_review,
(SELECT COUNT(*) FROM migration_errors
WHERE batch_id = '{{ $env.MIGRATION_BATCH_ID }}' AND DATE(created_at) = CURDATE()) AS errors
FROM migration_progress mp WHERE mp.batch_id = '{{ $env.MIGRATION_BATCH_ID }}';
Code Node (Build Report):
const s = $input.first().json;
const total = 20000;
const pct = ((s.total_progress / total) * 100).toFixed(1);
const nightsLeft = Math.ceil((total - s.total_progress) / (8 * 3600 / 3));
const report = `
📊 Migration Night Summary — ${new Date().toLocaleDateString('th-TH')}
${'─'.repeat(50)}
✅ Auto Ingested : ${s.auto_ingested}
🔍 Sent to Review : ${s.sent_to_review}
❌ Errors : ${s.errors}
─────────────────────────────────────────────────
📈 Progress : ${s.total_progress} / ${total} (${pct}%)
🌙 Est. Nights Left: ~${nightsLeft} คืน
${'─'.repeat(50)}
${s.errors > 50 ? '⚠️ WARNING: High error count — investigate before next run' : '✅ Error rate OK'}
`;
return [{ json: { report, stats: s } }];
📌 ส่วนที่ 7: Monitoring (Hourly Alert — เฉพาะเมื่อเกิน Threshold)
Code Node (Evaluate):
const s = $input.first().json;
const alerts = [];
if (s.minutes_since_update > 30)
alerts.push(`⚠️ No progress for ${s.minutes_since_update} min — may be stuck`);
if (s.is_fallback_active)
alerts.push(`⚠️ Fallback model active — errors: ${s.recent_error_count}`);
if (s.recent_error_count >= 20)
alerts.push(`🔴 Critical: ${s.recent_error_count} errors — consider stopping`);
return [{ json: { ...s, has_alerts: alerts.length > 0, alerts } }];
IF has_alerts = true → Email Alert ทันที
📌 ส่วนที่ 8: Pre-Production Checklist
| ลำดับ | รายการทดสอบ | ผลลัพธ์ที่คาดหวัง | ✅/❌ |
|---|---|---|---|
| 1 | Pre-flight ผ่านทุก Check | All green | |
| 2 | GET /api/meta/categories สำเร็จ |
categories array ไม่ว่าง | |
| 3 | Enum ใน Prompt ไม่ hardcode | ตรงกับ Backend | |
| 4 | Idempotency: รัน Batch ซ้ำ | ไม่สร้าง Revision ซ้ำ | |
| 5 | Storage path ตาม Spec | UUID + /year/month/ | |
| 6 | Audit Log มี action=IMPORT, source=MIGRATION |
Verified | |
| 7 | Review Queue ไม่สร้าง record อัตโนมัติ | Verified | |
| 8 | Revision drift → Review Queue | Verified | |
| 9 | Error ≥ 5 → Fallback Model สลับ | mistral:7b active | |
| 10 | Reject/Error CSV เขียนลง migration_logs/ |
ไม่ใช่ staging_ai/ |
|
| 11 | Rollback Guard ต้องพิมพ์ CONFIRM_ROLLBACK | Block ทำงาน | |
| 12 | Night Summary 06:30 + Est. nights left | Email ถึง Admin | |
| 13 | Monitoring Alert เฉพาะเกิน Threshold | ไม่ spam ทุกชั่วโมง | |
| 14 | Nginx Rate Limit burst=5 |
Configured | |
| 15 | Docker mem_limit=2g + log rotation |
Configured |
คำสั่งทดสอบ:
# Ollama
docker exec -it n8n-migration curl http://<ASUSTOR_IP>:11434/api/tags
# RO mount
docker exec -it n8n-migration ls /data/dms/staging_ai | head -5
# RW mount
docker exec -it n8n-migration sh -c "echo ok > /data/dms/migration_logs/test.txt && echo '✅ rw OK'"
# DB
docker exec -it n8n-migration mysql -h <DB_IP> -u migration_bot -p -e "SELECT 1"
# Backend + Category endpoint
curl -H "Authorization: Bearer <TOKEN>" https://<BACKEND>/api/meta/categories
📌 ส่วนที่ 9: การรันงานจริง
9.1 Daily Operation
| เวลา | กิจกรรม | ผู้รับผิดชอบ |
|---|---|---|
| 08:00 | ตรวจสอบ Night Summary Email | Admin |
| 09:00 | Approve/Reject ใน Review Queue | Document Controller |
| 17:00 | ตรวจ Disk Space + GPU Temp | DevOps |
| 22:00 | Workflow เริ่มรันอัตโนมัติ | System |
| 06:30 | Night Summary Report ส่ง Email | System |
9.2 Emergency Stop
# 1. หยุด n8n
docker stop n8n-migration
# 2. Disable Token
mysql -h <DB_IP> -u root -p \
-e "UPDATE users SET is_active = false WHERE username = 'migration_bot';"
# 3. Progress
mysql -h <DB_IP> -u root -p \
-e "SELECT * FROM migration_progress WHERE batch_id = 'migration_20260226';"
# 4. Errors
mysql -h <DB_IP> -u root -p \
-e "SELECT * FROM migration_errors ORDER BY created_at DESC LIMIT 20;"
# 5. Rollback ผ่าน Webhook
curl -X POST http://<NAS_IP>:5678/webhook/rollback \
-H "Content-Type: application/json" \
-d '{"confirmation":"CONFIRM_ROLLBACK"}'
📞 การติดต่อสนับสนุน
| ปัญหา | ช่องทางติดต่อ |
|---|---|
| Technical Issue | DevOps Team (Slack: #migration-support) |
| Data Issue | Document Controller (Email: dc@lcbp3.local) |
| Security Issue | Security Team (Email: security@lcbp3.local) |
เอกสารฉบับนี้จัดทำขึ้นเพื่อรองรับ Migration ตาม ADR-017 และ 03-04 Version: 1.8.0 | Last Updated: 2026-02-27 | Author: Development Team