260228:1412 20260228: setup n8n
All checks were successful
Build and Deploy / deploy (push) Successful in 2m49s
All checks were successful
Build and Deploy / deploy (push) Successful in 2m49s
This commit is contained in:
admin
File diff suppressed because it is too large
Load Diff
@@ -4,7 +4,7 @@
|
||||
| ------------------------------------------------------------------ | ------- |
|
||||
| legacy PDF document migration to system v1.8.0 uses n8n and Ollama | 1.8.0 |
|
||||
|
||||
> **Note:** Category Enum system-driven, Idempotency Contract, Duplicate Handling Clarification, Storage Enforcement, Audit Log Enhancement, Review Queue Integration, Revision Drift Protection, Execution Time, Encoding Normalization, Security Hardening, AI Physical Isolation (ASUSTOR), Folder Standard (/data/dms)
|
||||
> **Note:** Category Enum system-driven, Idempotency Contract, Duplicate Handling Clarification, Storage Enforcement, Audit Log Enhancement, Review Queue Integration, Revision Drift Protection, Execution Time, Encoding Normalization, Security Hardening, Orchestrator on QNAP, AI Physical Isolation (Desktop Desk-5439), Folder Standard (/share/np-dms/n8n)
|
||||
|
||||
---
|
||||
|
||||
@@ -21,8 +21,8 @@
|
||||
|
||||
## 2. โครงสร้างพื้นฐาน (Migration Infrastructure)
|
||||
|
||||
- **Migration Orchestrator:** n8n (รันจาก Docker Container บน ASUSTOR NAS)
|
||||
- **AI Validator:** Ollama (รันใน Internal Network บน ASUSTOR NAS)
|
||||
- **Migration Orchestrator:** n8n (รันจาก Docker Container บน QNAP NAS)
|
||||
- **AI Validator:** Ollama (รันใน Internal Network บน Desktop Desk-5439, RTX 2060 SUPER 8GB)
|
||||
- **Target Database:** MariaDB (`correspondences` table) บน QNAP NAS
|
||||
- **Target Storage:** QNAP File System — **ผ่าน Backend StorageService API เท่านั้น** (ห้าม move file โดยตรง)
|
||||
- **Connection:** 2.5G LAN + LACP / Internal VLAN
|
||||
@@ -35,18 +35,18 @@
|
||||
|
||||
**File Migration:**
|
||||
- ย้ายไฟล์ PDF ทั้งหมดจากแหล่งเก็บไปยัง Folder ชั่วคราวบน NAS (QNAP)
|
||||
- Target Path: `/data/dms/staging_ai/`
|
||||
- Target Path: `/share/np-dms/staging_ai/`
|
||||
|
||||
**Mount Folder:**
|
||||
- Bind Mount `/data/dms/staging_ai/` เข้ากับ n8n Container แบบ **read-only**
|
||||
- สร้าง `/data/dms/migration_logs/` Volume แยกสำหรับเขียน Log แบบ **read-write**
|
||||
- Bind Mount `/share/np-dms/staging_ai/` เข้ากับ n8n Container แบบ **read-only**
|
||||
- สร้าง `/share/np-dms/n8n/migration_logs/` Volume แยกสำหรับเขียน Log แบบ **read-write**
|
||||
|
||||
**Ollama Config:**
|
||||
- ติดตั้ง Ollama บน ASUSTOR NAS
|
||||
- ติดตั้ง Ollama บน Desktop (Desk-5439, RTX 2060 SUPER 8GB)
|
||||
- No DB credentials, Internal network only
|
||||
|
||||
```bash
|
||||
# แนะนำ: llama3.2:3b (เร็ว, VRAM ~3GB, เหมาะ Classification)
|
||||
# แนะนำ: llama3.2:3b (เร็ว, VRAM ~3GB, เหมาะ Classification) หรือ ollama run llama3.2:3b
|
||||
ollama pull llama3.2:3b
|
||||
|
||||
# Fallback: mistral:7b-instruct-q4_K_M (แม่นกว่า, VRAM ~5GB)
|
||||
@@ -55,7 +55,7 @@ ollama pull llama3.2:3b
|
||||
|
||||
**ทดสอบ Ollama:**
|
||||
```bash
|
||||
curl http://<OLLAMA_HOST>:11434/api/generate \
|
||||
curl http://192.168.20.100:11434/api/generate \
|
||||
-d '{"model":"llama3.2:3b","prompt":"reply: ok","stream":false}'
|
||||
```
|
||||
|
||||
@@ -165,7 +165,9 @@ return items.map(item => ({
|
||||
json: {
|
||||
...item.json,
|
||||
document_number: normalize(item.json.document_number),
|
||||
title: normalize(item.json.title)
|
||||
title: normalize(item.json.title),
|
||||
// Mapping เลขอ้างอิงเก่า (Legacy Number) เพื่อนำไปเก็บใน details JSON
|
||||
legacy_document_number: item.json.document_number
|
||||
}
|
||||
}));
|
||||
```
|
||||
@@ -174,7 +176,7 @@ return items.map(item => ({
|
||||
|
||||
- ตรวจสอบไฟล์ PDF มีอยู่จริงบน NAS
|
||||
- Normalize ชื่อไฟล์เป็น **UTF-8 NFC**
|
||||
- Path Traversal Guard: resolved path ต้องอยู่ใน `/data/dms/staging_ai` เท่านั้น
|
||||
- Path Traversal Guard: resolved path ต้องอยู่ใน `/share/np-dms/staging_ai` เท่านั้น
|
||||
- **Output 0** → valid → Node 3
|
||||
- **Output 1** → error → Node 5D (ไม่หายเงียบ)
|
||||
|
||||
@@ -246,7 +248,7 @@ if (item.json.excel_revision !== undefined) {
|
||||
|
||||
#### Node 5A: Auto Ingest — Backend API
|
||||
|
||||
> ⚠️ **Storage Enforcement:** n8n ส่งแค่ `source_file_path` — Backend จะ generate UUID, enforce path strategy (`/data/dms/uploads/YYYY/MM/{uuid}.pdf`), และ move file atomically ผ่าน StorageService
|
||||
> ⚠️ **Storage Enforcement:** n8n ส่งแค่ `source_file_path` — Backend จะ generate UUID, enforce path strategy (`/share/np-dms/staging_ai/...`), และ move file atomically ผ่าน StorageService
|
||||
|
||||
```http
|
||||
POST /api/correspondences/import
|
||||
@@ -303,9 +305,9 @@ Review → Admin Approve → POST /api/correspondences/import (เหมือ
|
||||
Admin Reject → ลบออกจาก queue ไม่สร้าง record
|
||||
```
|
||||
|
||||
#### Node 5C: Reject Log → `/data/dms/migration_logs/reject_log.csv`
|
||||
#### Node 5C: Reject Log → `/share/np-dms/n8n/migration_logs/reject_log.csv`
|
||||
|
||||
#### Node 5D: Error Log → `/data/dms/migration_logs/error_log.csv` + MariaDB
|
||||
#### Node 5D: Error Log → `/share/np-dms/n8n/migration_logs/error_log.csv` + MariaDB
|
||||
|
||||
---
|
||||
|
||||
@@ -370,7 +372,7 @@ SELECT ROW_COUNT();
|
||||
COMMIT;
|
||||
```
|
||||
|
||||
**Step 3:** ย้ายไฟล์กลับ `/data/dms/staging_ai/` ผ่าน Script แยก
|
||||
**Step 3:** ย้ายไฟล์กลับ `/share/np-dms/staging_ai/` ผ่าน Script แยก
|
||||
|
||||
**Step 4:** Reset State
|
||||
```sql
|
||||
@@ -424,4 +426,4 @@ GROUP BY idempotency_key HAVING COUNT(*) > 1;
|
||||
|
||||
---
|
||||
|
||||
> **ข้อแนะนำด้าน Physical Storage:** ไฟล์ PDF ทั้ง 20,000 ไฟล์จะถูก move โดย Backend StorageService ไปยัง path ที่ถูกต้องโดยอัตโนมัติ ไม่ปล่อยค้างไว้ที่ `/data/dms/staging_ai/`
|
||||
> **ข้อแนะนำด้าน Physical Storage:** ไฟล์ PDF ทั้ง 20,000 ไฟล์จะถูก move โดย Backend StorageService ไปยัง path ที่ถูกต้องโดยอัตโนมัติ ไม่ปล่อยค้างไว้ที่ `/share/np-dms/staging_ai/`
|
||||
|
||||
@@ -2,77 +2,103 @@
|
||||
|
||||
เอกสารนี้จัดทำขึ้นเพื่อรองรับการ Migration เอกสาร PDF 20,000 ฉบับ ตามแผนใน `03-04-legacy-data-migration.md` และ `ADR-017-ollama-data-migration.md`
|
||||
|
||||
> **Note:** Category Enum system-driven, Idempotency-Key Header, Storage Enforcement, Audit Log, Encoding Normalization, Security Hardening, Nginx Rate Limit, Docker Hardening, AI Physical Isolation (ASUSTOR), Folder Standard (/data/dms)
|
||||
> **Note:** Category Enum system-driven, Idempotency-Key Header, Storage Enforcement, Audit Log, Encoding Normalization, Security Hardening, Nginx Rate Limit, Docker Hardening, Orchestrator on QNAP, AI Physical Isolation (Desktop Desk-5439), Folder Standard (/share/np-dms/n8n)
|
||||
|
||||
---
|
||||
|
||||
## 📌 ส่วนที่ 1: การติดตั้งและตั้งค่าเบื้องต้น
|
||||
|
||||
### 1.1 ติดตั้ง n8n บน ASUSTOR NAS (Docker)
|
||||
### 1.1 ปรับปรุง n8n บน QNAP NAS (Docker)
|
||||
|
||||
```bash
|
||||
mkdir -p /data/dms/n8n
|
||||
cd /data/dms/n8n
|
||||
คุณสามารถเพิ่ม PostgreSQL Service เข้าไปใน `docker-compose-lcbp3-n8n.yml` ปัจจุบันบน QNAP NAS ได้ดังนี้:
|
||||
|
||||
cat > docker-compose.yml << 'EOF'
|
||||
```yaml
|
||||
version: '3.8'
|
||||
|
||||
x-restart: &restart_policy
|
||||
restart: unless-stopped
|
||||
|
||||
x-logging: &default_logging
|
||||
logging:
|
||||
driver: "json-file"
|
||||
options:
|
||||
max-size: "10m"
|
||||
max-file: "5"
|
||||
|
||||
services:
|
||||
n8n-db:
|
||||
<<: [*restart_policy, *default_logging]
|
||||
image: postgres:16-alpine
|
||||
container_name: n8n-db
|
||||
environment:
|
||||
- POSTGRES_USER=n8n
|
||||
- POSTGRES_PASSWORD=<strong_password>
|
||||
- POSTGRES_DB=n8n
|
||||
volumes:
|
||||
- "/share/np-dms/n8n/postgres-data:/var/lib/postgresql/data"
|
||||
networks:
|
||||
lcbp3: {}
|
||||
healthcheck:
|
||||
test: ['CMD-SHELL', 'pg_isready -h localhost -U n8n -d n8n']
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
|
||||
n8n:
|
||||
image: n8nio/n8n:latest
|
||||
container_name: n8n-migration
|
||||
restart: unless-stopped
|
||||
# Docker Hardening (Patch)
|
||||
mem_limit: 2g
|
||||
logging:
|
||||
driver: json-file
|
||||
options:
|
||||
max-size: "10m"
|
||||
max-file: "3"
|
||||
<<: [*restart_policy, *default_logging]
|
||||
image: n8nio/n8n:1.78.0
|
||||
container_name: n8n
|
||||
depends_on:
|
||||
n8n-db:
|
||||
condition: service_healthy
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: "1.5"
|
||||
memory: 2G
|
||||
environment:
|
||||
TZ: "Asia/Bangkok"
|
||||
NODE_ENV: "production"
|
||||
N8N_PUBLIC_URL: "https://n8n.np-dms.work/"
|
||||
WEBHOOK_URL: "https://n8n.np-dms.work/"
|
||||
N8N_EDITOR_BASE_URL: "https://n8n.np-dms.work/"
|
||||
N8N_PROTOCOL: "https"
|
||||
N8N_HOST: "n8n.np-dms.work"
|
||||
N8N_PORT: 5678
|
||||
N8N_PROXY_HOPS: "1"
|
||||
N8N_DIAGNOSTICS_ENABLED: 'false'
|
||||
N8N_SECURE_COOKIE: 'true'
|
||||
N8N_ENCRYPTION_KEY: "9AAIB7Da9DW1qAhJE5/Bz4SnbQjeAngI"
|
||||
N8N_ENFORCE_SETTINGS_FILE_PERMISSIONS: 'true'
|
||||
GENERIC_TIMEZONE: "Asia/Bangkok"
|
||||
# DB Setup
|
||||
DB_TYPE: postgresdb
|
||||
DB_POSTGRESDB_DATABASE: n8n
|
||||
DB_POSTGRESDB_HOST: n8n-db
|
||||
DB_POSTGRESDB_PORT: 5432
|
||||
DB_POSTGRESDB_USER: n8n
|
||||
DB_POSTGRESDB_PASSWORD: <strong_password>
|
||||
# Data Prune
|
||||
EXECUTIONS_DATA_PRUNE: 'true'
|
||||
EXECUTIONS_DATA_MAX_AGE: 168
|
||||
EXECUTIONS_DATA_PRUNE_TIMEOUT: 60
|
||||
ports:
|
||||
- "5678:5678"
|
||||
environment:
|
||||
- N8N_HOST=0.0.0.0
|
||||
- N8N_PORT=5678
|
||||
- N8N_PROTOCOL=http
|
||||
- NODE_ENV=production
|
||||
- WEBHOOK_URL=http://<NAS_IP>:5678/
|
||||
- GENERIC_TIMEZONE=Asia/Bangkok
|
||||
- TZ=Asia/Bangkok
|
||||
- N8N_SECURE_COOKIE=false
|
||||
- N8N_USER_FOLDER=/home/node/.n8n
|
||||
- N8N_PUBLIC_API_DISABLED=true
|
||||
- N8N_BASIC_AUTH_ACTIVE=true
|
||||
- N8N_BASIC_AUTH_USER=admin
|
||||
- N8N_BASIC_AUTH_PASSWORD=<strong_password>
|
||||
- N8N_PAYLOAD_SIZE_MAX=10485760
|
||||
- EXECUTIONS_DATA_PRUNE=true
|
||||
- EXECUTIONS_DATA_MAX_AGE=168
|
||||
- EXECUTIONS_DATA_PRUNE_TIMEOUT=60
|
||||
- DB_TYPE=postgresdb
|
||||
- DB_POSTGRESDB_HOST=<DB_IP>
|
||||
- DB_POSTGRESDB_PORT=5432
|
||||
- DB_POSTGRESDB_DATABASE=n8n
|
||||
- DB_POSTGRESDB_USER=n8n
|
||||
- DB_POSTGRESDB_PASSWORD=<password>
|
||||
volumes:
|
||||
- ./n8n_data:/home/node/.n8n
|
||||
# read-only: อ่านไฟล์ PDF ต้นฉบับเท่านั้น
|
||||
- /data/dms/staging_ai:/data/dms/staging_ai:ro
|
||||
# read-write: เขียน Log และ CSV ทั้งหมด
|
||||
- /data/dms/migration_logs:/data/dms/migration_logs:rw
|
||||
networks:
|
||||
- n8n-network
|
||||
|
||||
networks:
|
||||
n8n-network:
|
||||
driver: bridge
|
||||
EOF
|
||||
|
||||
docker-compose up -d
|
||||
lcbp3: {}
|
||||
volumes:
|
||||
- "/share/np-dms/n8n:/home/node/.n8n"
|
||||
- "/share/np-dms/n8n/cache:/home/node/.cache"
|
||||
- "/share/np-dms/n8n/scripts:/scripts"
|
||||
- "/share/np-dms/n8n/data:/data"
|
||||
- "/var/run/docker.sock:/var/run/docker.sock"
|
||||
# read-only: อ่านไฟล์ PDF ต้นฉบับเท่านั้น
|
||||
- "/share/np-dms/staging_ai:/share/np-dms/staging_ai:ro"
|
||||
# read-write: เขียน Log และ CSV ทั้งหมด
|
||||
- "/share/np-dms/n8n/migration_logs:/share/np-dms/n8n/migration_logs:rw"
|
||||
```
|
||||
|
||||
> ⚠️ **Volume หมายเหตุ:** `/data/dms/staging_ai` = **read-only** (อ่านไฟล์ต้นฉบับ) และ `/data/dms/migration_logs` = **read-write** (เขียน Log/CSV) — ห้ามเขียน CSV ลง `staging_ai` เพราะจะ Error ทันที
|
||||
> ⚠️ **Volume หมายเหตุ:** `/share/np-dms/staging_ai` = **read-only** (อ่านไฟล์ต้นฉบับ) และ `/share/np-dms/n8n/migration_logs` = **read-write** (เขียน Log/CSV) — ห้ามเขียน CSV ลง `staging_ai` เพราะจะ Error ทันที
|
||||
|
||||
### 1.2 Nginx Rate Limit
|
||||
|
||||
@@ -92,19 +118,19 @@ location /api/correspondences/import {
|
||||
|
||||
**Settings → Environment Variables ใน n8n UI:**
|
||||
|
||||
| Variable | ค่าที่แนะนำ | คำอธิบาย |
|
||||
| --------------------------- | ---------------------------- | ------------------------------------ |
|
||||
| `OLLAMA_HOST` | `http://<ASUSTOR_IP>:11434` | URL ของ Ollama (ใน internal network) |
|
||||
| `OLLAMA_MODEL_PRIMARY` | `llama3.2:3b` | Model หลัก |
|
||||
| `OLLAMA_MODEL_FALLBACK` | `mistral:7b-instruct-q4_K_M` | Model สำรอง |
|
||||
| `MIGRATION_BATCH_SIZE` | `10` | จำนวน Record ต่อ Batch |
|
||||
| `MIGRATION_DELAY_MS` | `2000` | Delay ระหว่าง Request (ms) |
|
||||
| `CONFIDENCE_THRESHOLD_HIGH` | `0.85` | Threshold Auto Ingest |
|
||||
| `CONFIDENCE_THRESHOLD_LOW` | `0.60` | Threshold Review Queue |
|
||||
| `MAX_RETRY_COUNT` | `3` | จำนวนครั้ง Retry |
|
||||
| `FALLBACK_ERROR_THRESHOLD` | `5` | Error ที่ trigger Fallback |
|
||||
| `BACKEND_URL` | `https://<BACKEND_URL>` | URL ของ LCBP3 Backend |
|
||||
| `MIGRATION_BATCH_ID` | `migration_20260226` | ID ของ Batch |
|
||||
| Variable | ค่าที่แนะนำ | คำอธิบาย |
|
||||
| --------------------------- | ----------------------------- | ---------------------------------- |
|
||||
| `OLLAMA_HOST` | `http://192.168.20.100:11434` | URL ของ Ollama (Desktop Desk-5439) |
|
||||
| `OLLAMA_MODEL_PRIMARY` | `llama3.2:3b` | Model หลัก |
|
||||
| `OLLAMA_MODEL_FALLBACK` | `mistral:7b-instruct-q4_K_M` | Model สำรอง |
|
||||
| `MIGRATION_BATCH_SIZE` | `10` | จำนวน Record ต่อ Batch |
|
||||
| `MIGRATION_DELAY_MS` | `2000` | Delay ระหว่าง Request (ms) |
|
||||
| `CONFIDENCE_THRESHOLD_HIGH` | `0.85` | Threshold Auto Ingest |
|
||||
| `CONFIDENCE_THRESHOLD_LOW` | `0.60` | Threshold Review Queue |
|
||||
| `MAX_RETRY_COUNT` | `3` | จำนวนครั้ง Retry |
|
||||
| `FALLBACK_ERROR_THRESHOLD` | `5` | Error ที่ trigger Fallback |
|
||||
| `BACKEND_URL` | `https://<BACKEND_URL>` | URL ของ LCBP3 Backend |
|
||||
| `MIGRATION_BATCH_ID` | `migration_20260226` | ID ของ Batch |
|
||||
|
||||
---
|
||||
|
||||
@@ -193,12 +219,12 @@ CREATE TABLE IF NOT EXISTS migration_daily_summary (
|
||||
**Credentials → Add New:**
|
||||
|
||||
#### 🔐 Ollama API
|
||||
| Field | ค่า |
|
||||
| -------------- | --------------------------- |
|
||||
| Name | `Ollama Local API` |
|
||||
| Type | `HTTP Request` |
|
||||
| Base URL | `http://<ASUSTOR_IP>:11434` |
|
||||
| Authentication | `None` |
|
||||
| Field | ค่า |
|
||||
| -------------- | ----------------------------- |
|
||||
| Name | `Ollama Local API` |
|
||||
| Type | `HTTP Request` |
|
||||
| Base URL | `http://192.168.20.100:11434` |
|
||||
| Authentication | `None` |
|
||||
|
||||
#### 🔐 LCBP3 Backend API
|
||||
| Field | ค่า |
|
||||
@@ -306,9 +332,9 @@ $workflow.variables.system_categories = categories;
|
||||
|
||||
// ตรวจ File Mount
|
||||
try {
|
||||
const files = fs.readdirSync('/data/dms/staging_ai');
|
||||
const files = fs.readdirSync('/share/np-dms/staging_ai');
|
||||
if (files.length === 0) throw new Error('staging_ai is empty');
|
||||
fs.writeFileSync('/data/dms/migration_logs/.preflight_ok', new Date().toISOString());
|
||||
fs.writeFileSync('/share/np-dms/n8n/migration_logs/.preflight_ok', new Date().toISOString());
|
||||
} catch (err) {
|
||||
throw new Error(`File mount check failed: ${err.message}`);
|
||||
}
|
||||
@@ -382,9 +408,9 @@ for (const item of items) {
|
||||
const safeName = path.basename(
|
||||
String(docNumber).replace(/[^a-zA-Z0-9\-_.]/g, '_')
|
||||
).normalize('NFC');
|
||||
const filePath = path.resolve('/data/dms/staging_ai', `${safeName}.pdf`);
|
||||
const filePath = path.resolve('/share/np-dms/staging_ai', `${safeName}.pdf`);
|
||||
|
||||
if (!filePath.startsWith('/data/dms/staging_ai/')) {
|
||||
if (!filePath.startsWith('/share/np-dms/staging_ai/')) {
|
||||
errorItems.push({ ...item, json: { ...item.json, error: 'Path traversal detected', error_type: 'FILE_NOT_FOUND' } });
|
||||
continue;
|
||||
}
|
||||
@@ -612,7 +638,10 @@ return [autoIngest, reviewQueue, rejectLog, errorLog];
|
||||
"ai_confidence": "={{ $json.ai_result.confidence }}",
|
||||
"ai_issues": "={{ $json.ai_result.detected_issues }}",
|
||||
"migrated_by": "SYSTEM_IMPORT",
|
||||
"batch_id": "={{ $env.MIGRATION_BATCH_ID }}"
|
||||
"batch_id": "={{ $env.MIGRATION_BATCH_ID }}",
|
||||
"details": {
|
||||
"legacy_number": "={{ $json.legacy_document_number }}"
|
||||
}
|
||||
},
|
||||
"options": { "timeout": 30000, "retry": { "count": 3, "delay": 5000 } }
|
||||
}
|
||||
@@ -662,12 +691,12 @@ ON DUPLICATE KEY UPDATE status = 'PENDING', review_reason = '{{ $json.review_rea
|
||||
|
||||
---
|
||||
|
||||
### 4.10 Node 5C: Reject Log → `/data/migration_logs/`
|
||||
#### 4.10 Node 5C: Reject Log → `/share/np-dms/n8n/migration_logs/`
|
||||
|
||||
```javascript
|
||||
const fs = require('fs');
|
||||
const item = $input.first();
|
||||
const csvPath = '/data/dms/migration_logs/reject_log.csv';
|
||||
const csvPath = '/share/np-dms/n8n/migration_logs/reject_log.csv';
|
||||
const header = 'timestamp,document_number,title,reject_reason,ai_confidence,ai_issues\n';
|
||||
const esc = (s) => `"${String(s||'').replace(/"/g,'""')}"`;
|
||||
|
||||
@@ -687,12 +716,12 @@ return [$input.first()];
|
||||
|
||||
---
|
||||
|
||||
### 4.11 Node 5D: Error Log → `/data/migration_logs/` + MariaDB
|
||||
#### 4.11 Node 5D: Error Log → `/share/np-dms/n8n/migration_logs/` + MariaDB
|
||||
|
||||
```javascript
|
||||
const fs = require('fs');
|
||||
const item = $input.first();
|
||||
const csvPath = '/data/dms/migration_logs/error_log.csv';
|
||||
const csvPath = '/share/np-dms/n8n/migration_logs/error_log.csv';
|
||||
const header = 'timestamp,document_number,error_type,error_message,raw_ai_response\n';
|
||||
const esc = (s) => `"${String(s||'').replace(/"/g,'""')}"`;
|
||||
|
||||
|
||||
@@ -1,29 +0,0 @@
|
||||
-- Fix Project Permissions
|
||||
-- File: specs/07-database/fix-project-permissions.sql
|
||||
-- 1. Ensure project.view permission exists
|
||||
INSERT IGNORE INTO permissions (
|
||||
permission_id,
|
||||
permission_name,
|
||||
description,
|
||||
module,
|
||||
is_active
|
||||
)
|
||||
VALUES (
|
||||
202,
|
||||
'project.view',
|
||||
'ดูรายการโครงการ',
|
||||
'project',
|
||||
1
|
||||
);
|
||||
-- 2. Grant project.view to Superadmin (Role 1)
|
||||
INSERT IGNORE INTO role_permissions (role_id, permission_id)
|
||||
VALUES (1, 202);
|
||||
-- 3. Grant project.view to Organization Admin (Role 2)
|
||||
INSERT IGNORE INTO role_permissions (role_id, permission_id)
|
||||
VALUES (2, 202);
|
||||
-- 4. Grant project.view to Project Manager (Role 6)
|
||||
INSERT IGNORE INTO role_permissions (role_id, permission_id)
|
||||
VALUES (6, 202);
|
||||
-- 5. Grant project.view to Viewer (Role 5)
|
||||
INSERT IGNORE INTO role_permissions (role_id, permission_id)
|
||||
VALUES (5, 202);
|
||||
@@ -16,12 +16,16 @@
|
||||
-- Major Changes:
|
||||
-- 1. ปรับปรุง:
|
||||
-- 1.1 TABLE correspondences
|
||||
-- - INDEX idx_doc_number (document_number),
|
||||
-- - INDEX idx_doc_number (correspondence_number),
|
||||
-- - INDEX idx_deleted_at (deleted_at),
|
||||
-- - INDEX idx_created_by (created_by),
|
||||
-- 2. เพิ่ม:
|
||||
-- 2.1 TABLE migration_progress
|
||||
-- 2.2 TABLE import_transactions
|
||||
-- 2.3 TABLE migration_review_queue
|
||||
-- 2.4 TABLE migration_errors
|
||||
-- 2.5 TABLE migration_fallback_state
|
||||
-- 2.6 TABLE migration_daily_summary
|
||||
-- ==========================================================
|
||||
SET NAMES utf8mb4;
|
||||
|
||||
@@ -50,8 +54,19 @@ DROP VIEW IF EXISTS v_current_correspondences;
|
||||
-- 🗑️ DROP TABLE SCRIPT: LCBP3-DMS v1.4.2
|
||||
-- คำเตือน: ข้อมูลทั้งหมดจะหายไป กรุณา Backup ก่อนรันบน Production
|
||||
SET FOREIGN_KEY_CHECKS = 0;
|
||||
|
||||
DROP TABLE IF EXISTS migration_progress;
|
||||
|
||||
DROP TABLE IF EXISTS import_transactions;
|
||||
|
||||
DROP TABLE IF EXISTS migration_review_queue;
|
||||
|
||||
DROP TABLE IF EXISTS migration_errors;
|
||||
|
||||
DROP TABLE IF EXISTS migration_fallback_state;
|
||||
|
||||
DROP TABLE IF EXISTS migration_daily_summary;
|
||||
|
||||
-- ============================================================
|
||||
-- ส่วนที่ 1: ตาราง System, Logs & Preferences (ตารางปลายทาง/ส่วนเสริม)
|
||||
-- ============================================================
|
||||
@@ -472,16 +487,19 @@ CREATE TABLE correspondences (
|
||||
created_at DATETIME DEFAULT CURRENT_TIMESTAMP COMMENT 'วันที่สร้าง',
|
||||
created_by INT COMMENT 'ผู้สร้าง',
|
||||
deleted_at DATETIME NULL COMMENT 'สำหรับ Soft Delete',
|
||||
INDEX idx_doc_number (document_number),
|
||||
INDEX idx_corr_number (correspondence_number),
|
||||
INDEX idx_deleted_at (deleted_at),
|
||||
INDEX idx_created_by (created_by),
|
||||
FOREIGN KEY (correspondence_type_id) REFERENCES correspondence_types (id) ON DELETE RESTRICT,
|
||||
FOREIGN KEY (project_id) REFERENCES projects (id) ON DELETE CASCADE,
|
||||
FOREIGN KEY (originator_id) REFERENCES organizations (id) ON DELETE SET NULL,
|
||||
FOREIGN KEY (created_by) REFERENCES users (user_id) ON DELETE SET NULL,
|
||||
FOREIGN KEY (originator_id) REFERENCES organizations (id) ON DELETE
|
||||
SET NULL,
|
||||
FOREIGN KEY (created_by) REFERENCES users (user_id) ON DELETE
|
||||
SET NULL,
|
||||
-- Foreign Key ที่รวมเข้ามาจาก ALTER (ระบุชื่อ Constraint ตามที่ต้องการ)
|
||||
CONSTRAINT fk_corr_discipline FOREIGN KEY (discipline_id) REFERENCES disciplines (id) ON DELETE SET NULL,
|
||||
UNIQUE KEY uq_corr_no_per_project (project_id, correspondence_number)
|
||||
CONSTRAINT fk_corr_discipline FOREIGN KEY (discipline_id) REFERENCES disciplines (id) ON DELETE
|
||||
SET NULL,
|
||||
UNIQUE KEY uq_corr_no_per_project (project_id, correspondence_number)
|
||||
) ENGINE = InnoDB DEFAULT CHARSET = utf8mb4 COLLATE = utf8mb4_general_ci COMMENT = 'ตาราง "แม่" ของเอกสารโต้ตอบ เก็บข้อมูลที่ไม่เปลี่ยนตาม Revision';
|
||||
|
||||
-- ตารางเชื่อมผู้รับ (TO/CC) สำหรับเอกสารแต่ละฉบับ (M:N)
|
||||
@@ -1545,11 +1563,12 @@ CREATE INDEX idx_wf_hist_user ON workflow_histories (action_by_user_id);
|
||||
|
||||
-- Checkpoint Table:
|
||||
CREATE TABLE IF NOT EXISTS migration_progress (
|
||||
batch_id VARCHAR(50) PRIMARY KEY,
|
||||
last_processed_index INT DEFAULT 0,
|
||||
status ENUM('RUNNING','COMPLETED','FAILED') DEFAULT 'RUNNING',
|
||||
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP
|
||||
batch_id VARCHAR(50) PRIMARY KEY,
|
||||
last_processed_index INT DEFAULT 0,
|
||||
STATUS ENUM('RUNNING', 'COMPLETED', 'FAILED') DEFAULT 'RUNNING',
|
||||
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP
|
||||
);
|
||||
|
||||
-- Idempotency Table :
|
||||
CREATE TABLE IF NOT EXISTS import_transactions (
|
||||
id INT AUTO_INCREMENT PRIMARY KEY,
|
||||
@@ -1560,6 +1579,7 @@ CREATE TABLE IF NOT EXISTS import_transactions (
|
||||
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||
INDEX idx_idem_key (idempotency_key)
|
||||
);
|
||||
|
||||
-- ============================================================
|
||||
-- 5. PARTITIONING PREPARATION (Advance - Optional)
|
||||
-- ============================================================
|
||||
@@ -2049,6 +2069,87 @@ CREATE INDEX idx_correspondences_type_project ON correspondences (correspondence
|
||||
|
||||
CREATE INDEX idx_corr_revisions_current_status ON correspondence_revisions (is_current, correspondence_status_id);
|
||||
|
||||
-- =====================================================
|
||||
-- Migration Tracking Tables (Temporary)
|
||||
-- =====================================================
|
||||
-- Checkpoint
|
||||
CREATE TABLE IF NOT EXISTS migration_progress (
|
||||
batch_id VARCHAR(50) PRIMARY KEY,
|
||||
last_processed_index INT DEFAULT 0,
|
||||
STATUS ENUM('RUNNING', 'COMPLETED', 'FAILED') DEFAULT 'RUNNING',
|
||||
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP
|
||||
);
|
||||
|
||||
-- Review Queue (Temporary — ไม่ใช่ Business Schema)
|
||||
CREATE TABLE IF NOT EXISTS migration_review_queue (
|
||||
id INT AUTO_INCREMENT PRIMARY KEY,
|
||||
document_number VARCHAR(100) NOT NULL,
|
||||
title TEXT,
|
||||
original_title TEXT,
|
||||
ai_suggested_category VARCHAR(50),
|
||||
ai_confidence DECIMAL(4, 3),
|
||||
ai_issues JSON,
|
||||
review_reason VARCHAR(255),
|
||||
STATUS ENUM('PENDING', 'APPROVED', 'REJECTED') DEFAULT 'PENDING',
|
||||
reviewed_by VARCHAR(100),
|
||||
reviewed_at TIMESTAMP NULL,
|
||||
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||
UNIQUE KEY uq_doc_number (document_number)
|
||||
);
|
||||
|
||||
-- Error Log
|
||||
CREATE TABLE IF NOT EXISTS migration_errors (
|
||||
id INT AUTO_INCREMENT PRIMARY KEY,
|
||||
batch_id VARCHAR(50),
|
||||
document_number VARCHAR(100),
|
||||
error_type ENUM(
|
||||
'FILE_NOT_FOUND',
|
||||
'AI_PARSE_ERROR',
|
||||
'API_ERROR',
|
||||
'DB_ERROR',
|
||||
'UNKNOWN'
|
||||
),
|
||||
error_message TEXT,
|
||||
raw_ai_response TEXT,
|
||||
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||
INDEX idx_batch_id (batch_id),
|
||||
INDEX idx_error_type (error_type)
|
||||
);
|
||||
|
||||
-- Fallback State
|
||||
CREATE TABLE IF NOT EXISTS migration_fallback_state (
|
||||
id INT AUTO_INCREMENT PRIMARY KEY,
|
||||
batch_id VARCHAR(50) UNIQUE,
|
||||
recent_error_count INT DEFAULT 0,
|
||||
is_fallback_active BOOLEAN DEFAULT FALSE,
|
||||
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP
|
||||
);
|
||||
|
||||
-- Idempotency (Patch)
|
||||
CREATE TABLE IF NOT EXISTS import_transactions (
|
||||
id INT AUTO_INCREMENT PRIMARY KEY,
|
||||
idempotency_key VARCHAR(255) UNIQUE NOT NULL,
|
||||
document_number VARCHAR(100),
|
||||
batch_id VARCHAR(100),
|
||||
status_code INT DEFAULT 201,
|
||||
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||
INDEX idx_idem_key (idempotency_key)
|
||||
);
|
||||
|
||||
-- Daily Summary
|
||||
CREATE TABLE IF NOT EXISTS migration_daily_summary (
|
||||
id INT AUTO_INCREMENT PRIMARY KEY,
|
||||
batch_id VARCHAR(50),
|
||||
summary_date DATE,
|
||||
total_processed INT DEFAULT 0,
|
||||
auto_ingested INT DEFAULT 0,
|
||||
sent_to_review INT DEFAULT 0,
|
||||
rejected INT DEFAULT 0,
|
||||
errors INT DEFAULT 0,
|
||||
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||
UNIQUE KEY uq_batch_date (batch_id, summary_date)
|
||||
);
|
||||
|
||||
CREATE INDEX idx_corr_revisions_correspondence_current ON correspondence_revisions (correspondence_id, is_current);
|
||||
|
||||
-- Indexes for v_current_rfas performance
|
||||
|
||||
@@ -18,7 +18,6 @@
|
||||
-- 2.1 username = migration_bot
|
||||
-- 2.2
|
||||
-- ==========================================================
|
||||
|
||||
INSERT INTO organization_roles (id, role_name)
|
||||
VALUES (1, 'OWNER'),
|
||||
(2, 'DESIGNER'),
|
||||
@@ -26,6 +25,7 @@ VALUES (1, 'OWNER'),
|
||||
(4, 'CONTRACTOR'),
|
||||
(5, 'THIRD PARTY'),
|
||||
(6, 'GUEST');
|
||||
|
||||
INSERT INTO organizations (
|
||||
id,
|
||||
organization_code,
|
||||
@@ -100,6 +100,7 @@ VALUES (1, 'กทท.', 'การท่าเรือแห่งประเ
|
||||
),
|
||||
(31, 'EN', 'Third Party Environment', 5),
|
||||
(32, 'CAR', 'Third Party Fishery Care', 5);
|
||||
|
||||
-- Seed project
|
||||
INSERT INTO projects (project_code, project_name)
|
||||
VALUES (
|
||||
@@ -126,6 +127,7 @@ VALUES (
|
||||
'LCBP3-EN',
|
||||
'โครงการพัฒนาท่าเรือแหลมฉบัง ระยะที่ 3 (ส่วนที่ 4) งานก่อสร้าง'
|
||||
);
|
||||
|
||||
-- Seed contract
|
||||
-- ใช้ Subquery เพื่อดึง project_id มาเชื่อมโยง ทำให้ไม่ต้องมานั่งจัดการ ID ด้วยตัวเอง
|
||||
INSERT INTO contracts (
|
||||
@@ -204,6 +206,7 @@ VALUES (
|
||||
),
|
||||
TRUE
|
||||
);
|
||||
|
||||
-- Seed user
|
||||
-- Initial SUPER_ADMIN user
|
||||
INSERT INTO users (
|
||||
@@ -252,18 +255,32 @@ VALUES (
|
||||
'$2b$10$MpKnf1UEvlu8hZcqMkhMsuWG3gYD/priWTUr71GpF/uuroaGxtose',
|
||||
'Viewer',
|
||||
'สคฉ.03',
|
||||
'viewer01 @example.com',
|
||||
'viewer01@example.com',
|
||||
NULL,
|
||||
10
|
||||
);
|
||||
|
||||
INSERT INTO users (username, email, role, is_active)
|
||||
VALUES (
|
||||
INSERT INTO users (
|
||||
user_id,
|
||||
username,
|
||||
password_hash,
|
||||
first_name,
|
||||
last_name,
|
||||
email,
|
||||
line_id,
|
||||
primary_organization_id
|
||||
)
|
||||
VALUES (
|
||||
5,
|
||||
'migration_bot',
|
||||
'$2b$10$MpKnf1UEvlu8hZcqMkhMsuWG3gYD/priWTUr71GpF/uuroaGxtose',
|
||||
'Migration',
|
||||
'Bot',
|
||||
'migration@system.internal',
|
||||
'SYSTEM_ADMIN',
|
||||
TRUE
|
||||
NULL,
|
||||
1
|
||||
);
|
||||
|
||||
-- ==========================================================
|
||||
-- Seed Roles (บทบาทพื้นฐาน 5 บทบาท ตาม Req 4.3)
|
||||
-- ==========================================================
|
||||
@@ -317,6 +334,7 @@ VALUES (
|
||||
'Contract',
|
||||
'ผู้ดูแลสัญญา: จัดการสมาชิกในสัญญา, สร้าง / จัดการข้อมูลหลักเฉพาะสัญญา, และอนุมัติเอกสารในสัญญา'
|
||||
);
|
||||
|
||||
-- ==========================================================
|
||||
-- Seed Role-Permissions Mapping (จับคู่สิทธิ์เริ่มต้น)
|
||||
-- ==========================================================
|
||||
@@ -343,8 +361,11 @@ VALUES (1, 1, 1, NULL, NULL, NULL, NULL),
|
||||
-- admin: Organization scope (org_id=1 = กทท.)
|
||||
(3, 3, 4, 41, NULL, NULL, 1),
|
||||
-- editor01: Editor role (role_id=4) at organization 41 (คคง.), assigned by superadmin
|
||||
(4, 4, 5, 10, NULL, NULL, 1);
|
||||
-- viewer01: Viewer role (role_id=5) at organization 10 (สคฉ.03), assigned by superadmin
|
||||
(4, 4, 5, 10, NULL, NULL, 1),
|
||||
-- viewer01: Viewer role (role_id=5) at organization 10 (สคฉ.03), assigned by superadmin
|
||||
(5, 5, 1, NULL, NULL, NULL, 1);
|
||||
|
||||
-- migration_bot: Superadmin role (role_id=1) for migration scripts, assigned by superadmin
|
||||
-- =====================================================
|
||||
-- == 4. การเชื่อมโยงโครงการกับองค์กร (project_organizations) ==
|
||||
-- =====================================================
|
||||
@@ -369,6 +390,7 @@ WHERE organization_code IN (
|
||||
'EN',
|
||||
'CAR'
|
||||
);
|
||||
|
||||
-- โครงการย่อย (LCBP3C1) จะมีเฉพาะองค์กรที่เกี่ยวข้อง
|
||||
INSERT INTO project_organizations (project_id, organization_id)
|
||||
SELECT (
|
||||
@@ -385,6 +407,7 @@ WHERE organization_code IN (
|
||||
'คคง.',
|
||||
'ผรม.1 '
|
||||
);
|
||||
|
||||
-- ทำเช่นเดียวกันสำหรับโครงการอื่นๆ (ตัวอย่าง)
|
||||
INSERT INTO project_organizations (project_id, organization_id)
|
||||
SELECT (
|
||||
@@ -401,6 +424,7 @@ WHERE organization_code IN (
|
||||
'คคง.',
|
||||
'ผรม.2'
|
||||
);
|
||||
|
||||
-- =====================================================
|
||||
-- == 5. การเชื่อมโยงสัญญากับองค์กร (contract_organizations) ==
|
||||
-- =====================================================
|
||||
@@ -432,6 +456,7 @@ VALUES (
|
||||
),
|
||||
'Designer'
|
||||
);
|
||||
|
||||
-- สัญญาที่ปรึกษาควบคุมงาน (PSLCBP3)
|
||||
INSERT INTO contract_organizations (contract_id, organization_id, role_in_contract)
|
||||
VALUES (
|
||||
@@ -460,6 +485,7 @@ VALUES (
|
||||
),
|
||||
'Consultant'
|
||||
);
|
||||
|
||||
-- สัญญางานก่อสร้าง ส่วนที่ 1 (LCBP3-C1)
|
||||
INSERT INTO contract_organizations (contract_id, organization_id, role_in_contract)
|
||||
VALUES (
|
||||
@@ -488,6 +514,7 @@ VALUES (
|
||||
),
|
||||
'Contractor'
|
||||
);
|
||||
|
||||
-- สัญญางานก่อสร้าง ส่วนที่ 2 (LCBP3-C2)
|
||||
INSERT INTO contract_organizations (contract_id, organization_id, role_in_contract)
|
||||
VALUES (
|
||||
@@ -516,6 +543,7 @@ VALUES (
|
||||
),
|
||||
'Contractor'
|
||||
);
|
||||
|
||||
-- สัญญาตรวจสอบสิ่งแวดล้อม (LCBP3-EN)
|
||||
INSERT INTO contract_organizations (contract_id, organization_id, role_in_contract)
|
||||
VALUES (
|
||||
@@ -544,6 +572,7 @@ VALUES (
|
||||
),
|
||||
'Consultant'
|
||||
);
|
||||
|
||||
-- Seed correspondence_status
|
||||
INSERT INTO correspondence_status (
|
||||
status_code,
|
||||
@@ -574,6 +603,7 @@ VALUES ('DRAFT', 'Draft', 10, 1),
|
||||
('CCBDSN', 'Canceled by Designer', 92, 1),
|
||||
('CCBCSC', 'Canceled by CSC', 93, 1),
|
||||
('CCBCON', 'Canceled by Contractor', 94, 1);
|
||||
|
||||
-- Seed correspondence_types
|
||||
INSERT INTO correspondence_types (type_code, type_name, sort_order, is_active)
|
||||
VALUES ('RFA', 'Request for Approval', 1, 1),
|
||||
@@ -586,6 +616,7 @@ VALUES ('RFA', 'Request for Approval', 1, 1),
|
||||
('MOM', 'Minutes of Meeting', 8, 1),
|
||||
('NOTICE', 'Notice', 9, 1),
|
||||
('OTHER', 'Other', 10, 1);
|
||||
|
||||
-- Seed rfa_types
|
||||
INSERT INTO rfa_types (
|
||||
contract_id,
|
||||
@@ -1075,6 +1106,7 @@ SELECT id,
|
||||
'รายงานการฝึกปฏิบัติ'
|
||||
FROM contracts
|
||||
WHERE contract_code = 'LCBP3-C2';
|
||||
|
||||
-- Seed rfa_status_codes
|
||||
INSERT INTO rfa_status_codes (
|
||||
status_code,
|
||||
@@ -1089,6 +1121,7 @@ VALUES ('DFT', 'Draft', 'ฉบับร่าง', 1),
|
||||
('ASB', 'AS - Built', 'แบบก่อสร้างจริง', 30),
|
||||
('OBS', 'Obsolete', 'ไม่ใช้งาน', 80),
|
||||
('CC', 'Canceled', 'ยกเลิก', 99);
|
||||
|
||||
INSERT INTO rfa_approve_codes (
|
||||
approve_code,
|
||||
approve_name,
|
||||
@@ -1103,12 +1136,14 @@ VALUES ('1A', 'Approved by Authority', 10, 1),
|
||||
('3R', 'Revise and Resubmit', 32, 1),
|
||||
('4X', 'Reject', 40, 1),
|
||||
('5N', 'No Further Action', 50, 1);
|
||||
|
||||
-- Seed circulation_status_codes
|
||||
INSERT INTO circulation_status_codes (code, description, sort_order)
|
||||
VALUES ('OPEN', 'Open', 1),
|
||||
('IN_REVIEW', 'In Review', 2),
|
||||
('COMPLETED', 'ปCompleted', 3),
|
||||
('CANCELLED', 'Cancelled / Withdrawn', 9);
|
||||
|
||||
-- ตาราง "แม่" ของ RFA (มีความสัมพันธ์ 1:N กับ rfa_revisions)
|
||||
-- ==========================================================
|
||||
-- SEED DATA 6B.md (Disciplines, RFA Types, Sub Types)
|
||||
@@ -1372,6 +1407,7 @@ SELECT id,
|
||||
'Other'
|
||||
FROM contracts
|
||||
WHERE contract_code = 'LCBP3-C1';
|
||||
|
||||
-- LCBP3-C2
|
||||
INSERT INTO disciplines (
|
||||
contract_id,
|
||||
@@ -1616,6 +1652,7 @@ SELECT id,
|
||||
'Others'
|
||||
FROM contracts
|
||||
WHERE contract_code = 'LCBP3-C2';
|
||||
|
||||
-- 2. Seed ข้อมูล Correspondence Sub Types (Mapping RFA Types กับ Number)
|
||||
-- เนื่องจาก sub_type_code ตรงกับ RFA Type Code แต่ Req ต้องการ Mapping เป็น Number
|
||||
-- LCBP3-C1
|
||||
@@ -1666,6 +1703,7 @@ FROM contracts c,
|
||||
correspondence_types ct
|
||||
WHERE c.contract_code = 'LCBP3-C1'
|
||||
AND ct.type_code = 'RFA';
|
||||
|
||||
-- LCBP3-C2
|
||||
INSERT INTO correspondence_sub_types (
|
||||
contract_id,
|
||||
@@ -1713,6 +1751,7 @@ FROM contracts c,
|
||||
correspondence_types ct
|
||||
WHERE c.contract_code = 'LCBP3-C2'
|
||||
AND ct.type_code = 'RFA';
|
||||
|
||||
-- LCBP3-C3
|
||||
INSERT INTO correspondence_sub_types (
|
||||
contract_id,
|
||||
@@ -1760,6 +1799,7 @@ FROM contracts c,
|
||||
correspondence_types ct
|
||||
WHERE c.contract_code = 'LCBP3-C3'
|
||||
AND ct.type_code = 'RFA';
|
||||
|
||||
-- LCBP3-C4
|
||||
INSERT INTO correspondence_sub_types (
|
||||
contract_id,
|
||||
@@ -1807,6 +1847,7 @@ FROM contracts c,
|
||||
correspondence_types ct
|
||||
WHERE c.contract_code = 'LCBP3-C4'
|
||||
AND ct.type_code = 'RFA';
|
||||
|
||||
INSERT INTO `correspondences` (
|
||||
`id`,
|
||||
`correspondence_number`,
|
||||
@@ -1843,6 +1884,7 @@ VALUES (
|
||||
1,
|
||||
NULL
|
||||
);
|
||||
|
||||
INSERT INTO `correspondence_revisions` (
|
||||
`id`,
|
||||
`correspondence_id`,
|
||||
@@ -1881,6 +1923,7 @@ VALUES (
|
||||
1,
|
||||
NULL
|
||||
);
|
||||
|
||||
INSERT INTO `rfas` (
|
||||
`id`,
|
||||
`rfa_type_id`,
|
||||
@@ -1889,6 +1932,7 @@ INSERT INTO `rfas` (
|
||||
`deleted_at`
|
||||
)
|
||||
VALUES (2, 68, '2025-12-06 05:40:02', 1, NULL);
|
||||
|
||||
INSERT INTO `rfa_revisions` (
|
||||
`id`,
|
||||
`rfa_id`,
|
||||
@@ -1929,6 +1973,7 @@ VALUES (
|
||||
NULL,
|
||||
NULL
|
||||
);
|
||||
|
||||
-- ==========================================================
|
||||
-- 20. Workflow Definitions (Unified Workflow Engine)
|
||||
-- ==========================================================
|
||||
@@ -2165,6 +2210,7 @@ VALUES (
|
||||
NOW(),
|
||||
NOW()
|
||||
);
|
||||
|
||||
INSERT INTO `document_number_formats` (
|
||||
`id`,
|
||||
`project_id`,
|
||||
|
||||
@@ -1,15 +1,14 @@
|
||||
-- ==========================================================
|
||||
-- DMS DMS v0.5.0
|
||||
-- Database v5.1 - Seed contract_dwg data
|
||||
-- Server: Container Station on QNAPQNAP TS-473A
|
||||
-- Database service: MariaDB 10.11
|
||||
-- database ui: phpmyadmin 5-apache
|
||||
-- backend sevice: node.js
|
||||
-- frontend sevice: next.js
|
||||
-- reverse proxy: nginx 1.27-alpine
|
||||
-- DMS v1.8.0 Document Management System Database
|
||||
-- Seed Contract Drawing data
|
||||
-- Server: Container Station on QNAP TS-473A
|
||||
-- Database service: MariaDB 11.8
|
||||
-- database web ui: phpmyadmin 5-apache
|
||||
-- database development ui: DBeaver
|
||||
-- backend service: NestJS
|
||||
-- frontend service: next.js
|
||||
-- reverse proxy: jc21/nginx-proxy-manager:latest
|
||||
-- cron service: n8n
|
||||
-- scripts: alpine:3.20
|
||||
-- Notes:
|
||||
-- ==========================================================
|
||||
|
||||
INSERT INTO contract_drawing_volumes (project_id, volume_code, volume_name, description)
|
||||
@@ -1,8 +1,9 @@
|
||||
-- ==========================================================
|
||||
-- DMS v1.6.0 - Permissions Seed Data (REORGANIZED)
|
||||
-- File: specs/07-database/permissions-seed-data.sql
|
||||
-- DMS v1.8.0 - Permissions Seed Data (REORGANIZED)
|
||||
-- File: specs/07-database/lcbp3-v1.8.0-seed-permissions.sql
|
||||
-- Total Permissions: 85 (Reorganized with systematic ID allocation)
|
||||
-- Created: 2025-12-13
|
||||
-- Updated: 2026-02-28 (v1.8.0 merge)
|
||||
-- ==========================================================
|
||||
-- Clear existing data
|
||||
TRUNCATE TABLE role_permissions;
|
||||
@@ -1065,3 +1066,37 @@ VALUES -- Contract Management
|
||||
-- ==========================================================
|
||||
-- VERIFICATION: Run permissions-verification.sql after this
|
||||
-- ==========================================================
|
||||
|
||||
|
||||
-- ==========================================================
|
||||
-- MERGED FROM fix-project-permissions.sql (v1.8.0 Update)
|
||||
-- ==========================================================
|
||||
-- Fix Project Permissions
|
||||
-- File: specs/07-database/fix-project-permissions.sql
|
||||
-- 1. Ensure project.view permission exists
|
||||
INSERT IGNORE INTO permissions (
|
||||
permission_id,
|
||||
permission_name,
|
||||
description,
|
||||
module,
|
||||
is_active
|
||||
)
|
||||
VALUES (
|
||||
202,
|
||||
'project.view',
|
||||
'ดูรายการโครงการ',
|
||||
'project',
|
||||
1
|
||||
);
|
||||
-- 2. Grant project.view to Superadmin (Role 1)
|
||||
INSERT IGNORE INTO role_permissions (role_id, permission_id)
|
||||
VALUES (1, 202);
|
||||
-- 3. Grant project.view to Organization Admin (Role 2)
|
||||
INSERT IGNORE INTO role_permissions (role_id, permission_id)
|
||||
VALUES (2, 202);
|
||||
-- 4. Grant project.view to Project Manager (Role 6)
|
||||
INSERT IGNORE INTO role_permissions (role_id, permission_id)
|
||||
VALUES (6, 202);
|
||||
-- 5. Grant project.view to Viewer (Role 5)
|
||||
INSERT IGNORE INTO role_permissions (role_id, permission_id)
|
||||
VALUES (5, 202);
|
||||
@@ -1,3 +1,15 @@
|
||||
-- ==========================================================
|
||||
-- DMS v1.8.0 Document Management System Database
|
||||
-- Seed Shop Drawing data
|
||||
-- Server: Container Station on QNAP TS-473A
|
||||
-- Database service: MariaDB 11.8
|
||||
-- database web ui: phpmyadmin 5-apache
|
||||
-- database development ui: DBeaver
|
||||
-- backend service: NestJS
|
||||
-- frontend service: next.js
|
||||
-- reverse proxy: jc21/nginx-proxy-manager:latest
|
||||
-- cron service: n8n
|
||||
-- ==========================================================
|
||||
INSERT INTO shop_drawing_sub_categories(
|
||||
project_id,
|
||||
sub_category_code,
|
||||
216
specs/03-Data-and-Storage/n8n.workflow
Normal file
216
specs/03-Data-and-Storage/n8n.workflow
Normal file
@@ -0,0 +1,216 @@
|
||||
{
|
||||
"meta": {
|
||||
"instanceId": "lcbp3-migration"
|
||||
},
|
||||
"nodes": [
|
||||
{
|
||||
"parameters": {},
|
||||
"id": "trigger-1",
|
||||
"name": "When clicking ‘Execute Workflow’",
|
||||
"type": "n8n-nodes-base.manualTrigger",
|
||||
"typeVersion": 1,
|
||||
"position": [0, 0]
|
||||
},
|
||||
{
|
||||
"parameters": {
|
||||
"operation": "read",
|
||||
"fileFormat": "xlsx",
|
||||
"options": {}
|
||||
},
|
||||
"id": "spreadsheet-1",
|
||||
"name": "Read Excel Data",
|
||||
"type": "n8n-nodes-base.spreadsheetFile",
|
||||
"typeVersion": 2,
|
||||
"position": [200, 0]
|
||||
},
|
||||
{
|
||||
"parameters": {
|
||||
"batchSize": 10,
|
||||
"options": {}
|
||||
},
|
||||
"id": "split-in-batches-1",
|
||||
"name": "Split In Batches",
|
||||
"type": "n8n-nodes-base.splitInBatches",
|
||||
"typeVersion": 3,
|
||||
"position": [400, 0]
|
||||
},
|
||||
{
|
||||
"parameters": {
|
||||
"jsCode": "const item = $input.first();\n\nconst prompt = `You are a Document Controller for a large construction project.\nYour task is to validate document metadata.\nYou MUST respond ONLY with valid JSON. No explanation, no markdown, no extra text.\n\nDocument Number: ${item.json.document_number}\nTitle: ${item.json.title}\nCategory List: [\"Correspondence\",\"RFA\",\"Drawing\",\"Transmittal\",\"Report\",\"Other\"]\n\nRespond ONLY with this exact JSON structure:\n{\n \"is_valid\": true,\n \"confidence\": 0.95,\n \"suggested_category\": \"Correspondence\",\n \"detected_issues\": [],\n \"suggested_title\": null\n}`;\n\nreturn [{\n json: {\n ...item.json,\n ollama_payload: {\n model: \"llama3.2:3b\",\n format: \"json\",\n stream: false,\n prompt: prompt\n }\n }\n}];"
|
||||
},
|
||||
"id": "code-1",
|
||||
"name": "Build Prompt",
|
||||
"type": "n8n-nodes-base.code",
|
||||
"typeVersion": 2,
|
||||
"position": [620, 0]
|
||||
},
|
||||
{
|
||||
"parameters": {
|
||||
"method": "POST",
|
||||
"url": "http://192.168.20.100:11434/api/generate",
|
||||
"sendBody": true,
|
||||
"specifyBody": "json",
|
||||
"jsonBody": "={{ $json.ollama_payload }}",
|
||||
"options": {
|
||||
"timeout": 30000
|
||||
}
|
||||
},
|
||||
"id": "http-1",
|
||||
"name": "Ollama Local API",
|
||||
"type": "n8n-nodes-base.httpRequest",
|
||||
"typeVersion": 4.1,
|
||||
"position": [840, 0]
|
||||
},
|
||||
{
|
||||
"parameters": {
|
||||
"jsCode": "const items = $input.all();\nconst parsed = [];\n\nfor (const item of items) {\n try {\n let raw = item.json.response || '';\n raw = raw.replace(/```json/gi, '').replace(/```/g, '').trim();\n const aiResult = JSON.parse(raw);\n parsed.push({ json: { ...item.json, ai_result: aiResult } });\n } catch (err) {\n parsed.push({ json: { ...item.json, ai_result: { confidence: 0, is_valid: false, error: err.message } } });\n }\n}\nreturn parsed;"
|
||||
},
|
||||
"id": "code-2",
|
||||
"name": "Parse JSON",
|
||||
"type": "n8n-nodes-base.code",
|
||||
"typeVersion": 2,
|
||||
"position": [1040, 0]
|
||||
},
|
||||
{
|
||||
"parameters": {
|
||||
"conditions": {
|
||||
"boolean": [
|
||||
{
|
||||
"value1": "={{ $json.ai_result.confidence >= 0.85 && $json.ai_result.is_valid }}",
|
||||
"value2": true
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"id": "if-1",
|
||||
"name": "Confidence >= 0.85?",
|
||||
"type": "n8n-nodes-base.if",
|
||||
"typeVersion": 1,
|
||||
"position": [1240, 0]
|
||||
},
|
||||
{
|
||||
"parameters": {
|
||||
"method": "POST",
|
||||
"url": "http://<YOUR_BACKEND_IP>:3000/api/migration/import",
|
||||
"sendHeaders": true,
|
||||
"headerParameters": {
|
||||
"parameters": [
|
||||
{
|
||||
"name": "Idempotency-Key",
|
||||
"value": "={{ $json.document_number }}:BATCH-001"
|
||||
},
|
||||
{
|
||||
"name": "Authorization",
|
||||
"value": "Bearer <YOUR_MIGRATION_TOKEN>"
|
||||
}
|
||||
]
|
||||
},
|
||||
"sendBody": true,
|
||||
"specifyBody": "json",
|
||||
"jsonBody": "={\n \"source_file_path\": \"/share/np-dms/staging_ai/{{$json.document_number}}.pdf\",\n \"document_number\": \"{{$json.document_number}}\",\n \"title\": \"{{$json.ai_result.suggested_title || $json.title}}\",\n \"category\": \"{{$json.ai_result.suggested_category}}\",\n \"revision\": 1, \n \"batch_id\": \"BATCH_001\",\n \"ai_confidence\": {{$json.ai_result.confidence}},\n \"ai_issues\": {{$json.ai_result.detected_issues}},\n \"legacy_document_number\": \"{{$json.legacy_number}}\"\n}",
|
||||
"options": {}
|
||||
},
|
||||
"id": "http-2",
|
||||
"name": "LCBP3 Backend (Auto Ingest)",
|
||||
"type": "n8n-nodes-base.httpRequest",
|
||||
"typeVersion": 4.1,
|
||||
"position": [1460, -100]
|
||||
},
|
||||
{
|
||||
"parameters": {
|
||||
"jsCode": "return [{ json: { message: \"Sent to Human Review Queue OR Check AI Error Log\", data: $input.first().json } }];"
|
||||
},
|
||||
"id": "code-3",
|
||||
"name": "Review Queue / Reject Log",
|
||||
"type": "n8n-nodes-base.code",
|
||||
"typeVersion": 2,
|
||||
"position": [1460, 100]
|
||||
}
|
||||
],
|
||||
"connections": {
|
||||
"When clicking ‘Execute Workflow’": {
|
||||
"main": [
|
||||
[
|
||||
{
|
||||
"node": "Read Excel Data",
|
||||
"type": "main",
|
||||
"index": 0
|
||||
}
|
||||
]
|
||||
]
|
||||
},
|
||||
"Read Excel Data": {
|
||||
"main": [
|
||||
[
|
||||
{
|
||||
"node": "Split In Batches",
|
||||
"type": "main",
|
||||
"index": 0
|
||||
}
|
||||
]
|
||||
]
|
||||
},
|
||||
"Split In Batches": {
|
||||
"main": [
|
||||
[
|
||||
{
|
||||
"node": "Build Prompt",
|
||||
"type": "main",
|
||||
"index": 0
|
||||
}
|
||||
]
|
||||
]
|
||||
},
|
||||
"Build Prompt": {
|
||||
"main": [
|
||||
[
|
||||
{
|
||||
"node": "Ollama Local API",
|
||||
"type": "main",
|
||||
"index": 0
|
||||
}
|
||||
]
|
||||
]
|
||||
},
|
||||
"Ollama Local API": {
|
||||
"main": [
|
||||
[
|
||||
{
|
||||
"node": "Parse JSON",
|
||||
"type": "main",
|
||||
"index": 0
|
||||
}
|
||||
]
|
||||
]
|
||||
},
|
||||
"Parse JSON": {
|
||||
"main": [
|
||||
[
|
||||
{
|
||||
"node": "Confidence >= 0.85?",
|
||||
"type": "main",
|
||||
"index": 0
|
||||
}
|
||||
]
|
||||
]
|
||||
},
|
||||
"Confidence >= 0.85?": {
|
||||
"main": [
|
||||
[
|
||||
{
|
||||
"node": "LCBP3 Backend (Auto Ingest)",
|
||||
"type": "main",
|
||||
"index": 0
|
||||
}
|
||||
],
|
||||
[
|
||||
{
|
||||
"node": "Review Queue / Reject Log",
|
||||
"type": "main",
|
||||
"index": 0
|
||||
}
|
||||
]
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,6 +1,6 @@
|
||||
-- ==========================================================
|
||||
-- Permission System Verification Queries
|
||||
-- File: specs/07-database/permissions-verification.sql
|
||||
-- Permission System Verification Queries (v1.8.0)
|
||||
-- File: specs/03-Data-and-Storage/permissions-verification.sql
|
||||
-- Purpose: Verify permissions setup after seed data deployment
|
||||
-- ==========================================================
|
||||
-- ==========================================================
|
||||
@@ -271,6 +271,8 @@ FROM (
|
||||
SELECT 'drawing.view'
|
||||
UNION
|
||||
SELECT 'workflow.action_review'
|
||||
UNION
|
||||
SELECT 'project.view'
|
||||
) required_perms
|
||||
LEFT JOIN permissions p USING (permission_name)
|
||||
ORDER BY permission_name;
|
||||
|
||||
@@ -10,12 +10,33 @@ x-logging: &default_logging
|
||||
max-size: "10m"
|
||||
max-file: "5"
|
||||
services:
|
||||
n8n-db:
|
||||
<<: [*restart_policy, *default_logging]
|
||||
image: postgres:16-alpine
|
||||
container_name: n8n-db
|
||||
environment:
|
||||
- POSTGRES_USER=n8n
|
||||
- POSTGRES_PASSWORD=Np721220$
|
||||
- POSTGRES_DB=n8n
|
||||
volumes:
|
||||
- "/share/np-dms/n8n/postgres-data:/var/lib/postgresql/data"
|
||||
networks:
|
||||
lcbp3: {}
|
||||
healthcheck:
|
||||
test: ['CMD-SHELL', 'pg_isready -h localhost -U n8n -d n8n']
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
|
||||
n8n:
|
||||
<<: [*restart_policy, *default_logging]
|
||||
image: n8nio/n8n:1.78.0
|
||||
image: n8nio/n8n:latest
|
||||
container_name: n8n
|
||||
stdin_open: true
|
||||
tty: true
|
||||
depends_on:
|
||||
n8n-db:
|
||||
condition: service_healthy
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
@@ -38,14 +59,19 @@ services:
|
||||
N8N_DIAGNOSTICS_ENABLED: 'false'
|
||||
N8N_SECURE_COOKIE: 'true'
|
||||
N8N_ENCRYPTION_KEY: "9AAIB7Da9DW1qAhJE5/Bz4SnbQjeAngI"
|
||||
N8N_BASIC_AUTH_ACTIVE: 'true'
|
||||
N8N_BASIC_AUTH_USER: admin
|
||||
N8N_BASIC_AUTH_PASSWORD: Center#2025
|
||||
N8N_ENFORCE_SETTINGS_FILE_PERMISSIONS: 'true'
|
||||
GENERIC_TIMEZONE: "Asia/Bangkok"
|
||||
# DB: MySQL/MariaDB removed in n8n v1.x — now using SQLite (default)
|
||||
# Data is persisted in /home/node/.n8n (mounted volume below)
|
||||
DB_TYPE: sqlite
|
||||
# DB Setup
|
||||
DB_TYPE: postgresdb
|
||||
DB_POSTGRESDB_DATABASE: n8n
|
||||
DB_POSTGRESDB_HOST: n8n-db
|
||||
DB_POSTGRESDB_PORT: 5432
|
||||
DB_POSTGRESDB_USER: n8n
|
||||
DB_POSTGRESDB_PASSWORD: Np721220$
|
||||
# Data Prune
|
||||
EXECUTIONS_DATA_PRUNE: 'true'
|
||||
EXECUTIONS_DATA_MAX_AGE: 168
|
||||
EXECUTIONS_DATA_PRUNE_TIMEOUT: 60
|
||||
|
||||
ports:
|
||||
- "5678:5678"
|
||||
@@ -57,6 +83,10 @@ services:
|
||||
- "/share/np-dms/n8n/scripts:/scripts"
|
||||
- "/share/np-dms/n8n/data:/data"
|
||||
- "/var/run/docker.sock:/var/run/docker.sock"
|
||||
# read-only: อ่านไฟล์ PDF ต้นฉบับเท่านั้น
|
||||
- "/share/np-dms-as/Legacy:/share/np-dms/staging_ai:ro"
|
||||
# read-write: เขียน Log และ CSV ทั้งหมด
|
||||
- "/share/np-dms/n8n/migration_logs:/share/np-dms/n8n/migration_logs:rw"
|
||||
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "wget -qO- http://127.0.0.1:5678/healthz || exit 1"]
|
||||
@@ -70,6 +100,8 @@ networks:
|
||||
external: true
|
||||
|
||||
# สำหรับ n8n volumes
|
||||
# chown -R 1000:1000 /share/Container/n8n
|
||||
# chmod -R 755 /share/Container/n8n
|
||||
# chown -R 1000:1000 /share/np-dms/n8n
|
||||
# chmod -R 755 /share/np-dms/n8n3
|
||||
# chown -R 999:999 /share/np-dms/n8n/postgres-data
|
||||
# chmod -R 700 /share/np-dms/n8n/postgres-data
|
||||
|
||||
|
||||
@@ -9,9 +9,7 @@
|
||||
- [n8n Migration Setup Guide](../03-Data-and-Storage/03-05-n8n-migration-setup-guide.md)
|
||||
- [Software Architecture](../02-Architecture/02-02-software-architecture.md)
|
||||
- [Data Dictionary](../03-Data-and-Storage/03-01-data-dictionary.md)
|
||||
|
||||
> **Note:** ADR-017 is clarified and hardened by ADR-018 regarding AI physical isolation. Category Enum system-driven, Idempotency Contract, Duplicate Handling Clarification, Storage Enforcement, Audit Log Enhancement, Review Queue Integration, Revision Drift Protection, Execution Time, Encoding Normalization, Security Hardening, AI Physical Isolation (ASUSTOR).
|
||||
|
||||
> **Note:** ADR-017 is clarified and hardened by ADR-018 regarding AI physical isolation. Category Enum system-driven, Idempotency Contract, Duplicate Handling Clarification, Storage Enforcement, Audit Log Enhancement, Review Queue Integration, Revision Drift Protection, Execution Time, Encoding Normalization, Security Hardening, Orchestrator on QNAP, AI Physical Isolation (Desktop Desk-5439).
|
||||
---
|
||||
|
||||
## Context and Problem Statement
|
||||
@@ -84,18 +82,18 @@
|
||||
|
||||
## Implementation Summary
|
||||
|
||||
| Component | รายละเอียด |
|
||||
| ---------------------- | ------------------------------------------------------------- |
|
||||
| Migration Orchestrator | n8n (Docker บน ASUSTOR NAS) |
|
||||
| AI Model Primary | Ollama `llama3.2:3b` |
|
||||
| AI Model Fallback | Ollama `mistral:7b-instruct-q4_K_M` |
|
||||
| Hardware | ASUSTOR NAS (AI Processing Only) |
|
||||
| Data Ingestion | RESTful API + Migration Token (7 วัน) + Idempotency-Key Header |
|
||||
| Concurrency | Sequential — 1 Request/ครั้ง, Delay 2 วินาที |
|
||||
| Checkpoint | MariaDB `migration_progress` |
|
||||
| Fallback | Auto-switch Model เมื่อ Error ≥ Threshold |
|
||||
| Storage | Backend StorageService เท่านั้น — ห้าม move file โดยตรง |
|
||||
| Expected Runtime | ~16.6 ชั่วโมง (~3–4 คืน) สำหรับ 20,000 records |
|
||||
| Component | รายละเอียด |
|
||||
| ---------------------- | ------------------------------------------------------------------------------- |
|
||||
| Migration Orchestrator | n8n (Docker บน QNAP NAS) |
|
||||
| AI Model Primary | Ollama `llama3.2:3b` |
|
||||
| AI Model Fallback | Ollama `mistral:7b-instruct-q4_K_M` |
|
||||
| Hardware | QNAP NAS (Orchestrator) + Desktop Desk-5439 (AI Processing, RTX 2060 SUPER 8GB) |
|
||||
| Data Ingestion | RESTful API + Migration Token (7 วัน) + Idempotency-Key Header |
|
||||
| Concurrency | Sequential — 1 Request/ครั้ง, Delay 2 วินาที |
|
||||
| Checkpoint | MariaDB `migration_progress` |
|
||||
| Fallback | Auto-switch Model เมื่อ Error ≥ Threshold |
|
||||
| Storage | Backend StorageService เท่านั้น — ห้าม move file โดยตรง |
|
||||
| Expected Runtime | ~16.6 ชั่วโมง (~3–4 คืน) สำหรับ 20,000 records |
|
||||
|
||||
---
|
||||
|
||||
|
||||
Reference in New Issue
Block a user