01. System Architecture & Ingestion
First, let me walk you through the end-to-end automated architecture.
The entry point is an HTTP Webhook trigger. When the LinkedIn scraper finishes running, it sends the raw payload directly to our n8n orchestration pipeline.
Before any processing, the payload passes through a validation and sanitization node to strip out messy HTML markup and verify non-empty text, preventing invalid API calls.
02. AI Processing & Schema Validation
Next comes the core analytical engine. The clean job description is passed into the Kimi API using a tailored system prompt.
It extracts core responsibilities, maps hard and soft skills, assigns relative weights, and flags any unclassified terms as Unmatched Skills.
To enforce database integrity, the LLM's response goes through a strict Schema Validation node before entering production storage.
03. Technical Pivot & Cost Efficiency
Initially, I explored fine-tuning an open-source model using ~120,000 Kaggle records. However, training overhead and lower accuracy made it impractical.
By pivoting to prompt engineering with the Kimi API and enforcing structured JSON output via n8n, I achieved significantly higher precision and consistency.
The operational cost is down to approximately one cent per record, making it scalable for large volume data ingestion.
04. Storage, Audit Queue & Completion
Finally, validated analyses are stored directly into MongoDB.
If any unmatched skills are detected during processing, a conditional branch routes those records to a dedicated Audit Queue for manual review and dictionary refinement.
Once complete, the pipeline returns a 200 OK response to the caller, completing the end-to-end execution.
