How to create a pipeline where i take each Instagram post and extract

How to create a pipeline where i take each Instagram post and extract its insights at 24 hour and 15 days and then at day 30.

Checking Date & Time

Asking for Preferences

Running Command

Writing File

All files are ready. Here's the complete picture:

Project files

file

File ·

instagram-insights-pipeline/
├── index.js          ← Entry point (Express server + optional inline cron)
├── package.json
├── schema.sql        ← DB schema (posts, insight_jobs, insights tables)
├── .env.example      ← Copy to .env and fill in your tokens
├── README.md
└── src/
    ├── webhook.js    ← Receives Instagram webhook events
    ├── scheduler.js  ← Registers 3 jobs per new post
    ├── runner.js     ← Cron job: fetches due insights from IG API
    ├── instagram.js  ← Graph API client (no external HTTP libs)
    └── db.js         ← SQLite adapter (swap for your own store)

How each piece fits together

Stage	File	What it does
Detection	`webhook.js`	Meta sends a `media` event → extract `media_id`
Registration	`scheduler.js`	`registerPost()` → insert post + 3 jobs (24h / 15d / 30d)
Execution	`runner.js`	Every 5 min → `SELECT jobs WHERE run_at <= NOW()` → fetch API → save
API	`instagram.js`	Calls `/{media_id}/insights` with correct metrics per media type
Storage	`db.js`	SQLite adapter — swap out for your existing DB

Setup steps

# 1. Install
npm install

# 2. Configure
cp .env.example .env
# Fill in: IG_ACCESS_TOKEN, IG_ACCOUNT_ID, WEBHOOK_VERIFY_TOKEN

# 3. Start (server + embedded cron every 5 min)
npm run start:cron

Then in Meta Developer Dashboard:
Webhooks → Instagram → Subscribe to media field → set callback to https://your-domain.com/webhook/instagram

Key design decisions

Idempotent — calling registerPost() twice for the same post ID is safe (no duplicate jobs)
Exact timing — jobs are scheduled from published_at, not from when you discovered the post
Retry logic — failed API calls retry with exponential backoff (5m → 30m → give up after 3 attempts)
Swap your DB — db.js exposes a clean interface; replace with Postgres/MySQL/your ORM without touching any other file
Media type aware — Instagram returns different metric names for VIDEO, REEL, STORY, IMAGE; instagram.js handles this automatically

This is a shared conversation. Sign in to Orris to start your own chat.