← History

Diagnostics

Deployment health

Confirms the configuration this app needs to pull leads. Values are never shown — only whether each is set correctly.

AI Ark API key (AI_ARK_API_KEY)

configured

OK

Public base URL (PUBLIC_BASE_URL)

https://ai-ark-lead-scraping.vercel.app

OK

Storage driver

kv

OK

KV reachable (live check)

reachable

OK

Webhook callback pattern

https://ai-ark-lead-scraping.vercel.app/api/webhooks/ai-ark/<per-job-token>

AI Ark POSTs results here. If it's unreachable (e.g. Vercel Deployment Protection), the app now also polls AI Ark directly, so results still arrive.

—

Dedup memory

116,075 leads already scraped

Every new search excludes these people at the source, so AI Ark won't return or charge for them again. Capped at 90,000 per search (10 slots minus 1 saved list); warns above 72,000.

Check

Saved recurring search: seen IDs vs exclusion lists

116,075 relevant seen IDs, room for 90,000, 30,000 would be left out

Relevant = people this search could return again (found by any overlapping search, or imported). While any are left out, exports are refused instead of re-pulling them. Free a list slot by removing an unneeded saved list from AI_ARK_EXCLUDE_LIST_IDS (each takes one of AI Ark's 10), or raise EXCLUSION_MAX_LEFT_OUT as a deliberate, temporary trade-off.

OK

Recurring pulls today

running normally

A recurring export that returns under 25% new leads halts the rest of that day (New York time).

—

Seen AI Ark people IDs

116,075

Stored in insertion order so 10,000-ID exclusion chunks keep stable boundaries.

—

Exclusion lists built today

0 / 50

AI Ark allows 50 list creations per day (UTC). Full 10,000-ID lists are rebuilt at most every 20h; between rebuilds only one extra list is created for newly added IDs.

OK

Last exclusion build

11/18/2026, 4:42 PM EST (slice-c80682575f32)

—

BrandJet email exclusions

191,009 emails

Imported via POST /api/dedup/import. Dropped from recurring daily pull results after they arrive.

—

Configured AI Ark exclude lists

1 list: 2e614503-ac30-49ba-8e56-f26dc5dc29ab

Saved AI Ark lists (from AI_ARK_EXCLUDE_LIST_IDS) excluded on every search — e.g. your team's master lists. Also excluded on top of dedup memory.

—

Compact stored jobs

Re-saves every stored job, dropping the old bulky raw-payload field and rebuilding the lightweight job index. Run this if History ever shows “Storage is temporarily unavailable” with a “max request size exceeded” error on this page — that means stored job data grew too large for a bulk read. Safe to run more than once.

Seed dedup memory from past searches

Loads people from your previous exports into the dedup memory so leads pulled before deduplication existed are also excluded from now on. Safe to run more than once — searches whose AI Ark results have expired are skipped.

Download the full dedupe list

Everything the scraper remembers as already deduped, as one CSV: the people from every stored job (email, name, title, company, website, LinkedIn, which job found them), the remembered person IDs that have no stored details, and the BrandJet / imported emails. The file is built while it downloads and can take a minute or two. Right now: 116,075 person IDs and 191,009 emails.

Download dedupe list (CSV)

Re-tag seen people from job history

Records which search found each person, using the filters stored on past jobs, so a search only has to exclude people another overlapping search could return. Run it once after deploying; safe to run more than once.

Import old AI Ark platform CSVs

Upload a Contact Export-*.csv downloaded from the AI Ark platform. Its “AI Ark People ID” and “Email Business” columns are added to the dedup memory (emails only if there is no ID column). Safe to upload the same file again.

Verify a live job

To see exactly what AI Ark returns for a specific job, request /api/diag?jobId=<id> (the id is in the URL of any export’s results page). It live-polls AI Ark and reports the state, row count, and any error — the fastest way to confirm the integration end to end.