Diagnostics
Deployment health
Confirms the configuration this app needs to pull leads. Values are never shown — only whether each is set correctly.
Dedup memory is at 116,075 of this app’s 90,000-person exclusion capacity.
One search can exclude at most 90,000 people (9 lists × 10,000; each saved list in AI_ARK_EXCLUDE_LIST_IDS takes one of the 10 slots). Past that, lists are built per search slice, and a slice that is still too big can let already-scraped leads come back and be charged again.
AI Ark API key (AI_ARK_API_KEY)
configured
Public base URL (PUBLIC_BASE_URL)
https://ai-ark-lead-scraping.vercel.app
Storage driver
kv
KV reachable (live check)
reachable
Webhook callback pattern
https://ai-ark-lead-scraping.vercel.app/api/webhooks/ai-ark/<per-job-token>
AI Ark POSTs results here. If it's unreachable (e.g. Vercel Deployment Protection), the app now also polls AI Ark directly, so results still arrive.
Dedup memory
116,075 leads already scraped
Every new search excludes these people at the source, so AI Ark won't return or charge for them again. Capped at 90,000 per search (10 slots minus 1 saved list); warns above 72,000.
Saved recurring search: seen IDs vs exclusion lists
116,075 relevant seen IDs, room for 90,000, 30,000 would be left out
Relevant = people this search could return again (found by any overlapping search, or imported). While any are left out, exports are refused instead of re-pulling them. Free a list slot by removing an unneeded saved list from AI_ARK_EXCLUDE_LIST_IDS (each takes one of AI Ark's 10), or raise EXCLUSION_MAX_LEFT_OUT as a deliberate, temporary trade-off.
Recurring pulls today
running normally
A recurring export that returns under 25% new leads halts the rest of that day (New York time).
Seen AI Ark people IDs
116,075
Stored in insertion order so 10,000-ID exclusion chunks keep stable boundaries.
Exclusion lists built today
0 / 50
AI Ark allows 50 list creations per day (UTC). Full 10,000-ID lists are rebuilt at most every 20h; between rebuilds only one extra list is created for newly added IDs.
Last exclusion build
11/18/2026, 4:42 PM EST (slice-c80682575f32)
BrandJet email exclusions
191,009 emails
Imported via POST /api/dedup/import. Dropped from recurring daily pull results after they arrive.
Configured AI Ark exclude lists
1 list: 2e614503-ac30-49ba-8e56-f26dc5dc29ab
Saved AI Ark lists (from AI_ARK_EXCLUDE_LIST_IDS) excluded on every search — e.g. your team's master lists. Also excluded on top of dedup memory.
Compact stored jobs
Re-saves every stored job, dropping the old bulky raw-payload field and rebuilding the lightweight job index. Run this if History ever shows “Storage is temporarily unavailable” with a “max request size exceeded” error on this page — that means stored job data grew too large for a bulk read. Safe to run more than once.
Seed dedup memory from past searches
Loads people from your previous exports into the dedup memory so leads pulled before deduplication existed are also excluded from now on. Safe to run more than once — searches whose AI Ark results have expired are skipped.
Download the full dedupe list
Everything the scraper remembers as already deduped, as one CSV: the people from every stored job (email, name, title, company, website, LinkedIn, which job found them), the remembered person IDs that have no stored details, and the BrandJet / imported emails. The file is built while it downloads and can take a minute or two. Right now: 116,075 person IDs and 191,009 emails.
Download dedupe list (CSV)Re-tag seen people from job history
Records which search found each person, using the filters stored on past jobs, so a search only has to exclude people another overlapping search could return. Run it once after deploying; safe to run more than once.
Import old AI Ark platform CSVs
Upload a Contact Export-*.csv downloaded from the AI Ark platform. Its “AI Ark People ID” and “Email Business” columns are added to the dedup memory (emails only if there is no ID column). Safe to upload the same file again.
Verify a live job
To see exactly what AI Ark returns for a specific job, request /api/diag?jobId=<id> (the id is in the URL of any export’s results page). It live-polls AI Ark and reports the state, row count, and any error — the fastest way to confirm the integration end to end.