Scrape your competitors’ posts. Transcribe their videos. Hand it all to Claude. Walk away with what’s already working in your niche, what nobody is touching, and a 90-day content calendar built from real engagement data, not vibes.
Most creators waste their best hours staring at a blank doc trying to invent ideas. Your next series is already public. The hits are in your competitors’ top posts, and the gaps are in what they keep missing. You just need a way to read it at scale.
Claude can’t pull posts from Instagram, TikTok or YouTube by itself. Apify can. It grabs the posts, captions, engagement numbers and transcripts from 5 to 10 accounts, and Claude does the analysis.
Content creator Marc Rutten used Apify’s Instagram Scraper to show brands why their big follower and like counts weren’t turning into sales. This is the same idea, tuned for creators and small business owners running their own content.
First time through, block about 90 minutes. After that it goes much faster because everything is set up.
01
Sign up for Apify
Create a free account at apify.com. No credit card. The Free plan includes $5 of usage a month, which resets each cycle and covers a small run like this one. (Apify pricing)
02
Pick your competitors with Claude
Get the list right before you spend a cent. This prompt checks your picks, adds direct and adjacent accounts, and hands you the final 8 in the exact format Apify wants.
03
Optional shortcut: connect Apify to Claude
Apify has a Claude connector. Search "Apify" in the connector directory, or add a custom connector with the URL https://mcp.apify.com, then sign in to Apify. Now you can ask Claude to run the scrapers below for you. It works in claude.ai and the desktop app. (Apify’s setup steps)
04
Scrape Instagram
Open apify/instagram-scraper. Paste each competitor’s profile URL into Instagram URLs, set Results limit per URL to 30 and click Start. You get the caption, hashtags, likes, comments, views, URL and timestamp for every post.
Cost: on the Free plan it’s $0.0027 per result, so 8 accounts at 30 posts each (240 posts) is about $0.65.
05
Scrape TikTok
Open clockworks/tiktok-scraper. Add the usernames under Usernames, set Number of videos per hashtag, profile or search to 30, and turn on Subtitles & Transcription so you get the words, not just the caption.
Cost: about $0.0037 per video on the Free plan, plus extra for transcription. Always cap the number of videos so one run can’t eat your whole credit.
06
Scrape YouTube
If anyone on your list has a channel, use streamers/youtube-scraper. Paste channel URLs into Direct URLs, cap the results, and use its Transcription & subtitles option for transcripts. About $0.004 per video on the Free plan.
07
Transcribe the top Reels and grab comments
Captions are the wrapper. The video is the product, so get the words. Paste the URLs of each competitor’s top Reels into invideoiq/video-transcriber ($0.035 per video). On the Free plan it transcribes 3 videos per run, so do your 3 best per account.
Then run the top posts through apify/instagram-comment-scraper. The comments are where people ask for what nobody is making.
08
Export and upload to Claude
In each Apify dataset, click Export and download a CSV. In Claude, make a project called "Gap Finder," and add every CSV to the project files. Keep code execution on (Settings, then Capabilities) so Claude can actually crunch the numbers.
09
Run the Gap Finder prompt
This is the deliverable the whole workflow was built for. Patterns to ride, gaps to own, a 12-piece calendar with hooks and a sharper positioning line, all tied to real posts.
Run It Once A Quarter
One run gives you 90 days of direction. Four a year and you have a content engine that compounds. The first run is the hard one, so block 90 minutes this weekend.
Once you have your ideas, pair them with Stop using AI to post more, start using AI to post better for the writing voice rules, and 10 Claude Connectors to get the content out the door faster.