Scrape your competitors' posts. Transcribe their videos. Hand it all to Claude. Walk away with what's already working in your niche, what nobody is touching, and a 90-day content calendar built from real engagement data, not vibes.
Why this works
Most creators waste their best hours staring at a blank doc trying to invent ideas. Your next series is already public. The hits are in your competitors' top posts. The gaps are in what they keep missing. You just need a way to read it at scale, and that is exactly what this workflow is for.
The trick: Claude cannot scrape Instagram, TikTok, or YouTube on its own. So we use Apify to pull every post, every caption, every engagement count, and every video transcript from 5 to 10 competitors. Then we drop the whole dataset into Google Drive and let Claude do the actual analysis.
A content consultant named Marc Rutten documented this exact workflow on Apify's blog: scrape competitor accounts, find the engagement-to-revenue gap, sell the analysis back to brands. The version below is the same playbook, tuned for creators and small business owners running their own content.
First time through, expect about 90 minutes end-to-end. Every run after the first one takes 20 minutes because everything is already wired up.
Step 01
Sign up for Apify (free)
Go to apify.com and create a free account. No credit card. The free plan gives you $5 of usage credit every month, which resets and is enough for a 5 to 10 competitor scrape with full video transcription.
Step 02
Pick your 5 to 10 competitors with Claude's help
Before scraping, get the list right. Open Claude and run the competitor picker prompt below. It surfaces the right mix of direct competitors, adjacent creators, and one or two wildcard accounts whose audience overlaps with yours.
Copy this prompt into Claude before you open Apify. It builds the exact 5 to 10 handles you'll feed into the scrapers in Steps 3 to 5, including the reasoning behind each pick so the final analysis is sharper.
Step 03
Scrape Instagram with Apify's Instagram Scraper
Open apify/instagram-scraper (231K users, most-used Instagram scraper on Apify). Paste your competitor handles into the directUrls or usernames field. Set resultsLimit to 30 per handle (enough to see what's working without blowing your free credit). Hit Start. About 5 to 10 minutes per run. You'll get a dataset with caption, hashtags, likes, comments, view count, video URL, image URL, and post timestamp for every post.
Pricing reality: $1.50 per 1,000 results. Your $5 free monthly credit covers about 3,300 posts, which is way more than you'll need.
Step 04
Scrape TikTok with Clockworks TikTok Scraper
If your competitors post on TikTok, open clockworks/tiktok-scraper (the consensus default TikTok actor in 2026). Same flow: paste handles, set maxItems to 30 per handle, hit Start. Captures caption, video URL, plays, hearts, comments, shares, hashtags, and music metadata.
Pricing reality: about $5 per 1,000 results. Always cap maxItems so a too-broad input doesn't blow your credit in one run.
Step 05
Scrape YouTube (if anyone in your list runs a channel)
Open streamers/youtube-scraper for video metadata (title, views, upload date, description, duration). YouTube transcripts are pulled separately: use topaz_sharingan/youtube-transcript-scraper with your top 20 to 30 videos per channel. Native captions cost almost nothing to scrape.
Step 06
Transcribe the videos that matter
For Instagram Reels and TikToks, you need the actual words said in each video. The cleanest 2026 actor for this is electrifying_haircut/instagram-reel-analyzer. It scrapes the Reel AND transcribes via Whisper in one run. Cost: about $0.007 per Reel, so transcribing the top 10 Reels for each of your 8 competitors (80 Reels total) runs about $0.56.
For TikTok, run invideoiq/video-transcriber on the top 10 TikToks per handle. Same logic, similar cost.
Why transcripts matter: captions are the marketing copy. The actual video is the content. Without transcripts you're analyzing the wrapper, not the product. This is the step most "competitor analysis" workflows skip, which is exactly why your analysis will be sharper than 95% of what creators do.
Step 07
Export everything to one Google Drive folder
In each Apify dataset, click Export → CSV. Download every CSV. Create a Google Drive folder called "Gap Finder / [Date]" and upload them all: instagram-scrape.csv, tiktok-scrape.csv, youtube-scrape.csv, instagram-transcripts.csv, tiktok-transcripts.csv, youtube-transcripts.csv. One folder, six files, ready for analysis.
Step 08
Open Claude with the Drive connector, run the Gap Finder prompt
Open Claude Cowork. Make sure your Google Drive connector is turned on (Settings → Connectors). Create a new project called "Gap Finder". Tell Claude to read from your "Gap Finder / [Date]" folder. Then paste the master Gap Finder prompt below.
Copy this prompt into Claude after the Drive folder is loaded. It runs the full analysis: top patterns to ride, gaps to own, a 90-day calendar of 12 specific pieces of content with hooks, AND your sharpened positioning statement. This is the deliverable the whole workflow was built for.
If you only do this once a quarter
One run gives you 90 days of content direction backed by real data. Stack 4 runs a year and you have a content engine that compounds. The hard part is the first run. Block 90 minutes this weekend and you have a quarter of clarity by Sunday night.
Stack with these
Once you have the list of ideas, pair them with Stop using AI to post more, start using AI to post better for the writing voice rules, and 10 Claude Connectors for the workflows that get the content out the door fast.
The Only AI Masterclass You Need
If this guide helped, but you’re looking to go deeper, I got you!! My 30-Day Challenge takes you from saving AI tips you never use to actually building with AI, step-by-step.
I show you exactly how I automated two e-commerce brands, my social media, and most of my personal life, then hand you the agents, workflows & systems to do the same. I’m teaching you every single thing I know with one lesson and one build a day.
Join the AI Masterclass →© 2026 Mariah Brunner. All rights reserved.