paper-scraper-windows — Automated academic paper scrapers for ScienceDirect and INFORMS PubsOnLine. Search papers by keyword, author, or journal, and batch-download PDFs using your institutional access.
Optimized for Windows with full cross-platform support (macOS / Linux).
| Script | Platform | PDF Download |
|---|---|---|
sd_scraper.py / sd_scraper_en.py |
ScienceDirect (Elsevier) | CDP printToPDF (bypasses Cloudflare) |
informs_scraper.py / informs_scraper_en.py |
INFORMS PubsOnLine | Direct HTTP with session cookie |
Files ending in _en.py are full English-interface versions; the others are Chinese-interface versions. Logic is identical.
| Script | Purpose |
|---|---|
cdp_scraper.py |
CDP search + PDF download (self-contained, recommended for mainland China users) |
cdp_search.py |
Lightweight CDP search only (no PDF download) |
extract_cookies_cdp.py |
Launch Chrome in debug mode + extract cookies via CDP |
The scrapers implement a complete four-layer fallback chain to maximize success rate:
| Layer | Preferred | Fallback | When Fallback Triggers |
|---|---|---|---|
| Cookie | browser_cookie3 (reads Chrome cookies from disk) |
CDP Chrome launch (extracts cookies via DevTools) | browser_cookie3 not installed or returns empty |
| Search | curl_cffi TLS fingerprint emulation |
CDP real Chrome window (bypasses JS Challenge) | Search returns 0 results (Cloudflare blocked) |
| PDF Download | CDP printToPDF / curl_cffi direct download |
Info table CSV (always saved before PDF attempt) | PDF download fails for any reason |
| Output Dir | User-specified --output path |
os.getcwd() (the directory where you launched the script) |
No --output flag provided |
Key guarantee: the CSV info table is saved before PDF download is attempted. Even if every PDF fails, you never lose the search results.
Both TLS and CDP methods require a valid institutional login to ScienceDirect.
The scrapers do NOT bypass the paywall — they "borrow" your existing login state:
- TLS method: reads cookies from your Chrome profile on disk (
browser_cookie3), then sends them with every HTTP request. - CDP method: copies your Chrome profile (including login cookies) and launches a real Chrome window — ScienceDirect sees a normal logged-in browser.
Without a valid login, you can still search and get metadata, but full-text PDFs will be blocked.
How to log in:
- Open Chrome, go to https://www.sciencedirect.com
- Click "Sign in" → "Access through your institution"
- Search for your university, complete SSO/CARSI login
- Verify your institution name appears in the top-right corner
pip install curl_cffi websocket-client browser-cookie3 openpyxlFor INFORMS scraper, also install:
pip install beautifulsoup4 lxmlFull dependency list: requirements.txt
# Interactive wizard (recommended for first use)
python sd_scraper_en.py
# Keyword search — save metadata as XLSX
python sd_scraper_en.py -m keyword -q "machine learning" -n 100 --browser-cookies
# Keyword search + download PDFs to Desktop
python sd_scraper_en.py -m keyword -q "patient capital" -n 10 --browser-cookies --download-pdfs --output "C:/Users/YourName/Desktop"
# Browse a journal (most recent first)
python sd_scraper_en.py -m journal -j "Energy" -n 200 --browser-cookies --sort date
# Keyword search within a specific journal
python sd_scraper_en.py -m journal_keyword -j "Renewable Energy" -q "solar cell" -n 50 --browser-cookies
# Search by author
python sd_scraper_en.py -m author -a "Zhang Wei" -n 30 --browser-cookies
# Advanced search (combine criteria)
python sd_scraper_en.py -m advanced -q "deep learning" --date 2021-2024 --type REV -n 50 --browser-cookies --download-pdfs# Interactive wizard
python informs_scraper_en.py
# Keyword search
python informs_scraper_en.py -m keyword -q "supply chain" -n 100 --browser-cookies
# Browse a journal
python informs_scraper_en.py -m journal -j mnsc -n 200 --browser-cookies
# Specific volume/issue TOC
python informs_scraper_en.py -m toc -j mnsc -v 71 -i 3 --browser-cookies
# Keyword search + download PDFs
python informs_scraper_en.py -m keyword -q "inventory" -n 50 --browser-cookies --download-pdf
# Best login method: pop up Chrome, log in manually
python informs_scraper_en.py -m keyword -q "machine learning" -n 50 --chrome-login --download-pdf| Code | Journal |
|---|---|
mnsc |
Management Science |
opre |
Operations Research |
ijoc |
INFORMS Journal on Computing |
mksc |
Marketing Science |
msom |
Manufacturing & Service Operations Management |
trsc |
Transportation Science |
isre |
Information Systems Research |
orsc |
Organization Science |
Cloudflare blocks direct HTTP requests to PDF endpoints in mainland China. The scrapers use two strategies:
Strategy 1 — TLS fingerprint (curl_cffi):
- Emulates Chrome's TLS fingerprint at the protocol level
- Fast, no browser window needed
- Works outside mainland China; inside China, often blocked by Cloudflare JS Challenge
Strategy 2 — Chrome DevTools Protocol (CDP):
- Launches a real Chrome window with your existing login profile
- Calls
Page.printToPDFCDP command on the article page - Decodes the base64 PDF response and writes it to disk
- Completely bypasses Cloudflare JS Challenge — ScienceDirect sees a normal browser
- Requires Chrome debug port (port 9222)
# If Chrome isn't already logged in, use this first:
python sd_scraper_en.py --open-browser-login
# Then run your actual scrape:
python sd_scraper_en.py -m keyword -q "turbine" -n 50 --browser-cookies --download-pdfsDue to GFW DNS poisoning and Cloudflare JS Challenges, the curl_cffi search path often does not work from mainland China. The standalone CDP tools drive Chrome directly:
# Step 1: Launch Chrome with debug port (auto-detects proxy)
python extract_cookies_cdp.py
# Step 2: CDP search + DOM extraction (metadata always saved as CSV)
python cdp_scraper.py "enterprise digital transformation" 5
# Step 3: Search + PDF download to custom directory
python cdp_scraper.py "enterprise digital transformation" 5 --download-pdfs -o "C:/Users/YourName/Desktop"
# Lightweight: search only (no PDF download)
python cdp_search.py "machine learning" 10cdp_scraper.py features:
- Bypasses Cloudflare JS Challenge completely via a real Chrome window
_DOM_EXTRACT_JS(~140 lines) extracts title/author/journal/year/PII/DOI from the DOM- PDF captured via
Page.printToPDFCDP command (base64-decoded, saved to disk) - Auto-detects proxy: env var
HTTPS_PROXY→ Windows registry → macOSscutil --proxy - CSV always saved before PDF attempt — failed downloads don't lose data
Important: CDP tools require you to be logged in to ScienceDirect in Chrome first. The script copies your Chrome profile to preserve the login session.
INFORMS (Atypon platform) does not enforce JS challenges on PDF endpoints — a valid session cookie is sufficient.
# Option 1: read cookies from Chrome (must be logged in already)
python informs_scraper_en.py -m keyword -q "inventory" -n 30 --browser-cookies --download-pdf
# Option 2: pop up Chrome, log in, then auto-extract cookies
python informs_scraper_en.py -m keyword -q "inventory" -n 30 --chrome-login --download-pdf
# Option 3: member credentials (direct login)
python informs_scraper_en.py -m keyword -q "inventory" -n 30 --member 123456 --password MyPwd --download-pdfOutput defaults to os.getcwd() — the directory where you launched the script. Use --output to customize.
<output_dir>/
└── keyword_machine_learning_20250628_120000/ ← timestamped subfolder
├── keyword_machine_learning_20250628_120000.xlsx ← metadata
├── keyword_machine_learning_20250628_120000.csv ← CSV backup
└── pdfs/
├── 001_Zhang_2024_Deep learning for...pdf
├── 002_Li_2023_Transfer learning in...pdf
└── ...
| Option | Description |
|---|---|
-m, --mode |
keyword / journal / journal_keyword / author / issn / advanced |
-q, --query |
Search keywords (supports AND / OR / NOT) |
-j, --journal |
Journal name |
-a, --author |
Author name |
--issn |
Journal ISSN |
-n, --count |
Max papers to fetch (default: 50) |
--date |
Year range, e.g. 2020-2024 |
--sort |
relevance (default) or date |
--type |
Article type: FLA / REV / SCO |
--browser-cookies |
Auto-read cookies from local Chrome |
--cookies |
Path to a cookie JSON file |
--format |
Output format: xlsx (default) / csv / json / all |
--download-pdfs |
Download PDFs after saving metadata |
--output |
Custom output directory (default: current working directory) |
--open-browser-login |
Open Chrome for manual institutional login |
--interactive |
Launch interactive wizard |
| Argument | Description |
|---|---|
query |
Search keywords (positional, required) |
count |
Max papers to fetch (positional, default: 3) |
--download-pdfs |
Download PDFs after search |
-o, --output |
Output directory (default: ./results/) |
| Option | Description |
|---|---|
-m, --mode |
keyword / journal / toc / advanced |
-q, --query |
Search keywords |
-j, --journal |
Journal code (e.g. mnsc) |
-v, --volume |
Volume number (toc mode) |
-i, --issue |
Issue number (toc mode) |
--author |
Author name (advanced mode) |
--date |
Year range, e.g. 2020-2024 |
-n, --count |
Max papers (default: 100) |
--chrome-login |
★ Pop up Chrome for manual login (most reliable) |
--browser-cookies |
Read cookies from local Chrome |
--cookies-file |
Load cookies from a JSON file |
--member |
INFORMS member ID |
--password |
Account password |
--format |
csv (default) / json / xlsx |
--download-pdf |
Download PDFs after scraping |
-o, --output-dir |
Output directory |
- Institutional access required for full-text PDF download. Open-access papers can be downloaded without login. Both TLS and CDP methods depend on your existing Chrome login session.
- In mainland China, use the CDP tools (
cdp_scraper.py) —curl_cffisearches are almost always blocked by the GFW + Cloudflare. - Rate limiting: the scraper adds random delays between requests (default 2–5 s). Do not reduce these aggressively.
- Cross-platform: Windows, macOS, and Linux are fully supported. The scripts auto-detect the OS and select the correct Chrome path automatically.
- Cookie expiry: session cookies expire (typically days to weeks). Re-login if you encounter 403 errors.
- CSV is always saved before PDF download — even if all PDFs fail, your search results are safe.
- Use responsibly and in accordance with your institution's and the publishers' terms of service.
This project is based on and heavily inspired by GAO-pooh/paper-scraper — the original ScienceDirect & INFORMS scraper for macOS. We are grateful for their pioneering work, which provided the foundation for the cross-platform adaptation, CDP-based Cloudflare bypass, and four-layer auto-fallback strategy implemented here.
MIT