Skip to content

CHANGELOG.md

Latest commit

 

History

History
657 lines (511 loc) · 24.4 KB

CHANGELOG.md

File metadata and controls

657 lines (511 loc) · 24.4 KB
Raw

Bright Data Python SDK Changelog

Version 2.5.2 - Safer CSV exports

  • Security fix: export_csv() (and export() for .csv files) now automatically escapes cell values that spreadsheet apps like Excel, Google Sheets, or LibreOffice could otherwise misinterpret as formulas, so scraped data can't unexpectedly trigger links or formulas when the exported file is opened.
  • New sanitize option: enabled by default (sanitize=True). Pass sanitize=False to export_csv() if you need the previous raw export behavior.

Version 2.5.1 - Lint cleanup

  • Chore: pinned an explicit ruff lint rule scope (select) and sorted imports repo-wide. No behavior changes.

Version 2.5.0 - CLI-credentials auth + scraper-core dedup

  • CLI-credentials auth fallback: one brightdata login now authenticates the SDK too. With no token given, BrightDataClient() / SyncBrightDataClient() fall back to the token already saved by the Bright Data CLI. Resolution order: token= parameter → BRIGHTDATA_API_TOKEN / BRIGHTDATA_API_KEY env vars → CLI login → actionable error mentioning brightdata login.
  • Auth-source reporting: every request now reports how the token was obtained via the User-Agent header (brightdata-sdk-python/{version} (auth=param|env|cli_credentials)) — the token itself is never logged.
  • Internal: shared ScraperCore: the 7 search scrapers (Amazon, ChatGPT, Instagram, LinkedIn, Pinterest, TikTok, YouTube) now share one construction core instead of hand-copied __init__s, fixing drift where Amazon/ChatGPT/LinkedIn search were missing the BRIGHTDATA_API_TOKEN environment fallback that the others already had (−174 lines of duplicated code).
  • Tests: 321 passing (57 new), covering every credential-resolution path, all 3 platform paths (Linux/macOS/Windows), and precedence order.

Version 2.4.0 - Sync parity, colorless job verbs, dataset error reporting

  • Sync client parity: SyncBrightDataClient now mirrors the async surface. Added client.datasets (fixes the SyncBrightDataClient datasets AttributeError), the 5 missing scrapers (scrape.tiktok / youtube / reddit / perplexity / digikey), the 2 missing search verticals (search.tiktok / youtube), Pinterest trigger/status/fetch, and Instagram-search profiles / reels_all.
  • Service-level job verbs (colorless pattern): every scraper now exposes generic status / wait / fetch / to_result(snapshot_id) (on BaseWebScraper), and DiscoverService gained status / wait / fetch / to_result(task_id) — so a triggered job can be driven by its id alone, like the crawler. Purely additive; the existing job.fetch() etc. are unchanged.
  • Discover sync manual path (new): SyncBrightDataClient adds discover_status / discover_wait / discover_fetch / discover_to_result(task_id), and a colorless DiscoverSnapshot handle.
  • Contract change: sync discover_trigger() now returns a DiscoverSnapshot (a typed, drivable handle) instead of the async-only DiscoverJob (which could not be used from sync). Migration: poll via client.discover_status(snap.task_id) / client.discover_fetch(snap.task_id).
  • Fixed: failed dataset snapshots now expose the API failure reason — and, when no recognized reason key is present, the raw snapshot status response as a fallback — plus the snapshot_id, instead of the unhelpful DatasetError: Snapshot failed: None. SnapshotStatus now retains the full API response (.raw) and matches more reason keys (error / error_message / message / failure_reason). The synchronous path inherits the fix.

Version 2.3.0 - Browser API, Scraper Studio, 175 Datasets

  • Browser API: Connect to cloud Chrome via CDP WebSocket. SDK builds the wss:// URL, you connect with Playwright/Puppeteer (client.browser.get_connect_url())
  • Scraper Studio: Trigger and fetch results from custom scrapers built in Bright Data's IDE (client.scraper_studio.run())
  • 75 more datasets: Agoda, AutoZone, BBC, Best Buy, Bluesky, Booking, Costco, eBay, Etsy, GitHub, Google News/Play/Shopping, Home Depot, Kroger, Lowe's, Macy's, Microcenter, Ozon, Quora, Realtor, Reddit, Snapchat, TikTok Shop, Tokopedia, Vimeo, Wayfair, Wikipedia, Wildberries, X/Twitter, Yahoo Finance, Zoopla, and more — 175 total
  • Codebase cleanup: Removed dead code and legacy abstractions — collapsed datasets/client.py from 1635 to 285 lines, fixed ScrapeJob.to_result() crash bug, cleaned up unused protocols, redundant config layers, and stale API modules
  • Test suite rewrite: Rebuilt test suite from scratch with 365 unit tests, shared fixtures via conftest.py, behavioral coverage focus — key modules now at 87–98% coverage (client, scrapers, SERP, sync client, job lifecycle)

Version 2.2.1 - 100 Datasets API

✨ New Features

Expanded Datasets Coverage

Added 92 new dataset integrations, bringing the total to 100 datasets:

  • Luxury Brands: Loewe, Berluti, Moynat, Hermes, Delvaux, Prada, Montblanc, YSL, Dior, Balenciaga, Bottega Veneta, Celine, Chanel, Fendi
  • E-commerce: Amazon (Reviews, Sellers), Walmart, Shopee, Lazada, Zalando, Sephora, Zara, Mango, Massimo Dutti, Asos, Shein, Ikea, H&M, Lego, Mouser, Digikey
  • Social Media: Instagram (Profiles, Posts), TikTok, Pinterest (Posts, Profiles), YouTube (Profiles, Videos, Comments), Facebook Pages Posts
  • Real Estate: Zillow, Airbnb, Australia Real Estate, Otodom Poland, Zonaprop Argentina, Metrocuadrado, Infocasas Uruguay, Properati, Toctoc, Inmuebles24 Mexico, Yapo Chile
  • Business Data: Glassdoor (Companies, Reviews, Jobs), Indeed (Companies, Jobs), ZoomInfo, PitchBook, G2, Trustpilot, TrustRadius, Owler, Slintel, Manta, VentureRadar, Companies Enriched, Employees Enriched
  • Other: World Zipcodes, US Lawyers, Google Maps Reviews, Yelp, Xing Profiles, OLX Brazil, Webmotors Brasil, Chileautos, LinkedIn Jobs

SERP Pagination Support

Added sequential querying to retrieve more than 10 search results from Google:

async with BrightDataClient() as client:
    # Get up to 50 results with automatic pagination
    results = await client.search.google(
        query="python programming",
        num_results=50  # Fetches multiple pages sequentially
    )

Version 2.2.0 - Datasets API

✨ New Features

Datasets API

Access Bright Data's pre-collected datasets with filtering and export capabilities.

async with BrightDataClient() as client:
    # Filter dataset records
    snapshot_id = await client.datasets.amazon_products(
        filter={"name": "rating", "operator": ">=", "value": 4.5},
        records_limit=100
    )
    # Download results
    data = await client.datasets.amazon_products.download(snapshot_id)

8 Datasets: LinkedIn Profiles, LinkedIn Companies, Amazon Products, Crunchbase Companies, IMDB Movies, NBA Players Stats, Goodreads Books, World Population

Export Utilities:

from brightdata.datasets import export_json, export_csv
export_json(data, "results.json")
export_csv(data, "results.csv")

📓 Notebooks

  • notebooks/datasets/linkedin/linkedin.ipynb - LinkedIn datasets (profiles & companies)
  • notebooks/datasets/amazon/amazon.ipynb - Amazon products dataset
  • notebooks/datasets/crunchbase/crunchbase.ipynb - Crunchbase companies dataset

Version 2.1.2 - Web Scrapers & Notebooks

🐛 Bug Fixes

LinkedIn Job Search

Fixed client.search.linkedin.jobs() to use the correct discovery dataset when searching by keyword/location. Previously it was incorrectly using the URL-based job scraper dataset which expected single job URLs, not search parameters.

📓 Notebooks

New Notebooks

  • notebooks/web_scrapers/linkedin.ipynb - Complete LinkedIn scraper tests for all endpoints
  • notebooks/03_serp.ipynb - Google Search API tests
  • notebooks/04_web_unlocker.ipynb - Web Unlocker HTML scraping tests

Updated Notebooks

  • notebooks/02_pandas_integration.ipynb - Efficient batch scraping with asyncio.gather() pattern

Version 2.1.1 - Instagram Scrapers & Version Centralization

✨ New Features

Complete Instagram Scraper Implementation

Full Instagram scraping support with URL-based extraction and discovery endpoints:

URL-based Scraping (client.scrape.instagram)

  • profiles(url) - Extract profile data from Instagram profile URL
  • posts(url) - Extract post data from Instagram post URL
  • comments(url) - Extract comments from Instagram post URL
  • reels(url) - Extract reel data from Instagram reel URL

Discovery/Search (client.search.instagram)

  • profiles(user_name) - Discover profile by exact username lookup
  • posts(url, num_of_posts, start_date, end_date, post_type) - Discover posts from profile
  • reels(url, num_of_posts, start_date, end_date) - Discover reels from profile
  • reels_all(url, num_of_posts, start_date, end_date) - Discover all reels from profile
async with BrightDataClient() as client:
    # URL-based scraping
    post = await client.scrape.instagram.posts("https://instagram.com/p/ABC123/")
    reel = await client.scrape.instagram.reels("https://instagram.com/reel/XYZ789/")

    # Discovery by username
    profile = await client.search.instagram.profiles(user_name="nasa")

    # Discover posts from profile with filters
    posts = await client.search.instagram.posts(
        url="https://instagram.com/nasa",
        num_of_posts=10,
        start_date="2024-01-01",
        end_date="2024-12-31"
    )

🔧 Internal Improvements

Version Centralization

Version is now managed from a single source (pyproject.toml). All other files read it dynamically via importlib.metadata.

Before (5 files to update):

  • pyproject.toml
  • src/brightdata/__init__.py
  • src/brightdata/_version.py
  • src/brightdata/core/engine.py
  • src/brightdata/cli/main.py

After (1 file to update):

  • pyproject.toml ← Single source of truth

Changes:

  • __init__.py now uses importlib.metadata.version("brightdata-sdk")
  • _version.py deleted (no longer needed)
  • engine.py imports __version__ for User-Agent header
  • cli/main.py imports __version__ for --version flag

Version 2.1.0 - Async Mode, API Simplification & Bug Fixes

✨ New Features

SERP Async Mode

Added non-blocking async mode for SERP API using Bright Data's unblocker endpoints:

from brightdata import BrightDataClient

async with BrightDataClient() as client:
    # Non-blocking - polls for results
    result = await client.search.google(
        query="python programming",
        mode="async",        # Enable async mode
        poll_interval=2,     # Check every 2 seconds
        poll_timeout=30      # Give up after 30 seconds
    )

Supported Engines: Google, Bing, Yandex

Performance: SERP async mode typically completes in ~3 seconds.

Web Unlocker Async Mode

Added non-blocking async mode for Web Unlocker API:

async with BrightDataClient() as client:
    result = await client.scrape_url(
        url="https://example.com",
        mode="async",
        poll_interval=5,     # Check every 5 seconds
        poll_timeout=180     # Web Unlocker async takes ~2 minutes
    )

    # Batch scraping multiple URLs
    urls = ["https://example.com", "https://example.org"]
    results = await client.scrape_url(url=urls, mode="async", poll_timeout=180)

Performance Warning: Web Unlocker async mode takes ~2 minutes to complete. For faster single-URL scraping, use the default sync mode.

How async mode works:

  1. Triggers request to /unblocker/req (returns immediately)
  2. Polls /unblocker/get_result until ready or timeout
  3. Returns same data structure as sync mode

Key Benefits:

  • ✅ Non-blocking requests - continue work while scraping
  • ✅ Batch optimization - trigger multiple URLs, collect later
  • ✅ Same data structure as sync mode
  • ✅ No extra configuration - works with existing zones
  • ✅ No customer_id required - derived from API token

See: Async Mode Guide for detailed usage

🐛 Bug Fixes

  • Fixed SyncBrightDataClient: Removed unused customer_id parameter that was incorrectly being passed to BrightDataClient
  • Fixed Web Unlocker async timeout: Changed default poll_timeout from 30s to 180s (Web Unlocker async takes ~145 seconds)

🚨 Breaking Changes

Removed GenericScraper

# OLD (v2.0.0)
result = await client.scrape.generic.url("https://example.com")

# NEW (v2.1.0) - Use scrape_url() directly
result = await client.scrape_url("https://example.com")

Async Method Naming Convention

The _async suffix has been removed. Now method() is async by default, and method_sync() is the synchronous version.

# OLD (v2.0.0)
result = await scraper.products_async(url)
await job.wait_async()
data = await job.fetch_async()

# NEW (v2.1.0)
result = await scraper.products(url)
await job.wait()
data = await job.fetch()

CLI Command Change

# OLD
brightdata scrape generic --url https://example.com

# NEW
brightdata scrape url --url https://example.com

✨ New Features

Complete SyncBrightDataClient

Added comprehensive sync_client.py with full coverage for all scrapers:

from brightdata import SyncBrightDataClient

with SyncBrightDataClient() as client:
    # All methods work synchronously
    result = client.scrape.amazon.products(url)
    result = client.scrape.linkedin.profiles(url)
    result = client.search.google("query")

Supported sync wrappers:

  • SyncAmazonScraper - products, reviews, sellers (+ trigger/status/fetch)
  • SyncLinkedInScraper - profiles, jobs, companies, posts
  • SyncInstagramScraper - profiles, posts, comments, reels
  • SyncFacebookScraper - posts_by_profile, posts_by_group, comments, reels
  • SyncChatGPTScraper - prompt, prompts
  • SyncSearchService - google, bing, yandex
  • SyncCrawlerService - crawl, scrape

Context Manager Enforcement

Client methods now require proper context manager initialization:

# Correct usage
async with BrightDataClient() as client:
    result = await client.scrape_url(url)

# Will raise RuntimeError
client = BrightDataClient()
result = await client.scrape_url(url)  # Error: not initialized

🔄 Migration Guide

Method Renames

Old (v2.0.0) New (v2.1.0)
products_async() products()
reviews_async() reviews()
profiles_async() profiles()
jobs_async() jobs()
wait_async() wait()
fetch_async() fetch()
to_result_async() to_result()
status_async() status()
scrape.generic.url() scrape_url()

Quick Migration

# Find and replace in your codebase:
_async() → ()
scrape.generic.url → scrape_url

📚 Documentation

  • Added Async Mode Guide - comprehensive guide to async mode
  • Simplified README with clearer examples
  • Updated all examples and tests to use new naming convention

🧪 Testing

  • Added unit tests for AsyncUnblockerClient
  • Added integration tests for SERP and Web Unlocker async modes
  • Verified backwards compatibility (existing code works unchanged)

Version 2.0.0 - Complete Architecture Rewrite

🚨 Breaking Changes

Client Initialization

# OLD (v1.1.3)
from brightdata import bdclient
client = bdclient(api_token="your_token")

# NEW (v2.0.0)
from brightdata import BrightDataClient
client = BrightDataClient(token="your_token")

API Structure Changes

  • Old: Flat API with methods directly on client (client.scrape(), client.search())
  • New: Hierarchical service-based API (client.scrape.amazon.products(), client.search.google())

Method Naming Convention

# OLD
client.scrape_linkedin.profiles(url)
client.search_linkedin.jobs()

# NEW
client.scrape.linkedin.profiles(url)
client.search.linkedin.jobs()

Return Types

  • Old: Raw dictionaries and strings
  • New: Structured ScrapeResult and SearchResult objects with metadata and timing metrics

Python Version Requirement

  • Old: Python 3.8+
  • New: Python 3.9+ (dropped Python 3.8 support)

🎯 Major Architectural Changes

1. Async-First Architecture

Old: Synchronous with ThreadPoolExecutor for concurrency

# Old approach - thread-based parallelism
with ThreadPoolExecutor(max_workers=10) as executor:
    results = executor.map(self.scrape, urls)

New: Native async/await throughout with sync wrappers

# New approach - native async (method() is async by default)
async def products(self, url):
    async with self.engine:
        return await self._execute_workflow(...)

# Sync client uses persistent event loop
with SyncBrightDataClient() as client:
    result = client.scrape.amazon.products(url)

2. Service-Based Architecture

Old: Monolithic bdclient class with all methods New: Layered architecture with specialized services

BrightDataClient
├── scrape (ScrapeService)
│   ├── amazon (AmazonScraper)
│   ├── linkedin (LinkedInScraper)
│   └── instagram (InstagramScraper)
├── search (SearchService)
│   ├── google
│   ├── bing
│   └── yandex
└── crawler (CrawlService)

3. Workflow Pattern Implementation

Old: Direct HTTP requests with immediate responses New: Trigger/Poll/Fetch workflow for long-running operations

# New workflow pattern
snapshot_id = await trigger(payload)     # Start job
status = await poll_until_ready(snapshot_id)  # Check progress
data = await fetch_results(snapshot_id)  # Get results

✨ New Features

1. Comprehensive Platform Support

Platform Old SDK New SDK New Capabilities
Amazon ❌ ✅ Products, Reviews, Sellers (separate datasets)
LinkedIn ✅ Basic ✅ Full Enhanced scraping and search methods
Instagram ❌ ✅ Profiles, Posts, Comments, Reels
Facebook ❌ ✅ Posts, Comments, Groups
ChatGPT ✅ Basic ✅ Enhanced Improved prompt interaction
Google Search ✅ ✅ Enhanced Dedicated service with better structure
Bing/Yandex ✅ ✅ Enhanced Separate service methods

2. Manual Job Control

# New capability - fine-grained control over scraping jobs
job = await scraper.products_trigger(url)
# Do other work...
status = await job.status()
if status == "ready":
    data = await job.fetch()

3. Type-Safe Payloads (Dataclasses)

# New - structured payloads with validation
from brightdata import AmazonProductPayload
payload = AmazonProductPayload(
    url="https://amazon.com/dp/B123",
    reviews_count=100
)

# Old - untyped dictionaries
payload = {"url": "...", "reviews_count": 100}

4. CLI Tool

# New - command-line interface
brightdata scrape amazon products --url https://amazon.com/dp/B123
brightdata search google --query "python sdk"
brightdata crawler discover --url https://example.com --depth 3

# Old - no CLI support

5. Registry Pattern for Scrapers

# New - self-registering scrapers
@register("amazon")
class AmazonScraper(BaseWebScraper):
    DATASET_ID = "gd_l7q7dkf244hwxbl93"

6. Advanced Telemetry

  • SDK function tracking via stack inspection
  • Microsecond-precision timestamps for all operations
  • Comprehensive cost tracking per platform
  • Detailed timing metrics in results

🚀 Performance Improvements

Connection Management

  • Old: New connection per request, basic session management
  • New: Advanced connection pooling (100 total, 30 per host) with keep-alive

Concurrency Model

  • Old: Thread-based with GIL limitations
  • New: Event loop-based with true async concurrency

Resource Management

  • Old: Basic cleanup with requests library
  • New: Triple-layer cleanup strategy with context managers and idempotent operations

Rate Limiting

  • Old: No built-in rate limiting
  • New: Optional AsyncLimiter integration (10 req/sec default)

📦 Dependency Changes

Removed Dependencies

  • beautifulsoup4 - Parsing moved to server-side
  • openai - Not needed for ChatGPT scraping

New Dependencies

  • tldextract - Domain extraction for registry
  • pydantic - Data validation (optional)
  • aiolimiter - Rate limiting support
  • click - CLI framework

Updated Dependencies

  • aiohttp>=3.8.0 - Core async HTTP client (was using requests for sync)

🔧 Configuration Changes

Environment Variables

# Supported in both old and new versions:
BRIGHTDATA_API_TOKEN=token
WEB_UNLOCKER_ZONE=zone
SERP_ZONE=zone
BROWSER_ZONE=zone
BRIGHTDATA_BROWSER_USERNAME=username
BRIGHTDATA_BROWSER_PASSWORD=password

# Note: Rate limiting is NOT configured via environment variable
# It must be set programmatically when creating the client

Client Parameters

# Old (v1.1.3)
client = bdclient(
    api_token="token",                  # Required parameter name
    auto_create_zones=True,              # Default: True
    web_unlocker_zone="sdk_unlocker",   # Default from env or 'sdk_unlocker'
    serp_zone="sdk_serp",               # Default from env or 'sdk_serp'
    browser_zone="sdk_browser",         # Default from env or 'sdk_browser'
    browser_username="username",
    browser_password="password",
    browser_type="playwright",
    log_level="INFO",
    structured_logging=True,
    verbose=False
)

# New (v2.0.0)
client = BrightDataClient(
    token="token",                       # Changed parameter name (was api_token)
    customer_id="id",                    # New parameter (optional)
    timeout=30,                          # New parameter (default: 30)
    auto_create_zones=False,             # Changed default: now False (was True)
    web_unlocker_zone="web_unlocker1",  # Changed default name
    serp_zone="serp_api1",              # Changed default name
    browser_zone="browser_api1",        # Changed default name
    validate_token=False,                # New parameter
    rate_limit=10,                      # New parameter (optional)
    rate_period=1.0                     # New parameter (default: 1.0)
)
# Note: browser credentials and logging config removed from client init

🔄 Migration Guide

Basic Scraping

# Old
result = client.scrape(url, zone="my_zone", response_format="json")

# New (minimal change)
result = client.scrape_url(url, zone="my_zone", response_format="json")

# New (recommended - platform-specific)
result = client.scrape.amazon.products(url)

LinkedIn Operations

# Old
profiles = client.scrape_linkedin.profiles(url)
jobs = client.search_linkedin.jobs(location="Paris")

# New
profiles = client.scrape.linkedin.profiles(url)
jobs = client.search.linkedin.jobs(location="Paris")

Search Operations

# Old
results = client.search(query, search_engine="google")

# New
results = client.search.google(query)

Async Migration

# Old (sync only)
result = client.scrape(url)

# New (async-first)
async def main():
    async with BrightDataClient(token="...") as client:
        result = await client.scrape_url(url)

# Or use sync client
with SyncBrightDataClient(token="...") as client:
    result = client.scrape_url(url)

🎯 Summary

Version 2.0.0 represents a complete rewrite of the Bright Data Python SDK, not an incremental update. The new architecture prioritizes:

  1. Modern Python patterns: Async-first with proper resource management
  2. Developer experience: Hierarchical APIs, type safety, CLI tools
  3. Production reliability: Comprehensive error handling, telemetry
  4. Platform coverage: All major platforms with specialized scrapers
  5. Flexibility: Three levels of control (simple, workflow, manual)

This is a breaking release requiring code changes. The migration effort is justified by:

  • 10x improvement in concurrent operation handling
  • 50+ new platform-specific methods
  • Proper async support for modern applications
  • Comprehensive timing and cost tracking
  • Future-proof architecture for new platforms

📝 Upgrade Checklist

  • Update Python to 3.9+
  • Update import statements from bdclient to BrightDataClient
  • Migrate to hierarchical API structure
  • Update method calls to new naming convention
  • Handle new ScrapeResult/SearchResult return types
  • Consider async-first approach for better performance
  • Review and update error handling for new exception types
  • Test rate limiting configuration if needed
  • Validate platform-specific scraper migrations