PathExtracting and exporting data

Step 5 of 7

How to Export Scraped Data to CSV and JSON

Pim · Scrappey Research

June 8, 2026 4 min read

Paste into ChatGPT, Claude, or any LLM

How to Export Scraped Data to CSV and JSON — conceptual illustration

On this page

Export scraped data to CSV when you need flat, spreadsheet-ready rows, and to JSON when you need to preserve nested structure. In Python, the built-in csv module writes rows with correct quoting and escaping, while json serializes objects directly. Because a web scraping API like Scrappey already returns JSON, the cleanest pipeline is: parse the response, write JSON for the raw structured record, and flatten to CSV only the fields you want in a spreadsheet.

Best for tabular data	CSV (one row per record, opens in Excel/Sheets)
Best for nested data	JSON or JSON Lines (preserves arrays and objects)
Excel encoding fix	Write UTF-8 with BOM (utf-8-sig) so accents render
Large datasets	Stream row-by-row or use JSON Lines, never load all in RAM
Escaping handled by	Pythons csv module quotes commas, quotes, and newlines

CSV vs JSON: which format to use

Pick CSV for flat tabular records and JSON when fields are nested. CSV is one row per item with a fixed set of columns; it opens directly in a spreadsheet and is ideal for prices, product listings, or contact rows. JSON keeps arrays and nested objects intact, so it is the right choice when a record has variable-length lists (tags, images, reviews) or sub-objects that would not fit cleanly into columns.

Need	Use	Why
Open in Excel/Google Sheets	CSV	Native, one row per record
Preserve nested fields	JSON	Arrays and objects survive intact
Append millions of records	JSON Lines	One JSON object per line, append-safe

A common mistake is forcing nested data into CSV: a list field gets stringified or dropped. If a record has both flat and nested parts, write the full record to JSON for fidelity and a flattened subset to CSV for analysis. See exporting to Excel for the .xlsx variant.

Encoding and escaping pitfalls

The two failures that corrupt CSV exports are wrong encoding and unescaped delimiters; the csv module and UTF-8 handle both if configured right. Always open files with encoding="utf-8" and newline="" so the csv module controls line endings. When the file is destined for Excel on Windows, use encoding="utf-8-sig" to write a byte-order mark (BOM) so accented characters and non-Latin scripts render instead of showing mojibake.

Commas and quotes in values: never join fields by hand. The csv writer wraps any field containing a comma, double-quote, or newline in quotes and doubles internal quotes automatically.
Inconsistent keys: use csv.DictWriter with extrasaction="ignore" so a record with a missing or extra key does not throw.
JSON unicode: set ensure_ascii=False in json.dump to keep readable UTF-8 instead of \uXXXX escapes, and indent=2 only for human-readable files (omit it for compact machine files).

CSV flattens structure: a nested list becomes a string. If you must keep a nested field in CSV, serialize just that cell with json.dumps() so it round-trips losslessly.

Streaming large datasets

For large jobs, write each record as it arrives instead of building one giant list in memory. Keep the output file open and call writer.writerow() (CSV) or write one JSON object per line (JSON Lines / NDJSON) per scraped item. This caps memory at one record and lets the job resume or append without rewriting the whole file.

JSON Lines is the scalable cousin of JSON: each line is a standalone JSON object, so you can append with mode "a", stream-read it back line by line, and never need to load the entire array. Plain json.dump of a list, by contrast, requires the whole dataset in RAM and a full rewrite to add records.

Append, do not overwrite: open in append mode for incremental runs; write the CSV header only once.
Deduplicate: track a key (URL or id) in a set, or post-process with sort -u / pandas drop_duplicates().
Pagination: reuse one Scrappey session across pages so cookies and session state persist, and flush each page to disk before fetching the next.

Pair this with dynamic content scraping when the source renders rows in the browser, so the data you stream out is already complete.

Code example

python

import csv
import json
import requests

API = "https://publisher.scrappey.com/api/v1?key=YOUR_API_KEY"

# 1. Fetch a page through Scrappey (handles JS, proxies, verification).
#    autoparse=true asks Scrappey to return structured data when it can.
r = requests.post(API, json={
    "cmd": "request.get",
    "url": "https://example.com/products",
    "proxyCountry": "UnitedStates",
    "session": "export-demo",
    "autoparse": True,
})
r.raise_for_status()

# The HTML / parsed body lives here. Scrappey already returns JSON natively.
body = r.json()["solution"]["response"]

# Pretend we extracted these records (replace with your own parsing).
records = [
    {"name": "Cafe au lait mug", "price": 12.5, "tags": ["kitchen", "ceramic"]},
    {"name": "Notebook, A5", "price": 6.0, "tags": ["office"]},
]

# 2. Write full structured records to JSON Lines (nested fields preserved,
#    append-safe, streams one object per line for large jobs).
with open("data.jsonl", "a", encoding="utf-8") as f:
    for rec in records:
        f.write(json.dumps(rec, ensure_ascii=False) + "\n")

# 3. Write a flat, spreadsheet-ready CSV. utf-8-sig adds a BOM so Excel shows
#    accents correctly; newline="" lets the csv module control line endings.
fields = ["name", "price", "tags"]
with open("data.csv", "w", encoding="utf-8-sig", newline="") as f:
    w = csv.DictWriter(f, fieldnames=fields, extrasaction="ignore")
    w.writeheader()
    for rec in records:
        row = dict(rec)
        # CSV cannot hold a list, so serialize the nested field to a string.
        row["tags"] = json.dumps(row["tags"], ensure_ascii=False)
        w.writerow(row)  # quoting/escaping handled automatically

print("wrote data.jsonl and data.csv")

Next in Extracting and exporting data · 6 of 7

Or straight into a spreadsheet.

How to Scrape Website Data to Excel

Related terms

What Is a Web Scraping API?

A web scraping API is a hosted HTTP service that visits a web page for you and hands back the result — rendered HTML, JSON, or already-parse…

How to Scrape JavaScript-Rendered Pages With Python (2026 Guide)

To scrape a JavaScript-rendered page in Python you need something that executes the page’s JavaScript before you read the HTML. A plain requ…

How to scrape dynamic JavaScript content? (2026 Guide)

Dynamic content is anything a page loads after the initial HTML arrives — usually pulled in by JavaScript running in your browser. Because t…

How to Scrape Website Data to Excel

To scrape website data into Excel, fetch the page through a scraping API that returns structured JSON, load the rows into a Python list of d…

Web Scraping to Google Sheets

To get scraped data into Google Sheets you either write rows from code with the gspread library and a Google service account, or pull a publ…

What is a REST API? (Complete Guide 2026)

A REST API is a standard way for programs to read and change data over the web using ordinary HTTP requests. This is the complete 2026 guide…

Concept map

How How to Export Scraped Data to CSV and JSON (Python) connects

The terms most directly tied to this one. Hover a node to see its neighbours, click to preview, drag to rearrange.

0 terms · 0 connections

You are here · Web Scraping APIs

Tools & solutions for this topic

Frequently asked questions

When should I export to CSV instead of JSON?

Use CSV when your records are flat and you want to open them in Excel or Google Sheets: one row per item, fixed columns. Use JSON (or JSON Lines) when records contain nested arrays or objects you need to preserve, since CSV flattens or stringifies nested fields. Many pipelines write JSON for full fidelity and a flattened CSV for analysis.

Why do accented characters look wrong when I open my CSV in Excel?

Excel on Windows assumes a legacy encoding unless the file starts with a UTF-8 byte-order mark. Write the file with encoding="utf-8-sig" in Python so the BOM is added, and accents and non-Latin scripts will render correctly. Also pass newline="" when opening the file so the csv module controls line endings.

How do I export a very large scrape without running out of memory?

Write each record to disk as it is scraped instead of collecting everything in one list. For CSV, keep the file open and call writer.writerow() per item; for JSON, use JSON Lines (one JSON object per line) so you can append in mode "a" and read it back line by line. Reuse one Scrappey session across paginated requests and flush each page before fetching the next.

Last updated: 2026-06-08