# Load job postings data into DuckDB

> Pull job posts into a local DuckDB file and analyze them with SQL in seconds - nested company data included.

Page: https://jobdataapi.com/integrations/duckdb/
Category: Databases and warehouses
You need: Python 3.9+, pip install requests duckdb and a jobdata API key (https://jobdataapi.com/accounts/pricing/).

## Steps

1. Set your API key and the filters you need.
2. The script writes newline-delimited JSON and loads it with read_json_auto, keeping nested fields as structs.
3. Query with plain SQL, e.g. company.name for the hiring company.

## Code

```python
import requests

API_KEY = "YOUR_API_KEY"


def fetch_jobs(**params):
    """Yield every job for the given filters, following pagination."""
    url = "https://jobdataapi.com/api/jobs/"
    params = {"page_size": 1000, **params}
    headers = {"Authorization": f"Api-Key {API_KEY}"}
    while url:
        resp = requests.get(url, headers=headers, params=params, timeout=60)
        resp.raise_for_status()
        data = resp.json()
        yield from data["results"]
        url, params = data["next"], None  # "next" already carries the filters


import json
import duckdb

with open("jobs.jsonl", "w") as f:
    for job in fetch_jobs(max_age=30, has_remote="true"):
        f.write(json.dumps(job) + "\n")

con = duckdb.connect("jobs.duckdb")
con.execute("CREATE OR REPLACE TABLE jobs AS SELECT * FROM read_json_auto('jobs.jsonl')")
print(con.sql("""
    SELECT company.name AS company, count(*) AS job_posts
    FROM jobs
    GROUP BY 1
    ORDER BY 2 DESC
    LIMIT 10
"""))
```

## Notes

- DuckDB reads the weekly Parquet downloads (access ultra) directly: SELECT * FROM 'jobs.parquet'.
- Run it daily with max_age=1 to pick up new job posts; on access lite, drop max_age - the feed covers the past 90 days and upserting by id keeps it free of duplicates.

## Related docs

- [Practical Parquet Analytics: Hiring Signals and Labor-Market Intelligence](https://jobdataapi.com/c/practical-parquet-analytics-hiring-signals-and-labor-market-intelligence/)
- [CSV and Parquet File Downloads Documentation](https://jobdataapi.com/c/csv-and-parquet-file-downloads-documentation/)
- [Jobs API Ingestion Guide: Reliable Historical Backfills and Incremental Syncs](https://jobdataapi.com/c/jobs-api-ingestion-guide-reliable-historical-backfills-and-incremental-syncs/)