Jobs API Ingestion Guide: Reliable Historical Backfills and Incremental Syncs

Table of contents

Historical ingestion is not just a matter of requesting every page until next is null. A production sync also needs an identity key, retry behavior, bounded work units, and a way to reconcile records that arrive while the backfill is running.

This guide presents a practical approach for historical backfills and incremental syncs against the jobdata API. The examples use Python and SQLite, but the same request and validation rules apply to PostgreSQL, a warehouse, or another destination.

The short version

For a small, stable response, ordinary pagination is fine. For a large historical extraction, sequential pagination is a practical best-effort mechanism, but not a mathematical completeness guarantee.

Use this operating model:

1. Split the requested history into bounded calendar windows.
2. Follow next sequentially within each window.
3. Run each window at least twice and union results by API id.
4. Repeat a window while its union is still growing or its observed count is changing.
5. Use bounded ID ranges for unusually large or unstable windows.
6. Store every job idempotently by API id.
7. Keep each window independently rerunnable and run periodic reconciliation passes.

This does not promise perfection against a live mutable dataset. It limits the blast radius of unstable page boundaries, makes failures manageable, and gives the importer a measurable stopping rule.

Understand the two timelines

The Jobs endpoint is:

GET https://jobdataapi.com/api/jobs/

Authenticate every request with:

Authorization: Api-Key YOUR_API_KEY

Each response contains:

• count: the number of matching rows at the time of the request
• next: a URL for the next page, or null
• previous: a URL for the previous page, or null
• results: the current page of job objects

There are two useful timelines in the data:

• published describes when a listing was published according to the source data.
• id describes when the listing entered the jobdata dataset.

Those timelines are not interchangeable. A listing can enter the dataset late with an old published value. Date windows are useful for historical scope, while ID ranges are useful for checkpoints and for subdividing an oversized date window.

Important pagination limitation

The endpoint currently uses page-number pagination and sorts normal job results primarily by published. Rows with equal published values do not have a guaranteed secondary order in every response.

That matters when a result set is larger than one page. Page 2 is an offset into a newly executed query, not a continuation token tied to the exact result set returned for page 1. If rows sharing a timestamp are arranged differently, a page boundary can contain a duplicate in one run and omit another row in a different run.

The dataset can also change during a multi-request backfill. New records can arrive or other fields can be updated. Sequential requests and a delay between requests do not create a database snapshot.

Until the API provides a stable total ordering and cursor or snapshot semantics, page-number pagination should be treated as best effort. Client-side deduplication is required. Multiple complete passes over a bounded window improve coverage because a later pass can return IDs omitted at a page boundary in an earlier pass, but multiple passes still cannot prove mathematical completeness.

Check access and make the scope explicit

Before building the importer, verify the subscription tier. API access lite and anonymous requests cannot use the historical slicing parameters:

• published_since
• published_until
• min_id
• max_id
• min_age
• max_age

For API access and higher plans, these parameters are available. If no slicing parameter is provided, the endpoint applies an implicit max_age=90 window. Always specify the intended scope instead of relying on that default.

When a date or ID slicing parameter is present, the implicit recent-window behavior is already disabled for non-lite accounts. max_age=0 is therefore unnecessary in those requests. It remains an option for an otherwise unsliced full-history request (not recommended).

For a query that should include expired jobs, do not add exclude_expired=true. Confirm the endpoint's visibility rules match your definition of a full dataset before starting: the Jobs endpoint is still subject to its normal job and company visibility rules.

Example:

curl -G 'https://jobdataapi.com/api/jobs/' \
  -H 'Authorization: Api-Key YOUR_API_KEY' \
  --data-urlencode 'published_since=2026-01-01' \
  --data-urlencode 'published_until=2026-01-02' \
  --data-urlencode 'page_size=4000'

Date boundaries are inclusive. Use an intentional boundary overlap and deduplicate by id; do not assume that published_until is an exclusive upper bound. Verify the behavior with a small test window before running a large extraction.

The practical historical backfill

1. Freeze the request definition

Record the complete set of filters used for the run, including:

• date boundaries
• ID boundaries
• country or region filters
• agency and expiration behavior
• fields requested, such as description_md=true
• the API key's plan and the run's UTC start time

Do not change filters halfway through a backfill. If the scope needs to change, start a new run with a new scope identifier.

2. Choose a manageable window

The right window is determined by volume and operational cost, not by a fixed calendar rule. Start by sampling a few days with page_size=4000 and record the response count, number of pages, elapsed time, and how much the ID set changes on a second pass.

At 100,000 jobs per day, one calendar day is approximately 25 pages. That is a reasonable default: it limits the time a window is exposed to data changes and makes failed work easy to restart. Use several days per window when the result is comfortably below roughly 100 pages. Use one day or smaller ID ranges when a window is larger, slow, or frequently changing.

For a high-volume feed, prefer a one-day window whose end is the next calendar day. For example, use published_since=2026-01-01 and published_until=2026-01-02, then use published_since=2026-01-02 for the next window. The boundary is intentionally repeated and deduplicated.

Do not make a window so small that orchestration becomes the main failure mode. A daily window is a good default for a feed producing around 100,000 jobs per day; measure and adjust from there.

3. Follow pages within each pass

For each date window, request the first page and follow the returned next URL until it is null. Save every page immediately using an upsert keyed by id.

Do not synthesize page URLs if a next URL is available. Keep the original window parameters with every request and record the page number, response count, result count, unique ID count, and whether next was present.

If a request fails after some pages have been saved, rerun the whole window. Idempotent writes make this simpler and safer than trying to resume at an offset that may have moved.

4. Run multiple passes over each window

Run every historical window twice. After each pass, merge the returned jobs into the destination by ID and compare the pass's unique ID set with the accumulated union.

Run a third or fourth pass only when useful. A practical stopping rule is:

• minimum of two complete passes
• stop after two consecutive passes add no new IDs and the observed count is not increasing
• continue or flag the window if the union is still growing or counts are changing materially

This is a coverage heuristic, not a guarantee. It is intentionally simple, observable, and usually more useful than attempting to prove completeness against a live mutable dataset.

5. Use ID ranges for unusually large windows

If a one-day window is too large to process comfortably, or if repeated passes continue to differ substantially, subdivide it by ID:

curl -G 'https://jobdataapi.com/api/jobs/' \
  -H 'Authorization: Api-Key YOUR_API_KEY' \
  --data-urlencode 'published_since=2026-01-01' \
  --data-urlencode 'published_until=2026-01-02' \
  --data-urlencode 'min_id=2000000' \
  --data-urlencode 'max_id=2099999' \
  --data-urlencode 'page_size=4000'

min_id and max_id filter the result set; they do not currently change the endpoint's primary ordering to ID order. Use them to make a difficult date window smaller, not as an assumption that the response is cursor-ordered. Run the same multi-pass strategy over the ID range if it still requires pagination.

The simple decision rule is:

if the window is manageable:
    follow next sequentially and run multiple passes
elif the window is too large or unstable:
    split it by min_id/max_id and repeat the same process

6. Validate every completed pass

Do not mark a pass complete merely because the HTTP requests succeeded. Record and check:

every page was fetched successfully
last_response.next is null
unique IDs are counted separately from result rows
the union of IDs is retained for comparison with later passes

If a query involving multi-valued relationship filters returns duplicate IDs inside one response, record that separately rather than silently treating count as a unique-job count. Narrow the filter set or investigate the query before claiming that the window is stable.

7. Reconcile after the first pass

After all windows have been ingested, repeat the same window plan. Compare the second pass with the first pass and with the destination:

• IDs returned by the second pass but absent from the destination indicate a write or coverage problem.
• IDs returned by the first pass but not the second may reflect a changing source or visibility state and should be logged.
• A window whose union continues growing should be rerun using smaller date or ID ranges.

Two full multi-page runs do not prove completeness. They are a pragmatic way to improve coverage while keeping the process understandable and operationally bounded.

A runnable multi-pass backfill example

The following script uses SQLite for demonstration. Replace save_jobs with your own upsert implementation in production. It follows next within each pass, saves pages as it goes, and repeats each date window while its union is still changing.

Save as multi_pass_backfill.py:

#!/usr/bin/env python3
import argparse
import json
import os
import sqlite3
import time
from datetime import date, timedelta
from typing import Dict, Iterable, List, Optional, Set, Tuple

import requests


API_URL = 'https://jobdataapi.com/api/jobs/'
PAGE_SIZE = 4000
MIN_PASSES = 2
MAX_PASSES = 4


def get_api_key() -> str:
    key = os.getenv('JOBDATA_API_KEY', '').strip()
    if not key:
        raise RuntimeError('Missing JOBDATA_API_KEY environment variable')
    return key


def get_connection() -> sqlite3.Connection:
    conn = sqlite3.connect('job_sync.sqlite3')
    conn.execute(
        '''
        CREATE TABLE IF NOT EXISTS jobs (
            id INTEGER PRIMARY KEY,
            published TEXT,
            title TEXT,
            raw_json TEXT NOT NULL,
            imported_at TEXT DEFAULT CURRENT_TIMESTAMP
        )
        '''
    )
    conn.commit()
    return conn


def request_json(
    session: requests.Session,
    url: str,
    api_key: str,
    params: Optional[Dict[str, str]] = None,
) -> Dict:
    headers = {'Authorization': f'Api-Key {api_key}'}

    for attempt in range(5):
        response = session.get(
            url,
            headers=headers,
            params=params,
            timeout=60,
        )
        if response.status_code == 429 or response.status_code >= 500:
            if attempt == 4:
                response.raise_for_status()
            wait_seconds = min(2 ** attempt, 30)
            print(
                f'[retry] status={response.status_code}, '
                f'sleeping={wait_seconds}s'
            )
            time.sleep(wait_seconds)
            continue

        response.raise_for_status()
        return response.json()

    raise RuntimeError('Request retry loop ended unexpectedly')


def save_jobs(conn: sqlite3.Connection, jobs: Iterable[Dict]) -> None:
    rows = [
        (
            int(job['id']),
            job.get('published'),
            job.get('title'),
            json.dumps(job, separators=(',', ':')),
        )
        for job in jobs
    ]
    conn.executemany(
        '''
        INSERT INTO jobs (id, published, title, raw_json)
        VALUES (?, ?, ?, ?)
        ON CONFLICT(id) DO UPDATE SET
            published=excluded.published,
            title=excluded.title,
            raw_json=excluded.raw_json,
            imported_at=CURRENT_TIMESTAMP
        ''',
        rows,
    )
    conn.commit()


def fetch_pass(
    session: requests.Session,
    conn: sqlite3.Connection,
    params: Dict[str, str],
    api_key: str,
) -> Tuple[Set[int], int, int]:
    url = API_URL
    query = dict(params)
    query['page_size'] = str(PAGE_SIZE)
    first_request = True
    page = 0
    unique_ids: Set[int] = set()
    max_count = 0

    while url:
        payload = request_json(
            session,
            url,
            api_key,
            params=query if first_request else None,
        )
        first_request = False
        page += 1
        results = payload.get('results', [])
        page_ids = [int(job['id']) for job in results]
        duplicate_rows = len(page_ids) - len(set(page_ids))
        unique_ids.update(page_ids)
        max_count = max(max_count, int(payload.get('count', 0)))
        save_jobs(conn, results)
        print(
            f'[page] page={page}, results={len(results)}, '
            f'unique={len(set(page_ids))}, duplicates={duplicate_rows}, '
            f'count={payload.get("count")}, '
            f'has_next={payload.get("next") is not None}'
        )
        url = payload.get('next')

    return unique_ids, max_count, page


def run_window(
    session: requests.Session,
    conn: sqlite3.Connection,
    params: Dict[str, str],
    api_key: str,
) -> Set[int]:
    union_ids: Set[int] = set()
    stable_passes = 0
    highest_count = 0

    for pass_number in range(1, MAX_PASSES + 1):
        pass_ids, observed_count, pages = fetch_pass(
            session, conn, params, api_key
        )
        new_ids = len(pass_ids - union_ids)
        union_ids.update(pass_ids)
        count_grew = observed_count > highest_count
        highest_count = max(highest_count, observed_count)
        print(
            f'[pass] number={pass_number}, pages={pages}, '
            f'pass_ids={len(pass_ids)}, union_ids={len(union_ids)}, '
            f'new_ids={new_ids}, observed_count={observed_count}'
        )

        if (
            pass_number >= MIN_PASSES and
            new_ids == 0 and
            not count_grew
        ):
            stable_passes += 1
        else:
            stable_passes = 0

        if stable_passes >= 2:
            break

    return union_ids


def iter_windows(start: date, until: date, days: int):
    cursor = start
    while cursor < until:
        window_end = min(cursor + timedelta(days=days), until)
        yield cursor, window_end
        cursor = window_end


def run_backfill(
    conn: sqlite3.Connection,
    api_key: str,
    start: date,
    until: date,
    window_days: int,
) -> int:
    session = requests.Session()
    all_ids: Set[int] = set()

    for window_start, window_end in iter_windows(start, until, window_days):
        params = {
            'published_since': window_start.isoformat(),
            'published_until': window_end.isoformat(),
            # Add stable scope filters here, for example:
            # 'country_code': 'US|CA|GB|DE',
        }
        ids = run_window(session, conn, params, api_key)
        all_ids.update(ids)
        print(
            f'[window] {window_start} to {window_end}: '
            f'window_ids={len(ids)}, global_union_ids={len(all_ids)}'
        )

    return len(all_ids)


if __name__ == '__main__':
    parser = argparse.ArgumentParser()
    parser.add_argument('--since', required=True, help='YYYY-MM-DD')
    parser.add_argument(
        '--until',
        required=True,
        help='Exclusive calendar boundary, YYYY-MM-DD',
    )
    parser.add_argument('--window-days', type=int, default=1)
    args = parser.parse_args()

    if args.window_days < 1:
        raise SystemExit('--window-days must be positive')

    conn = get_connection()
    total = run_backfill(
        conn,
        get_api_key(),
        date.fromisoformat(args.since),
        date.fromisoformat(args.until),
        args.window_days,
    )
    print(f'[complete] union IDs across windows={total}')

Run a small test first:

python multi_pass_backfill.py \
  --since 2025-01-01 \
  --until 2025-01-08 \
  --window-days 1

The --until value is a calendar boundary. Because the API date filters are inclusive, adjacent windows intentionally overlap at that boundary and the destination deduplicates the repeated IDs.

Incremental syncs

After the initial backfill, persist the greatest successfully observed ID as a durable checkpoint. The next run should begin at last_seen_id + 1.

For a bounded incremental range:

curl -G 'https://jobdataapi.com/api/jobs/' \
  -H 'Authorization: Api-Key YOUR_API_KEY' \
  --data-urlencode 'min_id=26400000' \
  --data-urlencode 'max_id=26409999' \
  --data-urlencode 'page_size=4000'

If this range spans several pages, follow next sequentially and apply the same multi-pass rule when missed records are costly. Do not advance the checkpoint until every page in the chosen range has been written successfully. If the range is too large or unstable, bisect it by ID and process the child ranges separately.

An ID checkpoint catches late-arriving listings whose published date is old. A short date overlap is still useful for refreshing records and recovering from delayed jobs, so mature pipelines commonly use both:

• ID checkpoint for new dataset entries
• short date overlap for refresh and recovery
• periodic reconciliation of older windows

At high volume, a max_age=2 request may itself span many pages. Use a one-day or bounded ID range for the regular pass, then resweep the previous one or two calendar days rather than assuming one huge rolling request is safer.

Regional and other filters

Country buckets are often easier to validate than broad region filters. For example:

country_code=US|CA|GB|DE|FR|NL
country_code=AU|NZ

Use the same multi-pass rules for every filtered stream. A filter that traverses a multi-valued relationship can produce duplicate SQL rows in some cases, so always validate API IDs rather than assuming count means unique jobs.

Keep filter definitions stable across a backfill. If you need a different market bucket or agency policy, give it a separate run identifier and destination scope.

Rate limits and request behavior

Keep requests sequential unless the API plan and your client have been explicitly designed for limited concurrency. Handle 429 responses with exponential backoff, and retry transient 5xx responses. Do not treat a successful retry as evidence that the page contents are identical to the previous attempt.

Use a requests.Session, a timeout on every request, and structured logs containing:

• window parameters
• pass number
• page number
• attempt number
• HTTP status
• response count
• number of results
• unique ID count
• duplicate row count
• whether next was present
• destination commit status

Never log the API key or full job descriptions in ordinary operational logs.

Bulk exports

If the account includes a CSV or Parquet bulk export suitable for the historical scope, prefer it for the initial multi-year load. Bulk exports avoid repeated page boundaries and are often simpler to checksum and archive. Use the API for incremental updates and targeted reconciliation after the bulk load.

Common failure modes

Treating page=2 as a continuation token

Page numbers are offsets, not immutable cursors. They are appropriate for browsing and for small result sets, but not a historical completeness guarantee when the result set is large or changing.

Running one giant historical query

A large query creates a long-lived moving target and makes a failed run expensive to restart. Use bounded calendar windows and make each window independently rerunnable.

Assuming ID filters imply ID ordering

min_id and max_id select an ID range. They do not guarantee that the response is sorted by ID. Use them to reduce a difficult window, then follow next and apply the same pass and reconciliation rules.

Deduplicating and assuming that fixes omissions

Deduplication protects the destination from repeated rows. It cannot identify a row that never appeared in any response. Multiple bounded passes improve practical coverage; ID and count metrics make remaining uncertainty visible.

Advancing a checkpoint too early

Persist the checkpoint only after all writes for the associated range have committed. A crash after advancing the checkpoint but before writing the data creates a permanent gap.

Relying on implicit defaults

Always specify the intended date, age, or ID scope. A request with no slicing parameter may be limited to the default recent window.

Running parallel requests without a recovery plan

Parallelism can increase rate-limit and timeout failures. If concurrency is necessary, make each worker own disjoint bounded windows and keep durable status for every window and pass.

Production checklist

Before a multi-year backfill:

• Confirm the subscription tier and endpoint visibility rules.
• Define whether expired jobs and agency jobs are included.
• Choose UTC calendar windows and record the exact filters.
• Sample several windows and measure page counts and pass-to-pass ID growth.
• Start with one-day windows for high-volume feeds.
• Run at least two passes per window.
• Continue passes only while they add useful IDs or counts are changing.
• Use bounded ID ranges for unusually large or unstable windows.
• Verify next completion for every pass.
• Use an idempotent destination keyed by API id.
• Keep date boundaries overlapped and deduplicate them.
• Retry 429 and transient 5xx responses with backoff.
• Persist window/pass completion only after successful writes.
• Run a second reconciliation sweep.
• Store request parameters and validation results for audit.

For ongoing syncs:

• Persist last_seen_id durably.
• Process bounded ID ranges or short date windows sequentially.
• Advance the checkpoint only after successful commits.
• Use a short date overlap for refreshes.
• Schedule periodic reconciliation of older windows.

Minimal validation plan

Run these tests before connecting the importer to production:

1. Fetch a small date window twice and compare unique ID sets.
2. Fetch adjacent overlapping windows and verify that the destination deduplicates the boundary.
3. Fetch a high-volume window with next and confirm every page is processed.
4. Run the same window twice and measure how many IDs the second pass adds.
5. Run the importer twice and confirm that destination row counts do not grow from duplicates.
6. Stop the importer after a committed page, restart the window, and confirm that upserts are safe.
7. Simulate a 429 and a transient 5xx response and verify backoff and retry behavior.
8. Compare the first backfill sweep with a reconciliation sweep and investigate every ID-set difference.

A reliable ingestion pipeline should make uncertainty visible, keep recovery cheap, and avoid claiming stronger guarantees than the API can provide.

Primary references:

• Jobs API Endpoint Documentation
• Date, ID, and Age Slicing Parameters
• Multi-value Parameters
• CSV and Parquet File Downloads
• Pricing and API access tiers
• jobdata API status

Related Docs