Practical Parquet Analytics: Hiring Signals and Labor-Market Intelligence

Table of contents

Why use the Parquet download?

The weekly jobs_companies.parquet download is a practical starting point when the question is analytical rather than transactional. It contains the full historical job archive, including expired jobs, and puts the most commonly needed company attributes next to each job. That means you can scan a large time range without paginating through the API or rebuilding the basic job-to-company join yourself.

The file has one row per job. A row contains job fields such as job_title, published_at, job_description_md, salary values, and remote-work flags. It also contains nested lists for locations, job types, and tags, plus fields such as company_name, company_industry_name, and company_website_url.

This tutorial uses that structure for two workflows:

1. Turn hiring activity into B2B intent signals and a ranked prospect list.
2. Build a labor-market benchmark for salary and remote-work patterns.

The examples use DuckDB because it can query Parquet directly, push filters down into the file, and return only the small aggregate result that you need. Do not start by loading the complete download into a Pandas DataFrame. A weekly snapshot can be many gigabytes even when the final report contains only a few hundred rows.

Prerequisites

File downloads are available to access ultra customers. Generate a download link from the dashboard and keep the download access key private. Do not put the complete URL in a public notebook, source repository, or client-side application.

Download the file on a machine with enough disk space for the compressed file and any outputs you plan to create:

curl -L -o jobs_companies.parquet "https://jobdataapi.com/download/<DOWNLOAD_ACCESS_KEY>/jobs_companies.parquet"

Install the small local analysis stack:

python -m pip install duckdb pandas matplotlib

Create a connection and a helper for returning aggregate queries as DataFrames:

from datetime import datetime, timedelta, timezone

import duckdb

PARQUET_PATH = 'jobs_companies.parquet'
con = duckdb.connect()


def query(sql, parameters=None):
    return con.execute(sql, parameters or []).fetchdf()

For a quick schema check, select no data and inspect the result:

print(con.execute(
    'DESCRIBE SELECT * FROM read_parquet(?)',
    [PARQUET_PATH],
).fetchdf()[['column_name', 'column_type']])

The Parquet export uses UTC timestamps. Use UTC boundaries in recurring reports so that a report does not change depending on the machine's local time zone.

Use case 1: Hiring signals for B2B lead generation

The idea

A company hiring for a particular capability may be a timely prospect for a vendor that helps with that capability. A Salesforce consultancy can look for Salesforce roles. A cloud security provider can look for security, Kubernetes, or infrastructure hiring. A recruiting agency can identify companies with a sudden increase in demand for a target job family.

This is a signal, not proof of buying intent. The useful output is a ranked list for human review, not an automatically sent message to every company in the result.

Define a signal and two comparison periods

The example below looks for a group of cloud and observability terms. Replace SIGNAL_PATTERN with terms that match your product. Keep the pattern reviewed and specific enough to avoid counting unrelated uses of a word.

We compare the most recent 30 days with the 30 days before that. Calculating the boundaries at runtime makes the same script useful next week. The Parquet file is refreshed weekly, so the most recent period will contain only the records available in the downloaded snapshot.

The Parquet export includes company_is_agency, a boolean company classification. Set it to false when recruiting agencies are unwanted hits:

end = datetime.now(timezone.utc)
current_start = end - timedelta(days=30)
previous_start = current_start - timedelta(days=30)

SIGNAL_PATTERN = (
    'datadog|observability|kubernetes|terraform|'
    'cloud security|application security'
)

Aggregate signals by company

The query filters the date range before testing the text pattern. It also projects only the columns required for the report. This keeps the scan smaller than selecting descriptions, locations, and every company field.

signal_sql = r'''
WITH scoped AS (
    SELECT
        company_id,
        company_name,
        company_website_url,
        company_industry_name,
        job_title,
        published_at,
        CASE
            WHEN published_at >= CAST(? AS TIMESTAMPTZ) THEN 'current'
            ELSE 'previous'
        END AS period,
        lower(
            coalesce(job_title, '') || ' ' ||
            coalesce(job_description_md, '')
        ) AS searchable_text
    FROM read_parquet(?)
    WHERE company_is_agency = false
      AND published_at >= CAST(? AS TIMESTAMPTZ)
      AND published_at < CAST(? AS TIMESTAMPTZ)
), matching AS (
    SELECT
        company_id,
        any_value(company_name) AS company_name,
        any_value(company_website_url) AS company_website_url,
        any_value(company_industry_name) AS company_industry_name,
        count(*) FILTER (WHERE period = 'current') AS current_signal_jobs,
        count(*) FILTER (WHERE period = 'previous') AS previous_signal_jobs,
        max(published_at) AS latest_signal_at,
        list(job_title ORDER BY published_at DESC)[:5] AS example_titles
    FROM scoped
    WHERE regexp_matches(searchable_text, ?)
    GROUP BY company_id
)
SELECT
    company_id,
    company_name,
    company_website_url,
    company_industry_name,
    current_signal_jobs,
    previous_signal_jobs,
    current_signal_jobs - previous_signal_jobs AS change_in_signal_jobs,
    latest_signal_at,
    example_titles
FROM matching
WHERE current_signal_jobs > 0
ORDER BY change_in_signal_jobs DESC, current_signal_jobs DESC
'''

leads = query(signal_sql, [
    current_start,
    PARQUET_PATH,
    previous_start,
    end,
    SIGNAL_PATTERN,
])

print(leads.head(20).to_string(index=False))

The result has one row per company_id, even if a company has posted many matching jobs. The company_is_agency = false predicate excludes recruiting agencies before aggregation. The two counts make the ranking more useful than a simple total: a company with six current matching jobs and no prior-period jobs is a different prospect from a company with six jobs in both periods.

Export the reviewed list for a CRM or sales-operations workflow:

leads.to_csv('hiring_signal_leads.csv', index=False)

Improve the signal before using it operationally

Start with a broad signal pattern to understand its volume, then add quality controls:

• Require at least two matching jobs or a minimum current-period count.
• Restrict by company_industry_name or a selected group of industries.
• Use job_application_url or the examples in example_titles for manual verification.
• Deduplicate on company_id, not on the company name, because names are not stable identifiers.
• Store the snapshot date and the query pattern with every export so the lead list is auditable.

The company fields company_num_jobs_open and company_num_jobs_total are useful context, but they are snapshots from export time. Do not use them as a historical time series unless you have saved multiple weekly exports.

Use case 2: Salary and remote-work benchmarking

The idea

Compensation research often starts with a title, an industry, or a location and asks questions such as:

• How does the advertised salary for a job family differ by industry?
• Is remote hiring becoming more or less common over time?
• Are salary ranges widening for a particular group of roles?

The Parquet export includes annual salary values in the original currency as well as converted USD and EUR values. For a cross-market comparison, use the converted fields and require both endpoints of the range. In this example, we use the midpoint of the USD range and report the median rather than relying only on an average, because a few unusually large ranges can pull the average upward.

Create comparable job families

The export has the original job title, not a universal job-family taxonomy. For a first benchmark, define a transparent set of title rules. For a production report, replace these rules with your own taxonomy and version it along with the report.

benchmark_sql = r'''
WITH prepared AS (
    SELECT
        published_year,
        company_industry_name,
        job_has_remote,
        (job_salary_min_usd + job_salary_max_usd) / 2.0 AS salary_midpoint_usd,
        CASE
            WHEN regexp_matches(lower(job_title),
                '(machine learning|ml engineer|artificial intelligence|ai engineer)')
                THEN 'AI and ML'
            WHEN regexp_matches(lower(job_title),
                '(data scientist|data analyst|analytics engineer)')
                THEN 'Data and Analytics'
            WHEN regexp_matches(lower(job_title),
                '(software engineer|software developer|backend engineer|frontend engineer)')
                THEN 'Software Engineering'
            WHEN regexp_matches(lower(job_title),
                '(product manager|product owner)')
                THEN 'Product'
            ELSE 'Other'
        END AS job_family
    FROM read_parquet(?)
    WHERE published_year IN (2025, 2026)
      AND job_salary_min_usd IS NOT NULL
      AND job_salary_max_usd IS NOT NULL
)
SELECT
    published_year,
    job_family,
    count(*) AS jobs,
    round(median(salary_midpoint_usd), 0) AS median_midpoint_usd,
    round(quantile_cont(salary_midpoint_usd, 0.10), 0) AS p10_midpoint_usd,
    round(quantile_cont(salary_midpoint_usd, 0.90), 0) AS p90_midpoint_usd,
    count(*) FILTER (WHERE job_has_remote) AS remote_jobs,
    round(
        100.0 * count(*) FILTER (WHERE job_has_remote) / count(*),
        1
    ) AS remote_share_percent
FROM prepared
WHERE job_family <> 'Other'
GROUP BY published_year, job_family
HAVING count(*) >= 100
ORDER BY published_year, median_midpoint_usd DESC
'''

benchmark = query(benchmark_sql, [PARQUET_PATH])
print(benchmark.to_string(index=False))

The HAVING clause prevents tiny groups from looking authoritative. Adjust the years and minimum count for the size of your report. If you need a benchmark for one industry, add company_industry_name to the selected columns and GROUP BY clause, then filter out null industry values.

The job_has_remote value is an indicator derived from the job listing. Treat a false value as not marked remote by the source data, not as proof that the employer never permits remote work. Also remember that a single job can have multiple countries or cities in its nested location lists.

Visualize the result

Because benchmark contains only aggregate rows, plotting it is inexpensive even when the source file is large:

import matplotlib.pyplot as plt

plot_data = benchmark[benchmark['published_year'] == 2026].copy()
plot_data = plot_data.sort_values('median_midpoint_usd')

ax = plot_data.plot.barh(
    x='job_family',
    y='median_midpoint_usd',
    legend=False,
    figsize=(9, 5),
    title='Median advertised salary midpoint by job family',
)
ax.set_xlabel('USD per year')
ax.set_ylabel('')
plt.tight_layout()
plt.show()

Save the benchmark as a data product rather than only as an image:

benchmark.to_csv('salary_remote_benchmark.csv', index=False)

Add geography without losing count accuracy

Location fields such as job_countries are native Parquet lists of structs. They can be unnested in DuckDB. A job with multiple countries then produces multiple country associations, so use count(DISTINCT job_id) for job counts and clearly document that multi-country jobs contribute to each associated country.

country_sql = r'''
SELECT
    country.code AS country_code,
    country.name AS country_name,
    count(DISTINCT jobs.job_id) AS jobs
FROM read_parquet(?) AS jobs
CROSS JOIN UNNEST(jobs.job_countries) AS countries(country)
WHERE jobs.published_year = 2026
GROUP BY country.code, country.name
ORDER BY jobs DESC
LIMIT 20
'''

country_counts = query(country_sql, [PARQUET_PATH])
print(country_counts.to_string(index=False))

Do not use the unnested query for a salary average without deciding how multi-country jobs should be weighted. If each job should count once in a global salary benchmark, aggregate salary at job level first. If the question is specifically about every location an employer advertised, explain that the same job may appear in more than one country group.

Operational notes

The download is a weekly snapshot, not a live table. Save the download date with every output and rerun the same query against each snapshot when you need a historical time series. The file includes historical and expired jobs, so add an explicit published_at or published_year filter for current-market reports.

Use column projection and date predicates whenever possible. DuckDB can read only the columns needed by a query, but a text search over job_description_md is still more expensive than a query over numeric or categorical columns. Test a narrow period first, inspect the output, and widen the range only when the result is correct.

The Parquet export intentionally does not include job embeddings or the original HTML description. Use the separate CSV downloads when you need those fields or the original relational export structure. Use the API when you need current endpoint filtering or application-style pagination. Use Parquet when you need repeatable scans, joins avoided by denormalized company fields, historical jobs, and local analytical control.

For the complete file list, access rules, field reference, and download-link guidance, see the CSV and Parquet file downloads documentation.

Related Docs