Using the jobdata API for Machine Learning with Cleaned Job Descriptions

Table of contents

This tutorial builds a small machine-learning pipeline using English-language job listings from the jobdata API. We will cluster similar roles and extract mentions of selected skills using description_md, the Markdown version of each job description.

Markdown retains headings, lists and meaningful punctuation without requiring you to process the default HTML description. You can use it directly as model input or convert it to visible text for phrase matching. Choose preprocessing for the task rather than stripping all punctuation: C#, C++, .NET and snake_case carry information that is easily lost.

Prerequisites

You need Python and an API key with an active access subscription. Set the key in your JOBDATA_API_KEY environment variable, and install:

pip install requests pandas scikit-learn Markdown beautifulsoup4

Step 1: Fetch a Bounded Sample

Request description_md=true to include Markdown and description_off=true to omit the default HTML description. This example fetches up to three pages of recent English-language listings, rather than attempting to download the full dataset.

import os
import requests
import pandas as pd

url = 'https://jobdataapi.com/api/jobs/'
params = {
    'language': 'en',
    'description_md': 'true',
    'description_off': 'true',
    'max_age': 30,
    'page_size': 500,
}
rows = []

with requests.Session() as session:
    session.headers.update({
        'Authorization': f'Api-Key {os.environ["JOBDATA_API_KEY"]}',
    })
    for _ in range(3):
        response = session.get(url, params=params, timeout=60)
        response.raise_for_status()
        data = response.json()
        rows.extend(data['results'])
        url = data.get('next')
        if not url:
            break
        params = None  # The next URL already contains the query parameters.

if not rows:
    raise ValueError('No jobs returned; adjust the query before continuing')

jobs = pd.DataFrame(rows).drop_duplicates(subset='id').copy()
jobs['title'] = jobs['title'].fillna('')
jobs['description_md'] = jobs['description_md'].fillna('')
jobs = jobs[jobs['description_md'].str.strip().ne('')].copy()
if len(jobs) < 2:
    raise ValueError('At least two jobs with descriptions are needed')

Each row includes id, title, description_md, company information and location fields. Store the original Markdown alongside derived features so you can change preprocessing without fetching the same descriptions again. See the pagination guide for larger retrievals and respect your plan's request limits.

Step 2: Cluster Similar Roles

Clustering groups similar documents without predefined labels. It does not automatically discover validated occupation categories: clusters may reflect technologies, seniority, employer boilerplate or location as well as job function.

For this baseline, combine titles and Markdown descriptions and use character-based TF-IDF. Unlike the default word tokenizer, character features can retain distinctions involving punctuation, such as C# versus C++.

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.cluster import KMeans

documents = jobs['title'] + '\n' + jobs['description_md']
vectorizer = TfidfVectorizer(
    analyzer='char_wb',
    ngram_range=(2, 5),
    lowercase=True,
    max_features=50000,
)
features = vectorizer.fit_transform(documents)

num_clusters = min(8, len(jobs), len(set(documents)))
model = KMeans(n_clusters=num_clusters, random_state=42, n_init=10)
jobs['cluster'] = model.fit_predict(features)

for cluster, group in jobs.groupby('cluster'):
    print(f'\nCluster {cluster}:')
    print(group[['id', 'title']].head(10).to_string(index=False))

The vocabulary cap bounds the number of retained features, not total memory usage. Start with a small sample; character n-grams can be more expensive than word features. Markdown syntax, URLs and repeated company text can also influence clusters, so inspect examples and compare alternative preprocessing before drawing conclusions.

For supervised classification, use reviewed labels, evaluate on held-out data and apply the same text preparation at training and prediction time. Do not treat exploratory cluster IDs as ground-truth job categories.

Step 3: Extract Skill Mentions

Phrase matching benefits from visible text: **machine** learning should match “Machine Learning,” while a keyword appearing only in a Markdown link destination should not count.

import re
import markdown
from bs4 import BeautifulSoup

def markdown_to_text(value):
    rendered = markdown.markdown(value or '', extensions=['extra'])
    soup = BeautifulSoup(rendered, 'html.parser')
    for element in soup(['script', 'style']):
        element.decompose()
    return ' '.join(soup.get_text(' ', strip=True).split())

skills = [
    'Python', 'Java', 'JavaScript', 'SQL', 'C#', 'C++', '.NET',
    'Node.js', 'Machine Learning', 'Data Analysis', 'Project Management',
]
skill_patterns = {}
for skill in skills:
    phrase = r'\s+'.join(re.escape(part) for part in skill.split())
    # Boundaries also account for punctuation used in technical skill names.
    skill_patterns[skill] = re.compile(
        r'(?<![\w#+.])' + phrase + r'(?![\w#+])',
        re.IGNORECASE,
    )

def extract_skills(text):
    return [skill for skill, pattern in skill_patterns.items() if pattern.search(text)]

jobs['analysis_text'] = jobs['description_md'].map(markdown_to_text)
jobs['skill_mentions'] = jobs['analysis_text'].map(extract_skills)

mentions = jobs['skill_mentions'].explode().dropna()
print(mentions.value_counts())

This avoids matching Java inside JavaScript and preserves punctuation in C#, C++ and .NET. Each skill is counted at most once per listing. The dictionary is deliberately small: expand it with aliases, spelling variants and domain-specific terminology for your use case.

A mention is not necessarily a requirement. “No Java experience needed” still mentions Java; company descriptions and optional qualifications can also produce matches. Use contextual rules, reviewed examples or a dedicated extraction model when you need required skills, proficiency levels or years of experience.

Next Steps

• Compare Markdown-based features with locally rendered text using the same evaluation sample.
• Inspect clusters and tune the feature representation rather than assuming eight groups fit every dataset.
• Expand and evaluate the skill dictionary, especially for ambiguous terms and negation.
• Keep preprocessing consistent when serving trained models. If an existing model used different cleaning, evaluate or retrain it before switching inputs.

The key is to retain the original description and derive only the text representation each task needs. Markdown provides a reusable starting point for both model features and readable, context-aware analysis.