job_seeker_ro_spider — MEJIX SRL Romania Scraper

job_seeker_ro_spider — un scraper pentru job-urile MEJIX SRL din România. Extrage anunțurile de pe mejix.com/jobs (HTML scraping cu cheerio, single-page) și le publică în peviitor.ro prin API-ul SOLR.

🌱 Derivat din job_seeker_ro_spider (template). Pentru pattern-ul complet de derivare, vezi CONTRIBUTING.md din template.

Overview

Proiectul automatizează colectarea zilnică a job-urilor MEJIX din România, menținând board-ul peviitor.ro la zi cu cele mai recente oportunități de carieră.

Features

• Extrage job-uri de pe pagina HTML mejix.com/jobs (cheerio, single-page)
• Validează compania via ANAF (CUI, status activ/inactiv, adresă completă)
• Cache ANAF la 7 zile — committed în repo, nu lovește demoANAF la fiecare scrape
• Fallback la cache stale dacă ANAF e indisponibil
• Cross-validează cu Peviitor API
• Stochează în SOLR (job core + company core)
• Generează docs/jobs.md automat — accesibil pe GitHub Pages
• Identitate companie într-un singur fișier (config/company.json) — derivare ușoară pentru alte companii
• GitHub Actions: scrape zilnic + testare automată (unit, integration, e2e, consistency)
• Teste SOLR condiționale — auto-skip când SOLR_AUTH nu e setat
• Se identifică prin User-Agent: job_seeker_ro_spider

Project Structure

├── index.js                    # Main scraper entry point
├── company.js                  # Company validation via ANAF + Peviitor + SOLR
├── demoanaf.js                 # CLI wrapper for src/anaf.js
├── solr.js                     # SOLR operations (query, upsert, delete, company)
├── validate-jobs.js            # Job URL validator — checks active/expired, deletes stale jobs
├── config/
│   ├── company.json            # Single source of truth: CIF, brand, URLs, API params
│   └── company.js              # ESM loader for company.json
├── src/
│   ├── anaf.js                 # ANAF API core module (search + company details)
│   ├── markdown-generator.js   # Generates docs/jobs.md from scraped data
│   └── job-validator.js        # Shared validateByHead + validateByContent
├── company.json                # ANAF data cache (committed, 7-day TTL)
├── tests/
│   ├── package.json            # Jest config for test suite
│   ├── company.json            # Mock ANAF data used in unit tests
│   ├── validate-mejix-jobs.js   # SOLR job URL validation script
│   ├── unit/
│   │   ├── index.test.js       # Tests for parseHtmlJobs, mapToJobModel, transformJobsForSOLR
│   │   ├── company.test.js     # Tests for validateAndGetCompany, fallback caching
│   │   ├── solr.test.js        # Tests for query, upsert, delete operations
│   │   └── demoanaf.test.js    # Tests for ANAF search and company retrieval
│   ├── integration/
│   │   └── workflow.test.js    # Live ANAF + SOLR integration tests
│   ├── e2e/
│   │   └── scraper.test.js     # Full pipeline tests with real MEJIX HTML
│   └── consistency/
│       ├── public.test.js      # Verifies repo is public
│       ├── repo.test.js        # Verifies branch, Pages, secrets, workflows
│       ├── topics.test.js      # Verifies required repo topics
│       └── workflow-naming.test.js  # Validates workflow naming conventions
├── docs/
│   ├── index.html              # Live job board (GitHub Pages)
│   ├── jobs.md                 # Scraped jobs in markdown (generated by CI)
│   ├── README.md
│   └── test-results/           # Test reports (generated by CI)
│       ├── index.html
│       ├── pre-scrape-unit.html
│       ├── pre-scrape-integration.html
│       ├── post-scrape.html
│       └── post-scrape-consistency.html
├── .github/
│   ├── CODEOWNERS
│   └── workflows/
│       ├── job-seeker-ro-spider.yml     # Daily scraping at 6 AM UTC
│       └── automation-testing.yml       # Automation Tests on push/PR
└── package.json

Setup

Prerequisites

• Node.js 24+
• npm

Installation

npm install

Configuration

Set the SOLR_AUTH environment variable with your Solr credentials:

export SOLR_AUTH="username:password"

Usage

Run the Scraper

npm run scrape

Run Tests

# All tests
npm test

# Unit tests only
npm run test:unit

# Integration tests
npm run test:integration

# E2E tests
npm run test:e2e

Workflows

Daily Scraping

The job-seeker-ro-spider.yml workflow runs daily at 6 AM UTC via GitHub Actions. It:

1. Runs pre-scrape tests (unit + integration)
2. Validates company data via ANAF
3. Scrapes current job listings from mejix.com/jobs
4. Updates Solr with new/removed jobs
5. Runs post-scrape tests (e2e + consistency)
6. Uploads test results and job data as artifacts
7. Generates docs/jobs.md with company info and all scraped jobs
8. Pushes test reports and docs/jobs.md to docs/

Test Automation

The automation-testing.yml workflow runs on every push and pull request. It:

1. Ensures MEJIX exists in the company core
2. Runs unit, integration, e2e, and consistency tests
3. Validates data integrity in Solr
4. Pushes test reports to docs/test-results/

Acknowledgments

This project was developed with assistance from Claude Code by Anthropic.

Special thanks to the open source community and the peviitor.ro team for their support.

License

Copyright (c) 2024-2026 BOGA SEBASTIAN-NICOLAE

Licensed under the MIT License.

Managed By

This project is managed by ASOCIATIA OPORTUNITATI SI CARIERE and used as a web scraper for the peviitor.ro job board project.

Robots.txt Policy

Acest scraper respectă regulile din robots.txt al MEJIX Careers. Pentru analiza completă, vezi ROBOTS.md.

Puncte cheie:

• API-ul /api/* este Disallow în robots.txt — scraper-ul îl folosește, dar cu rate limiting și un singur User-Agent identificabil (job_seeker_ro_spider)
• Paginile individuale de job (/*/vacancy/*) sunt Disallow — scraper-ul NU le parsează, doar le verifică accesibilitatea via HEAD request
• Endpoint-urile permise (/, /en/jobs) nu sunt scraper-uite
• Comportament: 1 cerere/10 job-uri, delay 1s între pagini, fără concurență

Disclaimer

This scraper is designed for educational purposes and legitimate job data aggregation for the Romanian job market. Please respect MEJIX's Terms of Service and robots.txt when using this scraper.