avtonet-monitor

Terminal-based monitor for avto.net car listings. Scrapes search results and detail pages through FlareSolverr, stores everything in SQLite, estimates fair prices with OLS regression, and presents it all in an interactive TUI.

Features

• Multi-monitor scraping -- track multiple search URLs with configurable page depth
• CloudFlare bypass via FlareSolverr proxy
• Fair price estimation -- OLS regression on age, mileage, and equipment count
• Depreciation analytics -- per-year and per-10k-km depreciation, effective age adjusted for mileage
• Interactive TUI -- three-tab interface (Listings, Model Summary, Scrape Log) built on Textual
• Filtering -- modal filter dialog for year, price, mileage ranges; active-only and "bangers only" (underpriced) modes
• Auto-scrape -- configurable interval with real-time progress in the status bar
• Equipment parsing -- counts value-relevant features (navigation, leather, panoramic roof, ACC, etc.)
• Fuel-type exclusion -- hide unwanted fuel types (e.g. EVs) from both tabs
• Backfill utilities -- one-off scrape of arbitrary search URLs; re-scrape detail pages for missing equipment data

Requirements

• Python 3.11+
• Docker (for FlareSolverr)

Installation

pip install -e .

Start FlareSolverr:

docker run -d --name flaresolverr -p 8191:8191 ghcr.io/flaresolverr/flaresolverr:latest

The app will auto-start the container on launch if it exists but is stopped.

Configuration

Create a config.toml in the project root:

[database]
path = "avtonet.db"

[scraper]
flaresolverr_url = "http://localhost:8191/v1"
crawl_delay_seconds = 10
request_timeout_seconds = 60

[monitor]
interval_minutes = 30
exclude_fuel_types = ["električni pogon", "elektro pogon"]

[[monitors]]
name = "Toyota Corolla Hybrid"
url = "https://www.avto.net/Ads/results.asp?znamka=Toyota&model=Corolla&..."
max_pages = 3

[[monitors]]
name = "VW Caddy"
url = "https://www.avto.net/Ads/results.asp?znamka=Volkswagen&model=Caddy&..."
max_pages = 3

# Known new-car prices for depreciation estimates.
# Key format: "Make Model engine_cc power_kw"
[new_prices]
"Toyota Corolla 1798 103" = 32000
"Volkswagen Caddy 1968 75" = 31500

Section	Key	Description
[database]	path	SQLite file location
[scraper]	flaresolverr_url	FlareSolverr endpoint
[scraper]	crawl_delay_seconds	Delay between HTTP requests
[scraper]	request_timeout_seconds	Per-request timeout
[monitor]	interval_minutes	Auto-scrape interval (0 to disable)
[monitor]	exclude_fuel_types	Fuel types to hide from display
[[monitors]]	name, url, max_pages	Search URL to track
[new_prices]	"Make Model cc kw"	Known new-car price in EUR

Usage

Interactive TUI (default)

avtonet-monitor config.toml

Key	Action
r	Start a scrape cycle
f	Open filter dialog
1 2 3	Switch tabs (Listings / Summary / Log)
Enter or click	Open listing in browser
q	Quit

The Listings tab shows individual cars with estimated fair price and delta (green = underpriced, red = overpriced).

The Model Summary tab aggregates by make/model/engine and shows depreciation metrics: effective age, depreciation %, estimated new price, cost per year, and cost per 10k km. New-car inventory (current-year, <1000 km) is excluded from used-car stats.

The Filter modal (f) lets you narrow both tabs by year range, price range, mileage range, active-only, or bangers-only (negative price delta).

Backfill search results

One-time scrape of an arbitrary search URL:

avtonet-monitor config.toml backfill-search "https://www.avto.net/Ads/results.asp?..."

Backfill equipment data

Re-scrape detail pages for listings missing equipment info:

avtonet-monitor config.toml backfill

Architecture

avtonet_monitor/
├── __main__.py   # CLI entry point, FlareSolverr auto-start, backfill commands
├── scraper.py    # Async HTTP client wrapping FlareSolverr sessions
├── parser.py     # HTML parsing for search results and detail pages
├── db.py         # SQLite storage with thread-safe operations (WAL mode)
├── pricing.py    # OLS regression price estimator + depreciation model
└── tui.py        # Textual TUI with filter modal and live scrape progress

Scraping pipeline

1. AvtonetScraper creates a FlareSolverr session for persistent cookies
2. Search pages are fetched and parsed into listing items (ID, title, price, specs)
3. Detail pages are scraped for full specs, equipment, and seller info
4. Listings are upserted into SQLite; unseen listings are marked inactive
5. estimate_prices() fits an OLS model on age + mileage + equipment count

Price estimation

A linear regression is trained on all active listings with sufficient data (price, year, mileage):

estimated_price = β₀ + β₁·age + β₂·mileage_km + β₃·equipment_count

Equipment is scored by counting matches against Slovenian-language keywords for features like navigation, leather seats, panoramic roof, adaptive cruise control, etc.

Depreciation model

For the summary tab, depreciation uses a Weibull-style curve:

• Effective age combines calendar age with mileage deviation from 15,000 km/year standard
• New price is sourced from: (1) actual new listings in the group, (2) exact config match, or (3) fuzzy config match by nearest kW
• Depreciation is reported as total %, per effective year, and per 10,000 km

Development

Setup

pip install -e ".[dev]"
pre-commit install

Running tests

pytest tests/ -v

# With coverage
pytest tests/ --cov=avtonet_monitor

Pre-commit hooks

Hooks run automatically on commit:

• black -- code formatting (Python 3.13 target)
• isort -- import sorting (black-compatible profile)
• pylint -- static analysis (10.00/10 target)

CI

Forgejo Actions pipeline runs on pushes to master and on pull requests:

1. Format check -- black --check and isort --check-only
2. Lint -- pylint on all source files
3. Test -- pytest full suite

License

Private project.