0010: One job pipeline, with Telegram as its report channel¶
- Date: 2026-11-11
- Status: accepted
Problem¶
Kairos finds jobs twice, and the two halves do not know about each other.
- The API pipeline (
kairos.api.background) harvests the boards every 12 hours, scores up to 100 new postings with Haiku after each harvest, and re-checks up to 100 saved ones a day. Everything lands in PostgreSQL, where postings can be analysed, tailored, written for and tracked. On production it holds 4,121 open jobs. - The watch (
kairos.watch, systemd timers six times a day plus an 08:00 digest) runs its own search. It uses headless Claude Code with web search, its own LinkedIn and Seek feeds, and the ATS company boards. It scores postings in its own prompt, stores them in its own SQLite file (695 postings), and alerts on Telegram (24 alerts so far).
The owner's rule is that Telegram is a channel for reporting the pipeline, not a pipeline of its own. Keeping two pipelines has a cost:
- A posting the watch alerts on is not in Kairos. It cannot be analysed, tailored,
given documents or tracked without being added again by hand with
kairos jobs add. - The same posting is scored twice, by different rules. The API scores against
[profile.fit], and the watch against its own prompt andmin_score. The two can disagree about the same job. - The LinkedIn and Seek feeds are read twice. That doubles the requests to boards that rate-limit and block scrapers.
- Nothing reports the API pipeline's health. Three problems found on 2026-11-11 were visible only in logs:
- Every Seek AU search fails with Cloudflare's bot challenge (
403,cf-mitigated: challenge). This happens from the server and from the Mac alike. - The availability check covers 100 of 4,121 open jobs a day, so a job is re-checked about every 41 days, and closed ads stay listed. Only 18 have been archived.
- Harvesting added 580 and 783 postings on two days, against 200 scoring slots a day.
Options¶
- Keep two pipelines. This needs no work, and none of the costs above go away.
- Have the watch post what it finds to the API (
POST /api/v1/jobs). Its postings would reach PostgreSQL. But there would still be two schedulers, two scorers and two readers of the same feeds, and Telegram would still report the watch, not Kairos. - One pipeline, reported to Telegram. The API pipeline becomes the only one. The watch's two sources that the API lacks, the ATS boards and web search, become harvest sources writing to PostgreSQL. Telegram reports what the pipeline found and how it is running. The watch's SQLite store and its systemd timers are retired.
Choice¶
Option 3.
1. Telegram is only a report interface¶
Telegram reads what the backend did; it never decides what the backend does. How often
the pipeline harvests, which sources it reads and how it scores are backend settings
([jobs]), and the report shows their results whatever they are. Only what is reported,
and when, is set in [report].
kairos.report holds a ReportChannel port with one adapter, Telegram, reusing
kairos.watch.telegram's client and message formatting. It sends two kinds of message.
- Alerts, after each harvest-and-score cycle. Every new posting whose fit score is at
or above
[report] alert_min_score(default 75, the watch's) gets one ranked message per cycle. A posting is sent once:jobs.notified_atrecords it, and is set only after every message went out, so a failed send is retried next cycle. A posting first seen more than 48 hours ago is left to the digest, as in the watch. "First seen" needs its own column,jobs.first_seen_at, becausescraped_atmoves on every re-search. Migration 022 adds both, takesfirst_seen_atfromscraped_atfor existing rows, and marks every existing row notified, so the first cycle after it does not alert on postings found before it. - A digest, once a day at
[report] digest_hour(default 8, local time). It lists the top postings of the last 24 hours at or abovedigest_min_score(default 60), and then the pipeline's health: - harvest searches that failed, grouped by board and reason, such as "Seek AU: all 18 searches blocked by Cloudflare";
- postings still waiting for a score;
- availability coverage: the oldest check, and how many were archived;
- the LLM spend of the day, from
llm_usage.
The bot token and chat id are read from KAIROS_TELEGRAM_BOT_TOKEN and
KAIROS_TELEGRAM_CHAT_ID, injected into the backend container from 1Password like the
other secrets. kairos report chat-id finds the chat id. The harvest log behind the
digest's failure lines is kept in the API process, since a failed search leaves no row;
a restart empties it. POST /api/v1/reports/digest (kairos report digest) sends the
digest at once. The fit bands a report marks postings with live in
kairos.core.fit_bands, and a test checks that the fit-scoring prompt states the same
floors.
2. The watch's company boards become a harvest source¶
kairos.platform.ats already reads Greenhouse, Lever and Ashby boards. The harvest reads
the boards listed in [jobs] ats_boards, keeping postings whose titles match the harvest
keywords. Like every source, it writes to jobs, so the cross-board dedupe by role_key
applies and the fit scorer scores it against [profile.fit].
The watch's web search is not carried over. Its LinkedIn and Seek results duplicate the harvest, and the rest costs a Claude Code session per keyword and region (27 a cycle on production) for ads that rarely survive the fit score. It can return later as a harvest source if the boards prove to miss too much.
3. The pipeline keeps up with itself¶
- Harvest every 4 hours (
[jobs] harvest_hours = 4, was 12). Alerts follow each harvest, so they arrive about six times a day, as the watch's did. That triples the board requests, which the platforms' rate limiters already pace. - Availability. Each day the check covers every open job not checked within
[jobs] availability_max_age_days(default 7), rather than a fixed 100.availability_batch(default raised from 100 to 1,000) is now only a ceiling on one run. On production that is about 590 checks a day, spread over the platforms' existing rate limiters. - Scoring. After each harvest every unscored posting is scored, up to
[jobs] fit_scoring_batch(its maximum raised from 1,000 to 5,000; production sets 1,000) as a cost ceiling. Anything over the cap is carried to the next cycle and appears in the digest. - Seek AU and NZ. Both are removed from
kairos/platform/seek/sites.toml(NZ answers with the same challenge). Getting past a bot challenge is not something Kairos does, and Australia is covered by Adzuna (the official API) and LinkedIn. Seek HK (JobsDB) and NZ are unaffected. A board whose every search fails in a cycle is reported in the digest, so the next block of this kind is visible the same day.
4. Retiring the watch¶
The systemd job-watch and job-watch-digest timers are disabled once the first API
digest has arrived. kairos watch run | digest are removed. kairos watch chat-id moves
to kairos report chat-id. The 695 postings in the SQLite store are discarded: most are
older than the harvest's 14-day window, and the live ones are found again by the
harvest. [watch] roles, locations and commute_area fold into the harvest
keywords and [profile] preferred_cities, which already describe the same search.
5. Tests¶
- Report formatting: an alert, a digest with and without failures, and the 48-hour window.
notified_at: a posting is sent once, and survives a failed send.- A cycle with a failing board appears in the digest.
- Availability selection by age instead of a count, and the scoring cap carrying over.
- The ATS source against fakes, with the dedupe.
Not in scope¶
- Other report channels (email, Slack). The
ReportChannelport leaves room for them. - Getting through Cloudflare or any other bot protection.
- Telegram commands that act on Kairos, such as "tailor this job". Reports only.
Owner's authority¶
The bot token and chat id were found in 1Password already, in 02 Personal Production
("Telegram job-watch bot"), with the same values as /etc/machines/job-watch.env
(compared by hash). The owner approved using it on 2026-11-11; it was tagged
project/kairos, provider/telegram, env/production, status/active. The env file
stays until the watch timers are off; deleting it is the owner's call.