Sign in Start free

Platform · Monitoring

Watch every model, and know why when it moves.

Log the predictions your stack already makes, or let Kaimei host the model and predict for you. Either way, every logged prediction can be explained after the fact, and drift arrives with the input that caused it.

Start for free Monitoring guide

Monitoring · Titanic example, notebook 02

A pipeline bug shows up as drift.

Two weeks of logged traffic with a shadow model, then a new feed that sends fares in pence. Nothing errors. Follow the timeline to see Kaimei catch it in one batch, and an explanation name it.

  1. 01Days 1 to 14

    Every prediction, logged

    A Titanic survival model is in production. The service that runs it logs each input and output to Kaimei; Kaggle's 418 test passengers stand in for two weeks of traffic. The band behind them is where the 891 reference passengers' fares sit.

  2. 02Days 1 to 14

    A shadow alongside

    A logistic regression scores the same passengers in shadow: logged, never acted on. The forest says 39% survive, the shadow 37%. They disagree on 28 passengers, which is where switching models would change real decisions.

  3. 03Day 14

    A clean check

    A drift analysis compares the latest logged inputs, up to 500, with the reference rows, one input at a time, using the population stability index. Below 0.1 is stable. Every input is: max PSI 0.03.

    Drift analysis · titanic-forestday 14

    Every input stable

    Recent inputs match the reference population (max PSI 0.03).

  4. 04Days 15 and 16

    A new feed goes live

    A new booking feed sends fares in pence, not pounds: the first passenger through it held a £7.75 ticket and is logged at 775. Nothing errors. Of its 300 passengers, 299 arrive with a fare higher than any in the reference data. Their average P(survived) is 51% as logged, against 41% with correct fares.

  5. 05Day 16

    Drift on one input

    The same check, two days later: 891 reference rows against the latest 500 logged predictions. Fare's PSI is 1.555, far past the 0.25 line for a significant shift. Every other input is stable. The predictions themselves have shifted too: PSI 0.499.

    Drift analysis · titanic-forestday 16

    891 reference rows vs the latest 500 logged predictions

    1.555Fare
    Age0.058
    Embarked0.028
    Title0.024
    FamSize0.010
    Parch0.002
    SibSp0.001
    Sex0.001
    Pclass0.000
    IsAlone0.000
    0.499Predictions

    Every other input 0.058 or less: stable

    stable < 0.1moderatesignificant > 0.25

  6. 06Day 16

    Flagged at high severity

    Kaimei records an Input drift insight at high severity, and the project page lists it under Needs attention. It says which input moved: Fare.

    Needs attention

    Input drift · Titanic survival · random forestRecent traffic has significant drift (PSI 1.55)… High

    Insightins_a85fe0d2be5820f47eb0

    Input drift

    DriftHighscore 1.555

    Recent traffic has significant drift (PSI 1.55). Shifted inputs: Fare.

    891 reference vs 500 recent rowspredictions PSI 0.499 · significant

  7. 07Day 16

    Named by an explanation

    Explain the first prediction logged from the new feed. For a third-class man, the “£775 fare” is the largest push toward survival, +14.3 points. The explanation is linked to the logged prediction by its id.

    Explanation · SHAP · titanic-forestprediction_id linked

    3rd class · male · Queenstown · travelling alone · Fare 775

    Fare = 775 GBP+14.3
    Sex = male−11.7
    Title = Mr−7.1
    Pclass = 3−4.0
    FamSize = 1+1.4
    SibSp = 0+1.2

    average 41.1%P(survived) 35.3% · died

  8. 08Day 16

    The fix

    With his fare in pounds, the same passenger's P(survived) falls from 35.3% to 9.4%. Fix the feed, re-score the 300 passengers it affected, and keep the drift check on a schedule.

    Same passenger, fare in poundstitanic-forest
    As logged775P(survived) 35.3%
    In pounds£7.75P(survived) 9.4%
    1. Fix the feed
    2. Re-score the 300 passengers it affected
    3. Keep the drift check on a schedule
    client.analyses.create(model="titanic-forest", type="drift")

From notebook 02 of the Titanic example. Passengers, fares and both models' outputs are real: re-running the notebook's code reproduces its printed results. The drift results, alert and explanation are the notebook's own. Staged: the notebook spreads its timestamps evenly over 16 days, and when each check runs and how the bars grow are illustrative.

Open notebook 02

In practice

Four pieces, one call each.

Drift needs at least 20 reference rows and 20 logged predictions; until then the insight says so.

Logged predictionsOne call per prediction, or batches. Inputs, outputs and latency, per model version.
Shadow modelsRun a candidate on live traffic beside the champion and compare before you switch.
Drift per inputPSI against the reference rows. Stable below 0.1, significant above 0.25, flagged within a batch.
VersionsShip v2 under the same model, keep v1's history, and compare their insights side by side.

Upload a model this afternoon.

The free plan covers the whole Titanic walkthrough with room to spare. Your own model is one call away.

Start for free Read the docs