Multi-Source Sensor ML Pipeline: Pest Population Forecasting
We compared 11 models across count forecasting and catch-day detection on 245 trap-days from 5 traps. A re-analysis found that a key input leaked the answer; without it, no model beats simple baselines.
245
Trap-days
5
Traps
21
Catch days
11
Models compared
Question
Can a few weeks of trap counts plus local weather warn of new insect catches, and which kind of model does it best when catch days are rare (21 of 245 trap-days)?
- Problem
- Growers check insect traps to decide when to treat crops, and a missed catch costs more than a false alarm. Can a few weeks of trap counts plus local weather warn of new catches when catch days are rare (21 of 245 trap-days)?
- My role
- Team of 3 (Information Systems and Business Intelligence, Federico II). I designed the model comparison and evaluation and did the re-analysis; Farshad Farahtaj led preprocessing, exploratory analysis and feature engineering; Giovanni Montanile built the Power BI reports.
- Result
- A negative result. With a leaking input removed, the Random Forest classifier scores AUC 0.49 and F1 0.36 on 49 test trap-days (9 with a catch), against the original 0.92 and 0.667; the regression's mean error rises from 0.355 to 0.496, worse than predicting yesterday's count (0.34).
Honest limits
- This was the only data available: 5 traps over 49 days (6 July to 23 August 2024), with 21 catch days in two bursts.
- The original input "recent activity" was 1 on every catch day including the day itself, so it leaked the answer; the original scores cannot be used.
- Synthetic catch days (SMOTE) made the classifier worse, not better.
Runs right here on this page. It loads from Hugging Face, which may set its own cookies, and a sleeping demo takes 30 to 60 seconds to wake up.
Weather and trap counts from five field traps
Growers check insect traps to decide when to treat crops, and a missed catch costs more than a false alarm. This was the only data available for the university project: 5 traps over 49 days (6 July to 23 August 2024), 245 trap-days with average temperature, humidity and temperature range, and 21 catch days in two bursts.
The data was split by date: the first 196 rows for training and the last 49 for testing. We engineered lag, 3-day rolling-mean, recency and calendar features.
Two tasks, 11 models, one time-ordered split
- Count regression (6 models): a regression task (estimate the day's catch count) with ARIMAX, SARIMAX, Prophet, Random Forest, XGBoost and LightGBM.
- Catch-day classification (5 models): whether any new insect is caught, with Random Forest, XGBoost, LightGBM, LSTM and GRU. Tree models were tuned with time-ordered cross-validation and
class_weight='balanced'. The threshold search found no useful value (validation F1 was 0 at every threshold), so it stayed at the grid's lowest value, 0.10. - Model choice: we ran both as tournaments; the winner was picked on the same test window, so even the original scores were optimistic.
A key input leaked the answer
The original classifier looked strong. When I re-checked the features, the input "recent activity" turned out to be 1 on every catch day, including the catch day itself, so it already contained the answer. It was the strongest feature in the model, and the regression used it too.
I re-ran both notebooks with that input removed, on the same split, the same search grid and the same threshold search, and compared the result with simple baselines.
Without the leak, no model beats simple baselines
With the leaking input removed, the Random Forest classifier scores AUC 0.49 and F1 0.36 on 49 test trap-days (9 with a catch), against the original 0.92 and 0.667. The regression's mean error rises from 0.355 to 0.496 insects per trap-day, worse than simply predicting yesterday's count (0.34). Adding synthetic catch days to training (SMOTE) made the classifier worse, not better.
Catch-day classifier
49 test trap-days, 9 with a catch (higher is better)
| Model | AUC | F1 |
|---|---|---|
| Original Random Forest (with the leaking input)Leaked, not a result | 0.92 | 0.667 |
| Random Forest, leaking input removedHonest score | 0.49 | 0.36 |
| Baseline: predict a catch every daySimple baseline | 0.5 | 0.31 |
An AUC near 0.5 means the model cannot tell catch days from other days. With the leak removed, the Random Forest ranks catch days no better than chance.
Daily count regression
Mean error in insects per trap-day, 50 test trap-days (lower is better)
| Model | Mean error |
|---|---|
| Original Random Forest (with the leaking input)Leaked, not a result | 0.355 |
| Random Forest, leaking input removedHonest score | 0.496 |
| Baseline: yesterday's count at the same trapSimple baseline | 0.34 |
| Baseline: predict 0Simple baseline | 0.52 |
With the leak removed, the Random Forest's error is worse than simply repeating yesterday's count.
The honest conclusion: on 245 trap-days with 21 catch days, none of these models forecasts catches better than a simple baseline. This is a negative result, reported as one. Scripts and tables for the re-analysis are in the reanalysis folder on GitHub.
The corrected dashboard
A Streamlit dashboard on Hugging Face with the data exploration, the corrected results next to the simple baselines, and what was withdrawn and why. The prediction form was removed, because no model beat the baselines.
Loads Pest forecasting dashboard from huggingface.co, which may set its own cookies. A sleeping demo takes 30 to 60 seconds to wake up.
Who did what
Team of 3 (Information Systems and Business Intelligence, Federico II). I designed the model comparison and evaluation and did the re-analysis; Farshad Farahtaj led data preprocessing, exploratory analysis and feature engineering; Giovanni Montanile built the Power BI reports.
Frameworks and infrastructure
Source code on GitHub.