What Karpathy's Autoresearch Unlocked for Me

Model stacking combines the predictions of multiple base learners — such as XGBoost and CatBoost — and feeds them into a meta-learner that learns the optimal way to blend their outputs. In one documented experiment, switching from a single model to an XGBoost + CatBoost + meta-learner stack jumped AUC from 0.581 to 0.628 on a CRM tabular classification problem. The technique works because each base model captures different patterns in the data, and the meta-learner exploits their complementary strengths.

An AUC plateau in tabular classification often signals that your current feature set or model architecture has been fully exploited. The approach described in the article used an automated experiment loop to systematically try new strategies: model stacking, temporal feature engineering (year, quarter, month), target encoding with CatBoost, and improved cross-validation schemes. Each intervention broke through a successive ceiling, ultimately lifting AUC from 0.58 to over 0.67 across 165 experiments.

Temporal decay weighting assigns higher sample weights to more recent training examples and lower weights to older ones, based on the assumption that recent data better reflects current patterns. In the article, a research sub-agent autonomously proposed temporal decay weighting as a strategy after the main experiment loop hit a ceiling, pushing AUC from 0.6719 to 0.6747 and still climbing. This technique is especially useful for CRM and business data where customer behavior shifts over time.

Karpathy's autoresearch concept involves setting up an AI agent to run an automated experiment loop — continuously iterating on model training, evaluating results, and proposing new hypotheses without constant human supervision. As demonstrated in the article, a software engineer with no data science background applied this method overnight, and the agent ran 165 experiments that improved AUC by 15.6% on a tabular classification task. The most effective dynamic was a human-in-the-loop pattern: the agent explores freely, hits a ceiling, receives a directional nudge, and then continues experimenting.

Unlike AutoML tools such as AutoGluon that search a predefined model space, the autoresearch approach uses an AI agent to run open-ended experiments iteratively, discovering strategies like model stacking and feature engineering that AutoML pipelines might not explore. In this case, the agent ran 165 experiments overnight and achieved a 15.6% AUC gain by combining XGBoost, CatBoost, and a meta-learner — a creative solution that goes beyond typical automated ML tools. The key difference is that autoresearch agents can propose and test novel hypotheses rather than just tuning hyperparameters within a fixed search space.

CatBoost target encoding uses ordered target statistics to encode categorical features without leaking future information into the training data, which is especially valuable for CRM and tabular classification tasks. In this project, adding CatBoost with target encoding pushed the AUC from 0.654 to 0.669, a meaningful jump that the AI agent discovered automatically during its experiment loop. The ordered approach processes each observation using only data that appeared before it in the training set, making it a safer and more effective encoding method than naive target encoding.

The human-in-the-loop dynamic proved essential in this autoresearch experiment: the AI agent would explore aggressively, hit a performance ceiling, and then the author would provide a targeted nudge to redirect its search. For example, after the agent plateaued with basic model stacking, the author suggested exploring temporal features, which the agent then used to push AUC from 0.628 to 0.654. This pattern of AI agent exploration combined with human-in-the-loop ML training guidance repeated across 165 experiments, demonstrating that neither the human nor the agent alone would have achieved the full 15.6% AUC improvement.

Yes. The author is a software engineer, not a data scientist, and was stuck at 0.58 AUC for three weeks using manual approaches. By setting up an AI agent to run an automated experimentation loop overnight, the agent discovered techniques like model stacking and CatBoost with target encoding that the author had never heard of, ultimately achieving a 15.6% AUC gain across 165 experiments. The key was constraining the agent to modify only the training script while keeping the validation metric fixed.

The author started a tmux session from an iPhone on a Friday night, pointed an AI agent at a tabular classification problem, and let it iterate against a fixed AUC validation metric while sleeping. The agent ran 165 experiments autonomously, modifying only the training script while the validation methodology stayed constant. Overnight autonomous ML experimentation produced model stacking, temporal feature engineering, and CatBoost integration — techniques the author would not have tried manually — jumping AUC from 0.581 to 0.6747.

The article demonstrates that CRM data predictive modeling without a data science team is achievable when an AI agent handles the experimentation loop. Working alone on a personal project with CRM data, the author went from a stalled 0.58 AUC to 0.6747 by letting the agent explore model architectures, feature engineering, and ensemble methods across 165 experiments. Human judgment still mattered at key plateaus — the author nudged the agent with new research directions — but the technical heavy lifting was entirely agent-driven.