Pipelines, Not Models, Break ML Systems

After 4+ years building ML systems, the core failure mode isn't weak models but unstable pipelines that produce inconsistent results. A one-time success turns into quiet failures without disciplined stability practices. Treat stability as a non-negotiable discipline, not an afterthought.

Enforce Reproducibility by Seeding Everything

Randomness turns models into unreliable slot machines—results vary per run, undermining debugging and deployment. Fix it with a global seed function covering all sources:

import random
import numpy as np
import torch

def set_seed(seed=42):
    random.seed(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)

set_seed(42)

Call this early. Key caveat: Seeds don't fully eliminate non-determinism in some GPU operations—explicitly configure those for true reproducibility.

Note: Article outlines 9 rules total but details only the first here.