The Layers We Bring To The Saddle