Castelnuovo theory and the geometric Schottky problem