We propose a framework for the free field construction of algebras of local observables which uses as an input the Bisognano-Wichmann relations and a representation of the Poincaré group on the one-particle Hilbert space. The abstract real Hilbert subspace version of the Tomita-Takesaki theory enables us to bypass some limitations of the Wigner formalism by introducing an intrinsic spacetime localization. Our approach works also for continuous spin representations to which we associate a net of von Neumann algebras on spacelike cones with the Reeh-Schlieder property. The positivity of the energy in the representation turns out to be equivalent to the isotony of the net, in the spirit of Borchers theorem. Our procedure extends to other spacetimes homogeneous under a group of geometric transformations as in the case of conformal symmetries and of de Sitter spacetime.