Brane instantons and fluxes in F-theory