Plans to create five new homes in a village near Dereham have been refused due to concerns over its remote location.