I don't think rabbits can clim fences, That must mean that the neighbors, nor the rabbit owner, did not have fences.
However, I know from my own yard that rabbits go through chain link fences with ease, so I assume they wold only need a small hole a small opening in any fence.
Fences make good neighbors.
500 is much too many rabbits in a home. Flopsy, Mopsy and Cottontail should be plenty for anybody.