The suttas record many instances of people leaving wives and children as they go to the homeless life.
That said, I expect that it was socially more acceptable to do this back then than it is now.
"Do not force others, including children, by any means whatsoever, to adopt your views, whether by authority, threat, money, propaganda, or even education." - Ven. Thich Nhat Hanh
"The uprooting of identity is seen by the noble ones as pleasurable; but this contradicts what the whole world sees." (Snp 3.12)