The Buddha notes that a householder's obligations prevent a householder from fully pursuing a monk's path. Thus, the Buddha articulates "the layman's duty" (Pali: gahatthavatta), what are essentially the Five Precepts, as follows:
Observe celibacy or at least do not have sex with another's wife
That sutta can be read here
Those that cannot observe the holy life, should at least not go to others' wives.
On the same page, but different sutta (#7 of the Sutta Nipata), the Buddha recollects ancient Brahmins who were moral, by not buying wives and by maintaining celibacy when "not in season", whatever that means:
290. Brahmins never went to another's wife, nor did they buy a wife
With mutual agreement they met, on equal terms.
291. Except in the season, at other times they abstained from sexuality,
Brahmins never went to women out of season.
292. . They honoured the holy life, virtues, straightforwardness, gentleness,
We should make a distinction here between householders seeking merit and householders seeking enlightenment. Sensual desire is a hindrance to enlightenment (for monks and laypeople alike) and sex is a form in which it is entertained, but sensual desire is not a hindrance to merit. Thus a householder seeking enlightenment would be wise to be celibate, but the Buddha did not require it. I would add that if a householder doesn't see the wisdom in celibacy of course
he shouldn't practice it. Seeing the true nature of things and trusting yourself is what enlightenment is all about; not arbitrary attachments to certain moral codes.