You should have no doubt that lust and affection are the enemies of loving-kindness (metta).
An entire chapter of the Dhammapada is devoted to the dangers of Affection
What the ordinary person with an undeveloped mind thinks of as love is defiled by affection, attachment, possessiveness, and lust. The Visuddhimagga goes into great detail on the way to develop the Brahmavihāra of Metta, saying that first one must get rid of all traces of anger and resentment — the far enemy — before one can begin the practice properly. Then, one should not choose a person of the opposite sex,¹ towards whom sexual desire could easily arise, nor a person who is a close relative, for whom affection and grief could arise. Rather, one should chose a person one holds in great respect such as a teacher or preceptor. It is easier to develop metta towards such a person, without the dangers of lust or affection. If developing metta towards your own mother or father, there might be no danger of lust, but the near enemy of affection could easily arise, and if any harm or danger threatens them, anger or fear might develop due to that affection.
¹ Presumably homosexuals should do though, as they would be likely to develop lust for a person of the same sex.