Buddhism is a practice, not a state of being.
One who practices Buddhism is a Buddhist.
You can undergo initiation as a monk one day and quit practicing the next, sometimes without even knowing that you've quit.
The Buddha stated that even after being accepted into the Order and receiving the precepts from the Buddha himself, one who acts in such and such a way "Is not my disciple and does not understand my teachings." One who acts in such and such a different way, is, and does understand.
People can say what they want, and as it seems you've already found out, they will. There is no Ministry of Buddhism in charge of handing out certificates and deciding who is a Buddhist and who is not.
There is no authority but the Buddha. Since we were not born in the Buddha's lifetime, we look to the sutras for instruction; "Rely on the Law and not upon persons." Reflecting deeply on our own conduct in the light of the Buddha's teaching we ourselves can borrow the wisdom of the Buddha and make our own judgment whether we are disciples of the Buddha and understand his teachings or not.