The Buddha to be was raised in a palace(s).
It is in the Pali Canon, including the EBT:
"Monks, I lived in refinement, utmost refinement, total refinement. My father even had lotus ponds made in our palace: one where red-lotuses bloomed, one where white lotuses bloomed, one where blue lotuses bloomed, all for my sake. I used no sandalwood that was not from Varanasi. My turban was from Varanasi, as were my tunic, my lower garments, & my outer cloak. A white sunshade was held over me day & night to protect me from cold, heat, dust, dirt, & dew.
"I had three palaces: one for the cold season, one for the hot season, one for the rainy season. During the four months of the rainy season I was entertained in the rainy-season palace by minstrels without a single man among them, and I did not once come down from the palace. Whereas the servants, workers, & retainers in other people's homes are fed meals of lentil soup & broken rice, in my father's home the servants, workers, & retainers were fed wheat, rice, and meat.
"Even though I was endowed with such fortune, such total refinement, the thought occurred to me: 'When an untaught, run-of-the-mill person, himself subject to aging, not beyond aging, sees another who is aged, he is horrified, humiliated, & disgusted, oblivious to himself that he too is subject to aging, not beyond aging. If I — who am subject to aging, not beyond aging — were to be horrified, humiliated, & disgusted on seeing another person who is aged, that would not be fitting for me.' As I noticed this, the [typical] young person's intoxication with youth entirely dropped away.
"Even though I was endowed with such fortune, such total refinement, the thought occurred to me: 'When an untaught, run-of-the-mill person, himself subject to illness, not beyond illness, sees another who is ill, he is horrified, humiliated, & disgusted, oblivious to himself that he too is subject to illness, not beyond illness. And if I — who am subject to illness, not beyond illness — were to be horrified, humiliated, & disgusted on seeing another person who is ill, that would not be fitting for me.' As I noticed this, the healthy person's intoxication with health entirely dropped away.
"Even though I was endowed with such fortune, such total refinement, the thought occurred to me: 'When an untaught, run-of-the-mill person, himself subject to death, not beyond death, sees another who is dead, he is horrified, humiliated, & disgusted, oblivious to himself that he too is subject to death, not beyond death. And if I — who am subject to death, not beyond death — were to be horrified, humiliated, & disgusted on seeing another person who is dead, that would not be fitting for me.' As I noticed this, the living person's intoxication with life entirely dropped away."
— AN 3.38