A good guideline is to consider whether your action would lead to or encourage wholesome or unwholesome qualities for yourself and/or others.
1. Would giving money to your brother be for your benefit, would it be for his benefit, would it be for the benefit of both of you or would it be neither for your nor for his benefit?
2. Would giving money to your brother be unwholesome for you, would it be unwholesome for him, would it be unwholesome for both you and your brother or would it be unwholesome neither for him nor for you?
Apart from that: Do you know for sure that your brother spends the money you give him on gambling?
The Buddha extolled generosity. He said "When asked, give even if you only have a little." (Dhp.224).
From my point of view: Just like not everything that is offered is acceptable under certain circumstances, not everything that is asked for is acceptable to be given under certain circumstance. If a woman would ask a monk to give her a child, that would not be something a monk would give. If a monk is offered alcohol, that would not be acceptable either. The rules for laypeople (and this particular situation) may be less clear, but generosity is a matter of the persons and the present involved imho. Providing shelter, food, medicine or clothes is something different than providing a drug to someone who is addicted and suffering from this addiction.
I just want to make sure I am doing the right thing by not giving money to a gambler. I haven't heard of generosity being discouraged, but surely giving to a gambler doesn't yield much fruit.
Do you think that some people deserve generosity, while others don't? I assume the conflict is not whether your brother is worty of generosity, but how generosity towards him should be expressed.
If you want to take an active stance, you could of course offer him help to find a job for extra cash or help him with financial matters or offer him advice on the gambling problem (or any other social / emotional support).
From my point of view: Denying an unreasonable request does not necessarily mean to be not generous and greedy, but just that generosity is applied with wisdom. Mind that there's a difference in intention between rejecting an unreasonable offer and rejecting a person.