IRS Publication 525 directly addresses what it calls holiday gifts. "If your employer gives you a turkey, ham, or other item of nominal value at Christmas or other holidays, do not include the value of the gift in your income. However, if your employer gives you cash, a gift certificate, or a similar item that you easily can exchange for cash, you include the value of that gift as extra salary or wages regardless of the amount involved."
At Christmas time, generous lay people often seek opportunities to give to those who have ministered to them during the year. Great idea! Gifts between individuals are neither taxable to the recipient, nor deductible to the donor.
The challenge comes when the entire congregation as the minister's employer (IRS Topic 417 and Publication 517) decides to take a collection and give him a Christmas bonus. This is viewed as an action by the minister's employer to compensate its employee--it's taxable. Of course, church revenues are almost entirely from the tax-deductible, charitable contributions of its members and guests. To rephrase my last sentence from the previous paragraph: Gifts between employers and employees are taxable to the recipient and deductible to the payer. Of course, a tax-exempt church "payer" is not concerned about receiving a tax deduction, but its contributors are.
I suppose that a church's lay leaders could remind its members: "Now we're getting close to Christmas. Don't forget to add our minister to your Christmas list. You can catch him in his office or here's his home address." But as soon as the congregation acts in concert as his employer ... well, refer to the previous paragraph.