diary-show-holidays-flag is a variable defined in `calendar.el'.
Its value is

Non-nil means include holidays in the diary display.
The holidays appear in the mode line of the diary buffer, or in the
fancy diary buffer next to the date. This slows down the diary functions
somewhat; setting it to nil makes the diary display faster.

You can customize this variable.