Rav Menashe Klein discusses (Mishneh Halachos 6:270) if it is permitted to go to the race track and bet on horses or even to watch horse racing just for fun. The Mekor for this Halacha is a Mishna in Sanhedrin (3:3) that says Mafrichei Yonim, pigeon flyers, are Pasul for Eidus. The gemara says that according to one explanation this means pigeon racing for money and it applies to any other animal racing including horses. The Rambam (Eidus 10) and Shulchan Aruch (CM 34:16) say this only applies to people who are not gainfully employed and do this for a living. However says Rav Klein that while the threshold to be pasul for eidus is that it must be your means of earning a livelihood, nevertheless it is assur to bet on these races even if it is just a hobby.This all implies that if you are merely going to be entertained by the spectacle and are not involved in any betting, then it would be permitted. However says Rav Klein, there is a Mahari Bruna that speaks about this. He says that watching horses race is not assur because it is educational and can teach you how to ride horses faster for a living and to run from your enemies. The Mahari Bruna then says that if your intention in going is simply to enjoy a jousting match then it might be assur. In our case says Rav Klein since there is no redeeming value whatsoever it is certainly assur.
This issur he says, would extend to all spectator sports and would fall under he issur in the gemara Avodah Zara (18b) of going to stadiums since it is considered Moshav Leitzim, which Rashi says is Bitul Torah. Moreover, he says, even if you were to watch it on television from the comfort of your own home, which would eliminate the issur of Moshav Leitzim, still you will be left with Bitul Torah and anyone with Yiras Shamayim should stay far away.