People care because they have an association with the college. Usually an alumnus themselves or a close family member. Big universities like Ohio State or Penn State graduate between 15K - 20K students each year.
Minor league teams are generally in smaller towns and areas with a smaller populations and smaller arenas and stadiums. If the Toledo Mudhens (MLB minor league) could pull 35K fans to a game they would, but it's a small town and can't. Why do you think New York City and LA have multiple NFL, NBA, NHL, and MLB teams? Because they have the large populations to support it.
Colleges grow there fans bases every year by admitting new students. Some new students may not be interested in sports, some may only be interested in sports while in college, but some will most certainly be proud fans for life. The Erie Otters (OHL minor league hockey) would love to get even 1K new potential fans every year, but that doesn't happen in minor league sports.
There are some well run minor league organizations throughout sports, but there are also many instances where teams fold and go bankrupt.