I think you will find the following are the real reasons why fans or want to be fans don't attend.
7pm Sunday games & to some extent Saturday night games, when 3pm Kick offs might be fun and leave the evenings free. Very child friendly time too.
Very expensive car parking at the stadium
Very expensive food and beverage at the stadium
Team results from the past.
Gate ticket and Online ticket prices.
I find it hard to believe low attendences have anything to do with racism or perceived racism. It's not like the place is full of skin heads and white power surpremisists. Yellow fever imho means yellow being one of the two major Wellington colours and fever as a play on the disease - football being infectuous. I strongly believe the founders of the YF did not even think that it was or would be a racist term. That suggestion would be just PC Bullshark dreamt up by someone.