Is my goldfish a white supremesist?

I bought a whitish goldfish. And I also have black and white pebbles in the tank he swims in. and for some reason he tends to stay away from the black pebbles. Should I be concerned about my fishe's bigotry?