Who Asked The Monkeys?

If an old man steps on an apple core and falls, it's a tragedy. If the same old man steps on a banana peel and falls, it's hilarity.

How did the banana become the wacky fruit, one associated with things zainy and screwball? Other fruits that have associations are kind of negative:

Lemon = poor quality
Grapes = potentially sour
Tomatoes* = potentially rotten
Apples = not oranges
Oranges = not apples

*It's got seeds, it's a fruit