Citrus greening.

https://en.wikipedia.org/wiki/Citrus_greening_disease

Orange harvests have been getting smaller and smaller in Florida due to citrus greening disease, which has more than likely been ignored on purpose in order to allow increased real estate development. I say this because I remember reading about how Florida is getting hit really hard but other citrus growing regions are not. I might be wrong about this.

I've never seen California bananas, but also never any Florida bananas, generally they're from central and south america. I assume they are imported from there because of exploitative labor conditions.

One thing not mentioned: California has 4 grow seasons a year, instead of 1 (or 2 at most) I'm other regions.

So starting in the early 20th century they started taking advantage of this.

Over time this made it hard to compete with California growers for many things - there was a continual grow and distribution cycle. At one time the East Coast had orchards of peaches, pears, and apples everywhere.

Then add political motivations.

Because we've been lied to and the whole food supply chain is rigged for a variety of complex and intertwined political reasons that make no sense and never did but served political purposes at various times. It's all fucked up, and that's on purpose, and that's the way they like it and as a reward for looking the other way, we get all the fresh fruits and vegetables we want at any time of year and if we have to tariff or bomb a few countries to make sure that happens then by god you'd better believe we will.

Adding to the other comments, even when certain fruits can indeed be grown in places besides California, there's also the matter of infrastructure. Not specific to oranges, Pacific Fruit Express (PFE) supplied refrigerated train cars for long-haul distribution of fruits from California's Central Valley out to the East Coast. Because this was the early 20th Century before mechanical refrigeration was widely available, cooling had to be done using the same approach for centuries: ice.

In this regard, California was blessed with the Sierra Nevada mountain range, where water could be frozen into ice and then transported by rail on the now-Union Pacific transcontinental railroad to Sacramento and then down to the entire Central Valley for keeping food from spoiling during the long journey out east. The fact that these fruit-laden railcars had to go through the mountain pass again meant they could be topped up with more ice, and when the empty train car returned from the East Coast, it could carry ice back down to the Central Valley again. A virtuous cycle.

Basically, fruits not only need to be growable, but also the transportation infrastructure must exist. Sure, Florida also had railroads in the early 1900s but it was not really well connected to the rest of the Eastern Seaboard. As a side note, this is a contributing factor to the Confederacy's loss in the American Civil War, since different railroad gauges meant they had more difficulty mobilizing by rail, specifically for materiel. Whereas the industrialized North already used standard gauge everywhere for their mainline trains. So even if Florida did have standard gauge going into the 1900s, the different rail companies involved would not necessarily have had good enough relationships to easily schedule the necessary cargo trains to do a full run from Florida to the population centers up north. These are all frictions that never plagued the now-Union Pacific, which could run basically effortlessly from California to Chicago and eastward beyond. And of course, Florida is not known to have ice-making weather.

And now with an advantage of over a century, with the California fruit industry already built up, what would be the point to build up the same infrastructure but in Florida? It would be expensive and there's no need for it, not unless California is about to secede from the Union.