Small size means a smaller battery. If they make the phone’s processor too powerful, the battery will run out in less than a day, and then everyone will be mad about that. There’s also less surface to dissipate heat.

Making things smaller is harder and more expensive, but people who want small phones don’t want to pay more than large phones.