If your towns grew bigger, then everything looks alright?
Town growth is not linear in diameter, and not even in size, it will slow down more the larger a town grows:
* Big houses with a lot of population usually are in the city centers. Thus towns can grow by replacing smaller houses with bigger ones. Replacing old houses by new is more common for bigger towns.
* A small town adding 10 houses to the outskirts grows visibly a lot while adding 10 new houses at the rim of a large town is barely noticable. Yet both grew by the same amount.
Do you know if there is a possibility to completely delete some zones to uniform borders between some cities ?