I'm surprised at this, even if the tree is suffering from extreme drought as Victoria's seem to thrive and produce large fruit even in difficult conditions. Is it a well established tree or newly planted?
The only other possibility that occurred to me is that somehow the rootstock has taken over and that what is growing is fruit from the rootstock rather than the Victoria part which has been grafted on to it. Is that a possibility?
If not, just feed and water your tree and hope for a better harvest next year.