Too little room on the earth to feed all humans?

Recently I read a statement from an organisation for vegetarians in which it said that there isn't enough ground on the earth to grow all the crops necessary to feed all the animals necessary to feed humans.
This actually makes no sense at all.
There are large amounts of land in this world, about two thirds, on which it will not be able to grow any crops to feed humans. The animals that we mainly eat are animals that eat grass.
Cows, sheep and goats f.i. should not be mainly fed grains, that is bad for their health, they need a large amount of grass, weeds, etc., this goes for most of the animals that we feed ourselves with.
Therefore it is total rubbish to say that when the western world would eat less meat that this would give the third world countries more food and the people there would not starve to death. Basically the western world should feed the animals what they need to feed and then there wouldn't be such problems. I personally think it would be pretty sick to not use about two thirds of the land and overuse one third which would then end up being infertile.
We would also need to eat higher quantities of food to get the required proteins and fats if we were vegetarians and no longer getting concentrated protein and fat from animal products, so we would need to greatly increase vegetable and grain production and wearing this land out. As a result I can see starvation for all.
Of course vegetable growing can be more productive using organic techniques instead of factory farming, which is true, but that this would require a lot more human labor, which I would love to see happening. The majority of us would have to go back to hard labour in the fields instead of just a few now. This would of course benefit the health of most of us, but its unlikely to happen in practice as most people do not want to do hard labour. I personally love having a vegetable garden for our family, saves money and feeds all 6 of us easily throughout the year.