An austere man - Hard, severe, oppressive. The word is commonly applied to unripe fruit, and means "sour," unpleasant; harsh. In this case it means that the man was taking every advantage, and, while "he" lived in idleness, was making his living out of the toils of others.
Thou takest up - Thou dost exact of others what thou didst not give. The phrase is applied to a man who "finds" what has been lost by another, and keeps it himself, and refuses to return it to the owner. All this is designed to show the sinner's view of God. He regards him as unjust, demanding more than man has "power" to render, and more, therefore, than God has a "right" to demand. See the notes at Matthew 25:24.

For I feared thee,.... Not with a right fear, with a fear of his goodness, who had bestowed such an excellent gift on him; for this would have taught him to have departed from evil, and have put him on doing his master's will, and making use of his gift to his glory: his fear was not of the right kind, and was ill grounded, as appears by what follows:
because thou art an austere man; cruel and uncompassionate to his servants, and hard to be pleased; than which nothing is more false, since it is evident, that Christ is compassionate both to the bodies and souls of men; is a merciful high priest, and is one that has compassion on the ignorant, and them that are out of the way, and cannot but be touched with the feeling of his people's infirmities; and is mild and gentle in his whole deportment, and in all his administrations:
thou takest up that thou layest not down, and reapest that thou didst not sow; suggesting, that he was covetous of that which did not belong to him, and withheld what was due to his servants, and rigorously exacted service that could not be performed; a most iniquitous charge, since none so liberal as he, giving gifts, grace and glory, freely; imposing no grievous commands on men; his yoke being easy, and his burden light; never sending a man to a warfare at his own charge; but always giving grace and strength proportionable to the service he calls to, and rewarding his servants in a most bountiful manner, infinitely beyond their deserts.