Jones was born in 1683 or 1684 at Penboyr, Carmarthenshire, and christened on 1 May 1684. His father was John Ap Gruffydd, "a godly father", and his mother Elinor John. Later in life, he married Margaret, who was described as a charitable and pious woman.
After village school, Jones became a shepherd, but then entered Carmarthen Grammar School, with the aim of becoming a clergyman. About 1707 he applied for ordination, and according to John Evans of Eglwys Cymyn (1702–1782), was rejected more than once, but owing to the influence of Evan Evans, vicar of Clydeu, Pembrokeshire, he was at last ordained as a priest by Bishop George Bull in 1708.
Jones was appointed in 1716 rector of Llanddowror, where he remained for the rest of his working life. He was an enthusiastic member of the Society for the Promotion of Christian Knowledge. In 1731, he started what were known as circulating schools, in order to teach people to read. These were held in one location for about three months before moving ("circulating") to another place in Carmarthenshire. The language of instruction in these schools was the language of the people, Welsh. The idea met with enthusiasm, and by his death in 1761, it is estimated that over 200,000 people had learnt to read in schools organised by Jones throughout the country.
Jones taught people to read that they might know those things that were necessary for their salvation – the curriculum at his schools consisted only in the study of the Bible and the catechism of the Church of England. But in doing so he joined in creating a country with a literate population and a deeper knowledge of the Christian scriptures. This may have played a key role in making the people of Wales ready to accept Methodism. He is often thought of as the forerunner of Methodist ideas in Wales. He was a powerful preacher and he would preach in the open air, as later the Methodists would do, and bring upon himself the censure of bishops for addressing irregular meetings. He lent his critical support to the Methodist revival, and was associated with the early leaders.
On 8 April 1761, aged 77, he died at the home of Bridget Bevan in Laugharne, where he had resided since his wife's death. Bridget Bevan continued to manage and support the schools until she died in 1779. Both were buried in Llanddowror Church.