Holy Trinity Church was built by a committee in 1840-1 to serve the south-eastern part of the town. Subscribers included George Salter and William Chance, and the site was given by George Silvester. An endowment of £1,000 was received from Thomas Hood and Edward Bullock, and in 1850 a grant of £200 was made from Queen Anne’s Bounty. The right of nominating the minister was granted by the bishop to trustees.  A parish was assigned out of Christ Church parish in 1842. The living, at first a perpetual curacy and a vicarage from 1868, has remained in the gift of trustees (now the Peache Trustees). A vicarage house was built on part of the site in 1844.

In 1872 the schoolroom in Lower Trinity Street was licensed for divine service. It was used as a mission room until c. 1937. Open-air services were held in the parish during Robert Thompson’s time as vicar (1902-16); he was helped by Ethel Silvester, granddaughter of the donor of the site, who worked in the parish as a deaconess from 1906 to 1918.

Holy Trinity Church is of brick and was designed by S. W. Dawkes in an Early English style.  It stands in a walled churchyard with streets on three sides and the vicarage house on the fourth. It consists of chancel, nave with galleries on three sides, and west tower; there is a small north porch near the east end, and the porches on either side of the tower were added c. 1872. The chancel was burnt down early in 1861 but was at once rebuilt. The position of the organ was then changed from the chancel to the west gallery; soon after Robert Thompson’s appointment as vicar in 1902 the organ was moved to the south side of the chancel, new stalls were provided, and a surpliced choir was introduced. The pews were arranged in a large central block and two smaller blocks under the galleries, apparently dated from 1884 when the existing ‘close high-backed pews’ were removed.