It was once thought that about 1216 this estate belonged to a family named de Kelsey, but our more recent historians say that there is no proof that John and William de Kelsey of that time had anything to do with this Kelsey. They say that the beginning of Kelsey lies in 1408 when Maurice Ie Brun, Lord of the Manor of Beckenham, granted to William Kelshulle a Lease of two meadows, and there Kelshulle built a mansion, near to the lower waterfall.
The Brograves, in the person first of William Brograve, took over from Kelshulle and in 1479 were granted a Licence from the Bishop of Rochester, to erect a Chapel, or Oratory, at "Kelsies". The Brograve family remained in possession until 1688 when the property passed to the first Peter Burrell.
The first mansion fell into dis-use towards the end of the 18th century and was pulled down some time between 1790 and 1820, for there is no reference to a building there in the Gwydir Sale of 1820. A new mansion was built by the upper lake by Richard Bennett, who had married one of the Burrell daughters in 1766. The ownership remained with the Burrells right through to 1820 when the estate was bought by Edward Gross Smith, and in 1835 he sold to Peter Richard Hoare, the elder; he died on 10th September 1849 at Kelsey and there is a memorial to him in the Parish Church. The property then went to his son, Peter Richard Hoare, the younger; he was made a partner in the Hoare's Bank in 1842, and inherited the Luscombe Estate in Devon on the death of his uncle Charles in 1851, and thereafter his interest in his Beckenham estate diminished.
Charles A.R. Hoare, second son of the younger P.R. Hoare, then came into possession of Kelsey but he only resided there until about 1895 because of his interest in the training ship "Mercury" down at the Hamble river. The good work he did as Commander, in running this training ship, is recalled in chapter 9 of C.B. Fry's "Life worth living". He was the last of the Hoare family to reside at Kelsey and died in 1908.
When the Hoares took possession of Kelsey they converted the house to a Scottish baronial-style mansion by re-facing and adding to the old Bennett House, and added a chapel, dedicated to St. Agatha in 1869. In later years choir boys for this chapel came from Sandhills School, in Kelsey Lane. Sandhills was demolished in 1956 when houses were erected in Kelsey Lane.
The approach to the mansion was through the main lodge in Kelsey Square, off the centre of the High Street, past a gravel pit (shown on maps in 1623), along the present site of Greenways, Manor Way not being developed until about 1911. The only surviving building of either of the two mansions is the cottage with Gothic windows, at the lower end of Manor Way. This is over 200 years old and was once the estate bailiff's office.
From 1895 the mansion was used as a Convent by the Sisters of All Saints, Margaret Street, until 1901 when it was used by the Kepplestone School for the Daughters of Gentlemen for some years, but according to records had gone by 1909; Kepplestone School had then moved to Overbury Avenue. The estate was then bought and developed by Hay Walker, a consulting engineer, who was the chief engineer for the London Tube Railways.
In 1911 the Local Council, through the working of the Mr. T.W. Thornton, purchased Kelsey Park which was opened to the public in 1913. During the first world war the house was used for military purposes and was demolished in 1921. In 1933 the Council purchased part of the grounds of Cedar Lawn in Wickham Road and made an extension to the Park with a new entrance opposite Tudor Road that opened in 1936.