Founded in 1964, Southwold Rugby Club initially fielded one team. Today the club has over 300 members and comprises a 1st XV and 2nd XV, a ladies team (The Swallows), U13 to U17 Youth teams (Sharks) as well as U7s to U12 Minis teams (Seagulls).

In the early days of the club facilities were somewhat rudimentary. Southwold's Harbour Inn served as a changing facility, club house and match tea venue. A clubhouse was built in the late 60s and was repeatedly extended. In 2008 the new William Godell Pavilion was opened and is now the proud home of Southwold RFC