The 57-year-old music mogul was participating in a Q&A session on 'Xtra Factor', a spin-off show of 'The X Factor' in the UK, when he made the joke to gay TV presenter Rylan Clark-Neal, who was responsible for opening doors behind which fans could ask Simon a question.
During the segment, Rylan said: "I love a front door."
To which Simon quickly replied: "I thought you liked a back door."
The remark prompted gasps from the TV audience, but Rylan shot back: "Oh Simon! Shall I show you how much?"
Then, realising his mistake, Simon issued an apology for his comment and said it's why he "should never do live TV".
Simon said: "I apologise. I shouldn't have said that. That's why I should never do live TV."
Rylan also downplayed the incident after the show, saying it was a "joke between friends".
The controversy comes shortly after it was announced Simon will continue as a judge on 'America's Got Talent' until 2019.
The Brit is one of the co-creators of the show and has proved a hit with his first stint on the panel this year, so NBC bosses have extended his contract for another three seasons.
Paul Telegdy, NBC Entertainment alternative and reality group president, said in a statement: "It was terrific to see Simon in his judge's chair this past season, bringing his unique expertise and enthusiasm to the show, night after night.
"His brilliance has made 'America's Got Talent' the industry leader that it is today and his ability to evaluate and spot talent is unmatched, so we couldn't be happier to have him seated at the judge's table for the next three years."