Mr Richardson, who is a patron of the show society, said he was concerned that if the show was forced to stop for a year or two during the rail line's construction, it may not continue.
He said the lease for the showground was up for renewal in 2014 and it was unlikely to be renewed if the show wasn't running.
"All I said was that if the show ceased then the government might think there's no point in retaining the showground," Mr Richardson said.
In that event Mr Richardson said the land could be looked at for development.
"What are you going to do with land right next to a railway?" he said.
Baulkham Hills MP David Elliott, who is also a member of the RSL sub branch and attended the meeting, said the speculation could lead to "unnecessary public anxiety".
"The main reason the Liberal government is going to the expense of building the railway underground until it reaches Kellyville is to protect the open spaces synonymous with the 'Garden Shire'," he said.
"Suggestions that we will lose our wonderful parklands are false and mischievous and I am deeply disappointed that we are forced to rebuke such claims which only serve to create anxiety and concern among the community."
Castle Hill MP Dominic Perrottet said the government did not have any plans to develop the showground into high density housing.
"I have held discussions with various stakeholders and the North West project team and I am confident the showground will be available for use by the community for years to come," Mr Perrottet said.
"The Government continues to consult with the community and anyone with any concerns or queries in relation to the North West Rail Line can visit the Community Information Centre at Castle Hill."