Huddersfield were thought to be eyeing a swoop for Brown following his successful loan campaign at the John Smith’s Stadium last season.
Brown scored five goals for the Terriers to help them to Premier League promotion in May, having only joined them on a six-month loan deal in January.
But Town are not on the verge of bringing him back to Leeds Road.
The Huddersfield Examiner report that no contact has been made with Chelsea over a proposed permanent transfer.
Huddersfield Town are have made just one big-money signing so far this summer, with Aaron Mooy coming in on a £10m deal from Manchester City.
Brown is supposedly valued at £8million and, whether or not the player does eventually return to the club, Huddersfield are likely to break their transfer record again at some point this summer.
Brown, 20, joined Chelsea from West Brom in 2013 but has made just one league appearance for the Blues.
He previously spent a season-long loan spell at Vitesse Arnheim before playing for Rotherham and then Huddersfield.
The Examiner add that any deal between Huddersfield and Chelsea for the playmaker would likely be another loan agreement.
Discussing his Chelsea future after signing for Town, Brown said in February: “Being 16 and supporting Chelsea as a kid, when a team like that tells you they want you it’s hard for any teenager to say no.
“It’s all about timing, I’m still 20 and not too worried.”