RESIDENTS of the small Victorian hamlet of Flowerdale are returning to their devastated community for the first time since the bushfires razed their homes. 

Four days after the blaze wiped out the town, killing at least four people, residents have been allowed back under police escort.

Police led a convoy of anxious refugees from the Yea recovery centre through road blocks to Flowerdale from 10am (AEDT).

Many will return to find nothing remains of their once idyllic rural township.

Sergeant Trevor Connell, of Yea Police, broke the news to residents holed up at the Yea Recreation Centre around breakfast time.

He said that while many people would be unprepared for the devastation that would confront them, it was time to let them see for themselves.

"Since Saturday, they haven't been allowed back to where their home is. There is total devastation and we're still looking for people that are unaccounted for," Sgt Connell said.

"They've delayed letting them back in there until it's safe and they're obviously frustrated. At some stage they've got to go back and look and see what the devastation is and I think it's now time to do that and let them start moving on."

Sgt Connell said there was still a lot of debris on the road and people unaccounted for.

Cars would be allowed back in, two at a time, in a controlled convoy to avoid chaotic scenes, he said.

Police are urging people whose homes have been lost to return to the recovery centre.

"I can't tell them to come back, I can only encourage them to come back because in there at the moment there's no water, no electricity and no phone, no food," Sgt Connell said.

"So, in reality, they're better off back here where the services are."

AAP