Nah, that logic sucks, because if you follow that logic it doesn't have a bivalve creature living in a shell either, so it's not a scallop.
But also a scallop involving potatoes already exists, so it's overloading the term, ergo cake is less-bad (there are kids of cakes that don't involve icing). If you want to argue there's a better term, the South Aussies *technically* have a more sound claim.
TL;DR: VIC might be wrong, but NSW isn't right.