They're by default saved in the same folders as ROMs, under an .SGM file extension.
If you've changed directory paths in VBA's options, they will be stored in the location set for savestate reading and saving.
Batteries (.SAV) are the GBA's actual saves, so you dont want to delete these.
.SGM files are the savestates. Theyre named with a numeral appended to the normal rom's name, like for a game called pokemon, savestate 1 being labelled pokemon1.sgm