Yeah, I was also thinking about a CD player but then burning CDs is kinda wasteful compared to just flashing an SD-card or USB stick. Cassette sounds like a nice idea though.
To your last point: my daughter (3) already knows how to remove and re-attach the back plate of my reform (all 10 screws by herself) so they can do surprising things if you teach them early. :)