With sonic the hedgehog related stuff, I actually built my own tools from scratch (clarification: I used the community decompression tools in a few cases). What that means is basically treating the rom like a giant block of memory then examining byte offsets, packing them into more conventional variables into custom data structures, then interpreting them through something like SDL or OpenGL.
example:
As to how one finds offsets for relevant information themselves, it's (generally) a process using emulators or some sort of hardware that can dump RAM (like a devkit). For example, what you can do is take a saved state of an emulated game in one instance, do some action which would de-increment the value you want to isolate, then take another saved state, and compare the two files to see what offsets have changed.
An easy to understand example, say you want to figure out where in memory the game stores your lives in a game. You could do that by creating a saved state, then dying, then creating another, and looking for values that decreased by one. You'll undoubtedly get many such offsets that had their values decreased by one, so you just have to pick around at the results until your figure it out. Once you figure that out, you can start looking around for things that call to that memory location. Say you want to change it so that, like, an extra life box gives you 3 lives instead of 1. You can find things that modify that memory location that stores the lives, and change it there. It's obviously a
lot of trial and error doing this, mind you.
Once you get a handle on where the game stores and pulls data from, you can then go about checking those areas and seeing if there are unused things. Like lost badnik art in Sonic games get stored in groups, so finding art for a never before seen badnik, for example, means going through a block of ROM and basically testing it out piece by piece to see if there's anything unused. That's obviously an extremely simple example, and it gets way more difficult when you talk about things like compressed files (like nearly all Sonic art is) and things that don't necessarily live in ROM (like dynamically created objects), but that's the super basic gist.
This is for example how Wood Zone was found way before the Simon Wai Sonic 2 beta popped up. We were save state hacking with KGen or some emulator from back then on the final Sonic 2 built ROM. All art assets and level layout (collision, etc) data related to Wood Zone no longer existed in the game, as they'd stripped that data out for ROM sizing reasons. But by watching RAM states taken at the end of one level, and then again at the beginning of another level, lead to understanding where the game stores palettes, which wound up being a large block of memory. By offsetting into this block of memory by the size of a single zone's palette, you could break that block of memory up into zones, like an array. Emerald Hill Zone might have been block [1], Chemical Plant Zone might have been block [2], etc. But it became apparent that the block was larger than the number of zones actually present in the game. By contrasting the palette information with what you could actually see in game, a mapping of which blocks went to which zones was created, and there were the existence of palettes in the game not used. One belonged to Hidden Palace Zone, which was easy to recognize thanks to screenshots. By changing which palette was loaded into levels, you could actually test out other zone's palette. And that's how this was found:
This palette didn't match anything in the game or any known pre-release screenshots. We called it the "Lost" zone for a while until the Simon Wai Sonic 2 Beta was found and we realized it belonged to Wood Zone, a zone nobody had ever heard of before.
