Welcome, Guest!

Here are some external resources you may find helpful

Dreamcast GDEMU info

FamilyGuy

2049 Donator
Donator
Registered
Joined
May 31, 2019
Messages
304
Reaction score
297
Points
63
AG User Name
-=FamilyGuy=-
AG Join Date
March 3, 2007
No problem! I did test one shrunk WinCE game called "Armada" that worked well, at least for the 45 minutes or so that I played it. My plan is to shrink literally everything in the US TOSEC set and try each one, but I can probably start with just the WinCE titles to see how all the rest of them work.

Would you want the "blacklist" to be based on Product ID only, or also the MD5 of the IP.BIN? I ask because changing any flags in the IP.BIN (region, VGA, etc.) will obviously result in an MD5 difference. I observed that games managed using MadSheep's GDEMU SD Card Manager seemed to have different MD5s than the originals, but I haven't done a proper diff of two of them to figure out what's changed.
Thank you for that.

I'd say that it's better to gather more info than not enough, then we can choose what we will use without wasting many hours on your side.

You're right that mods could change the ip.bin or executable MD5, but they could also change the Product ID (bootsector swap). I'm note sure what the preferred approach would be.

If you could gather, either in a properly formatted CSV or a list of python dict saved with np.savez:
  1. ip.bin informations (title, product ID, etc) and MD5
  2. ip.bin MD5 with the VGA flag forced to 1 (should be a matter of changing a single character in memory)
  3. Main executable size and MD5
  4. Any other identifying information you can think of that'd survive after mods.
  5. Working or not after shrinking
That'd be greatly helpful to me to implement a blacklist, with many fallback to identify the games.

My plan is to only do the bin2iso part of the shrinking if a game is in the blacklist. I'll allow an override option of course.
 

Xerxes3rd

Donator
Donator
Registered
Joined
Jun 2, 2019
Donations
£7.36
Messages
12
Reaction score
5
Points
3
AG User Name
Xerxes3rd
Thank you for that.

I'd say that it's better to gather more info than not enough, then we can choose what we will use without wasting many hours on your side.

You're right that mods could change the ip.bin or executable MD5, but they could also change the Product ID (bootsector swap). I'm note sure what the preferred approach would be.

If you could gather, either in a properly formatted CSV or a list of python dict saved with np.savez:
  1. ip.bin informations (title, product ID, etc) and MD5
  2. ip.bin MD5 with the VGA flag forced to 1 (should be a matter of changing a single character in memory)
  3. Main executable size and MD5
  4. Any other identifying information you can think of that'd survive after mods.
  5. Working or not after shrinking
That'd be greatly helpful to me to implement a blacklist, with many fallback to identify the games.

My plan is to only do the bin2iso part of the shrinking if a game is in the blacklist. I'll allow an override option of course.
I'm working on this now. I created my own version of parse_gdi() that can read in-memory zip files so I can extract the ip.bin data directly from them, which seems to parse fine. My question is regarding the access of the ip.bin data: it looks like the ip.bin is stored a the beginning of the high-density area (track 3, LBA 45000). When I open a track03.bin file in a hex editor, it looks like the ip.bin data starts at offset 0x0010, instead of right at the beginning of the file. In order to extract the ip.bin data, should I always just read 32K of data from the track at 45000, and skip the first 16 bytes? I'm trying to avoid extracting all these zip files just to get at the stored ip.bin data. =)
 

megavolt85

DreamShell Developer
Registered
Joined
Jun 17, 2019
Messages
198
Reaction score
545
Points
93
Location
Russia
Website
www.dc-swat.ru
AG User Name
megavolt85
AG Join Date
01.09.2015
IP.BIN is first 16 sectors of track03

bin tracks have sector size 2352 bytes

1) open image
2) sector counter = 0
3) skip 16 bytes
4) read 2048 bytes
5) skip 288 bytes
6) sector counter = sector counter + 1
7) if sector counter < 16, goto step 3
8) end
 

FamilyGuy

2049 Donator
Donator
Registered
Joined
May 31, 2019
Messages
304
Reaction score
297
Points
63
AG User Name
-=FamilyGuy=-
AG Join Date
March 3, 2007
I'm working on this now. I created my own version of parse_gdi() that can read in-memory zip files so I can extract the ip.bin data directly from them, which seems to parse fine. My question is regarding the access of the ip.bin data: it looks like the ip.bin is stored a the beginning of the high-density area (track 3, LBA 45000). When I open a track03.bin file in a hex editor, it looks like the ip.bin data starts at offset 0x0010, instead of right at the beginning of the file. In order to extract the ip.bin data, should I always just read 32K of data from the track at 45000, and skip the first 16 bytes? I'm trying to avoid extracting all these zip files just to get at the stored ip.bin data. =)

IP.BIN is first 16 sectors of track03

bin tracks have sector size 2352 bytes

1) open image
2) sector counter = 0
3) skip 16 bytes
4) read 2048 bytes
5) skip 288 bytes
6) sector counter = sector counter + 1
7) if sector counter < 16, goto step 3
8) end
@megavolt85 is right, as always, but GDITools should take care of that for you.

If you're only extracting track03, you should be able to use the CDImage class to read a bin file just as if it was a iso file. See the bin2iso implementation that comes with GDITools for the details.
 

Xerxes3rd

Donator
Donator
Registered
Joined
Jun 2, 2019
Donations
£7.36
Messages
12
Reaction score
5
Points
3
AG User Name
Xerxes3rd
@megavolt85 is right, as always, but GDITools should take care of that for you.

If you're only extracting track03, you should be able to use the CDImage class to read a bin file just as if it was a iso file. See the bin2iso implementation that comes with GDITools for the details.
Thanks- I was trying to avoid doing a whole bunch of subclassing just to read the ip.bin data out of an in-memory track03, but I can try only subclassing CDImage and replacing the seek() calls with byte array math.

Worst case I'll use @megavolt85 's solution which should be easy to implement; I'll be sure to check the sector sizes in the .gdi file for consistency.
 

FamilyGuy

2049 Donator
Donator
Registered
Joined
May 31, 2019
Messages
304
Reaction score
297
Points
63
AG User Name
-=FamilyGuy=-
AG Join Date
March 3, 2007
Thanks- I was trying to avoid doing a whole bunch of subclassing just to read the ip.bin data out of an in-memory track03, but I can try only subclassing CDImage and replacing the seek() calls with byte array math.

Worst case I'll use @megavolt85 's solution which should be easy to implement; I'll be sure to check the sector sizes in the .gdi file for consistency.
Oh yeah, then just read 32/2048*2352 bytes in memory from lba 45000 and directly apply what megavolt said to that buffer.
 
Last edited:

megavolt85

DreamShell Developer
Registered
Joined
Jun 17, 2019
Messages
198
Reaction score
545
Points
93
Location
Russia
Website
www.dc-swat.ru
AG User Name
megavolt85
AG Join Date
01.09.2015
WINCE games have three types of protection, but they all check for the existence of the WARNING.DA or WARN_EN.DA file
1) the game just checks for the existence of the WARNING.DA file
2) the game checks the LBA of the WARNING.DA file, the LBA is stored in 0WINCEOS.BIN in the LBA + 150 format
3) the game just checks for the existence of the WARNING.DA file, but instead of DIR RECORD information about the disk structure is stored in a special file on the disk
 

FamilyGuy

2049 Donator
Donator
Registered
Joined
May 31, 2019
Messages
304
Reaction score
297
Points
63
AG User Name
-=FamilyGuy=-
AG Join Date
March 3, 2007
WINCE games have three types of protection, but they all check for the existence of the WARNING.DA or WARN_EN.DA file
1) the game just checks for the existence of the WARNING.DA file
2) the game checks the LBA of the WARNING.DA file, the LBA is stored in 0WINCEOS.BIN in the LBA + 150 format
3) the game just checks for the existence of the WARNING.DA file, but instead of DIR RECORD information about the disk structure is stored in a special file on the disk
So what would I need to automate the workaround?

Create a dummy record with proper LBA? Read that special file?

Any help is appreciated, it'd be nice to have a fully working method.

Also, Merry Christmas!
 

Xerxes3rd

Donator
Donator
Registered
Joined
Jun 2, 2019
Donations
£7.36
Messages
12
Reaction score
5
Points
3
AG User Name
Xerxes3rd
By the way, when I noted "gdishrink fails on Q-bert" I meant that the python script itself exits with an error when attempting to run gdishrink on that title.

Thank you both for the help, Merry Christmas!
 

megavolt85

DreamShell Developer
Registered
Joined
Jun 17, 2019
Messages
198
Reaction score
545
Points
93
Location
Russia
Website
www.dc-swat.ru
AG User Name
megavolt85
AG Join Date
01.09.2015
So what would I need to automate the workaround?
1) trim track03
2) add a dummy file to the end of track03
3) change to dir record LBA dummy file
4) find the old LBA + 150 value in the last data track and replace it with a new one
 

Xerxes3rd

Donator
Donator
Registered
Joined
Jun 2, 2019
Donations
£7.36
Messages
12
Reaction score
5
Points
3
AG User Name
Xerxes3rd
If you could gather, either in a properly formatted CSV or a list of python dict saved with np.savez:
  1. ip.bin informations (title, product ID, etc) and MD5
  2. ip.bin MD5 with the VGA flag forced to 1 (should be a matter of changing a single character in memory)
  3. Main executable size and MD5
  4. Any other identifying information you can think of that'd survive after mods.
  5. Working or not after shrinking
I updated the previous spreadsheet (link):
For WinCE titles, it looks like the main executable is often (if not always) 0WINCEOS.BIN, and likely always the same binary. I only checked two titles, but the 0WINCEOS.BIN file was identical on both (I checked '4x4 EVO' and 'Atari Anniversary Edition'). I'm not sure what else would generate a unique signature that would survive shrinking, unless there's a specific additional file that exists on every disc that's guaranteed to be unique.

Other than that complication, I ran GDIShrink on every title (excluding the titles previously listed) and tested each of the 298 US TOSEC titles. For each title, I started the game and played it for at least a minute or two.

My testing resulted in a number of additional titles that hang/reboot the system after shrinking, and all but 2 of them are WinCE games. Additionally, I found 11 titles that would exhibit an annoying 'audio buzzing' after shrinking. The audio buzz generally occurs only in-game, and in some titles, only during specific events, which suggests there's an audio file or two getting corrupted. The unmodified GDI for each of these works fine.
 
Last edited:

Xerxes3rd

Donator
Donator
Registered
Joined
Jun 2, 2019
Donations
£7.36
Messages
12
Reaction score
5
Points
3
AG User Name
Xerxes3rd
After digging around some more and comparing the original and shrunk versions that megavolt85 provided, my new plan for generating a "modified" md5sum of the ip.bin would be to take the original and zero out the following fields:

  • VGA flag (offset 0x3D, 1 byte)
  • Region flags (offset 0x30, technically 8 bytes but only 3 are ever used; perhaps set these all to 0x20 instead of 0x00)
  • TOC Sector offsets/addresses (0x100 through 0x2FF should cover all the TOC stuff; could leave 0x100-0x107 and 0x288-0x28F intact, since they shouldn't change?)
Those seem to be the only likely things to change. Another option is to calculate the TOC data for a shrink operation, then use the result for the "modified" md5sum; however, @FamilyGuy I also tried running GDIShink multiple times on a GDI, and sometimes it seems to be cutting 2K off the total file size each time. I assume it shouldn't be doing anything to an already-shrunk GDI?

Also, since a number of non-WInCE titles seem to have audio buzzing issues after being shrunk, perhaps the tool should ignore .RAW files when shrinking.
 
Last edited:

FamilyGuy

2049 Donator
Donator
Registered
Joined
May 31, 2019
Messages
304
Reaction score
297
Points
63
AG User Name
-=FamilyGuy=-
AG Join Date
March 3, 2007
After digging around some more and comparing the original and shrunk versions that megavolt85 provided, my new plan for generating a "modified" md5sum of the ip.bin would be to take the original and zero out the following fields:

  • VGA flag (offset 0x3D, 1 byte)
  • Region flags (offset 0x30, technically 8 bytes but only 3 are ever used; perhaps set these all to 0x20 instead of 0x00)
  • TOC Sector offsets/addresses (0x100 through 0x2FF should cover all the TOC stuff; could leave 0x100-0x107 and 0x288-0x28F intact, since they shouldn't change?)
Those seem to be the only likely things to change. Another option is to calculate the TOC data for a shrink operation, then use the result for the "modified" md5sum; however, @FamilyGuy I also tried running GDIShink multiple times on a GDI, and sometimes it seems to be cutting 2K off the total file size each time. I assume it shouldn't be doing anything to an already-shrunk GDI?

Also, since a number of non-WInCE titles seem to have audio buzzing issues after being shrunk, perhaps the tool should ignore .RAW files when shrinking.

Blanking the entries that can change is an elegant solution. I like it.

I'm not sure why it would cut a sector out of already shrunk dump though. I'd have to look into it. It's most likely a bug.

The tool does ignore raw files.
As far as audio issues, thiat common when a game interprets data as audio. There might be some funky business happening.

Sadly I'm very busy these days with life and everything (finishing school, starting a job, I guess I'm an adult now; 2020 was crazy all around), so don't hold your breath.
 

Xerxes3rd

Donator
Donator
Registered
Joined
Jun 2, 2019
Donations
£7.36
Messages
12
Reaction score
5
Points
3
AG User Name
Xerxes3rd
Blanking the entries that can change is an elegant solution. I like it.

I'm not sure why it would cut a sector out of already shrunk dump though. I'd have to look into it. It's most likely a bug.

The tool does ignore raw files.
As far as audio issues, thiat common when a game interprets data as audio. There might be some funky business happening.

Sadly I'm very busy these days with life and everything (finishing school, starting a job, I guess I'm an adult now; 2020 was crazy all around), so don't hold your breath.
Thanks- I modified my scripts to operate this way and it seems to work well- I'm able to uniquely identify each title & version in the US TOSEC set.

The "repeated shrinking cuts out a sector" bug doesn't happen to every dump, so I'll work on finding a couple repeatable examples of when it happens.

I would have thought the tool would ignore raw files as well, but some of them are sized differently after shrinking. Does the tool try to convert the sector size from 2352 to 2048 for raw files?

Absolutely no worries! I'm posting updates here more for documenting my findings. As I understand the GDI format better and get more comfortable in python, I might be able to take a crack at resolving the issues I find.
 

FamilyGuy

2049 Donator
Donator
Registered
Joined
May 31, 2019
Messages
304
Reaction score
297
Points
63
AG User Name
-=FamilyGuy=-
AG Join Date
March 3, 2007
I would have thought the tool would ignore raw files as well, but some of them are sized differently after shrinking. Does the tool try to convert the sector size from 2352 to 2048 for raw files?
It's been a while, but I'm fairly certain that they should simply be copy/pasted. If there's any conversion happening, it's a bug.

If you end up fixing stuff or adding features, please submit a pull request on SourceForge so that I can merge it with the official code. Contributions are welcome!
 

Xerxes3rd

Donator
Donator
Registered
Joined
Jun 2, 2019
Donations
£7.36
Messages
12
Reaction score
5
Points
3
AG User Name
Xerxes3rd
Will do! A couple follow-up questions:
  1. It looks like the low-density tracks 1 & 2 are replaced with dummy tracks. Is this just to save space, and if so, is it really saving much?
  2. Would you be opposed to me modifying the scripts to support Python 3? Ideally the scripts would work with both Python 2 & 3.
 

FamilyGuy

2049 Donator
Donator
Registered
Joined
May 31, 2019
Messages
304
Reaction score
297
Points
63
AG User Name
-=FamilyGuy=-
AG Join Date
March 3, 2007
Will do! A couple follow-up questions:
  1. It looks like the low-density tracks 1 & 2 are replaced with dummy tracks. Is this just to save space, and if so, is it really saving much?
  2. Would you be opposed to me modifying the scripts to support Python 3? Ideally the scripts would work with both Python 2 & 3.
1. Yes it was to save space, and it depends on the game. I was planning on changing it so that it's reversible though.
2. Not at all, as long as the features are the same. Again though, I'd prefer pull requests so that I merge the code in the main repository, instead of having many forks which can be confusing for users.
 
Last edited:

sonik

Registered
Registered
Joined
Jul 24, 2019
Messages
39
Reaction score
21
Points
8
AG User Name
sonik
AG Join Date
Mar 15, 2004
There's a version of gdmenu in gdi format on Deunan's site.
It works. If I rebuild the GDI it still works. But if I rebuild it with adding the list.ini file it seems to ignore it when booting. Why?
@FamilyGuy or @megavolt85 might know?
 
Top