Another fundraiser..

This fundraiser is multiple lots we have been offered for our museum, most of these are extremely early systems, and many contain unreleased games and history that would other wise be lost.

The goal for this is split into groups, we have awhile to sort this one and lots would be paid as funded. The total includes paypal fee's added and shipping costs included excluding taxes, we put 10% as we're often charged different rates depending on donators locations.
Goal
£4,800.00
Received
£610.36
Latest donor
Kempy161 Kempy161
12%
Custom LBP Developer PS3 containing multiple unreleased games, and multiple alpha builds.
£1,300.00
An early prototype PlayStation 3 dev Kit, in an early shell design which includes a disc filled with build(s? exact contents unknown)
£1,230.00
3x Prototype XBOX's, plus a boxed system believed to be used at E3. + Misc items to be added that are unknown.
£1,230.00
Misc Nintendo systems.
£500.00
The above prices doesn't include paypal fees, please see goal amount for total with 10% added for fee's we're charged.
£0.00
Expenses
Welcome, Guest!

Here are some links you may find helpful

Dreamcast GDEMU info

sonik

Well-known member
Registered
Jul 24, 2019
51
31
18
AGName
sonik
AG Join Date
Mar 15, 2004
Interesting. Data needs to be in low density area then?
 

megavolt85

DreamShell Developer
Registered
Jun 17, 2019
214
560
93
www.dc-swat.ru
AGName
megavolt85
AG Join Date
01.09.2015
GDMENU mount low density as directory cd, high density as directory gd
list.ini find only in cd
 

megavolt85

DreamShell Developer
Registered
Jun 17, 2019
214
560
93
www.dc-swat.ru
AGName
megavolt85
AG Join Date
01.09.2015
@sonik you are reading my thoughts, I have long wanted to suggest switching to gdi for GDMENU
this will allow you to easily change themes and you can create a combined GDMENU + codebreaker image
 
  • Like
Reactions: Anthony817

sonik

Well-known member
Registered
Jul 24, 2019
51
31
18
AGName
sonik
AG Join Date
Mar 15, 2004
Cool!
I'm changing it to GDI in my GDMenuCardManager app
 
  • Like
Reactions: Anthony817

Anthony817

Well-known member
Community Contributor
Registered
Jun 2, 2019
415
563
93
AG Join Date
May 12, 2010
@sonik you are reading my thoughts, I have long wanted to suggest switching to gdi for GDMENU
this will allow you to easily change themes and you can create a combined GDMENU + codebreaker image

Wait, how would that work vs the way it is in .CDI? Just curious how making it in .GDI will allow us to have a combined image with codebreaker? How is this possible as one format over the other? Just really curious. I think you know myself and many others over the years have long wanted codebreaker support in GDMENU. That would be such a dream come true if this could happen. ;)

Edit: Ok I think I understand now after reading this post again.

GDMENU mount low density as directory cd, high density as directory gd
list.ini find only in cd

So basically the GDMENU would be in the low density part of the disc, while the Codebreaker would be in the high density part? This in effect would allow the Codebreaker to always load after we make our game selection in GDMENU thus allowing us to add any codes we wanted right?
 

megavolt85

DreamShell Developer
Registered
Jun 17, 2019
214
560
93
www.dc-swat.ru
AGName
megavolt85
AG Join Date
01.09.2015
@Anthony817 no, this is not the same as what you saw in the screenshots of the unreleased version of neirocaid
I am constantly bothered by the first item of GDMENU, why launch the menu from the menu
1) we will change the name in IP.BIN to CodeBreaker
2) I will change the GDMENU code so that it would consider the first image as a native codebreaker disk
3) I will make a GDI image with GDMENU and codebreaker

codebreaker will be in track03, GDMENU in track04 for quick replacement so that you can update the theme without rebuilding the image, and list.ini will be in track01
 

Anthony817

Well-known member
Community Contributor
Registered
Jun 2, 2019
415
563
93
AG Join Date
May 12, 2010
Oh yes I figured it would not be like how neuroacids built in menu was going to be. I know expecting that is not really so easy. But the way you explain it to be done using the actual codebreaker disc menu sounds like a much needed alternative for it. Man I just want to say, you are really on a roll already this year. If 2020 was great for Dreamcast, I can't wait to see the stuff you have planned for this year.
 

sonik

Well-known member
Registered
Jul 24, 2019
51
31
18
AGName
sonik
AG Join Date
Mar 15, 2004
Menu worked with list in track01! Yay!
When updating track01 the start address still needs to be updated in the gdi file right?
 

FamilyGuy

2049 Donator
Donator
Registered
May 31, 2019
345
337
63
AGName
-=FamilyGuy=-
AG Join Date
March 3, 2007
Menu worked with list in track01! Yay!
When updating track01 the start address still needs to be updated in the gdi file right?
As long as some zero padding is made in advance, there should be no need to move the tracks around. Just rebuild the wanted track and merge the TOC, or directly edit track03 TOC to inject the new file size.
 
  • Like
Reactions: sonik

syntax

Member
Registered
Mar 24, 2022
16
2
3
AGName
syntax
AG Join Date
25032022
eBay Username
none
Ran exclusives thru your neat little program and made some lists for you.
I found most blacklisted titles would work fine if I added some extra buffer to the end of track03.
Resident Evil 2 was my first test and from memory needed around 62k extra padding to get past the movie loop,
but I went and added 1mb padding to offending titles OR used the track03 from a game processed with gdiopt.exe(DC-Swat release).
I think gdiopt.exe handles a few edge cases safer than GDmenu, Im pretty sure it wont allow a track03 under 300k.

I put everything thru shrink, and unless some of the blacklisted games are audio issues or late game issues they are loading fine to the first menu.
I will only list titles that failed to load unless I fixed them.

Code:
Used track03.iso from gdiopt.exe or orignal track03.iso if it was tiny.

GET!! Colonies
Hello Kitty - Magical Block
Hydra Castle Labyrinth (Homebrew)
Super Runabout - San Francisco Edition
Who Wants to Beat Up a Millionaire


Added buffer 30,720 bytes long to the end of track03.iso

4X4 EVO Revival
Defense Commander (Homebrew)
Floigan Bros
Ooga Booga
Puzzle Bobble 4
Resident Evil 2 - Disc 1
Resident Evil 2 - Disc 2
Shinseiki Evangelion Typing E-Keikaku
SnoCross - Championship Racing
Taxi 2 - The Game

HL 007 GOLDENEYE        (Homebrew) 
HL BLACK OPS            (Homebrew)
HL COUNTER-STRIKE       (Homebrew)
HL GRUNT                (Homebrew)
HL GUNMAN CHRONICLES    (Homebrew)
HL OPPOSING FORCES      (Homebrew)
HL PARANOIA             (Homebrew)
HL THEY HUNGER          (Homebrew)
HL USS DARKSTAR         (Homebrew)

Data actually removed, something goes wrong here and 2k of data gets wiped from the end of track03.
SILENT SCOPE (2KB)

When adding hex some titles will fail to compress to CHD, and then I realised the emulator I was using to test (ReDream) could play some of these titles only in CHD format, so I am going back thru the list to remove the titles that shouldnt be there. Resident Evil 2 still compress' to CHD fine with the hex edit and plays to the menu.
EDIT
Im going back over the list and adding only 30,720 bytes to the end of titles that need a buffer, it seems that size works to compress to CHD and works GDI and CHD also on ReDream.
 
Last edited:

syntax

Member
Registered
Mar 24, 2022
16
2
3
AGName
syntax
AG Join Date
25032022
eBay Username
none
I learned pretty fast that simply slamming some 0s at the end of track03 breaks alot of stuff.
ODE wouldn't see the images as games and GDMENU refused to shrink iso edited this way.

So I took RE2, and replaced a 30,720 byte long block of 0s with 1s, then shrunk it, took the shrunk track03 and replaced the 1s with 0s.

https://imgur.com/Jyd0UfH

This now works fine on emulators and GDemu, saving around 240mb.
Is there an easier way about it? I noticed the GDI didn't change between my 2 builds.
How hard would it be to set GDShrink to leave more padding at the end of track03 in general?
For what its worth something like 30720 to every track03 as a safety net seems like a viable option.
https://drive.google.com/file/d/1iACpnh7eZ97d-7HQiPBU8Lyx-zBaYYaT/view?usp=sharing
 
Last edited:

FamilyGuy

2049 Donator
Donator
Registered
May 31, 2019
345
337
63
AGName
-=FamilyGuy=-
AG Join Date
March 3, 2007
I learned pretty fast that simply slamming some 0s at the end of track03 breaks alot of stuff.
ODE wouldn't see the images as games and GDMENU refused to shrink iso edited this way.

So I took RE2, and replaced a 30,720 byte long block of 0s with 1s, then shrunk it, took the shrunk track03 and replaced the 1s with 0s.

https://imgur.com/Jyd0UfH

This now works fine on emulators and GDemu, saving around 240mb.
Is there an easier way about it? I noticed the GDI didn't change between my 2 builds.
How hard would it be to set GDShrink to leave more padding at the end of track03 in general?
For what its worth something like 30720 to every track03 as a safety net seems like a viable option.
https://drive.google.com/file/d/1iACpnh7eZ97d-7HQiPBU8Lyx-zBaYYaT/view?usp=sharing
So you'd like track03 to be at least 16 sectors long? It should already be the case. Or do you want to pas track03 with 16 sectors worth of 0x00s?

It shouldn't be hard at all to do that in GDIShrink.
 
  • Like
Reactions: syntax

syntax

Member
Registered
Mar 24, 2022
16
2
3
AGName
syntax
AG Join Date
25032022
eBay Username
none
So you'd like track03 to be at least 16 sectors long? It should already be the case. Or do you want to pas track03 with 16 sectors worth of 0x00s?

It shouldn't be hard at all to do that in GDIShrink.

16 sectors worth of 0x00s at the end of every track03. It seems to help problem titles load and I'm not sure why.
Im still going thru HEX checking out some titles that fail on USBGDrom, one that jumps out and not on the black list is DRACONUS - CULT OF THE WYRM

It has 2 blocks of repeating data that get wiped completely from the end of the file. I tried adding some random data over the last block to see if it was the shrink algorithm nulling the repeating data but the result was the first block of repeating data now gets kept and the last one I made gets wiped, like anything after a large block of 0s gets wiped.
 

syntax

Member
Registered
Mar 24, 2022
16
2
3
AGName
syntax
AG Join Date
25032022
eBay Username
none
Tested DRACONUS - CULT OF THE WYRM in 4 different ways
Tosec release BOOTS
Tosec release gdiopt.exe BOOTS
Tosec release gdiopt.exe GDmenu shrink FAILS
Tosec release GDmenu shrink FAILS

Drac Tosec size of 50mb, gdiopt.exe 45mb, GDMenu 312kb

Im just not sure what to do to get this one going shrunk as I havent been make a track with atleast 16 sectors worth of 0x00s at the end of track03 to test if that method works.
I can only brute force that when there is enough trailing space for me to fill with data before GDmenu shrink.
Im going to stop using GDmenu shrink after gdiopt.exe to keep things easier to follow.

What im finding is gdiopt.exe has some way of knowing these problem titles and it wont split the game into more than 3 tracks and barely takes anything off the track03.
It still consolidates the data for this title but has large gaps of 00x0 and keeps the 2 blocks of repeating data and 476k of 00x0 after the last bit of data. much larger than the 30720 I have been using.
Im unsure if this track is actually expecting its repeating data thats being wiped or it just needs a longer trailing 00x0 section like every other title I have fixed.
I can then search and find the games with a large track03 and from there made a new blacklist.
It would be really cool for me to have a way to test what different buffers are required for different titles and if there is a sweet spot for all.
Im not that great at compiling else I would of had a crack at making a variable option for testing. I did try some edits to gdiopt.exe but couldnt compile :(
 
Last edited:

syntax

Member
Registered
Mar 24, 2022
16
2
3
AGName
syntax
AG Join Date
25032022
eBay Username
none
OK I managed to figure out a way to brute force cos I couldnt wait.
(Im only using gdiopt.exe to consolidate data and make it easier for me to have a free 30720 block to replace during tests)

Two more tests.
DRACONUS - CULT OF THE WYRM
Tosec release gdiopt.exe - Nulled the end random data to test if the game needed it. Booted but is still a large 45mb track. So that confirms there's nothing wrong with the shrink algorithm and that the game doesnt need the repeating data that GDMenu wipes.

But does it need a huge 45mb 0x00 buffer?? Or can we get away with a 30720 byte one?

DRACONUS - CULT OF THE WYRM
Tosec release gdiopt.exe - Took track 03 and replaced 30720 bytes after the last "real" block of data because I knew this shouldnt get wiped, put thru GDmenu shrink, and yep my random data was still there with the track ending exactly on it. Turned it back to 0s so I had a nice 30720 buffer after the last real block of data. BOOTS!

Im really starting to think padding every track03.iso after the last bit of data by an extra 30720 or more or whatever is a good number (yet to determine) would make compatibility skyrocket.
I only chose 30720 because it was over the minimum the most problematic game gave me (around 14000) and was divisible by 512 and 2048 so i guessed it was 2048 mode friendly.
Im have 0 clue when it comes to math and ISO standards but is 30720/2048=15 what you would call 15 sectors?

The reason this title jumped out at me I think is because it was one of a few that refused to compress to CHD, resulting in a tiny file, but after applying my fix it now compress fine.

I had a look at editing gditools.py but got stuck in a wormhole...

Code:
    # 4- We copy the relevent data from input tracks to output tracks
    # *** THIS IS WHERE THE SHRINKING REALLY HAPPENS ***
    # TRACK03
    with ISO9660(itracks) as gdi, open(otracks[2]['filename'], 'wb') as ofile:
       gdifile = gdi._gdifile
       begin_offset = otracks[2]['lba']*2048
       length = last_toc_sector*2048 - begin_offset
       gdifile.seek(begin_offset)
       _copy_buffered(gdifile, ofile, length=length)
 
Last edited:

FamilyGuy

2049 Donator
Donator
Registered
May 31, 2019
345
337
63
AGName
-=FamilyGuy=-
AG Join Date
March 3, 2007
OK I managed to figure out a way to brute force cos I couldnt wait.
(Im only using gdiopt.exe to consolidate data and make it easier for me to have a free 30720 block to replace during tests)

Two more tests.
DRACONUS - CULT OF THE WYRM
Tosec release gdiopt.exe - Nulled the end random data to test if the game needed it. Booted but is still a large 45mb track. So that confirms there's nothing wrong with the shrink algorithm and that the game doesnt need the repeating data that GDMenu wipes.

But does it need a huge 45mb 0x00 buffer?? Or can we get away with a 30720 byte one?

DRACONUS - CULT OF THE WYRM
Tosec release gdiopt.exe - Took track 03 and replaced 30720 bytes after the last "real" block of data because I knew this shouldnt get wiped, put thru GDmenu shrink, and yep my random data was still there with the track ending exactly on it. Turned it back to 0s so I had a nice 30720 buffer after the last real block of data. BOOTS!

Im really starting to think padding every track03.iso after the last bit of data by an extra 30720 or more or whatever is a good number (yet to determine) would make compatibility skyrocket.
I only chose 30720 because it was over the minimum the most problematic game gave me (around 14000) and was divisible by 512 and 2048 so i guessed it was 2048 mode friendly.
Im have 0 clue when it comes to math and ISO standards but is 30720/2048=15 what you would call 15 sectors?

The reason this title jumped out at me I think is because it was one of a few that refused to compress to CHD, resulting in a tiny file, but after applying my fix it now compress fine.

I had a look at editing gditools.py but got stuck in a wormhole...

Code:
    # 4- We copy the relevent data from input tracks to output tracks
    # *** THIS IS WHERE THE SHRINKING REALLY HAPPENS ***
    # TRACK03
    with ISO9660(itracks) as gdi, open(otracks[2]['filename'], 'wb') as ofile:
       gdifile = gdi._gdifile
       begin_offset = otracks[2]['lba']*2048
       length = last_toc_sector*2048 - begin_offset
       gdifile.seek(begin_offset)
       _copy_buffered(gdifile, ofile, length=length)
I'm a little confused about your method. What is GDmemu shrink? I don't know that. Do you mean GDIShrink from GDItools?

Afaik gdiopt is very basic 2352 to 2048 conversion + some track removing I think. It's very different than GDIShrink.

I'm the author of GDItools (and GDIShrink), so I can help you with it if you can clearly describe:

1. A bug, with a description of the actual behavior and the expected behavior

2. A feature request, with an example of expected behavior


GDItools isn't the cleanest code I've written, but I guarantee you it's not the dirtiest code you'll find floating around. You do need a fair grasp of python and object oriented programming to understand and modify it yourself though.

The code excerpt simply reads from the gdi dump as if it was a single large image, starting at the track03 offset and for a length corresponding the part we want to keep. It does the mode (2353 or 2048) conversion on the fly. It's funny that you say you're lost in a wormhole, as there's actually a wormhole file class in GDItools.


Yes, a sector of user-data is 2048 bytes. A 2352 image just contains some supplemental error detection and correction data for the optical disc format. A length of 32768 bytes is 16 sectors, that's the length of the bootsector of track03, aka the ip.bin. It's the data between the start of the track and the TOC.
 
Last edited:

syntax

Member
Registered
Mar 24, 2022
16
2
3
AGName
syntax
AG Join Date
25032022
eBay Username
none
Sorry for the confusion.
I am using GDmenu to shrink my games, it uses GDItools to do this I thought which is why I reached out to you.

I have found that if we add 16 sectors of 0x00 to the end of track03 of any title it then becomes 100% compatible with shrink.
Like it works on anything, GDemu, clones, Emulators, HDD, USBGDrom. All blacklisted titles get fixed.
It even fixed games that CHDMAN.exe refused to compress to chd. No idea why.

I cant figure out how to patch GDItools so im asking if you could for me so I can do more testing as I havent been able to try this on games with tiny 40kb track03.
Like I said I have been using GDMenu to do my shrinks with GUI, but now have python setup so am able to use yours .py scripts.

Yeah I was making fun of my lack of understanding with the wormhole class, I think I went to jump over it before it started and got stuck in it :p

Kinda reminded me of the documentation in SBItools

Code:
'Barry: What? What is this...?			
			'Jill: What is it?
			'Barry: A CUE!
			'Barry: Jill, see if you can't find any other CUES, I'll be examining this...
			'Barry: I hope this is not a BIN file!
 

syntax

Member
Registered
Mar 24, 2022
16
2
3
AGName
syntax
AG Join Date
25032022
eBay Username
none
Bug? (unsure)
Description of behaviour
Track03 of DRACONUS - CULT OF THE WYRM has 2 blocks of repeating data at the end of the file that get removed when shrinking with GDItools.
Any attempt to add random data to the leading or trailing edges of these blocks is also destroyed.
It seems anything after a certain offset gets wiped regardless of the datas structure.

Expected behaviour
GDI tools should keep these 2 blocks.
 

FamilyGuy

2049 Donator
Donator
Registered
May 31, 2019
345
337
63
AGName
-=FamilyGuy=-
AG Join Date
March 3, 2007
Bug? (unsure)
Description of behaviour
Track03 of DRACONUS - CULT OF THE WYRM has 2 blocks of repeating data at the end of the file that get removed when shrinking with GDItools.
Any attempt to add random data to the leading or trailing edges of these blocks is also destroyed.
It seems anything after a certain offset gets wiped regardless of the datas structure.

Expected behaviour
GDI tools should keep these 2 blocks.
Alright I'll try to look into this. Normally, GDItools should ensure that only 0x00s are removed.

Be sure to use the latest version found on SourceForge on the code section. Not the release zip.

E.g get the up to date code with:

Code:
git clone https://git.code.sf.net/p/dcisotools/code dcisotools-code

The version included in GDMenu is likely way out of date. I'm still confused about the actual program you're using here, you mean GDMenu builder? GDMenu itself is the GDEMU menu Dreamcast program.
Please link to the actual program your using.
 
Last edited:

Another fundraiser..

This fundraiser is multiple lots we have been offered for our museum, most of these are extremely early systems, and many contain unreleased games and history that would other wise be lost.

The goal for this is split into groups, we have awhile to sort this one and lots would be paid as funded. The total includes paypal fee's added and shipping costs included excluding taxes, we put 10% as we're often charged different rates depending on donators locations.
Goal
£4,800.00
Received
£610.36
Latest donor
Kempy161 Kempy161
12%
Custom LBP Developer PS3 containing multiple unreleased games, and multiple alpha builds.
£1,300.00
An early prototype PlayStation 3 dev Kit, in an early shell design which includes a disc filled with build(s? exact contents unknown)
£1,230.00
3x Prototype XBOX's, plus a boxed system believed to be used at E3. + Misc items to be added that are unknown.
£1,230.00
Misc Nintendo systems.
£500.00
The above prices doesn't include paypal fees, please see goal amount for total with 10% added for fee's we're charged.
£0.00
Expenses