Let's see if I can possibly answer some of these questions...
On the one hand, Sega clearly wanted to move away from certain franchises established during the 16-bit era (Streets Of Rage, anyone?) by creating several CS Teams with the purpose of creating new IP, while others were updated to have more of a "next generation" look, including Shinobi-X, Hang On GP and Golden Axe: The Duel. Even the company's mascot was stuck waiting to make the leap from Mega Drive to Saturn, and when he did finally return, what did we get? A more polished version of Sonic 3D, the admittedly impressive Sonic Jam compilation and the technically remarkable Sonic R, but none of these were really a worthy follow-up to Sonic 3 & Knuckles - of course, Sonic Adventure
was in development and would surface early into the Dreamcast's life, though it should be noted that Sonic Xtreme nearly became a thing long before even that. Similarly, I'm aware that we very nearly got a 3D Streets Of Rage in the form of what became Fighting Force, but the general point still stands: Sega desperately hoped its fans would embrace the likes of Panzer Dragoon, Clockwork Knight, Victory Goal and even BUG! over what it felt were outdated ideas, which most would have preferred. Why couldn't they have given us both, as might have been the case with Revenge Of Death Adder and Golden Axe: The Duel?
My understanding is that a lot of the partial or complete source code lost (for games such as Virtua Fighter 2 and Panzer Dragoon Saga) occurred during Sega's corporate restructuring in the early 2000s, though it's also believed that Tantalus might have lost a hard drive of original data for The House Of The Dead that had been provided by AM1 to help in the process of converting this game to the Saturn. With most of the affected titles being the kind it's possible to continue releasing by simply porting other versions, Sega probably doesn't have much great need or financial incentive to hunt down any assets that may still exist. If they ever did want an archivist to go through their vaults, I'd happily donate my services in return for a comfortable wage and a place to live in Japan...
I've already answered this to a degree, but it should also be noted that a lot of the 16-bit games that did receive updates or sequels on Saturn were those best suited to making the jump from two to three dimensions, such as Hang On and After Burner, the latter receiving a spiritual successor of sorts in Sky Target. Come to think of it, why did that pre-rendered Ecco footage surface (heh) early in the Saturn's life if Sega had no intention of continuing this series? Also, wouldn't it have made a lot more sense to have a new Eternal Champions exist for those who didn't think Virtua Fighter was all that? How come the powers that be were only prepared to push one flagship fighting series rather than having two? Just a few years later, they'd have Virtua Fighter 3, Fighting Vipers, Sonic The Fighters, Last Bronx and even Fighters Megamix all competing for the attention of gamers, with all but one of these coming from AM2. Was there really no space for a new Eternal Champions, not to mention Golden Axe, Shinobi, Streets Of Rage or Ecco? Also, instead of the Sega Ages campaign, what if proper new sequels to After Burner, Space Harrier, OutRun, Power Drift and Galaxy Force had been developed for the Saturn instead? Even though I brought it up just a few sentences ago, Sky Target doesn't really count, and while I'm a massive fan of the series, couldn't there have been a next generation Space Harrier to accompany the first two Panzer Dragoon games?
"Why didn't I launch with a solid library of truly arcade-perfect Model 3 games? Why am I always looking over my shoulder for the PS2?"
I see what you did there! For a while later in their careers, it does seem as if the remaining guys in Genesis had something of a self-deprecating sense of humour, and although a cross marketing campaign with Sega might have worked in the US, they're a British group, and the joke would have been lost on us. Between the "Six Of The Best" reunion concert in 1982 and a 1999 remake of The Carpet Crawlers (which turned out to be the group's last new release overall), did Peter Gabriel even still want to be associated with Genesis in the 1990s? Whatever the case, I
do agree that it's interesting to imagine the possibilities if only the Sega-Genesis connection had been taken to its somewhat logical conclusion.
Between his anti-2D stance, an issue with RPGs (especially those by Working Designs in particular), introducing the five star policy then just as quickly declaring that the Saturn wasn't Sega's future at a time when rumours were already in full force about its next generation plans, it looks as if Bernie Stolar was hellbent on poisoning the chalice he took over from Tom Kalinske. As for whether Bernie was a mole from Sony, I doubt it - just look at how much the PlayStation's software library blossomed once he left that company. For one, would he have allowed the likes of Final Fantasy VII and Metal Gear Solid to become major hardware-sellers?
I believe you're referring to the announced PS versions of Virtua Fighter 2, Panzer Dragoon Zwei and NiGHTS, plus a new fully polygonal 2.5D update to the original Sonic The Hedgehog? Now there's an interesting question... Did any of these ever go into development? Now that I think about it, what happened to the similarly promised M2 version of Virtua Fighter 3, or its counterpart for the Game.com, never mind that Sonic Xtreme was also supposedly heading to Tiger's handheld?
I've already briefly covered Fighting Force, but what of the other two attempts, indeed? That prototype footage of a planned Dreamcast sequel always felt so pointless to me, especially when a conversion of Spikeout would have done many of the same things and looked infinitely better. More recently, wasn't there also a 2D sequel pitched? Someone high up the food chain at Sega was once quoted as saying they rejected a fourth Streets Of Rage because they didn't recognise just how big this series once was for the company, possibly because it was being proposed under its original Japanese title of Bare Knuckle. Whatever the case, it's clear those in charge don't care much for this franchise, which leads me on to the next question...
The simple answer? Takashi Iizuka. No, seriously. While I can appreciate that he's one of the most enduring figures within Sonic Team, this guy is also responsible for a lot of the design elements that have become a foundation of sorts for entire games, rather than being mere gimmicks. I seem to recall one of his ideas being to have short tube-like sections connecting larger hub style areas in Sonic Adventure to hide load times, and where that was once necessary to mask technical limitations, now it's regularly exploited as a call back to the iconic special stage of Sonic The Hedgehog 2. Similarly, the grinding mechanic that's so often one of the more annoying parts about modern day Sonic games came from his mind. At least he learned from the massive failure experienced by Journey Of Dreams to leave NiGHTS alone, though I fear my words may be a little premature, since there's currently a lot of talk about a possible new game in that series being unveiled soon. In short, Sonic Team desperately needs Yuji Naka, Naoto Ohshima and Hirokazu Yasuhara back!
As for why the average fan can do better, it's probably because the wider community has spent literally years dissecting each of what they deservedly consider the classic games, using various software creators to refine and reinvent certain elements without the burden of needing to keep churning out new product that must sell a justifiable amount to keep their developers in business. From this highly understanding, productive and fertile environment, Sega finally reached out to select talents, though I still don't feel their efforts are enough. Sure, it was great to see Taxman and Stealth go from message board heroes to the saviors of Sega, but why end there? Sonic Mania had the right idea by trying to offer a "parallel universe" type scenario where the Saturn launched with an evolution of the improvements first introduced with Knuckles Chaotix, but wasn't the plan originally to have Sonic Team develop something that demonstrated more of the Saturn's 3D capabilities while retaining a side-on perspective, much like Clockwork Knight, only presumably a lot faster? Taking this concept further, why hasn't Sonic Team reached out to Andrew75 or XL2 to finally show what could be done with all the potential behind Sonic Xtreme?
As someone who was a member of the SSRG community back in the late 1990s/early 2000s, I fondly remember the enthusiasm when Simon Wai discovered that Sonic 2 prototype. Before long, everyone was quick to find whatever they could in this ROM's code, and I even briefly had a site dedicated to sharing some of my own findings in that build along with the Sonic Crackers alpha. From this limited experience, I saw the wider fanbase already splintering between Emulation Zone, Sonic Cult and countless other spin-off (sorry!) groups. I wouldn't dare paint everyone with the same brush, yet a large amount of people cling to the belief that they alone know what's best for Sonic, which naturally leads to division and hostility. Sonic Retro is definitely the closest thing I've seen to a truly unified approach, concentrating on the documentation of fact and leaving such discussion to a forum-based format.
I'm not convinced... Lost World might have recycled a few basic ideas from the ill-fated Xtreme project, but even that never appeared to have a singular direction, so it's possible many of the aspects that are thought of as being direct nods were purely coincidental. On the other hand, perhaps the inspiration many see as coming from Xtreme were the result of Sega merely taking note of what worked in Super Mario Galaxy?
I've often seen the Yakuza series described as being Sega keeping the spirit of Yu Suzuki's more ambitious Shenmue alive without the record-shattering production budget, though it should be noted that AM2 never paid a small fortune to have its characters voiced by relatively known actors. I suspect the reason even more recent games in the greater Yakuza franchise have still to branch out from the same limited area is because it would be financially risky to have its scope expanded beyond introducing a few smaller locations, new characters and minor additional gameplay elements, particularly when you factor in how much of a drain on resources modern development can be. I can't imagine the creation of the in-house Dragon technology came cheap, while Shenmue III is being produced on Unreal Engine 4 to keep its own costs down.
There are so many factors at play here beyond the obvious cultural differences... For one, I was surprised to learn that Sonic never really took off in Japan during the Mega Drive era, with the entire Project Sonic campaign being an opportunity to raise awareness of the company's mascot in his native country. By contrast, American gamers didn't need reintroducing to the character through the Saturn version of Sonic 3D or Sonic R, though I can imagine that many saw the Sonic World portion of Sonic Jam as a teaser of what they'd soon be playing when Sonic Adventure finally arrived. Similarly, it's incredible to look back and think that management within Sega's Japanese headquarters initially resisted Tom Kalinske's proposal to have the original Sonic packed in with the
Mega Drive Genesis, which they'd continue with Sonic 2 and even Sonic 3 in some regions. On the one hand, I can appreciate the argument coming from Japan that you just don't release hardware with a killer app included, but then the Saturn included Virtua Fighter in many parts of the world. Sega is a confusing company, and it is indeed a shame there was so much division within the company at a time it could have done with showing a more united front in order to counter Sony, never mind Nintendo as well.
If anything, Burning Rangers should have been even more of a spectacular send-off for the Saturn. Part of its problem was development being significantly delayed while core gameplay aspects were revised, including the removal of a planned co-operative feature that can still be seen in early media along with a very different HUD that was to have shown a compass very similar to that later seen in the prototype Saturn footage of Shenmue, plus such games as Harley-Davidson & L.A. Riders and perhaps most notably Crazy Taxi. Although it pushes the basic NiGHTS engine to previously unseen levels, I always got the impression that BR would have benefited from even longer to iron out some of the polygon flickering also seen much less prominently in Sonic Team's previous full game (and Christmas NiGHTS, for that matter), but Sega was never going to let such a high profile title slip into the latter half of 1998, especially when this would have not only jeopardised its possibility of release outside Japan, but might have almost certainly ensured it wouldn't be able to make back the large amounts I can only assume it cost to produce then promote in the first place.
Now, would BR have been more suited to the Dreamcast? As seen in Sonic Adventure, also built on an even more heavily modified version of the same basic underlying engine technology, it's likely Sonic Team wouldn't have needed to worry about such factors as glitching polygons, difficulty generating transparency effects or limiting the amount of data it could store on a disc - just try and picture BR with a far more organic and flowing voice navigation system, plus longer video sequences. In a way, it looks as if Sega wasn't quite done with the underlying concept behind BR, since Brave Firefighters took this concept even further. However, where BR could have taken full advantage of the extra power the Saturn's follow-up offered, BF was only possible because of AM1 using the even more recent Hikaru board, which built on Sega's home console of the time to allow for flame, water and smoke effects that I'm sure would have required considerable downgrading if converted back to Dreamcast.
P.S. I really enjoyed writing this!