Results 1 to 18 of 18

Thread: About the .PBP file format

  1. #1
    Join Date
    Apr 2015
    Location
    France, Paris
    Timezone
    GMT + 1
    Posts
    310

    Exclamation About the .PBP file format

    UPDATE 4

    Take a look at notes in update 3 then,

    https://github.com/aybe/WXX-Rebirth/releases/tag/PBPv2

    • TIMs detection
    • bug fixes


    At this point you are getting pretty much the same picture as in the attached Excel file, what is not an image is game data; things are clearer.

    There won't be anything new until quite some time now, things needs to be reverse-engineered.






    Update 3

    • it has been simplified, accepts one file and an output directory
    • all bugs are fixed AFAIK
    • better interface/reporting with colors
    • shipped with an Unpack.bat to make your life easier


    The older version was trying to be smart and failed somehow, now there are two tools that each do a very specific task and both do work well. I've ditched the multiple input files feature as this was limiting the output options and was cumbersome, with the current version you can write your own batch file and keep control of things as you wish.

    Please disregard all previous versions.

    There will be a last update (or unless bugs are found), the detection of TIM files; for the time being they are easy to spot : every file that has two sequence of numbers (e.g. TRACKM0A.PBP.00011.00000) is a TIM. You can easily group them within Explorer by sorting by modified date :



    (in this example I've already renamed them to TIMs but they would pop in the exact same order)

    grab it here

    PS1: to preview TIM files in Explorer, grab SageThumbs
    PS2: your AV might suspect these EXEs, there's nothing I can do about that and you have to allow them, also you can always check the code with JustDecompile

    Waiting for your feedback now !








    UPDATE 2

    Two new tools (UnpackCMPEx and UnpackPBPEx)
    1. searches for CMP files and unpacks them
    2. unpacks a PBP (simplified version of the first tool, better suited for batch operations)


    So far, it successfully unpacks a PBP and the CMPs inside it, no false-positive AFAIK



    See USAGE.TXT for an example !

    PS1: TIMs are not yet renamed to .TIM as explained in USAGE.TXT, working on another tool for that.
    PS2: you can use SageThumbs to get a preview of TIMs within Explorer





    UPDATE 1

    Here is (finally) the first version of the unpacker : https://github.com/aybe/WXX-Rebirth/releases

    I guess it works properly since it extracted 10990 items out of 197 files (every PBP file from every version: WO3E, WO3J, WO3SE, WO3U, WO3UDEMO)

    NOTES:

    • full instructions on how to use it are on the README.TXT
    • archives of TIM images (aka CMPs) are not decompressed by the utility -> the UnpackCMP utility is on its way for this.
    • in the web link above there is also an Excel file, it contains a couple of informations already figured from the unique PBP file in the Wipeout 3 US Demo Disc (highly recommended since most files will harbor this structure)
    • the 'unofficial' preferred version of Wipeout 3 to study is the SE version, it is the most recent, the most featured and surprisingly there are less files


    Let me know if you encounter any problem !




    Never say never, all things have a beginning but if we don't start we have zero chance to succeed

    I took a quick look at the format and here are my findings:





    Header (in UINT32 units)
    • at offset 80 it is the number of entries in the container
    • at offset 88 starts the 12 bytes struct of each entry
    • 1st value is absolute offset in file
    • 2nd value is unknown
    • 3rd is the length of the entry
    • i've checked on multiple PBP and they fit perfectly with their length


    We can already get the individual parts of these PBPs and try to get to next step

    IMO these files are certainly compressed but not necessarily encrypted, there used to be some entropy detection software that could help but can't remember the name actually ...

    EDIT

    Just wrote a tiny unpacker, it swallowed every PBP out of WO3SE from \WIPEOUTC and \WIPEOUT3; there is always between 58 and 60 files in each PBP. With time and patience we'll decipher them

    EDIT 2

    I don't know where he got these infos from but they're hintful.

    EDIT 3

    Some progress !

    There are VAG files in these containers: (ship thruster sound)



    There are also a bunch of files which I suspect are textures since there are many entries in their header have a 0x40 value (64 pixels), not .TIM files though. We should prioritize in searching for files from the official formats in the SDK, if not then it could simply be a form of RLE encoding.

    There are also files that looks like polygon data but it's a bit premature IMO :



    The left-most pixel rarely changes and could indicate the polygon type as it can be seen in no$psx debugger.


    Also many of these files are duplicate, basing myself on studying a group of files from the link posted in edit 2.

    2 be continued

    - - - Updated - - -

    EDIT 5

    Houston, we have TIMs from PBPs !






    - - - Updated - - -

    EDIT 6



    - - - Updated - - -

    EDIT 7

    Houston, we're about to crash on vertex data !





    basically, files with no headers are simply LZ77 streams

    EDIT 8

    • Large 2 groups on bottom : ship/team data and I guess textures
    • Bigger files seems to be vertex data as they have some kind of stride

    Last edited by aybe; 30th April 2015 at 11:13 PM.

  2. #2
    Join Date
    Apr 2015
    Location
    France, Paris
    Timezone
    GMT + 1
    Posts
    310

    Default

    Last update for today

    The content of the 8 last files :


  3. #3
    Join Date
    Apr 2010
    Posts
    1,529

    Default

    Wow, great work, man!

    Just out of curiosity, what programs did you use? I only have an hex editor and gcc... xD Makes things so much tougher.
    Last edited by Xpand; 26th April 2015 at 11:19 AM.

  4. #4
    Join Date
    Mar 2015
    Posts
    58

    Default

    Yep, great work. If you keep going that way we will have a Wipeout3 model/track viewer soon!

  5. #5
    Join Date
    Apr 2015
    Location
    France, Paris
    Timezone
    GMT + 1
    Posts
    310

    Default

    Quote Originally Posted by Xpand View Post
    Wow, great work, man!

    Just out of curiosity, what programs did you use? I only have an hex editor and gcc... xD Makes things so much tougher.
    Only an hex editor but it's quite featured as you can see above (coloring, etc) and the infos from wipeout.js; the rest I wrote it in C#

    - - - Updated - - -

    Quote Originally Posted by tigrou View Post
    Yep, great work. If you keep going that way we will have a Wipeout3 model/track viewer soon!
    this is going to take time, I think I'll post my results somewhere so people can try by themselves.

    - - - Updated - - -

    To the admins:

    Can we post a decompressor of these files at the forum, or not ?

  6. #6
    Join Date
    Mar 2015
    Posts
    58

    Default

    Quote Originally Posted by aybe View Post
    Only an hex editor but it's quite featured as you can see above (coloring, etc) and the infos from wipeout.js; the rest I wrote it in C#
    To the admins:
    Can we post a decompressor of these files at the forum, or not ?
    What about creating a github repository ?

  7. #7
    Join Date
    Apr 2015
    Location
    France, Paris
    Timezone
    GMT + 1
    Posts
    310

    Default

    Yes, probably within the next days, there are a couple of things I would like to sort before:

    - references because I've just been coding against Monogame and WPF and those utilities do drag lots of unneeded DLLs
    - get my renderer on par with Dominic's one (textured objects + Wipeout 2), shouldn't be too hard.
    - clean up the thing a bit, write a README etc ...
    (those are imminent, easy tasks)

    I'm done with the PBP/CMP/LZ77 unpackers, still trying to figure out whether I can properly detect LZ77 streams so one wouldn't have to bother to manually check whether it is encoded or not (to get a 1-click PBP unpack)

    other things of interests (later)
    - add viewers to the GUI app. basically you select a file and you can see 3d objects etc ...
    - implement a viewer with triangle selection, we'd instantly know what object is what data hence help us understand the thing better
    - begin the coding related to the game

    regarding previous talks it seems that the way to go would be to have another name than Wipeout, been thinking about WXX-Rebirth (like http://www.dxx-rebirth.com/), what do you think, sounds cool isn't it ?

    - - - Updated - - -

    Just 'secured' the thing : https://github.com/aybe/WXX-Rebirth

    I'll post the WO3 unpackers there ASAP ...

  8. #8
    Join Date
    Apr 2015
    Location
    France, Paris
    Timezone
    GMT + 1
    Posts
    310

    Default

    About to post an unpacker, which I guess properly understands the format now (its behavior matches my manual checking of every file, while previously it was not). The final step is to properly detect CMP streams and I think we'll be done with the outer shell. Will push command-line tools hopefully by tonight

  9. #9
    Join Date
    Mar 2015
    Posts
    58

    Default

    @aybe :

    one question : is Wipeout 3 simply using a container (PBP) and then the inside is files similar to WO1/WOXL ? Or is the inside of PBP something totally different from what we have seen before ?
    I'm not talking about common PSX files (like TIM textures) but rather wipeout specific track formats (like TRS/TRV/TRF, ...)

  10. #10
    Join Date
    Apr 2015
    Location
    France, Paris
    Timezone
    GMT + 1
    Posts
    310

    Default

    Some things differs definitely, some not.

    PS: just found out that US versions are PBN instead, but they;re working as well (they have different content though)

    - I guess the vertices are the same and mostly for polys, though we'll only know once we progressed.
    (from what I've seen with that little app that renders as bitmap, i.e. PCM sounds are instantly recognisable and those that I suspect being the above twos do look like WO1/2)

    - there is an object system with names (cubeXX, goteki, etc ...) my initial thoughts were to attack the format by studying the 2nd prototype track as it's very simple and contains cubes, these are small structs that once we figure one of them we should progress quite a bit. I think I've figured where are the indices but it's a bit puzzling.

    - i do not suspect other things to differ that much since as we're on a PS1 and as you can see in no$psx, the rendering possibilities are known, there aren't 100s of possible combinations (tri, quad, face, vert, etc ...)

    This in conjunction with a vertices renderer should significantly help, basically if you render an array of vertices as spheres and look at them from a certain point of view you'd know very quickly if's vertex data or not; e.g. the 2nd prototype track will be of help because there are orphan cubes in the circuit, a significant hint. (will test that theory with WOXL data first as we know what to expect)

    for the rest you'll discover that by yourself

    |
    |
    V

    UnpackPBP is done and works well, still would like to throw all the pb(n/p)s to test it, writing a little doc, a few other things then I'll put on GitHub.

    It's been quite time consuming to get from a pseudo-format to a well understood (i guess by now) format. Maybe tonight or tomorrow, actually time runs pretty fast when you do something interesting !

  11. #11
    Join Date
    Apr 2015
    Location
    France, Paris
    Timezone
    GMT + 1
    Posts
    310

    Default

    Unpacker released at the top of this page

  12. #12
    Join Date
    Apr 2015
    Location
    France, Paris
    Timezone
    GMT + 1
    Posts
    310

    Default

    The detection and unpacking of CMP files inside PBPs is mostly done:



    - - - Updated - - -

    New update ! (see 1st post)

  13. #13
    Join Date
    Feb 2004
    Location
    Germany
    Timezone
    GMT + 1
    PSN ID
    sausehuhn
    Posts
    3,329

    Default

    This thread feels like watching a detective solving his case. Keeps me checking it every day

  14. #14
    Join Date
    Apr 2015
    Location
    France, Paris
    Timezone
    GMT + 1
    Posts
    310

    Default

    I think I'm done by now, just shipped what I hope will be the latest update; now comes the real reverse-engineering work !

  15. #15
    Join Date
    Apr 2015
    Location
    France, Paris
    Timezone
    GMT + 1
    Posts
    310

    Default

    Another and last update ! no more until a major milestone now ...

  16. #16
    Join Date
    May 2012
    Timezone
    GMT -6
    Posts
    863

    Default

    I hate to be a buzzkill, but the Github links are broken.

  17. #17
    Join Date
    Apr 2015
    Location
    France, Paris
    Timezone
    GMT + 1
    Posts
    310

    Default

    Sorry ! I'm not receiving any notifications for some reason ...

    The repository has gone private, and they're gone anyway !

    But I have a backup here : https://1drv.ms/u/s!AqAtR29m3cqGtR6XE-PtUrbLNpg7

  18. #18
    Join Date
    May 2012
    Timezone
    GMT -6
    Posts
    863

    Default

    Thanks.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •