LoginLogin

SmileBASIC file format

Root / Submissions / [.]

MasterR3C0RDCreated:
Quick note: I didn't figure out most of the format of SB files. Trinitro21 did most of the work figuring out the file structure, and the footer was thanks to code plutooo had for smilehax. I just wrote this document, which sums everything we know. DISCLAIMER: This is not 100% correct and may change at any time. By the way, me and Trinitro21 (triangle) wrote an API to download programs from the SmileBASIC servers. It can also display GRPs as PNGs. It's available at http://sbapi.me/ SmileBASIC has its own format for storing its files. There are 2 main types of files, TXT and DAT. Both have a common header and a footer, but DAT files have a secondary header after the common header to store metadata like how many dimensions it has, how large each dimension is, and what type of data it stores. All files are stored in SmileBASIC's ExtData archive stored on the SD card. The default folder is stored as ### in the ExtData, and filenames are prefixed with T or B, for Text (TXT) and Binary (DAT), respectively.

Common Header

The shared part of every SB file is the common header. This contains information such as the username of who wrote it, how large the data stored is, and when it was last modified. The common header is 80 bytes long on the 3DS and 112 bytes on Switch. All values are little-endian. 
Offset | Size (bytes) | Description
-----------------------------------
&H00   | 2            | Version of SmileBASIC the file was created under (00 or 01 = 3DS/WiiU, 04 = Switch)
&H02   | 1            | File type (00 = TXT, 01 = DAT. Switch: 02 = GRP, 04 = META)
&H04   | 1            | Zlib compression (0 = no, 1 = yes)
&H06   | 1            | Icon shown in the project browser. For TXT files, 00 = TXT and 01 = PRG. For DAT, 00 = DAT and 02 = GRP
&H08   | 4            | 32-bit value storing the size of the file, not including the common header or the footer
&H0C   | 2            | 16-bit value storing the year the file was last modified
&H0E   | 1            | 8-bit value storing the month the file was last modified
&H0F   | 1            | 8-bit value storing the day of the month the file was last modified
&H10   | 1            | 8-bit value storing the hour the file was last modified
&H11   | 1            | 8-bit value storing the minute the file was last modified
&H12   | 1            | 8-bit value storing the second the file was last modified
&H13   | 1            | Unknown, might be part of date/time
A very important difference to note is that in SB4, uploader information is slightly longer. Offsets for 3DS: 
&H14   | 18           | The first author's (the original uploader)  NNID. This one isn't shown in the project browser
&H26   | 18           | The second author's (the last editor) NNID. This one is displayed in the project browser
&H38   | 4            | The first author's user ID. Used for controlling the blacklist editable in the project download area
&H3C   | 4            | The second author's user ID
Offsets for Switch: 
&H14   | 32           | The first author's (the original uploader)  NNID. This one isn't shown in the project browser
&H34   | 32           | The second author's (the last editor) NNID. This one is displayed in the project browser
&H54   | 4            | The first author's user ID. Used for controlling the blacklist editable in the project download area
&H58   | 4            | The second author's user ID
The header ends at &H50 on 3DS and &H70 on Switch.

DAT Secondary Header (Petit Computer BiNary)

TXT and PRG files just place the text after the footer. The DAT and GRP files need more information, which is why they have a secondary header. This secondary header stores information for SB to parse the file properly. (offset is the offset after the header, which changes depending on the version of SmileBASIC in use.) 
Offset | Size (bytes) | Description
-----------------------------------
&H00   | 8            | Always the ASCII string "PCBN000n", where n is the device type (offset &H00)
&H08   | 1            | Data type (03 = 16 bit unsigned (colors as RGBA5551, used for GRPs), 04 = Signed 32 bit integers (int%), 05 = 64-bit double (real#)). GRPs in SB4 are stored as integer (data type &H04) DAT files since they use RGBA8888 encoding.
&H0A   | 1            | Number of dimensions (1-4)
&H0C   | 4            | 32-bit value storing the size of the first dimension
&H10   | 4            | 32-bit value storing the size of the second dimension if applicable
&H14   | 4            | 32-bit value storing the size of the third dimension if applicable
&H18   | 4            | 32-bit value storing the size of the fourth dimension if applicable
Afterward, the data is stored in row-major order (https://en.wikipedia.org/wiki/Row-_and_column-major_order).

META Project file (Petit Computer Project Metadata

META files are used in projects to store metadata about a project, including icon, name, and description. (offset is the offset after the header, which changes depending on the version of SmileBASIC in use.) 
Offset | Size (bytes) | Description
-----------------------------------
&H00   | 8            | Always the ASCII string "PCPM0005"
&H08   | 48           | The project name (UCS-2)
&H38   | 4576         | The project description (UCS-2)
&H1218 | 4            | The width of the project icon
The data following is the icon's pixel data, encoded in BGRA8888. Icons are always square, so you can find the length of the icon data by squaring the icon width and multiplying by 4.

Projects

File type 2 in SB3 and file type 3 on SB4 are reserved for project files that are used for uploading/downloading projects from the server. They are unpacked by SB when downloaded into a proper file structure. As such, this file format should never be encountered unless you're talking to the SB servers yourself. If you're doing that, figuring out project file format is left as an exercise for you :)

Footer

The footer is a 20 byte HMAC-SHA1 hash of the header and data using this HMAC key: nqmby+e9S?{%U*-V]51n%^xZMk8>b{?x]&?(NmmV[,g85:%6Sqd"'U")/8u77UL2 The footer must be valid in order to download or upload a program/project, not having a valid footer will cause an error when doing either of these.
ResourcesSystem Guides
dankrause#1
So I don't have access to any files exported from SmileBASIC, but if someone can provide me with a handful, I can probably figure out what that 20 byte footer is.
JustGreat#2
Replying to:dankrause
So I don't have access to any files exported from SmileBASIC, but if someone can provide me with a handful, I can probably figure out what that 20 byte footer is.
Petit Movie Player's converter program has the exported PRG of the Movie Player included in the download. http://rei.to/PetitMovieCreator1.0.0.zip
dankrause#3
Replying to:dankrause
So I don't have access to any files exported from SmileBASIC, but if someone can provide me with a handful, I can probably figure out what that 20 byte footer is.
Looks like it doesn't contain a 20 byte footer though.
MasterR3C0RD#4
Replying to:dankrause
So I don't have access to any files exported from SmileBASIC, but if someone can provide me with a handful, I can probably figure out what that 20 byte footer is.
I'll get you one right now, should only take a second. EDIT: Here's 4, two of each file type: http://oboy.smilebasicsource.com:49000/brokenr3c0rd/sbfiles.zip Lemme know if you need any more.
dankrause#5
Replying to:dankrause
So I don't have access to any files exported from SmileBASIC, but if someone can provide me with a handful, I can probably figure out what that 20 byte footer is.
My money is on an sha1 signature, meaning we'd need the signing key that SmileBASIC is using if we wanted to generate it. It should be possible to extract it from the .cia or .3ds file, but I'm not sure that's going to be easy.
MasterR3C0RD#6
Replying to:dankrause
So I don't have access to any files exported from SmileBASIC, but if someone can provide me with a handful, I can probably figure out what that 20 byte footer is.
I was trying to get someone to look through the code.bin in IDA Pro and find where the footer was, but they never ended up doing it sadly.
CyberYoshi64#7
Well, that explains the 28 bytes on any DAT file when looking at their size in the file browser.
snail_#8
Replying to:dankrause
So I don't have access to any files exported from SmileBASIC, but if someone can provide me with a handful, I can probably figure out what that 20 byte footer is.
The footer isn't apparently a SHA-1 of the file contents from what I can gather; no idea what you guys are on about a signature though.
Yolkai#9
Replying to:dankrause
So I don't have access to any files exported from SmileBASIC, but if someone can provide me with a handful, I can probably figure out what that 20 byte footer is.
SmileBASIC spends five years shuffling registers around with 1732584193 -271733879 -1732584194 271733878 -1009589776 loaded when saving files. "I ran a SHA-1 on it and it didn't work" isn't really helpful.
DrZog#10
The 20 byte footer at the end of smilebasic files is a sha1 hmac. The code to calculate it has been public for 3 years, right under our noses: https://github.com/plutooo/smilehax/blob/master/scripts/make_script.py I went ahead and adapted the script to more inline with what we'd want here (to fix the footer to something smilebasic would accept - without overwriting the header). https://gist.github.com/zoogie/ccae68b60d86b25ee8801f3fbc212301 Usage: python SMILEBASIC_FILENAME output_filename (needs python 3)
12Me21#11
Replying to:DrZog
The 20 byte footer at the end of smilebasic files is a sha1 hmac. The code to calculate it has been public for 3 years, right under our noses: https://github.com/plutooo/smilehax/blob/master/scripts/make_script.py I went ahead and adapted the script to more inline with what we'd want here (to fix the footer to something smilebasic would accept - without overwriting the header). https://gist.github.com/zoogie/ccae68b60d86b25ee8801f3fbc212301 Usage: python SMILEBASIC_FILENAME output_filename (needs python 3)
oh nice I think line 38 should be out.write(message+footer_calculated) instead of out.write(buf+footer_calculated)
DrZog#12
Replying to:DrZog
The 20 byte footer at the end of smilebasic files is a sha1 hmac. The code to calculate it has been public for 3 years, right under our noses: https://github.com/plutooo/smilehax/blob/master/scripts/make_script.py I went ahead and adapted the script to more inline with what we'd want here (to fix the footer to something smilebasic would accept - without overwriting the header). https://gist.github.com/zoogie/ccae68b60d86b25ee8801f3fbc212301 Usage: python SMILEBASIC_FILENAME output_filename (needs python 3)
Yeah, I've already caught that mistake. Check rev 3.
12Me21#13
Replying to:DrZog
The 20 byte footer at the end of smilebasic files is a sha1 hmac. The code to calculate it has been public for 3 years, right under our noses: https://github.com/plutooo/smilehax/blob/master/scripts/make_script.py I went ahead and adapted the script to more inline with what we'd want here (to fix the footer to something smilebasic would accept - without overwriting the header). https://gist.github.com/zoogie/ccae68b60d86b25ee8801f3fbc212301 Usage: python SMILEBASIC_FILENAME output_filename (needs python 3)
Here's a GRP file that's 420x69 pixels (instead of 512x512) RW4532D
snail_#14
Replying to:DrZog
The 20 byte footer at the end of smilebasic files is a sha1 hmac. The code to calculate it has been public for 3 years, right under our noses: https://github.com/plutooo/smilehax/blob/master/scripts/make_script.py I went ahead and adapted the script to more inline with what we'd want here (to fix the footer to something smilebasic would accept - without overwriting the header). https://gist.github.com/zoogie/ccae68b60d86b25ee8801f3fbc212301 Usage: python SMILEBASIC_FILENAME output_filename (needs python 3)
oh jeez... y and record figured out it was a sha1hmac independently in chat recently but the debugging and stack trace plumbing never got us anywhere. good find.
Yolkai#15
Replying to:DrZog
The 20 byte footer at the end of smilebasic files is a sha1 hmac. The code to calculate it has been public for 3 years, right under our noses: https://github.com/plutooo/smilehax/blob/master/scripts/make_script.py I went ahead and adapted the script to more inline with what we'd want here (to fix the footer to something smilebasic would accept - without overwriting the header). https://gist.github.com/zoogie/ccae68b60d86b25ee8801f3fbc212301 Usage: python SMILEBASIC_FILENAME output_filename (needs python 3)
actually we had everything: after guessing that it was HMAC, the key I posted (as integers) was correct (actually, I had seen it a year and a half ago as a suspicious looking static string in the binary but didn't know its significance), but I guess record and 12 didn't get the byte order correct.
snail_#16
Replying to:DrZog
The 20 byte footer at the end of smilebasic files is a sha1 hmac. The code to calculate it has been public for 3 years, right under our noses: https://github.com/plutooo/smilehax/blob/master/scripts/make_script.py I went ahead and adapted the script to more inline with what we'd want here (to fix the footer to something smilebasic would accept - without overwriting the header). https://gist.github.com/zoogie/ccae68b60d86b25ee8801f3fbc212301 Usage: python SMILEBASIC_FILENAME output_filename (needs python 3)
either way now we know
Yolkai#17
Replying to:DrZog
The 20 byte footer at the end of smilebasic files is a sha1 hmac. The code to calculate it has been public for 3 years, right under our noses: https://github.com/plutooo/smilehax/blob/master/scripts/make_script.py I went ahead and adapted the script to more inline with what we'd want here (to fix the footer to something smilebasic would accept - without overwriting the header). https://gist.github.com/zoogie/ccae68b60d86b25ee8801f3fbc212301 Usage: python SMILEBASIC_FILENAME output_filename (needs python 3)
yeah, just have to add it to smilebasic-mode and sbfm I think 12 is almost done with a SmileBASIC Tools replacement edit: actually I should open an issue right now
12Me21#18
Replying to:DrZog
The 20 byte footer at the end of smilebasic files is a sha1 hmac. The code to calculate it has been public for 3 years, right under our noses: https://github.com/plutooo/smilehax/blob/master/scripts/make_script.py I went ahead and adapted the script to more inline with what we'd want here (to fix the footer to something smilebasic would accept - without overwriting the header). https://gist.github.com/zoogie/ccae68b60d86b25ee8801f3fbc212301 Usage: python SMILEBASIC_FILENAME output_filename (needs python 3)
https://12me21.github.io/sbtools/ IDK how compression works yet
DrZog#19
Replying to:DrZog
The 20 byte footer at the end of smilebasic files is a sha1 hmac. The code to calculate it has been public for 3 years, right under our noses: https://github.com/plutooo/smilehax/blob/master/scripts/make_script.py I went ahead and adapted the script to more inline with what we'd want here (to fix the footer to something smilebasic would accept - without overwriting the header). https://gist.github.com/zoogie/ccae68b60d86b25ee8801f3fbc212301 Usage: python SMILEBASIC_FILENAME output_filename (needs python 3)
12Me21, it seems you don't need to know. I've encountered a couple of 04 flagged SB files and they aren't compressed on extdata. I would assume it's there to indicate that the file should be compressed on Smileboom's end, or to decompress on the 3ds client after download. You can safely just ignore the variable I would bet. fwiw, I've also encountered an 05, no idea what that would mean. It was decompressed as well.
MasterR3C0RD#20
Replying to:DrZog
The 20 byte footer at the end of smilebasic files is a sha1 hmac. The code to calculate it has been public for 3 years, right under our noses: https://github.com/plutooo/smilehax/blob/master/scripts/make_script.py I went ahead and adapted the script to more inline with what we'd want here (to fix the footer to something smilebasic would accept - without overwriting the header). https://gist.github.com/zoogie/ccae68b60d86b25ee8801f3fbc212301 Usage: python SMILEBASIC_FILENAME output_filename (needs python 3)
I updated the page to reflect this discovery. I can't believe it was there this entire time. @DrZog If that's true, that likely means it's a bitmask. 0x4 (0b0100) might mean it's compressed, but 0x1 (0b0001, 0x4 | 0x1 = 0x5) might mean something else. This definitely means some more research may be needed. Either way, we got the footer, so we can basically do anything now. Even a custom SmileBASIC server is possible with this information. It's pretty exciting to see that we've finally reached this point. EDIT: There is one more thing that we need to work on, which would be server tokens. SmileBASIC uses these tokens for the server API, but they expire every couple of hours. This might require more reverse engineering but I'm hoping we can figure this out so that we can create tools that can speak to the servers.
12