? SmileBASIC file format ● SmileBASIC Source

Sign In

Register
*Usernames are case-sensitive
Forgot my password

SmileBASIC file format

Quick note: I didn't figure out most of the format of SB files. Trinitro21 did most of the work figuring out the file structure, and the footer was thanks to code plutooo had for smilehax. I just wrote this document, which sums everything we know. DISCLAIMER: This is not 100% correct and may change at any time. By the way, me and Trinitro21 (triangle) wrote an API to download programs from the SmileBASIC servers. It can also display GRPs as PNGs. It's available at http://sbapi.me/ SmileBASIC has its own format for storing its files. There are 2 main types of files, TXT and DAT. Both have a common header and a footer, but DAT files have a secondary header after the common header to store metadata like how many dimensions it has, how large each dimension is, and what type of data it stores. All files are stored in SmileBASIC's ExtData archive stored on the SD card. The default folder is stored as ### in the ExtData, and filenames are prefixed with T or B, for Text (TXT) and Binary (DAT), respectively.

Common Header

The shared part of every SB file is the common header. This contains information such as the username of who wrote it, how large the data stored is, and when it was last modified. The common header is 80 bytes long, but 16 bytes at the end are always zero, so it's assumed it's reserved. All values are little-endian. Offset | Size (bytes) | Description ----------------------------------- &H00 | 2 | Always &H0100, the magic number &H02 | 1 | File type (00 = TXT, 01 = DAT) &H04 | 1 | Zlib compression (0 = no, 1 = yes) &H06 | 1 | Icon shown in the project browser. For TXT files, 00 = TXT and 01 = PRG. For DAT, 00 = DAT and 02 = GRP &H08 | 4 | 32-bit value storing the size of the file, not including the common header or the footer &H0C | 2 | 16-bit value storing the year the file was last modified &H0E | 1 | 8-bit value storing the month the file was last modified &H0F | 1 | 8-bit value storing the day of the month the file was last modified &H10 | 1 | 8-bit value storing the hour the file was last modified &H11 | 1 | 8-bit value storing the minute the file was last modified &H12 | 1 | 8-bit value storing the second the file was last modified &H13 | 1 | Unknown, might be part of datetime &H14 | 18 | The first author's (the original uploader) NNID. This one isn't shown in the project browser &H26 | 18 | The second author's (the last editor) NNID. This one is displayed in the project browser &H38 | 4 | The first author's user ID. Used for controlling the blacklist editable in the project download area &H3C | 4 | The second author's user ID

DAT Secondary Header

TXT and PRG files just put the text after the footer. The DAT and GRP files need more information, which is why they have a secondary header. This secondary header stores information for SB to parse the file properly. Offset | Size (bytes) | Description ----------------------------------- &H50 | 8 | Always "PCBN0001", magic number for the DAT header &H58 | 1 | Data type (03 = 16 bit unsigned (colors as RGBA5551, used for GRPs), 04 = Signed 32 bit integers (int%), 05 = 64-bit double (real#)) &H5A | 1 | Number of dimensions (1-4) &H5C | 4 | 32-bit value storing the size of the first dimension &H60 | 4 | 32-bit value storing the size of the second dimension if applicable &H64 | 4 | 32-bit value storing the size of the third dimension if applicable &H68 | 4 | 32-bit value storing the size of the fourth dimension if applicable Afterwords, the data is stored in row-major order (https://en.wikipedia.org/wiki/Row-_and_column-major_order).

Footer

The footer is a 20 byte HMAC-SHA1 hash of the header and data using this HMAC key: nqmby+e9S?{%U*-V]51n%^xZMk8>b{?x]&?(NmmV[,g85:%6Sqd"'U")/8u77UL2 The footer must be valid in order to download or upload a program/project, not having a valid footer will cause an error when doing either of these.
Author
MasterR3C0RD
Updated
Rating
12 votes
Categories
Keywords
  • fileformat
  • sbfm
  • smilebasic
  • files
  • format
  • file
  • compression
  • dat
  • txt
  • prg
  • grp
  • sb
32 Comment(s) 12Me21 12Me21 Power User Syntax Highlighter Received for creating the code syntax highlighter on SBS Night Person I like the quiet night and sleep late. Express Yourself It looks like icons are actually: 0 = DAT/TXT (depending on file type) 1 = PRG 2 = GRP ralsei ralsei Forum Leader Hidden Achievements Second Year My account is over 2 years old Website Expert Programmer Programming no longer gives me any trouble. Come to me for help, if you like! Programming Strength What happens when a DAT has a PRG icon or a TXT has a GRP icon? 12Me21 12Me21 Power User Syntax Highlighter Received for creating the code syntax highlighter on SBS Night Person I like the quiet night and sleep late. Express Yourself it just shows that icon, nothing special happens ralsei ralsei Forum Leader Hidden Achievements Second Year My account is over 2 years old Website Expert Programmer Programming no longer gives me any trouble. Come to me for help, if you like! Programming Strength oh not sure what i expecting but that was underwhelming DrZog DrZog The 20 byte footer at the end of smilebasic files is a sha1 hmac. The code to calculate it has been public for 3 years, right under our noses: https://github.com/plutooo/smilehax/blob/master/scripts/make_script.py I went ahead and adapted the script to more inline with what we'd want here (to fix the footer to something smilebasic would accept - without overwriting the header). https://gist.github.com/zoogie/ccae68b60d86b25ee8801f3fbc212301 Usage: python SMILEBASIC_FILENAME output_filename (needs python 3) 12Me21 12Me21 Power User Syntax Highlighter Received for creating the code syntax highlighter on SBS Night Person I like the quiet night and sleep late. Express Yourself oh nice I think line 38 should be out.write(message+footer_calculated) instead of out.write(buf+footer_calculated) DrZog DrZog Yeah, I've already caught that mistake. Check rev 3. 12Me21 12Me21 Power User Syntax Highlighter Received for creating the code syntax highlighter on SBS Night Person I like the quiet night and sleep late. Express Yourself Here's a GRP file that's 420x69 pixels (instead of 512x512) RW4532D snail_ snail_ QSP Contest 1 Contest Participant I participated in the first SmileBASIC Source QSP Contest! Helper Received for being very helpful around SmileBASIC Source Achievements Amazing Contributor Someone thinks I'm an awesome person who has done so much for the community! Achievements oh jeez... y and record figured out it was a sha1hmac independently in chat recently but the debugging and stack trace plumbing never got us anywhere. good find. Yttria Yttria Head Admin actually we had everything: after guessing that it was HMAC, the key I posted (as integers) was correct (actually, I had seen it a year and a half ago as a suspicious looking static string in the binary but didn't know its significance), but I guess record and 12 didn't get the byte order correct. snail_ snail_ QSP Contest 1 Contest Participant I participated in the first SmileBASIC Source QSP Contest! Helper Received for being very helpful around SmileBASIC Source Achievements Amazing Contributor Someone thinks I'm an awesome person who has done so much for the community! Achievements either way now we know Yttria Yttria Head Admin yeah, just have to add it to smilebasic-mode and sbfm I think 12 is almost done with a SmileBASIC Tools replacement edit: actually I should open an issue right now 12Me21 12Me21 Power User Syntax Highlighter Received for creating the code syntax highlighter on SBS Night Person I like the quiet night and sleep late. Express Yourself https://12me21.github.io/sbtools/ IDK how compression works yet DrZog DrZog 12Me21, it seems you don't need to know. I've encountered a couple of 04 flagged SB files and they aren't compressed on extdata. I would assume it's there to indicate that the file should be compressed on Smileboom's end, or to decompress on the 3ds client after download. You can safely just ignore the variable I would bet. fwiw, I've also encountered an 05, no idea what that would mean. It was decompressed as well. MasterR3C0RD MasterR3C0RD Amazing Contributor Someone thinks I'm an awesome person who has done so much for the community! Achievements Third Year My account is over 3 years old Website osu! Is Awesome! I love osu! Express Yourself I updated the page to reflect this discovery. I can't believe it was there this entire time. @DrZog If that's true, that likely means it's a bitmask. 0x4 (0b0100) might mean it's compressed, but 0x1 (0b0001, 0x4 | 0x1 = 0x5) might mean something else. This definitely means some more research may be needed. Either way, we got the footer, so we can basically do anything now. Even a custom SmileBASIC server is possible with this information. It's pretty exciting to see that we've finally reached this point. EDIT: There is one more thing that we need to work on, which would be server tokens. SmileBASIC uses these tokens for the server API, but they expire every couple of hours. This might require more reverse engineering but I'm hoping we can figure this out so that we can create tools that can speak to the servers. 12Me21 12Me21 Power User Syntax Highlighter Received for creating the code syntax highlighter on SBS Night Person I like the quiet night and sleep late. Express Yourself I'm pretty sure we've proved that certain GRP files are decompressed when loaded (I think the actual compression flag might be 2, not 4) compression is probably the most important reason for figuring out the header (differently sized GRP files are nice, but we could already just save image data as DAT, so whatever) snail_ snail_ QSP Contest 1 Contest Participant I participated in the first SmileBASIC Source QSP Contest! Helper Received for being very helpful around SmileBASIC Source Achievements Amazing Contributor Someone thinks I'm an awesome person who has done so much for the community! Achievements The GRPs on BIG are smaller than they should be normally so there is definitely compression of some form. DrZog DrZog Interesting thing i just discovered. Every sample demo file in the smilebasic main game file romfs is flagged 05. Every demo file in the 3.6.0 update romfs is 04, however. The update romfs .GRP files were actually compressed (flagged 04), but the text files were not. And in the main game romfs (flagged 05), nothing was compressed. Strange. snail_ snail_ QSP Contest 1 Contest Participant I participated in the first SmileBASIC Source QSP Contest! Helper Received for being very helpful around SmileBASIC Source Achievements Amazing Contributor Someone thinks I'm an awesome person who has done so much for the community! Achievements Compression was seemingly introduced in an update Yttria Yttria Head Admin text files most likely ignore the compression flag. edit: false, but because we didn't know where the REAL compression flag was MasterR3C0RD MasterR3C0RD Amazing Contributor Someone thinks I'm an awesome person who has done so much for the community! Achievements Third Year My account is over 3 years old Website osu! Is Awesome! I love osu! Express Yourself Zlib compression is confirmed to be a different byte (offset &H04) snail_ snail_ QSP Contest 1 Contest Participant I participated in the first SmileBASIC Source QSP Contest! Helper Received for being very helpful around SmileBASIC Source Achievements Amazing Contributor Someone thinks I'm an awesome person who has done so much for the community! Achievements 12Me21 12Me21 Power User Syntax Highlighter Received for creating the code syntax highlighter on SBS Night Person I like the quiet night and sleep late. Express Yourself https://12me21.github.io/sbtools/ added compression support CyberYoshi64 CyberYoshi64 Intermediate Programmer I can make programs, but I still have trouble here and there. Programming Strength Night Person I like the quiet night and sleep late. Express Yourself Great Page Hidden Achievements Well, that explains the 28 bytes on any DAT file when looking at their size in the file browser. dankrause dankrause Expert Programmer Programming no longer gives me any trouble. Come to me for help, if you like! Programming Strength Second Year My account is over 2 years old Website So I don't have access to any files exported from SmileBASIC, but if someone can provide me with a handful, I can probably figure out what that 20 byte footer is. JustGreat JustGreat First Month Joined in the very first month of SmileBASIC Source Website Avatar Taboo I didn't change my avatar for 180 days Website Night Person I like the quiet night and sleep late. Express Yourself Petit Movie Player's converter program has the exported PRG of the Movie Player included in the download. http://rei.to/PetitMovieCreator1.0.0.zip dankrause dankrause Expert Programmer Programming no longer gives me any trouble. Come to me for help, if you like! Programming Strength Second Year My account is over 2 years old Website Looks like it doesn't contain a 20 byte footer though. MasterR3C0RD MasterR3C0RD Amazing Contributor Someone thinks I'm an awesome person who has done so much for the community! Achievements Third Year My account is over 3 years old Website osu! Is Awesome! I love osu! Express Yourself I'll get you one right now, should only take a second. EDIT: Here's 4, two of each file type: http://oboy.smilebasicsource.com:49000/brokenr3c0rd/sbfiles.zip Lemme know if you need any more. dankrause dankrause Expert Programmer Programming no longer gives me any trouble. Come to me for help, if you like! Programming Strength Second Year My account is over 2 years old Website My money is on an sha1 signature, meaning we'd need the signing key that SmileBASIC is using if we wanted to generate it. It should be possible to extract it from the .cia or .3ds file, but I'm not sure that's going to be easy. MasterR3C0RD MasterR3C0RD Amazing Contributor Someone thinks I'm an awesome person who has done so much for the community! Achievements Third Year My account is over 3 years old Website osu! Is Awesome! I love osu! Express Yourself I was trying to get someone to look through the code.bin in IDA Pro and find where the footer was, but they never ended up doing it sadly. snail_ snail_ QSP Contest 1 Contest Participant I participated in the first SmileBASIC Source QSP Contest! Helper Received for being very helpful around SmileBASIC Source Achievements Amazing Contributor Someone thinks I'm an awesome person who has done so much for the community! Achievements The footer isn't apparently a SHA-1 of the file contents from what I can gather; no idea what you guys are on about a signature though. Yttria Yttria Head Admin SmileBASIC spends five years shuffling registers around with 1732584193 -271733879 -1732584194 271733878 -1009589776 loaded when saving files. "I ran a SHA-1 on it and it didn't work" isn't really helpful.