What is PSSE?
PSSE stands for PlayStation Suite Encrypted. It is an encryption layer that encrypts the PSM game's filesystem.
The encryption layer is comparable to the PS Vita Filesystem in many ways; however, it seemed that there was a bug at Sony, where besides executable files, the first directory level of a PSM app is not encrypted with PSSE, whose directory level is below the root of RO/. Similarly, the PSM SaveData is not encrypted by such layer either.
|0x00||0x50535345 (ASCII "PSSE" / "PSME")||Magic Number / File Header|
|0x8||int64||Decrypted file size|
|0x10||int32||PSSE Type (always 0x1)|
|0x14||char[0x24] CONTENT_ID||Contents of /RW/System/content_id|
|0x40||char[0x10]||MD5 Of Original File (Plaintext)|
|0x50||char[0x20]||(Twice?) Encrypted Filename|
|0x680||Content Data||Actural encrypted file data|
AES-128-CBC decrypt 0x10 bytes from 0x70 using PSSE Header IV and PSSE Header Key the resulting decryption is the Contents IV, now take 0x10 bytes from 0x120 of the game's NoPsmDrm RIF, this is the Content Key (if its one of the DLL's in the PSM Runtime Package, then the content key is Runtime PSSE App Key)
the content data section (0x680 onwards) is made up of blocks, each 0x8000 bytes in size however the first block is only 0x7980 bytes, (0x8000 - 0x680) as if the block started at 0x00, however you start reading from 0x680 all subsequent blocks are 0x8000 bytes, so block 0 is at 0x680, block 1 at 0x8000, block 2 at 0x10000 etc, and, at every interval of 0x80000 bytes there is an 0x400 byte gap before the next block starts
(that next block is then 0x7C00 bytes long, as if the block started at 0x80000, even though it starts at 0x800400) and of course all subsequent ones are then 0x8000 past there again,
when reading each block the IV used to decrypt it changes slightly, usually just the first byte changes. to calculate the IV for any given block, take that block's number (counting sequentially from 0x800, following all above rules) as an int32, little endain, put it into a byte-array. and make that byte array be 0x10 bytes long, with the block number at the very start eg, for block 4 it would be 0x04000000000000000000000000000000 then, XOR the original IV from the PSSE Header, with that byte array,
Decrypt each block using AES-128-CBC with the content IV you calculated earlier from header + block ID and the content KEY from the rif,
to decrypt the entire file, decrypt each block in-order.
A script that implements the above algorithm can be found here: https://github.com/KuromeSan/psse-decrypt/blob/master/decrypt.py it is in the public domain, so feel free to use it for whatever you want