How the ECDSA algorithm works

To popular demand, I have decided to try and explain how the ECDSA algorithm works. I’ve been struggling a bit to understand it properly and while I found a lot of documentation about it, I haven’t really found any “ECDSA for newbies” anywhere. So I thought it would be good to explain in simple terms how it works so others can learn from my research. I have found some websites that explain the basic principles but nowhere near enough to actually understand it, others that explains things without any basics, making it incomprehensible, and others that go way too deep into the the mathematics behind it.

ECDSA stands for “Elliptic Curve Digital Signature Algorithm”, it’s used to create a digital signature of data (a file for example) in order to allow you to verify its authenticity without compromising its security. Think of it like a real signature, you can recognize someone’s signature, but you can’t forge it without others knowing. The ECDSA algorithm is basically all about mathematics.. so I think it’s important to start by saying : “hey kids, don’t slack off at school, listen to your teachers, that stuff might be useful for you some day!” ūüôā But these maths are fairly complicated, so while I’ll try to vulgarize it and make it understandable for non technical people, you will still probably need some knowledge in mathematics to understand it properly. I will do this in two parts, one that is a sort of high level explanation about how it works, and another where I dig deeper into its inner workings to complete your understanding. Note however that I’ve just recently learned this stuff, so I’m definitely not an expert on the matter.

So the principle is simple, you have a mathematical equation which draws a curve on a graph, and you choose a random point on that curve and consider that your point of origin. Then you generate a random number, this is your private key, you do some magical mathematical equation using that random number and that “point of origin” and you get a second point on the curve, that’s your public key. When you want to sign a file, you will use this private key (the random number) with a hash of the file (a unique number to represent the file) into a magical equation and that will give you your signature. The signature itself is divided into two parts, called R and S. In order to verify that the signature is correct, you only need the public key (that point on the curve that was generated using the private key) and you put that into another magical equation with one part of the signature (S), and if it was signed correctly using the the private key, it will give you the other part of the signature (R). So to make it short, a signature consists of two numbers, R and S, and you use a private key to generate R and S, and if a mathematical equation using the public key and S gives you R, then the signature is valid. There is no way to know the private key or to create a signature using only the public key.

Alright, now for the more in depth understanding, I suggest you take an aspirin right now as this might hurt! ūüėõ

Let’s start with the basics (which may be boring for people who know about it, but is mandatory for those who don’t) : ECDSA uses only integer mathematics, there are no floating points (this means possible values are 1, 2, 3, etc.. but not 1.5..), ¬†also, the range of the numbers is bound by how many bits are used in the signature (more bits means higher numbers, means more security as it becomes harder to ‘guess’ the critical numbers used in the equation), as you should know, computers use ‘bits’ to represent data, a bit is a ‘digit’ in binary notation (0 and 1) and 8 bits represent one byte. Every time you add one bit, the maximum number that can be represented doubles, with 4 bits you can represent values 0 to 15 (for a total of 16 possible values), with 5 bits, you can represent 32 values, with 6 bits, you can represent 64 values, etc.. one byte (8 bits) can represent 256 values, and 32 bits can represent 4294967296 values (4 Giga).. Usually ECDSA will use 160 bits total, so that makes… well, a very huge number with 49 digits in it…

ECDSA is used with a SHA1 cryptographic hash of the message to sign (the file). A hash is simply another mathematical equation that you apply on every byte of data which will give you a number that is unique to your data. Like for example, the sum of the values of all bytes may be considered a very dumb hash function. So if anything changes in the message (the file) then the hash will be completely different. In the case of the SHA1 hash algorithm, it will always be 20 bytes (160 bits). It’s very useful to validate that a file has not been modified or corrupted, you get the 20 bytes hash for a file of any size, and you can easily recalculate that hash to make sure it matches. What ECDSA signs is actually that hash, so if the data changes, the hash changes, and the signature isn’t valid anymore.

Now, how does it work? Well Elliptic Curve cryptography is based on an equation of the form :

y^2 = (x^3 + a * x + b) mod p

First thing you notice is that there is a modulo and that the ‘y‘ is a square. This means that for any x coordinate, you will have two values of y and that the curve is symmetric on the X axis. The modulo is a prime number and makes sure that all the values are within our range of 160 bits and it allows the use of “modular square root” and “modular multiplicative inverse” mathematics which make calculating stuff easier (I think). Since we have a modulo (p) , it means that the possible values of y^2 are between ¬†0 and p-1, which gives us p total possible values. However, since we are dealing with integers, only a smaller subset of those values will be a “perfect square” (the square value of two integers), which gives us N possible points on the curve where N < p (N being the number of perfect squares between 0 and p). Since each¬†x will yield two points (positive and negative values of the square-root of y^2), this means that there are N/2 possible ‘x‘ coordinates that are valid and that give a point on the curve. So this elliptic curve has a finite number of points on it, and it’s all because of the integer calculations and the modulus. Another thing you need to know about Elliptic curves, is the notion of “point addition“. It is defined as adding one point P to another point Q will lead to a point S such that if you draw a line from P to Q, it will intersect the curve on a third point R which is the negative value of S¬†(remember that the curve is symmetric on the X axis). In this case, we define R = -S¬†to represent the symmetrical point of¬†R on the X axis. This is easier to illustrate with an image : So you can see a curve of the form y^2 = x^3 + ax + b (where a = -4 and b = 0), which is symmetric on the X axis, and where P+Q¬†is the symmetrical point through X of the point R¬†which is the third intersection of a line going from P to Q. In the same manner, if you do P + P, ¬†it will be the symmetrical point of R¬†which is the intersection of the line that is a tangent to the point P.. And P + P + P is the addition between the resulting point of P+P with the point P¬†since P + P + P can be written as (P+P) + P.. This defines the “point multiplication” where k*P is the addition of the point P to itself k times… here are two examples showing this : ¬†

Here, you can see two elliptic curves, and a point P¬†from which you draw the tangent, it intersects the curve with a third point, and its symmetric point it 2P, then from there, you draw a line from 2P and P¬†and it will intersect the curve, and the symmetrical point is 3P. etc… you can keep doing that for the point multiplication. You can also already guess why you need to take the symmetric point of¬†R when doing the addition, otherwise, multiple additions of the same point will always give the same line and the same three intersections.

One particularity of this point multiplication is that if you have a point R = k*P, where you know R and you know P, there is no way to find out what the value of ‘k‘ is. Since there is no point subtraction or point division, you cannot just resolve k = R/P. Also, since you could be doing millions of ¬†point additions, you will just end up on another point on the curve, and you’d have no way of knowing “how” you got there. You can’t reverse this operation, and you can’t find the value ‘k‘ which was multiplied with your point P to give you the resulting point R.

This thing where you can’t find the multiplicand even when you know the original and destination points is the whole basis of the security behind the ECDSA algorithm, and the principle is called a “trap door function“.

Now that we’ve handled the “basics”, let’s talk about the actual ECDSA signature algorithm. For ECDSA, you first need to know your curve parameters, those are a, b, p, N and G. You already know that ‘a‘ and ‘b‘ are the parameters of the curve function (y^2 = x^3 + ax + b), that ‘p‘ is the prime modulus, ¬†and that ‘N‘ is the number of points of the curve, but there is also ‘G‘ that is needed for ECDSA, and it represents a ‘reference point’ or a point of origin if you prefer. Those curve parameters are important and without knowing them, you obviously can’t sign or verify a signature. Yes, verifying a signature isn’t just about knowing the public key, you also need to know the curve parameters for which this public key is derived from.

So first of all, you will have a private and a public key.. the private key is a random number (of 20 bytes) that is generated, and the public key is a point on the curve generated from the point multiplication of G with the private key. We set ‘dA‘ as the private key (random number) and ‘Qa‘ as the public key (a point), so we have : Qa = dA * G (where G is the point of reference in the curve parameters).

So how do you sign a file/message ? First, you need to know that the signature is 40 bytes and is represented by two values of 20 bytes each, the first one is called R and the second one is called S.. so the pair (R, S) together is your ECDSA signature.. now here’s how you can create those two values in order to sign a file.. first you must generate a random value ‘k‘ (of 20 byes), and use point multiplication to calculate the point P=k*G. That point’s x value will represent ‘R‘. Since the point on the curve P is represented by its (x, y) coordinates (each being 20 bytes long), you only need the ‘x‘ value (20 bytes) for the signature, and that value will be called ‘R‘. Now all you need is the ‘S‘ value.

To calculate S, you must make a SHA1 hash of the message, this gives you a 20 bytes value that you will consider as a very huge integer number and we’ll call it ‘z‘. Now you can calculate S¬†using the equation :

S = k^-1 (z + dA * R) mod p

Note here the k^-1 which is the ‘modular multiplicative inverse‘ of k… it’s basically the inverse of k, but since we are dealing with integer numbers, then that’s not possible, so it’s a number such that (k^-1 * k ) mod p is equal to 1. And again, I remind you that k is the random number used to generate R, z is the hash of the message to sign, dA is the private key and R is the x coordinate of k*G (where G is the point of origin of the curve parameters).

Now that you have your signature, you want to verify it, it’s also quite simple, and you only need the public key (and curve parameters of course) to do that. You use this equation to calculate a point P :

P=  S^-1*z*G + S^-1 * R * Qa

If the x coordinate of the point P is equal to R, that means that the signature is valid, otherwise it’s not.

Pretty simple, huh? now let’s see why and how… and this is going to require some mathematics to verify :

We have :

P = S^-1*z*G + S^-1 * R *Qa

but Qa = dA*G, so:

P = S^-1*z*G + S^-1 * R * dA*G = S^-1 (z + dA* R) * G

But the x coordinate of P must match R and R is the x coordinate of k * G, which means that :

k*G = S^-1 (z + dA * R) *G

we can simplify by removing G which gives us :

k = S^-1(z + dA * R)

by inverting k and S, we get :

S = k^-1 (z + dA *R)

and that is the equation used to generate the signature.. so it matches, and that is the reason why you can verify the signature with it.

You can note that you need both ‘k‘ (random number) and ‘dA‘ (the private key) in order to calculate S, but you only need R and Qa (public key) to validate the signature. And since R=k*G and Qa = dA*G and because of the trap door function in the ECDSA point multiplication (explained above), we cannot calculate dA or k from knowing Qa and R, this makes the ECDSA algorithm secure, there is no way of finding the private keys, and there is no way of faking a signature without knowing the private key.

The ECDSA algorithm is used everywhere and has not been cracked and it is a vital part of most of today’s security.

Now I’ll discuss on how and why the ECDSA signatures that Sony ¬†used in the PS3 were faulty and how it allowed us to gain access to their private key.

So you remember the equations needed to generate a signature.. R = k*G and S= k^-1(z + dA*R) mod p.. well this equation’s strength is in the fact that you have one equation with two unknowns (k and dA) so there is no way to determine either one of those. However, the security of the algorithm is based on its implementation and it’s important to make sure that ‘k‘ is randomly generated and that there is no way that someone can guess, calculate, or use a timing attack or any other type of attack in order to find the random value ‘k‘. But Sony made a huge mistake in their implementation, they used the same value for ‘k‘ everywhere, which means that if you have two signatures, both with the same k, then they will both have the same R value, and it means that you can calculate k using two S signatures of two files with hashes z and z’ and signatures S and S’ respectively :

S – S’ = k^-1 (z + dA*R) – k^-1 (z’ + da*R) = k^-1 (z + da*R – z’ -dA*R) = k^-1 (z – z’)

So : k = (z – z’) / (S – S’)

Once you know k, then the equation  for S because one equation with one unknown and is then easily resolved for dA :

dA = (S*k – z) / R

Once you know the private key dA, you can now sign your files and the PS3 will recognize it as an authentic file signed by Sony. This is why it’s important to make sure that the random number used for generating the signature is actually “cryptographically random”. ¬†This is also the reason why it is impossible to have a custom firmware above 3.56, simply because since the 3.56 version, Sony have fixed their ECDSA algorithm implementation and used new keys for which it is impossible to find the private key.. if there was a way to find that key, then the security of every computer, website, system may be compromised since a lot of systems are relying on ECDSA for their security, and it is impossible to crack.

Finally! I hope this makes the whole algorithm clearer to many of you.. I know that this is still very complicated and hard to understand. I usually try to make things easy to understand for non technical people, but this algorithm is too complex to be able to explain in any simpler terms. After all that’s why I prefer to call it the MFET algorithm (Mathematics For Extra Terrestrials) ūüôā

But if you are a developer or a mathematician or someone interested in learning about this because you want to help or simple gain knowledge, then I’m sure that this contains enough information for you to get started or to at least understand the concept behind this unknown beast called “ECDSA”.

That being said, I’d like to thank a few people who helped me understand all of this, one particularly who wishes to remain anonymous, as well as the many wikipedia pages I linked to throughout this article, and Avi Kak thanks to his paper explaining the mathematics behind ECDSA, and from which I have taken those graph images aboves.

P.s: In this article, I used ’20 bytes’ in my text to talk about the ECDSA signature because that’s what is usually used as it matches the SHA1 hash size of 20 bytes and that’s what the PS3 security uses, but the algorithm itself can be used with any size of numbers. There may be other inaccuracies in this article, but like I said, I’m not an expert, I just barely learned all of this in the past week.

Posted in Development, PS3 | Tagged , , , , , , | 158 Comments

Status update on the PS3 4.0 HEN

Here’s a “quick” status update on the 4.00 HEN (Homebrew ENabler) for PS3.

Following my clarifications from almost 2 months ago here, there has been a lot of progress. We have not been slacking off, we’re a group of about 10 developers working together for the last 2 months, for sometimes 15 hours everyday in order to bring back homebrew support to the latest version of the PS3.

There are three major parts to the HEN, first, getting the packages to install on the PS3, that part is done, completed, tested, debugged, etc.. the second part is to get the apps to run, that one still has major issues… the last part is something I will not discuss for now (it’s a surprise) but it’s about 60% to 70% done (and it has nothing to do with peek&poke and has nothing to do with backup managers or anything like that. This is and will stay a piracy-free solution for the PS3).

Now, running apps is the biggest challenge that we’ve been working on for the past 2 months. As some of you know, if you’ve been following me on Twitter, we originally had hoped for Mathieulh to give us the “npdrm hash algorithm” that was necessary to run the apps, but he was reluctant, he kept doing his usual whore so people would kiss his feet (or something else) so he’d feel good about himself. But in the end, he said that he refuses to give us the needed “npdrm hash algorithm” to make it work… So what I initially thought would be “this will be released next week” ended up taking a lot more time than expected, and we’re still nowhere near ready to make it work.

Mathieulh kept tossing his usual “riddles” which he thinks are “very helpful for those who have a brain”, and which pisses off anyone who actually does… so he told us that the solution to all our problems was to look in appldr of the 3.56 firmware.. and that it was something lv1 was sending appldr which made the “hash check” verified or not… so we spent one month and a lot of sweat and after killing a few of our brain cells out of exhaustion, we finally concluded that it was all bullshit. After one month of reading assembly code and checking and double-checking our results, we finally were able to confirm that that hash algorithm was NOT in the 3.56 firmware like he told us (at all).

He said ¬†that it was an AES OMAC hash, but after tracking all the uses of the OMAC functions in appldr, we found that it was not used for the “hash”… ¬†he then said “oh, I meant HMAC“, so we do that again and again come up with the same conclusion, then we’re sure it’s not in appldr, and then he says “ah no, it’s in lv1“.. have a look for yourself to what he decided to write :¬†http://www.ps3devwiki.com/index.php?title=Talk:KaKaRoTo_Kind_of_%C2%B4Jailbreak%C2%B4

That happened after the huge twitter fight I had with him for being his usual arrogant ass and claiming that he “shared” something (For your information, the code that he shared was not his own, I have proof of that too (can’t show you the proof because even if I don’t respect him, I gave him my word to not share what he gave me, and I respect my word) since he forgot to remove the name of the original developer from one of the files… also it was completely useless and was not used at all, just made me waste a day reading the crappy undocumented code. So why is he still trying to force his “advice” through these riddles even after we had that fight? Well to sabotage us and make us lose all those months of hard work!

So anyways, we had all accepted that Mathieulh was full of shit (we knew before, but we gave him the benefit of the doubt) and decided to continue working without considering any of his useless riddles. So we then tried to exploit/decrypt the 3.60+ firmware in order to get the algorithm from there.

Now, a few more weeks later, we finally have succeeded in fully understanding that missing piece from the “npdrm hash algorithm”, ¬†and here it is for everyone’s pleasure with some prerequisite explanation :

A game on the PS3 is an executable file in a format called a “SELF“file (kind of like .exe on windows), those “self” files are cryptographically signed and encrypted.. For PSN games (games that do not run from a bluray disc), they need to have an additional security layer called “NPDRM”. So a “npdrm self” is basically an executable that is encrypted and signed, then re-encrypetd again with some additional information. On 3.55 and lower, we were able to encrypt and sign our own self files so they would look like original (made by sony) “npdrm self” files, and the PS3 would run them without problem. However, it wasn’t really like an original file.. a real NPDRM self file had some additional information that the PS3 simply ignored, it did not check for that information, so we could put anything in it, and it worked. Since the 3.60 version, the PS3 now also validates this additional information, so it can now differentiate between NPDRM self files created by sony and the ones that we create ourselves for homebrew. That’s the “npdrm hash algorithm” that we have been trying to figure out, because once we can duplicate that information in the proper manner, then the PS3 will again think that those files are authentic and will let us play them.

Another important point to explain, I said a few times that the files are “signed”.. this means that there is an “ECDSA signature” in the file which the PS3 can verify. The ECDSA signature is something that allows the PS3 to verify if the file has been modified or not.. it is easy to validate the signature, but impossible to create one without having access to the “private keys” (think of it like a real signature, you can see your dad’s signature and recognize it, but you can’t sign it exactly like him, and you can recognize if your brother tried to forge his signature). So how were we able to sign the self files that were properly authenticated on 3.55? That’s because this “ECDSA signature” is just a very complicated mathematical equation (my head still hurts trying to fully understand it, but I might blog about it in the future and try to explain it in simple terms if people are interested you can learn about it here), and one very important part of this mathematical equation is that you need to use a random number to generate the signature, but Sony had failed and used the same number every time.. by doing that, it was easy to just find the private key (which allows us to forge perfectly the signature) by doing some mathematical equation on it. So to summarize, a “signed file” is a file which is digitally signed with an “ECDSA signature” that cannot be forged, unless you have the “private key” for it, which is impossible to obtain usually, but we were able to obtain it because Sony failed in implementing it properly.

Now, back on topic.. so what is this missing “npdrm hash algorithm” that we need? well it turns out that the “npdrm self” has a second signature, so it’s a “encrypted and signed self file” with an additional layer of security (the NPDRM layer) which re-encrypts it and re-signs it again. That second signature was not verified in 3.55 and is now verified since the 3.60 version of the PS3 firmware.

One important thing to note is that Sony did NOT make the same mistake with this signature, they always used a random number, so it it technically impossible to figure out the private key for it. To be more exact, this is the exact same case as the .pkg packages you install on the PS3, you need to patch the firmware (making it cfw) so that those .pkg files can be installed, and that’s because the .pkg files are signed with an ECDSA signature for which no one was able to get the private key. That’s why we call them “pseudo-retail packages” or “unsigned packages”.

The signature on the NPDRM self file uses the exact same ECDSA curve and the same key as the one used in PS3 .pkg files, so no one has (or could have) the private key for it. What this means is that, even though we finally figured out the missing piece and we now know how the NPDRM self is built, we simply cannot duplicate it.

The reason we wasted 2 months on this is because Mathieulh lied by saying that he can do it.. remember when the 4.0 was out and I said “I can confirm that my method still works” then he also confirmed that his “npdrm hash algorithm” still works too? well he didn’t do anything to confirm, he just lied about it because there is no way that he could have verified it because he doesn’t have the private key.

I said I will provide proof of the lies that Mathieulh gave us, so here they are : he said it’s in 3.56, that was a lie, he said it’s an AES OMAC, that was a lie, ¬†he said it’s an HMAC, that was a lie, he said it’s in appldr, that was a lie, he said it’s in lv1, that was a lie, he said that he can do it, that was a lie, he said that “it takes one hour to figure it out if you have a brain”, that was a lie, he said that he verified it to work on 4.0, that was a lie, he said that he had the algorithm/keys, that was a lie, he said that once we know the algorithm used, we can reproduce it, that was a lie, he kept referring to it as “the hash”, that was wrong. The proof ? It’s an ECDSA signature, it’s not a hash (two very different terms for different things), it was verified by vsh.self, it was not in lv2, or lv1, or appldr, and the private key is unaccessible, so there is no way he could build his own npdrm self files. Now you know the real reason why he refused to “share” what he had.. it’s because he didn’t have it…

So why do all this? was it because his arrogance didn’t allow him to admit not knowing something? or was it because he wanted to make us lose all this time? To me, it looks like pure sabotage, it was misleading information to steer us away from the real part of the code that holds the solution…. That is of course, if we are kind enough to assume that he knew what/where it was in the first place.¬† In the end, he wasn’t smart enough to only lie about things that we could not verify.. now we know (we always knew, but now we have proof to back it) that he’s a liar, and I do not think that anyone will believe his lies anymore.

 

Enough talking about liars and drama queens, back to the 4.0 HEN solution… so what next? well, we now know that we can’t sign the file, so we can’t run our apps on 3.60+ (it can work on 3.56 though). What we will do is look for a different way, a completely new exploit that would allow the files we install to actual run on the PS3. We will also be looking for possible “signature collisions” and for that we will need the help of the community, hopefully there is a collision (same random number used twice) which will allow us to calculate the private key, and if that happens, then we can move forward with a release.

When will the “jailbreak” be released? If I knew, I’d tell you, but I don’t know.. I would have said in last november, then december, then before christmas, then before new year, etc… but as you can see, it’s impossible to predict what we will find.. we might get lucky and have it ready in a couple of days, or we may not and it will not be ready for another couple of months.. so all you need to do is : BE PATIENT (and please stop asking me about an estimated release date)!

I would like to thank the team who helped on this task for all this time and who never got discouraged, and I’d like to thank an anonymous contributor who recently joined us and who was instrumental in figuring it all out. We all believe that freedom starts with knowledge, and that knowledge should be open and available to all, that is why we are sharing this information with the world. We got the confirmation (by finding the public key used and verifying the signatures) yesterday and since sharing this information will not help Sony in any way to block our efforts in a future release, we have decided to share it with you. ¬†We believe in transparency, we believe in openness, we believe in a free world, and we want you to be part of it.

If you want to know more about this ECDSA signature algorithm, I tried to explain it in a blog post here, also, you can read this interesting paper that explains it in detail, and you can also watch Team Fail0verflow’s CCC presentation that first explained Sony’s mistake in their implementation, which made custom firmwares possible.

 

Thanks for reading,

KaKaRoTo

 

Posted in Development, PS3 | Tagged , , , , , , , | 297 Comments

GstFilters library released!

After the various blog posts about it, and the talk I gave at the GStreamer Conference, there was a lot of interest in the GstFilters library that I’ve been working on. The original plan was for it to get merged into gst-plugins-base, however, it seems like that’s not going to happen. The GStreamer developers would prefer seeing some of its features integrated into the core, but they don’t want the library itself. So I have finally decided to release it as a standalone package so everyone interested can already start using it.

As features from GstFilters will slowly get merged into the core of GStreamer, I will adapt the library to make use of these new features, reducing its internal code. However I believe it is still very useful to have Gstfilters as it’s a very simple library for those who are not familiar with GStreamer. Also the concept of the ‘filters’ is very different from the GstElements because an element can only be added once in a pipeline but filters can be added any number of times in a pipeline (a GstFilter doesn’t represent an actual element, it’s more like a helper function for “create and link these elements for me”). Also the points I’ve made about the steep learning curve and the robustness checks will still be valid even after the Gst core makes dynamic pipeline modifications easier.

GstFilters are now released and will be hosted on freedesktop.org under Farstream’s project. While Farstream users will be the most interested in this library and it is very useful for VoIP/Farstream users, it can also be used for non VoIP applications.

On a similar note, the Farsight-Utils library and API that I presented at the GStreamer Conference has been modified to make it even simpler. The library has been renamed into Farstream-IO since it basically takes care of all the Input/Output to the Farstream conference. The new API is based on a single object now, a FsIoPipeline that you create (which is a subclass of GstPipeline) and to which you register the FsConference/FsSession/FsStream. All the methods from the previous Farsight-Utils classes (FsuConference, FsuSession and FsuStream) will stay the same but will be merged into this single FsIoPipeline class, making everything easier and you’d only need to keep track of a single object.

The FsIo API will be merged into Farstream and released for the next version.

Here is the link to the new GstFilters page : http://www.freedesktop.org/wiki/Software/GstFilters 

And you can get the release tarball from here : http://freedesktop.org/software/farstream/releases/gstfilters/

And browse its documentation here : http://www.freedesktop.org/software/farstream/apidoc/gstfilters/ 

 

Posted in Development, Farsight | Tagged , , , , , , , , , , , | 3 Comments

ExpoLibre 2011 in Talca/Chile

Hi all,

About 2 weeks ago, I was in Talca, Chile for the ExpoLibre 2011 conference. It was really awesome, I had one of the best experiences as a speaker!

One of the particularities of that conference, is that it’s organized by the university and its target audience is students, teachers and enthusiasts in open source. The majority of the attendees were not open source developers, but they were people who wanted to learn more about it.
For that reason, this was my very first “motivational talk” rather than my usual technical talks that I’ve given in the past, and I loved it!

Another interesting point was that the audience was mostly speaking Spanish, and not everyone understood English, so I had my colleagues (Reynaldo Verdejo and Thibault Saunier) there to translate what I was saying. That created a very pleasant experience as I had time to relax between each slide while they were translating, and it also made the talk more casual and interactive. I wasn’t nervous for the first time, and it felt great! ūüôā

After the talk, I received some very interesting questions and I thoroughly enjoyed answered everyone of them. I saw a lot of people who were interested and I felt like I connected with everyone and I was able to touch them with my ideas. If I was able to change at least one attendee’s perception of open source, and hopefully get them involved in various FLOSS projects, then my mission is a success!

Today, the organizers of the ExpoLibre conference sent me the video recording of my talk, and I’ve shared it on youtube so everyone can listen to what I had stay. I hope everyone enjoys it as much as I enjoyed doing it.

On a final note, I’d like to say that Chile is a beautiful country. I stayed there for almost two weeks, and even though travel from/to Canada is a pain, it was totally worth it! I can’t wait for the next opportunity for me to go there.

Update : Some people complained about the rhythm being broken because of the translation to spanish,  so I asked here for anyone who wants to contribute, to edit the video and crop the non-english sections, so english-only speaking people can view the talk in one constant rhythm/flow without the interruptions by the translators.

Patrick Donnelly, one of the people who saw the video (and my request for an edit) did it and commented below  with a link to an english-only version of my talk (the intro and questions part were left untouched at my request). Here it is for those who need it :

 

And here is the original, unedited version of my talk I gave, enjoy it!

Ps: The video I tried to show to the audience (around 6:30) which did not work, was this one : http://www.youtube.com/watch?v=20ClL3mL8Gc

And here are the slides used during the talk, in PDF format : http://people.collabora.co.uk/~kakaroto/expolibre-2011.pdf 

 

KaKaRoTo

 

Posted in Development | Tagged , , , , , , , , | 43 Comments

Clarifications about 3.73 (and 4.0) “jailbreak”

Update:
I tested the jailbreak on the latest firmware 4.0 since it was released and I can confirm that it still works.

Hi all,

I’ve been flooded with questions on twitter and I’ve read many posts on news sites and ¬†I’ve seen some stuff being said on IRC and I thought I needed to clarify a few things…

First of all, I didn’t expect to see my tweet front paged on all ps3 hacking news sites.. although I should have expected it.. but anyways, the “jailbreak” is not ready to be used, at all. I only tweeted that because I was excited having it working and I wanted to share my excitement with everyone. But this is a bit equivalent to the day I released that create_cfw.sh script that created the very first CFW/MFW but it still took a couple of months before a real, easy, multiplatform and fully fledged solution was released : PS3MFW.

We are currently at the same state, I have the proof of concept, it works, but a solution that anyone can use where they just click a button and their PS3 gets jailbroken is still far from ready.

I’ve seen people say (and even write it in their front page news) that I’ll release it in two weeks after I come back from vacation. That is not true and I never said that. What I said was that for the next 2 weeks, the project is on hold until I get back.. but when I get back, then I will continue working on it, and it will then take some more time before it’s ready and released.

Some asked if it’s based on what gitbrew was doing/suggesting or if I used someone else’s exploit or work. No, this solution is my own idea and 100% my own implementation. However, the actual solution for the full jailbreak involves some components on which I will not work, and I expect/hope that someone else will provide the solution for that.

Some speculated it might be what I spoke about back in March which I later said I wasn’t pursuing by lack of motivation.. and yes, you are right. The same hack I had in March is still valid today, I told a few people about it (rms, Mathieulh, an0nym0us, and a couple more), but no one was interested in pursuing it further and actually exploiting that flaw (mainly because it requires a huge amount of work to get a proof of concept working). 10 days ago (I started on the 11th), I got bored and decided to start poking at it again, and yesterday (a lot faster than I thought it would take), I got my first pkg installed on 3.73 firmware.

On twitter, I said “do not update if you are on 3.55”, I said that in response to someone who said he would update.¬†¬†Because of that, people speculated that you need to be on 3.55 first, and then install something before doing the upgrade. No, that’s not it, that would be useless. The purpose of my solution is to jailbreak a ps3 that is already on 3.73 firmware and which had never been jailbroken before. I told people not to update because, first of all, it’s not yet ready, and second of all, the 3.55 firmware gives you a lot more possibilities than what can be achieved on 3.73.

So what is this jailbreak? I won’t say because I don’t want Sony to block it in a firmware update (and yes, they potentially could) before it’s even released (and yes, I will release it when it’s ready). But I will explain this to you : in order to run your homebrew apps, you need two things. First, to be able to install them on the ps3, and second to be able to run it once installed. I did only one of these two things.

Some may say it’s not a real jailbreak, but the way I see it, there are three ‘jails’ on the ps3, I broke the first one which prevents you from installing anything, so now you can install your .pkg, great, but it won’t run, that’s the second jail. The third jail is being able to modify the firmware (peek&poke).

The second jail (running apps) is something that can be done, but it’s not my area of expertise (npdrm algo), so I will not be working on that. I am waiting for someone else to achieve it (some have succeeded but do not wish to release it, at least not for now) then I will release.

The third jail (modifying the firmware) is not possible with my method, this means that you will ¬†not have a “CFW”, you will run your homebrew applications and games on an official firmware. This also means that without peek&poke support, none of the backup managers will work. So, again, my solution is piracy-free, and as always, I do not plan on working on a way to enable piracy (or even legal backups).

Overall, the purpose will be to allow people who are on 3.73 firmware to enjoy the homebrew games that were released, to play a bit with Eskiss, and to use Showtime for playing their movies. This should be more than enough for everyone.

Finally, I will conclude by replying to another question I received : Do you accept donations? The answer is yes. I do accept donations but I do not seek them out. I will include a donate button to the bottom of this post, so if anyone wishes to donate, they can do so, however, I want to make it clear that whether or not you donate does not and will not affect in any way, the release, or the progress of the work I’m doing. If you donate, you would do it as a sign of appreciation of my efforts, and not in exchange of any favors or anything crazy like that.

That’s about it I think… If you have any more questions, please refrain from asking them, I get enough as it is already.. I also said everything I needed to say and I don’t want to give any more information than that (for now).

KaKaRoTo

Posted in PS3 | Tagged , , , , , , , | 435 Comments

GStreamer Conference 2011 videos

The videos of the presentations given at the GStreamer Conference from last month in Prague are finally available online! So if you missed the conference, you can still catch all the interesting talks on video. Thanks to the great work of Ubicast who used a GStreamer-based system for capturing the videos and slides, and it looks awesome!

I gave a presentation in which I introduced two libraries : GstFilters and FsUtils.

Both libraries are  convenience libraries that sit on top of Gstreamer and Farstream respectively, and they should make your lives much easier. I discussed them a bit before in a blog post, but now you can see the full talk with all the details and explanation.

Here’s a link to the conference talks : http://gstconf.ubicast.tv/channels/#conferences2011¬†

And a link to my presentation about GstFilters and FsUtils : http://gstconf.ubicast.tv/videos/gstreamer-and-farsight/ 

 

Posted in Development, Farsight | Tagged , , , , , , , , , , | 5 Comments

Eskiss for PS3 with PS Move support

Hi all,

I’m releasing Eskiss with Move support and I think the instructions on how ¬†to use it require a bit more than what twitter allows (from my usual small updates).

You can download here the Eskiss package for PS3 3.55, and here the package for PS3 3.41.

The instructions are simple, you can still play with a normal mouse if you want, or use the controller to emulate the mouse, just like before. But, if you have a PS Eye camera plugged in, then it will also be ready to handle the Move.

If it detects a move controller, the ball on the controller will be white, at that point, you must press the Action button while pointing the controller to the camera (there’s no image feedback on the screen, so just point and press the action button). This will calibrate the controller and the ball will change color. At this point, moving the controller will also move the cursor on screen.

You can press the Action button at any time to recalibrate the controller (useful if the tracking stops working correctly, or camera falls off), and you can press the Start button at anytime to center the cursor on screen. Pressing the T button trigger will emulate a click.

You have the choice between two tracking modes, the first one (the one selected by default) is the 3D coordinate system, which means the cursor appears on screen with 1 to 1 precision (kind of) with where the controller is located in the room, so you have to move the whole controller to move the cursor (and even maybe stretch your arms to get to the corners), the second tracking mode is using the internal gyroscope of the controller, in other words, you can move the cursor just by pointing or rotating the controller without moving the whole controller in 3D space.

You can switch from one tracking mode to another at any time by pressing the Select button. Try them both and see which one you like best.

P.s: ¬†When you press the Action button to calibrate, the ball will change colors a few times, you must not move the controller while it’s doing that, do not move until it becomes a solid, stable color. If the ball becomes white again, it means you moved and the calibration failed… in that case, try again.

P.p.s: In this release, I have also fixed the crash that you might have had in the previous version, so the game should be a lot more stable. While it still might crash, it is now very rare and shouldn’t break the gameplay like it did before.

And here’s a video demo of the game running with the Move controller, courtesy of fungos :

Enjoy,

KaKaRoTo

Posted in Development, PS3 | Tagged , , , , , , | 9 Comments

Network emulator tool for Linux

I have finally decided to blog about my netem tool that I wrote a couple of months ago.
First, the introductions, netem is a kernel component for controlling QoS, rate control and various network properties that allows you to emulate a network by modifying the kernel’s IP stack’s queue disciplines. You can read more about it here : http://www.linuxfoundation.org/collaborate/workgroups/networking/netem

The issue I had with the netem queue was that it was hard/complicated to use and required a fair bit of reading and understanding of how the Linux IP stack worked in order to even use it properly. I needed an easy to use tool in order to test multiple network properties quickly. I looked around for a tool that would help me with that and only found phpnetemgui which is a very old piece of code, not even compatible with the latest php versions and which requires you to run a server on localhost and give sudo access to the web server… I didn’t like that, so I wrote my own tool for easy netem configuration (thanks to the phpnetemgui code, it was helpful in providing some of the commands).

You can find my netem tool here : http://cgit.collabora.com/git/user/kakaroto/netem.git/

The README has all the information you need in order to use it, so make sure you read it, but let me summarize a bit how it works.
Netem uses a CSV file in which you can set multiple rules, each with its own set of properties (10% packet loss, 5% duplicated packets, 100ms delay with 25ms of jitter, limit bandwidth to 256Kbps, etc..). Each rule has a name and you can have multiple rules with the same name (limit bandwidth to 256Kbps for IP 1.2.3.4, and 512Kbps to IP 1.2.3.5). All these sub-rules with the same name will be considered as being a single rule. You can run netem on an interface, giving it the CSV filename and the name of the rule that you want to activate and it will output all the commands you need to execute in order to emulate the network as specified in the rules from the CSV file.

To actually run the network emulation, just pipe the output to ‘sh’, for example : ./netem eth0 my_rules.csv 256kbps | sudo sh

The reason I did this was to help my colleague Olivier Crete who was working on TFRC (Tcp-Friendly Rate Control) for RTP in Farsight. He needed to be able to emulate various network configurations, change the bandwidth limitations, introduce packet drop, etc.. and see how TFRC would react to make sure that the video/audio stream’s quality stays acceptable and the bitrate calculation adapts correctly to changing network conditions. I’ve also been recently working on HLS (HTTP Live Streaming) support in GStreamer and I’ve used the tool to make sure that the HLS stream correctly adapts to the network bandwidth and switches the bitrate/resolution correctly. This tool has been a great help in doing all these tests, so it’s time now to share it with whoever it might interest.

I’ll conclude with these example outputs for three different rules (taken from the provided test.csv in git) :

  • Limit inbound and outbound bandwidth to 1024Kbps (2 sub-rules)

kakaroto@kakaroto:~/coding/netem$ ./netem wlan0 test.csv 1024kbps
modprobe ifb
ip link set dev ifb0 up
tc qdisc del dev wlan0 ingress
tc qdisc add dev wlan0 ingress
tc filter add dev wlan0 parent ffff: protocol ip u32 match u32 0 0 flowid 1:1 action mirred egress redirect dev ifb0
tc qdisc del dev ifb0 root
tc qdisc add dev ifb0 root handle 1: prio bands 10
tc qdisc del dev wlan0 root
tc qdisc add dev wlan0 root handle 1: prio bands 10
tc qdisc add dev ifb0 parent 1:1 handle 10: htb default 1
tc class add dev ifb0 parent 10: classid 0:1 htb rate 1024kbit ceil 1024kbit burst 0 cburst 0
tc qdisc add dev wlan0 parent 1:1 handle 10: htb default 1
tc class add dev wlan0 parent 10: classid 0:1 htb rate 1024kbit ceil 1024kbit burst 0 cburst 0
tc filter add dev wlan0 protocol ip parent 1:0 prio 1 u32 match ip src 0.0.0.0/0 match ip dst 0.0.0.0/0 flowid 10:1
tc filter add dev ifb0 protocol ip parent 1:0 prio 1 u32 match ip src 0.0.0.0/0 match ip dst 0.0.0.0/0 flowid 10:1

  • A rule to add 100ms of delay with 25ms of jitter using a normal distribution with 25% of correlation

kakaroto@kakaroto:~/coding/netem$ ./netem wlan0 test.csv delay
tc qdisc del dev wlan0 root
tc qdisc add dev wlan0 root handle 1: prio bands 10
tc qdisc add dev wlan0 parent 1:1 handle 10: netem delay 100ms 25ms 25% distribution normal
tc filter add dev wlan0 protocol ip parent 1:0 prio 1 u32 match ip src 0.0.0.0/0 match ip dst 0.0.0.0/0 flowid 10:1

  • A rule that emulates various packet loss, delay, duplication, packet reordering, rate control, for both inbound and outbound connection with IP and port matching (3 sub-rules)

kakaroto@kakaroto:~/coding/netem$ ./netem wlan0 test.csv test1
modprobe ifb
ip link set dev ifb0 up
tc qdisc del dev wlan0 ingress
tc qdisc add dev wlan0 ingress
tc filter add dev wlan0 parent ffff: protocol ip u32 match u32 0 0 flowid 1:1 action mirred egress redirect dev ifb0
tc qdisc del dev ifb0 root
tc qdisc add dev ifb0 root handle 1: prio bands 10
tc qdisc del dev wlan0 root
tc qdisc add dev wlan0 root handle 1: prio bands 10
tc qdisc add dev wlan0 parent 1:1 handle 10: htb default 1
tc class add dev wlan0 parent 10: classid 0:1 htb rate 256kbit ceil 256kbit burst 0 cburst 0
tc qdisc add dev wlan0 parent 10:1 handle 11: netem loss 0.5% 25% duplicate 5% delay 100ms 50ms 25% distribution pareto reorder 1% limit 1000
tc qdisc add dev wlan0 parent 1:2 handle 20: netem loss 0.5% 50% limit 1000
tc qdisc add dev ifb0 parent 1:1 handle 10: netem loss 5% reorder 5% limit 1000
tc filter add dev wlan0 protocol ip parent 1:0 prio 1 u32 match ip dst 1.2.3.4/32 match ip dport 1234 0xffff flowid 10:1
tc filter add dev wlan0 protocol ip parent 1:0 prio 2 u32 match ip sport 4321 0xffff flowid 10:1
tc qdisc add dev wlan0 parent 1:3 handle 30: pfifo
tc filter add dev wlan0 protocol ip parent 1:0 prio 3 u32 match ip src 0.0.0.0/0 match ip dst 0.0.0.0/0 flowid 30:3
tc filter add dev ifb0 protocol ip parent 1:0 prio 1 u32 match ip src 1.2.3.4/32 match ip sport 1234 0xffff flowid 10:1
tc qdisc add dev ifb0 parent 1:2 handle 20: pfifo
tc filter add dev ifb0 protocol ip parent 1:0 prio 2 u32 match ip src 0.0.0.0/0 match ip dst 0.0.0.0/0 flowid 20:2

Posted in Development, Farsight | Tagged , , , , , , , , , , , | 5 Comments

Programming, Open Source, Hacking and Greedy Corporations

I’m a programmer, a developer, a hacker. I’m mostly involved with the Open Source community and I try to promote open source development as much as I can. Unfortunately, most of the time when I tell someone that I’m a “developer”, they don’t understand the concept, and when I start talking about open source, they understand me even less.

The world is full of people with different background, with deferent references and we don’t always understand each other. As most of you who read my blog would probably know, I’m involved in the PS3 hacking scene, and I see a lot of misinformed people, and I read a lot of things that don’t make any sense to me. This is because most people don’t understand the world that we (developers/hackers) come from and things tend to be misinterpreted.

This message is for everybody, it’s intent is to open a window into our world so people can understand us better. I don’t have the audacity to explain everything about programming in this text, but I will try to formulate in terms easy to understand the general idea behind it. While most of this post will be generic and intended to anyone, there will be a paragraph that will address some of the recent issues surrounding the PS3 and Sony. This post will probably be very long and I’m sorry, I don’t think I have a shorter version for those who get bored easily.

1 – Programming

If you’re familiar with or understand programming, you may skip this section, as it might be a bit boring, otherwise, read on, it should explain what you need to know to understand the rest of this blog post.

What is a “program”? Let’s put it simply : “It’s a set of instructions that produce a result”. A program is what you run on your computer, phone, gaming console, or even your alarm clock. It tells the computer to do something, for example “if the user pressed the ‘up’ button, advance the minutes by one. If the time reaches this specific value, sound the alarm” (alarm clock programming) or “Draw a red circle. If the user clicks inside the circle, change the color to blue”. With many simple instructions, you end up with a complex program that can achieve a multitude of tasks, like for example Microsoft Office, or Skype. But the basic definition is that a program is “a set of instruction that produce a result”.

Now what is a “source code”? This mystical thing you keep hearing about is nothing more than “a set of instructions that produce a result”.. sounds familiar? Basically, a “source code” is the text that the programmer writes in order to tell the computer the instructions it wants the program to achieve. The source code is in itself, the program, but it’s in a readable and understandable format : a text file using a language that the programmer understands. The computer however doesn’t understand the source code, it only understand mathematics, numbers. A program’s instructions are written with “numbers” that the computer understands, for example 1 might mean “copy this” and 2 might mean “write that” and 3 might mean “show this”, etc.. (very simplistic view, but you get the idea). So the difference between a program that you run and a source code is that they are both the exact same thing, but the program you run is made up of numbers representing instructions to the computer (this is what we call the “Assembly” language or “machine code”) while the source code is the same instructions written in a more readable format, text, using a language that is easy to understand.. so instead of “1 4 185 353 532” (machine code) you would see “if the user clicks on the circle, change the color to blue” (source code).

What is a “programming language”? The source code can be written in different languages, just like spoken language, we have english, french, italian, russian, etc.. in the programming world, there are multiple different languages to define the instructions for the computer. These programming languages differ in the vocabulary (commands/functions) and in grammar (syntax). Explanation more than that is not relevant to the current topic so I’ll leave it at that.

How do you get an application (a program) from source code? It’s simple, there is a program called a “compiler” which reads this source code (the text), understands it, and rewrites it into machine code (the numbers). When you download an application, you only get these ‘numbers’ that the computer understands because that’s all you need to run your application.

I’d like to remind people to not make the confusion, thinking that the source code is the recipe and the program is the final meal, you have to think of the programs themselves as being recipes, the ingredient is the electricity used and the result is whatever appears on your screen.

2 – Open Source

So.. what is this “open source” everyone keeps talking about? Well now that you know the basics about programming, let me put it simply : a program (all those numbers) is open source, when the source code used to generate the program is publicly available.

And here is the juicy part of this blog post. Remember when I said that a program is “a set of instructions that produce a result”? Well, here’s an absolutely superb analogy: A program is like a recipe. What is a recipe? Well, isn’t it a set of instructions that you must follow in order to produce something? This analogy comes from Richard Stallman in the documentary The code (this one, not the 2011 movie) and I think it’s absolutely brilliant.

You can listen to it in his own words here : https://www.youtube.com/watch?v=20ClL3mL8Gc

So now, with this analogy in mind (which I’ll keep referring to throughout this blog post), back to the question at hand. A closed source (or “proprietary”) program is like going to a restaurant where they serve this dish that you like, but when you ask the waiter/waitress what’s in it, they refuse to tell you the recipe for it. And open source is when you go to your friend’s house, you eat something that you like, and when you ask what’s in it, your friend tells you “oh, let me give you the recipe”.

Now imagine a world where no one could ever get a recipe for anything, you want to cook something, you have to relearn from scratch, experiment yourself with everything and see if the result is satisfactory, without having any references. Unfortunately, you’ll end up mixing two things together that you never should have done, and you’d be thinking how sad it is that every person in the world has to reinvent something that should be ‘common knowledge’. Thankfully, this isn’t the world we live in, and in the same way as you might enjoy cooking and exchanging recipes with your friends and family, we, programmers, enjoy sharing source code with each other, making our ‘recipes’ publicly available to everyone.

If you eat a delicious cheesecake at your friend’s house, and he/she gives you the recipe, then you try it, but then you realize it’s too sweet and you decide to decrease the amount of sugar, you have just “modified the code”, then you realize that adding a bit of lemon juice will make it better, and it does. You tell your friend about your changes, and he/she likes it and says “I’ve always wondered what it was missing”. You have just “contributed” to the program and now all your friends and family can enjoy this improved cheesecake (I love cheesecake by the way).

This is what Open Source is all about, it’s about sharing your recipes, anyone being able to improve on them and contribute his changes and slowly, thanks to the original recipe, new recipes will be born and people will enjoy more great products. It’s all thanks to this simple of idea of sharing. This applies to the programming world in the same way, we write programs, we share the source code, others can improve them (add features, fix bugs, add translations, make a better/easier user interface, etc..) and everyone benefits from it.

My journey into this wonderful world started more than 10 years ago, I was using a program that I liked but I wanted something that it didn’t do. Thankfully, it was open source, so I added the feature that I wanted, gave my changes back to the project, the other users loved it which made the program more popular and some new users decided to do the same thing and improve the program, and in the end (I’ll say it again) everyone benefits from it.

3 – Hacking

What is “hacking”? Again, let’s put it simply: hacking basically means “working around a problem”. In a broader definition, it could also be viewed as “modifying something to make it do a task it wasn’t intended to do”. I have headphones and one of the wires got cut.. so I taped it and it worked.. in my definition, that counts as “hacking” because I worked around the problem. The term “hacking” has been publicized as being ‘evil’ or a bad thing, but people confuse it too much with what it really means. I hack everyday and you probably do without knowing it. Back to the food/recipe analogy. Did you ever go to someone’s home and were served a meal, then you took the salt from the table and added some to your plate? You have just “worked around a problem” (not salty enough) and you just modified something (the meal) from its intended purpose (the ‘view/taste’ of the one who cooked it). In my definition, you “hacked” the meal to make it fit more to your taste.

This is the reality of things, when you modify something that you own to make it more to your taste (everyone has different tastes after all), you are “hacking” it. When you decide that 200g of sugar is better than 250g of sugar in your cupcake recipe, you are “hacking” the recipe. But in the terms of the computer world, the term has been used widely to describe pretty much anything we do, but mostly things we do in a hurry. My friend programmed his computer to play a sound (an alarm) when his girlfriend connects on MSN so it wakes him up, but he would say “I hacked it” because he did it in 5 minutes and didn’t spend months setting up a whole infrastructure behind this “wake me up when my girlfriend is online” system. Nowadays, the simple fact of “programming” is called “hacking”, it’s nothing illegal, it’s nothing harmful, but most of the time we say “I’m hacking” rather than “I’m programming” simply because the act of programming is all about finding solutions and working around problems. You should read the definition of the term as explained in wikipedia.

The problem is that there are those who use their talent for criminal behavior and when it’s related to anything “computer-y”, people decide to call it “hacking”. It’s like saying that “cooking” is evil and anyone who “cooks” is a criminal because someone, somewhere put a drop of poison in someone else’s food. Isn’t that ridiculous? I very often see people saying “death to the hackers” or “those hackers are criminals and should rot in jail forever” without knowing what they are talking about. It’s funny how people get emotional and suddenly they become judge, jury and executioner. To all these people, let me tell you something : The next time that you add some salt to your meal, watch your back because the FBI just might lock you up for your crime!

Now here’s another thing that we, programmers and hackers, often do, it’s called “reverse engineering”, it’s basically about understanding how something works without being told by the original maker. Whenever you try to understand how something works, you are ‘reverse engineering’ it. In the recipes analogy, this would mean that when you taste something and you start wondering if there’s garlic in it, or say “is that cinnamon?”, you are basically reverse engineering the meal by trying to recreate the recipe (or part of it) by looking at the final product.

Yes, that is what reverse engineering is, you receive a finished product and you try to understand how it was made. This is equivalent to going to a restaurant and trying to make the same dish that they served without them giving you the recipe. If you ever did that, then you definitely know what a reverse engineer is.

 

4  РThe Greedy Corporations

Now, ¬†this is the interesting part, the ‘greedy corporations’. I’m saying it like this because I didn’t want to say “Sony” because they are clearly not the only ones playing this game. Why are they greedy? because they want to have total control over you and your freedom, thus allowing them to generate more profit. I’ll go back to the recipe analogy: What Sony/Microsoft/Apple/etc.. are doing is basically the equivalent of LG selling you a kitchen appliance and saying you can only use it with their products! Imagine buying a kitchen stove that only allowed you to cook using ‘LG and Tefal” pans… or imagine a pan or a pot that only allowed you to cook food from some specific brands. No, you can’t buy the cheap, equivalent (and sometimes better) “no name” brand or buy your fresh vegetables at the market.. no, those vegetables have to have been processed by those giant corporations that put some sort of label on it allowing the pan to cook them. This is my analogy, it may sound stupid, but I believe this is what it is.

Did you ever wonder what “DRM” (Digital Rights Management) is? ¬†well to put it simply, it’s like having a microchip inside your Tefal pan, and it continuously detects what’s in it.. if you ever dare to put in the pan an ingredient (a tomato!) that wasn’t pre-selected and pre-accepted by Tefal, then the pan would automatically and instantly cool down and stop absording heat. Hell, it could even send a signal to the stove which will simply shut it down. That’s what DRM is.. and why is it there? Well, they would tell you that it’s “For your own good”, it’s because they want to deter people from stealing food from the supermarket. But what it really does is that it prevents you from using your fresh vegetables that you proudly grew yourself in your backyard, so that you have to buy their product. Even worse, DRM means that you can only use ‘pre made’ cookie dough in your oven.. if you get a better cookie recipe from your friend and try to make those cookies yourself, the oven will not turn on. And for those “super awesome, elite, we are the nice guys” oven brands that tell you “wow, you can bake your own cookies! Here is the recipe!”, you have to read the fine print, the recipe says 250g of flour, and it’s unfortunate, but the oven will not turn on if you made the mistake of puttin 255g of flour in your dough. And I’m not even talking about the LG microwave that will only heat meals that were cooked on an LG appliance, or the fridge that will not cool anything without the “Kraft” label on it…

The irony is that when you buy your pan, you’re buying it for 100$, because do you think that these greedy corporations will pay the fee for the DRM? no, YOU are! The pan should cost 20$, but they are charging you 100$ because you have to pay for that microchip in your pan, you don’t want it, but you are paying for it.. you have no choice! And if someone comes along and creates a new, DRM-free pan and wants to sell it, they’ll label him a “pirate” (ouh, that’s a scary word) and pay millions in propaganda and in advertisement (that conveniently appear at the bottom of your pan and on the front glass of your oven) to tell you how this DRM-free oven/pan is ‘evil’, will eat your babies at night and will kill your dog. The funny thing is, the first time you hear it, you’re thinking “wtf?”, then after hearing it 1000 times a day, you believe in it as being the absolute truth. You will eventually get used to verifying the “compatibility list” of your new oven before you buy it.. make sure that you can borrow plates from your neighbor because they are “authorized/licensed accessories” to the oven. You will get used to checking the label on your vegetables when you go to the supermarket to make sure that they are compatible with your pan, and you will get used to not buying a specific brand because your fridge’s manufacturer never made a deal with that brand so you can’t put it in your fridge…

I know what you’re thinking : “what the hell?”. Yes, what I just said sounds absolutely absurd, it sounds crazy and it doesn’t make any sense. After all, who would accept that? Who would even think of doing some crazy things like that? Well here’s the thing, the reason I love this analogy between programming and recipes is simply because not only is it quite accurate, but it’s also something that everyone can relate to. I think pretty much everyone knows how to cook, if even just an omelette. And if you don’t, you probably saw or know someone who can. If not, then at least cooking isn’t a concept that is so “obscure” that you can’t comprehend it. If the kitchen appliances tried to force all those restrictions, or if people tried to outlaw exchanging recipes, then pretty much 99.99% of the population will say “this is bullshit, we refuse!”. But in the computer world, this is exactly what is happening, only nobody cares because nobody can understand it… all this “computer-y” stuff is not something that interests most people, so they don’t try to understand it and they don’t care about it, and for those who do, well, unfortunately, they prefer to program rather than go on trial against all the corporations.

Here’s a real life example, here is a ‘hack’ that i’ve done a couple of days ago :

This is indeed a ‘hack’, I used two tools that weren’t made to be used together in order to work around a problem that I had at that time. There’s nothing wrong with it!¬†Both the whisk and the drill are mine, they are my property and I should be able to do what I want with them. However, if a similar situation was happening in the computer world, then I’d already be getting a lawsuit, because for some reason I don’t own the drill, I only paid to be “authorized to use it the way they allow it”. They would call me a “pirate” because I’m “killing the industry”, because by doing that hack, Black&Decker are losing money. They would be right, because since I did that hack, I didn’t have to spend another 100$ to buy an electric mixer. The funny thing is that I’d probably lose in court because there are no real laws to protect me as a consumer into using my tools any way I want, at least not in the programming world.

I read this last paragraph again and I’m thinking “I’m a lunatic” and I perfectly understand if you’re thinking the same thing. At least now we have something in common, we both think that the current situation in the programming world is completely crazy, and I’m glad you are able to see it.

5 – My angry rant

Yes, I’m angry! I am angry because I see the world evolving at an alarming rate but the laws (and people’s common sense) isn’t. I will dedicate this paragraph to rant about all the things that I recently saw and that got me angry. If you don’t want to see some angry dude raging, then skip it ūüôā

First of all, there are many people who are associating us, the jailbreakers, the programmers, the hackers, with what recently happened to the PSN data leak. Because they couldn’t play their games online for a few weeks, they decide to throw their anger at us, put us all in the same boat, and label us criminals. Every time we speak, I see comments saying “ah, these criminals are now trying to justify their crime”. But.. what crime? What crime did we do that you should label us criminals? Don’t throw words like that without understanding their meaning! Or at least, use your common sense before thinking that anything deemed ‘illegal’ is a ‘crime’! Do you know that in France, a woman must wear a dress, and that, by law, she’s a criminal if she wears pants/jeans? It’s an old law when only men wore pants and a women who did was considered a ‘transvestite’… this is a stupid example, but I’m using it to show you that common sense should overcome stupid laws.
If you think we’re criminals for jailbreaking the PS3, then how is it a crime to want to use your backyard-grown tomatoes to cook your meals? If it’s because of the PSN hack, then here’s another analogy for you: when you go to a restaurant and someone orders food, eats it and runs without paying the bill, how would you feel if the restaurant’s owner puts all the blame on you, you, who were sitting all the way to the other side of the restaurant, who didn’t even see or notice the thief, but you had the audacity of adding a bit of ketchup to your burger. As you know.. you “modified the vision of the chef” and that is a huge criminal offense and you should rot in jail you filthy criminal. No need to answer me, but just think about it.. how would you feel? (and yes, I believe that this analogy is very representative of the situation).

Now here’s another thing that makes us criminals: reverse engineering. We are ‘criminals’ because we reverse engineer products? Back to the recipe analogy: the next time you taste a meal and say or even think “humm, I think they put garlic in it”, then consider yourself a criminal and you should rot in jail.

If one million PS3 users (I’m being generous) told Sony that they don’t agree with them, that would still only be 1% or 2% of their user base, so they keep doing what they’re doing because 1 million people is an “insignificant number”. ¬†What happened last year when Sony removed OtherOS support from the PS3 is the equivalent of Frigidaire selling you a fridge then a couple of months later, tell you that “On the 1st of april, your freezer will stop working, we suggest you remove any food from the freezer and stop using it. You have a choice though, if you don’t want your freezer to automatically stop working, then empty the top 2 shelves of your fridge because those sections in the fridge will be at room temperature now. If you ever put something back into the top shelves of your fridge, then the freezer will be disabled permanently”… sure we have a choice, thank you for your generosity!!! The worst thing, the most heartbreaking thing is that going to Frigidaire’s website to complain about their unlawful practice, you find those thousands of people cheering and saying “who cares? it’s A FRIDGE, it’s not a freezer!! who uses the freezer anyway? just buy a dedicated freezer instead!” or “I wasn’t using the freezer, after all it does say “refrigirator” on the machine, so that freezer was a BONUS, be happy you got to use it for free all this time”, etc.. Let me ask you a question… if you accept that Sony removes OtherOS from your PS3, then you will have absolutely no problem in Frigidaire disabling your freezer right? even if you don’t use it, I might but who cares right? you’re not egotistical after all, if you don’t use it then no one in the world is? And again “DEATH TO THOSE DAMN HACKERS”.. how dare they put a cheesecake in the fridge when Frigidaire specifically said “no pastries”!! After all, they clearly wrote it in page 258 of their user manual!!!! After all, it’s Frigidaire’s fridge (no you didn’t buy it, you only ‘rented’ it for 2000$, it’s clearly written on page 531 of the manual!) and they have all the rights to it, they have all the rights to defend their interests… I mean, they never made any sort of deal with the bakeries!!! You know what this “deal” means? it means that the bakeries had to accept paying Frigidaire to allow their pastries in the freezer, so every time you buy something from them, you are paying 50% to the bakery and 50% to Frigidaire, and this allows you to put your cheesecake in the fridge and you’ve always been wondering why the prices doubled recently..

Anyways, you get the idea… but what pisses me off the most is how all these people think that their mission on earth is to defend Sony… like they say where I come from “is it your father’s company?”… seriously, why do you feel the need to go all over the internet, in every forum that you find and yell hate messages against ‘us’? why do you feel like you need to repeat Sony’s propaganda everywhere and why do you hope that we die and/or spend our life in jail? What do YOU gain from that? Why do you think that this multi-billion dollar company needs you to defend it? It’s like walking in the street at night and seeing a mob of 10+ huge guys beating an innocent child in an alley and you’re rooting for the mob… where is your common sense?

6 – Conclusion

I wrote this post because I wanted to make people understand our world a bit better. I know that some people might disagree with some of the things I said, but remember, this  is not meant to be an exhaustive explanation of how computers work but rather simply a glimpse into it, in terms that non-initiated people can, hopefully, understand.

I hope that I have achieved my goal: make a few people understand us and most importantly, make a few more people think about these issues. I know that I will continue to see misinformed posts everywhere, and nothing can change that, but to those who are willing to listen to others and accept differing views, then I’m glad I could help you with that (if I did). If you have questions or want to start a debate on something I said, feel free to comment.

And for your information, I am not saying that closed source is evil, I believe in freedom, and if you want to keep your code closed, then you are free to do so. I also do understand the need for closed source sometimes, in order to stay competitive for example, but I think that if everything was open source, then competition would become different. I simply believe that the world would be a better place if everything was always shared. Knowledge is for everyone, and I just can’t imagine where the world would be today if people shared all their ideas/code/recipes/etc.. with each other. It would certainly be a wonderful world. I find it truly pathetic to know that every company is recreating the same thing that others did before them.

Finally, I’d like to point people to the EFF, the Electronic Frontier Foundation. It’s a group that protects us and defends our digital rights every day. ¬†Right now, we are still under the mercy of the giant greedy corporations, but thanks to the EFF’s efforts, I hope that some day soon, we will be free to code the way we want, just like we are free to cook the way we want.

Thank you for reading!

KaKaRoTo

 

Update: After reading a few comments about this post, I thought I should clarify a few things.

First of all, this post isn’t about Sony or the PS3, which is why my title and fourth paragraph says “Greedy corporations”. While I do address the PS3 subject in my rant, it is only because it’s a subject that is dear to me and for which I have a lot to say. But what I outline is and should be considered generic and the main purpose remains to “open a window into our world” for those who are not computer savvy and who may not understand the issues at hand. I want people to understand that, from our point of view, the world is a crazy place, and you can draw parallels with many things, not just with the recent issues with Sony.

Also, like I’ve found myself saying a few times in the comments, there’s a saying that should govern us all : “One’s freedom stops where someone else’s freedom starts”. I believe that you are free to do whatever you want. As a consumer, you should be free to use your legally bought devices any way you wish (as long as you don’t infringe on other’s freedom, whether it is other’s freedom to gain money from their work or freedom of a fellow customer to enjoy their product (online cheating as an example)), but also, as a product manufacturer or a company, you are free to put the restrictions you want and you are entitled to use anything you feel is needed to protect your investment, but again, as long as it doesn’t infringe on other’s freedom.

I’ve had a few comments about DRM, but I never said that DRM is bad and this post isn’t at all about DRM. I have personally no issues with DRM as long as it’s reasonable but when you think that your own needs are more important than the needs of others, that’s where I see a problem. If I ever got an idea for something that could potentially make me rich, I would pursue it and I probably would try to protect my investment and intellectual property as much as I can, but there is a moral barrier that remains and I will never allow myself to be controlled by greed in such a way that I would sacrifice other’s freedom to further my goals.

In the same way, you are free to do whatever you want with your work, I have absolutely no problem with closed source applications, I simply prefer open source and I believe that the world would be a much better place and our civilization would be much more advanced if everything was open source.

One example of the above is the fact that advertisement exist as a sort of ‘payment’ for things you watch. When I watch a movie on public TV, I see ads and that’s what’s paying for the movies I’m watching “for free”, but then, why is it that when I buy a DVD, I am forced to watch ads before accessing its content? Didn’t I already pay for the DVD so why are you forcing me to watch ads? And even if you put ads in there, and it’s ok, then why can’t I skip them? If I watched the movie 10 times, do I still need to see the same ads? And why would I be forced to watch a trailer for a movie that I might have already bought (or which I already saw and hated)? Why is it that if a friend comes over and I want to show him a 30 second scene from a movie, do I need to wait 10 minutes until all your trailers finish just to show him that? This “you cannot skip the trailers in a DVD” is something unrelated to DRM but is still something caused by companies’ greed (get more money from each sale) which is infringing on my freedom of using the DVD I legally bought the way I want (in this case, watch it without having to suffer through all those trailers).

Finally, this post contains information, it contains knowledge, and my belief is that knowledge should be free and available to all. I am not trying to generate any page views (my poor server would hate me) and I don’t have any ads on my blog, so if anyone wants to publish this whole article somewhere else, where others could benefit from its content, then you are permitted and encouraged to do so. I’d be quite happy to see this published in its entirety on sites such as Arstechnica, Kotaku, Joystiq, the New York Times, or whatever other media that would reach more people than this humble blog.

Don’t forget, share, and everyone benefits from it ūüôā

Thank you (and congratulations :p) for reading!

 

Posted in Development | Tagged , , , , , , , , , , | 85 Comments

The Humble Homebrew Collection

Finally, after almost 2 months of hard work, I’m proud and happy to announce the release of the Homebrew game I’ve been working on : SGT Puzzles. It’s a collection of portable puzzle games for Windows, Mac, Linux, Android, PocketPC, Android, etc.. and I’ve ported it to the PS3 too!

The release of this homebrew game comes with the ¬†release of The Humble Homebrew Collection which is inspired by the Humble Indie Bundle Initiative (but not endorsed by it). The difference here is that you don’t have to pay anything in order to enjoy the games, they are free to download by anyone, but you are also able to donate any amount to the developer of the puzzle games (Simon Tatham) as well as the PS3 port developer (me!) and the EFF. You decide who to send the money to just like with the Humble Bundle. I’ve also linked to the game’s Windows, Mac and Android ports if you want them (they are already available in most Linux distributions).

The addition here and probably the most important part is a petition where yo get to sign and send a message to Sony asking for a legitimate way of having homebrew games on the PS3. Every signature will send an email to SCEE, SCEA, SCE Australia, SCE New Zealand and Kazuo Hirai, the CEO of Sony Computer Entertainment. ¬†This is done in the hopes that Sony will finally see the light, learn from the mistakes they’ve been doing these past few years, and finally give us a legitimate and officially supported way of developing homebrew applications for our PS3 Systems.

Sony would be stupid not to answer to that, considering that Apple complied, Microsoft complied and Google complied, and they are all generating huge revenues thanks to homebrewers, with zero investment from their part. I know that the Sony execs only understand when you talk about money, so I hope this is a good enough incentive for them. Clearly, they do not care about their customers, so I don’t think they’ll change anything only to do what is right.

The SGT Puzzles game includes 33 puzzles, which are excellent for the most part. My favorite is and always will be Pattern, as I’ve spent countless hours playing it. I’ve recently also discovered Rectangles and Net which are also very good (in higher difficulties). I suggest you give those puzzles a try. Above all, I hope everyone can enjoy these games.

This all started about 2 months ago when I found a copy of Pattern on my PC and started playing it again. I tweeted about it and asked if someone wanted to port it to the PS3. Clement Bouvet (@TeToNN) quickly made a proof of concept using cairo. That got me excited and I decided to help him. We ended up writing a PS3 application over Simon Tatham’s Portable Puzzle Collection which, I must say, is very well written and made porting it to the PS3 very easy. It took maybe a day or two and the first game was playable on the PS3. At that point, I discovered the Cairo Drawing API which I loved and and I decided to invest myself entirely in this. It took 3 more weeks of hard work to get the whole system working (choose your puzzle game, change difficulty (Select) and writing the whole menu system for the game). I’ve received various help, Surenix made the designs for the menu graphics and buttons, and BeGamer helped design the HHC website.

The game still lacks a few things, and I will continue to work on it and improve it so everyone can enjoy a quality homebrew game, that, I hope, will make the anti-homebrew purists jealous.

The funny thing is that since day one, the source code for this game was available on my github account, but no one noticed it. Only a few people who accidently ended up on my github page found it, but no news website author found it or reported on it. I’m glad, because it allowed me to make this happen the way I wanted it to and launch this HHC initiative when it became ready. I’d like to ask the various websites out there not to link directly to the games (even if you are allowed to) and instead link to humblehomebrew.com so people can sign the petition while downloading.

Most of the code is licensed under the MIT license. Parts of the code (the cairo menu system) is licensed under the LGPL license and I plan on extracting that into its own library for other developers to use in their applications.

The website took about 3 weeks to code. I learned two valuable lessons.. first, HTML coding is crap… secondly, it’s much more complicated than it looks. I hope people will appreciate this effort and I hope the Humble Homebrew Collection will make a difference.

In the future, I hope to enhance it by adding new homebrew games whenever I find something of quality, and keep the website and this whole initiative going for a long time, for as long as necessary.

 

So.. go ahead, download the games, sign the petition, maybe donate if you’re feeling generous, and most importantly, have fun!

Thank you!

 

Posted in Development, PS3 | Tagged , , , , , , , , , , , | 35 Comments