One of the principles of the Singularity is that the first intelligent machine is the last thing that humans will ever need to invent. Once a machine has the intelligence to improve itself, it will create a more intelligent version of itself that is beyond the ability of humans to create. Many fiction stories involve the danger that such an AI no longer under human control would pose to civilization. Dr. Stephen Omohundro's research into the mathematical and behavioral characteristics that these self improving AI's will exhibit will help us to avoid those bad scenarios and harness their nature for human good.

The main principle that will drive self improving AIs is economic rational theory. The "homo economicus", that hasn't actually done a good job of predicting human behavior, will fit these AIs much better. In order to make economically rational decisions, an AI would need a utility function (the goal it tries to achieve) and a subjective probability distribution (the set of beliefs about the world that it uses to evaluate how choices will affect the utility function). The way to make sure an AI would exhibit behavior consistent with human values is to carefully select the utility function. Much like corporations single-mindedly pursue profit, the AI will pursue maximization of its utility function, whether it be winning at chess, making money or bringing world peace.

Regardless of how the utility function or beliefs are written, there four main methods of maximizing utility that emerge from them. First, the AI will use resources more efficiently, leading to more utility per resource. Second, since dying or being shut down is the lowest possible utility, they will exhibit strong self preservation, both for their continued existence and the the preservation of the utility function. Third, they will seek to acquire more resources, since more resources will let them improve utility. Fourth, they will creatively seek new possibilities for increasing utility, especially for open ended goals. Further research into economics, mathematics, behavioral science, computer science, and other fields are needed before this can be achieved, but even more important than the research is including the values of all kinds of people. We need to find what's most valuable to us so we can build it into our systems.