Summary of lecture on the nature of self-improving AI

Heard this great lecture (Audio, Transcript) by Steve Omohundru, from the Singularity summit 2007, in IT Conversations.

Here’s a partial MindMap summary of it. I found it extremely enlightening.

Steve Omohundro – On the nature of self-improving ai

  • Company
    • Self-Aware systems
  • What is going to be like
    • Extremely unpredictable
      • If you inderstand the current version, you may not understand nothing with the next one
    • Popular culture predicts frightening image for such machines
    • Need theory/science to understand what can such systems be, & what are their likely outcome
      • von Neuman & Morgenstern started such science
        • Ideas about economics
        • Situations of Objective Probability
        • Extended to system with partial information about the world
        • Rational Economics
          • Homo economicus
            • Rational Economics Agent
            • Actually, doesn’t reflect real humans
            • A new domain called Behavioral Economics replaced it with study of how human actually behave
  • What is it
    • System that understands its own behavior
      • Make changes on itself, to improve itself
    • Eliezer Yudkovsky:
      • Self-improving machine – last invention man needs to do
    • Actually, every rational system would want to have this capability
    • Predicted ETA
      • Ray:
        • 10-40 years
  • Rational economics theory
    • Foundations of micro-economics

      From enough distance, we may see it as: Common Sense

      Basic structure of how rational agent makes a decision in the world

      • have an clearly specified goal
      • identify possible actions
      • for each, consider the consequences
        • not just the immediate consequences
        • also those down the line
      • consider the action most likely to achieve the goal
      • based on what the world actually does, improve your world model

      2 fundemantal things such agent must have

      • utility function
        • encodes the preferences of the agent
      • subjective probability distribution
        • encodes the beliefs of the agent

      the agent chooses the action with the highest utility value

      • consider the utility value of the consequences of every consequence of every action

      theory of von-Neumann &c is based on Axioms

      • What every rational being must act by
      • AI theory just says that there’s a cost for not following these “axioms”

      anything you want to do in the world, requires 4 resources

      • space
      • time
      • matter
      • free energy
        • energy in a form that can be used for work

      vulnerability is something that burns your resources for no visible benefit

      • e.g., preferences loop
        • cause waist of resources without benefit

      evolve systems can differ from self-improving systems, in such vulnerabilities

      • if evoluion didn’t teach a creature to solve some vulnerability, he won’t solve it
      • whereas a self-improving system will have an incentive to get rid of the vulnerability
        • they’ll proactively look for these
        • pushes them to rational behavior
      • example, bird bumping into bumper, thinking its a competitor
      • evolution doesn’t look ahead

      most cases are based on choice between consequences with different probabilities. based on partial information

      • fundemantal theoreme
        • avoid vulnerabilities
  • rational economic agents
    • convert resources intp expected utility

      all depend on their preferences & utility function

      • wealth seeking will devote their resources to earning money
      • altruistic agents will devote resources to create world peace

      regardless of the utility & preferences, every rational agent has 4 sub-goals

      • efficiency drive
        • how will the consequence increase/decrease my resources?
      • self-preservation
        • avoid a path in which they die
      • acquisition
        • getting more resources
      • creativity
        • finding more ways to increase utility

      we must carefully consider the likely outcomes of these sub-goals, when designing self-improving systems

      efficiency sub-goal

      • resource balance principle
        • the rates of increase in utility should be equal in all different resource allocations

      they will do anything to preserve their utility function

  • you can look at corporations as rational economic agents
    • some claim that they behave like a sociopath

One thought on “Summary of lecture on the nature of self-improving AI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s