Architecture of Emergence & Memetics, Part II

Here’s a core concept of my emergence engine: model based evolutionary value creation. Basically it means the following:
- You get stories depicting some domain, i.e., user answers or tweets
- You translate them to semantic concepts & statements
- You give these concepts & statements (model parts) behavior, which basically tries to create value for the end users, or the end users organization
– Simplest example is when some statement can bring value to some user if he learns about it
– Another example is when several statements reason the causal relations between them & infer what is the root cause to some phenomenon
- Having all of the model parts behave all the time is compute intensive, which has a cost
- Not all of the model parts have the potential to create value
- So, an evolutionary process can take place, in which only the model parts that succeed in creating value survive & get resources.

See also my initial post on the base concept.

 

Blood vessels inspired mechanism for driving social interaction across the enterprise & accelerating change

A. The business problem
Large organizations don’t work as one body. This is problematic, because of numerous reasons. To name just a few common ones:

• The agenda, priorities, focus & policies of higher management may not be known or implemented by the lower level units & employees, leading to garbage factories, missed targets & activities not aligned with company policies & agenda
• Adapting to changing business environment is slow in large organizations, because of the difficulty in changing both processes and value streams, as well as the mindset & knowledge of the people running them
• Problems and threats, or on the other hand innovative ideas and opportunities are not handled because they’re usually not propagated from the lower level units to the executive level
• A need for activities in one unit, with not enough resources, can’t use free resources in another unit, as well as existing know-how or goods already achieved in other units

B. An analogy: blood vessels
Blood is the source of life to all organs and cells in an organism’s body. It supplies all cells with the inputs required & returns their outputs. It carries commands sent from the brain to organs, via hormones, as well as the necessary chemicals & food the body tissues require. It also spreads solutions for disease agents across the body.
In order to reach every cell, in all organs of the body, the blood vessels are organized in an hierarchical structure, starting from the heart & lungs, and spreading downstream till every cell, & back to the lungs & heart.
Blood flows in frequent cycles, circulating the means of life continuously across the body. It interfaces with the cells via special membrane controlling the flow of materials in & out of cells.

A business enterprise is a large multi-cell organism, that uses knowledge to transform supplies into goods, in a complex value stream. What ties the multiple humans working in an enterprise into a large super-organism, are the communications between them, that coordinate the processes comprising the value stream to drive the enterprise to survive & grow in a dynamic market environment.
However, we believe that these vital human communications are based on paradigms that were established & formed in the pre-computer-mediated era, & for sure before the new social interactions paradigms of the Web 2.0.
We suggest a simple mechanism, inspired by the architecture of blood vessels, for making the communications inside the company flow from the top management to all employees, and also across the entire organizations, in order to help the enterprise work as one body.
C. Social media capabilities
We propose new social interaction “vessels”, flowing from the CEO, through the organization structure, to every employee, and also aggregating the responses from the employees all the way back to the CEO. The basic component of social interactions is a simple short text (the kind passed in services such as Twitter, normally limited to the size of Short Text Messages), along with simple discussion semantics such as “in-reply-to”, “for”, “re” prefixes. It’s highly important that the input from employees will be in this open-ended form, & not forced into any structured schema form, in order to promote the discovery of insights not known in advance, and the emergence of bottom-up new usages of the system. The text can contain marked tags, that characterizes its content, either given by the user, or automatically marked by the system. The system will query users every day for status messages, that may contain such information as their:

• priorities & focus
• risks, undesired effects & problems
• achievements, opportunities & value drivers
• work status, load & health
• requests & questions

Querying can be done in any communication channel used by employees (IM, SMS & Email, &c).
Every day, each employee will also receive a collection of messages from the hierarchy path above it, from the CEO, to its direct supervisors.
Using the tags, messages can be passed between users not in an hierarchical path, just on the basis of similar tags. Tags may also be collected in user’s profiles, so that a user who once wrote a message containing a certain tag, will receive future messages with that tag by other users.
Every manager will receive daily a collection of messages from the hierarchy path beneath it. Aggregation, clustering & classification can be done on the responses, to make the results arriving upwards summarized. Visualization methods can be used for presenting a GUI that enables both topsight view of the aggregated results, and ability to explore area of interest.
Messages may be marked with an importance indicator. The higher the importance a message is ranked, the more exposure it will have across the enterprise.
Users will also be able to vote on other users messages, & increase their rank & exposure.

An important use case is with risk management. A user may enter a message describing a risk it wants to put to discussion, and rank it with high importance. The message will be exposed to many other employees across the enterprise, that are related to the tags in the message. Their responses will form a multiple stake-holders discussion, which is the best known way to prepare & address risks.

When considering the extended enterprise, certain units & employees interact with external people (suppliers, partners, customers &c). The flow of social interaction can flow through them to the external people, & back to the CEO. Similarly, the chain can start from above the CEO, e.g., main share holders, allowing them to both interact with, & learn on the overall, current status of the company.

The frequency cycle of this process should be as high as possible, such as 1-2 days, but can also be once or twice a week.

D. The utility of solving the business problem social media capabilities
Unlike the common communications practice in today’s common enterprises, in which the the frequency of communications between the top executives and the employees of the lower level units is limited to 1-4 times per year, and usually also limited to unidirectional communications, the suggested capabilities can foster bidirectional communications on a daily basis, flowing from top management to all employees, and also between employees across the enterprise. This is very likely to address the common reasons given above why enterprises don’t work as one body:

• The agenda, priorities, focus & policies of higher management will be known & implemented by the lower level units & employees, leading to the elimination of garbage factories, met targets & aligning all activities with the company policies & agenda
• The frequent circulation of knowledge on changing business environment, and the way the company adapts to them, can create positive feedback loops, that incrementally change & adapt processes and value streams, as well as the mindset & knowledge of the people running them, to the changes in the business environment. Using today’s practices such adaptations can take years, but may be reduced to only weeks using the suggested capabilities.
• Problems and threats, or, on the other hand, innovative ideas and opportunities will be properly propagated from the lower level units to the executive level, which will enable their effective handling & value extraction
• A need for activities in one unit, with not enough resources, will be communicated to other units with free resources, or relevant existing know-how or goods already achieved, which will enable collaboration, higher resource efficiency & considerable time saving.

The main benefit for the end-users will arrive from the much stronger bonding & involvement with the entire organization, that can make employees feel more motivated and appreciated. Other obvious benefits are greater responsiveness from their management, & stronger collaboration with other employees across the enterprise.

E. The information that will be collected and how it will be useful for the enterprise
The communications that will circulate in the social vessels may contain valuable information, of many kinds:

• Risks & threats that may be handled immediately after they are discovered, instead of after they cause their damage
• Opportunities, value-drivers and innovative ideas that will arrive to the executives that can understand their value & decide on their implementation
• Trends that may point out problems, and emerging changes, that can be handled before they are reaching a critical mass or serious effect
• Insights on external factors outside the company, that can arrive even from low-level employees, and hold important strategic opportunities or threats

Once the collected information reaches a sustainable size, it can be applied to analytics that can provide both macro insights on the status & health of the entire organization, and well as micro insights on bottle-necks and inefficiencies that can be removed to effect bottom-line profits. For example:

• Analyzing negative sentiment in messages, based on Natural Language Processing, can indicate trends in cultural health problems, in certain area of the company or across the organization
• Aggregated messages tagged with over-load, at certain units or type of resources, may indicate a bottle-neck that delays other units & processes

F. Ensuring privacy and security of information gathered
While the querying interface of the system to employees are common media channels, such as SMS, IM & Email, the outputs of the system (collection of message from the hierarchy above, and aggregated responses from the hierarchy below) are presented in a Web application, that can be protected by any standard of security, for both authentication protection and authorization. Messages can also be marked with security level, to make them available only to employees with access to this level. Any other organization policy, taking into account organization structure or roles, can be applied to determine authorization to view messages.

 

Architectures of Memetics & Emergence (part I)

Motivation: autonomous value creating applications

I’ve been working on a very pretentious platform which I hope can prove useful for innovative applications. The platform is based on 2 main principles, Memetics & Emergence. Both are originally taken from the world of human culture & sociology. In the architecture of this platform they are applied to a complex composite Multi Agent System. The motivation behind it is to try mimic the way human individuals, organizations & societies succeed in very large complex tasks, whereas it be a single human, small team, business corporation or a whole society. The fact is that a single human, or any organization of humans is usually good in doing something, called Creating Value. The fact that I earn money is because I create value for my employer; the fact that some company makes money is because it creates value for its customers. So the general motivation for an architecture that tries to mimic human or human organization is to enable software to create value. It isn’t that existing software today doesn’t create value, the only reason software exists is because it creates value. But, unlike software, humans aren’t (explicitly) programmed – they are given some initial knowledge (education/training), they are assigned some jobs, & they create value while collecting the knowledge & expertise in doing it. And this is the motivation. A task such as enabling applications to create value without being programmed seems complete Science Fiction today. So we require something very novel & innovative & something very new to basically be able to claim that we can build such applications, that create value without being programmed, except for some basic education: when assigned with a job, performing it, improving in it & creating value without being specifically programmed as to how to solve each case. The motivation is to create software that just like Humans, even when provided only with basic knowledge of what to do in each case, still can:

• Solve unexpected situations,

• Create value in unexpected ways,

• employ both common sense &

• the ability to learn from situations &

• improve its performance, i.e., the value created,

• by merely performing the job for sufficient amount of time.

 

Memetics & Emergence

So, what are the architecture components that we claim may produce this?

Let’s start with Memetics. Well, memetics basically is the theory that there is an evolution of ideas, where ideas are taken in the broadest sense of things that you copy/learn from others. This evolution is for what is called human culture, science, art, & basically our whole social life is based on memetics. For an ultimate introduction to Memetics, I highly recommend hearing or reading the proponent of this field, Dr. Susan Blackmore. This is the basic idea. This idea can apply not only to the humans world, but also to general intelligent agents. When applied to software, Memetics basically means Evolutionary Knowledge Engineering. The idea is that whereas in knowledge engineering we produce knowledge representations of a domain, including also knowledge required to perform tasks, i.e., Behavioral Knowledge, in Evolutionary Knowledge Engineering, we apply the evolutionary algorithm to this process of knowledge engineering. Meaning that, if we have variations of knowledge representations & if we have different versions of how to perform tasks, only the fittest of these pieces of knowledge will survive & be the base of the knowledge base population. The effect of this is improvement in our knowledge, which becomes more adapted & effective in it’s domain environment. So, to recap, memetics is all about people spreading ideas, & the ideas that are the fittest – most fruitful & valuable – are the ones that survive & base the population of ideas. Similarly, Evolutionary Knowledge Engineering is just Memetics applied not to the culture of humans, but to any society of agents performing knowledge engineering.

 

 

The 2^nd concept called Emergence, is basically a claim that high-level intelligent behavior can be obtained from low-level simple agents, whether it be animals, software or any object with some behavior, when you combine them into a group, that works together. So any time you take a bunch of agents & combine them into a group, even though each of them has a very simple low level behavior, that may not present any intelligence whatsoever, i.e., any complexity, any reasoning behind it, nevertheless, when you combine them into a group, that works together & collaborate, suddenly the group has an higher-level intelligent behavior, in other words, the intelligence emerges from nowhere, by just combining the agents into a bigger unit. For a great introduction to this concept, with numerous eye-opening examples, I highly recommend reading the book on this concept, by Steven Berlin Johnson. Normally, we think of emergence in situations when the intelligent behavior emerges unexpectedly, but I prefer to include any high-level behavior formed by the collaboration of lower-level parts. E.g., a Power Ranger has this amount of power, but when a team of Power Rangers connect together & morph into a giant all-mighty robot, I also see it as emergence. Now this of course may recurse, for example, if you take a group of A-type agents, & combine them into a group, called B-type, & then combine several B-type agents, into a group called C-type. Now the C-type agent can then manifest even higher level of intelligence than B-types agents, & this is like multiplication of the power of emergence, because we start from simple very low-level unintelligent A-type agents, & multiple the emergence effect & get C-type intelligent agents.

Illustration of emergence: combination of many simple pixels into a group, creates complex intelligent picture (Original image by Matt Champlin)

And another illustration: it’s hard to model a 3D shape, e.g.:

 

But if you zoom to a much lower level, you can model the shape, e.g., using many simple triangles:

 

Emergence examples are all around us, everywhere you look, & it’s enough to mention the extreme intelligence (learn & behold) of Ant colonies as a very obvious example. Each ant doesn’t manifest high-level intelligent behavior, but when you combine them into a group, you get a very powerful & successful intelligent behavior. Ants are a very good example, but if you think about it, take any group of humans, whether it be a family, team, community, organization, city, nation, any group of people, is strong because it has more intelligence & more power, that is ability to solve large problems (i.e., Intelligence), only by combining individuals into a higher-level group. In corporations, or hierarchical organizations, we see the emergence multiplication effect, where we take several people into a team, & then take several teams into a department, & then take several department into a division & so forth, we see that more power & more intelligence, more high-level behavior, come out of the group as we multiply the emergence effect. We must understand that it is not the sum of power of the individual components. Take for example a branch of a fast-food chain. The power & intelligence of it, isn’t the sum of the power of the staff running it. The added power & intelligence of these workers isn’t enough to feed thousands of people each day. These young people don’t necessarily understand the process & knowledge, & the sum of their intelligence isn’t enough. Put them all in a room, & you get no special intelligence & power to feed many people. The intelligence is in the fact that working together they create some higher-level machine. They create something that is very powerful, feed thousands of people, but it is not the sum of their intelligence & power. The intelligence is in the combination of them into a collaborating team. Everyone are doing their low-level job, & you get a very powerful higher-level machine. Once they combine you get the emergence effect. Suddenly a bunch of teen-agers feed thousands of people. (This is just an illustration, please don’t take it personal if you happen to be a teen working in a fast-food branch…)

 

 

You could say that both Emergence & Memetics, are nothing but metaphores, ways to see things, which humans have always known. But as any science theory is just a way to see stuff, judged by its fruitfulness in predicting measurements, I believe with these concepts you understand how come human ideas & knowledge improves all the time, & how come the teaming of humans into special types of groups yields so much power, & once you understand it, you can harness this in human life, to create new types of mechanisms, for example as the social services harness the concept of emergence, file-sharing networks, Web2.0 social services, all exemplify it in numerous examples. You can also harness these principles into architecture of software agents, which is what the platform I’ve been working on is all about.

 

Illustration of a simple composite architecture based on Memetics & Emergence

 

 

Test case for yielding emergence

I was thinking on the simplest way to test my emergence engine, & came up with an extremely simple task – the reactive algorithm of a thermostat: measure the temperature, & turn the heating on & off to maintain a given temperature. It sounds indeed very simple to code a program that does that, but what I’m going to experiment is how to do it without any programming.

Emergence engine is a kind of general AI, capable of achieving goals, without being programmed how to solve them. It’s based on the assumption that you don’t need to build real intelligence, rather just create many many simple software workers, having only very simple tools & logic, & let them swarm their way toward the system’s given goals.

So, here’s how I hope my engine will handle the test case:

• It should 1st learn by elicitation the model of a room, having a temperature, thermometer & heating unit.
• It should also learn the relevant beliefs on the effect of using the thermometer on the accuracy of the model, & the effect of turning the heating on & off on the room’s temperature
• It should then learn what’s the desired temperature
• From this it should start deriving action plans & execute activities to achieve the goal of maintaining the desired temperature
• It should also adapt to changes in the room, e.g., a door is open & there’s need to use more heating, or alternatively the heating doesn’t work & we need an alternative heating unit

I’m saying it but of course what’s doing all this are many collaborating agents, working together to achieve the goal. This is done by breaking the value in the goal into smaller value “summs” given to states & activities leading to the goal, & having the agents collaborate on creating all these summs.

Although the design is very simple, & intended for complete autonomous behavior, I noticed that I’ll be able to effect the engine & help it reach its goal, by changing the knowledge driving it, i.e., the learnt beliefs, according to which the agents work.

So, I can’t wait to see how the engine will handle this, which will actually test whether the simple emergence design is enough to yield emergence, even if the value it delivers is so small & simple.

 

KQML for SOA

I’ve been using this simple pattern for the past year & a half now, to facilitate SOA, & didn’t yet stop to ponder & evaluate how valuable is it.

KQML stands for Knowledge Querying & Modelling Language, & basically defines a simple message structure, that contains fields like:

• Speech Act
• Recipient
• Sender
• Language
• Content
• Ontology

(there are many more fields, but these are the most important.)

I use KQML together with a distributed shared memory (from GigaSpaces), to create a distributed architecture in which many service providers & applications basically converse with each other in order to achieve almost anything. Any piece of code is devoted to sending & receiving messages, calling services, processing them a bit, & sending them forward.

This allows more complex grid like distribution, using mechanisms that load new service providers according to the accumulation of unhandled messages.

What I learnt using this architecture is that the use of semantic-enabled content (e.g., OWL) made the required message processing much simpler: usually, the content is meaningful to all conversing service providers, so they don’t need to modify it much, just behave according to the requested speech act.

It turned out quite useful & simple for integration with service providers: they usually liked the simplicity of the pattern, requirring them only to know which speech acts are supported & what logical content they need.

Overall, KQML yields quite nice SOA, but it isn’t so common & adopted, because most people turn to well adopted standards (e.g., WS, which can be made semantic using OWL-S). I’m going to start working soon on a new architecture requiring much more dynamic & collaborative conversation, which will put KQML to a real test.

 

Summary notes from Joshua Schachter talk

[My notes from the recorded talk of del.icio.us author at the Carson Workshop Summit on the future of Web Apps:]

• Scaling
  • Recommends reading: Cal Handerson & Brad Fitzpatrick (Flickr & LiveJournal) presentation on how to make things fast (scaling) – helped Joshua a lot
  • Nagios - to figure what went wrong, get alerts
  • Use caching everywhere possible
  • figure where you can be latent (slopy), & be latent there
  • wait to see what actually breaks before it actually does
  • Know how to tune DB & Apache
  • Put Proxy – not Apache – before it: prioritizing services, load balancing
  • e.g., someone saves the site to his disk – this will blow the site
  • e.g., Greasemonkey script that checks delicious on every web page
  • throttlling
• API – help adoption
  • the easier API is, the more they’re used
  • don’t expose the internal identifier, especially if its sequential or computable (people will iterate on it)
• what features to add?
  • things that are crucial for success (eg tagging) – very usable, must be added
  • don’t add something that exist elsewhere (e.g. messages)
  • if something is asked, try understand the reason, what’s the real problem that needs to be solved
  • some features have a too heavy performance price (eg query calculus)
• put RSS every possible thing
  • always be able to answer: is there new data in this RSS (by time)? this will save access to data
  • RSS is the heaviest traffic
• hide everything of the underlying framework (e.g., filename.php)
• watch for new behavior in the application, & decide what to do with it
• solve a problem you really have (Joshua had a text file with 26K bookmarks) – because you really understand the problem & passionate on solving it better than whoever don’t have it
• every day your system isn’t open, you’re loosing users & input. get it out there asap
• aggregation of attention (what’s popular today) is cool when the population is small, if its big there’s too much bias. create piles of attention
• spam – people will try to get to the pile of attention
  • when you track spammers, don’t give them any feedback (error messages &c)
• tagging:
  • useful for recall
  • ok for discovery
  • bad for distribution
  • not all metadata is tags
  • make people make the minimal amount of work
• understand the motivation of the people
  • user 1 have to find the system useful (from selfish reasons)
  • if the value is from many users, it’s problematic
  • the tip: the users community should want to get more users to the system (evangalize, viral)
• beaware where you spend your efforts (e.g., a feature no one ever uses)
• watch your system carefully
  • intuition backed by data
  • measure everything
  • how people react to features, what they do
  • measure behavior rather than claims
• testing is very important
  • user acceptance testing
  • everybody on the team should look behind the mirror on actual user labs
  • when doing labs, don’t give them todo’s
  • they worked with “Creative Good“
• use the users language
  • don’t make them use your language
• registration for seeing is a roadblock
  • give as much functionality possible without registration
  • users want to get a good idea of what they’ll get before registering
  • you must show them, they won’t read about it
  • registration should be as fast as possible, & get you to where you were before
• understand where you’re breaking the current paradigm, but other than that, learn & use how the world/internet work (emulate it)
• you have to develop a set of morals
  • its the users data, not your data
  • CRUD their data
  • up to remove all your account data
• infection vectors
  • Joshua spend $0
  • promote evangalism
  • RSS is good – you evade channels, get to applications & users
• understand how a community uses your system
  • Joshua don’t want to own the community – the community is elsewhere
  • just enable communities use your system, don’t force them anything

[See also:

• a collaborative summary of this talk I found on the Web
• Mindmap summary of this talk from Larsz
• a lecture I gave on Web2.0 & the Semantic Web]

 

Finally understood the architecture of the world

The famous programmers quote that “to iterate is human, to recurse, divine” has some good insight. The architecture of the world is simply: recursive multi-agent system. The pattern is so visible & clear: watch every piece of the world & you’ll see many agents, interacting with each other & pursuing their goals. Zoom in & you’ll see that each agent is composed of many smaller agents, that are similarily interacting & pursuing their goals. Zoom out & you’ll see that from the agents interactions & goals pursuing, emerges a higher-level behavior, which turns all of them into a larger agent.

(the great Powers of Ten movie by IBM)

If this is how the world is designed, all the OOP/AOP architectures have a long way until they’ll be able to model anything properly, instead of just stuffing tons of logic into useless huge complexity. Only some Emergence based new architectures (such as Echo & StarLogo) are starting to model the world as it really is.