Collective Intelligence and open source

For the past 20 years, I have been for an open source author and contributor (I will use the term ‘open source’ but I could say ‘free software’). I had the occasion to think about proper collaborative environments for group collaboration around writing some piece of software. When I hold a leadership position in the Tikiwiki community my main concern was to enable an efficient context for massive participation. At that time I was called a ‘reluctant leader’ for the project, while I was considering myself an ‘attentive gardener’.

Collaboration can be instinctive in some contexts. But engineering projects need some flavor of coordination to make things go smoothly. The goal being to reach the threshold of the sum. It happens when it’s more efficient to join the collective than to work in isolation. The collective action is more than the sum of the parts. It’s quite easy to achieve and it happens in every business. But in an environment without enforcement of an authority, like in open source projects, the setup has to rely on autonomy of the participants and their willingness to play their part.

In such context, I immediately got seduced by my encounter with collective intelligence. The emerging whole, the autonomy of the parts, the object-link, the need for holopticism and defined standards are many aspects that we could find in an open source development team.

Since that time, Github, in 2008, changed the way open source is done. Before, on sourceforge or such platforms, contributors needed to ‘join a group’, request access, whatever the protocol was decided by the maintainers that already had access to CSV or SVN repositories. Now on git-based projects, contributions don’t require joining any group, Pull Request are just an easier way to submit patches without the need to agree on any prerequisite. Collaboration began to become punctual, sporadic, in many cases just a one-shot contribution. Of course there are still communities that keep a strong context beyond that sporadic aspect. Core teams still exist following various models (I saw a nice video recently explaining some of those models). In most cases the link between participants was not as tight as it could have been previously.

But from the perspective of the gardener, it doesn’t change the equation. Variables are different, but the logic is the same. We still want, in a loosely linked relationship, to extract more impact than if the relationship was not there. And the recipe for designing an environment will follow the same principles.

Here are the elements that characterize an emerging collective intelligence, they apply pretty well to the context of community-oriented open source software development:

  • an emerging whole: the result of the collaboration cannot be predicted, it emerges from the autonomous actions from the participants. Steering is possible, but if the given direction doesn’t match individual interests of the participants, it will just fail. In that regard, open participation and flexibility make the emergence easier.

  • holopticism: this weird word is synonym of transparency, visibility and reachability. Each participant, being autonomous, needs to have a view on what others are doing and where the whole is going. It’s all about creating bridges instead of walls. The opposite of holopticism is panopticism, well known in hierarchical companies where everybody knows their part and cubicle protects from distractions. Such closed context creates a strong dependence to a ‘manager’ and negates any possibility of autonomy.

  • rules of the game: in a strongly oriented technical context, there are various rules that are supposed to be already known by the participants, related to the conventions of the craft. But any other non technical rules, defining a social contract, need to be clear and explicit for each participant. It’s about license of the common good, ways to interact, maybe code of conduct and any other convention that make the collaboration clear and rules enforceable by each participant.

  • switchable roles: because nobody can be hold accountable for completion of any task, several participants should be able to accomplish any task required for the accomplishment of the shared goals. For example a release process, code review, mentoring of newcomers etc. Those tasks cannot be exclusive to designated people, because if they got stuck, the whole collective gets stuck. This polymorphism of the group is necessary for its resilience.

  • linking object: in order to help autonomous participants to synchronize their efforts, a common object is required to be defined. The term ‘object’ is wide in the sense that it can be a milestone, a todo list, an objective that is likely to attract the approbation of all participants, or simply the maintenance of a codebase or an infrastructure. This object can be evolving if this evolution follows the rules of the holopticism: everybody has to be aware of the change (release done, infrastructure in crisis, milestone passed, changes in a todo list, …)

  • transactive memory system: this is the way for newcomers to acquire knowledge of past iterations of the project, of the life of the group and the linking object. This can be served by availability of logs, mentoring sessions, conversational approach of story-telling, stored minutes of group interactions, etc. But it also includes specific parts of knowledge required for activity in the group, possibly tutorials, guides and any learning material. It also includes clear description of the procedures to achieve any tasks, enabling the ‘switchable roles’ principle.

  • collaborative economy: also called a gift economy, as opposed to a contractual economy. In a collaborative economy, the gift comes first, and the reward comes from the relevance of the gift to the common goal. In a contractual economy, the reward is agreed first and the task is accomplished in the conditions agreed upon. For nurturing a rewarding collaborative economy, a culture of recognition of peers is required. Good actions need to be acknowledged and praised.

Those 7 principles are extracted and adapted from the definition of the original collective intelligence as presented in the Collective Intelligence, the invisible revolution from JF Noubel.

Now that those principles are identified, I can think of some more point that need attention.

  • architects, catalysts and gardeners: all the principles above are not occurring naturally, they need to be designed, engineered, nudged, maintained and renewed. A great part of the job is at the start of a project to prepare the ground for the incoming collaborators, like a seed that will possibly be modified under the influence of the future participants. But it has to start somewhere. And during the lifetime of the collaboration, attention has to be dedicated to keep all those elements active and vibrant. Some participants, being concerned by the core principles, have to step up, pay attention to the environment, and operate towards the balance of the whole for its harmonious development.

  • gradual involvement pathways: there should be a choice for any participant to chose their degree of participation. It means that someone that just wants to fix a typo on a text should have one or two paragraphs to read before proposing their help. Someone willing to help in translations should have not more than a page to read to understand the translation process and workflow. Someone that wants to dig deeper should be able to enter the cycle of self-learning with all available sources from the collective.

  • specialized sub-collectives: some tasks cannot be shared widely, like a root access to a server for example. But each group in charge of a specific task (system administration, finances, legal, etc.) should be composed by more than one person and follow the same rules, being closed doors, as the rest of the collective. Pragmatically this setup is only realist for projects of a certain size. Contingency often dictates that only one person is holding responsibility for some specialized task. But it should be kept in mind as a vivid truth that this is not an optimal situation. It is a vulnerability in the setup when only one person is holding a specialized role in the collective. But in any case, the default position regarding information should remain the path of sharing with others, and the secrecy is being decided as an exception for very specific reasons (security, privacy, legality, etc.).

  • arbitration mechanism: when vision fails to converge, arbitration can be required. Its procedure should be clear as part of the rules of the game, and can involve voting, call for a referee or a jury, consultation of participants whose reputation is above all questioning if there are any, mediation by someone playing the role of a catalyst, etc. It’s not required, and it would actually be harmful, to write a full book of law. A couple of paragraphs describing the arbitration convention is enough.

  • tools and platforms: various tools need to be available for synchronous and asynchronous exchange between participant, collective gathering of activity logs, centralized or replicated availability of the rules of the game and transactive memory elements. Depending the nature of the project, some more specialized tools may be required, especially for the holopticism part. Transparency is not a natural state and it requires a good toolbox for its enforcement: graphs and statistics, monitoring of activity, track records, audit logs, periodic summaries of activity, subscription-based diffusion of activity logs, reports builders, etc. There is never too much of it. Abundant information can be filtered selectively, but missing information cannot be guessed.

  • interpersonal bonding: it’s human nature. It’s much more enjoyable to get involved in a collective initiative when it creates personal links. The collaboration space should not only be geared toward efficiency, but should leave space for random and personal interactions. While it’s common wisdom to avoid speaking about religion or politics, any other subject can be shared among participants (well, in various cases I also have seen religion and politics debated without harm among mature participants). Well, like in real life. We are social beings, after all. And the social link is an extra element of satisfaction.

There are various other details but those are the main ones, at least in my experience. The reality of things is certainly far from perfect and there is a majority of open source projects that miss many of the points I list here. But for those who match this list of characteristic, I notice that they are much more successful, efficient and healthy. When the state of Collective Intelligence is reached, the shared benefit can be felt by all participants. Beyond the fact that one does participate in full autonomy and voluntarily, perceiving the added value of the collaboration is a form of accomplishment.


Collective Intelligence or Collecting Intelligence

Because I had some time recently I watched the whole batch of videos of the “Collective Intelligence Conference” from 2014 on youtube. It’s part of my exploration of that field, see what changed in the past years.

Well, those were pretty interesting talks but it stroke me that there was a clear ambiguity of terms there. It’s related to the english usage of the “intelligence” word. The exact same word in french only means the capacity to process information and take decisions. This is a human trait, somehow a way to distinguish us from the rest of the animal realm.

But in english it also is used by government as “data collection”, as in “business intelligence”, or “the intelligence community”. In french, we call that “renseignement”, so it’s a totally separate concept. So, from my french perspective, I didn’t anticipate the impact of this double meaning for the word “intelligence” while watching talks about “Collective Intelligence”.

So there was a lot of talks about crowdsourcing, about new ways to gather information from previously untaped sources by the way of massive internet participation. And not that much about the other intelligence which is about organizing data and structural design of human interaction.

They held conferences in 2012 but the links to videos failed, in 2014, which is the one I found videos for, in 2015 found no videos, in 2016 videos are in google drive. The 2017 edition is just happening right now and I hope there will be videos.

There are a lot of academic papers linked to those talks. Pretty good content.

But what I’m interested in seems to be a very small subset of what is called Collective Intelligence nowadays. I may need to define the term more explicitly. Let’s call it Intentional Collective Intelligence Engineering. Because as a software developer and a systems architect, I’m interested in building things, designing its cogs, anticipating its impact on usage and on norms.

While I like collecting feedback from the users of my solutions, I also know that the numerous choices you do when designing a system impact on the usages in very deep ways. Ways that the user cannot anticipate or imagine. Those are not always conscious habits that are influenced. There are always natural (or systemic) forces that drive a group to adopt specific strategies after having analyzed and matured the knowledge of its context.

And I’m pretty convinced that the Collective Intelligence I seek can be engineered. A proper environment, set of conventions, incentives and obstacles, can be deliberately designed for the purpose of emergence of that collective intelligence in a group of participants.

I have seen glimpses of it happen while participating in open source community development. I know there are bricks already existing to create various conditions necessary to the emergence. It’s not an utopia.


bye bye Green Ruby

Today I’m closing Green Ruby. That newsletter, started on february 2013, has been a self-imposed weekly duty for more than 4 years. I take a lot of pride not to have missed one single week during those 225 weeks.

But now that we decided, my friends and I, to put it to an end, there is an interesting question popping up. What will happen with the list of 2,000 subscribers?

It’s common knowledge that nowadays, when you don’t pay for a service, you are the product. But honestly, I hate this statement, however relevant it may be. It eludes any possibility of generosity, and any chance to enjoy a gift economy. Open source software is an example of this. You don’t pay for it, but by using it, you are not used as an asset or a resource.

So as a symbolic gesture of rebellion, I decided that, by ending the publication, I will delete the list of subscribers. Forever.

I always felt raped when I subscribe to a service under a certain context, and then the service provider gets acquired by another company and my account gets automatically transferred. The contract changes, I have agreed to the initial setup, but why presume I would automatically agree to any other context? This is the real annoyance in that user-as-a-product thing. You are not considered as a thinking being anymore.

I hate this!

So the deletion of that list of email is a symbolic sacrifice. Why did you invest all that time and effort during those 4 years then, will you ask me? Well, I was willing to contribute to a common interest, out of generosity and love for my peers and friends. Why should I need a money-related reason? Money kills the social link, it neutralize the unmeasurable debt that we establish between each others as a community.

A society without generosity, is a society of isolated people. Gifts creates a feeling of debt, that debt is a bond. If the gift is not personal, it relates you to a community. Never forget about it. Pay it forward.


The virtues of transitions

A collection of events occurred since my last post. I was really hoping to get some deep dive into writing more about collective intelligence, in the background. But various things came and distracted me from that initial trajectory.

Hubot revival

First, there has been a revival of the Hubot project. I wrote a bunch of plugins for that chatbot for internal usage at Gandi in the past year. I have been poking around in the community complaining that it became some sort of abandonware. Issues on github were not addressed, PR unanswered, irc was silent.

Then one day 3 weeks ago, bkeepers came up and woke it up. It appears that now Github has plan for this codebase. They want to include it in its products more officially, says the rumor (nothing official yet). But it makes a lot of sense. So then he gathered people that could be involved in the revival of the project. I was contacted to join the core team. That’s what you get by nagging people.

Since then it has been quite great to clean up backlog, re-examine old ideas, help setup new process and tooling for community gardening. I never get tired to see how informal communities can shape up and self-organize.

Quiting is not a bad thing

Around the same time, I felt inspired by handing my resignation of my current job at Gandi. There is nothing wrong with the job, and most reasonable people could argue that I’m just stupid. But there are a combination of factors that led me to think I was not using my potential at full capacity.

Working on a large infrastructure as a sysadmin was great, but it’s pretty hard to build new things properly in the service industry. It’s always about contingency management, and reacting to the environment. It’s more about doing the fireman than the gardener. Well, honestly, I think it’s a question of intent and culture and that could have been addressed. But I didn’t see any hint that could let me think this environment would become intentional about gardening anytime soon.

So, out of the blue, I left. I chose a moment where there was no crisis, no trouble, no real reason to quit except a strong feeling that it was time to move on. That way, transition is painless, there is no frustration involved. I hate conflicts. My advice: never wait the burn out to make your move out.

Green ruby end of life

In the mood of transitionning, this week I proposed that we close the Green Ruby adventure. For some months Xenor has taken over the publication process. But he’s himself bunt out a bit. So after 4 years of publication, it’s time to close this properly and move on.

That weekly links collection was a fun exercise. And I’m really happy to have refused any business side track on this operation, it would have been much harder to close.

What next?

I decided that I could stay funemployed for a while (that’s the proper term, right?). I’m planning to look around, play with techs, talk with friends, hold on any decision to engage in anything. It’s time to re-assess my roadmap. This is what is great in transitions. By losing contact with the ground, you can feel the direction of the wind.

Maybe I will get back to entrepreneurship, and setup some business venture. Maybe I will freelance for a while. Maybe I will find a context in a company where I feel comfortable contributing. But honestly I have no clue. And yet, this feels great.

Getting back to old friends

In the first days after my decision to renounce my day job, I got back in contact with a lot of old friends. Transitions have that side effect that you have to re-evaluate your environment, your network, your values.

Getting into a full-time job doesn’t prevent that, of course, but you cannot give the same attention. The center of gravity of an employed person is static. Now that I’m floating, I have to dedicate much more attention to my environment. And it’s an excellent thing.

I will certainly have occasion to talk about my adventures very soon. Maybe I will force myself to write more.


How I came to collective intelligence

I plan to share some thoughts about Collective Intelligence from a software perspective. But this comes from an old story that I thought I should make public first.

The origins

When I first began to program web applications, it was mostly for my own needs as an activist (freedom of speech on internet, cryptography, and such matters). Around 1998 I had a fully working solution called ‘Collaborative writer’ or cWriter for short, written in PHP. It was like a sort of wiki geared towards small groups self-organization.

When I got to Claranet I deployed it as an internal solution, and worked on a full company intranet called ‘in’. It was the occasion to release cWriter as open source in 2000 on Sourceforge.

At that time it appeared obvious that my area of preference was about engineering collaborative work. I was putting a lot of thoughts into that. I had this clear understanding that the tool shapes the usages, especially in this new realm that was internet communication.

Later on, I dedicated my time in various projects, mostly Tikiwiki groupware in 2003. That experience was also rich in valuable experiments. I became leader in that community because being unemployed at that time, I took the occasion to just invest time in open source (being payed by labor insurance money). And I was just very prolific. I established various bases for that community that still thrives today. I had a recorded interview with the guys for the 10 years for the project in 2012.

The discovery

Somewhere in 2004 I met with JF Noubel, that had some need for help on his own Tikiwiki instance, called thetransitionner.org. His website (not available anymore nowadays) was dedicated to promotion and exploration of Collective Intelligence, following the work, among others, of Pierre Lévy.

I quickly identified this area of research as something that I was practicing and aiming at, without having formally identified it. So I got more and more interested into that field in the following years.

Jean-Francois became a friend, his approach evolved since then but I still got stuck with his original presentation describing the basics of Collective Intelligence.

Anyways it led me to some other kind of activism, revolving around some alternative currency projects (like openmoney), other projects concerning governance (like democracy 2.0, from Mikael Nordfors cited in the transcript on archive.org), and various other topics that would be too long to list here.

Ultimately it led to get involved in Angenius with Than Nghiem in a background of sustainable development between 2006 and 2008. Including going to live in a farm in Normandy. Which was amazing but I was just not made for it.

The migration

In 2008 I decided to leave France and go live in Taiwan. I wanted to take a break and learn Chinese. It was my decision for my 40th birthday. Since then I got various jobs in development and then in system administration, grinding more programming languages and technical skills. But I still kept all this time the background thinking on collective intelligence.

What I wanted to experience was just another cultural background. I wanted to see patterns a little better about open source communities and collaborative work in a different context. Well. This could be the topic for another post. I got some surprises.

What’s next?

Now, 9 years after, I feel it’s more than time I get back to my past ventures. Well, I didn’t live like an hermit, I’m involved in various local and global communities. But I feel I could share a little more about what I learned in my past experiments. Now that I reach my 50’s, maybe I just feel like I have to begin writing stuff.


Interview-based knowledge sharing

For years I noticed the difficulty to extract decent information from developers and craftsmen about their work. Developers clearly lack the skill or the taste for documentation. It extends to the specifications, which also provides an occasion to notice bad performance.

To work this around, specialized project managers have to fill this role, but it can create a gap between developers ownership and the final result. Sometimes luck creates the inspiration and developers produce a decent amount of documentation. But honestly, this is pure luck, and not a rule.

I found a way to mitigate this issue, that I began to experiment some time ago at work. It’s based on a recorded interaction between a project person and a person of the craft.

Here are the rules:

  • the project person organizes interviews with stakeholders from all specialties including developers, business people, operations people, sometimes users, the more diverse population possible. But it can also be done for no reason for sharing a certain kind of knowledge with an expert.
  • interviews take place by written interactive communication in a chat, in a way that enables logging (irc, slack, whatever)
  • it lasts 20 minutes more or less, but the format can extend to hours if the need is felt
  • once the interview is finished, the interviewer cleans up the logs, removes out of topic details, fixes typoes, removes elements that are purely belonging to the chat way of communication
  • the cleaned log is reviewed by the interviewee for consensual agreement on its publication
  • the log can then be added to whatever project space is dedicated to the topic at hand (documentation, specification annexes, study, paper)

There are various beneficial side effects to this endeavor:

  • actors have a better feeling of engagement in the project or topic at hand
  • it creates a bond between actors, facilitates future exchanges
  • it gives an equal chance for everyone to speak, even the shy ones, because the 1 to 1 context is much less frightening. In a meeting, there are people that never speak. Are they stupid? they are not!
  • it creates content that can be shared with other actors so that they have a better chance to understand the point of views of other parties.
  • it creates a useful reference for the project, a raw material which can be annexed and can be used for summaries. It also creates more content for an eventual search engine if the publication space has one.
  • there is less risk to cite someone out of context because the full context is provided.
  • it is much easier to organize interviews or hold them on the fly than setup meetings.
  • there is no feeling of loss of time like in a meeting: the time dedicated is intense and very interactive, there is no waste.

I had the occasion to try that technique in a context where internal communication is not optimal, for knowledge sharing at first, then for exploring options on a new project and set up specifications. I have been happy to notice that this formula creates pretty good output and generates a nice feeling in all parties involved.

Certainly more study will come on this approach. It’s a lot of fun.


Enough sleeping

A few months ago, I got tired of leading Greenruby publication and Xenor accepted to jump in and take the publisher role. I’m very grateful about it. But as a secundary effect, I lost that ranting channel I got used to abuse. Each week, I was getting somehow acustomed to send some rant with the list of links.

Now it’s time to get back into a publisher mood. I have various thoughts that could deserve sharing, or at least, publication. Well I don’t tweet much and I hate facebook, so my media coverage may be a little tiny.

But that’s not really a problem. It’s not like I have some agenda. I just wish to exercise my right to reflect aloud on various topics, bring to the open some personal and/or original thoughts. Nothing more. Whoever reads, do it in silence.


My lovely Chatbots, sick already?

Recently I have been to a meetup about chatbots. I was expecting something technical. But it seems that I was totally mislead on what chatbots are. They are visibly now more of a marketing tool than just a cool piece of technology.

I’m pretty convinced that web interfaces are going to die slowly in favor of more conversational interfaces. But in past technology shift we had some time between appearance and marketability. Like with the web, the social networks, etc. No more. Chatbots are immediate slaves for sales volumes. All the blooming of new services for ‘building your own bot’ are just totally geared towards customers management and acquisition.

My naive assumption was that assistive conversational interface could have a larger scope. But well, the market is driven by what sells, and what sells is what make more sales. I just hope it will have a nice border effect to improve chatbots as a tool or as a general interface to access information.

But my instinct warns me against the potential invasion of our social world online by all sort of aggressive automated peddlers. The same way we get robotic phone calls. Chatbots are going to be the new color of spam?


Is Facebook the new NPM?

This week there is a lot of noise in the JS ecosystem, on various trends. But the most noticeable is about Yarn. Yet another package manager for js, that states it opens a war against NPM (in soft terms but clearly enough). The long post from Facebook about it demonstrates a clear effort to push things forward about NPM shortcomings. The reaction on hacker news is pretty verbose.

It provoked a public response from NPM to try to explain that, nop, it’s not a war. But I keep feeling that Facebook is slowly taking over another piece of the javascript world. For now, it doesn’t seem harmful at all, but my instinct lights a various number of warning bonfires.

But for real, it was more than time that the supremacy of NPM was questioned. It has a lot of flaws and the number of posts about this event proves that the community was kind of waiting for something like this to happen. Like this cheatsheet, various articles like this one, this enthusiastic post from Yehuda Kats. But not everybody is happy about how things are done.

Personally, I like the technical direction it takes. But I hope that like io.js went back to node.js, it will eventually merge back to NPM, really. But I have the feeling it is not going to happen.


The JS toolback hell

This week I laughed a lot while reading How it feels to learn JavaScript in 2016. That article had a pretty good response, like it hits a nerve.

But seriously it’s clear that we are at a transition time in Javascript evolution, and there is a huge chaos of possible alternatives to everything. It feels like an ecosystem where the natural selection didn’t operate its magic yet. It’s like there are things in suspension that are going to fall in order at some point eventually.

My personal bet is that things like elm will win the race, because with its embrace of functional programming it seems like it opens the door to interpretation with yet-to-be-written faster and more direct compilers (rather than transpiling to js). But that’s just a hunch.

In the while, front-end craft is now a wizard arcane art. It can’t be acquired by pure reasoning and logic, or reading a doc. You need the map of the landscape for knowing the possible choices and alternative intermediary solutions. For part-time front-end people, it’s just hell. For now.