What is a Unconference

Last weekend I was at WordCamp UK it was described as an "unconference", I spend most of my weekends at "unconferences" and Hackdays, I've helped organised them I consider myself a BarCamper though perhaps not quite to the level of my friend Alistair! In my mind WordCamp UK wasn't an unconference the schedule was pre-determined with gaps to be made up by "adhoc" sessions.

Wikipedia described an Unconference as "An unconference is a facilitated, participant-driven conference centered on a theme or purpose." I'm not sure I'm comfortable with that definition I like to think of them as simply attendee driven conferences. The problem is that while Attendees could start a session, they did hit a few barriers in organising them the board was hidden, and went more or less un-promoted indeed the board could best be described as...

"But the plans were on display ..."

"On display? I eventually had to go down to the cellar to find them."

"That's the display department."

"With a flashlight."

"Ah, well the lights had probably gone."

"So had the stairs."

"But look, you found the notice didn't you?"

"Yes," said Arthur, "yes I did. It was on display in the bottom of a
locked filing cabinet stuck in a disused lavatory with a sign on the
door saying 'Beware of the Leopard'."
Hitch Hikers Guide to the Galaxy

As a Barcamper where the board is the focal point of an event and the spirit of it this was very depressing, I loved some of the talks but I couldn't help think I would have liked to seen more then Buddypress , buddypress again etc and if the board had been the focal point I strongly believe adhoc sessions would have appeared to counter this. I went with the intention of running a session, but left having not.

This isn't a rant about WordCampuk otherwise I would ask questions like where the money go etc, but it highlights that peoples ideas are different, I'm sure the organisers have been to a barcamp and see how "we" do it, they chose a half way house at it didn't work.

Can Conference and Un conferences work together?

I think they can, if you have a 2 day conference, I see nothing wrong with have structured tracks on day 1 and unstructured on day 2, likewise in a 1 day conference simply run a scheduled and unscheduled tracks. The key though is treat them as different tracks, keep the focal point "the board" and don't be afraid the community won't deliver, I have been to barcamps where I have looked at the board and around the room, and realised there were more slots then people. Those have been some of the best camps I have been to, all the slots filled, people step up if your a traditional conference organiser its frightening to not know who speaking before the day but thats the point trust your attendees!

So about the controversy at WordCampUK

To sum up, a suggestion was made by Jane who represented both wordcamp.org and Automatic the company that the UK had reached the point where it perhaps had reach enough mass to break out into regional and city level camps much like Barcamp has and if this was to happen a central camp maybe harming this, or creating a hierarchy. I had left but it was quite disgusting watching the tweets and for Jane to describe herself as near to tears is just not acceptable. That said, let's take a look at the Barcamp model, there have been dozen of barcamps every year here in the UK, London is the biggest, some appear, some go, but year on year they have grown both in number of camps and in number of attendees. Each barcamp is organised independently there are "campers" who will be found at most of these volunteering and helping out, but their is no formal structure organisers just pick it up and run with it.

In contrast their are lots PHP user groups, PHPLondon use to host a conference it was nice, then PHPNW ran a conference, suddenly Londons conference became PHPUK it was a mess, this year wasn't much better. There was no need to attach the UK monkier, or try to get a larger venue, from the outside it just came across as vain and silly.

WordCamp is a great idea, what's more there is plenty of sponsorship out there, really if someone wants to run WordCamp Leeds (and I would love to do that) I don't think it should be looked down on, if it only gets 8 attendees (and I do believe that there are more users then many give credit for) then so what! If they learn something new, enjoy themselves and go away having had a good time what does it matter if it was small?

As for the UK one as an outsider it doesn't appear to work it came across as having ideas of grandeur which fell flat, focused on a small organising team who have done a hard job but really saying it's time to discuss disbanding or rebranding is not a bad thing it just proves their success.

One thing I find about a barcamp is that its a greater leveler, they work best when egos are left at the door, and you roll up your sleeves and muck in, maybe thats the lesson that needs to be learn't to that end if their are Wordpress users in Leeds and the surrounding area, let's talk

Idle fact of the day: over a 3rd of the web is porn, according to a study by Optenet.

That's quite a lot and its increasing. So. imagine if you ran a popular membership management software for Wordpress and you want to segment your adult industry specialists in your mailing list to better deal with their "needs". Oddly, it's not something we ask at checkout!

So How Do You Find Out If a Site Has Porn?

Well, you could use your eyes, I guess, but when you have a couple of thousand URLs to process it's going to take some time. You would also need to visit future sites, so viewing each site is not really feasible. Instead, the answer is clearly NLP and brute force checking. Don't look at me like that, thats clearly the answer!

Keyword Extraction
There are 3 broad types of keywords extraction analysis:

• Statistical
• Linguistic
• Mixed

Statistical - Instead of looking the words themselves, this method looks at aspects like frequency and the position of  the word in the text. These are probably the most common method of analysis, in part because it is easy to program and relative good results.

Linguistic - Looking at the actual words, the structure, and the parts-of-speech. The benefit of using such methods is a greater accuracy in identifying individual words and collections, but not necessarily any major advantage in terms of results. Indeed, linguistic methods are rarely deployed on their own.

Mixed - This method uses a combination of statistical and linguistic methods to identify of collection of words. It has a higher degree of accuracy than statistical methods, but it can be combined with term frequency and inverse document frequency to produce reasonable results.

For most of my projects, I rely on pure statistical methods for extraction, combined with a stop list to prevent 'this', 'the', and 'a' turning up as principle keywords. However, for this project, I chose to cheat.

AlchemyAPI

AlchemyAPI describes itself as a suite of content analysis and meta-data annotation tools. It's a pure REST based system that is quite cool and free, even for commercial use. One of the methods it has is Keyword / Terminology Extraction, which it describes as 'sophisticated statistical algorithms and natural language processing technology'. So, using mixed methodology on a basic level, we can assume some grouping of words, but the main part of the API is still using term frequency combined with a stop word list. You will notice AlchemyAPI produces two word terms moreso than single words.

Keyword / Term Extraction: Web API

To use the API, you need an API key, and that's about it. A simple post based curl request is needed with:

http://access.alchemyapi.com/calls/url/URLGetRankedKeywords

• apikey
• url
• outputMode

Be careful to capitalise the M in output otherwise you get xml returned regardless.

The return is basically a list of keywords found in the text and a "relevance" score. For example, on my blog homepage:


[url] => http://www.timnash.co.uk/
[language] => english
[keywords] => Array
(
[0] => stdClass Object
(
[text] => Continue Reading
[relevance] => 0.943294
)

[1] => stdClass Object
(
[text] => search engine
[relevance] => 0.629734
)

[2] => stdClass Object
(
[text] => Search Engine Optimisation
[relevance] => 0.610243
)

[3] => stdClass Object
(
[text] => twitter spammer
[relevance] => 0.599565
)

[4] => stdClass Object
(
[text] => ranking factor
[relevance] => 0.583604
)

[5] => stdClass Object
(
[text] => Information Architecture
[relevance] => 0.551172
)

Now, as you can see, it's picked up 'Continue Reading' as the primary topic. Oh well, guess I need to do a little optimising on my site. My own name came in 6th with a relevancy of 0.541318! Still, as a simple tool it will fit our purposes nicely.

Building Our Check Word List

Now that we have extracted the terms and associated some sort of scoring with them, it's time to build our check list. This acts almost like a reverse stop list and becomes the list of terms we are focusing on. We'll assign each term a rating of how associated with porn it is.

At this point, you could introduce some machine learning into the mixture to dynamically update the otherwise arbitrary value associated with each term. However, in this example, it's a bit of an overkill.

Once I have the list, it's a case of processing through for each topic and comparing them to see if it has words in our check list. Then, we'll need to perform a very simple fomulae:

porn likelihood = Relevance * pornyness

The total 'porn likelihood' for all topics is added up and a final page score is derived. If it exceeds a certain value, the page is considered porn!

To make things interesting, I initially gave the system an arbitrary starting point for determining when is a page porn based on testing against a small set of 10 porn sites and 10 sites like my blog and BBC home page. The system stores the average score of all porn pages and uses that as a modifier to dynamically adjust the threshold of what is and isn't porn.

Classification Of Porn

@tnash im proud of you mate..... can it determine size and species?
http://twitter.com/seoidiot/statuses/16926576279

Was what greeted my announcement that my porn bot was having more fun than me, but within reason, the topic keyword indicator can be used to broadly identify sites by associating keywords with a category. You can then use the relevance score to determine which category the site is most likely to fit in.

If you really want to, point it to a site with an alternate species. I'm fairly confident it would pick it up and place it in beastiality. One of the interesting problems is when a site discusses porn, such as this post. In such scenarios, it would return a false positive for our little bot, which was only looking at a single url instead of the entire site.

The obvious way around this is to use a full crawler. You can use AlchemyAPI for this to along with a combination of Content Scraping and xpath. We'll use it to pull all links on a page that cross link on that page and slowly build a score up over time. If it's an anti-porn site or therapist site, it will still throw up false positive, but for a site like this one, it should balance out.

So, there you go. A simple way to classify large sets of sites to determine if they are porn, without burning the eyes out of your sales team, or destroying the innocence of your intern.

Not quite sure why they feel the need to check the accuracy of my system though....

First of all let's get the big announcement out of the way...

Days till oauthpocalypse

August 16th 2010 is oauthpocalypse. Please note: the powers that be discovered you can't have an apocalypse during the world cup because apparently, no one notices (the old date is 30th of June). Also, for those panicking, unlike the original plan, the guys at Twitter have decided we can't go cold turkey so they will start to reduce the rate limit over a few days. All non Oauth/Xauth apps will die after August 31, unless something else catches our attention.

World Cup Killing Twitter?

Who would have thought kicking a cows bladder around could cause so much trouble? It would seem the World Cup is mostly to blame for Twitter's current fail whale issues. In response, today's second important public service announcement is: For the time being, applications will be subjected to variable rate limits!

What Does This Matter?
Well, normally an Oauth'd user can make 350 calls/hour, but during the high load time (so I'm guessing between 3pm - 7pm BST) this limit maybe dropped, indeed as this post launches the current rate limit is 175 for all.

Now, this is an emergency measure, and for it to have any real effect, Twitter clients have to start adapting and dynamically reducing or increasing their calls.

Three ways to tell if your hitting the rate limit:

• 400 HTTP Status code - means you hit your limit
• X-ratelimit- Headers sent in return to all rate limited API calls
• account/rate_limit_status API Call

Now, your average PHP developer making CURL requests will start using number 3. I know. It was my first thought because you would save having to parse through headers. However, there is an obvious flaw with this of course of action.

The whole point of this exercise is to reduce the number of calls against Twitter. One developer I was speaking to yesterday used a rate_limit API call after every action. In effect, he was definitely not reducing the number of calls. He wa doubling them!

Developing Sensible Twitter Spamming Patterns

If you're like me and run intensive Twitter clients that send out hundreds of tweets an hour and monitor lots of accounts etc, you are going to get caught, if you're not careful. Repeatedly sending tweets to 400 HTTP will result in blacklisting, so this clearly is not a solution. Now, I'm assuming you are using php/CURL for the rest of this article as (no offense guys) you are most likely the people causing problems.

First Of All Stop Dead the chance of blacklisting
Really, you want to completely stop your twitter client's activities the moment you receive a 400. So, after the

curl_exec($ch);


and before we look at the response message, we need to know if everything is ok. This is particularly important if your application is threaded or on a 1min cron job etc.

$status_code = curl_getinfo($ch, CURLINFO_HTTP_CODE);

if($status_code != '200'){
//some thing has gone utterly wrong
die();
}

Now, obviously the above just kills the current process. You will want to switch off the tweets. I use a simple on/off switch for my systems that allow me to kill twitter apps from posting dead. Then, I use other techniques on top. This is normally a tiny int stored in a db or memcached. It is fine except for an obvious problem...

When Do You Restart Your Spamming?

At this point, you would normally interogate the header for the rate limit information, except doh! The 400 doesn't have any! However, the good news is the account/rate_limit_status calls is still available to you. Make sure you re pass your Oauth credentials when calling, otherwise you will get the IP rate limit back. This call, you will be pleased to know, won't count against your rate limit.

The two reset values, reset_time and reset_time_in_seconds, will give you the time to resume your spamming. Add at least 30 seconds to that before you start.

I would suggest using reset_time and storing it against your dead switch, or indeed as your dead switch. So far, it's fairly simple, but this is very much reactive management, not proactive. If you are sending out large amounts of data, you will want to have a high delivery rate, and the only way to do that is traffic shaping.

Managing the Twitter API Yourself

Assuming you are on the edge of the API limit, you are going to have a clue as to how much data you will need to pass in a given hour because an application (IP address) is limited to 20,000 calls. While it hasn't been ruled out, this will be reduced. The current rate limit adjustments seem to be aimed at clients, who are limited to a maximum of 350/hour.

This means an IP can have 57 clients using the maximum limit per hour (unless it's white listed).

Now, if your clients are receiving and not posting data, there is a very simple way around the rate limit: increase the number of clients. As one client gets close to its limit, it hands over to the next client, and so on. I think I can hear someone screaming at me now, and of course this is not a nice option for Twitter, but an option none the less.

If this is not really an option, you will end up having to limit your posting schedule. Most of my applications target a UK audience, so it "posts" most often between 7am and 6pm GMT.

I'm also taking several additional steps with high posting accounts. Because I want people to see information, I am currently avoiding the dreaded Whale Zone from around 4-7pm BST, I also try to get most of the posts out prior to it. This means the account is putting a higher load on Twitter during early afternoon, but is vastly reducing that load during Twitter's peak and during the night.

You will notice a peak around 10pm. This is due to pushing off last high priority tweets.

Now, ahead of traffic shaping, I have been working on equalizing tweets, basically this means more constant tweeting, with "low" value tweets being fired off at night ,and high value tweets during the day.

Dealing With World Cup Matches

Normally, the above is fine, but what about world cup matches? Initially, I was more worried about the run up and post games (both show high spikes), but a match itself causes a fair amount of traffic. Therefore, I reduce my tweets during this time. Gareth Poole has provided World Cup data on fixtures and matches as sql and json. If you are taking matches into account, I suggest you take 15 minutes before kick off and 30 after as reduced times.

What if you do all this

It can still go utterly wrong as demonstrated above. However, you will just have to live with it, (That page scrolled through a lot of whales oops )

While it is a lot of work, the above will save you in the long run. With Twitter having high loads, its error reporting will get flaky. One issue I have been having recently is lots of duplicate content error messages caused by posting at times when the API is on edge of breaking. The best way to avoid such things is to simply not post at these times, which is why I posted this post at oh 16.31 BST sorry Twitter.

Are you using Twitter on the edge of the API limits? What is your experience with doing your own API shaping?

The World Bank has opened up an awful lot of its data. Here are some of the interesting stats from their “World Development Indicators”. There is a caveat, however. The data is out of date since the 2009/2010 data has yet to be released.

Here are some examples from the 2008 version:

Top 10 Countries by Number of Patent Application (by residents)

1. Japan - 330,110
2. USA - 231,588
3. China - 194,579
4. South Korea - 127,114
5. Germany - 49,240
6. Russia - 27,712
7. United Kingdom - 16,52
8. France - 14,743
9. Canada – 5,061
10. Spain - 3,632

Top 10 Countries by Internet Users (Per 100 people)

1. Iceland – 90
2. Sweden – 87
3. Netherlands – 86
4. Denmark – 83
5. Norway – 82
6. Finland – 79
7. Bermuda – 79
8. Luxembourg – 76
9. United Kingdom – 76
10. Switzerland – 76
11. United States – 76

Note: The numbers are rounded. See data.worldbank.org for exact figures. Also bare in mind that was in 2008

Top 10 Countries by cost of Mobile phone subscription plan

This could be used as an alternate to Big Mac index and  potentially for subscription services, which require bandwidth usage. I want to note that countries at the top of the list are there principally because the infrastructure cost is being passed on to customer.

The plan is defined as “which includes the cost of monthly mobile use for 25 outgoing calls per month spread over the same mobile network, other mobile networks, and mobile to fixed-line calls and during peak, off-peak, and weekend times as well as 30 text messages per month.”

1. Brazil - $37
2. France - $36
3. Switzerland - $36
4. Spain - $33
5. Japan - $32
6. Australia - $26
7. Portugal - $26
8. Greece - $25
9. Venezuela, RB - $25
10. Austria - $24
11. New Zealand - $23
12. Cuba - $23

For those interested the UK came in at $20 while the USA $15

Why is this data interesting?

Well, it's the same data one of the largest data mining groups use to determine how solvent, economically viable, and wealthy a country is. However, it's also a great snapshot of life in different countries. While it may have no direct use in eccomerce, it can be used to weigh several factors. When entering new markets, determining cell phone costs and comparing them to how much money people have to spend will ultimately tell you how big a market it is.

One of the more interesting sections for me is 'Economic Rankings'. It's a fascinating place for anyone looking to start, run and close companies abroad. Each country is ranked by various factors ranging from how easy it is to set up a company to how hard it is to sue people. This is actually a whole microsite in its own right.

Getting hold of the data

All this data and hundreds of other indicators and economic data is available from data.worldbank.org. They have an API as well as XML, Excel and various other data types including RFD. I can’t wait to see this data being used! Together with the national data sources for the UK and the USA, its never been easier for data lovers to get their fix!

I wish to pose a simple question:

How would you determine the price of a product or service for multiple countries?

• You could go through each country and work out a value.
• You choose a base currency and do real time conversions.
• You could use comparable item counting.

Each method has merits, but they are reliant on a lot of manual and repetitive work. They may also cost you when the method fails to take local conditions such as taxes into account.

Work out individual value - This is a long process, but probably the best when working with 1 or 2 currencies in familiar markets. For example, I'm likely to set Pounds, Dollars and Euros because I am familiar with costs in the US and in Europe, but I'm not so sure about prices in South Korea, for example.

Real time counting - (Determining a price in a default currency and converting it into alternative currencies on the fly.) This method has several issues: the price will change constantly, people are nervous of products with constant price changes, and it doesn't take local taxes and other costs into account. Customers in the EU, for instance, wouldn't be charged VAT.

Product Counting - (Finding a product similar to your own and using its price in each country to set your own.) The problem is that the product has to be the same in each country, and it doesn't account for import costs.

So how do burgers help? Well, let's start with an example:

I'm running a subscription site in multiple countries, and I want to display subscriptions to people in their own language and currency.

Identify the costs per country

For each currency, identify any additional costs such as sales taxes, shipping , and any additional costs associated with selling products or services in a country. If it costs you more to supply a country, be sure to include it in your price.

Choose a stable currency as your base

This does not have to be your country's currency, since it's just something to work from. For instance, if you use the Big Mac index, the default currency is dollars, so you may wish to use dollars. Or, you might choose the currency you do your accounts in.

Define the default currency levels

Next, work out the subscription levels in your default currency. Lets say we use $10, $20 and $30 as our levels. Obviously, how you determine the initial price is up to you, but I suggest testing multiple prices before settling on one. Also, remember when choosing a currency conversion system that you can override it when you need to.

Start counting Big Macs

No, seriously! A Big Mac is a perfect comparison tool. It's made of the same ingredients in every country, and is competitively priced. This means that, while McDonalds makes a profit on each burger, it's carefully calculated to be roughly the same % in ever country. For a list of the costs of a Big Mac in every country, you'll need a subscription to the Observer, or you can go to www.oanda.com/currency/big-mac-index

source: Economist

Our next step is to count how many Big Macs are in our subscription. So, a Big Mac in the US costs $3.57 so 10/3.75, and we would need to multiply it by approximately 2.8 to equal our subscription. So, to determine the relative value of a $10 subscription in other countries, we find the value of a Big Mac and multiply it by 2.8.

Lets do a simple example: A Big Mac in the UK is 2.29, and when we multiply that by 2.8, it would be 6.412.


function bigmac($value,$cost,$rate){
return round((floor($value/$cost) * $rate),0,PHP_ROUND_HALF_UP);
}


For PHP users a quick function where $value is default subscription, $cost of the burger in the default country and $rate is the current exchange rate. Works only currencies with similar exchange rates

The real economics of the Big Mac index

The example above is an extremely simple way to use the Big Mac Index. Economists will use the index to determine the purchasing-power parity, which was first proposed by Gustav Cassel. (Purchasing-power parity is when a countries exchange rate is determined by the price of a shopping basket from a super market)

The video from the (http://www.metalproject.co.uk/) explains how economists can compare Big Macs by country to determine the relative value of almost anything. However, this rate often varies wildly from the actual rate, which is affected by many other factors.

Always round up

Now, of course you'll want to convert 6.412 to something a little less programmed, so round to 7,13,19. You'll notice in this example that the bigger the value is, the more of a discount it's getting, and the further it will drift from market demand.

Low inflation currencies probably should be rounded to the next 5 or 0 to maintain a nice pattern.

Note: This only works for low inflation currencies, so don't try this with Zimbabwean dollars. In the example, our Pounds are now 10, 15, 20. Suddenly, the site is making a profit on the lower price and losing on the higher price. Ultimately, it works out even, so long as there is an even distribution.

What about larger subscriptions

Sometimes, Big Macs are just to small to be useful. As you saw in the previous examples, the more Big Macs you work with, the greater the rounding error will be. So, when dealing with larger numbers, we could use an alternate index.

One possible example is the iPod Index. Initially proposed back in 2007 by Comsec, the iPod has become as ubiquitous as the MP3. Unfortunately, they do not maintain the index, but it's possible to recreate it using the Apple store and some googling.

Other methods

One technique I've used is what I call 'over pegged middle'. First, you need to know the number of subscribers for each level and the total income they bring in. Then, whichever total brings in the most income is the one you do the Big Mac check against.

Next, round that figure up to a 5 or zero and match the other subscriptions by using the same ratio as the original subscription. This will cause an imbalance because the majority of users will pay more, but it will also make higher earning subscriptions more tempting.

Any suggestions for how you would work out subscription prices?

The two big releases from Paypal have been Adaptive Payments and Adaptive Accounts, Adaptive Payments is a collection of methods of payments that changes the way you do business online (at least that’s what the marketing department at Paypal said I still think they are pretty useful though) while Adaptive Accounts will allow you to generate your customers with Paypal accounts on the fly (yes you read that right) This article I’m going to focus on Adaptive Payments and in particular split payments, though fear not I will be talking about the other features over the next few weeks and months.

Split (parellel) or Chained Payments

One of the features that is going to appeal to Internet Marketers and Ecommerce developers is the Split and Chained payment feature, this allows someone to make a singular payment but the payment is split amongst several receivers this immediately opens up some amazing possibilities. With Adative payments you have:

• A Purchaser or Buyer - the party who is buying the product
• A Receiver - a party receiving money from the Purchase
• An API Caller/handler - a Party which instigated the transaction and determines where funds are sent, the API caller maybe a receiver but can't be the purchaser.

Split Payment

A singular Payment is split so the money is divided amongst one or more parties, each party is shown on the invoice, in effect while their has been one payment their are several transactions. The purchaser can see each party and interact with them, for example requesting a refund against one of the payments rather then all.

Chained Payments

A Chained Payment a singular party becomes the API Caller the purchase is made in the API caller name and the payment is then split to other parties which could include the original API caller. In this scenario the buyer only sees the API Caller in the purchase even if several transactions have occurred.

Paypal Split and Chained Payment Usage Examples

Trials and Upsells

Using Split payments a merchant can provide multiple payment cycles for different products for example:

A user visits a site and purchases the online version of a product and is also offered access to a Joint Venture Partners product as part of an upsell if he agrees a split payment is made one to the original merchant and the other to the JV partner, while the user has made just one payment his invoice clearly shows two separate transactions. A problem with one of those transaction that results in a refund only effects that transaction. Allowing JV partners to have a direct financial relationship with the customer during the merchants sales person. Transactions do not have to be of the same type, while the original purchase maybe a one off payment a JV transaction might be a pre approval subscription.

Drop Shipping

Dropshipping is where a site actually does not have any stock but purchases directly from a drop shipper on purchase. When the purchase is made the dropshipper is sent the shipping details of the client and the order as well as the wholesale price of the product plus their commission. They then ship the product on behalf of the original site. With Adaptive Payments Chained payments this can be instantly automated into just 2 payments rather then just one. When the purchaser sends payment to the Merchant the merchant then generates a split payment to all drop shippers along with sales information.

The advantage is that within Paypal every item purchased by the purchaser is passed to multiple dropshippers as one job making tracking processing a lot easier and far fewer payments.

Instant Affiliate Payments

Have a direct relationship with Affiliates you can literally split the payment, Why is this better then a monthly pay back, if their is a refund you can refund from both your Merchant account and the Affiliate account, either partially or fully! Allowing Affiliates instant access to their cash as long as they have the understanding it maybe refunded, of course you can choose to take the entire refund or indeed in part.

Is your Ecommerce system ready for Adaptive Payments?

Almost certainly not but over the next few months and years the larger e-commerce software will start to adopt this technology but by then a new breed of payment systems will be in place which can take maximum benefit. For Eccomerce systems to accept split payments they will need to create Adaptive Payments Gateway as a payment provider however to make maximum use then it will require a change in the way eccomerce systems work.

Ways current Ecommerce Systems could use Adaptive Payments?

So you want to use these right now with Zen Cart or OS commerce and similar? Well with a little work you could get something working, here is how I would implement Adaptive Payments Split Payments to notify 3rd Parties.

Let us assume certain products within your shopping cart require paying 3rd Parties, these are treated as regular products we just store their Product ID for future use along with who we have to pay and what call we need to make.

Assuming your Ecommerce system has flexible gateway system, building an Adaptive Payments gateway is very similar to building any Paypal based API gateway their are a couple of strange authentication changes meaning you will need to send Public Keys in Headers rather then in Post data. To generate the Payment details we would need to look at the Product IDs in the basket and check for third party products. Create a Payment button with the correct data and send the chained or Split Payment with the IPN return set to the cart, on return of the IPN request make additional calls as required to notify the third party suppliers, this is not unusual most carts have the ability to allow such calls after purchase for autoresponders. Of course this does require the third party to accept such calls and have some sort of headless Ecommerce system.

Why Headless Ecommerce Systems are the future of Ecommerce!

A headless system is one with no user interface, so no pretty web pages, rather its designed to be interacted with by web services. Think of them as listeners when data is sent to them they process the data and return if the transaction is completed. When combined with split payments these systems can be used to automate the purchase and delivery of digitial products or order of physical product without the user ever having to go to the companies own web site in effect allowing them to expand beyond their own user base. With Headless Eccomerce Systems a web site procurement program can be added directly into other peoples systems.

Examples

Membership site - A headless Application manages membership to a site allowing site owners to subscribe users to the site, after payment the API generates the user with subscription appropriate to whatever was purchased. The Membership can be integrated directly into the original merchants checkout process as a cross sale.

Printing on Demand API - A headless API which accepts print on demand orders from Publishers who as part of their upsell process offer hard copies of their book, on purchase the Publisher makes a split payment with the royalties going to the Author, Printer and themselves, after payment is complete they call the Printers API to fire off the print job.

This was a quick introduction to what split and chained payments can do as well as introducing the idea of headless ecommerce systems, over the next few weeks and months I plan to show more and more of the ideas as well as more practical examples of usage.

Passive Profiling

From the moment a visitor enters a site a wealth of information is provided by them including hopefully where they have come from, their browser, their physical location (of their ISP at least), their language all of this can and is faked by a small minority but for the vast majority this information can be treated as reasonably accurate. Even with this limited information we can make educated guesses about a user for example we can guess if the user is at work or not.

Once the visitor starts to interact with the site we gain even more information for example once a suitable number of clicks have been recorded we can start to use behavioral models to attempt to estimate gender and age See Post notes for details. Return visitors provide even more information how do they return direct or through the original referrer? All of this builds up a picture which helps to determine how to present the site to the visitor and change them from a visitor to a “user” but this sort of profiling requires the user to stay on the site long enough to profile the visitor. 10 clicks may not sound much but that could easily be a completed transaction or a lost sale before passive profiling has had a chance to determine the best route for the visitor.See Post notes for details

Active Profiling

Active profiling in this context is confronting the visitor into a decision making process rather then allowing them to wander around the site on their own. This sort of profiling is not without risk people baulk at being told what to do and especially when a site does not operate in preconceived boundaries of how a site should behave. For many people the website asking questions just isn't cricket!

The key to active profiling it to force the user into making a decision which can then be profiled this is usually done by removing navigations or hiding content behind some sort of form. By giving the user no real choice (other then to bail) but to tell us something we can then use to show them the next step in our profiling. Let's take a really simple example if I am selling software I may want to identify larger corporate customers from small businesses I therefore present visitors with a screen with two options one for small businesses and one for corporate customers. Each leads to a different landing page selling the same software but highlighting different features of the software simple and pretty un-invasive we are adding a step between getting the information and this will increase the number of exits at that point but if those that stay convert higher percentage it's worth losing a few visitors to gain the information needed to improve conversions.

Pros of Active Profiling

• Gather information that you just couldn't gather through other means
• Confirm information gathered through passive profiling
• Can be introduced directly into visitors funneling process
• Much quicker and more reliable data then passive profiling

Cons of Active Profiling

• Higher bounce and exit rates
• Can cause confusion to visitors

The best method is of course to combine the two methods to gather as much information as possible while keeping the active profiling limited and filling only vital gaps.

Landing Pages - Case Study

Lets take an example of a site selling laptops to households it's target demographic is 18-30yrs olds in the UK.

On arrival to the site notices the top of the site has a loading icon in reality the site is well optimised however the site is generating the new arrivals profile to put them initially into one of 6 groups. Male/Female/Neutral UK/Not UK to do this the page is pulling the IP address to confirm country if not able to identify assume not UK, and using css history to determine gender (if this fails pesky firefox user! Then gender neutral)

Once the initial profiling is generated the assets are loaded a “British male” is presented with a female “Kirsty” , “British female” would get “Craig” while other would get “Barry” each have certain charms designed to entice and more importantly distract the visitor similar but more generic characters appear for the other 3. These characters then ask a question with four choices (this is our active profiling) Once loaded the characters have a hover element track on them to help determine how distracting the characters are being so that on future pages they can be brought to the fold or are pushed back into the background. The only two pages where the visitor is not determining the character position (though of course the visitor has no idea they are or it would ruin the effect) are this initial page and the last checkout page where they are always at the front almost egging the user on.

Once the user has “answered” the characters question they are further subdivided into groups from this point the wording, colours and location of buttons are determined by the profile they are in. In effect it determines if “Kirsty” talks tech to you or blue is your colour!

This sort of segmentation is highly effective if you already have a tight demographic you are targeting.

Preventing Comment Spam and Payment Fraud – Case Study

Something we have been working on for a little while is the way to identify users that are most likely to be spammers prior to them actually leaving the comment rather then just relying on identifying the comment as spam. For truly automated spam this is pretty easy to do, for human spammers (those people being paid through systems like amazon turk to leave comments) it's a little harder to identify but not much. By using a first past post point scoring system so no one issue will cause the visitor to be considered a problem you can build up a passive profile and compare it to a typical spammer who will have a specific click pattern, hover over search words and many using blocks of known IP, or indeed from a certain country in some blog cases. Once the passive profile has been matched the active profiling is engaged and forces the user into a simple question “Are you a spammer?” ok so not quite however when the visitor makes a comment rather then an immediate submission the user is presented with a logic puzzle or simply a question this is not to prove they are human as we actually know that already! But to see if they have engaged with the site or the post. If they pass through this active profiling then the normal spam prevention kicks in, if they don't then they can be subjected to what ever torments the webmaster has in store for them (a java applet is always cruel).

Similar techniques can be used to identify potential fraudulent transactions where users passive profiling might indicate their behavior is “suspicious” the active aspect would be used to reiterate part of the transaction process.

Over using Active Profiling

This post was inspired when my friend Andy Beard pointed me to a marketing tool which uses active profiling (though being marketers they added 400 additional buzz words) via a quiz to generate sales pages in what you could consider to be traditional quizes. When he initially pointed me to the program I was presented with a quiz asking quite leading and obvious questions before I could “see the video” after the second question I simply started randomly clicking until it went away it was only then did I realise the thing that was irritating and annoying was in fact what Andy wanted to show me!

So why did it fail? Well over use of active profiling results in “oh get out of the way” mentality where the user blindly clicks or feeds false information to just get rid or past the profiling system. This is a triple disaster you not only haven't gathered useless information, you don't know you have gathered duff information (unless you are confirming passive profiling and are double checking for errors) and you have an upset visitor more likely to bail. The solution is to use active profiling sparingly and only as part of a funnel sales process.

Do you use any of these techniques? If so have they helped?
If you have found this article interesting you might also like to take a look at my recent introduction to this type of user profiling.

<

Heat/Click maps

Most readers will be familiar with these but if your not basically a click heat map measures the number of clicks occurring on specific parts of a page (normally links) and then creates a pretty density match the more clicks the brighter the colour (normally they look like heat maps hence the name) they are used to see areas of interest for users and where users are leaving. These maps are really quite useful if you know how to use them and you can easily get started using open source software like Clickheat and there are also paid for services out there.

Temporal Click Density maps

So I'm sure the title (time based click maps) has given away what our current project is all about, you see clickmaps give a very two dimensional view of click density by introducing time, we get a much better picture of what areas are being clicked on and some motivation behind them. Unfortunately introducing time into web analytics is notoriously difficult perhaps more so then it would appear on first glance.

Time periods
The first problem is accurately recording time, either you rely on the server or the clients browser to record the time, both come with problems and neither can be considered accurate the solution in this case is simply to probe in time periods in our case we have chosen a near logarithmic scale for determining time periods. It would be unhelpful if 99% of all our clicks were in our first time period because we chose to long a period and likewise we wish to keep processing and data storage overheads down so polling data every second is unrealistic.

In focus or idle?
The next hurdle is determining if they are even looking at the page all modern browsers are capable of tabs and multiple windows, and so users could leave a page open and be surfing for hours on another window never to return to your site. Likewise a user referencing your site can be coming backwards and forwards between tabs or windows. Most browsers now support some javascript focus option which will allow you limited knowledge when combined with blur. However this does require that we first access the browsers capabilities . In reality we actually use compiled action script assuming the user has flash and fall back to javascript only really for iphone browser and similar.

Some Temporal Click Density Map goodies

ok so now for some totally unhelpful yet oddly cool tidbits we have discoverd while running temporal clickmaps on a couple of our sites. Remember now this is just initial findings on a couple of sites.

Greater the density the earlier the click – Afraid so from our initial finding the click area with the greatest number of clicks also had the clicks in a short period of time relative to their arrival. The positive side is a good call to action is likely to be clicked on quickly.

Adverts are clicked late – This was a little surprising I had always assumed advert clicks occurred quite early on but adverts tended to be clicked later, is this showing a subconscious desire to complete the primary call to action? Well possibly and (warning blatant plug ahead) its certainly the theory we have put forward for why newmedias.co.uk Your CAPTCHA advert wordpress plugin works (end of plug)

Further down the page the later – yep your hocked but the lower down the page the longer it takes to get clicked, there is an exception to this and that is pre known call to actions a good example being long sales pages where we saw a split between quick clicks vs long clicks. So some people realising the style of page shot to the bottom while others waited.
Interesting and this is when this sort of mapping comes into its own, those who waded through the long sales page and clicked to purchase were statistically less likely to complete the transaction

Stumblers really do hate you – a couple of years ago I posted about the fact that the most clicked area on a page during a stumble was the top left hand side and theorised that this increase was due to stumblers missing the stumble button. With the new system we can see the different density levels for the top left for stumblers vs non and with a noticable difference between the two the theory still stands to make things worse... nearly all those clicks are in the first time period meaning you really don't have much chance to impress.

So what do you need to get temporal data going on your site, well clickheat map system is a good start I recommend labsmedia clickheat php class as a starting point. Then you need to measure time and focus we use an actionscript system but I would be interested in seeing or hearing other ideas. Indeed if people want to play with our code I maywell clean it up and throw it up on newmedias.co.uk

• Your Name
• Age
• Shipping Address
• Some basic Credit details.

They can almost certain make educated and pretty accurate guesses to:

• If you are at work
• your income bracket
• Gender

They might even in a few users cases be able to tell you what you had for breakfast all in a few seconds. Scared?

We recent worked with a client who was looking to improve conversions on their alternative payment methods in this case primarily Google Checkout. They were disappointed by the take up on their Google Checkout option (with such low commission for the first year I suspect they were expecting significant savings) we were asked to develop a method to target existing Google Checkout users and present them with using the one click checkout option in favour of the traditional credit card approach.

Solution

We ran a very short questionnaire, of sales made to ask a simple question:

Do you use any of the following Google Services (tick as many as you use):

The results were pretty staggering 55% used one of the above services with Google personalised homepage making up the majority followed by Google checkout!

Making it worthwhile
ok so we now know there is a feasible amount of users accessing Google services and a reasonable (12.4%) using Google Checkout to make targeting while 12.4 of participants to the survey we can normally add an error margin of + or – 2% which means theoretically 14.5% (approx) could be using checkout thats not a small majority. If we could up the number of completed checkout by even a few percent it would make a difference.

The idea was fairly simple instead of presenting a Pay with Credit Card button with a google checkout button as one of the alternate smaller payments, we would show the Google checkout button as the primary option and pay with credit card as the alternate.

This idea could backfire, users may have made a single one off payment with Google Checkout and absolutely hated it! We have no way of knowing we maybe suggesting something they loathe but lets for now assume that paying via Google Checkout is the preferred option for all those who have previously used checkout but if its a new user how do we know?

Iframes, Javascript and the DOM

Using just Javascript to do this sort of XSS is no longer really viable, however there are plenty of alternates using flash and PHP proxies.

If I was to get you to click this link https://www.google.com/accounts/ManageAccount?service=profiles&hl=en chances are that it has just opened your Google profile. Why is that? Well mainly because the people reading this are likely to use one or more of Google services with a fair few using Gmail. Now whats the first thing you do when you get online?

Ok so this page is going to be accessible via javascript for a lot of users, to access it we simply open it in an iframe. We then can use javascript to pull data out, by calling the document in the iframe using something similar to:

window.frames['myframe'].document

where myframe is the id of the frame, once we have access to the document we can use a range of techniques for extracting data from getelementbyid through to implementations of xpath (since Google can't manage nicely designed web pages the later is needed)

note – Its worth remembering this sort of implementation is subject to the whims of the site your grabbing the data from! If they change the page layout you will need to match it. Also you are reliant on the user a) being logged in, b) their browser playing fair, each browser handles cross domain requests differently for example IE is a lot more strict then say Google Chrome the above code will not work on most browsers without some modification

If you take a look at the above you can see the section marked default payment you will forgive me if I have removed some personal details but it shows the default Card type and last 4 numbers if any card is associated with the Google account. So all we need to do is check to see if there is a 4 digit code there if there is then we can assume the user is a Checkout user if there is not or if we couldn't log in then we assume they are not.

Once they have made their first purchase we can ask them to create an account and allow them a preference check as to how they wish to pay.

We have only been running the new system a few days but the initial data is positive sadly we can't do a similar thing for Paypal and would need to rely on CSS history trick (see below)

All your data is mine

Lets go look at that Google account page again, I'm pretty frugal with my details yet on that page there is my Full name, email address, a shipping address (probably my home even) if you go down to the services you would have quickly realised that I use this account for work as some of my services include Google Web Optimiser, Adwords, Analytics.

So what about other sites and other information for example...

MSN / Yahoo
Similar to Google both have a central account location lots of information to grab should you choose to.

Facebook

http://www.facebook.com/home.php?ref=home

Gives basic login info including full name and their Facebook ID via the profile link which allows us to access:
http://www.facebook.com/profile.php?id=FBID&ref=profile#/profile.php?id=FBID&v=info&viewas= FBID

Which gets you the users complete profile, dates of birth, mobile phone numbers etc etc, oh and all their friends information to (well what they shared with your original victim sorry customer)

Twitter
I did say find out what they had for lunch, while Twitter information can be obtained through the Iframe method its much simpler but my friend TheHodge pointed out an alternate method utilising a bit of javascript and Twitter API the downside to this is that like the rest of Twitter you are reliant on the thing being up! Still I did promise we could find out what some people had for lunch and what better place, once you got the username, just pull in their feed and mine.

Profiling and Data guessing

Once we have all this data we can start making some guesses and profiling users, we could go the whole hog and do credit checks after all we have name and current address but in the UK that would probably abusing your credit license plus time consuming etc. We could also pull details from the land registry on their property and then estimate an income bracket, or we could look up their job description. I can't find the study but the BBC ran a report a while back claiming people were less likely to lie about their Job title and description on facebook then on other sites such as LinkedIn etc always worth knowing.

Identifying if some one is at work
This is pretty easy, verify they have a job, check current time in their location is it between 9 and 5? reverse DNS on their hostname check against a list of common terms such as residential etc if their ISP shows no sign of being primarily for non commercial use, its within normal hours of work and they themselves do work its a reasonable assumption to assume they are at work! Of course they might not be but chances are they are.

What if they use a secure session?
Some sites like Paypal won't let you stay logged in so you can't grab information from the browser window unless they happen to be using Paypal within a few minutes of visiting your site. If we want to know if they have visited Paypal we will have to go down an alternate and less reliable route.

CSS Browser History

Your browser can identify visited links, so when a page renders it knows when you have been to certain pages before. Web designers can even style these visited pages with a different colour link, which means its equally easy for us to identify where a user has been. The first step is to generate a list of possible locations, then create a very specific visited colour. When the user visits the page it will show the other pages they visited in your specific colour, we then run a piece of script to identify elements with that specific colour on the page and we now have a history of where you have been.

Note: This is flaky and I mean really flaky I have used the term URL but I should really have said URI it is the exact location you have been for example http://www.example.com and http://www.example.com/#hello are not the same and in some browsers (such as chrome they would not both show in css browser history unless you visited both)

For more information and for the original script that see the Ha.ckers.org CSS History Hack

Final thoughts
If this has worried you a little then remember the solution is simple, log in and out of your sites, if you must stay logged in use a different browser (or in Chrome and new Firefox user their privacy mode)

However for marketers the ability to know as much information as possible about a user is gold but only if you can properly utilise it also keeping in mind data protection and retention regulations means a lot of the data you may find you can't keep so can only be used for one off judgments (such as assigning a profile group, or changing checkout info around)

So now I will leave you with a simple question, how much are you worth? Your data I mean how much do you think all the data your exposing is worth?

Why make a browser

Its a simple if some what open ended question if you are a large search engine why would you invest time and money into a browser. Stand back from one of the cleverest “oops did we leak that?” launches ever and ponder just how Evil is Google?

Google just made millions from people like you!

It might be a drop in the ocean, but the browser development probably didn't cost a great deal to scale from an internal system to a large project except a small team and plenty of man hours but for Google man hours are cheap, However even a drop needs to pay for itself, so how does an open source browser make Google millions overnight?

Well who is going to adopt Google Chrome first? Let's see is it Gran in her home just firing up the big blue E or perhaps the web developer with FF 3 and perhaps more crucially pimped out plugins?

What is one of the single most adopted plugins in the world for Firefox 3? Still don't get it ok spot the difference from fellow northern SEO bloggers DaveN post on Google Chrome!

I didn't even know he ran Adsense on his site but in surfing the web with Chrome in the last hour I have clicked by accident on at least 2 Google ads. Think about that for a second? Suddenly you have 100's of thousand if not millions of people suddenly being re-introduced to Google ad's having to remember what the things look like and pay attention to where you click less you wander into an Adsense trap. But if you think it's us poor chrome users that will suffer you are mistaken, PPC campaigns in technology niches will no doubt see a steep rise but will their sales? This strategy is bordering on genius inflated PPC see's really only Google winning and everyone else paying.

But that doesn't answer the question why?
Well I don't have an answer, I do have a thought of why I would do it, but first let me speculate. A while ago I saw a rather unusual useragent in my logs something calling itself GoogleBot but coming with a webkit user agent string.

Total aside but for your records Chrome user agent is: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/525.13 (KHTML, like Gecko) Chrome/0.X.Y.Z Safari/525.13.

I didn't think much of it and it's still just as likely to be a Mac user badly spoofing Googlebot but today it got me thinking.
I have covered Simple Block Segmentation analysis before and given a brief overview of its big brother Visual segmentation analysis at the time I focused on Microsofts research but it would be naive to think they were alone in their ideas. Since then we have worked on developing our own system using Opera which has screenshot grabbing via command line allowing batching another post to write about. But the problem is Opera renders very differently then say IE if I was a search engine I would find it much more convenient to use only one rendering agent to do such analysis (it's worth remembering that Google “new found” Flash search ability is based on similar modified browser technology) and not have to deal with 3 or 4. If I can saturate the market I can start really developing modified versions of my browser with a search crawler not a human driving.

Just some thoughts, to the tin foil hat brigade a very quick couple of checks no obvious phone home devices it's not looking like spyware, though contrary to other reports I'm finding my copy in real world tests to be sluggish.

Quick apologies for absence family crisis and so bad bits prevented blogging this post was rushed set of thoughts hopefully I will explore them more fully in future..