In July, the BBC launched the Dr Who streaming service. For a few Facebook credits, non-UK residents can watch a select set of Dr Who episodes for up to 72 hours. (This is just one example of how the garden is changing.)

When the Beeb launched their service, Twitter and Facebook exploded. Why? It’s the BBC. Naturally. everything they touch has to be investigated in minute detail, discussed on social networks and in the wider media here in the UK. It’s also a change of direction for the BBC, which is now relying on a 3rd party. Indeed, it opened a debate on should the BBC be in part paying a third party to put content in a closed garden?

Over the weekend of the launch, I had to stifle laughs. Friends were explaining how much it cost, how they wouldn’t make their money back, and most importantly, why the BBC would give 30% of money to Facebook!

You see, as the technical director and part of the team at Coding Futures, I’ve helped several major clients launch large Facebook applications. We had also been working on a way to take that knowledge to put it into our retail software, Your Members.

The Your Members WordPress membership plugin leans heavily on developments coming from our work with enterprise clients. Over the last few months, we have been building a feature-rich Facebook addon that allows you to create applications, protect content and pay for it with multiple gateways, including Facebook credits while in Facebook.

What’s more, the same software already has a streaming addon, which allows you to securely stream video from Amazon s3 via progressive download, or create an Amazon Cloudfront distribution and do real RTMP streaming.

Now, you would expect the Beeb to use it’s massive resources to hire a development team like Coding Futures to create something unique. They didn’t. In fact, they could have created the exact same thing for less than $200 using Your Members. Which, while great they have helped prove a concept, have they really pushed the boundaries?

Now, basically anyone could create their own pay-to-view Facebook video service. If you want to give it a go, here is a step-by-step tutorial for mimicking the BBC Facebook application in Your Members.

Of course, to use Facebook, you don’t have to pay them 30% of earnings. That’s only when using Facebook credits. You don’t need to take payments at all, and for the moment, only “games” are required to sell via credits, so other applications can use different payment methods.

Which leads to one of the more interesting concepts to developing within Facebook: the “likewall”. Just like a paywall, content is protected, but it only costs the user a single click to access content rather than a financial payment. For digital brands and ad agencies, the idea of a likewall is only just taking off mainstream, and it will be interesting to see how Facebook deals with incentivize likes. Currently, their terms and conditions on the subject are woolly at best.

So, the obvious question is where to next for the BBC? Is this first foray their last? Or, will we one day see iPlayer inside Facebook? (For non-UK peeps, this isn’t some strange apple device you haven’t heard of, but rather a BBC streaming service.) Probably not. If only because the people in the BBC working with Facebook are not the corporation we know here in the UK. They belong to BBC Worldwide, which is basically the BBC’s non-UK commercial arm.

So, if not iPlayer, more classic streaming episodes, perhaps. Certainly, if you can get over that 30%, using both Facebook Credits and another payment gateway could be one way to save costs (plug: Your Members can do that ), and in turn, make this an effective medium for any company with media content.

Another interesting possibility is organisations like Open University putting their online course material on Facebook for their students. And not just big organisations. Suddenly, anyone running an online course could have them available to individual users within Facebook. How about an online course on Facebook in Facebook, very meta!

For many people today, Facebook is the web. It’s application platform looks ready for prime time, and Facebook Credits are starting to be used regularly, even by non-farming related gamers. Is this the time Facebook marketing really grows up? Is it possible to run a Facebook-only business?

Should the BBC put money into a closed garden like Facebook, given that large companies owning closed gardens tend to start imposing more restrictive rules (cough apple)? Can any business truly survive by operating solely in Facebook? While it’s certainly technically possible, do you think it’s feasible from a business perspective?

I think it is, but having watched businesses been burned by similar platforms (Twitter/Apple/Microsoft) suddenly changing the rules, it’s not there yet. I think, until the platform fully matures, people will continue to make small steps and build small corners of their sites into Facebook. And now that it has become possible to do so easily, do agencies and businesses really have an excuse to not take those steps?

Disclaimer: I work for Coding Futures, the developers of Your Members.

Image: http://www.flickr.com/photos/ingythewingy/5178899845/

East Coast Rail and Failing credit cards

Like many northerners I regularly travel down south and prefer to take the train from Leeds the only real choice is East Coast Main Line, gone are the days of GNER and instead we are stuck with a nasty money pinching publicly owned disaster but this is not a rant about no more free wifi (or how you can just swap your mac address to get access) or cutting stations from the map but about their website.

East coast website is the normal place to buy advance tickets and it offers up to 20% discount on advance fares which are the only way to travel at a vaguely reasonable price. Recently at Coding Futures we have had a nightmare of a time booking tickets today I decided to experiment and work out why.

Card Declined for no reason?

We would go through the ticket flow something we have done a hundred times and put in all our details, as always the billing address details would be empty (though the card number and other bits were there) and we would populate with the office address, go through the dreaded and totally insecure 3d secure and come back to be told there was an error. A few hours later the charge will appear in the bank and within a day the bank will reverse it.

Several other cards and all the same thing, so what’s the problem?

Incorrect Billing address

In frustration I contacted their web support team I’m sure it was on the paper infront of him but the poor guy was utterly illogical telling me I had used two cards my bank was linking the two cards and using the wrong address. Strangely enough my bank has not linked my personal and corporate card and it was utter tosh but it did spark an idea.

The account I had been using, was my existing account on East Coast which had in the past had my home address what if it wasn’t my bank but East Coast who was linking the cards, easy to test so I quickly created a new account added the corporate card and as if by magic it worked. So that could be a fluke so I added my personal card (with a different address) filled in the address as normal guess what came back declined.

It would appear to not be the banks linking cards but that East Coast is ignoring what Address you have been putting in during the booking process and taking the one from the first booking of your account…

Intuitive nope, but if you have been having the same issues the solution is to create a new account for your card with your new address, while we have written and suggested strongly to East Coast the problem exists I don’t imagine it to be fixed anytime soon.

I’m currently midway through a major behavioural modelling project. It’s complete with large scale re-targeting, both within the site and via advertising networks, as well as using techniques such as the CSS history hack to collect data about whether a visitor has visited our competitors.

Indeed. Looking at the sort of stats we are collecting:

Everyone coming into one of 3 sites is being tagged, using a browser fingerprint. It’s similar to the level Panopticlick project uses in their efforts to educate users on how to be on the safe side. We also use a persistent storage in the form of persist.js to “cookie” the visitor. Lastly, any third party software, which is capable of accepting custom values, has the user hash added, making tracking a user across the board as easy as possible.

To put this in perspective we can at 1 click retrieve:

• If the user has purchased
• How they arrived on the site
• A rough idea of age
• Rough idea gender
• Where they are coming from (not just country but are they at home or work)
• If they have visited our competitors
• What pages they have visited
• What offers enticed them
• If reinforcement marketing is working
• Any lead mechanisms (email/twitter etc) we may have them subcribed to
• If they are part of a focus or test group
• What ad group they arrived in
• Which split tests they have been set up with
• Where they clicked on a page and when

Basically, everything they have done on the site to the tiniest detail can be looked at, analysed and dissected. What’s more, the average user will not have a clue. That list would terrify many. I mean, if little peeps like us are doing it, then imagine what people like Google are doing. Best get your foil hats now!

Privacy at heart of Behavioural driven campaigns

One of the things that has been important for us from the start of the campaign is for our visitors to be in control, well a bit anyway. If possible, we want them to be able to opt out of our orwellian vision. The problem is how?

Removing data

Let’s assume we have a user who does want to opt out. The first stage is to remove their data. Since our system has a database, this is fairly simple. Just delete their row in the visitors table and any associated data in the meta table. Small snag: this doesn’t remove the data in third party applications and causes data corruption in the master table. Really, there is not much we can do about the 3rd party applications. Where possible, you can try to automate them, but normally the only option you are left with is giving a user a link to the application’s opt out procedures, if indeed they have one at all!

With your own applications, we have gone down the route of what we term “anonymous annihilation”. All our users are split into testing groups, and the user’s information is overwritten by an average of all those in the test group. The only data we keep exact is country. The IP is overwritten to 999.999.999.999, which makes an easy way for us to exclude the data in reporting, and their user agent finger print is reduced by us removing all the plugin data. Suddenly, we can’t tell them from Adam, except for that Persistent “cookie”, which actually is quite a pain to remove. But hey ho! That was the point. The issue is how do we not track them in the future?

Cooking the excluded

The only real way to exclude someone from an opt out system is to know they have opt’d out! But, to know they have opt’d out, we need to either maintain some information, or tag them in some way. Neither of these options are very palatable to the end user, but ultimately, at the moment, it is the only real solution. When opting out, I suggest using a traditional cookie rather then a persistent storage, clearly named within the cookie and make it clear this is what you have done. The downside, if they clear their cookies and come back, you generate a new profile and the circle starts again. But hey, you tried!

Looking to the future

Right now, there is a lot of talk about “Do not track” methods, especially amongst browser manufacturers. Google is releasing a new extension to allow you to prevent tracking (The irony sure will not be lost on them) and there is a more public discussion from the Mozilla team. Both seem to be heading down the route of the browser making the decision to prevent storage, which is great in principal, but has 2 major obstacles to overcome:

Persistent storage is all about hiding things in the most obscure places such as flash storage, where browsers do not have control. Therefore, simply assuming the browser is in control of all storage would be a mistake.

Carpet banning of data, would be frustrating and would effectively break a lot of the modern web. Cookies and storage are used in every aspect of web development, from ad tracking through to analytics, to storing shopping carts, to changing the colour of a site. Users are not going to want to be prompted every time, so they are likely to adopt an on or off approach.

One of the more interesting and hopeful projects is the idea of using headers. Proposed by Mozilla, the idea is that the client browser sends a HTTP header to the server, telling the server the user does not want to be tracked.
X-Tracking-Choice: do-not-track
It’s then up to the server to determine how to handle this. I think this is a great step forward with one major addition.

Telling a browser to send the header, I would like to see a method that allows sites to instruct a browser to send the do not track header. In effect, when someone clicks opt out, the site tells the browser the user wishes to opt out. Now, obviously, you don’t want a site to be able to opt people in, so the mechanism should be one way, and not mandatory for the browser (i.e it shouldn’t override an existing user preference).

The mechanism I propose has one major issue, at the start I explained this was a multi site campaign, but the mechanism is for only one site, and I can’t see a safe way around.

What do you think? Should we adopt Do not click header? What about the ability for a site to ask a browser to enforce it? Would other advertisers use it?

I wish to pose a simple question:

How would you determine the price of a product or service for multiple countries?

• You could go through each country and work out a value.
• You choose a base currency and do real time conversions.
• You could use comparable item counting.

Each method has merits, but they are reliant on a lot of manual and repetitive work. They may also cost you when the method fails to take local conditions such as taxes into account.

Work out individual value – This is a long process, but probably the best when working with 1 or 2 currencies in familiar markets. For example, I’m likely to set Pounds, Dollars and Euros because I am familiar with costs in the US and in Europe, but I’m not so sure about prices in South Korea, for example.

Real time counting – (Determining a price in a default currency and converting it into alternative currencies on the fly.) This method has several issues: the price will change constantly, people are nervous of products with constant price changes, and it doesn’t take local taxes and other costs into account. Customers in the EU, for instance, wouldn’t be charged VAT.

Product Counting – (Finding a product similar to your own and using its price in each country to set your own.) The problem is that the product has to be the same in each country, and it doesn’t account for import costs.

So how do burgers help? Well, let’s start with an example:

I’m running a subscription site in multiple countries, and I want to display subscriptions to people in their own language and currency.

Identify the costs per country

For each currency, identify any additional costs such as sales taxes, shipping , and any additional costs associated with selling products or services in a country. If it costs you more to supply a country, be sure to include it in your price.

Choose a stable currency as your base

This does not have to be your country’s currency, since it’s just something to work from. For instance, if you use the Big Mac index, the default currency is dollars, so you may wish to use dollars. Or, you might choose the currency you do your accounts in.

Define the default currency levels

Next, work out the subscription levels in your default currency. Lets say we use $10, $20 and $30 as our levels. Obviously, how you determine the initial price is up to you, but I suggest testing multiple prices before settling on one. Also, remember when choosing a currency conversion system that you can override it when you need to.

Start counting Big Macs

No, seriously! A Big Mac is a perfect comparison tool. It’s made of the same ingredients in every country, and is competitively priced. This means that, while McDonalds makes a profit on each burger, it’s carefully calculated to be roughly the same % in ever country. For a list of the costs of a Big Mac in every country, you’ll need a subscription to the Observer, or you can go to www.oanda.com/currency/big-mac-index

source: Economist

Our next step is to count how many Big Macs are in our subscription. So, a Big Mac in the US costs $3.57 so 10/3.75, and we would need to multiply it by approximately 2.8 to equal our subscription. So, to determine the relative value of a $10 subscription in other countries, we find the value of a Big Mac and multiply it by 2.8.

Lets do a simple example: A Big Mac in the UK is 2.29, and when we multiply that by 2.8, it would be 6.412.


function bigmac($value,$cost,$rate){
return round((floor($value/$cost) * $rate),0,PHP_ROUND_HALF_UP);
}


For PHP users a quick function where $value is default subscription, $cost of the burger in the default country and $rate is the current exchange rate. Works only currencies with similar exchange rates

The real economics of the Big Mac index

The example above is an extremely simple way to use the Big Mac Index. Economists will use the index to determine the purchasing-power parity, which was first proposed by Gustav Cassel. (Purchasing-power parity is when a countries exchange rate is determined by the price of a shopping basket from a super market)

The video from the (http://www.metalproject.co.uk/) explains how economists can compare Big Macs by country to determine the relative value of almost anything. However, this rate often varies wildly from the actual rate, which is affected by many other factors.

Always round up

Now, of course you’ll want to convert 6.412 to something a little less programmed, so round to 7,13,19. You’ll notice in this example that the bigger the value is, the more of a discount it’s getting, and the further it will drift from market demand.

Low inflation currencies probably should be rounded to the next 5 or 0 to maintain a nice pattern.

Note: This only works for low inflation currencies, so don’t try this with Zimbabwean dollars. In the example, our Pounds are now 10, 15, 20. Suddenly, the site is making a profit on the lower price and losing on the higher price. Ultimately, it works out even, so long as there is an even distribution.

What about larger subscriptions

Sometimes, Big Macs are just to small to be useful. As you saw in the previous examples, the more Big Macs you work with, the greater the rounding error will be. So, when dealing with larger numbers, we could use an alternate index.

One possible example is the iPod Index. Initially proposed back in 2007 by Comsec, the iPod has become as ubiquitous as the MP3. Unfortunately, they do not maintain the index, but it’s possible to recreate it using the Apple store and some googling.

Other methods

One technique I’ve used is what I call ‘over pegged middle’. First, you need to know the number of subscribers for each level and the total income they bring in. Then, whichever total brings in the most income is the one you do the Big Mac check against.

Next, round that figure up to a 5 or zero and match the other subscriptions by using the same ratio as the original subscription. This will cause an imbalance because the majority of users will pay more, but it will also make higher earning subscriptions more tempting.

Any suggestions for how you would work out subscription prices?

Currently I run:

• Google Analytics
• Google Weboptimiser and other A/B testing
• GetClicky
• Heatmap software
• Occasional CSS History profiler
• Surveys

If I want to track a user across all these currently I can’t for example if I want to see the clickmap of a user I can’t compare it with Google Analytical data for example. This is fine but the more work I do the more I want to be able to follow a user through the entire experience, now at this stage many people may start to think ok, reduce the number of third party software and this thought has occured to me. The reason I use getclicky and Google Analytics is I can’t do better it’s that simple.

Privacy Concerns

The biggest issue when linking multiple systems together is the inevitable extra privacy issues, while these systems are separate they are psuedo anonymous combining them makes it much easier to identify a user especially when linked with a login/commenting system where they have to give their email and other information like name. However in many ways I think centralising your data makes dealing with concerns easier to deal with for example you can set up a single “remove me from your tracking” service (also you can track how many people have opt’d out! oh wait is that wrong?) so centralising my data not only will make things easier for me it will make it easier for visitors who have privacy concerns.

Central storage area

The obvious way to centralise all the data is to create a central storage repository and put data in it, of course this immediately prevents several obvious problems.

• Replication
• Single Point of Failure
• Reliability

Replication – There is rarely a good reason in life to have two working copies of something, your analytics data included, apart from the fact you have to maintain both copies you also have to check data integrity and it’s taking up space and therefore costing more to store.

Single Point of Failure – while not normally a problem, when something is being continually used both for read write it”s life expectancy is limited made worse by the fact that several parts of the site will be reliant on the system to make choices, if the system falls over or worse is just slow it will cause issues throughout the site.

Reliability – One of the reasons to use third party services is so I don’t have to handle such things as uptime and reliability any benefit in getting someone else to do the work is lost if I then redo it.

the advantage is speed and as long as it’s up we should be able to access everything instantly.

Decentralised with key link

The second approach to look at it is linking all the various services with a common key. Most third party services worth anything will allow you to store a custom value against a visitor. If the same custom value is used per visitor for all the services then they can be tracked through various calls to each services API. This is easier said then done…

A couple of problems that immediately come to mind

• Identifying the unique visitor
• Linking a visitor after the fact
• What controls the initial identification

It also has the potential for a single point of failure of the totally centralised solution, the service that tags visitors is down the data is lost. This however seems a much smaller risk, at worse some visitors are not tagged correctly and it probably means the site has far worse problems!

Identifying the unique visitor – This at first glance seems easy but to be accurate is actually more difficult and is a post in it’s own right. Once identified the next problem is choosing a naming strategy for a visitor Id if we had a centralised relational database this would be easy it would be the id of the row but we don’t. Some ideas I played with was timestamp, IP and profile type or some combination of these.

Once the ID of the unique user is set and stored on their machine either through a session, cookie or some more hardy persist storage they can be simply picked up in the future.

Linking to a user after the fact – There can be times where a user maybe identified after a service has stored data about the individual some systems will automatically tie in the old data with the new, others won’t unfortunately there is not much you can do barring a recursive check and additions. For example let’s assume a user visits a site on a laptop from home, then visits at work. We treat his work log in as a different instance, when he logs in, we can identify this new visitor under the same user. However we have already sent a pile of custom keys to all our analytical packages.

What controls the initial identification – here is a more tricky issue in the scenario to my blog, a simple wordpress plugin that checks to see if a persistent storage or cookie is on the users machine, determines ID and adds a cookie as needed.

so two competing systems both with problems the solution seems to be a blend between the two.

Decentralised in a Centralised way

I’m going to run through two examples of the way I’m going to centralise my data, one for here timnash.co.uk and the other for a membership site.

On timnash.co.uk I’m going for a totally decentralised approach, a wordpress plugin, will identify users based on if they have been tagged before as I have no easy way to identify if they are previous user on a different browser machine, except if they comment there is no major advantage of maintaining any form of database control. Users will be tagged with a combination of timestamp+profileid+random number
This is then included as custom data to all the stats gathering packages and stored on the users machine using browsers persistent storage. If a user wishes me not to collate individual data they can opt out via the privacy page, this will place a persistent storage cookie, telling the system to not attach the key to their pages, to opt out entirely they will still need to individual drop out of each service.

For a Membership I run I plan a similar system however as it has a login system, individual browser profiles (unique keys) will be stored against a logged in user. This will allow these profiles to be linked via the username and has the advantage of spotting password sharers if their are a large quantity of browser combinations (it should be able to detect even if users use proxies or are on a corporate network)

so that’s the plan, anyone see any major issues with it? let me know, ideally before I fully build it! How are you managing your various data services?

For example if you were going to run an extortion service let’s say which at best a stupid idea and at worse criminal then putting the address of your legitimate company would be stupid. However being a smart fellow you might choose to then change it to a holding address for your mail. Unfortunately Google Cache saves the original address and well your pretty much screwed at that point and look just a little silly.

Let’s take an example, who could we pick on…

Verified SEO

Lovely site, offering to sell you a verified SEO status at just $199 a month, no wait $99 they must be having a sale. Now let’s compare two versions of their footers the current one, and this one from Google Cache

The current footer

The old one from cache

look they moved…
I wonder who else is at the old address well according to Merchant Circle a Mr Charles Preston, wonder if he has a site… oh look so he does!

Charles Preston SEO

What a nice chap, apparently according to his site…

His record of success speaks for itself. In the early nineties he taught himself computer programming and website development. Using only books and a burning desire to change his fortune, he became a certified and sought after adept at several website and database development programming technologies.

He invented time travel as well, this man talents show no bounds! It is of cause quite possible Charles Preston has been a very unlucky chap who just happened to work in the same office as the person trying to hide their real details, and if he is well think of the links, oh wait you didn’t get one, doh!

Takeaway hints:

• Don’t upset someone enough to make them come look for you
• Don’t use your business address for your nefarious deeds
• Don’t think because you have whois-guard on mean people won’t find you
• don’t rip people off!

p.s anyone wanting to become a Verified SEO please click here for your shiny new badge!

The two big releases from Paypal have been Adaptive Payments and Adaptive Accounts, Adaptive Payments is a collection of methods of payments that changes the way you do business online (at least that’s what the marketing department at Paypal said I still think they are pretty useful though) while Adaptive Accounts will allow you to generate your customers with Paypal accounts on the fly (yes you read that right) This article I’m going to focus on Adaptive Payments and in particular split payments, though fear not I will be talking about the other features over the next few weeks and months.

Split (parellel) or Chained Payments

One of the features that is going to appeal to Internet Marketers and Ecommerce developers is the Split and Chained payment feature, this allows someone to make a singular payment but the payment is split amongst several receivers this immediately opens up some amazing possibilities. With Adative payments you have:

• A Purchaser or Buyer – the party who is buying the product
• A Receiver – a party receiving money from the Purchase
• An API Caller/handler – a Party which instigated the transaction and determines where funds are sent, the API caller maybe a receiver but can’t be the purchaser.

Split Payment

A singular Payment is split so the money is divided amongst one or more parties, each party is shown on the invoice, in effect while their has been one payment their are several transactions. The purchaser can see each party and interact with them, for example requesting a refund against one of the payments rather then all.

Chained Payments

A Chained Payment a singular party becomes the API Caller the purchase is made in the API caller name and the payment is then split to other parties which could include the original API caller. In this scenario the buyer only sees the API Caller in the purchase even if several transactions have occurred.

Paypal Split and Chained Payment Usage Examples

Trials and Upsells

Using Split payments a merchant can provide multiple payment cycles for different products for example:

A user visits a site and purchases the online version of a product and is also offered access to a Joint Venture Partners product as part of an upsell if he agrees a split payment is made one to the original merchant and the other to the JV partner, while the user has made just one payment his invoice clearly shows two separate transactions. A problem with one of those transaction that results in a refund only effects that transaction. Allowing JV partners to have a direct financial relationship with the customer during the merchants sales person. Transactions do not have to be of the same type, while the original purchase maybe a one off payment a JV transaction might be a pre approval subscription.

Drop Shipping

Dropshipping is where a site actually does not have any stock but purchases directly from a drop shipper on purchase. When the purchase is made the dropshipper is sent the shipping details of the client and the order as well as the wholesale price of the product plus their commission. They then ship the product on behalf of the original site. With Adaptive Payments Chained payments this can be instantly automated into just 2 payments rather then just one. When the purchaser sends payment to the Merchant the merchant then generates a split payment to all drop shippers along with sales information.

The advantage is that within Paypal every item purchased by the purchaser is passed to multiple dropshippers as one job making tracking processing a lot easier and far fewer payments.

Instant Affiliate Payments

Have a direct relationship with Affiliates you can literally split the payment, Why is this better then a monthly pay back, if their is a refund you can refund from both your Merchant account and the Affiliate account, either partially or fully! Allowing Affiliates instant access to their cash as long as they have the understanding it maybe refunded, of course you can choose to take the entire refund or indeed in part.

Is your Ecommerce system ready for Adaptive Payments?

Almost certainly not but over the next few months and years the larger e-commerce software will start to adopt this technology but by then a new breed of payment systems will be in place which can take maximum benefit. For Eccomerce systems to accept split payments they will need to create Adaptive Payments Gateway as a payment provider however to make maximum use then it will require a change in the way eccomerce systems work.

Ways current Ecommerce Systems could use Adaptive Payments?

So you want to use these right now with Zen Cart or OS commerce and similar? Well with a little work you could get something working, here is how I would implement Adaptive Payments Split Payments to notify 3rd Parties.

Let us assume certain products within your shopping cart require paying 3rd Parties, these are treated as regular products we just store their Product ID for future use along with who we have to pay and what call we need to make.

Assuming your Ecommerce system has flexible gateway system, building an Adaptive Payments gateway is very similar to building any Paypal based API gateway their are a couple of strange authentication changes meaning you will need to send Public Keys in Headers rather then in Post data. To generate the Payment details we would need to look at the Product IDs in the basket and check for third party products. Create a Payment button with the correct data and send the chained or Split Payment with the IPN return set to the cart, on return of the IPN request make additional calls as required to notify the third party suppliers, this is not unusual most carts have the ability to allow such calls after purchase for autoresponders. Of course this does require the third party to accept such calls and have some sort of headless Ecommerce system.

Why Headless Ecommerce Systems are the future of Ecommerce!

A headless system is one with no user interface, so no pretty web pages, rather its designed to be interacted with by web services. Think of them as listeners when data is sent to them they process the data and return if the transaction is completed. When combined with split payments these systems can be used to automate the purchase and delivery of digitial products or order of physical product without the user ever having to go to the companies own web site in effect allowing them to expand beyond their own user base. With Headless Eccomerce Systems a web site procurement program can be added directly into other peoples systems.

Examples

Membership site – A headless Application manages membership to a site allowing site owners to subscribe users to the site, after payment the API generates the user with subscription appropriate to whatever was purchased. The Membership can be integrated directly into the original merchants checkout process as a cross sale.

Printing on Demand API – A headless API which accepts print on demand orders from Publishers who as part of their upsell process offer hard copies of their book, on purchase the Publisher makes a split payment with the royalties going to the Author, Printer and themselves, after payment is complete they call the Printers API to fire off the print job.

This was a quick introduction to what split and chained payments can do as well as introducing the idea of headless ecommerce systems, over the next few weeks and months I plan to show more and more of the ideas as well as more practical examples of usage.

Hours prior to to Geocities closure, Reocities was born a project to archive its content, it got a lot of twitter time including myself and they manage to scrape a hefty chunk. However unknown to them Yahoo had been working to save Geocities through archive.org

For legal reasons Yahoo felt they could not wholesale dump Geocities content to archive.org but did provide the site with every assistance to allow them to store and and access all publicly available sites at time of closure, they also allowed their bots unlimited connections and bandwidth to grab as much content as possible.

Why didn’t they publicise it?

Good question, but they were quite keen when I asked that the message was made clear they had worked with archive.org and that this was a long term plan, as David said back in June. With archive.org not being indexed however it does mean a large part of the web has still vanished from day to day searches and so potentially their still is a place for projects like reocities.com.

Sometimes watching television can be frustrating and I don’t mean because of the silliness of CSI awesome computing and forensic capabilities No what I find silly is the way advertisers try to leverage the web. So I would like to introduce you to a case example.

Park Hampers – Park Hampers are a Christmas saving scheme based in the UK they work by you paying for your hamper in the weeks months prior to Christmas when it comes to Christmas time they deliver your hamper. Its aimed more at low income families or people who are terrible at saving. Their SEO team should be shot!

Their recent TV campaign tells you to visit park11.tv this is of course a redirect to a truly terrible landing page, but it is not the landing page that concerns me but the fact other then this URL there is no mention of Park11 or Park11.tv anywhere on there site this results in a Google Search for Park11 being:

Christmas Hampers? No. Parkisons disease for Christmas? No thanks

What has gone wrong for Park?

Park obviously want to track number of hits their TV campaign is bringing to their website so have created a unique url for that campaign the aforementioned park11.tv this I assume logs the user before redirecting to the landing page. Park have numerous other URLs leading to various landing pages and mini sites so this seems to be part of an overall strategy the problem is that as the unique url has diversified away from the core brand they haven’t laid any foundation on their site to cope with it.

So they don’t show in a Google search for Park or Park11 so what?

How many people put urls into Google not the url bar on a web browsers? I would suggest quite a lot especially those people who perhaps are not as web savy which are in part the customers Park Hampers are aiming at. So by having a sponsored listing they are simply losing traffic and what traffic they are picking up they are paying for via PPC. Not only are they paying for an advert on tv they are then paying to pick up the traffic from that advert!

In the case of Park they have a small reputation issue as well, while badly researched by them Park11 association with Parkinsons can’t be helping as users arrive at Google but find no results for hampers they do read a pile of scientific jargon if that doesn’t put the user off they may well visit the wikipedia page on Parkinsons. Which to be fair I think every one should so here is the link again.

How do you solve the problem

First off all sorting out the rest of Park Hampers site so that its pages are ranking, then reference Park11 campaign create a dedicated landing page for it with the keywords in use and at least internal navigation pointing to the site. Just to be safe surely you would optimise for keywords such as Park11 & “Park Eleven”

Park are not the only company not to do this, but since they are on tele right now advertising I thought they were the perfect example.

Update: Myself and Carolyn were bored over the weekend and so have been playing with look at Park Hampers redirect strategy to say its a mess is an understatement, they own park1.tv through to park40.tv and seem to be using most in various tracking, about 20 of the Google results for park1-40 actually have Park anywhere in the top 10, examples of other Park redirects include park5.tv and park15.tv. This itself wouldn’t be that bad but in addition Park own another 8 or so domains including myparkmag.co.uk, gopark.co.uk etc these are not minisites but rather badly designed redirects. In short a mess, don’t fall for the same mistakes.

Update2: I incorrectly identified gopark.co.uk as belonging to Park Hampers, this is in fact a site owned by a third party with no link to Park Hampers who has asked I clarify rather then remove the reference, please see my about page for my Disclaimer on Facts and Figures. I have also written a more fuller apology to gopark.co.uk.

At the time I first double checked our own bots don’t do anything quite so stupid before suggesting that I thought it unlikely but would happily test it. Dave suggested a wager oddly enough one I never took him up on and I’m glad I didn’t!

How we crawl robots.txt file

The need for speed is paramount when crawling a site, a bot is taking up server resources and you want it to complete its required action in as short a point as possible. If your bot follows REP (what’s REP See Post notes for details )it’s first action should be to download the available robots.txt file, on average these files are 2-4kb in size very small and take no time to download, however a file sent with a 404 is closer to 22-40kb assuming it also sends the associated html. a much larger size given the majority of sites do not have a robots.txt file this means if you are not careful your robot will spend more time downloading a useless file then anything else. The method we use is to simply ask initially for packets if the return is a status 200 we proceed to download the file, anything else and the status is stored and is ignored.

Is the way you do your crawler the correct way Tim?

There is no “official” recommendation within the RFC governing REP that covers how you should treat status codes and which you should follow to only follow Status 200 is by far the most efficient method but it comes at a cost as you could be ignoring the file! It also doesn’t totally protect against downloading 404 pages as some servers send out a status 200 not 404 when a page can not be found.
A draft proprosal did suggest that other status codes should be followed including 3xx related to moved documents temporary or permanent it did not explicitly mention dealing with cross domains.
I have started to make changes to our own bots See Post notes for details

How does Google deal with cross domain 301 of a robots.txt file?

It reads the file at least according to webmaster tools, in Bronco follow up post they show Google Webmaster tools accepting bit.ly/robots.txt file nice should we be alarmed potentially though only if your allowing custom urls to your user at a root level on your domain with dots in them so if your running a URL shortener then yes perhaps something to check.

Did you do your own tests?

yes I had already done some tests last night which backed up what they did here is how I tested.

Experiment 1 Cross 301 oh please say this doesn’t work!
We created two domains domain A and domain B with a robots.txt on Domain A and 301 to a file on Domain B the robots.txt dissallowed access to /test/ folder, a test folder was put on both domains and index file was put in both, each domain was given a root index cross referencing each and each of the test files.

If Google crawled the robots.txt then Domain A should have 1 indexed page, Domain B 2 when finished.
with a monitor attached to the logs doing reverse DNS looking for a Google IP so we could watch the interaction some links were thrown at Domain A.

Result: Domain A – 1 page indexed, Domain B – 2 page Indexed

In Webmaster Tools a status 200

Experiment 2 – Let’s give google the benefit of the doubt
Ok so maybe they have indeed adopted the 1997 draft and are therefore obeying redirects it will ignore a Status 666 right?
Fresh domain this time our robots.txt file will be in the correct location but will send a http status of 666

Result: Domain A – indexed 1 page

In Webmaster Tools – status 200

Experiment 3 – given you a robots.txt file regardless
Ok so what if we tell you our server is broken i.e 500 but we give you a correct robots.txt file?
Fresh domain, correct location but headers sent are http 503 – Service Unavailable we are telling it we are not available the server is buggered in effect.

Result: Domain A – indexed 1 page

In Webmaster Tools – status 200

Tim – If you think about this it actually supports the belief google have actually programmed in the ability to follow 3xx as otherwise it would have for the 3xx returned a 404 or a 200 and blank file

Experiment 4 – I’m not here even though I’m here
Final test send http status 404 but also a valid robots.txt file what you going to do Google!

Result: Domain A – indexed 2 page

In Webmaster Tools – status 404

Only in the final test did Google behave as if it was paying the blindest notice to http status codes, can we assume 404 is hard coded and it will accept anything else?

Why should you care?

While the potential for abuse is small unless you run something akin to a URL shortner what happens when your site is producing an intermittent 500 error. From playing with status codes it would seem Google if shown a invalid or unreachable robots.txt will continue to use the old file could this be a potential for abuse what about a sneaky redirect only google on a 301 from your robots.txt by a very mischievous hacker. food for thought, and I’m glad I didn’t take that bet of with DaveN.