I read Antonio García Martínez's excellent memoir "Chaos Monkeys: Obscene Fortune and Random Failure in Silicon Valley" and it can be one of the best book I have read in this year.

This book describes around 5 years when the author Martinez moved from Wall Street to Silicon Valley in 2008 when financial crisis reached climax and left Facebook in 2013. Martinez’ field in Sillicon Valley is advertising technology (AdTech). He worked for AdTech startup, and then found YC backed AdTech startup. His startup survived an IP lawsuit from the company he used to work for, and eventually was acquired by Twitter.

Adtech was one of my field of my previous career, digital marketing analyst, involved in launching Japanese branch of US digital marketing information service. It was from 2015 to 2017. I found out that I can compose rough AdTech business history after the financial crisis, through merging “Chaos Monkeys”(2008~2013) and my experience. The change of Facebook's advertising products is best storytelling on the transition of the Internet economy. Although there will be a technical explanation by all means, I think that this blog will be valuable information for person working for the tech industry, the media / advertising industry, entrepreneurs, investors, job seekers in the industry. Even if you skip the technical part, it will be valuable to know history of the internet advertisement.

History of Internet Advertising

Let's look back over the history of Adtech. In 1994, the world's first banner ad was born. Internet advertisement at the time was the same as the transpotation advertisement. The internet advertisement has greatly evolved by using a "ad server" different from the server that delivers the original web page. Two important things are 1) Gathering a lot of users for your website 2) securing performative ad server cluster suppling advertisements there. It was very simple. Players who have a lot of users such as Yahoo and Microsoft have secured ad server function by acquiring.

Yahoo, Microsoft, Google, AOL shared ad server market in the middle of the 2000s. Google bought NY's ad-tech company DoubleClick in 2008 and Google began to dominate Yahoo and others. Financial crisis had a great impact on internet advertising. So many software engineers lost job at Wall Street, and some of them moved to advertising industry. Founding and investment of the startup increased dramatically. The idea was conceived seriously to conduct an auction for a few milliseconds before web page display and "ad server" delivers the advertisement immediately. This aimed at building Wall Street like financial market in old-fashion advertising industry. Even the application of algorithmic trading was challenged. AdTech became a buzzword.

Thousands of AdTech startup emerged and competed, but “Smartphone” proposed by Steve Jobs was definitive factor drawing a line between winners and losers. People shifted to prefer using mobile app. AdTech tactics for desktop browsers have become useless, and native advertisements in Mobile apps will be in the spotlight. People start to use multi connected devices, so tracking became not easy. Situation handed a big advantage for Google and Facebook, with cross-device user identification capability. In 2017 digital advertising revenues surpassed TV advertising revenue in the United States, digital advertisement became the center of the world in 13 years since the birth of banner advertisement in 1994.

Now we live in the duoply world that Google and Facebook accounts for more than 60% of the US digital advertising market. Snapchat, which might become challenger for Duopoly, was completely defeated by Marc Zuckerberg's cruel copycat strategy. Amazon, the other challenger, mainly dealing with demand around digital commerce, succeed to build steady business, but its size seems lack of the impact as third parties penetrating duopoly market. Yahoo and AOL, former giants who were defeated by Google and Facebook, merged as Verizon property last year, but I do not think their future is bright.

Ad-Tech startup has begun to sell business since around 2016, the main buyers are a Chinese company with tax avoidance purposes. They are Manufacturer or PE not interested in advertisements. In total, there were 77 mergers and acquisitions in the ad-tech, marketing-tech and digital content sectors in the U.S. in the second quarter of 2018, according to investment bank LUMA Partners. It is evidence of loser that sell business this year(too late). Market already signaled negative sign in second half of 2016. It is in 2011 that Martinez sold the company.

Ad-Tech under control by ad agency is still alive but it seems that employees are constantly searching for opportunities outside.

Open vs Proprietary

Now look at Facebook Exchange (FBX) that Martinez managed as product manager. This help you understand how internet advertising evolve.

Martinez explained FBX competed with Custom Audience (CA) inside the company at the time. The relationship between FBX and CA should be expressed as “open vs proprietary". It is a conflict that always comes out in every competition over software. It is generally said that open are preferable. Linux is a prime example of successful open strategy. However, it is also true that proprietary system have generated enomous profit. Apple, Microsoft, and Oracle are typical example.

FBX was an approach familiar with trend of Ad-tech at that time. Advertisers can buy ads displaying desktop browser with "real-time-bidding (RTB)". The FBX is open to third party DSP (Demand Side Platform). Facebook is just one of the advertiser position as well as a number of DSPs. Third-party DSPs often stood in information superiority through matching data they gather outside and Facebook cookie data.‌‌ Because of this, third-party DSPs could bit higher price than advertisers Facebook represent. Facebook could be beaten sometimes, though it supplies inventories and organizes auctions.

Martinez's approach was trying to bring the structure of financial markets into the world of internet advertising. In Wall Street, algorithmic trading becomes commonplace, traders are losing necessity. The approach is to bring similar things to advertising transactions.

The advantage of this approach is that external ecosystems such as DSP, SSP and third party data vendors had already grown, so if you have interoperability with the ecosystem, you can quickly expand your business. Facebook attract a lot of demand through FBX, so This also benefits Facebook on taking fee of transaction and adding pressure to hike contract price.

The downside on this approach is that external DSPs take information advantage over Facebook. In fact, they matched their data with external data sources and could be aggressive to valuable inventories for advertiser DSPs represent. Executive of Facebook advertising department dislike the situation because they were able to issue higher bids than ordinary customers and Facebook itself.

According to the author 's recollection, the Facebook advertising department was unfamiliar about Ad-Tech when he joined in 2011. He was surprised at the low CPM(Cost per thousand) of Facebook ad, which was under average price of the US market. Perhaps there was a possibility that Executives could not understand FBX in detail.

On the other hand, Custom Audience (CA) is an advertisement product that allows matching of the behaviour data on a proprietary platform developed by Facebook. It was considered that it took two or three years to develop the platform then, so it is assumed that there were just anonymized IDs associated with behavioural data on Facebook.

DSPs participating in the bid on the FBX can match various data sources inside and outside Facebook, and they can enjoy Information asymmetry.

Downside on CA was that data Facebook had was not important information the advertiser likes, although Facebook has a large number of users and their demographic and behavioral data. So it is necessary to sync Facebook data with data outside, but a proprietary mechanism was under development (now it’s done).

What is Upside of CA ? Facebook can fully control data usage, limiting connection with the external system. It is possible for Facebook to pressure external players make profits using its data. In addition, if advertisers and agencies continue to purchase ad at CA (there are SaaS offering expensive external tools for big enterprises or some agency/vendor hybrid who substitute for buying ads using their own tools) , Facebook can lock advertiser's advertising budget in its platform.

It seems that it was CA where Facebook concentrate human resources. For sales selling CA, It was not fun for their clients to be a loser due to the bidding with advertisers who uses external DSPs connecting Facebook property via FBX. Perhaps there might be an organizational conflict within the company. For Facebook sales and operations, FBX must be like an enemy, denying their activity and automating business process. This is also what I felt as one of the introducers of Buzzwords "Digital Marketing" to Japan.

CA was more advantageous than FBX from point of view by business side, because advertising is more profitable by monopolising distribution process. Facebook did not know anything about Adtech compared with thousands of startups. There are not rational reasons to affirm an FBX that push Facebook outside wild Ad-Tech startups were competing brutally. Martinez points out that Facebook is similar to Goldman Sachs he used to work for. The investment bank prioritized monopolizing the market by maintaining imbalance of information rather than innovation. Facebook was fond of proprietary ad selling process and monopoly of data.

Multi-Device Identification

After Martinez quit Facebook(April 2013), Facebook has expanded advertising sales inclined to mobile application ad while people are moving to mobile applications. Users really liked mobile apps, and then mobile native ad formats were born, which became Facebook's cash cow.

Users now have multiple devices, making the game more difficult for existing players who had served ads to desktop browsers, but Facebook was able to identify cross-device users. For example, if there are users who use the four devices, desktop, laptop, smartphone, tablet, existing players recognize four device as four users differently. If a user use internet only through apps on smartphone and tablet, cookies doesn’t work. The visibility of behaviour is so bad that advertiser cannot use it for campaign. However, Facebook recognizes four users as one person, and links behavioural information to just one ID. Only Google and Facebook can deploy this cross device tracking globally. It is clear that these will bring huge advantage to Google and Facebook in digital advertising market. Duo can target internet users with confidence such as "this device belong to ID ******(anonymous)". This is called a "deterministic ID". On the other hand, other players must infer from various uncertain information that "probably this ID use this device”. It is like detective novel. This is called a "probabilistic ID". low accuracy rate often yields low performance campaign. The deterministic ID is always better the probabilistic ID.

It is natural to design advertising products, leveraging its data dominance. Facebook limited the connectivity to the other advertising platforms and concentrated their resource to build the Walled Garden. Outside the mobile castle, Facebook acquired Atlas from Microsoft. Despite employing Adtech talents for atlas, atlas was like an fort isolated from Facebook mobile castle, and defeated by Google DoubleClick. Google was taking same strategy as Facebook. Search ads has been the most profitable among digital ad products. The proprietary ad have never showed indication they lose the position.

Google DoubleClick, which FBX aimed to take advantage, also have a financial institution-like feature. Google closed a door to external buyers on Youtube ad inventory at its own exchange, putting google in the same orientation as Facebook.‌‌ Youtube has been getting more and more valuable to become a pillar of Google's revenue.

Thus, Google and Facebook account for more than 60% of advertising revenues in the United States, and the market expansion every year has been swallowed by two giants. There are two huge castles in the internet economy and those continues to expand gradually. Outside of the castle is “Dog eat dog world”, costing more expensive than tax made by the duo. Most of advertiser chooses being in the castles and paying tax.

A Textbook tells us that the market expands with a free competitive environment, and it is beneficial for stakeholders, but the reality has settled to proprietary . It is clear that the advertising expenditure, which is gasoline of the Internet medias, is controlled by twin giants. It is obvious that next-gen players must have other business model than an advertisement.

Bad Market

Now, some readers have impression that “Ad-Tech was ideal way to evolve digital advertising, but it was defeated by Google and Facebook unfortunately” . If you are a bit familiar with finance, Ad-Tech's mechanism is exciting as an analogy of financial transactions. It is considerable that trading ad inventories on digital is far more efficient than the way old-fashioned sales person visiting clients repeatedly and make a small talk.

Ad-tech can deliver advertisements matching human intent. This is correct when you limit discussion scope to theory. In fact, the practical Ad-Tech and advertising industry are full of various difficulties made by human beings who tend to follow immediate gains.

The critical factor is that the advertising market is not a “perfect market" in economics, and Ad inventory is not a "commodity". Impression means that ads are displayed on the screen, but each impression is too unique to commoditize. Ad is not displayed in screen, but counted as an impression. It is not displayed because of network instability of network or confusion on Ad-Tech stack. Scale doesn’t guarantee correctness of statistical approach in this circumstances.

Industries has been showing interest in behavioral economics and behavioral psychology approach to the complexity. How can we draw a line between intent and behaviour? human is social animal at the same time. They pretend to have nice personality in social media, where their digital activity is captured in his or her own name. Some Ad-Tech exec, who can see the law data on various content platform, told me “What people are interested in is sex and money, the other categories are very small compared with these needs representing human nature”. In conclusions, contexts are extremely varied when people are seeing ads, and human’s mind and behaviour are so complicated that we haven't had reasonable explanation (we haven't figure out how our brains are working, or how genetic information affect  nature of ourselves).

Then, fungibility is also matter, the term defined as the ability of a good or asset to be interchanged with other individual goods or assets of the same type in economics. For example, We can exchange 10 million Toyota stocks owned by Company A and 10 million Toyota stocks owned by Company B. This means Toyota stock is fungibile. Since digital display ads has no funability as I explain, it is doubtful to trade them as collective,  homogeneous commodity. When advertisement trading is done by placing digital ads in a large pool, someone intends to buy Toyota stocks but actually he/she bought Honda stocks. Machine driven ultra-high speed bidding can amplify this gap, and result a low performance of the campaign. The more participants gather in the market, the more confusion occurs. “Market” needs veterans who understand this broken and super wild pool and they grab unignorable influence in the end.

programmatic advertising means that it is efficient and quick way to dealing ad products without hiring labour force, but actually it really needs help of human’s hand.

Inefficient Ad-Tech

Basically, Ad-Tech's business ecosystem is very inefficient.

Everything just falls into the dark as transactions enter the Ad-Tech stack, contrary to its technical beauty that the millisecond interaction between agents that represent demand and supply side and the ad server's ultra-fast ad serving . The advertiser gives money to agency. Agency receives it, and finally hands to the publisher through the Ad-Tech.

‌                                                                         Upload

Ad-tech Chaos Map

• ‌

This can be explained as fragmentation of distribution, like , “A sells to B and B sells to C”, and each take its own margin from the original advertisement spending. Advertisers can only imagine how much margin was taken inside the box.

From the viewpoint of the advertiser, the starting point of the distribution flow, this is crazy system. There is no way to speculate on how much of the budget advertiser invest work effectively as “practical budget”.Programmatic buying is executed in milliseconds. The result is returned as a strange report or as a numerical value on the dashboard.

In recent years, annoyed advertisers emphasized measurement of advertisement effectiveness, viewability of ad, and verification of impression. Terribly, costs for these monitoring owe both advertisers and publishers.

Publishers, which position the end point of the distribution flow, can't know how is transaction made before they receive bids.Software such as Supply Side Platform (SSP)  represents their trading, but SSP also like to leverage information asymmetry towards publishers, though they are agent representing publishers interest.

Publishers must prepare an ad server for having digital ad business, but ad server requires you to have unique expertise, so people who originally sold newspapers and magazines must have difficulty equipping it.Therefore, publishers adopt ad server vendor, where Google holds 70 to 80% of the share.

Since publishers have suspicions about Ad-Tech which always pass them Joker, publishers usually  sets  adserver as defensive “Waterfall”. A certain number of premium inventories are always kept for sales to sell them manually in waterfall setting. Publishers hand non-premium inventories called “remnant” to Ad-Tech, which is similar with that fashion brand sells unpopular items at low price in suburban shopping mall.

Ad server sets selling priority of buyers. What happens under this configuration? Ad server asks order to “buyer A” first. If A skips order, ad server asks “buyer B” next. If B skips order, adserver asks..... buyers sequence is usually decided, based on average price they have ordered. Buyer A can be very powerful influence in series of transactions, and Buyer A is always Google, monopolising adserver market, and dominative in every part of Ad-Tech stack.

Waterfall brought risk for user experience. If several buyers skipped ordering, ad serve would delay page load. In worst case, ad server failed to serve content.

This publisher strategy ruins the upside of programmatic advertising. Ad-Tech has claimed efficient free-market full of premium inventory, benefits every stakeholders, but it is very primitive way of market design in “before internet age” ironically. There is also a factor that worsens the publisher's position. People around the world are surfing the internet without any specific purpose, creating countless low-quality ad inventories. A price collapse must occur due to oversupply.

There is another problem causing serious headache: low transparency. For example, if agencies and Ad-Techs take 70% margin of advertising expenses, only 30% works normally. 30% should be spent for "King of remnant". Advertisers, who do not know about the situation,  doubt effect of the digital advertisement. Publishers can not enjoy reward for the value they create. Advertisers and publishers at both ends of ecosystem don't have means to know the transition of inventory price. In other words, Middleman is exploiting publishers who provide value to users consuming contents, and advertiser who supply resources to supplychain in exchange for the marketing opportunities. It is doubtful that middleman create value for users and supply chain.

Middleman battled competition for margin. The more advertising agencies and Ad-Tech try to maximize their margins, the less practical advertising budgets is. As a result, Competition on margin causes bad campaign result, eventually the customers are gone.

This Battle Royale system is extremely inefficient. Google and Facebook charge premium price for advertiser. They take bold margin, but everything is comfortable in Walled Garden. on the other hand, outside of Walled Garden or Ad-Tech, everything is so wild like “dog eat dog world”. It is natural that Google and Facebook win the game.

I want to point out that I don't mean analog advertising is better than digital advertising through this blog. Analog advertising has own unique background, and we cannot compared digital and analog in quantitative approach. I am happy if this blog improves your decision making. Thank you for reading long article.

Image via "Planet Ape 2017"