Embracing change was never an easy thing, even if you are the engine of that change or you have to adjust to a change others made. 2016 was the year of a major change for the former 2Parale, now 2Performant team, as well as for all the platform’s users.
Today we are getting ready to face a new challenge: Black Friday. Of course, it is not the first one we are spending together, but it is the first one for the new 2Performant platform.
Surely there are a lot of questions gravitting in the minds of the 2Performant users. That’s why we decided to sit down and talk to Tudor – Product Manager and Cătălin – Technical Lead about how the network was prepared for this important event.
How was the 2Performant system prepared for Black Friday?
We have deployed loads of performance optimizations starting with September.
My colleagues in the Development and Infrastructure departments have analyzed each type of request that our users make, they picked the “heaviest” of the lot and have done all they could to lower its response time, its memory footprint and connections to other services that were not vital to its execution. Where possible, they’ve decoupled some processing routines into asynchronous jobs, that are executed after the response has been served back to the requesting user.
After that, they’ve done the same for the second heaviest request and so on, until we ran out of “spikes” on our performance monitoring graphs. All this in order to make sure everything happens as quickly and efficiently as possible, for both users and the platform itself.
We have conducted some load tests for the redirection service and we know how many requests per second our servers can handle. Therefore, we will know in real time whether we have enough servers or if we need to add some more. Being hosted on the Amazon Web Services infrastructure, adding them is immediately available, we don’t need to add physical servers.
We know that after merging 2Parale in 2Performant many things turned out to be in need of improvements. Black Friday was an excellent motivator for performance optimization, since the last thing any of us wanted was to worry about servers.
Will the cookie placement and commission registration (the event) be functional?
We are expecting 100% uptime. The “event” (i.e. the redirection and commission tracking service) was one of the key issues that the migration has concerned. We are now safe to say it’s blisteringly fast, extremely well isolated from any other service that has a chance of falling prey to heavy loads, and that it will function flawlessly.
Besides, we have found a pretty good recipe for horizontal scaling, which means that we can easily add more servers with the exact same role in case the existing “events” are nearing their max potential, so that we can distribute the load.
Will the statistics be displayed live?
Our stats are never really live. There’s always a minuscule delay due to the fact that most processing goes on asynchronously. Under normal conditions, these delays are a few hundreths of a second long, but it’s likely that during Black Friday the traffic volume is so great that they will increase slightly.
The dedicated stats section, as well as the numbers users see in their dashboards will be displayed nearly in real time, with possible delays as high as a couple of minutes.
The graph showing clicks and commissions in the dashboard will suffer delays as high as 15 minutes, but the actual values within it will be refreshed more frequently elsewhere.
In order to make sure that loads on our systems doesn’t increase that much, we have used caching for certain bits of information.
That is, after we compute – for example – the number of commissions in the current day, we keep that number in a cache that we serve for a certain amount of time (e.g. 1 minute). After that minute, the cache expires and the number of commissions is calculated again. This warrants a very low response time for any second (third, etc.) query made in that 1 minute interval.
If the expiration timeframe for a cache is 1 minute long, then that cached statistic can only be 1 minute old at most, after which, on the next query, when the cache expired, it will be refreshed with live information.
Our caching policy (i.e. the amount of time we cache each resource) will depend on the load, but we can promise we’ll do all we can to serve the freshest information we can upon each request.
Will the commissions be displayed in the platform in real time?
As I was saying, even in normal circumstances, there is a slight delay between the precise moment a commission is generated and its appearance in the commission list. During the heavy load we predict during Black Friday, these delays could extend to a few minutes, but we will do everything possible to prevent them from exceeding that threshold.
If there are any delays in displaying the commissions, will there be messages sent about them in real time?
Sending notification messages is a significantly lower priority process, compared to registering commissions and generating stats. We are expecting slight delays in the process of sending these emails (there are very many of them), but we don’t think these delays will exceed 15-20 minutes.
If statistics, especially clicks, are not displayed in real time, does that mean that the redirect is not functional?
Visitor redirection and cookie placement work independently from storing click information.
In the extremely unlikely case that the load increases to such extent that it will determine long delays in clicks being displayed (we’re talking about minutes or tens of minutes here), visitors will be given cookies and will be redirected just as quickly as they have been until now.
If commissions are not displayed in real time, does that mean they are not being registered?
Just like clicks, commissions are registered asynchronously. That is, all the data required in order to store a commission in the database is temporarily stored in a work queue, from where they are processed in the same order they entered the queue.
Imagine a line at the cash register in a supermarket. Even if the queue is very long, each of the people waiting in line will be eventually served. The difference is that in our queues, the commissions are patient and they don’t have anything better to do, so they’re guaranteed to reach the cash register sooner or later. And our “clerks” are very efficient 🙂
Is there a risk for some network features to be restricted? If so, which ones?
We are prepared to temporarily deactivate certain functionalities in case the stress they generate on the servers could potentially affect other more important functionalities.
The way we see it, this is the list of functionalities (not just in the interface), ranked by their importance:
- Registering clicks and commissions, redirection
- Generating promotional links
- Allowing pre-paid advertisers to deposit money
- Sending and receiving messages
- Statistics
- Commissions listing
- Affiliate program listing
- Applying to/exiting from/banning affiliate programs
- Serving affiliate feeds
- Dashboard statistics
- Creating payment profiles as an affiliate
- Requesting withdrawals as affiliates
Among those listed above, stats and affiliate feeds are by far the biggers resource eaters, and will therefore be the most likely to fail in case of a high load. Before we deactivate them, though, we are prepared to limit their resource consumption (e.g. smaller timeframes for stats, deactivating certain stat filters etc.).
This hierarchy is also valid throughout the year, not just during Black Friday, but the point is that we will do our best to keep the services running in approximately that order.
If certain interface features are restricted, does that mean cookie placement and/or commission registration will be limited?
Redirection, cookie placement and commission registration are almost completely isolated fom the rest of the services, which makes them nearly invulnerable to issues that affect the interface.
We hope that Tudor and Cătălin managed to clarify as many of your questions as possible, so you can focus on generating tons of sales this Black Friday as well.
May the force of conversions be with you!
