How to set up analytics using Yandex Cloud and get a convenient business tool that saves employees time and allows track all data?

We set up and automated analytics for the Business Drive Academy project, which helped to get data on key indicators in a convenient way, which made it possible to make decisions on the further development of the project.

  • Rostelecom approached our team with a request to customize and automate analytics for the Business Drive Academy project
  • The customer wanted a tool that would allow regular monitoring of relevant data in a single dashboard

What goals did the customer want to achieve?

  • The previous team’s attempts to customize analytics were not successful. The Google analytics counter used previously collected inaccurate data, and data from multiple sources had to be aggregated manually to calculate the cost per user.
  • Address management’s concerns about the vulnerability of data being transmitted through external services
  • Avoid additional recurring fees for various services

The main conditions for choosing the service were:

  • avoid the use of additional paid services in the data chain, in particularly connectors for transferring costs from advertising systems;
  • the tools used should not be at risk of being shut down due to external events (sanctions or blocking).

Choice of tools

To implement the project, we chose Yandex Cloud. The platform meets the customer’s objectives and contains a full range of compatible tools from data collection and storage (Yandex.Metrica, ClickHouse) to visualization (DataLens).

Scripts in Python were used to transfer raw data from Yandex.Metrica and AppMetrica to ClickHouse. Also, to avoid paid third-party connector services, they unloaded expenses from advertising accounts using Python.

Yandex.Metrica and AppMetrica have direct integrations with ClickHouse, which greatly simplifies a number of analytics tasks. But the transfer of some of the data we needed, such as expenses or, was not supported by such integrations at the time of work, and we had to resort to writing scripts.

Stages of work

After we decided on the platform, the team started the main stages of work

What did we do?
1. Installed the counter “Yandex. Metrica” on sites
The project uses 2 websites:

  • A welcoming lending which attracts new users;
  • The main platform with courses, where users get to after registration.

2. Installed AppMetrica counter in iOS and Android apps

3. Set up conversions in “Yandex.Metrics”
Which conversions have been emphasized?

  • Conversions to introductory lessons.
  • Conversions to registration.
  • Registration on platform.
  • Lesson views on the platform.

4. We have configured the transfer of user event parameters
Each person registered on the platform is assigned a unique user identifier (UserID) ➡ Transfer them to Yandex. Metrics” as parameters of visits ➡ Bind them with ClientID of “Yandex. Metrics.

5. Created a database with the help of Managed Service for ClickHouse inside Yandex Cloud

6. Wrote scripts in Python
What has been finalized?

  • Transfer of expenses from “Yandex.Direct” and “VKontakte” advertising cabinet to ClickHouse.
  • Data transfer from Yandex.Metrics to ClickHouse.
  • Data transfer from AppMetrica to ClickHouse.

7. Created a virtual machine through Compute Cloud, downloaded the data transfer scripts and configured them to run regularly
How did you set up regular data upload to ClickHouse?

  • We created a virtual machine with Compute Cloud – a service inside Yandex Cloud for creating and managing virtual machines.
  • Uploaded data transfer scripts to the virtual machine and configured them to run on schedule.

8. We set up visualization in DataLens so that the client receives the final result of the analysis in the form of a dashboard
During the work created:

  • connection to ClickHouse;
  • datasets and merge data at dataset level;
  • charts;
  • dashboards.