Paytm Insider Cuts Costs with Loki for Event Logging
Paytm Insider’s motto is “to make every day less everyday,” says Hitesh Pachpor, Technical Manager for the popular platform for buying event tickets in India. “The platform is driven by the fan that is inside of us, who wants to have the most hassle-free experience when booking tickets online.”
So when tickets for a big cricket match or a major music artist’s concert go on sale, the company’s DevOps/SRE team knows it has to be prepared for a big surge in traffic, which on any given day already tops 1 million hits.
The company, which started as an event management business in Mumbai four years ago, runs a homegrown AWS, Kubernetes, and Grafana stack. “Our goal is to make sure that the systems are always up, so that our users on the mobile and web apps always have access to the platform,” says DevOps Engineer Aayush Anand.
Paytm Insider Challenges Caused by Growing Scale
But as the platform grew in scale, the team started running into problems.
In order to debug any issues with the platform that affected user experience, “we needed logs at all times,” says DevOps Engineer Piyush Baderia. “With our growing scale, the cost of our logging and monitoring stack was increasing along with our scale, which was not ideal for us.”
Secondly, during those spikes in traffic caused by big on-sale events, “the volume of logs would increase exponentially, as well as the log-drop rate and the time to buffer and ship logs,” says Baderia. “So an issue could be occurring for customers, but we were not aware of it or we were not able to debug it properly.”
“With the high volume of logs, scaling the Elastic cluster while controlling costs was very difficult,” he adds, “and often nodes would run out of memory, causing restarts and dropped logs.”
Paytm Insider Monitoring Everywhere
Perhaps most crucially, “we had no central place for all of our monitoring solutions,” says Baderia.
Paytm Insider used different monitoring solutions for different services. Says Anand: “We had CloudWatch for logging our different applications that run within the AWS environment. Then we had our Kubernetes stack Prometheus monitoring, where we would track metrics such as the pod usage, CPU utilization, and such and such. And then we also had our application stack where we used ELK for tracing the application logs and for finding bugs and performance limitations of different services.”
As a result, “there were many different dashboards that you had to go to for looking at issues,” says Baderia. “It was becoming very difficult for us to correlate. And our alerts were not centralized at all. For a single issue, we would get three different alerts from three different places, which would also create a lot of noise and make it more difficult for us to debug those issues.”
Paytm Insider Finding a Centralized Solution
The team realized it needed a way to “map all these logs together so that they could make more sense and so that we could find the problems in our production environment in a much faster and smoother way,” says Anand.
They were already using Grafana with Prometheus, and began to explore Loki. “We found out that with promtail, we can actually tail application logs and have the metrics for them in the same dashboard in paytm Insider,” says Anand. “Using Loki deployed on Kubernetes with Grafana made sense to us.”
This solution was implemented around July 2019. “Loki was fairly easy to use and integrate into our stack,” says Baderia. “It comes with a default EBS store on AWS and has a good estimation of the resource usage and everything that we might need in the default Helm chart. But after looking through the docs and a couple of blogs, we figured that we can also use AWS DynamoDB (to store indices) and AWS S3 as a backend (to store logs), and that lets us scale horizontally as much as possible without actually worrying if our logging stack will go down.”
With that in place, the team began to reap the benefits. “Once we had all the logs as well as the metrics in the same dashboard, we were able to very quickly find out what caused a spike in the CPU utilization by just mapping the timeframe for the application logs with the metrics,” says Anand. “That decreased our time to response for different production usage substantially. We were able to figure out what’s causing the latencies and what’s causing the bugs in our application in a much faster, smarter way than what we were doing before.”
For example, one of Paytm Insider’s applications was experiencing a slow response time. “With Loki, we were able to find that this call is taking this much amount of time, but when we ran it locally it didn’t take as much time,” says Anand. “So by investigating further we found that the middleware was calling another API, which was taking longer.”
Stellar Results
Today, the team gets 2,000 logs per minute per service, and Loki centralizes logs for 25 services. They use a single Loki pod deployed as a stateful set and have promtail deployed as a Daemonset.
Learn How to start dropshipping.
Before Loki, the average response time for debugging a latency issue was about 30 minutes. With Loki, used with Grafana and Prometheus, it’s down to 10 minutes. “And it’s less than that in most cases,” says Baderia.
Plus, the team has achieved nearly a 75% reduction in cost for the logging and monitoring stack by using Loki in its environment. “That has been pretty great for us because now, even though we are scaling in terms of user base, our costs are not scaling that much at least for our monitoring and logging stack,” says Baderia.
Centralized alerts have helped save time. “Now with logs and metrics in a single data source, and a single visualization tool, we can have a centralized alerting using Grafana,” says Baderia. “All of our alerts now come from a single place, and we don’t have to look anywhere else. We can just go directly to the alert and look at the status of the metrics and logs for that duration.”
And now that there’s centralized monitoring, the team has a way to correlate infrastructure and application performance.
Looking Ahead
The Paytm Insider team is working on further cost optimizations, and hoping to add more functionality. For instance, the team would like to add Loki as a data source in its dashboards, use Loki labels to enable per-API monitoring, and integrate panels into its stack to achieve horizontal scalability.
But already, Loki has “worked great for us,” says Anand.
“Since the time we installed Loki, there has not been a single incident or an instance where Loki went down for us, regardless of the volume of application logs that we were getting into Paytm Insider,” adds Baderia.
Baderia mentions in passing that the platform had just experienced a huge spike in traffic that day, because tickets for an India vs. Bangladesh T20 cricket match went on sale.
It’s no big deal now. “It went amazing,” he says. “We did not have a single log drop on Loki or a single metric drop on Prometheus.”
You can create an on-ground event for Paytm Insider on our platform by registering and getting access to our publish portal.
How to set up an on-ground event on Paytm Insider
This section shows you on how to set up for event in paytm insider.
1. Click on 3 lines on the left side of your screen for Paytm Insider
2. Select “Create an event” and choose “On-ground event.”
3. Upload a creative for your event page in paytm Insider
NOTE: Requirements: Image size: 1316x720px @ 144dpi
Saved for web at 60% or lower – file size under 180kb.
Format: JPG
Please ensure the creative does not have logos of other ticketing platforms.
4. Add your “Event name’ and ‘Description’
Use this space to let users know what your event is about.
Include artist lineups, a writeup about the organiser and guidelines that you’d like the attendees to follow.
NOTE: Do not mention any phone numbers or email ID’s.
5. Add images and/or a video
Add an image gallery or embed a video on your event page to give ticket buyers a sense of what’s in store.
Event pages with rich content such as images and videos see more traction and purchases than those without.
Image Requirements: Image size: 1000X1000px @ 144dpi
Saved for web at 60% or lower – file size under 180kb.
Format: JPG
Video: Copy paste the YouTube link to the video you want to add to your event description page
NOTE: You can only embed a YouTube video which is already uploaded on YouTube.
6. Enter your event details
Event start & end time: Enter these details carefully and accurately as it will be displayed on your event page.
City: Select the city where you are hosting the event, your event will be displayed on that city’s page on Paytm Insider.
Group: Choose a relevant group that your event falls under. For example: events, workshops, travel etc.
Category: Pick a category, this will help users discover your event. For eg. Quiz, Kids, Comedy, Music.
Organiser: You will see your brand name here (only if it’s already registered with Paytm Insider)
Venue: From the drop-down menu, select ‘Yes’ and search for the address on the google map displayed on your publisher dashboard. Similarly, select ‘No’ If you haven’t decided on a venue yet and your event page will display ‘Venue to be Announced’ until you update it.
Note: If you change your venue after the event is published; here’s what to do:
1. You need to unpublish the event
2. Change the venue
3. Publish your event again
7. Enter the Details of the person managing your event
It could be you or someone from your team.
This is required in case we need to get in touch.
8. Include Health and Safety Precautions
Highlight the safety precautions that will be taken at your event, to re-assure your customers and keep them updated on what to expect at the venue.
All about Tickets for Paytm Insider!
This section will show you how to add tickets to events.
1. Head to the ‘Manage your event section on Publisher
2. Add a Show
Click on ‘add show’
Remember to mention the date accurately, the start and the end time of the event.
NOTE: If your event lasts more than a day – Create multiple shows, for each day.
3. Adding tickets for your event
Click on “add tickets” under that particular show
NOTE: Add a Regular ticket or RSVP (sign up for one) or a Custom ticket (which allows you to customize the name of the ticket). Then please add description, Price, Time of the event, Quantity i.e., inventory of the tickets, max purchase amount, and expiry date.
GST: Select from the drop down (Tickets above Rs.499 will attract GST)
4. Collect Additional Information
We already collect Name, email address and contact numbers.
Please use this feature if your event requires additional information from customers.
For example: Marathons need customer blood group, T-Shirt size etc.
You can add the questions for extra information.
You can choose to collect this info for each ticket (if someone buys 5 tickets, they will have to answer 5 times) OR for each order (if they buy 5 tickets, they will have to answer only once)
5. Changing Ticket Prices
If you wish to change the ticket price after your event is already published, you need to disable the current ticket and create a new one with the updated price.
You’re all set! Let’s get the event Published.
When you create the event, it is unpublished.
Once you feel all the information has been added, change the event status to Published using ‘event status’.
You can mark your event Sold Out or Cancelled using the same feature.
NOTE: Once you mark your event cancelled, communication regarding the cancellation is automatically triggered and refunds will be initiated to all the customers.
If you wish to see how your event looks on the Paytm Insider App/Website, use the View Event on Paytm Insider feature.
Managing Your Event for Paytm Insider
Once your event is live, you can manage various aspects of it through the publish dashboard. This includes adding a new show, introducing a differently priced ticket, managing discounts, shutting sales for your event or even sending out a reminder or update to your attendees.
The Publisher dashboard includes these features to make managing your event easy
- Orders: See all the orders for your event, including revenue and fees along with fetching basic customer data (Name, email ID, phone number etc.)
- Add attendees: Helps you to Issue tickets to customers and collect payments offline by filling in the buyer details.
- Guestlist: Give complimentary tickets to guests by filling in the guest details.
- Check in: you can easily Check in attendees by scanning QR code (via mobile browser) on tickets or by using ticket short-code or Email. This indicates that the customer has attended the event.
- Send communication: If you want to share any information about the event with all the ticket holders, you can use this feature and send communication via email or Whatsapp. You can use this for communicating Venue change/date change info, or even for sending event guidelines/ event reminder one day before the event.
Managing Discounts
If you wish to give discounts to customers, you can create the discount codes using this feature.
Integrations
iframe: You can embed the Paytm Insider Ticket selling page on your website with the help of the iframe.
Webhooks: Get notified about important updates for your event via Slack, Discord or POST Api call.
Paytm Insider rules for #FAQ
Frequently Answered Questions is a game show for and by the audience. A chaotic and fun quiz where there is a buzzer. Are you fast enough to frequently answer these questions?
1. Only one individual per team. That doesn’t mean you shouldn’t bring friends in Paytm Insider. Bring them and compete against them.
2. All the questions are buzzer based. First one to buzz in will get to answer the question into Paytm insider.
3. Don’t look stuff up on google –Strict rule from Paytm insider. It’s a quiz and it’s more fun to play than to win.
4. The quiz master is GOD. All final decisions will be made by him.