Cost Management

Cost Management is a simple tool providing cost transparency and cost control.

Users create log categories, set planned quotas, and then specify which logfiles or servers to include in each category. Scalyr displays your log volume by category for the past 24 hours, and for the month-to-date. You can also receive email notifications if you exceed your planned volume - either overall or for a single category - over the past 24 hours.

Cost control is provided through the specification of various discard filters. These filters, applied to a log category and expressed in Scalyr query language, identify logs you wish to discard. You will not be charged for discarded logs.

Discard filters can be temporarily disabled for a specified number of hours. For example, you can discard your detailed debugging logs, and turn them on only when necessary. You can also adjust discard filters throughout the day or month to control costs.

Subsampling will be added in a future update.

Overview

(1) To access Cost Management, select your Account.

(2) Then select Billing Plan.

(3) Cost Management allows you to define log categories for your log volume, as well as a planned quota for each category. Once these categories are defined, Scalyr displays them in a simple table containing the following information:

  • Log Category: The name of the category you have created.
  • Planned/day: The planned quota, in GB per day, of the category you have created.
  • Average/day (month to date): The total amount of data ingested for the current month divided by the number of days elapsed. If this value is greater than your planned quota it is highlighted red. If the value is lower it is highlighted green. This excludes logs you choose to discard below in steps (7) through (9).
  • Over last 24 hours: The actual amount of data ingested for that category over the past 24 hours. If this value is greater than your planned quota it is highlighted red. If the value is lower it is highlighted green. This excludes logs you choose to discard below in steps (7) through (9).

Note that there is an Other category in the table, without a planned quota, which contains all logs other than those grouped into the budget categories you have created.

To define a new or additional Log Category, select Add New Category.

(4) Enter the name of the log category you wish to create.

(5) Enter a planned usage quota for the category, in GB per day.

(6) Specify which server(s) and/or logfile(s) to include in the log category. If you're using Kubernetes these are cluster and controller name, respectively.

These two fields ($serverHost and $logfile) are the only fields with dedicated dropdowns on the Search view page (see (6) in the Search Overview).

You can use a single * as a wildcard at the beginning or end (but not the middle) of the server or log file name.

  • $serverHost contains "Prod": groups logs from all Production servers into a category.
  • $logfile = "/var/log/nginx/access": groups all access logs into a category
  • $serverHost = "Back" && $logfile = "/var/log/scalyr-agent-2/linux_system_metrics.log": groups all Linux system metrics logs on all Backend servers into a category.

(7) Once you have specified your log categories, each is listed with the option of creating one or several discard filters for that category. Expressed in Scalyr Query Language, the discard filter selects logs to discard. You will not be charged for discarded logs.

Select Add New Filter to create a discard filter.

(8) Enter your discard filter in the text box.

  • $status==200: discards all OK transactions.
  • $status = "INFO": discards all "INFO" logs.
  • $index > 50: discards logs with an index greater than 50.

(9) Clicking on Test Filter takes you to the Search view page, filtered on your entry in (8), allowing you to check the accuracy of the filter, and view the logs it selects. These are the logs you will be discarding.

(Note that it is often useful to work from Search view to construct your filter and view the specific logs you wish to discard. You can then copy and paste your filter into the text box.)

Select Save when finished.

In the above example we have created two log categories, Prod ($serverHost contains "Prod") and Garbage Collection ($logfile="/var/log/tomcat-gc-log"). We have also created a discard filter for each. The Production category is discarding all logs for which $status=200, while Garbage Collection is discarding histogram output with an index greater than 50 ($index>50), thus retaining output for only the top 50 Classes.

(10) You can reorder your list of log categories by selecting the hamburger button and dragging your selection to the desired order in the list.

This is important because rules you define in (6) above are processed according to the first match. If you specify the same logs for two categories, those logs will be grouped with the first matching category. Reordering the hierarchy of categories via the hamburger button lets you set the precedence of a log category over another.

(11) Select the magnifying-glass icon to go to the Search view page, filtered for the logs in your category.

(12) Select the pencil icon to edit the information for a category.

(13) Select the garbage icon to delete a category.

(14) After setting up a discard filter, you may wish to temporarily disable it. For example, you are rolling out an update for a certain category and wish to turn on detailed debugging logs.

To disable a discard filter, toggle the Discard button from On to Off.

(15) Enter the number of hours you wish to disable the discard filter and select Disable Filter.

The category will now be highlighted and greyed out until the amount of time you specify is up. At the end of the time period, the button reverts back to On.

Toggling the Discard button back to on re-enables the discard filter.

(16) Cost Management allows for notifications if your total log volume exceeds a certain size in a day, or if any log category exceeds its planned amount in a day.

(17) You can toggle notifications on or off for both log overages and for log category overages.

(18) Enter one or multiple email address(es) to receive notifications. Use a comma or semi-colon to differentiate multiple emails.

(19) For log overage notifications, enter a planned daily quota (GB per day). For log category overages, this quota will be specified in (5) above.

Select Save Changes when you are finished entering your email address(es).