Cloud Usage History

For some reports, uploading usage history from cloud providers is required. This doc covers how to obtain that information from the major cloud providers, like AWS, Azure, and Google Cloud.

Sharing cloud usage history

Sharing cloud usage history helps us create an accurate representation of the cost-benefits of migrating to a new service (e.g.: AWS S3 ➡️ Azure Blob Storage) or adopting a new product (e.g.: forecast APM cost). Once generated, you can download your usage history from a cloud provider and submit to Taloflow via our secure upload service.

Cost and usage reports only contain billing-related information.

AWS

Your AWS usage history is available in AWS Cost and Usage Reports (sometimes abbreviated as CURs).

You must have AWS Billing and Cost Management Console Access to perform the following steps.

Generate AWS Cost and Usage Reports

To start generating AWS Cost and Usage Reports, please follow these steps:

1. Sign in to the AWS Billing and Cost Management console.

2. On the navigation pane, choose Cost & Usage Reports.

3. Click Create Report. Give your report a name. For example, general-cost-report . Make sure that both Include resource IDs and Data refresh settings are selected.

4. Under delivery options, select the S3 bucket where your reports currently reside and click Verify OR create a new S3 bucket if you do not currently have one for the report. If you get a prompt to add a default bucket policy, accept it.

5. We recommend you create a path prefix in the next field (e.g.: main ). Make sure that Hourly, GZIP and Create New Report Version are selected. Click Next, then click Review & Complete.

AWS should generate your first Cost and Usage Reports in 8-24 hours time.

Download AWS Cost and Usage Reports

1. Sign in to the AWS Billing and Cost Management console

2. On the navigation pane, choose Cost & Usage Reports.

3. A list of your Cost and Usage Reports will be available. Click on the relevant one.

4. Click-through the S3 bucket folders until you select the latest report (ideally full month), and download to csv.gz file.

If it's available, it's much better to have a full-month report from the last month (not the current/ongoing month) so we can capture more variance. The report path will indicate the period like in this example: 20210501-20210601/

Google Cloud Platform

To export billing data from Google Cloud Platform (GCP) you have to create a billing report table in BigQuery.

You must have Billing enabled on your project to perform the following steps.

Generate the Billing Report in BigQuery

1. Set up Cloud Billing data export to BigQuery by following these instructions.

2. Run the following query in BigQuery after replacing <PROJECT>, <DATASET>, <BILLING_ACCOUNT_ID> and <INVOICE_MONTH> with the name of the relevant project, the relevant BigQuery dataset you created in the previous step, the Billing Account ID, and the last month in the following format yyyyMM (e.g.: 202105):

This query will extract the relevant information for Cloud Storage.

SELECT
  billing_account_id, service.id AS service_id,
  service.description AS service_description, sku.id AS sku_id,
  sku.description AS sku_description, usage_start_time,
  usage_end_time, project.id AS project_id, location.location,
  location.country, location.region, location.zone, cost, currency,
  currency_conversion_rate, usage.amount AS usage, usage.unit,
  usage.amount_in_pricing_units AS usage_in_pricing_units,
  usage.pricing_unit, TO_JSON_STRING(credits) AS credits_array,
  IFNULL((SELECT SUM(CAST(c.amount * 1000000 as int64)) FROM UNNEST(credits) c), 0) / 1000000 AS total_credits,
  invoice.month, cost_type, adjustment_info.description AS adjustment_info_description,
  adjustment_info.mode AS adjustment_info_mode, adjustment_info.type AS adjustment_info_type
FROM `<PROJECT>.<DATASET>.gcp_billing_export_v1_<BILLING_ACCOUNT_ID>`
WHERE service.description = "Cloud Storage" AND invoice.month = '<INVOICE_MONTH>' ORDER BY invoice.month, usage_start_time, usage_end_time;

Download the Billing Report from BigQuery

1. If there are less than 16k records from the query, then you can simply click Save Results, then select CSV (local file) to download the report.

2. If there are more than 16k records from the query, then you'll have to perform these additional steps:

   1. Click on Query History

   2. Click on the query you just ran

   3. Click on the Temporary Table link (Destination Table) and click export to GCS

   4. Chose GZIP compression

   5. For file location, browse and select a GCS bucket where you want to save the file

   6. For the file name it would be better to use a wildcard in case more than one file needs be created, so you can put something like report-*.csv.gz (Read more about this here)

   7. When the export finishes you can download the files from your GCS bucket directly.

If there is more than 1 report file, you need to put all the files into a .zip or .tar.gz file before uploading to Taloflow.

Microsoft Azure

If you are on Microsoft Azure startup credits and can only access your billing data from the Microsoft Azure Sponsorships portal, you cannot get an analysis performed by Taloflow. This is because this limited report does not have critical data or report columns necessary for cost analysis, including: bandwidth, chargeType, unitOfMeasure and productName.

Microsoft Azure has many different account types and some of these have their own specific ways of accessing the billing report for export. The following two sets of linked instructions work in the vast majority of cases. However, if these steps are not relevant to your account type, please contact us at [email protected] and we'll provide different instructions.

1. Download from Cost Management + Billing console

2. Billing Export Method

If you're using the Billing Export method, please use Actual Cost and a Daily export of last-month costs (or month-to-date costs if last-month is unavailable).