# Cloud Usage History

## Sharing cloud usage history

Sharing cloud usage history helps us create an accurate representation of the cost-benefits of migrating to a new service (e.g.: AWS S3 :arrow\_right: Azure Blob Storage) or adopting a new product (e.g.: forecast APM cost). Once generated, you can download your usage history from a cloud provider and submit to Taloflow via our secure upload service.

{% hint style="success" %}
Cost and usage reports only contain billing-related information.
{% endhint %}

## AWS

Your AWS usage history is available in AWS Cost and Usage Reports (sometimes abbreviated as CURs).

{% hint style="info" %}
You must have[ AWS Billing and Cost Management Console Access](https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/control-access-billing.html) to perform the following steps.
{% endhint %}

### Generate AWS Cost and Usage Reports

To start generating AWS Cost and Usage Reports, please follow these steps:

1. Sign in to the [AWS Billing and Cost Management console](https://console.aws.amazon.com/billing/home#/).
2. On the navigation pane, choose **Cost & Usage Reports**.
3. Click **Create Report**. Give your report a name. For example, `general-cost-report` . Make sure that both `Include resource IDs` and `Data refresh` settings are selected.
4. Under delivery options, select the S3 bucket where your reports currently reside and click **Verify** OR create a new S3 bucket if you do not currently have one for the report. If you get a prompt to add a default bucket policy, accept it.
5. We recommend you create a path prefix in the next field (e.g.: `main` ). Make sure that `Hourly`, `GZIP` and `Create New Report Version` are selected. Click **Next**, then click **Review & Complete**.

{% hint style="info" %}
AWS should generate your first Cost and Usage Reports in 8-24 hours time.
{% endhint %}

### Download AWS Cost and Usage Reports

1. Sign in to the [AWS Billing and Cost Management console](https://console.aws.amazon.com/billing/home#/)
2. On the navigation pane, choose **Cost & Usage Reports**.
3. A list of your Cost and Usage Reports will be available. Click on the relevant one.
4. Click-through the S3 bucket folders until you select the latest report (ideally full month), and download to `csv.gz` file.

{% hint style="warning" %}
If it's available, it's much better to have a full-month report from the last month (not the current/ongoing month) so we can capture more variance. The report path will indicate the period like in this example: `20210501-20210601/`
{% endhint %}

## Google Cloud Platform

To export billing data from Google Cloud Platform (GCP) you have to create a billing report table in BigQuery.

{% hint style="info" %}
You must have [Billing enabled](https://cloud.google.com/billing/docs/how-to/modify-project#confirm_billing_is_enabled_on_a_project) on your project to perform the following steps.
{% endhint %}

### Generate the Billing Report in BigQuery

1. Set up Cloud Billing data export to BigQuery by following [these instructions](https://cloud.google.com/billing/docs/how-to/export-data-bigquery-setup).
2. Run the following query in [BigQuery](https://console.cloud.google.com/bigquery) after replacing `<PROJECT>`, `<DATASET>`, `<BILLING_ACCOUNT_ID>` and `<INVOICE_MONTH>` with the name of the relevant project, the relevant BigQuery dataset you created in the previous step, the [Billing Account ID](https://console.cloud.google.com/billing), and the last month in the following format `yyyyMM` (e.g.: `202105`):

{% hint style="info" %}
This query will extract the relevant information for **Cloud Storage**.
{% endhint %}

```sql
SELECT
  billing_account_id, service.id AS service_id,
  service.description AS service_description, sku.id AS sku_id,
  sku.description AS sku_description, usage_start_time,
  usage_end_time, project.id AS project_id, location.location,
  location.country, location.region, location.zone, cost, currency,
  currency_conversion_rate, usage.amount AS usage, usage.unit,
  usage.amount_in_pricing_units AS usage_in_pricing_units,
  usage.pricing_unit, TO_JSON_STRING(credits) AS credits_array,
  IFNULL((SELECT SUM(CAST(c.amount * 1000000 as int64)) FROM UNNEST(credits) c), 0) / 1000000 AS total_credits,
  invoice.month, cost_type, adjustment_info.description AS adjustment_info_description,
  adjustment_info.mode AS adjustment_info_mode, adjustment_info.type AS adjustment_info_type
FROM `<PROJECT>.<DATASET>.gcp_billing_export_v1_<BILLING_ACCOUNT_ID>`
WHERE service.description = "Cloud Storage" AND invoice.month = '<INVOICE_MONTH>' ORDER BY invoice.month, usage_start_time, usage_end_time;
```

### Download the Billing Report from BigQuery

1. If there are *less than* 16k records from the query, then you can simply click **Save Results**, then select `CSV` (local file) to download the report.
2. If there are *more than* 16k records from the query, then you'll have to perform these additional steps:
   1. Click on **Query History**
   2. Click on the query you just ran
   3. Click on the **Temporary Table** link (Destination Table) and click export to GCS
   4. Chose `GZIP` compression
   5. For file location, browse and select a GCS bucket where you want to save the file
   6. For the file name it would be better to use a wildcard in case more than one file needs be created, so you can put something like `report-*.csv.gz` (Read more about this [here](https://cloud.google.com/bigquery/docs/exporting-data#exporting_data_into_one_or_more_files))
   7. When the export finishes you can download the files from your GCS bucket directly.

{% hint style="warning" %}
If there is more than 1 report file, you need to put all the files into a `.zip` or `.tar.gz` file before uploading to Taloflow.
{% endhint %}

## Microsoft Azure

{% hint style="danger" %}
If you are on Microsoft Azure startup credits and can only access your billing data from the Microsoft Azure Sponsorships portal, **you cannot get an analysis performed by Taloflow**. This is because this limited report does not have critical data or report columns necessary for cost analysis, including: `bandwidth`, `chargeType`, `unitOfMeasure` and `productName`.
{% endhint %}

Microsoft Azure has many different account types and some of these have their own specific ways of accessing the billing report for export. The following two sets of linked instructions work in the vast majority of cases. However, if these steps are not relevant to your account type, please contact us at <help@taloflow.ai> and we'll provide different instructions.

1. [Download from Cost Management + Billing console](https://docs.microsoft.com/en-us/azure/cost-management-billing/understand/download-azure-daily-usage#download-usage-for-pending-charges)
2. [Billing Export Method](https://docs.microsoft.com/en-us/azure/cost-management-billing/costs/tutorial-export-acm-data?tabs=azure-portal#create-a-daily-export)

{% hint style="warning" %}
If you're using the Billing Export method, please use **Actual Cost** and a **Daily export of last-month costs** (or month-to-date costs if last-month is unavailable).
{% endhint %}