Cloud Usage History
For some reports, uploading usage history from cloud providers is required. This doc covers how to obtain that information from the major cloud providers, like AWS, Azure, and Google Cloud.
Sharing cloud usage history helps us create an accurate representation of the cost-benefits of migrating to a new service (e.g.: AWS S3 Azure Blob Storage) or adopting a new product (e.g.: forecast APM cost). Once generated, you can download your usage history from a cloud provider and submit to Taloflow via our secure upload service.
➡
Cost and usage reports only contain billing-related information.
Your AWS usage history is available in AWS Cost and Usage Reports (sometimes abbreviated as CURs).
To start generating AWS Cost and Usage Reports, please follow these steps:
- 1.
- 2.On the navigation pane, choose Cost & Usage Reports.
- 3.Click Create Report. Give your report a name. For example,
general-cost-report
. Make sure that bothInclude resource IDs
andData refresh
settings are selected. - 4.Under delivery options, select the S3 bucket where your reports currently reside and click Verify OR create a new S3 bucket if you do not currently have one for the report. If you get a prompt to add a default bucket policy, accept it.
- 5.We recommend you create a path prefix in the next field (e.g.:
main
). Make sure thatHourly
,GZIP
andCreate New Report Version
are selected. Click Next, then click Review & Complete.
AWS should generate your first Cost and Usage Reports in 8-24 hours time.
- 1.
- 2.On the navigation pane, choose Cost & Usage Reports.
- 3.A list of your Cost and Usage Reports will be available. Click on the relevant one.
- 4.Click-through the S3 bucket folders until you select the latest report (ideally full month), and download to
csv.gz
file.
If it's available, it's much better to have a full-month report from the last month (not the current/ongoing month) so we can capture more variance. The report path will indicate the period like in this example:
20210501-20210601/
To export billing data from Google Cloud Platform (GCP) you have to create a billing report table in BigQuery.
- 1.
- 2.Run the following query in BigQuery after replacing
<PROJECT>
,<DATASET>
,<BILLING_ACCOUNT_ID>
and<INVOICE_MONTH>
with the name of the relevant project, the relevant BigQuery dataset you created in the previous step, the Billing Account ID, and the last month in the following formatyyyyMM
(e.g.:202105
):
This query will extract the relevant information for Cloud Storage.
SELECT
billing_account_id, service.id AS service_id,
service.description AS service_description, sku.id AS sku_id,
sku.description AS sku_description, usage_start_time,
usage_end_time, project.id AS project_id, location.location,
location.country, location.region, location.zone, cost, currency,
currency_conversion_rate, usage.amount AS usage, usage.unit,
usage.amount_in_pricing_units AS usage_in_pricing_units,
usage.pricing_unit, TO_JSON_STRING(credits) AS credits_array,
IFNULL((SELECT SUM(CAST(c.amount * 1000000 as int64)) FROM UNNEST(credits) c), 0) / 1000000 AS total_credits,
invoice.month, cost_type, adjustment_info.description AS adjustment_info_description,
adjustment_info.mode AS adjustment_info_mode, adjustment_info.type AS adjustment_info_type
FROM `<PROJECT>.<DATASET>.gcp_billing_export_v1_<BILLING_ACCOUNT_ID>`
WHERE service.description = "Cloud Storage" AND invoice.month = '<INVOICE_MONTH>' ORDER BY invoice.month, usage_start_time, usage_end_time;
- 1.If there are less than 16k records from the query, then you can simply click Save Results, then select
CSV
(local file) to download the report. - 2.If there are more than 16k records from the query, then you'll have to perform these additional steps:
- 1.Click on Query History
- 2.Click on the query you just ran
- 3.Click on the Temporary Table link (Destination Table) and click export to GCS
- 4.Chose
GZIP
compression - 5.For file location, browse and select a GCS bucket where you want to save the file
- 6.For the file name it would be better to use a wildcard in case more than one file needs be created, so you can put something like
report-*.csv.gz
(Read more about this here) - 7.When the export finishes you can download the files from your GCS bucket directly.
If there is more than 1 report file, you need to put all the files into a
.zip
or .tar.gz
file before uploading to Taloflow.If you are on Microsoft Azure startup credits and can only access your billing data from the Microsoft Azure Sponsorships portal, you cannot get an analysis performed by Taloflow. This is because this limited report does not have critical data or report columns necessary for cost analysis, including:
bandwidth
, chargeType
, unitOfMeasure
and productName
.Microsoft Azure has many different account types and some of these have their own specific ways of accessing the billing report for export. The following two sets of linked instructions work in the vast majority of cases. However, if these steps are not relevant to your account type, please contact us at [email protected] and we'll provide different instructions.
If you're using the Billing Export method, please use Actual Cost and a Daily export of last-month costs (or month-to-date costs if last-month is unavailable).
Last modified 1yr ago