What is Raw Logs?
Export to an S3 storage
• Amazon storage
• Non-Amazon storage
Export to an FTP/SFTP storage
Export time intervals
Log path example
Log format
Log example
Log fields
What is Raw Logs?
Raw Logs is an option that enables an automatic export of CDN resource logs to your storage. Logs contain information about user requests sent to cache servers and pre-cache servers (if origin shielding is enabled).
The feature is paid. To activate, contact us via support@gcore.com. After activation, enable Raw Logs in your control panel and configure export to S3, FTP or SFTP storage.
Export logs to an S3 storage
Amazon storage
1. If you use our shielding option, you can see a checkbox "Add logs from origin shielding". Check the box if you want to receive logs from both edge servers and pre-cache shielding servers.
2. For storage type, select "Amazon".
3. Specify your access key ID. In your Amazon personal account, it is called "AWS access Key ID". You can find it using the instruction. An access key ID and secret access key are required to configure log export to your storage.
4. Specify your secret access key. In your Amazon personal account, it is called "AWS secret access Key". You can find it using the instruction.
5. Specify your AWS region — location of a server where your storage is hosted. This is optional: for most storages, the region is determined automatically. You can leave the field empty. But we recommend filling it out so to ensure that your logs are exported successfully.
6. Choose how to organize storage: to put logs of all CDN resources into one bucket or to use separate buckets for each CDN resource.
7. Specify bucket(s) for log export. Make sure you indicate an existing bucket. Otherwise, your logs cannot be exported. If you want to export logs to a specific folder within a bucket, specify a folder name.
8. Click "Save changes".
Non-Amazon storage
1. If you use our shielding option, you can see a checkbox "Add logs from origin shielding". Check the box if you want to receive logs from both edge servers and pre-cache shielding servers.
2. For storage type, select "Other".
3. Specify a hostname — a name that is assigned to a storage server within a network and is used instead of an IP address. If you use Gcore S3 storage, you can find its access key ID in your personal account in the "Hostname" field.
4. Specify your access key ID. Along with a secret access key, it is required to configure log export to your storage. If you use Gcore S3 storage, you can find its access key ID in your personal account in the "Access key" field.
5. Specify your secret access key. If you use Gcore S3 storage, you can find its secret access key in your personal account in the "Secret key" field.
6. Specify a bucket hostname — a bucket ID that is used by your S3 storage system in the {bucket_name}.{hostname} format. It is required to ensure that logs are exported to a correct bucket within a storage. This field is optional: for some storages, a bucket hostname is determined automatically. If you use Gcore or Yandex.Cloud storage, a bucket hostname is required. A bucket hostname of the Gcore storage looks as follows: {bucket name}.{hostname from step 3}. For example: examplename.s-ed1.cloud.gcore.lu. A bucket hostname of Yandex.Cloud storage looks as follows: {bucket name}.{Yandex.Cloud hostname}}. For example: examplename.storage.yandexcloud.net.
5. Specify a region — location ID of a server where your storage is hosted. This is optional: for some storages, the region is determined automatically. You can leave the field empty. If you use Gcore S3 storage, a location ID is required. You can find it in the "Details" of the storage. Your location ID is a part of your hostname to the first dot.
7. Choose how to organize storage: put logs of all CDN resources into one bucket or to use separate buckets for each CDN resource.
8. Specify bucket(s) for log export. Make sure you indicate an existing bucket. Otherwise, your logs cannot be exported. If you want to export logs to a specific folder within a bucket, specify a folder name.
9. Click "Save changes".
Export logs to an FTP/SFTP storage
1. If you use our shielding option, you can see a checkbox "Add logs from origin shielding". Check the box if you want to receive logs from both edge servers and pre-cache shielding servers.
2. Specify a hostname — a name that is assigned to a storage server within a network and is used instead of an IP address. If you use Gcore SFTP storage, you can find its hostname in the "Details" of the storage in the "Hostname" field. It looks as follows: drf.origin.edgecore.ru . Additionally, you can specify an FTP or SFTP storage port by adding a colon after the hostname. For example: mia.origin.gcore.lu:2200.
3. Specify a storage username. If you use Gcore SFTP storage, you can find its username in the "Details" of your storage in the "Storage/User name" field.
4. Enter your storage password.
5. Specify a folder for export. If you use Gcore SFTP storage, specify the root (home) folder where other folders originate from. You can find its name in the "Details" of your SFTP storage at the end of the "Upload path" field.
If you use an SFTP storage from another provider, clarify whether a root folder that includes other folders is created by default. If not, leave the field empty. If yes, specify a folder name.
6. Choose how to organize storage: put logs of all CDN resources into one folder or to use separate folders for each CDN resource. Then specify a folder name. If you specify a non-existent folder, logs will be exported to a root folder.
7. Click "Save changes".
Export time intervals
Logs are exported at the end of each hour. If you activate Raw Logs at 00:30, the first logs will be exported between 00:45 and 01:00, the next ones — between 01:45 and 02:00.
If CDN servers are not requested, an empty log file (± 20 bytes) will be sent to your storage.
You can see the status of the Raw Logs option in your control panel:
- "Pending" is a status for the time interval between the connection to a storage and the very first log export
- "OK" is a status showing that logs are exported from at least one CDN server
- "Failed" is a status indicating that an error occurred while connecting to a storage or that the service failed to export logs within 24 hours
- "Pause" is a status showing that the option is paused
Log path example
s3://log-bucket-name/2019/08/20/15/nodename_primarycname.domain.ru_access.log.gz
Log format
"$remote_addr" "-" "$remote_user" "[$time_local]" "$request" "$status"
"$body_bytes_sent" "$http_referer" "$http_user_agent" "$bytes_sent"
"$edgename" "$scheme" "$host" "$request_time"
"$upstream_response_time" "$request_length" "$http_range" "[$responding_node]"
"$upstream_cache_status" "$upstream_response_length" "$upstream_addr"
"$gcdn_api_client_id" "$gcdn_api_resource_id" "$uid_got" "$uid_set"
"$geoip_city_country_code" "$geoip_city" "$shield_type" "$server_addr" "$server_port"
"$upstream_status" "-" "$upstream_connect_time" "$upstream_header_time"
"$shard_addr" "$geoip2_data_asnumber" "$connection" "$connection_requests"
"$request_id" "$http_x_forwarded_proto" "$http_x_forwarded_request_id" "$ssl_cipher"
"$ssl_session_id" "$ssl_session_reused"
"$sent_http_content_type" "$tcpinfo_rtt"
Please don’t be surprised if you see a field that is not listed above. We occasionally add new fields. If some fields are added to logs, you will receive an email about it. New fields are added to the end of the line.
Log example
"0.0.0.0" "-" "-" "[26/Apr/2019:09:47:40 +0000]" "GET /ContentCommon/images/image.png HTTP/1.1"
"200" "1514283" "https://example.com/videos/10" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 YaBrowser/16.10.0.2309 Safari/537.36"
"1514848" "[dh-up-gc18]" "https" "origin.cdn.com" "1.500" "0.445" "157" "bytes=0-1901653" "[dh]"
"MISS" "10485760" "0.0.0.0:80" "2510" "7399" "-" "-" "KZ" "-" "shield_no" "0.0.0.0" "80" "206" "-" "0.000"
"0.200" "0.0.0.0" "asnumber" "106980391" "1" "c1c0f12ab35b7cccccd5dc0a454879c5" "-" "-"
"ECDHE-RSA-AES256-GCM-SHA384" "28a4184139cb43cdc79006cf2d1a4ac93bdc****" "r"
"application/json" "21"
Log fields
Not all fields are important. Some of them relate to our internal CDN system and are not meaningful for you. In the table below, we have highlighted such system fields in italics. Other fields can be helpful for traffic analysis or statistics.
| Field | Log value example |
Description |
|
$remote_addr |
0.0.0.0 |
User's IP address |
|
$remote_user (internal system variable) |
- |
Username used in Basic authentication |
|
[$time_local] |
[26/Apr/2019:09:47:40 +0000] |
Local time in Common Log Format |
|
$request |
GET /ContentCommon/images/image.png HTTP/1.1 |
HTTP method, requested file path, HTTP version |
|
$status |
200 |
Response status code from a CDN server |
|
$body_bytes_sent |
1514283 |
Number of bytes sent to a user, excluding the response header size |
|
$http_referer |
https://example.com/videos/10 |
Referrer - a URL requested by a user |
|
$http_user_agent |
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 YaBrowser/16.10.0.2309 Safari/537.36 |
User agent that was used to send a request (browser or other application) |
|
$bytes_sent |
1514848 |
Number of bytes sent to a user |
|
$edgename |
[dh-up-gc18] |
CDN server that forwarded a requested file |
|
$scheme |
https |
Protocol (HTTP or HTTPS) of a request |
|
$host |
cdn.example.com |
Requested hostname of a CDN resource |
|
$request_time |
1.500 |
Request processing time in seconds (accurate to milliseconds); time elapsed between the first bytes of a request were processed and logging after the last bytes were sent to a user |
|
$upstream_response_time |
0.445 |
Number of seconds (accurate to milliseconds) it took to receive a response from an origin. In case of multiple responses, commas and colons are used |
|
$request_length |
157 |
Request length (including request line, header, and request body) |
|
$http_range |
bytes=0-1901653 |
File fragment size in a Range request |
|
[$responding_node] |
dh |
Responding data center |
|
$upstream_cache_status |
MISS |
Status of a requested file in CDN cache: HIT is a status of a response served from CDN cache. STALE is a status of an outdated response that failed to update because an origin was not responding or responding incorrectly. UPDATING is a status of an outdated response that is still updating since a previous request. REVALIDATED is a status of a response that is identical to the one on an origin based on the proxy_cache_revalidate directive. EXPIRED is a status of a response that has expired in cache, but still matches the one on an origin. A request has been sent to an origin for re-caching. MISS is a status of a response that has been served directly from an origin, rather than from cache |
|
$upstream_response_length |
10485760 |
Response length from an origin in bytes. In case of multiple responses, commas and colons are used |
|
$upstream_addr |
0.0.0.0:80 |
Origin's IP address and port |
|
$gcdn_api_client_id (internal system variable) |
123 |
Your ID in our system |
|
$gcdn_api_resource_id (internal system variable) |
01 |
Your CDN-resource ID in our system |
|
$uid_got (internal system variable) |
- |
Cookie name and received user ID |
|
$uid_set (internal system variable) |
- |
Cookie name and provided user ID |
|
$geoip_country_code |
KZ |
User’s country code |
|
$geoip_city |
- |
User’s city code |
|
$shield_type (internal system variable) |
shield_no |
This field shows whether the shielding option is enabled: shield_old - enabled shield_no - disabled |
|
$server_addr (internal system variable) |
0.0.0.0 |
IP address of an Anycast zone or CDN server |
|
$server_port (internal system variable) |
80 |
Requested port |
|
$upstream_status |
206 |
Origin response code |
|
$upstream_connect_time |
0.000 |
Number of seconds (accurate to milliseconds) it took to access an origin server |
|
$upstream_header_time |
0.200 |
Number of seconds (accurate to milliseconds) it took to receive a response header from an origin server |
|
$shard_addr (internal system variable) |
0.0.0.0 |
IP address of a CDN server that was first to accept a request if the Cache Sharding feature is enabled |
|
$geoip2_data_asnumber |
asnumber |
Number of an autonomous system that sent a request |
|
$connection (internal system variable) |
2897494295 |
Connection serial number |
|
$connection_requests (internal system variable) |
1 |
Current number of requests made through a connection |
|
$request_id (internal system variable) |
c1c0f12ab35b7cccccd5dc0a454879c5 |
Unique request identifier generated from 16 random bytes, in hexadecimal form |
|
$http_x_forwarded_proto |
- |
Initial protocol of an incoming request (HTTP or HTTPS) |
|
$http_x_forwarded_request_id (internal system variable) |
- |
Initial ID of an incoming request |
|
$ssl_cipher (internal system variable) |
ECDHE-RSA-AES256-GCM-SHA384 |
Cipher name used for an established SSL connection |
|
$ssl_session_id (internal system variable) |
28a4184139cb43cdc79006cf2d1a4ac93bdc**** |
Session ID of an established SSL connection |
|
$ssl_session_reused (internal system variable) |
r |
The filed shows whether a session was reused (“ |
|
$sent_http_content_type |
application/json |
Value of the Content-Type HTTP header, indicating the MIME type of a transmitted file |
|
$tcpinfo_rtt |
21 |
Average time (latency) it takes to transfer a packet to/from a server. |