It is a paid option

You can receive CDN resource logs from CDN servers and shielding (if it is activated).

Contact us via chat or email to support@gcorelabs.com to enable the option.

Once the feature is enabled you will be able to set it up in your personal account->CDN section->Raw logs->Receive raw logs.

We use two protocols to deliver logs: S3 and FTP.

You should set up a storage on your side to start receiving the logs.

 

Configure logs receiving on S3

• Storage is Amazon 
• Storage is not Amazon  

Configure logs receiving on FTP

Log path example

Log format

Example log

Log Fields

Configure logs receiving on S3

Storage is Amazon 

If you want to receive logs to your Amazon storage, specify AWS Access Key ID, AWS Secret Access Key and Region (optional).

Check the box Add logs from origin shielding to receive logs from CDN servers and shielding.

Select where you want to send logs:

• into one folder

•  into different folders

Note! The system doesn't create folders (bucket) in your storage. Specify an existing folder (bucket).

Click Save Changes.

 

Storage is not Amazon  

If you choose Other, fill in Hostname, Access Key ID and Secret Access Key.

If you use Yandex.Cloud fill in Bucket hostname.

If you use our S3 storage, fill in Bucket hostname and Region.

Check the box Add logs from origin shielding to receive logs from CDN servers and shielding.

Select where you want to send logs:

• into one folder

•  into different folders

Note! The system doesn't create folders (bucket) in your storage. Specify an existing folder (bucket).

Click Save Changes.

Configure logs receiving on FTP

Specify Hostname, Login, Password and Prepend folder (optional).

Note! A port number can be added to your Hostname if needed: <hostname>:<port>.

Check the box Add logs from origin shielding to receive logs from CDN servers and shielding.

Select where you want to send logs:

• into one folder

•  into different folders

Note! The system doesn't create folders (buckets) in your storage. Specify an existing folder (bucket).

Click Save Changes.

 

Log path example 

 s3://log-bucket-name/2019/08/20/15/nodename_primarycname.gcdn.co_access.log.gz 

Log Format

"$remote_addr" "-" "$remote_user" "[$time_local]" "$request" "$status" 
"$body_bytes_sent" "$http_referer" "$http_user_agent" "$bytes_sent" 
"$edgename" "$scheme" "$host" "$request_time" 
"$upstream_response_time" "$request_length" "$http_range" "[$responding_node]" 
"$upstream_cache_status" "$upstream_response_length" "$upstream_addr" 
"$gcdn_api_client_id" "$gcdn_api_resource_id" "$uid_got" "$uid_set" 
"$geoip_city_country_code" "$geoip_city" "$shield_type" "$server_addr" "$server_port"
"$upstream_status" "-" "$upstream_connect_time" "$upstream_header_time" 
"$shard_addr" "$geoip2_data_asnumber" "$connection" "$connection_requests" 
"$request_id" "$http_x_forwarded_proto" "$http_x_forwarded_request_id" "$ssl_cipher" 
"$ssl_session_id" "$ssl_session_reused";

The log format can be changed at any time. New fields are always added to the end of the line. When analyzing logs, only the known fields from the beginning of the line should be taken into account.

Example Log

"0.0.0.0" "-" "-" "[26/Apr/2019:09:47:40 +0000]" "GET /ContentCommon/images/image.png HTTP/1.1" "200" "1514283" "https://example.com/videos/10" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 YaBrowser/16.10.0.2309 Safari/537.36" "1514848" "[dh-up-gc18]" "https" "origin.cdn.com" "1.500" "0.445" "157" "bytes=0-1901653" "[dh]" "MISS" "10485760" "0.0.0.0:80" "2510" "7399" "-" "-" "KZ" "-" "shield_no" "0.0.0.0" "80" "206" "-" "0.000" "0.200" "0.0.0.0" "asnumber" "106980391" "1" "c1c0f12ab35b7cccccd5dc0a454879c5" "-" "-" "ECDHE-RSA-AES256-GCM-SHA384" "28a4184139cb43cdc79006cf2d1a4ac93bdc****" "r"

Log Fields

Not all variables are important, some of them relate to the internal system of our CDN and are not useful for client's analysis. They are shown in small print. All other variables can be useful for analyzing traffic or collecting statistics.

Field	Example log value	Description
$remote_addr	0.0.0.0	IP address of the client
$remote_user (internal system variable)	-	User name supplied with the Basic authentication
[$time_local]	[26/Apr/2019:09:47:40 +0000]	Local time in the Common Log Format
$request	GET /ContentCommon/images/image.png HTTP/1.1	HTTP request type, requested path to a file, HTTP version
$status	200	Response status
$body_bytes_sent	1514283	Number of bytes sent to a client, disregarding the response header
$http_referer	https://example.com/videos/10	Referrer: specify the URL of the user
$http_user_agent	Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 YaBrowser/16.10.0.2309 Safari/537.36	User Agent which a user applied to request content
$bytes_sent	1514848	Number of bytes sent to a client
$edgename	[dh-up-gc18]	The name of the responded CDN-node
$scheme	https	Request's protocol (HTTP ot HTTPS)
$host	cdn.example.com	CDN-resource's Hostname received from a request
$request_time	1.500	Request processing time counted in seconds with a milliseconds resolution; time elapsed from the first bytes sent by the client till the log record, made after a client sent last bytes
$upstream_response_time	0.445	Time spent on receiving the response from the upstream server; time counted in seconds with millisecond resolution. Time of several responses is shown in commas and colon
$request_length	157	Request length (including request line, header and request body)
$http_range	bytes=0-1901653	File fragment value received by Range request
[$responding_node]	dh	The name of the responded DC
$upstream_cache_status	MISS	CDN cache status of a requested file: HIT –  file's given from the cache STALE – file outdated as the origin server didn't respond or responded incorrectly due to cache's being updated UPDATING - file outdated as it's being updated after the earlier request was sent REVALIDATED – the proxy_cache_revalidate directive was enabled and NGINX verified that the currently cached content is still valid EXPIRED - cache lifetime ended, but the file is relevant to the origin's file. A request is sent to the origin server for the recache MISS – file isn't given from the cache, it's proxied from the origin server
$upstream_response_length	10485760	Length of the response obtained from the origin server; stored in bytes. Lengths of several responses are separated by commas and colons
$upstream_addr	0.0.0.0:80	IP address and port of the prigin server
$gcdn_api_client_id (internal system variable)	123	Your ID in our system
$gcdn_api_resource_id (internal system variable)	01	Your CDN-resource ID in our system
$uid_got (internal system variable)	-	Cookie name and received user ID
$uid_set (internal system variable)	-	Cookie name and provided user ID
$geoip_country_code	KZ	User’s country code
$geoip_city	-	User’s city code
$shield_type (internal system variable)	shield_no	Specifies if a shield is enabled or not shield_old - enabled shield_no - disabled
$server_addr (internal system variable)	0.0.0.0	IP address of anycast zone or edge-server
$server_port (internal system variable)	80	Port of the server which accepted a request
$upstream_status	206	Origin server's response code
$upstream_connect_time	0.000	Time spent to access the origin server; time counted in seconds with millisecond resolution.
$upstream_header_time	0.200	Time spent to receive a header from the origin server;  time counted in seconds with millisecond resolution.
$shard_addr (internal system variable)	0.0.0.0	IP address of an edge-server that received the first client request if the Cache Sharding feature is enabled.
$geoip2_data_asnumber	asnumber	The autonomous system number where the user request came from.
$connection (internal system variable)	2897494295	Sequential number of connection.
$connection_requests (internal system variable)	1	Current number of requests made through a connection.
$request_id (internal system variable)	c1c0f12ab35b7cccccd5dc0a454879c5	Unique request identifier generated from 16 random bytes, in hexadecimal.
$http_x_forwarded_proto	-	Initial protocol of the incoming request (http or https).
$http_x_forwarded_request_id (internal system variable)	-	Initial ID of the incoming request.
$ssl_cipher (internal system variable)	ECDHE-RSA-AES256-GCM-SHA384	Returns the name of the cipher used for an established SSL connection.
$ssl_session_id (internal system variable)	28a4184139cb43cdc79006cf2d1a4ac93bdc****	Returns the session identifier of an established SSL connection.
$ssl_session_reused (internal system variable)	r	Returns “r” if an SSL session was reused, or “.” otherwise.