apache Custom Log Formats – 好用的參數

URL : http://httpd.apache.org/docs/2.2/mod/mod_log_config.html#formats

我的 sample code :

ServerName test.monster.tw
DocumentRoot "/m2/test"
LogFormat "%{%Y-%m-%d}t , %{%H:%M:%S}t , %D , %a , \"%q\" , \"%{Referer}i\" , \"%{User-agent}i\"" MONSTER_format
CustomLog "/m2/log/test.monster.tw/access.log" MONSTER_format

MTS 的 http config

SetEnvIf Request_URI /dot.gif MTS_icon
LogFormat "%{%Y-%m-%d}t , %{%H:%M:%S}t , %a , \"%q\" , \"%{Referer}i\" , \"%{User-agent}i\"" MTS_format
CustomLog "| /usr/sbin/cronolog /var/www/ts.monster.com.tw/log/access-%Y%m%d%H.log" MTS_format env=MTS_icon

apache default 的 combined format

LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\"" combined
CustomLog log/acces_log combined

以下是 apache 的資料

Format String Description
%% The percent sign
%a Remote IP-address
%A Local IP-address
%B Size of response in bytes, excluding HTTP headers.
%b Size of response in bytes, excluding HTTP headers. In CLF format, i.e. a ‘-‘ rather than a 0 when no bytes are sent.
%{Foobar}C The contents of cookie Foobar in the request sent to the server.
%D The time taken to serve the request, in microseconds.
%{FOOBAR}e The contents of the environment variable FOOBAR
%f Filename
%h Remote host
%H The request protocol
%{Foobar}i The contents of Foobar: header line(s) in the request sent to the server. Changes made by other modules (e.g. mod_headers) affect this.
%k Number of keepalive requests handled on this connection. Interesting if KeepAlive is being used, so that, for example, a ‘1’ means the first keepalive request after the initial one, ‘2’ the second, etc…; otherwise this is always 0 (indicating the initial request).
%l Remote logname (from identd, if supplied). This will return a dash unless mod_ident is present and IdentityCheck is set On.
%m The request method
%{Foobar}n The contents of note Foobar from another module.
%{Foobar}o The contents of Foobar: header line(s) in the reply.
%p The canonical port of the server serving the request
%{format}p The canonical port of the server serving the request or the server’s actual port or the client’s actual port. Valid formats are canonical, local, or remote.
%P The process ID of the child that serviced the request.
%{format}P The process ID or thread id of the child that serviced the request. Valid formats are pid, tid, and hextid. hextid requires APR 1.2.0 or higher.
%q The query string (prepended with a ? if a query string exists, otherwise an empty string)
%r First line of request
%s Status. For requests that got internally redirected, this is the status of the *original* request — %>s for the last.
%t Time the request was received (standard english format)
%{format}t The time, in the form given by format, which should be in strftime(3) format. (potentially localized)
%T The time taken to serve the request, in seconds.
%u Remote user (from auth; may be bogus if return status (%s) is 401)
%U The URL path requested, not including any query string.
%v The canonical ServerName of the server serving the request.
%V The server name according to the UseCanonicalName setting.
%X Connection status when response is completed:

X = connection aborted before the response completed.
+ = connection may be kept alive after the response is sent.
- = connection will be closed after the response is sent.

(This directive was %c in late versions of Apache 1.3, but this conflicted with the historical ssl %{var}c syntax.)

%I Bytes received, including request and headers, cannot be zero. You need to enable mod_logio to use this.
%O Bytes sent, including headers, cannot be zero. You need to enable mod_logio to use this.

防止盜圖的 code

SetEnvIf Referer "^http://www.example.com/" local_referal
# Allow browsers that do not send Referer info
SetEnvIf Referer "^$" local_referal
[directory images web]
   Order Deny,Allow
   Deny from all
   Allow from env=local_referal
[/directory]

這個更讚! 排除掉 gif / jpg / png / css / swf … 等等不要log的檔案

SetEnvIf Request_URI \.gif misc-request
SetEnvIf Request_URI \.jpg misc-request
SetEnvIf Request_URI \.png misc-request
SetEnvIf Request_URI \.css misc-request
SetEnvIf Request_URI \.swf misc-request
CustomLog logs/access_log env=!misc-request

整合一下, 這是 lazy 的 config

  ServerName lazy.monster.tw
  DocumentRoot "/home/webuser/lazy.monster.tw"
  SetEnvIf Request_URI \.gif misc-request
  SetEnvIf Request_URI \.jpg misc-request
  SetEnvIf Request_URI \.png misc-request
  SetEnvIf Request_URI \.css misc-request
  SetEnvIf Request_URI \.swf misc-request

  LogFormat "%{%Y-%m-%d}t , %{%H:%M:%S}t , %a , %u , %D , \"%r\" , \"%{Referer}i\" , \"%{User-agent}i\"" MONSTER_format

  CustomLog /m2/log/lazy.monster.tw/access.log MONSTER_format env=!misc-request

Simple Monster Tracking System – step by step

mts.js , 網上範例很多 , 這個是基本型:
70a13e19e02a2c85496d095bec62ba1b

這段 code 前面就是一連串的組出 z 變數(包括 OS , BROWSER , SCREEN WIDTH/HEIGHT …) , 然後用一個小點透過 apache 的 log 記錄下來.

這個 apache 要裝 mod_setenvif 或 mod_rewrite modules , 在 apache config 中設定 只 log 特定的 tracking data.

SetEnvIf Request_URI /dot.gif MTS_icon
LogFormat "%{%Y-%m-%d}t , %{%H:%M:%S}t , %a , \"%q\" , \"%{Referer}i\" , \"%{User-agent}i\"" MTS_format
CustomLog "| /usr/sbin/cronolog /var/www/ts.monster.com.tw/log/access-%Y%m%d%H.log" MTS_format env=MTS_icon

引用例:

fce31197e4af4ae225f2a76fb86e84cb

然後 apache log 會長這個樣子:

32e8d2437c1ca7c201d84d7e14f56c72

寫一段 PHP code:

$fp = fopen($mts_logfile,"r");

while ( $temp = fgetcsv($fp,$max_size)  ) {
  if ( count($temp)<1 ) continue;
  print_r($temp);
}
fclose($fp);

結果就類似這樣:

237de7c2773e273a224561d7bfcba4c7

再來就是處理 [3] 那邊的各種 data …

Monster LAMP Pack Lite – ver.317

Basic install emerge
lilo dhcpcd openssh syslog-ng vixie-cron screen ntp cronolog net-mail/mpack app-arch/sharutils unzip bind-tools trafshow traceroute

Linux  : 2.6.24-gentoo-r7
Apache : 2.2.10
Mysql  : 5.0.70-r1
PHP    : 5.2.8-pl2

PHP 的編法是

USE="apache2 berkdb bzip2 calendar cjk cli crypt curl gd gdbm hash iconv json mysql mysqli ncurses nls oci8-instant-client pcre readline reflection session simplexml spell spl ssl truetype unicode xml zlib" emerge -av php

package 有

samba :
postfix :
oracle instant client :
open-vm-tools : vmware 的 tools

emerge 這些 package:

重要軟體:
mysql php apache postfix

中等重要:
screen ntp samba

工具類軟體:
subversion vim open-vm-tools cronolog net-mail/mpack app-arch/sharutils unzip

記得

  • /etc/udev/rules.d/70-persistent-net.rule 砍掉
  • 改 net_DHCP
  • 砍 /tmp/*
  • check /etc/conf.d/clock , /etc/hosts , /etc/resolv.conf

安裝 apache / mod_memcache

準備好這幾個 tarball, 並且解開:

drwxr-xr-x  6 rimmon rimmon  576 Mar 14 17:36 mod_memcached_cache-0.1.0
drwxr-xr-x  7 rimmon rimmon  584 Mar 14 17:09 apr_memcache-0.7.0
drwxr-xr-x 12 rimmon rimmon 1416 Mar 14 16:55 httpd-2.2.11

Hypertext Transfer Protocol — HTTP/1.1 – 這可是現今最偉大的 protocol 呀!

The Hypertext Transfer Protocol (HTTP) is an application-level
protocol for distributed, collaborative, hypermedia information
systems. It is a generic, stateless, protocol which can be used for
many tasks beyond its use for hypertext, such as name servers and
distributed object management systems, through extension of its
request methods, error codes and headers
http://tools.ietf.org/html/rfc2616

Status Code Definitions
http://tools.ietf.org/html/rfc2616#section-10

204 No Content

The server has fulfilled the request but does not need to return an
entity-body, and might want to return updated metainformation. The
response MAY include new or updated metainformation in the form of
entity-headers, which if present SHOULD be associated with the
requested variant.

If the client is a user agent, it SHOULD NOT change its document view
from that which caused the request to be sent. This response is
primarily intended to allow input for actions to take place without
causing a change to the user agent’s active document view, although
any new or updated metainformation SHOULD be applied to the document
currently in the user agent’s active view.

The 204 response MUST NOT include a message-body, and thus is always
terminated by the first empty line after the header fields.

可以利用這點作一些事, 譬如記錄 user 的訪問, log 等等

apache 有用的 mod

*  www-apache/mod_chroot
      Latest version available: 0.5
      Latest version installed: [ Not Installed ]
      Size of files: 14 kB
      Homepage:      http://core.segfault.pl/~hobbit/mod_chroot/
      Description:   mod_chroot allows you to run Apache in a chroot jail with no additional files.
      License:       GPL-2

*  www-apache/mod_extract_forwarded
      Latest version available: 2.0.2
      Latest version installed: [ Not Installed ]
      Size of files: 13 kB
      Homepage:      http://www.openinfo.co.uk/apache/index.html
      Description:   Apache module that rewrites X-Forwarded-For to REMOTE_ADDR for reverse proxy configurations.
      License:       Apache-2.0 Apache-1.1

*  www-apache/mod_log_rotate
      Latest version available: 1.00
      Latest version installed: [ Not Installed ]
      Size of files: 4 kB
      Homepage:      http://www.hexten.net/wiki/index.php/Mod-log-rotate
      Description:   mod_log_rotate adds log rotation support to mod_log_config based on strftime(3)
      License:       Apache-2.0

oci / compile php with oracle instant client

小筆記一下…

oracle 那邊 download 這幾個 file

-rw-rw-r– 1 root portage 44414338 Feb 24 13:41 instantclient-basic-linux32-11.1.0.7.zip
-rw-rw-r– 1 root portage   607196 Feb 24 13:41 instantclient-sdk-linux32-11.1.0.7.zip

放到 /usr/portage/distfiles 這底下

然後 PHP 是這麼編的

USE="apache2 berkdb bzip2 calendar cjk cli crypt curl gd gdbm hash iconv json mysql mysqli ncurses nls oci8-instant-client pcre readline reflection session simplexml spell spl ssl truetype unicode xml zlib" emerge -av php