Saturday, November 24, 2007

Corporate IM gateway

Corporate Instant Messaging system.
Task: installation of corporate Instant Messaging gateway with possibility of logging of all conversations.
Including adjustment of transport for others IM networks (ICQ, MSN, AIM ) As well as authentication of users in LDAP.
The choice between LCS (Live Communication Server M$) and XMPP/Jabber has been made instantly ;) .
It is necessary to choose only which Jabber server to use.
From the considered variants:
chime (Java) http://www.codecobra.com/chime/
DJabberd (Perl) http://danga.com/djabberd/
ejabberd (Erlang) Home page and Community Site
jabberd14 (C) http://jabberd.org/
jabberd2 (C) http://jabberd2.xiaoka.com/
Openfire (Wildfire Server) (Java) http://www.igniterealtime.org/projects/openfire/
OpenIM (Java) http://www.open-im.net/
pretzel (Python) http://code.google.com/p/pretzel/
psyced (LPC) http://www.psyced.org/
Tigase (Java) http://www.tigase.org/
WPJabber (C) http://spik.wp.pl/jabber.html
xmppd.py (Python) http://xmpppy.sourceforge.net/

For comparison the following document has been taken: http://www.jabber.org/admin/jsc/

From the advantages listed above - ejabberd option has been choosen mostly for the following reasons :
1.Protocol-standards compatibility (XMPP Core, XMPP IM),
2.fully distributable
3.database can be replicated to many nodes .
4.The default database, Mnesia, is suitable for small, as well as big deployments.
5.code can be updated while ejabberd is running (feature of Erlang)
6.loading and unloading of modules while ejabberd is running is possible (feature of Erlang)
7.Modular design
Installation
I have install ejabberd on FreeBSD 6.2 , I think that there won`t be large differences in Linux operating system.
Installation performed by ports using “portinstall”.
Before starting installation - it is necessary to download following software:
diablo-jdk
tzupdater (JDK US DST Timezone Update Tool)
Let's execute following actions:
cd /usr/ports/distfiles/
lynx 'http://www.FreeBSDFoundation.org/cgi-bin/download?download=diablo-caffe-freebsd6-i386-1.5.0_07-b01.tar.bz2'
(Read and follow the license agreement)
Download via your web browser http://java.sun.com/javase/downloads/index.jsp
and copy tzupdater-x_x_x_x_x.zip to /usr/ports/distfiles/
installing ejabberd:
cd /usr/ports/
portinstall -p ejabberd-1.1.4
Don't enable support ODBC .
If all these components successfully compiled and installed - the following line should appear in /etc/rc.conf

ejabberd_enable="YES"

DNS Records

eJabberd server and clients, are able to use DNS SRV records for hostname resolution. DNS SRV records allow for delegation of services — by port — to other hosts.
I described setting for BIND , if You use other DNS server - please check following documentation: http://jabberd.jabberstudio.org/2/docs/section05.html#5_7
There are 3 SRV records that could be created for a eJabberd server installation:
_jabber._tcp.your_domain.com. 86400 IN SRV 5 0 5269 host.your_domain.com.
_xmpp-server._tcp.your_domain.com. 86400 IN SRV 5 0 5269 host.your_domain.com.
_xmpp-client._tcp.your_domain.com. 86400 IN SRV 5 0 5222 host.your_domain.com.

Replace “you_domain” with Your Domain Name and host with hostname, and don't forget to put “.” after the domain name.

Let's configure eJabber.
cp /usr/local/etc/ejabberd/ejabberd.cfg.example /usr/local/etc/ejabberd/ejabberd.cfg
cp /usr/local/etc/ejabberd/ejabberd.defaults.example /usr/local/etc/ejabberd/ejabberd.defaults
and edit fail:
vi /usr/local/etc/ejabberd/ejabberd.cfg
{acl, admin, {user, "alex"}}.
Let's add here our user who will have the Administrator permission.
Change “localhost” to Your Server name.
% Host name:
{hosts, ["host.your_domain.com"]}.
Configure support SSL:
vi /usr/local/etc/ejabberd/ejabberd.cfg
% Listened ports:
{listen,
[{5222, ejabberd_c2s, [{access, c2s},
{max_stanza_size, 65536},
starttls, {certfile, "/usr/local/etc/ejabberd/server.pem"},
{shaper, c2s_shaper}]},
{5223, ejabberd_c2s, [{access, c2s},
{max_stanza_size, 65536},
tls, {certfile, "/usr/local/etc/ejabberd/server.pem"},
{shaper, c2s_shaper}]},
% Use STARTTLS+Dialback for S2S connections
{s2s_use_starttls, true}.
{s2s_certfile, "/usr/local/etc/ejabberd/server.pem"}. Full path to file sertificate !!!
Now We will create the certificate :
cd /usr/local/etc/ejabberd/
openssl req -new -x509 -nodes -newkey rsa:1024 -days 3650 -keyout privkey.pem -out server.pem -subj
"/C=XX/ST=XX/L=XX/O=XX/OU=XX/CN=host.your_domain.com/emailAddress="postmaster@your_domain.com
cat privkey.pem >> server.pem
rm privkey.pem

...and start our server by: /usr/local/etc/rc.d/ejabberd start
(Try to telnet localhost 5222 or 5223 for SSL in order to check this.)

Now you have working and reliable jabber server.

LDAP

What about authentication abilities ? We need to authenticate users against LDAP server and to use LDAP Directory as vCard storage.
Edit the file - ejabberd.cfg

 vi /usr/local/etc/ejabberd/ejabberd.cfg
% Authentication method.
%{auth_method, internal}.
<---!!! Comment this line !!!!

% For LDAP authentication use these lines instead of above one:
{auth_method, ldap}.
{ldap_servers, ["ldap.your_domain.com"]}. % List of LDAP servers
{ldap_uidattr, "uid"}. % LDAP attribute that holds user ID
{ldap_base, "ou=People,dc=
your_domain,dc=com"}. % Search base of LDAP directory
{ldap_rootdn, "cn=jabber,ou=DSA,dc=
your_domain
,dc=com"}. % LDAP manager
{ldap_password, "password"}. % Password to LDAP manager

Now We want to use user`s LDAP-info as their vCard. In order to implement this - find and edit the following
line in section

% Used modules:

{mod_vcard_ldap, [{host, "ldapyour_domain.com"}]},
and then just restart eJabber /usr/local/etc/rc.d/ejabberd restart

Logging Messages

I usеd for this purpose "Bandersnatch" software. http://funkypenguin.co.za/
Originally , this soft works perfectly , but have a little issue with different encoding.
I need to log different languages such a France, Russian, Hebrew, Spain in my company.
So, I made a little correction in the Perl script and Data Base structure for "Bandersnatch".
If someone will be interested - please ask me by e-mail, I will send you by the reply
attached files.

...unpack "Bandersnatch" archive to special directory and make executable bandersnatch Perl script

chmod +x bandersnatch

Cofigure ejabberd:

Edit ejabberd.cfg and add this line to the 'modules' section:

{mod_service_log, [{loggers, ["bandlog.your_domain.com"]}]},

Add the following lines to 'listen' section to give eJabberd ability to listen for Bandersnatch connect.
% listen for Bandersnatch connections
{5526, ejabberd_service, [{ip, {127, 0, 0, 1}}, {access, all},
{hosts, ["bandlog.your_domain.com"],
[{password, "password"}]}]}


Create DB for Log System
mysql -u root -pPassword bandlog < bandersnatch.sql (I recommend use my file bandersnatch.sql)

Configure Bandersnatch:
edit config.xml
<server>
<connectiontype>tcpip</connectiontype>
<hostname>localhost</hostname>
<port>5526</port>
<secret>password</secret>
</server>
<component>
<name>bandersnatch@bandlog.your_domain.com</name>
</component>
<mysql>
<server>localhost</server>
<dbname>bandlog</dbname>
<username>bandlog</username>
<password>bandlog</password>
</mysql>

Do not forget to add to DNS a new A record ( like this example ):

bandlog A 192.168.1.1 ( jabber server's IP)

Now restart Your eJabberd server:
/usr/local/etc/rc.d/ejabberd restart
And run in "screen" bandersnatch:
./path/to/bandersnatch/bandrsnatch config.xml
if You see output like this:
Bandersnatch: Connected to Jabber server (localhost) ...
Bandersnatch: Connected to MySQL database (bandlog@localhost) ...
You have successfully installed the system !


Transports
I srongly recomend read befor:
http://wiki.blathersource.org/wiki/index.php/PyICQt
http://wiki.blathersource.org/wiki/index.php/PyAIMt
http://delx.cjb.net/pymsnt/docs/user.html
but if you have reached this point without a problem, with "transport" installation You should not have any problems.
I describe it together (ICQ, MSN, AIM) . At the moment of a writing of this article in OS FreeBSD port jabber-yahoo-2.3.2_2 has been marked IGNORE (broken)
please check for status of Yahoo transport on site http://yahoo-transport-2.jabberstudio.org/

So, let's install all needed software:
portinstall -p jabber-pyicq-transport-0.8a
portinstall -p jabber-pymsn-transport-0.11.2_2,1
portinstall -p jabber-pyaim-transport-0.8a


Add following "A" records to DNS
icq.your_domain.com
msn.
your_domain.com
aim.your_domain.com

Edit following rc scripts in /usr/local/etc/rc.d:
jabber-pyaim-transport
: ${jabber_pyaim_enable="YES"}
: ${jabber_pyaim_dir="/usr/local/lib/jabber/pyaim"}
: ${jabber_pyaim_piddir="/var/spool/ejabberd/pid"}
: ${jabber_pyaim_user="ejabberd"}

jabber-pyicq-transport
: ${jabber_pyicq_enable="YES"}
: ${jabber_pyicq_dir="/usr/local/lib/jabber/pyicq"}
: ${jabber_pyicq_piddir="/var/spool/ejabberd/pid"}
: ${jabber_pyicq_user="ejabberd"}

jabber-pymsn-transport
: ${jabber_pymsn_enable="YES"}
: ${jabber_pymsn_dir="/usr/local/lib/jabber/pymsn"}
: ${jabber_pymsn_piddir="/var/spool/ejabberd/pid"}
: ${jabber_pymsn_user="ejabberd"}

Do not forgot make symlink and change owner:
chown -R
/usr/local/lib/jabber/
chown -R
/var/spool/ejabberd/
ln -s /var/spool/ejabberd /var/spool/jabber

Now let's edit config of eJabberd server:
vi /usr/local/etc/ejabberd/ejabberd.cfg

in section
% Listened ports:
add
% listen for PyICQt connections
{5347, ejabberd_service, [{access, all},{host, "icq.your_domain.com",
[{password, "preved"}]}]},

% listen for PyMSNt connections
{5348, ejabberd_service, [{host, "msn.
your_domain.com",
[{password, "password"}]}]},

% listen for PyAIMt connections
{5349, ejabberd_service, [{host, "aim.
your_domain.com",
[{password, "password"}]}]}

Edit following config of transports:

jabber-pyicq.xml
<jid>icq.your_domain.com</jid>
<spooldir>/var/spool/ejabberd</spooldir>
<pid>/var/spool/ejabberd/pid/PyICQt.pid</pid>
<mainServer>127.0.0.1</mainServer>
<port>5347</port>
<secret>preved</secret>
jabber-pymsn.xml
<jid>msn.your_domain.com</jid>
<spooldir>/var/spool/ejabberd</spooldir>
<pid>/var/spool/ejabberd/pid/PyMSNt.pid</pid>
<mainServer>127.0.0.1</mainServer>
<port>5348</port>
<secret>password</secret>

<getAllAvatars/>
jabber-pyaim.xml
<jid>aim.your_domain.com</jid>
<spooldir>/var/spool/ejabberd</spooldir>
<pid>/var/spool/ejabberd/pid/PyAIMt.pid</pid>
<mainServer>127.0.0.1</mainServer>
<port>5349</port>
<secret>password</secret>


Last stap - add to /etc/rc.conf following lines:
jabber_pyicq_enable="YES"
jabber_pymsn_enable="YES"
jabber_pyaim_enable="YES"

Now restart Your eJabberd server:
/usr/local/etc/rc.d/ejabberd restart

and run all transports:
/usr/local/etc/rc.d/jabber-pyaim-transport start
/usr/local/etc/rc.d/jabber-pyicq-transport start
/usr/local/etc/rc.d/jabber-pymsn-transport start



P.S.
about a Jabber Client software You can find more here http://en.wikipedia.org/wiki/List_of_Jabber_client_software
I recommend "PSI" and "Gajem".







Sunday, November 11, 2007

PostgreSQL checkpoints

PostgreSQL checkpoints.

In documentation ( http://www.postgresql.org/docs/ ) PostgreSQL checkpoints are described as follows:
checkpoint_segments
Maximum distance between automatic WAL checkpoints, in log file segments (each segment is normally 16 megabytes). The default is three segments. This parameter can only be set in the postgresql.conf file or on the server command line.
checkpoint_timeout
Maximum time between automatic WAL checkpoints, in seconds. The default is five minutes (5min). This parameter can only be set in the postgresql.conf file or on the server command line.
checkpoint_warning
Write a message to the server log if checkpoints caused by the filling of checkpoint segment files happen closer together than this many seconds (which suggests that checkpoint_segments ought to be raised). The default is 30 seconds (30s). Zero disables the warning. This parameter can only be set in the postgresql.conf file or on the server command line.
I shall describe it is more expanded:

When the transaction log no longer has available space, the installation of a so-called control checkpoint takes place. This serves as an order for the system to remove all the unprocessed material to a disk, so the log can become available for use once again. Additionally, the control checkpoint may be installed not upon occasion, but after a certain period of time, typically standing at 5 min. When there is massive logging to the database, the transaction log can become overloaded too rapidly. This, in turn, will lead to a substantial deceleration of the aforementioned material removal to a disk.

Establishing the installation frequency of control checkpoints:

The installation of control checkpoints must take place every couple of minutes. If the installation takes place too frequently (for instance, every minute), the productivity of the system would notably decrease. To establish the current frequency, at which the system installs control checkpoints, you can use the log's analysis of timestamps. First, however, make sure that you've allowed access to the log. Check follow options in the file postgresql.conf:

log_timestamp = true

After that changes the file of a configuration will be automatically re-read, and you can observe in a logfile of server PostgreSQL following records:
2007-02-11 21:17:32 LOG: recycled transaction log file 0000000000000000
2007-02-11 21:17:33 LOG: recycled transaction log file 0000000000000001
2007-02-11 21:17:33 LOG: recycled transaction log file 0000000000000002
Estimate the period between two control points. It will allow you to define their frequency.
In the example resulted above you can see, that control points were made each 40 seconds that is too frequent operation ,
and that slows down productivity of system as a whole. By the way, be not surprised, having found out records with the same
time mark (see above). Frequently the same control point enters the name in the log several times.

Reduction of frequency of installation of Control points

Reduction of frequency of Control points leads to increase in number of files of anticipatory caching, created in data/pg_xlog. Each file has the size of 16 mbyte, that as a result can affect considerably the general free space on a disk. Installation by default means minimization of number of such files. For reduction of frequency of control points you should change following parameter:

checkpoint_segments = 3
Its initial value is equal 3.
Gradually increase this value until the interval between control
points not begins to be equaled to several minutes.
The next record in the logfile , which you can observe, can be like this:
LOG: XLogWrite: new log file created - consider increasing WAL_FILES

It means, that the parameter wal_files requiers increase in a file
postgresql.conf.





Tuesday, November 6, 2007

Postfix and Cyrus TLS howto

Cyrus Postfix AUTH TLS
Here comes TLS into play. Before we put it to work we need a certificate. Either you get it from a Certificate Authority whom you have to pay, or you create it for yourself. I will not describe how to become a Certificate Authority; instead I will show you how you can create a self signed .pem certificate.

This howto is meant as a practical guide, it does not cover the theoretical backgrounds.

And I would recommend will familiarize with the Documentation at leisure:

http://www.postfix.org/TLS_README.html


+++++++++ CERTIFICATE ++++++++++++++

Now we have to generate the certificate files needed for TLS:
mkdir /var/imap
cd /var/imap

openssl genrsa -des3 -rand /etc/hosts -out smtpd.key 1024

Enter a password for smtpd.key.
chmod 600 smtpd.key
openssl req -new -key smtpd.key -out smtpd.csr

Again, enter your password for smtpd.key.
Enter your Country Name (e.g., "CA").
Enter your State or Province Name.
Enter your City.
Enter your Organization Name (e.g., the name of your company).
Enter your Organizational Unit Name (e.g. "IT Department").
Enter the Fully Qualified Domain Name of the system (e.g. "mail.mydomain.com").
Enter your Email Address. (postmaster@mydomain.com)
The following information is optional:
Enter a challenge password.
Enter an optional company name.

openssl x509 -req -days 3650 -in smtpd.csr -signkey smtpd.key -out smtpd.crt
Again, enter your password for smtpd.key.
openssl rsa -in smtpd.key -out smtpd.key.unencrypted
Again, enter your password for smtpd.key.
mv -f smtpd.key.unencrypted smtpd.key
openssl req -new -x509 -extensions v3_ca -keyout cakey.pem -out ca-cert.pem -days 3650

Again, enter your password for smtpd.key.
Enter your Country Name (e.g., "DE").
Enter your State or Province Name.
Enter your City
Enter your Organization Name (e.g., the name of your company).
Enter your Organizational Unit Name (e.g. "IT Department").
Enter the Fully Qualified Domain Name of the system (e.g. "mail.mydomain.com").
Enter your Email Address. (postmaster@mydomain.com)



Make certificat and key file accessible by postfix and Cyrus
chown -R cyrus:mail /var/imap/


+++++++++++++++++++++++++++++++++++++++

+++++++++++++ POSTFIX +++++++++++++++++

Let's enable SMTP AUTH and TLS in Postfix

Add to the file /etc/postfix/main.cf following lines:

smtp_use_tls = yes
smtpd_use_tls = yes
smtpd_tls_CApath = /var/imap
smtpd_tls_CAfile = /var/imap/ca-cert.pem
smtpd_tls_cert_file = /var/imap/smtpd.crt
smtpd_tls_key_file = /var/imap/smtpd.key
smtpd_tls_loglevel = 1
smtpd_tls_received_header = yes


And uncomment in file /etc/postfix/master.cf lines:

smtps inet n - n - - smtpd
-o smtpd_tls_wrappermode=yes
-o smtpd_sasl_auth_enable=yes

And restart MTA /etc/init.d/postfix restart

++++++++++++++++++++++++++++++++++++++++


+++++++++++ Cyrus ++++++++++++++++

For access to IMAP server using secure authentcation add/uncomment following lines:

/etc/imapd.conf

tls_cert_file: /var/imap/smtpd.crt
tls_key_file: /var/imap/smtpd.key
tls_ca_file: /var/imap/ca-cert.pem
tls_ca_path: /var/imap/

/etc/cyrus.conf

imaps cmd="imapd -s -U 30" listen="imaps" prefork=0 maxchild=100
pop3s cmd="pop3d -s -U 30" listen="pop3s" prefork=0 maxchild=50

Then restart IMAP server /etc/init.d/cyrus2.2 restart

.



Friday, November 2, 2007

Filtering Image Spam With FuzzyOCR And SpamAssassin

Struggle against a spam.... An episode the third.


This article describes how to scan emails for image spam with FuzzyOCR. FuzzyOCR is a plugin for SpamAssassin which is aimed at unsolicited bulk mail containing images using different methods, it analyzes the content and properties of images to distinguish between normal and spam mails. Installation will reviewed on Debian(Etch).

I assume that SpamAssassin (and MailServer) is already installed and working :) and exist symlink /etc/mail/spamassassin (other case ln -s /etc/spamassassin /etc/mail/spamassassin ).
For the beginning We install necessary dependences:
aptitude install netpbm gifsicle libungif-bin gocr ocrad libstring-approx-perl libmldbm-sync-perl imagemagick tesseract-ocr libdbd-mysql-perl libdbi-perl libtie-cache-perl
Next step We download unpack and install the latest FuzzyOCR :
cd /usr/src/
wget http://users.own-hero.net/~decoder/fuzzyocr/fuzzyocr-3.5.1-devel.tar.gz
tar -zxvf fuzzyocr-3.5.1-devel.tar.gz
cd FuzzyOcr-3.5.1/

cp -r FuzzyOcr* /etc/spamassassin/ (include directory FuzzyOcr/ !!! )

Source directory /usr/src/FuzzyOcr-3.5.1/ contain directory sapmples/ with sample spam emails, that we need later for testing.
So, installation finished, now we start to configure it. All configs are in /etc/spamassassin/
In the file /etc/mail/spamassassin/FuzzyOcr.cf
uncomment follow line:
focr_global_wordlist /etc/mail/spamassassin/FuzzyOcr.words

The file /etc/mail/spamassassin/FuzzyOcr.words
is the predetermined list of words which goes with FuzzyOCR. You can recustomize оr to add it under the needs.
Replace these two lines
focr_bin_helper pnmnorm, pnminvert, pamthreshold, ppmtopgm, pamtopnm

focr_bin_helper tesseract
on following
focr_bin_helper pnmnorm, pnminvert, convert, ppmtopgm, tesseract

Finally we add or uncomment next lines:
focr_path_bin /usr/local/netpbm/bin:/usr/local/bin:/usr/bin

focr_preprocessor_file /etc/mail/spamassassin/FuzzyOcr.preps
focr_scanset_file /etc/mail/spamassassin/FuzzyOcr.scansets
focr_enable_image_hashing 2
focr_digest_db /etc/mail/spamassassin/FuzzyOcr.hashdb
focr_db_hash /etc/mail/spamassassin/FuzzyOcr.db
focr_db_safe /etc/mail/spamassassin/FuzzyOcr.safe.db
The last 4 lines - adjustment hashing instead of MySQL.
Now we can feed all samples-spam mails to spamassassin, for check of its connected with Fuzzy.
/usr/bin/spamassassin --debug FuzzyOcr < /usr/src/FuzzyOcr-3.5.1/samples/ocr-gif.eml > /dev/null

As you see FuzzyOCR is working.

Now restart Spamassassin and closely check (tail -f /var/log/mail.info) on presence of errors from spamassassin or Perl modules.
........Your SpamAssassin is now able to recognize image spam!