Development notes: May 2008

Bash arithmetic/command expansion

Command expansion, I use a lot

echo something $( date +%s )

Backticks also work for command expansion

echo something `date +%s`

...but do they work any differently to $( )?

Conditionals, like an if statement, use square brackets

if [ "$end_hour" = "24" ]; then ....; fi

Arithmetic, like 1 + 2, uses this:

i=$(( i + 1 ))
let i+=1

See http://www.softpanorama.org/Scripting/Shellorama/arithmetic_expressions.shtml

Temporarily suspend a process

At times, you may find it necessary to temporarily suspend a process, and then resume its execution at a later time. The following two commands will suspend a process, and the resume it, respectively:

# kill -STOP 945
# kill -CONT 945

Bash getopts

Use named command line parameters in Bash scripts.

Evidence Based Scheduling

Currently only available in Fogbugz 6.

This is a system that learns from your work estimates, decides how good you are at estimating, and automatically adjusts as you go along to give a more realistic estimate. It works on the principle that people tend to consistently either over-estimate, under-estimate by the same factor. It will also automatically factor in and balance out any delays that occur.

Test Coverage Vim Mappings

From the vi text editor, see which tests cover the file you're working on, or which modules are covered by the test you're in.

Features on the way: Being able to know which tests cover a particular line of a file!

http://search.cpan.org/dist/Devel-CoverX-Covered/lib/Devel/CoverX/Covered.pm

Perforce: Diff -f only on files that have changed

p4 diff -f ./... | egrep -B 2 '^<|^>' | less

MySQL: Log to a file

tee [filename]

[commands to log]

notee

Show size of MySQL tables

show table status

Displays size of tables and indexes, in bytes.

Profiling MySQL

SET profiling = 1;

SET profiling_history_size = 100; -- maximum is 100

SHOW PROFILES; -- shows a list of recent commands

SHOW PROFILE; -- shows a very detailed breakdown of time spent on each database task

For example, if a lot of time is spent 'Sending data', try improving your indexes.

http://dev.mysql.com/doc/refman/5.0/en/show-profiles.html

MySQL auto increment tip

TRUNCATE a table to reset its auto_increment number

MySQL JOIN syntax

(remember the brackets):

SELECT COUNT(*) FROM search_fact JOIN (date_dm) USING (date_id) WHERE day = 22 AND month = 4 AND year = 2008;

or equivalently:

SELECT COUNT(*) from search_fact, date_dm WHERE search_fact.date_id = date_dm.date_id AND day = 22 AND month = 4 AND year = 2008;

Select all the search_fact rows that have a date_id that corresponds to 22/04/2008.

MySQL DELETE syntax

DELETE FROM result_set USING search, result_set WHERE search.id = result_set.id AND occurred >= '2008-04-26 00:00:00' AND occurred < '2008-04-27 00:00:00';

Delete from result_set, using a range of dates in search, matching on id.

SELECT COUNT(*) FROM result_set JOIN search WHERE search.id = result_set.id AND occurred >= '2008-04-26 00:00:00' AND occurred < '2008-04-27 00:00:00';

Select the rows that would be deleted, as above.

MySQL LEFT JOIN syntax

SELECT count(*) FROM search LEFT JOIN click ON search.id = click.id WHERE click.id IS NOT NULL;

or

SELECT count(*) FROM search, click WHERE search.id = click.id AND click.id IS NOT NULL;

or

SELECT count(*) FROM search, click WHERE search.id = click.id;

Count the rows in search that have a corresponding matching id row in click.

Five Ways to Improve Your Perl Programming

Article

MySQL tools

MySQL Workbench

Free, works on Windows only at time of writing.

MySQL Administrator, MySQL Query Browser and MySQL Migration Toolkit

All free, and work on Windows, Mac and Linux.
Source code also available.

Installing MySQL on linux

Download these files:

MySQL-server-community-5.0.45-0.rhel3.i386.rpm
MySQL-client-community-5.0.45-0.rhel3.i386.rpm
MySQL-shared-community-5.0.45-0.rhel3.i386.rpm
MySQL-devel-community-5.0.45-0.rhel3.i386.rpm

Run this command:

rpm -i MySQL*

NOTE: The default data directory is /var/lib/mysql
Make sure the /var partition has enough space for your database!

You can get the files from redhat.

validating xml files from the command line

It may need to be installed, but it really good at checking encodings, xml schemas etc..

http://xmlsoft.org/xmllint.html

xmllint --valid --noout test.xml

- worked for Rob H.

Hanging commands

If a command seems to be taking a very long time, e.g. an SQL query:

Check the diskspace: df -h

It could be that the /var parition is full (this is the default location for MySQL tables).

XSL regular expressions

NOTE: Variables must be declared in a root <xsl:template> node.


<xsl:variable name="id">
<xsl:analyze-string select="doc:entry/doc:id" regex="id/(\d+)">
<xsl:matching-substring>
<xsl:value-of select="regex-group(1)"/>
</xsl:matching-substring>
</xsl:analyze-string>
</xsl:variable>

XSL: generate the current date

For stylesheets processed using Java:

<xsl:stylesheet version="2.0" xmlns:java="java">

<today-date>
<xsl:value-of select="java:util.Date.new()" />
</today-date>

XSL: use a variable

<xsl:value-of select="$location"/>

XSL lookup table

The lookup file:

<?xml version="1.0"?>
<lookup>
<location id="1769" guide="TT00379a" average="TT123456"/>
<location id="1230" guide="TT003999" average="TT000001"/>
</lookup>

The code: NOTE: $id is a local variable.


<xsl:variable name="guide">
<xsl:for-each
select="document('lookup.xml')">
<xsl:value-of select="key('map',$id)/@guide"/>
</xsl:for-each>
</xsl:variable>

XSL: http requests

Getting the hostname given in the current http request can be done using XSP.

GNU screen

Type 'screen' after you log into a server.

Or type byobu for an enhancement to GNU screen!

"Allows users to quickly switch their .screenrc to any of the available profiles"
See also http://launchpad.net/byobu
byobu version 3.5
Screen version 4.00.03jw4 (FAU) 2-May-06

Byobu features:

Visible at bottom of screen at all times:

All screen tabs
Server welcome message
Uptime
Load
Hardware spec
[configurable to display anything you want]

Tab names show special characters to provide more info:

- (dash) means that window will be switched to with Ctrl-A, A
@ (at) means that tab has seen activity since you last saw it

Screen keyboard commands:

Ctrl-A, c -- to create a new terminal session
Ctrl-A, Ctrl-A -- to alt-tab
Ctrl-A, " -- to list the terminals
Ctrl-A, A -- to rename the terminal
Ctrl-A M -- (monitor for activity) Toggles monitoring of the current window
Ctrl-A _ -- (monitor for silence) Toggle

Best screen feature:

Log into the server.
Type screen to start a session.
Work as normal.
When finished, just close/kill the terminal program, don't log off.
Later, on any (other) PC, log into the server again.
Type screen -D -R to retrieve your session, just as you left it.

I don't know of a way to change the order of the tabs.

vi syntax higlighting

Use

:set syn=perl

:set syn=tt2

and

:highlight Comment ctermfg=Green guifg=Green

Combine STDOUT and STDERR

It's supposed to be done by adding 2>&1 to the end of the command, but that doesn't work for certain commands like time.

Instead make a script combine.sh containing this:

$@ 2>&1

Call it like this:

./combine.sh command_to_capture with parameters 2>&1 > outputfile

mount a directory from another host

in /etc/fstab:
//example.com/inetpub2 /nolmcs02/inetpub smbfs credentials=/root/nolmcs2-auth.txt,domain=NATIONAL,uid=noladmin,gid=noladmin,directio,hard 0 0

where /root/nolmcs2-auth.txt (or whatever you name it) is a file containing the username and password to use:

[root@nolappsdc01 ~]# cat /root/nolmcs2-auth.txt
username=
password=

You can mount this manually to test using:

mount -t smbfs //example.com/inetpub2 /nolmcs02/inetpub2 -o credentials=/root/nolmcs2-auth.txt,domain=NATIONAL,uid=noladmin,gid=noladmin,directio,hard

The credentials file is used so it will work on restart automatically. Instead of credentials=, you can use username=, etc. temporarily.

restart mysql

Usage: /etc/init.d/mysql {start|stop|restart|reload|force-reload|status} [ MySQL server options ]

Development notes