Perl subroutine parameters

Call subs like this:

sub_name(

      param1 => 'value1',

      param2 => 'value2',

);

Define subs like this:

sub sub_name {
    my $p1      = {@_}->{param1};
    my $p2      = {@_}->{param2};
    # etc

}

Media players for Linux

• Rhythmbox. Great for internet radio and mp3s
• VLC
• Movie Player (Ubuntu). Sucks for audio.

Text-based web browsers

• http://w3m.sourceforge.net/
• http://lynx.browser.org/
• http://www.jikos.cz/~mikulas/links/

How to search PDFs on ubuntu

1. sudo aptitude install poppler-utils
2. pdftotext file.pdf
3. grep [search text] file.txt

Run an SQL script

• (a) pipe it into the program:  mysql < script.sql
• (b) source it from within mysql:  source script.sql

See also batch commands

Ways to access Google Reader on Blackberry

• Blackberry browser - Google Reader site doesn't work, site says it's loading too slowly and fails.
• Blackberry browser - add all RSS feeds manually as bookmarks - too tedious.
• Opera Mini - Google Reader works. Opera settings:
• Mobile view: off
• Full screen: on
• View Google Reader in: Mobile view (link at bottom of web page)
• Apps on Blackberry Curve 8900, Platform 5.2.0.67, App Version 5.0.0.681
• BeReader 1.1.49 (BerryReader, free trial) - I logged in and selected 'choose feed', it crashed with "Uncaught exception: Application berryreader(421) is not responding; process terminated".
• Viigo 3.1.507 - Works great for its built-in feeds, but adding my Google Reader results in "no feeds" warning and "authentication error". Also can't change font size or wrap titles, so you only see the first 4-6 words of the title. Loading the article is fast though.

Things wrong with FosWiki WYSIWYG editor

http://foswiki.org/ (Not FossWiki)

• Sometimes this invisible string appears at the bottom of the document, and you can't add anything more in WYSIWYG mode, until you go into Wiki Text mode and delete it: \x{feff}\x{feff}\x{feff}
• When an empty bullet point is at the end of a list, it often gets stuck there and cannot be removed in WYSIWYG mode.
• Often, pressing enter in part of the document in WYSIWYG mode causes the focus to jump up to the start of the document, and the enter is put in there, instead of where the cursor was.

Test MySQL date calculations

This is a way to test certain SQL queries involving dates. Say a query is supposed to select "Last Saturday", regardless of the date on which it runs:

SELECT DATE_SUB(CURDATE(), INTERVAL WEEKDAY(DATE_SUB(CURDATE(), INTERVAL 5 DAY)) DAY) AS 'Last Saturday';

How do you know if it really works on all days? This command will iterate through all the days in the month, and run the query with each date to see what it gets:

for day in $(seq 1 30)

do

    echo Nov $day, $(mysql -u sdkain -h db21 lf_wh -e"SELECT DATE_SUB('2010-11-$day', INTERVAL WEEKDAY(DATE_SUB('2010-11-$day', INTERVAL 6 DAY)) DAY) AS 'Last Sunday'")

done

Remember to test for when the year changes, and on leap years.

Automatically diagram an SQL database

Use cases:

• Designing a new schema from scratch, using a GUI
• Automatically rendering a diagram of an existing SQL schema

2010:

• The least worst option is MySQL Workbench. The GUI is poor: http://wb.mysql.com/
• This one uses Javascript. No features, but it works: http://code.google.com/p/database-diagram/

See also database-schema-diagrams-in-2022

Making a Perl module

Object oriented convention in Perl:

sub new
{  
    my ($class, %params) = @_;
    my $self = \%params;
    bless $self => $class;
    return $self;
}

Use Firefox as if it were WWW::Mechanize::Shell

Depends on plugin Mozrepl: https://github.com/bard/mozrepl/wiki
Use WWW::Mechanize::Firefox

HTTPS in Perl

After Crypt::SSLeay or IO::Socket::SSL is installed, LWP will automatically work for URLs starting https://

Have vi detect perl mason .html files

In ~/.vim/ftdetect/mason.vim:

" Mason detection
au BufRead,BufNewFile *
\   if search('^\s*<%\(args\|once\|init\|perl\|shared\|attr\)>\s*$','n')
\|    set filetype=mason
\|  endif

Eclipse features

• Syntax highlighting
• Code folding
• Go to source of library
• Look up Javadoc/Perldoc
• Automatic refactoring
• File navigator

Perl tools

Perl Needs Better Tools

IDEs

• Padre is free, supposedly it knows more about Perl internals than other IDEs.
• Syntax highlighting: View|Style|Evening
• Current line highlight colour: Tools|Preferences|Appearance|Editor Current Line Background Colour
• Eclipse is free but, seems very heavyweight. Extensible with plugins (Version control, etc. See EPIC for Perl)
• Komodo is paid, but is explicitly designed for Perl. It has a remote debugging feature, which works! You have to install a script on the remote server, and have access to the remote fileystem to display the source code.
• NetBeans is free
• IntelliJ IDEA is paid, and has no Perl plugin

Text editors

• vim - I use this a lot. It has keyword lookup (if you can configure it), a powerful but difficult-to-use scripting language, and tabs, but no debugging support. Lots of scripts/add-ons available.
• Emacs - some of my colleagues use it, can be very powerful with add-ons/perl modules
• UltraEdit
• Textpad - Windows only. Nice to use, has plugins
• Sublime - A nice text editor

Further reading

• Why should I use an IDE?
• TODO: [an article on coding horror or somewhere "what would a good IDE look like?"] where is it?

Catch runaway processes on Mac

#!/bin/sh

# kill runaway processes at 100% CPU usage (Mac OSX)
export PNAME=$1
if [[ ! $PNAME ]]
then
    echo "usage: runaways.sh [process name]"
    exit
fi

export PPERCENT='100.0'
SECONDS_TO_TEST=20
THRESHOLD=15

SECONDS_RUNAWAY=$(top -l $SECONDS_TO_TEST | perl -lne'BEGIN{$c=0};/$ENV{PNAME}\s+$ENV{PPERCENT}\%/ && do { $c++ };END{print$c}');
if [[ $SECONDS_RUNAWAY -gt $THRESHOLD ]]
then
    echo "killing $PNAME at $PPERCENT"%...
    PID=$(top -o cpu -l 2 | perl -lne'/$ENV{PNAME}\s+$ENV{PPERCENT}\%/ && print $1')
    if [[ $PID ]]
    then

        echo "Killing $PNAME"

        kill -9 $PID

    else

        echo "$PNAME does not exceed the threshold for a runaway"
    fi
else
    echo "$PNAME is not runaway"
fi

Combine STDOUT and STDERR

It's supposed to be done by adding 2>&1 to the end of the command, but that doesn't work for certain commands like time.

Instead make a script combine.sh containing this:

$1 $2 $3 $4 $5 $6 $7 $8 $9 ${10} ${11} ${12} ${13} ${14} ${15} ${16} ${17} ${18} ${19} ${20} 2>&1

Call it like this:

./combine.sh command_to_capture with parameters 2>&1 > outputfile 

Mount a directory using Samba

mount -t smbfs //host/subdir hostname -o username=yourusername,fmask=555,dmask=555,ro 

Don't use Samba though, use SSHFS.

XSL array/hash/lookup table

The lookup file:

<?xml version="1.0"?>

<lookup>

<location id="1769" guide="TT00379a" average="TT123456"/>

<location id="1230" guide="TT003999" average="TT000001"/>

</lookup>

The code: NOTE: $id is a local variable:

<!-- look up the country guide ID -->

<xsl:variable name="guide">

<xsl:for-each select="document('lookup.xml')">

<xsl:value-of select="key('map',$id)/@guide"/>

</xsl:for-each>

</xsl:variable>

Using Apache::Registry

When a script uses a config file that is a perl package, and that package gets changed, it must be reloaded. This can happen either through Apache::StatINC, or Apache::Reload. With both methods you must wait for about a second for perl to 'let go' of the config packages before performing any actions on those files. Otherwise all sort of intermittent weirdness will occur, (e.g. when running the script multiple times in a test and changing the config files inbetween).

Using StatINC_UndefOnReload On can cause all manner of strange errors if you run a script/module, then do :w on the module in vim but don't change anything, then run the script/module again.

Using PerlFreshRestart On causes lots of "Attempt to free unreferenced scalar" errors

Also, when changing the contents of perl configuration 'packages', the modification date on the file must be changed for any Apache module to reload it. Remember that move doesn't change the modification date of files, but copy does. So copy and unlink files, don't move them.

Make vi recognise different filetypes

In ~/.vimrc:

" load custom filetypes

:filetype on
au BufNewFile,BufRead *.tt set filetype=html

Perl test coverage

Select files to cover and run the tests, capturing coverage data:

perl -MDevel::Cover=,+ignore,.*,-select,^lib t/path/to/test.t

Meaning:

• ignore all paths...
• except ones starting with "lib"

After it's run, at the command prompt, type this to generate the HTML report:
cover

And then open this file in your web browser:
cover_db/coverage.html

(docs)

Timing out a system call

as seen in perldoc perlipc

 

#!/usr/bin/perl

use warnings;

my @cmd = @ARGV;
my @ls = ();
my $t = 10;

    eval {
        local $SIG{ALRM} = sub { die "alarm went off" };
        alarm $t;
            @ls = `@cmd`;
        alarm 0;
    };

if (! @ls) {
        print "No response after $t seconds\n";
}
else {
        foreach (@ls) { print $_; }
}

# actually never gets to here
if ($@ and $@ !~ /alarm went off/) { die "Command timed out after $t seconds\n"; };

Perl: Tailing a file

seek() can remove the end-of-file marker:

while (1) {
    while () {
        # do something
    }
    sleep $for_a_while;
    seek( FILE, 0, 1 );
}

Perl: Simplest way to get a formatted date

use POSIX 'strftime';
my $now_string = strftime( "%Y%m%d", localtime );

Perl: convert a number of seconds into days, hours, mins, secs

printf("That took %d"."d %d"."h %d"."m %d"."s",(gmtime($seconds))[7,2,1,0])); 

URI encoding in Java

Deprecated Apache Commons HttpClient 3.x:

 

import org.apache.commons.httpclient.URIException;

import org.apache.commons.httpclient.util.URIUtil;

String uriTest1(String uri) {

    String encodedUri = null;

    try {

        encodedUri = URIUtil.encodePath(uri,"UTF-8");

    } catch (URIException e) {

        System.err.println("Caught URI Exception");

        e.printStackTrace();

    }

    return encodedUri;

}

URIEncoder encodes to application/x-www-form-urlencoded (i.e. spaces turn to +)

import java.io.UnsupportedEncodingException;
import java.net.URISyntaxException;
import java.net.URLEncoder;

String uriTest2(String uri) {
    String encodedUri = null;
    String encoding = "UTF-8";
    try {
        encodedUri = URLEncoder.encode(uri, encoding);
    } catch (UnsupportedEncodingException e) {
        System.err.println("Unsuported encoding: "+encoding);
        e.printStackTrace();
    }
    return encodedUri;
}

java.net.URI requires the whole URI to be input, but you can extract just the URI encoded query part (turns spaces to %20)

import java.net.URI;

private static String uriTest3(String path) {
    String encodedUri = null;
    URI uri = null;
    try {
        uri = new URI("http","bbc.co.uk","/search/news/",path,null);
    } catch (URISyntaxException e) {
        System.err.println("Caught URI Syntax Exception");
        e.printStackTrace();
    }
    encodedUri = uri.getRawQuery();
    return encodedUri;
}
See also.

Have unattended scripts look after themselves

# Send all output to a logfile and supress input
typeset LOG="/tmp/${0##*/}.out"
mv $LOG ${LOG}.old >/dev/null 2>&1
[[ -t 1 ]] && echo "Writing to logfile '$LOG'."
exec > $LOG 2>&1
exec < /dev/null 2<&1

OpenZaurus documentation

• Wiki: http://wiki.openzaurus.org/Main_Page (No longer accepting edits - I have a mediawiki backup)
• Blog: http://www.openzaurus.org/wordpress/ (Last post: April 26th, 2007 -- post says OpenZaurus is dead and suggests that the future is Angstrom)
• Sourceforge: http://sourceforge.net/projects/openzaurus/ (No downloads, no issue tracking, nothing is accessible)

Sending an email to yourself

You can't send an email from your blackberry phone, through a gmail account, and expect to see it arrive.
It just doesn't work. For reasons explained here:
http://www.blackberry.com/btsc/search.do?cmd=displayKC&docType=kc&externalId=KB18070
http://www.google.com/support/forum/p/Google+Mobile/thread?tid=196b521f8b4b7a76&hl=en

SSH on Blackberry

MidpSSH

I want to use SSH on my Blackberry, for example with the program MidpSSH.
I am using a Blackberry 8900 on the Orange UK network.

When I try to connect to a server using MidpSSH, I get the error: "Session error: Writer: Invalid parameter".
Apparently I need to set my APN to fix this. What is an APN? List of APNs

On my device in Options | Advanced Options | TCP/IP there's a tickbox for "APN Settings Enabled".
There I can set APN: orangeinternet But this doesn't help. Tried rebooting.

Perhaps it won't work because:

Orange would have had to setup the normal Orange GPRS APN on your account, ... which they don't do for blackberry users (understandably, we're all meant to use BIS).

(I can also see in Advanced Options | Host Routing Table, all the "Host Routing Information" entries have APN: blackberry.net)


Orange


UPDATE 10/09/10:
I called Orange and spoke to the "online services" team (439). A nice man called Ben (CMTS Darlington) found some instructions on their system for using SSH from a Blackberry. They recommended downloading the client from xk72.com. But then he spoke to the 2nd line support team for me and they said that SSH is "Not supported anymore".
Well, what does that mean? That I won't be able to get an SSH connection? Or that I will be able to get a connection, but they won't give me any help should a problem arise? How do I know if their system is working properly for me to try and make an SSH connection?

Here's an unrelated typical example of poor Orange customer service.

PaderSyncSSH

I tried connecting with PaderSyncSSH free trial from the Blackberry App Store.

When I set the Networking connection type in 'Misc. Settings' to "Direct TCP", and set my APN to orangeinternet, I get the error: Connect failed: Error opening socket. java.io.IOException: Peer refused the connection. (Note: This is the same message as I get when connecting to a server that does not offer SSH).
With Networking connection type 'BES/MD5' I got the error: Connect failed: Error opening socket. java.io.IOException: Invalid URL parameter (that's the same as MidpSSH!)
With Networking connection type 'BIS-B' I got the error: Connect failed: Connect failed. ab: Session.connect: java.io.IOException: BIS-B connection failed. (and a recommendation to try Direct-TCP, or have the server listen on a port above 1024 because some carriers block lower ports).

UPDATE 14/09/10:

After connecting once via wifi, I tried connecting to an SSH server via Orange's GRPS phone data network again, and it worked perfectly. Success! I discovered that sometimes I get the error "Connect failed: Error opening socket. java.io.IOException: Peer refused the connection", but other times it works just fine.




Telnet

I succeeded in making a telnet connection to bbc.co.uk:80 through GPRS and issuing a GET HTTP/1.1 command, which resulted in the HTML text of the BBC homepage.



Wifi

I succeeded in using PaderSyncSSH SSHing to shellmix.com:30 (newuser/newuser) when I connected my BlackBerry to a wifi network. It worked really well! MidpSSH gave the same error (but then it doesn't have a network type selection option like PaderSync does).

Configuring log4j for maven

To debug: Add -Dlog4j.debug to the jvm parameters.

The files log4j.properties or log4j.xml must be on the classpath.

Easily import properties file in JUnit

Pass this parameter to the jvm:
-DPropertyManager.file=/path/to/props.properties

Then in the program:
import junitx.util.PropertyManager; // from http://sourceforge.net/projects/junit-addons/ ?
PropertyManager.getProperty("my.key");

Subversion on a Mac 10.4 (Tiger), behind a firewall

The Subversion project recommends:

• THIS ONE WORKS: MacPorts - has a .dmg installer for 10.4 
• need to set environment variable http_proxy if behind a firewall
• then the sudo port -v selfupdate command
• then do sudo port install subversion 
• Warning: The installed version of Xcode (2.3) is known to cause problems. Version 2.5 or later is recommended on Mac OS X 10.4. Download Xcode tools.
• Fink - wouldn't install for me, saying "Cannot perform symlink test on this volume because of a permissions problem". Running "Repair Disk Permissions" utility as recommended doesn't help at all.
• Apple's Developer Tools - I don't think it's on the CD. Couldn't find a list of apps anywhere. When I tried, it installed a lot of stuff but I couldn't see svn anywhere.
• OpenCollabNet - ?

Remove all login restrictions for MySQL

Start mysqld (or mysqld_safe) with the --skip-grant-tables option

Cocoon documentation

Current:

• Wiki: http://wiki.apache.org/cocoon/FrontPage (question about docs. Answer: Understand How ASF works)
• Official v2 site: http://cocoon.apache.org/
• Jira bug tracking: https://issues.apache.org/jira/browse/COCOON (issue about docs - there are several more issues regarding out-of-date contribution procedures)
• Cocoon 3 alpha: http://people.apache.org/~reinhard/c3-ref/html/
• List of mailing lists: http://svn.apache.org/repos/asf/cocoon/site/site/mail-lists.html (also a browsable SVN repository of the documentation website - will updating this fix the live site?)
• Main site mailing lists: http://cocoon.apache.org/2.1/1175.html
• Searchable mailing list archives: http://markmail.org/search/?q=list%3Aorg.apache.cocoon.users
• and http://markmail.org/search/?q=list%3Aorg.apache.cocoon.dev

Dead or out-of-date:

• org.apache.cocoon.docs - is this dead?
• This link to Cocoon Bugzilla is dead (it should be Jira now anyway): http://cocoon.apache.org/2.0/howto/howto-bugzilla.html
• Many of the links on this menu are dead: http://cocoon.apache.org/2.0/
• The "View, Edit or comment" link at the bottom of this page is dead: http://cocoon.apache.org/

Criticism:

• Struts vs. Cocoon: Why Cocoon lost the battle.
• The Death of XSLT in Web Frameworks

Use DBUnit

Add to the classpath:

• dbunit-2.4.7
• slf4j-api-1.6.1
• slf4j-simple-1.6.1
• mysql-connector-java-5.0.8

Example program:

import java.io.FileOutputStream;
import java.sql.*;
import org.dbunit.database.*;
import org.dbunit.dataset.*;
import org.dbunit.dataset.xml.*;
// database connection
Class driverClass = Class.forName("com.mysql.jdbc.Driver");
Connection jdbcConnection = DriverManager.getConnection("jdbc:mysql://host:port/dbname","user","pass");
IDatabaseConnection connection = new DatabaseConnection(jdbcConnection);
// prevent error: Potential problem found: The configured data type factory
// 'class org.dbunit.dataset.datatype.DefaultDataTypeFactory' might cause problems with the current database 'MySQL'
// (e.g. some datatypes may not be supported properly).
connection.getConfig().setProperty(DatabaseConfig.PROPERTY_DATATYPE_FACTORY, new MySqlDataTypeFactory());
// full database export
IDataSet fullDataSet = connection.createDataSet();
FlatXmlDataSet.write(fullDataSet, new FileOutputStream("dbname"));

Sprintf in Java

import java.text.MessageFormat;

int a = 3;
String b = "coders";

System.out.println(
    MessageFormat.format("{0} Java {1}.", String.format("%04d", a), b)
);

Sample XMLUnit test class

import java.io.IOException;
import org.apache.commons.httpclient.*;
import org.apache.commons.httpclient.methods.*;
import java.util.HashMap;
import org.custommonkey.xmlunit.*;

public class MyTest extends XMLTestCase {
    private HttpClient client;
    private String pageTitle = "something";
  
    protected void setUp() {
         client = new HttpClient();        
        // declare XML namespaces
        HashMap m = new HashMap();
        m.put("agg", "http://www.bbc.co.uk/aggregation/1.0");
        m.put("atom", "http://www.w3.org/2005/Atom");
        NamespaceContext ctx = new SimpleNamespaceContext(m);
        XMLUnit.setXpathNamespaceContext(ctx);
    }

    public void testVersion() throws Exception {
        String pageXML = getXML("http://example.com/version");
        assertXpathEvaluatesTo("2010-07-29", "/version/release-date", pageXML);
    }
  
    public void testTitle() throws Exception {
        String pageXML = getXML("http://example.com/test");
        assertXpathEvaluatesTo("items about "+pageTitle, "/agg:page//atom:feed/atom:title[1]", pageXML);
    }
  
     private String getXML(String uri) throws IOException {
        GetMethod method = new GetMethod(uri);
        int statusCode = client.executeMethod(method);
        assertEquals("HTTP GET " + uri, HttpStatus.SC_OK, statusCode); // 200 OK
        return method.getResponseBodyAsString();
     }

     public void TearDown() {
     }

}

Testing XML with Java

Testing

• XMLUnit
• JUnit - most popular

XML

• org.w3c.dom
• javax.xml.xpath (Parsing an XML Document with XPath, The Java XPath API)
• org.jdom.xpath (JDom and Xpath, required libs)

with Java

• Selenium
• Eclipse
• Maven + Jetty

3rd party javadocs in Eclipse

• Download and unzip the 3rd party package's source
• Open the Package Explorer view in Eclipse
• Navigate to [your project] | Referenced libraries | [3rd party library] | right-click | Properties
• Enter the source directory in the Java Source Attachment | Location path

There is also a Javadoc Location field that accepts a URL (didn't work for XMLUnit or JUnit though)

How to run a Selenium/JUnit test

• Install Eclipse 3.5
• Download JUnit
• Download Selenium RC
• Run java -jar selenium-server-1.0.3/selenium-server.jar
• Create a Java project in Eclipse
• Right-click on the project name and go to Properties | Java Build Path | Add External JARs
• add JUnit JAR
• all the Selenium Server JARs
• all the Selenium Java JARs
• Run a program like this (This one doesn't actually work; the assertion fails)

package com.example;

import junit.framework.TestCase;
import com.thoughtworks.selenium.DefaultSelenium;
import com.thoughtworks.selenium.Selenium;

public class GoogleTest extends TestCase {
 private static Selenium selenium;
 protected void setUp() {
  selenium = new DefaultSelenium("localhost" , 4444 , "*firefox", "http://www.google.com");
  selenium.start();
  selenium.setSpeed("2000");
 }
 public final void testTitle() {
  selenium.open("http://www.google.com");
  selenium.type("//input[@title='Google Search']", "I am using selenium");
  selenium.click("//input[@value = 'Google Search']");
  assertTrue(selenium.isElementPresent("/html/head/title"));
 }
 protected void tearDown() {
  selenium.stop();
 }
} 

oXygen Eclipse plugin

oXygen XML/XSLT: http://www.oxygenxml.com/download_oxygenxml_editor.html#Eclipse

• it's not all that, but step-through-debugging does work
• have to manually set the file association defaults to Oxygen types
• unreadable default colour scheme for Oxygen filetypes. It is difficult to import syntax-highlighting preferences in Eclipse. You're better off setting them yourself.
• to use step-through debugging, you need to:
• 'Configure Transformation Scenario' (under XML/XSL menu - it appears when you open an XML or XSL file)
• open the 'Debug Scenario' perspective in the XML/XSL menu
• setting the input parameters for debugging is fiddly
• the XML file being used as input must be "part of the project"

Nginx high-performance webserver

Nginx is a high-performance HTTP server. http://wiki.nginx.org/Main

Seems to be used by quite a lot of sites and a few people have noticed nginx 404 error pages by facebook and youtube indicating they may also be using it? http://wiki.nginx.org/NginxWhyUseIt

Comes also with a memcached module to talk with memcached directly: http://www.igvita.com/2008/02/11/nginx-and-memcached-a-400-boost/

And has embedded perl in it as standard: http://wiki.nginx.org/NginxEmbeddedPerlModule

And call perl directly from SSI
<!- # perl sub="module::function" arg="parameter1" arg="parameter2"... >

For XSLT fans: http://wiki.nginx.org/NginxHttpXsltModule

Eclipse notes

• File associations are under Preferences | General | Editors | File Associations
• Background colour is under: Preferences | General | Editors | Text editors 
• and Preferences | oXygen | Editor
• Syntax highlighting is under: Preferences | XML | XML Files | Editor | Syntax Coloring
• and Preferences | oXygen | Editor | Syntax highlighting
• Eclipse plugins site: http://eclipse-plugins.info/eclipse/
• How to import/export syntax highlighting colours
• No word wrap!

Start Apache with a custom configuration

/path/to/main/httpd -f /path/to/your/custom/httpd.conf -k start

Uptime

99.8% uptime allows for 17.52 hours downtime per year.
That seems reasonable.

For-each over static values in XSL

When you want to do this:

<xsl:for-each select="$some_value in ('a','b','c','d')"><!-- invalid? -->

You have to do this:

<xsl:variable name="items">
    <item>a</item>
    <item>b</item>
    <item>c</item>
    <item>d</item>
</xsl:variable>

<xsl:for-each select="$items/item">
    <xsl:choose>
    <xsl:when test="$some_value = .">

Remote file browsing

• Mount a remote filesystem via SFTP
• (Mac): http://pqrs.org/macosx/sshfs/
• (Ubuntu): Use software centre
• GUI
• Try http://sshfs-gui.darwinports.com/
• or Mac ports: sudo port install sshfs-gui
• To start: sshfs hostname: mountpoint
• To stop: fusermount -u mountpoint
• Target Management (Remote file browsing in Eclipse via SSH): http://www.eclipse.org/dsdp/tm/

Eclipse plugins

Eclipse 3.5

• Target Management (Remote file browsing, via SSH): http://www.eclipse.org/dsdp/tm/
• works great (but sshfs is simpler)
• Instead of that, see also http://wills-tech-notes.blogspot.com/2010/07/remote-file-browsing.html
• XSLT: http://www.eclipse.org/webtools/
• can't download through update manager, it's too slow
• supposed to offer XML/XSL features, but I can't see any
• Oxygen XML/XSLT: http://www.oxygenxml.com/download_oxygenxml_editor.html#Eclipse
• Works. To use step-through debugging,you need to open the 'Debug Scenario' in the XML or XSL menu (it appears when you open an XML or XSL file)
• vim -- why not use vim editor inside Eclipse? No good solution, apparently (none of these were tested):
• http://www.viplugin.com/viplugin/ (2009 - costs a few pounds)
• http://eclim.org/vim/ (recent)
• http://vrapper.sourceforge.net/ (recent)
• http://sourceforge.net/projects/vimplugin/ (2007, but states it supports syntax files)
• Full Screen: http://code.google.com/p/eclipse-fullscreen/ (just works) 
• P4WSAD: Perforce plugin: http://www.perforce.com/perforce/products/p4wsad.html (had to download zip file from FTP site and install manually, as automatic install URL was too slow)
• Plugin can be slow to use if you browse the repository directly, but is faster if you check out, then do right-click | Import as project, and use the standard Eclipse Navigator or Project tab to work with the files.

Installing Catalyst

http://wiki.catalystframework.org/wiki/installingcatalyst#Debian.2C_Ubuntu_.28.2A.deb.29

apt-cache search catalyst

How to set up apt-get on ubuntu

NOTE: Yum is designed for use on redhat-like systems using RPMs.
Ubuntu is based on debian, so uses apt-get/aptitude and .deb files.

On a fresh linux install:

• sudo apt-get update
• "You will have to enable the component called 'universe'": Go to System > Administration > Software Sources. Under the Ubuntu Software tab, tick the box for Community maintained Open Source Software (universe).
• sudo apt-get install XXX

How to create a PDF

• Save as PDF from OpenOffice
• http://www.pdfonline.com/convert-pdf/

To where has yum installed a package?

Yum works with RPMs:

rpm -ql packagename

How to use Saxon XSL 2 processor

java -jar saxon9he.jar -xsl:/path/to/file.xsl -s:/path/to/file.xml -o:/path/to/output.xml parameter1='value1' parameter2='value2'

How to get your subdomains to resolve

Mac:

• The wrong way: Edit /etc/hosts
• The right way: Add subdomains to your network preferences 'Search domains' (in TCP/IP tab)

Windows:

• No idea - someone please tell me

Burning .gbi files to CD

In Windows:

• MagicISO
• gBurner
  

In Linux:

• K3B - doesn't seem to work
  
• You can't burn .gbi files natively in Linux, you can only burn .iso files
• There is daa2iso (gbi to iso), but it may need to be compiled for Linux.
• Or use the Windows version like this: wine daa2iso.exe name.gbi name.iso

POST contents of a file

#!/usr/bin/perl

use strict;
use warnings;
require LWP::UserAgent;
use HTTP::Request;

my $ua = LWP::UserAgent->new;
$ua->timeout(10);
$ua->env_proxy;

my $url = $ARGV[0];
my $file = $ARGV[1];
my $head = $ARGV[2];

my $text = do { local( @ARGV, $/ ) = $file or die $!; <> } ;

my $headers;
if ($head) {
$headers = HTTP::Headers->new(Accept=> $head);
}

my $req = HTTP::Request->new('POST', $url, $headers);
$req->content($text);
my $response = $ua->request($req);

print $response->content;

Convert a string to UTF8

If the string is in iso-8859-1 encoding:

my $string_iso_8859_1 = decode("iso-8859-1", $string);
my $string_utf8 = encode("utf8", $string_iso_8859_1);

Text file endings

Mac OS: \r
Unix: \n
Windows: \r\n

parser error : Start tag expected, '<' not found

What this error means:
You're not giving the parser the XML you think you are.

Check for non-ascii characters

perl -lne'until (! $_) { s/^(.)//; $c = $1; print $_ if ord($c) > 127 }' file.html

Roll an 'if' statement into a single line in vim

Ctrl-V
s/\n\t\+/ /g

or

type JJJ

Which Perl modules are installed?

To run: perl script.pl $(cat list_of_modules.txt)

#!/usr/bin/perl
use strict;
use warnings;

foreach my $mod (@ARGV) {
(my $fn="$mod.pm")=~s|::|/|g; # Foo::Bar::Baz => Foo/Bar/Baz.pm
if (eval { require $fn; 1; }) {
print "Module $mod loaded ok\n";
} else {
print "Could not load $mod. Error Message: $@\n";
}
}

# Thanks Perl Monks

Also try typing perldoc perllocal to see a list of all installed modules.

XSL tips

• In XSL templates, when generating new static elements with attributes, instead of <xsl:element name="foo"><xsl:attribute name="bar"><xsl:value-of select="'baz'"> etc, you can just do <foo bar="baz">! The main use for <xsl:element> and <xsl:attribute> is for dynamically named elements and attributes.

• Print debug statements like this: <xsl:message><xsl:text>min_score = </xsl:text><xsl:value-of select="$min_score"/></xsl:message>
  

Stop XML::LibXSLT and XML::LibXML connecting to w3.org

If you have this line in your XML/XSL:

<!DOCTYPE xml PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">

Then your parser may connect to w3.org for validation. If w3 blocks you for hammering their server, then you may get this error message: http error : Operation in progress

http error : Operation in progress

If you don't want/need validation against the DTD (and you probably don't), you can turn off that feature in XML::LibXML:

1.70
$parser = XML::LibXML->new(load_ext_dtd => 0);

< 1.70
$parser = XML::LibXML->new();
$parser->load_ext_dtd(0);

Tailing all the logs files

Truncate all the log files:
find . -name "*.log" -exec cp /dev/null {} \;

Tail all the log files at once:
tail -f $(find . -name "*.log")

Free counter

Needs no registration, works with Blogger: http://www.e-zeeinternet.com/

Working with iptables on Redhat linux

iptables is the default firewall software on Redhat Linux

Display a list of ports it's restricting:
sudo /sbin/iptables -L

This is what it looks like when it's not restricting anything:

Chain INPUT (policy ACCEPT)
target prot opt source destination

Chain FORWARD (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination


If it doesn't look like that and you need to open a port, edit this file:
/etc/sysconfig/iptables

It might look something like this:

*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:RH-Firewall-1-INPUT - [0:0]
-A INPUT -j RH-Firewall-1-INPUT
-A FORWARD -j RH-Firewall-1-INPUT
-A RH-Firewall-1-INPUT -i lo -j ACCEPT
-A RH-Firewall-1-INPUT -p icmp --icmp-type any -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m tcp --dport 80 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT
-A RH-Firewall-1-INPUT -j REJECT --reject-with icmp-host-prohibited
COMMIT


Add the line in bold above to open up port 80.

Restart like this:
/etc/init.d/iptables restart

Min/Max for XSLT

Obtaining the MIN:
<!-- Create a variable named $minEventDate containing the MIN date -->
<xsl:variable name="minEventDate">
<xsl:for-each select="event">
<xsl:sort select="@date" data-type="text" order="ascending" />
<xsl:if test="position() = 1">
<xsl:value-of select="@date" />
</xsl:if>
</xsl:for-each>
</xsl:variable>

Obtaining the MAX:
<!-- Create a variable named $maxEventDate containing the MAX date -->
<xsl:variable name="maxEventDate">
<xsl:for-each select="event">
<xsl:sort select="@date" data-type="text" order="descending" />
<xsl:if test="position() = 1">
<xsl:value-of select="@date" />
</xsl:if>
</xsl:for-each>
</xsl:variable>

XSLT1 processing, using Perl

Example taken from XML::LibXSLT docs, code commented and replaced because of old versions of libs:
$ perl -MXML::LibXML -le'print $XML::LibXML::VERSION'

1.69

$ perl -MXML::LibXSLT -le'print $XML::LibXSLT::VERSION'

1.59


Code:

#!/usr/bin/perl

use strict;
use warnings;
use XML::LibXSLT;
use XML::LibXML;

my $xslt = XML::LibXSLT->new();

#my $source = XML::LibXML->load_xml(location => 'file.xml');
my $parser = XML::LibXML-> new();
my $source = $parser->parse_file('file.xml');

#my $style_doc = XML::LibXML->load_xml(location=>'file.xsl', no_cdata=>1);
my $parser2 = XML::LibXML-> new();
my $style_doc = $parser2->parse_file('file.xsl');

my $stylesheet = $xslt->parse_stylesheet($style_doc);

my $results = $stylesheet->transform($source);

#print $stylesheet->output_as_bytes($results);
print $results->toString;

Hide number sending on Blackberry Bold

• Open phone application
• Press menu
• Choose 'options'

Rename multiple files in bash

for i in *.JPG; do mv "$i" "${i/.JPG}".jpg; done

or

for i in *.jpg; do mv "$i" "`basename $i .jpg`.JPG"; done

Ping a host, then sleep and loop in Windows

@echo off
for /L %%a in (1,1,2016) do (
date /t
time /t
ping -n 30 yahoo.co.uk
ping localhost -n 360 > nul
)

Pipe this into a file to test your internet connection.

It pings yahoo.co.uk 30 times, then sleeps for 5 minutes.

It repeats 2016 times; That is one week.

Using a custom Apache config

apachectl -f ~/httpd/conf/httpd.conf -k graceful

tail -f ~/httpd/logs/error_log

Select fields from a hashref in Perl

my @wanted_fields = qw/link id title image/;
my %display_fields;
@display_fields{@wanted_fields} = @{ %{$thing->to_hashref} }{@wanted_fields};

Add a password to a MySQL login

Many applications require there to be a password.

SELECT host, user FROM mysql.user;

SET PASSWORD FOR 'bob'@'%.loc.gov' = PASSWORD('newpass');

Perl DBI

This is the one you want:

$ary_ref = $dbh->selectcol_arrayref($statement);

Perl map command

%hash = map { get_a_key_for($_) => $_ } @array;

http://perldoc.perl.org/functions/map.html

How to convert a tab separated file into a Perl hashref


open(LIST,'<',$datafile) || die "Failed to open ".$datafile;
my @header = split("\t", <list>);
chomp(@header);
my @nodes = ();
while (my $line = <LIST>) {
chomp($line);
my $i=0; my %node = map { $header[$i++] => ($_?$_:'[empty field]') } split("\t", $line);
push @nodes, \%node;
}
close(LIST);

print Dumper(\@nodes);


Simple log4perl config

log4perl.rootLogger=DEBUG, STDERR
log4perl.appender.STDERR=Log::Log4perl::Appender::Screen
log4perl.appender.STDERR.layout=PatternLayout
log4perl.appender.STDERR.layout.ConversionPattern=[%d] [%p %c] - %m%n

Poor man's sleep command for Windows batch files

for roughly 20 seconds wait:

ping 127.0.0.1 -n 20 > nul

Using an XML configuration file for Log::Log4perl

XML configuration file:

<!--
* General configuration for log4perl
* LOGFILE.filename may be set by the application
* All logging goes to STDOUT (CONSOLE)
-->
<log4perl>
<log4perl.rootlogger>WARN, CONSOLE, LOGFILE</log4perl.rootLogger>

<log4perl.appender.console>Log::Log4perl::Appender::Screen</log4perl.appender.CONSOLE>
<log4perl.appender.console.layout>PatternLayout</log4perl.appender.CONSOLE.layout>
<log4perl.appender.console.layout.conversionpattern>[%d] [%p %c] - %m%n</log4perl.appender.CONSOLE.layout.ConversionPattern>

<log4perl.appender.logfile>Log::Log4perl::Appender::File</log4perl.appender.LOGFILE>
<!-- Let logging come to STDERR on the console, and *pipe* into a log so that unexpected errors are caught
<log4perl.appender.logfile.filename>/optional/absolute/path/to/logfile.log</log4perl.appender.LOGFILE.filename>
-->
<log4perl.appender.logfile.mode>append</log4perl.appender.LOGFILE.mode>
<log4perl.appender.logfile.layout>PatternLayout</log4perl.appender.LOGFILE.layout>
<log4perl.appender.logfile.layout.conversionpattern>[%d] [%p %c] - %m%n</log4perl.appender.LOGFILE.layout.ConversionPattern>

<!-- Add different log levels for specified modules
<log4perl.logger.clickthrough.transform>INFO</log4perl.logger.ClickThrough.Transform>
-->
</log4perl>

How to read the configuration:

my $log4perl = XML::Simple->new(ForceArray=>0, KeyAttr=>[])->XMLin( $log4perl_config_filename );
my $logname = $0;
$logname =~ s/\.pl$/.log/;
$log4perl->{'log4perl.appender.LOGFILE.filename'} = $logname;
Log::Log4perl::init($log4perl);
my $logger = Log::Log4perl->get_logger(__PACKAGE__);

Confluence calendar


{calendar:id=nameOfCalendar|title=Title of Calendar|defaultView=month}

It publishes in iCal.

Simple Perl CGI and LWP

#!/usr/bin/perl

use CGI;

my $q = CGI->new;
print $q->header();

my $url = $q->param('url');

###

require LWP::UserAgent;

my $ua = LWP::UserAgent->new;
$ua->timeout(10);
$ua->proxy('http', 'http://your_proxy_here:80');

my $response = $ua->get($url);

if ($response->is_success) {
print $response->content; # or do fancy stuff here
}
else {
die $response->status_line;
}

Record and playback macros in vi

qN - start recording, where N is a-z
@N - play back your recording, where N is a-z
@@ play back the same recording again

Ruby Net::HTTP cheat sheet

http://august.lilleaas.net/blog/ruby-net-http-cheat-sheet

Kill a process in Windows

taskkill /f /im AnnoyingProcess.exe

List all processes in Windows

wmic /output:c:\process.txt process get description,executablepath

Date/Time perl modules

• Date::Manip - Big. Needs several step sto convert to/from eopch seconds. But you could just subtract 1 week's worth of seconds.
• Date::Calc - Less big, simpler. Has Add_Delta_Days. Doesn't generate BST from epoch seconds (on Mac).
• POSIX - has strftime. Can easily generate epoch seconds from the current time. Lacking in conversion to date?
• DateTime - The best and easiest.
  

How to list differences between Perforce labels

#!/bin/bash

NOW="/tmp/now_tmp"
BEFORE="/tmp/before_tmp"
LABEL=$1

USAGE="usage: "${0%}" [label]"
if [ "x$LABEL" == "x" ]; then echo $USAGE; exit; fi

p4 changes -l -i ./... > $NOW
p4 changes -l -i ./...@$LABEL &gt $BEFORE
for change_num in $(diff $NOW $BEFORE | perl -lne'/. Change (\d+)/ && print $1');
do
    p4 describe $change_num |perl -pne'do { print "-----------------------------------------------------------\n\n"; exit } if m{Differences \.\.\.}'
done

XML parsing fun with Perl

• XML::Simple - No XPath, just simple nested hashrefs. Good options for collapsing keys. Useful for configuration files.

• XML::Twig - Designed to handle sub-trees of very large documents, fast.

• XML::XPath - find() method produces a nodelist, which is just an arrayref (see XML::XPath::XMLParser). It's dumb to have to "re-parse" nodes in order to look further down. Maybe I'm missing something?

• XML::TreeBuilder or HTML::TreeBuilder - No XPath, but a splendid look_down() method. Returns objects on which you can also call look_down().

• XML::LibXML and XML::LibXML::XPathContext - The best.
• But HTML source often requires tidying into XML
• And there is fun to be has converting entities

use XML::LibXML;
use XML::LibXML::XPathContext;
$parser = XML::LibXML->new();
$xml = $parser->parse_string($self->raw_xml);
$xpath = XML::LibXML::XPathContext->new($xml->documentElement);
$xpath->registerNs('atom', 'http://www.w3.org/2005/Atom');
print $xpath->findvalue("/atom:feed/atom:title");


Nothing seems good for testing XML paths, tags and attributes.

Misc Ruby stuff

• Net::HTTP - fetch a webpage

• Write to a file:
File.open(local_filename, 'w') {|f| f.write(doc) }

• Read from a file:
File.open(local_filename, 'w') {|f| f.write(doc) }

• stringify: .inspect
  

Simple Perl parallelisation

@waitlist = (10,20,16,12,8,22);

foreach $item (@waitlist) {
    pipe *{$item},RETHAND;
    unless ($pid = fork()) {
        # Child process
        sleep $item;
        print RETHAND "Completed $item\n";
        exit();
    }
}

foreach $item (@waitlist) {
    $response = <$item>;
    $now = localtime();
    print "$now - $response";
}

Replace sleep with a more useful command.

Apache::Registry / Perl config files

• Apache::Reload causes more problems than it fixes
• Apache::StatINC only works on the @INC path which is set by Apache before scripts are run

• Apache::Registry - just don't bother
  

• Don't use Apache's PerlFreshRestart On, it will break things
• If parsing Perl syntax config files from Apache, remember it takes a few seconds for Perl to 'let go' after making changes.
  

Perl testing/profiling

• Devel::DProf - subroutine profiler
• Devel::NYTProf - the best one, apparently
• Devel::SmallProf - line-by-line profiler
• Devel::Cover - Test coverage
• HTML::Mason::Plugin - manual method for Mason components

See what a linux process is doing

strace -p [pid]

Log into a server using secure keys instead of typing a password

On the client:

1. ssh-keygen -t rsa
2. accept the default filename
3. press enter for a blank password
4. a private (id_rsa) and public key (id_rsa.pub) will be created
5. copy the public key to the server
   

On the server

1. cat id_rsa.pub >> ~/.ssh/authorized_keys
2. chmod 700 ~/.ssh
3. chmod 644 ~/.ssh/authorized_keys

Now back up your keys.

See also this and this.
(Upon error "Agent admitted failure to sign using the key", log out of the client and log back in again to fix).

How to customise the "content edit form" in Drupal

/**
* Catch the theme_node_form function, and redirect through the template api
* Allows us to customise the fields displayed on the "create content" form
*/
function phptemplate_node_form($form) {
$admin_role_id = 3;

if ((! $user->roles[ $admin_role_id ]) && ($user->uid != 1)) {
// user is not an admin, hide all the advanced fields

// Default Drupal fields
unset($form['attachments']);
unset($form['options']);
unset($form['author']);
unset($form['comment_settings']);
unset($form['revision_information']);
unset($form['menu']);
unset($form['body_field']['format']);
unset($form['path']);
//$form['format']['#title'] = 'Advanced setting: Input format'; // e.g. rename fields

// Ubercart fields
//unset($form['base']); // note: can't submit without this because display price and SKU are required
//unset($form['taxonomy']);
//unset($form['field_image_cache']);
//unset($form['google_checkout']);

// call the original function
return theme_node_form($form);
}

Set up sendmail on linux desktop

Setting up email sending (not receiving) for XAMPP in Ubuntu.

Install Postfix:

• sudo apt-get install postfix
• Choose satellite system
• Choose example.com for the system mail name (it will mean the 'hidden' Return-Path address goes to nothing, not the visible From/Reply-to address)
• Type your ISP's SMTP address
• Type a user for incoming email
• Just accept the defaults for other questions

If you need to reconfigure, do sudo dpkg-reconfigure postfix

To check the version, do postconf -d | grep mail_version

PHP/XAMPP settings:

• In php.ini, add: sendmail_path = /usr/sbin/sendmail -t

Test like this:



$email = 'you@yourdomain.com';
$headers = "From: $email\r\n";
$headers .= "Reply-To: $email\r\n";
$subject = "test subject";
$message = "test body";
print mail($email,$subject,$message,$headers) . "\n";

?>

Simple PHP email form

The 'hidden' Return-Path header (not visible Reply-To) will be your name at your domain.

Email form:

<form method="post" action="s.php">
From: <input name="from" type="text" /><br />
To: <input name="to" type="text" /><br />
Subject: <input name="subject" type="text" /><br />
Body:<br />
<textarea name="body" rows="15" cols="40">
</textarea><br />
<input type="submit" value="send"/>
</form>


s.php:

<?php
$to = $_REQUEST['to'] ;
$from = $_REQUEST['from'] ;
$subject = $_REQUEST['subject'] ;
$body = $_REQUEST['body'] ;

$return = mail( $to, $subject, $body, "From: $from\r\nReply-To: $from" );

if ($return == 1) {
print "sent.";
}
else {
print "error.";
}
?>

Get all past podcasts in iTunes

• Display the podcasts in list view (details view)
  
• Ctrl-click on the triangle next to the podcasts
• Voila! You will see a list of the past episodes to select