www.techmind.org logo
by W.A. Steer  PhD
Back to contentsAbout...

 

 

Projects blog

This page is basically intended to be a glorified 'to do' list and progress report for the various technical projects which I have on-the-go, or in-mind. I'll also try to include links to source material I found useful at the time. Along the way I'll drop in some tech hints and tips, and may or may not include comment ('opinion') on any science-related current-affairs... let's see how things go!

Projects in progress (30th Nov. 2005)

In my rough projected order of likely completion (ie most-likely-to-complete first, first).

Wednesday ~ 30th November 2005

Started the project-blog!

Hosts file (on Windows)

Only in the past few weeks did I discover the Windows hosts file. Unix/Linux old-hands will know what this file does; I don't have a deep understanding but it's a sort-of hostname-to-IP lookup/redirect table... and may be readily-used to prevent unwanted internet accesses from your PC to hostnames you'd rather not communicate with... such as animated advertisements :-) Certain technews-sites (and one rail journey-planner) I regularly visit have huge distracting, flashing, animating, bandwidth-wasting adverts which really annoy me. Simply view the offending html page source (in IE, View|Source), find the annoying ad-serving hostname, and add it to your hosts file alongside IP address 127.0.0.1 (a dummy / home address). From then on, in place of the ad, you'll just get a broken image icon or a 404 error! In a standard WinXP installation, you'll find the hosts file in C:\WINDOWS\system32\drivers\etc\, and it can be opened using any basic text-editor such as Notepad or Wordpad (it won't auto-associate since the hosts filename has no extension)

My hosts file presently looks like: 
# Copyright (c) 1993-1999 Microsoft Corp.
#
# This is a sample HOSTS file used by Microsoft TCP/IP for Windows.
#
# This file contains the mappings of IP addresses to host names. Each
# entry should be kept on an individual line. The IP address should
# be placed in the first column followed by the corresponding host name.
# The IP address and the host name should be separated by at least one
# space.
#
# Additionally, comments (such as these) may be inserted on individual
# lines or following the machine name denoted by a '#' symbol.
#
# For example:
#
#      102.54.94.97     rhino.acme.com          # source server
#       38.25.63.10     x.acme.com              # x client host

# Everything after localhost added by WAS summer/autumn 2005 onwards
#   - advertising sites which wind me up by using ActiveX code, animated ads etc
#   Note ... .realmedia.com  and  ... .exactadvertising.com are both highly dubious activities too

127.0.0.1       localhost

# The Register et al
127.0.0.1       ad.uk.doubleclick.net
127.0.0.1       ad.doubleclick.net
127.0.0.1       a.as-eu.falkag.net
127.0.0.1       sel.as-eu.falkag.net

# on National Rail Enquiries www.nationalrail.co.uk
127.0.0.1       ads.adviva.net
127.0.0.1       etype.adbureau.net
127.0.0.1       view.atdmt.com
127.0.0.1       spe.atdmt.com
127.0.0.1       spd.atdmt.com

GDI+ (Windows Programming enhanced graphics API)

This Sunday, while flicking through the latest (Dec.2005) MSDN magazine, I stumbled across the existance of GDI+ - an enhanced graphics API which is built into Windows XP [and can be retrofitted to older versions of Windows (back to Win95), by installing the gdiplus.dll]. GDI+ adds loads of features you'll probably never use as a programmer, but it does provide easy-to-use functions for opening and saving GIF, JPG, PNG, and TIFF graphics ...which definitely is useful to know about. Apparently it can also do lossless 90/180-degree rotations of JPGs (assuming their size is a multiple of 16 pixels in both dimensions).
I've now already written a quick demo program in BCB 6 to load and display GIFs/JPGs using GDI+ to prove to myself it can be done (thanks also to the guidance at http://www.leunen.com/cbuilder/gdiplus.html).

Incidentally, it was in this GDI+ module that a buffer-overrun error was discovered in the JPG decode module in September 2004. This now makes a whole lot more sense to me! If you fear you may have an old version of the GDI+ DLL lurking on your system, best take a look at http://support.microsoft.com/?id=873374

The Microsoft documentation for GDI+ can be found at http://msdn.microsoft.com/library/default.asp?url=/library/en-us/gdicpp/gdiplus/gdiplus.asp

Friday ~ 9th December 2005

GIF-reading continued...

Not a lot to report! I got my GIF-reading built into my required application, which actually loads a large number (100's) of small gifs. Although the performance is fairly respectable now, I still haven't worked out if I'm doing my coding in the most efficient way, and whether I'm sufficiently minimising the 'housekeeping' (creating/destroying/recreating objects etc). The fact that gifs are palette-based, and that GDI+ uses some new Palette structures doesn't make passing an HPALETTE to BCB's TImage easy; after quite a few hours messing around, and reading Microsoft's online documentation, I've hacked the result I want, but again I don't hold huge confidence in the robustness of my solution. Then again, since the application will not be deployed beyond a few friends, I can live with that (for now)!

35mm negative photo-scanning

Earlier in the week I went back to having another go at scanning a nice 35mm negative, to improve on the previous scan, now that I understand more about the scanner... and realised that (being used to the fairly good scan) that the raw negative was somewhat 'out' being slightly overexposed and with a bit of a bluey cast... but eventually made a scan which improves upon the previous attempt by quite a considerable margin. Having studied the negative (including the order of the pictures) I was surprised to have to conclude that the picture was taken with my compact camera, rather than my SLR. On the other hand, that helps to explain the slight exposure error, and the mild vignetting in the corners [along with colour aberation, something I could write correction for in my vapourware image-manipulation program]. The picture is still remarkably sharp. Unusually, this was taken on Kodak Gold 400 (I normally use 100 or 200 ASA films, Kodak HD200 recently). Despite the huge grain of KG400, with tasteful (as opposed to over-the-top) software grain-reduction, the result is still very pleasing.

 
The image on the right (or below) shows actual pixels (4000dpi negative scan). On this scale, the total picture is 21 megapixels!

Now show me a compact, lightweight, digital camera which rivals the quality and can take a couple of hundred pictures on a single pair of Duracell AA batteries... ;-)

I've got more pictures waiting to be scanned, but both technically and artistically it can be a slow process! See also the gallery.

'Grey' lighting from fluorescent tubes

I've been bothered for ages by the general office fluorescent lighting at my workplace. The light just has a weird 'grey' quality to it. I don't mean it's dim. It's just frustratingly lifeless! Morbid even. Following slight added impetus from a talk from a guy in the Lighting division of the company (telling us about all the attractive $$$ artistic lighting they're doing with LEDs), I finally got the spectrophotometer out to measure the light and understand what was wrong with it. Turns out the spectrum has just about 4 narrow emission-lines, and big gaps with virtually no energy in between [plot to follow]. These 'triphosphor' tubes are to sunlight as a low-bitrate MP3 is to a CD. Lacking. Lacking detail, definition and subtlety. The tubes probably have a colour-rendering index (CRI, or sometimes Ra) of 80 to 85, but that's nothing compared to a tungsten bulb (including quartz-halogen) or daylight which is pretty much 100. Will have to investigate getting some 'full-spectrum' replacement tubes, or quartz-halogen spotlights, to bring life to my office through the winter :-)


Spectrum of ambient light in my office, illuminated by 'triphosphor'-type fluorescent tubes. Note the gaps in the spectrum!

Antivirus software

Hmmmm. The 'window of opportunity' between a virus first mass-mailing itself and detection by the Anti-Virus software is very real. Last Saturday and Sunday I received by email 7 copies of a new Mytob varient, yet this was invisible to Norton until it did its weekly update on Thursday evening. Fortunately I use (fairly virus-immune) Eudora, and anyway as my first line of defence have my own procmail script running on my mail server which tosses aside most things that look even vaguely virus-like -- long before Windows gets to see it. Without procmail, I'd be downloading 1-2MB of virus-laden email daily. Does this not justify ISPs doing virus filtering, rather than leaving it to the (possibly) clueless user at the end of a 56kbps phone line?

Sunday ~ 11th December 2005

Colour-temperature and whitepoints - visualisation program

This weekend I finally got around to modifying/extending my CIE-related programs to calculate and show colours corresponding to 'black-body' sources. Discovered in the process that it is not possible to accurately represent colour-temperatures below 1900K on a (sRGB / Rec.709) monitor as the display primary red is not deep enough!

See the full write-up on this site at http://www.techmind.org/colour/coltemp.html.

Friday ~ 6th January 2006

I hate spam

Hmmmm. The amount of spam I'm receiving to my inbox has taken an significant and annoying upward trend since Christmas. Will have to put some more effort into a) spam filtering by tweaking procmail scripts (tho' manually analysing spam for means of identification is quite soul-destroying), and b) further coding or obfuscating my email address shown at the foot of the pages on this site (a few bits of character- encoding . to display an '@' etc. no-longer seem to do the job). Possibly a 'contact me' link, rather than email address on each page, would reduce the opportunities for addresses to lurk in web-caches and bait virus-email too...? But I prefer the principle of having my address in human-readable form on each page, especially as it remains useful on a printout. A gif image could be used, but inconveniences the user by not being cut-and-paste friendly. I see Steve Gibson of http://www.grc.com/ uses a JavaScript to deal with the email-harvesting problem...

I've recently been receiving over 10 megabytes of virus-loaded email each day, virtually all of which is redirected to /dev/null or to various holding-files. Unfortuately legitimate (but html-heavy) emails very occasionally fall foul of my aggressive filters. To be on the safe side, please send plain-text emails without attachments.

Should I add a (usually pointless) 'Privacy Policy' page to my site?

Project shortlist 2006

Friday ~ 10th February 2006

House-buying economics

I don't seem to have done that much worthy of report recently. Since Christmas a few of my work-friends have been looking rather more seriously at buying houses; I've been holding off buying for a long time, but have put some time into looking at the relative finances of buying at renting in the present UK economic climate. House prices are very high, having doubled in the past 6 or 7 years, though interest rates (and thus mortgage rates) are relatively low. Mortgage providers (with a vested interest, of course) reckon prices will continue to rise at around 4% per year for the foreseeable future, while doomsayers (such as Captial Economics) reckon prices will fall by up to 30% over the next 2 or 3 years (see www.housepricecrash.co.uk for more of that line of thinking). I may write more on this another time, but basically housesharing is a very cheap way to live (if you can bear the lifestyle), buying on your own in the south-east of England as a 30-year old, even on a reasonable professional salary will really stretch the finances (and gets very nasty if interest rates rise by even a couple of percent, or if prices fall), buying as a couple or between two friends would be financially comfortable, but is still a big commitment to a given job or part of the country.

Looking at trends in the FTSE stock-market index

A couple of times recently I've seen comments in the financial section of the weekend papers about stock-exchange indexes having a monthly periodicity, and making most of their rises around the time of the turn of the month when coorporate pension schemes typically invest. This all sounds very reasonable, but how true is it really, and is the effect actually significant? Using data from one particular FTSE All-Share tracker-fund, I set out to find out.


I calculated a smoothed rolling-average of the index-value over 1-2months, then plotted the daily values relative to the local average (black traces). I then plotted in green the average of the data for all 11 years, and lo and behold there is a distinct trend!

The market is typically most over-valued on the 6th of the month, by about 0.66%, remains about 0.5% over-valued for the 7th and 8th, before reverting to norm for the 10th-14th. There's then a few fluctuations, but from about 23rd onwards it tends to be undervalued. The greatest undervalue is typically -0.45%, on the 25th and 26th of the month. So, while the general scatter is around ±1.5% there are still significant statistical trends. The original hypothesis appears to be borne out.

The next thing to check is annual trends... does the index typically climb in certain seasons and stagnate or fall in others? This will be harder to verify since you only get one trace per year instead of 12, so the statistical validity of any results will be poorer. Watch this space!

Thursday ~ 2nd March 2006

Visit to Photo-Me Service Centre

I'd had an invite to visit a local Service Centre for Photo-Me kiosks and digitial mini-labs. Had an interesting visit and found out more about thermal printing technology (used in home printers and serve-yourself shop kiosks), and had a look inside a Photo-Me / Kis / Kodak DKS1550 mini-lab machine which uses an LCD-based projection system to expose photographic paper to make 'digital' prints. It seems that much of the 'innovation' these days is in digital image enhancement, particularly for 'souping up' underexposed flash photographs... which may be a good thing for average snapshots that consumers bring in, but is rather less good for pro-ams who already have perfected their image and don't want it messed about! Even without 'enhancements', it was revealing to see how much difference different gamut-mapping settings made. How many regular machine-opertors in high-street shops will understand the implications of such settings?

I'll try and add some postscript to my photoprinting pages with my new findings/understanding.

Friday ~ 3rd March 2006

Strange colours on CanoScan LiDE scanner

It all began a year or more ago when, rather than take a photograph, I attempted to 'scan' one of our polymer-LED displays at work using the available Canon CanoScan LiDE 30 flatbed scanners. The particular unpowered display appears unambiguously olive green to the eye, but came out distinctly dark blue on the scan. Weird. I didn't figure out what was going on, and eventually forgot about it. Then earlier this week it suddenly came to me: fluorescence! Our displays as well as being electroluminescent (glow when you drive an electric current through them) are also fluorescent (glow when exposed to ultraviolet - or just deep blue light). Unlike older generation scanners, the LiDE scanners use colour-sequential LED illumination and an unfiltered array of photodiodes to detect the light. Our display was registering a much higher signal than it should have been with the blue illumination not due to reflectance (being blue) but owing to fluorescence! I tested this by 'scanning' a small crop of Stabilo highlighter pens from the office stationary cupboard (see picture). From the top down, the real pen colours are: fluorescent green, fluorescent orange, fluorescent-ish blue, fluorescent pink.

The older bulkier scanners (typically 3.5" or 10cm deep) use a CCFL illumination source and 3 CCDs (one for each of red/green/blue). These aren't really fooled by fluorescent colours and have a much greater depth-of-field than the cheaper LiDE-style. With old-style scanners you can have a fair go at scanning 3D things, like the component-side of a circuit board. With the LiDE, anything more than a millimetre above the glass is hopelessly out-of-focus.

There's some nice (Flash) graphics illustrating scanner optics on the Canon website {insert link}.

Tuesday ~ 7th March 2006

"Display Usability" meeting at the National Physical Laboratory

This was a meeting organised by the NPL Optical Radiation and Measurement (ORM) Club together with the UK branch of the Society for Information Display (SID) which I attended on Company business. For me, the most useful talk was from Paul Boynton of the American NIST about display measurement and in particular about avoiding measurement errors due to "veiling glare" (where stray light from bright parts of the display scatters within the measuring instrument optical system and contaminates 'black' measurements). The NIST site has some reading material, and some limited consumer advice on the purchase of flat panel TVs... www.fpdl.nist.gov/. The other notable trend which I saw was towards proper photometrically-accurate ray-tracing imaging to enable product designers to fully visualise how their products will look (particularly products such as mobile phones, featuring displays and lit keyboards etc) in different environments. Along with everything else, the software accurately models viewing-angle and reflection characteristics of displays, and is available from the French company Optis. A similar methodology was shown by a representative of the Bartlett School of Architecture at University College London, to aid visualisation of lighting within and around buildings at different times of day and of the year.

Thursday ~ 9th September 2006

Time has flown - what have I to show for it?

I realised a few weeks ago that it had been absolutely ages since I'd done any C (or C++) programming. Just checked my file-dates, and shows nothing between February 20th and 23rd August (in both cases, programs generating Colour plots for my Colour Science / Colour Temperature webpages). No wonder I'm rusty at it, and the thought of programming seems too much like hard work!!!

In August I took a solar spectra using an instrument at work; I hope to get permission to use that data (and some more besides) to make some nice graphics to add to my web-pages on Colour and Colour Temperature. Before that I found some academic work on "Lateral Ion Transport in LCDs" - a new (to me) cause of "image-sticking" in LCD panels. I've been on holiday, taken some photos, scanned some photos - few new ones in my gallery. I've also serviced my bicycle - found the chain had "stretched", necessitating its replacement and replacement of the rear sprocket set (hint: in future, replace chain sooner!).

I've been drafting a web-page on "Visualising Energy" with the aim of offering the public some genuine and worthwhile hints on how to estimate and save energy usage for the purpose of saving "the environment" or merely saving money. Sorting the hype from the reality etc. Trying to make energy "real". Key point being that heating and transport are the key energy-guzzlers, and most else pales into relative insignificance.

Accuracy/precision of mains electricity 50Hz (60Hz) frequency?

image of UK grid frequency meter Work has been a bit more interesting recently, and all of a sudden 2 out of the 3 projects I'm working on require some signal-processing (DSP) analysis. As a result of one of these, we asked the question "how stable (or accurate) is the 50Hz (or US/Japan, 60Hz) AC mains power frequency?" It turns out that in the UK it's governed by law to be within ±1%, i.e. 49.5-50.5Hz, although in practice excursions beyond 49.8 or 50.2Hz are fairly rare. The system is rigged so that over a day or two the average frequency is spot-on, so that electric clocks don't have a net gain or loss of time. The instantaneous frequency of the mains speeds up or slows down slightly according to the balence of generated power to demand (higher frequency when there's excess power, lower when the demand is too high).
I found a nice real-time UK grid frequency meter: www.dynamicdemand.co.uk/grid.htm.
Google also turned up two independent (non real-time) studies: www.leapsecond.com/pages/mains/ and wwwhome.cs.utwente.nl/~ptdeboer/misc/mains.html.

A nice little project when I have a spare afternoon would be to modify my PC soundcard-based frequency counter software for more precision and devise a simple hardware pickup to connect to the soundcard microphone input to make a complete DIY mains frequency meter application...

The National Grid has some Real Time Demand data on their website at: http://www.nationalgrid.com/uk/Electricity/Data/Realtime/ (although when I tested it on 21 June 2007 the data was incorrect and/or the graphs were not updating properly...)

September 2007

Software-initiated web-downlaods, and anti-spam techniques [written retrospectively]

I finally got around to putting together a software routine to perform http requests and hence retrieve web documents and images etc. under automatic/programmatic control. I view this as a really useful tool to have to hand these days, but the immediate application was anti-spam.

I've probably previously commented that much spam contains URL links to websites whose "disposable" domain-names were only registered in the few days prior to the spam run. A powerful anti-spam technique therefore is to look up the domain-name WHOIS for URLs found in email, and if the name was registered extremely recently to discard the email as spam. This is effectively what I did. I used a web-based WHOIS service to do the lookups. Unfortunately the different domain-name registrars use different date-formats, which adds complexity to the parsing of the WHOIS records - but nothing very complicated.

Here's a selection of spammy URLs from emails around August/Sept 2007, followed in brackets by the number of days between registration of the domain name and receipt of the containing email.
(A time of "-1 days" merely indicates crossing of international date zones, and is not significant.) 
http://www.watches9122.info  (7 days)
http://potapsoftonline.com  (4 days)
http://rxpilmisav.com  (6 days)
http://multioemstore.com  (6 days)
http://oemsoftgriz.com  (1 days)
http://asnolyke.com  (12 days)
http://kunchakoem.net  (4 days)
http://prevedkosoft.com  (5 days)
http://singelgirls.info  (1 days)
http://qbiit.oemforfreeforyou.com  (8 days)
http://printerrcannerr.com  (2 days)
http://cellphoesmells.com  (3 days)
http://Dutton.alsoother.com  (6 days)
http://www.musodash.net  (1 days)
http://www.ratherthin.com  (2 days)
http://padderhi.com  (5 days)
http://www.plixuen.net  (3 days)
http://meloiret.com  (4 days)
http://friskijahna.cn  (0 days)
http://www.serduim.net  (0 days)
http://www.podveg.com  (16 days)
http://www.flowerswim.com  (2 days)
http://cagospel.com  (2 days)
http://iechedd.cn  (2 days)
http://Washington.enseol.com  (1 days)
http://www.elpigs.com  (5 days)
http://www.mamuuko.com  (2 days)
http://deguxa.bffidchipsto.com  (18 days)
http://maximoit.com  (2 days)
http://mankine.com  (2 days)
http://dsgg.kccsilver.com  (2 days)
http://www.pkgsolar.com  (2 days)
http://www.dugmu.com  (1 days)
http://www.dwebmax.com  (1 days)
http://drugsrog61.com  (5 days)
http://wertorest.com  (13 days)
http://suavnc.com  (2 days)

http://dugmu.com  (1 days)
http://droegt.com  (2 days)
http://hjpercor.com  (0 days)
http://casinorfhwcc.cn  (28 days)
http://hihoem.com  (0 days)
http://adyms.com  (1 days)
http://www.toptdt.com  (1 days)
http://www.topxbox1.com  (1 days)
http://esayaid.com  (2 days)
http://eskybiz.com  (2 days)
http://eqiqbbs.com  (2 days)
http://spoaueyaat.cn  (0 days)
http://mbvrt.com  (0 days)
http://molarweb.com  (0 days)
http://mohesr.com  (0 days)
http://motatv.com  (0 days)
http://meblepl.com  (1 days)
http://www.motatv.com  (1 days)
http://mohesr.com  (1 days)
http://www.mediaign.com  (1 days)
http://mhtlhelp.com  (1 days)
http://www.derheini.com  (0 days)
http://www.disnui.com  (0 days)
http://mdeps.com  (1 days)
http://www.mbvrt.com  (1 days)
http://dggslaw.com  (0 days)
http://www.devildj.com  (0 days)
http://atatayiu.cn  (2 days)
http://desibany.com  (1 days)
http://diasoul.com  (1 days)
http://www.dhlvlj.com  (1 days)
http://www.dicnay.com  (1 days)
http://www.sharkfu.com  (0 days)
http://antlernc.com  (10 days)
http://www.postrilk.com  (1 days)
http://www.incolk.com  (1 days)
http://senomax.com  (0 days)
Obviously I don't recommend you visit any of these domains, as they will be up to no good.

Having got this far, I then discovered the URIBL (URL-blacklist), a freely-available spam-URI blacklist which is presumably created by similar methods to that I used above. This blacklist is queried by a DNS-lookup mechanism, which required further internet-software techniques to be developed to probe. This was done.

See also http://www.uribl.com/

Unfortunately these methods only run locally on my PC and I don't have the authority to run them on my mail server. Even so, this is an effective way to cleanse my email archive.

April 2008

Restoration of original television colour to 16mm black-and-white film archives

Numerous BBC television programmes of cult and classic status from the late 1960's and 1970's which were originally made in colour now only exist in the form of black and white 16mm film 'telerecordings', the original videotapes having been destroyed or overwritten. These programmes include a number of episodes of "Doctor Who", "Steptoe and Son", and "Dad's Army". It has long been believed that we'd never again see these programmes in their original colour. Intriguingly however, for technical reasons, most of these black and white recordings actually contain remnants of the PAL colour-coding signal (in the form of 'chromadot' patterning) - but until recently it was not thought possible to decode them.

I stumbled across the above problem through my newgroup-reading, and found the informal Colour Recovery Working Group which was looking into it. With my previous experience at software PAL-decoding I thought I'd be well-placed to work on this. Indeed within 2-3 weeks I'd demonstrated practical image-processing colour-recovery approaches which superceeded anything they'd done before.

See my web page PAL Colour Recovery for more information.

This is a fascinating problem in practical image-processing.

As at end May 2008 this work is still ongoing...

Created: November 2005
Last modified: 24 May 2008
Source: http://www.techmind.org/blog/

©2008 William Andrew Steer
andrew@techmind.org