This is a description of protocols used on the various serial ports that exist in the xda-1. These ports are:
devicenamedriveruart addressirqconnects to
COM1:serial.dlla80300000010 external serial cable
COM3:ircomm.dll infrared port
COM2:serial3.dlla8050000000f gsm multiplexed protocols
COM8:usb.dlla8000000000d serial over USB
COM9:serial2.dlla80100000011 gsm bootloader
To be able to talk the multichannel protocol over COM2, you have to send this ioctl: DeviceIoControl (hCom,0xAAAA5679L, {0x84,0}, 2,0,0,0,0)

multi channel protocol

This protocol is what is sent over the internal port ( the strongarm serial port 1 at virtual address 0xa8050000, or physical 0x80050000 ). and it is what atdbg ( or rilcomtemp.exe ) dumps.
commandprotocol channels supported
normal 11 14 16 17
dualtrace11 14 17
dual11 14 17
protocoldescription
11trace messages
12other gsm trace
13PCO trace
14unknown
16gsm at commands
17peripheral control ( led, vibrator, battery)
the number you can specify after 'normal' and 'atcmd' seems to indicate something related to baud rate, looking at the type of crap that you see with different values.
Wait for turn on GSM...
GSM already on -> RESET !!
GSM turn on successed!!

-----------------------ULYSSE MODE---------------------------
Normal mode(UART2 --- UART3)


******************************************************
InitDebugSerial using SERIAL PORT 2
******************************************************
GSM RESET...
after which things get interesting: in hex:
02 11 00 00 00 20 03 "Trace module started by the environment" 02
02 11 00 00 00 00 01 "RVT: Lost Message" 02
02 17 "4D01FF00004D" 02
02 16 "AT-Command Interpreter ready\r\n" 02
now there are several things I can do:
requestreply
11 02 Exception: Unexpected Software Interrupt this seems to crash the gsm
14 02 02 14 00 10 10 10 10 02
17 02 02 17 "0000" 02
16 "at%ureg?0,4\r" 02
02 16 "a" 02 
02 16 "t%ureg?0,4\r\r\n\r\n" 
      "+EXT_UREG EA000446\r\n\r\n\r\n" 
      "OK\r\n" 02

channel 17 - peripheral control

in this protocol everything is sent encoded as hexadecimal characters. the first digit indicates the length of the command sequence, the second digit is probably a device address. followed by the number of bytes ( in hex, so 2 digits per byte ) indicated by the first digit.

these devices are available I think:
addressdescription
1led/vibrator
2unknown
5battery via cal
7unknown
8unknown
9phone audio control
bunknown
dbattery via ssp

in replies, the length+device nyble are followed by a hex-status byte, followed by <length> result bytes
sequenceaction
51 00 00 00 00 00 led off
51 00 01 01 05 05 red led on
51 00 00 01 05 05 red led off
51 01 01 01 05 05 green led on
51 01 00 01 05 05 green led off
51 02 01 01 05 05 orange led on
51 02 00 01 05 05 orange led off
51 03 01 00 01 00 vibrator on
51 03 00 00 01 00 vibrator off
19 xx phone mode ? something audio related
0d request battery status via scc
answer: 4D0180021FEF
05 request battery status via cal
answer: 45FFFF0274B9
the last 2 bytes for the led/vibrator are the 'on', and 'off' time, so 510101000105 will give brief flashes and long spaces. or 510301000101 will alternate the vibrator


rsupgrade/monitor protocol

This protocol is what rsupgrade.exe uses to upload and execute the monitor to the gsm, used to flash a new radio stack.
(this talks to the strongarm serial port 3 at virtual address 0xa8010000, or physical 0x80010000 ) you can also use this protocol on the external serial port, when entering the utilities menu, selecting the option 'gsmrom to sd', with an sd card inserted. the xda then prompts the user to start 'pc-monitor'. (after close inspection of the bootloader code, it turns out that you are talking to an emulation of the gsm, in the bootloader.)

the code in the gsm rom handling this protocol is in the first 4K of the gsm rom. this code is never updated(well, not that I have seen) by radio stack upgrades, thus always leaving an option to rescue your radio stack if something went wrong upgrading.

I have not yet figured out what rsupgrade does to be able to talk to com9:
I think I did everything the same as I see rsupgrade doing, it just doesn't work for me.

the general request format is:
positionvaluedescription
00xaaheader byte
1 length byte
20x00 .. 0x0bcommand byte
>=3 data, if 0xaa occurs, it is escaped by doubling it
the general answer format is:
positionvaluedescription
00xaaheader byte
1 length byte
20x00 .. 0x0bcommand byte
3 status byte, 0x00= OK
>4 data, if 0xaa occurs, it is escaped by doubling it

below the packet formats, ( leaving out the '0xaa' )
with the number of 'x' specifying how many digits in each parameter.
commandrequest formatanswer format
cmd00get_monitor_id 01 00 05 00 00 ii mi ma
info byte, minor, major version nr
cmd01get_flash_memory_id 01 01 03 01 00 xx
cmd02get_chip_id 01 02 03 02 00 xx
cmd03get_board_id 01 03 03 03 00 xx
cmd04erase_flash_memory 01 04 01 04
cmd05write_flash_memory 01 05 10 05 00 xxxx xxxx xxxxxxxx
first 16bit word is a checksum, last 32bit word is a crc
cmd06 01 06 03 06 00 xx
cmd07read_parameters 04 07 xxxx xx ?? 07 00 xx [xxxx ...]
cmd08write_parameters ?? 08 xxxx xx [xxxx ...]01 08
cmd09write srecords 01 09 06 09 00 xxxx xxxx
first word is file checksum, second is ram checksum
cmd0aexecute monitor 05 0a xxxxxxxx 01 0a
cmd0bwrite_flash_memory 01 0b 0a 05 00 xxxx xxxx xxxxxxxx

test program

I created this pocket pc tool to be able to test various bits and ioctls in the xda. the source is browsable here

on the top line, there are checkboxes, to control this: otherwise it is sent as ascii, followed by a CR (carriage return)
DTR, RTS controls these rs232 lines
BREAK sends a break
Infrared-mode sets the uart in Infrared mode.
hex the input edit box is interpreted as hexadecimal data.

the next line contains status indicators for this:
DSR,CTS,DCD,RING the status of these rs232 lines
BRK break was detected. - MS Comm api does not tell when the break ends
RLSD ? - [see ms docs]
rcv,xmit these toggle on and off, with each packet received/sent

below each of these indicators is a counter, counting how often they have changed

then follows the userinput box, with a 'transmit' button

then a big output window

followed by 2 lines of bit indicators, the first line are the gpio bits, the second line the port at 0xa4000068 ( with the now I think wrong assumption, that 0xa4000064 controls the direction of these bits ). red means it is an input bit, black means it is an output bit. you can switch from input to output by tap+holding the bit. you can toggle its output value by tapping ig. a filled square is a '1' bit, an empty squere is a '0'.
playing with these bits may damage your device, so first research by reverseengineering code that touches them, before just randomly fidling with them.

the bottom is a bit chaotic, and contains experimental checkboxes enable.disable it. for COM2: this enables shared use of the GSM at command channel. this seems to control the power of the gsm part. this seems to enable control of the power of the gsm part.
ril send ioctl 0x03000318/0x03000314 to the RIL1: device,
com send ioctl 0xaaaa5679 to the current device
dbg enable/disable debugging info in the output window
gsm controls bit 6 of 0xa4000068
mode controls bit 6 of 0xa4000064
mem toggles the value of 0xac0203c8 between 0 and 1
open opens or closes the selected device

trace tools

on the xda-ii you can see raw output from the multichannel protocol in \Storage\XPanelLog*.pco files.
the at commands are logged in \Storage\Atdbg*.log
problem is however that these log files are not -always- written. or maybe only closed after they reach their maximum size ( about 2M )

to enable logging, set these registry keys:

[HKLM\SOFTWARE\HTC\ATDbgLog]
Enable=1

[HKLM\SOFTWARE\HTC\XPanel]
Enable=1