Toppy.org.uk ~ View topic - CrashTrace -- better debug info

Author

Message

< TAP and patch development ~ CrashTrace -- better debug info

Page 1 of 2

Goto page 1, 2 Next

R2-D2

Posted: Mon Nov 24, 2008 3:45 pm

Frequent contributor Joined: 18 Dec 2006 Posts: 12148

CrashTrace is a first attempt at providing some more info on crashes on the serial port. I've tried to make it quite defensive, but if something does go wrong at this level you'll probably get a never-ending cycle of exceptions (until you press the power button and the Front Panel shuts it down).

Extra info includes a couple of short memory dumps around the exception code and stack points, a trace back of function calls that lead to the exception, and a list of current TAP locations. There's also a simple CrashTest TAP which will cause a crash when the 0 button is pressed (or press Exit to quit it). The initial register dump and the trace back should show the values assigned (explicitly) to $s1, as an indication of how this new information might be useful.

_________________
Troubleshooting -- User Manual -- Dark Side of the Matrix: Firmwares and Patches

R2-D2

Posted: Mon Nov 24, 2008 11:22 pm

Frequent contributor Joined: 18 Dec 2006 Posts: 12148

V0.2 adds a memory usage dump, reformats things a bit and adds a little more info. Sample output is now:

Code:

COMMAND> exec crashtracec.tap
CrashTrace: hook added @ 000010a4
COMMAND> exec crashtest.tap
CrashTest: start @ 00001339
CrashTest: crashing @ 0000170b
**CRASH**

$sp=804f6c98
$at=00000000 $v0=804dc8f8 $v1=00000078 $a0=0000000a
$a1=82d39580 $a2=804f6d18 $a3=0000000f $t0=80256894
$t1=00000000 $t2=82bc9288 $t3=804f640c $t4=80425e5c
$t5=00000000 $t6=804f6c0e $t7=0000000a $t8=0000000d
$t9=80179628 $s0=00010000 $s1=12cafe34 $s2=82d39602
$s3=0000e300 $s4=00000009 $s5=80179628 $s6=00010000
$s7=82d39602 $s8=804f6dc8 $ra=82d38710 $t0=00000000
$t1=00000000 $gp=82d39602 $k1=82d38654

<<< Unexpected Exception >>>
CAUSE = 0000040c
EPC = 82d38654
*EPC = ac000000
SR = 10000403
badVAddr = 00000000
task = 0
_runCount=1
lastTask(0) = 0

Dumping bytes from 82d385d0 for EPC=82d38654
82d385d0: 00000000 00000000 00000000 00000000
82d385e0: 00000000 00000000 00000000 00000000
82d385f0: 00000000 00000000 00000000 00000000
82d38600: 27bdffe0 afbf0018 afb70014 afb10010
82d38610: 04100001 00000000 3c170000 26f70fea
82d38620: 02ffb821 3c1112ca 3631fe34 8f8201c6
82d38630: 00000000 8c420000 8ee40016 00000000
82d38640: 24841580 0040c821 0320f809 00000000
82d38650: 8fbf0018 *ac000000* 8fb70014 8fb10010
82d38660: 03e00008 27bd0020 27bdffe0 afbf001c
82d38670: afb70018 afb10014 afb00010 04100001
82d38680: 00000000 3c170000 26f70f7e 02ffb821
82d38690: 00a08021 3084ffff 24020100 1482003b
82d386a0: 00000000 3c020001 1602001b 00000000
82d386b0: 3c11dead 3631beef 8f82002e 00000000
82d386c0: 8c420000 00000000 0040c821 0320f809
82d386d0: 00000000 8ee40016 00000000 2484158c

Dumping bytes from 804f6c90 for fn SP=804f6d14
804f6c90: 00010000 804f6dc8 0000040c 804dc8f8
804f6ca0: 00000078 0000000a 82d39580 804f6d18
804f6cb0: 0000000f 80256894 00000000 82bc9288
804f6cc0: 804f640c 80425e5c 00000000 804f6c0e
804f6cd0: 0000000a 0000000d 80179628 00010000
804f6ce0: 12cafe34 82d39602 0000e300 00000009
804f6cf0: 80179628 00010000 82d39602 804f6dc8
804f6d00: 82d38710 00000000 00000000 82d39602
804f6d10: 82d38654 *82d39580* 82d3958c 00000004
804f6d20: 0000000f deadbeef 82d39602 82d38710
804f6d30: 804f6dc8 82d3958c 0000170b 00010000
804f6d40: 00000003 82d38668 00010000 8025e6d8
804f6d50: 8013324c 801a2a9c 00010000 0000e300
804f6d60: 83011664 00000010 804f6dcc 804dc8f8
804f6d70: 80005f84 00010010 801021f4 80000220
804f6d80: 80000290 81fffe98 0000e300 0000e300
804f6d90: 83011664 00000010 00000000 81fffe98

** Stack back trace @ 22:07 (uptime: 0d 0h 0m 58s) **
1) fn=82d38600 ra=82d38654 sp=804f6d14 in TAP @ 82d38000 CrashTest
s1=deadbeef s7=82d39602
2) fn=82d38668 ra=82d38710 sp=804f6d34 in TAP @ 82d38000 CrashTest
s0=82d38668 s1=00010000 s7=8025e6d8
3) fn=80132eb4 ra=8013324c sp=804f6d54 in FIRMWARE CODE
s0=80000220 s1=80000290 s2=81fffe98 s3=0000e300 s4=0000e300 s5=83011664
s6=00000010 fp=00000000
4) fn=80101b4c ra=801021f4 sp=804f6d9c in FIRMWARE CODE
s3=80002800 s4=80000654 s5=8000056c s6=06045025 fp=74000640
5) fn=800041d4 ra=80004444 sp=804f6e0c in TASK ROOT *end of trace*

** TAP list **
0] 827e8000 -> 827ed7e0 (827ef5e0) SecCache (UK) v0.4
1] 829d0000 -> 829d643c (829d8b80) XStartTap 0.9R
2] 829e8000 -> 829f01d8 (82a057a8) Font Manager 1.0
3] 82a08000 -> 82a160f8 (82a1a2e0) Extend v1.5
4] 82a40000 -> 82a584b0 (82a67b88) mei2archive BETA 3.8
5] 82a78000 -> 82a90850 (82ad0e38) eit2mei BETA 7.8m5
6] 82ad8000 -> 82b9fc48 (82bb1320) MyStuff 5.62c
7] 82bb8000 -> 82bc3310 (82bda968) TF5000 Display v1.49
8] 83010000 -> 83016368 (83018b40) CrashTrace v0.2
9] 82d38000 -> 82d3957c (82d39ae4) CrashTest

** Memory **
free=16,055,040 avail=13,667,840

_________________
Troubleshooting -- User Manual -- Dark Side of the Matrix: Firmwares and Patches

neilpercy

Posted: Tue Nov 25, 2008 12:00 am

Frequent contributor Joined: 17 Oct 2006 Posts: 224

I only feel qualified to say thanks for working on all this stuff.

I've no doubt it will come in very useful to you (and ultimately the general Toppy community) in the future for solving problems with all the other marvellous work on patches/TAPS that you do.

As always, many thanks for alll your efforts.

neil

_________________
TF5800 456, TS On, F/W: MS6 Recommended F/W 12/9/2009 -FmSyXp+BmEzPfR3WfZ
TAPs: (EIT Sub v0.6 SnG); Goldfish 0.5; MyStuff 6.4; Tap Launcher 3.10; SecCache (UK) v0.4; EPG2MEI v0.96; TAP Commander 1.34; Extend v1.7; Skip 1.34a (TF5800PVR); MHEG On/Off A3; mei2archive BETA 3.8l7; MyInfo B5.6; FFWD v0.7; TSSaver v0.5;
Sig generated by MyInfo? on 14/10/11

shinyford

Posted: Wed Nov 26, 2008 11:58 am

Frequent contributor Joined: 12 Dec 2005 Posts: 726 Location: London

This looks fantastically useful. Many thanks, R2!

Cheers

Nic

_________________
iGuide (EPG, now-and-next, series links) and iGuideSMS (sms connectivity for the Toppy) TAPs. Latest manual, alpha spike and beta

bdb

Posted: Wed Nov 26, 2008 9:57 pm

Frequent contributor Joined: 18 Oct 2005 Posts: 499

looks handy - any chance of the source?

bdb

R2-D2

Posted: Wed Nov 26, 2008 10:05 pm

Frequent contributor Joined: 18 Dec 2006 Posts: 12148

bdb wrote:

any chance of the source?

Source is available to those who want it (PM me), but it's not supplied by default because (like many of my recent TAPs) it requires my modified compiler in order to work properly (which basically gets rid of the $gp "problem") -- just using the normal compiler will make a (sometimes subtle) mess.

_________________
Troubleshooting -- User Manual -- Dark Side of the Matrix: Firmwares and Patches

R2-D2

Posted: Thu Jan 15, 2009 8:00 pm

Frequent contributor Joined: 18 Dec 2006 Posts: 12148

Tiny update which is probably of very little interest -- I noticed that I'd put the return address for leaf functions in the wrong place and so if a crash happened there it was actually being listed as 0.

_________________
Troubleshooting -- User Manual -- Dark Side of the Matrix: Firmwares and Patches

R2-D2

Posted: Mon Feb 02, 2009 9:57 pm

Frequent contributor Joined: 18 Dec 2006 Posts: 12148

Another small update -- some exhaustive testing [ahem...] showed that sometimes the EPC might be trashed but $ra might be OK, so there are additional memory dumps for around $ra (and associated stack, if possible). Plus, it became clear that it would be helpful if the trace back showed offsets into the TAP code to make it a little easier to find the crash point (and function) in the TAP disassembly and/or .map file.

_________________
Troubleshooting -- User Manual -- Dark Side of the Matrix: Firmwares and Patches

shinyford

Posted: Tue May 18, 2010 11:02 am

Frequent contributor Joined: 12 Dec 2005 Posts: 726 Location: London

Hullo again Mr D2, and all - revisiting this after many months.

I use this all the time, and very useful it is too. However, what would be even more useful would be a way of having some of the crash information available after a reboot. (Without leaving my PC on and attached to the serial port 24hours a day, which my wife objects to.)

Is there any way this TAP could be adapted so that, in addition to what it already does, it'd stick a small piece of information into Flash memory? For example, the ID of the running TAP at the time of the crash, and the EPC (and its contents?) at that point. So, two dwords of information that could be extracted by another TAP and pushed into a log file on reboot.

I note that, according to the TF Internal Info pdf, there's an unused Flash area between 0x000269A0 and 0x7FFFFFF. This seems quite large, so I was wondering if it was addressable through the shadow, and writable?

If not, could one or two of the standard variables be subverted? I'd personally be happy for SYSVAR_SubLan and SYSVAR_AudLan to be bent to my evil will, for the purposes of debugging, so long as there was an autostart TAP (which I'd be happy to write) which logged whatever values they held and reverted them to English.

(I'd be happy to rework this TAP myself, given the source code - but your modified compiler dependencies worry me a little.)

Anyway, I'd be interested in any insights into this idea.

Cheers

Nic

_________________
iGuide (EPG, now-and-next, series links) and iGuideSMS (sms connectivity for the Toppy) TAPs. Latest manual, alpha spike and beta

R2-D2

Posted: Tue May 18, 2010 12:59 pm

Frequent contributor Joined: 18 Dec 2006 Posts: 12148

I've shied away from doing anything more complicated than just dumping stuff on the serial port for a simple reason: the Toppy has crashed and so we can't be sure what state the stack and other important bits of the system are in. There's already the possibility that the output from CrashTrace can result in an endless loop of crashes (until you power off) or worse, but that's only marginally more likely than the same happening via the normal exception dump. If attempts were made to log to disk or flash then there's increased likelihood of significant corruption. I believe that the last time this was mooted, FireBird made the good suggestion of reserving and using a known, fixed location on disk -- this would alleviate some of the issues but still requires things like the disk subsystem to be in a reasonable state.

When I was running FpLogger and needed constant logs, I had the Toppy's serial port connected constantly to my Asus (via a serial<->USB adapter). That connection is now promoted out to the network using ser2net, so I can monitor on a PC, too.

_________________
Troubleshooting -- User Manual -- Dark Side of the Matrix: Firmwares and Patches

shinyford

Posted: Tue May 18, 2010 7:15 pm

Frequent contributor Joined: 12 Dec 2005 Posts: 726 Location: London

Problem is, for that I'd need an Asus Smile

(Probably time I bit the bullet and bought one.)

In the meantime... is there a version of this TAP that will compile with the standard compiler? If so, I'm willing to take on the job of adding such functionality in, and taking the risk of using it.

Cheers

Nic

_________________
iGuide (EPG, now-and-next, series links) and iGuideSMS (sms connectivity for the Toppy) TAPs. Latest manual, alpha spike and beta

R2-D2

Posted: Tue May 18, 2010 8:30 pm

Frequent contributor Joined: 18 Dec 2006 Posts: 12148

shinyford wrote:

is there a version of this TAP that will compile with the standard compiler?

No, but if you really want the pain of coping with the $gp issue you can have a crack at adapting the code. (You might, like me, find it much simpler to build your own version of the compiler without the issue!)

_________________
Troubleshooting -- User Manual -- Dark Side of the Matrix: Firmwares and Patches

ccs

Posted: Thu Jun 17, 2010 7:18 pm

Frequent contributor Joined: 30 Oct 2007 Posts: 2584

I'm looking to setup crashtrace to catch a very infrequent reboot/VF&F when the signal fails on a channel being recorded.

I don't fancy moving my PC to another room everyday for the next few months, so.....

How long can a serial-usb cable be? Is 40ft too much?

And while I'm asking, what usb monitor should I be using? I've searched the forum without success.

_________________
TF5810, F/W: MS6 Recommended F/W 12/9/2009 -FmXl+CtEzIScVdZ
TAPs: EIT Sub v0.6; EPG2MEI v0.96; MPDisplayLITE V1.2; MyInfo B5.6; SecCache (UK) v0.4; Extend v1.7; MyStuff 6.6;
Sig generated by MyInfo on 20/10/14 ccsx

alan_m

Posted: Thu Jun 17, 2010 8:06 pm

Frequent contributor Joined: 18 Oct 2006 Posts: 3496

ccs wrote:

How long can a serial-usb cable be? Is 40ft too much?

USB is 5 metres but only with a good quality cable. Anything longer requires an active repeater per 5m maximum. 40 ft would be too much.

BUT

I suspect that the facility outputs on the Toppy's RS232 serial port which I believe defaults to 115,200 baud, 8-bit, no parity, 1 stop bit.

At 19,200 baud you may get 50ft BUT again only with a good quality cable where the capacitance is a lot less than 50pF/foot. At 115,200 you stand no chance. If the facility sets the baud rate to something like 9600 baud you may may stand a chance.

Quote:

And while I'm asking, what usb monitor should I be using? I've searched the forum without success.

Hyperterminal? Part of some(all?) versions of Windoze. Look in Accessories -> Communications.

_________________
Ex Toppy 5800 user - now migrated to Extrend ET10000 Enigma 2 box with 2 terrestrial and 2 satellite tuners

ccs

Posted: Thu Jun 17, 2010 9:20 pm

Frequent contributor Joined: 30 Oct 2007 Posts: 2584

Thanks for your help - much appreciated.

Looks like I'll try and get a very old laptap working again - Hyperterminal isn't part of vista anymore so windows 2000 might be a bonus!

All times are GMT + 1 Hour

View next topic View previous topic Page 1 of 2 Goto page 1, 2 Next Toppy.org.uk Forum Index ~ TAP and patch development

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum