Click here to go to the forum index Click here for the home page
 
Author Message

<  TAP and patch development  ~  CrashTrace -- better debug info

Page 1 of 2
Goto page 1, 2  Next
R2-D2
Posted: Mon Nov 24, 2008 3:45 pm Reply with quote
Frequent contributor Joined: 18 Dec 2006 Posts: 12148
CrashTrace is a first attempt at providing some more info on crashes on the serial port. I've tried to make it quite defensive, but if something does go wrong at this level you'll probably get a never-ending cycle of exceptions (until you press the power button and the Front Panel shuts it down).

Extra info includes a couple of short memory dumps around the exception code and stack points, a trace back of function calls that lead to the exception, and a list of current TAP locations. There's also a simple CrashTest TAP which will cause a crash when the 0 button is pressed (or press Exit to quit it). The initial register dump and the trace back should show the values assigned (explicitly) to $s1, as an indication of how this new information might be useful.

_________________
Troubleshooting -- User Manual -- Dark Side of the Matrix: Firmwares and Patches
View user's profile Send private message Visit poster's website
R2-D2
Posted: Mon Nov 24, 2008 11:22 pm Reply with quote
Frequent contributor Joined: 18 Dec 2006 Posts: 12148
V0.2 adds a memory usage dump, reformats things a bit and adds a little more info. Sample output is now:
Code:
COMMAND> exec crashtracec.tap
CrashTrace: hook added @ 000010a4
COMMAND> exec crashtest.tap
CrashTest: start @ 00001339
CrashTest: crashing @ 0000170b
**CRASH**

$sp=804f6c98
$at=00000000    $v0=804dc8f8    $v1=00000078    $a0=0000000a
$a1=82d39580    $a2=804f6d18    $a3=0000000f    $t0=80256894
$t1=00000000    $t2=82bc9288    $t3=804f640c    $t4=80425e5c
$t5=00000000    $t6=804f6c0e    $t7=0000000a    $t8=0000000d
$t9=80179628    $s0=00010000    $s1=12cafe34    $s2=82d39602
$s3=0000e300    $s4=00000009    $s5=80179628    $s6=00010000
$s7=82d39602    $s8=804f6dc8    $ra=82d38710    $t0=00000000
$t1=00000000    $gp=82d39602    $k1=82d38654


<<< Unexpected Exception >>>
CAUSE   = 0000040c
  EPC = 82d38654
 *EPC   = ac000000
   SR   = 10000403
badVAddr = 00000000
 task = 0
_runCount=1
lastTask(0) = 0

Dumping bytes from 82d385d0 for EPC=82d38654
82d385d0:  00000000  00000000  00000000  00000000
82d385e0:  00000000  00000000  00000000  00000000
82d385f0:  00000000  00000000  00000000  00000000
82d38600:  27bdffe0  afbf0018  afb70014  afb10010
82d38610:  04100001  00000000  3c170000  26f70fea
82d38620:  02ffb821  3c1112ca  3631fe34  8f8201c6
82d38630:  00000000  8c420000  8ee40016  00000000
82d38640:  24841580  0040c821  0320f809  00000000
82d38650:  8fbf0018 *ac000000* 8fb70014  8fb10010
82d38660:  03e00008  27bd0020  27bdffe0  afbf001c
82d38670:  afb70018  afb10014  afb00010  04100001
82d38680:  00000000  3c170000  26f70f7e  02ffb821
82d38690:  00a08021  3084ffff  24020100  1482003b
82d386a0:  00000000  3c020001  1602001b  00000000
82d386b0:  3c11dead  3631beef  8f82002e  00000000
82d386c0:  8c420000  00000000  0040c821  0320f809
82d386d0:  00000000  8ee40016  00000000  2484158c

Dumping bytes from 804f6c90 for fn SP=804f6d14
804f6c90:  00010000  804f6dc8  0000040c  804dc8f8
804f6ca0:  00000078  0000000a  82d39580  804f6d18
804f6cb0:  0000000f  80256894  00000000  82bc9288
804f6cc0:  804f640c  80425e5c  00000000  804f6c0e
804f6cd0:  0000000a  0000000d  80179628  00010000
804f6ce0:  12cafe34  82d39602  0000e300  00000009
804f6cf0:  80179628  00010000  82d39602  804f6dc8
804f6d00:  82d38710  00000000  00000000  82d39602
804f6d10:  82d38654 *82d39580* 82d3958c  00000004
804f6d20:  0000000f  deadbeef  82d39602  82d38710
804f6d30:  804f6dc8  82d3958c  0000170b  00010000
804f6d40:  00000003  82d38668  00010000  8025e6d8
804f6d50:  8013324c  801a2a9c  00010000  0000e300
804f6d60:  83011664  00000010  804f6dcc  804dc8f8
804f6d70:  80005f84  00010010  801021f4  80000220
804f6d80:  80000290  81fffe98  0000e300  0000e300
804f6d90:  83011664  00000010  00000000  81fffe98

** Stack back trace @ 22:07 (uptime: 0d 0h 0m 58s) **
1) fn=82d38600 ra=82d38654 sp=804f6d14 in TAP @ 82d38000 CrashTest
     s1=deadbeef s7=82d39602
2) fn=82d38668 ra=82d38710 sp=804f6d34 in TAP @ 82d38000 CrashTest
     s0=82d38668 s1=00010000 s7=8025e6d8
3) fn=80132eb4 ra=8013324c sp=804f6d54 in FIRMWARE CODE
     s0=80000220 s1=80000290 s2=81fffe98 s3=0000e300 s4=0000e300 s5=83011664
     s6=00000010 fp=00000000
4) fn=80101b4c ra=801021f4 sp=804f6d9c in FIRMWARE CODE
     s3=80002800 s4=80000654 s5=8000056c s6=06045025 fp=74000640
5) fn=800041d4 ra=80004444 sp=804f6e0c in TASK ROOT *end of trace*

** TAP list **
 0] 827e8000 -> 827ed7e0 (827ef5e0) SecCache (UK) v0.4
 1] 829d0000 -> 829d643c (829d8b80) XStartTap 0.9R
 2] 829e8000 -> 829f01d8 (82a057a8) Font Manager 1.0
 3] 82a08000 -> 82a160f8 (82a1a2e0) Extend v1.5
 4] 82a40000 -> 82a584b0 (82a67b88) mei2archive BETA 3.8
 5] 82a78000 -> 82a90850 (82ad0e38) eit2mei BETA 7.8m5
 6] 82ad8000 -> 82b9fc48 (82bb1320) MyStuff 5.62c
 7] 82bb8000 -> 82bc3310 (82bda968) TF5000 Display v1.49
 8] 83010000 -> 83016368 (83018b40) CrashTrace v0.2
 9] 82d38000 -> 82d3957c (82d39ae4) CrashTest

** Memory **
free=16,055,040 avail=13,667,840

_________________
Troubleshooting -- User Manual -- Dark Side of the Matrix: Firmwares and Patches
View user's profile Send private message Visit poster's website
neilpercy
Posted: Tue Nov 25, 2008 12:00 am Reply with quote
Frequent contributor Joined: 17 Oct 2006 Posts: 224
I only feel qualified to say thanks for working on all this stuff.

I've no doubt it will come in very useful to you (and ultimately the general Toppy community) in the future for solving problems with all the other marvellous work on patches/TAPS that you do.

As always, many thanks for alll your efforts.




neil

_________________
TF5800 456, TS On, F/W: MS6 Recommended F/W 12/9/2009 -FmSyXp+BmEzPfR3WfZ
TAPs: (EIT Sub v0.6 SnG); Goldfish 0.5; MyStuff 6.4; Tap Launcher 3.10; SecCache (UK) v0.4; EPG2MEI v0.96; TAP Commander 1.34; Extend v1.7; Skip 1.34a (TF5800PVR); MHEG On/Off A3; mei2archive BETA 3.8l7; MyInfo B5.6; FFWD v0.7; TSSaver v0.5;
Sig generated by MyInfo? on 14/10/11
View user's profile Send private message
shinyford
Posted: Wed Nov 26, 2008 11:58 am Reply with quote
Frequent contributor Joined: 12 Dec 2005 Posts: 726 Location: London
This looks fantastically useful. Many thanks, R2!

Cheers

Nic

_________________
iGuide (EPG, now-and-next, series links) and iGuideSMS (sms connectivity for the Toppy) TAPs. Latest manual, alpha spike and beta
View user's profile Send private message Visit poster's website
bdb
Posted: Wed Nov 26, 2008 9:57 pm Reply with quote
Frequent contributor Joined: 18 Oct 2005 Posts: 499
looks handy - any chance of the source?

bdb
View user's profile Send private message
R2-D2
Posted: Wed Nov 26, 2008 10:05 pm Reply with quote
Frequent contributor Joined: 18 Dec 2006 Posts: 12148
bdb wrote:
any chance of the source?
Source is available to those who want it (PM me), but it's not supplied by default because (like many of my recent TAPs) it requires my modified compiler in order to work properly (which basically gets rid of the $gp "problem") -- just using the normal compiler will make a (sometimes subtle) mess.

_________________
Troubleshooting -- User Manual -- Dark Side of the Matrix: Firmwares and Patches
View user's profile Send private message Visit poster's website
R2-D2
Posted: Thu Jan 15, 2009 8:00 pm Reply with quote
Frequent contributor Joined: 18 Dec 2006 Posts: 12148
Tiny update which is probably of very little interest -- I noticed that I'd put the return address for leaf functions in the wrong place and so if a crash happened there it was actually being listed as 0.

_________________
Troubleshooting -- User Manual -- Dark Side of the Matrix: Firmwares and Patches
View user's profile Send private message Visit poster's website
R2-D2
Posted: Mon Feb 02, 2009 9:57 pm Reply with quote
Frequent contributor Joined: 18 Dec 2006 Posts: 12148
Another small update -- some exhaustive testing [ahem...] showed that sometimes the EPC might be trashed but $ra might be OK, so there are additional memory dumps for around $ra (and associated stack, if possible). Plus, it became clear that it would be helpful if the trace back showed offsets into the TAP code to make it a little easier to find the crash point (and function) in the TAP disassembly and/or .map file.

_________________
Troubleshooting -- User Manual -- Dark Side of the Matrix: Firmwares and Patches
View user's profile Send private message Visit poster's website
shinyford
Posted: Tue May 18, 2010 11:02 am Reply with quote
Frequent contributor Joined: 12 Dec 2005 Posts: 726 Location: London
Hullo again Mr D2, and all - revisiting this after many months.

I use this all the time, and very useful it is too. However, what would be even more useful would be a way of having some of the crash information available after a reboot. (Without leaving my PC on and attached to the serial port 24hours a day, which my wife objects to.)

Is there any way this TAP could be adapted so that, in addition to what it already does, it'd stick a small piece of information into Flash memory? For example, the ID of the running TAP at the time of the crash, and the EPC (and its contents?) at that point. So, two dwords of information that could be extracted by another TAP and pushed into a log file on reboot.

I note that, according to the TF Internal Info pdf, there's an unused Flash area between 0x000269A0 and 0x7FFFFFF. This seems quite large, so I was wondering if it was addressable through the shadow, and writable?

If not, could one or two of the standard variables be subverted? I'd personally be happy for SYSVAR_SubLan and SYSVAR_AudLan to be bent to my evil will, for the purposes of debugging, so long as there was an autostart TAP (which I'd be happy to write) which logged whatever values they held and reverted them to English.

(I'd be happy to rework this TAP myself, given the source code - but your modified compiler dependencies worry me a little.)

Anyway, I'd be interested in any insights into this idea.

Cheers

Nic

_________________
iGuide (EPG, now-and-next, series links) and iGuideSMS (sms connectivity for the Toppy) TAPs. Latest manual, alpha spike and beta
View user's profile Send private message Visit poster's website
R2-D2
Posted: Tue May 18, 2010 12:59 pm Reply with quote
Frequent contributor Joined: 18 Dec 2006 Posts: 12148
I've shied away from doing anything more complicated than just dumping stuff on the serial port for a simple reason: the Toppy has crashed and so we can't be sure what state the stack and other important bits of the system are in. There's already the possibility that the output from CrashTrace can result in an endless loop of crashes (until you power off) or worse, but that's only marginally more likely than the same happening via the normal exception dump. If attempts were made to log to disk or flash then there's increased likelihood of significant corruption. I believe that the last time this was mooted, FireBird made the good suggestion of reserving and using a known, fixed location on disk -- this would alleviate some of the issues but still requires things like the disk subsystem to be in a reasonable state.

When I was running FpLogger and needed constant logs, I had the Toppy's serial port connected constantly to my Asus (via a serial<->USB adapter). That connection is now promoted out to the network using ser2net, so I can monitor on a PC, too.

_________________
Troubleshooting -- User Manual -- Dark Side of the Matrix: Firmwares and Patches
View user's profile Send private message Visit poster's website
shinyford
Posted: Tue May 18, 2010 7:15 pm Reply with quote
Frequent contributor Joined: 12 Dec 2005 Posts: 726 Location: London
Problem is, for that I'd need an Asus Smile (Probably time I bit the bullet and bought one.)

In the meantime... is there a version of this TAP that will compile with the standard compiler? If so, I'm willing to take on the job of adding such functionality in, and taking the risk of using it.

Cheers

Nic

_________________
iGuide (EPG, now-and-next, series links) and iGuideSMS (sms connectivity for the Toppy) TAPs. Latest manual, alpha spike and beta
View user's profile Send private message Visit poster's website
R2-D2
Posted: Tue May 18, 2010 8:30 pm Reply with quote
Frequent contributor Joined: 18 Dec 2006 Posts: 12148
shinyford wrote:
is there a version of this TAP that will compile with the standard compiler?
No, but if you really want the pain of coping with the $gp issue you can have a crack at adapting the code. (You might, like me, find it much simpler to build your own version of the compiler without the issue!)

_________________
Troubleshooting -- User Manual -- Dark Side of the Matrix: Firmwares and Patches
View user's profile Send private message Visit poster's website
ccs
Posted: Thu Jun 17, 2010 7:18 pm Reply with quote
Frequent contributor Joined: 30 Oct 2007 Posts: 2584
I'm looking to setup crashtrace to catch a very infrequent reboot/VF&F when the signal fails on a channel being recorded.

I don't fancy moving my PC to another room everyday for the next few months, so.....

How long can a serial-usb cable be? Is 40ft too much?

And while I'm asking, what usb monitor should I be using? I've searched the forum without success.

_________________
TF5810, F/W: MS6 Recommended F/W 12/9/2009 -FmXl+CtEzIScVdZ
TAPs: EIT Sub v0.6; EPG2MEI v0.96; MPDisplayLITE V1.2; MyInfo B5.6; SecCache (UK) v0.4; Extend v1.7; MyStuff 6.6;
Sig generated by MyInfo on 20/10/14
ccsx
View user's profile Send private message
alan_m
Posted: Thu Jun 17, 2010 8:06 pm Reply with quote
Frequent contributor Joined: 18 Oct 2006 Posts: 3496
ccs wrote:

How long can a serial-usb cable be? Is 40ft too much?


USB is 5 metres but only with a good quality cable. Anything longer requires an active repeater per 5m maximum. 40 ft would be too much.

BUT

I suspect that the facility outputs on the Toppy's RS232 serial port which I believe defaults to 115,200 baud, 8-bit, no parity, 1 stop bit.

At 19,200 baud you may get 50ft BUT again only with a good quality cable where the capacitance is a lot less than 50pF/foot. At 115,200 you stand no chance. If the facility sets the baud rate to something like 9600 baud you may may stand a chance.

Quote:

And while I'm asking, what usb monitor should I be using? I've searched the forum without success.


Hyperterminal? Part of some(all?) versions of Windoze. Look in Accessories -> Communications.

_________________
Ex Toppy 5800 user - now migrated to Extrend ET10000 Enigma 2 box with 2 terrestrial and 2 satellite tuners
View user's profile Send private message
ccs
Posted: Thu Jun 17, 2010 9:20 pm Reply with quote
Frequent contributor Joined: 30 Oct 2007 Posts: 2584
Thanks for your help - much appreciated.

Looks like I'll try and get a very old laptap working again - Hyperterminal isn't part of vista anymore so windows 2000 might be a bonus!

_________________
TF5810, F/W: MS6 Recommended F/W 12/9/2009 -FmXl+CtEzIScVdZ
TAPs: EIT Sub v0.6; EPG2MEI v0.96; MPDisplayLITE V1.2; MyInfo B5.6; SecCache (UK) v0.4; Extend v1.7; MyStuff 6.6;
Sig generated by MyInfo on 20/10/14
ccsx
View user's profile Send private message

Display posts from previous:  

All times are GMT + 1 Hour
Page 1 of 2
Goto page 1, 2  Next

Jump to:  

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum