Click here to go to the forum index Click here for the home page
 
Author Message

<  TAP and patch development  ~  How to save or access mheg5/mhp DSMCC files ?

Page 1 of 4
Goto page 1, 2, 3, 4  Next
fblasot
Posted: Wed Dec 09, 2009 12:53 am Reply with quote
Regular contributor Joined: 18 Mar 2008 Posts: 99 Location: Turin, Italy
Hi,
I'm using a TF5800 in Italy with fw 5.13.65B patched.

Here we have, as interactive service, the MHP system and, of course, it is not working on toppy (no red button...) since it is different from UK MHEG5.
I don't need interactive service but specific files transmitted in the DSMCC object carousel.
We have a broadcaster that transmit full EPG data (for 7 days) but not using DVB standard; STB compatible with that EPG have a specific software that reads data (they are transmitted zipped so in a couple of minutes are available) and show them.
Using DVBStreamExplorer I already get that data files.

I'm a novice in TAP programmer but I have already written 2 (simple) taps (I have a good experience with delphi programming).

My idea is to write a TSR Tap that, when GUIDE key on rc is pressed, and only if it is the 1st time pressed, it should:
a) save the current channel (TAP_Channel_GetCurrent)
b) switch to a specific channel (TAP_Channel_Start), dialog box showing "Updating EPG", maybe with no video and audio
c) leave the system collect all file transmitted (better should be filtered for CarouselId that I need) in DSMCC carousel up to when after 10 seconds of no new files added, and save them to disk
d) reads files (for specific CarouselId) and fill the epg standard (TAP_PutEvent)
e) switch back to saved channel
f) return to system as key GUIDE not processed so the original EPG guide grid is showned.

I have no idea how to do the c) point...

Maybe I can write a TAP that hook the original process that handle MHEG5/DSMCC ?
I now that in Firebird lib there are function used to set/unset hook. Are existing source example how to hook ?

Any suggestion will be much appreciated !

_________________
TF5800 Firmware 5.13.65B Patch
Taps: TAPCommander v1.34, QuickJump v1.72, RecCopy v4.5a, SDS 1.3e, HDFW v2.4, ChkEncryptedRecs v0.1 (my 1st TAP !), FixEITPremium v0.1 (my 2nd TAP !)
View user's profile Send private message
R2-D2
Posted: Wed Dec 09, 2009 10:21 am Reply with quote
Frequent contributor Joined: 18 Dec 2006 Posts: 12148
bdb is the expert on this, I think. His epg_extender has code to extract 4TV EPG data which, from what you've described, is broadcast in a similar way.

_________________
Troubleshooting -- User Manual -- Dark Side of the Matrix: Firmwares and Patches
View user's profile Send private message Visit poster's website
fblasot
Posted: Wed Dec 09, 2009 1:56 pm Reply with quote
Regular contributor Joined: 18 Mar 2008 Posts: 99 Location: Turin, Italy
Hi R2-D2,

I already read posts by bdb and I agree that he should be the expert.

To start I will take a look at his "epg extender".

_________________
TF5800 Firmware 5.13.65B Patch
Taps: TAPCommander v1.34, QuickJump v1.72, RecCopy v4.5a, SDS 1.3e, HDFW v2.4, ChkEncryptedRecs v0.1 (my 1st TAP !), FixEITPremium v0.1 (my 2nd TAP !)
View user's profile Send private message
fblasot
Posted: Wed Dec 09, 2009 3:52 pm Reply with quote
Regular contributor Joined: 18 Mar 2008 Posts: 99 Location: Turin, Italy
R2-D2,

I studied "epg extender" and understood how to hook original fw function, it seems easy, apart to found the correct address for newer fw !

Source is a little bit old and refer only up to 5.12.88 (the 0x1288 value in code...), right ?

Code:

type_hook dsmcc_hook_patch[] = {
// sysid, firmware, enable, address,    length, original_data
    {456, 0x1209,   1,      0x801aacfc, 8,      {0x27, 0xbd, 0xff, 0xe0,  0xaf, 0xb6, 0x00, 0x18, }, },  //process_dsmcc
    {456, 0x1225,   1,      0x801ab730, 8,      {0x27, 0xbd, 0xff, 0x78,  0xaf, 0xb6, 0x00, 0x80, }, },  //process_dsmcc
    {456, 0x1288,   1,      0x801b3124, 8,      {0x27, 0xbd, 0xff, 0x78,  0xaf, 0xb6, 0x00, 0x80, }, },  //process_dsmcc
};


Another things that can be usefull for my FixEITPremium is to hook the function that receive EIT data...

Code:

type_hook eit_hook_patch[] = {
// sysid, firmware, enable, address,    length, original_data
    {456, 0x1209,   1,      0x8014ed60, 8,      {0x27, 0xbd, 0xff, 0x28,  0xaf, 0xb0, 0x00, 0xb8, }, },  //process_eit
    {456, 0x1225,   1,      0x8014fc2c, 8,      {0x27, 0xbd, 0xff, 0x28,  0xaf, 0xb0, 0x00, 0xb8, }, },  //process_eit
    {456, 0x1288,   1,      0x80155194, 8,      {0x27, 0xbd, 0xff, 0x18,  0xaf, 0xb0, 0x00, 0xc8, }, },  //process_eit
};


Is it easy for you (or who reads...) found the necessary information for fw 5.13.65B so I can try something ?

Thanks.

EDIT: I found the address (in a file of exTap\patches.c) but still missing "original data"...
Sure I have to unpack firmware and go to that address, right ?

EDIT bis: I use TFDPack-GUI to unpack 5.13.65B (block=32762 and ID=456) and I got UNPACKED.ASM.
Then I edit it and I went to address 0x1b67fc (should be the process_dsmcc's address from esTap\patches.c for fw 5.13.65) and found the bytes 0x27 0xBD 0xFF 0x78 0xAF 0xB6 0x00 0x80.

Is this way correct ?

Edit ter: (sorry...)
Hook procedure seems more easy with FireBirdLib (that I already used for my taps...), it needs only the address of the original fw function, right ?

_________________
TF5800 Firmware 5.13.65B Patch
Taps: TAPCommander v1.34, QuickJump v1.72, RecCopy v4.5a, SDS 1.3e, HDFW v2.4, ChkEncryptedRecs v0.1 (my 1st TAP !), FixEITPremium v0.1 (my 2nd TAP !)
View user's profile Send private message
R2-D2
Posted: Wed Dec 09, 2009 8:51 pm Reply with quote
Frequent contributor Joined: 18 Dec 2006 Posts: 12148
fblasot wrote:
Source is a little bit old and refer only up to 5.12.88 (the 0x1288 value in code...), right ?
Ah yes, bdb's exTAP library is built for certain firmwares I think -- there may be a version that supports 5.13.65 around, or you might be able to build one yourself. I think the key bit you're looking for are the signatures that are used to find the firmware routines... something like "generate_patches". You should be able to easily adapt those signatures to something more generic using FireBirdLib's FindInstructionSequence() to dynamically locate the code.

_________________
Troubleshooting -- User Manual -- Dark Side of the Matrix: Firmwares and Patches
View user's profile Send private message Visit poster's website
fblasot
Posted: Wed Dec 09, 2009 10:24 pm Reply with quote
Regular contributor Joined: 18 Mar 2008 Posts: 99 Location: Turin, Italy
I created a small tap to set/unset a fw hook (using FireBird lib) for Process_DSMCC but my hook procedure is never called; Of course interactive service are enabled...

So I try my small tap with address of original EIT process and my hook is called regularly (of course it does nothing and then EPG is empty...).

I think that the original Process_DSMCC is called only if the stream is recognized as MHEG5...

For me is still to early to go with disassembler to find and patch when Process_DSMCC is called...
R2-D2 could you help me ? Of course if it takes small time to do...

_________________
TF5800 Firmware 5.13.65B Patch
Taps: TAPCommander v1.34, QuickJump v1.72, RecCopy v4.5a, SDS 1.3e, HDFW v2.4, ChkEncryptedRecs v0.1 (my 1st TAP !), FixEITPremium v0.1 (my 2nd TAP !)
View user's profile Send private message
bdb
Posted: Wed Dec 09, 2009 10:54 pm Reply with quote
Frequent contributor Joined: 18 Oct 2005 Posts: 499
Quote:
EDIT bis: I use TFDPack-GUI to unpack 5.13.65B (block=32762 and ID=456) and I got UNPACKED.ASM.
Then I edit it and I went to address 0x1b67fc (should be the process_dsmcc's address from esTap\patches.c for fw 5.13.65) and found the bytes 0x27 0xBD 0xFF 0x78 0xAF 0xB6 0x00 0x80.

Is this way correct ?

yes;
for 13.65, process_dsmcc = 801B67FC
process_eit = 80157e74

epg_extender has never been updated to support later exTaps, so you will need to add additional lines containing the addresses to the structures you have identified.

Quote:
Maybe I can write a TAP that hook the original process that handle MHEG5/DSMCC ?

The 'process_dsmcc' hook captures the data as it is passed into the toppy firmware from the hardware section filter.
It should be possible to capture the data a little later, once the toppy firmware has extracted it; but all the data structiures are dynamically malloed, with lots of linked lists - so very difficult to
find a suitable place to intercept it.

The epg_extender code will give you a file containing the raw section data (see here for a sample).
To get the epg data you will need to process the dsmcc stack. If the data you are after is carried with just the thin dsmcc headers, then exracting the EIT section data is easy; just delete them.
However most data is burried beneath many more layers, including compresion and a filing system. This is less trivial to extract.

I never bothered getting any code on the toppy to do this; just exported the raw files back to a pc for decoding.

bdb


Quote:
I think that the original Process_DSMCC is called only if the stream is recognized as MHEG5...

possibly ... the firmware may analyse the pmt, and only enable the section filters for the streams that it understands; it may depend on how the MHP is marked. Adding additional section filters is more painful than hooking an existing one.
View user's profile Send private message
fblasot
Posted: Wed Dec 09, 2009 11:14 pm Reply with quote
Regular contributor Joined: 18 Mar 2008 Posts: 99 Location: Turin, Italy
Hi bdb, happy to read you...

So you think that a small patch where original Process_DSMCC is called is not enough ?
I'm just installing IDA...

I know that MHEG5 and MHP are different but the caurosel DMSCC should be a lot similar, from documents I've found in internet seem that MHP is more restrictive (it allow only 2 section per TS packet while MHEG5 is 4...).

I know you wrote Record-A-Mux tap, in a way similar I could record the whole mux (or better only a part since really I would need only a specific service) and then process looking what I need...


EDIT: I load the decompressed fw in IDA at 0x80000000 with MIPSB big endian. Is it correct ?
Which is the first entry point for execution ?

_________________
TF5800 Firmware 5.13.65B Patch
Taps: TAPCommander v1.34, QuickJump v1.72, RecCopy v4.5a, SDS 1.3e, HDFW v2.4, ChkEncryptedRecs v0.1 (my 1st TAP !), FixEITPremium v0.1 (my 2nd TAP !)
View user's profile Send private message
bdb
Posted: Wed Dec 09, 2009 11:41 pm Reply with quote
Frequent contributor Joined: 18 Oct 2005 Posts: 499
if you are not getting any calls to 'process_dsmcc'; then no.
A quick look at the firmware suggests it checks the pmt for a 'data broadcast id descriptor' (0x66)
If this is 0x106, or 0x111 (I think these are used by the UK MHEG, and for the OTA software downloads) it extracts the relevent pids and adds a section filter.
There is also a reference to a 'MHP Object Carousel data_broadcast_id' (0xf0) - not sure why.

It shouldn't be too hard to patch this bit of code to work with your stream.

Once you know which pids to use, you can save the raw TS data to disk, then read back the file (or intercept it on the way to disk [aka ad_skipper...]).
You would then need to do you own TS to section reconstruction.

If you could extract + post some raw PMT TS packets for a relevent channel (30 seconds of a normal .rec file should be fine), I can see how the MHP is signalled.

bdb
View user's profile Send private message
fblasot
Posted: Wed Dec 09, 2009 11:58 pm Reply with quote
Regular contributor Joined: 18 Mar 2008 Posts: 99 Location: Turin, Italy
Quote:
It shouldn't be too hard to patch this bit of code to work with your stream.


very happy...

I'm uploading a .rec file of 1 minute, if less toppy doesn't saves the file...

EDIT: mmm, I think I'm wrong..you need a full transport stream not done with toppy, right ?

This is a normal rec done with toppy http://rapidshare.com/files/318678956/test.zip.html

while this is a TS file filtered with only necessary PIDs http://rapidshare.com/files/318683986/TS_test.zip.html


Last edited by fblasot on Thu Dec 10, 2009 12:29 am; edited 2 times in total

_________________
TF5800 Firmware 5.13.65B Patch
Taps: TAPCommander v1.34, QuickJump v1.72, RecCopy v4.5a, SDS 1.3e, HDFW v2.4, ChkEncryptedRecs v0.1 (my 1st TAP !), FixEITPremium v0.1 (my 2nd TAP !)
View user's profile Send private message
bdb
Posted: Thu Dec 10, 2009 12:16 am Reply with quote
Frequent contributor Joined: 18 Oct 2005 Posts: 499
I _think_ that the .rec files contain the pmt
View user's profile Send private message
fblasot
Posted: Thu Dec 10, 2009 12:24 am Reply with quote
Regular contributor Joined: 18 Mar 2008 Posts: 99 Location: Turin, Italy
Quote:
There is also a reference to a 'MHP Object Carousel data_broadcast_id' (0xf0) - not sure why


That is what I see with DVBStreamExplorer !

Quote:
If this is 0x106, or 0x111


That refer to what ?
I don't think to 'data broadcast id descriptor' since you said is 0x66...
Maybe are ELEMENTARY PID ?


PS: I updated previus post with link to a TS rec file...

_________________
TF5800 Firmware 5.13.65B Patch
Taps: TAPCommander v1.34, QuickJump v1.72, RecCopy v4.5a, SDS 1.3e, HDFW v2.4, ChkEncryptedRecs v0.1 (my 1st TAP !), FixEITPremium v0.1 (my 2nd TAP !)
View user's profile Send private message
bdb
Posted: Thu Dec 10, 2009 1:19 am Reply with quote
Frequent contributor Joined: 18 Oct 2005 Posts: 499
The data_broadcast_id indicates what is in the dsmcc stream.
0x0106 means a UK MHEG service
0x0111 means an OTA download
0x00f0 means some MHP

The toppy firmware only enables the section filter for the MHEG/OTA streams.

try this (13.65 firmware):
it will force it to accept 0x00f0 rather than 0x106
- make sure you have 'interactive services enabled
- change channels to make sure it wakes up

*(unsigned long *)0x801B58DC = 0x241900f0; //was 0x24190106


to restore, reboot, or:
*(unsigned long *)0x801B58DC = 0x24190106;


bdb
View user's profile Send private message
fblasot
Posted: Thu Dec 10, 2009 9:27 am Reply with quote
Regular contributor Joined: 18 Mar 2008 Posts: 99 Location: Turin, Italy
thanks bdb,
this evening I will try...

_________________
TF5800 Firmware 5.13.65B Patch
Taps: TAPCommander v1.34, QuickJump v1.72, RecCopy v4.5a, SDS 1.3e, HDFW v2.4, ChkEncryptedRecs v0.1 (my 1st TAP !), FixEITPremium v0.1 (my 2nd TAP !)
View user's profile Send private message
fblasot
Posted: Thu Dec 10, 2009 10:24 pm Reply with quote
Regular contributor Joined: 18 Mar 2008 Posts: 99 Location: Turin, Italy
bdb,

thanks for your help, all is working right, data from DSMCC are saved.

Next step now is extract what I need from the saved file and for that I will also study dsmcc_stuff.zip from your web page.

_________________
TF5800 Firmware 5.13.65B Patch
Taps: TAPCommander v1.34, QuickJump v1.72, RecCopy v4.5a, SDS 1.3e, HDFW v2.4, ChkEncryptedRecs v0.1 (my 1st TAP !), FixEITPremium v0.1 (my 2nd TAP !)
View user's profile Send private message

Display posts from previous:  

All times are GMT + 1 Hour
Page 1 of 4
Goto page 1, 2, 3, 4  Next

Jump to:  

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum